Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| db25b469ba | |||
| b228b44f0e | |||
| 91b9405c81 | |||
| 4c69c25b48 | |||
| c9013bb47d | |||
| 2d1e5d0774 | |||
| f09c53efc1 | |||
| 92d647a6f6 | |||
| d03ab7f1f5 | |||
| b17d1c597d | |||
| 0ae01dfb52 | |||
| dd7cdc09e7 | |||
| 7b0a8b234b | |||
| ce1a4b4758 | |||
| ac90169a5d | |||
| 154d6dcaa9 | |||
| 2f7072050f | |||
| 85e799f360 |
@@ -23,7 +23,8 @@ unconditional: true
|
|||||||
customer data; anything that changes a promise the product makes to a customer; anything that
|
customer data; anything that changes a promise the product makes to a customer; anything that
|
||||||
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
|
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
|
||||||
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
|
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
|
||||||
catalog version; a new external dependency.
|
catalog version; a new external dependency; **a hub image build or hub deploy in a session the operator does not
|
||||||
|
attend** (operator ruling 2026-10-07, `09` §3 decision 162).
|
||||||
|
|
||||||
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
|
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
|
||||||
|
|
||||||
|
|||||||
+157
-4
@@ -1,4 +1,157 @@
|
|||||||
## Unreleased (2026-10-06 night, later) — after a restart the agent remembers the last backup per tier (R-894); three more SMART counters on the wire (R-330)
|
## Unreleased (2026-10-08) — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane
|
||||||
|
|
||||||
|
**Delivery: the agent binary AND the config bundle** — the wrapper `felhom-os-apply` changed (2026-10-09).
|
||||||
|
|
||||||
|
- **The hub's backup evidence survives a restart (2026-10-09, found in the kernel-night read-back).** The host report's
|
||||||
|
`backups` list came only from memory, which is empty after an agent restart. On demo-felhom the night backup landed
|
||||||
|
02:40 UTC and the kernel step restarted the host at 02:44 — no report fell in between, two nights running — so the hub
|
||||||
|
alarmed „host tier: newest backup is 48h old" at 03:00 after a good backup. The report now adds R-894's saved newest
|
||||||
|
success per tier and guest (`backup-success-state.json`, ONE instance shared with the local API) where memory holds no
|
||||||
|
success at or after it. Only successes are saved, so a failure is never hidden. `hub.KnownBackupReporter`,
|
||||||
|
`BackupSuccessState.KnownBackupSuccesses`; tests `TestCollectBackups_SavedSuccessSurvivesARestart` (red-proved: the
|
||||||
|
restart case reported nothing), `TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen`, and
|
||||||
|
`TestR894_LastKnownBackupsIsWiredIntoTheDaemon` now also asserts the collector gets the same instance (red-proved).
|
||||||
|
- **`proxmox-secure-boot-support` is a boot-chain package (2026-10-09).** On demo-hp (Secure Boot on) its upgrade pulled
|
||||||
|
`shim-signed-common`; in the ring-0 Proxmox plan that refused the whole step (R6) and skipped the night's kernel step,
|
||||||
|
after the household had been told „tonight". It joins `HOST_SLOW_RE` with shim and GRUB (they stay out of every lane,
|
||||||
|
as before). `test_ring0_pending_pve_leaves_the_secure_boot_meta_out` (red-proved).
|
||||||
|
|
||||||
|
- **R-899 (operator ruling 2026-10-08, option A):** `POST /backup?trigger=manual` (a „Mentés most" press, sent by the
|
||||||
|
controller from its next release) runs no OS leg after it: the leg belongs to the night, after the night's own copy.
|
||||||
|
Before, a press ran the leg at once (in the day; on 2026-10-07 08:51 demo-hp skipped it only by the 20-hour rule). A
|
||||||
|
request without the parameter (an older controller, the scheduled path) behaves as before.
|
||||||
|
`internal/localapi/server.go`; `TestAfterPrimaryBackup` gained two sub-cases (red-proved: ignoring `trigger`, the leg
|
||||||
|
ran once after a press).
|
||||||
|
- **R-304 — „wrong code" only when every earlier package was tried.** `POST /escrow/recover-offsite-password` answered
|
||||||
|
400 („the recovery code did not open the sealed bundle") whenever the current package and the TRIED earlier packages
|
||||||
|
refused the code — even when the hub withheld earlier packages (rows with no key material, rows over its serve cap), a
|
||||||
|
served package was malformed, the 6-attempt cap stopped the loop, or the retained list could not be read at all. Now
|
||||||
|
those cases answer **424** with `older_unchecked` (the count, -1 = unknown) and a sentence that does not call the code
|
||||||
|
wrong. `escrow.ErrRetainedUnchecked` / `RetainedUncheckedError`; the retained fetcher's second value is now every
|
||||||
|
withheld package (unopenable + truncated + malformed). Tests `TestR304_*` (real age crypto; red-proved: without the
|
||||||
|
check, four cases returned the wrong-code error) and a 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus`.
|
||||||
|
An older controller maps the unknown 424 to its neutral „we do not know why" sentence.
|
||||||
|
- **The „ring 1" label after a boot:** in the first second after a reboot the agent has not fetched the hub's block, and
|
||||||
|
the after-boot kernel reports (`judging`, `good`, `revert`, `fell_back`…) said ring 1 on a ring-0 box (demo-felhom,
|
||||||
|
2026-10-08 night). Now they read the fetched block, else the block the daemon saved on disk before the reboot (R-866),
|
||||||
|
else ring 1 as before. A label only — the hub's approval reads its own ring list. `kernelReportRing`,
|
||||||
|
`TestKernelReportRing_BeforeFirstFetch` (red-proved: ring 1, want 0).
|
||||||
|
|
||||||
|
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
|
||||||
|
|
||||||
|
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
|
||||||
|
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
|
||||||
|
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
|
||||||
|
|
||||||
|
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
|
||||||
|
|
||||||
|
- `internal/osupdate/kernel.go`: ring 0's night kernel step stages EXACTLY the kernel the household was told about
|
||||||
|
(select `listed`, `KernelSet(kver)` = the series meta-package and the signed image at the kernel's own version) instead
|
||||||
|
of "whatever is pending tonight". Seen 2026-10-07: demo-felhom was told about 7.0.14-20 while its sources offered
|
||||||
|
7.0.14-22 by night — the old code staged `pending-kernel` and the wrapper refused it (R23), losing the night. A told
|
||||||
|
version that is no longer installable is refused by the wrapper before any change (R7) and the hub tells the household
|
||||||
|
again for the newer kernel (hub v0.143.1). Tests `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the
|
||||||
|
old select), `TestKernelSet`; wrapper `test_ring0_listed_installs_the_told_kernel_not_the_newest`,
|
||||||
|
`test_ring0_told_kernel_gone_is_refused_before_any_change`.
|
||||||
|
|
||||||
|
## v0.152.0 — the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
|
||||||
|
|
||||||
|
Released by `scripts/release-agent.sh`: binary sha256 `95ff42208e36ba49b6e2b09a97e81a6fa11562ecc8042f1ed18d378b8b1f88b1`,
|
||||||
|
config bundle sha256 `f0c2cec374b711b3c131c955012b33fd0ad495337d049b7a743c3eef9e85c20b` (tag `v0.152.0` = `d03ab7f`).
|
||||||
|
Step bundle `0.152.0-step1` sha256 `0b71d32b054cf3b7ade0234ffcbb0df159901f542cde540adaee411db466f48e` (the 0.151.0 bundle
|
||||||
|
with only `felhom-os-apply` replaced; `scripts/build-step-bundle.py`), published as package version `0.152.0-step1`.
|
||||||
|
|
||||||
|
**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths
|
||||||
|
(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880).
|
||||||
|
|
||||||
|
A new kernel boots ONCE; if it crashes the box comes back on the old kernel by itself; it becomes the default only after
|
||||||
|
a healthy boot; a booted-but-unhealthy kernel is reverted ONCE by the agent with no person. Built on the spike's
|
||||||
|
candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot entry.
|
||||||
|
|
||||||
|
- `configs/felhom-grub-oneshot.sh` → `/etc/grub.d/01_felhom_oneshot` (bundle): reads `felhom_next` from a GRUB env block
|
||||||
|
on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and sets the default to that kernel's
|
||||||
|
one-shot entry only when the name is an installed kernel. No vfat ESP → prints nothing.
|
||||||
|
- `configs/felhom-grub-oneshot-entries.sh` → `/etc/grub.d/42_felhom_oneshot` (bundle): one entry per installed kernel,
|
||||||
|
id `felhom-oneshot-<ver>`, the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
|
||||||
|
(option C). Sorted after `10_linux`: never entry 0, never the default.
|
||||||
|
- `configs/felhom-os-apply`: layer `kernel` (lane slow; an appliance; authority = a signed `os_kernel_step` or the
|
||||||
|
root-owned ring-0 mark). Modes: `apply` STAGES (select `pending-kernel` or a signed `listed` set; `expect_kver` = the
|
||||||
|
kernel the household was told about): pins the GRUB default to the RUNNING kernel in
|
||||||
|
`/etc/default/grub.d/zz-felhom-kernel-default.cfg` and proves it from grub.cfg, installs, proves the default did not
|
||||||
|
move and the one-shot entry exists, writes the flag; never reboots. `kernel-reboot` (a staged step only),
|
||||||
|
`kernel-boot` (judging | fell_back | self_reverted | revert_failed), `kernel-good` (the new kernel becomes the
|
||||||
|
default, proved), `kernel-revert` (ONE per step; refused when the default is not the old kernel), `kernel-cancel`,
|
||||||
|
`kernel-status`. New refusals: R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB
|
||||||
|
without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or an unclean boot in its
|
||||||
|
window), R22 (the phase does not allow the mode; never two steps within 20 h), R23 (not exactly one newer kernel, or
|
||||||
|
not the one signed / told). State `/var/lib/felhom-kernel/state.json`. Facts carry `kernel_lane`; the next-boot
|
||||||
|
kernel reads the flag and the grub.cfg default. Tests: `KernelLane` (27), red-proof
|
||||||
|
`felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
|
||||||
|
- `configs/felhom-crash-guard` unchanged; `KernelStepCannotLeaveTheBoxOff` (3 tests) pins that a step's planned reboot,
|
||||||
|
one crash and one self-revert add ONE unclean boot (a panic before userspace adds none), so the box cannot stay off.
|
||||||
|
- `internal/osupdate/kernel.go`: the night leg ends with the kernel step — after a healthy host step (and a healthy
|
||||||
|
Proxmox step when one ran), trigger `night` only, on a night the hub's `os_update.kernel` block marks `tonight` (the
|
||||||
|
household was mailed the day before — no mail, no step). Ring 0 stages + reboots; ring 1 reboots only a kernel a
|
||||||
|
signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged`
|
||||||
|
BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the
|
||||||
|
host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured:
|
||||||
|
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 45-minute `host_stale` (`alerting.stale_threshold`)).
|
||||||
|
Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`.
|
||||||
|
Tests: `TestKernel*` (13); red-proofs in the same file.
|
||||||
|
- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is
|
||||||
|
destructive-class. `cmd/felhom-opsign`: the op is listed.
|
||||||
|
|
||||||
|
## v0.151.0 — the agent can no longer hand the guest any image; the Proxmox package lane; the other-key archives reported; the DR directive retired (R-861, R-812 A, R-366, R-105; `09` §3 163, 165, 168, 169) (2026-10-07)
|
||||||
|
|
||||||
|
Released by `scripts/release-agent.sh`: binary sha256 `0464354f2cdf452a7c5d2a74d9191fe91415fcfa244154480d26d5b30e10b194`
|
||||||
|
config bundle sha256 `bacd1d175a9392bc1755341d01a106abbd72aba48ab575e7a32dd819d8f5da4c` (tag `v0.151.0` = `dd7cdc0`).
|
||||||
|
**Deliver the binary FIRST, then the bundle:** the bundle's sudoers removes the `tee` grant the 0.150.0 binary still uses.
|
||||||
|
No path added (26 → 26), so no step bundle.
|
||||||
|
|
||||||
|
### Part of v0.151.0 — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
|
||||||
|
|
||||||
|
**Delivery order: agent binary FIRST, then the config bundle.** The new sudoers drops the agent's in-guest `tee`
|
||||||
|
grant; an older binary still calls `tee`, so a bundle that lands before the binary would stop managed controller
|
||||||
|
updates (and the old binary's capability probe would read `controllerswap-write` degraded). No bundle path is added
|
||||||
|
(`felhom-priv-apply` and `/etc/sudoers.d/felhom-op` are already bundle files), so no step bundle.
|
||||||
|
|
||||||
|
- `configs/felhom-priv-apply`: new verb `controller-image <vmid>` — reads the ref on stdin (≤ 256 bytes, ASCII, one
|
||||||
|
optional trailing newline), requires `^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$` (the agent's
|
||||||
|
own `controllerImageRe`), then runs `pct exec <vmid> -- tee /etc/felhom-controller-image` AS ROOT; refusal rule `I1`
|
||||||
|
(rc 3), a bad vmid `A1` (rc 2); listed in `--self-check`.
|
||||||
|
- `configs/felhom-agent.sudoers` `FELHOM_CONTROLLERSWAP`: `pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$`
|
||||||
|
REMOVED; `/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$` added.
|
||||||
|
- `internal/localapi`: `GuestExecutor.GuestExecStdin` replaced by `WriteControllerImage`; `GuestBinder.WriteControllerImage`
|
||||||
|
pipes `ref\n` to the verb through the fenced runner; the swap's `writeImage` calls it. Capability `controllerswap-write`
|
||||||
|
now probes the verb.
|
||||||
|
- `configs/felhom-op.sudoers` (B2, hygiene): `pct start|stop|unlock [0-9]*` → `pct ^start [0-9]+$` etc. (the glob's `*`
|
||||||
|
matched spaces: `pct stop 9201 --skiplock 1` passed).
|
||||||
|
- Tests: `ControllerImage` (5, `configs/test_felhom_priv_apply.py`), `TestSudoersRefusesTheR861Injections` (+3 lines),
|
||||||
|
`TestSudoersAllowsTheControllerImageVerb`, `TestFelhomOpSudoersPctIsExact`, `TestR861_WriteControllerImageUsesTheRootVerb`,
|
||||||
|
`TestControllerSwap_WriteViaRootVerb_NoShell`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/C/`.
|
||||||
|
- `README.md`: the controller-swap paragraph described the removed `tee` path — corrected.
|
||||||
|
### Part of v0.151.0 — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
|
||||||
|
|
||||||
|
**MinAgent impact: none** (a new layer; an older hub ignores the pve report). **The bundle carries the new
|
||||||
|
`felhom-os-apply` — deliver it with the binary** (signed `agent_update`, then signed `agent_config_update`).
|
||||||
|
|
||||||
|
- `configs/felhom-os-apply`: new layer `pve`, lane `slow` only — the host's Proxmox USERSPACE packages: origin `Proxmox Debian Repository` only (R2), never a kernel / boot / firmware / microcode name (R14, `HOST_SLOW_RE` — the kernel is R-836's lane), no removal (R4), no undo (R5), a new package only from `PVE_NEW_ALLOW` (`proxmox-firewall-data`, measured on demo-felhom; R6 otherwise), an appliance only (R12), authority = a signed `os_pve_step` or the root-owned ring-0 mark (R3). Select `pending-pve` (ring 0): installed Proxmox-origin packages with a pending upgrade. The report carries `pve_manager` (pveversion after the step).
|
||||||
|
- `internal/pvegate` (new): the agent's own writes to /etc/pve wait while a pve step runs (pmxcfs restarts); the step waits for writes in flight (bounded, 2 min — then it fails and does not run). Wired at `proxmox.Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile).
|
||||||
|
- `internal/osupdate`: `LayerPVE`; the night leg runs the pve step in ring 0 after a healthy host step (an appliance; ring 1 never in the night leg); `PVEHealthVerdict` = the host rule + every running container keeps its id + pveversion reads the installed pve-manager; the pve report carries Proxmox userspace only (the hub's candidate set). `PVEStepExecutor` (signed `os_pve_step`, ring 1, under the heavy-op gate and the /etc/pve gate); `reconcile.ClassOSPVEStep` (destructive-class); `felhom-opsign -op os_pve_step` (params by `-params`).
|
||||||
|
- Tests: wrapper `PVELane` (17; red first — the `pve-manager` plan was refused R12 on the old code), `pvegate` (5), `TestPVEGate_*` + `TestWritesEtcPVE`, `TestPVE_*`, `TestPVEHealthVerdict`, `TestPVEStepExecutor_*`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/B/`.
|
||||||
|
### Part of v0.151.0
|
||||||
|
|
||||||
|
- R-366 slice 2 (`09` §3 decision 168): the restore-test pick records, per tier, the archives it skipped as written with another key (count, oldest, newest — no key material) in a `ForeignKeyLedger`; the host report carries it as `foreign_key_archives.tiers` (the stanza absent until a tier was evaluated since start, `tiers: []` when none — no null on the wire, the report contract forbids it). The hub turns a change into one operator line. Tests `TestR366_PickRecordsArchivesWrittenWithAnotherKey`, `TestR366_EvaluatedWithNoneIsAnEmptyList` (red-proved, `felhom.eu/documentation/audits/day-2026-10-07/E/`).
|
||||||
|
- R-105 option A (`09` §3 decision 169): the `--selftest=escrow-create -directive <file>` flag and the escrow upload's `directive` field are removed — nothing read the directive; the DR path reads the recipe, tenantsync and the escrow blob. The hub ignores a `directive` from an older agent.
|
||||||
|
|
||||||
|
## v0.150.0 — the Docker step proves the engine reports a memory kill; after a restart the agent remembers the last backup per tier; three more SMART counters on the wire (R-528, R-894, R-330; `09` §3 decisions 157, 161) (2026-10-07)
|
||||||
|
|
||||||
|
Released by `scripts/release-agent.sh`: binary sha256 `a23d1c9085bc7fd4fc48fe0327f6504aa83e6331510fb4a3d23e042dddb26f9c`
|
||||||
|
config bundle sha256 `88456b386d9b1027bd22861cac8c23df004bf9fd9f67644d6595bfca8c94498e` (tag `v0.150.0` = `3a72a48`).
|
||||||
|
**The bundle carries the new `felhom-os-apply` (the memory-kill check) — deliver it with the binary:** signed
|
||||||
|
`agent_update`, then signed `agent_config_update`. No path added (26 → 26), so no step bundle.
|
||||||
|
|
||||||
|
### Part of v0.150.0 (2026-10-06 night, later) — after a restart the agent remembers the last backup per tier (R-894); three more SMART counters on the wire (R-330)
|
||||||
|
|
||||||
Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night read-back. Nothing delivered tonight.
|
Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night read-back. Nothing delivered tonight.
|
||||||
|
|
||||||
@@ -9,7 +162,7 @@ Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night r
|
|||||||
- Tests: `TestBackupDue_R894_*` (restart = a new server and a new state from the same file; fresh / old / none / storage answers / failed backup not saved), `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon` (AST). Four red-proofs observed (`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`).
|
- Tests: `TestBackupDue_R894_*` (restart = a new server and a new state from the same file; fresh / old / none / storage answers / failed backup not saved), `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon` (AST). Four red-proofs observed (`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`).
|
||||||
- **R-330 (disk health Phase 2, the wire only):** the SMART summary carries three more SATA raw counters — `reported_uncorrect` (187), `command_timeout` (188, carried as the vendor reports it; some pack several counters), `udma_crc_errors` (199). Pointer + omitempty: an attribute the drive does not report is OMITTED (unknown), never 0. No verdict reads them yet. Tests `TestParseSMART_R330_*` (two red-proofs, `felhom.eu/documentation/audits/night-burndown-2026-10-06/r330/`).
|
- **R-330 (disk health Phase 2, the wire only):** the SMART summary carries three more SATA raw counters — `reported_uncorrect` (187), `command_timeout` (188, carried as the vendor reports it; some pack several counters), `udma_crc_errors` (199). Pointer + omitempty: an attribute the drive does not report is OMITTED (unknown), never 0. No verdict reads them yet. Tests `TestParseSMART_R330_*` (two red-proofs, `felhom.eu/documentation/audits/night-burndown-2026-10-06/r330/`).
|
||||||
|
|
||||||
## Unreleased (2026-10-06 night) — the Docker step proves the engine reports a memory kill (`09` §3 decision 157, R-528)
|
### Part of v0.150.0 (2026-10-06 night) — the Docker step proves the engine reports a memory kill (`09` §3 decision 157, R-528)
|
||||||
|
|
||||||
To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night read-back (the night of 2026-10-06 runs v0.149.0 on purpose).
|
To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night read-back (the night of 2026-10-06 runs v0.149.0 on purpose).
|
||||||
|
|
||||||
@@ -18,11 +171,11 @@ To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night rea
|
|||||||
- `configs/test_felhom_os_apply.py`: its `unittest.main()` sat in the middle of the file, so 11 tests (UnsentReport, SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran — moved to the end; all pass.
|
- `configs/test_felhom_os_apply.py`: its `unittest.main()` sat in the middle of the file, so 11 tests (UnsentReport, SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran — moved to the end; all pass.
|
||||||
- Tests: the OOMCheck class (pass, OOMKilled=false, no event, unreadable image, removal on an inspect error, not on other layers or in health mode, the events window after the settle wait, mode oom-check alone and its refusals); TestDocker_OOMCheckReachesTheHubUnchanged, TestR868_KeptCopyCarriesTheOOMCheck. 12 red-proofs in `felhom.eu/documentation/audits/readback-2026-10-07/F/`.
|
- Tests: the OOMCheck class (pass, OOMKilled=false, no event, unreadable image, removal on an inspect error, not on other layers or in health mode, the events window after the settle wait, mode oom-check alone and its refusals); TestDocker_OOMCheckReachesTheHubUnchanged, TestR868_KeptCopyCarriesTheOOMCheck. 12 red-proofs in `felhom.eu/documentation/audits/readback-2026-10-07/F/`.
|
||||||
|
|
||||||
## Unreleased (2026-10-06 evening) — the shared rule file (`09` §3 decision 152); no code change
|
### Part of v0.150.0 (2026-10-06 evening) — the shared rule file (`09` §3 decision 152); no code change
|
||||||
|
|
||||||
- `.claude/rules/unprompted-work.md` added, byte-identical to the copies in felhom.eu, felhom-controller, app-catalog-felhom.eu and the workspace root (checked with `diff` against the controller's copy and one md5 across all five). Its copies line names five copies.
|
- `.claude/rules/unprompted-work.md` added, byte-identical to the copies in felhom.eu, felhom-controller, app-catalog-felhom.eu and the workspace root (checked with `diff` against the controller's copy and one md5 across all five). Its copies line names five copies.
|
||||||
|
|
||||||
## Unreleased (2026-10-06 afternoon) — instruction files kept true (`09` §3 decision 150); no code change
|
### Part of v0.150.0 (2026-10-06 afternoon) — instruction files kept true (`09` §3 decision 150); no code change
|
||||||
|
|
||||||
- `CLAUDE.md` „Gates — ONE entry point": the runner runs every gate in its `GATES` table (five: three shared, `published`, `release-complete`); `--fast` skips `published` (network). It said two gates and „all of them".
|
- `CLAUDE.md` „Gates — ONE entry point": the runner runs every gate in its `GATES` table (five: three shared, `published`, `release-complete`); `--fast` skips `published` (network). It said two gates and „all of them".
|
||||||
- `CLAUDE.md`: the decoy gate and its audit are named with their `felhom.eu/` prefix (they do not exist in this repo).
|
- `CLAUDE.md`: the decoy gate and its audit are named with their `felhom.eu/` prefix (they do not exist in this repo).
|
||||||
|
|||||||
@@ -1,5 +1,12 @@
|
|||||||
# CONTEXT — felhom-agent working state
|
# CONTEXT — felhom-agent working state
|
||||||
|
|
||||||
|
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
|
||||||
|
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
|
||||||
|
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
|
||||||
|
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
|
||||||
|
|
||||||
|
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
|
||||||
|
|
||||||
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
||||||
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
|
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
|
||||||
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
|
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
|
||||||
|
|||||||
@@ -44,13 +44,14 @@ unnoticed until a user hit them. `internal/capability` makes that loud:
|
|||||||
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
|
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
|
||||||
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
|
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
|
||||||
|
|
||||||
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
|
**Controller-swap under non-root (v0.45.0; the write since R-861 (a) A1).** The agent-owned controller image swap
|
||||||
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
|
(`internal/localapi/controllerswap.go`) no longer shells out. The write goes on **stdin** to the ROOT verb
|
||||||
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
|
`felhom-priv-apply controller-image <vmid>` (`GuestBinder.WriteControllerImage` → `Runner.RunStdin`, the same fenced
|
||||||
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
|
`sudo -n` runner), which re-checks the ref against our registry + repository + an x.y.z tag and writes
|
||||||
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
|
`/etc/felhom-controller-image` inside the guest itself; the agent has no in-guest `tee` grant any more (before, a
|
||||||
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
|
compromised agent could feed any image — sudo cannot see stdin). Its grants live in the `FELHOM_CONTROLLERSWAP` sudoers
|
||||||
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
|
alias (read-only or fixed-target) and in the capability manifest (Critical), so a dropped grant is a build failure + a
|
||||||
|
live degraded signal. No general `pct exec` is granted.
|
||||||
|
|
||||||
## The `storage` package — observe + watchdog (slice 5)
|
## The `storage` package — observe + watchdog (slice 5)
|
||||||
|
|
||||||
|
|||||||
@@ -1,14 +1,16 @@
|
|||||||
# REPORT — the second burn-down night (2026-10-06 → 07): R-894 and the R-330 wire half, on main, unreleased
|
# REPORT — agent, 2026-10-08 (day): R-899 and R-304 on main, unreleased
|
||||||
|
|
||||||
Nothing was released or delivered (brief fence). Both changes ship with the memory-kill check as **v0.150.0**, after the
|
**No release, no delivery today** (kernel night 8→9; `09` §3 decision 178). Binary-only; ships with tomorrow's release.
|
||||||
2026-10-07 night read-back. Session report: `felhom.eu/REPORT-night-burndown-2026-10-06.md`; night log
|
|
||||||
`felhom.eu/documentation/audits/night-burndown-2026-10-06/NIGHT-LOG.md`.
|
|
||||||
|
|
||||||
- **R-894** (`74b5eae`): the newest successful backup per tier is kept on disk (`backup-success-state.json` in the OOB
|
| Item | Commit | Tests | Red-proof |
|
||||||
state dir) and read by `/backup/due` **only when the tier's storage cannot be read**: fresh → not due, older than the
|
|---|---|---|---|
|
||||||
cadence → due, none → due with age unknown (as before). A storage that answers stays the ground truth. Tests
|
| R-899 — `trigger=manual` (a press) runs no OS leg | `4c69c25` | `TestAfterPrimaryBackup` + 2 sub-cases | ignoring `trigger` → the leg ran once after a press (`[8200]`) |
|
||||||
`TestBackupDue_R894_*`, `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon`; four red-proofs
|
| „ring 1" label in after-boot kernel reports (seen 2026-10-08 night, demo-felhom) | `4c69c25` | `TestKernelReportRing_BeforeFirstFetch` | `Block().Ring` → `ring 1, want 0` |
|
||||||
`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`. Architecture: `07` §6.1 updated.
|
| R-304 — 424 `older_unchecked` instead of „did not open the sealed bundle" when earlier packages were not all tried | `91b9405` | `TestR304_*` (real age crypto), 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus` | check off → four cases returned the wrong-code error |
|
||||||
- **R-330, wire only** (`adaf86a`): the host report's SMART summary carries 187 `reported_uncorrect`, 188
|
|
||||||
`command_timeout`, 199 `udma_crc_errors` (pointers, omitted when unknown). No verdict reads them.
|
**Read before the change (read-only, demo-hp):** on 2026-10-07 08:51 the morning press DID reach the OS leg
|
||||||
- Suite `go test ./...` rc 0; `scripts/agent_gates.py` (full) all OK; CI green on each push (runs 1451, 1467).
|
(`osupdate: skipped — already ran tonight`, saved by the 20-hour rule only); `os-update-block.json` on demo-hp holds
|
||||||
|
`ring 0` (the fallback the label fix reads).
|
||||||
|
|
||||||
|
**Gates:** `go build/vet/test ./...` rc 0; `agent_gates.py --fast` rc 0. **CI:** job 1529 (`4c69c25`) success; job 1530
|
||||||
|
(`91b9405`) success.
|
||||||
|
|||||||
@@ -15,7 +15,7 @@
|
|||||||
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
|
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
|
||||||
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
|
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
|
||||||
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
|
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
|
||||||
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key` | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
|
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
|
||||||
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
|
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
|
||||||
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
|
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
|
||||||
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
|
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
|
||||||
@@ -88,6 +88,8 @@
|
|||||||
|
|
||||||
| Symbol | File | Short signature | Use for | Gotchas |
|
| Symbol | File | Short signature | Use for | Gotchas |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
|
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
|
||||||
|
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
|
||||||
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
||||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||||
@@ -168,7 +170,7 @@
|
|||||||
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
||||||
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
||||||
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
||||||
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
|
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec; the image write via `felhom-priv-apply controller-image`) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
|
||||||
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
|
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
|
||||||
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
|
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
|
||||||
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
|
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
|
||||||
|
|||||||
+90
-61
@@ -135,39 +135,38 @@ func main() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
var (
|
var (
|
||||||
cfgPath string
|
cfgPath string
|
||||||
selftest selftestFlag
|
selftest selftestFlag
|
||||||
vmid int
|
vmid int
|
||||||
watch time.Duration
|
watch time.Duration
|
||||||
archive string
|
archive string
|
||||||
mode string
|
mode string
|
||||||
hostname string
|
hostname string
|
||||||
keep bool
|
keep bool
|
||||||
rootfsGrow int
|
rootfsGrow int
|
||||||
dataVolGrow int
|
dataVolGrow int
|
||||||
dataVolMount string
|
dataVolMount string
|
||||||
sysDataGrow int
|
sysDataGrow int
|
||||||
sysDataMount string
|
sysDataMount string
|
||||||
cores int
|
cores int
|
||||||
memoryMB int
|
memoryMB int
|
||||||
pbsStorage string
|
pbsStorage string
|
||||||
paperkey bool
|
paperkey bool
|
||||||
offline bool
|
offline bool
|
||||||
upload bool
|
upload bool
|
||||||
custID string
|
custID string
|
||||||
custDomain string
|
custDomain string
|
||||||
custName string
|
custName string
|
||||||
custEmail string
|
custEmail string
|
||||||
hubPassword string
|
hubPassword string
|
||||||
blobPath string
|
blobPath string
|
||||||
expectedFP string
|
expectedFP string
|
||||||
keyDest string
|
keyDest string
|
||||||
installWGKey bool
|
installWGKey bool
|
||||||
idBundlePath string
|
idBundlePath string
|
||||||
directivePath string
|
swapImage string
|
||||||
swapImage string
|
outputMode string
|
||||||
outputMode string
|
showVersion bool
|
||||||
showVersion bool
|
|
||||||
)
|
)
|
||||||
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
|
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
|
||||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `restore-test-due` = READ-ONLY: print the per-tier due verdict the scheduler would act on, with its cost; `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
|
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `restore-test-due` = READ-ONLY: print the per-tier due verdict the scheduler would act on, with its cost; `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
|
||||||
@@ -198,7 +197,6 @@ func main() {
|
|||||||
flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)")
|
flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)")
|
||||||
flag.BoolVar(&installWGKey, "install-wg-key", false, "for --selftest=identity-consume: ALSO install the recovered wg_private_key into wgtunnel's key file (S5 DR; create-only, refuses to overwrite)")
|
flag.BoolVar(&installWGKey, "install-wg-key", false, "for --selftest=identity-consume: ALSO install the recovered wg_private_key into wgtunnel's key file (S5 DR; create-only, refuses to overwrite)")
|
||||||
flag.StringVar(&idBundlePath, "identity-bundle", "", "for --selftest=escrow-create: a 0600 JSON file {tunnel_token,pbs_token} to ALSO escrow under R (10D)")
|
flag.StringVar(&idBundlePath, "identity-bundle", "", "for --selftest=escrow-create: a 0600 JSON file {tunnel_token,pbs_token} to ALSO escrow under R (10D)")
|
||||||
flag.StringVar(&directivePath, "directive", "", "for --selftest=escrow-create: a JSON file with the non-secret DR directive (pbs repo/ns, expected fingerprint, tunnel id)")
|
|
||||||
flag.StringVar(&custID, "customer-id", "", "for --selftest=provision: the customer id — the hub config-pull target, baked into the guest's bootstrap")
|
flag.StringVar(&custID, "customer-id", "", "for --selftest=provision: the customer id — the hub config-pull target, baked into the guest's bootstrap")
|
||||||
flag.StringVar(&hubPassword, "hub-password", "", "for --selftest=provision: the customer's hub RETRIEVAL PASSPHRASE (SECRET) — baked into bootstrap.json so the controller pulls its config (and the customer-scoped hub key) from the hub. The customer must already exist in the hub.")
|
flag.StringVar(&hubPassword, "hub-password", "", "for --selftest=provision: the customer's hub RETRIEVAL PASSPHRASE (SECRET) — baked into bootstrap.json so the controller pulls its config (and the customer-scoped hub key) from the hub. The customer must already exist in the hub.")
|
||||||
flag.StringVar(&swapImage, "image", "", "for --selftest=controller-swap: the target controller image ref (gitea.dooplex.hu/admin/felhom-controller:<semver>) — must already be pulled in the guest")
|
flag.StringVar(&swapImage, "image", "", "for --selftest=controller-swap: the target controller image ref (gitea.dooplex.hu/admin/felhom-controller:<semver>) — must already be pulled in the guest")
|
||||||
@@ -269,7 +267,7 @@ func main() {
|
|||||||
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount, Cores: cores, MemoryMB: memoryMB},
|
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount, Cores: cores, MemoryMB: memoryMB},
|
||||||
}))
|
}))
|
||||||
case "escrow-create":
|
case "escrow-create":
|
||||||
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath, outputMode))
|
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, outputMode))
|
||||||
case "escrow-consume":
|
case "escrow-consume":
|
||||||
os.Exit(runSelftestEscrowConsume(context.Background(), logger, blobPath, expectedFP, keyDest))
|
os.Exit(runSelftestEscrowConsume(context.Background(), logger, blobPath, expectedFP, keyDest))
|
||||||
case "identity-consume":
|
case "identity-consume":
|
||||||
@@ -792,6 +790,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
|
pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
|
||||||
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
|
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
|
||||||
collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
|
collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
|
||||||
|
// R-366 slice 2: the restore-test's ledger of archives written with another key → the host report.
|
||||||
|
foreignKeys := backup.NewForeignKeyLedger()
|
||||||
|
collector.SetForeignKeyArchiveReporter(foreignKeys)
|
||||||
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
|
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
|
||||||
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
|
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
|
||||||
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
|
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
|
||||||
@@ -876,6 +877,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
|
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
|
||||||
logger.Info("osupdate: sent kept report(s)", "count", n)
|
logger.Info("osupdate: sent kept report(s)", "count", n)
|
||||||
})
|
})
|
||||||
|
// R-836 (`09` §3 decision 172): what became of a kernel step across this boot; on a one-shot boot of a new kernel,
|
||||||
|
// judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE.
|
||||||
|
// The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the
|
||||||
|
// reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow
|
||||||
|
// network and stays under the hub's 45-minute host_stale (its
|
||||||
|
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||||
|
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
|
||||||
|
|
||||||
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
|
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
|
||||||
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
|
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
|
||||||
@@ -1017,7 +1025,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
// with the local API so a backup and a restore-test can never run together.
|
// with the local API so a backup and a restore-test can never run together.
|
||||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||||
heavyOps := &backup.InFlight{}
|
heavyOps := &backup.InFlight{}
|
||||||
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
|
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, foreignKeys, logger)
|
||||||
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
|
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
|
||||||
// holds only this process's latest run, and under per-archive due-ness the agent will not
|
// holds only this process's latest run, and under per-archive due-ness the agent will not
|
||||||
// re-test an archive it has already proven — so without this the hub can report a tier unproven
|
// re-test an archive it has already proven — so without this the hub can report a tier unproven
|
||||||
@@ -1109,6 +1117,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
}
|
}
|
||||||
return release, nil
|
return release, nil
|
||||||
}}
|
}}
|
||||||
|
// R-812 option A: a signed Proxmox package step (`11` §5.10) — ring 1; under the heavy-op gate and the /etc/pve gate.
|
||||||
|
pveExec := osupdate.PVEStepExecutor{Leg: osLeg, Guest: firstGuest(px),
|
||||||
|
Gate: func(ctx context.Context) (func(), error) {
|
||||||
|
release, busy, ok := heavyOps.TryAcquire("os-pve-step")
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("busy: %s", busy)
|
||||||
|
}
|
||||||
|
return release, nil
|
||||||
|
}}
|
||||||
|
// R-836 (`09` §3 decision 172): a signed kernel step STAGES a kernel on a ring-1 box (install + the one-shot flag,
|
||||||
|
// never a reboot — the night leg reboots it on a night the household was told about); under the heavy-op gate.
|
||||||
|
kernelExec := osupdate.KernelStepExecutor{Leg: osLeg, Guest: firstGuest(px),
|
||||||
|
Gate: func(ctx context.Context) (func(), error) {
|
||||||
|
release, busy, ok := heavyOps.TryAcquire("os-kernel-step")
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("busy: %s", busy)
|
||||||
|
}
|
||||||
|
return release, nil
|
||||||
|
}}
|
||||||
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
||||||
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
||||||
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
||||||
@@ -1120,7 +1147,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
}
|
}
|
||||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||||
}}
|
}}
|
||||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, bundleExec}, cfg.Hub.HostID, logger)
|
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec}, cfg.Hub.HostID, logger)
|
||||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
|
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
|
||||||
|
|
||||||
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
|
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
|
||||||
@@ -1700,7 +1727,7 @@ func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget
|
|||||||
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the
|
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the
|
||||||
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
|
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
|
||||||
// machinery still works on-demand via --selftest=restore-test.
|
// machinery still works on-demand via --selftest=restore-test.
|
||||||
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler {
|
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, foreign *backup.ForeignKeyLedger, logger *slog.Logger) *backup.Scheduler {
|
||||||
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
|
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
|
||||||
// per-archive due-check in internal/backup/restoretest_due.go.
|
// per-archive due-check in internal/backup/restoretest_due.go.
|
||||||
cadence := cfg.Backup.RestoreTestEvalInterval()
|
cadence := cfg.Backup.RestoreTestEvalInterval()
|
||||||
@@ -1719,6 +1746,9 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
|
|||||||
min, max := cfg.Backup.ScratchBand()
|
min, max := cfg.Backup.ScratchBand()
|
||||||
target := cfg.Backup.BackupTarget()
|
target := cfg.Backup.BackupTarget()
|
||||||
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
|
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
|
||||||
|
if foreign != nil {
|
||||||
|
runner.SetForeignKeyLedger(foreign) // R-366 slice 2: the pick records archives written with another key
|
||||||
|
}
|
||||||
// Every configured tier is a rotation candidate, not just the primary.
|
// Every configured tier is a rotation candidate, not just the primary.
|
||||||
cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed
|
cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed
|
||||||
tierIDs := make([]string, 0, len(cfgTiers))
|
tierIDs := make([]string, 0, len(cfgTiers))
|
||||||
@@ -1874,11 +1904,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
|||||||
return nil, 0, ferr
|
return nil, 0, ferr
|
||||||
}
|
}
|
||||||
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
|
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
|
||||||
|
// R-304: every package the hub holds and the code will NOT be tried against — no key material,
|
||||||
|
// over the hub's cap, or malformed here. A refusal may call the code wrong only when this is 0.
|
||||||
|
withheld := resp.UnopenableCount + resp.TruncatedCount
|
||||||
for _, p := range resp.Packages {
|
for _, p := range resp.Packages {
|
||||||
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
|
||||||
if derr != nil || len(blob) == 0 {
|
if derr != nil || len(blob) == 0 {
|
||||||
// One malformed package must not sink the rest — the customer's code may open a
|
// One malformed package must not sink the rest — the customer's code may open a
|
||||||
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
|
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
|
||||||
|
withheld++
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
out = append(out, escrow.RetainedBlob{
|
out = append(out, escrow.RetainedBlob{
|
||||||
@@ -1888,9 +1922,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
|||||||
Index: p.Index,
|
Index: p.Index,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return out, resp.UnopenableCount, nil
|
return out, withheld, nil
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
// R-894: ONE instance — the local API writes it, the host report reads it (2026-10-09: a restart right
|
||||||
|
// after the night backup erased the hub's evidence of it).
|
||||||
|
lastKnownBackups := backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json"))
|
||||||
|
collector.SetKnownBackupReporter(lastKnownBackups)
|
||||||
srv, err := localapi.NewServer(localapi.Options{
|
srv, err := localapi.NewServer(localapi.Options{
|
||||||
EscrowRecovery: escrowRecoverer,
|
EscrowRecovery: escrowRecoverer,
|
||||||
ListenAddr: cfg.LocalAPI.ListenAddr,
|
ListenAddr: cfg.LocalAPI.ListenAddr,
|
||||||
@@ -1903,7 +1941,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
|||||||
Store: store,
|
Store: store,
|
||||||
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
|
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
|
||||||
// read right after a restart. Same state dir as restore-test-state.json.
|
// read right after a restart. Same state dir as restore-test-state.json.
|
||||||
LastKnownBackups: backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json")),
|
LastKnownBackups: lastKnownBackups,
|
||||||
Storage: observer,
|
Storage: observer,
|
||||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||||
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
|
||||||
@@ -2264,7 +2302,7 @@ func runSelftestRestoreTestDue(ctx context.Context, cfg config.Config, logger *s
|
|||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
|
||||||
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, logger)
|
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, nil, logger)
|
||||||
|
|
||||||
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
|
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
@@ -2690,7 +2728,6 @@ type escrowCeremonyOpts struct {
|
|||||||
offline bool
|
offline bool
|
||||||
upload bool
|
upload bool
|
||||||
identityBundlePath string
|
identityBundlePath string
|
||||||
directivePath string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the
|
// escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the
|
||||||
@@ -2749,11 +2786,10 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
|
|||||||
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("PBS key for %q not found (%s): %v", storage, keyPath, err)}
|
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("PBS key for %q not found (%s): %v", storage, keyPath, err)}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Slice 10D.1: optionally ALSO wrap the identity bundle under the same R, and carry the non-secret
|
// Slice 10D.1: optionally ALSO wrap the identity bundle under the same R. The bundle file is a 0600 secret
|
||||||
// directive for the hub. The bundle file is a 0600 secret (tunnel/pbs tokens); the directive is
|
// (tunnel/pbs tokens). The non-secret "directive" that used to ride along is retired (R-105, `09` §3 decision 169:
|
||||||
// non-secret (pbs repo/ns, expected fingerprint, tunnel id).
|
// nothing read it; the DR path reads the recipe, tenantsync and the escrow blob).
|
||||||
var identity *escrow.IdentityBundle
|
var identity *escrow.IdentityBundle
|
||||||
var directive json.RawMessage
|
|
||||||
if opts.identityBundlePath != "" {
|
if opts.identityBundlePath != "" {
|
||||||
raw, err := os.ReadFile(opts.identityBundlePath)
|
raw, err := os.ReadFile(opts.identityBundlePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -2764,11 +2800,6 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
|
|||||||
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("identity bundle is not valid JSON {tunnel_token,pbs_token}: %v", err)}
|
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("identity bundle is not valid JSON {tunnel_token,pbs_token}: %v", err)}
|
||||||
}
|
}
|
||||||
identity = &b
|
identity = &b
|
||||||
if opts.directivePath != "" {
|
|
||||||
if d, err := os.ReadFile(opts.directivePath); err == nil && json.Valid(d) {
|
|
||||||
directive = d
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
// S3: auto-inject the offsite WG private key into the escrowed identity when the key file
|
// S3: auto-inject the offsite WG private key into the escrowed identity when the key file
|
||||||
// exists — a NEW escrow run should always capture the live tunnel identity. Field NAME only
|
// exists — a NEW escrow run should always capture the live tunnel identity. Field NAME only
|
||||||
@@ -2847,7 +2878,7 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
|
|||||||
ResticPwSealed: resticStaged,
|
ResticPwSealed: resticStaged,
|
||||||
}
|
}
|
||||||
if opts.upload {
|
if opts.upload {
|
||||||
if err := uploadEscrowBlob(ctx, cfg, res, directive, resticPwSHA256); err != nil {
|
if err := uploadEscrowBlob(ctx, cfg, res, resticPwSHA256); err != nil {
|
||||||
// R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets
|
// R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets
|
||||||
// the text shell keep the pre-extraction order (R surfaced, THEN the failure).
|
// the text shell keep the pre-extraction order (R surfaced, THEN the failure).
|
||||||
return out, &escrowCeremonyErr{kind: "upload", err: err}
|
return out, &escrowCeremonyErr{kind: "upload", err: err}
|
||||||
@@ -2904,7 +2935,7 @@ func printEscrowTextRBlock(out *escrowCeremonyOutcome) {
|
|||||||
// NOTHING else there; every human/info line goes to stderr; failures exit non-zero with no
|
// NOTHING else there; every human/info line goes to stderr; failures exit non-zero with no
|
||||||
// partial JSON. This is the controller-driven ceremony's parse surface (spike §2.3: the text
|
// partial JSON. This is the controller-driven ceremony's parse surface (spike §2.3: the text
|
||||||
// banner is positionally brittle).
|
// banner is positionally brittle).
|
||||||
func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, directivePath, outputMode string) int {
|
func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, outputMode string) int {
|
||||||
switch outputMode {
|
switch outputMode {
|
||||||
case "", "text", "json":
|
case "", "text", "json":
|
||||||
default:
|
default:
|
||||||
@@ -2922,7 +2953,7 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
|
|||||||
|
|
||||||
out, cerr := escrowCeremony(ctx, cfg, logger, escrowCeremonyOpts{
|
out, cerr := escrowCeremony(ctx, cfg, logger, escrowCeremonyOpts{
|
||||||
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
|
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
|
||||||
identityBundlePath: identityBundlePath, directivePath: directivePath,
|
identityBundlePath: identityBundlePath,
|
||||||
})
|
})
|
||||||
if cerr != nil {
|
if cerr != nil {
|
||||||
switch cerr.kind {
|
switch cerr.kind {
|
||||||
@@ -3105,20 +3136,19 @@ type escrowUploadRequest struct {
|
|||||||
BlobB64 string `json:"blob_b64"` // base64 of the opaque R-wrapped blob (ciphertext)
|
BlobB64 string `json:"blob_b64"` // base64 of the opaque R-wrapped blob (ciphertext)
|
||||||
KeyFingerprint string `json:"key_fingerprint"` // for operator display only
|
KeyFingerprint string `json:"key_fingerprint"` // for operator display only
|
||||||
Posture string `json:"posture"` // e.g. "zero_knowledge"
|
Posture string `json:"posture"` // e.g. "zero_knowledge"
|
||||||
// Slice 10D.1 — optional DR bundle (identity escrow + non-secret directive). Omitted in slice-7.
|
// Slice 10D.1 — optional identity escrow. Omitted in slice-7. (The `directive` is retired — R-105.)
|
||||||
IdentityBlobB64 string `json:"identity_blob_b64,omitempty"`
|
IdentityBlobB64 string `json:"identity_blob_b64,omitempty"`
|
||||||
DirectiveJSON json.RawMessage `json:"directive,omitempty"`
|
CreatedAt string `json:"created_at"` // RFC3339
|
||||||
CreatedAt string `json:"created_at"` // RFC3339
|
|
||||||
// SLICE 3 — sha256 hex of the offsite restic repo password sealed in the identity blob (present only
|
// SLICE 3 — sha256 hex of the offsite restic repo password sealed in the identity blob (present only
|
||||||
// when a staged password was folded in). Non-reversible hash of a 256-bit random secret — safe to
|
// when a staged password was folded in). Non-reversible hash of a 256-bit random secret — safe to
|
||||||
// store/serve; lets the controller VERIFY "the escrow covers the CURRENT key" and auto-confirm.
|
// store/serve; lets the controller VERIFY "the escrow covers the CURRENT key" and auto-confirm.
|
||||||
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob + non-secret directive) to
|
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob) to
|
||||||
// the hub, authed with the per-host key. The hub stores ciphertext + non-secret fields; no usable
|
// the hub, authed with the per-host key. The hub stores ciphertext + non-secret fields; no usable
|
||||||
// secret leaves the agent.
|
// secret leaves the agent.
|
||||||
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, directive json.RawMessage, resticPwSHA256 string) error {
|
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string) error {
|
||||||
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
||||||
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
||||||
}
|
}
|
||||||
@@ -3131,7 +3161,6 @@ func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateR
|
|||||||
}
|
}
|
||||||
if len(res.IdentityBlob) > 0 {
|
if len(res.IdentityBlob) > 0 {
|
||||||
upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob)
|
upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob)
|
||||||
upReq.DirectiveJSON = directive
|
|
||||||
}
|
}
|
||||||
body, _ := json.Marshal(upReq)
|
body, _ := json.Marshal(upReq)
|
||||||
url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow"
|
url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow"
|
||||||
|
|||||||
@@ -12,12 +12,18 @@ import (
|
|||||||
//
|
//
|
||||||
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
|
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
|
||||||
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
|
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
|
||||||
|
//
|
||||||
|
// 2026-10-09: the SAME instance also feeds the host report (collector.SetKnownBackupReporter), so a
|
||||||
|
// restart right after a backup no longer erases the hub's evidence of it. The field may name a local
|
||||||
|
// variable; the variable must be built by backup.NewBackupSuccessState and be the one handed to the
|
||||||
|
// collector. RED-PROOF (observed): drop the SetKnownBackupReporter call → "not handed to the collector".
|
||||||
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
||||||
_, f := parseMain(t)
|
_, f := parseMain(t)
|
||||||
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
|
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
|
||||||
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
|
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
|
||||||
}
|
}
|
||||||
var field, built bool
|
var field, built bool
|
||||||
|
var fieldVar, handed string
|
||||||
for _, d := range f.Decls {
|
for _, d := range f.Decls {
|
||||||
fd, ok := d.(*ast.FuncDecl)
|
fd, ok := d.(*ast.FuncDecl)
|
||||||
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
|
||||||
@@ -45,6 +51,28 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
|||||||
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
|
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
|
||||||
built = true
|
built = true
|
||||||
}
|
}
|
||||||
|
if id, ok := kv.Value.(*ast.Ident); ok {
|
||||||
|
fieldVar = id.Name
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
// a local variable: built by NewBackupSuccessState, and handed to the collector
|
||||||
|
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||||
|
switch x := n.(type) {
|
||||||
|
case *ast.AssignStmt:
|
||||||
|
for i, l := range x.Lhs {
|
||||||
|
if id, ok := l.(*ast.Ident); ok && fieldVar != "" && id.Name == fieldVar && i < len(x.Rhs) &&
|
||||||
|
callsIn(x.Rhs[i])["backup.NewBackupSuccessState"] {
|
||||||
|
built = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case *ast.CallExpr:
|
||||||
|
if fn, ok := x.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "SetKnownBackupReporter" && len(x.Args) == 1 {
|
||||||
|
if id, ok := x.Args[0].(*ast.Ident); ok {
|
||||||
|
handed = id.Name
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return true
|
return true
|
||||||
@@ -56,6 +84,9 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
|
|||||||
if !built {
|
if !built {
|
||||||
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
|
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
|
||||||
}
|
}
|
||||||
|
if handed == "" || handed != fieldVar {
|
||||||
|
t.Fatalf("the saved backups (%q) are not handed to the collector (SetKnownBackupReporter got %q)", fieldVar, handed)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func callsIn(n ast.Node) map[string]bool {
|
func callsIn(n ast.Node) map[string]bool {
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ func main() {
|
|||||||
|
|
||||||
func run() error {
|
func run() error {
|
||||||
var (
|
var (
|
||||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update")
|
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update")
|
||||||
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
|
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
|
||||||
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
|
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
|
||||||
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
|
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
|
||||||
|
|||||||
@@ -111,15 +111,17 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
|
|||||||
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
||||||
# + container across spaces, spike-confirmed)
|
# + container across spaces, spike-confirmed)
|
||||||
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
||||||
# tee <FIXED image file> — WRITE the ref; content is fed on STDIN (no shell, no interpolation),
|
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
|
||||||
# the agent strict-validates the ref (controllerImageRe) before the write.
|
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
|
||||||
|
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
|
||||||
|
# compromised agent could hand the guest's bootstrap ANY image (sudo cannot see stdin).
|
||||||
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
|
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
|
||||||
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
||||||
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
|
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
|
||||||
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
||||||
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
||||||
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
||||||
/usr/sbin/pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$
|
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
|
||||||
|
|
||||||
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
||||||
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
|
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11).
|
||||||
|
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it.
|
||||||
|
#
|
||||||
|
# One menu entry per installed Proxmox kernel, id `felhom-oneshot-<version>`, booted ONLY when 01_felhom_oneshot found
|
||||||
|
# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1
|
||||||
|
# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into
|
||||||
|
# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes).
|
||||||
|
# It sorts AFTER 10_linux, so it is never entry 0 and never the default.
|
||||||
|
#
|
||||||
|
# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries).
|
||||||
|
set -e
|
||||||
|
prefix="/usr"
|
||||||
|
exec_prefix="/usr"
|
||||||
|
datarootdir="/usr/share"
|
||||||
|
. "$datarootdir/grub/grub-mkconfig_lib"
|
||||||
|
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
|
||||||
|
[ -n "$esp_uuid" ] || exit 0
|
||||||
|
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
|
||||||
|
[ -n "$kernels" ] || exit 0
|
||||||
|
case "${GRUB_DEVICE}" in
|
||||||
|
/dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;;
|
||||||
|
*) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;;
|
||||||
|
esac
|
||||||
|
[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)"
|
||||||
|
rel=$(make_system_path_relative_to_its_root /boot)
|
||||||
|
prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /')
|
||||||
|
for k in $kernels; do
|
||||||
|
[ -f "/boot/initrd.img-$k" ] || continue
|
||||||
|
cat <<EOF
|
||||||
|
menuentry 'Felhom one-shot: $k' --class proxmox --id felhom-oneshot-$k {
|
||||||
|
insmod gzio
|
||||||
|
$prep
|
||||||
|
echo 'Loading Linux $k (felhom one-shot) ...'
|
||||||
|
linux $rel/vmlinuz-$k $root_arg ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
|
||||||
|
initrd $rel/initrd.img-$k
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
done
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# /etc/grub.d/01_felhom_oneshot — the kernel lane's ONE-SHOT boot (R-836, `09` §3 decisions 164 + 172, `11` §5.11).
|
||||||
|
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it; it prints GRUB script.
|
||||||
|
#
|
||||||
|
# At boot, GRUB reads `felhom_next` from an environment block on the ESP (vfat — GRUB can rewrite a file there; it
|
||||||
|
# cannot on the LVM /boot, R-836), CLEARS it, and — only if it names an installed kernel — boots that kernel's one-shot
|
||||||
|
# entry (42_felhom_oneshot) instead of the default. The next boot uses the default again whatever happens: a new kernel
|
||||||
|
# that panics comes back on the old one by itself. felhom-os-apply writes the flag (mode apply) and never the default
|
||||||
|
# for a new kernel. Measured on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
|
||||||
|
# `audits/kernel-spike-2026-10-07/` (candidate 2).
|
||||||
|
#
|
||||||
|
# No vfat ESP at /boot/efi, or no Proxmox kernel → prints nothing (the box boots exactly as before).
|
||||||
|
set -e
|
||||||
|
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
|
||||||
|
[ -n "$esp_uuid" ] || exit 0
|
||||||
|
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
|
||||||
|
[ -n "$kernels" ] || exit 0
|
||||||
|
cat <<EOF
|
||||||
|
# felhom kernel lane: a one-shot kernel named on the ESP, read and cleared before the menu
|
||||||
|
insmod part_gpt
|
||||||
|
insmod fat
|
||||||
|
search --no-floppy --fs-uuid --set=felhom_esp $esp_uuid
|
||||||
|
if [ -f (\$felhom_esp)/EFI/felhom/oneshot.env ]; then
|
||||||
|
load_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
|
||||||
|
if [ "\${felhom_next}" ]; then
|
||||||
|
set felhom_boot="\${felhom_next}"
|
||||||
|
set felhom_next=
|
||||||
|
save_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
|
||||||
|
EOF
|
||||||
|
for k in $kernels; do
|
||||||
|
printf ' if [ "${felhom_boot}" = "%s" ]; then set default="felhom-oneshot-%s"; fi\n' "$k" "$k"
|
||||||
|
done
|
||||||
|
cat <<EOF
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
EOF
|
||||||
@@ -16,8 +16,10 @@ Cmnd_Alias FELHOM_OP_REPAIR = \
|
|||||||
/usr/bin/systemctl reset-failed felhom-sshd, \
|
/usr/bin/systemctl reset-failed felhom-sshd, \
|
||||||
/usr/bin/systemctl restart felhom-sshd, \
|
/usr/bin/systemctl restart felhom-sshd, \
|
||||||
/usr/sbin/pct list, \
|
/usr/sbin/pct list, \
|
||||||
/usr/sbin/pct start [0-9]*, \
|
/usr/sbin/pct ^start [0-9]+$, \
|
||||||
/usr/sbin/pct stop [0-9]*, \
|
/usr/sbin/pct ^stop [0-9]+$, \
|
||||||
/usr/sbin/pct unlock [0-9]*
|
/usr/sbin/pct ^unlock [0-9]+$
|
||||||
|
|
||||||
|
# R-861 (b) B2 (`09` §3 decision 165, hygiene): one numeric vmid per pct verb, anchored — the old glob `[0-9]*` also
|
||||||
|
# matched spaces, so `pct stop 9201 --skiplock 1` passed. Pinned by TestFelhomOpSudoersPctIsExact.
|
||||||
felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR
|
felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR
|
||||||
|
|||||||
+554
-19
@@ -21,6 +21,11 @@
|
|||||||
# or, for an unsigned ring-0 step, the root-owned TRUST_FILE saying `"ring0_slow_lane": true` (set by hand on the demo
|
# or, for an unsigned ring-0 step, the root-owned TRUST_FILE saying `"ring0_slow_lane": true` (set by hand on the demo
|
||||||
# boxes only). The agent's own config is NOT trusted for either: the agent can write it. A Docker step also needs
|
# boxes only). The agent's own config is NOT trusted for either: the agent can write it. A Docker step also needs
|
||||||
# `live-restore` ON (R15) — without it every container restarts.
|
# `live-restore` ON (R15) — without it every container restarts.
|
||||||
|
# Agent v0.151.0 (R-812 option A, `09` §3 decision 163) adds the layer "pve": the HOST's Proxmox USERSPACE packages,
|
||||||
|
# lane "slow" only, origin "Proxmox Debian Repository" only, never a kernel / boot / firmware / microcode name (R14 —
|
||||||
|
# the kernel is R-836's lane), no removal, no undo, a NEW package only from PVE_NEW_ALLOW, an appliance only (R12), and
|
||||||
|
# the same authority as the Docker step (R3: a signed `os_pve_step`, or the root-owned ring-0 mark). Select
|
||||||
|
# "pending-pve" (ring 0): every installed Proxmox-origin package with a pending upgrade, minus HOST_SLOW_RE.
|
||||||
#
|
#
|
||||||
# Modes (plan field "mode"):
|
# Modes (plan field "mode"):
|
||||||
# inventory `apt-get update`, then report what is installed (with origin), what is pending, and health.
|
# inventory `apt-get update`, then report what is installed (with origin), what is pending, and health.
|
||||||
@@ -88,8 +93,13 @@ OOMCHECK_TIMEOUTS = {"image": 10, "clock": 5, "run": 30, "inspect": 10, "events"
|
|||||||
OOMCHECK_SETTLE = 2
|
OOMCHECK_SETTLE = 2
|
||||||
# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
|
# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
|
||||||
# reboot is needed for them to take effect, and a bad one can stop the box from booting.
|
# reboot is needed for them to take effect, and a bad one can stop the box from booting.
|
||||||
|
# proxmox-secure-boot-support is the Secure Boot meta-package: its only job is to pull shim-signed and the signed GRUB,
|
||||||
|
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
||||||
|
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
||||||
|
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
||||||
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
||||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)")
|
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
||||||
|
r"proxmox-secure-boot-support)")
|
||||||
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
||||||
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
||||||
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
||||||
@@ -100,6 +110,12 @@ HOST_SERVICES = ["pveproxy", "pvedaemon", "pvestatd", "pve-cluster", "felhom-age
|
|||||||
DOCKER_NAMES = ("containerd.io", "docker-buildx-plugin", "docker-ce", "docker-ce-cli", "docker-ce-rootless-extras",
|
DOCKER_NAMES = ("containerd.io", "docker-buildx-plugin", "docker-ce", "docker-ce-cli", "docker-ce-rootless-extras",
|
||||||
"docker-compose-plugin")
|
"docker-compose-plugin")
|
||||||
DOCKER_ORIGIN = "Docker CE"
|
DOCKER_ORIGIN = "Docker CE"
|
||||||
|
# The Proxmox package lane (R-812 option A): the origin apt prints for download.proxmox.com, and the ONLY new packages
|
||||||
|
# a pve step may add (measured on demo-felhom 2026-10-07: a full upgrade adds proxmox-firewall-data and the kernel;
|
||||||
|
# the kernel is refused by R14 whatever this list says).
|
||||||
|
PVE_ORIGIN = "Proxmox Debian Repository"
|
||||||
|
PVE_NEW_ALLOW = ("proxmox-firewall-data",)
|
||||||
|
PVE_SIGNED_OP = "os_pve_step"
|
||||||
# ROOT-OWNED trust anchors (the installer writes them; the demo boxes got them by hand, R-840). Never the agent's config.
|
# ROOT-OWNED trust anchors (the installer writes them; the demo boxes got them by hand, R-840). Never the agent's config.
|
||||||
TRUST_FILE = "/etc/felhom/os-trust.json" # {"host_id": "...", "ring0_slow_lane": false}
|
TRUST_FILE = "/etc/felhom/os-trust.json" # {"host_id": "...", "ring0_slow_lane": false}
|
||||||
TRUST_SIGNERS = "/etc/felhom/operator-signers" # ssh allowed_signers: <key_id> namespaces="felhom-op-v1" <key>
|
TRUST_SIGNERS = "/etc/felhom/operator-signers" # ssh allowed_signers: <key_id> namespaces="felhom-op-v1" <key>
|
||||||
@@ -113,6 +129,32 @@ DAEMON_JSON = "/etc/docker/daemon.json"
|
|||||||
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
|
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
|
||||||
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
|
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
|
||||||
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
|
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
|
||||||
|
# ---------- the kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11) ----------
|
||||||
|
# A new kernel boots ONCE through a flag in a GRUB environment block on the ESP (vfat — GRUB can rewrite it there; on
|
||||||
|
# the LVM /boot it cannot, R-836). The bundle's two GRUB generators read and clear the flag (01_felhom_oneshot) and
|
||||||
|
# give each installed kernel a one-shot entry with the lockup-to-panic options (42_felhom_oneshot, option C). The GRUB
|
||||||
|
# default is the kernel the box RUNS, pinned in KERNEL_DEFAULT_CFG; only "kernel-good" (after a healthy one-shot boot)
|
||||||
|
# moves it to the new kernel. Measured in the spike on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
|
||||||
|
# `audits/kernel-spike-2026-10-07/`.
|
||||||
|
KERNEL_OP = "os_kernel_step"
|
||||||
|
KERNEL_STATE = "/var/lib/felhom-kernel/state.json" # 0644 root: the step's phase (the agent reads it)
|
||||||
|
KERNEL_DEFAULT_CFG = "/etc/default/grub.d/zz-felhom-kernel-default.cfg" # sourced last: GRUB_DEFAULT = this kernel
|
||||||
|
ONESHOT_SNIPPET = "/etc/grub.d/01_felhom_oneshot" # bundle-owned: read + clear the flag, pick the entry
|
||||||
|
ONESHOT_ENTRIES = "/etc/grub.d/42_felhom_oneshot" # bundle-owned: the one-shot entries (option C options)
|
||||||
|
GRUB_CFG = "/boot/grub/grub.cfg"
|
||||||
|
ESP_MOUNT = "/boot/efi"
|
||||||
|
ONESHOT_ENV = ESP_MOUNT + "/EFI/felhom/oneshot.env"
|
||||||
|
ONESHOT_ARGS = "softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10"
|
||||||
|
KERNEL_PIN_FILE = "/etc/kernel/proxmox-boot-pin" # an operator's `proxmox-boot-tool kernel pin` — never fought
|
||||||
|
KVER_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$")
|
||||||
|
KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)(-signed)?$")
|
||||||
|
# the packages a kernel step may UPGRADE (never add): the kernel series meta-package, the default-kernel meta, the boot
|
||||||
|
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
|
||||||
|
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
|
||||||
|
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
|
||||||
|
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
|
||||||
|
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
|
||||||
|
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||||
|
|
||||||
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
||||||
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
||||||
@@ -159,6 +201,10 @@ BUNDLE_FILES = [
|
|||||||
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
|
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
|
||||||
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
|
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
|
||||||
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
|
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
|
||||||
|
# the kernel lane's two GRUB generators (R-836, `11` §5.11): they take effect at the next update-grub, which the
|
||||||
|
# first kernel step runs itself; with no flag on the ESP they change nothing about how the box boots.
|
||||||
|
("/etc/grub.d/01_felhom_oneshot", "felhom-grub-oneshot.sh", 0o755, "sh", "replace"),
|
||||||
|
("/etc/grub.d/42_felhom_oneshot", "felhom-grub-oneshot-entries.sh", 0o755, "sh", "replace"),
|
||||||
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
|
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
|
||||||
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
|
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
|
||||||
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
|
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
|
||||||
@@ -423,7 +469,8 @@ class Apply:
|
|||||||
|
|
||||||
def check_plan(self, plan):
|
def check_plan(self, plan):
|
||||||
mode = plan.get("mode", "apply")
|
mode = plan.get("mode", "apply")
|
||||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check"):
|
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
|
||||||
|
+ KERNEL_MODES:
|
||||||
raise Refused("R11", f"unknown mode {mode!r}")
|
raise Refused("R11", f"unknown mode {mode!r}")
|
||||||
if mode == "agent_update":
|
if mode == "agent_update":
|
||||||
if plan.get("layer") != "host":
|
if plan.get("layer") != "host":
|
||||||
@@ -434,12 +481,18 @@ class Apply:
|
|||||||
raise Refused("R11", "bundle is a host-layer mode")
|
raise Refused("R11", "bundle is a host-layer mode")
|
||||||
return mode, "host", 0, "bundle"
|
return mode, "host", 0, "bundle"
|
||||||
layer = plan.get("layer")
|
layer = plan.get("layer")
|
||||||
if layer not in ("guest", "host", "docker"):
|
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
||||||
raise Refused("R12", f"layer {layer!r} is not guest, host or docker")
|
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
||||||
|
if (mode in KERNEL_MODES) != (layer == "kernel" and mode not in ("apply", "health")):
|
||||||
|
raise Refused("R11", f"mode {mode!r} does not fit layer {layer!r}")
|
||||||
lane = plan.get("lane", "fast")
|
lane = plan.get("lane", "fast")
|
||||||
if layer == "docker" and lane != "slow":
|
if layer == "docker" and lane != "slow":
|
||||||
raise Refused("R3", "the Docker engine is the slow lane (`11` §5.8); a fast-lane Docker plan is refused")
|
raise Refused("R3", "the Docker engine is the slow lane (`11` §5.8); a fast-lane Docker plan is refused")
|
||||||
if layer != "docker" and lane != "fast":
|
if layer == "pve" and lane != "slow":
|
||||||
|
raise Refused("R3", "the Proxmox packages are the slow lane (`11` §5.10); a fast-lane pve plan is refused")
|
||||||
|
if layer == "kernel" and lane != "slow":
|
||||||
|
raise Refused("R3", "the kernel is the slow lane (`11` §5.11); a fast-lane kernel plan is refused")
|
||||||
|
if layer not in ("docker", "pve", "kernel") and lane != "fast":
|
||||||
raise Refused("R3", f"the {layer} layer has no slow lane in this release (kernel, Proxmox: `11` §8 step 6)")
|
raise Refused("R3", f"the {layer} layer has no slow lane in this release (kernel, Proxmox: `11` §8 step 6)")
|
||||||
if mode == "facts" and layer != "host":
|
if mode == "facts" and layer != "host":
|
||||||
raise Refused("R11", "facts is a host-layer mode (it reads the host and the guest)")
|
raise Refused("R11", "facts is a host-layer mode (it reads the host and the guest)")
|
||||||
@@ -447,10 +500,12 @@ class Apply:
|
|||||||
raise Refused("R11", "live-restore-on is a guest-layer mode")
|
raise Refused("R11", "live-restore-on is a guest-layer mode")
|
||||||
if mode == "oom-check" and layer != "docker":
|
if mode == "oom-check" and layer != "docker":
|
||||||
raise Refused("R11", "oom-check is a docker-layer mode (it checks the guest's Docker engine)")
|
raise Refused("R11", "oom-check is a docker-layer mode (it checks the guest's Docker engine)")
|
||||||
if plan.get("undo") and layer != "docker":
|
if plan.get("undo") and layer != "docker": # the pve layer has no undo in this release (R-812 option A)
|
||||||
raise Refused("R5", "an undo (downgrade) exists only for the Docker layer, inside a signed job")
|
raise Refused("R5", "an undo (downgrade) exists only for the Docker layer, inside a signed job")
|
||||||
vmid = plan.get("vmid")
|
vmid = plan.get("vmid")
|
||||||
if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
|
if layer == "kernel" and mode in KERNEL_MODES and mode != "kernel-reboot" and vmid == 0:
|
||||||
|
pass # after a boot the guest may not run (that is what is judged); these modes never touch it
|
||||||
|
elif not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
|
||||||
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
|
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
|
||||||
rid = plan.get("release_id", "")
|
rid = plan.get("release_id", "")
|
||||||
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
||||||
@@ -458,17 +513,24 @@ class Apply:
|
|||||||
if plan.get("allow_new"):
|
if plan.get("allow_new"):
|
||||||
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
||||||
select = plan.get("select", "listed")
|
select = plan.get("select", "listed")
|
||||||
if select not in ("listed", "pending-fast", "pending-docker"):
|
if select not in ("listed", "pending-fast", "pending-docker", "pending-pve", "pending-kernel"):
|
||||||
raise Refused("R11", f"unknown select {select!r}")
|
raise Refused("R11", f"unknown select {select!r}")
|
||||||
if (select == "pending-docker") != (layer == "docker" and select != "listed"):
|
if (select == "pending-docker") != (layer == "docker" and select != "listed"):
|
||||||
if select == "pending-docker" or layer == "docker":
|
if select == "pending-docker" or layer == "docker":
|
||||||
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||||
|
if (select == "pending-pve") != (layer == "pve" and select != "listed"):
|
||||||
|
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||||
|
if (select == "pending-kernel") != (layer == "kernel" and select != "listed"):
|
||||||
|
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||||
|
ek = plan.get("expect_kver")
|
||||||
|
if ek is not None and (layer != "kernel" or not isinstance(ek, str) or not KVER_RE.match(ek)):
|
||||||
|
raise Refused("R11", f"expect_kver {ek!r} is not a kernel version of the kernel layer")
|
||||||
pk = plan.get("packages", [])
|
pk = plan.get("packages", [])
|
||||||
if not isinstance(pk, list):
|
if not isinstance(pk, list):
|
||||||
raise Refused("R11", "packages must be a list")
|
raise Refused("R11", "packages must be a list")
|
||||||
if mode == "apply" and select == "listed" and not pk:
|
if mode == "apply" and select == "listed" and not pk:
|
||||||
raise Refused("R11", "packages must be a non-empty list in apply mode (select listed)")
|
raise Refused("R11", "packages must be a non-empty list in apply mode (select listed)")
|
||||||
if select in ("pending-fast", "pending-docker") and pk:
|
if select in ("pending-fast", "pending-docker", "pending-pve", "pending-kernel") and pk:
|
||||||
raise Refused("R11", f"select {select} takes no package list")
|
raise Refused("R11", f"select {select} takes no package list")
|
||||||
seen = set()
|
seen = set()
|
||||||
for e in pk:
|
for e in pk:
|
||||||
@@ -488,6 +550,19 @@ class Apply:
|
|||||||
continue
|
continue
|
||||||
if n in DOCKER_NAMES:
|
if n in DOCKER_NAMES:
|
||||||
raise Refused("R2", f"{n} is a Docker package — the slow lane (`11` §5.8), never in a {layer} plan")
|
raise Refused("R2", f"{n} is a Docker package — the slow lane (`11` §5.8), never in a {layer} plan")
|
||||||
|
if layer == "kernel":
|
||||||
|
if o != PVE_ORIGIN:
|
||||||
|
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the kernel layer)")
|
||||||
|
if not (KERNEL_UPGRADE_RE.match(n) or KERNEL_IMAGE_RE.match(n)):
|
||||||
|
raise Refused("R23", f"{n} is not a kernel package (a kernel image, the kernel meta-packages, "
|
||||||
|
f"proxmox-kernel-helper or pve-firmware)")
|
||||||
|
continue
|
||||||
|
if layer == "pve":
|
||||||
|
if o != PVE_ORIGIN:
|
||||||
|
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the pve layer)")
|
||||||
|
if HOST_SLOW_RE.match(n):
|
||||||
|
raise Refused("R14", f"{n} is a kernel / boot / firmware package — never the pve lane (R-836)")
|
||||||
|
continue
|
||||||
if o not in FAST_ORIGINS:
|
if o not in FAST_ORIGINS:
|
||||||
raise Refused("R2", f"{n}: origin {o!r} is not Debian / Debian-Security (the fast lane, `11` C3)")
|
raise Refused("R2", f"{n}: origin {o!r} is not Debian / Debian-Security (the fast lane, `11` C3)")
|
||||||
if layer == "host" and HOST_SLOW_RE.match(n):
|
if layer == "host" and HOST_SLOW_RE.match(n):
|
||||||
@@ -630,12 +705,13 @@ class Apply:
|
|||||||
now = self.r.now()
|
now = self.r.now()
|
||||||
self.r.write_nonces({k: v for k, v in seen.items() if v > now})
|
self.r.write_nonces({k: v for k, v in seen.items() if v > now})
|
||||||
|
|
||||||
def docker_authority(self, plan):
|
def docker_authority(self, plan, op_name=SIGNED_OP):
|
||||||
"""R3 for the docker layer: returns (who, undo). A signed job binds the EXACT package list and the undo flag."""
|
"""R3 for the docker (and, op_name os_pve_step, the pve) layer: returns (who, undo). A signed job binds the EXACT
|
||||||
|
package list and the undo flag."""
|
||||||
trust = self.load_trust()
|
trust = self.load_trust()
|
||||||
signed = plan.get("signed")
|
signed = plan.get("signed")
|
||||||
if signed:
|
if signed:
|
||||||
params = self.verify_signed(signed, trust)
|
params = self.verify_signed(signed, trust, op_name=op_name)
|
||||||
want = sorted(f"{e.get('name')}={e.get('version')}" for e in params.get("packages") or [])
|
want = sorted(f"{e.get('name')}={e.get('version')}" for e in params.get("packages") or [])
|
||||||
got = sorted(f"{e['name']}={e['version']}" for e in plan.get("packages", []))
|
got = sorted(f"{e['name']}={e['version']}" for e in plan.get("packages", []))
|
||||||
if not want or want != got:
|
if not want or want != got:
|
||||||
@@ -649,7 +725,7 @@ class Apply:
|
|||||||
raise Refused("R3", "an undo (downgrade) needs a signed operator job")
|
raise Refused("R3", "an undo (downgrade) needs a signed operator job")
|
||||||
if trust.get("ring0_slow_lane") is True:
|
if trust.get("ring0_slow_lane") is True:
|
||||||
return "ring0", False
|
return "ring0", False
|
||||||
raise Refused("R3", "a Docker step needs a signed operator job (ring 1) or this box's root-owned ring-0 mark")
|
raise Refused("R3", f"a {self.layer} slow-lane step needs a signed operator job (ring 1) or this box's root-owned ring-0 mark")
|
||||||
|
|
||||||
def live_restore(self):
|
def live_restore(self):
|
||||||
rc, out, _ = self.g(["docker", "info", "--format", "{{.LiveRestoreEnabled}}"], timeout=60)
|
rc, out, _ = self.g(["docker", "info", "--format", "{{.LiveRestoreEnabled}}"], timeout=60)
|
||||||
@@ -737,6 +813,17 @@ class Apply:
|
|||||||
return v or "unknown"
|
return v or "unknown"
|
||||||
h = {"debian": first(["cat", "/etc/debian_version"]), "kernel_running": first(["uname", "-r"])}
|
h = {"debian": first(["cat", "/etc/debian_version"]), "kernel_running": first(["uname", "-r"])}
|
||||||
h["kernel_next_boot"], h["kernel_next_boot_source"] = self.kernel_next_boot()
|
h["kernel_next_boot"], h["kernel_next_boot_source"] = self.kernel_next_boot()
|
||||||
|
try:
|
||||||
|
# the kernel lane (R-836): the default and the one-shot flag, read from grub.cfg and the ESP themselves
|
||||||
|
kv = Kernel(self, {}).view()
|
||||||
|
kv["setup_problems"] = Kernel(self, {}).setup_problems()
|
||||||
|
h["kernel_lane"] = kv
|
||||||
|
if kv.get("flag"):
|
||||||
|
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["flag"], "felhom one-shot flag (once; then the default)"
|
||||||
|
elif kv.get("default") not in (None, "unknown"):
|
||||||
|
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["default"], "grub.cfg default"
|
||||||
|
except Exception as e: # never cost the System page its other facts
|
||||||
|
h["kernel_lane"] = {"error": str(e)[:200]}
|
||||||
rc, out, _ = self.r.host(["apt-mark", "showhold"], 60)
|
rc, out, _ = self.r.host(["apt-mark", "showhold"], 60)
|
||||||
h["held"] = sorted(out.split()) if rc == 0 else None
|
h["held"] = sorted(out.split()) if rc == 0 else None
|
||||||
try:
|
try:
|
||||||
@@ -795,8 +882,8 @@ class Apply:
|
|||||||
|
|
||||||
# ---------- target helpers ----------
|
# ---------- target helpers ----------
|
||||||
def x(self, argv, timeout=1800):
|
def x(self, argv, timeout=1800):
|
||||||
"""Run in the TARGET layer: the guest via pct exec, or the host directly."""
|
"""Run in the TARGET layer: the guest via pct exec, or the host directly (host, pve and kernel)."""
|
||||||
if self.layer == "host":
|
if self.layer in ("host", "pve", "kernel"):
|
||||||
return self.r.host(argv, timeout)
|
return self.r.host(argv, timeout)
|
||||||
return self.r.guest(self.vmid, argv, timeout) # guest and docker both live in the customer guest
|
return self.r.guest(self.vmid, argv, timeout) # guest and docker both live in the customer guest
|
||||||
|
|
||||||
@@ -891,7 +978,7 @@ class Apply:
|
|||||||
def restart_needed(self):
|
def restart_needed(self):
|
||||||
"""Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
|
"""Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
|
||||||
LXC guests (the host's /proc shows guest processes too)."""
|
LXC guests (the host's /proc shows guest processes too)."""
|
||||||
skip = RESTART_SKIP_CGROUP["host" if self.layer == "host" else "guest"]
|
skip = RESTART_SKIP_CGROUP["host" if self.layer in ("host", "pve", "kernel") else "guest"]
|
||||||
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
|
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
|
||||||
'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
|
'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
|
||||||
rc, out, _ = self.x(["sh", "-c", script], timeout=120)
|
rc, out, _ = self.x(["sh", "-c", script], timeout=120)
|
||||||
@@ -965,8 +1052,12 @@ class Apply:
|
|||||||
return Bundle(self).from_plan(plan)
|
return Bundle(self).from_plan(plan)
|
||||||
if self.mode == "agent_update":
|
if self.mode == "agent_update":
|
||||||
return self.agent_update(plan)
|
return self.agent_update(plan)
|
||||||
if self.layer == "host":
|
if self.layer in ("host", "pve", "kernel"):
|
||||||
self.check_appliance()
|
self.check_appliance()
|
||||||
|
if self.layer == "kernel" and self.mode != "health":
|
||||||
|
# the kernel lane's own modes (R-836): most of them run while the guest is still starting after a boot, so
|
||||||
|
# the guest check is the stage's and the reboot's own (Kernel.run), not every mode's
|
||||||
|
return Kernel(self, plan).run()
|
||||||
self.check_guest(self.vmid)
|
self.check_guest(self.vmid)
|
||||||
log = self.r.log
|
log = self.r.log
|
||||||
if self.mode == "live-restore-on":
|
if self.mode == "live-restore-on":
|
||||||
@@ -976,6 +1067,9 @@ class Apply:
|
|||||||
self.report["oom_check"] = self.oom_check()
|
self.report["oom_check"] = self.oom_check()
|
||||||
return 0
|
return 0
|
||||||
self.who, self.allow_downgrade = ("fast", False)
|
self.who, self.allow_downgrade = ("fast", False)
|
||||||
|
if self.layer == "pve" and self.mode == "apply":
|
||||||
|
self.who, self.allow_downgrade = self.docker_authority(plan, op_name=PVE_SIGNED_OP)
|
||||||
|
self.report["authority"] = self.who
|
||||||
if self.layer == "docker" and self.mode == "apply":
|
if self.layer == "docker" and self.mode == "apply":
|
||||||
self.who, self.allow_downgrade = self.docker_authority(plan)
|
self.who, self.allow_downgrade = self.docker_authority(plan)
|
||||||
if self.live_restore() != "true":
|
if self.live_restore() != "true":
|
||||||
@@ -988,7 +1082,7 @@ class Apply:
|
|||||||
log(f"os-apply: START release={plan.get('release_id')} layer={self.layer}" +
|
log(f"os-apply: START release={plan.get('release_id')} layer={self.layer}" +
|
||||||
(f":{self.vmid}" if self.layer != "host" else "") +
|
(f":{self.vmid}" if self.layer != "host" else "") +
|
||||||
f" lane={plan.get('lane', 'fast')} mode={self.mode} select={self.select} packages={len(plan.get('packages', []))}" +
|
f" lane={plan.get('lane', 'fast')} mode={self.mode} select={self.select} packages={len(plan.get('packages', []))}" +
|
||||||
(f" authority={self.who}{' UNDO' if self.allow_downgrade else ''}" if self.layer == "docker" else ""))
|
(f" authority={self.who}{' UNDO' if self.allow_downgrade else ''}" if self.layer in ("docker", "pve") else ""))
|
||||||
if self.apt_lock_held():
|
if self.apt_lock_held():
|
||||||
raise Refused("R9", f"another apt/dpkg holds the lock on the {self.layer}")
|
raise Refused("R9", f"another apt/dpkg holds the lock on the {self.layer}")
|
||||||
self.report["health_before"] = self.health()
|
self.report["health_before"] = self.health()
|
||||||
@@ -1012,6 +1106,8 @@ class Apply:
|
|||||||
if self.layer == "docker":
|
if self.layer == "docker":
|
||||||
rc_v, out_v, _ = self.g(["docker", "version", "--format", "{{.Server.Version}}"], timeout=60)
|
rc_v, out_v, _ = self.g(["docker", "version", "--format", "{{.Server.Version}}"], timeout=60)
|
||||||
self.report["docker_engine"] = out_v.strip() if rc_v == 0 and out_v.strip() else "unknown"
|
self.report["docker_engine"] = out_v.strip() if rc_v == 0 and out_v.strip() else "unknown"
|
||||||
|
if self.layer == "pve":
|
||||||
|
self.report["pve_manager"] = self.pve_manager()
|
||||||
self.report["reboot_scanned"] = "reboot_needed" in self.report
|
self.report["reboot_scanned"] = "reboot_needed" in self.report
|
||||||
self.report["health_after"] = self.health()
|
self.report["health_after"] = self.health()
|
||||||
if self.layer == "docker" and self.mode == "apply":
|
if self.layer == "docker" and self.mode == "apply":
|
||||||
@@ -1161,10 +1257,26 @@ class Apply:
|
|||||||
return [{"name": p["name"], "version": p["to"], "origin": DOCKER_ORIGIN} for p in pend
|
return [{"name": p["name"], "version": p["to"], "origin": DOCKER_ORIGIN} for p in pend
|
||||||
if p["from"] is not None and p["name"] in DOCKER_NAMES and self.origin_name(p["origin"]) == {DOCKER_ORIGIN}]
|
if p["from"] is not None and p["name"] in DOCKER_NAMES and self.origin_name(p["origin"]) == {DOCKER_ORIGIN}]
|
||||||
|
|
||||||
|
def pending_pve(self):
|
||||||
|
"""Ring 0 (select pending-pve): the newest pending version of each INSTALLED Proxmox-origin package, never a
|
||||||
|
kernel / boot / firmware / microcode name (HOST_SLOW_RE, R-836's lane), never another origin."""
|
||||||
|
rc, pend, remv, _ = self.simulate(["dist-upgrade"])
|
||||||
|
return [{"name": p["name"], "version": p["to"], "origin": PVE_ORIGIN} for p in pend
|
||||||
|
if p["from"] is not None and not HOST_SLOW_RE.match(p["name"]) and p["name"] not in DOCKER_NAMES
|
||||||
|
and self.origin_name(p["origin"]) == {PVE_ORIGIN}]
|
||||||
|
|
||||||
|
def pve_manager(self):
|
||||||
|
"""pveversion's pve-manager version ("unknown" when it cannot be read)."""
|
||||||
|
rc, out, _ = self.r.host(["pveversion"], 60)
|
||||||
|
m = re.match(r"^pve-manager/([^/\s]+)", out.strip()) if rc == 0 else None
|
||||||
|
return m.group(1) if m else "unknown"
|
||||||
|
|
||||||
def origin_ok(self, origin):
|
def origin_ok(self, origin):
|
||||||
o = self.origin_name(origin)
|
o = self.origin_name(origin)
|
||||||
if self.layer == "docker":
|
if self.layer == "docker":
|
||||||
return o == {DOCKER_ORIGIN}
|
return o == {DOCKER_ORIGIN}
|
||||||
|
if self.layer == "pve":
|
||||||
|
return o == {PVE_ORIGIN}
|
||||||
return bool(o & set(FAST_ORIGINS))
|
return bool(o & set(FAST_ORIGINS))
|
||||||
|
|
||||||
def apply(self, plan):
|
def apply(self, plan):
|
||||||
@@ -1173,6 +1285,8 @@ class Apply:
|
|||||||
packages = plan["packages"]
|
packages = plan["packages"]
|
||||||
elif self.select == "pending-docker":
|
elif self.select == "pending-docker":
|
||||||
packages = self.pending_docker()
|
packages = self.pending_docker()
|
||||||
|
elif self.select == "pending-pve":
|
||||||
|
packages = self.pending_pve()
|
||||||
else:
|
else:
|
||||||
packages = self.pending_fast()
|
packages = self.pending_fast()
|
||||||
cmp_op = "ne" if self.allow_downgrade else "gt"
|
cmp_op = "ne" if self.allow_downgrade else "gt"
|
||||||
@@ -1221,6 +1335,11 @@ class Apply:
|
|||||||
raise Refused("R4", f"the plan would remove {', '.join(remv[:5])}")
|
raise Refused("R4", f"the plan would remove {', '.join(remv[:5])}")
|
||||||
want = dict(upgrade)
|
want = dict(upgrade)
|
||||||
for p in sim:
|
for p in sim:
|
||||||
|
if p["from"] is None and self.layer == "pve" and p["name"] in PVE_NEW_ALLOW and self.origin_ok(p["origin"]) \
|
||||||
|
and not HOST_SLOW_RE.match(p["name"]):
|
||||||
|
self.r.log(f"os-apply: NEW {p['name']}={p['to']} (on the pve lane's allow-list)")
|
||||||
|
self.report.setdefault("added", []).append({"name": p["name"], "version": p["to"]})
|
||||||
|
continue
|
||||||
if p["from"] is None:
|
if p["from"] is None:
|
||||||
raise Refused("R6", f"the plan would add a package that is not installed: {p['name']}")
|
raise Refused("R6", f"the plan would add a package that is not installed: {p['name']}")
|
||||||
if p["name"] not in want:
|
if p["name"] not in want:
|
||||||
@@ -1231,7 +1350,7 @@ class Apply:
|
|||||||
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
|
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
|
||||||
if not self.origin_ok(p["origin"]):
|
if not self.origin_ok(p["origin"]):
|
||||||
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not the {self.layer} layer's origin")
|
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not the {self.layer} layer's origin")
|
||||||
if self.layer == "host" and HOST_SLOW_RE.match(p["name"]):
|
if self.layer in ("host", "pve") and HOST_SLOW_RE.match(p["name"]):
|
||||||
raise Refused("R14", f"{p['name']} is a kernel / boot / firmware package — the host's slow lane")
|
raise Refused("R14", f"{p['name']} is a kernel / boot / firmware package — the host's slow lane")
|
||||||
need = self.download_bytes(args)
|
need = self.download_bytes(args)
|
||||||
free = self.free_bytes()
|
free = self.free_bytes()
|
||||||
@@ -1314,6 +1433,422 @@ class Apply:
|
|||||||
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||||
|
|
||||||
|
|
||||||
|
def _iso(t):
|
||||||
|
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t))
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_iso(s):
|
||||||
|
try:
|
||||||
|
return calendar.timegm(time.strptime(s, "%Y-%m-%dT%H:%M:%SZ"))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class Kernel:
|
||||||
|
"""The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). Refusal codes: R3 (authority), R4 (removal),
|
||||||
|
R6 (a package outside the step), R8 (space), R9 (locks), R12 (appliance), R20 (the box's boot setup cannot do a
|
||||||
|
one-shot), R21 (the crash guard is tripped or saw an unclean boot within its window), R22 (the step's phase does not
|
||||||
|
allow this mode), R23 (the kernel set is not one exact new kernel).
|
||||||
|
|
||||||
|
Modes (plan "mode", layer "kernel", lane "slow"):
|
||||||
|
apply STAGE: install the kernel set, keep the GRUB default on the kernel the box runs, write the flag.
|
||||||
|
Never reboots. select "pending-kernel" (ring 0, the root-owned mark) or "listed" (a signed
|
||||||
|
os_kernel_step). expect_kver: the kernel the hub told the household about — any other is R23.
|
||||||
|
kernel-reboot a STAGED step's reboot (the night leg, after the household was told): phase oneshot, then reboot.
|
||||||
|
kernel-boot after a boot: what became of the step (judging | fell_back | self_reverted | revert_failed).
|
||||||
|
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
|
||||||
|
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
|
||||||
|
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
|
||||||
|
kernel-status read only."""
|
||||||
|
|
||||||
|
def __init__(self, apply, plan):
|
||||||
|
self.a, self.r, self.plan = apply, apply.r, plan
|
||||||
|
self.report = apply.report
|
||||||
|
self.log = apply.r.log
|
||||||
|
|
||||||
|
# ---------- reading the box ----------
|
||||||
|
def running(self):
|
||||||
|
rc, out, _ = self.r.host(["uname", "-r"], 30)
|
||||||
|
v = out.strip() if rc == 0 else ""
|
||||||
|
return v if KVER_RE.match(v) else ""
|
||||||
|
|
||||||
|
def state(self):
|
||||||
|
try:
|
||||||
|
s = json.loads(self.r.read_file(KERNEL_STATE))
|
||||||
|
return s if isinstance(s, dict) else {}
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
|
||||||
|
def save_state(self, s):
|
||||||
|
s["updated_at"] = _iso(self.r.now())
|
||||||
|
self.r.put_file(KERNEL_STATE, (json.dumps(s, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||||
|
|
||||||
|
def flag(self):
|
||||||
|
"""The one-shot flag: the kernel named, "" when the env block holds none, None when there is no env block."""
|
||||||
|
rc, out, _ = self.r.host(["grub-editenv", ONESHOT_ENV, "list"], 30)
|
||||||
|
if rc != 0:
|
||||||
|
return None
|
||||||
|
for l in out.splitlines():
|
||||||
|
if l.startswith("felhom_next="):
|
||||||
|
return l.split("=", 1)[1].strip()
|
||||||
|
return ""
|
||||||
|
|
||||||
|
def grub_cfg(self):
|
||||||
|
try:
|
||||||
|
return self.r.read_file(GRUB_CFG)
|
||||||
|
except OSError:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def default_kver(cfg):
|
||||||
|
"""The kernel grub.cfg boots by default (00_header's `set default=`), or "unknown"."""
|
||||||
|
m = re.search(r'^\s*set default="(?:gnulinux-advanced-[^>"]*>)?gnulinux-([0-9][^"]*?-pve)-advanced-[^"]*"', cfg, re.M)
|
||||||
|
return m.group(1) if m else "unknown"
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def entry_id(cfg, kver):
|
||||||
|
"""The GRUB_DEFAULT value that names kver's normal entry, read from grub.cfg itself (10_linux's ids)."""
|
||||||
|
sub = re.search(r"\$menuentry_id_option '(gnulinux-advanced-[^']+)'", cfg)
|
||||||
|
m = re.search(r"\$menuentry_id_option '(gnulinux-" + re.escape(kver) + r"-advanced-[^']+)'", cfg)
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
return f"{sub.group(1)}>{m.group(1)}" if sub else m.group(1)
|
||||||
|
|
||||||
|
def setup_problems(self):
|
||||||
|
"""R20: why this box cannot do a one-shot boot (empty = it can). Measured shape: UEFI, a vfat ESP at /boot/efi,
|
||||||
|
/boot on the root filesystem, GRUB with fat + loadenv, the bundle's two generators, no hand pin."""
|
||||||
|
why = []
|
||||||
|
if not self.r.lexists("/sys/firmware/efi"):
|
||||||
|
why.append("the box does not boot UEFI")
|
||||||
|
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "FSTYPE", ESP_MOUNT], 30)
|
||||||
|
if rc != 0 or out.strip() != "vfat":
|
||||||
|
why.append(f"{ESP_MOUNT} is not a mounted vfat ESP ({out.strip() or 'not mounted'})")
|
||||||
|
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "TARGET", "/boot"], 30)
|
||||||
|
if rc == 0 and out.strip():
|
||||||
|
why.append("/boot is a separate filesystem (the one-shot entries assume /boot on the root filesystem)")
|
||||||
|
for m in ("fat", "loadenv"):
|
||||||
|
if not self.r.lexists(f"/usr/lib/grub/x86_64-efi/{m}.mod"):
|
||||||
|
why.append(f"GRUB has no {m} module")
|
||||||
|
for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES):
|
||||||
|
if not self.r.lexists(p):
|
||||||
|
why.append(f"{p} is missing (the config bundle installs it)")
|
||||||
|
if self.r.lexists(KERNEL_PIN_FILE):
|
||||||
|
why.append("a kernel is pinned by hand (proxmox-boot-tool kernel pin) — the lane never fights it")
|
||||||
|
return why
|
||||||
|
|
||||||
|
def guard(self):
|
||||||
|
try:
|
||||||
|
return json.loads(self.r.read_file(CRASH_GUARD_STATE))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def check_guard(self):
|
||||||
|
"""R21: a kernel step only on a box whose crash guard is armed and saw no unclean boot within its window — so
|
||||||
|
the step's own reboots (clean), one crash and one self-revert (clean) can never reach the 3rd unclean boot that
|
||||||
|
leaves the box off (`11` §5.9). Pinned by test_felhom_crash_guard KernelStepCannotLeaveTheBoxOff."""
|
||||||
|
g = self.guard()
|
||||||
|
if not isinstance(g, dict):
|
||||||
|
raise Refused("R21", f"no crash guard state ({CRASH_GUARD_STATE}) — a kernel step needs the guard")
|
||||||
|
if g.get("tripped") or not g.get("armed"):
|
||||||
|
raise Refused("R21", "the crash guard is tripped — no kernel step until it re-arms")
|
||||||
|
if (g.get("unclean_boots_in_window") or 0) > 0:
|
||||||
|
raise Refused("R21", f"{g.get('unclean_boots_in_window')} unclean boot(s) within the guard's window — wait")
|
||||||
|
|
||||||
|
def view(self, st=None, cfg=None):
|
||||||
|
st = self.state() if st is None else st
|
||||||
|
cfg = self.grub_cfg() if cfg is None else cfg
|
||||||
|
return {"running": self.running() or "unknown", "default": self.default_kver(cfg), "flag": self.flag(),
|
||||||
|
"phase": st.get("phase", "none"), "from": st.get("from"), "to": st.get("to"),
|
||||||
|
"step_id": st.get("step_id"), "self_revert_used": bool(st.get("self_revert_used")), "vmid": st.get("vmid"),
|
||||||
|
"staged_at": st.get("staged_at"), "rebooted_at": st.get("rebooted_at"),
|
||||||
|
"result_at": st.get("result_at"), "reason": st.get("reason")}
|
||||||
|
|
||||||
|
# ---------- writing the box ----------
|
||||||
|
def write_default(self, kver):
|
||||||
|
"""Pin the GRUB default to kver's normal entry, regenerate grub.cfg, and PROVE it (the default read back)."""
|
||||||
|
cfg = self.grub_cfg()
|
||||||
|
eid = self.entry_id(cfg, kver)
|
||||||
|
if not eid:
|
||||||
|
raise Refused("R20", f"grub.cfg has no normal entry for {kver}")
|
||||||
|
body = ("# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily\n"
|
||||||
|
f'GRUB_DEFAULT="{eid}"\n')
|
||||||
|
self.r.put_file(KERNEL_DEFAULT_CFG, body.encode(), 0o644)
|
||||||
|
rc, out, err = self.r.host(["update-grub"], 300)
|
||||||
|
got = self.default_kver(self.grub_cfg())
|
||||||
|
if rc != 0 or got != kver:
|
||||||
|
raise Refused("R20", f"update-grub rc={rc}: the default reads {got}, not {kver}: {(out + err).strip()[-200:]}")
|
||||||
|
self.log(f"os-apply: KERNEL default = {kver} (proved from grub.cfg)")
|
||||||
|
|
||||||
|
def set_flag(self, kver):
|
||||||
|
self.r.host(["mkdir", "-p", os.path.dirname(ONESHOT_ENV)], 30)
|
||||||
|
if self.flag() is None:
|
||||||
|
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "create"], 30)
|
||||||
|
if rc != 0:
|
||||||
|
raise Refused("R20", f"cannot create the one-shot env block on the ESP: {(out + err).strip()[-200:]}")
|
||||||
|
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "set", f"felhom_next={kver}"], 30)
|
||||||
|
if rc != 0 or self.flag() != kver:
|
||||||
|
raise Refused("R20", f"the one-shot flag did not read back as {kver}: {(out + err).strip()[-200:]}")
|
||||||
|
|
||||||
|
def clear_flag(self):
|
||||||
|
if self.flag():
|
||||||
|
self.r.host(["grub-editenv", ONESHOT_ENV, "unset", "felhom_next"], 30)
|
||||||
|
|
||||||
|
def reboot(self, why):
|
||||||
|
self.log(f"os-apply: KERNEL REBOOT — {why}")
|
||||||
|
rc, out, err = self.r.host(["systemctl", "reboot"], 60)
|
||||||
|
self.report["reboot_rc"] = rc
|
||||||
|
if rc != 0:
|
||||||
|
self.report["failed"] = {"rc": 3, "step": "reboot", "reason": (out + err).strip()[-200:]}
|
||||||
|
return 3
|
||||||
|
return 0
|
||||||
|
|
||||||
|
# ---------- the modes ----------
|
||||||
|
def run(self):
|
||||||
|
mode = self.a.mode
|
||||||
|
self.report["kernel_mode"] = mode
|
||||||
|
if mode == "kernel-status":
|
||||||
|
v = self.view()
|
||||||
|
v["setup_problems"] = self.setup_problems()
|
||||||
|
self.report["kernel"] = v
|
||||||
|
return 0
|
||||||
|
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
|
||||||
|
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
|
||||||
|
rc = fn()
|
||||||
|
self.report["kernel"] = self.view()
|
||||||
|
return rc
|
||||||
|
|
||||||
|
def phase_is(self, st, *phases):
|
||||||
|
if st.get("phase") not in phases:
|
||||||
|
raise Refused("R22", f"the kernel step is {st.get('phase', 'none')!r}, not {' or '.join(phases)} — "
|
||||||
|
f"{self.a.mode} does not apply")
|
||||||
|
|
||||||
|
def stage(self):
|
||||||
|
a = self.a
|
||||||
|
st = self.state()
|
||||||
|
if st.get("phase") in KERNEL_ACTIVE:
|
||||||
|
raise Refused("R22", f"a kernel step is already {st['phase']} ({st.get('from')} -> {st.get('to')})")
|
||||||
|
last = _parse_iso(st.get("staged_at"))
|
||||||
|
if last is not None and self.r.now() - last < KERNEL_MIN_GAP:
|
||||||
|
raise Refused("R22", "a kernel step was staged within the last 20 hours — never two in one night")
|
||||||
|
why = self.setup_problems()
|
||||||
|
if why:
|
||||||
|
raise Refused("R20", "; ".join(why))
|
||||||
|
self.check_guard()
|
||||||
|
a.check_guest(a.vmid)
|
||||||
|
who, _ = a.docker_authority(self.plan, op_name=KERNEL_OP)
|
||||||
|
self.report["authority"] = who
|
||||||
|
old = self.running()
|
||||||
|
if not old:
|
||||||
|
raise Refused("R20", "the running kernel is not a Proxmox kernel version")
|
||||||
|
self.log(f"os-apply: START release={self.plan.get('release_id')} layer=kernel lane=slow mode=apply "
|
||||||
|
f"select={a.select} authority={who} running={old}")
|
||||||
|
if a.apt_lock_held():
|
||||||
|
raise Refused("R9", "another apt/dpkg holds the lock on the host")
|
||||||
|
self.report["health_before"] = a.health()
|
||||||
|
a.repair()
|
||||||
|
rc, out, err = a.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||||
|
if rc != 0:
|
||||||
|
raise Refused("R7", f"apt-get update failed on the host: {(out + err).strip().splitlines()[-1:]}")
|
||||||
|
inst = a.installed()
|
||||||
|
if a.select == "pending-kernel":
|
||||||
|
_, pend, _, _ = a.simulate(["dist-upgrade"])
|
||||||
|
want = [(p["name"], p["to"]) for p in pend if p["from"] is not None and KERNEL_UPGRADE_RE.match(p["name"])
|
||||||
|
and a.origin_name(p["origin"]) == {PVE_ORIGIN}]
|
||||||
|
else:
|
||||||
|
want = [(e["name"], e["version"]) for e in self.plan["packages"]]
|
||||||
|
# upgrades of installed names (never a downgrade), and at most the listed new kernel image
|
||||||
|
args, upg = [], {}
|
||||||
|
for n, v in want:
|
||||||
|
if n in inst:
|
||||||
|
if a.dpkg_cmp(v, "gt", inst[n]):
|
||||||
|
upg[n] = v
|
||||||
|
args.append(f"{n}={v}")
|
||||||
|
elif KERNEL_IMAGE_RE.match(n):
|
||||||
|
args.append(f"{n}={v}")
|
||||||
|
else:
|
||||||
|
raise Refused("R6", f"{n} is not installed and is not a kernel image")
|
||||||
|
if not args:
|
||||||
|
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
|
||||||
|
self.log("os-apply: DONE rc=0 upgraded=0 (no pending kernel)")
|
||||||
|
return 0
|
||||||
|
rc, sim, remv, text = a.simulate(["install", "--no-install-recommends"] + args)
|
||||||
|
if rc != 0:
|
||||||
|
raise Refused("R7", "the simulation failed: " + (text.strip().splitlines()[-1] if text.strip() else ""))
|
||||||
|
if remv:
|
||||||
|
raise Refused("R4", f"the kernel step would remove {', '.join(remv[:5])}")
|
||||||
|
images = []
|
||||||
|
listed = dict(want)
|
||||||
|
for p in sim:
|
||||||
|
if a.origin_name(p["origin"]) != {PVE_ORIGIN}:
|
||||||
|
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not {PVE_ORIGIN!r}")
|
||||||
|
m = KERNEL_IMAGE_RE.match(p["name"])
|
||||||
|
if p["from"] is None:
|
||||||
|
if not m:
|
||||||
|
raise Refused("R6", f"the kernel step would add {p['name']}, which is not a kernel image")
|
||||||
|
if a.select == "listed" and listed.get(p["name"]) != p["to"]:
|
||||||
|
raise Refused("R23", f"the kernel step would add {p['name']}={p['to']}, not the signed set")
|
||||||
|
images.append((m.group(1), p["to"]))
|
||||||
|
continue
|
||||||
|
if p["name"] not in upg or p["to"] != upg[p["name"]]:
|
||||||
|
raise Refused("R6", f"the kernel step would touch {p['name']} ({p['to']}), which is not in the step")
|
||||||
|
if not a.dpkg_cmp(p["to"], "gt", p["from"]):
|
||||||
|
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
|
||||||
|
if len(images) > 1:
|
||||||
|
raise Refused("R23", f"the kernel step would add {len(images)} kernels — one at a time")
|
||||||
|
# the target kernel: the new image, else the series meta-package's version (its image is already installed)
|
||||||
|
if images:
|
||||||
|
new = images[0][0]
|
||||||
|
if images[0][1] + "-pve" != new:
|
||||||
|
raise Refused("R23", f"the image version {images[0][1]} does not name the kernel {new}")
|
||||||
|
else:
|
||||||
|
metas = [(n, v) for n, v in upg.items() if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)]
|
||||||
|
if len(metas) != 1:
|
||||||
|
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
|
||||||
|
self.log("os-apply: DONE rc=0 upgraded=0 (the pending set names no kernel to boot)")
|
||||||
|
return 0
|
||||||
|
new = metas[0][1] + "-pve"
|
||||||
|
if not KVER_RE.match(new) or not a.dpkg_cmp(new[:-4], "gt", old[:-4]):
|
||||||
|
raise Refused("R23", f"the kernel {new} is not newer than the running {old}")
|
||||||
|
ek = self.plan.get("expect_kver")
|
||||||
|
if ek and ek != new:
|
||||||
|
raise Refused("R23", f"the step would boot {new}, but the household was told about {ek}")
|
||||||
|
need = a.download_bytes(["install", "--no-install-recommends"] + args)
|
||||||
|
free = a.free_bytes()
|
||||||
|
if free >= 0 and free < max(MIN_FREE, 3 * need):
|
||||||
|
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
|
||||||
|
# 1. the default = the kernel the box RUNS (it booted healthily), proved from grub.cfg BEFORE the install
|
||||||
|
default_before = self.default_kver(self.grub_cfg())
|
||||||
|
self.write_default(old)
|
||||||
|
# 2. install (the kernel's own postinst runs update-grub; our default file keeps the default on `old`)
|
||||||
|
t0 = time.time()
|
||||||
|
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + ["install", "--no-install-recommends"] + args)
|
||||||
|
a.x(["apt-get", "clean"])
|
||||||
|
if rc != 0:
|
||||||
|
_, aud, _ = a.x(["dpkg", "--audit"])
|
||||||
|
self.report["failed"] = {"rc": rc, "step": "install", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
|
||||||
|
"tail": (out + err).strip().splitlines()[-3:]}
|
||||||
|
self.log(f"os-apply: FAILED rc={rc} step=install (the default stays {old}; no flag written)")
|
||||||
|
return 3
|
||||||
|
self.report["upgraded"] = [{"name": x.split("=", 1)[0], "version": x.split("=", 1)[1]} for x in args]
|
||||||
|
self.report["seconds"] = round(time.time() - t0, 1)
|
||||||
|
# 3. prove: the image is there, the default is still `old`, the one-shot entry for `new` exists
|
||||||
|
cfg = self.grub_cfg()
|
||||||
|
if f"felhom-oneshot-{new}" not in cfg or self.default_kver(cfg) != old:
|
||||||
|
self.r.host(["update-grub"], 300)
|
||||||
|
cfg = self.grub_cfg()
|
||||||
|
for f in (f"/boot/vmlinuz-{new}", f"/boot/initrd.img-{new}"):
|
||||||
|
if not self.r.lexists(f):
|
||||||
|
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"{f} is missing after the install"}
|
||||||
|
return 3
|
||||||
|
if f"felhom-oneshot-{new}" not in cfg or "felhom_next" not in cfg:
|
||||||
|
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"grub.cfg has no one-shot entry for {new}"}
|
||||||
|
return 3
|
||||||
|
if self.default_kver(cfg) != old:
|
||||||
|
self.report["failed"] = {"rc": 3, "step": "verify",
|
||||||
|
"reason": f"the install moved the default to {self.default_kver(cfg)} — no flag written"}
|
||||||
|
return 3
|
||||||
|
# 4. the flag — the ONLY thing that makes the next boot use `new`, and only once
|
||||||
|
self.set_flag(new)
|
||||||
|
self.save_state({"phase": "staged", "step_id": self.plan.get("release_id"), "from": old, "to": new, "vmid": a.vmid,
|
||||||
|
"staged_at": _iso(self.r.now()), "authority": who, "default_before": default_before,
|
||||||
|
"packages": self.report["upgraded"], "self_revert_used": False})
|
||||||
|
self.report["reboot_needed"] = True
|
||||||
|
self.log(f"os-apply: KERNEL STAGED {old} -> {new} (default {old}, one-shot flag {new}); upgraded={len(args)} "
|
||||||
|
f"seconds={self.report['seconds']}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def reboot_staged(self):
|
||||||
|
st = self.state()
|
||||||
|
self.phase_is(st, "staged")
|
||||||
|
if self.flag() != st.get("to"):
|
||||||
|
raise Refused("R22", f"the one-shot flag reads {self.flag()!r}, not {st.get('to')!r}")
|
||||||
|
if self.running() != st.get("from"):
|
||||||
|
raise Refused("R22", f"the box runs {self.running()!r}, not the step's old kernel {st.get('from')!r}")
|
||||||
|
cfg = self.grub_cfg()
|
||||||
|
if self.default_kver(cfg) != st["from"] or f"felhom-oneshot-{st['to']}" not in cfg:
|
||||||
|
raise Refused("R20", "grub.cfg no longer keeps the old default with a one-shot entry for the new kernel")
|
||||||
|
if self.setup_problems():
|
||||||
|
raise Refused("R20", "; ".join(self.setup_problems()))
|
||||||
|
self.check_guard()
|
||||||
|
self.a.check_guest(self.a.vmid)
|
||||||
|
st["health_before"] = self.a.health()
|
||||||
|
st["phase"], st["rebooted_at"] = "oneshot", _iso(self.r.now())
|
||||||
|
self.save_state(st)
|
||||||
|
return self.reboot(f"one-shot boot of {st['to']} (the default stays {st['from']})")
|
||||||
|
|
||||||
|
def after_boot(self):
|
||||||
|
st = self.state()
|
||||||
|
ph, run = st.get("phase"), self.running()
|
||||||
|
old, new = st.get("from"), st.get("to")
|
||||||
|
event = "none"
|
||||||
|
if ph in ("oneshot", "staged", "judging") and run == new and new:
|
||||||
|
if ph != "judging":
|
||||||
|
st["phase"], st["judging_since"], event = "judging", _iso(self.r.now()), "judging"
|
||||||
|
else:
|
||||||
|
event = "judging"
|
||||||
|
elif ph in ("oneshot", "judging") and run == old:
|
||||||
|
# the new kernel did not come up, or crashed: GRUB already booted the default (the old kernel)
|
||||||
|
self.clear_flag()
|
||||||
|
st["phase"], st["result_at"], event = "fell_back", _iso(self.r.now()), "fell_back"
|
||||||
|
st["reason"] = "the box came back on the old kernel by itself (the new one did not boot, or crashed)"
|
||||||
|
elif ph == "reverting" and run == old:
|
||||||
|
st["phase"], st["result_at"], event = "self_reverted", _iso(self.r.now()), "self_reverted"
|
||||||
|
elif ph == "reverting" and run == new:
|
||||||
|
st["phase"], st["result_at"], event = "revert_failed", _iso(self.r.now()), "revert_failed"
|
||||||
|
st["reason"] = "the self-revert came back on the NEW kernel — never retried (one self-revert per step)"
|
||||||
|
if event not in ("none",) and st.get("phase") != ph:
|
||||||
|
self.save_state(st)
|
||||||
|
self.log(f"os-apply: KERNEL AFTER-BOOT {ph} -> {st['phase']} running={run} ({old} -> {new})")
|
||||||
|
self.report["kernel_event"] = event
|
||||||
|
if event == "judging":
|
||||||
|
self.report["health_before"] = st.get("health_before") # the agent judges the boot against it
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def good(self):
|
||||||
|
st = self.state()
|
||||||
|
self.phase_is(st, "judging")
|
||||||
|
if self.running() != st.get("to"):
|
||||||
|
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
|
||||||
|
try:
|
||||||
|
self.write_default(st["to"])
|
||||||
|
except Refused:
|
||||||
|
# put the old default back — the box must never be left without a proved default
|
||||||
|
self.write_default(st["from"])
|
||||||
|
raise
|
||||||
|
self.clear_flag()
|
||||||
|
st["phase"], st["result_at"] = "good", _iso(self.r.now())
|
||||||
|
self.save_state(st)
|
||||||
|
self.log(f"os-apply: KERNEL GOOD {st['to']} is the default now (was {st['from']})")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def revert(self):
|
||||||
|
st = self.state()
|
||||||
|
self.phase_is(st, "judging")
|
||||||
|
if st.get("self_revert_used"):
|
||||||
|
raise Refused("R22", "this kernel step already used its one self-revert")
|
||||||
|
if self.running() != st.get("to"):
|
||||||
|
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
|
||||||
|
self.clear_flag()
|
||||||
|
cfg = self.grub_cfg()
|
||||||
|
if self.default_kver(cfg) != st.get("from"):
|
||||||
|
raise Refused("R20", f"the GRUB default reads {self.default_kver(cfg)}, not the old {st.get('from')} — "
|
||||||
|
f"no self-revert into an unknown kernel")
|
||||||
|
reason = self.plan.get("reason")
|
||||||
|
st["reason"] = reason[:300] if isinstance(reason, str) else "the one-shot boot was not healthy"
|
||||||
|
st["self_revert_used"], st["phase"], st["reverted_at"] = True, "reverting", _iso(self.r.now())
|
||||||
|
self.save_state(st)
|
||||||
|
return self.reboot(f"self-revert to {st['from']}: {st['reason']}")
|
||||||
|
|
||||||
|
def cancel(self):
|
||||||
|
st = self.state()
|
||||||
|
self.phase_is(st, "staged")
|
||||||
|
self.clear_flag()
|
||||||
|
st["phase"], st["result_at"], st["reason"] = "cancelled", _iso(self.r.now()), "cancelled before the reboot"
|
||||||
|
self.save_state(st)
|
||||||
|
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
class Bundle:
|
class Bundle:
|
||||||
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
||||||
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
|
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
|
||||||
|
|||||||
@@ -17,6 +17,8 @@ Verbs (each one sudoers line, exact-match pattern):
|
|||||||
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
|
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
|
||||||
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
|
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
|
||||||
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
|
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
|
||||||
|
controller-image <vmid> the ref on STDIN -> /etc/felhom-controller-image INSIDE guest <vmid> (R-861 (a) A1): only
|
||||||
|
our registry + our repository + an x.y.z tag; the agent no longer has a `tee` grant
|
||||||
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
|
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
|
||||||
|
|
||||||
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
|
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
|
||||||
@@ -39,7 +41,14 @@ WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
|
|||||||
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
|
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
|
||||||
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
|
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
|
||||||
MAX_BYTES = 64 * 1024
|
MAX_BYTES = 64 * 1024
|
||||||
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key")
|
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
|
||||||
|
# R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/
|
||||||
|
# controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by
|
||||||
|
# configs/test_felhom_priv_apply.py ControllerImage.
|
||||||
|
CONTROLLER_IMAGE_RE = re.compile(r"^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$")
|
||||||
|
CONTROLLER_IMAGE_FILE = "/etc/felhom-controller-image"
|
||||||
|
CONTROLLER_IMAGE_MAX = 256
|
||||||
|
VMID_RE = re.compile(r"^[0-9]{1,9}$")
|
||||||
|
|
||||||
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
|
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
|
||||||
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
|
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
|
||||||
@@ -123,6 +132,16 @@ class Host:
|
|||||||
pass
|
pass
|
||||||
raise
|
raise
|
||||||
|
|
||||||
|
def read_stdin(self, limit):
|
||||||
|
return sys.stdin.buffer.read(limit + 1)
|
||||||
|
|
||||||
|
def write_guest_image(self, vmid, data):
|
||||||
|
"""As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell)."""
|
||||||
|
r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE],
|
||||||
|
input=data, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=60)
|
||||||
|
if r.returncode != 0:
|
||||||
|
raise OSError(f"pct exec {vmid} tee exited {r.returncode}: {r.stderr.decode(errors='replace').strip()[:200]}")
|
||||||
|
|
||||||
def log(self, line):
|
def log(self, line):
|
||||||
print(line, file=sys.stderr)
|
print(line, file=sys.stderr)
|
||||||
try:
|
try:
|
||||||
@@ -377,6 +396,34 @@ def plan(argv):
|
|||||||
raise Refused("A1", f"wrong arguments for {v}")
|
raise Refused("A1", f"wrong arguments for {v}")
|
||||||
|
|
||||||
|
|
||||||
|
def controller_image(rest, host):
|
||||||
|
"""R-861 (a) A1: read the ref on stdin, check it, write it INSIDE the guest as root."""
|
||||||
|
try:
|
||||||
|
if len(rest) != 1 or not VMID_RE.match(rest[0]):
|
||||||
|
raise Refused("A1", "usage: felhom-priv-apply controller-image <vmid> (the ref on stdin)")
|
||||||
|
raw = host.read_stdin(CONTROLLER_IMAGE_MAX)
|
||||||
|
if len(raw) > CONTROLLER_IMAGE_MAX:
|
||||||
|
raise Refused("I1", f"the image ref is longer than {CONTROLLER_IMAGE_MAX} bytes")
|
||||||
|
try:
|
||||||
|
text = raw.decode("ascii")
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
raise Refused("I1", "the image ref is not ASCII")
|
||||||
|
ref = text[:-1] if text.endswith("\n") else text
|
||||||
|
if not CONTROLLER_IMAGE_RE.match(ref) or "\n" in ref:
|
||||||
|
raise Refused("I1", "the image ref is not gitea.dooplex.hu/admin/felhom-controller:<x.y.z>")
|
||||||
|
except Refused as e:
|
||||||
|
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] controller-image {' '.join(rest)[:40]}: {e.reason}")
|
||||||
|
return 2 if e.rule == "A1" else 3
|
||||||
|
vmid = int(rest[0])
|
||||||
|
try:
|
||||||
|
host.write_guest_image(vmid, (ref + "\n").encode())
|
||||||
|
except (OSError, subprocess.SubprocessError) as e:
|
||||||
|
host.log(f"felhom-priv-apply: FAILED controller-image {vmid}: {e}")
|
||||||
|
return 4
|
||||||
|
host.log(f"felhom-priv-apply: WROTE controller-image {vmid} {ref}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def main(argv, host=None):
|
def main(argv, host=None):
|
||||||
host = host or Host()
|
host = host or Host()
|
||||||
if argv == ["--self-check"]:
|
if argv == ["--self-check"]:
|
||||||
@@ -396,6 +443,8 @@ def main(argv, host=None):
|
|||||||
return 3
|
return 3
|
||||||
print("OK")
|
print("OK")
|
||||||
return 0
|
return 0
|
||||||
|
if argv and argv[0] == "controller-image":
|
||||||
|
return controller_image(argv[1:], host)
|
||||||
try:
|
try:
|
||||||
verb, src, dest, mode, checker = plan(argv)
|
verb, src, dest, mode, checker = plan(argv)
|
||||||
data = host.read_source(src)
|
data = host.read_source(src)
|
||||||
|
|||||||
@@ -139,5 +139,62 @@ class Guard(unittest.TestCase):
|
|||||||
self.assertEqual(mode, 0o644)
|
self.assertEqual(mode, 0o644)
|
||||||
|
|
||||||
|
|
||||||
|
class KernelStepCannotLeaveTheBoxOff(unittest.TestCase):
|
||||||
|
"""R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the
|
||||||
|
box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window
|
||||||
|
(felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` —
|
||||||
|
the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th
|
||||||
|
(3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
self.d = tempfile.TemporaryDirectory()
|
||||||
|
self.e = FakeEnv(self.d.name)
|
||||||
|
cg.main(["x", "boot"], self.e)
|
||||||
|
s = self.e.state()
|
||||||
|
self.assertTrue(s["armed"])
|
||||||
|
self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.d.cleanup()
|
||||||
|
|
||||||
|
def planned(self, minutes):
|
||||||
|
cg.main(["x", "clean-stop"], self.e)
|
||||||
|
self.e.t += minutes * 60
|
||||||
|
cg.main(["x", "boot"], self.e)
|
||||||
|
|
||||||
|
def crash(self, minutes):
|
||||||
|
self.e.t += minutes * 60
|
||||||
|
cg.main(["x", "boot"], self.e)
|
||||||
|
|
||||||
|
def test_planned_reboot_one_crash_one_self_revert(self):
|
||||||
|
self.planned(2) # the step's one-shot reboot (orderly)
|
||||||
|
self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10)
|
||||||
|
self.planned(2) # the self-revert (orderly)
|
||||||
|
s = self.e.state()
|
||||||
|
self.assertFalse(s["tripped"], s)
|
||||||
|
self.assertTrue(s["armed"])
|
||||||
|
self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash")
|
||||||
|
self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot")
|
||||||
|
|
||||||
|
def test_a_panic_before_userspace_is_not_even_counted(self):
|
||||||
|
# the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned
|
||||||
|
# reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean.
|
||||||
|
cg.main(["x", "clean-stop"], self.e)
|
||||||
|
self.e.t += 120 # the panicking boot: no userspace, the guard never ran
|
||||||
|
cg.main(["x", "boot"], self.e)
|
||||||
|
s = self.e.state()
|
||||||
|
self.assertEqual(s["unclean_boots_in_window"], 0, s)
|
||||||
|
self.assertEqual(self.e.panic(), 10)
|
||||||
|
|
||||||
|
def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self):
|
||||||
|
self.planned(2)
|
||||||
|
self.crash(3) # the step's one crash
|
||||||
|
self.planned(2) # the self-revert
|
||||||
|
self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off)
|
||||||
|
s = self.e.state()
|
||||||
|
self.assertTrue(s["tripped"])
|
||||||
|
self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -232,6 +232,8 @@ class Fake:
|
|||||||
return (0, self.daemon_json, "") if self.daemon_json is not None else (1, "", "No such file")
|
return (0, self.daemon_json, "") if self.daemon_json is not None else (1, "", "No such file")
|
||||||
if cmd == "cat" and a[1] == "/etc/debian_version":
|
if cmd == "cat" and a[1] == "/etc/debian_version":
|
||||||
return 0, "13.7\n", ""
|
return 0, "13.7\n", ""
|
||||||
|
if cmd == "pveversion":
|
||||||
|
return 0, f"pve-manager/{self.installed.get('pve-manager', '9.2.2')}/abcdef (running kernel: 7.0.14-20-pve)\n", ""
|
||||||
if cmd == "uname":
|
if cmd == "uname":
|
||||||
return 0, "7.0.14-20-pve\n", ""
|
return 0, "7.0.14-20-pve\n", ""
|
||||||
if cmd == "apt-mark":
|
if cmd == "apt-mark":
|
||||||
@@ -284,7 +286,7 @@ class Fake:
|
|||||||
n, v = x.split("=", 1)
|
n, v = x.split("=", 1)
|
||||||
if v not in self.avail(n):
|
if v not in self.avail(n):
|
||||||
return 100, "", f"E: Version '{v}' for '{n}' was not found"
|
return 100, "", f"E: Version '{v}' for '{n}' was not found"
|
||||||
origin = "Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB
|
origin = getattr(self, "origins", {}).get(n) or ("Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB)
|
||||||
out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
|
out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
|
||||||
out += "".join(l + "\n" for l in self.extra_sim)
|
out += "".join(l + "\n" for l in self.extra_sim)
|
||||||
return 0, out, ""
|
return 0, out, ""
|
||||||
@@ -1324,5 +1326,684 @@ class OOMCheck(unittest.TestCase):
|
|||||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
|
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
|
||||||
|
|
||||||
|
|
||||||
|
PVE = "Proxmox Debian Repository:stable"
|
||||||
|
PVE_SET = [{"name": "pve-manager", "version": "9.2.21", "origin": "Proxmox Debian Repository"},
|
||||||
|
{"name": "libpve-common-perl", "version": "9.1.9", "origin": "Proxmox Debian Repository"}]
|
||||||
|
|
||||||
|
|
||||||
|
def pve_fake(signed=None, ring0=False):
|
||||||
|
f = Fake()
|
||||||
|
f.installed.update({"pve-manager": "9.2.2", "libpve-common-perl": "9.1.1", "proxmox-kernel-helper": "9.0.4"})
|
||||||
|
f.live["pve-manager"] = {"9.2.21", "9.2.2"}
|
||||||
|
f.live["libpve-common-perl"] = {"9.1.9", "9.1.1"}
|
||||||
|
f.origins = {"pve-manager": PVE, "libpve-common-perl": PVE, "proxmox-kernel-helper": PVE, "shim-signed": PVE,
|
||||||
|
"proxmox-firewall-data": PVE}
|
||||||
|
f.plan = {"release_id": "os-pve-t1", "layer": "pve", "lane": "slow", "vmid": 9201, "mode": "apply",
|
||||||
|
"packages": [dict(p) for p in PVE_SET]}
|
||||||
|
if ring0:
|
||||||
|
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
|
||||||
|
if signed is not None:
|
||||||
|
f.plan["signed"] = signed
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
class PVELane(unittest.TestCase):
|
||||||
|
"""R-812 option A (`09` §3 decision 163): the host's Proxmox USERSPACE packages, slow lane, no kernel / boot /
|
||||||
|
firmware / microcode (R14), no removal, a new package only from PVE_NEW_ALLOW. Red-proof: audits/day-2026-10-07/B/."""
|
||||||
|
|
||||||
|
def refused(self, f, code):
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 2, rep)
|
||||||
|
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||||
|
self.assertEqual(f.installed["pve-manager"], "9.2.2", "nothing may be installed on a refusal")
|
||||||
|
return rep
|
||||||
|
|
||||||
|
# THE RED TEST (design-R-812 §5): before the pve layer existed this plan was refused R12.
|
||||||
|
def test_pve_manager_plan_is_installed_on_the_host(self):
|
||||||
|
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(f.installed["pve-manager"], "9.2.21")
|
||||||
|
self.assertEqual(rep["authority"], "signed")
|
||||||
|
self.assertEqual(rep["pve_manager"], "9.2.21", "pveversion after the step is reported")
|
||||||
|
inst = [c for c in f.calls if c[0] == "host" and "install" in c[1] and "-s" not in c[1] and "-f" not in c[1]
|
||||||
|
and "--print-uris" not in c[1]]
|
||||||
|
self.assertTrue(inst, "the pve layer installs on the HOST")
|
||||||
|
self.assertFalse([c for c in f.calls if c[0] == "guest" and "install" in c[2]], "nothing installed in the guest")
|
||||||
|
self.assertEqual(sorted(rep["health_after"]["host_services"]), sorted(osapply.HOST_SERVICES))
|
||||||
|
|
||||||
|
def test_kernel_in_a_pve_plan_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["packages"].append({"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"})
|
||||||
|
self.refused(f, "R14")
|
||||||
|
|
||||||
|
def test_shim_in_a_pve_plan_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["packages"].append({"name": "shim-signed", "version": "1.47+pmx1", "origin": "Proxmox Debian Repository"})
|
||||||
|
self.refused(f, "R14")
|
||||||
|
|
||||||
|
def test_kernel_pulled_in_by_the_simulation_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.installed["proxmox-kernel-helper"] = "9.0.4"
|
||||||
|
f.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
|
||||||
|
self.refused(f, "R6") # not in the plan — refused before the name check; R14 below when it IS in the plan
|
||||||
|
g = pve_fake(ring0=True)
|
||||||
|
g.live["proxmox-kernel-helper"] = {"9.0.6"}
|
||||||
|
g.plan["packages"] = [dict(PVE_SET[0])]
|
||||||
|
g.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
|
||||||
|
# a kernel-helper the plan did not name is R6; the R14 name check covers a listed one (test above)
|
||||||
|
self.refused(g, "R6")
|
||||||
|
|
||||||
|
def test_debian_package_in_a_pve_plan_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["packages"].append({"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"})
|
||||||
|
self.refused(f, "R2")
|
||||||
|
|
||||||
|
def test_debian_origin_in_the_pve_simulation_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.origins["libpve-common-perl"] = DEB
|
||||||
|
self.refused(f, "R2")
|
||||||
|
|
||||||
|
def test_docker_package_in_a_pve_plan_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["packages"].append({"name": "docker-ce", "version": "5:29.8.2-1~debian.13~trixie", "origin": "Proxmox Debian Repository"})
|
||||||
|
self.refused(f, "R2")
|
||||||
|
|
||||||
|
def test_pve_in_the_fast_lane_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["lane"] = "fast"
|
||||||
|
self.refused(f, "R3")
|
||||||
|
|
||||||
|
def test_no_authority_is_refused(self):
|
||||||
|
self.refused(pve_fake(), "R3")
|
||||||
|
|
||||||
|
def test_docker_signed_op_does_not_authorize_a_pve_step(self):
|
||||||
|
self.refused(pve_fake(signed=signed_job(packages=PVE_SET, op="os_docker_step")), "R3")
|
||||||
|
|
||||||
|
def test_pve_on_a_byo_box_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||||
|
self.refused(f, "R12")
|
||||||
|
|
||||||
|
def test_unlisted_new_package_is_refused(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.extra_sim = ["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"]
|
||||||
|
self.refused(f, "R6")
|
||||||
|
|
||||||
|
def test_allow_listed_new_package_is_accepted(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(f.installed["pve-manager"], "9.2.21")
|
||||||
|
|
||||||
|
def test_allow_new_is_never_an_open_door_in_the_fast_lane(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["layer"] = "host"
|
||||||
|
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["refused"]["code"]), (2, "R6"), rep)
|
||||||
|
|
||||||
|
def test_undo_is_refused(self):
|
||||||
|
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
|
||||||
|
f.plan["undo"] = True
|
||||||
|
self.refused(f, "R5")
|
||||||
|
|
||||||
|
def test_ring0_pending_pve_takes_only_installed_proxmox_userspace(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||||
|
f.installed.update({"linux-image-amd64": "6.12.1", "tailscale": "1.102.2"})
|
||||||
|
f.pending_sim = [
|
||||||
|
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst libpve-common-perl [9.1.1] (9.1.9 Proxmox Debian Repository:stable [all])",
|
||||||
|
"Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])",
|
||||||
|
"Inst proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])",
|
||||||
|
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])",
|
||||||
|
"Inst tailscale [1.102.2] (1.102.5 Tailscale:pkgs.tailscale.com [amd64])",
|
||||||
|
]
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["authority"], "ring0")
|
||||||
|
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["libpve-common-perl", "pve-manager"],
|
||||||
|
"pending-pve: installed, Proxmox-origin, never kernel/boot/firmware, never Debian or other origins")
|
||||||
|
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
|
||||||
|
|
||||||
|
# 2026-10-09, demo-hp (Secure Boot on): proxmox-secure-boot-support's upgrade pulls shim-signed-common; in the pve
|
||||||
|
# plan that refused the whole step (R6) and skipped the night's kernel step. It is a boot-chain package: left out.
|
||||||
|
# RED-PROOF: drop proxmox-secure-boot-support from HOST_SLOW_RE -> it is in the plan -> this test fails.
|
||||||
|
def test_ring0_pending_pve_leaves_the_secure_boot_meta_out(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||||
|
f.installed.update({"proxmox-secure-boot-support": "9.0.1"})
|
||||||
|
f.pending_sim = [
|
||||||
|
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst proxmox-secure-boot-support [9.0.1] (9.0.2 Proxmox Debian Repository:stable [all])",
|
||||||
|
"Inst shim-signed-common [1.48+pmx1+16.1-1+pmx1] (1.51+pmx1+16.1-2+pmx1 Proxmox Debian Repository:stable [all])",
|
||||||
|
]
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||||
|
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
||||||
|
|
||||||
|
def test_select_pending_pve_needs_the_pve_layer(self):
|
||||||
|
f = Fake()
|
||||||
|
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----------
|
||||||
|
U = "1af1fcc6-639c-416b-a7e5-c4470d41a502"
|
||||||
|
OLD, NEW = "7.0.2-6-pve", "7.0.14-22-pve"
|
||||||
|
KERNEL_SET = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"},
|
||||||
|
{"name": "proxmox-kernel-7.0.14-22-pve-signed", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}]
|
||||||
|
|
||||||
|
|
||||||
|
class KFake(Fake):
|
||||||
|
"""A Proxmox host with GRUB on UEFI: /boot on the root LV, a vfat ESP, the bundle's two generators. update-grub is
|
||||||
|
emulated like the real 10_linux: WITHOUT the felhom default file the NEWEST kernel becomes the default (measured,
|
||||||
|
R-836 — that is the defect the lane exists for); with it, the kernel it names."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self.running = OLD
|
||||||
|
self.boot = {OLD}
|
||||||
|
self.efi, self.esp_fs, self.boot_mount, self.mods, self.snippets, self.pin = True, "vfat", "", True, True, False
|
||||||
|
self.env = None # None = no env block on the ESP; else {"felhom_next": ...}
|
||||||
|
self.tree = {} # files put_file wrote
|
||||||
|
self.reboots = []
|
||||||
|
self.update_grubs = 0
|
||||||
|
self.grub_runs_in_postinst = True
|
||||||
|
self.installed.update({"proxmox-kernel-7.0": "7.0.2-6", "proxmox-default-kernel": "2.1.0",
|
||||||
|
"pve-firmware": "3.18-3", "proxmox-kernel-7.0.2-6-pve-signed": "7.0.2-6",
|
||||||
|
"pve-manager": "9.2.21"})
|
||||||
|
self.live.update({"proxmox-kernel-7.0": {"7.0.14-22", "7.0.2-6"},
|
||||||
|
"proxmox-kernel-7.0.14-22-pve-signed": {"7.0.14-22"},
|
||||||
|
"proxmox-kernel-7.0.14-23-pve-signed": {"7.0.14-23"},
|
||||||
|
"pve-firmware": {"3.18-7", "3.18-3"}})
|
||||||
|
self.origins = {}
|
||||||
|
self.kernel_pending = ["Inst pve-firmware [3.18-3] (3.18-7 Proxmox Debian Repository:stable [all])",
|
||||||
|
"Inst proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst proxmox-kernel-7.0 [7.0.2-6] (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst pve-manager [9.2.21] (9.2.22 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])"]
|
||||||
|
self.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "apply",
|
||||||
|
"select": "pending-kernel", "packages": []}
|
||||||
|
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
|
||||||
|
self.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 0})
|
||||||
|
self.cfg = self.render()
|
||||||
|
|
||||||
|
# -- GRUB --
|
||||||
|
def newest(self):
|
||||||
|
best = None
|
||||||
|
for k in self.boot:
|
||||||
|
if best is None or dpkg_cmp(k[:-4], "gt", best[:-4]):
|
||||||
|
best = k
|
||||||
|
return best
|
||||||
|
|
||||||
|
def render(self):
|
||||||
|
d = self.tree.get(osapply.KERNEL_DEFAULT_CFG)
|
||||||
|
if d:
|
||||||
|
dflt = re.search(r'GRUB_DEFAULT="([^"]+)"', d).group(1)
|
||||||
|
else:
|
||||||
|
dflt = f"gnulinux-advanced-{U}>gnulinux-{self.newest()}-advanced-{U}"
|
||||||
|
cfg = ('if [ "${next_entry}" ] ; then\n set default="${next_entry}"\nelse\n'
|
||||||
|
f' set default="{dflt}"\nfi\n'
|
||||||
|
f"submenu 'Advanced options' $menuentry_id_option 'gnulinux-advanced-{U}' {{\n")
|
||||||
|
for k in sorted(self.boot):
|
||||||
|
cfg += f" menuentry 'Proxmox VE, with Linux {k}' $menuentry_id_option 'gnulinux-{k}-advanced-{U}' {{ }}\n"
|
||||||
|
cfg += "}\n"
|
||||||
|
if self.snippets:
|
||||||
|
cfg += "### BEGIN /etc/grub.d/01_felhom_oneshot ###\nload_env felhom_next\n"
|
||||||
|
cfg += "".join(f"menuentry 'Felhom one-shot: {k}' --id felhom-oneshot-{k} {{ }}\n" for k in sorted(self.boot))
|
||||||
|
return cfg
|
||||||
|
|
||||||
|
def lexists(self, p):
|
||||||
|
if p == "/sys/firmware/efi":
|
||||||
|
return self.efi
|
||||||
|
if p.startswith("/usr/lib/grub/x86_64-efi/"):
|
||||||
|
return self.mods
|
||||||
|
if p in (osapply.ONESHOT_SNIPPET, osapply.ONESHOT_ENTRIES):
|
||||||
|
return self.snippets
|
||||||
|
if p == osapply.KERNEL_PIN_FILE:
|
||||||
|
return self.pin
|
||||||
|
m = re.match(r"^/boot/(vmlinuz|initrd\.img)-(.+)$", p)
|
||||||
|
if m:
|
||||||
|
return m.group(2) in self.boot
|
||||||
|
return p in self.tree
|
||||||
|
|
||||||
|
def put_file(self, path, data, mode):
|
||||||
|
self.tree[path] = data.decode()
|
||||||
|
|
||||||
|
def read_file(self, p):
|
||||||
|
if p == osapply.GRUB_CFG:
|
||||||
|
return self.cfg
|
||||||
|
if p in self.tree:
|
||||||
|
return self.tree[p]
|
||||||
|
return super().read_file(p)
|
||||||
|
|
||||||
|
def host(self, argv, timeout=600, stdin=None):
|
||||||
|
if argv[0] == "uname":
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
return 0, self.running + "\n", ""
|
||||||
|
if argv[0] == "findmnt":
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
if argv[-1] == osapply.ESP_MOUNT:
|
||||||
|
return (0, self.esp_fs + "\n", "") if self.esp_fs else (1, "", "")
|
||||||
|
return (0, self.boot_mount + "\n", "") if self.boot_mount else (1, "", "")
|
||||||
|
if argv[0] == "grub-editenv":
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
if argv[2] == "list":
|
||||||
|
return (1, "", "no such file") if self.env is None else \
|
||||||
|
(0, "".join(f"{k}={v}\n" for k, v in self.env.items()), "")
|
||||||
|
if argv[2] == "create":
|
||||||
|
self.env = {}
|
||||||
|
return 0, "", ""
|
||||||
|
if argv[2] == "set":
|
||||||
|
k, v = argv[3].split("=", 1)
|
||||||
|
self.env[k] = v
|
||||||
|
return 0, "", ""
|
||||||
|
if argv[2] == "unset":
|
||||||
|
self.env.pop(argv[3], None)
|
||||||
|
return 0, "", ""
|
||||||
|
if argv[0] == "update-grub":
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
self.update_grubs += 1
|
||||||
|
self.cfg = self.render()
|
||||||
|
return 0, "", ""
|
||||||
|
if argv[0] == "mkdir":
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
return 0, "", ""
|
||||||
|
if argv[:2] == ["systemctl", "reboot"]:
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
self.reboots.append(self.running)
|
||||||
|
return 0, "", ""
|
||||||
|
return super().host(argv, timeout, stdin)
|
||||||
|
|
||||||
|
def emulate(self, argv):
|
||||||
|
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||||
|
if a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a and "-f" not in a:
|
||||||
|
if self.install_rc:
|
||||||
|
return self.install_rc, "", "E: boom"
|
||||||
|
for x in a:
|
||||||
|
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||||
|
n, v = x.split("=", 1)
|
||||||
|
self.installed[n] = v
|
||||||
|
m = re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)
|
||||||
|
if m:
|
||||||
|
self.installed[f"proxmox-kernel-{v}-pve-signed"] = v
|
||||||
|
if m or osapply.KERNEL_IMAGE_RE.match(n):
|
||||||
|
self.boot.add(v + "-pve")
|
||||||
|
if self.grub_runs_in_postinst:
|
||||||
|
self.cfg = self.render() # the kernel's postinst runs update-grub (zz-update-grub)
|
||||||
|
return 0, "Setting up proxmox-kernel ...\n", ""
|
||||||
|
return super().emulate(argv)
|
||||||
|
|
||||||
|
def sim(self, a):
|
||||||
|
if "--print-uris" in a:
|
||||||
|
return 0, "", ""
|
||||||
|
if "dist-upgrade" in a:
|
||||||
|
return 0, "\n".join(self.kernel_pending) + "\n", ""
|
||||||
|
out, named = "", set()
|
||||||
|
for x in a:
|
||||||
|
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||||
|
named.add(x.split("=", 1)[0])
|
||||||
|
for x in a:
|
||||||
|
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||||
|
n, v = x.split("=", 1)
|
||||||
|
if v not in self.avail(n):
|
||||||
|
return 100, "", f"E: Version '{v}' for '{n}' was not found"
|
||||||
|
o = self.origins.get(n, PVE)
|
||||||
|
out += f"Inst {n} [{self.installed[n]}] ({v} {o} [amd64])\n" if n in self.installed else f"Inst {n} ({v} {o} [amd64])\n"
|
||||||
|
if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n):
|
||||||
|
img = f"proxmox-kernel-{v}-pve-signed"
|
||||||
|
if img not in self.installed and img not in named:
|
||||||
|
out += f"Inst {img} ({v} {PVE} [amd64])\n"
|
||||||
|
out += "".join(l + "\n" for l in self.extra_sim)
|
||||||
|
return 0, out, ""
|
||||||
|
|
||||||
|
|
||||||
|
def kfake(**kw):
|
||||||
|
f = KFake()
|
||||||
|
for k, v in kw.items():
|
||||||
|
setattr(f, k, v)
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
def kmode(f, mode, **extra):
|
||||||
|
f.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": mode, "packages": []}
|
||||||
|
f.plan.update(extra)
|
||||||
|
return run(f)
|
||||||
|
|
||||||
|
|
||||||
|
def staged(f=None):
|
||||||
|
f = f or kfake()
|
||||||
|
rc, rep = run(f)
|
||||||
|
assert rc == 0, rep
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
class KernelLane(unittest.TestCase):
|
||||||
|
"""R-836 / `09` §3 decision 172: stage a kernel once through the ESP flag; the default moves only after a healthy
|
||||||
|
one-shot boot. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
|
||||||
|
|
||||||
|
def refused(self, f, code, mode=None, **extra):
|
||||||
|
rc, rep = kmode(f, mode, **extra) if mode else run(f)
|
||||||
|
self.assertEqual(rc, 2, rep)
|
||||||
|
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||||
|
return rep
|
||||||
|
|
||||||
|
# --- the four red tests the brief names (Part A 3) ---
|
||||||
|
def test_installing_a_kernel_does_not_change_the_grub_default(self):
|
||||||
|
f = kfake()
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertIn(NEW, f.boot, "the new kernel was installed")
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD,
|
||||||
|
"the default must stay the kernel the box runs (R-836: an install made the new one default)")
|
||||||
|
self.assertIn(f"gnulinux-{OLD}-advanced-{U}", f.tree[osapply.KERNEL_DEFAULT_CFG])
|
||||||
|
self.assertEqual(f.env, {"felhom_next": NEW}, "the ONLY way to the new kernel is the one-shot flag")
|
||||||
|
self.assertEqual(f.reboots, [], "staging never reboots")
|
||||||
|
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||||
|
self.assertEqual((st["phase"], st["from"], st["to"]), ("staged", OLD, NEW))
|
||||||
|
self.assertEqual(rep["kernel"]["default"], OLD)
|
||||||
|
|
||||||
|
def test_a_box_without_the_esp_flag_is_refused(self):
|
||||||
|
for attr, val, frag in (("esp_fs", "ext4", "vfat"), ("efi", False, "UEFI"), ("snippets", False, "bundle"),
|
||||||
|
("mods", False, "module"), ("boot_mount", "/boot", "separate"), ("pin", True, "pinned")):
|
||||||
|
f = kfake(**{attr: val})
|
||||||
|
rep = self.refused(f, "R20")
|
||||||
|
self.assertIn(frag, rep["refused"]["reason"], attr)
|
||||||
|
self.assertNotIn(NEW, f.boot, f"{attr}: nothing may be installed on a refusal")
|
||||||
|
self.assertIsNone(f.env, f"{attr}: no flag on a refusal")
|
||||||
|
|
||||||
|
def test_a_reboot_without_the_flag_is_refused(self):
|
||||||
|
f = staged()
|
||||||
|
f.env = {"felhom_next": ""}
|
||||||
|
self.refused(f, "R22", mode="kernel-reboot")
|
||||||
|
self.assertEqual(f.reboots, [])
|
||||||
|
|
||||||
|
def test_a_kernel_outside_the_approved_set_is_refused(self):
|
||||||
|
# a signed set that names only the meta-package: the image the sources pull in was never signed for
|
||||||
|
f = kfake()
|
||||||
|
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||||
|
f.plan.update(select="listed", packages=[dict(KERNEL_SET[0])],
|
||||||
|
signed=signed_job(packages=[KERNEL_SET[0]], op="os_kernel_step"))
|
||||||
|
self.refused(f, "R23")
|
||||||
|
self.assertNotIn(NEW, f.boot)
|
||||||
|
# a signed set names 7.0.14-22; the sources would ALSO bring 7.0.14-23
|
||||||
|
f2 = kfake()
|
||||||
|
f2.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||||
|
f2.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||||
|
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
|
||||||
|
f2.extra_sim = ["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"]
|
||||||
|
self.refused(f2, "R23")
|
||||||
|
# a name that is not a kernel package at all
|
||||||
|
g = kfake()
|
||||||
|
g.plan.update(select="listed", packages=[{"name": "pve-manager", "version": "9.2.22", "origin": "Proxmox Debian Repository"}])
|
||||||
|
self.refused(g, "R23")
|
||||||
|
# the household was told about another kernel
|
||||||
|
h = kfake()
|
||||||
|
h.plan["expect_kver"] = "7.0.14-23-pve"
|
||||||
|
self.refused(h, "R23")
|
||||||
|
self.assertNotIn(NEW, h.boot)
|
||||||
|
|
||||||
|
def test_the_snippet_clears_the_flag_on_use(self):
|
||||||
|
"""01_felhom_oneshot: the flag is copied, CLEARED and SAVED before any `set default`, all inside the one branch
|
||||||
|
that runs only when the flag is set; a default is set only for an installed kernel's own entry."""
|
||||||
|
text = (HERE / "felhom-grub-oneshot.sh").read_text()
|
||||||
|
body = text[text.index("cat <<EOF"):]
|
||||||
|
i_copy = body.index('set felhom_boot="\\${felhom_next}"')
|
||||||
|
i_clear = body.index("set felhom_next=\n")
|
||||||
|
i_save = body.index("save_env -f (\\$felhom_esp)/EFI/felhom/oneshot.env felhom_next")
|
||||||
|
i_default = body.index('set default="felhom-oneshot-%s"')
|
||||||
|
self.assertLess(i_copy, i_clear)
|
||||||
|
self.assertLess(i_clear, i_save)
|
||||||
|
self.assertLess(i_save, i_default, "the flag must be cleared on disk BEFORE GRUB boots anything")
|
||||||
|
self.assertIn('if [ "${felhom_boot}" = "%s" ]', body, "a default only for a kernel that is installed")
|
||||||
|
self.assertIn(osapply.ONESHOT_ENV[len(osapply.ESP_MOUNT):], text, "the wrapper and GRUB name the same env file")
|
||||||
|
|
||||||
|
# --- the rest of the stage ---
|
||||||
|
def test_option_c_options_are_on_the_one_shot_entry_only(self):
|
||||||
|
text = (HERE / "felhom-grub-oneshot-entries.sh").read_text()
|
||||||
|
self.assertIn(osapply.ONESHOT_ARGS, text)
|
||||||
|
self.assertIn("--id felhom-oneshot-$k", text)
|
||||||
|
self.assertNotIn("set default", text, "the entries file never sets the default")
|
||||||
|
|
||||||
|
def test_both_generators_ride_the_bundle(self):
|
||||||
|
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_SNIPPET][1:4], ("felhom-grub-oneshot.sh", 0o755, "sh"))
|
||||||
|
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_ENTRIES][1:4], ("felhom-grub-oneshot-entries.sh", 0o755, "sh"))
|
||||||
|
|
||||||
|
def test_ring0_pending_kernel_takes_only_the_kernel_set(self):
|
||||||
|
f = kfake()
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["authority"], "ring0")
|
||||||
|
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["proxmox-kernel-7.0", "pve-firmware"])
|
||||||
|
self.assertEqual(f.installed["pve-manager"], "9.2.21", "a Proxmox userspace package is the pve lane's")
|
||||||
|
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3", "a Debian package is the fast lane's")
|
||||||
|
|
||||||
|
def test_signed_listed_set_is_installed(self):
|
||||||
|
f = kfake()
|
||||||
|
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||||
|
f.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||||
|
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["authority"], "signed")
|
||||||
|
self.assertEqual(f.env, {"felhom_next": NEW})
|
||||||
|
|
||||||
|
def test_no_authority_and_the_wrong_op_are_refused(self):
|
||||||
|
f = kfake()
|
||||||
|
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||||
|
self.refused(f, "R3")
|
||||||
|
g = kfake()
|
||||||
|
g.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||||
|
g.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||||
|
signed=signed_job(packages=KERNEL_SET, op="os_pve_step"))
|
||||||
|
self.refused(g, "R3")
|
||||||
|
|
||||||
|
def test_fast_lane_byo_and_wrong_select_are_refused(self):
|
||||||
|
f = kfake()
|
||||||
|
f.plan["lane"] = "fast"
|
||||||
|
self.refused(f, "R3")
|
||||||
|
g = kfake()
|
||||||
|
g.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||||
|
self.refused(g, "R12")
|
||||||
|
h = kfake()
|
||||||
|
h.plan["select"] = "pending-pve"
|
||||||
|
self.refused(h, "R11")
|
||||||
|
k = kfake()
|
||||||
|
k.plan.update(layer="host", lane="fast", mode="kernel-status")
|
||||||
|
self.refused(k, "R11")
|
||||||
|
|
||||||
|
def test_the_crash_guard_must_be_armed_and_quiet(self):
|
||||||
|
f = kfake()
|
||||||
|
f.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": False, "tripped": True, "unclean_boots_in_window": 2})
|
||||||
|
self.refused(f, "R21")
|
||||||
|
g = kfake()
|
||||||
|
g.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 1})
|
||||||
|
self.refused(g, "R21")
|
||||||
|
h = kfake()
|
||||||
|
del h.files[osapply.CRASH_GUARD_STATE]
|
||||||
|
self.refused(h, "R21")
|
||||||
|
self.assertNotIn(NEW, h.boot)
|
||||||
|
|
||||||
|
def test_never_two_steps_in_one_night(self):
|
||||||
|
f = staged()
|
||||||
|
self.refused(f, "R22") # still staged
|
||||||
|
g = staged()
|
||||||
|
st = json.loads(g.tree[osapply.KERNEL_STATE])
|
||||||
|
st["phase"] = "good"
|
||||||
|
g.tree[osapply.KERNEL_STATE] = json.dumps(st)
|
||||||
|
g.clock += 3600
|
||||||
|
self.refused(g, "R22") # done, but within 20 h
|
||||||
|
|
||||||
|
def test_removal_new_non_kernel_two_kernels_and_older_are_refused(self):
|
||||||
|
f = kfake(extra_sim=["Remv pve-firmware [3.18-3]"])
|
||||||
|
self.refused(f, "R4")
|
||||||
|
g = kfake(extra_sim=["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"])
|
||||||
|
self.refused(g, "R6")
|
||||||
|
h = kfake(extra_sim=["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"])
|
||||||
|
self.refused(h, "R23")
|
||||||
|
k = kfake(running="7.0.14-23-pve")
|
||||||
|
k.boot = {"7.0.14-23-pve"}
|
||||||
|
k.cfg = k.render()
|
||||||
|
self.refused(k, "R23")
|
||||||
|
m = kfake()
|
||||||
|
m.origins = {"proxmox-kernel-7.0": DEB}
|
||||||
|
self.refused(m, "R2")
|
||||||
|
|
||||||
|
# R-898: ring 0 stages EXACTLY the told kernel (select listed, the set derived from it) — even when the sources
|
||||||
|
# offer a newer one by night; a told version that can no longer be installed is refused BEFORE any change (R7).
|
||||||
|
def test_ring0_listed_installs_the_told_kernel_not_the_newest(self):
|
||||||
|
f = kfake()
|
||||||
|
f.live["proxmox-kernel-7.0"] = {"7.0.14-20", "7.0.14-22", "7.0.2-6"}
|
||||||
|
f.live["proxmox-kernel-7.0.14-20-pve-signed"] = {"7.0.14-20"}
|
||||||
|
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||||
|
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||||
|
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["authority"], "ring0")
|
||||||
|
self.assertEqual(f.env, {"felhom_next": "7.0.14-20-pve"})
|
||||||
|
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.14-20")
|
||||||
|
self.assertNotIn("7.0.14-22-pve", f.boot)
|
||||||
|
|
||||||
|
def test_ring0_told_kernel_gone_is_refused_before_any_change(self):
|
||||||
|
f = kfake()
|
||||||
|
f.live["proxmox-kernel-7.0"] = {"7.0.14-22"} # 7.0.14-20 is no longer in the archive
|
||||||
|
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||||
|
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||||
|
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||||
|
rep = self.refused(f, "R7")
|
||||||
|
self.assertIsNone(f.env)
|
||||||
|
self.assertNotIn(osapply.KERNEL_DEFAULT_CFG, f.tree, "refused before the default was even pinned")
|
||||||
|
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.2-6")
|
||||||
|
|
||||||
|
def test_nothing_pending_changes_nothing(self):
|
||||||
|
f = kfake(kernel_pending=[])
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["upgraded"], [])
|
||||||
|
self.assertIsNone(f.env)
|
||||||
|
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
|
||||||
|
|
||||||
|
def test_a_failed_install_writes_no_flag(self):
|
||||||
|
f = kfake(install_rc=100)
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 3, rep)
|
||||||
|
self.assertIsNone(f.env)
|
||||||
|
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||||
|
|
||||||
|
def test_a_postinst_without_update_grub_is_regenerated_and_proved(self):
|
||||||
|
f = kfake(grub_runs_in_postinst=False)
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertIn(f"felhom-oneshot-{NEW}", f.cfg)
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||||
|
|
||||||
|
# --- the reboot, the boot, good / revert ---
|
||||||
|
def test_reboot_of_a_staged_step(self):
|
||||||
|
f = staged()
|
||||||
|
rc, rep = kmode(f, "kernel-reboot")
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(f.reboots, [OLD])
|
||||||
|
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||||
|
self.assertEqual(st["phase"], "oneshot")
|
||||||
|
self.assertIn("host_services", st["health_before"])
|
||||||
|
|
||||||
|
def test_reboot_needs_a_staged_step(self):
|
||||||
|
self.refused(kfake(), "R22", mode="kernel-reboot")
|
||||||
|
|
||||||
|
def boot_into(self, f, kver):
|
||||||
|
f.running = kver
|
||||||
|
if f.env and f.env.get("felhom_next"):
|
||||||
|
f.env["felhom_next"] = "" # GRUB cleared it (01_felhom_oneshot)
|
||||||
|
return kmode(f, "kernel-boot")
|
||||||
|
|
||||||
|
def test_healthy_one_shot_becomes_the_default(self):
|
||||||
|
f = staged()
|
||||||
|
kmode(f, "kernel-reboot")
|
||||||
|
rc, rep = self.boot_into(f, NEW)
|
||||||
|
self.assertEqual((rc, rep["kernel_event"], rep["kernel"]["phase"]), (0, "judging", "judging"), rep)
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "judging does not move the default")
|
||||||
|
rc, rep = kmode(f, "kernel-good")
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), NEW)
|
||||||
|
self.assertEqual(rep["kernel"]["phase"], "good")
|
||||||
|
|
||||||
|
def test_a_panic_falls_back_and_is_recorded(self):
|
||||||
|
f = staged()
|
||||||
|
kmode(f, "kernel-reboot")
|
||||||
|
rc, rep = self.boot_into(f, OLD)
|
||||||
|
self.assertEqual((rc, rep["kernel_event"]), (0, "fell_back"), rep)
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||||
|
self.refused(f, "R22", mode="kernel-good")
|
||||||
|
|
||||||
|
def test_one_self_revert_then_never_again(self):
|
||||||
|
f = staged()
|
||||||
|
kmode(f, "kernel-reboot")
|
||||||
|
self.boot_into(f, NEW)
|
||||||
|
rc, rep = kmode(f, "kernel-revert", reason="the customer guest is not running")
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(f.reboots, [OLD, NEW])
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "the self-revert boots the default, the old kernel")
|
||||||
|
rc, rep = self.boot_into(f, OLD)
|
||||||
|
self.assertEqual(rep["kernel_event"], "self_reverted")
|
||||||
|
# the same step can never revert again
|
||||||
|
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||||
|
st["phase"] = "judging"
|
||||||
|
f.tree[osapply.KERNEL_STATE] = json.dumps(st)
|
||||||
|
f.running = NEW
|
||||||
|
self.refused(f, "R22", mode="kernel-revert")
|
||||||
|
self.assertEqual(len(f.reboots), 2)
|
||||||
|
|
||||||
|
def test_a_revert_that_comes_back_new_is_never_retried(self):
|
||||||
|
f = staged()
|
||||||
|
kmode(f, "kernel-reboot")
|
||||||
|
self.boot_into(f, NEW)
|
||||||
|
kmode(f, "kernel-revert")
|
||||||
|
rc, rep = self.boot_into(f, NEW)
|
||||||
|
self.assertEqual(rep["kernel_event"], "revert_failed")
|
||||||
|
self.refused(f, "R22", mode="kernel-revert")
|
||||||
|
|
||||||
|
def test_revert_refuses_an_unknown_default(self):
|
||||||
|
f = staged()
|
||||||
|
kmode(f, "kernel-reboot")
|
||||||
|
self.boot_into(f, NEW)
|
||||||
|
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-9.9.9-1-pve-advanced-{U}"\n'
|
||||||
|
f.cfg = f.render()
|
||||||
|
self.refused(f, "R20", mode="kernel-revert")
|
||||||
|
self.assertEqual(len(f.reboots), 1)
|
||||||
|
|
||||||
|
def test_cancel_clears_the_flag(self):
|
||||||
|
f = staged()
|
||||||
|
rc, rep = kmode(f, "kernel-cancel")
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertFalse(f.env.get("felhom_next"), "the flag is gone")
|
||||||
|
self.assertEqual(rep["kernel"]["phase"], "cancelled")
|
||||||
|
|
||||||
|
def test_status_is_read_only_and_names_setup_problems(self):
|
||||||
|
f = kfake(esp_fs="ext4")
|
||||||
|
rc, rep = kmode(f, "kernel-status")
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertTrue(rep["kernel"]["setup_problems"])
|
||||||
|
self.assertEqual(f.tree, {})
|
||||||
|
self.assertFalse([c for c in f.calls if c[1][0] in ("update-grub", "systemctl", "apt-get")])
|
||||||
|
|
||||||
|
def test_facts_carry_the_kernel_lane(self):
|
||||||
|
f = staged()
|
||||||
|
f.plan = {"release_id": "facts", "layer": "host", "lane": "fast", "vmid": 9201, "mode": "facts", "packages": []}
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
h = rep["facts"]["host"]
|
||||||
|
self.assertEqual(h["kernel_lane"]["phase"], "staged")
|
||||||
|
self.assertEqual(h["kernel_next_boot"], NEW)
|
||||||
|
self.assertIn("one-shot", h["kernel_next_boot_source"])
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -56,6 +56,18 @@ class FakeHost:
|
|||||||
def log(self, line):
|
def log(self, line):
|
||||||
self.logs.append(line)
|
self.logs.append(line)
|
||||||
|
|
||||||
|
# controller-image (R-861 (a) A1): the ref arrives on stdin and is written INSIDE the guest by root.
|
||||||
|
stdin = b""
|
||||||
|
guest_writes = None
|
||||||
|
|
||||||
|
def read_stdin(self, limit):
|
||||||
|
return self.stdin[:limit + 1]
|
||||||
|
|
||||||
|
def write_guest_image(self, vmid, data):
|
||||||
|
if self.guest_writes is None:
|
||||||
|
self.guest_writes = []
|
||||||
|
self.guest_writes.append((vmid, data))
|
||||||
|
|
||||||
|
|
||||||
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
|
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
|
||||||
[Unit]
|
[Unit]
|
||||||
@@ -316,5 +328,53 @@ class Refuses(unittest.TestCase):
|
|||||||
self.refused(h, ["wg", "/etc/shadow"], "A1")
|
self.refused(h, ["wg", "/etc/shadow"], "A1")
|
||||||
|
|
||||||
|
|
||||||
|
class ControllerImage(unittest.TestCase):
|
||||||
|
"""R-861 (a) A1 (`09` §3 decision 165): the agent can no longer `tee` any image ref into the guest. The root verb
|
||||||
|
reads the ref on stdin, requires our registry + our repository + an x.y.z tag, and writes the guest file itself.
|
||||||
|
RED-PROOF: on the pre-A1 wrapper `controller-image` is not a verb (A1 usage, rc 2) — the accepted case fails."""
|
||||||
|
|
||||||
|
def go(self, ref, *argv):
|
||||||
|
h = FakeHost()
|
||||||
|
h.stdin = ref.encode() if isinstance(ref, str) else ref
|
||||||
|
return h, run(h, *(argv or ("controller-image", "9201")))
|
||||||
|
|
||||||
|
def test_our_controller_ref_is_written_in_the_guest(self):
|
||||||
|
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")
|
||||||
|
self.assertEqual(rc, 0, h.logs)
|
||||||
|
self.assertEqual(h.guest_writes, [(9201, b"gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")])
|
||||||
|
|
||||||
|
def test_a_foreign_image_is_refused(self):
|
||||||
|
for ref in ("docker.io/library/alpine:latest\n", "alpine\n",
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller:latest\n",
|
||||||
|
"gitea.dooplex.hu/admin/other:0.1.0\n",
|
||||||
|
"evil.example/admin/felhom-controller:0.301.0\n",
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller:0.301.0\nalpine\n",
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller:0.301.0 x\n",
|
||||||
|
"", "\n"):
|
||||||
|
h, rc = self.go(ref)
|
||||||
|
self.assertEqual(rc, 3, f"{ref!r} was accepted")
|
||||||
|
self.assertFalse(h.guest_writes, f"{ref!r} wrote the guest file")
|
||||||
|
self.assertTrue(any("[I1]" in l for l in h.logs), h.logs)
|
||||||
|
|
||||||
|
def test_oversize_stdin_is_refused(self):
|
||||||
|
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0" + " " * 300)
|
||||||
|
self.assertEqual(rc, 3)
|
||||||
|
self.assertFalse(h.guest_writes)
|
||||||
|
|
||||||
|
def test_vmid_must_be_numeric(self):
|
||||||
|
for argv in (("controller-image", "9201;id"), ("controller-image", "-1"), ("controller-image",),
|
||||||
|
("controller-image", "9201", "9202")):
|
||||||
|
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n", *argv)
|
||||||
|
self.assertIn(rc, (2, 3), argv)
|
||||||
|
self.assertFalse(h.guest_writes, argv)
|
||||||
|
|
||||||
|
def test_self_check_names_the_verb(self):
|
||||||
|
import io, contextlib
|
||||||
|
buf = io.StringIO()
|
||||||
|
with contextlib.redirect_stdout(buf):
|
||||||
|
pa.main(["--self-check"])
|
||||||
|
self.assertIn("controller-image", buf.getvalue())
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main(verbosity=2)
|
unittest.main(verbosity=2)
|
||||||
|
|||||||
@@ -103,6 +103,29 @@ func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Tim
|
|||||||
return e.at, ok
|
return e.at, ok
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// KnownBackupSuccesses returns every saved success as a host-report record (the collector's
|
||||||
|
// KnownBackupReporter, so the hub keeps its evidence across a restart). Only the tier, guest and start
|
||||||
|
// time are known; the archive name is not saved and stays empty. Sorted for a stable report.
|
||||||
|
func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
|
||||||
|
if s == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
out := make([]hub.Backup, 0, len(s.last))
|
||||||
|
for _, e := range s.last {
|
||||||
|
out = append(out, hub.Backup{TargetID: e.target, VMID: e.vmid, Success: true, CrashConsistent: true,
|
||||||
|
StartedAt: e.at.Format(time.RFC3339), UncoveredVolumes: []string{}})
|
||||||
|
}
|
||||||
|
sort.Slice(out, func(i, j int) bool {
|
||||||
|
if out[i].TargetID != out[j].TargetID {
|
||||||
|
return out[i].TargetID < out[j].TargetID
|
||||||
|
}
|
||||||
|
return out[i].VMID < out[j].VMID
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func (s *BackupSuccessState) saveLocked() error {
|
func (s *BackupSuccessState) saveLocked() error {
|
||||||
entries := make([]backupSuccessJSON, 0, len(s.last))
|
entries := make([]backupSuccessJSON, 0, len(s.last))
|
||||||
for _, e := range s.last {
|
for _, e := range s.last {
|
||||||
|
|||||||
@@ -57,3 +57,20 @@ func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
|
|||||||
t.Fatal("a corrupt file must read as nothing known")
|
t.Fatal("a corrupt file must read as nothing known")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The saved successes come back as host-report records: success, tier, guest, start time (R-894 → the
|
||||||
|
// host report, 2026-10-09). A new state read from the same file gives the same records (the restart case).
|
||||||
|
func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||||
|
s := NewBackupSuccessState(path)
|
||||||
|
if err := s.RecordBackupSuccess("felhom-backup", hub.Backup{VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got := NewBackupSuccessState(path).KnownBackupSuccesses()
|
||||||
|
if len(got) != 1 || !got[0].Success || got[0].TargetID != "felhom-backup" || got[0].VMID != 9201 || got[0].StartedAt != "2026-10-09T02:40:16Z" {
|
||||||
|
t.Fatalf("got %+v", got)
|
||||||
|
}
|
||||||
|
if got[0].UncoveredVolumes == nil {
|
||||||
|
t.Fatal("uncovered_volumes must marshal as [], not null")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sort"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ForeignKeyLedger (R-366 slice 2, `09` §3 decision 168) holds, per backup tier, the whole-guest archives the
|
||||||
|
// restore-test pick skipped because another key wrote them (R-727 — an earlier install of this box). Since R-727 the
|
||||||
|
// skip was one INFO log line per archive and nothing else, so after a reinstall the operator was never told that the
|
||||||
|
// box's older whole-guest copies are unreadable to it. The host report carries this ledger; the hub raises ONE
|
||||||
|
// operator event when it changes.
|
||||||
|
//
|
||||||
|
// It reports nil until a tier has been evaluated since the agent started, so a restart does not read as "the set
|
||||||
|
// changed to empty" (the hub keeps its last state for an absent field).
|
||||||
|
type ForeignKeyLedger struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
byTarget map[string]hub.ForeignKeyArchives
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewForeignKeyLedger builds an empty ledger.
|
||||||
|
func NewForeignKeyLedger() *ForeignKeyLedger {
|
||||||
|
return &ForeignKeyLedger{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *ForeignKeyLedger) set(target string, n int, oldest, newest int64) {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
if l.byTarget == nil {
|
||||||
|
l.byTarget = map[string]hub.ForeignKeyArchives{}
|
||||||
|
}
|
||||||
|
e := hub.ForeignKeyArchives{Target: target, Count: n}
|
||||||
|
if n > 0 {
|
||||||
|
e.Oldest = time.Unix(oldest, 0).UTC().Format(time.RFC3339)
|
||||||
|
e.Newest = time.Unix(newest, 0).UTC().Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
l.byTarget[target] = e
|
||||||
|
}
|
||||||
|
|
||||||
|
// ForeignKeyArchives implements hub.ForeignKeyArchiveReporter: nil before any evaluation; otherwise the tiers that
|
||||||
|
// hold such archives (`Tiers` empty, never nil, when none do), sorted by tier.
|
||||||
|
func (l *ForeignKeyLedger) ForeignKeyArchives(context.Context) *hub.ForeignKeyArchivesStanza {
|
||||||
|
l.mu.Lock()
|
||||||
|
defer l.mu.Unlock()
|
||||||
|
if l.byTarget == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := []hub.ForeignKeyArchives{}
|
||||||
|
for _, e := range l.byTarget {
|
||||||
|
if e.Count > 0 {
|
||||||
|
out = append(out, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Slice(out, func(i, j int) bool { return out[i].Target < out[j].Target })
|
||||||
|
return &hub.ForeignKeyArchivesStanza{Tiers: out}
|
||||||
|
}
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-366 slice 2 (`09` §3 decision 168) — the restore-test's skip of an archive written with another key stops being
|
||||||
|
// silent: the pick records, per tier, how many it skipped and their time range, and the host report carries it.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): remove the `r.foreign.set(...)` call from PickSettledRestoreCandidateOn → this fails
|
||||||
|
// with "after one evaluation the ledger must report felhom-pbs: 2 archives …; got []". Restored.
|
||||||
|
func TestR366_PickRecordsArchivesWrittenWithAnotherKey(t *testing.T) {
|
||||||
|
api := &fakeBackupAPI{
|
||||||
|
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||||
|
content: []proxmox.StorageContent{
|
||||||
|
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
|
||||||
|
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
|
||||||
|
{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
l := NewForeignKeyLedger()
|
||||||
|
r.SetForeignKeyLedger(l)
|
||||||
|
|
||||||
|
if got := l.ForeignKeyArchives(context.Background()); got != nil {
|
||||||
|
t.Fatalf("before any evaluation the ledger must be nil (the hub keeps its state); got %v", got)
|
||||||
|
}
|
||||||
|
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
st := l.ForeignKeyArchives(context.Background())
|
||||||
|
want := hub.ForeignKeyArchives{Target: "felhom-pbs", Count: 2, Oldest: "2026-09-16T17:27:32Z", Newest: "2026-09-16T21:59:54Z"}
|
||||||
|
var got []hub.ForeignKeyArchives
|
||||||
|
if st != nil {
|
||||||
|
got = st.Tiers
|
||||||
|
}
|
||||||
|
if len(got) != 1 || got[0] != want {
|
||||||
|
t.Fatalf("after one evaluation the ledger must report felhom-pbs: 2 archives 2026-09-16T17:27:32Z…21:59:54Z; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Evaluated and none found → the stanza with `tiers: []`; not evaluated → no stanza at all. Never a null on the wire.
|
||||||
|
func TestR366_EvaluatedWithNoneIsAnEmptyList(t *testing.T) {
|
||||||
|
api := &fakeBackupAPI{
|
||||||
|
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||||
|
content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}},
|
||||||
|
}
|
||||||
|
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||||
|
l := NewForeignKeyLedger()
|
||||||
|
r.SetForeignKeyLedger(l)
|
||||||
|
b, _ := json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
|
||||||
|
if strings.Contains(string(b), "foreign_key_archives") {
|
||||||
|
t.Fatalf("not evaluated must omit the stanza; got %s", b)
|
||||||
|
}
|
||||||
|
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
b, _ = json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
|
||||||
|
if !strings.Contains(string(b), `"foreign_key_archives":{"tiers":[]}`) {
|
||||||
|
t.Fatalf("evaluated with none must report tiers: []; got %s", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -66,8 +66,13 @@ type BackupRunner struct {
|
|||||||
// due-check is served from the local-API handler goroutines.
|
// due-check is served from the local-API handler goroutines.
|
||||||
rejectedMu sync.Mutex
|
rejectedMu sync.Mutex
|
||||||
rejected map[string]struct{}
|
rejected map[string]struct{}
|
||||||
|
// foreign (R-366 slice 2) records, per tier, the archives the pick skipped as another key's. nil = not wired.
|
||||||
|
foreign *ForeignKeyLedger
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetForeignKeyLedger wires the R-366 slice-2 ledger the host report reads.
|
||||||
|
func (r *BackupRunner) SetForeignKeyLedger(l *ForeignKeyLedger) { r.foreign = l }
|
||||||
|
|
||||||
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
|
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
|
||||||
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
|
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
|
||||||
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
|
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
|
||||||
@@ -370,6 +375,8 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
|||||||
var best string
|
var best string
|
||||||
var bestCTime int64 = -1
|
var bestCTime int64 = -1
|
||||||
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
|
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
|
||||||
|
var foreignN int // R-366 slice 2: archives skipped as another key's, and their time range
|
||||||
|
var foreignMin, foreignMax int64
|
||||||
for _, e := range contents {
|
for _, e := range contents {
|
||||||
if e.Content != "backup" {
|
if e.Content != "backup" {
|
||||||
continue
|
continue
|
||||||
@@ -384,6 +391,13 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
|||||||
}
|
}
|
||||||
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
|
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
|
||||||
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
|
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
|
||||||
|
foreignN++
|
||||||
|
if foreignMin == 0 || e.CTime < foreignMin {
|
||||||
|
foreignMin = e.CTime
|
||||||
|
}
|
||||||
|
if e.CTime > foreignMax {
|
||||||
|
foreignMax = e.CTime
|
||||||
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
|
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
|
||||||
@@ -420,6 +434,9 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
|||||||
bestCTime, best = e.CTime, e.VolID
|
bestCTime, best = e.CTime, e.VolID
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if r.foreign != nil {
|
||||||
|
r.foreign.set(target, foreignN, foreignMin, foreignMax)
|
||||||
|
}
|
||||||
if best == "" {
|
if best == "" {
|
||||||
return "", time.Time{}, nil
|
return "", time.Time{}, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -145,7 +145,8 @@ var manifest = []Capability{
|
|||||||
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
||||||
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
||||||
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
||||||
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
|
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
|
||||||
|
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
|
||||||
|
|
||||||
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
|
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
|
||||||
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
|
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
|
||||||
|
|||||||
@@ -200,7 +200,8 @@ func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
|
// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
|
||||||
// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the
|
// FELHOM_CONTROLLERSWAP grants: with the write grant removed (since R-861 (a) A1 the `felhom-priv-apply controller-image`
|
||||||
|
// line; before it, an agent `tee /etc/felhom-controller-image`), the
|
||||||
// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
|
// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
|
||||||
// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
|
// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
|
||||||
func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
|
func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
|
||||||
@@ -210,7 +211,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
|
|||||||
}
|
}
|
||||||
var kept []string
|
var kept []string
|
||||||
for _, ln := range strings.Split(string(data), "\n") {
|
for _, ln := range strings.Split(string(data), "\n") {
|
||||||
if strings.Contains(ln, "tee /etc/felhom-controller-image") {
|
if strings.Contains(ln, "felhom-priv-apply ^controller-image") { // R-861 (a) A1: the write's grant
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
kept = append(kept, ln)
|
kept = append(kept, ln)
|
||||||
@@ -229,7 +230,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
|
|||||||
}
|
}
|
||||||
cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
|
cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
|
||||||
if matchesAny(cmdline, entries) {
|
if matchesAny(cmdline, entries) {
|
||||||
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant")
|
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping its grant")
|
||||||
}
|
}
|
||||||
if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
|
if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
|
||||||
t.Errorf("controllerswap-write should be covered by the real sudoers")
|
t.Errorf("controllerswap-write should be covered by the real sudoers")
|
||||||
|
|||||||
@@ -49,6 +49,10 @@ var r861Injections = []string{
|
|||||||
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
|
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
|
||||||
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
|
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
|
||||||
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
|
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
|
||||||
|
// R-861 (a) A1 (decision 165): the agent wrote ANY image ref into the guest by `tee` — now only the root verb may
|
||||||
|
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
|
||||||
|
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
|
||||||
|
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSudoersRefusesTheR861Injections(t *testing.T) {
|
func TestSudoersRefusesTheR861Injections(t *testing.T) {
|
||||||
@@ -93,3 +97,42 @@ func TestSudoersFstrimRuleIsExact(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-861 (a) A1: the managed controller update still has its route — the root verb, one numeric vmid.
|
||||||
|
func TestSudoersAllowsTheControllerImageVerb(t *testing.T) {
|
||||||
|
data, err := os.ReadFile(sudoersPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !matchesAny("/usr/local/sbin/felhom-priv-apply controller-image 9201", parseSudoersEntries(t, string(data))) {
|
||||||
|
t.Fatal("the sudoers does not allow `felhom-priv-apply controller-image 9201` — a managed controller update cannot write its image")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-861 (b) B2 (decision 165, hygiene): felhom-op's `pct start|stop|unlock` grants are ONE numeric vmid each. The old
|
||||||
|
// glob `[0-9]*` eats spaces, so `pct stop 9201 --skiplock 1` and two vmids matched.
|
||||||
|
// RED-PROOF: on the pre-B2 felhom-op.sudoers (`/usr/sbin/pct stop [0-9]*`) the decoys match.
|
||||||
|
func TestFelhomOpSudoersPctIsExact(t *testing.T) {
|
||||||
|
data, err := os.ReadFile("../../configs/felhom-op.sudoers")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
entries := parseSudoersEntries(t, string(data))
|
||||||
|
for _, ok := range []string{"/usr/sbin/pct start 9201", "/usr/sbin/pct stop 9201", "/usr/sbin/pct unlock 9201", "/usr/sbin/pct list"} {
|
||||||
|
if !matchesAny(ok, entries) {
|
||||||
|
t.Errorf("felhom-op lost a repair verb: %s", ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, bad := range []string{
|
||||||
|
"/usr/sbin/pct stop 9201 --skiplock 1",
|
||||||
|
"/usr/sbin/pct start 9201 9202",
|
||||||
|
"/usr/sbin/pct unlock 9201 --whatever",
|
||||||
|
"/usr/sbin/pct start 92a1",
|
||||||
|
"/usr/sbin/pct stop ",
|
||||||
|
"/usr/sbin/pct destroy 9201",
|
||||||
|
} {
|
||||||
|
if matchesAny(bad, entries) {
|
||||||
|
t.Errorf("felhom-op's sudoers allows %q", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+50
-10
@@ -59,8 +59,26 @@ var (
|
|||||||
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
|
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
|
||||||
// which is the one fact the customer needs to recognise it.
|
// which is the one fact the customer needs to recognise it.
|
||||||
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
|
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
|
||||||
|
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
|
||||||
|
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
|
||||||
|
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
|
||||||
|
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
|
||||||
|
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
|
||||||
|
// something we did not check.
|
||||||
|
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
|
||||||
|
// (-1 = unknown: the retained list itself could not be read). No secret.
|
||||||
|
type RetainedUncheckedError struct {
|
||||||
|
Unchecked int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *RetainedUncheckedError) Error() string {
|
||||||
|
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
|
||||||
|
}
|
||||||
|
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
|
||||||
|
|
||||||
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
|
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
|
||||||
// carries no secret — not the code, not the bundle, not the repository password.
|
// carries no secret — not the code, not the bundle, not the repository password.
|
||||||
type RetainedMatch struct {
|
type RetainedMatch struct {
|
||||||
@@ -102,8 +120,10 @@ type RetainedBlob struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
|
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
|
||||||
// slice is a clean "none". R-311.
|
// slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
|
||||||
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
|
// blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
|
||||||
|
// (R-304): each is a package the code was never tried against.
|
||||||
|
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
|
||||||
|
|
||||||
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
|
||||||
type OffsiteKeyRecoverer struct {
|
type OffsiteKeyRecoverer struct {
|
||||||
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
|||||||
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
|
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
|
||||||
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
|
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
|
||||||
// about our own incuriosity, read by the customer as a statement about their code.
|
// about our own incuriosity, read by the customer as a statement about their code.
|
||||||
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
|
m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
|
||||||
|
if ok {
|
||||||
return "", &RetainedOpenedError{Match: m}
|
return "", &RetainedOpenedError{Match: m}
|
||||||
}
|
}
|
||||||
|
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
|
||||||
|
if unchecked != 0 {
|
||||||
|
return "", &RetainedUncheckedError{Unchecked: unchecked}
|
||||||
|
}
|
||||||
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
|
||||||
}
|
}
|
||||||
if bundle.ResticRepoPassword == "" {
|
if bundle.ResticRepoPassword == "" {
|
||||||
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
|
|||||||
// function breaking is the behaviour we had before it existed.
|
// function breaking is the behaviour we had before it existed.
|
||||||
//
|
//
|
||||||
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
|
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
|
||||||
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
|
//
|
||||||
|
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
|
||||||
|
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
|
||||||
|
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
|
||||||
|
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
|
||||||
if r.FetchRetained == nil {
|
if r.FetchRetained == nil {
|
||||||
return RetainedMatch{}, false
|
return RetainedMatch{}, false, 0
|
||||||
}
|
}
|
||||||
blobs, _, err := r.FetchRetained(ctx)
|
blobs, withheld, err := r.FetchRetained(ctx)
|
||||||
if err != nil || len(blobs) == 0 {
|
if err != nil {
|
||||||
return RetainedMatch{}, false
|
return RetainedMatch{}, false, -1
|
||||||
|
}
|
||||||
|
if withheld < 0 {
|
||||||
|
withheld = 0
|
||||||
|
}
|
||||||
|
if len(blobs) == 0 {
|
||||||
|
return RetainedMatch{}, false, withheld
|
||||||
}
|
}
|
||||||
limit := r.MaxRetainedTried
|
limit := r.MaxRetainedTried
|
||||||
if limit <= 0 {
|
if limit <= 0 {
|
||||||
limit = defaultMaxRetainedTried
|
limit = defaultMaxRetainedTried
|
||||||
}
|
}
|
||||||
|
unchecked := withheld
|
||||||
|
if len(blobs) > limit {
|
||||||
|
unchecked += len(blobs) - limit
|
||||||
|
}
|
||||||
for i, rb := range blobs {
|
for i, rb := range blobs {
|
||||||
if i >= limit {
|
if i >= limit {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
if len(rb.Blob) == 0 {
|
if len(rb.Blob) == 0 {
|
||||||
|
unchecked++
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
|
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
|
||||||
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
|
|||||||
// correct and must be told so — but the history behind it still cannot be reopened, and
|
// correct and must be told so — but the history behind it still cannot be reopened, and
|
||||||
// saying otherwise would be a promise this path cannot keep.
|
// saying otherwise would be a promise this path cannot keep.
|
||||||
HasResticPassword: bundle.ResticRepoPassword != "",
|
HasResticPassword: bundle.ResticRepoPassword != "",
|
||||||
}, true
|
}, true, 0
|
||||||
}
|
}
|
||||||
return RetainedMatch{}, false
|
return RetainedMatch{}, false, unchecked
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
|
// FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
|
||||||
// worst outcome of this feature breaking is the behaviour we had before it.
|
// retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
|
||||||
|
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
|
||||||
//
|
//
|
||||||
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
|
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
|
||||||
// gets a new, unexplained failure mode → this FAILS.
|
// gets a new, unexplained failure mode → this FAILS.
|
||||||
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
|
|||||||
if containsStr(err.Error(), "hub exploded") {
|
if containsStr(err.Error(), "hub exploded") {
|
||||||
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
|
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
|
||||||
}
|
}
|
||||||
|
var ue *RetainedUncheckedError
|
||||||
|
if !errors.As(err, &ue) || ue.Unchecked != -1 {
|
||||||
|
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
|
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
|
||||||
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
|
|||||||
return false
|
return false
|
||||||
})()
|
})()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
|
||||||
|
//
|
||||||
|
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
|
||||||
|
// typing")? It may be only when the hub withheld nothing and every served package was tried.
|
||||||
|
//
|
||||||
|
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
|
||||||
|
// → they FAIL; the all-tried case keeps passing.
|
||||||
|
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||||
|
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||||
|
const code = "a code that opens nothing whatsoever in this test"
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
blobs int
|
||||||
|
withheld int
|
||||||
|
limit int
|
||||||
|
unchecked int // 0 = must be a plain wrong code
|
||||||
|
}{
|
||||||
|
{"all tried, nothing withheld", 2, 0, 6, 0},
|
||||||
|
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
|
||||||
|
{"only withheld rows, nothing served", 0, 2, 6, 2},
|
||||||
|
{"the attempt cap stopped the loop", 5, 0, 2, 3},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
blobs := make([]RetainedBlob, 0, c.blobs)
|
||||||
|
for i := 0; i < c.blobs; i++ {
|
||||||
|
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
|
||||||
|
}
|
||||||
|
_, err := OffsiteKeyRecoverer{
|
||||||
|
Fetch: fetcherFor(current),
|
||||||
|
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
|
||||||
|
return blobs, c.withheld, nil
|
||||||
|
},
|
||||||
|
MaxRetainedTried: c.limit,
|
||||||
|
}.RecoverOffsiteRepoPassword(context.Background(), code)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("a code that opens nothing succeeded")
|
||||||
|
}
|
||||||
|
var ue *RetainedUncheckedError
|
||||||
|
isUnchecked := errors.As(err, &ue)
|
||||||
|
if c.unchecked == 0 {
|
||||||
|
if isUnchecked {
|
||||||
|
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
|
||||||
|
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A malformed (empty) served package counts as not tried.
|
||||||
|
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
|
||||||
|
ensureAge(t)
|
||||||
|
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
|
||||||
|
_, err := OffsiteKeyRecoverer{
|
||||||
|
Fetch: fetcherFor(current),
|
||||||
|
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
|
||||||
|
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
|
||||||
|
var ue *RetainedUncheckedError
|
||||||
|
if !errors.As(err, &ue) || ue.Unchecked != 1 {
|
||||||
|
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+66
-5
@@ -66,6 +66,13 @@ type ProvenRestoreTestReporter interface {
|
|||||||
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// KnownBackupReporter is the newest SUCCESSFUL backup per tier and guest kept on disk (R-894's
|
||||||
|
// backup-success-state.json). collectBackups folds it into the report so a restart does not erase the
|
||||||
|
// hub's evidence of a backup that ran minutes before it (the kernel-night shape, 2026-10-09).
|
||||||
|
type KnownBackupReporter interface {
|
||||||
|
KnownBackupSuccesses() []Backup
|
||||||
|
}
|
||||||
|
|
||||||
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
||||||
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
||||||
type PBSReporter interface {
|
type PBSReporter interface {
|
||||||
@@ -105,6 +112,7 @@ type Collector struct {
|
|||||||
backups BackupReporter
|
backups BackupReporter
|
||||||
restoreTests RestoreTestReporter
|
restoreTests RestoreTestReporter
|
||||||
provenTests ProvenRestoreTestReporter
|
provenTests ProvenRestoreTestReporter
|
||||||
|
knownBackups KnownBackupReporter // R-894 file → host report after a restart (nil → in-memory only)
|
||||||
pbs PBSReporter
|
pbs PBSReporter
|
||||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||||
@@ -115,6 +123,7 @@ type Collector struct {
|
|||||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||||
|
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
|
||||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||||
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
|
||||||
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
|
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
|
||||||
@@ -228,6 +237,18 @@ func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
|
|||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ForeignKeyArchiveReporter is the R-366 slice-2 seam: the restore-test's ledger of archives written with another
|
||||||
|
// key. nil = not evaluated yet (the stanza is omitted and the hub keeps its state).
|
||||||
|
type ForeignKeyArchiveReporter interface {
|
||||||
|
ForeignKeyArchives(ctx context.Context) *ForeignKeyArchivesStanza
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetForeignKeyArchiveReporter wires the restore-test's foreign-key ledger (R-366 slice 2; nil-safe → omitted).
|
||||||
|
func (c *Collector) SetForeignKeyArchiveReporter(r ForeignKeyArchiveReporter) *Collector {
|
||||||
|
c.foreignKey = r
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
// SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted).
|
// SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted).
|
||||||
func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector {
|
func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector {
|
||||||
c.diskTrim = r
|
c.diskTrim = r
|
||||||
@@ -394,6 +415,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
|||||||
if c.diskTrim != nil {
|
if c.diskTrim != nil {
|
||||||
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
|
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
|
||||||
}
|
}
|
||||||
|
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
|
||||||
|
if c.foreignKey != nil {
|
||||||
|
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
|
||||||
|
}
|
||||||
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
|
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
|
||||||
if c.selfUpdate != nil {
|
if c.selfUpdate != nil {
|
||||||
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
|
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
|
||||||
@@ -559,14 +584,50 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
|
|||||||
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
|
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
|
||||||
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
|
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
|
||||||
// list so the collection always marshals as [].
|
// list so the collection always marshals as [].
|
||||||
|
//
|
||||||
|
// The saved successes (SetKnownBackupReporter) are added for each tier and guest the in-memory list has
|
||||||
|
// no success for at or after the saved time. Why: the in-memory list is empty after an agent restart,
|
||||||
|
// and the hub's backup-freshness check reads only what the reports carried. Measured 2026-10-09 on
|
||||||
|
// demo-felhom: the night backup landed 02:40 UTC, the kernel step restarted the host at 02:44, no report
|
||||||
|
// fell in between, so two kernel nights in a row left no trace and the hub alarmed „newest backup is 48h
|
||||||
|
// old" at 03:00. Only SUCCESSES are saved, so a failure is never hidden and never invented.
|
||||||
|
// Pinned by TestCollectBackups_SavedSuccessSurvivesARestart.
|
||||||
func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
||||||
if c.backups == nil {
|
out := []Backup{}
|
||||||
return []Backup{}
|
if c.backups != nil {
|
||||||
|
if b := c.backups.Backups(ctx); b != nil {
|
||||||
|
out = append(out, b...)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if b := c.backups.Backups(ctx); b != nil {
|
if c.knownBackups == nil {
|
||||||
return b
|
return out
|
||||||
}
|
}
|
||||||
return []Backup{}
|
for _, k := range c.knownBackups.KnownBackupSuccesses() {
|
||||||
|
kt, err := time.Parse(time.RFC3339, k.StartedAt)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
covered := false
|
||||||
|
for _, b := range out {
|
||||||
|
if !b.Success || b.TargetID != k.TargetID || b.VMID != k.VMID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if bt, err := time.Parse(time.RFC3339, b.StartedAt); err == nil && !bt.Before(kt) {
|
||||||
|
covered = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !covered {
|
||||||
|
out = append(out, k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetKnownBackupReporter wires the R-894 saved successes into the host report (nil-safe → in-memory only).
|
||||||
|
func (c *Collector) SetKnownBackupReporter(r KnownBackupReporter) *Collector {
|
||||||
|
c.knownBackups = r
|
||||||
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
||||||
|
|||||||
@@ -0,0 +1,62 @@
|
|||||||
|
package hub
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeMemBackups []Backup
|
||||||
|
|
||||||
|
func (f fakeMemBackups) Backups(context.Context) []Backup { return f }
|
||||||
|
|
||||||
|
type fakeKnownBackups []Backup
|
||||||
|
|
||||||
|
func (f fakeKnownBackups) KnownBackupSuccesses() []Backup { return f }
|
||||||
|
|
||||||
|
// The consequence, not the mechanism: after a restart (empty in-memory list) the host report still carries
|
||||||
|
// the night's backup, so the hub's freshness check sees it. Measured 2026-10-09 on demo-felhom: without this
|
||||||
|
// the report carried backups: [] and the hub alarmed „newest backup is 48h old" an hour after a good backup.
|
||||||
|
// RED-PROOF: return before the saved-success loop in collectBackups → the first case reports nothing → fails.
|
||||||
|
func TestCollectBackups_SavedSuccessSurvivesARestart(t *testing.T) {
|
||||||
|
saved := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}
|
||||||
|
|
||||||
|
t.Run("restart: memory empty, the saved success is reported", func(t *testing.T) {
|
||||||
|
c := &Collector{backups: fakeMemBackups(nil), knownBackups: fakeKnownBackups{saved}}
|
||||||
|
got := c.collectBackups(context.Background())
|
||||||
|
if len(got) != 1 || got[0].StartedAt != saved.StartedAt || !got[0].Success || got[0].TargetID != "felhom-backup" {
|
||||||
|
t.Fatalf("want the saved success, got %+v", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("memory holds the same or a newer success: no second entry", func(t *testing.T) {
|
||||||
|
mem := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z", Archive: "a"}
|
||||||
|
c := &Collector{backups: fakeMemBackups{mem}, knownBackups: fakeKnownBackups{saved}}
|
||||||
|
if got := c.collectBackups(context.Background()); len(got) != 1 || got[0].Archive != "a" {
|
||||||
|
t.Fatalf("want only the in-memory record, got %+v", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("a newer FAILURE in memory never hides the saved success, and is kept", func(t *testing.T) {
|
||||||
|
fail := Backup{TargetID: "felhom-backup", VMID: 9201, Success: false, StartedAt: "2026-10-10T02:40:00Z", Error: "x"}
|
||||||
|
c := &Collector{backups: fakeMemBackups{fail}, knownBackups: fakeKnownBackups{saved}}
|
||||||
|
got := c.collectBackups(context.Background())
|
||||||
|
if len(got) != 2 || got[0].Success || !got[1].Success {
|
||||||
|
t.Fatalf("want the failure and the saved success, got %+v", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("another tier's success does not cover this tier", func(t *testing.T) {
|
||||||
|
other := Backup{TargetID: "felhom-pbs", VMID: 9201, Success: true, StartedAt: "2026-10-10T00:00:00Z"}
|
||||||
|
c := &Collector{backups: fakeMemBackups{other}, knownBackups: fakeKnownBackups{saved}}
|
||||||
|
if got := c.collectBackups(context.Background()); len(got) != 2 {
|
||||||
|
t.Fatalf("want both tiers, got %+v", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("no saved state wired: the in-memory list as before, never nil", func(t *testing.T) {
|
||||||
|
c := &Collector{}
|
||||||
|
if got := c.collectBackups(context.Background()); got == nil || len(got) != 0 {
|
||||||
|
t.Fatalf("want an empty non-nil list, got %#v", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -129,6 +129,12 @@ type HostReport struct {
|
|||||||
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
|
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
|
||||||
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
|
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
|
||||||
|
|
||||||
|
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
|
||||||
|
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
|
||||||
|
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
|
||||||
|
// the agent started (the hub keeps its last state); `tiers: []` = evaluated, none found.
|
||||||
|
ForeignKeyArchives *ForeignKeyArchivesStanza `json:"foreign_key_archives,omitempty"`
|
||||||
|
|
||||||
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
|
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
|
||||||
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
|
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
|
||||||
// right after the control envelope requested it (log_tail_requested); consume-once on
|
// right after the control envelope requested it (log_tail_requested); consume-once on
|
||||||
@@ -630,6 +636,17 @@ type WireOSUpdate struct {
|
|||||||
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
|
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
|
||||||
// approved for the guest is not approved for the host by that fact alone.
|
// approved for the guest is not approved for the host by that fact alone.
|
||||||
HostRelease *WireOSRelease `json:"host_release,omitempty"`
|
HostRelease *WireOSRelease `json:"host_release,omitempty"`
|
||||||
|
// Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the
|
||||||
|
// household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel
|
||||||
|
// step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step.
|
||||||
|
Kernel *WireKernelStep `json:"kernel,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo).
|
||||||
|
type WireKernelStep struct {
|
||||||
|
Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23)
|
||||||
|
Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot
|
||||||
|
NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log
|
||||||
}
|
}
|
||||||
|
|
||||||
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
|
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
|
||||||
@@ -726,3 +743,18 @@ type WireRestoreDirective struct {
|
|||||||
Archive string `json:"archive,omitempty"` // source archive/snapshot to restore from
|
Archive string `json:"archive,omitempty"` // source archive/snapshot to restore from
|
||||||
VMID int `json:"vmid,omitempty"`
|
VMID int `json:"vmid,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ForeignKeyArchivesStanza wraps the per-tier list so "evaluated, none" (`tiers: []`) differs from "not evaluated"
|
||||||
|
// (the stanza absent) without a null on the wire.
|
||||||
|
type ForeignKeyArchivesStanza struct {
|
||||||
|
Tiers []ForeignKeyArchives `json:"tiers"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ForeignKeyArchives is one tier's count of archives written with another key (R-366 slice 2): the count and the
|
||||||
|
// newest/oldest archive time (RFC3339, UTC). No key material — the fingerprints stay on the box.
|
||||||
|
type ForeignKeyArchives struct {
|
||||||
|
Target string `json:"target"`
|
||||||
|
Count int `json:"count"`
|
||||||
|
Oldest string `json:"oldest"`
|
||||||
|
Newest string `json:"newest"`
|
||||||
|
}
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ import (
|
|||||||
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
|
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
|
||||||
// and the gate sub-case fails.
|
// and the gate sub-case fails.
|
||||||
func TestAfterPrimaryBackup(t *testing.T) {
|
func TestAfterPrimaryBackup(t *testing.T) {
|
||||||
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
|
run := func(t *testing.T, failErr, path string) (calls []int, gateHeld bool) {
|
||||||
gate := &backup.InFlight{}
|
gate := &backup.InFlight{}
|
||||||
b := &fakeBackups{failErr: failErr}
|
b := &fakeBackups{failErr: failErr}
|
||||||
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
|
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
|
||||||
@@ -34,7 +34,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
|||||||
done <- struct{}{}
|
done <- struct{}{}
|
||||||
})
|
})
|
||||||
h := srv.Handler()
|
h := srv.Handler()
|
||||||
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
|
if do(t, h, "POST", path, "A", "").Code != http.StatusAccepted {
|
||||||
t.Fatal("POST /backup not accepted")
|
t.Fatal("POST /backup not accepted")
|
||||||
}
|
}
|
||||||
select {
|
select {
|
||||||
@@ -47,7 +47,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
|||||||
return calls, gateHeld
|
return calls, gateHeld
|
||||||
}
|
}
|
||||||
t.Run("success runs the leg under the gate", func(t *testing.T) {
|
t.Run("success runs the leg under the gate", func(t *testing.T) {
|
||||||
calls, held := run(t, "")
|
calls, held := run(t, "", "/backup")
|
||||||
if len(calls) != 1 {
|
if len(calls) != 1 {
|
||||||
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
|
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
|
||||||
}
|
}
|
||||||
@@ -56,8 +56,21 @@ func TestAfterPrimaryBackup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
t.Run("a failed backup runs nothing", func(t *testing.T) {
|
t.Run("a failed backup runs nothing", func(t *testing.T) {
|
||||||
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
|
if calls, _ := run(t, "vzdump exploded", "/backup"); len(calls) != 0 {
|
||||||
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
|
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
// R-899: a household press is not the night's backup — no OS leg after it. Same fake, same successful backup as
|
||||||
|
// the first sub-case; only the query differs. Red-proof: make handleBackup ignore `trigger` and this sub-case
|
||||||
|
// fails with the leg run once.
|
||||||
|
t.Run("a manual press runs nothing", func(t *testing.T) {
|
||||||
|
if calls, _ := run(t, "", "/backup?trigger=manual"); len(calls) != 0 {
|
||||||
|
t.Fatalf("the OS leg ran after a manual press: %v", calls)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("the scheduled path with the new query still runs the leg", func(t *testing.T) {
|
||||||
|
if calls, _ := run(t, "", "/backup?trigger=night"); len(calls) != 1 {
|
||||||
|
t.Fatalf("the leg ran %d time(s) after a non-manual backup, want 1", len(calls))
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -54,8 +53,8 @@ func (f *supExec) GuestExec(_ context.Context, vmid int, args ...string) (string
|
|||||||
}
|
}
|
||||||
return "", errors.New("supExec: unexpected args")
|
return "", errors.New("supExec: unexpected args")
|
||||||
}
|
}
|
||||||
func (f *supExec) GuestExecStdin(context.Context, int, io.Reader, ...string) (string, error) {
|
func (f *supExec) WriteControllerImage(context.Context, int, string) error {
|
||||||
return "", errors.New("supExec: no stdin exec expected")
|
return errors.New("supExec: no image write expected")
|
||||||
}
|
}
|
||||||
func (f *supExec) count(vmid int) int {
|
func (f *supExec) count(vmid int) int {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
@@ -41,9 +40,10 @@ func ValidControllerImage(ref string) bool { return controllerImageRe.MatchStrin
|
|||||||
// faked in tests. The single seam the swap composes over (no hand-rolled pct).
|
// faked in tests. The single seam the swap composes over (no hand-rolled pct).
|
||||||
type GuestExecutor interface {
|
type GuestExecutor interface {
|
||||||
GuestExec(ctx context.Context, vmid int, args ...string) (string, error)
|
GuestExec(ctx context.Context, vmid int, args ...string) (string, error)
|
||||||
// GuestExecStdin is GuestExec with the command's stdin fed from stdin — the swap write pipes the
|
// WriteControllerImage writes the image ref into the guest's /etc/felhom-controller-image through the ROOT
|
||||||
// image ref into an in-guest `tee` (no shell vector).
|
// verb `felhom-priv-apply controller-image <vmid>` (R-861 (a) A1, `09` §3 decision 165), which re-checks the
|
||||||
GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error)
|
// ref against our registry + repository + x.y.z. The agent no longer holds a `tee` grant into the guest.
|
||||||
|
WriteControllerImage(ctx context.Context, vmid int, image string) error
|
||||||
}
|
}
|
||||||
|
|
||||||
// ControllerSwapState is the durable record of a swap (crash-safety + status). Written before the swap
|
// ControllerSwapState is the durable record of a swap (crash-safety + status). Written before the swap
|
||||||
@@ -141,14 +141,11 @@ func (c *ControllerSwapper) imagePresent(ctx context.Context, vmid int, image st
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c *ControllerSwapper) writeImage(ctx context.Context, vmid int, image string) error {
|
func (c *ControllerSwapper) writeImage(ctx context.Context, vmid int, image string) error {
|
||||||
// Non-root path: pipe the image ref into an in-guest `tee` over stdin — no shell, no
|
// R-861 (a) A1: the ROOT verb writes the file (it re-checks the ref — a compromised agent cannot hand the guest's
|
||||||
// interpolation, no `bash -c` (the only swap vector that would have needed an arbitrary-exec
|
// bootstrap another image). The bytes are `image\n`, byte-identical to the golden's `printf '%s\n'`; the bootstrap
|
||||||
// grant). The trailing "\n" makes the on-disk bytes byte-identical to the golden's
|
// reads `IMAGE=$(cat …)` so the newline is stripped on read. image is also strict-validated (controllerImageRe)
|
||||||
// `printf '%s\n'`; the bootstrap reads `IMAGE=$(cat …)` so the newline is stripped on read
|
// upstream in Swap.
|
||||||
// (spike SPIKE-controllerswap-narrow-grants-2026-06-29). image is strict-validated
|
return c.exec.WriteControllerImage(ctx, vmid, image)
|
||||||
// (controllerImageRe) upstream in Swap; defence-in-depth, the stdin path can't smuggle anyway.
|
|
||||||
_, err := c.exec.GuestExecStdin(ctx, vmid, strings.NewReader(image+"\n"), "tee", controllerImageFile)
|
|
||||||
return err
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *ControllerSwapper) restartBootstrap(ctx context.Context, vmid int) error {
|
func (c *ControllerSwapper) restartBootstrap(ctx context.Context, vmid int) error {
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ type fakeGuestExec struct {
|
|||||||
present map[string]bool // images pulled into the guest
|
present map[string]bool // images pulled into the guest
|
||||||
good map[string]bool // images that report healthy when running
|
good map[string]bool // images that report healthy when running
|
||||||
containerImg string // image the running container currently has
|
containerImg string // image the running container currently has
|
||||||
teeStdin []string // raw bytes piped into each `tee` write (the swap's write vector)
|
teeStdin []string // image refs handed to WriteControllerImage (the root verb, R-861 (a) A1)
|
||||||
failRestart bool
|
failRestart bool
|
||||||
noHealthBlock bool // if set, .State.Health is absent ("none")
|
noHealthBlock bool // if set, .State.Health is absent ("none")
|
||||||
restartCount int // F1: .RestartCount reported by docker inspect (a crash-looper has >0)
|
restartCount int // F1: .RestartCount reported by docker inspect (a crash-looper has >0)
|
||||||
@@ -75,20 +75,15 @@ func (f *fakeGuestExec) GuestExec(_ context.Context, _ int, args ...string) (str
|
|||||||
return "", fmt.Errorf("fake: unexpected exec %v", args)
|
return "", fmt.Errorf("fake: unexpected exec %v", args)
|
||||||
}
|
}
|
||||||
|
|
||||||
// GuestExecStdin models the swap's write vector: `tee /etc/felhom-controller-image` with the image
|
// WriteControllerImage models the swap's write: the root verb `felhom-priv-apply controller-image <vmid>` (R-861
|
||||||
// piped on stdin. It records the raw stdin bytes and sets the modeled file content (newline-stripped,
|
// (a) A1). It records the ref and sets the modeled file content, as the bootstrap's `IMAGE=$(cat …)` would read it.
|
||||||
// as the bootstrap's `IMAGE=$(cat …)` read would see it).
|
func (f *fakeGuestExec) WriteControllerImage(_ context.Context, _ int, image string) error {
|
||||||
func (f *fakeGuestExec) GuestExecStdin(_ context.Context, _ int, stdin io.Reader, args ...string) (string, error) {
|
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
defer f.mu.Unlock()
|
defer f.mu.Unlock()
|
||||||
f.calls = append(f.calls, args)
|
f.calls = append(f.calls, []string{"felhom-priv-apply", "controller-image", image})
|
||||||
b, _ := io.ReadAll(stdin)
|
f.teeStdin = append(f.teeStdin, image+"\n")
|
||||||
if len(args) >= 2 && args[0] == "tee" && args[1] == controllerImageFile {
|
f.imageFile = image
|
||||||
f.teeStdin = append(f.teeStdin, string(b))
|
return nil
|
||||||
f.imageFile = strings.TrimSpace(string(b))
|
|
||||||
return string(b), nil // tee echoes stdin to stdout
|
|
||||||
}
|
|
||||||
return "", fmt.Errorf("fake: unexpected exec-stdin args=%v stdin=%q", args, string(b))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// wrote reports whether the image was written via the stdin `tee` vector with the exact `image\n`
|
// wrote reports whether the image was written via the stdin `tee` vector with the exact `image\n`
|
||||||
@@ -162,7 +157,7 @@ func TestControllerSwap_Happy(t *testing.T) {
|
|||||||
|
|
||||||
// The write vector must be the stdin `tee` with byte-identical `image\n` and NO shell — the
|
// The write vector must be the stdin `tee` with byte-identical `image\n` and NO shell — the
|
||||||
// controllerswap.go writeImage rewrite. This would FAIL on the pre-change `bash -c "printf … >"` impl.
|
// controllerswap.go writeImage rewrite. This would FAIL on the pre-change `bash -c "printf … >"` impl.
|
||||||
func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) {
|
func TestControllerSwap_WriteViaRootVerb_NoShell(t *testing.T) {
|
||||||
fe := &fakeGuestExec{
|
fe := &fakeGuestExec{
|
||||||
imageFile: prevImg,
|
imageFile: prevImg,
|
||||||
present: map[string]bool{newImg: true},
|
present: map[string]bool{newImg: true},
|
||||||
@@ -173,22 +168,15 @@ func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) {
|
|||||||
t.Fatalf("state = %q, want done", st.State)
|
t.Fatalf("state = %q, want done", st.State)
|
||||||
}
|
}
|
||||||
if !fe.wrote(newImg) {
|
if !fe.wrote(newImg) {
|
||||||
t.Errorf("expected a tee write of %q+\\n; teeStdin=%q", newImg, fe.teeStdin)
|
t.Errorf("expected the root verb to write %q; writes=%q", newImg, fe.teeStdin)
|
||||||
}
|
}
|
||||||
sawTee := false
|
|
||||||
for _, c := range fe.calls {
|
for _, c := range fe.calls {
|
||||||
if len(c) >= 2 && c[0] == "tee" {
|
if len(c) >= 1 && c[0] == "tee" {
|
||||||
sawTee = true
|
t.Errorf("the swap still uses an in-guest tee (R-861 (a) A1 removed that grant): %v", c)
|
||||||
if c[1] != controllerImageFile {
|
|
||||||
t.Errorf("tee target = %q, want fixed %q", c[1], controllerImageFile)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !sawTee {
|
|
||||||
t.Error("no tee call recorded — writeImage did not use the stdin tee vector")
|
|
||||||
}
|
|
||||||
if fe.usedShell() {
|
if fe.usedShell() {
|
||||||
t.Errorf("swap used a shell vector (bash/-c/printf) — must be stdin tee only; calls=%v", fe.calls)
|
t.Errorf("swap used a shell vector (bash/-c/printf); calls=%v", fe.calls)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -300,9 +288,7 @@ func (s *inspectScript) GuestExec(_ context.Context, _ int, args ...string) (str
|
|||||||
}
|
}
|
||||||
return "", nil
|
return "", nil
|
||||||
}
|
}
|
||||||
func (s *inspectScript) GuestExecStdin(_ context.Context, _ int, _ io.Reader, _ ...string) (string, error) {
|
func (s *inspectScript) WriteControllerImage(context.Context, int, string) error { return nil }
|
||||||
return "", nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func fastSwapper(exec GuestExecutor) *ControllerSwapper {
|
func fastSwapper(exec GuestExecutor) *ControllerSwapper {
|
||||||
s := NewControllerSwapper(exec, "", discardLogger())
|
s := NewControllerSwapper(exec, "", discardLogger())
|
||||||
|
|||||||
@@ -127,6 +127,22 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
|
|||||||
"retained_has_restic_pw": match.HasResticPassword,
|
"retained_has_restic_pw": match.HasResticPassword,
|
||||||
},
|
},
|
||||||
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
|
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
|
||||||
|
// ── R-304 (2026-10-08) — NOT EVERY EARLIER PACKAGE WAS CHECKED. ─────────────────────────
|
||||||
|
//
|
||||||
|
// The current package refused the code, no retained package opened it — and at least one earlier
|
||||||
|
// package the hub holds was never tried (or the list could not be read). Saying „the code is
|
||||||
|
// wrong" here would claim a check that did not happen. 424 (Failed Dependency): the verdict
|
||||||
|
// depends on packages we could not try. The controller classifies on the status, never on this
|
||||||
|
// sentence; an older controller maps an unknown status to its neutral „we do not know why".
|
||||||
|
case errors.Is(err, escrow.ErrRetainedUnchecked):
|
||||||
|
n := -1
|
||||||
|
var ue *escrow.RetainedUncheckedError
|
||||||
|
if errors.As(err, &ue) {
|
||||||
|
n = ue.Unchecked
|
||||||
|
}
|
||||||
|
s.logger.Warn("local-api: offsite key recovery: the code did not open the current package and earlier packages were NOT all checked — not reported as a wrong code (R-304)", "vmid", vmid, "unchecked", n)
|
||||||
|
writeStatus(w, http.StatusFailedDependency, false, map[string]any{"older_unchecked": n},
|
||||||
|
"the recovery code did not open the current sealed package, and earlier packages the hub holds were not all checked — the code may belong to one of them; nothing was written")
|
||||||
case errors.Is(err, escrow.ErrNoResticPassword):
|
case errors.Is(err, escrow.ErrNoResticPassword):
|
||||||
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
|
||||||
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
|
||||||
|
|||||||
@@ -54,6 +54,13 @@ func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
|
|||||||
wantStatus: 404,
|
wantStatus: 404,
|
||||||
mustNotSay: []string{"did not open"},
|
mustNotSay: []string{"did not open"},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
// R-304: earlier packages were not all tried — never „did not open the sealed bundle" (the wrong-code words).
|
||||||
|
name: "earlier packages not all checked — not a wrong code",
|
||||||
|
err: &escrow.RetainedUncheckedError{Unchecked: 2},
|
||||||
|
wantStatus: 424,
|
||||||
|
mustNotSay: []string{"did not open the sealed bundle", "could not be fetched"},
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "the bundle predates the repository-password field",
|
name: "the bundle predates the repository-password field",
|
||||||
err: escrow.ErrNoResticPassword,
|
err: escrow.ErrNoResticPassword,
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package localapi
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -126,14 +125,16 @@ func (b *GuestBinder) GuestExec(ctx context.Context, vmid int, args ...string) (
|
|||||||
return string(out), nil
|
return string(out), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GuestExecStdin is GuestExec with the in-guest command's stdin fed from stdin. The controller-swap
|
// privApplyBin is the root content checker (R-861); its `controller-image` verb writes the guest's image file.
|
||||||
// write uses it to pipe the image ref into an in-guest `tee` (no shell vector, no interpolation),
|
const privApplyBin = "/usr/local/sbin/felhom-priv-apply"
|
||||||
// through the same fenced runner so the `sudo -n` prefix stays in one place.
|
|
||||||
func (b *GuestBinder) GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error) {
|
// WriteControllerImage pipes `image\n` to `felhom-priv-apply controller-image <vmid>` through the same fenced runner
|
||||||
pctArgs := append([]string{"exec", strconv.Itoa(vmid), "--"}, args...)
|
// (the `sudo -n` prefix stays in one place). The verb checks the ref as root and writes the guest file itself
|
||||||
out, stderr, err := b.runner.RunStdin(ctx, stdin, "pct", pctArgs...)
|
// (R-861 (a) A1, `09` §3 decision 165). Pinned by TestR861_WriteControllerImageUsesTheRootVerb.
|
||||||
|
func (b *GuestBinder) WriteControllerImage(ctx context.Context, vmid int, image string) error {
|
||||||
|
_, stderr, err := b.runner.RunStdin(ctx, strings.NewReader(image+"\n"), privApplyBin, "controller-image", strconv.Itoa(vmid))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return string(out), fmt.Errorf("pct exec %d %v: %w: %s", vmid, args, err, strings.TrimSpace(string(stderr)))
|
return fmt.Errorf("felhom-priv-apply controller-image %d: %w: %s", vmid, err, strings.TrimSpace(string(stderr)))
|
||||||
}
|
}
|
||||||
return string(out), nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// stdinRecorder is a proxmox.Runner that records each call and the stdin it was handed.
|
||||||
|
type stdinRecorder struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
stdin string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *stdinRecorder) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
r.name, r.args = name, args
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *stdinRecorder) RunStdin(_ context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
b, _ := io.ReadAll(stdin)
|
||||||
|
r.name, r.args, r.stdin = name, args, string(b)
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-861 (a) A1 (`09` §3 decision 165): the managed controller update writes the guest's image file through the ROOT
|
||||||
|
// verb, never through an in-guest `tee` the agent could feed any image.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): restore the pre-A1 body (`b.runner.RunStdin(ctx, …, "pct", "exec", vmid, "--",
|
||||||
|
// "tee", controllerImageFile)`) → this fails with "the image write ran pct …, want felhom-priv-apply". Restored.
|
||||||
|
func TestR861_WriteControllerImageUsesTheRootVerb(t *testing.T) {
|
||||||
|
rec := &stdinRecorder{}
|
||||||
|
b := NewGuestBinder(rec, discardLogger())
|
||||||
|
const img = "gitea.dooplex.hu/admin/felhom-controller:0.302.0"
|
||||||
|
if err := b.WriteControllerImage(context.Background(), 9201, img); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rec.name != privApplyBin {
|
||||||
|
t.Fatalf("the image write ran %s %v, want felhom-priv-apply", rec.name, rec.args)
|
||||||
|
}
|
||||||
|
if len(rec.args) != 2 || rec.args[0] != "controller-image" || rec.args[1] != "9201" {
|
||||||
|
t.Fatalf("argv = %v, want [controller-image 9201] (the sudoers line `^controller-image [0-9]+$`)", rec.args)
|
||||||
|
}
|
||||||
|
if rec.stdin != img+"\n" {
|
||||||
|
t.Fatalf("stdin = %q, want the ref plus one newline", rec.stdin)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -825,6 +825,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
key := backupJobKey{vmid: vmid, target: tier.TargetID}
|
||||||
|
// R-899 (operator ruling 2026-10-08): a household press („Mentés most") is not the night's backup. A controller
|
||||||
|
// that knows sends `trigger=manual`; then the OS leg does not follow (it belongs to the night, after the night's
|
||||||
|
// own copy). An older controller sends nothing and keeps the old behaviour.
|
||||||
|
manual := r.URL.Query().Get("trigger") == "manual"
|
||||||
|
|
||||||
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
|
||||||
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
|
||||||
@@ -926,7 +930,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
|||||||
}
|
}
|
||||||
s.finishJob(key, jobID, b)
|
s.finishJob(key, jobID, b)
|
||||||
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
|
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
|
||||||
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
if b.Success && tier.Primary && s.afterPrimaryBackup != nil && manual {
|
||||||
|
s.logger.Info("local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)", "vmid", vmid, "job", jobID)
|
||||||
|
} else if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
|
||||||
s.afterPrimaryBackup(base, vmid)
|
s.afterPrimaryBackup(base, vmid)
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|||||||
@@ -0,0 +1,446 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
// The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). The root half is felhom-os-apply's layer "kernel"
|
||||||
|
// (configs/, its own tests); this file decides WHEN and judges the boot:
|
||||||
|
//
|
||||||
|
// - the night leg (Run → runKernel): after a healthy host step, on a night the hub marks as told (the household was
|
||||||
|
// mailed the day before — no mail, no step): ring 0 STAGES the pending kernel (select pending-kernel, the root-owned
|
||||||
|
// ring-0 mark) and reboots; ring 1 reboots only a step a signed os_kernel_step staged earlier (KernelStepExecutor).
|
||||||
|
// - after a boot (KernelAfterBoot, at daemon start): the wrapper says what became of the step. On the new kernel the
|
||||||
|
// agent JUDGES the boot — the host health rule (`11` §8.2: the Proxmox daemons, the guest running and healthy, the
|
||||||
|
// tunnel) AND the box reaching the hub — for KernelJudgeWait. Healthy → kernel-good (the new kernel becomes the
|
||||||
|
// default). Not healthy by the deadline → ONE self-revert (kernel-revert: a reboot into the old kernel, still the
|
||||||
|
// default). A crash on the new kernel needs nothing from the agent: GRUB already boots the old default.
|
||||||
|
//
|
||||||
|
// The host is rebooted by this file only through the wrapper (kernel-reboot, kernel-revert), and only for a staged step.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"regexp"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// OpKernelStep is the signed op class that STAGES a kernel set on a ring-1 box (it never reboots: the night leg does,
|
||||||
|
// once the household was told). CC may sign it until the first paying customer (R-530 ruling).
|
||||||
|
const OpKernelStep = "os_kernel_step"
|
||||||
|
|
||||||
|
// DefaultKernelJudgeWait is how long a one-shot boot may take to come back healthy before the agent reverts it ONCE.
|
||||||
|
// Measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after a reboot on demo-felhom,
|
||||||
|
// 272 s on demo-hp; 20 minutes stays under the hub's 45-minute host_stale (a box that never comes back alarms after it).
|
||||||
|
const DefaultKernelJudgeWait = 20 * time.Minute
|
||||||
|
|
||||||
|
var kverRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$`)
|
||||||
|
|
||||||
|
// KernelView is the wrapper's kernel object (felhom-os-apply Kernel.view).
|
||||||
|
type KernelView struct {
|
||||||
|
Running string `json:"running"`
|
||||||
|
Default string `json:"default"`
|
||||||
|
Flag *string `json:"flag"`
|
||||||
|
Phase string `json:"phase"`
|
||||||
|
From string `json:"from"`
|
||||||
|
To string `json:"to"`
|
||||||
|
SelfRevertUsed bool `json:"self_revert_used"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
|
||||||
|
}
|
||||||
|
|
||||||
|
// kernelReportRing is the ring an after-boot report carries. In the first second after a boot the agent has not
|
||||||
|
// fetched the hub's block yet, and Block() then answers ring 1 — so a ring-0 box's „judging" report said ring 1
|
||||||
|
// (seen on demo-felhom, 2026-10-08 night, `audits/kernel-night-2026-10-07/readback/`). Order: the fetched block; the
|
||||||
|
// block the daemon saved on disk before the reboot (R-866); ring 1 as before. A label only: the hub's approval reads its
|
||||||
|
// own ring list. Pinned by TestKernelReportRing_BeforeFirstFetch.
|
||||||
|
func (l *Leg) kernelReportRing() int {
|
||||||
|
l.mu.Lock()
|
||||||
|
fetched := l.block
|
||||||
|
l.mu.Unlock()
|
||||||
|
if fetched != nil {
|
||||||
|
return fetched.Ring
|
||||||
|
}
|
||||||
|
if b, _, ok := LoadSavedBlock(l.planDir()); ok && b != nil {
|
||||||
|
return b.Ring
|
||||||
|
}
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseKernel(raw json.RawMessage) KernelView {
|
||||||
|
var v KernelView
|
||||||
|
_ = json.Unmarshal(raw, &v)
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
|
||||||
|
// kernelPlan is one kernel-layer wrapper call.
|
||||||
|
func kernelPlan(mode string, vmid int, extra map[string]any) map[string]any {
|
||||||
|
p := map[string]any{"release_id": "kernel", "layer": LayerKernel, "lane": "slow", "vmid": vmid, "mode": mode,
|
||||||
|
"packages": []Package{}}
|
||||||
|
for k, v := range extra {
|
||||||
|
p[k] = v
|
||||||
|
}
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// KernelStatus reads the kernel lane's state (read only).
|
||||||
|
func (l *Leg) KernelStatus(ctx context.Context, vmid int) (KernelView, error) {
|
||||||
|
wr, err := l.call(ctx, "kstatus"+l.now().UTC().Format("150405"), kernelPlan("kernel-status", vmid, nil))
|
||||||
|
if err != nil {
|
||||||
|
return KernelView{}, err
|
||||||
|
}
|
||||||
|
if wr.refused() {
|
||||||
|
return KernelView{}, fmt.Errorf("kernel-status refused: %s", wr.Refused)
|
||||||
|
}
|
||||||
|
return parseKernel(wr.Kernel), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// kernelDue reports whether tonight's leg may take a kernel step, and why not.
|
||||||
|
func kernelDue(blk hub.WireOSUpdate, trigger string) (bool, string) {
|
||||||
|
switch {
|
||||||
|
case trigger != "night":
|
||||||
|
return false, "a kernel step runs only in the night leg (never a debug pass)"
|
||||||
|
case blk.Kernel == nil:
|
||||||
|
return false, "the hub names no kernel step for this box"
|
||||||
|
case !blk.Enabled:
|
||||||
|
return false, "OS updates are switched off for this box"
|
||||||
|
case !kverRE.MatchString(blk.Kernel.Kver):
|
||||||
|
return false, "the hub's kernel " + blk.Kernel.Kver + " is not a kernel version"
|
||||||
|
case !blk.Kernel.Tonight:
|
||||||
|
return false, "the household has not been told about tonight (no mail, no step — `09` §3 decision 172)"
|
||||||
|
}
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// runKernel is the night leg's last step. It returns the stage report (Layer "" when nothing ran). On success the box
|
||||||
|
// is rebooting when it returns.
|
||||||
|
func (l *Leg) runKernel(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
|
||||||
|
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", trigger, "ring", blk.Ring)
|
||||||
|
if ok, why := kernelDue(blk, trigger); !ok {
|
||||||
|
lg.Info("osupdate: kernel step skipped — " + why)
|
||||||
|
return Report{}
|
||||||
|
}
|
||||||
|
want := blk.Kernel.Kver
|
||||||
|
st, err := l.KernelStatus(ctx, vmid)
|
||||||
|
if err != nil {
|
||||||
|
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||||
|
Mode: "apply", Outcome: "failed", HealthReason: "kernel status unreadable: " + err.Error()})
|
||||||
|
}
|
||||||
|
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply", ReleaseID: want}
|
||||||
|
switch {
|
||||||
|
case st.Phase == "staged" && st.To == want:
|
||||||
|
lg.Info("osupdate: kernel step — a staged kernel waits for tonight", "from", st.From, "to", st.To)
|
||||||
|
rep.Outcome, rep.Healthy = "staged", true
|
||||||
|
case blk.Ring != 0:
|
||||||
|
lg.Info("osupdate: kernel step skipped — ring 1 boots only a kernel a signed os_kernel_step staged", "phase", st.Phase, "staged", st.To, "want", want)
|
||||||
|
return Report{}
|
||||||
|
default:
|
||||||
|
// R-898: EXACTLY the kernel the household was told about — never "whatever is pending tonight" (the sources can
|
||||||
|
// offer a newer one by night; the step then refused, R23, and the night was lost). A version no longer
|
||||||
|
// installable is refused by the wrapper before any change (R7) and the hub tells the household again.
|
||||||
|
wr, cerr := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": "ring0-" + runID,
|
||||||
|
"select": "listed", "packages": KernelSet(want), "expect_kver": want, "run_id": runID, "trigger": trigger,
|
||||||
|
"ring": blk.Ring}))
|
||||||
|
rep.unsent = reportFile(l.planDir(), runID, LayerKernel, "apply")
|
||||||
|
rep.Kernel = rawOrNil(wr.Kernel)
|
||||||
|
switch {
|
||||||
|
case cerr != nil:
|
||||||
|
rep.Outcome, rep.HealthReason = "failed", cerr.Error()
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case wr.refused():
|
||||||
|
rep.Outcome, rep.Refused = "refused", wr.Refused
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case wr.failed():
|
||||||
|
rep.Outcome, rep.Refused = "failed", wr.Failed
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
|
||||||
|
rep.Outcome, rep.Healthy = "nothing", true
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
}
|
||||||
|
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
|
||||||
|
rep.RebootNeeded = true
|
||||||
|
}
|
||||||
|
rep = l.finish(ctx, lg, rep) // the hub hears "staged" BEFORE the box goes down
|
||||||
|
wr, err := l.call(ctx, runID, kernelPlan("kernel-reboot", vmid, nil))
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||||
|
Mode: "kernel-reboot", ReleaseID: want, Outcome: "failed", HealthReason: "kernel-reboot: " + err.Error()})
|
||||||
|
case wr.refused() || wr.failed():
|
||||||
|
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||||
|
Mode: "kernel-reboot", ReleaseID: want, Outcome: "refused", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||||
|
Kernel: rawOrNil(wr.Kernel)})
|
||||||
|
}
|
||||||
|
lg.Warn("osupdate: kernel step — the box restarts now for its one-shot boot", "to", want)
|
||||||
|
return rep
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstRaw(a, b json.RawMessage) json.RawMessage {
|
||||||
|
if r := rawOrNil(a); r != nil {
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
return rawOrNil(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// KernelJudge is what KernelAfterBoot needs besides the leg: the hub reachability probe is the "judging" report itself.
|
||||||
|
type KernelJudge struct {
|
||||||
|
Wait time.Duration // default DefaultKernelJudgeWait
|
||||||
|
Poll time.Duration // default 30 s
|
||||||
|
}
|
||||||
|
|
||||||
|
// KernelAfterBoot runs once at daemon start: what became of a kernel step across the boot. On the new kernel it judges
|
||||||
|
// the boot (blocking up to the wait — run it in a goroutine). vmid 0 = the guest the step recorded (it may not run yet).
|
||||||
|
func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Report {
|
||||||
|
runID := "boot-" + l.now().UTC().Format("20060102T150405Z")
|
||||||
|
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid)
|
||||||
|
wr, err := l.call(ctx, runID, kernelPlan("kernel-boot", vmid, nil))
|
||||||
|
if err != nil {
|
||||||
|
lg.Warn("osupdate: kernel after-boot check failed", "err", err)
|
||||||
|
return Report{}
|
||||||
|
}
|
||||||
|
if wr.refused() {
|
||||||
|
lg.Info("osupdate: kernel after-boot check refused (an older wrapper, or a BYO host)", "refused", string(wr.Refused))
|
||||||
|
return Report{}
|
||||||
|
}
|
||||||
|
v := parseKernel(wr.Kernel)
|
||||||
|
if vmid <= 0 {
|
||||||
|
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
|
||||||
|
}
|
||||||
|
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-boot",
|
||||||
|
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
|
||||||
|
switch wr.KernelEvent {
|
||||||
|
case "fell_back":
|
||||||
|
rep.Outcome, rep.HealthReason = "fell_back", v.Reason
|
||||||
|
lg.Warn("osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one", "from", v.From, "to", v.To)
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case "self_reverted":
|
||||||
|
rep.Outcome, rep.HealthReason = "self_reverted", v.Reason
|
||||||
|
lg.Warn("osupdate: kernel step SELF-REVERTED — back on the old kernel", "from", v.From, "to", v.To, "reason", v.Reason)
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case "revert_failed":
|
||||||
|
rep.Outcome, rep.HealthReason = "revert_failed", v.Reason
|
||||||
|
lg.Error("osupdate: kernel self-revert came back on the NEW kernel — no second revert; the operator decides", "to", v.To)
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case "judging":
|
||||||
|
return l.judgeKernel(ctx, runID, vmid, v, wr.HealthBefore, j, lg)
|
||||||
|
}
|
||||||
|
return Report{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// KernelVerdict is THE one-shot boot rule (R-836; pinned by TestKernelVerdict): the host health rule (`11` §8.2 —
|
||||||
|
// the Proxmox daemons and the agent active, the customer guest running and its own rule passing, the tunnel running)
|
||||||
|
// AND the box reached the hub since this boot.
|
||||||
|
func KernelVerdict(before, after *Health, tunnel string, hubReached bool) (bool, string) {
|
||||||
|
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
|
||||||
|
return false, why
|
||||||
|
}
|
||||||
|
if !hubReached {
|
||||||
|
return false, "the box has not reached the hub since the boot"
|
||||||
|
}
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelView, before *Health, j KernelJudge, lg *slog.Logger) Report {
|
||||||
|
wait, poll := j.Wait, j.Poll
|
||||||
|
if wait <= 0 {
|
||||||
|
wait = DefaultKernelJudgeWait
|
||||||
|
}
|
||||||
|
if poll <= 0 {
|
||||||
|
poll = 30 * time.Second
|
||||||
|
}
|
||||||
|
lg.Info("osupdate: kernel step — judging the one-shot boot", "from", v.From, "to", v.To, "wait", wait.String())
|
||||||
|
start := l.now()
|
||||||
|
deadline := start.Add(wait)
|
||||||
|
hubReached := false
|
||||||
|
var why string
|
||||||
|
for {
|
||||||
|
if !hubReached && l.Hub != nil {
|
||||||
|
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
|
||||||
|
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||||
|
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
|
||||||
|
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||||
|
if err := l.Hub.PostOSReport(rctx, body); err == nil {
|
||||||
|
hubReached = true
|
||||||
|
lg.Info("osupdate: kernel step — the box reached the hub on the new kernel", "after", l.now().Sub(start).Round(time.Second).String())
|
||||||
|
}
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
var h *Health
|
||||||
|
hr, err := l.call(ctx, runID, kernelPlan("health", vmid, nil))
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
why = "no health reading: " + err.Error()
|
||||||
|
case hr.refused():
|
||||||
|
why = "no health reading: " + string(hr.Refused)
|
||||||
|
default:
|
||||||
|
h = hr.Health
|
||||||
|
}
|
||||||
|
if h != nil {
|
||||||
|
t := hub.TunnelUnknown
|
||||||
|
if l.Tunnel != nil {
|
||||||
|
t, _ = l.Tunnel.Status(ctx)
|
||||||
|
}
|
||||||
|
var ok bool
|
||||||
|
ok, why = KernelVerdict(before, h, t, hubReached)
|
||||||
|
if ok {
|
||||||
|
return l.kernelGood(ctx, runID, vmid, v, start, lg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !l.now().Before(deadline) || ctx.Err() != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
l.sleep(ctx, poll)
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
lg.Warn("osupdate: kernel judging stopped (the agent is stopping) — the next start judges again", "reason", why)
|
||||||
|
return Report{}
|
||||||
|
}
|
||||||
|
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
|
||||||
|
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||||
|
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
|
||||||
|
Kernel: mustRaw(v)})
|
||||||
|
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
|
||||||
|
"waited", wait.String(), "from", v.From, "to", v.To)
|
||||||
|
wr, err := l.call(ctx, runID, kernelPlan("kernel-revert", vmid, map[string]any{"reason": truncate(why, 280)}))
|
||||||
|
if err != nil || wr.refused() || wr.failed() {
|
||||||
|
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
|
||||||
|
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
|
||||||
|
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
|
||||||
|
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||||
|
HealthReason: "the self-revert did not start"})
|
||||||
|
}
|
||||||
|
return rep
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
|
||||||
|
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
|
||||||
|
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-good",
|
||||||
|
ReleaseID: v.To}
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
rep.Outcome, rep.HealthReason = "failed", "kernel-good: "+err.Error()
|
||||||
|
case wr.refused() || wr.failed():
|
||||||
|
rep.Outcome, rep.Refused, rep.HealthReason = "failed", firstRaw(wr.Refused, wr.Failed), "kernel-good did not move the default"
|
||||||
|
default:
|
||||||
|
rep.Outcome, rep.Healthy = "applied", true
|
||||||
|
rep.HealthReason = fmt.Sprintf("healthy %s after the agent started; the new kernel is the default", l.now().Sub(start).Round(time.Second))
|
||||||
|
}
|
||||||
|
rep.Kernel = rawOrNil(wr.Kernel)
|
||||||
|
lg.Info("osupdate: kernel step — "+rep.Outcome, "to", v.To, "reason", rep.HealthReason)
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustRaw(v any) json.RawMessage {
|
||||||
|
b, _ := json.Marshal(v)
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
func truncate(s string, n int) string {
|
||||||
|
if len(s) <= n {
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
return s[:n]
|
||||||
|
}
|
||||||
|
|
||||||
|
// KernelSet is the package set that installs exactly kver (R-898; the hub's kernelSet, field-exact): the series
|
||||||
|
// meta-package and the signed image, both at the kernel's own version. Proxmox keeps old kernel versions in its archive.
|
||||||
|
// nil for a string that is not a kernel version.
|
||||||
|
func KernelSet(kver string) []Package {
|
||||||
|
m := kverSeriesRE.FindStringSubmatch(kver)
|
||||||
|
if m == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
v := kver[:len(kver)-len("-pve")]
|
||||||
|
return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: PVEOrigin},
|
||||||
|
{Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: PVEOrigin}}
|
||||||
|
}
|
||||||
|
|
||||||
|
var kverSeriesRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`)
|
||||||
|
|
||||||
|
// KernelStepParams are a signed os_kernel_step's params: the exact kernel set (the wrapper compares it with the plan).
|
||||||
|
type KernelStepParams struct {
|
||||||
|
ReleaseID string `json:"release_id"`
|
||||||
|
Packages []Package `json:"packages"`
|
||||||
|
Kver string `json:"kver"`
|
||||||
|
VMID int `json:"vmid,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// KernelStepExecutor STAGES a verified os_kernel_step (signedjobs.Executor) under the heavy-op gate. It never reboots:
|
||||||
|
// the night leg reboots a staged kernel on a night the household was told about.
|
||||||
|
type KernelStepExecutor struct {
|
||||||
|
Leg *Leg
|
||||||
|
Guest func(ctx context.Context) (int, error)
|
||||||
|
Gate func(ctx context.Context) (release func(), err error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Execute implements signedjobs.Executor.
|
||||||
|
func (e KernelStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||||
|
if op != OpKernelStep {
|
||||||
|
return signedjobs.ErrNoExecutor
|
||||||
|
}
|
||||||
|
so, ok := signedjobs.SignedOpFrom(ctx)
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("os_kernel_step: no signed envelope in the context — the wrapper could not verify it")
|
||||||
|
}
|
||||||
|
var p KernelStepParams
|
||||||
|
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 || !kverRE.MatchString(p.Kver) {
|
||||||
|
return fmt.Errorf("os_kernel_step: params must name the kernel set and its kver: %v", err)
|
||||||
|
}
|
||||||
|
vmid := p.VMID
|
||||||
|
if vmid == 0 {
|
||||||
|
if e.Guest == nil {
|
||||||
|
return fmt.Errorf("os_kernel_step: no vmid and no guest finder")
|
||||||
|
}
|
||||||
|
v, err := e.Guest(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("os_kernel_step: find the customer guest: %w", err)
|
||||||
|
}
|
||||||
|
vmid = v
|
||||||
|
}
|
||||||
|
if e.Gate != nil {
|
||||||
|
release, err := e.Gate(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("os_kernel_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
}
|
||||||
|
rep := e.Leg.StageKernelSigned(ctx, vmid, p, so.Blob, string(so.Sig))
|
||||||
|
if rep.Outcome == "staged" || rep.Outcome == "nothing" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return fmt.Errorf("os_kernel_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
|
||||||
|
}
|
||||||
|
|
||||||
|
// StageKernelSigned stages a signed kernel set (ring 1): install + flag, no reboot.
|
||||||
|
func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParams, blob []byte, sig string) Report {
|
||||||
|
unlock := l.lockPass(true)
|
||||||
|
defer unlock()
|
||||||
|
l.sendUnsentLocked(ctx) // R-868
|
||||||
|
runID := l.now().UTC().Format("20060102T150405Z")
|
||||||
|
rid := p.ReleaseID
|
||||||
|
if rid == "" {
|
||||||
|
rid = "signed-" + runID
|
||||||
|
}
|
||||||
|
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", "signed", "release", rid)
|
||||||
|
wr, err := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": rid, "select": "listed",
|
||||||
|
"packages": p.Packages, "expect_kver": p.Kver, "run_id": runID, "trigger": "signed", "ring": l.Block().Ring,
|
||||||
|
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}}))
|
||||||
|
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "signed", Ring: l.Block().Ring, VMID: vmid, Mode: "apply",
|
||||||
|
ReleaseID: rid, Kernel: rawOrNil(wr.Kernel), unsent: reportFile(l.planDir(), runID, LayerKernel, "apply")}
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
rep.Outcome, rep.HealthReason = "failed", err.Error()
|
||||||
|
case wr.refused():
|
||||||
|
rep.Outcome, rep.Refused = "refused", wr.Refused
|
||||||
|
case wr.failed():
|
||||||
|
rep.Outcome, rep.Refused = "failed", wr.Failed
|
||||||
|
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
|
||||||
|
rep.Outcome, rep.Healthy = "nothing", true
|
||||||
|
default:
|
||||||
|
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
|
||||||
|
rep.RebootNeeded = true
|
||||||
|
}
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
}
|
||||||
@@ -0,0 +1,328 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ---- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----
|
||||||
|
|
||||||
|
const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve"
|
||||||
|
|
||||||
|
func kview(phase string) json.RawMessage {
|
||||||
|
return mustRaw(KernelView{Running: kOld, Default: kOld, Phase: phase, From: kOld, To: kNew, VMID: 9201})
|
||||||
|
}
|
||||||
|
|
||||||
|
func tonight(ring int) *hub.WireOSUpdate {
|
||||||
|
return &hub.WireOSUpdate{Ring: ring, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func kernelCalls(w *fakeWrapper) []string {
|
||||||
|
var m []string
|
||||||
|
for _, p := range w.plans {
|
||||||
|
if p["layer"] == LayerKernel {
|
||||||
|
m = append(m, p["mode"].(string))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
|
||||||
|
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
|
||||||
|
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{
|
||||||
|
"kernel-status": {{Kernel: kview("none")}},
|
||||||
|
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
|
||||||
|
"kernel-reboot": {{Kernel: kview("oneshot")}},
|
||||||
|
}}
|
||||||
|
l, h := newLeg(t, w, tonight(0))
|
||||||
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
|
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply,kernel-reboot" {
|
||||||
|
t.Fatalf("kernel calls = %s", got)
|
||||||
|
}
|
||||||
|
var ap map[string]any
|
||||||
|
for _, x := range w.plans {
|
||||||
|
if x["layer"] == LayerKernel && x["mode"] == "apply" {
|
||||||
|
ap = x
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// R-898: EXACTLY the told kernel — the listed set derived from it, never "pending" (red before the fix: select was
|
||||||
|
// pending-kernel, so a newer kernel in the sources by night was refused R23 and the night was lost)
|
||||||
|
pk, _ := json.Marshal(ap["packages"])
|
||||||
|
if ap["select"] != "listed" || ap["expect_kver"] != kNew || ap["lane"] != "slow" ||
|
||||||
|
string(pk) != `[{"name":"proxmox-kernel-7.0","origin":"Proxmox Debian Repository","version":"7.0.14-22"},{"name":"proxmox-kernel-7.0.14-22-pve-signed","origin":"Proxmox Debian Repository","version":"7.0.14-22"}]` {
|
||||||
|
t.Fatalf("stage plan = %v (packages %s)", ap, pk)
|
||||||
|
}
|
||||||
|
if p.Kernel.Outcome != "staged" || !p.Kernel.Healthy {
|
||||||
|
t.Fatalf("kernel report = %+v", p.Kernel)
|
||||||
|
}
|
||||||
|
last := h.reports[len(h.reports)-1]
|
||||||
|
if last.Layer != LayerKernel || last.Outcome != "staged" {
|
||||||
|
t.Fatalf("the hub must hear 'staged' before the reboot: %+v", h.reports)
|
||||||
|
}
|
||||||
|
// the kernel step is the LAST wrapper call of the night
|
||||||
|
if lp := w.plans[len(w.plans)-1]; lp["layer"] != LayerKernel || lp["mode"] != "kernel-reboot" {
|
||||||
|
t.Fatalf("the reboot must end the night, last call = %v", lp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// No mail, no step: a kernel the household was NOT told about never runs; nor in a debug pass; nor without a block.
|
||||||
|
// COMPANION RED-PROOF (observed): drop the `!blk.Kernel.Tonight` case in kernelDue → the first sub-case fails.
|
||||||
|
func TestKernel_NoMailNoStep(t *testing.T) {
|
||||||
|
cases := map[string]struct {
|
||||||
|
blk *hub.WireOSUpdate
|
||||||
|
trigger string
|
||||||
|
}{
|
||||||
|
"not told": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: false}}, "night"},
|
||||||
|
"debug pass": {tonight(0), "debug"},
|
||||||
|
"no block": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night"},
|
||||||
|
"switch off": {&hub.WireOSUpdate{Ring: 0, Enabled: false, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}, "night"},
|
||||||
|
"bad kver": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: "7.0; reboot", Tonight: true}}, "night"},
|
||||||
|
}
|
||||||
|
for name, c := range cases {
|
||||||
|
w := &fakeWrapper{t: t}
|
||||||
|
l, _ := newLeg(t, w, c.blk)
|
||||||
|
p := l.Run(context.Background(), 9201, c.trigger)
|
||||||
|
if len(kernelCalls(w)) != 0 || p.Kernel.Layer != "" {
|
||||||
|
t.Fatalf("%s: a kernel step ran: %v", name, kernelCalls(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The kernel step needs a healthy host step on an appliance, and a healthy Proxmox step when one ran.
|
||||||
|
func TestKernel_SkippedWithoutHealthyEarlierSteps(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t}
|
||||||
|
l, _ := newLeg(t, w, tonight(0))
|
||||||
|
l.Appliance = false
|
||||||
|
l.Run(context.Background(), 9201, "night")
|
||||||
|
if len(kernelCalls(w)) != 0 {
|
||||||
|
t.Fatalf("a BYO box took a kernel step: %v", kernelCalls(w))
|
||||||
|
}
|
||||||
|
w2 := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
|
||||||
|
PVEManager: "9.2.2"}}} // pveversion still old → the pve step is unhealthy
|
||||||
|
l2, _ := newLeg(t, w2, tonight(0))
|
||||||
|
l2.Run(context.Background(), 9201, "night")
|
||||||
|
if len(kernelCalls(w2)) != 0 {
|
||||||
|
t.Fatalf("a kernel step ran after an unhealthy Proxmox step: %v", kernelCalls(w2))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ring 1 reboots only a kernel a signed job staged — never stages one itself in the night leg.
|
||||||
|
func TestKernel_Ring1RebootsOnlyASignedStage(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}}
|
||||||
|
l, _ := newLeg(t, w, tonight(1))
|
||||||
|
l.Run(context.Background(), 9201, "night")
|
||||||
|
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status" {
|
||||||
|
t.Fatalf("ring 1 without a staged kernel: calls = %s", got)
|
||||||
|
}
|
||||||
|
w2 := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("staged")}},
|
||||||
|
"kernel-reboot": {{Kernel: kview("oneshot")}}}}
|
||||||
|
l2, _ := newLeg(t, w2, tonight(1))
|
||||||
|
p := l2.Run(context.Background(), 9201, "night")
|
||||||
|
if got := strings.Join(kernelCalls(w2), ","); got != "kernel-status,kernel-reboot" || p.Kernel.Outcome != "staged" {
|
||||||
|
t.Fatalf("ring 1 with a staged kernel: calls = %s report = %+v", got, p.Kernel)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A refused stage never reboots.
|
||||||
|
func TestKernel_RefusedStageNeverReboots(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
|
||||||
|
"apply": {{Refused: json.RawMessage(`{"code":"R20","reason":"/boot/efi is not a mounted vfat ESP"}`)}}}}
|
||||||
|
l, h := newLeg(t, w, tonight(0))
|
||||||
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
|
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply" || p.Kernel.Outcome != "refused" {
|
||||||
|
t.Fatalf("calls = %s report = %+v", got, p.Kernel)
|
||||||
|
}
|
||||||
|
if last := h.reports[len(h.reports)-1]; last.Layer != LayerKernel || last.Outcome != "refused" {
|
||||||
|
t.Fatalf("the hub must hear the refusal: %+v", last)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE one-shot boot rule: the host rule AND the hub reached. COMPANION RED-PROOF (observed): drop the hubReached
|
||||||
|
// check in KernelVerdict → the second case fails.
|
||||||
|
func TestKernelVerdict(t *testing.T) {
|
||||||
|
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, true); !ok {
|
||||||
|
t.Fatalf("a healthy boot read unhealthy: %s", why)
|
||||||
|
}
|
||||||
|
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, false); ok || !strings.Contains(why, "hub") {
|
||||||
|
t.Fatalf("a box that has not reached the hub must not pass: ok=%v %q", ok, why)
|
||||||
|
}
|
||||||
|
down := hostOK()
|
||||||
|
down.GuestRunning = new(bool)
|
||||||
|
if ok, _ := KernelVerdict(hostOK(), down, hub.TunnelRunning, true); ok {
|
||||||
|
t.Fatal("a guest that does not run must fail")
|
||||||
|
}
|
||||||
|
if ok, _ := KernelVerdict(hostOK(), hostOK(), hub.TunnelUnknown, true); ok {
|
||||||
|
t.Fatal("an unknown tunnel must fail (the host rule)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func judgingLeg(t *testing.T, w *fakeWrapper) (*Leg, *fakeHub) {
|
||||||
|
if w.kernelRep == nil {
|
||||||
|
w.kernelRep = map[string][]WrapperReport{}
|
||||||
|
}
|
||||||
|
if _, ok := w.kernelRep["kernel-boot"]; !ok {
|
||||||
|
w.kernelRep["kernel-boot"] = []WrapperReport{{KernelEvent: "judging", Kernel: mustRaw(KernelView{Running: kNew,
|
||||||
|
Default: kOld, Phase: "judging", From: kOld, To: kNew, VMID: 9201}), HealthBefore: hostOK()}}
|
||||||
|
}
|
||||||
|
return newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
}
|
||||||
|
|
||||||
|
// A healthy one-shot boot: the hub hears "judging", then kernel-good, then "applied".
|
||||||
|
func TestKernelAfterBoot_HealthyBecomesTheDefault(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-good": {{Kernel: kview("good")}}}}
|
||||||
|
l, h := judgingLeg(t, w)
|
||||||
|
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: 30 * time.Second})
|
||||||
|
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot,health,kernel-good" {
|
||||||
|
t.Fatalf("calls = %s", got)
|
||||||
|
}
|
||||||
|
if r.Outcome != "applied" || !r.Healthy {
|
||||||
|
t.Fatalf("report = %+v", r)
|
||||||
|
}
|
||||||
|
if len(h.reports) != 2 || h.reports[0].Outcome != "judging" || h.reports[1].Outcome != "applied" {
|
||||||
|
t.Fatalf("hub reports = %+v", h.reports)
|
||||||
|
}
|
||||||
|
if w.plans[1]["vmid"] != float64(9201) {
|
||||||
|
t.Fatalf("the health reading must use the step's own guest, got %v", w.plans[1]["vmid"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An unhealthy one-shot boot: wait the full judge time, tell the hub, then ONE kernel-revert.
|
||||||
|
// COMPANION RED-PROOF (observed): return before the kernel-revert call in judgeKernel → "calls" fails.
|
||||||
|
func TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait(t *testing.T) {
|
||||||
|
down := hostOK()
|
||||||
|
down.GuestRunning = new(bool)
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"health": {{Health: down}},
|
||||||
|
"kernel-revert": {{Kernel: kview("reverting")}}}}
|
||||||
|
l, h := judgingLeg(t, w)
|
||||||
|
start := l.now()
|
||||||
|
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: time.Minute})
|
||||||
|
calls := kernelCalls(w)
|
||||||
|
if calls[len(calls)-1] != "kernel-revert" || strings.Count(strings.Join(calls, ","), "kernel-revert") != 1 {
|
||||||
|
t.Fatalf("calls = %v", calls)
|
||||||
|
}
|
||||||
|
if waited := l.now().Sub(start); waited < 10*time.Minute {
|
||||||
|
t.Fatalf("reverted after %s — before the judge wait", waited)
|
||||||
|
}
|
||||||
|
if r.Outcome != "health_failed" || !strings.Contains(r.HealthReason, "not running") {
|
||||||
|
t.Fatalf("report = %+v", r)
|
||||||
|
}
|
||||||
|
if last := h.reports[len(h.reports)-1]; last.Outcome != "health_failed" {
|
||||||
|
t.Fatalf("the hub must hear health_failed before the revert reboot: %+v", h.reports)
|
||||||
|
}
|
||||||
|
for _, p := range w.plans {
|
||||||
|
if p["mode"] == "kernel-revert" && !strings.Contains(p["reason"].(string), "not running") {
|
||||||
|
t.Fatalf("the revert must carry the reason: %v", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A box that never reaches the hub is not "healthy" — it reverts too.
|
||||||
|
func TestKernelAfterBoot_NoHubMeansRevert(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-revert": {{Kernel: kview("reverting")}}}}
|
||||||
|
l, _ := judgingLeg(t, w)
|
||||||
|
l.Hub = unreachableHub{}
|
||||||
|
l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 5 * time.Minute, Poll: time.Minute})
|
||||||
|
if c := kernelCalls(w); c[len(c)-1] != "kernel-revert" {
|
||||||
|
t.Fatalf("calls = %v", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type unreachableHub struct{}
|
||||||
|
|
||||||
|
func (unreachableHub) PostOSReport(context.Context, []byte) error { return context.DeadlineExceeded }
|
||||||
|
|
||||||
|
// What kernel-boot found becomes the hub's outcome, with no judging and no reboot.
|
||||||
|
func TestKernelAfterBoot_FallBackAndRevertResultsAreReported(t *testing.T) {
|
||||||
|
for ev, want := range map[string]string{"fell_back": "fell_back", "self_reverted": "self_reverted", "revert_failed": "revert_failed"} {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: ev,
|
||||||
|
Kernel: mustRaw(KernelView{Running: kOld, Default: kOld, Phase: ev, From: kOld, To: kNew, Reason: "r", VMID: 9201})}}}}
|
||||||
|
l, h := judgingLeg(t, w)
|
||||||
|
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{})
|
||||||
|
if r.Outcome != want || len(h.reports) != 1 || h.reports[0].Outcome != want {
|
||||||
|
t.Fatalf("%s: report %+v hub %+v", ev, r, h.reports)
|
||||||
|
}
|
||||||
|
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot" {
|
||||||
|
t.Fatalf("%s: calls = %s", ev, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// nothing to do → nothing reported
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: "none", Kernel: kview("good")}}}}
|
||||||
|
l, h := judgingLeg(t, w)
|
||||||
|
if r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{}); r.Layer != "" || len(h.reports) != 0 {
|
||||||
|
t.Fatalf("an ordinary boot must report nothing: %+v %+v", r, h.reports)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The signed executor STAGES (listed + the raw envelope + the kver) and never reboots.
|
||||||
|
func TestKernelStepExecutor_StagesNeverReboots(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0",
|
||||||
|
Version: "7.0.14-22"}}, Authority: "signed", Kernel: kview("staged")}}}}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
e := KernelStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
|
||||||
|
params, _ := json.Marshal(KernelStepParams{ReleaseID: "os-kernel-1", Kver: kNew,
|
||||||
|
Packages: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22", Origin: PVEOrigin}}})
|
||||||
|
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_kernel_step"}`), Sig: []byte("SIG")})
|
||||||
|
if err := e.Execute(ctx, OpKernelStep, params); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
pp := w.plans[len(w.plans)-1]
|
||||||
|
sg, _ := pp["signed"].(map[string]any)
|
||||||
|
if pp["mode"] != "apply" || pp["select"] != "listed" || pp["expect_kver"] != kNew || sg == nil ||
|
||||||
|
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_kernel_step"}`)) {
|
||||||
|
t.Fatalf("plan = %v", pp)
|
||||||
|
}
|
||||||
|
if got := strings.Join(kernelCalls(w), ","); got != "apply" {
|
||||||
|
t.Fatalf("a signed stage must never reboot: %s", got)
|
||||||
|
}
|
||||||
|
if len(h.reports) != 1 || h.reports[0].Outcome != "staged" {
|
||||||
|
t.Fatalf("hub = %+v", h.reports)
|
||||||
|
}
|
||||||
|
if err := e.Execute(context.Background(), OpKernelStep, params); err == nil {
|
||||||
|
t.Fatal("no envelope must refuse")
|
||||||
|
}
|
||||||
|
bad, _ := json.Marshal(KernelStepParams{Kver: "x", Packages: []Package{{Name: "a"}}})
|
||||||
|
if err := e.Execute(ctx, OpKernelStep, bad); err == nil {
|
||||||
|
t.Fatal("a bad kver must refuse")
|
||||||
|
}
|
||||||
|
if err := e.Execute(ctx, OpPVEStep, params); err != signedjobs.ErrNoExecutor {
|
||||||
|
t.Fatalf("another op must pass through the chain: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// os_kernel_step is never benign.
|
||||||
|
func TestKernelStep_IsDestructiveClass(t *testing.T) {
|
||||||
|
if reconcile.Classify(reconcile.ClassOSKernelStep, reconcile.Provenance{}) != reconcile.Destructive {
|
||||||
|
t.Fatal("os_kernel_step must be destructive-class (signed, operational key)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A kept stage report (the agent was killed mid-stage) reaches the hub as "staged" with its kernel view.
|
||||||
|
func TestKernel_KeptStageReportIsStaged(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t}
|
||||||
|
l, _ := newLeg(t, w, tonight(0))
|
||||||
|
ring := 0
|
||||||
|
rep := l.reportFromKept(context.Background(), WrapperReport{Layer: LayerKernel, Mode: "apply", Ring: &ring,
|
||||||
|
Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Kernel: kview("staged")}, "/x/report-r-kernel-apply.json")
|
||||||
|
if rep.Outcome != "staged" || !rep.Healthy || len(rep.Kernel) == 0 {
|
||||||
|
t.Fatalf("kept = %+v", rep)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestKernelSet(t *testing.T) {
|
||||||
|
if got := KernelSet("7.0.14-20-pve"); len(got) != 2 || got[0].Name != "proxmox-kernel-7.0" || got[0].Version != "7.0.14-20" ||
|
||||||
|
got[1].Name != "proxmox-kernel-7.0.14-20-pve-signed" {
|
||||||
|
t.Fatalf("%+v", got)
|
||||||
|
}
|
||||||
|
if KernelSet("7.0; reboot") != nil || KernelSet("") != nil {
|
||||||
|
t.Fatal("a non-kernel string must give no set")
|
||||||
|
}
|
||||||
|
}
|
||||||
+153
-6
@@ -20,6 +20,7 @@ import (
|
|||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -27,6 +28,7 @@ import (
|
|||||||
|
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||||
)
|
)
|
||||||
|
|
||||||
// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
|
// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
|
||||||
@@ -40,8 +42,24 @@ const (
|
|||||||
LayerGuest = "guest"
|
LayerGuest = "guest"
|
||||||
LayerHost = "host"
|
LayerHost = "host"
|
||||||
LayerDocker = "docker" // the guest's Docker engine set — slow lane (`11` §5.8)
|
LayerDocker = "docker" // the guest's Docker engine set — slow lane (`11` §5.8)
|
||||||
|
// LayerPVE is the HOST's Proxmox userspace packages — slow lane (R-812 option A, `09` §3 decision 163, `11` §5.10):
|
||||||
|
// ring 0 in the night leg after a healthy host step, ring 1 only inside a signed os_pve_step. Never a kernel.
|
||||||
|
LayerPVE = "pve"
|
||||||
|
// LayerKernel is the HOST's kernel — the kernel lane (R-836, `09` §3 decision 172, `11` §5.11): a one-shot boot of
|
||||||
|
// the new kernel through the ESP flag, the default moved only after a healthy boot (kernel.go).
|
||||||
|
LayerKernel = "kernel"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// PVEOrigin is the origin apt prints for download.proxmox.com (the wrapper's PVE_ORIGIN); InstalledPVEOrigin is how
|
||||||
|
// the wrapper's inventory names the same source.
|
||||||
|
const (
|
||||||
|
PVEOrigin = "Proxmox Debian Repository"
|
||||||
|
InstalledPVEOrigin = "Proxmox"
|
||||||
|
)
|
||||||
|
|
||||||
|
// hostSlowRE mirrors the wrapper's HOST_SLOW_RE: kernel, boot, firmware and microcode names never ride the pve lane.
|
||||||
|
var hostSlowRE = regexp.MustCompile(`^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)`)
|
||||||
|
|
||||||
// DockerNames are the six packages of the Docker engine set (the wrapper's DOCKER_NAMES).
|
// DockerNames are the six packages of the Docker engine set (the wrapper's DOCKER_NAMES).
|
||||||
var DockerNames = map[string]bool{"containerd.io": true, "docker-buildx-plugin": true, "docker-ce": true,
|
var DockerNames = map[string]bool{"containerd.io": true, "docker-buildx-plugin": true, "docker-ce": true,
|
||||||
"docker-ce-cli": true, "docker-ce-rootless-extras": true, "docker-compose-plugin": true}
|
"docker-ce-cli": true, "docker-ce-rootless-extras": true, "docker-compose-plugin": true}
|
||||||
@@ -109,6 +127,13 @@ type WrapperReport struct {
|
|||||||
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
|
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
|
||||||
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
|
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
|
||||||
OOMCheck json.RawMessage `json:"oom_check"`
|
OOMCheck json.RawMessage `json:"oom_check"`
|
||||||
|
// PVEManager: the pve layer — pveversion's pve-manager version after the step ("unknown" = unreadable).
|
||||||
|
PVEManager string `json:"pve_manager"`
|
||||||
|
// Kernel (R-836): the kernel layer's view {running, default, flag, phase, from, to, …}; KernelEvent what kernel-boot
|
||||||
|
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
|
||||||
|
Kernel json.RawMessage `json:"kernel"`
|
||||||
|
KernelEvent string `json:"kernel_event"`
|
||||||
|
OutcomeHint string `json:"outcome_hint"`
|
||||||
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
||||||
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
||||||
RunID string `json:"run_id"`
|
RunID string `json:"run_id"`
|
||||||
@@ -149,6 +174,12 @@ type Report struct {
|
|||||||
// OOMCheck: docker layer — the wrapper's oom_check object, byte-for-byte (R-528; the hub decides approval on it).
|
// OOMCheck: docker layer — the wrapper's oom_check object, byte-for-byte (R-528; the hub decides approval on it).
|
||||||
// Pinned by TestDocker_OOMCheckReachesTheHubUnchanged and TestR868_KeptCopyCarriesTheOOMCheck.
|
// Pinned by TestDocker_OOMCheckReachesTheHubUnchanged and TestR868_KeptCopyCarriesTheOOMCheck.
|
||||||
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
|
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
|
||||||
|
// PVEManager: pve layer — pve-manager's version after the step (the hub's System page; R-812 option A).
|
||||||
|
PVEManager string `json:"pve_manager,omitempty"`
|
||||||
|
// Kernel: kernel layer — the wrapper's kernel view, byte for byte (running, default, flag, phase, from, to). The
|
||||||
|
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
|
||||||
|
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
|
||||||
|
Kernel json.RawMessage `json:"kernel,omitempty"`
|
||||||
|
|
||||||
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
|
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
|
||||||
}
|
}
|
||||||
@@ -386,6 +417,36 @@ func EngineOf(pkgVersion string) string {
|
|||||||
return v
|
return v
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// PVEHealthVerdict is THE Proxmox-package-step health rule (R-812 option A; pinned by TestPVEHealthVerdict): the host
|
||||||
|
// rule (every host service active, the guest running and healthy, the tunnel running), plus every container running at
|
||||||
|
// the start still runs as the SAME container (a Proxmox step must not restart the household's apps), plus pveversion
|
||||||
|
// now reports the pve-manager the step installed (wantPVE "" = pve-manager was not in the step).
|
||||||
|
func PVEHealthVerdict(before, after *Health, tunnel, wantPVE, gotPVE string) (bool, string) {
|
||||||
|
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
|
||||||
|
return false, why
|
||||||
|
}
|
||||||
|
if before != nil && before.Guest != nil && after.Guest != nil {
|
||||||
|
names := make([]string, 0, len(before.Guest.Containers))
|
||||||
|
for n := range before.Guest.Containers {
|
||||||
|
names = append(names, n)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
for _, n := range names {
|
||||||
|
b := before.Guest.Containers[n]
|
||||||
|
if b.State != "running" || b.ID == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if a := after.Guest.Containers[n]; a.ID != b.ID {
|
||||||
|
return false, n + " is a new container (id changed) — the Proxmox step restarted the household's app"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if wantPVE != "" && gotPVE != wantPVE {
|
||||||
|
return false, "pveversion reads pve-manager " + gotPVE + ", not " + wantPVE
|
||||||
|
}
|
||||||
|
return true, ""
|
||||||
|
}
|
||||||
|
|
||||||
// DockerHealthVerdict is THE Docker-step health rule (`11` §5.8; pinned by TestDockerHealthVerdict): the guest rule,
|
// DockerHealthVerdict is THE Docker-step health rule (`11` §5.8; pinned by TestDockerHealthVerdict): the guest rule,
|
||||||
// plus every container running at the start still runs as the SAME container (same id — a changed id means the
|
// plus every container running at the start still runs as the SAME container (same id — a changed id means the
|
||||||
// household's apps restarted, which `live-restore` exists to prevent), plus the engine now reports the version the step
|
// household's apps restarted, which `live-restore` exists to prevent), plus the engine now reports the version the step
|
||||||
@@ -451,7 +512,7 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
|
|||||||
|
|
||||||
// Pass is one leg's reports; an empty Layer means the step did not run.
|
// Pass is one leg's reports; an empty Layer means the step did not run.
|
||||||
type Pass struct {
|
type Pass struct {
|
||||||
Guest, Host, Docker Report
|
Guest, Host, Docker, PVE, Kernel Report
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
||||||
@@ -479,13 +540,54 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
|
|||||||
if err := l.EnsureLiveRestore(ctx, g.RunID, vmid); err != nil {
|
if err := l.EnsureLiveRestore(ctx, g.RunID, vmid); err != nil {
|
||||||
p.Docker = l.finish(ctx, lg, Report{RunID: g.RunID, Layer: LayerDocker, Trigger: trigger, Ring: 0, VMID: vmid,
|
p.Docker = l.finish(ctx, lg, Report{RunID: g.RunID, Layer: LayerDocker, Trigger: trigger, Ring: 0, VMID: vmid,
|
||||||
Mode: "apply", Outcome: "failed", HealthReason: "live-restore could not be turned on: " + err.Error()})
|
Mode: "apply", Outcome: "failed", HealthReason: "live-restore could not be turned on: " + err.Error()})
|
||||||
return p
|
} else {
|
||||||
|
p.Docker = l.runLayer(ctx, g.RunID, LayerDocker, vmid, trigger, blk, dockerOpts{})
|
||||||
}
|
}
|
||||||
p.Docker = l.runLayer(ctx, g.RunID, LayerDocker, vmid, trigger, blk, dockerOpts{})
|
}
|
||||||
|
// R-812 option A: the Proxmox package step — ring 0, an appliance, after a HEALTHY host step (it is a host change).
|
||||||
|
// A Docker step's outcome does not gate it (the Docker set lives in the guest). Pinned by TestPVE_*.
|
||||||
|
switch {
|
||||||
|
case blk.Ring != 0 || !blk.Enabled:
|
||||||
|
lg.Info("osupdate: pve step skipped — ring 1 takes a Proxmox set only inside a signed operator job (`11` §5.10)", "ring", blk.Ring, "enabled", blk.Enabled)
|
||||||
|
case !l.Appliance || h.Layer == "" || !okStep(h):
|
||||||
|
lg.Info("osupdate: pve step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
|
||||||
|
default:
|
||||||
|
p.PVE = l.runPVE(ctx, g.RunID, vmid, trigger, blk, dockerOpts{})
|
||||||
|
}
|
||||||
|
// R-836 (`09` §3 decision 172): the kernel step ENDS the night — an appliance, after a healthy host step (and a
|
||||||
|
// healthy Proxmox step when one ran), only on a night the hub marks as told. It reboots the box. Pinned by TestKernel_*.
|
||||||
|
switch {
|
||||||
|
case !l.Appliance || h.Layer == "" || !okStep(h):
|
||||||
|
lg.Info("osupdate: kernel step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
|
||||||
|
case p.PVE.Layer != "" && !okStep(p.PVE):
|
||||||
|
lg.Warn("osupdate: kernel step skipped — the Proxmox step did not end healthy", "pve_outcome", p.PVE.Outcome)
|
||||||
|
default:
|
||||||
|
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
|
||||||
}
|
}
|
||||||
return p
|
return p
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// pveDrainWait bounds how long a pve step waits for the agent's own /etc/pve writes in flight (pvegate).
|
||||||
|
var pveDrainWait = 2 * time.Minute
|
||||||
|
|
||||||
|
// runPVE runs the pve layer while holding pvegate: the agent's own /etc/pve writes wait until it ends.
|
||||||
|
func (l *Leg) runPVE(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate, do dockerOpts) Report {
|
||||||
|
lg := l.log().With("run", runID, "layer", LayerPVE, "vmid", vmid, "trigger", trigger)
|
||||||
|
dctx, cancel := context.WithTimeout(ctx, pveDrainWait)
|
||||||
|
end, err := pvegate.Step(dctx)
|
||||||
|
cancel()
|
||||||
|
if err != nil {
|
||||||
|
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerPVE, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply",
|
||||||
|
ReleaseID: do.releaseID, Outcome: "failed", HealthReason: "an agent write to /etc/pve did not finish in time (pvegate): " + err.Error()})
|
||||||
|
}
|
||||||
|
lg.Info("osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends")
|
||||||
|
defer func() {
|
||||||
|
end()
|
||||||
|
lg.Info("osupdate: pve step released the /etc/pve write gate")
|
||||||
|
}()
|
||||||
|
return l.runLayer(ctx, runID, LayerPVE, vmid, trigger, blk, do)
|
||||||
|
}
|
||||||
|
|
||||||
// dockerOpts is a signed Docker step (DockerStepExecutor); the zero value is ring 0's unsigned "pending-docker".
|
// dockerOpts is a signed Docker step (DockerStepExecutor); the zero value is ring 0's unsigned "pending-docker".
|
||||||
type dockerOpts struct {
|
type dockerOpts struct {
|
||||||
releaseID string
|
releaseID string
|
||||||
@@ -571,7 +673,7 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
|||||||
wire = blk.HostRelease
|
wire = blk.HostRelease
|
||||||
}
|
}
|
||||||
lane := "fast"
|
lane := "fast"
|
||||||
if layer == LayerDocker {
|
if layer == LayerDocker || layer == LayerPVE {
|
||||||
lane = "slow"
|
lane = "slow"
|
||||||
if do.signed != nil {
|
if do.signed != nil {
|
||||||
rel = hub.WireOSRelease{ID: do.releaseID}
|
rel = hub.WireOSRelease{ID: do.releaseID}
|
||||||
@@ -604,6 +706,13 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
|||||||
}
|
}
|
||||||
case layer == LayerDocker:
|
case layer == LayerDocker:
|
||||||
plan["select"] = "pending-docker" // ring 0: the wrapper checks the box's ROOT-OWNED ring-0 mark itself
|
plan["select"] = "pending-docker" // ring 0: the wrapper checks the box's ROOT-OWNED ring-0 mark itself
|
||||||
|
case layer == LayerPVE && do.signed != nil:
|
||||||
|
plan["packages"], plan["signed"] = do.packages, do.signed
|
||||||
|
for _, p := range do.packages {
|
||||||
|
planned[p.Name] = true
|
||||||
|
}
|
||||||
|
case layer == LayerPVE:
|
||||||
|
plan["select"] = "pending-pve" // ring 0: the same root-owned mark; the wrapper picks installed Proxmox userspace
|
||||||
case !blk.Enabled:
|
case !blk.Enabled:
|
||||||
plan["mode"] = "inventory"
|
plan["mode"] = "inventory"
|
||||||
lg.Info("osupdate: switched OFF for this box — reporting only")
|
lg.Info("osupdate: switched OFF for this box — reporting only")
|
||||||
@@ -637,6 +746,7 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
|||||||
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
||||||
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
||||||
rep.OOMCheck = rawOrNil(wr.OOMCheck)
|
rep.OOMCheck = rawOrNil(wr.OOMCheck)
|
||||||
|
rep.PVEManager = wr.PVEManager
|
||||||
if rep.Outcome == "" {
|
if rep.Outcome == "" {
|
||||||
switch {
|
switch {
|
||||||
case rep.Mode == "inventory" && !blk.Enabled:
|
case rep.Mode == "inventory" && !blk.Enabled:
|
||||||
@@ -654,21 +764,27 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
|||||||
}
|
}
|
||||||
// Health: compare with the start of the pass; give restarted services time (only after an install).
|
// Health: compare with the start of the pass; give restarted services time (only after an install).
|
||||||
cur := wr.HealthAfter
|
cur := wr.HealthAfter
|
||||||
wantEngine := ""
|
wantEngine, wantPVE := "", ""
|
||||||
for _, u := range wr.Upgraded {
|
for _, u := range wr.Upgraded {
|
||||||
if u.Name == "docker-ce" {
|
if u.Name == "docker-ce" {
|
||||||
wantEngine = EngineOf(u.Version)
|
wantEngine = EngineOf(u.Version)
|
||||||
}
|
}
|
||||||
|
if u.Name == "pve-manager" {
|
||||||
|
wantPVE = u.Version
|
||||||
|
}
|
||||||
}
|
}
|
||||||
verdict := func(h *Health) (bool, string) {
|
verdict := func(h *Health) (bool, string) {
|
||||||
if layer == LayerDocker {
|
if layer == LayerDocker {
|
||||||
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
|
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
|
||||||
}
|
}
|
||||||
if layer == LayerHost {
|
if layer == LayerHost || layer == LayerPVE {
|
||||||
t := hub.TunnelUnknown
|
t := hub.TunnelUnknown
|
||||||
if l.Tunnel != nil {
|
if l.Tunnel != nil {
|
||||||
t, _ = l.Tunnel.Status(ctx)
|
t, _ = l.Tunnel.Status(ctx)
|
||||||
}
|
}
|
||||||
|
if layer == LayerPVE {
|
||||||
|
return PVEHealthVerdict(wr.HealthBefore, h, t, wantPVE, wr.PVEManager)
|
||||||
|
}
|
||||||
return HostHealthVerdict(wr.HealthBefore, h, t)
|
return HostHealthVerdict(wr.HealthBefore, h, t)
|
||||||
}
|
}
|
||||||
return HealthVerdict(wr.HealthBefore, h)
|
return HealthVerdict(wr.HealthBefore, h)
|
||||||
@@ -708,6 +824,10 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
|
|||||||
// the docker report carries the engine set only (the guest report already carries the Debian packages)
|
// the docker report carries the engine set only (the guest report already carries the Debian packages)
|
||||||
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
|
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
|
||||||
rep.NotCovered = nil
|
rep.NotCovered = nil
|
||||||
|
} else if layer == LayerPVE {
|
||||||
|
// the pve report carries the Proxmox userspace set only — the hub's candidate is built from it
|
||||||
|
rep.Installed, rep.Pending = onlyPVE(wr.Installed), onlyPVEPending(wr.Pending)
|
||||||
|
rep.NotCovered = nil
|
||||||
} else {
|
} else {
|
||||||
rep.NotCovered = notCovered(wr.Pending, blk.Ring, planned)
|
rep.NotCovered = notCovered(wr.Pending, blk.Ring, planned)
|
||||||
}
|
}
|
||||||
@@ -732,6 +852,33 @@ func onlyDocker(in []Package) []Package {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// onlyPVE keeps the installed Proxmox-origin packages the pve lane may touch (never a kernel / boot / firmware name).
|
||||||
|
func onlyPVE(in []Package) []Package {
|
||||||
|
var out []Package
|
||||||
|
for _, p := range in {
|
||||||
|
if (p.Origin == InstalledPVEOrigin || p.Origin == PVEOrigin) && !hostSlowRE.MatchString(p.Name) && !DockerNames[p.Name] {
|
||||||
|
out = append(out, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func onlyPVEPending(in []Pending) []Pending {
|
||||||
|
var out []Pending
|
||||||
|
for _, p := range in {
|
||||||
|
if p.From == "" || hostSlowRE.MatchString(p.Name) || DockerNames[p.Name] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, o := range p.Origin {
|
||||||
|
if o == PVEOrigin {
|
||||||
|
out = append(out, p)
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func onlyDockerPending(in []Pending) []Pending {
|
func onlyDockerPending(in []Pending) []Pending {
|
||||||
var out []Pending
|
var out []Pending
|
||||||
for _, p := range in {
|
for _, p := range in {
|
||||||
|
|||||||
@@ -13,16 +13,19 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||||
)
|
)
|
||||||
|
|
||||||
// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per layer and mode.
|
// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per layer and mode.
|
||||||
type fakeWrapper struct {
|
type fakeWrapper struct {
|
||||||
t *testing.T
|
t *testing.T
|
||||||
pending []Pending
|
pending []Pending
|
||||||
applyRep map[string]WrapperReport // per layer
|
applyRep map[string]WrapperReport // per layer
|
||||||
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
|
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
|
||||||
plans []map[string]any
|
plans []map[string]any
|
||||||
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
||||||
|
pveGateHeld bool
|
||||||
|
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
|
||||||
}
|
}
|
||||||
|
|
||||||
func yes() *bool { b := true; return &b }
|
func yes() *bool { b := true; return &b }
|
||||||
@@ -50,9 +53,22 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
|
|||||||
f.plans = append(f.plans, plan)
|
f.plans = append(f.plans, plan)
|
||||||
layer := plan["layer"].(string)
|
layer := plan["layer"].(string)
|
||||||
ok := guestOK()
|
ok := guestOK()
|
||||||
if layer == LayerHost {
|
if layer == LayerHost || layer == LayerPVE || layer == LayerKernel {
|
||||||
ok = hostOK()
|
ok = hostOK()
|
||||||
}
|
}
|
||||||
|
if layer == LayerKernel {
|
||||||
|
if seq := f.kernelRep[plan["mode"].(string)]; len(seq) > 0 {
|
||||||
|
rep := seq[0]
|
||||||
|
if len(seq) > 1 {
|
||||||
|
f.kernelRep[plan["mode"].(string)] = seq[1:]
|
||||||
|
}
|
||||||
|
out, _ := json.Marshal(rep)
|
||||||
|
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if layer == LayerPVE && plan["mode"] == "apply" {
|
||||||
|
f.pveGateHeld = pvegate.Stepping() // R-812: the /etc/pve write gate must be held while the pve step runs
|
||||||
|
}
|
||||||
var rep WrapperReport
|
var rep WrapperReport
|
||||||
switch plan["mode"] {
|
switch plan["mode"] {
|
||||||
case "inventory":
|
case "inventory":
|
||||||
@@ -160,8 +176,8 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
|
|||||||
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
|
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
|
||||||
t.Fatalf("guest %+v\nhost %+v", g, ho)
|
t.Fatalf("guest %+v\nhost %+v", g, ho)
|
||||||
}
|
}
|
||||||
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply" {
|
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply" {
|
||||||
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore and the ring-0 docker step", calls(w))
|
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step and the pve step", calls(w))
|
||||||
}
|
}
|
||||||
for _, p := range w.plans[:2] {
|
for _, p := range w.plans[:2] {
|
||||||
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
|
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
|
||||||
@@ -171,7 +187,7 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
|
|||||||
if len(g.NotCovered) != 1 || g.NotCovered[0] != "docker-ce" {
|
if len(g.NotCovered) != 1 || g.NotCovered[0] != "docker-ce" {
|
||||||
t.Fatalf("not covered = %v", g.NotCovered)
|
t.Fatalf("not covered = %v", g.NotCovered)
|
||||||
}
|
}
|
||||||
if len(h.reports) != 3 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost || h.reports[2].Layer != LayerDocker {
|
if len(h.reports) != 4 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost || h.reports[2].Layer != LayerDocker || h.reports[3].Layer != LayerPVE {
|
||||||
t.Fatalf("hub got %+v", h.reports)
|
t.Fatalf("hub got %+v", h.reports)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -389,7 +405,7 @@ func TestHostReport_CarriesRebootScanned(t *testing.T) {
|
|||||||
}}
|
}}
|
||||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
run2(l, "night")
|
run2(l, "night")
|
||||||
if len(h.reports) != 3 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
|
if len(h.reports) != 4 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
|
||||||
t.Fatalf("hub got %+v", h.reports)
|
t.Fatalf("hub got %+v", h.reports)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -436,7 +452,12 @@ func TestDocker_Ring0PlanAndReport(t *testing.T) {
|
|||||||
DockerEngine: "29.8.2", Authority: "ring0"}}}
|
DockerEngine: "29.8.2", Authority: "ring0"}}}
|
||||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
p := l.Run(context.Background(), 9201, "night")
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
dp := w.plans[len(w.plans)-1]
|
var dp map[string]any
|
||||||
|
for _, x := range w.plans {
|
||||||
|
if x["layer"] == "docker" {
|
||||||
|
dp = x
|
||||||
|
}
|
||||||
|
}
|
||||||
if dp["layer"] != "docker" || dp["lane"] != "slow" || dp["select"] != "pending-docker" {
|
if dp["layer"] != "docker" || dp["lane"] != "slow" || dp["select"] != "pending-docker" {
|
||||||
t.Fatalf("docker plan = %v", dp)
|
t.Fatalf("docker plan = %v", dp)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,152 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ---- the Proxmox package step (R-812 option A, `09` §3 decision 163, `11` §5.10) ----
|
||||||
|
|
||||||
|
// Ring 0: after a healthy host step the leg runs the pve layer — slow lane, select pending-pve — while holding the
|
||||||
|
// /etc/pve write gate; the report carries only Proxmox userspace packages and pve-manager's version.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): call runLayer instead of runPVE in Run → "the /etc/pve write gate was not held".
|
||||||
|
func TestPVE_Ring0PlanGateAndReport(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {
|
||||||
|
Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
|
||||||
|
Installed: []Package{{Name: "pve-manager", Version: "9.2.21", Origin: "Proxmox"},
|
||||||
|
{Name: "proxmox-kernel-helper", Version: "9.0.4", Origin: "Proxmox"}, {Name: "libc6", Version: "u4", Origin: "Debian"}},
|
||||||
|
Pending: []Pending{{Name: "qemu-server", From: "9.0.1", To: "9.0.9", Origin: []string{PVEOrigin}},
|
||||||
|
{Name: "proxmox-kernel-7.0", From: "7.0.2", To: "7.0.14", Origin: []string{PVEOrigin}},
|
||||||
|
{Name: "libc6", From: "u3", To: "u4", Origin: []string{"Debian"}}},
|
||||||
|
PVEManager: "9.2.21", Authority: "ring0"}}}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
|
var pp map[string]any
|
||||||
|
for _, x := range w.plans {
|
||||||
|
if x["layer"] == LayerPVE {
|
||||||
|
pp = x
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if pp == nil || pp["lane"] != "slow" || pp["select"] != "pending-pve" {
|
||||||
|
t.Fatalf("pve plan = %v (calls %s)", pp, calls(w))
|
||||||
|
}
|
||||||
|
if !w.pveGateHeld {
|
||||||
|
t.Fatal("the /etc/pve write gate was not held while the pve step ran")
|
||||||
|
}
|
||||||
|
if pvegate.Stepping() {
|
||||||
|
t.Fatal("the gate must be released after the step")
|
||||||
|
}
|
||||||
|
r := p.PVE
|
||||||
|
if r.Outcome != "applied" || !r.Healthy || r.PVEManager != "9.2.21" {
|
||||||
|
t.Fatalf("pve report = %+v", r)
|
||||||
|
}
|
||||||
|
if len(r.Installed) != 1 || r.Installed[0].Name != "pve-manager" || len(r.Pending) != 1 || r.Pending[0].Name != "qemu-server" {
|
||||||
|
t.Fatalf("the pve report must carry Proxmox userspace only: installed=%v pending=%v", r.Installed, r.Pending)
|
||||||
|
}
|
||||||
|
if h.reports[len(h.reports)-1].Layer != LayerPVE {
|
||||||
|
t.Fatalf("the hub must get the pve report: %+v", h.reports)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ring 1 never takes a Proxmox step in the night leg.
|
||||||
|
func TestPVE_Ring1NightLegNeverSteps(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
if p := l.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w), "pve") {
|
||||||
|
t.Fatalf("ring 1 took a pve step: %s", calls(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// No healthy host step (a BYO box, or an unhealthy host step) → no pve step.
|
||||||
|
func TestPVE_SkippedWithoutAHealthyHostStep(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
l.Appliance = false
|
||||||
|
if p := l.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w), "pve") {
|
||||||
|
t.Fatalf("a BYO box took a pve step: %s", calls(w))
|
||||||
|
}
|
||||||
|
w2 := &fakeWrapper{t: t}
|
||||||
|
l2, _ := newLeg(t, w2, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
l2.Tunnel = fakeTunnel{"stopped"} // the host step reads unhealthy
|
||||||
|
w2.applyRep = map[string]WrapperReport{LayerHost: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}
|
||||||
|
if p := l2.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w2), "pve") {
|
||||||
|
t.Fatalf("a pve step ran after an unhealthy host step: %s", calls(w2))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A write in flight that never finishes makes the pve step give up (failed), never run without the gate.
|
||||||
|
func TestPVE_GivesUpWhenAWriteDoesNotFinish(t *testing.T) {
|
||||||
|
old := pveDrainWait
|
||||||
|
pveDrainWait = 50_000_000 // 50 ms
|
||||||
|
defer func() { pveDrainWait = old }()
|
||||||
|
rel, _, _ := pvegate.Write(context.Background())
|
||||||
|
defer rel()
|
||||||
|
w := &fakeWrapper{t: t}
|
||||||
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
|
if p.PVE.Outcome != "failed" || strings.Contains(calls(w), "pve") {
|
||||||
|
t.Fatalf("the pve step must fail without a wrapper call: %+v calls=%s", p.PVE, calls(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE pve health rule. COMPANION RED-PROOF (observed): drop the container-id loop or the pve-manager check in
|
||||||
|
// PVEHealthVerdict → the matching case below fails.
|
||||||
|
func TestPVEHealthVerdict(t *testing.T) {
|
||||||
|
before, after := hostOK(), hostOK()
|
||||||
|
before.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "a1"}
|
||||||
|
after.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "a1"}
|
||||||
|
if ok, why := PVEHealthVerdict(before, after, hub.TunnelRunning, "9.2.21", "9.2.21"); !ok {
|
||||||
|
t.Fatalf("healthy step read unhealthy: %s", why)
|
||||||
|
}
|
||||||
|
if ok, _ := PVEHealthVerdict(before, after, hub.TunnelRunning, "9.2.21", "9.2.2"); ok {
|
||||||
|
t.Fatal("pveversion still on the old pve-manager must fail")
|
||||||
|
}
|
||||||
|
after.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "b2"}
|
||||||
|
if ok, why := PVEHealthVerdict(before, after, hub.TunnelRunning, "", "9.2.2"); ok || !strings.Contains(why, "id changed") {
|
||||||
|
t.Fatalf("an app restarted by the Proxmox step must fail, got ok=%v %q", ok, why)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The signed executor hands the RAW envelope and the exact list to the wrapper's pve layer.
|
||||||
|
func TestPVEStepExecutor_PassesTheSignedEnvelope(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {
|
||||||
|
Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}}, PVEManager: "9.2.21", Authority: "signed"}}}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
e := PVEStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
|
||||||
|
params, _ := json.Marshal(PVEStepParams{ReleaseID: "os-pve-1", Packages: []Package{{Name: "pve-manager", Version: "9.2.21", Origin: PVEOrigin}}})
|
||||||
|
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_pve_step"}`), Sig: []byte("SIG")})
|
||||||
|
if err := e.Execute(ctx, OpPVEStep, params); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
pp := w.plans[len(w.plans)-1]
|
||||||
|
sg, _ := pp["signed"].(map[string]any)
|
||||||
|
if pp["layer"] != LayerPVE || pp["lane"] != "slow" || pp["release_id"] != "os-pve-1" || sg == nil ||
|
||||||
|
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_pve_step"}`)) || sg["sig"] != "SIG" {
|
||||||
|
t.Fatalf("pve plan = %v", pp)
|
||||||
|
}
|
||||||
|
if !w.pveGateHeld || calls(w) != "pve:apply" || len(h.reports) != 1 || h.reports[0].Trigger != "signed" {
|
||||||
|
t.Fatalf("gate=%v calls=%s reports=%+v", w.pveGateHeld, calls(w), h.reports)
|
||||||
|
}
|
||||||
|
if err := e.Execute(context.Background(), OpPVEStep, params); err == nil {
|
||||||
|
t.Fatal("no envelope must refuse")
|
||||||
|
}
|
||||||
|
if err := e.Execute(context.Background(), OpDockerStep, params); err != signedjobs.ErrNoExecutor {
|
||||||
|
t.Fatalf("another op must pass through the chain: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// os_pve_step is never benign.
|
||||||
|
func TestPVEStep_IsDestructiveClass(t *testing.T) {
|
||||||
|
if reconcile.Classify(reconcile.ClassOSPVEStep, reconcile.Provenance{}) != reconcile.Destructive {
|
||||||
|
t.Fatal("os_pve_step must be destructive-class (signed, operational key)")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// OpPVEStep is the signed op class of a Proxmox package step (R-812 option A, `11` §5.10): a ring-1 box takes an
|
||||||
|
// approved Proxmox set only through it. No undo in this release. CC may sign it until the first paying customer.
|
||||||
|
const OpPVEStep = "os_pve_step"
|
||||||
|
|
||||||
|
// PVEStepParams are the signed params. The wrapper compares Packages with the plan byte-for-byte.
|
||||||
|
type PVEStepParams struct {
|
||||||
|
ReleaseID string `json:"release_id"`
|
||||||
|
Packages []Package `json:"packages"`
|
||||||
|
VMID int `json:"vmid,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// PVEStepExecutor runs a verified os_pve_step (signedjobs.Executor) under the host-wide heavy-op gate (Gate) and the
|
||||||
|
// /etc/pve write gate (inside runPVE).
|
||||||
|
type PVEStepExecutor struct {
|
||||||
|
Leg *Leg
|
||||||
|
Guest func(ctx context.Context) (int, error)
|
||||||
|
Gate func(ctx context.Context) (release func(), err error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Execute implements signedjobs.Executor.
|
||||||
|
func (e PVEStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||||
|
if op != OpPVEStep {
|
||||||
|
return signedjobs.ErrNoExecutor
|
||||||
|
}
|
||||||
|
so, ok := signedjobs.SignedOpFrom(ctx)
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("os_pve_step: no signed envelope in the context — the wrapper could not verify it")
|
||||||
|
}
|
||||||
|
var p PVEStepParams
|
||||||
|
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 {
|
||||||
|
return fmt.Errorf("os_pve_step: params must name the Proxmox set: %v", err)
|
||||||
|
}
|
||||||
|
vmid := p.VMID
|
||||||
|
if vmid == 0 {
|
||||||
|
if e.Guest == nil {
|
||||||
|
return fmt.Errorf("os_pve_step: no vmid and no guest finder")
|
||||||
|
}
|
||||||
|
v, err := e.Guest(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("os_pve_step: find the customer guest: %w", err)
|
||||||
|
}
|
||||||
|
vmid = v
|
||||||
|
}
|
||||||
|
if e.Gate != nil {
|
||||||
|
release, err := e.Gate(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("os_pve_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
}
|
||||||
|
rep := e.Leg.RunPVESigned(ctx, vmid, p, so.Blob, string(so.Sig))
|
||||||
|
switch rep.Outcome {
|
||||||
|
case "applied", "nothing":
|
||||||
|
if rep.Healthy {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fmt.Errorf("os_pve_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunPVESigned is one signed Proxmox step (ring 1): the pve layer with the signed envelope, which the wrapper verifies
|
||||||
|
// itself, holding the /etc/pve write gate.
|
||||||
|
func (l *Leg) RunPVESigned(ctx context.Context, vmid int, p PVEStepParams, blob []byte, sig string) Report {
|
||||||
|
unlock := l.lockPass(true)
|
||||||
|
defer unlock()
|
||||||
|
l.sendUnsentLocked(ctx) // R-868
|
||||||
|
runID := l.now().UTC().Format("20060102T150405Z")
|
||||||
|
rid := p.ReleaseID
|
||||||
|
if rid == "" {
|
||||||
|
rid = "signed-" + runID
|
||||||
|
}
|
||||||
|
return l.runPVE(ctx, runID, vmid, "signed", l.Block(), dockerOpts{releaseID: rid, packages: p.Packages,
|
||||||
|
signed: map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}})
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// After a boot the agent has not fetched the hub's block yet; the after-boot kernel report must still carry the
|
||||||
|
// box's real ring (seen 2026-10-08: a ring-0 box's „judging" report said ring 1). Red-proof: return Block().Ring
|
||||||
|
// from kernelReportRing and the saved-block case fails with 1.
|
||||||
|
func TestKernelReportRing_BeforeFirstFetch(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
l := &Leg{PlanDir: dir}
|
||||||
|
if got := l.kernelReportRing(); got != 1 {
|
||||||
|
t.Fatalf("nothing fetched, nothing saved: ring %d, want 1 (the old default)", got)
|
||||||
|
}
|
||||||
|
l.saveBlock(&hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
l2 := &Leg{PlanDir: dir} // a fresh daemon after the reboot: no block fetched yet
|
||||||
|
if got := l2.kernelReportRing(); got != 0 {
|
||||||
|
t.Fatalf("ring-0 block saved before the reboot: ring %d, want 0", got)
|
||||||
|
}
|
||||||
|
l2.SetBlock(&hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
if got := l2.kernelReportRing(); got != 1 {
|
||||||
|
t.Fatalf("a fetched block wins over the saved one: ring %d, want 1", got)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(filepath.Join(dir, SavedBlockFile)); err != nil {
|
||||||
|
t.Fatalf("saved block missing: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -142,24 +142,42 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
|
|||||||
default:
|
default:
|
||||||
rep.Outcome = "applied"
|
rep.Outcome = "applied"
|
||||||
}
|
}
|
||||||
|
if wr.Layer == LayerKernel {
|
||||||
|
// a kernel STAGE changes nothing the box runs (the new kernel only boots once, at the night's reboot), so its
|
||||||
|
// kept copy needs no fresh health reading (R-836)
|
||||||
|
rep.Kernel = rawOrNil(wr.Kernel)
|
||||||
|
rep.Upgraded, rep.PassSeconds, rep.Authority = wr.Upgraded, wr.PassSeconds, wr.Authority
|
||||||
|
if rep.Outcome == "applied" {
|
||||||
|
rep.Outcome = "staged"
|
||||||
|
}
|
||||||
|
rep.Healthy, rep.HealthReason = rep.Outcome == "staged" || rep.Outcome == "nothing", prefix
|
||||||
|
return rep
|
||||||
|
}
|
||||||
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
||||||
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
||||||
rep.OOMCheck = rawOrNil(wr.OOMCheck)
|
rep.OOMCheck = rawOrNil(wr.OOMCheck)
|
||||||
wantEngine := ""
|
rep.PVEManager = wr.PVEManager
|
||||||
|
wantEngine, wantPVE := "", ""
|
||||||
for _, u := range wr.Upgraded {
|
for _, u := range wr.Upgraded {
|
||||||
if u.Name == "docker-ce" {
|
if u.Name == "docker-ce" {
|
||||||
wantEngine = EngineOf(u.Version)
|
wantEngine = EngineOf(u.Version)
|
||||||
}
|
}
|
||||||
|
if u.Name == "pve-manager" {
|
||||||
|
wantPVE = u.Version
|
||||||
|
}
|
||||||
}
|
}
|
||||||
verdict := func(h *Health) (bool, string) {
|
verdict := func(h *Health) (bool, string) {
|
||||||
switch wr.Layer {
|
switch wr.Layer {
|
||||||
case LayerDocker:
|
case LayerDocker:
|
||||||
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
|
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
|
||||||
case LayerHost:
|
case LayerHost, LayerPVE:
|
||||||
t := hub.TunnelUnknown
|
t := hub.TunnelUnknown
|
||||||
if l.Tunnel != nil {
|
if l.Tunnel != nil {
|
||||||
t, _ = l.Tunnel.Status(ctx)
|
t, _ = l.Tunnel.Status(ctx)
|
||||||
}
|
}
|
||||||
|
if wr.Layer == LayerPVE {
|
||||||
|
return PVEHealthVerdict(wr.HealthBefore, h, t, wantPVE, wr.PVEManager)
|
||||||
|
}
|
||||||
return HostHealthVerdict(wr.HealthBefore, h, t)
|
return HostHealthVerdict(wr.HealthBefore, h, t)
|
||||||
}
|
}
|
||||||
return HealthVerdict(wr.HealthBefore, h)
|
return HealthVerdict(wr.HealthBefore, h)
|
||||||
@@ -167,7 +185,7 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
|
|||||||
ok, why := verdict(wr.HealthAfter)
|
ok, why := verdict(wr.HealthAfter)
|
||||||
if !ok && len(wr.Upgraded) > 0 && wr.VMID > 0 {
|
if !ok && len(wr.Upgraded) > 0 && wr.VMID > 0 {
|
||||||
lane := "fast"
|
lane := "fast"
|
||||||
if wr.Layer == LayerDocker {
|
if wr.Layer == LayerDocker || wr.Layer == LayerPVE {
|
||||||
lane = "slow"
|
lane = "slow"
|
||||||
}
|
}
|
||||||
if hr, err := l.call(ctx, "kept-"+runID, map[string]any{"release_id": "kept", "layer": wr.Layer, "lane": lane,
|
if hr, err := l.call(ctx, "kept-"+runID, map[string]any{"release_id": "kept", "layer": wr.Layer, "lane": lane,
|
||||||
@@ -187,6 +205,8 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
|
|||||||
rep.RebootScanned = wr.RebootScanned
|
rep.RebootScanned = wr.RebootScanned
|
||||||
if wr.Layer == LayerDocker {
|
if wr.Layer == LayerDocker {
|
||||||
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
|
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
|
||||||
|
} else if wr.Layer == LayerPVE {
|
||||||
|
rep.Installed, rep.Pending = onlyPVE(wr.Installed), onlyPVEPending(wr.Pending)
|
||||||
} else {
|
} else {
|
||||||
planned := map[string]bool{}
|
planned := map[string]bool{}
|
||||||
for _, u := range wr.Upgraded {
|
for _, u := range wr.Upgraded {
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||||
"io"
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"net/url"
|
"net/url"
|
||||||
@@ -105,6 +106,15 @@ func (c *Client) do(ctx context.Context, method, path string, body io.Reader, ou
|
|||||||
// doBody is the single HTTP chokepoint: builds the request, sets auth, executes,
|
// doBody is the single HTTP chokepoint: builds the request, sets auth, executes,
|
||||||
// maps non-2xx to APIError, and decodes the data envelope.
|
// maps non-2xx to APIError, and decodes the data envelope.
|
||||||
func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader, contentType string, out any) error {
|
func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader, contentType string, out any) error {
|
||||||
|
// R-812 option A: every non-GET call may write /etc/pve — it waits while a Proxmox package step restarts pmxcfs
|
||||||
|
// (pvegate). Pinned by TestPVEGate_ClientWriteWaitsGetDoesNot.
|
||||||
|
if method != http.MethodGet {
|
||||||
|
release, _, gerr := pvegate.Write(ctx)
|
||||||
|
if gerr != nil {
|
||||||
|
return fmt.Errorf("proxmox: %s %s held back by a Proxmox package step: %w", method, path, gerr)
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
}
|
||||||
req, err := http.NewRequestWithContext(ctx, method, c.base+path, body)
|
req, err := http.NewRequestWithContext(ctx, method, c.base+path, body)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("proxmox: building request: %w", err)
|
return fmt.Errorf("proxmox: building request: %w", err)
|
||||||
|
|||||||
@@ -4,8 +4,10 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||||
"io"
|
"io"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
"strconv"
|
"strconv"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -59,6 +61,15 @@ func (r *ExecRunner) Run(ctx context.Context, name string, args ...string) ([]by
|
|||||||
// RunStdin is Run with the process stdin fed from stdin (nil = no stdin). The sudo-prefix/mode
|
// RunStdin is Run with the process stdin fed from stdin (nil = no stdin). The sudo-prefix/mode
|
||||||
// handling is identical to Run — kept here so both paths share one place.
|
// handling is identical to Run — kept here so both paths share one place.
|
||||||
func (r *ExecRunner) RunStdin(ctx context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
func (r *ExecRunner) RunStdin(ctx context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
// R-812 option A: a root CLI that writes /etc/pve waits while a Proxmox package step runs (pvegate).
|
||||||
|
// Pinned by TestPVEGate_ExecRunnerPctSetWaits / TestWritesEtcPVE.
|
||||||
|
if WritesEtcPVE(name, args) {
|
||||||
|
release, _, gerr := pvegate.Write(ctx)
|
||||||
|
if gerr != nil {
|
||||||
|
return nil, nil, fmt.Errorf("proxmox: %s held back by a Proxmox package step: %w", name, gerr)
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
}
|
||||||
var cmd *exec.Cmd
|
var cmd *exec.Cmd
|
||||||
if r.Mode == RunnerSudo {
|
if r.Mode == RunnerSudo {
|
||||||
sudo := r.SudoPath
|
sudo := r.SudoPath
|
||||||
@@ -77,6 +88,28 @@ func (r *ExecRunner) RunStdin(ctx context.Context, stdin io.Reader, name string,
|
|||||||
return stdout.b, stderr.b, err
|
return stdout.b, stderr.b, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// WritesEtcPVE reports whether a root command writes /etc/pve: `pct` with a config-changing verb, `pvesm`, `pveum`,
|
||||||
|
// and the PBS storage wrapper's create / reconcile verbs. `pct exec|status|list|config` and every other command do not
|
||||||
|
// (the os-update wrapper itself must never wait on the gate its own step holds). Pinned by TestWritesEtcPVE.
|
||||||
|
func WritesEtcPVE(name string, args []string) bool {
|
||||||
|
base := filepath.Base(name)
|
||||||
|
switch base {
|
||||||
|
case "pvesm", "pveum":
|
||||||
|
return true
|
||||||
|
case "pct":
|
||||||
|
if len(args) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
switch args[0] {
|
||||||
|
case "set", "create", "destroy", "restore", "unlock", "resize", "snapshot", "delsnapshot", "rollback", "move-volume", "start", "stop", "reboot", "shutdown":
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
case "felhom-pbs-apply":
|
||||||
|
return len(args) > 0 && (args[0] == "create" || args[0] == "reconcile")
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// Privileged is the root-CLI backend.
|
// Privileged is the root-CLI backend.
|
||||||
type Privileged struct {
|
type Privileged struct {
|
||||||
runner Runner
|
runner Runner
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
package proxmox
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-812 option A: while a Proxmox package step holds the gate, a non-GET API call waits and a GET does not.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): delete the pvegate.Write block in doBody → this fails with "a PUT reached the API
|
||||||
|
// while the Proxmox step held the gate". Restored. (audits/day-2026-10-07/B/red-pvegate-chokepoints.txt)
|
||||||
|
func TestPVEGate_ClientWriteWaitsGetDoesNot(t *testing.T) {
|
||||||
|
d := &mockDoer{fn: func(*http.Request) (*http.Response, error) { return jsonResp(200, `{"data":null}`), nil }}
|
||||||
|
c := newTestClient(d)
|
||||||
|
end, err := pvegate.Step(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := c.get(context.Background(), "/nodes", nil); err != nil {
|
||||||
|
t.Fatalf("a GET must not wait on the gate: %v", err)
|
||||||
|
}
|
||||||
|
if d.calls != 1 {
|
||||||
|
t.Fatalf("the GET must reach the API, calls=%d", d.calls)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
err = c.postForm(ctx, http.MethodPut, "/nodes/x/lxc/9201/config", nil, nil)
|
||||||
|
if d.calls != 1 {
|
||||||
|
end()
|
||||||
|
t.Fatal("a PUT reached the API while the Proxmox step held the gate")
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
end()
|
||||||
|
t.Fatal("a PUT held back past its deadline must fail")
|
||||||
|
}
|
||||||
|
end()
|
||||||
|
if err := c.postForm(context.Background(), http.MethodPut, "/nodes/x/lxc/9201/config", nil, nil); err != nil || d.calls != 2 {
|
||||||
|
t.Fatalf("after the step the PUT must go through (err=%v calls=%d)", err, d.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A root `pct set` waits on the gate; `pct exec` does not.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): delete the WritesEtcPVE block in RunStdin → this fails with "pct set ran while the
|
||||||
|
// Proxmox step held the gate". Restored.
|
||||||
|
func TestPVEGate_ExecRunnerPctSetWaits(t *testing.T) {
|
||||||
|
r := &ExecRunner{Mode: RunnerDirect}
|
||||||
|
end, err := pvegate.Step(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer end()
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
start := time.Now()
|
||||||
|
_, _, err = r.Run(ctx, "/nonexistent/pct", "set", "9201", "-mp8", "/x")
|
||||||
|
if err == nil || time.Since(start) < 90*time.Millisecond {
|
||||||
|
t.Fatalf("pct set ran while the Proxmox step held the gate (err=%v after %s)", err, time.Since(start))
|
||||||
|
}
|
||||||
|
start = time.Now()
|
||||||
|
_, _, _ = r.Run(context.Background(), "/nonexistent/pct", "exec", "9201", "--", "true")
|
||||||
|
if time.Since(start) > 80*time.Millisecond {
|
||||||
|
t.Fatal("pct exec must not wait on the gate")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWritesEtcPVE(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"pct", []string{"set", "9201", "-mp8", "x"}, true},
|
||||||
|
{"/usr/sbin/pct", []string{"create", "9201"}, true},
|
||||||
|
{"pct", []string{"exec", "9201", "--", "true"}, false},
|
||||||
|
{"pct", []string{"status", "9201"}, false},
|
||||||
|
{"pvesm", []string{"add", "dir", "x"}, true},
|
||||||
|
{"pveum", []string{"acl", "modify"}, true},
|
||||||
|
{"/usr/local/sbin/felhom-pbs-apply", []string{"reconcile"}, true},
|
||||||
|
{"/usr/local/sbin/felhom-pbs-apply", []string{"read"}, false},
|
||||||
|
{"/usr/local/sbin/felhom-os-apply", []string{"--plan", "x"}, false},
|
||||||
|
{"pct", nil, false},
|
||||||
|
} {
|
||||||
|
if got := WritesEtcPVE(c.name, c.args); got != c.want {
|
||||||
|
t.Errorf("WritesEtcPVE(%s %v) = %v, want %v", c.name, c.args, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
// Package pvegate keeps the agent's own writes to /etc/pve out of the way of a Proxmox package step (R-812 option A,
|
||||||
|
// `09` §3 decision 163, `11` §5.10).
|
||||||
|
//
|
||||||
|
// WHY. A `pve` step upgrades pve-cluster / pve-manager / qemu-server / pve-container; their postinst scripts restart
|
||||||
|
// pmxcfs (the FUSE filesystem behind /etc/pve) and the API daemons. A write that lands while pmxcfs restarts fails or,
|
||||||
|
// worse, half-lands (design-R-812 §3 A, "can go wrong"). Backups and restore-tests are already kept out by the
|
||||||
|
// host-wide heavy-op gate; this gate covers everything else the agent writes: every non-GET Proxmox API call
|
||||||
|
// (proxmox.Client.doBody) and every root CLI that writes /etc/pve (proxmox.ExecRunner — `pct set|create|…`, `pvesm`,
|
||||||
|
// `pveum`, `felhom-pbs-apply create|reconcile`).
|
||||||
|
//
|
||||||
|
// THE RULE. Write waits while a step runs (bounded by its own context). Step marks the step and then waits until every
|
||||||
|
// write already in flight has finished; it never waits forever (its context bounds it, and the caller gives up and
|
||||||
|
// does not run the step). One step at a time. Pinned by pvegate_test.go and, at the two chokepoints, by
|
||||||
|
// proxmox TestPVEGate_*.
|
||||||
|
package pvegate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
inFlight int
|
||||||
|
stepping bool
|
||||||
|
stepDone chan struct{}
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrStepRunning is returned by Step when another step already holds the gate.
|
||||||
|
var ErrStepRunning = errors.New("pvegate: a Proxmox package step is already running")
|
||||||
|
|
||||||
|
// Write marks one /etc/pve write in flight, first waiting while a Proxmox package step runs. The returned release must
|
||||||
|
// be called when the write has finished. waited reports how long the write was held back.
|
||||||
|
func Write(ctx context.Context) (release func(), waited time.Duration, err error) {
|
||||||
|
start := time.Now()
|
||||||
|
for {
|
||||||
|
mu.Lock()
|
||||||
|
if !stepping {
|
||||||
|
inFlight++
|
||||||
|
mu.Unlock()
|
||||||
|
var once sync.Once
|
||||||
|
return func() {
|
||||||
|
once.Do(func() {
|
||||||
|
mu.Lock()
|
||||||
|
inFlight--
|
||||||
|
mu.Unlock()
|
||||||
|
})
|
||||||
|
}, time.Since(start), nil
|
||||||
|
}
|
||||||
|
ch := stepDone
|
||||||
|
mu.Unlock()
|
||||||
|
select {
|
||||||
|
case <-ch:
|
||||||
|
case <-ctx.Done():
|
||||||
|
return nil, time.Since(start), ctx.Err()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step marks a Proxmox package step and waits until every /etc/pve write already in flight has finished. On error the
|
||||||
|
// gate is released again and the step must not run. end releases the gate and lets the held-back writes go.
|
||||||
|
func Step(ctx context.Context) (end func(), err error) {
|
||||||
|
mu.Lock()
|
||||||
|
if stepping {
|
||||||
|
mu.Unlock()
|
||||||
|
return nil, ErrStepRunning
|
||||||
|
}
|
||||||
|
stepping = true
|
||||||
|
done := make(chan struct{})
|
||||||
|
stepDone = done
|
||||||
|
mu.Unlock()
|
||||||
|
var once sync.Once
|
||||||
|
end = func() {
|
||||||
|
once.Do(func() {
|
||||||
|
mu.Lock()
|
||||||
|
stepping = false
|
||||||
|
close(done)
|
||||||
|
mu.Unlock()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
for {
|
||||||
|
mu.Lock()
|
||||||
|
n := inFlight
|
||||||
|
mu.Unlock()
|
||||||
|
if n == 0 {
|
||||||
|
return end, nil
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
end()
|
||||||
|
return nil, ctx.Err()
|
||||||
|
case <-time.After(50 * time.Millisecond):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stepping reports whether a Proxmox package step holds the gate (for logs).
|
||||||
|
func Stepping() bool {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
return stepping
|
||||||
|
}
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
package pvegate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A write that starts while a step runs waits until the step ends.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): make Write ignore `stepping` → this fails with "the write went through while the
|
||||||
|
// Proxmox step held the gate". Restored. (audits/day-2026-10-07/B/red-pvegate.txt)
|
||||||
|
func TestWrite_WaitsWhileAStepRuns(t *testing.T) {
|
||||||
|
end, err := Step(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got := make(chan time.Time, 1)
|
||||||
|
go func() {
|
||||||
|
rel, _, err := Write(context.Background())
|
||||||
|
if err == nil {
|
||||||
|
rel()
|
||||||
|
}
|
||||||
|
got <- time.Now()
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-got:
|
||||||
|
end()
|
||||||
|
t.Fatal("the write went through while the Proxmox step held the gate")
|
||||||
|
case <-time.After(150 * time.Millisecond):
|
||||||
|
}
|
||||||
|
ended := time.Now()
|
||||||
|
end()
|
||||||
|
select {
|
||||||
|
case at := <-got:
|
||||||
|
if at.Before(ended) {
|
||||||
|
t.Fatal("the write finished before the step ended")
|
||||||
|
}
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("the write never went through after the step ended")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A step waits for a write already in flight before it starts.
|
||||||
|
func TestStep_WaitsForAWriteInFlight(t *testing.T) {
|
||||||
|
rel, _, err := Write(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
started := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
end, err := Step(context.Background())
|
||||||
|
if err == nil {
|
||||||
|
close(started)
|
||||||
|
end()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-started:
|
||||||
|
rel()
|
||||||
|
t.Fatal("the step started while a write was in flight")
|
||||||
|
case <-time.After(150 * time.Millisecond):
|
||||||
|
}
|
||||||
|
rel()
|
||||||
|
select {
|
||||||
|
case <-started:
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("the step never started after the write finished")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A step that cannot drain the writes in time gives up and releases the gate (it never waits forever).
|
||||||
|
func TestStep_GivesUpAndReleases(t *testing.T) {
|
||||||
|
rel, _, _ := Write(context.Background())
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
if _, err := Step(ctx); err == nil {
|
||||||
|
t.Fatal("the step must give up while a write is in flight past its deadline")
|
||||||
|
}
|
||||||
|
if Stepping() {
|
||||||
|
t.Fatal("a step that gave up must release the gate")
|
||||||
|
}
|
||||||
|
rel()
|
||||||
|
}
|
||||||
|
|
||||||
|
// A write held back past its own deadline returns the context's error.
|
||||||
|
func TestWrite_HonoursItsContext(t *testing.T) {
|
||||||
|
end, _ := Step(context.Background())
|
||||||
|
defer end()
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
if _, _, err := Write(ctx); err == nil {
|
||||||
|
t.Fatal("a write held back past its deadline must fail")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// One step at a time.
|
||||||
|
func TestStep_OneAtATime(t *testing.T) {
|
||||||
|
end, _ := Step(context.Background())
|
||||||
|
defer end()
|
||||||
|
if _, err := Step(context.Background()); err != ErrStepRunning {
|
||||||
|
t.Fatalf("a second step must be refused, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -52,6 +52,15 @@ const (
|
|||||||
// (signed, operational key) like agent_update; the root wrapper re-verifies the same signature itself.
|
// (signed, operational key) like agent_update; the root wrapper re-verifies the same signature itself.
|
||||||
ClassOSDockerStep OpClass = "os_docker_step"
|
ClassOSDockerStep OpClass = "os_docker_step"
|
||||||
|
|
||||||
|
// A Proxmox package step on the host (R-812 option A, `11` §5.10) — ring 1. Destructive-class (signed, operational
|
||||||
|
// key) like os_docker_step; the root wrapper re-verifies the same signature itself.
|
||||||
|
ClassOSPVEStep OpClass = "os_pve_step"
|
||||||
|
|
||||||
|
// A kernel step on the host (R-836, `09` §3 decision 172, `11` §5.11) — ring 1: it STAGES a kernel (install + the
|
||||||
|
// one-shot flag; the night leg reboots it). Destructive-class (signed, operational key) like os_pve_step; the root
|
||||||
|
// wrapper re-verifies the same signature itself.
|
||||||
|
ClassOSKernelStep OpClass = "os_kernel_step"
|
||||||
|
|
||||||
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
|
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
|
||||||
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
|
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
|
||||||
ClassAgentConfigUpdate OpClass = "agent_config_update"
|
ClassAgentConfigUpdate OpClass = "agent_config_update"
|
||||||
@@ -123,7 +132,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
|
|||||||
return Destructive
|
return Destructive
|
||||||
case ClassKeyRotation:
|
case ClassKeyRotation:
|
||||||
return Destructive
|
return Destructive
|
||||||
case ClassAgentUpdate, ClassOSDockerStep, ClassAgentConfigUpdate:
|
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate:
|
||||||
// Never benign — no agent-internal provenance can make replacing the agent binary
|
// Never benign — no agent-internal provenance can make replacing the agent binary
|
||||||
// unsigned-safe (a compromised process must not be able to self-bless an update).
|
// unsigned-safe (a compromised process must not be able to self-bless an update).
|
||||||
return Destructive
|
return Destructive
|
||||||
|
|||||||
Reference in New Issue
Block a user