Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
db01ae0f2a | ||
|
|
54ac2c2ac9 | ||
|
|
ec7e699fd7 | ||
|
|
61feb95c06 | ||
|
|
c0bf97de80 | ||
|
|
d5ce5c364a | ||
|
|
a1135018c7 | ||
|
|
ede5d0e6c7 | ||
|
|
386a0c4cf4 | ||
|
|
a9ab47ec71 | ||
|
|
2706e35198 |
No files matched your search
+167
@@ -1,3 +1,170 @@
|
||||
## Unreleased (2026-11-15) — a cut write of `agent.json` cannot leave a blind box (R-1380; found before the release)
|
||||
|
||||
- **`config.ReplaceInPlace` + `config.RecoverCutWrite`** (`internal/config/cutwrite.go`). `agent.json` is the agent's
|
||||
file in a root-owned folder, so the trust move, the PBS seed and the backup-target move write it in place
|
||||
(truncate, write). A power cut in between left an empty or half file and an agent that does not start. Now the old
|
||||
bytes go into a journal in the state folder first (`agent.json.write-in-flight`, durable), the config is written,
|
||||
synced and read back, and the journal is removed; the daemon's start repairs a config that does not parse from the
|
||||
journal BEFORE it reads it. A selftest never repairs. No journal folder → no write. Tests: a cut staged at each of
|
||||
four steps, the wiring in `main`, and a source pin that the three writers use the helper and open no file by hand.
|
||||
Red-proofs: felhom.eu `documentation/audits/release-2b-3-2026-11-15/part0/`. Not proven: a real power cut.
|
||||
|
||||
## Unreleased (2026-11-14 night) — release 3 („the hub tells you”) and the clock-step fix, the agent's parts, built and NOT released
|
||||
|
||||
Commit only. Each part can be released alone; none needs release 2b. No `configs/` change in this section.
|
||||
|
||||
- **Fixes after an independent review of the night's code (2026-11-14; felhom.eu
|
||||
`documentation/audits/night-build-2026-11-14/review/`).** (1) `config.FileMu`: the three functions that read
|
||||
`agent.json` whole and write it whole (the trust move, the PBS seed, the backup-target move) take one lock — two of
|
||||
them could interleave and the second put back the document it read before the first one's change; pinned from the
|
||||
source by `TestAgentJSONWritersTakeTheSharedLock`. (2) `pvetrust.Manager.Ensure` runs one at a time (start, timer,
|
||||
a failed collect and a bundle install could overlap, and the second parked the already-moved file as the „before”
|
||||
copy), and its root step has a 90-second deadline (it runs on the start path and inside the report loop). (3) **The
|
||||
CA copy keeps its previous certificate behind the new one** (`configs/felhom-os-apply`, mode `pve-ca-copy`): the
|
||||
copy was replaced before anything could be proven with the new root, and Proxmox serves its old certificate until
|
||||
pveproxy is reloaded — an agent that restarted in that window had no root that verified it. **Only for the
|
||||
changeover:** at every later run the step makes one handshake to the local Proxmox with the NEW root alone, and
|
||||
drops the previous root as soon as that verifies, or when the previous root is no longer a CA that has not ended.
|
||||
The copy is compared exactly, so nothing the step did not check stays in it. To drop the old root at once by hand
|
||||
(a root changed because it leaked): delete the copy. `configs/` changes again, so the release needs both signed jobs.
|
||||
(4) The profile-refresh stanza is built from the wrapper's ANSWER (its record of the last refresh comes back at
|
||||
every ask); the record file is a bonus — its folder is root-only on a box where a signed job made it first (read in
|
||||
source, not on a box). `rewritten` is absent when not known. (5) A watchdog give-up is forgotten when the guest is
|
||||
no longer the watchdog's (onboot off, locked, destroyed) — it was reported for ever. (6) `/disks` sends both boot
|
||||
tokens from ONE read of the guest's init (`GuestBootTokens`). 13 mutations seen red, two after a mutant that did
|
||||
not compile was done again.
|
||||
- **The hub is told what the kernel clean-up did and what the profile refresh did (R-1383; felhom.eu `11` §5.11.1,
|
||||
§5.12).** (1) The clean-up's report carries `kernel_cleanup` (`osupdate.CleanupDetail`: removed, kept, bytes_gained,
|
||||
leftovers). It still has no `kernel` view and `release_id` stays `kernel-cleanup` — the hub reads only those two as
|
||||
„the kernel this step was for” (the test asserts the wire form). (2) The host report carries `profile_refresh`
|
||||
(`osupdate.ProfileRefreshStatus`, from `Leg.ProfileRefreshStanza`, wired in `main.go`): the root wrapper's own record
|
||||
of the last refresh — so it outlives an agent restart — and how this process's last ask ended. One small file read,
|
||||
no wrapper call, names checked and capped. No OS report is sent for the refresh (an unknown layer reads as a guest
|
||||
step on the hub). Tests `TestKernelCleanup_NightOrderOnceAndNoNames` (extended), `TestProfileRefreshStanza_RecordAndLastAsk`,
|
||||
`TestMainWiresPVETrust`; 9 mutations seen red (one mutant did not compile at first and was done again).
|
||||
- **A boot token that a clock step cannot move (R-1395).** `/disks` answers `guest_boot_key` =
|
||||
`<kernel boot_id>-<guest init start>` beside the unchanged `guest_boot_id` (`<boot TIME>-<guest init start>`). The
|
||||
kernel computes the boot time from the wall clock, so a step of the host's clock moved the old token and the
|
||||
controller recreated every drive-backed app (the N100, 2026-11-14, +32.7 s). `GuestBinder.GuestBootKey`; `""` when
|
||||
the guest does not run or the boot_id is unreadable. Only a controller that knows the key (built there, not
|
||||
released) reads it. Tests `TestGuestBootKey_AClockStepDoesNotMoveIt`, `TestDisks_CarriesBothBootTokens`; 6 mutations
|
||||
seen red (felhom.eu `documentation/audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt`; two mutants did not
|
||||
compile at first and were done again).
|
||||
- **A refused signed job and a watchdog give-up reach the hub (R-1383, D78, felhom.eu `09` §3 decision 299; `05`
|
||||
§16.6).** (1) `signedjobs.Runner` clears a job with `hub.Client.CompleteJobWithOutcome`: `outcome` (done · rejected
|
||||
· failed · malformed), a `reason` from the closed `signedjobs.ReasonCodes` (the two clock reasons `expired` and
|
||||
`not_yet_valid` first — a box with a wrong clock refuses every job) and the op's name, in the `DELETE`'s query. No
|
||||
signature, nonce or job parameter is sent. (2) The guest-power watchdog's give-up — one ERROR line in the journal
|
||||
until now — rides the host report as `guest_power_gave_up` (`localapi.Server.GuestPowerGaveUpStanza`, wired in
|
||||
`main.go`); the list empties when the guest runs. Tests `TestRunner_TellsTheHubHowEachJobEnded` (the real gate
|
||||
and verifier), `TestReasonCode_ClosedList`, `TestCompleteJob_OutcomeRidesTheQuery`,
|
||||
`TestGuestPower_GiveUpIsReportedUntilTheGuestRuns`, `TestMainWiresPVETrust`; 12 mutations seen red (felhom.eu
|
||||
`documentation/audits/night-build-2026-11-14/B/redproof-agent-{job-outcome,guest-power}.txt`).
|
||||
- **The report carries the box's clock at the moment of sending (R-1384, D79, felhom.eu `09` §3 decision 300).**
|
||||
`HostReport.SentAt` (`sent_at`, RFC3339 with nanoseconds) is stamped by `Client.Report` at every send, never by the
|
||||
collector. `reported_at` could not serve: it is stamped when the collect starts. The hub (its own unreleased part)
|
||||
compares it with its clock and alarms above one minute. Test `TestReport_StampsSentAtAtSending`; 3 mutations seen
|
||||
red (felhom.eu `documentation/audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt`).
|
||||
|
||||
## Unreleased (2026-11-14 night) — release 2b, built and NOT released: trust in Proxmox by a copy of the box's own root CA (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a)
|
||||
|
||||
Commit only. No box ran any of this; the proof on a box (demo-hp, a forced new certificate, twice) is the next attended
|
||||
session's. `configs/` changes, so the release needs BOTH signed jobs.
|
||||
|
||||
- **The root step that keeps the copy** (`configs/felhom-os-apply`, host-layer mode `pve-ca-copy`). It copies exactly
|
||||
`/etc/pve/pve-root-ca.pem` to exactly `/etc/felhom-agent/pve-root-ca.pem` (0644 root), only if the source is ONE
|
||||
certificate with `CA:TRUE` that has not ended; written whole or not at all; „same" when nothing changed. The plan may
|
||||
name nothing (T1); no source (T2); not one CA certificate (T3). No appliance check: the copy is the agent's own trust
|
||||
file and changes nothing of a BYO owner's host. Tests `PVECACopy` (9, one with the real openssl through the real
|
||||
runner); 7 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-wrapper.txt`).
|
||||
- **The move of a box that exists, in a safe order** (`internal/pvetrust`, new; wired in `cmd/felhom-agent/main.go`
|
||||
at the agent's start, after a config bundle install, on a timer — hourly until moved, daily after — and on every
|
||||
call to Proxmox that fails on trust). (1) the root step makes the copy; (2) ONE real call to Proxmox with the copy
|
||||
ALONE, while the pinned fingerprint is still the configured trust; (3) only then ONE write of `agent.json` sets
|
||||
`proxmox.tls.ca_file` and clears `fingerprint` (the agent's own file, rewritten in place after a recovery copy
|
||||
`agent.json.pre-pve-trust` was parked, read back and checked); (4) the RUNNING client takes the new trust
|
||||
(`proxmox.Client.SetTLS`, new) — no restart. A failed copy, proof or write changes nothing: the fingerprint stays.
|
||||
**`proxmox.tls` keeps its order (a set fingerprint wins over a CA file), so the fingerprint is cleared LAST.** An
|
||||
operator's own CA file, a box with checks switched off and a config that is not a file are never moved.
|
||||
- **A pinned box whose certificate changed heals by itself:** the failed call starts the same order, the box moves to
|
||||
the CA copy and the next call works. A refreshed copy (a renewed root CA) is proven and reloaded by the running
|
||||
client.
|
||||
- **The cause is named** (`proxmox.TrustProblem`, new): the pinned certificate changed; the CA copy did not sign the
|
||||
served certificate; a certificate ended; the copy is unreadable; the copy ends within 30 days; the root step gave no
|
||||
copy; the proof failed. An ERROR line in the journal, and the new `proxmox_trust` stanza of the host report
|
||||
(`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`). **The hub does not read the stanza yet**
|
||||
(felhom.eu R-1396), and a box that is fully blind sends no host report at all — that case still shows only as
|
||||
„host stale" on the hub.
|
||||
- Tests: `internal/pvetrust` (11, against a real TLS server that changes its certificate and renews its CA), the AST
|
||||
wiring test `TestMainWiresPVETrust`; 12 + 3 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-agent.txt`;
|
||||
one mutation is not a rule and says so).
|
||||
|
||||
## v0.174.0 (2026-11-14) — the box removes its old kernels (R-1368); security profiles are refreshed after a kernel change (R-1388, R-1301); Go 1.27.4 (R-1391) — RELEASED 2026-11-14 (release-agent.sh from the clean working clone at `cc63dd5`, CI run id 2641; agent sha256 6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92, bundle sha256 7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f) and VOUCHED 2026-11-14 15:21:45 CET
|
||||
|
||||
Release 2a of the build order (felhom.eu `09` §3 decision 306). Design: felhom.eu `11-os-updates.md` §5.11.1 and §5.12.
|
||||
`configs/` changed (the wrapper), so a box needs BOTH signed jobs: `agent_update`, then `agent_config_update`.
|
||||
|
||||
**Part A — old kernels (`09` §3 decision 286).** Nothing removed old kernels; a ring-0 box got about one a week, 1 GB
|
||||
each. Now:
|
||||
|
||||
- **The wrapper** (`configs/felhom-os-apply`, layer `kernel`, new mode `kernel-cleanup`): the hand-proven rule of
|
||||
2026-11-10 as a program. Keep the running kernel, the step's `from`, `to` and `default_before`, both defaults, the
|
||||
flag's kernel, an operator's boot pin and the newest installed; remove every other installed kernel image in ONE
|
||||
`apt-get purge` by exact name. Refusals P1, P2, P3, K1, N1, S1, S2 (the design's table), C1 (the plan names
|
||||
something — it may carry only `dry_run`), R9, R12. „Not run" (exit 0): no root-owned ring-0 mark, no kernel step yet,
|
||||
the newest step did not end `good`, the box does not run its newest kernel, a step within the last 20 hours. K1 reads
|
||||
the box a second time, by itself. Repair-first. After the purge it proves the default, the pinned file, the two
|
||||
generators and the state file unchanged; a left-over folder is reported, never deleted. **Rule R4 and every update
|
||||
plan are unchanged; `apt-get autoremove` is not used.**
|
||||
- **The agent** (`internal/osupdate/kernel.go`): the clean-up is the night leg's last call, after a healthy host step,
|
||||
never in a night with a kernel step, never in a debug pass, never on a BYO host. The hub hears `cleaned`, `refused`
|
||||
or `failed` (a kernel-layer report with NO kernel version — the hub would read that as a kernel step); „nothing to
|
||||
remove" and „not run" are log lines. `--selftest=kernel-cleanup-dry-run` prints what it would do and changes nothing.
|
||||
- **No new sudo line.** The wrapper is reached by the one `--plan` line (pinned by `test_cleanup_needs_no_new_sudo_line`).
|
||||
|
||||
**Part B — security profiles (spike first: felhom.eu `audits/profile-refresh-spike-2026-11-14/`).** After a kernel
|
||||
went back and forth a box can load an old compiled AppArmor profile; on the N100 this stopped the time service.
|
||||
|
||||
- **The wrapper**, new host-layer mode `profile-refresh` (an appliance only; no package change; the plan names
|
||||
nothing, F1): once per kernel the box starts on (a root-owned record), `apparmor_parser --replace --skip-read-cache
|
||||
--write-cache -- /etc/apparmor.d`; then ONE restart for a `failed` service whose own program's cached profile the
|
||||
rewrite changed (at most 3). Measured on demo-hp: 0.5 s, no guest or container touched.
|
||||
- **The agent** (`internal/osupdate/profilerefresh.go`) asks at every start and after a config bundle install, under
|
||||
the heavy-op gate; nothing is sent to the hub.
|
||||
|
||||
**Part C — Go 1.27.4** (GO-2026-6859; hygiene: the agent's one `MaxBytesReader` place does not match the advisory).
|
||||
|
||||
**Tests, each seen red** (felhom.eu `audits/release-2a-2026-11-14/A/redproof*.txt`, `B/redproof*.txt`): 31 wrapper
|
||||
cases for the clean-up, one per acceptance scenario and one per kept-kernel kind (14 mutations, all red); 11 for the
|
||||
profile refresh (10 mutations); 5 + 4 agent tests (3 + 3 mutations). Two existing agent tests list the night's calls
|
||||
exactly and gained the clean-up call (`TestRing0_OneCallPerLayer`, `TestRing1_NoReleaseIsInventory`).
|
||||
|
||||
**Delivered 2026-11-14 to the three test boxes by signed `agent_update` + `agent_config_update` each** (Tester 1,
|
||||
demo-hp, the N100): update committed, bundle installed (written 1 of 31), capability probe 77/77, the running binary
|
||||
reads `go1.27.4`. **On boxes** (felhom.eu `audits/release-2a-2026-11-14/D/`):
|
||||
|
||||
- **Profile refresh, by the product's own path, on all three:** refused by the OLD wrapper at the agent's start
|
||||
(R11, logged), then run right after the bundle install: 0.5 s each, nothing restarted on Tester 1 and demo-hp.
|
||||
**On the N100 it repaired the time service:** the cached chrony profile was rewritten, `chrony.service` (failed
|
||||
since 2026-10-16) was restarted and is active, the clock stepped +32.7 s; offset to DooPlex −32.55 s → +0.18 s. After
|
||||
a restart of Tester 1 the agent answered „not needed" (the same kernel).
|
||||
- **Clean-up:** dry runs through the released agent on all three (demo-hp: nothing to remove; the N100: 6 of 8 will
|
||||
go tonight; Tester 1: „not run", it has no ring-0 mark). Refusals P1, P2 and P3 by dry run on demo-hp against states
|
||||
set and put back. **A real removal on Tester 1**, with the ring-0 mark set for the test on the operator's yes and
|
||||
put back: `proxmox-kernel-7.0.14-20-pve-signed` removed, 1.12 GB gained, 8.5 s, the default, the pinned file, the
|
||||
generators and the state file unchanged, no left-over; the second call removed nothing; one orderly restart came
|
||||
back on the same kernel, healthy, a clean stop for the crash guard. **This real run was started by hand** (the
|
||||
agent user, the one sudo line, the night leg's own plan), not by the night leg.
|
||||
- **Not proven:** the step by the agent at night (the N100's and demo-hp's own night, 2026-11-15); a box whose running
|
||||
kernel is not its newest and a self-revert into the kept kernel (tests only); refusal K1 on a box (it guards a wrong
|
||||
computation; tests only); a restart by the profile refresh of any program but chrony.
|
||||
- **Found:** a clock step makes the CONTROLLER read a new boot (its boot identity is `/proc/stat` btime): on the N100
|
||||
it recreated its three drive-backed apps 9 s after the step (felhom.eu R-1395). The crash guard (kernel `boot_id`)
|
||||
was not fooled.
|
||||
|
||||
**Vouched by the operator 2026-11-14 15:21:45 CET** (the hub log: `Artifact manifest set: agent=0.174.0`, the bundle
|
||||
sha = the release; the hub's System page: „vouched agent: 0.174.0"). `pinned.day0_agent` = 0.174.0 (D50).
|
||||
|
||||
## Unreleased (2026-11-11) — docs only: a repository address holds no credential (R-925, Part C of the clock/end-dates run)
|
||||
|
||||
- `scripts/release-agent.sh` header: the tag push needs `GITEA_GIT_TOKEN` from the secret store (`sec run`). The origin
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-11-14 (day, attended) — v0.174.0 RELEASED and VOUCHED (15:21:45 CET); on the three test boxes (agent + bundle, 77/77).** sha `6efbb978…`, bundle `7d87d058…`. Two new wrapper modes: `kernel-cleanup` (layer kernel; old kernel images by the hand-proven rule; ring-0 mark only; the plan names nothing; the night leg's last call, never with a kernel step) and `profile-refresh` (layer host; AppArmor profiles from a fresh compile once per kernel; one restart for a failed service whose cached profile was stale). Go 1.27.4. The N100's time service was repaired by the product path (+32.7 s step). A real kernel removal ran on Tester 1 by hand with a temporary ring-0 mark (operator's yes). **Next:** read the N100's and demo-hp's night (the clean-up by the agent at night is not proven yet); a ruling for ring-1 boxes; release 2b (trust in Proxmox by the root CA copy, D80). Report: felhom.eu `REPORT-release-2a-2026-11-14.md`.
|
||||
|
||||
> **2026-11-10 (afternoon, attended) — v0.173.0 RELEASED and VOUCHED (16:29:16 CET); on the three test boxes.** sha `752b2727…`, bundle `51ebc1f4…` (`configs/` unchanged since 0.172.1, so no bundle job). Measured: a box cleans its self-update folder at the commit of the update that BRINGS the code (49 → 1, 48 → 1, 33 → 1 entries; about 16 MB stays). The line below („at the first good update AFTER that release") was wrong. A rollback after the clean-up was proven on Tester 1 with a signed update to 0.172.1 and one crash in the dwell. `pinned.day0_agent` = 0.173.0. Open: the `.bak-*` files (R-1368 part 4).
|
||||
|
||||
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
|
||||
|
||||
@@ -1,56 +1,31 @@
|
||||
# REPORT — v0.173.0 released, vouched and delivered to the three test boxes (2026-11-10, attended)
|
||||
# REPORT — v0.174.0 released, vouched and delivered to the three test boxes (2026-11-14, attended)
|
||||
|
||||
**Plain words first.** Each agent update left its downloaded file on the box for ever. Version 0.173.0 removes the old
|
||||
files after an update that ended well. It is released and it runs on the three test boxes. The folders went from
|
||||
49, 48 and 33 files to 1 file each. A rollback still works after the clean-up. The operator vouched it at 16:29 CET.
|
||||
**Plain words first.** A box now removes its old kernels by itself, and it reloads its security profiles after it
|
||||
starts on another kernel. Version 0.174.0 is released and runs on Tester 1, demo-hp and the N100. The operator vouched
|
||||
it at 15:21 CET. The N100's time service runs again; the product repaired it, with no hand step.
|
||||
|
||||
The full report of the run is in `felhom.eu`: `REPORT-release-space-fixes-2026-11-10.md`.
|
||||
The full report of the run is in `felhom.eu`: `REPORT-release-2a-2026-11-14.md`.
|
||||
|
||||
## One claim that was wrong
|
||||
## What a reader must know
|
||||
|
||||
The CHANGELOG entry and the report of the morning said: a box cleans its folder at the first good update AFTER the
|
||||
release. That was wrong. The commit runs in the new binary. So the box cleans the folder at the commit of the update
|
||||
that brings the code. Measured on three boxes; the CHANGELOG entry is corrected.
|
||||
|
||||
## The release
|
||||
1. **The clean-up runs on a box with the root-owned ring-0 mark only.** Tester 1 has no mark, so the program answers
|
||||
„not run" there. For the proof the operator allowed the mark on Tester 1 for some minutes; it is back as before.
|
||||
2. **The real removal on Tester 1 was started by hand**, with the agent's own command line. The night leg cannot be
|
||||
started by day: it follows only the night's own backup (R-899). So „the step by the agent at night" is **not proven
|
||||
yet**; the N100's and demo-hp's night of 2026-11-15 is the proof.
|
||||
3. **The profile refresh was spiked first** on demo-hp (felhom.eu `audits/profile-refresh-spike-2026-11-14/`). Three
|
||||
things it found are built in: a stale copy is told by the cached file changing, not by the kernel's hash; one
|
||||
profile never compiles the same twice; a refresh hides other kernel audit lines for a few seconds.
|
||||
4. **A clock step makes the controller read a new boot** (felhom.eu R-1395, a controller row): on the N100 it
|
||||
recreated three apps once.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Version | 0.173.0, `scripts/release-agent.sh` from a clean clone of `776025c8` (tag `v0.173.0`) |
|
||||
| Agent sha256 | `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea` |
|
||||
| Bundle sha256 | `51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` |
|
||||
| CI | `776025c8`: run 2608, `gates #311`, success (read by commit on the public Actions page) |
|
||||
| Delivery | one signed `agent_update` per box. No bundle job: `configs/` did not change since v0.172.1 |
|
||||
| Vouch | the operator, 2026-11-10 16:29:16 CET. Read from the hub log: `Artifact manifest set: agent=0.173.0`, bundle sha = the release. `pinned.day0_agent` = 0.173.0 (D50) |
|
||||
|
||||
## The folder `/var/lib/felhom-agent/selfupdate`, before and after
|
||||
|
||||
| Box | Before (on 0.172.1) | After the commit of 0.173.0 | Commit (UTC) |
|
||||
|---|---|---|---|
|
||||
| demo-hp | 49 entries, 722,596 KB | 1 entry, 16,028 KB | 14:59:19, `removed=49` |
|
||||
| demo-felhom | 48 entries, 706,968 KB | 1 entry, 16,028 KB | 14:51:50, `removed=48` |
|
||||
| Tester 1 | 33 entries, 497,244 KB | 1 entry, 16,028 KB | 15:00:19, `removed=33` |
|
||||
|
||||
After each commit: `pending.json` absent, `/usr/local/bin/felhom-agent.prev` = 0.172.1, the unit active, no restart.
|
||||
|
||||
## The rollback proof (Tester 1)
|
||||
|
||||
1. A signed `agent_update` to 0.172.1 (the vouched version) was uploaded at 15:01:19 UTC. The box took it at its next
|
||||
report, 15:14:31 UTC. The wrapper installed 0.172.1 and wrote the pending marker.
|
||||
2. At 15:14:40 UTC, inside the 60-second dwell, one `systemctl kill -s SEGV` hit the agent's main process.
|
||||
3. The unit's `OnFailure` ran the wrapper's `rollback`. Journal: `rolled back to previous binary and restarted
|
||||
felhom-agent.service`. 0.173.0 started at 15:14:40 UTC.
|
||||
4. Read at 15:15:56 UTC: agent 0.173.0, active, `pending.json` absent, no failed unit. The folder holds 2 files
|
||||
(32,040 KB): the running version's and the failed 0.172.1. The failed file stays by design, until the next good
|
||||
update.
|
||||
|
||||
This is the path the repo already has. No second version was released for the proof.
|
||||
|
||||
## What stays open
|
||||
|
||||
- `/usr/local/bin/felhom-agent.bak-*` (R-1368 part 4): demo-felhom holds 11 such files, demo-hp 1. The agent cannot
|
||||
reach them.
|
||||
- After a rollback, `/usr/local/bin/felhom-agent.prev` holds the same version as the running binary. That is how the
|
||||
wrapper works; it is not new.
|
||||
|
||||
Evidence: felhom.eu `documentation/audits/release-space-fixes-2026-11-10/agent/`.
|
||||
| Version | 0.174.0, sha256 `6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92`, bundle `7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f`, Go 1.27.4 |
|
||||
| Code commit, CI | `cc63dd5`, run id 2641 success |
|
||||
| Tests | wrapper 191 (42 new), agent `internal/osupdate` green; 30 mutations, each seen red (felhom.eu `audits/release-2a-2026-11-14/A`, `B`) |
|
||||
| Tester 1 | 3 kernels → 2, 1.12 GB gained, 8.5 s; second run: nothing; one restart, same kernel, healthy, clean stop |
|
||||
| demo-hp | nothing to remove; refusals P1, P2, P3 by dry run |
|
||||
| The N100 | time service repaired (offset −32.55 s → +0.18 s); tonight it removes 6 of 8 kernels |
|
||||
| Vouch | the operator, 2026-11-14 15:21:45 CET; `pinned.day0_agent` = 0.174.0 |
|
||||
| Not proven | the step at night by the agent; a fell-back box and a self-revert into the kept kernel on a box; K1 on a box |
|
||||
@@ -62,6 +62,7 @@
|
||||
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
|
||||
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
|
||||
| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
|
||||
| `GuestBinder.GuestBootKey` (R-1395) | internal/localapi/intermediary.go | `GuestBootKey(ctx, vmid) string` | "is this a new boot of the guest?" — `<kernel boot_id>-<guest init start>` | **`GuestBootID` carries `/proc/stat` `btime`, which the kernel computes from the wall clock: a clock STEP moves it.** Use the key for any „same boot?” question; `""` = unreadable. Seams `hostBtimeFn` / `hostBootIDFn` / `procStarttimeFn` |
|
||||
| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
|
||||
| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
|
||||
| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
|
||||
@@ -100,6 +101,8 @@
|
||||
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
|
||||
| `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. |
|
||||
| `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. |
|
||||
| `pvetrust.Manager` (`Ensure`, `OnCallError`, `Status`) + `proxmox.Client.SetTLS` / `proxmox.TrustProblem` + `Leg.PVECACopy` + the wrapper's `pve_ca_copy` | internal/pvetrust/pvetrust.go; internal/proxmox/{client,trust}.go; internal/osupdate/pvecacopy.go; configs/felhom-os-apply (mode `pve-ca-copy`) | `Ensure(ctx, why) (changed bool)`; `TrustProblem(err) string` | R-1380 (felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA; the move from a pinned certificate; the cause of a trust failure | THE ORDER: the copy, a real call with the copy ALONE, then ONE config write that clears the fingerprint LAST (a set fingerprint wins over `ca_file` in `TLSConfig.build`). Never clear the fingerprint anywhere else. The wrapper's plan names nothing (T1). `agent.json` is written by the agent in place, with a parked recovery copy — the same way as the backup-target move. |
|
||||
| `config.ReplaceInPlace` / `config.RecoverCutWrite` (+ `config.FileMu`) | internal/config/cutwrite.go; internal/config/filelock.go | `ReplaceInPlace(path, prev, next) error`; `RecoverCutWrite(path) (note string, err error)` | R-1380 (felhom.eu `03` §12a item 7): the only way to write `agent.json` — the folder is root-owned, so no rename; the old bytes go to a journal in the state folder first, and the daemon repairs a cut write at start, before it reads the config | A new writer takes `FileMu`, calls `ReplaceInPlace`, opens no file by hand, and is added to the two source pins in `cmd/felhom-agent` (`…TakeTheSharedLock`, `…UseTheJournalledWrite`). A package whose tests write the config sets `config.CutWriteDir` in `TestMain` |
|
||||
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// A box whose agent.json was cut in the middle of a write must start (R-1380): main repairs the file from the
|
||||
// journal BEFORE it reads it. The staging is the state a power cut leaves: a durable journal of the old bytes and a
|
||||
// half-written config.
|
||||
func TestMainRecoversACutConfigWrite(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
prev := config.CutWriteDir
|
||||
config.CutWriteDir = filepath.Join(dir, "state")
|
||||
t.Cleanup(func() { config.CutWriteDir = prev })
|
||||
if err := os.MkdirAll(config.CutWriteDir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfgPath := filepath.Join(dir, "agent.json")
|
||||
old := `{"proxmox":{"node":"pve","tls":{"fingerprint":"AA:BB"}}}`
|
||||
if err := os.WriteFile(filepath.Join(config.CutWriteDir, "agent.json.write-in-flight"), []byte(old), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(cfgPath, []byte(`{"proxmox":{"node":"pv`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := config.Load(cfgPath); err == nil {
|
||||
t.Fatal("the staging is wrong: the cut config loads")
|
||||
}
|
||||
note, err := recoverCutConfigWrite(cfgPath)
|
||||
if err != nil || note == "" {
|
||||
t.Fatalf("note=%q err=%v", note, err)
|
||||
}
|
||||
cfg, err := config.Load(cfgPath)
|
||||
if err != nil {
|
||||
t.Fatalf("the agent does not start after the repair: %v", err)
|
||||
}
|
||||
if cfg.Proxmox.TLS.Fingerprint != "AA:BB" {
|
||||
t.Fatalf("the repaired config lost the fingerprint: %+v", cfg.Proxmox.TLS)
|
||||
}
|
||||
}
|
||||
|
||||
// main must call the repair before config.Load, or the repair never runs on the box that needs it (a seam that is
|
||||
// built and not wired). Read from the source: the call's position in main's body.
|
||||
func TestMainCallsTheConfigRepairBeforeItLoadsTheConfig(t *testing.T) {
|
||||
src, err := os.ReadFile("main.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f, err := parser.ParseFile(token.NewFileSet(), "main.go", src, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var repair, load token.Pos
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != "main" || fn.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fn.Body, func(n ast.Node) bool {
|
||||
c, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "recoverCutConfigWrite" && repair == 0 {
|
||||
repair = c.Pos()
|
||||
}
|
||||
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Load" {
|
||||
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" && load == 0 {
|
||||
load = c.Pos()
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if repair == 0 || load == 0 {
|
||||
t.Fatalf("main: repair call found=%v, config.Load found=%v", repair != 0, load != 0)
|
||||
}
|
||||
if repair > load {
|
||||
t.Fatal("main reads the config BEFORE it repairs a cut write — a cut config stops the agent at start")
|
||||
}
|
||||
}
|
||||
|
||||
// Every writer of agent.json goes through config.ReplaceInPlace and none truncates the file itself: a writer that
|
||||
// opens the config with O_TRUNC by hand is the unprotected write again.
|
||||
func TestAgentJSONWritersUseTheJournalledWrite(t *testing.T) {
|
||||
for _, w := range []struct{ file, fn string }{
|
||||
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
|
||||
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
|
||||
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
|
||||
} {
|
||||
src, err := os.ReadFile(w.file)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, w.file, src, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
found := false
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
body := string(src[fset.Position(fn.Body.Pos()).Offset:fset.Position(fn.Body.End()).Offset])
|
||||
calls := 0
|
||||
ast.Inspect(fn.Body, func(n ast.Node) bool {
|
||||
if c, ok := n.(*ast.CallExpr); ok {
|
||||
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "ReplaceInPlace" {
|
||||
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" {
|
||||
calls++
|
||||
}
|
||||
}
|
||||
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "OpenFile" {
|
||||
t.Errorf("%s %s opens a file by hand — the config is written only by config.ReplaceInPlace", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if calls != 1 {
|
||||
t.Errorf("%s %s: %d call(s) to config.ReplaceInPlace, want 1", w.file, w.fn, calls)
|
||||
}
|
||||
if strings.Contains(body, "O_TRUNC") {
|
||||
t.Errorf("%s %s truncates a file itself", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("%s: function %s not found — the pin reads nothing", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Three functions in this process read agent.json whole, change one section and write it whole: the trust move, the
|
||||
// PBS seed and the backup-target move. Without one shared lock two of them interleave and the second puts back the
|
||||
// document it read before the first one's change (found by review, 2026-11-14). Each must take config.FileMu as its
|
||||
// FIRST statement and release it by defer — asserted from the source, because the race itself is not a thing a unit
|
||||
// test can stage reliably.
|
||||
func TestAgentJSONWritersTakeTheSharedLock(t *testing.T) {
|
||||
for _, w := range []struct{ file, fn string }{
|
||||
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
|
||||
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
|
||||
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
|
||||
} {
|
||||
f, err := parser.ParseFile(token.NewFileSet(), w.file, nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", w.file, err)
|
||||
}
|
||||
found := false
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
isLockCall := func(e ast.Expr, name string) bool {
|
||||
c, ok := e.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
sel, ok := c.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != name {
|
||||
return false
|
||||
}
|
||||
inner, ok := sel.X.(*ast.SelectorExpr)
|
||||
if !ok || inner.Sel.Name != "FileMu" {
|
||||
return false
|
||||
}
|
||||
pkg, ok := inner.X.(*ast.Ident)
|
||||
return ok && pkg.Name == "config"
|
||||
}
|
||||
if len(fn.Body.List) < 2 {
|
||||
t.Errorf("%s %s: too short to hold the lock", w.file, w.fn)
|
||||
continue
|
||||
}
|
||||
first, ok1 := fn.Body.List[0].(*ast.ExprStmt)
|
||||
second, ok2 := fn.Body.List[1].(*ast.DeferStmt)
|
||||
if !ok1 || !isLockCall(first.X, "Lock") || !ok2 || !isLockCall(second.Call, "Unlock") {
|
||||
t.Errorf("%s %s: must start with config.FileMu.Lock() and defer config.FileMu.Unlock()", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("%s: function %s not found — a writer of agent.json was renamed; keep it under the shared lock and update this test", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -54,6 +54,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfheal"
|
||||
@@ -216,6 +217,15 @@ func main() {
|
||||
return
|
||||
}
|
||||
|
||||
// Only the daemon repairs: a selftest can run beside a live agent that is in the middle of a write.
|
||||
var cutWriteNote string
|
||||
var cutWriteErr error
|
||||
if selftest.mode == "" {
|
||||
cutWriteNote, cutWriteErr = recoverCutConfigWrite(cfgPath)
|
||||
if cutWriteErr != nil {
|
||||
fmt.Fprintln(os.Stderr, "config repair:", cutWriteErr)
|
||||
}
|
||||
}
|
||||
cfg, err := config.Load(cfgPath)
|
||||
if err != nil {
|
||||
// A missing default config file is fine if env provides the values; only a
|
||||
@@ -229,6 +239,11 @@ func main() {
|
||||
// logRing is the always-DEBUG capture ring (v0.83.0 observability): served by the
|
||||
// local API's GET /debug/logs and the heartbeat log-pull. Selftests ignore it.
|
||||
logger, logRing := applog.New(cfg.LogLevel)
|
||||
if cutWriteErr != nil {
|
||||
logger.Error("config: the last write of the config was cut and could not be repaired", "err", cutWriteErr)
|
||||
} else if cutWriteNote != "" {
|
||||
logger.Warn("config: a write of the config was cut by a stop — repaired at start", "result", cutWriteNote)
|
||||
}
|
||||
|
||||
switch selftest.mode {
|
||||
case "":
|
||||
@@ -988,6 +1003,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
osLeg := newOSLeg(cfg, client, px, logger)
|
||||
collector.SetSystemReporter(&factsReporter{leg: osLeg, guest: firstGuest(px)}) // R-852: the versions
|
||||
desiredSyncer.AddConsumer(osLeg)
|
||||
// R-1380 (D80, felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA. At this start the root
|
||||
// step makes the copy, a REAL call proves it while the pinned fingerprint is still the trust, and only then ONE
|
||||
// write of the config sets ca_file and clears the fingerprint. A call that fails on trust later names the cause and
|
||||
// runs the same order again (a pinned box whose certificate changed moves right then, with no restart).
|
||||
pveTrust := newPVETrust(cfg, px, osLeg, logger)
|
||||
collector.SetProxmoxTrustSource(func() any { return pveTrust.Status() })
|
||||
loop.SetCollectErrorHook(pveTrust.OnCallError)
|
||||
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
|
||||
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
|
||||
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
|
||||
@@ -1018,6 +1040,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// network and stays under the hub's 45-minute host_stale (its
|
||||
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
|
||||
// R-1380: the trust order at this start (the copy, the proof, then the config) — before the first report cycle, so
|
||||
// a pinned box whose certificate changed while the agent was down is not blind for a cycle. Then on a timer.
|
||||
pveTrust.Ensure(ctx, "start")
|
||||
go pveTrust.Run(ctx, time.Hour, 24*time.Hour)
|
||||
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
|
||||
// after start so the guest has settled; a failure is logged and the next start retries.
|
||||
go func() {
|
||||
@@ -1301,9 +1327,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// R-1388: a new wrapper may know the profile refresh the old one refused — ask again (its own goroutine:
|
||||
// it may wait for the heavy-op gate, and the signed job must not wait with it).
|
||||
go osLeg.ProfileRefresh(daemonCtx, "config bundle installed", profileGate, 12, 5*time.Minute)
|
||||
// R-1380: a new wrapper may know the CA-copy step the old one refused — run the trust order again.
|
||||
go pveTrust.Ensure(daemonCtx, "config bundle installed")
|
||||
}}
|
||||
// R-1388 / R-1301 (`11` §5.12): after a start, load the security profiles from a fresh compile when the box runs
|
||||
// another kernel than at the last refresh (the wrapper keeps that record and answers "not needed" otherwise).
|
||||
// R-1383 (release 3): the last refresh, and how this process's last ask ended, ride the host report.
|
||||
collector.SetProfileRefreshSource(osLeg.ProfileRefreshStanza)
|
||||
go osLeg.ProfileRefresh(daemonCtx, "agent start", profileGate, 12, 5*time.Minute)
|
||||
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
||||
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
||||
@@ -1355,6 +1385,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
},
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
// R-1383 (D78): the guest-power watchdog's give-up rides the host report (it was one journal line).
|
||||
collector.SetGuestPowerGaveUpSource(localSrv.GuestPowerGaveUpStanza)
|
||||
}
|
||||
if localSrv != nil {
|
||||
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
|
||||
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
|
||||
@@ -3948,6 +3982,29 @@ func (f *selftestFlag) Set(v string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// newPVETrust wires internal/pvetrust (R-1380): the root step through the OS wrapper, the proof as a throwaway client
|
||||
// with EXACTLY the trust under test, and the running client's SetTLS.
|
||||
func newPVETrust(cfg config.Config, px *proxmox.Client, leg *osupdate.Leg, logger *slog.Logger) *pvetrust.Manager {
|
||||
m := &pvetrust.Manager{ConfigPath: cfg.SourcePath, StateDir: agentStateDir(cfg), Logger: logger,
|
||||
EnsureCopy: func(ctx context.Context) (pvetrust.Copy, error) {
|
||||
r, err := leg.PVECACopy(ctx)
|
||||
return pvetrust.Copy{Path: r.Path, Outcome: r.Outcome, NotAfter: r.NotAfter, DaysLeft: r.DaysLeft}, err
|
||||
},
|
||||
Probe: func(ctx context.Context, t proxmox.TLSConfig) error {
|
||||
p, err := proxmox.NewClient(proxmox.Config{Endpoint: cfg.Proxmox.Endpoint, Node: cfg.Proxmox.Node, Token: cfg.Proxmox.Token, TLS: t})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = p.Version(ctx)
|
||||
return err
|
||||
},
|
||||
Apply: px.SetTLS,
|
||||
}
|
||||
m.Start(proxmox.TLSConfig{CAFile: cfg.Proxmox.TLS.CAFile, Fingerprint: cfg.Proxmox.TLS.Fingerprint,
|
||||
InsecureSkipVerify: cfg.Proxmox.TLS.InsecureSkipVerify})
|
||||
return m
|
||||
}
|
||||
|
||||
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
|
||||
// the once-per-night marker live in the agent's own os/ dir.
|
||||
func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg {
|
||||
@@ -4203,3 +4260,13 @@ func agentStateDir(cfg config.Config) string {
|
||||
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
|
||||
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
|
||||
}
|
||||
|
||||
// recoverCutConfigWrite repairs agent.json BEFORE it is read, when the last write of it was cut by a power loss
|
||||
// (config.RecoverCutWrite; R-1380). It runs before the logger exists, so main logs what it returns.
|
||||
// Pinned by TestMainRecoversACutConfigWrite.
|
||||
func recoverCutConfigWrite(cfgPath string) (string, error) {
|
||||
if cfgPath == "" {
|
||||
return "", nil
|
||||
}
|
||||
return config.RecoverCutWrite(cfgPath)
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1380: the trust move is a component with seams (the root step, the proof, the running client, the report stanza,
|
||||
// the failed-call hook). A unit-green pvetrust that main.go never starts is the inert-seam defect, so the wiring is
|
||||
// asserted from the AST: built once, run at start and after a bundle install, on a timer, fed by a failed collect, and
|
||||
// reported. The same for the profile refresh (R-1388), which had no such test.
|
||||
func TestMainWiresPVETrust(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
seen := map[string]int{}
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.CallExpr:
|
||||
sel, ok := node.Fun.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
x, _ := sel.X.(*ast.Ident)
|
||||
switch {
|
||||
case x != nil && x.Name == "pveTrust" && sel.Sel.Name == "Ensure":
|
||||
seen["ensure"]++
|
||||
case sel.Sel.Name == "SetProxmoxTrustSource":
|
||||
seen["stanza"]++
|
||||
case sel.Sel.Name == "SetProfileRefreshSource" && len(node.Args) == 1:
|
||||
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "ProfileRefreshStanza" {
|
||||
seen["profile refresh → report"]++ // R-1383
|
||||
}
|
||||
case sel.Sel.Name == "SetGuestPowerGaveUpSource" && len(node.Args) == 1:
|
||||
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "GuestPowerGaveUpStanza" {
|
||||
seen["guest-power give-up → report"]++ // R-1383
|
||||
}
|
||||
case sel.Sel.Name == "SetCollectErrorHook" && len(node.Args) == 1:
|
||||
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "OnCallError" {
|
||||
seen["hook"]++
|
||||
}
|
||||
case x != nil && x.Name == "osLeg" && sel.Sel.Name == "ProfileRefresh":
|
||||
seen["profile-refresh"]++
|
||||
}
|
||||
case *ast.GoStmt:
|
||||
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
|
||||
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "pveTrust" {
|
||||
seen["timer"]++
|
||||
}
|
||||
}
|
||||
case *ast.KeyValueExpr:
|
||||
if k, ok := node.Key.(*ast.Ident); ok && k.Name == "Apply" {
|
||||
if v, ok := node.Value.(*ast.SelectorExpr); ok && v.Sel.Name == "SetTLS" {
|
||||
seen["apply=SetTLS"]++
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
want := map[string]int{"ensure": 2, "stanza": 1, "hook": 1, "timer": 1, "apply=SetTLS": 1, "profile-refresh": 2,
|
||||
"guest-power give-up → report": 1, "profile refresh → report": 1}
|
||||
for k, n := range want {
|
||||
if seen[k] < n {
|
||||
t.Errorf("main.go wires %q %d time(s), want at least %d (the two-call ones: at start and after a bundle install)", k, seen[k], n)
|
||||
}
|
||||
}
|
||||
}
|
||||
+113
-1
@@ -161,6 +161,14 @@ KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good",
|
||||
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||
# The clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1). A step that ENDED in one of these phases
|
||||
# and is not "good" keeps every kernel (the box fell back or reverted); any other phase is "not ended" (P1).
|
||||
# Trust in Proxmox by a copy of the box's own root CA (R-1380, `09` §3 decision 303, felhom.eu `03` §12a). The agent's
|
||||
# user cannot read Proxmox's CA file (0640 root:www-data); this root step keeps a readable copy of that ONE public
|
||||
# certificate where the agent's `proxmox.tls.ca_file` points.
|
||||
PVE_CA_SOURCE = "/etc/pve/pve-root-ca.pem"
|
||||
PVE_CA_COPY = "/etc/felhom-agent/pve-root-ca.pem" # 0644 root: a CA certificate is public; the agent only reads it
|
||||
PVE_CA_MAX_BYTES = 64 * 1024
|
||||
PVE_LOCAL_API = "127.0.0.1:8006" # pveproxy on this host (the installer's proof uses the same)
|
||||
PEM_CERT_RE = re.compile(rb"-----BEGIN CERTIFICATE-----\s[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----")
|
||||
# The security-profile refresh (R-1388, R-1301, `11` §5.12; measured on demo-hp 2026-11-14,
|
||||
# felhom.eu audits/profile-refresh-spike-2026-11-14/). AppArmor keeps one cache folder per kernel feature set; a
|
||||
# profile that changed while ANOTHER kernel ran leaves an old compiled copy in this kernel's folder, with a time that
|
||||
@@ -514,7 +522,7 @@ class Apply:
|
||||
def check_plan(self, plan):
|
||||
mode = plan.get("mode", "apply")
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
|
||||
"oom-check", "profile-refresh") \
|
||||
"oom-check", "profile-refresh", "pve-ca-copy") \
|
||||
+ KERNEL_MODES:
|
||||
raise Refused("R11", f"unknown mode {mode!r}")
|
||||
if mode == "agent_update":
|
||||
@@ -529,6 +537,15 @@ class Apply:
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "bundle-policy is a host-layer mode")
|
||||
return mode, "host", 0, "bundle-policy"
|
||||
if mode == "pve-ca-copy":
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "pve-ca-copy is a host-layer mode")
|
||||
# R-1380 (D80): ONE fixed source, ONE fixed place — the plan may name NOTHING (no path, no file, no content)
|
||||
extra = sorted(k for k in plan if k not in ("mode", "layer", "release_id", "run_id", "trigger", "ring"))
|
||||
if extra:
|
||||
raise Refused("T1", f"a pve-ca-copy plan carries {', '.join(extra)} — it takes nothing from a plan: "
|
||||
f"the source is {PVE_CA_SOURCE} and the copy is {PVE_CA_COPY}, both fixed")
|
||||
return mode, "host", 0, "pve-ca-copy"
|
||||
if mode == "profile-refresh":
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "profile-refresh is a host-layer mode")
|
||||
@@ -1124,6 +1141,10 @@ class Apply:
|
||||
return Bundle(self).from_plan(plan)
|
||||
if self.mode == "bundle-policy":
|
||||
return Bundle(self).policy_pass()
|
||||
if self.mode == "pve-ca-copy":
|
||||
# NO appliance check, on purpose (like the bundle): the copy is the agent's OWN trust file in the agent's own
|
||||
# config folder, and it changes nothing of the host's — a BYO host's agent talks to Proxmox the same way.
|
||||
return self.pve_ca_copy()
|
||||
if self.mode == "profile-refresh":
|
||||
self.check_appliance()
|
||||
return self.profile_refresh(plan)
|
||||
@@ -1192,6 +1213,97 @@ class Apply:
|
||||
self.report["oom_check"] = self.oom_check()
|
||||
return 0
|
||||
|
||||
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80) ----------
|
||||
def is_live_ca(self, block):
|
||||
"""One PEM certificate block: is it a CA certificate that has not ended? (openssl, the bytes on stdin)."""
|
||||
try:
|
||||
rc, out, _ = self.r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-checkend", "0"], 30,
|
||||
stdin=block.decode("ascii") + "\n")
|
||||
except (UnicodeDecodeError, subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
return rc == 0 and re.search(r"\bCA:TRUE\b", out) is not None
|
||||
|
||||
def pve_serves_with(self, cafile):
|
||||
"""Does the local Proxmox API serve a certificate that verifies with this CA file ALONE? One handshake to
|
||||
PVE_LOCAL_API; nothing is sent. Any doubt (no answer, a timeout) is "no" — the previous root then stays."""
|
||||
try:
|
||||
rc, out, err = self.r.host(["openssl", "s_client", "-connect", PVE_LOCAL_API, "-CAfile", cafile, "-no-CApath",
|
||||
"-verify_return_error"], 20, stdin="")
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
return rc == 0 and "Verify return code: 0 (ok)" in (out + err)
|
||||
|
||||
def pve_ca_copy(self):
|
||||
"""Copy exactly PVE_CA_SOURCE to exactly PVE_CA_COPY — only if the source is ONE CA certificate that has not
|
||||
ended — written whole or not at all (a root-owned temp file beside the target, fsync, rename). "same" when the
|
||||
copy is exactly the source's bytes. The bytes that are checked are the bytes that are written first; only
|
||||
during a change of the root the copy also holds its previous certificate behind them (see below)."""
|
||||
r = self.r
|
||||
res = {"outcome": None, "path": PVE_CA_COPY}
|
||||
self.report["pve_ca_copy"] = res
|
||||
try:
|
||||
data = r.read_bytes(PVE_CA_SOURCE)
|
||||
except OSError as e:
|
||||
raise Refused("T2", f"{PVE_CA_SOURCE} cannot be read ({e.__class__.__name__}) — is this a Proxmox host?")
|
||||
if not data or len(data) > PVE_CA_MAX_BYTES:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is {len(data)} bytes — not one certificate")
|
||||
certs = PEM_CERT_RE.findall(data)
|
||||
if len(certs) != 1 or b"PRIVATE KEY" in data:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} holds {len(certs)} certificate(s)"
|
||||
+ (" and a private key" if b"PRIVATE KEY" in data else "") + " — exactly one CA certificate is expected")
|
||||
try:
|
||||
text = data.decode("ascii") # PEM is ASCII; the runner hands text to openssl's stdin
|
||||
except UnicodeDecodeError:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is not a PEM text file")
|
||||
rc, out, err = r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-enddate"], 30, stdin=text)
|
||||
if rc != 0:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is not a readable certificate: {(out + err).strip()[-160:]}")
|
||||
if not re.search(r"\bCA:TRUE\b", out):
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is not a CA certificate (no CA:TRUE) — a leaf or any other file is never copied")
|
||||
m = re.search(r"notAfter=(.+)", out)
|
||||
try:
|
||||
import calendar
|
||||
not_after = calendar.timegm(time.strptime(m.group(1).strip(), "%b %d %H:%M:%S %Y GMT")) # UTC, no DST
|
||||
except (AttributeError, ValueError):
|
||||
raise Refused("T3", f"the end date of {PVE_CA_SOURCE} cannot be read")
|
||||
days = int((not_after - r.now()) // 86400)
|
||||
if days < 0:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} ended {-days} day(s) ago — an ended CA is never copied")
|
||||
res.update(sha256=sha256_hex(data), not_after=_iso(not_after), days_left=days)
|
||||
try:
|
||||
have = r.read_bytes(PVE_CA_COPY)
|
||||
except OSError:
|
||||
have = None
|
||||
if have == data:
|
||||
res["outcome"] = "same"
|
||||
return 0
|
||||
# The copy the agent trusts by is replaced BEFORE anything can be proven with the new root (Proxmox serves its
|
||||
# old certificate until pveproxy is reloaded). So at a CHANGE the copy's previous first certificate — written
|
||||
# by this same step — is kept behind the new one, and the agent trusts both roots across the change, also
|
||||
# after a restart. That second root is kept ONLY for the changeover: it is dropped as soon as Proxmox serves a
|
||||
# certificate that verifies with the new root alone, and at once when it is no longer a CA that has not ended.
|
||||
# The copy is compared EXACTLY: nothing this step did not check is ever left in it. (Review, 2026-11-14.)
|
||||
# To drop the old root at once by hand (a root that was changed because it leaked): delete the copy.
|
||||
prev = PEM_CERT_RE.findall(have) if have and b"PRIVATE KEY" not in have else []
|
||||
if len(prev) == 2 and have == data + prev[1] + b"\n":
|
||||
over = self.pve_serves_with(PVE_CA_SOURCE)
|
||||
if not over and self.is_live_ca(prev[1]):
|
||||
res["outcome"], res["kept_previous"] = "same", True # the changeover is still on
|
||||
return 0
|
||||
r.put_file(PVE_CA_COPY, data, 0o644)
|
||||
res["outcome"], res["replaced"], res["kept_previous"], res["dropped_previous"] = "written", True, False, True
|
||||
r.log(f"os-apply: PVE CA COPY the previous root is dropped from {PVE_CA_COPY} "
|
||||
f"({'Proxmox serves a certificate of the new root' if over else 'the previous root is no longer a live CA'})")
|
||||
return 0
|
||||
want, kept = data, False
|
||||
if prev and prev[0].strip() != certs[0].strip() and self.is_live_ca(prev[0]):
|
||||
want, kept = data + prev[0] + b"\n", True
|
||||
r.put_file(PVE_CA_COPY, want, 0o644)
|
||||
res["outcome"], res["replaced"], res["kept_previous"] = "written", have is not None, kept
|
||||
r.log(f"os-apply: PVE CA COPY written {PVE_CA_COPY} sha256={res['sha256'][:16]} ends={res['not_after']} "
|
||||
f"({'replaced an older copy' if have is not None else 'first copy'})")
|
||||
return 0
|
||||
|
||||
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
|
||||
def profile_cache(self):
|
||||
"""{"<folder>/<file>": sha256} of every cached profile, every kernel's folder."""
|
||||
|
||||
@@ -2633,5 +2633,245 @@ class ProfileRefresh(unittest.TestCase):
|
||||
run(f)
|
||||
self.assertFalse([c for c in f.calls if c[1][0] in ("apt-get", "dpkg", "apt")], "no package command at all")
|
||||
|
||||
|
||||
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80, felhom.eu `03` §12a) ----------
|
||||
CA_PEM = b"-----BEGIN CERTIFICATE-----\nQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=\n-----END CERTIFICATE-----\n"
|
||||
CA_PEM_2 = b"-----BEGIN CERTIFICATE-----\nWllYV1ZVVFNSUVBPTk1MS0pJSEdGRURDQkE=\n-----END CERTIFICATE-----\n"
|
||||
|
||||
|
||||
class TFake(Fake):
|
||||
"""A Proxmox host: the root CA file, the copy (or none), and openssl's answer about the bytes it is given."""
|
||||
|
||||
def __init__(self, source=CA_PEM, copy=None, ca=True, ends="Aug 17 15:44:42 2036 GMT"):
|
||||
super().__init__()
|
||||
self.source, self.tree, self.ca, self.ends, self.openssl_rc = source, {}, ca, ends, 0
|
||||
if copy is not None:
|
||||
self.tree[osapply.PVE_CA_COPY] = copy
|
||||
self.openssl_stdin, self.puts = [], []
|
||||
self.serves_new = False # does pveproxy serve a certificate that verifies with the NEW root alone?
|
||||
self.dead_ca = set() # certificate texts openssl says are not a live CA (ended)
|
||||
self.handshakes = []
|
||||
self.plan = {"release_id": "pve-ca", "layer": "host", "mode": "pve-ca-copy"}
|
||||
|
||||
def read_bytes(self, p):
|
||||
if p == osapply.PVE_CA_SOURCE:
|
||||
if self.source is None:
|
||||
raise FileNotFoundError(p)
|
||||
return self.source
|
||||
if p in self.tree:
|
||||
return self.tree[p]
|
||||
raise FileNotFoundError(p)
|
||||
|
||||
def put_file(self, path, data, mode):
|
||||
self.puts.append((path, data, mode))
|
||||
self.tree[path] = data
|
||||
|
||||
def host(self, argv, timeout=600, stdin=None):
|
||||
if argv[0] == "openssl" and argv[1] == "s_client":
|
||||
self.handshakes.append(argv)
|
||||
return (0, "Verify return code: 0 (ok)\n", "") if self.serves_new else (1, "", "Verify return code: 19 (self-signed certificate in certificate chain)\n")
|
||||
if argv[0] == "openssl" and "-checkend" in argv:
|
||||
return (1, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will expire\n", "") if stdin.strip() in self.dead_ca else \
|
||||
(0, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will not expire\n", "")
|
||||
if argv[0] == "openssl":
|
||||
self.calls.append(("host", argv))
|
||||
self.openssl_stdin.append(stdin)
|
||||
if self.openssl_rc:
|
||||
return self.openssl_rc, "", "Could not read certificate from <stdin>"
|
||||
return 0, ("X509v3 Basic Constraints: critical\n CA:%s\nnotAfter=%s\n" % ("TRUE" if self.ca else "FALSE", self.ends)), ""
|
||||
return super().host(argv, timeout, stdin)
|
||||
|
||||
|
||||
class PVECACopy(unittest.TestCase):
|
||||
"""Red-proofs: felhom.eu audits/night-build-2026-11-14/A/redproof-wrapper.txt."""
|
||||
|
||||
def refused(self, f, code):
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, code), rep)
|
||||
self.assertEqual(f.puts, [], "nothing is written on a refusal")
|
||||
return rep
|
||||
|
||||
def test_a_new_box_gets_the_copy(self):
|
||||
f = TFake()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.puts, [(osapply.PVE_CA_COPY, CA_PEM, 0o644)], "ONE file, the fixed place, the source's bytes, 0644")
|
||||
self.assertEqual(f.openssl_stdin, [CA_PEM.decode()], "the bytes that are checked are the bytes that are written")
|
||||
c = rep["pve_ca_copy"]
|
||||
self.assertEqual((c["outcome"], c["path"], c["replaced"]), ("written", "/etc/felhom-agent/pve-root-ca.pem", False))
|
||||
self.assertEqual(c["not_after"], "2036-08-17T15:44:42Z")
|
||||
self.assertGreater(c["days_left"], 3000)
|
||||
|
||||
def test_the_same_copy_is_not_written_again(self):
|
||||
f = TFake(copy=CA_PEM)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
|
||||
|
||||
def test_a_renewed_ca_refreshes_the_copy(self):
|
||||
f = TFake(source=CA_PEM_2, copy=CA_PEM)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["replaced"]), (0, "written", True), rep)
|
||||
# the new certificate FIRST, and the copy's previous one kept behind it: Proxmox serves its old certificate until
|
||||
# pveproxy is reloaded, and an agent that restarts in that window must still have a root that verifies it
|
||||
bundle = f.tree[osapply.PVE_CA_COPY]
|
||||
self.assertEqual(bundle, CA_PEM_2 + osapply.PEM_CERT_RE.findall(CA_PEM)[0] + b"\n")
|
||||
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
|
||||
self.assertEqual(f.openssl_stdin, [CA_PEM_2.decode()], "the SOURCE is checked in full; the kept one by its own check")
|
||||
|
||||
def test_the_previous_root_stays_only_for_the_changeover(self):
|
||||
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
|
||||
bundle = CA_PEM_2 + old + b"\n"
|
||||
# Proxmox still serves the OLD certificate: nothing is written, the old root stays
|
||||
f = TFake(source=CA_PEM_2, copy=bundle)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
|
||||
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
|
||||
self.assertEqual(len(f.handshakes), 1)
|
||||
self.assertIn(osapply.PVE_CA_SOURCE, f.handshakes[0], "the handshake is verified with the NEW root alone (the source file)")
|
||||
self.assertIn("-no-CApath", f.handshakes[0])
|
||||
self.assertIn("-verify_return_error", f.handshakes[0])
|
||||
# pveproxy was reloaded and serves a certificate of the NEW root: the old root is dropped
|
||||
f = TFake(source=CA_PEM_2, copy=bundle)
|
||||
f.serves_new = True
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2, "after the changeover the copy is the new root alone")
|
||||
self.assertEqual((rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["dropped_previous"], rep["pve_ca_copy"]["kept_previous"]), ("written", True, False))
|
||||
# the old root has ENDED while Proxmox still serves the old certificate: it verifies nothing, so it is dropped
|
||||
f = TFake(source=CA_PEM_2, copy=bundle)
|
||||
f.dead_ca = {old.decode()}
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
|
||||
# an ended previous root is not kept at the change itself either
|
||||
f = TFake(source=CA_PEM_2, copy=CA_PEM)
|
||||
f.dead_ca = {old.decode()}
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
|
||||
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
|
||||
|
||||
def test_a_copy_with_anything_unchecked_in_it_is_rewritten(self):
|
||||
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
|
||||
for name, copy in (("text after the source", CA_PEM_2 + b"anything at all\n"),
|
||||
("a third certificate", CA_PEM_2 + old + b"\n" + old + b"\n"),
|
||||
("a key in the file", CA_PEM_2 + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"),
|
||||
("the two in the wrong order", CA_PEM + osapply.PEM_CERT_RE.findall(CA_PEM_2)[0] + b"\n")):
|
||||
f = TFake(source=CA_PEM_2, copy=copy)
|
||||
f.serves_new = True
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, (name, rep))
|
||||
got = f.tree[osapply.PVE_CA_COPY]
|
||||
self.assertTrue(got.startswith(CA_PEM_2), name)
|
||||
self.assertNotIn(b"anything", got, name)
|
||||
self.assertNotIn(b"PRIVATE KEY", got, name)
|
||||
self.assertLessEqual(len(osapply.PEM_CERT_RE.findall(got)), 2, name)
|
||||
# a first copy keeps nothing behind it, and makes no handshake
|
||||
f4 = TFake()
|
||||
rc, rep = run(f4)
|
||||
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
|
||||
self.assertEqual((f4.tree[osapply.PVE_CA_COPY], f4.handshakes), (CA_PEM, []))
|
||||
|
||||
def test_a_source_that_is_not_a_ca_certificate_is_refused(self):
|
||||
self.refused(TFake(ca=False), "T3") # a leaf certificate
|
||||
self.refused(TFake(source=b"hello\n"), "T3") # not a certificate at all
|
||||
self.refused(TFake(source=CA_PEM + CA_PEM_2), "T3") # two certificates (a chain, a bundle)
|
||||
self.refused(TFake(source=CA_PEM + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"), "T3")
|
||||
self.refused(TFake(source=b""), "T3")
|
||||
self.refused(TFake(source=CA_PEM + b"x" * (osapply.PVE_CA_MAX_BYTES + 1)), "T3")
|
||||
f = TFake()
|
||||
f.openssl_rc = 1 # openssl cannot parse it
|
||||
self.refused(f, "T3")
|
||||
self.refused(TFake(ends="Aug 17 15:44:42 2020 GMT"), "T3") # an ended CA
|
||||
self.refused(TFake(ends="garbage"), "T3")
|
||||
|
||||
def test_no_source_is_refused(self):
|
||||
self.refused(TFake(source=None), "T2")
|
||||
|
||||
def test_with_the_real_openssl_and_the_real_runner(self):
|
||||
"""NOT a fake: a CA and a leaf made by openssl in a temp folder, judged through Runner.host as on a box (the
|
||||
fake cannot show a wrong argument or a bytes/text mix-up). Skipped where openssl is missing."""
|
||||
import shutil, tempfile
|
||||
if not shutil.which("openssl"):
|
||||
self.skipTest("no openssl here")
|
||||
d = tempfile.mkdtemp()
|
||||
try:
|
||||
def sh(*a):
|
||||
subprocess.run(a, check=True, capture_output=True, cwd=d)
|
||||
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "ca.key", "-out", "ca.pem", "-days", "30",
|
||||
"-subj", "/CN=test CA", "-addext", "basicConstraints=critical,CA:TRUE")
|
||||
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "leaf.key", "-out", "leaf.pem", "-days", "30",
|
||||
"-subj", "/CN=leaf", "-addext", "basicConstraints=CA:FALSE")
|
||||
real = osapply.Runner()
|
||||
for name, want in (("ca.pem", "written"), ("leaf.pem", "T3")):
|
||||
data = open(os.path.join(d, name), "rb").read()
|
||||
f = TFake(source=data)
|
||||
f.host = lambda argv, timeout=600, stdin=None: real.host(argv, timeout, stdin) if argv[0] == "openssl" else Fake.host(f, argv, timeout, stdin)
|
||||
f.clock = __import__("time").time()
|
||||
rc, rep = run(f)
|
||||
got = rep["pve_ca_copy"]["outcome"] if rc == 0 else rep["refused"]["code"]
|
||||
self.assertEqual(got, want, f"{name}: {rep}")
|
||||
if want == "written":
|
||||
self.assertIn(rep["pve_ca_copy"]["days_left"], (29, 30))
|
||||
# the two checks the changeover uses, with the REAL openssl and a REAL TLS server
|
||||
ended = True
|
||||
try: # an ENDED CA needs openssl 3.4+ (-not_after); older: that one check is not made
|
||||
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "old.key", "-out", "old.pem",
|
||||
"-not_before", "20190101000000Z", "-not_after", "20200101000000Z",
|
||||
"-subj", "/CN=ended CA", "-addext", "basicConstraints=critical,CA:TRUE")
|
||||
except subprocess.CalledProcessError:
|
||||
ended = False
|
||||
sh("openssl", "req", "-newkey", "rsa:2048", "-nodes", "-keyout", "srv.key", "-out", "srv.csr", "-subj", "/CN=pve")
|
||||
sh("openssl", "x509", "-req", "-in", "srv.csr", "-CA", "ca.pem", "-CAkey", "ca.key", "-CAcreateserial", "-out", "srv.pem", "-days", "5")
|
||||
app = osapply.Apply.__new__(osapply.Apply)
|
||||
app.r = real
|
||||
block = lambda n: osapply.PEM_CERT_RE.findall(open(os.path.join(d, n), "rb").read())[0]
|
||||
self.assertTrue(app.is_live_ca(block("ca.pem")), "a CA that has not ended")
|
||||
self.assertFalse(app.is_live_ca(block("leaf.pem")), "a leaf is not a CA")
|
||||
if ended:
|
||||
self.assertFalse(app.is_live_ca(block("old.pem")), "an ended CA is not a live one")
|
||||
import socket
|
||||
s0 = socket.socket(); s0.bind(("127.0.0.1", 0)); port = s0.getsockname()[1]; s0.close()
|
||||
srv = subprocess.Popen(["openssl", "s_server", "-accept", f"127.0.0.1:{port}", "-cert", "srv.pem", "-key", "srv.key", "-quiet"],
|
||||
cwd=d, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
saved, osapply.PVE_LOCAL_API = osapply.PVE_LOCAL_API, f"127.0.0.1:{port}"
|
||||
ok = False
|
||||
for _ in range(40): # the server needs a moment to listen
|
||||
if app.pve_serves_with(os.path.join(d, "ca.pem")):
|
||||
ok = True
|
||||
break
|
||||
__import__("time").sleep(0.1)
|
||||
self.assertTrue(ok, "the server's certificate is signed by ca.pem: the handshake must verify with it alone")
|
||||
self.assertFalse(app.pve_serves_with(os.path.join(d, "leaf.pem")), "another certificate as the CA file must NOT verify")
|
||||
osapply.PVE_LOCAL_API = "127.0.0.1:1"
|
||||
self.assertFalse(app.pve_serves_with(os.path.join(d, "ca.pem")), "no answer is a no")
|
||||
finally:
|
||||
osapply.PVE_LOCAL_API = saved
|
||||
srv.kill(); srv.wait()
|
||||
finally:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
def test_a_plan_that_names_a_path_or_a_file_is_refused(self):
|
||||
for extra in ({"source": "/etc/shadow"}, {"path": "/etc/sudoers.d/x"}, {"dest": "/root/.ssh/authorized_keys"},
|
||||
{"content_b64": "QQ=="}, {"packages": []}, {"file": "x"}, {"vmid": 9201}, {"dry_run": True}):
|
||||
f = TFake()
|
||||
f.plan.update(extra)
|
||||
rep = self.refused(f, "T1")
|
||||
self.assertEqual(f.openssl_stdin, [], "refused before the source is even read")
|
||||
g = TFake()
|
||||
g.plan["layer"] = "kernel"
|
||||
self.refused(g, "R11")
|
||||
|
||||
def test_a_refused_source_keeps_an_older_copy(self):
|
||||
f = TFake(source=b"not a certificate", copy=CA_PEM)
|
||||
self.refused(f, "T3")
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM, "written whole or not at all")
|
||||
|
||||
def test_it_runs_on_a_byo_host_too_and_changes_nothing_else(self):
|
||||
f = TFake()
|
||||
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"]), (0, "written"), rep)
|
||||
self.assertEqual([c[1][0] for c in f.calls], ["openssl"], "one openssl call; no package, no service, no other file")
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,188 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// A cut write of agent.json must never leave a blind box (R-1380, 2026-11-15).
|
||||
//
|
||||
// agent.json is the agent's own file inside a ROOT-owned folder, so the agent cannot write a new file and rename it
|
||||
// over the old one: every writer truncates the file and writes it again. A power cut between the truncate and the
|
||||
// sync leaves an empty or half file, the agent does not start, and nothing can repair a box whose agent does not
|
||||
// start from the hub. So the three writers (the trust move, the PBS seed, the backup-target move) go through
|
||||
// ReplaceInPlace: the OLD bytes are first made durable in a journal file in the agent's own state folder (a new file
|
||||
// and a rename, which the agent CAN do there), then the config is written, synced and read back, then the journal is
|
||||
// removed. RecoverCutWrite runs at every start, before the config is read: a journal beside a config that does not
|
||||
// parse means the write was cut, and the old bytes go back. A journal beside a config that parses means the cut fell
|
||||
// before the truncate or after the sync — the file is whole, old or new, and only the journal is removed.
|
||||
//
|
||||
// Pinned by TestReplaceInPlace_* and TestRecoverCutWrite_* (this package) and by TestMainRecoversACutConfigWrite
|
||||
// (cmd/felhom-agent).
|
||||
|
||||
// CutWriteDir is where the journal lives. It is NOT read from the config: the recovery runs before the config can be
|
||||
// read. /var/lib/felhom-agent is the unit's StateDirectory. A test points it at a temp folder.
|
||||
var CutWriteDir = "/var/lib/felhom-agent"
|
||||
|
||||
// cutWriteJournal is the journal's file name: the bytes agent.json held before the write that is in flight.
|
||||
const cutWriteJournal = "agent.json.write-in-flight"
|
||||
|
||||
// cutPoint lets a test stop a write where a power cut would (the names are the steps of ReplaceInPlace).
|
||||
var cutPoint = func(step string) error { return nil }
|
||||
|
||||
// ReplaceInPlace writes next over the config at path, whose present content is prev. The caller holds FileMu.
|
||||
// With no journal folder there is no write: an unprotected write is the defect this exists to remove.
|
||||
func ReplaceInPlace(path string, prev, next []byte) error {
|
||||
if !json.Valid(prev) || !json.Valid(next) {
|
||||
return fmt.Errorf("config: refusing to write %s: the old or the new content is not JSON", path)
|
||||
}
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(CutWriteDir, 0o700); err != nil {
|
||||
return fmt.Errorf("config: the journal folder: %w", err)
|
||||
}
|
||||
journal := filepath.Join(CutWriteDir, cutWriteJournal)
|
||||
if err := writeDurable(journal, prev); err != nil {
|
||||
return fmt.Errorf("config: the journal of the old content: %w", err)
|
||||
}
|
||||
if err := cutPoint("journal-written"); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
|
||||
if err != nil {
|
||||
_ = removeDurable(journal)
|
||||
return err
|
||||
}
|
||||
if err := cutPoint("truncated"); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
half := len(next) / 2
|
||||
_, werr := f.Write(next[:half])
|
||||
if werr == nil {
|
||||
if err := cutPoint("half-written"); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
_, werr = f.Write(next[half:])
|
||||
}
|
||||
if werr == nil {
|
||||
werr = f.Sync()
|
||||
}
|
||||
if cerr := f.Close(); werr == nil {
|
||||
werr = cerr
|
||||
}
|
||||
if werr == nil {
|
||||
var back []byte
|
||||
if back, werr = os.ReadFile(path); werr == nil && !bytes.Equal(back, next) {
|
||||
werr = fmt.Errorf("the file did not read back as written")
|
||||
}
|
||||
}
|
||||
if werr != nil {
|
||||
// The write failed with the process alive: put the old bytes back now. If that fails too, the journal stays
|
||||
// and the next start repairs the file.
|
||||
if rerr := restoreInPlace(path, prev); rerr != nil {
|
||||
return fmt.Errorf("config: writing %s failed (%v) and the old content could not be put back (%v) — the next start repairs it from the journal", path, werr, rerr)
|
||||
}
|
||||
_ = removeDurable(journal)
|
||||
return fmt.Errorf("config: writing %s failed, the old content was put back: %w", path, werr)
|
||||
}
|
||||
if err := cutPoint("synced"); err != nil {
|
||||
return err
|
||||
}
|
||||
return removeDurable(journal)
|
||||
}
|
||||
|
||||
// RecoverCutWrite repairs a config whose last write was cut. It returns what it did, for the start's log line:
|
||||
// "" (no journal), "journal removed — the config is whole", or "the old config was put back".
|
||||
func RecoverCutWrite(path string) (string, error) {
|
||||
journal := filepath.Join(CutWriteDir, cutWriteJournal)
|
||||
prev, err := os.ReadFile(journal)
|
||||
if os.IsNotExist(err) {
|
||||
return "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("config: reading the write journal: %w", err)
|
||||
}
|
||||
cur, cerr := os.ReadFile(path)
|
||||
if cerr == nil && isJSONObject(cur) {
|
||||
return "journal removed — the config is whole", removeDurable(journal)
|
||||
}
|
||||
if !isJSONObject(prev) {
|
||||
// A journal that is itself broken cannot have been followed by a truncate (it is made durable first), so
|
||||
// this is not a cut write. Leave both for a person.
|
||||
return "", fmt.Errorf("config: %s does not parse and the write journal does not parse either", path)
|
||||
}
|
||||
if err := restoreInPlace(path, prev); err != nil {
|
||||
return "", fmt.Errorf("config: putting the old content back into %s: %w", path, err)
|
||||
}
|
||||
return "the old config was put back", removeDurable(journal)
|
||||
}
|
||||
|
||||
func isJSONObject(b []byte) bool {
|
||||
var doc map[string]json.RawMessage
|
||||
return json.Unmarshal(b, &doc) == nil && doc != nil
|
||||
}
|
||||
|
||||
// restoreInPlace writes content into the EXISTING file (no create: the agent cannot make a file in the config's
|
||||
// folder), so the file keeps its owner and mode.
|
||||
func restoreInPlace(path string, content []byte) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(content); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Sync(); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
// writeDurable makes path hold content, whole, across a power cut: a new file, synced, renamed, the folder synced.
|
||||
func writeDurable(path string, content []byte) error {
|
||||
tmp := path + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(content); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Sync(); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
return err
|
||||
}
|
||||
return syncDir(filepath.Dir(path))
|
||||
}
|
||||
|
||||
func removeDurable(path string) error {
|
||||
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
|
||||
return err
|
||||
}
|
||||
return syncDir(filepath.Dir(path))
|
||||
}
|
||||
|
||||
func syncDir(dir string) error {
|
||||
d, err := os.Open(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer d.Close()
|
||||
return d.Sync()
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const (
|
||||
oldCfg = `{"proxmox":{"tls":{"fingerprint":"AA:BB"}},"keep":"me"}`
|
||||
newCfg = `{"proxmox":{"tls":{"ca_file":"/etc/felhom-agent/pve-root-ca.pem","fingerprint":""}},"keep":"me"}`
|
||||
)
|
||||
|
||||
// cutBox is a config file and a journal folder in a temp dir, with the write cut at one named step.
|
||||
func cutBox(t *testing.T) (cfg string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
cfg = filepath.Join(dir, "etc", "agent.json")
|
||||
if err := os.MkdirAll(filepath.Dir(cfg), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(cfg, []byte(oldCfg), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prevDir, prevCut := CutWriteDir, cutPoint
|
||||
CutWriteDir = filepath.Join(dir, "state")
|
||||
t.Cleanup(func() { CutWriteDir, cutPoint = prevDir, prevCut })
|
||||
return cfg
|
||||
}
|
||||
|
||||
func cutAt(step string) {
|
||||
cutPoint = func(s string) error {
|
||||
if s == step {
|
||||
return errors.New("power cut at " + s)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func read(t *testing.T, p string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// The consequence that matters: after a power cut at ANY step of the write and the repair at the next start, the
|
||||
// config is the whole old file or the whole new file — never a broken one — and Load reads it.
|
||||
func TestRecoverCutWrite_AfterACutAtEveryStepTheConfigIsWholeAndLoads(t *testing.T) {
|
||||
want := map[string]string{
|
||||
"journal-written": oldCfg, // nothing was touched yet
|
||||
"truncated": oldCfg, // the file is EMPTY at the cut
|
||||
"half-written": oldCfg, // the file is HALF at the cut
|
||||
"synced": newCfg, // the new file is whole; only the journal is left
|
||||
}
|
||||
for step, wantAfter := range want {
|
||||
t.Run(step, func(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
cutAt(step)
|
||||
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
|
||||
t.Fatal("the staged cut did not stop the write")
|
||||
}
|
||||
atCut := read(t, cfg)
|
||||
if (step == "truncated" || step == "half-written") && isJSONObject([]byte(atCut)) {
|
||||
t.Fatalf("the staging is wrong: at %s the file must be broken, it is %q", step, atCut)
|
||||
}
|
||||
cutPoint = func(string) error { return nil } // the next start: no cut
|
||||
note, err := RecoverCutWrite(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("repair: %v", err)
|
||||
}
|
||||
if note == "" {
|
||||
t.Fatal("the repair saw no journal — a cut write would stay broken")
|
||||
}
|
||||
if got := read(t, cfg); got != wantAfter {
|
||||
t.Fatalf("after a cut at %s and the repair the config is %q, want %q", step, got, wantAfter)
|
||||
}
|
||||
if _, err := Load(cfg); err != nil {
|
||||
t.Fatalf("the agent does not start with the repaired config: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
|
||||
t.Fatalf("the journal stayed after the repair (%v) — an old journal would undo a later change", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A write that is not cut leaves the new file and NO journal: a journal that stayed would put an old config back
|
||||
// at some later start.
|
||||
func TestReplaceInPlace_AWholeWriteLeavesTheNewFileAndNoJournal(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := read(t, cfg); got != newCfg {
|
||||
t.Fatalf("config = %q", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
|
||||
t.Fatalf("the journal stayed after a whole write: %v", err)
|
||||
}
|
||||
if note, err := RecoverCutWrite(cfg); err != nil || note != "" {
|
||||
t.Fatalf("a start after a whole write must do nothing: note=%q err=%v", note, err)
|
||||
}
|
||||
if got := read(t, cfg); got != newCfg {
|
||||
t.Fatalf("the start changed a whole config to %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// No journal folder, no write: the old file stays as it was.
|
||||
func TestReplaceInPlace_WithoutAJournalNothingIsWritten(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
// the journal cannot be made: a folder sits where its new file must be written
|
||||
if err := os.MkdirAll(filepath.Join(CutWriteDir, cutWriteJournal+".tmp"), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
|
||||
t.Fatal("the write went on with no journal")
|
||||
}
|
||||
if got := read(t, cfg); got != oldCfg {
|
||||
t.Fatalf("the config changed with no journal: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A config that is broken with NO journal is not a cut write: the repair does not touch it, and says nothing.
|
||||
func TestRecoverCutWrite_ABrokenConfigWithNoJournalIsLeftAlone(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
if err := os.WriteFile(cfg, []byte(`{"half":`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
note, err := RecoverCutWrite(cfg)
|
||||
if err != nil || note != "" {
|
||||
t.Fatalf("note=%q err=%v", note, err)
|
||||
}
|
||||
if got := read(t, cfg); got != `{"half":` {
|
||||
t.Fatalf("the repair wrote a config it had no journal for: %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package config
|
||||
|
||||
import "sync"
|
||||
|
||||
// FileMu is held by EVERY read-modify-write of agent.json inside this process: the trust move (pvetrust), the PBS
|
||||
// seed (pbsdr) and the backup-target move (localapi). Each of them reads the whole file, changes one section and
|
||||
// writes the whole file back in place — without one lock two of them could interleave, and the second writer would
|
||||
// put back the document it read BEFORE the first one's change (a silently reverted `escrow.pbs_storage_id`,
|
||||
// `backup.local_backup_target` or trust section). Found by review on 2026-11-14, the night the third writer was
|
||||
// added. Hold it for the whole read … write, never across a call to Proxmox or the root wrapper.
|
||||
// Pinned by TestAgentJSONWritersTakeTheSharedLock (cmd/felhom-agent).
|
||||
var FileMu sync.Mutex
|
||||
+30
-1
@@ -10,6 +10,7 @@ import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -86,6 +87,7 @@ func (e *HTTPError) Error() string {
|
||||
// IS the heartbeat (locked decision 1). Errors are typed (transport vs HTTP) and
|
||||
// never include the bearer token.
|
||||
func (c *Client) Report(ctx context.Context, r *HostReport) (*ControlEnvelope, error) {
|
||||
r.SentAt = time.Now().UTC().Format(time.RFC3339Nano) // the box clock at SENDING (R-1384); see HostReport.SentAt
|
||||
body, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("hub: marshaling report: %w", err)
|
||||
@@ -285,10 +287,37 @@ func (c *Client) Jobs(ctx context.Context) ([]JobWire, error) {
|
||||
// /hosts/{id}/jobs/{job_id}, self-scoped. Called after a job is executed OR permanently rejected
|
||||
// (the nonce is already durably burned on a passing verify, so re-processing is replay-safe).
|
||||
func (c *Client) CompleteJob(ctx context.Context, jobID string) error {
|
||||
return c.CompleteJobWithOutcome(ctx, jobID, JobOutcome{})
|
||||
}
|
||||
|
||||
// JobOutcome says HOW a signed job ended when the agent clears it (R-1383, D78): Outcome is done | rejected | failed
|
||||
// | malformed; Reason is one code of signedjobs.ReasonCodes (empty for done); Op is the job's op class. It rides the
|
||||
// DELETE's query, so a hub that does not know it ignores it. Until this, „ran” and „refused” looked the same to the
|
||||
// hub, and a refused agent update left no trace outside the box's journal. No secret: no signature, no nonce, no
|
||||
// parameter of the job is in it.
|
||||
type JobOutcome struct {
|
||||
Outcome string
|
||||
Reason string
|
||||
Op string
|
||||
}
|
||||
|
||||
// CompleteJobWithOutcome clears the job and tells the hub how it ended. The zero JobOutcome sends no query.
|
||||
func (c *Client) CompleteJobWithOutcome(ctx context.Context, jobID string, o JobOutcome) error {
|
||||
if c.hostID == "" || jobID == "" {
|
||||
return fmt.Errorf("hub: CompleteJob requires host_id + job_id")
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.baseURL+"/api/v1/hosts/"+c.hostID+"/jobs/"+jobID, nil)
|
||||
u := c.baseURL + "/api/v1/hosts/" + c.hostID + "/jobs/" + jobID
|
||||
if o.Outcome != "" {
|
||||
q := url.Values{"outcome": {o.Outcome}}
|
||||
if o.Reason != "" {
|
||||
q.Set("reason", o.Reason)
|
||||
}
|
||||
if o.Op != "" {
|
||||
q.Set("op", o.Op)
|
||||
}
|
||||
u += "?" + q.Encode()
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, u, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1384 (D79): the hub tells a wrong box clock by the time stamped at the moment of SENDING. A report whose collect
|
||||
// started long ago (an old reported_at) must still leave with a fresh sent_at, and a second send of the SAME report
|
||||
// struct must carry a new one — or a report that waited on the box reads as a clock that is behind.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt).
|
||||
func TestReport_StampsSentAtAtSending(t *testing.T) {
|
||||
var bodies []map[string]any
|
||||
c := testClient(func(r *http.Request) (*http.Response, error) {
|
||||
raw, _ := io.ReadAll(r.Body)
|
||||
var m map[string]any
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatalf("body: %v", err)
|
||||
}
|
||||
bodies = append(bodies, m)
|
||||
return httpResp(200, `{"status":"ok"}`), nil
|
||||
})
|
||||
old := time.Now().Add(-40 * time.Minute).UTC()
|
||||
rep := &HostReport{HostID: "h", ReportedAt: old.Format(time.RFC3339), SentAt: old.Format(time.RFC3339Nano)}
|
||||
for i := 0; i < 2; i++ {
|
||||
if _, err := c.Report(context.Background(), rep); err != nil {
|
||||
t.Fatalf("Report: %v", err)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
var sent []time.Time
|
||||
for i, m := range bodies {
|
||||
s, _ := m["sent_at"].(string)
|
||||
at, err := time.Parse(time.RFC3339Nano, s)
|
||||
if err != nil {
|
||||
t.Fatalf("send %d: sent_at = %q, want an RFC3339 time", i, s)
|
||||
}
|
||||
if d := time.Since(at); d < 0 || d > 5*time.Second {
|
||||
t.Errorf("send %d: sent_at is %s old, want the moment of sending (reported_at was 40 minutes old)", i, d)
|
||||
}
|
||||
if m["reported_at"] != old.Format(time.RFC3339) {
|
||||
t.Errorf("send %d: reported_at changed to %v — the collect time is not the send time's to touch", i, m["reported_at"])
|
||||
}
|
||||
sent = append(sent, at)
|
||||
}
|
||||
if len(sent) == 2 && !sent[1].After(sent[0]) {
|
||||
t.Errorf("the second send carries %s, not after the first %s — the stamp is not taken per send", sent[1], sent[0])
|
||||
}
|
||||
}
|
||||
|
||||
// R-1383 (D78): the clearing call says how the job ended, in the DELETE's query; a plain CompleteJob sends none (an
|
||||
// older hub, and the „nothing to say” case, see the same request as before).
|
||||
func TestCompleteJob_OutcomeRidesTheQuery(t *testing.T) {
|
||||
var urls []string
|
||||
c := testClient(func(r *http.Request) (*http.Response, error) {
|
||||
if r.Method != http.MethodDelete {
|
||||
t.Errorf("method = %s", r.Method)
|
||||
}
|
||||
urls = append(urls, r.URL.Path+"?"+r.URL.RawQuery)
|
||||
return httpResp(200, `{"status":"ok"}`), nil
|
||||
})
|
||||
if err := c.CompleteJob(context.Background(), "j1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.CompleteJobWithOutcome(context.Background(), "j2", JobOutcome{Outcome: "rejected", Reason: "expired", Op: "agent_update"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"/api/v1/hosts/demo-host-01/jobs/j1?", "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&reason=expired"}
|
||||
if len(urls) != 2 || urls[0] != want[0] || urls[1] != want[1] {
|
||||
t.Fatalf("requests = %q, want %q", urls, want)
|
||||
}
|
||||
}
|
||||
+51
-6
@@ -124,6 +124,9 @@ type Collector struct {
|
||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
|
||||
pveTrust func() any // R-1380: the proxmox_trust stanza (nil = omitted)
|
||||
guestPowerGaveUp func() any // R-1383: the guest_power_gave_up stanza (nil = omitted)
|
||||
profileRefresh func() any // R-1383: the profile_refresh stanza (nil = omitted)
|
||||
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
|
||||
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||
@@ -209,6 +212,24 @@ func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capabi
|
||||
return c
|
||||
}
|
||||
|
||||
// SetProxmoxTrustSource wires the `proxmox_trust` stanza (R-1380). Optional; nil = the stanza is omitted.
|
||||
func (c *Collector) SetProxmoxTrustSource(f func() any) *Collector {
|
||||
c.pveTrust = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetProfileRefreshSource wires the `profile_refresh` stanza (R-1383). Optional; nil, or a nil result, omits it.
|
||||
func (c *Collector) SetProfileRefreshSource(f func() any) *Collector {
|
||||
c.profileRefresh = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetGuestPowerGaveUpSource wires the `guest_power_gave_up` stanza (R-1383). Optional; nil, or a nil result, omits it.
|
||||
func (c *Collector) SetGuestPowerGaveUpSource(f func() any) *Collector {
|
||||
c.guestPowerGaveUp = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub
|
||||
// watches it for a re-key). Static per process — set once at startup. "" when the local API is
|
||||
// disabled. Returns the collector for chaining.
|
||||
@@ -391,6 +412,27 @@ func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
|
||||
// error (no useful report — the cycle skips the POST); a failed per-guest
|
||||
// GuestConfig degrades that guest to status="unknown" without spec but still sends;
|
||||
// a cloudflared probe failure yields status="unknown" and is never fatal.
|
||||
func (c *Collector) profileRefreshStanza() any {
|
||||
if c.profileRefresh == nil {
|
||||
return nil
|
||||
}
|
||||
return c.profileRefresh()
|
||||
}
|
||||
|
||||
func (c *Collector) guestPowerGaveUpStanza() any {
|
||||
if c.guestPowerGaveUp == nil {
|
||||
return nil
|
||||
}
|
||||
return c.guestPowerGaveUp()
|
||||
}
|
||||
|
||||
func (c *Collector) proxmoxTrust() any {
|
||||
if c.pveTrust == nil {
|
||||
return nil
|
||||
}
|
||||
return c.pveTrust()
|
||||
}
|
||||
|
||||
func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
ns, err := c.px.NodeStatus(ctx)
|
||||
if err != nil {
|
||||
@@ -416,12 +458,15 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
RestoreTests: c.collectRestoreTests(ctx),
|
||||
PBSSnapshots: c.collectPBSSnapshots(ctx),
|
||||
|
||||
AuditTail: []AuditEntry{},
|
||||
Cloudflared: c.cloudflared(ctx),
|
||||
Capabilities: c.capabilities(ctx),
|
||||
LeafFingerprint: c.leafFP,
|
||||
Addresses: c.collectAddresses(),
|
||||
System: c.systemInfo(ctx, ns),
|
||||
AuditTail: []AuditEntry{},
|
||||
Cloudflared: c.cloudflared(ctx),
|
||||
Capabilities: c.capabilities(ctx),
|
||||
LeafFingerprint: c.leafFP,
|
||||
ProxmoxTrust: c.proxmoxTrust(),
|
||||
GuestPowerGaveUp: c.guestPowerGaveUpStanza(),
|
||||
ProfileRefresh: c.profileRefreshStanza(),
|
||||
Addresses: c.collectAddresses(),
|
||||
System: c.systemInfo(ctx, ns),
|
||||
}
|
||||
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
|
||||
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
|
||||
|
||||
@@ -70,8 +70,14 @@ type Loop struct {
|
||||
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
|
||||
// path waits on it so the hub has the deliberate-off record before the box goes down.
|
||||
reported func(r *HostReport)
|
||||
// collectErr (R-1380) sees every failed collect.
|
||||
collectErr func(ctx context.Context, err error)
|
||||
}
|
||||
|
||||
// SetCollectErrorHook wires an observer of a failed collect (R-1380: a call to Proxmox that fails on TRUST starts the
|
||||
// move to the CA copy, or a reload of a refreshed copy). Optional; it must not block for long.
|
||||
func (l *Loop) SetCollectErrorHook(f func(ctx context.Context, err error)) { l.collectErr = f }
|
||||
|
||||
// SetReportedHook wires the R-998 after-report observer. Optional.
|
||||
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
|
||||
|
||||
@@ -144,6 +150,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
|
||||
report, err := l.collector.Collect(ctx)
|
||||
if err != nil {
|
||||
l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err)
|
||||
if l.collectErr != nil {
|
||||
l.collectErr(ctx, err)
|
||||
}
|
||||
return current
|
||||
}
|
||||
// Fulfill a pending log-pull: attach the ring tail to THIS report and clear the
|
||||
|
||||
+20
-2
@@ -15,8 +15,13 @@ import (
|
||||
// server-side received_at is the hub's dead-man's-switch liveness signal. There is
|
||||
// no separate heartbeat endpoint.
|
||||
type HostReport struct {
|
||||
HostID string `json:"host_id"` // echoes config.Hub.HostID
|
||||
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
|
||||
HostID string `json:"host_id"` // echoes config.Hub.HostID
|
||||
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
|
||||
// SentAt (R-1384, D79) is this box's clock at the moment the report is SENT — stamped by Client.Report, never by
|
||||
// the collector. The hub compares it with its own clock to tell a wrong box clock. ReportedAt cannot serve: it is
|
||||
// stamped when the collect STARTS, and a collect that waits on Proxmox makes it old on a box whose clock is right.
|
||||
// Pinned by TestReport_StampsSentAtAtSending.
|
||||
SentAt string `json:"sent_at,omitempty"`
|
||||
AgentVersion string `json:"agent_version"`
|
||||
// AgentSHA256 is the sha256 of the binary this process is RUNNING (read through /proc/self/exe,
|
||||
// once per process), R-349. The version string cannot tell a hand-built proof binary from the
|
||||
@@ -53,6 +58,11 @@ type HostReport struct {
|
||||
// alert. Not a secret (the fp is public; the token is never reported).
|
||||
LeafFingerprint string `json:"leaf_fingerprint"`
|
||||
|
||||
// ProxmoxTrust (R-1380, felhom.eu `03` §12a): how the agent trusts Proxmox's certificate (`mode`: ca | fingerprint |
|
||||
// …), and the CAUSE when something about it is wrong (`problem`, `detail`): the proof with the CA copy failed, the
|
||||
// copy ends soon, the root step gave no copy. Omitted when not wired. No secret: a mode, a cause, a date.
|
||||
ProxmoxTrust any `json:"proxmox_trust,omitempty"`
|
||||
|
||||
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
|
||||
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
|
||||
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
|
||||
@@ -172,6 +182,14 @@ type HostReport struct {
|
||||
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
|
||||
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
|
||||
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
|
||||
// ProfileRefresh (R-1383, release 3): the box's last security-profile refresh and how the agent's last ask ended
|
||||
// (osupdate.ProfileRefreshStatus, field-exact; hub sysfacts.ProfileRefresh). `any` + omitempty: absent when there
|
||||
// is nothing to say or the source is not wired.
|
||||
ProfileRefresh any `json:"profile_refresh,omitempty"`
|
||||
// GuestPowerGaveUp (R-1383, D78): the guests the guest-power watchdog stopped trying to start (localapi
|
||||
// GuestPowerGiveUp, field-exact: vmid, since, attempts). The hub raises one operator alarm per guest and clears it
|
||||
// when the guest runs. `any` + omitempty: absent when there is none or the source is not wired.
|
||||
GuestPowerGaveUp any `json:"guest_power_gave_up,omitempty"`
|
||||
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
|
||||
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
|
||||
// when there is nothing to answer, so the golden stays byte-stable.
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -145,9 +146,11 @@ func (s *Server) validateBackupTargetMount(ctx context.Context, where string) er
|
||||
// begin with. A typed round-trip would silently drop any key this build does not know about.
|
||||
//
|
||||
// Written IN PLACE (O_TRUNC), not tmp+rename: /etc/felhom-agent is root-owned while agent.json is
|
||||
// agent-owned 0600, so the non-root agent cannot rename into that directory. A recovery copy is
|
||||
// parked first, so a torn write is recoverable.
|
||||
// agent-owned 0600, so the non-root agent cannot rename into that directory. config.ReplaceInPlace
|
||||
// journals the old bytes first; the next start repairs a cut write (TestRecoverCutWrite_*).
|
||||
func (s *Server) setConfiguredBackupTarget(id string) error {
|
||||
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
|
||||
defer config.FileMu.Unlock()
|
||||
path := s.configPath
|
||||
if path == "" {
|
||||
return fmt.Errorf("no config path known to this agent (env-only config)")
|
||||
@@ -179,24 +182,14 @@ func (s *Server) setConfiguredBackupTarget(id string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if s.stateDir != "" {
|
||||
if err := os.MkdirAll(s.stateDir, 0o700); err == nil {
|
||||
_ = os.WriteFile(filepath.Join(s.stateDir, "agent.json.pre-backup-target"), raw, 0o600)
|
||||
}
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(out); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
// config.ReplaceInPlace: the old bytes go into a journal first, so a power cut in the middle of the write is
|
||||
// repaired at the next start (R-1380).
|
||||
return config.ReplaceInPlace(path, raw, out)
|
||||
}
|
||||
|
||||
// wrapperReason surfaces the wrapper's own REFUSED line when it produced one — it explains WHY in
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
|
||||
// agent's state folder on a box, and a test must not need it.
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
config.CutWriteDir = dir
|
||||
code := m.Run()
|
||||
_ = os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
@@ -87,6 +87,11 @@ type GuestAttacher interface {
|
||||
// GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a
|
||||
// controller-only restart — the deterministic guest-reboot signal the controller recreates apps on.
|
||||
GuestBootID(ctx context.Context, vmid int) string
|
||||
// GuestBootKey is the same signal built from the kernel's boot_id, which a step of the host's clock cannot
|
||||
// move (R-1395). "" = not readable; the controller then reads GuestBootID.
|
||||
GuestBootKey(ctx context.Context, vmid int) string
|
||||
// GuestBootTokens returns (GuestBootID, GuestBootKey) from one read — what /disks sends.
|
||||
GuestBootTokens(ctx context.Context, vmid int) (id, key string)
|
||||
// AttachBind is the LEGACY per-drive `pct set -mpN` bind (pre-intermediary). Retained for the
|
||||
// transition; new attaches use AttachDrive.
|
||||
AttachBind(ctx context.Context, vmid int, mountKey, where string) error
|
||||
@@ -431,11 +436,12 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
|
||||
// Guest boot-id (intermediary model): changes on every guest boot, stable across controller restarts.
|
||||
// The controller persists it and deterministically recreates drive-backed apps when it changes.
|
||||
bootID := ""
|
||||
// R-1395: guest_boot_key is the clock-proof form (kernel boot_id); guest_boot_id stays for older controllers.
|
||||
bootID, bootKey := "", ""
|
||||
if s.guestAttach != nil {
|
||||
bootID = s.guestAttach.GuestBootID(r.Context(), vmid)
|
||||
bootID, bootKey = s.guestAttach.GuestBootTokens(r.Context(), vmid)
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID})
|
||||
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID, "guest_boot_key": bootKey})
|
||||
}
|
||||
|
||||
// handleDiskCandidates (Impl-2a) lists the host disks FREE for Felhom to enroll — the raw-device
|
||||
|
||||
@@ -466,6 +466,10 @@ func (f *fakeGuestAttacher) AttachDrive(_ context.Context, _ int, where string)
|
||||
}
|
||||
func (f *fakeGuestAttacher) GuestSeesMount(_ context.Context, _ int, _ string) bool { return true }
|
||||
func (f *fakeGuestAttacher) GuestBootID(_ context.Context, _ int) string { return "boot-1" }
|
||||
func (f *fakeGuestAttacher) GuestBootKey(_ context.Context, _ int) string { return "key-1" }
|
||||
func (f *fakeGuestAttacher) GuestBootTokens(_ context.Context, _ int) (string, string) {
|
||||
return "boot-1", "key-1"
|
||||
}
|
||||
func (f *fakeGuestAttacher) attachDriveCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
type initPIDRunner struct{ pid string }
|
||||
|
||||
func (r initPIDRunner) Run(_ context.Context, name string, _ ...string) ([]byte, []byte, error) {
|
||||
if name == "lxc-info" {
|
||||
return []byte(r.pid + "\n"), nil, nil
|
||||
}
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func (r initPIDRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
// R-1395: on 2026-11-14 the N100's time service came back and stepped the host's clock by +32.7 s. The kernel
|
||||
// computes the boot TIME from the wall clock, so guest_boot_id moved and the controller recreated every drive-backed
|
||||
// app. The consequence asserted: the boot KEY does not move when the boot time moves; it does move on a host reboot
|
||||
// and on a guest reboot; and it is "" (so the controller falls back to the old token) when the boot_id is unreadable.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt).
|
||||
func TestGuestBootKey_AClockStepDoesNotMoveIt(t *testing.T) {
|
||||
oldB, oldK, oldS := hostBtimeFn, hostBootIDFn, procStarttimeFn
|
||||
t.Cleanup(func() { hostBtimeFn, hostBootIDFn, procStarttimeFn = oldB, oldK, oldS })
|
||||
btime, bootID, start := "1794112518", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
|
||||
hostBtimeFn = func() string { return btime }
|
||||
hostBootIDFn = func() string { return bootID }
|
||||
procStarttimeFn = func(pid string) string {
|
||||
if pid != "31337" {
|
||||
t.Errorf("start time read for pid %q, want the guest's init", pid)
|
||||
}
|
||||
return start
|
||||
}
|
||||
b := NewGuestBinder(initPIDRunner{"31337"}, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
ctx := context.Background()
|
||||
|
||||
id0, key0 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
|
||||
if id0 != "1794112518-4242" || key0 != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" {
|
||||
t.Fatalf("tokens = %q / %q", id0, key0)
|
||||
}
|
||||
btime = "1794112550" // THE STEP: the wall clock jumped +32 s, so the kernel's btime did too
|
||||
id1, key1 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
|
||||
if id1 == id0 {
|
||||
t.Fatalf("control: the old token did not move with the boot time (%q) — the test does not stage the defect", id1)
|
||||
}
|
||||
if key1 != key0 {
|
||||
t.Fatalf("the boot key moved with the clock: %q → %q — a clock step reads as a new boot", key0, key1)
|
||||
}
|
||||
if strings.Contains(key1, "1794") {
|
||||
t.Errorf("the boot key carries the boot time: %q", key1)
|
||||
}
|
||||
bootID = "c0ffee00-1111-2222-3333-444455556666" // a host reboot
|
||||
if k := b.GuestBootKey(ctx, 9201); k == key0 {
|
||||
t.Errorf("a host reboot (new boot_id) did not change the key: %q", k)
|
||||
}
|
||||
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "991234" // a guest reboot
|
||||
if k := b.GuestBootKey(ctx, 9201); k == key0 || !strings.HasSuffix(k, "-991234") {
|
||||
t.Errorf("a guest reboot (new init start) did not change the key: %q", k)
|
||||
}
|
||||
for _, bad := range []string{"", "not-a-boot-id", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11\nx"} {
|
||||
bootID = bad
|
||||
if k := b.GuestBootKey(ctx, 9201); k != "" {
|
||||
t.Errorf("boot_id %q gave the key %q, want \"\" (the controller then reads the old token)", bad, k)
|
||||
}
|
||||
if id := b.GuestBootID(ctx, 9201); id == "" {
|
||||
t.Errorf("the old token must still be sent when the boot_id is unreadable")
|
||||
}
|
||||
}
|
||||
// what /disks sends: both tokens from ONE read of the guest's init — the init is asked for once, not twice
|
||||
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
|
||||
reads := 0
|
||||
procStarttimeFn = func(string) string { reads++; return start }
|
||||
if id, key := b.GuestBootTokens(ctx, 9201); id != "1794112550-4242" || key != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" || reads != 1 {
|
||||
t.Errorf("GuestBootTokens = %q, %q after %d read(s) of the init — want both tokens from ONE read", id, key, reads)
|
||||
}
|
||||
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "" // the guest does not run
|
||||
if k := b.GuestBootKey(ctx, 9201); k != "" {
|
||||
t.Errorf("a guest that does not run gave the key %q", k)
|
||||
}
|
||||
}
|
||||
|
||||
// The key is no use if /disks does not carry it: both tokens are on the wire, the old one unchanged for controllers
|
||||
// that do not know the key.
|
||||
func TestDisks_CarriesBothBootTokens(t *testing.T) {
|
||||
srv := targetRowServer(t, "", []hub.StorageTarget{})
|
||||
srv.guestAttach = &fakeGuestAttacher{}
|
||||
body := do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()
|
||||
var w struct {
|
||||
Data struct {
|
||||
ID string `json:"guest_boot_id"`
|
||||
Key string `json:"guest_boot_key"`
|
||||
} `json:"data"`
|
||||
ID string `json:"guest_boot_id"`
|
||||
Key string `json:"guest_boot_key"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &w); err != nil {
|
||||
t.Fatalf("decode: %v (%s)", err, body)
|
||||
}
|
||||
if w.Data.ID+w.ID != "boot-1" || w.Data.Key+w.Key != "key-1" {
|
||||
t.Fatalf("/disks boot tokens: id=%q key=%q, want boot-1 and key-1 (%s)", w.Data.ID+w.ID, w.Data.Key+w.Key, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The real binder is what main.go hands the server: it must have the method (the interface makes this a build
|
||||
// error if not — stated here so the reason is on record).
|
||||
var _ GuestAttacher = (*GuestBinder)(nil)
|
||||
@@ -3,6 +3,7 @@ package localapi
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -82,6 +83,40 @@ type guestPowerState struct {
|
||||
attempts int
|
||||
nextAt time.Time
|
||||
raised bool // the give-up fault has already been raised for this run
|
||||
raisedAt time.Time
|
||||
}
|
||||
|
||||
// GuestPowerGiveUp is one guest the watchdog stopped trying to start (R-1383, D78). It rides the host report
|
||||
// (`guest_power_gave_up`) so the operator hears it: until this, the give-up was one ERROR line in the host's journal
|
||||
// while the household's apps were off. Memory only — an agent restart forgets it (and tries three times again).
|
||||
type GuestPowerGiveUp struct {
|
||||
VMID int `json:"vmid"`
|
||||
Since string `json:"since"` // RFC3339 UTC, when the watchdog gave up
|
||||
Attempts int `json:"attempts"`
|
||||
}
|
||||
|
||||
// GuestPowerGaveUpStanza is GuestPowerGaveUp shaped for the host report: an UNTYPED nil when there is none, so the
|
||||
// report omits the field (a typed nil slice in an `any` would be sent as `null`).
|
||||
func (s *Server) GuestPowerGaveUpStanza() any {
|
||||
if g := s.GuestPowerGaveUp(); len(g) > 0 {
|
||||
return g
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GuestPowerGaveUp returns the guests the watchdog has given up on, by vmid. nil when there is none — the report
|
||||
// stanza is then omitted. Pinned by TestGuestPower_GiveUpIsReportedUntilTheGuestRuns.
|
||||
func (s *Server) GuestPowerGaveUp() []GuestPowerGiveUp {
|
||||
guestPowerMu.Lock()
|
||||
defer guestPowerMu.Unlock()
|
||||
var out []GuestPowerGiveUp
|
||||
for vmid, st := range s.guestPower {
|
||||
if st.raised {
|
||||
out = append(out, GuestPowerGiveUp{VMID: vmid, Since: st.raisedAt.UTC().Format(time.RFC3339), Attempts: st.attempts})
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].VMID < out[j].VMID })
|
||||
return out
|
||||
}
|
||||
|
||||
// WatchGuestPower runs the guest-power sweep every guestPowerInterval until ctx is done. No-op when
|
||||
@@ -117,6 +152,19 @@ func (s *Server) GuestPowerTick(ctx context.Context) {
|
||||
s.logger.Warn("guest-power: guest list unavailable — skipping sweep (ownership unproven)", "err", err)
|
||||
return
|
||||
}
|
||||
// A guest that left the list (destroyed, or re-made under another number) takes its attempt record with it — a
|
||||
// give-up for a guest that no longer exists would ride every host report until the agent restarts.
|
||||
present := make(map[int]bool, len(guests))
|
||||
for _, g := range guests {
|
||||
present[g.VMID] = true
|
||||
}
|
||||
guestPowerMu.Lock()
|
||||
for vmid := range s.guestPower {
|
||||
if !present[vmid] {
|
||||
delete(s.guestPower, vmid)
|
||||
}
|
||||
}
|
||||
guestPowerMu.Unlock()
|
||||
var stopped int
|
||||
for _, g := range guests {
|
||||
if ctx.Err() != nil {
|
||||
@@ -167,6 +215,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
// SCENARIO B — a deliberately stopped guest is left alone, forever. onboot:0 means the operator
|
||||
// (or the golden-image provisioning) does not want this guest running.
|
||||
if !onboot {
|
||||
s.forgetGuestPower(g.VMID) // not ours to start any more: a give-up for it must not be reported for ever
|
||||
return
|
||||
}
|
||||
|
||||
@@ -176,6 +225,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
if lock != "" {
|
||||
s.logger.Info("guest-power: guest is stopped but LOCKED — leaving it to the stale-lock path",
|
||||
"vmid", g.VMID, "lock", lock)
|
||||
s.forgetGuestPower(g.VMID) // another operation owns it now; if it ends stopped, the three tries start again
|
||||
return
|
||||
}
|
||||
|
||||
@@ -260,6 +310,7 @@ func (s *Server) markGuestPowerRaised(vmid int) {
|
||||
defer guestPowerMu.Unlock()
|
||||
st := s.guestPower[vmid]
|
||||
st.raised = true
|
||||
st.raisedAt = s.now()
|
||||
s.guestPower[vmid] = st
|
||||
}
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"testing"
|
||||
@@ -237,3 +238,81 @@ func TestGuestPower_RunningGuestClearsAttemptHistory(t *testing.T) {
|
||||
t.Error("attempt history survived the guest coming back healthy")
|
||||
}
|
||||
}
|
||||
|
||||
// R-1383 (D78): the give-up was ONE journal line while the household's apps were off. It now rides the host report:
|
||||
// nothing before the give-up (and an untyped nil, so the report omits the field), the guest with the time and the
|
||||
// attempt count after it, and nothing again once the guest runs.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-guest-power.txt).
|
||||
func TestGuestPower_GiveUpIsReportedUntilTheGuestRuns(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
startErr: errors.New("cannot start: storage offline"),
|
||||
}
|
||||
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
|
||||
s := gpServer(t, ctl, func() time.Time { return now })
|
||||
if st := s.GuestPowerGaveUpStanza(); st != nil {
|
||||
t.Fatalf("before any attempt the stanza is %#v, want an untyped nil (the report omits the field)", st)
|
||||
}
|
||||
for i := 0; i < guestPowerMaxAttempts; i++ { // the three failed starts
|
||||
s.GuestPowerTick(context.Background())
|
||||
if g := s.GuestPowerGaveUp(); len(g) != 0 {
|
||||
t.Fatalf("after %d failed start(s) the watchdog still tries, but it reports a give-up: %+v", i+1, g)
|
||||
}
|
||||
now = now.Add(10 * time.Minute)
|
||||
}
|
||||
gaveUpAt := now
|
||||
s.GuestPowerTick(context.Background()) // the tick that gives up
|
||||
now = now.Add(time.Hour)
|
||||
s.GuestPowerTick(context.Background())
|
||||
g := s.GuestPowerGaveUp()
|
||||
if len(g) != 1 || g[0].VMID != 9201 || g[0].Attempts != guestPowerMaxAttempts || g[0].Since != gaveUpAt.Format(time.RFC3339) {
|
||||
t.Fatalf("give-up list = %+v, want guest 9201, %d attempts, since %s (the give-up's time, not a later tick's)", g, guestPowerMaxAttempts, gaveUpAt.Format(time.RFC3339))
|
||||
}
|
||||
b, _ := json.Marshal(s.GuestPowerGaveUpStanza())
|
||||
if string(b) != `[{"vmid":9201,"since":"2026-11-14T02:30:00Z","attempts":3}]` {
|
||||
t.Fatalf("the stanza on the wire = %s", b)
|
||||
}
|
||||
// the operator starts the guest: the next sweep sees it running and the list is empty again
|
||||
ctl.guests = []proxmox.Guest{{VMID: 9201, Status: "running"}}
|
||||
s.GuestPowerTick(context.Background())
|
||||
if st := s.GuestPowerGaveUpStanza(); st != nil {
|
||||
t.Fatalf("the guest runs again and the stanza is still %#v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// Found by review, 2026-11-14: the give-up was forgotten only when the guest was seen RUNNING. A guest the operator
|
||||
// then switches to onboot:0, a guest another operation locks, and a guest that is destroyed would be reported as
|
||||
// „given up” for ever (the hub's alarm could never clear). Each of the three ends the report.
|
||||
func TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
change func(ctl *fakeGuestPowerCtl)
|
||||
}{
|
||||
{"onboot switched off", func(ctl *fakeGuestPowerCtl) { ctl.onboot = map[int]bool{9201: false} }},
|
||||
{"locked by another operation", func(ctl *fakeGuestPowerCtl) { ctl.locks = map[int]string{9201: "backup"} }},
|
||||
{"destroyed", func(ctl *fakeGuestPowerCtl) { ctl.guests = nil }},
|
||||
} {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
startErr: errors.New("cannot start"),
|
||||
}
|
||||
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
|
||||
s := gpServer(t, ctl, func() time.Time { return now })
|
||||
for i := 0; i < guestPowerMaxAttempts+1; i++ {
|
||||
s.GuestPowerTick(context.Background())
|
||||
now = now.Add(10 * time.Minute)
|
||||
}
|
||||
if len(s.GuestPowerGaveUp()) != 1 {
|
||||
t.Fatalf("%s: setup — no give-up recorded", c.name)
|
||||
}
|
||||
c.change(ctl)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if g := s.GuestPowerGaveUp(); len(g) != 0 {
|
||||
t.Errorf("%s: the give-up is still reported: %+v", c.name, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
@@ -288,21 +289,82 @@ func (b *GuestBinder) guestInitPID(ctx context.Context, vmid int) string {
|
||||
// /proc/<pid>/stat — ticks since host boot, unique per process launch) changes on a guest reboot. ""
|
||||
// on any read error (the controller then keeps its last-seen → no spurious recreate).
|
||||
func (b *GuestBinder) GuestBootID(ctx context.Context, vmid int) string {
|
||||
pid := b.guestInitPID(ctx, vmid)
|
||||
if pid == "" {
|
||||
return ""
|
||||
}
|
||||
start := procStarttime(pid)
|
||||
start := b.guestInitStart(ctx, vmid)
|
||||
if start == "" {
|
||||
return ""
|
||||
}
|
||||
bt := hostBtime()
|
||||
bt := hostBtimeFn()
|
||||
if bt == "" {
|
||||
bt = "0"
|
||||
}
|
||||
return bt + "-" + start
|
||||
}
|
||||
|
||||
// guestInitStart is the guest init's start time in ticks since the host's boot ("" = the guest does not run, or a
|
||||
// read error). Both boot tokens end with it.
|
||||
func (b *GuestBinder) guestInitStart(ctx context.Context, vmid int) string {
|
||||
pid := b.guestInitPID(ctx, vmid)
|
||||
if pid == "" {
|
||||
return ""
|
||||
}
|
||||
return procStarttimeFn(pid)
|
||||
}
|
||||
|
||||
// GuestBootKey (R-1395) is GuestBootID with the host's boot TIME replaced by the kernel's boot_id:
|
||||
// "<boot_id>-<guest-init-starttime>". The kernel computes the boot time (`btime`) from the wall clock, so a STEP of
|
||||
// the host's clock moves GuestBootID — on the N100 on 2026-11-14 the time service came back, stepped +32.7 s, and the
|
||||
// controller recreated every drive-backed app as if the guest had booted. The kernel's boot_id is a random value made
|
||||
// once per boot; no clock can move it. Same meaning as before: it changes on a host reboot (new boot_id) and on a
|
||||
// guest reboot (new init start), and is stable across an agent or controller restart.
|
||||
//
|
||||
// "" when the guest does not run or the boot_id cannot be read — the controller then decides by GuestBootID, which
|
||||
// is still sent (controllers that do not know the key read only that). Pinned by TestGuestBootKey_*.
|
||||
func (b *GuestBinder) GuestBootKey(ctx context.Context, vmid int) string {
|
||||
start := b.guestInitStart(ctx, vmid)
|
||||
return guestBootKey(hostBootIDFn(), start)
|
||||
}
|
||||
|
||||
// GuestBootTokens returns both tokens from ONE read of the guest's init: the old token and the key always describe
|
||||
// the same moment. Read separately, one could succeed and the other fail (the guest stops between the two) — the
|
||||
// controller would then get an old token with no key, read „an older agent”, and recreate every drive-backed app.
|
||||
func (b *GuestBinder) GuestBootTokens(ctx context.Context, vmid int) (id, key string) {
|
||||
start := b.guestInitStart(ctx, vmid)
|
||||
if start == "" {
|
||||
return "", ""
|
||||
}
|
||||
bt := hostBtimeFn()
|
||||
if bt == "" {
|
||||
bt = "0"
|
||||
}
|
||||
return bt + "-" + start, guestBootKey(hostBootIDFn(), start)
|
||||
}
|
||||
|
||||
// guestBootKey joins the two halves; "" when either is missing or the boot_id is not the kernel's shape.
|
||||
func guestBootKey(bootID, start string) string {
|
||||
if start == "" || !kernelBootIDRe.MatchString(bootID) {
|
||||
return ""
|
||||
}
|
||||
return bootID + "-" + start
|
||||
}
|
||||
|
||||
var kernelBootIDRe = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
|
||||
|
||||
// The three reads of /proc behind the boot tokens, as seams (the tests move the boot time and keep the boot_id).
|
||||
var (
|
||||
hostBtimeFn = hostBtime
|
||||
hostBootIDFn = hostBootID
|
||||
procStarttimeFn = procStarttime
|
||||
)
|
||||
|
||||
// hostBootID returns the kernel's boot_id (/proc/sys/kernel/random/boot_id): random, made once per boot. "" on error.
|
||||
func hostBootID() string {
|
||||
data, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(data))
|
||||
}
|
||||
|
||||
// procStarttime returns field 22 (starttime) of /proc/<pid>/stat. The comm field (2) can contain spaces
|
||||
// and parentheses, so we split AFTER the last ')': field 22 is index 19 of the post-comm fields
|
||||
// (field 3 = state is index 0). "" on any error.
|
||||
|
||||
@@ -531,6 +531,8 @@ func (l *Leg) runKernelCleanup(ctx context.Context, runID string, vmid int, trig
|
||||
gained = fmt.Sprintf("%.1f GiB", float64(*res.BytesGained)/(1<<30))
|
||||
}
|
||||
rep.Outcome, rep.Healthy = "cleaned", true
|
||||
rep.KernelCleanup = &CleanupDetail{Removed: capNames(res.Removed, 40), Kept: capNames(sortedKeys(res.Kept), 40),
|
||||
BytesGained: res.BytesGained, Leftovers: len(res.Leftovers)}
|
||||
rep.HealthReason = truncate(fmt.Sprintf("removed %d old kernel package(s): %s; kept %s; %s gained; %d left-over path(s)",
|
||||
len(res.Removed), strings.Join(res.Removed, ", "), strings.Join(sortedKeys(res.Kept), ", "), gained, len(res.Leftovers)), 600)
|
||||
lg.Info("osupdate: kernel clean-up — old kernels removed", "removed", len(res.Removed), "names", strings.Join(res.Removed, ","),
|
||||
@@ -564,6 +566,22 @@ func (l *Leg) KernelCleanupDryRun(ctx context.Context, vmid int) (json.RawMessag
|
||||
return wr.KernelCleanup, nil
|
||||
}
|
||||
|
||||
// capNames bounds a list that goes to the hub: at most n names, each at most 120 bytes; never nil (the hub reads
|
||||
// "removed: []" as "none", not as "unknown").
|
||||
func capNames(in []string, n int) []string {
|
||||
out := make([]string, 0, len(in))
|
||||
for _, x := range in {
|
||||
if len(out) == n {
|
||||
break
|
||||
}
|
||||
if len(x) > 120 {
|
||||
x = x[:120]
|
||||
}
|
||||
out = append(out, x)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string][]string) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
|
||||
@@ -400,6 +400,25 @@ func TestKernelCleanup_NightOrderOnceAndNoNames(t *testing.T) {
|
||||
if len(lastRep.Kernel) != 0 || kverRE.MatchString(lastRep.ReleaseID) {
|
||||
t.Fatalf("the clean-up's report names a kernel (the hub reads that as a kernel step): %+v", lastRep)
|
||||
}
|
||||
// R-1383 (release 3): the detail the hub's System page shows — removed, kept, gained, left-overs — rides the same
|
||||
// report as its own object. On the wire it is `kernel_cleanup`, and the report STILL has no `kernel` key.
|
||||
d := lastRep.KernelCleanup
|
||||
if d == nil || len(d.Removed) != 1 || d.Removed[0] != "proxmox-kernel-7.0.14-20-pve-signed" ||
|
||||
strings.Join(d.Kept, ",") != "7.0.14-22-pve,7.0.2-6-pve" || d.BytesGained == nil || *d.BytesGained != 1073741824 || d.Leftovers != 0 {
|
||||
t.Fatalf("the clean-up's detail did not reach the hub: %+v", d)
|
||||
}
|
||||
wire, _ := json.Marshal(lastRep)
|
||||
var keys map[string]json.RawMessage
|
||||
_ = json.Unmarshal(wire, &keys)
|
||||
if _, has := keys["kernel"]; has {
|
||||
t.Fatalf("the clean-up's report carries a `kernel` view on the wire: %s", wire)
|
||||
}
|
||||
if string(keys["kernel_cleanup"]) != `{"removed":["proxmox-kernel-7.0.14-20-pve-signed"],"kept":["7.0.14-22-pve","7.0.2-6-pve"],"bytes_gained":1073741824,"leftovers":0}` {
|
||||
t.Fatalf("kernel_cleanup on the wire = %s", keys["kernel_cleanup"])
|
||||
}
|
||||
if string(keys["release_id"]) != `"kernel-cleanup"` {
|
||||
t.Fatalf("release_id on the wire = %s (a kernel version there is read as a step for it)", keys["release_id"])
|
||||
}
|
||||
// the same night again: the pass is skipped as a whole — never a second clean-up
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
if len(cleanupCalls(w)) != 1 {
|
||||
|
||||
@@ -139,7 +139,9 @@ type WrapperReport struct {
|
||||
KernelCleanup json.RawMessage `json:"kernel_cleanup"`
|
||||
// ProfileRefresh (R-1388, `11` §5.12): mode profile-refresh's result {outcome, kernel, rewritten, restarted, seconds}.
|
||||
ProfileRefresh json.RawMessage `json:"profile_refresh"`
|
||||
OutcomeHint string `json:"outcome_hint"`
|
||||
// PVECACopy (R-1380, felhom.eu `03` §12a): mode pve-ca-copy's result {outcome, path, sha256, not_after, days_left}.
|
||||
PVECACopy json.RawMessage `json:"pve_ca_copy"`
|
||||
OutcomeHint string `json:"outcome_hint"`
|
||||
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
||||
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
||||
RunID string `json:"run_id"`
|
||||
@@ -186,10 +188,23 @@ type Report struct {
|
||||
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
|
||||
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
|
||||
Kernel json.RawMessage `json:"kernel,omitempty"`
|
||||
// KernelCleanup (R-1383, release 3): ONLY on the mode kernel-cleanup report with outcome "cleaned" — what the
|
||||
// clean-up removed, kept and gained, for the hub's System page. It is not the `kernel` view: this report still
|
||||
// carries no kernel view and no kernel as its release_id, or the hub would read it as a kernel STEP for that kernel
|
||||
// (the hub's kernelOf reads only those two). Pinned by TestKernelCleanup_NightOrderOnceAndNoNames.
|
||||
KernelCleanup *CleanupDetail `json:"kernel_cleanup,omitempty"`
|
||||
|
||||
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
|
||||
}
|
||||
|
||||
// CleanupDetail is the clean-up's result as the hub shows it (hub osupdates.CleanupDetail, field-exact).
|
||||
type CleanupDetail struct {
|
||||
Removed []string `json:"removed"` // package names
|
||||
Kept []string `json:"kept"` // kernel versions that stay
|
||||
BytesGained *int64 `json:"bytes_gained,omitempty"` // nil = the wrapper could not measure it
|
||||
Leftovers int `json:"leftovers"` // paths of a removed kernel that are still there
|
||||
}
|
||||
|
||||
// Reporter posts a report to the hub (*hub.Client).
|
||||
type Reporter interface {
|
||||
PostOSReport(ctx context.Context, body []byte) error
|
||||
@@ -209,9 +224,15 @@ type Leg struct {
|
||||
MinGap time.Duration // between night runs (default 20 h)
|
||||
Now func() time.Time
|
||||
Sleep func(context.Context, time.Duration)
|
||||
// ProfileRecordPath is the root wrapper's record of the last security-profile refresh (0644, read-only for the
|
||||
// agent); "" = DefaultProfileRecordPath. A seam for the tests.
|
||||
ProfileRecordPath string
|
||||
|
||||
mu sync.Mutex
|
||||
block *hub.WireOSUpdate
|
||||
mu sync.Mutex
|
||||
block *hub.WireOSUpdate
|
||||
profileAsk string // how the last ask of the profile refresh ended in THIS process ("" = never asked)
|
||||
profileAskAt time.Time
|
||||
profileLast ProfileRefreshStatus // the last refresh as the wrapper answered it to this process
|
||||
}
|
||||
|
||||
// planFile / reportFile: the plan the agent writes and the copy of the report the wrapper keeps beside it (R-868).
|
||||
|
||||
@@ -5,11 +5,14 @@ package osupdate
|
||||
// profile-refresh loads every profile from a fresh compile, once per kernel the box starts on, and restarts only a
|
||||
// failed service whose own program's profile was stale. This file decides WHEN to ask: at every agent start and after
|
||||
// a config bundle was installed (a new wrapper may know the mode the old one refused) — the wrapper itself answers
|
||||
// "not needed" when this kernel was already done. Nothing is sent to the hub (no hub field exists for it; R-1383).
|
||||
// "not needed" when this kernel was already done. No OS report is sent for it (an unknown layer reads as a GUEST step on
|
||||
// the hub); since release 3 the result rides the host report as the `profile_refresh` stanza (ProfileRefreshStanza).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -27,6 +30,113 @@ type ProfileRefreshResult struct {
|
||||
Active bool `json:"active"`
|
||||
} `json:"restarted"`
|
||||
Seconds float64 `json:"seconds"`
|
||||
// The wrapper's own record of the refresh BEFORE this ask (it answers them at every ask, also "not needed").
|
||||
LastRefreshKernel string `json:"last_refresh_kernel"`
|
||||
LastRefreshAt string `json:"last_refresh_at"`
|
||||
}
|
||||
|
||||
// DefaultProfileRecordPath is where the root wrapper keeps the record of its last refresh (PROFILE_STATE, 0644).
|
||||
const DefaultProfileRecordPath = "/var/lib/felhom-os-apply/profile-refresh.json"
|
||||
|
||||
// ProfileRefreshStatus is the host report's `profile_refresh` stanza (R-1383, release 3; hub sysfacts.ProfileRefresh,
|
||||
// field-exact): the last refresh the box did — from the root wrapper's own record, so it outlives an agent restart —
|
||||
// and how the last ask of THIS agent process ended. The refresh restarted a failed service on the N100 and nobody
|
||||
// outside the box's journal could see it.
|
||||
type ProfileRefreshStatus struct {
|
||||
Kernel string `json:"kernel,omitempty"` // the kernel the last refresh ran on
|
||||
At string `json:"at,omitempty"` // RFC3339, when
|
||||
Rewritten *int `json:"rewritten,omitempty"` // how many compiled profiles changed; nil = not known to this process
|
||||
Restarted []string `json:"restarted,omitempty"` // units it restarted (failed, with a stale profile)
|
||||
LastAsk string `json:"last_ask,omitempty"` // refreshed | not_needed | failed | refused | no_answer
|
||||
LastAskAt string `json:"last_ask_at,omitempty"`
|
||||
}
|
||||
|
||||
var profileNameRe = regexp.MustCompile(`^[A-Za-z0-9@._:+-]{1,120}$`)
|
||||
|
||||
// ProfileRefreshStanza builds the stanza, or an UNTYPED nil when there is nothing to say (never asked and no record: a
|
||||
// BYO host, or an agent that did not ask yet) — the report then omits the field.
|
||||
//
|
||||
// Two sources, the first that knows wins per field. (1) What the wrapper ANSWERED to this process's last ask: a
|
||||
// refresh gives everything; "not needed" gives the kernel and time of the earlier refresh (the wrapper answers its own
|
||||
// record at every ask) — so the last refresh is still told after an agent restart, without the detail. (2) The
|
||||
// wrapper's record file, when this process may read it: on boxes where its folder was first made by a signed job it
|
||||
// is root-only (0700), so the file is a bonus, never the only source (found by review, 2026-11-14; not read on a box).
|
||||
// It never calls the wrapper. Names that are not plain unit / kernel names are dropped, the list is capped.
|
||||
func (l *Leg) ProfileRefreshStanza() any {
|
||||
l.mu.Lock()
|
||||
st, ask, at := l.profileLast, l.profileAsk, l.profileAskAt
|
||||
l.mu.Unlock()
|
||||
have := ask != ""
|
||||
if have {
|
||||
st.LastAsk, st.LastAskAt = ask, at.UTC().Format(time.RFC3339)
|
||||
}
|
||||
path := l.ProfileRecordPath
|
||||
if path == "" {
|
||||
path = DefaultProfileRecordPath
|
||||
}
|
||||
if raw, err := os.ReadFile(path); err == nil && len(raw) < 64<<10 {
|
||||
var rec struct {
|
||||
Kernel string `json:"kernel"`
|
||||
At string `json:"at"`
|
||||
Rewritten []string `json:"rewritten"`
|
||||
Restarted []string `json:"restarted"`
|
||||
}
|
||||
if json.Unmarshal(raw, &rec) == nil && (rec.Kernel != "" || rec.At != "") {
|
||||
have = true
|
||||
fromRec := profileStatusFrom(rec.Kernel, rec.At, rec.Rewritten, rec.Restarted, true)
|
||||
if st.At == "" || fromRec.At == st.At || st.Rewritten == nil {
|
||||
fromRec.LastAsk, fromRec.LastAskAt = st.LastAsk, st.LastAskAt
|
||||
st = fromRec
|
||||
}
|
||||
}
|
||||
}
|
||||
if !have {
|
||||
return nil
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// profileStatusFrom checks and bounds what goes to the hub. detail=false: the count and the units are not known.
|
||||
func profileStatusFrom(kernel, at string, rewritten, restarted []string, detail bool) ProfileRefreshStatus {
|
||||
var st ProfileRefreshStatus
|
||||
if profileNameRe.MatchString(kernel) {
|
||||
st.Kernel = kernel
|
||||
}
|
||||
if _, err := time.Parse(time.RFC3339, at); err == nil {
|
||||
st.At = at
|
||||
}
|
||||
if detail {
|
||||
n := len(rewritten)
|
||||
st.Rewritten = &n
|
||||
for _, u := range restarted {
|
||||
if len(st.Restarted) < 16 && profileNameRe.MatchString(u) {
|
||||
st.Restarted = append(st.Restarted, u)
|
||||
}
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// noteProfileAsk remembers how an ask ended and, when the wrapper answered, what it said about the last refresh.
|
||||
func (l *Leg) noteProfileAsk(outcome string, res *ProfileRefreshResult) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.profileAsk, l.profileAskAt = outcome, l.now()
|
||||
if res == nil {
|
||||
return
|
||||
}
|
||||
switch outcome {
|
||||
case "refreshed":
|
||||
units := make([]string, 0, len(res.Restarted))
|
||||
for _, u := range res.Restarted {
|
||||
units = append(units, u.Unit)
|
||||
}
|
||||
l.profileLast = profileStatusFrom(res.Kernel, l.now().UTC().Format(time.RFC3339), res.Rewritten, units, true)
|
||||
case "not_needed":
|
||||
if res.LastRefreshAt != "" || res.LastRefreshKernel != "" {
|
||||
l.profileLast = profileStatusFrom(res.LastRefreshKernel, res.LastRefreshAt, nil, nil, false)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ProfileGate takes the heavy-op gate (a backup, a restore-test or an OS step must not run beside the refresh).
|
||||
@@ -69,12 +179,15 @@ func (l *Leg) ProfileRefresh(ctx context.Context, why string, gate ProfileGate,
|
||||
switch {
|
||||
case err != nil:
|
||||
lg.Warn("osupdate: profile refresh — the wrapper gave no answer; the next agent start asks again", "err", err, "took", took)
|
||||
l.noteProfileAsk("no_answer", nil)
|
||||
return ""
|
||||
case wr.refused():
|
||||
lg.Info("osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)", "refused", string(wr.Refused))
|
||||
l.noteProfileAsk("refused", nil)
|
||||
return ""
|
||||
case wr.failed():
|
||||
lg.Error("osupdate: profile refresh FAILED — the profiles are as before; the next agent start asks again", "failed", string(wr.Failed), "took", took)
|
||||
l.noteProfileAsk("failed", nil)
|
||||
return "failed"
|
||||
}
|
||||
var res ProfileRefreshResult
|
||||
@@ -97,5 +210,8 @@ func (l *Leg) ProfileRefresh(ctx context.Context, why string, gate ProfileGate,
|
||||
default:
|
||||
lg.Warn("osupdate: profile refresh — the wrapper answered with no outcome this agent knows", "outcome", res.Outcome)
|
||||
}
|
||||
if res.Outcome == "refreshed" || res.Outcome == "not_needed" {
|
||||
l.noteProfileAsk(res.Outcome, &res)
|
||||
}
|
||||
return res.Outcome
|
||||
}
|
||||
@@ -3,6 +3,9 @@ package osupdate
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -95,3 +98,72 @@ func TestProfileRefresh_RefusedAndFailed(t *testing.T) {
|
||||
t.Fatalf("failed → %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1383 (release 3): the refresh restarted a failed service on the N100 and only the box's journal knew. The stanza
|
||||
// that rides the host report: nothing before any ask (an UNTYPED nil — the report omits the field); after a refresh,
|
||||
// everything; after an agent RESTART, the kernel and time of that refresh again — from the wrapper's answer to the new
|
||||
// process's ask, because the wrapper's record FILE may be root-only on a box (its folder is 0700 where a signed job
|
||||
// made it first). Where the file can be read, it adds the detail back.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-b3.txt).
|
||||
func TestProfileRefreshStanza_RecordAndLastAsk(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
unreadable := filepath.Join(t.TempDir(), "root-only", "profile-refresh.json") // does not exist: as a 0700 folder reads
|
||||
l.ProfileRecordPath = unreadable
|
||||
now := time.Date(2026, 11, 14, 11, 30, 11, 0, time.UTC)
|
||||
l.Now = func() time.Time { return now }
|
||||
if st := l.ProfileRefreshStanza(); st != nil {
|
||||
t.Fatalf("before any ask the stanza is %#v, want an untyped nil", st)
|
||||
}
|
||||
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "refreshed" {
|
||||
t.Fatalf("outcome = %q", got)
|
||||
}
|
||||
b, _ := json.Marshal(l.ProfileRefreshStanza())
|
||||
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":1,"restarted":["chrony.service"],"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"}` {
|
||||
t.Fatalf("stanza after a refresh, record file unreadable = %s", b)
|
||||
}
|
||||
// an agent restart: a new process asks, the wrapper answers "not needed" with ITS record of the last refresh
|
||||
l2, _ := newLeg(t, &fakeWrapper{t: t, profileRep: &WrapperReport{ProfileRefresh: json.RawMessage(
|
||||
`{"outcome":"not_needed","reason":"this kernel was done","kernel":"7.0.14-28-pve","last_refresh_kernel":"7.0.14-28-pve","last_refresh_at":"2026-11-14T11:30:11Z"}`)}},
|
||||
&hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
l2.ProfileRecordPath = unreadable
|
||||
l2.Now = func() time.Time { return now.Add(3 * time.Hour) }
|
||||
if st := l2.ProfileRefreshStanza(); st != nil {
|
||||
t.Fatalf("a new process that has not asked and cannot read the record: %#v, want nil", st)
|
||||
}
|
||||
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "not_needed" {
|
||||
t.Fatalf("outcome = %q", got)
|
||||
}
|
||||
b, _ = json.Marshal(l2.ProfileRefreshStanza())
|
||||
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
|
||||
t.Fatalf("stanza after an agent restart (the wrapper's answer, no detail) = %s", b)
|
||||
}
|
||||
// where the record file CAN be read, the detail is back — and a name that is not a unit name is dropped
|
||||
rec := filepath.Join(t.TempDir(), "profile-refresh.json")
|
||||
if err := os.WriteFile(rec, []byte(`{"at":"2026-11-14T11:30:11Z","kernel":"7.0.14-28-pve","restarted":["chrony.service","bad name; rm -rf"],`+
|
||||
`"rewritten":["a","b"]}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l2.ProfileRecordPath = rec
|
||||
b, _ = json.Marshal(l2.ProfileRefreshStanza())
|
||||
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":2,"restarted":["chrony.service"],"last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
|
||||
t.Fatalf("stanza with a readable record = %s", b)
|
||||
}
|
||||
// a failed ask is told, beside the older record
|
||||
l2.Runner = (&fakeWrapper{t: t, profileRep: &WrapperReport{Failed: json.RawMessage(`{"code":"F1","reason":"apparmor_parser failed"}`)}})
|
||||
l2.Now = func() time.Time { return now.Add(4 * time.Hour) }
|
||||
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "failed" {
|
||||
t.Fatalf("outcome = %q", got)
|
||||
}
|
||||
b, _ = json.Marshal(l2.ProfileRefreshStanza())
|
||||
if !strings.Contains(string(b), `"last_ask":"failed","last_ask_at":"2026-11-14T15:30:11Z"`) || !strings.Contains(string(b), `"kernel":"7.0.14-28-pve"`) {
|
||||
t.Fatalf("stanza after a failed ask = %s", b)
|
||||
}
|
||||
// a record that is not JSON says nothing and breaks nothing
|
||||
_ = os.WriteFile(rec, []byte("not json"), 0o644)
|
||||
l3, _ := newLeg(t, &fakeWrapper{t: t}, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
l3.ProfileRecordPath = rec
|
||||
if st := l3.ProfileRefreshStanza(); st != nil {
|
||||
t.Fatalf("an unreadable record gave the stanza %#v", st)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package osupdate
|
||||
|
||||
// The root step that keeps a readable copy of Proxmox's root CA for the agent (R-1380, D80, felhom.eu `03` §12a).
|
||||
// The rule lives in the wrapper (mode pve-ca-copy: one fixed source, one fixed place, a CA certificate only); the plan
|
||||
// names nothing. internal/pvetrust decides when to ask and what to do with the answer.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// PVECACopyPath is where the wrapper puts the copy (felhom-os-apply PVE_CA_COPY) — the agent's `proxmox.tls.ca_file`.
|
||||
const PVECACopyPath = "/etc/felhom-agent/pve-root-ca.pem"
|
||||
|
||||
// PVECACopyResult is the wrapper's pve_ca_copy object.
|
||||
type PVECACopyResult struct {
|
||||
Outcome string `json:"outcome"` // written | same
|
||||
Path string `json:"path"`
|
||||
SHA256 string `json:"sha256"`
|
||||
NotAfter string `json:"not_after"`
|
||||
DaysLeft int `json:"days_left"`
|
||||
Replaced bool `json:"replaced"`
|
||||
}
|
||||
|
||||
// PVECACopy asks the wrapper to make or refresh the copy. An error names a refusal (T1–T3, or R11 from a wrapper that
|
||||
// does not know the mode yet) or a wrapper that gave no answer.
|
||||
func (l *Leg) PVECACopy(ctx context.Context) (PVECACopyResult, error) {
|
||||
wr, err := l.call(ctx, "pveca"+l.now().UTC().Format("150405.000"), map[string]any{"release_id": "pve-ca-copy",
|
||||
"layer": LayerHost, "mode": "pve-ca-copy"})
|
||||
if err != nil {
|
||||
return PVECACopyResult{}, err
|
||||
}
|
||||
if wr.refused() {
|
||||
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy refused: %s", wr.Refused)
|
||||
}
|
||||
if wr.failed() {
|
||||
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy failed: %s", wr.Failed)
|
||||
}
|
||||
var res PVECACopyResult
|
||||
if jerr := json.Unmarshal(wr.PVECACopy, &res); jerr != nil || res.Outcome == "" || res.Path == "" {
|
||||
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy: the wrapper gave no result (%v)", jerr)
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package pbsdr
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
|
||||
// agent's state folder on a box, and a test must not need it.
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
config.CutWriteDir = dir
|
||||
code := m.Run()
|
||||
_ = os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
@@ -27,6 +27,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
"log/slog"
|
||||
"net/netip"
|
||||
"os"
|
||||
@@ -483,8 +484,10 @@ func (m *Manager) consumedFail(h, msg string, block *hub.WirePBSDR) {
|
||||
// seedEscrowStorageID sets escrow.pbs_storage_id in agent.json when empty/absent — the bare
|
||||
// `--selftest=escrow-create` one-liner must find its storage with no flags. A different existing
|
||||
// value is NEVER clobbered (warn-and-keep). Unknown config keys are preserved verbatim
|
||||
// (map[string]RawMessage read-modify-write, atomic rename).
|
||||
// (map[string]RawMessage read-modify-write, in place behind a journal — config.ReplaceInPlace).
|
||||
func (m *Manager) seedEscrowStorageID(storageID string) error {
|
||||
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
|
||||
defer config.FileMu.Unlock()
|
||||
if m.configPath == "" {
|
||||
return nil
|
||||
}
|
||||
@@ -527,30 +530,18 @@ func (m *Manager) seedEscrowStorageID(storageID string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
st, err := os.Stat(m.configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// The config DIRECTORY (/etc/felhom-agent) is root-owned while the FILE is agent-owned —
|
||||
// tmp+rename is impossible for the non-root agent (rename mutates the directory; live
|
||||
// finding on the demo). So: park a recovery copy in the agent state dir, then rewrite the
|
||||
// file IN PLACE (O_TRUNC). The agent is the file's only writer and the content is small;
|
||||
// a torn write is recoverable from the parked copy.
|
||||
// finding on the demo). So: park a copy for a person in the agent state dir, then rewrite the
|
||||
// file IN PLACE through config.ReplaceInPlace, which journals the old bytes first — the next
|
||||
// start repairs a cut write (R-1380, TestRecoverCutWrite_*).
|
||||
if err := os.MkdirAll(m.stateDir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(m.stateDir, "agent.json.pre-seed"), raw, 0o600); err != nil {
|
||||
return fmt.Errorf("parking the pre-seed copy: %w", err)
|
||||
}
|
||||
f, err := os.OpenFile(m.configPath, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(out); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
if err := config.ReplaceInPlace(m.configPath, raw, out); err != nil {
|
||||
return err
|
||||
}
|
||||
m.logger.Info("pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)", "storage_id", storageID)
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
|
||||
@@ -45,6 +46,39 @@ type Client struct {
|
||||
node string
|
||||
token string
|
||||
http doer
|
||||
|
||||
// tlsMu guards http: SetTLS swaps the transport while calls run (R-1380 — the move from a pinned certificate to the
|
||||
// box's own root CA, and a refreshed CA copy, take effect with no agent restart).
|
||||
tlsMu sync.RWMutex
|
||||
timeout time.Duration
|
||||
}
|
||||
|
||||
func (c *Client) httpDoer() doer {
|
||||
c.tlsMu.RLock()
|
||||
defer c.tlsMu.RUnlock()
|
||||
return c.http
|
||||
}
|
||||
|
||||
// SetTLS replaces how this client trusts the Proxmox certificate, for every call that starts after it returns. On an
|
||||
// error (an unreadable CA file, a bad fingerprint) nothing changes.
|
||||
func (c *Client) SetTLS(t TLSConfig) error {
|
||||
tlsCfg, err := t.build()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
timeout := c.timeout
|
||||
if timeout == 0 {
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
hc := &http.Client{Timeout: timeout, Transport: httpx.NewTransport(tlsCfg, 0)}
|
||||
c.tlsMu.Lock()
|
||||
old := c.http
|
||||
c.http = hc
|
||||
c.tlsMu.Unlock()
|
||||
if o, ok := old.(*http.Client); ok && o != nil {
|
||||
o.CloseIdleConnections() // a kept-alive connection made under the old trust must not serve the next call
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NewClient builds an API client. It validates required config and constructs the
|
||||
@@ -74,10 +108,11 @@ func NewClient(cfg Config) (*Client, error) {
|
||||
Transport: httpx.NewTransport(tlsCfg, 0),
|
||||
}
|
||||
return &Client{
|
||||
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
|
||||
node: cfg.Node,
|
||||
token: cfg.Token,
|
||||
http: hc,
|
||||
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
|
||||
node: cfg.Node,
|
||||
token: cfg.Token,
|
||||
http: hc,
|
||||
timeout: timeout,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -125,7 +160,7 @@ func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader
|
||||
req.Header.Set("Content-Type", contentType)
|
||||
}
|
||||
|
||||
resp, err := c.http.Do(req)
|
||||
resp, err := c.httpDoer().Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("proxmox: %s %s: %w", method, path, err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"crypto/x509"
|
||||
"errors"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The causes a call to Proxmox can fail on TRUST (R-1380, felhom.eu `03` §12a). A symptom alarm („the agent channel is
|
||||
// down") exists; these name the cause.
|
||||
const (
|
||||
TrustPinnedCertChanged = "pinned_cert_changed" // the pinned certificate is not the one Proxmox serves any more
|
||||
TrustCAMismatch = "ca_mismatch" // the CA file did not sign the served certificate
|
||||
TrustCertExpired = "cert_expired" // the served certificate (or the CA) has ended
|
||||
TrustCAFileUnreadable = "ca_file_unreadable" // the CA file is missing or holds no certificate
|
||||
TrustNameMismatch = "name_mismatch" // the certificate does not name the address the agent calls
|
||||
)
|
||||
|
||||
// TrustProblem names why err is a TRUST failure, or "" when it is none (a refused connection, a 401, a timeout).
|
||||
func TrustProblem(err error) string {
|
||||
if err == nil {
|
||||
return ""
|
||||
}
|
||||
var ua x509.UnknownAuthorityError
|
||||
var inv x509.CertificateInvalidError
|
||||
var hn x509.HostnameError
|
||||
switch {
|
||||
case strings.Contains(err.Error(), "TLS pin mismatch"):
|
||||
return TrustPinnedCertChanged
|
||||
case errors.As(err, &ua):
|
||||
return TrustCAMismatch
|
||||
case errors.As(err, &inv):
|
||||
if inv.Reason == x509.Expired {
|
||||
return TrustCertExpired
|
||||
}
|
||||
return TrustCAMismatch
|
||||
case errors.As(err, &hn):
|
||||
return TrustNameMismatch
|
||||
case strings.Contains(err.Error(), "reading TLS CA file"), strings.Contains(err.Error(), "contained no usable certificates"):
|
||||
return TrustCAFileUnreadable
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package pvetrust
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
|
||||
// agent's state folder on a box, and a test must not need it.
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
config.CutWriteDir = dir
|
||||
code := m.Run()
|
||||
_ = os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
@@ -0,0 +1,406 @@
|
||||
// Package pvetrust moves the agent's trust in Proxmox from ONE pinned certificate to the box's own root CA, and names
|
||||
// the cause when a call fails on trust (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a).
|
||||
//
|
||||
// WHY. Proxmox makes a new node certificate by itself, at most every two years. An agent that pins the old one is then
|
||||
// blind: every call fails, no host report reaches the hub, and the only alarm names a symptom. A copy of the box's own
|
||||
// root CA verifies every certificate that CA signs.
|
||||
//
|
||||
// THE ORDER, and it is the whole point (a box must never be left with neither way working):
|
||||
//
|
||||
// 1. the root step makes or refreshes the copy (it refuses anything that is not one CA certificate);
|
||||
// 2. a REAL call to Proxmox is made with the copy ALONE — while the fingerprint still is the configured trust;
|
||||
// 3. only after that call worked, ONE write of the agent's config sets `ca_file` and clears the fingerprint;
|
||||
// 4. the running client takes the new trust (no restart).
|
||||
//
|
||||
// If 1 or 2 fails nothing is written and the fingerprint stays. If 3 fails the fingerprint stays, in the file and in
|
||||
// the running client. `proxmox.tls` keeps its order in code — a set fingerprint wins over a CA file — so the
|
||||
// fingerprint is cleared LAST, in the same write that sets `ca_file`.
|
||||
//
|
||||
// WHO WRITES THE CONFIG. The agent itself: `agent.json` is the agent's own file (0600) in a root-owned folder, so it is
|
||||
// rewritten in place after a recovery copy was parked in the agent's state folder — the same way the backup-target move
|
||||
// and the PBS seed write it. No root step touches it.
|
||||
package pvetrust
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// EndsSoonDays: the copy's certificate ends within this many days → the problem „ca_copy_ends_soon". Proxmox renews its
|
||||
// root CA when less than two years are left, so a healthy box never comes near it.
|
||||
const EndsSoonDays = 30
|
||||
|
||||
// Problems this package names beyond proxmox.Trust*.
|
||||
const (
|
||||
ProblemCopyUnavailable = "ca_copy_unavailable" // the root step refused or did not answer
|
||||
ProblemCopyNoProof = "ca_copy_does_not_verify" // a call with the copy alone failed — the fingerprint stays
|
||||
ProblemCopyEndsSoon = "ca_copy_ends_soon"
|
||||
ProblemConfigNotSaved = "config_not_written" // the proof worked but the config could not be written — the fingerprint stays
|
||||
)
|
||||
|
||||
// Copy is what the root step answered.
|
||||
type Copy struct {
|
||||
Path string
|
||||
Outcome string // written | same
|
||||
NotAfter string
|
||||
DaysLeft int
|
||||
}
|
||||
|
||||
// Status is the stanza that rides the host report (`proxmox_trust`). No secret: a mode, a cause, a date.
|
||||
type Status struct {
|
||||
Mode string `json:"mode"` // ca | fingerprint | ca_file_own | insecure | system
|
||||
Problem string `json:"problem,omitempty"` // "" = none
|
||||
Detail string `json:"detail,omitempty"`
|
||||
CANotAfter string `json:"ca_not_after,omitempty"`
|
||||
CADaysLeft *int `json:"ca_days_left,omitempty"`
|
||||
MovedAt string `json:"moved_at,omitempty"` // when this box moved from the pinned certificate to the CA copy
|
||||
CheckedAt string `json:"checked_at,omitempty"`
|
||||
}
|
||||
|
||||
// Manager holds the trust state of one agent process.
|
||||
type Manager struct {
|
||||
// EnsureCopy runs the root step (osupdate.Leg.PVECACopy).
|
||||
EnsureCopy func(ctx context.Context) (Copy, error)
|
||||
// Probe makes one real call to Proxmox with EXACTLY this trust (a throwaway client).
|
||||
Probe func(ctx context.Context, t proxmox.TLSConfig) error
|
||||
// Apply gives the running client the new trust (proxmox.Client.SetTLS).
|
||||
Apply func(t proxmox.TLSConfig) error
|
||||
// ConfigPath is agent.json ("" = the config came from the environment: nothing can be saved, so nothing is moved).
|
||||
ConfigPath string
|
||||
StateDir string
|
||||
Logger *slog.Logger
|
||||
Now func() time.Time
|
||||
// MinGap between two runs that a failed call asked for (default 10 minutes).
|
||||
MinGap time.Duration
|
||||
|
||||
runMu sync.Mutex // serialises Ensure
|
||||
mu sync.Mutex
|
||||
cur proxmox.TLSConfig
|
||||
status Status
|
||||
lastRun time.Time
|
||||
}
|
||||
|
||||
// copyStepTimeout bounds the root step that makes the CA copy.
|
||||
const copyStepTimeout = 90 * time.Second
|
||||
|
||||
func (m *Manager) now() time.Time {
|
||||
if m.Now != nil {
|
||||
return m.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (m *Manager) log() *slog.Logger {
|
||||
if m.Logger != nil {
|
||||
return m.Logger
|
||||
}
|
||||
return slog.Default()
|
||||
}
|
||||
|
||||
// Start records the trust the process started with. Call once, before Ensure.
|
||||
func (m *Manager) Start(t proxmox.TLSConfig) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.cur = t
|
||||
m.status = Status{Mode: modeOf(t)}
|
||||
}
|
||||
|
||||
func modeOf(t proxmox.TLSConfig) string {
|
||||
switch {
|
||||
case t.InsecureSkipVerify:
|
||||
return "insecure"
|
||||
case t.Fingerprint != "":
|
||||
return "fingerprint"
|
||||
case t.CAFile != "":
|
||||
return "ca"
|
||||
}
|
||||
return "system"
|
||||
}
|
||||
|
||||
// Status is the current stanza (nil-safe).
|
||||
func (m *Manager) Status() *Status {
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s := m.status
|
||||
return &s
|
||||
}
|
||||
|
||||
// Moved reports whether the box trusts Proxmox by the CA copy (nothing is left to move).
|
||||
func (m *Manager) Moved() bool {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.cur.Fingerprint == "" && m.cur.CAFile != ""
|
||||
}
|
||||
|
||||
// Run asks again on a timer until ctx ends: every `pending` while the box is not moved yet (a wrapper that did not
|
||||
// know the mode, a Proxmox that did not answer), every `settled` after (a renewed CA refreshes the copy).
|
||||
func (m *Manager) Run(ctx context.Context, pending, settled time.Duration) {
|
||||
for {
|
||||
d := settled
|
||||
if !m.Moved() {
|
||||
d = pending
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(d):
|
||||
m.Ensure(ctx, "timer")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Current is the trust the running client uses.
|
||||
func (m *Manager) Current() proxmox.TLSConfig {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.cur
|
||||
}
|
||||
|
||||
// OnCallError is the hook for a failed Proxmox call: when the failure is one of TRUST, it names the cause and runs
|
||||
// Ensure (at most once per MinGap) — a pinned box whose certificate changed moves to the CA copy right then, and a box
|
||||
// whose CA copy was refreshed reloads it. A failure of another kind does nothing.
|
||||
func (m *Manager) OnCallError(ctx context.Context, err error) {
|
||||
cause := proxmox.TrustProblem(err)
|
||||
if cause == "" {
|
||||
return
|
||||
}
|
||||
m.mu.Lock()
|
||||
gap := m.MinGap
|
||||
if gap <= 0 {
|
||||
gap = 10 * time.Minute
|
||||
}
|
||||
tooSoon := !m.lastRun.IsZero() && m.now().Sub(m.lastRun) < gap
|
||||
if m.status.Problem == "" {
|
||||
m.status.Problem, m.status.Detail = cause, causeText(cause)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
m.log().Error("pvetrust: a call to Proxmox failed on TRUST — "+causeText(cause), "cause", cause, "err", err)
|
||||
if tooSoon {
|
||||
return
|
||||
}
|
||||
m.Ensure(ctx, cause)
|
||||
}
|
||||
|
||||
func causeText(cause string) string {
|
||||
switch cause {
|
||||
case proxmox.TrustPinnedCertChanged:
|
||||
return "the pinned certificate is not the one Proxmox serves any more (Proxmox made a new certificate)"
|
||||
case proxmox.TrustCAMismatch:
|
||||
return "the CA copy did not sign the certificate Proxmox serves (the box's root CA changed, or the copy is not this box's)"
|
||||
case proxmox.TrustCertExpired:
|
||||
return "the certificate Proxmox serves, or its CA, has ended"
|
||||
case proxmox.TrustCAFileUnreadable:
|
||||
return "the CA copy is missing or holds no certificate"
|
||||
case proxmox.TrustNameMismatch:
|
||||
return "the certificate Proxmox serves does not name the address the agent calls"
|
||||
}
|
||||
return cause
|
||||
}
|
||||
|
||||
// Ensure runs the order of the package comment once. why is a log word ("start", "daily", a trust cause). It returns
|
||||
// true when the running trust changed.
|
||||
func (m *Manager) Ensure(ctx context.Context, why string) bool {
|
||||
// One run at a time. Ensure is called from the start, the timer, a failed collect and after a bundle install; two
|
||||
// overlapping runs would both read the config, and the second would park the ALREADY-MOVED file as the "before"
|
||||
// copy (losing the fingerprint from the recovery copy). Found by review, 2026-11-14.
|
||||
m.runMu.Lock()
|
||||
defer m.runMu.Unlock()
|
||||
m.mu.Lock()
|
||||
cur := m.cur
|
||||
m.lastRun = m.now()
|
||||
m.mu.Unlock()
|
||||
lg := m.log().With("why", why, "mode", modeOf(cur))
|
||||
set := func(mode, problem, detail string, c *Copy) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
moved := m.status.MovedAt
|
||||
m.status = Status{Mode: mode, Problem: problem, Detail: detail, MovedAt: moved, CheckedAt: m.now().UTC().Format(time.RFC3339)}
|
||||
if c != nil {
|
||||
d := c.DaysLeft
|
||||
m.status.CANotAfter, m.status.CADaysLeft = c.NotAfter, &d
|
||||
}
|
||||
}
|
||||
if cur.InsecureSkipVerify {
|
||||
lg.Warn("pvetrust: certificate checks are switched off in the config (insecure_skip_verify) — nothing is moved")
|
||||
set("insecure", "", "", nil)
|
||||
return false
|
||||
}
|
||||
if m.EnsureCopy == nil || m.Probe == nil {
|
||||
return false
|
||||
}
|
||||
// 1. the copy. The root step gets a deadline of its own: Ensure runs on the daemon's start path and inside the
|
||||
// report loop, and a wrapper that hangs (sudo, a stuck read of /etc/pve) must not stop either.
|
||||
cctx, ccancel := context.WithTimeout(ctx, copyStepTimeout)
|
||||
c, err := m.EnsureCopy(cctx)
|
||||
ccancel()
|
||||
if err != nil {
|
||||
lg.Warn("pvetrust: the root step gave no CA copy — the trust stays as it is", "err", err)
|
||||
set(modeOf(cur), ProblemCopyUnavailable, truncate(err.Error(), 240), nil)
|
||||
return false
|
||||
}
|
||||
ca := proxmox.TLSConfig{CAFile: c.Path}
|
||||
onCopy := cur.Fingerprint == "" && cur.CAFile == c.Path
|
||||
if cur.Fingerprint == "" && cur.CAFile != "" && cur.CAFile != c.Path {
|
||||
// an operator's own CA file: never replaced
|
||||
lg.Info("pvetrust: the config names its own CA file — left as it is", "ca_file", cur.CAFile)
|
||||
set("ca_file_own", "", "", &c)
|
||||
return false
|
||||
}
|
||||
if onCopy && c.Outcome != "written" && why != proxmox.TrustCAMismatch && why != proxmox.TrustCAFileUnreadable && why != proxmox.TrustCertExpired {
|
||||
// the box already trusts by the copy and the copy did not change: nothing to prove again
|
||||
problem, detail := "", ""
|
||||
if c.DaysLeft < EndsSoonDays {
|
||||
problem, detail = ProblemCopyEndsSoon, fmt.Sprintf("the CA copy ends in %d day(s), on %s", c.DaysLeft, c.NotAfter)
|
||||
lg.Error("pvetrust: "+detail, "days_left", c.DaysLeft)
|
||||
}
|
||||
set("ca", problem, detail, &c)
|
||||
return false
|
||||
}
|
||||
// 2. the proof: a real call with the copy ALONE — the fingerprint is still the configured trust
|
||||
pctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
perr := m.Probe(pctx, ca)
|
||||
cancel()
|
||||
if perr != nil {
|
||||
cause := proxmox.TrustProblem(perr)
|
||||
detail := "a call to Proxmox with the CA copy alone failed"
|
||||
if cause != "" {
|
||||
detail += ": " + causeText(cause)
|
||||
} else {
|
||||
detail += " (not a trust failure: " + truncate(perr.Error(), 160) + ")"
|
||||
}
|
||||
if onCopy {
|
||||
lg.Error("pvetrust: the box trusts Proxmox by the CA copy and the copy does not verify the served certificate", "err", perr)
|
||||
set("ca", firstNonEmpty(cause, ProblemCopyNoProof), detail, &c)
|
||||
return false
|
||||
}
|
||||
if cause == "" {
|
||||
// Proxmox did not answer at all (it restarts, the box just booted): nothing is known about the copy, so no
|
||||
// cause is named; the next run asks again.
|
||||
lg.Warn("pvetrust: the proof could not be made now (not a trust failure) — the trust stays as it is; asked again later", "err", perr)
|
||||
set(modeOf(cur), "", "", &c)
|
||||
return false
|
||||
}
|
||||
lg.Error("pvetrust: the proof with the CA copy FAILED — the fingerprint STAYS, nothing was written", "err", perr)
|
||||
set(modeOf(cur), ProblemCopyNoProof, detail+" — the fingerprint stays", &c)
|
||||
return false
|
||||
}
|
||||
if onCopy {
|
||||
// the copy was refreshed (or a trust failure asked): the running client reloads it
|
||||
if m.Apply != nil {
|
||||
if aerr := m.Apply(ca); aerr != nil {
|
||||
set("ca", proxmox.TrustCAFileUnreadable, truncate(aerr.Error(), 200), &c)
|
||||
return false
|
||||
}
|
||||
}
|
||||
lg.Warn("pvetrust: the CA copy was refreshed and proven — the running client uses it now", "outcome", c.Outcome)
|
||||
set("ca", "", "", &c)
|
||||
return true
|
||||
}
|
||||
// 3. the config: ONE write sets ca_file and clears the fingerprint — only now, after the proof
|
||||
if m.ConfigPath == "" {
|
||||
lg.Warn("pvetrust: the proof worked, but the config did not come from a file — nothing can be saved, so the trust stays as it is")
|
||||
set(modeOf(cur), ProblemConfigNotSaved, "the config is not a file; the move was not made", &c)
|
||||
return false
|
||||
}
|
||||
if werr := m.writeConfig(c.Path); werr != nil {
|
||||
lg.Error("pvetrust: the proof worked but the config could NOT be written — the fingerprint stays", "err", werr)
|
||||
set(modeOf(cur), ProblemConfigNotSaved, truncate(werr.Error(), 200), &c)
|
||||
return false
|
||||
}
|
||||
// 4. the running client
|
||||
if m.Apply != nil {
|
||||
if aerr := m.Apply(ca); aerr != nil {
|
||||
lg.Error("pvetrust: the config is written but the running client could not take the CA copy — it keeps the old trust until the next start", "err", aerr)
|
||||
}
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.cur = ca
|
||||
m.mu.Unlock()
|
||||
lg.Warn("pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy",
|
||||
"ca_file", c.Path, "ca_not_after", c.NotAfter)
|
||||
set("ca", "", "", &c)
|
||||
m.mu.Lock()
|
||||
m.status.MovedAt = m.now().UTC().Format(time.RFC3339)
|
||||
m.mu.Unlock()
|
||||
return true
|
||||
}
|
||||
|
||||
func firstNonEmpty(a, b string) string {
|
||||
if a != "" {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n]
|
||||
}
|
||||
|
||||
// writeConfig sets proxmox.tls.ca_file and clears proxmox.tls.fingerprint in ONE write of agent.json. Every other key is
|
||||
// kept byte for byte. A recovery copy is parked in the state folder first; the result is read back and checked.
|
||||
func (m *Manager) writeConfig(caPath string) error {
|
||||
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
|
||||
defer config.FileMu.Unlock()
|
||||
raw, err := os.ReadFile(m.ConfigPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var doc map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return fmt.Errorf("parse %s: %w", m.ConfigPath, err)
|
||||
}
|
||||
px := map[string]json.RawMessage{}
|
||||
if cur, ok := doc["proxmox"]; ok {
|
||||
if err := json.Unmarshal(cur, &px); err != nil {
|
||||
return fmt.Errorf("parse the proxmox section: %w", err)
|
||||
}
|
||||
}
|
||||
tlsSec := map[string]json.RawMessage{}
|
||||
if cur, ok := px["tls"]; ok {
|
||||
if err := json.Unmarshal(cur, &tlsSec); err != nil {
|
||||
return fmt.Errorf("parse the proxmox.tls section: %w", err)
|
||||
}
|
||||
}
|
||||
caJSON, _ := json.Marshal(caPath)
|
||||
tlsSec["ca_file"] = caJSON
|
||||
tlsSec["fingerprint"] = json.RawMessage(`""`)
|
||||
b, _ := json.Marshal(tlsSec)
|
||||
px["tls"] = b
|
||||
b, _ = json.Marshal(px)
|
||||
doc["proxmox"] = b
|
||||
out, err := json.MarshalIndent(doc, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out = append(out, '\n')
|
||||
if m.StateDir != "" {
|
||||
if err := os.MkdirAll(m.StateDir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(m.StateDir, "agent.json.pre-pve-trust"), raw, 0o600); err != nil {
|
||||
return fmt.Errorf("parking the recovery copy: %w", err)
|
||||
}
|
||||
}
|
||||
// In place, not tmp+rename: the folder is root-owned and the file is the agent's. config.ReplaceInPlace keeps the
|
||||
// old bytes in a journal first, so a power cut in the middle is repaired at the next start (R-1380); it also reads
|
||||
// the file back and puts the old bytes back when the write fails with the agent alive.
|
||||
if err := config.ReplaceInPlace(m.ConfigPath, raw, out); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,489 @@
|
||||
package pvetrust
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// ---- a small Proxmox: a root CA that signs node certificates, and a server that can change its certificate ----
|
||||
|
||||
type pki struct {
|
||||
t *testing.T
|
||||
caKey *ecdsa.PrivateKey
|
||||
caCert *x509.Certificate
|
||||
caPEM []byte
|
||||
}
|
||||
|
||||
func newPKI(t *testing.T, cn string) *pki {
|
||||
t.Helper()
|
||||
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
return (&pki{t: t, caKey: k}).renewCA(cn)
|
||||
}
|
||||
|
||||
// renewCA makes a new CA certificate with the SAME key and name (what Proxmox does when its root CA is renewed).
|
||||
func (p *pki) renewCA(cn string) *pki {
|
||||
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: cn},
|
||||
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true,
|
||||
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign}
|
||||
der, err := x509.CreateCertificate(rand.Reader, tpl, tpl, &p.caKey.PublicKey, p.caKey)
|
||||
if err != nil {
|
||||
p.t.Fatal(err)
|
||||
}
|
||||
p.caCert, _ = x509.ParseCertificate(der)
|
||||
p.caPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *pki) leaf() tls.Certificate {
|
||||
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: "node"},
|
||||
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(12 * time.Hour), IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
|
||||
KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}
|
||||
der, err := x509.CreateCertificate(rand.Reader, tpl, p.caCert, &k.PublicKey, p.caKey)
|
||||
if err != nil {
|
||||
p.t.Fatal(err)
|
||||
}
|
||||
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: k}
|
||||
}
|
||||
|
||||
func fp(c tls.Certificate) string {
|
||||
s := sha256.Sum256(c.Certificate[0])
|
||||
return hex.EncodeToString(s[:])
|
||||
}
|
||||
|
||||
type pve struct {
|
||||
srv *httptest.Server
|
||||
mu sync.Mutex
|
||||
cert tls.Certificate
|
||||
}
|
||||
|
||||
func newPVE(t *testing.T, c tls.Certificate) *pve {
|
||||
p := &pve{cert: c}
|
||||
p.srv = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
fmt.Fprint(w, `{"data":{"version":"9.2.2","release":"9.2","repoid":"x"}}`)
|
||||
}))
|
||||
// GetConfigForClient, not GetCertificate: httptest adds its own certificate, and for a call by IP address (no
|
||||
// server name) that one would win over GetCertificate.
|
||||
p.srv.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
return &tls.Config{Certificates: []tls.Certificate{p.cert}}, nil
|
||||
}}
|
||||
p.srv.StartTLS()
|
||||
t.Cleanup(p.srv.Close)
|
||||
return p
|
||||
}
|
||||
|
||||
// newCert is `pveupdate` making a new node certificate: served from the next connection on.
|
||||
func (p *pve) newCert(c tls.Certificate) {
|
||||
p.mu.Lock()
|
||||
p.cert = c
|
||||
p.mu.Unlock()
|
||||
p.srv.CloseClientConnections()
|
||||
}
|
||||
|
||||
// ---- the agent's side ----
|
||||
|
||||
type box struct {
|
||||
t *testing.T
|
||||
m *Manager
|
||||
px *proxmox.Client
|
||||
cfgPath string
|
||||
copyPath string
|
||||
source []byte // what the root step would copy (nil = the root step refuses)
|
||||
events []string
|
||||
applyErr error
|
||||
}
|
||||
|
||||
func cfgJSON(fingerprint, caFile string) string {
|
||||
return fmt.Sprintf(`{
|
||||
"hub": {"url": "https://hub.example", "host_id": "box-1", "api_key": "KEEP-ME"},
|
||||
"proxmox": {"endpoint": "X", "node": "n", "token": "u@pve!t=s", "tls": {"fingerprint": %q, "ca_file": %q, "insecure_skip_verify": false}, "custom": [1, 2, 3]},
|
||||
"unknown_key": {"stays": true}
|
||||
}`, fingerprint, caFile)
|
||||
}
|
||||
|
||||
func newBox(t *testing.T, srv *pve, start proxmox.TLSConfig, source []byte) *box {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
b := &box{t: t, cfgPath: filepath.Join(dir, "agent.json"), copyPath: filepath.Join(dir, "pve-root-ca.pem"), source: source}
|
||||
if start.CAFile == "COPY" {
|
||||
start.CAFile = b.copyPath
|
||||
if err := os.WriteFile(b.copyPath, source, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(b.cfgPath, []byte(cfgJSON(start.Fingerprint, start.CAFile)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pcfg := proxmox.Config{Endpoint: srv.srv.URL, Node: "n", Token: "u@pve!t=s", TLS: start, HTTPTimeout: 5 * time.Second}
|
||||
px, err := proxmox.NewClient(pcfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b.px = px
|
||||
b.m = &Manager{ConfigPath: b.cfgPath, StateDir: filepath.Join(dir, "state"),
|
||||
EnsureCopy: func(context.Context) (Copy, error) {
|
||||
b.events = append(b.events, "copy")
|
||||
if b.source == nil {
|
||||
return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3", "reason": "not a CA certificate"}`)
|
||||
}
|
||||
outcome := "same"
|
||||
if cur, _ := os.ReadFile(b.copyPath); string(cur) != string(b.source) {
|
||||
outcome = "written"
|
||||
if err := os.WriteFile(b.copyPath, b.source, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return Copy{Path: b.copyPath, Outcome: outcome, NotAfter: "2036-08-17T15:44:42Z", DaysLeft: 3560}, nil
|
||||
},
|
||||
Probe: func(ctx context.Context, tl proxmox.TLSConfig) error {
|
||||
b.events = append(b.events, fmt.Sprintf("probe(fp=%q,ca=%v) while the file says fp=%q", tl.Fingerprint, tl.CAFile != "", b.fileTLS().Fingerprint))
|
||||
c := pcfg
|
||||
c.TLS = tl
|
||||
p, err := proxmox.NewClient(c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = p.Version(ctx)
|
||||
return err
|
||||
},
|
||||
Apply: func(tl proxmox.TLSConfig) error {
|
||||
b.events = append(b.events, fmt.Sprintf("apply(fp=%q,ca=%v)", tl.Fingerprint, tl.CAFile != ""))
|
||||
if b.applyErr != nil {
|
||||
return b.applyErr
|
||||
}
|
||||
return b.px.SetTLS(tl)
|
||||
},
|
||||
}
|
||||
b.m.Start(start)
|
||||
return b
|
||||
}
|
||||
|
||||
type fileTLS struct {
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
CAFile string `json:"ca_file"`
|
||||
}
|
||||
|
||||
func (b *box) fileTLS() fileTLS {
|
||||
var d struct {
|
||||
Proxmox struct {
|
||||
TLS fileTLS `json:"tls"`
|
||||
} `json:"proxmox"`
|
||||
}
|
||||
raw, _ := os.ReadFile(b.cfgPath)
|
||||
if err := json.Unmarshal(raw, &d); err != nil {
|
||||
b.t.Fatalf("agent.json is not JSON any more: %v\n%s", err, raw)
|
||||
}
|
||||
return d.Proxmox.TLS
|
||||
}
|
||||
|
||||
func (b *box) call() error {
|
||||
_, err := b.px.Version(context.Background())
|
||||
return err
|
||||
}
|
||||
|
||||
func (b *box) has(prefix string) bool {
|
||||
for _, e := range b.events {
|
||||
if strings.HasPrefix(e, prefix) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Scenario: a NEW box — the installer wrote the copy and ca_file, no fingerprint. Nothing is proven again, nothing written.
|
||||
func TestNewBox_CAOnly_NothingToMove(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
srv := newPVE(t, p.leaf())
|
||||
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatal("a box that already trusts by the copy changed its trust")
|
||||
}
|
||||
after, _ := os.ReadFile(b.cfgPath)
|
||||
if string(before) != string(after) || b.has("probe") || b.has("apply") {
|
||||
t.Fatalf("the config was touched or a proof ran: %v", b.events)
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "ca" || st.Problem != "" || st.CADaysLeft == nil || *st.CADaysLeft != 3560 {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the call fails: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: an OLD box moves. The proof runs with the CA copy ALONE while the file still holds the fingerprint; only
|
||||
// then ONE write sets ca_file and clears the fingerprint; every other key stays; a recovery copy is parked.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/A/redproof-agent.txt): write the config
|
||||
// BEFORE the probe → the order check fails; skip the probe → TestProofFails_TheFingerprintStays fails.
|
||||
func TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
old, _ := os.ReadFile(b.cfgPath)
|
||||
if !b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatalf("the box did not move: %+v %v", b.m.Status(), b.events)
|
||||
}
|
||||
want := []string{"copy", fmt.Sprintf("probe(fp=\"\",ca=true) while the file says fp=%q", fp(leaf)), "apply(fp=\"\",ca=true)"}
|
||||
if strings.Join(b.events, " | ") != strings.Join(want, " | ") {
|
||||
t.Fatalf("order = %v\nwant %v", b.events, want)
|
||||
}
|
||||
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
|
||||
t.Fatalf("the config's tls = %+v", got)
|
||||
}
|
||||
raw, _ := os.ReadFile(b.cfgPath)
|
||||
for _, keep := range []string{`"KEEP-ME"`, `"unknown_key"`, `"stays"`, `"custom"`, `"u@pve!t=s"`, `"insecure_skip_verify"`} {
|
||||
if !strings.Contains(string(raw), keep) {
|
||||
t.Fatalf("the write lost %s:\n%s", keep, raw)
|
||||
}
|
||||
}
|
||||
if parked, _ := os.ReadFile(filepath.Join(b.m.StateDir, "agent.json.pre-pve-trust")); string(parked) != string(old) {
|
||||
t.Fatal("no recovery copy of the old config was parked")
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "ca" || st.MovedAt == "" || st.Problem != "" {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
// and now Proxmox makes a new certificate: the agent goes on, with no restart and no new client
|
||||
srv.newCert(p.leaf())
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("after a new Proxmox certificate the moved box fails: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: the proof fails (the copy is another box's CA). The fingerprint stays — in the file, byte for byte, and in
|
||||
// the running client — and the cause is named.
|
||||
func TestProofFails_TheFingerprintStays(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
other := newPKI(t, "another box")
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatal("the trust changed although the proof failed")
|
||||
}
|
||||
after, _ := os.ReadFile(b.cfgPath)
|
||||
if string(before) != string(after) {
|
||||
t.Fatal("the config was written although the proof failed")
|
||||
}
|
||||
if b.has("apply") {
|
||||
t.Fatalf("the running client was changed: %v", b.events)
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemCopyNoProof || !strings.Contains(st.Detail, "did not sign") {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the fingerprint no longer works: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: Proxmox makes a new certificate on a box that still PINS the old one. The failed call names the cause, the
|
||||
// box moves to the CA copy right then, and the next call works — no restart.
|
||||
func TestPinnedBox_NewCertificate_HealsWithNoRestart(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
srv.newCert(p.leaf())
|
||||
err := b.call()
|
||||
if proxmox.TrustProblem(err) != proxmox.TrustPinnedCertChanged {
|
||||
t.Fatalf("fixture: want a pin mismatch, got %v", err)
|
||||
}
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the box is still blind after the move: %v (%v)", err, b.events)
|
||||
}
|
||||
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
|
||||
t.Fatalf("the config's tls = %+v", got)
|
||||
}
|
||||
// a failure that is NOT one of trust does nothing
|
||||
n := len(b.events)
|
||||
b.m.OnCallError(context.Background(), errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused"))
|
||||
if len(b.events) != n {
|
||||
t.Fatalf("a refused connection started a trust run: %v", b.events[n:])
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: the root CA itself is renewed with the same key. The copy is refreshed, proven, and the running client
|
||||
// reloads it; the config is not written again.
|
||||
func TestCARenewedWithTheSameKey_TheCopyIsRefreshedAndReloaded(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
srv := newPVE(t, p.leaf())
|
||||
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
|
||||
oldCA := string(p.caPEM)
|
||||
p.renewCA("PVE CA")
|
||||
srv.newCert(p.leaf())
|
||||
b.source = p.caPEM
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if !b.m.Ensure(context.Background(), "daily") {
|
||||
t.Fatalf("a refreshed copy was not taken: %+v %v", b.m.Status(), b.events)
|
||||
}
|
||||
if cur, _ := os.ReadFile(b.copyPath); string(cur) == oldCA || string(cur) != string(p.caPEM) {
|
||||
t.Fatal("the copy was not refreshed")
|
||||
}
|
||||
if !b.has("probe") || !b.has("apply") {
|
||||
t.Fatalf("a refreshed copy must be proven and reloaded: %v", b.events)
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatal("the config was written again")
|
||||
}
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the call fails after the CA's renewal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The root step refuses (the source is not a CA certificate): nothing changes, the cause is named.
|
||||
func TestRootStepRefuses_NothingChanges(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("probe") || b.has("apply") {
|
||||
t.Fatalf("something ran without a copy: %v", b.events)
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatal("the config was written")
|
||||
}
|
||||
if st := b.m.Status(); st.Problem != ProblemCopyUnavailable || !strings.Contains(st.Detail, "T3") {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// The proof works but the config cannot be written: the fingerprint stays, in the file and in the running client.
|
||||
func TestConfigCannotBeWritten_TheFingerprintStays(t *testing.T) {
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("root writes a read-only file")
|
||||
}
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
if err := os.Chmod(b.cfgPath, 0o400); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
|
||||
t.Fatalf("the trust changed without a saved config: %v", b.events)
|
||||
}
|
||||
if got := b.fileTLS(); got.Fingerprint != fp(leaf) {
|
||||
t.Fatalf("the fingerprint is gone from the file: %+v", got)
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemConfigNotSaved {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// What is never moved: an operator's own CA file, a box with checks switched off, a config that is not a file.
|
||||
func TestNeverMoved(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
own := filepath.Join(t.TempDir(), "own-ca.pem")
|
||||
os.WriteFile(own, p.caPEM, 0o644)
|
||||
for name, start := range map[string]proxmox.TLSConfig{"own CA file": {CAFile: own}, "insecure": {InsecureSkipVerify: true}} {
|
||||
b := newBox(t, srv, start, p.caPEM)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
|
||||
t.Fatalf("%s: moved (%v)", name, b.events)
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatalf("%s: the config was written", name)
|
||||
}
|
||||
}
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
b.m.ConfigPath = ""
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("apply") || b.m.Status().Problem != ProblemConfigNotSaved {
|
||||
t.Fatalf("a config that is not a file was 'moved': %v %+v", b.events, b.m.Status())
|
||||
}
|
||||
}
|
||||
|
||||
// The copy ends soon: the cause is named (the box still works).
|
||||
func TestCopyEndsSoon(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
srv := newPVE(t, p.leaf())
|
||||
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
|
||||
inner := b.m.EnsureCopy
|
||||
b.m.EnsureCopy = func(ctx context.Context) (Copy, error) {
|
||||
c, err := inner(ctx)
|
||||
c.DaysLeft, c.NotAfter = 12, "2026-11-26T00:00:00Z"
|
||||
return c, err
|
||||
}
|
||||
b.m.Ensure(context.Background(), "daily")
|
||||
if st := b.m.Status(); st.Problem != ProblemCopyEndsSoon || !strings.Contains(st.Detail, "12 day") {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// A trust failure asks at most once per MinGap.
|
||||
func TestOnCallError_Throttled(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
other := newPKI(t, "another box")
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
|
||||
srv.newCert(p.leaf())
|
||||
err := b.call()
|
||||
now := time.Date(2026, 11, 15, 3, 0, 0, 0, time.UTC)
|
||||
b.m.Now = func() time.Time { return now }
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
n := len(b.events)
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
if len(b.events) != n {
|
||||
t.Fatalf("a second failure inside the gap ran again: %v", b.events[n:])
|
||||
}
|
||||
now = now.Add(11 * time.Minute)
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
if len(b.events) == n {
|
||||
t.Fatal("no new run after the gap")
|
||||
}
|
||||
if st := b.m.Status(); st.Problem == "" {
|
||||
t.Fatalf("a blind box names no cause: %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// Proxmox does not answer at all (it restarts): nothing is known about the copy — no cause is named, nothing is written,
|
||||
// and the box counts as not moved, so the timer asks again soon.
|
||||
func TestProofCannotBeMadeNow_NoCauseNamed(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
b.m.Probe = func(context.Context, proxmox.TLSConfig) error {
|
||||
return errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused")
|
||||
}
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatal("moved without a proof")
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatal("the config was written")
|
||||
}
|
||||
if st := b.m.Status(); st.Problem != "" || st.Mode != "fingerprint" || b.m.Moved() {
|
||||
t.Fatalf("status = %+v moved=%v", st, b.m.Moved())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package signedjobs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// outcomeQueue is a jobsQueue that also has the optional half (OutcomeSource), as *hub.Client does.
|
||||
type outcomeQueue struct {
|
||||
jobsQueue
|
||||
omu sync.Mutex
|
||||
outcomes map[string]hub.JobOutcome
|
||||
}
|
||||
|
||||
func (q *outcomeQueue) CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error {
|
||||
q.omu.Lock()
|
||||
if q.outcomes == nil {
|
||||
q.outcomes = map[string]hub.JobOutcome{}
|
||||
}
|
||||
q.outcomes[jobID] = o
|
||||
q.omu.Unlock()
|
||||
return q.jobsQueue.CompleteJob(ctx, jobID)
|
||||
}
|
||||
|
||||
// R-1383 (D78): a signed job that is refused or fails was cleared from the hub's queue exactly like one that ran —
|
||||
// „an agent update never arrives” left one journal line on the box. The runner now says how each job ended, through
|
||||
// the REAL gate and verifier: the stored effect is the outcome the hub receives with the clearing call.
|
||||
// The two clock reasons are the ones the ruling is about: a box with a wrong clock refuses every job.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-job-outcome.txt).
|
||||
func TestRunner_TellsTheHubHowEachJobEnded(t *testing.T) {
|
||||
s := newTestSigner(t)
|
||||
_, _, exec := newRealGateRunner(t, s)
|
||||
base, _, _ := newRealGateRunner(t, s)
|
||||
src := &outcomeQueue{}
|
||||
r := NewRunner(src, base.gate, exec, testHost, quiet())
|
||||
now := time.Now().UTC()
|
||||
|
||||
src.add(mintJob(t, s, "ok", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
|
||||
src.add(mintJob(t, s, "expired", testHost, "", "ops-1", wipeParamsJSON, now.Add(-3*time.Hour), now.Add(-2*time.Hour)))
|
||||
src.add(mintJob(t, s, "early", testHost, "", "ops-1", wipeParamsJSON, now.Add(2*time.Hour), now.Add(3*time.Hour)))
|
||||
src.add(mintJob(t, s, "other-box", "some-other-host", "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
|
||||
src.add(corruptSig(t, mintJob(t, s, "forged", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))))
|
||||
src.add(hub.JobWire{JobID: "garbage", BlobB64: "!!!not-base64!!!"})
|
||||
if _, err := r.RunOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// and a job that is accepted and whose run fails
|
||||
exec.err = errors.New("the wrapper refused")
|
||||
src.add(mintJob(t, s, "run-fails", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
|
||||
if _, err := r.RunOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]hub.JobOutcome{
|
||||
"ok": {Outcome: "done", Op: "storage_wipe"},
|
||||
"expired": {Outcome: "rejected", Reason: "expired", Op: "storage_wipe"},
|
||||
"early": {Outcome: "rejected", Reason: "not_yet_valid", Op: "storage_wipe"},
|
||||
"other-box": {Outcome: "rejected", Reason: "target", Op: "storage_wipe"},
|
||||
"forged": {Outcome: "rejected", Reason: "malformed", Op: "storage_wipe"}, // corruptSig breaks the armour itself
|
||||
"garbage": {Outcome: "malformed", Reason: "malformed"},
|
||||
"run-fails": {Outcome: "failed", Reason: "execution_failed", Op: "storage_wipe"},
|
||||
}
|
||||
for id, w := range want {
|
||||
if got, ok := src.outcomes[id]; !ok || got != w {
|
||||
t.Errorf("job %q: the hub was told %+v (told=%v), want %+v", id, got, ok, w)
|
||||
}
|
||||
}
|
||||
if len(src.outcomes) != len(want) {
|
||||
t.Errorf("outcomes for %d jobs, want %d: %+v", len(src.outcomes), len(want), src.outcomes)
|
||||
}
|
||||
if exec.count() != 2 {
|
||||
t.Errorf("executor ran %d times, want 2 (ok and run-fails only)", exec.count())
|
||||
}
|
||||
}
|
||||
|
||||
// Every code reasonCode can return is on the closed list the hub holds a sentence for.
|
||||
func TestReasonCode_ClosedList(t *testing.T) {
|
||||
known := map[string]bool{}
|
||||
for _, c := range ReasonCodes {
|
||||
known[c] = true
|
||||
}
|
||||
for _, d := range []reconcile.Decision{
|
||||
{}, {Reason: reconcile.ReasonRejected, Err: errors.New("something new")},
|
||||
{Reason: reconcile.ReasonRoleDenied}, {Reason: reconcile.ReasonBindingMismatch}, {Reason: reconcile.ReasonPendingSignature},
|
||||
} {
|
||||
if c := reasonCode(d); !known[c] {
|
||||
t.Errorf("reasonCode(%+v) = %q, not on ReasonCodes", d, c)
|
||||
}
|
||||
}
|
||||
if reasonCode(reconcile.Decision{Reason: reconcile.ReasonRejected, Err: errors.New("x")}) != "other" {
|
||||
t.Error("an unknown refusal must be told as „other”, never as the error's own text")
|
||||
}
|
||||
if !known["execution_failed"] || !known["malformed"] {
|
||||
t.Error("the two codes the runner sets itself are not on the list")
|
||||
}
|
||||
}
|
||||
@@ -35,6 +35,42 @@ type JobSource interface {
|
||||
CompleteJob(ctx context.Context, jobID string) error
|
||||
}
|
||||
|
||||
// OutcomeSource is the optional half of a JobSource: it clears a job AND tells the hub how it ended (R-1383, D78).
|
||||
// *hub.Client has it. A source without it is cleared as before.
|
||||
type OutcomeSource interface {
|
||||
CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error
|
||||
}
|
||||
|
||||
// ReasonCodes is the CLOSED list of reasons the agent sends with a job that did not run to a good end. The hub holds
|
||||
// one operator sentence per code (felhom.eu hub/internal/api, signedJobReasons); a code the hub does not know is
|
||||
// told as „other”. Pinned by TestReasonCode_ClosedList.
|
||||
var ReasonCodes = []string{"expired", "not_yet_valid", "window_too_long", "replay", "bad_signature", "unknown_signer",
|
||||
"target", "namespace", "malformed", "role_denied", "binding_mismatch", "execution_failed", "other"}
|
||||
|
||||
// reasonCode names why the gate refused: the authz error first (the two clock reasons matter most — a box with a
|
||||
// wrong clock refuses EVERY job), then the gate's own reason.
|
||||
func reasonCode(dec reconcile.Decision) string {
|
||||
for _, m := range []struct {
|
||||
err error
|
||||
code string
|
||||
}{
|
||||
{authz.ErrExpired, "expired"}, {authz.ErrNotYetValid, "not_yet_valid"}, {authz.ErrWindowTooLong, "window_too_long"},
|
||||
{authz.ErrReplay, "replay"}, {authz.ErrBadSignature, "bad_signature"}, {authz.ErrUnknownSigner, "unknown_signer"},
|
||||
{authz.ErrTarget, "target"}, {authz.ErrNamespace, "namespace"}, {authz.ErrMalformed, "malformed"},
|
||||
} {
|
||||
if errorsIs(dec.Err, m.err) {
|
||||
return m.code
|
||||
}
|
||||
}
|
||||
switch dec.Reason {
|
||||
case reconcile.ReasonRoleDenied:
|
||||
return "role_denied"
|
||||
case reconcile.ReasonBindingMismatch:
|
||||
return "binding_mismatch"
|
||||
}
|
||||
return "other"
|
||||
}
|
||||
|
||||
// Authorizer is the reversibility gate. Satisfied by *reconcile.Gate.
|
||||
type Authorizer interface {
|
||||
Authorize(intent reconcile.Intent, signed *reconcile.SignedOp) reconcile.Decision
|
||||
@@ -141,7 +177,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
// A malformed/forged envelope is permanently bad — clear it loudly (a queued garbage job
|
||||
// must not wedge the queue). No nonce was consumed (we never reached Verify).
|
||||
r.logger.Error("signedjobs: malformed job envelope — clearing", "job", j.JobID)
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "malformed", Reason: "malformed"})
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -166,7 +202,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
// it as a security event (a compromised hub queuing a forged op lands here).
|
||||
r.logger.Warn("signedjobs: REJECTED signed op — executor not called",
|
||||
"job", j.JobID, "op", ob.Op, "reason", dec.Reason, "err", errStr(dec.Err))
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "rejected", Reason: reasonCode(dec), Op: ob.Op})
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -177,7 +213,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
switch {
|
||||
case err == nil:
|
||||
r.logger.Warn("signedjobs: signed op COMPLETED", "job", j.JobID, "op", ob.Op)
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "done", Op: ob.Op})
|
||||
return true
|
||||
case errorsIs(err, ErrNoExecutor):
|
||||
// No executor for this op class in this build — leave it queued for the owning slice.
|
||||
@@ -189,7 +225,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
// nonce to retry) and log the failure loudly.
|
||||
r.logger.Error("signedjobs: signed op execution FAILED (nonce spent — clearing)",
|
||||
"job", j.JobID, "op", ob.Op, "err", err)
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "failed", Reason: "execution_failed", Op: ob.Op})
|
||||
return true
|
||||
}
|
||||
}
|
||||
@@ -215,8 +251,14 @@ func (r *Runner) decode(j hub.JobWire) (*reconcile.SignedOp, authz.OpBlob, bool)
|
||||
return &reconcile.SignedOp{Blob: opBlob, Sig: []byte(env.SigArmored)}, ob, true
|
||||
}
|
||||
|
||||
func (r *Runner) complete(ctx context.Context, jobID string) {
|
||||
if err := r.source.CompleteJob(ctx, jobID); err != nil {
|
||||
func (r *Runner) complete(ctx context.Context, jobID string, o hub.JobOutcome) {
|
||||
var err error
|
||||
if os, ok := r.source.(OutcomeSource); ok {
|
||||
err = os.CompleteJobWithOutcome(ctx, jobID, o) // R-1383: the hub hears HOW the job ended
|
||||
} else {
|
||||
err = r.source.CompleteJob(ctx, jobID)
|
||||
}
|
||||
if err != nil {
|
||||
r.logger.Warn("signedjobs: clearing job failed (will retry next pass)", "job", jobID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,10 +24,10 @@
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-11-10",
|
||||
"recorded_by": "CC 2026-11-10: day0_agent 0.173.0 (operator vouch 2026-11-10 16:29:16 CET, read from the hub log: agent=0.173.0, bundle 51ebc1f4… = the release; the save took 232 ms); earlier 0.172.1 (CC 2026-11-07) (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"recorded": "2026-11-14",
|
||||
"recorded_by": "CC 2026-11-14: day0_agent 0.174.0 (operator vouch 2026-11-14 15:21:45 CET, read from the hub log and from the hub's System page: agent=0.174.0, bundle 7d87d058… = the release; the save took 428 ms); earlier 0.173.0 (CC 2026-11-10) (operator vouch 2026-11-10 16:29:16 CET, read from the hub log: agent=0.173.0, bundle 51ebc1f4… = the release; the save took 232 ms); earlier 0.172.1 (CC 2026-11-07) (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.173.0"
|
||||
"day0_agent": "0.174.0"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user