Compare commits

..
11 Commits
Author SHA1 Message Date
admin db01ae0f2a agent (unreleased): a cut write of agent.json cannot leave a blind box — a journal of the old bytes, a repair at start (R-1380)
gates / gates (push) Successful in 42s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-15 08:20:49 +01:00
admin 54ac2c2ac9 agent (unreleased): the CA copy keeps its previous root only for the changeover, and is compared exactly (the commit's security review)
gates / gates (push) Successful in 47s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 19:20:07 +01:00
admin ec7e699fd7 agent (unreleased): fixes after an independent review — one lock for the writers of agent.json, the trust run serialised and bounded, the CA copy keeps its previous root, the profile stanza from the wrapper's answer, a stale give-up ends, both boot tokens from one read
gates / gates (push) Successful in 48s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 19:09:21 +01:00
admin 61feb95c06 agent (unreleased): the hub is told what the kernel clean-up did and what the profile refresh did (R-1383)
gates / gates (push) Successful in 58s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 18:06:12 +01:00
admin c0bf97de80 agent: gofmt (formatting only)
gates / gates (push) Successful in 40s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:47:42 +01:00
admin d5ce5c364a agent (unreleased): a boot token that a clock step cannot move (R-1395)
/disks answers guest_boot_key (the kernel's boot_id and the guest init's start) beside the unchanged guest_boot_id.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:43:33 +01:00
admin a1135018c7 agent (unreleased): a refused signed job and a watchdog give-up reach the hub (R-1383, D78)
gates / gates (push) Successful in 42s
The job runner clears a job with its outcome, a closed reason code and the op's name. The guest-power watchdog's
give-up rides the host report until the guest runs.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:29:53 +01:00
admin ede5d0e6c7 agent (unreleased): the report carries the box's clock at the moment of sending (R-1384, D79)
gates / gates (push) Successful in 46s
sent_at is stamped by Client.Report at every send, never by the collector. The hub reads it to tell a wrong box clock.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:09:12 +01:00
admin 386a0c4cf4 2b (unreleased): the agent moves from a pinned Proxmox certificate to the box's root CA copy — copy, proof, then the config; the cause of a trust failure is named (R-1380)
gates / gates (push) Successful in 38s
2026-11-14 16:35:22 +01:00
admin a9ab47ec71 2b (unreleased): the root step that keeps a copy of Proxmox's root CA (wrapper mode pve-ca-copy, R-1380)
gates / gates (push) Successful in 44s
2026-11-14 16:28:04 +01:00
admin 2706e35198 v0.174.0 RELEASED and VOUCHED: CHANGELOG, the day-0 pin, CONTEXT, REPORT
gates / gates (push) Successful in 40s
2026-11-14 15:23:16 +01:00
42 changed files with 3247 additions and 118 deletions

No files matched your search

+167
View File
@@ -1,3 +1,170 @@
## Unreleased (2026-11-15) — a cut write of `agent.json` cannot leave a blind box (R-1380; found before the release)
- **`config.ReplaceInPlace` + `config.RecoverCutWrite`** (`internal/config/cutwrite.go`). `agent.json` is the agent's
file in a root-owned folder, so the trust move, the PBS seed and the backup-target move write it in place
(truncate, write). A power cut in between left an empty or half file and an agent that does not start. Now the old
bytes go into a journal in the state folder first (`agent.json.write-in-flight`, durable), the config is written,
synced and read back, and the journal is removed; the daemon's start repairs a config that does not parse from the
journal BEFORE it reads it. A selftest never repairs. No journal folder → no write. Tests: a cut staged at each of
four steps, the wiring in `main`, and a source pin that the three writers use the helper and open no file by hand.
Red-proofs: felhom.eu `documentation/audits/release-2b-3-2026-11-15/part0/`. Not proven: a real power cut.
## Unreleased (2026-11-14 night) — release 3 („the hub tells you”) and the clock-step fix, the agent's parts, built and NOT released
Commit only. Each part can be released alone; none needs release 2b. No `configs/` change in this section.
- **Fixes after an independent review of the night's code (2026-11-14; felhom.eu
`documentation/audits/night-build-2026-11-14/review/`).** (1) `config.FileMu`: the three functions that read
`agent.json` whole and write it whole (the trust move, the PBS seed, the backup-target move) take one lock — two of
them could interleave and the second put back the document it read before the first one's change; pinned from the
source by `TestAgentJSONWritersTakeTheSharedLock`. (2) `pvetrust.Manager.Ensure` runs one at a time (start, timer,
a failed collect and a bundle install could overlap, and the second parked the already-moved file as the „before”
copy), and its root step has a 90-second deadline (it runs on the start path and inside the report loop). (3) **The
CA copy keeps its previous certificate behind the new one** (`configs/felhom-os-apply`, mode `pve-ca-copy`): the
copy was replaced before anything could be proven with the new root, and Proxmox serves its old certificate until
pveproxy is reloaded — an agent that restarted in that window had no root that verified it. **Only for the
changeover:** at every later run the step makes one handshake to the local Proxmox with the NEW root alone, and
drops the previous root as soon as that verifies, or when the previous root is no longer a CA that has not ended.
The copy is compared exactly, so nothing the step did not check stays in it. To drop the old root at once by hand
(a root changed because it leaked): delete the copy. `configs/` changes again, so the release needs both signed jobs.
(4) The profile-refresh stanza is built from the wrapper's ANSWER (its record of the last refresh comes back at
every ask); the record file is a bonus — its folder is root-only on a box where a signed job made it first (read in
source, not on a box). `rewritten` is absent when not known. (5) A watchdog give-up is forgotten when the guest is
no longer the watchdog's (onboot off, locked, destroyed) — it was reported for ever. (6) `/disks` sends both boot
tokens from ONE read of the guest's init (`GuestBootTokens`). 13 mutations seen red, two after a mutant that did
not compile was done again.
- **The hub is told what the kernel clean-up did and what the profile refresh did (R-1383; felhom.eu `11` §5.11.1,
§5.12).** (1) The clean-up's report carries `kernel_cleanup` (`osupdate.CleanupDetail`: removed, kept, bytes_gained,
leftovers). It still has no `kernel` view and `release_id` stays `kernel-cleanup` — the hub reads only those two as
„the kernel this step was for” (the test asserts the wire form). (2) The host report carries `profile_refresh`
(`osupdate.ProfileRefreshStatus`, from `Leg.ProfileRefreshStanza`, wired in `main.go`): the root wrapper's own record
of the last refresh — so it outlives an agent restart — and how this process's last ask ended. One small file read,
no wrapper call, names checked and capped. No OS report is sent for the refresh (an unknown layer reads as a guest
step on the hub). Tests `TestKernelCleanup_NightOrderOnceAndNoNames` (extended), `TestProfileRefreshStanza_RecordAndLastAsk`,
`TestMainWiresPVETrust`; 9 mutations seen red (one mutant did not compile at first and was done again).
- **A boot token that a clock step cannot move (R-1395).** `/disks` answers `guest_boot_key` =
`<kernel boot_id>-<guest init start>` beside the unchanged `guest_boot_id` (`<boot TIME>-<guest init start>`). The
kernel computes the boot time from the wall clock, so a step of the host's clock moved the old token and the
controller recreated every drive-backed app (the N100, 2026-11-14, +32.7 s). `GuestBinder.GuestBootKey`; `""` when
the guest does not run or the boot_id is unreadable. Only a controller that knows the key (built there, not
released) reads it. Tests `TestGuestBootKey_AClockStepDoesNotMoveIt`, `TestDisks_CarriesBothBootTokens`; 6 mutations
seen red (felhom.eu `documentation/audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt`; two mutants did not
compile at first and were done again).
- **A refused signed job and a watchdog give-up reach the hub (R-1383, D78, felhom.eu `09` §3 decision 299; `05`
§16.6).** (1) `signedjobs.Runner` clears a job with `hub.Client.CompleteJobWithOutcome`: `outcome` (done · rejected
· failed · malformed), a `reason` from the closed `signedjobs.ReasonCodes` (the two clock reasons `expired` and
`not_yet_valid` first — a box with a wrong clock refuses every job) and the op's name, in the `DELETE`'s query. No
signature, nonce or job parameter is sent. (2) The guest-power watchdog's give-up — one ERROR line in the journal
until now — rides the host report as `guest_power_gave_up` (`localapi.Server.GuestPowerGaveUpStanza`, wired in
`main.go`); the list empties when the guest runs. Tests `TestRunner_TellsTheHubHowEachJobEnded` (the real gate
and verifier), `TestReasonCode_ClosedList`, `TestCompleteJob_OutcomeRidesTheQuery`,
`TestGuestPower_GiveUpIsReportedUntilTheGuestRuns`, `TestMainWiresPVETrust`; 12 mutations seen red (felhom.eu
`documentation/audits/night-build-2026-11-14/B/redproof-agent-{job-outcome,guest-power}.txt`).
- **The report carries the box's clock at the moment of sending (R-1384, D79, felhom.eu `09` §3 decision 300).**
`HostReport.SentAt` (`sent_at`, RFC3339 with nanoseconds) is stamped by `Client.Report` at every send, never by the
collector. `reported_at` could not serve: it is stamped when the collect starts. The hub (its own unreleased part)
compares it with its clock and alarms above one minute. Test `TestReport_StampsSentAtAtSending`; 3 mutations seen
red (felhom.eu `documentation/audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt`).
## Unreleased (2026-11-14 night) — release 2b, built and NOT released: trust in Proxmox by a copy of the box's own root CA (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a)
Commit only. No box ran any of this; the proof on a box (demo-hp, a forced new certificate, twice) is the next attended
session's. `configs/` changes, so the release needs BOTH signed jobs.
- **The root step that keeps the copy** (`configs/felhom-os-apply`, host-layer mode `pve-ca-copy`). It copies exactly
`/etc/pve/pve-root-ca.pem` to exactly `/etc/felhom-agent/pve-root-ca.pem` (0644 root), only if the source is ONE
certificate with `CA:TRUE` that has not ended; written whole or not at all; „same" when nothing changed. The plan may
name nothing (T1); no source (T2); not one CA certificate (T3). No appliance check: the copy is the agent's own trust
file and changes nothing of a BYO owner's host. Tests `PVECACopy` (9, one with the real openssl through the real
runner); 7 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-wrapper.txt`).
- **The move of a box that exists, in a safe order** (`internal/pvetrust`, new; wired in `cmd/felhom-agent/main.go`
at the agent's start, after a config bundle install, on a timer — hourly until moved, daily after — and on every
call to Proxmox that fails on trust). (1) the root step makes the copy; (2) ONE real call to Proxmox with the copy
ALONE, while the pinned fingerprint is still the configured trust; (3) only then ONE write of `agent.json` sets
`proxmox.tls.ca_file` and clears `fingerprint` (the agent's own file, rewritten in place after a recovery copy
`agent.json.pre-pve-trust` was parked, read back and checked); (4) the RUNNING client takes the new trust
(`proxmox.Client.SetTLS`, new) — no restart. A failed copy, proof or write changes nothing: the fingerprint stays.
**`proxmox.tls` keeps its order (a set fingerprint wins over a CA file), so the fingerprint is cleared LAST.** An
operator's own CA file, a box with checks switched off and a config that is not a file are never moved.
- **A pinned box whose certificate changed heals by itself:** the failed call starts the same order, the box moves to
the CA copy and the next call works. A refreshed copy (a renewed root CA) is proven and reloaded by the running
client.
- **The cause is named** (`proxmox.TrustProblem`, new): the pinned certificate changed; the CA copy did not sign the
served certificate; a certificate ended; the copy is unreadable; the copy ends within 30 days; the root step gave no
copy; the proof failed. An ERROR line in the journal, and the new `proxmox_trust` stanza of the host report
(`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`). **The hub does not read the stanza yet**
(felhom.eu R-1396), and a box that is fully blind sends no host report at all — that case still shows only as
„host stale" on the hub.
- Tests: `internal/pvetrust` (11, against a real TLS server that changes its certificate and renews its CA), the AST
wiring test `TestMainWiresPVETrust`; 12 + 3 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-agent.txt`;
one mutation is not a rule and says so).
## v0.174.0 (2026-11-14) — the box removes its old kernels (R-1368); security profiles are refreshed after a kernel change (R-1388, R-1301); Go 1.27.4 (R-1391) — RELEASED 2026-11-14 (release-agent.sh from the clean working clone at `cc63dd5`, CI run id 2641; agent sha256 6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92, bundle sha256 7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f) and VOUCHED 2026-11-14 15:21:45 CET
Release 2a of the build order (felhom.eu `09` §3 decision 306). Design: felhom.eu `11-os-updates.md` §5.11.1 and §5.12.
`configs/` changed (the wrapper), so a box needs BOTH signed jobs: `agent_update`, then `agent_config_update`.
**Part A — old kernels (`09` §3 decision 286).** Nothing removed old kernels; a ring-0 box got about one a week, 1 GB
each. Now:
- **The wrapper** (`configs/felhom-os-apply`, layer `kernel`, new mode `kernel-cleanup`): the hand-proven rule of
2026-11-10 as a program. Keep the running kernel, the step's `from`, `to` and `default_before`, both defaults, the
flag's kernel, an operator's boot pin and the newest installed; remove every other installed kernel image in ONE
`apt-get purge` by exact name. Refusals P1, P2, P3, K1, N1, S1, S2 (the design's table), C1 (the plan names
something — it may carry only `dry_run`), R9, R12. „Not run" (exit 0): no root-owned ring-0 mark, no kernel step yet,
the newest step did not end `good`, the box does not run its newest kernel, a step within the last 20 hours. K1 reads
the box a second time, by itself. Repair-first. After the purge it proves the default, the pinned file, the two
generators and the state file unchanged; a left-over folder is reported, never deleted. **Rule R4 and every update
plan are unchanged; `apt-get autoremove` is not used.**
- **The agent** (`internal/osupdate/kernel.go`): the clean-up is the night leg's last call, after a healthy host step,
never in a night with a kernel step, never in a debug pass, never on a BYO host. The hub hears `cleaned`, `refused`
or `failed` (a kernel-layer report with NO kernel version — the hub would read that as a kernel step); „nothing to
remove" and „not run" are log lines. `--selftest=kernel-cleanup-dry-run` prints what it would do and changes nothing.
- **No new sudo line.** The wrapper is reached by the one `--plan` line (pinned by `test_cleanup_needs_no_new_sudo_line`).
**Part B — security profiles (spike first: felhom.eu `audits/profile-refresh-spike-2026-11-14/`).** After a kernel
went back and forth a box can load an old compiled AppArmor profile; on the N100 this stopped the time service.
- **The wrapper**, new host-layer mode `profile-refresh` (an appliance only; no package change; the plan names
nothing, F1): once per kernel the box starts on (a root-owned record), `apparmor_parser --replace --skip-read-cache
--write-cache -- /etc/apparmor.d`; then ONE restart for a `failed` service whose own program's cached profile the
rewrite changed (at most 3). Measured on demo-hp: 0.5 s, no guest or container touched.
- **The agent** (`internal/osupdate/profilerefresh.go`) asks at every start and after a config bundle install, under
the heavy-op gate; nothing is sent to the hub.
**Part C — Go 1.27.4** (GO-2026-6859; hygiene: the agent's one `MaxBytesReader` place does not match the advisory).
**Tests, each seen red** (felhom.eu `audits/release-2a-2026-11-14/A/redproof*.txt`, `B/redproof*.txt`): 31 wrapper
cases for the clean-up, one per acceptance scenario and one per kept-kernel kind (14 mutations, all red); 11 for the
profile refresh (10 mutations); 5 + 4 agent tests (3 + 3 mutations). Two existing agent tests list the night's calls
exactly and gained the clean-up call (`TestRing0_OneCallPerLayer`, `TestRing1_NoReleaseIsInventory`).
**Delivered 2026-11-14 to the three test boxes by signed `agent_update` + `agent_config_update` each** (Tester 1,
demo-hp, the N100): update committed, bundle installed (written 1 of 31), capability probe 77/77, the running binary
reads `go1.27.4`. **On boxes** (felhom.eu `audits/release-2a-2026-11-14/D/`):
- **Profile refresh, by the product's own path, on all three:** refused by the OLD wrapper at the agent's start
(R11, logged), then run right after the bundle install: 0.5 s each, nothing restarted on Tester 1 and demo-hp.
**On the N100 it repaired the time service:** the cached chrony profile was rewritten, `chrony.service` (failed
since 2026-10-16) was restarted and is active, the clock stepped +32.7 s; offset to DooPlex −32.55 s → +0.18 s. After
a restart of Tester 1 the agent answered „not needed" (the same kernel).
- **Clean-up:** dry runs through the released agent on all three (demo-hp: nothing to remove; the N100: 6 of 8 will
go tonight; Tester 1: „not run", it has no ring-0 mark). Refusals P1, P2 and P3 by dry run on demo-hp against states
set and put back. **A real removal on Tester 1**, with the ring-0 mark set for the test on the operator's yes and
put back: `proxmox-kernel-7.0.14-20-pve-signed` removed, 1.12 GB gained, 8.5 s, the default, the pinned file, the
generators and the state file unchanged, no left-over; the second call removed nothing; one orderly restart came
back on the same kernel, healthy, a clean stop for the crash guard. **This real run was started by hand** (the
agent user, the one sudo line, the night leg's own plan), not by the night leg.
- **Not proven:** the step by the agent at night (the N100's and demo-hp's own night, 2026-11-15); a box whose running
kernel is not its newest and a self-revert into the kept kernel (tests only); refusal K1 on a box (it guards a wrong
computation; tests only); a restart by the profile refresh of any program but chrony.
- **Found:** a clock step makes the CONTROLLER read a new boot (its boot identity is `/proc/stat` btime): on the N100
it recreated its three drive-backed apps 9 s after the step (felhom.eu R-1395). The crash guard (kernel `boot_id`)
was not fooled.
**Vouched by the operator 2026-11-14 15:21:45 CET** (the hub log: `Artifact manifest set: agent=0.174.0`, the bundle
sha = the release; the hub's System page: „vouched agent: 0.174.0"). `pinned.day0_agent` = 0.174.0 (D50).
## Unreleased (2026-11-11) — docs only: a repository address holds no credential (R-925, Part C of the clock/end-dates run)
- `scripts/release-agent.sh` header: the tag push needs `GITEA_GIT_TOKEN` from the secret store (`sec run`). The origin
+2
View File
@@ -1,5 +1,7 @@
# CONTEXT — felhom-agent working state
> **2026-11-14 (day, attended) — v0.174.0 RELEASED and VOUCHED (15:21:45 CET); on the three test boxes (agent + bundle, 77/77).** sha `6efbb978…`, bundle `7d87d058…`. Two new wrapper modes: `kernel-cleanup` (layer kernel; old kernel images by the hand-proven rule; ring-0 mark only; the plan names nothing; the night leg's last call, never with a kernel step) and `profile-refresh` (layer host; AppArmor profiles from a fresh compile once per kernel; one restart for a failed service whose cached profile was stale). Go 1.27.4. The N100's time service was repaired by the product path (+32.7 s step). A real kernel removal ran on Tester 1 by hand with a temporary ring-0 mark (operator's yes). **Next:** read the N100's and demo-hp's night (the clean-up by the agent at night is not proven yet); a ruling for ring-1 boxes; release 2b (trust in Proxmox by the root CA copy, D80). Report: felhom.eu `REPORT-release-2a-2026-11-14.md`.
> **2026-11-10 (afternoon, attended) — v0.173.0 RELEASED and VOUCHED (16:29:16 CET); on the three test boxes.** sha `752b2727…`, bundle `51ebc1f4…` (`configs/` unchanged since 0.172.1, so no bundle job). Measured: a box cleans its self-update folder at the commit of the update that BRINGS the code (49 → 1, 48 → 1, 33 → 1 entries; about 16 MB stays). The line below („at the first good update AFTER that release") was wrong. A rollback after the clean-up was proven on Tester 1 with a signed update to 0.172.1 and one crash in the dwell. `pinned.day0_agent` = 0.173.0. Open: the `.bak-*` files (R-1368 part 4).
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
+24 -49
View File
@@ -1,56 +1,31 @@
# REPORT — v0.173.0 released, vouched and delivered to the three test boxes (2026-11-10, attended)
# REPORT — v0.174.0 released, vouched and delivered to the three test boxes (2026-11-14, attended)
**Plain words first.** Each agent update left its downloaded file on the box for ever. Version 0.173.0 removes the old
files after an update that ended well. It is released and it runs on the three test boxes. The folders went from
49, 48 and 33 files to 1 file each. A rollback still works after the clean-up. The operator vouched it at 16:29 CET.
**Plain words first.** A box now removes its old kernels by itself, and it reloads its security profiles after it
starts on another kernel. Version 0.174.0 is released and runs on Tester 1, demo-hp and the N100. The operator vouched
it at 15:21 CET. The N100's time service runs again; the product repaired it, with no hand step.
The full report of the run is in `felhom.eu`: `REPORT-release-space-fixes-2026-11-10.md`.
The full report of the run is in `felhom.eu`: `REPORT-release-2a-2026-11-14.md`.
## One claim that was wrong
## What a reader must know
The CHANGELOG entry and the report of the morning said: a box cleans its folder at the first good update AFTER the
release. That was wrong. The commit runs in the new binary. So the box cleans the folder at the commit of the update
that brings the code. Measured on three boxes; the CHANGELOG entry is corrected.
## The release
1. **The clean-up runs on a box with the root-owned ring-0 mark only.** Tester 1 has no mark, so the program answers
„not run" there. For the proof the operator allowed the mark on Tester 1 for some minutes; it is back as before.
2. **The real removal on Tester 1 was started by hand**, with the agent's own command line. The night leg cannot be
started by day: it follows only the night's own backup (R-899). So „the step by the agent at night" is **not proven
yet**; the N100's and demo-hp's night of 2026-11-15 is the proof.
3. **The profile refresh was spiked first** on demo-hp (felhom.eu `audits/profile-refresh-spike-2026-11-14/`). Three
things it found are built in: a stale copy is told by the cached file changing, not by the kernel's hash; one
profile never compiles the same twice; a refresh hides other kernel audit lines for a few seconds.
4. **A clock step makes the controller read a new boot** (felhom.eu R-1395, a controller row): on the N100 it
recreated three apps once.
| | |
|---|---|
| Version | 0.173.0, `scripts/release-agent.sh` from a clean clone of `776025c8` (tag `v0.173.0`) |
| Agent sha256 | `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea` |
| Bundle sha256 | `51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` |
| CI | `776025c8`: run 2608, `gates #311`, success (read by commit on the public Actions page) |
| Delivery | one signed `agent_update` per box. No bundle job: `configs/` did not change since v0.172.1 |
| Vouch | the operator, 2026-11-10 16:29:16 CET. Read from the hub log: `Artifact manifest set: agent=0.173.0`, bundle sha = the release. `pinned.day0_agent` = 0.173.0 (D50) |
## The folder `/var/lib/felhom-agent/selfupdate`, before and after
| Box | Before (on 0.172.1) | After the commit of 0.173.0 | Commit (UTC) |
|---|---|---|---|
| demo-hp | 49 entries, 722,596 KB | 1 entry, 16,028 KB | 14:59:19, `removed=49` |
| demo-felhom | 48 entries, 706,968 KB | 1 entry, 16,028 KB | 14:51:50, `removed=48` |
| Tester 1 | 33 entries, 497,244 KB | 1 entry, 16,028 KB | 15:00:19, `removed=33` |
After each commit: `pending.json` absent, `/usr/local/bin/felhom-agent.prev` = 0.172.1, the unit active, no restart.
## The rollback proof (Tester 1)
1. A signed `agent_update` to 0.172.1 (the vouched version) was uploaded at 15:01:19 UTC. The box took it at its next
report, 15:14:31 UTC. The wrapper installed 0.172.1 and wrote the pending marker.
2. At 15:14:40 UTC, inside the 60-second dwell, one `systemctl kill -s SEGV` hit the agent's main process.
3. The unit's `OnFailure` ran the wrapper's `rollback`. Journal: `rolled back to previous binary and restarted
felhom-agent.service`. 0.173.0 started at 15:14:40 UTC.
4. Read at 15:15:56 UTC: agent 0.173.0, active, `pending.json` absent, no failed unit. The folder holds 2 files
(32,040 KB): the running version's and the failed 0.172.1. The failed file stays by design, until the next good
update.
This is the path the repo already has. No second version was released for the proof.
## What stays open
- `/usr/local/bin/felhom-agent.bak-*` (R-1368 part 4): demo-felhom holds 11 such files, demo-hp 1. The agent cannot
reach them.
- After a rollback, `/usr/local/bin/felhom-agent.prev` holds the same version as the running binary. That is how the
wrapper works; it is not new.
Evidence: felhom.eu `documentation/audits/release-space-fixes-2026-11-10/agent/`.
| Version | 0.174.0, sha256 `6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92`, bundle `7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f`, Go 1.27.4 |
| Code commit, CI | `cc63dd5`, run id 2641 success |
| Tests | wrapper 191 (42 new), agent `internal/osupdate` green; 30 mutations, each seen red (felhom.eu `audits/release-2a-2026-11-14/A`, `B`) |
| Tester 1 | 3 kernels → 2, 1.12 GB gained, 8.5 s; second run: nothing; one restart, same kernel, healthy, clean stop |
| demo-hp | nothing to remove; refusals P1, P2, P3 by dry run |
| The N100 | time service repaired (offset −32.55 s → +0.18 s); tonight it removes 6 of 8 kernels |
| Vouch | the operator, 2026-11-14 15:21:45 CET; `pinned.day0_agent` = 0.174.0 |
| Not proven | the step at night by the agent; a fell-back box and a self-revert into the kept kernel on a box; K1 on a box |
+3
View File
@@ -62,6 +62,7 @@
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
| `GuestBinder.GuestBootKey` (R-1395) | internal/localapi/intermediary.go | `GuestBootKey(ctx, vmid) string` | "is this a new boot of the guest?" — `<kernel boot_id>-<guest init start>` | **`GuestBootID` carries `/proc/stat` `btime`, which the kernel computes from the wall clock: a clock STEP moves it.** Use the key for any „same boot?” question; `""` = unreadable. Seams `hostBtimeFn` / `hostBootIDFn` / `procStarttimeFn` |
| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
@@ -100,6 +101,8 @@
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
| `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. |
| `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. |
| `pvetrust.Manager` (`Ensure`, `OnCallError`, `Status`) + `proxmox.Client.SetTLS` / `proxmox.TrustProblem` + `Leg.PVECACopy` + the wrapper's `pve_ca_copy` | internal/pvetrust/pvetrust.go; internal/proxmox/{client,trust}.go; internal/osupdate/pvecacopy.go; configs/felhom-os-apply (mode `pve-ca-copy`) | `Ensure(ctx, why) (changed bool)`; `TrustProblem(err) string` | R-1380 (felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA; the move from a pinned certificate; the cause of a trust failure | THE ORDER: the copy, a real call with the copy ALONE, then ONE config write that clears the fingerprint LAST (a set fingerprint wins over `ca_file` in `TLSConfig.build`). Never clear the fingerprint anywhere else. The wrapper's plan names nothing (T1). `agent.json` is written by the agent in place, with a parked recovery copy — the same way as the backup-target move. |
| `config.ReplaceInPlace` / `config.RecoverCutWrite` (+ `config.FileMu`) | internal/config/cutwrite.go; internal/config/filelock.go | `ReplaceInPlace(path, prev, next) error`; `RecoverCutWrite(path) (note string, err error)` | R-1380 (felhom.eu `03` §12a item 7): the only way to write `agent.json` — the folder is root-owned, so no rename; the old bytes go to a journal in the state folder first, and the daemon repairs a cut write at start, before it reads the config | A new writer takes `FileMu`, calls `ReplaceInPlace`, opens no file by hand, and is added to the two source pins in `cmd/felhom-agent` (`…TakeTheSharedLock`, `…UseTheJournalledWrite`). A package whose tests write the config sets `config.CutWriteDir` in `TestMain` |
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
+141
View File
@@ -0,0 +1,141 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// A box whose agent.json was cut in the middle of a write must start (R-1380): main repairs the file from the
// journal BEFORE it reads it. The staging is the state a power cut leaves: a durable journal of the old bytes and a
// half-written config.
func TestMainRecoversACutConfigWrite(t *testing.T) {
dir := t.TempDir()
prev := config.CutWriteDir
config.CutWriteDir = filepath.Join(dir, "state")
t.Cleanup(func() { config.CutWriteDir = prev })
if err := os.MkdirAll(config.CutWriteDir, 0o700); err != nil {
t.Fatal(err)
}
cfgPath := filepath.Join(dir, "agent.json")
old := `{"proxmox":{"node":"pve","tls":{"fingerprint":"AA:BB"}}}`
if err := os.WriteFile(filepath.Join(config.CutWriteDir, "agent.json.write-in-flight"), []byte(old), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(cfgPath, []byte(`{"proxmox":{"node":"pv`), 0o600); err != nil {
t.Fatal(err)
}
if _, err := config.Load(cfgPath); err == nil {
t.Fatal("the staging is wrong: the cut config loads")
}
note, err := recoverCutConfigWrite(cfgPath)
if err != nil || note == "" {
t.Fatalf("note=%q err=%v", note, err)
}
cfg, err := config.Load(cfgPath)
if err != nil {
t.Fatalf("the agent does not start after the repair: %v", err)
}
if cfg.Proxmox.TLS.Fingerprint != "AA:BB" {
t.Fatalf("the repaired config lost the fingerprint: %+v", cfg.Proxmox.TLS)
}
}
// main must call the repair before config.Load, or the repair never runs on the box that needs it (a seam that is
// built and not wired). Read from the source: the call's position in main's body.
func TestMainCallsTheConfigRepairBeforeItLoadsTheConfig(t *testing.T) {
src, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
f, err := parser.ParseFile(token.NewFileSet(), "main.go", src, 0)
if err != nil {
t.Fatal(err)
}
var repair, load token.Pos
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != "main" || fn.Body == nil {
continue
}
ast.Inspect(fn.Body, func(n ast.Node) bool {
c, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "recoverCutConfigWrite" && repair == 0 {
repair = c.Pos()
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Load" {
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" && load == 0 {
load = c.Pos()
}
}
return true
})
}
if repair == 0 || load == 0 {
t.Fatalf("main: repair call found=%v, config.Load found=%v", repair != 0, load != 0)
}
if repair > load {
t.Fatal("main reads the config BEFORE it repairs a cut write — a cut config stops the agent at start")
}
}
// Every writer of agent.json goes through config.ReplaceInPlace and none truncates the file itself: a writer that
// opens the config with O_TRUNC by hand is the unprotected write again.
func TestAgentJSONWritersUseTheJournalledWrite(t *testing.T) {
for _, w := range []struct{ file, fn string }{
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
} {
src, err := os.ReadFile(w.file)
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, w.file, src, 0)
if err != nil {
t.Fatal(err)
}
found := false
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
continue
}
found = true
body := string(src[fset.Position(fn.Body.Pos()).Offset:fset.Position(fn.Body.End()).Offset])
calls := 0
ast.Inspect(fn.Body, func(n ast.Node) bool {
if c, ok := n.(*ast.CallExpr); ok {
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "ReplaceInPlace" {
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" {
calls++
}
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "OpenFile" {
t.Errorf("%s %s opens a file by hand — the config is written only by config.ReplaceInPlace", w.file, w.fn)
}
}
return true
})
if calls != 1 {
t.Errorf("%s %s: %d call(s) to config.ReplaceInPlace, want 1", w.file, w.fn, calls)
}
if strings.Contains(body, "O_TRUNC") {
t.Errorf("%s %s truncates a file itself", w.file, w.fn)
}
}
if !found {
t.Errorf("%s: function %s not found — the pin reads nothing", w.file, w.fn)
}
}
}
@@ -0,0 +1,62 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// Three functions in this process read agent.json whole, change one section and write it whole: the trust move, the
// PBS seed and the backup-target move. Without one shared lock two of them interleave and the second puts back the
// document it read before the first one's change (found by review, 2026-11-14). Each must take config.FileMu as its
// FIRST statement and release it by defer — asserted from the source, because the race itself is not a thing a unit
// test can stage reliably.
func TestAgentJSONWritersTakeTheSharedLock(t *testing.T) {
for _, w := range []struct{ file, fn string }{
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
} {
f, err := parser.ParseFile(token.NewFileSet(), w.file, nil, 0)
if err != nil {
t.Fatalf("%s: %v", w.file, err)
}
found := false
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
continue
}
found = true
isLockCall := func(e ast.Expr, name string) bool {
c, ok := e.(*ast.CallExpr)
if !ok {
return false
}
sel, ok := c.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != name {
return false
}
inner, ok := sel.X.(*ast.SelectorExpr)
if !ok || inner.Sel.Name != "FileMu" {
return false
}
pkg, ok := inner.X.(*ast.Ident)
return ok && pkg.Name == "config"
}
if len(fn.Body.List) < 2 {
t.Errorf("%s %s: too short to hold the lock", w.file, w.fn)
continue
}
first, ok1 := fn.Body.List[0].(*ast.ExprStmt)
second, ok2 := fn.Body.List[1].(*ast.DeferStmt)
if !ok1 || !isLockCall(first.X, "Lock") || !ok2 || !isLockCall(second.Call, "Unlock") {
t.Errorf("%s %s: must start with config.FileMu.Lock() and defer config.FileMu.Unlock()", w.file, w.fn)
}
}
if !found {
t.Errorf("%s: function %s not found — a writer of agent.json was renamed; keep it under the shared lock and update this test", w.file, w.fn)
}
}
}
+67
View File
@@ -54,6 +54,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfheal"
@@ -216,6 +217,15 @@ func main() {
return
}
// Only the daemon repairs: a selftest can run beside a live agent that is in the middle of a write.
var cutWriteNote string
var cutWriteErr error
if selftest.mode == "" {
cutWriteNote, cutWriteErr = recoverCutConfigWrite(cfgPath)
if cutWriteErr != nil {
fmt.Fprintln(os.Stderr, "config repair:", cutWriteErr)
}
}
cfg, err := config.Load(cfgPath)
if err != nil {
// A missing default config file is fine if env provides the values; only a
@@ -229,6 +239,11 @@ func main() {
// logRing is the always-DEBUG capture ring (v0.83.0 observability): served by the
// local API's GET /debug/logs and the heartbeat log-pull. Selftests ignore it.
logger, logRing := applog.New(cfg.LogLevel)
if cutWriteErr != nil {
logger.Error("config: the last write of the config was cut and could not be repaired", "err", cutWriteErr)
} else if cutWriteNote != "" {
logger.Warn("config: a write of the config was cut by a stop — repaired at start", "result", cutWriteNote)
}
switch selftest.mode {
case "":
@@ -988,6 +1003,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
osLeg := newOSLeg(cfg, client, px, logger)
collector.SetSystemReporter(&factsReporter{leg: osLeg, guest: firstGuest(px)}) // R-852: the versions
desiredSyncer.AddConsumer(osLeg)
// R-1380 (D80, felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA. At this start the root
// step makes the copy, a REAL call proves it while the pinned fingerprint is still the trust, and only then ONE
// write of the config sets ca_file and clears the fingerprint. A call that fails on trust later names the cause and
// runs the same order again (a pinned box whose certificate changed moves right then, with no restart).
pveTrust := newPVETrust(cfg, px, osLeg, logger)
collector.SetProxmoxTrustSource(func() any { return pveTrust.Status() })
loop.SetCollectErrorHook(pveTrust.OnCallError)
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
@@ -1018,6 +1040,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1380: the trust order at this start (the copy, the proof, then the config) — before the first report cycle, so
// a pinned box whose certificate changed while the agent was down is not blind for a cycle. Then on a timer.
pveTrust.Ensure(ctx, "start")
go pveTrust.Run(ctx, time.Hour, 24*time.Hour)
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
@@ -1301,9 +1327,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// R-1388: a new wrapper may know the profile refresh the old one refused — ask again (its own goroutine:
// it may wait for the heavy-op gate, and the signed job must not wait with it).
go osLeg.ProfileRefresh(daemonCtx, "config bundle installed", profileGate, 12, 5*time.Minute)
// R-1380: a new wrapper may know the CA-copy step the old one refused — run the trust order again.
go pveTrust.Ensure(daemonCtx, "config bundle installed")
}}
// R-1388 / R-1301 (`11` §5.12): after a start, load the security profiles from a fresh compile when the box runs
// another kernel than at the last refresh (the wrapper keeps that record and answers "not needed" otherwise).
// R-1383 (release 3): the last refresh, and how this process's last ask ended, ride the host report.
collector.SetProfileRefreshSource(osLeg.ProfileRefreshStanza)
go osLeg.ProfileRefresh(daemonCtx, "agent start", profileGate, 12, 5*time.Minute)
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
@@ -1355,6 +1385,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
},
}
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil {
// R-1383 (D78): the guest-power watchdog's give-up rides the host report (it was one journal line).
collector.SetGuestPowerGaveUpSource(localSrv.GuestPowerGaveUpStanza)
}
if localSrv != nil {
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
@@ -3948,6 +3982,29 @@ func (f *selftestFlag) Set(v string) error {
return nil
}
// newPVETrust wires internal/pvetrust (R-1380): the root step through the OS wrapper, the proof as a throwaway client
// with EXACTLY the trust under test, and the running client's SetTLS.
func newPVETrust(cfg config.Config, px *proxmox.Client, leg *osupdate.Leg, logger *slog.Logger) *pvetrust.Manager {
m := &pvetrust.Manager{ConfigPath: cfg.SourcePath, StateDir: agentStateDir(cfg), Logger: logger,
EnsureCopy: func(ctx context.Context) (pvetrust.Copy, error) {
r, err := leg.PVECACopy(ctx)
return pvetrust.Copy{Path: r.Path, Outcome: r.Outcome, NotAfter: r.NotAfter, DaysLeft: r.DaysLeft}, err
},
Probe: func(ctx context.Context, t proxmox.TLSConfig) error {
p, err := proxmox.NewClient(proxmox.Config{Endpoint: cfg.Proxmox.Endpoint, Node: cfg.Proxmox.Node, Token: cfg.Proxmox.Token, TLS: t})
if err != nil {
return err
}
_, err = p.Version(ctx)
return err
},
Apply: px.SetTLS,
}
m.Start(proxmox.TLSConfig{CAFile: cfg.Proxmox.TLS.CAFile, Fingerprint: cfg.Proxmox.TLS.Fingerprint,
InsecureSkipVerify: cfg.Proxmox.TLS.InsecureSkipVerify})
return m
}
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
// the once-per-night marker live in the agent's own os/ dir.
func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg {
@@ -4203,3 +4260,13 @@ func agentStateDir(cfg config.Config) string {
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
}
// recoverCutConfigWrite repairs agent.json BEFORE it is read, when the last write of it was cut by a power loss
// (config.RecoverCutWrite; R-1380). It runs before the logger exists, so main logs what it returns.
// Pinned by TestMainRecoversACutConfigWrite.
func recoverCutConfigWrite(cfgPath string) (string, error) {
if cfgPath == "" {
return "", nil
}
return config.RecoverCutWrite(cfgPath)
}
+71
View File
@@ -0,0 +1,71 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-1380: the trust move is a component with seams (the root step, the proof, the running client, the report stanza,
// the failed-call hook). A unit-green pvetrust that main.go never starts is the inert-seam defect, so the wiring is
// asserted from the AST: built once, run at start and after a bundle install, on a timer, fed by a failed collect, and
// reported. The same for the profile refresh (R-1388), which had no such test.
func TestMainWiresPVETrust(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
seen := map[string]int{}
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
sel, ok := node.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
x, _ := sel.X.(*ast.Ident)
switch {
case x != nil && x.Name == "pveTrust" && sel.Sel.Name == "Ensure":
seen["ensure"]++
case sel.Sel.Name == "SetProxmoxTrustSource":
seen["stanza"]++
case sel.Sel.Name == "SetProfileRefreshSource" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "ProfileRefreshStanza" {
seen["profile refresh → report"]++ // R-1383
}
case sel.Sel.Name == "SetGuestPowerGaveUpSource" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "GuestPowerGaveUpStanza" {
seen["guest-power give-up → report"]++ // R-1383
}
case sel.Sel.Name == "SetCollectErrorHook" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "OnCallError" {
seen["hook"]++
}
case x != nil && x.Name == "osLeg" && sel.Sel.Name == "ProfileRefresh":
seen["profile-refresh"]++
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "pveTrust" {
seen["timer"]++
}
}
case *ast.KeyValueExpr:
if k, ok := node.Key.(*ast.Ident); ok && k.Name == "Apply" {
if v, ok := node.Value.(*ast.SelectorExpr); ok && v.Sel.Name == "SetTLS" {
seen["apply=SetTLS"]++
}
}
}
return true
})
want := map[string]int{"ensure": 2, "stanza": 1, "hook": 1, "timer": 1, "apply=SetTLS": 1, "profile-refresh": 2,
"guest-power give-up → report": 1, "profile refresh → report": 1}
for k, n := range want {
if seen[k] < n {
t.Errorf("main.go wires %q %d time(s), want at least %d (the two-call ones: at start and after a bundle install)", k, seen[k], n)
}
}
}
+113 -1
View File
@@ -161,6 +161,14 @@ KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good",
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# The clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1). A step that ENDED in one of these phases
# and is not "good" keeps every kernel (the box fell back or reverted); any other phase is "not ended" (P1).
# Trust in Proxmox by a copy of the box's own root CA (R-1380, `09` §3 decision 303, felhom.eu `03` §12a). The agent's
# user cannot read Proxmox's CA file (0640 root:www-data); this root step keeps a readable copy of that ONE public
# certificate where the agent's `proxmox.tls.ca_file` points.
PVE_CA_SOURCE = "/etc/pve/pve-root-ca.pem"
PVE_CA_COPY = "/etc/felhom-agent/pve-root-ca.pem" # 0644 root: a CA certificate is public; the agent only reads it
PVE_CA_MAX_BYTES = 64 * 1024
PVE_LOCAL_API = "127.0.0.1:8006" # pveproxy on this host (the installer's proof uses the same)
PEM_CERT_RE = re.compile(rb"-----BEGIN CERTIFICATE-----\s[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----")
# The security-profile refresh (R-1388, R-1301, `11` §5.12; measured on demo-hp 2026-11-14,
# felhom.eu audits/profile-refresh-spike-2026-11-14/). AppArmor keeps one cache folder per kernel feature set; a
# profile that changed while ANOTHER kernel ran leaves an old compiled copy in this kernel's folder, with a time that
@@ -514,7 +522,7 @@ class Apply:
def check_plan(self, plan):
mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
"oom-check", "profile-refresh") \
"oom-check", "profile-refresh", "pve-ca-copy") \
+ KERNEL_MODES:
raise Refused("R11", f"unknown mode {mode!r}")
if mode == "agent_update":
@@ -529,6 +537,15 @@ class Apply:
if plan.get("layer") != "host":
raise Refused("R11", "bundle-policy is a host-layer mode")
return mode, "host", 0, "bundle-policy"
if mode == "pve-ca-copy":
if plan.get("layer") != "host":
raise Refused("R11", "pve-ca-copy is a host-layer mode")
# R-1380 (D80): ONE fixed source, ONE fixed place — the plan may name NOTHING (no path, no file, no content)
extra = sorted(k for k in plan if k not in ("mode", "layer", "release_id", "run_id", "trigger", "ring"))
if extra:
raise Refused("T1", f"a pve-ca-copy plan carries {', '.join(extra)} — it takes nothing from a plan: "
f"the source is {PVE_CA_SOURCE} and the copy is {PVE_CA_COPY}, both fixed")
return mode, "host", 0, "pve-ca-copy"
if mode == "profile-refresh":
if plan.get("layer") != "host":
raise Refused("R11", "profile-refresh is a host-layer mode")
@@ -1124,6 +1141,10 @@ class Apply:
return Bundle(self).from_plan(plan)
if self.mode == "bundle-policy":
return Bundle(self).policy_pass()
if self.mode == "pve-ca-copy":
# NO appliance check, on purpose (like the bundle): the copy is the agent's OWN trust file in the agent's own
# config folder, and it changes nothing of the host's — a BYO host's agent talks to Proxmox the same way.
return self.pve_ca_copy()
if self.mode == "profile-refresh":
self.check_appliance()
return self.profile_refresh(plan)
@@ -1192,6 +1213,97 @@ class Apply:
self.report["oom_check"] = self.oom_check()
return 0
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80) ----------
def is_live_ca(self, block):
"""One PEM certificate block: is it a CA certificate that has not ended? (openssl, the bytes on stdin)."""
try:
rc, out, _ = self.r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-checkend", "0"], 30,
stdin=block.decode("ascii") + "\n")
except (UnicodeDecodeError, subprocess.SubprocessError, OSError):
return False
return rc == 0 and re.search(r"\bCA:TRUE\b", out) is not None
def pve_serves_with(self, cafile):
"""Does the local Proxmox API serve a certificate that verifies with this CA file ALONE? One handshake to
PVE_LOCAL_API; nothing is sent. Any doubt (no answer, a timeout) is "no" — the previous root then stays."""
try:
rc, out, err = self.r.host(["openssl", "s_client", "-connect", PVE_LOCAL_API, "-CAfile", cafile, "-no-CApath",
"-verify_return_error"], 20, stdin="")
except (subprocess.SubprocessError, OSError):
return False
return rc == 0 and "Verify return code: 0 (ok)" in (out + err)
def pve_ca_copy(self):
"""Copy exactly PVE_CA_SOURCE to exactly PVE_CA_COPY — only if the source is ONE CA certificate that has not
ended — written whole or not at all (a root-owned temp file beside the target, fsync, rename). "same" when the
copy is exactly the source's bytes. The bytes that are checked are the bytes that are written first; only
during a change of the root the copy also holds its previous certificate behind them (see below)."""
r = self.r
res = {"outcome": None, "path": PVE_CA_COPY}
self.report["pve_ca_copy"] = res
try:
data = r.read_bytes(PVE_CA_SOURCE)
except OSError as e:
raise Refused("T2", f"{PVE_CA_SOURCE} cannot be read ({e.__class__.__name__}) — is this a Proxmox host?")
if not data or len(data) > PVE_CA_MAX_BYTES:
raise Refused("T3", f"{PVE_CA_SOURCE} is {len(data)} bytes — not one certificate")
certs = PEM_CERT_RE.findall(data)
if len(certs) != 1 or b"PRIVATE KEY" in data:
raise Refused("T3", f"{PVE_CA_SOURCE} holds {len(certs)} certificate(s)"
+ (" and a private key" if b"PRIVATE KEY" in data else "") + " — exactly one CA certificate is expected")
try:
text = data.decode("ascii") # PEM is ASCII; the runner hands text to openssl's stdin
except UnicodeDecodeError:
raise Refused("T3", f"{PVE_CA_SOURCE} is not a PEM text file")
rc, out, err = r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-enddate"], 30, stdin=text)
if rc != 0:
raise Refused("T3", f"{PVE_CA_SOURCE} is not a readable certificate: {(out + err).strip()[-160:]}")
if not re.search(r"\bCA:TRUE\b", out):
raise Refused("T3", f"{PVE_CA_SOURCE} is not a CA certificate (no CA:TRUE) — a leaf or any other file is never copied")
m = re.search(r"notAfter=(.+)", out)
try:
import calendar
not_after = calendar.timegm(time.strptime(m.group(1).strip(), "%b %d %H:%M:%S %Y GMT")) # UTC, no DST
except (AttributeError, ValueError):
raise Refused("T3", f"the end date of {PVE_CA_SOURCE} cannot be read")
days = int((not_after - r.now()) // 86400)
if days < 0:
raise Refused("T3", f"{PVE_CA_SOURCE} ended {-days} day(s) ago — an ended CA is never copied")
res.update(sha256=sha256_hex(data), not_after=_iso(not_after), days_left=days)
try:
have = r.read_bytes(PVE_CA_COPY)
except OSError:
have = None
if have == data:
res["outcome"] = "same"
return 0
# The copy the agent trusts by is replaced BEFORE anything can be proven with the new root (Proxmox serves its
# old certificate until pveproxy is reloaded). So at a CHANGE the copy's previous first certificate — written
# by this same step — is kept behind the new one, and the agent trusts both roots across the change, also
# after a restart. That second root is kept ONLY for the changeover: it is dropped as soon as Proxmox serves a
# certificate that verifies with the new root alone, and at once when it is no longer a CA that has not ended.
# The copy is compared EXACTLY: nothing this step did not check is ever left in it. (Review, 2026-11-14.)
# To drop the old root at once by hand (a root that was changed because it leaked): delete the copy.
prev = PEM_CERT_RE.findall(have) if have and b"PRIVATE KEY" not in have else []
if len(prev) == 2 and have == data + prev[1] + b"\n":
over = self.pve_serves_with(PVE_CA_SOURCE)
if not over and self.is_live_ca(prev[1]):
res["outcome"], res["kept_previous"] = "same", True # the changeover is still on
return 0
r.put_file(PVE_CA_COPY, data, 0o644)
res["outcome"], res["replaced"], res["kept_previous"], res["dropped_previous"] = "written", True, False, True
r.log(f"os-apply: PVE CA COPY the previous root is dropped from {PVE_CA_COPY} "
f"({'Proxmox serves a certificate of the new root' if over else 'the previous root is no longer a live CA'})")
return 0
want, kept = data, False
if prev and prev[0].strip() != certs[0].strip() and self.is_live_ca(prev[0]):
want, kept = data + prev[0] + b"\n", True
r.put_file(PVE_CA_COPY, want, 0o644)
res["outcome"], res["replaced"], res["kept_previous"] = "written", have is not None, kept
r.log(f"os-apply: PVE CA COPY written {PVE_CA_COPY} sha256={res['sha256'][:16]} ends={res['not_after']} "
f"({'replaced an older copy' if have is not None else 'first copy'})")
return 0
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
def profile_cache(self):
"""{"<folder>/<file>": sha256} of every cached profile, every kernel's folder."""
+240
View File
@@ -2633,5 +2633,245 @@ class ProfileRefresh(unittest.TestCase):
run(f)
self.assertFalse([c for c in f.calls if c[1][0] in ("apt-get", "dpkg", "apt")], "no package command at all")
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80, felhom.eu `03` §12a) ----------
CA_PEM = b"-----BEGIN CERTIFICATE-----\nQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=\n-----END CERTIFICATE-----\n"
CA_PEM_2 = b"-----BEGIN CERTIFICATE-----\nWllYV1ZVVFNSUVBPTk1MS0pJSEdGRURDQkE=\n-----END CERTIFICATE-----\n"
class TFake(Fake):
"""A Proxmox host: the root CA file, the copy (or none), and openssl's answer about the bytes it is given."""
def __init__(self, source=CA_PEM, copy=None, ca=True, ends="Aug 17 15:44:42 2036 GMT"):
super().__init__()
self.source, self.tree, self.ca, self.ends, self.openssl_rc = source, {}, ca, ends, 0
if copy is not None:
self.tree[osapply.PVE_CA_COPY] = copy
self.openssl_stdin, self.puts = [], []
self.serves_new = False # does pveproxy serve a certificate that verifies with the NEW root alone?
self.dead_ca = set() # certificate texts openssl says are not a live CA (ended)
self.handshakes = []
self.plan = {"release_id": "pve-ca", "layer": "host", "mode": "pve-ca-copy"}
def read_bytes(self, p):
if p == osapply.PVE_CA_SOURCE:
if self.source is None:
raise FileNotFoundError(p)
return self.source
if p in self.tree:
return self.tree[p]
raise FileNotFoundError(p)
def put_file(self, path, data, mode):
self.puts.append((path, data, mode))
self.tree[path] = data
def host(self, argv, timeout=600, stdin=None):
if argv[0] == "openssl" and argv[1] == "s_client":
self.handshakes.append(argv)
return (0, "Verify return code: 0 (ok)\n", "") if self.serves_new else (1, "", "Verify return code: 19 (self-signed certificate in certificate chain)\n")
if argv[0] == "openssl" and "-checkend" in argv:
return (1, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will expire\n", "") if stdin.strip() in self.dead_ca else \
(0, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will not expire\n", "")
if argv[0] == "openssl":
self.calls.append(("host", argv))
self.openssl_stdin.append(stdin)
if self.openssl_rc:
return self.openssl_rc, "", "Could not read certificate from <stdin>"
return 0, ("X509v3 Basic Constraints: critical\n CA:%s\nnotAfter=%s\n" % ("TRUE" if self.ca else "FALSE", self.ends)), ""
return super().host(argv, timeout, stdin)
class PVECACopy(unittest.TestCase):
"""Red-proofs: felhom.eu audits/night-build-2026-11-14/A/redproof-wrapper.txt."""
def refused(self, f, code):
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, code), rep)
self.assertEqual(f.puts, [], "nothing is written on a refusal")
return rep
def test_a_new_box_gets_the_copy(self):
f = TFake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.puts, [(osapply.PVE_CA_COPY, CA_PEM, 0o644)], "ONE file, the fixed place, the source's bytes, 0644")
self.assertEqual(f.openssl_stdin, [CA_PEM.decode()], "the bytes that are checked are the bytes that are written")
c = rep["pve_ca_copy"]
self.assertEqual((c["outcome"], c["path"], c["replaced"]), ("written", "/etc/felhom-agent/pve-root-ca.pem", False))
self.assertEqual(c["not_after"], "2036-08-17T15:44:42Z")
self.assertGreater(c["days_left"], 3000)
def test_the_same_copy_is_not_written_again(self):
f = TFake(copy=CA_PEM)
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
def test_a_renewed_ca_refreshes_the_copy(self):
f = TFake(source=CA_PEM_2, copy=CA_PEM)
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["replaced"]), (0, "written", True), rep)
# the new certificate FIRST, and the copy's previous one kept behind it: Proxmox serves its old certificate until
# pveproxy is reloaded, and an agent that restarts in that window must still have a root that verifies it
bundle = f.tree[osapply.PVE_CA_COPY]
self.assertEqual(bundle, CA_PEM_2 + osapply.PEM_CERT_RE.findall(CA_PEM)[0] + b"\n")
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
self.assertEqual(f.openssl_stdin, [CA_PEM_2.decode()], "the SOURCE is checked in full; the kept one by its own check")
def test_the_previous_root_stays_only_for_the_changeover(self):
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
bundle = CA_PEM_2 + old + b"\n"
# Proxmox still serves the OLD certificate: nothing is written, the old root stays
f = TFake(source=CA_PEM_2, copy=bundle)
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
self.assertEqual(len(f.handshakes), 1)
self.assertIn(osapply.PVE_CA_SOURCE, f.handshakes[0], "the handshake is verified with the NEW root alone (the source file)")
self.assertIn("-no-CApath", f.handshakes[0])
self.assertIn("-verify_return_error", f.handshakes[0])
# pveproxy was reloaded and serves a certificate of the NEW root: the old root is dropped
f = TFake(source=CA_PEM_2, copy=bundle)
f.serves_new = True
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2, "after the changeover the copy is the new root alone")
self.assertEqual((rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["dropped_previous"], rep["pve_ca_copy"]["kept_previous"]), ("written", True, False))
# the old root has ENDED while Proxmox still serves the old certificate: it verifies nothing, so it is dropped
f = TFake(source=CA_PEM_2, copy=bundle)
f.dead_ca = {old.decode()}
rc, rep = run(f)
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
# an ended previous root is not kept at the change itself either
f = TFake(source=CA_PEM_2, copy=CA_PEM)
f.dead_ca = {old.decode()}
rc, rep = run(f)
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
def test_a_copy_with_anything_unchecked_in_it_is_rewritten(self):
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
for name, copy in (("text after the source", CA_PEM_2 + b"anything at all\n"),
("a third certificate", CA_PEM_2 + old + b"\n" + old + b"\n"),
("a key in the file", CA_PEM_2 + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"),
("the two in the wrong order", CA_PEM + osapply.PEM_CERT_RE.findall(CA_PEM_2)[0] + b"\n")):
f = TFake(source=CA_PEM_2, copy=copy)
f.serves_new = True
rc, rep = run(f)
self.assertEqual(rc, 0, (name, rep))
got = f.tree[osapply.PVE_CA_COPY]
self.assertTrue(got.startswith(CA_PEM_2), name)
self.assertNotIn(b"anything", got, name)
self.assertNotIn(b"PRIVATE KEY", got, name)
self.assertLessEqual(len(osapply.PEM_CERT_RE.findall(got)), 2, name)
# a first copy keeps nothing behind it, and makes no handshake
f4 = TFake()
rc, rep = run(f4)
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
self.assertEqual((f4.tree[osapply.PVE_CA_COPY], f4.handshakes), (CA_PEM, []))
def test_a_source_that_is_not_a_ca_certificate_is_refused(self):
self.refused(TFake(ca=False), "T3") # a leaf certificate
self.refused(TFake(source=b"hello\n"), "T3") # not a certificate at all
self.refused(TFake(source=CA_PEM + CA_PEM_2), "T3") # two certificates (a chain, a bundle)
self.refused(TFake(source=CA_PEM + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"), "T3")
self.refused(TFake(source=b""), "T3")
self.refused(TFake(source=CA_PEM + b"x" * (osapply.PVE_CA_MAX_BYTES + 1)), "T3")
f = TFake()
f.openssl_rc = 1 # openssl cannot parse it
self.refused(f, "T3")
self.refused(TFake(ends="Aug 17 15:44:42 2020 GMT"), "T3") # an ended CA
self.refused(TFake(ends="garbage"), "T3")
def test_no_source_is_refused(self):
self.refused(TFake(source=None), "T2")
def test_with_the_real_openssl_and_the_real_runner(self):
"""NOT a fake: a CA and a leaf made by openssl in a temp folder, judged through Runner.host as on a box (the
fake cannot show a wrong argument or a bytes/text mix-up). Skipped where openssl is missing."""
import shutil, tempfile
if not shutil.which("openssl"):
self.skipTest("no openssl here")
d = tempfile.mkdtemp()
try:
def sh(*a):
subprocess.run(a, check=True, capture_output=True, cwd=d)
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "ca.key", "-out", "ca.pem", "-days", "30",
"-subj", "/CN=test CA", "-addext", "basicConstraints=critical,CA:TRUE")
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "leaf.key", "-out", "leaf.pem", "-days", "30",
"-subj", "/CN=leaf", "-addext", "basicConstraints=CA:FALSE")
real = osapply.Runner()
for name, want in (("ca.pem", "written"), ("leaf.pem", "T3")):
data = open(os.path.join(d, name), "rb").read()
f = TFake(source=data)
f.host = lambda argv, timeout=600, stdin=None: real.host(argv, timeout, stdin) if argv[0] == "openssl" else Fake.host(f, argv, timeout, stdin)
f.clock = __import__("time").time()
rc, rep = run(f)
got = rep["pve_ca_copy"]["outcome"] if rc == 0 else rep["refused"]["code"]
self.assertEqual(got, want, f"{name}: {rep}")
if want == "written":
self.assertIn(rep["pve_ca_copy"]["days_left"], (29, 30))
# the two checks the changeover uses, with the REAL openssl and a REAL TLS server
ended = True
try: # an ENDED CA needs openssl 3.4+ (-not_after); older: that one check is not made
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "old.key", "-out", "old.pem",
"-not_before", "20190101000000Z", "-not_after", "20200101000000Z",
"-subj", "/CN=ended CA", "-addext", "basicConstraints=critical,CA:TRUE")
except subprocess.CalledProcessError:
ended = False
sh("openssl", "req", "-newkey", "rsa:2048", "-nodes", "-keyout", "srv.key", "-out", "srv.csr", "-subj", "/CN=pve")
sh("openssl", "x509", "-req", "-in", "srv.csr", "-CA", "ca.pem", "-CAkey", "ca.key", "-CAcreateserial", "-out", "srv.pem", "-days", "5")
app = osapply.Apply.__new__(osapply.Apply)
app.r = real
block = lambda n: osapply.PEM_CERT_RE.findall(open(os.path.join(d, n), "rb").read())[0]
self.assertTrue(app.is_live_ca(block("ca.pem")), "a CA that has not ended")
self.assertFalse(app.is_live_ca(block("leaf.pem")), "a leaf is not a CA")
if ended:
self.assertFalse(app.is_live_ca(block("old.pem")), "an ended CA is not a live one")
import socket
s0 = socket.socket(); s0.bind(("127.0.0.1", 0)); port = s0.getsockname()[1]; s0.close()
srv = subprocess.Popen(["openssl", "s_server", "-accept", f"127.0.0.1:{port}", "-cert", "srv.pem", "-key", "srv.key", "-quiet"],
cwd=d, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
saved, osapply.PVE_LOCAL_API = osapply.PVE_LOCAL_API, f"127.0.0.1:{port}"
ok = False
for _ in range(40): # the server needs a moment to listen
if app.pve_serves_with(os.path.join(d, "ca.pem")):
ok = True
break
__import__("time").sleep(0.1)
self.assertTrue(ok, "the server's certificate is signed by ca.pem: the handshake must verify with it alone")
self.assertFalse(app.pve_serves_with(os.path.join(d, "leaf.pem")), "another certificate as the CA file must NOT verify")
osapply.PVE_LOCAL_API = "127.0.0.1:1"
self.assertFalse(app.pve_serves_with(os.path.join(d, "ca.pem")), "no answer is a no")
finally:
osapply.PVE_LOCAL_API = saved
srv.kill(); srv.wait()
finally:
shutil.rmtree(d, ignore_errors=True)
def test_a_plan_that_names_a_path_or_a_file_is_refused(self):
for extra in ({"source": "/etc/shadow"}, {"path": "/etc/sudoers.d/x"}, {"dest": "/root/.ssh/authorized_keys"},
{"content_b64": "QQ=="}, {"packages": []}, {"file": "x"}, {"vmid": 9201}, {"dry_run": True}):
f = TFake()
f.plan.update(extra)
rep = self.refused(f, "T1")
self.assertEqual(f.openssl_stdin, [], "refused before the source is even read")
g = TFake()
g.plan["layer"] = "kernel"
self.refused(g, "R11")
def test_a_refused_source_keeps_an_older_copy(self):
f = TFake(source=b"not a certificate", copy=CA_PEM)
self.refused(f, "T3")
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM, "written whole or not at all")
def test_it_runs_on_a_byo_host_too_and_changes_nothing_else(self):
f = TFake()
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"]), (0, "written"), rep)
self.assertEqual([c[1][0] for c in f.calls], ["openssl"], "one openssl call; no package, no service, no other file")
if __name__ == "__main__":
unittest.main()
+188
View File
@@ -0,0 +1,188 @@
package config
import (
"bytes"
"encoding/json"
"fmt"
"os"
"path/filepath"
)
// A cut write of agent.json must never leave a blind box (R-1380, 2026-11-15).
//
// agent.json is the agent's own file inside a ROOT-owned folder, so the agent cannot write a new file and rename it
// over the old one: every writer truncates the file and writes it again. A power cut between the truncate and the
// sync leaves an empty or half file, the agent does not start, and nothing can repair a box whose agent does not
// start from the hub. So the three writers (the trust move, the PBS seed, the backup-target move) go through
// ReplaceInPlace: the OLD bytes are first made durable in a journal file in the agent's own state folder (a new file
// and a rename, which the agent CAN do there), then the config is written, synced and read back, then the journal is
// removed. RecoverCutWrite runs at every start, before the config is read: a journal beside a config that does not
// parse means the write was cut, and the old bytes go back. A journal beside a config that parses means the cut fell
// before the truncate or after the sync — the file is whole, old or new, and only the journal is removed.
//
// Pinned by TestReplaceInPlace_* and TestRecoverCutWrite_* (this package) and by TestMainRecoversACutConfigWrite
// (cmd/felhom-agent).
// CutWriteDir is where the journal lives. It is NOT read from the config: the recovery runs before the config can be
// read. /var/lib/felhom-agent is the unit's StateDirectory. A test points it at a temp folder.
var CutWriteDir = "/var/lib/felhom-agent"
// cutWriteJournal is the journal's file name: the bytes agent.json held before the write that is in flight.
const cutWriteJournal = "agent.json.write-in-flight"
// cutPoint lets a test stop a write where a power cut would (the names are the steps of ReplaceInPlace).
var cutPoint = func(step string) error { return nil }
// ReplaceInPlace writes next over the config at path, whose present content is prev. The caller holds FileMu.
// With no journal folder there is no write: an unprotected write is the defect this exists to remove.
func ReplaceInPlace(path string, prev, next []byte) error {
if !json.Valid(prev) || !json.Valid(next) {
return fmt.Errorf("config: refusing to write %s: the old or the new content is not JSON", path)
}
st, err := os.Stat(path)
if err != nil {
return err
}
if err := os.MkdirAll(CutWriteDir, 0o700); err != nil {
return fmt.Errorf("config: the journal folder: %w", err)
}
journal := filepath.Join(CutWriteDir, cutWriteJournal)
if err := writeDurable(journal, prev); err != nil {
return fmt.Errorf("config: the journal of the old content: %w", err)
}
if err := cutPoint("journal-written"); err != nil {
return err
}
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
_ = removeDurable(journal)
return err
}
if err := cutPoint("truncated"); err != nil {
f.Close()
return err
}
half := len(next) / 2
_, werr := f.Write(next[:half])
if werr == nil {
if err := cutPoint("half-written"); err != nil {
f.Close()
return err
}
_, werr = f.Write(next[half:])
}
if werr == nil {
werr = f.Sync()
}
if cerr := f.Close(); werr == nil {
werr = cerr
}
if werr == nil {
var back []byte
if back, werr = os.ReadFile(path); werr == nil && !bytes.Equal(back, next) {
werr = fmt.Errorf("the file did not read back as written")
}
}
if werr != nil {
// The write failed with the process alive: put the old bytes back now. If that fails too, the journal stays
// and the next start repairs the file.
if rerr := restoreInPlace(path, prev); rerr != nil {
return fmt.Errorf("config: writing %s failed (%v) and the old content could not be put back (%v) — the next start repairs it from the journal", path, werr, rerr)
}
_ = removeDurable(journal)
return fmt.Errorf("config: writing %s failed, the old content was put back: %w", path, werr)
}
if err := cutPoint("synced"); err != nil {
return err
}
return removeDurable(journal)
}
// RecoverCutWrite repairs a config whose last write was cut. It returns what it did, for the start's log line:
// "" (no journal), "journal removed — the config is whole", or "the old config was put back".
func RecoverCutWrite(path string) (string, error) {
journal := filepath.Join(CutWriteDir, cutWriteJournal)
prev, err := os.ReadFile(journal)
if os.IsNotExist(err) {
return "", nil
}
if err != nil {
return "", fmt.Errorf("config: reading the write journal: %w", err)
}
cur, cerr := os.ReadFile(path)
if cerr == nil && isJSONObject(cur) {
return "journal removed — the config is whole", removeDurable(journal)
}
if !isJSONObject(prev) {
// A journal that is itself broken cannot have been followed by a truncate (it is made durable first), so
// this is not a cut write. Leave both for a person.
return "", fmt.Errorf("config: %s does not parse and the write journal does not parse either", path)
}
if err := restoreInPlace(path, prev); err != nil {
return "", fmt.Errorf("config: putting the old content back into %s: %w", path, err)
}
return "the old config was put back", removeDurable(journal)
}
func isJSONObject(b []byte) bool {
var doc map[string]json.RawMessage
return json.Unmarshal(b, &doc) == nil && doc != nil
}
// restoreInPlace writes content into the EXISTING file (no create: the agent cannot make a file in the config's
// folder), so the file keeps its owner and mode.
func restoreInPlace(path string, content []byte) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, 0)
if err != nil {
return err
}
if _, err := f.Write(content); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
return f.Close()
}
// writeDurable makes path hold content, whole, across a power cut: a new file, synced, renamed, the folder synced.
func writeDurable(path string, content []byte) error {
tmp := path + ".tmp"
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return err
}
if _, err := f.Write(content); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
if err := os.Rename(tmp, path); err != nil {
return err
}
return syncDir(filepath.Dir(path))
}
func removeDurable(path string) error {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return err
}
return syncDir(filepath.Dir(path))
}
func syncDir(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}
+140
View File
@@ -0,0 +1,140 @@
package config
import (
"errors"
"os"
"path/filepath"
"testing"
)
const (
oldCfg = `{"proxmox":{"tls":{"fingerprint":"AA:BB"}},"keep":"me"}`
newCfg = `{"proxmox":{"tls":{"ca_file":"/etc/felhom-agent/pve-root-ca.pem","fingerprint":""}},"keep":"me"}`
)
// cutBox is a config file and a journal folder in a temp dir, with the write cut at one named step.
func cutBox(t *testing.T) (cfg string) {
t.Helper()
dir := t.TempDir()
cfg = filepath.Join(dir, "etc", "agent.json")
if err := os.MkdirAll(filepath.Dir(cfg), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(cfg, []byte(oldCfg), 0o600); err != nil {
t.Fatal(err)
}
prevDir, prevCut := CutWriteDir, cutPoint
CutWriteDir = filepath.Join(dir, "state")
t.Cleanup(func() { CutWriteDir, cutPoint = prevDir, prevCut })
return cfg
}
func cutAt(step string) {
cutPoint = func(s string) error {
if s == step {
return errors.New("power cut at " + s)
}
return nil
}
}
func read(t *testing.T, p string) string {
t.Helper()
b, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
return string(b)
}
// The consequence that matters: after a power cut at ANY step of the write and the repair at the next start, the
// config is the whole old file or the whole new file — never a broken one — and Load reads it.
func TestRecoverCutWrite_AfterACutAtEveryStepTheConfigIsWholeAndLoads(t *testing.T) {
want := map[string]string{
"journal-written": oldCfg, // nothing was touched yet
"truncated": oldCfg, // the file is EMPTY at the cut
"half-written": oldCfg, // the file is HALF at the cut
"synced": newCfg, // the new file is whole; only the journal is left
}
for step, wantAfter := range want {
t.Run(step, func(t *testing.T) {
cfg := cutBox(t)
cutAt(step)
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
t.Fatal("the staged cut did not stop the write")
}
atCut := read(t, cfg)
if (step == "truncated" || step == "half-written") && isJSONObject([]byte(atCut)) {
t.Fatalf("the staging is wrong: at %s the file must be broken, it is %q", step, atCut)
}
cutPoint = func(string) error { return nil } // the next start: no cut
note, err := RecoverCutWrite(cfg)
if err != nil {
t.Fatalf("repair: %v", err)
}
if note == "" {
t.Fatal("the repair saw no journal — a cut write would stay broken")
}
if got := read(t, cfg); got != wantAfter {
t.Fatalf("after a cut at %s and the repair the config is %q, want %q", step, got, wantAfter)
}
if _, err := Load(cfg); err != nil {
t.Fatalf("the agent does not start with the repaired config: %v", err)
}
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
t.Fatalf("the journal stayed after the repair (%v) — an old journal would undo a later change", err)
}
})
}
}
// A write that is not cut leaves the new file and NO journal: a journal that stayed would put an old config back
// at some later start.
func TestReplaceInPlace_AWholeWriteLeavesTheNewFileAndNoJournal(t *testing.T) {
cfg := cutBox(t)
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err != nil {
t.Fatal(err)
}
if got := read(t, cfg); got != newCfg {
t.Fatalf("config = %q", got)
}
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
t.Fatalf("the journal stayed after a whole write: %v", err)
}
if note, err := RecoverCutWrite(cfg); err != nil || note != "" {
t.Fatalf("a start after a whole write must do nothing: note=%q err=%v", note, err)
}
if got := read(t, cfg); got != newCfg {
t.Fatalf("the start changed a whole config to %q", got)
}
}
// No journal folder, no write: the old file stays as it was.
func TestReplaceInPlace_WithoutAJournalNothingIsWritten(t *testing.T) {
cfg := cutBox(t)
// the journal cannot be made: a folder sits where its new file must be written
if err := os.MkdirAll(filepath.Join(CutWriteDir, cutWriteJournal+".tmp"), 0o700); err != nil {
t.Fatal(err)
}
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
t.Fatal("the write went on with no journal")
}
if got := read(t, cfg); got != oldCfg {
t.Fatalf("the config changed with no journal: %q", got)
}
}
// A config that is broken with NO journal is not a cut write: the repair does not touch it, and says nothing.
func TestRecoverCutWrite_ABrokenConfigWithNoJournalIsLeftAlone(t *testing.T) {
cfg := cutBox(t)
if err := os.WriteFile(cfg, []byte(`{"half":`), 0o600); err != nil {
t.Fatal(err)
}
note, err := RecoverCutWrite(cfg)
if err != nil || note != "" {
t.Fatalf("note=%q err=%v", note, err)
}
if got := read(t, cfg); got != `{"half":` {
t.Fatalf("the repair wrote a config it had no journal for: %q", got)
}
}
+12
View File
@@ -0,0 +1,12 @@
package config
import "sync"
// FileMu is held by EVERY read-modify-write of agent.json inside this process: the trust move (pvetrust), the PBS
// seed (pbsdr) and the backup-target move (localapi). Each of them reads the whole file, changes one section and
// writes the whole file back in place — without one lock two of them could interleave, and the second writer would
// put back the document it read BEFORE the first one's change (a silently reverted `escrow.pbs_storage_id`,
// `backup.local_backup_target` or trust section). Found by review on 2026-11-14, the night the third writer was
// added. Hold it for the whole read … write, never across a call to Proxmox or the root wrapper.
// Pinned by TestAgentJSONWritersTakeTheSharedLock (cmd/felhom-agent).
var FileMu sync.Mutex
+30 -1
View File
@@ -10,6 +10,7 @@ import (
"io"
"log/slog"
"net/http"
"net/url"
"os"
"strings"
"time"
@@ -86,6 +87,7 @@ func (e *HTTPError) Error() string {
// IS the heartbeat (locked decision 1). Errors are typed (transport vs HTTP) and
// never include the bearer token.
func (c *Client) Report(ctx context.Context, r *HostReport) (*ControlEnvelope, error) {
r.SentAt = time.Now().UTC().Format(time.RFC3339Nano) // the box clock at SENDING (R-1384); see HostReport.SentAt
body, err := json.Marshal(r)
if err != nil {
return nil, fmt.Errorf("hub: marshaling report: %w", err)
@@ -285,10 +287,37 @@ func (c *Client) Jobs(ctx context.Context) ([]JobWire, error) {
// /hosts/{id}/jobs/{job_id}, self-scoped. Called after a job is executed OR permanently rejected
// (the nonce is already durably burned on a passing verify, so re-processing is replay-safe).
func (c *Client) CompleteJob(ctx context.Context, jobID string) error {
return c.CompleteJobWithOutcome(ctx, jobID, JobOutcome{})
}
// JobOutcome says HOW a signed job ended when the agent clears it (R-1383, D78): Outcome is done | rejected | failed
// | malformed; Reason is one code of signedjobs.ReasonCodes (empty for done); Op is the job's op class. It rides the
// DELETE's query, so a hub that does not know it ignores it. Until this, „ran” and „refused” looked the same to the
// hub, and a refused agent update left no trace outside the box's journal. No secret: no signature, no nonce, no
// parameter of the job is in it.
type JobOutcome struct {
Outcome string
Reason string
Op string
}
// CompleteJobWithOutcome clears the job and tells the hub how it ended. The zero JobOutcome sends no query.
func (c *Client) CompleteJobWithOutcome(ctx context.Context, jobID string, o JobOutcome) error {
if c.hostID == "" || jobID == "" {
return fmt.Errorf("hub: CompleteJob requires host_id + job_id")
}
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.baseURL+"/api/v1/hosts/"+c.hostID+"/jobs/"+jobID, nil)
u := c.baseURL + "/api/v1/hosts/" + c.hostID + "/jobs/" + jobID
if o.Outcome != "" {
q := url.Values{"outcome": {o.Outcome}}
if o.Reason != "" {
q.Set("reason", o.Reason)
}
if o.Op != "" {
q.Set("op", o.Op)
}
u += "?" + q.Encode()
}
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, u, nil)
if err != nil {
return err
}
+76
View File
@@ -0,0 +1,76 @@
package hub
import (
"context"
"encoding/json"
"io"
"net/http"
"testing"
"time"
)
// R-1384 (D79): the hub tells a wrong box clock by the time stamped at the moment of SENDING. A report whose collect
// started long ago (an old reported_at) must still leave with a fresh sent_at, and a second send of the SAME report
// struct must carry a new one — or a report that waited on the box reads as a clock that is behind.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt).
func TestReport_StampsSentAtAtSending(t *testing.T) {
var bodies []map[string]any
c := testClient(func(r *http.Request) (*http.Response, error) {
raw, _ := io.ReadAll(r.Body)
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatalf("body: %v", err)
}
bodies = append(bodies, m)
return httpResp(200, `{"status":"ok"}`), nil
})
old := time.Now().Add(-40 * time.Minute).UTC()
rep := &HostReport{HostID: "h", ReportedAt: old.Format(time.RFC3339), SentAt: old.Format(time.RFC3339Nano)}
for i := 0; i < 2; i++ {
if _, err := c.Report(context.Background(), rep); err != nil {
t.Fatalf("Report: %v", err)
}
time.Sleep(5 * time.Millisecond)
}
var sent []time.Time
for i, m := range bodies {
s, _ := m["sent_at"].(string)
at, err := time.Parse(time.RFC3339Nano, s)
if err != nil {
t.Fatalf("send %d: sent_at = %q, want an RFC3339 time", i, s)
}
if d := time.Since(at); d < 0 || d > 5*time.Second {
t.Errorf("send %d: sent_at is %s old, want the moment of sending (reported_at was 40 minutes old)", i, d)
}
if m["reported_at"] != old.Format(time.RFC3339) {
t.Errorf("send %d: reported_at changed to %v — the collect time is not the send time's to touch", i, m["reported_at"])
}
sent = append(sent, at)
}
if len(sent) == 2 && !sent[1].After(sent[0]) {
t.Errorf("the second send carries %s, not after the first %s — the stamp is not taken per send", sent[1], sent[0])
}
}
// R-1383 (D78): the clearing call says how the job ended, in the DELETE's query; a plain CompleteJob sends none (an
// older hub, and the „nothing to say” case, see the same request as before).
func TestCompleteJob_OutcomeRidesTheQuery(t *testing.T) {
var urls []string
c := testClient(func(r *http.Request) (*http.Response, error) {
if r.Method != http.MethodDelete {
t.Errorf("method = %s", r.Method)
}
urls = append(urls, r.URL.Path+"?"+r.URL.RawQuery)
return httpResp(200, `{"status":"ok"}`), nil
})
if err := c.CompleteJob(context.Background(), "j1"); err != nil {
t.Fatal(err)
}
if err := c.CompleteJobWithOutcome(context.Background(), "j2", JobOutcome{Outcome: "rejected", Reason: "expired", Op: "agent_update"}); err != nil {
t.Fatal(err)
}
want := []string{"/api/v1/hosts/demo-host-01/jobs/j1?", "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&reason=expired"}
if len(urls) != 2 || urls[0] != want[0] || urls[1] != want[1] {
t.Fatalf("requests = %q, want %q", urls, want)
}
}
+51 -6
View File
@@ -124,6 +124,9 @@ type Collector struct {
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
pveTrust func() any // R-1380: the proxmox_trust stanza (nil = omitted)
guestPowerGaveUp func() any // R-1383: the guest_power_gave_up stanza (nil = omitted)
profileRefresh func() any // R-1383: the profile_refresh stanza (nil = omitted)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
@@ -209,6 +212,24 @@ func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capabi
return c
}
// SetProxmoxTrustSource wires the `proxmox_trust` stanza (R-1380). Optional; nil = the stanza is omitted.
func (c *Collector) SetProxmoxTrustSource(f func() any) *Collector {
c.pveTrust = f
return c
}
// SetProfileRefreshSource wires the `profile_refresh` stanza (R-1383). Optional; nil, or a nil result, omits it.
func (c *Collector) SetProfileRefreshSource(f func() any) *Collector {
c.profileRefresh = f
return c
}
// SetGuestPowerGaveUpSource wires the `guest_power_gave_up` stanza (R-1383). Optional; nil, or a nil result, omits it.
func (c *Collector) SetGuestPowerGaveUpSource(f func() any) *Collector {
c.guestPowerGaveUp = f
return c
}
// SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub
// watches it for a re-key). Static per process — set once at startup. "" when the local API is
// disabled. Returns the collector for chaining.
@@ -391,6 +412,27 @@ func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
// error (no useful report — the cycle skips the POST); a failed per-guest
// GuestConfig degrades that guest to status="unknown" without spec but still sends;
// a cloudflared probe failure yields status="unknown" and is never fatal.
func (c *Collector) profileRefreshStanza() any {
if c.profileRefresh == nil {
return nil
}
return c.profileRefresh()
}
func (c *Collector) guestPowerGaveUpStanza() any {
if c.guestPowerGaveUp == nil {
return nil
}
return c.guestPowerGaveUp()
}
func (c *Collector) proxmoxTrust() any {
if c.pveTrust == nil {
return nil
}
return c.pveTrust()
}
func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
ns, err := c.px.NodeStatus(ctx)
if err != nil {
@@ -416,12 +458,15 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
RestoreTests: c.collectRestoreTests(ctx),
PBSSnapshots: c.collectPBSSnapshots(ctx),
AuditTail: []AuditEntry{},
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
AuditTail: []AuditEntry{},
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
ProxmoxTrust: c.proxmoxTrust(),
GuestPowerGaveUp: c.guestPowerGaveUpStanza(),
ProfileRefresh: c.profileRefreshStanza(),
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
}
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
+9
View File
@@ -70,8 +70,14 @@ type Loop struct {
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
// path waits on it so the hub has the deliberate-off record before the box goes down.
reported func(r *HostReport)
// collectErr (R-1380) sees every failed collect.
collectErr func(ctx context.Context, err error)
}
// SetCollectErrorHook wires an observer of a failed collect (R-1380: a call to Proxmox that fails on TRUST starts the
// move to the CA copy, or a reload of a refreshed copy). Optional; it must not block for long.
func (l *Loop) SetCollectErrorHook(f func(ctx context.Context, err error)) { l.collectErr = f }
// SetReportedHook wires the R-998 after-report observer. Optional.
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
@@ -144,6 +150,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
report, err := l.collector.Collect(ctx)
if err != nil {
l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err)
if l.collectErr != nil {
l.collectErr(ctx, err)
}
return current
}
// Fulfill a pending log-pull: attach the ring tail to THIS report and clear the
+20 -2
View File
@@ -15,8 +15,13 @@ import (
// server-side received_at is the hub's dead-man's-switch liveness signal. There is
// no separate heartbeat endpoint.
type HostReport struct {
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
// SentAt (R-1384, D79) is this box's clock at the moment the report is SENT — stamped by Client.Report, never by
// the collector. The hub compares it with its own clock to tell a wrong box clock. ReportedAt cannot serve: it is
// stamped when the collect STARTS, and a collect that waits on Proxmox makes it old on a box whose clock is right.
// Pinned by TestReport_StampsSentAtAtSending.
SentAt string `json:"sent_at,omitempty"`
AgentVersion string `json:"agent_version"`
// AgentSHA256 is the sha256 of the binary this process is RUNNING (read through /proc/self/exe,
// once per process), R-349. The version string cannot tell a hand-built proof binary from the
@@ -53,6 +58,11 @@ type HostReport struct {
// alert. Not a secret (the fp is public; the token is never reported).
LeafFingerprint string `json:"leaf_fingerprint"`
// ProxmoxTrust (R-1380, felhom.eu `03` §12a): how the agent trusts Proxmox's certificate (`mode`: ca | fingerprint |
// …), and the CAUSE when something about it is wrong (`problem`, `detail`): the proof with the CA copy failed, the
// copy ends soon, the root step gave no copy. Omitted when not wired. No secret: a mode, a cause, a date.
ProxmoxTrust any `json:"proxmox_trust,omitempty"`
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
@@ -172,6 +182,14 @@ type HostReport struct {
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
// ProfileRefresh (R-1383, release 3): the box's last security-profile refresh and how the agent's last ask ended
// (osupdate.ProfileRefreshStatus, field-exact; hub sysfacts.ProfileRefresh). `any` + omitempty: absent when there
// is nothing to say or the source is not wired.
ProfileRefresh any `json:"profile_refresh,omitempty"`
// GuestPowerGaveUp (R-1383, D78): the guests the guest-power watchdog stopped trying to start (localapi
// GuestPowerGiveUp, field-exact: vmid, since, attempts). The hub raises one operator alarm per guest and clears it
// when the guest runs. `any` + omitempty: absent when there is none or the source is not wired.
GuestPowerGaveUp any `json:"guest_power_gave_up,omitempty"`
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
// when there is nothing to answer, so the golden stays byte-stable.
+8 -15
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"net/http"
"os"
"path/filepath"
@@ -145,9 +146,11 @@ func (s *Server) validateBackupTargetMount(ctx context.Context, where string) er
// begin with. A typed round-trip would silently drop any key this build does not know about.
//
// Written IN PLACE (O_TRUNC), not tmp+rename: /etc/felhom-agent is root-owned while agent.json is
// agent-owned 0600, so the non-root agent cannot rename into that directory. A recovery copy is
// parked first, so a torn write is recoverable.
// agent-owned 0600, so the non-root agent cannot rename into that directory. config.ReplaceInPlace
// journals the old bytes first; the next start repairs a cut write (TestRecoverCutWrite_*).
func (s *Server) setConfiguredBackupTarget(id string) error {
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
defer config.FileMu.Unlock()
path := s.configPath
if path == "" {
return fmt.Errorf("no config path known to this agent (env-only config)")
@@ -179,24 +182,14 @@ func (s *Server) setConfiguredBackupTarget(id string) error {
if err != nil {
return err
}
st, err := os.Stat(path)
if err != nil {
return err
}
if s.stateDir != "" {
if err := os.MkdirAll(s.stateDir, 0o700); err == nil {
_ = os.WriteFile(filepath.Join(s.stateDir, "agent.json.pre-backup-target"), raw, 0o600)
}
}
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
return err
}
if _, err := f.Write(out); err != nil {
f.Close()
return err
}
return f.Close()
// config.ReplaceInPlace: the old bytes go into a journal first, so a power cut in the middle of the write is
// repaired at the next start (R-1380).
return config.ReplaceInPlace(path, raw, out)
}
// wrapperReason surfaces the wrapper's own REFUSED line when it produced one — it explains WHY in
+21
View File
@@ -0,0 +1,21 @@
package localapi
import (
"os"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
// agent's state folder on a box, and a test must not need it.
func TestMain(m *testing.M) {
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
if err != nil {
panic(err)
}
config.CutWriteDir = dir
code := m.Run()
_ = os.RemoveAll(dir)
os.Exit(code)
}
+9 -3
View File
@@ -87,6 +87,11 @@ type GuestAttacher interface {
// GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a
// controller-only restart — the deterministic guest-reboot signal the controller recreates apps on.
GuestBootID(ctx context.Context, vmid int) string
// GuestBootKey is the same signal built from the kernel's boot_id, which a step of the host's clock cannot
// move (R-1395). "" = not readable; the controller then reads GuestBootID.
GuestBootKey(ctx context.Context, vmid int) string
// GuestBootTokens returns (GuestBootID, GuestBootKey) from one read — what /disks sends.
GuestBootTokens(ctx context.Context, vmid int) (id, key string)
// AttachBind is the LEGACY per-drive `pct set -mpN` bind (pre-intermediary). Retained for the
// transition; new attaches use AttachDrive.
AttachBind(ctx context.Context, vmid int, mountKey, where string) error
@@ -431,11 +436,12 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// Guest boot-id (intermediary model): changes on every guest boot, stable across controller restarts.
// The controller persists it and deterministically recreates drive-backed apps when it changes.
bootID := ""
// R-1395: guest_boot_key is the clock-proof form (kernel boot_id); guest_boot_id stays for older controllers.
bootID, bootKey := "", ""
if s.guestAttach != nil {
bootID = s.guestAttach.GuestBootID(r.Context(), vmid)
bootID, bootKey = s.guestAttach.GuestBootTokens(r.Context(), vmid)
}
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID})
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID, "guest_boot_key": bootKey})
}
// handleDiskCandidates (Impl-2a) lists the host disks FREE for Felhom to enroll — the raw-device
+4
View File
@@ -466,6 +466,10 @@ func (f *fakeGuestAttacher) AttachDrive(_ context.Context, _ int, where string)
}
func (f *fakeGuestAttacher) GuestSeesMount(_ context.Context, _ int, _ string) bool { return true }
func (f *fakeGuestAttacher) GuestBootID(_ context.Context, _ int) string { return "boot-1" }
func (f *fakeGuestAttacher) GuestBootKey(_ context.Context, _ int) string { return "key-1" }
func (f *fakeGuestAttacher) GuestBootTokens(_ context.Context, _ int) (string, string) {
return "boot-1", "key-1"
}
func (f *fakeGuestAttacher) attachDriveCount() int {
f.mu.Lock()
defer f.mu.Unlock()
+116
View File
@@ -0,0 +1,116 @@
package localapi
import (
"context"
"encoding/json"
"io"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
type initPIDRunner struct{ pid string }
func (r initPIDRunner) Run(_ context.Context, name string, _ ...string) ([]byte, []byte, error) {
if name == "lxc-info" {
return []byte(r.pid + "\n"), nil, nil
}
return nil, nil, nil
}
func (r initPIDRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
// R-1395: on 2026-11-14 the N100's time service came back and stepped the host's clock by +32.7 s. The kernel
// computes the boot TIME from the wall clock, so guest_boot_id moved and the controller recreated every drive-backed
// app. The consequence asserted: the boot KEY does not move when the boot time moves; it does move on a host reboot
// and on a guest reboot; and it is "" (so the controller falls back to the old token) when the boot_id is unreadable.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt).
func TestGuestBootKey_AClockStepDoesNotMoveIt(t *testing.T) {
oldB, oldK, oldS := hostBtimeFn, hostBootIDFn, procStarttimeFn
t.Cleanup(func() { hostBtimeFn, hostBootIDFn, procStarttimeFn = oldB, oldK, oldS })
btime, bootID, start := "1794112518", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
hostBtimeFn = func() string { return btime }
hostBootIDFn = func() string { return bootID }
procStarttimeFn = func(pid string) string {
if pid != "31337" {
t.Errorf("start time read for pid %q, want the guest's init", pid)
}
return start
}
b := NewGuestBinder(initPIDRunner{"31337"}, slog.New(slog.NewTextHandler(io.Discard, nil)))
ctx := context.Background()
id0, key0 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
if id0 != "1794112518-4242" || key0 != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" {
t.Fatalf("tokens = %q / %q", id0, key0)
}
btime = "1794112550" // THE STEP: the wall clock jumped +32 s, so the kernel's btime did too
id1, key1 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
if id1 == id0 {
t.Fatalf("control: the old token did not move with the boot time (%q) — the test does not stage the defect", id1)
}
if key1 != key0 {
t.Fatalf("the boot key moved with the clock: %q → %q — a clock step reads as a new boot", key0, key1)
}
if strings.Contains(key1, "1794") {
t.Errorf("the boot key carries the boot time: %q", key1)
}
bootID = "c0ffee00-1111-2222-3333-444455556666" // a host reboot
if k := b.GuestBootKey(ctx, 9201); k == key0 {
t.Errorf("a host reboot (new boot_id) did not change the key: %q", k)
}
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "991234" // a guest reboot
if k := b.GuestBootKey(ctx, 9201); k == key0 || !strings.HasSuffix(k, "-991234") {
t.Errorf("a guest reboot (new init start) did not change the key: %q", k)
}
for _, bad := range []string{"", "not-a-boot-id", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11\nx"} {
bootID = bad
if k := b.GuestBootKey(ctx, 9201); k != "" {
t.Errorf("boot_id %q gave the key %q, want \"\" (the controller then reads the old token)", bad, k)
}
if id := b.GuestBootID(ctx, 9201); id == "" {
t.Errorf("the old token must still be sent when the boot_id is unreadable")
}
}
// what /disks sends: both tokens from ONE read of the guest's init — the init is asked for once, not twice
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
reads := 0
procStarttimeFn = func(string) string { reads++; return start }
if id, key := b.GuestBootTokens(ctx, 9201); id != "1794112550-4242" || key != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" || reads != 1 {
t.Errorf("GuestBootTokens = %q, %q after %d read(s) of the init — want both tokens from ONE read", id, key, reads)
}
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "" // the guest does not run
if k := b.GuestBootKey(ctx, 9201); k != "" {
t.Errorf("a guest that does not run gave the key %q", k)
}
}
// The key is no use if /disks does not carry it: both tokens are on the wire, the old one unchanged for controllers
// that do not know the key.
func TestDisks_CarriesBothBootTokens(t *testing.T) {
srv := targetRowServer(t, "", []hub.StorageTarget{})
srv.guestAttach = &fakeGuestAttacher{}
body := do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()
var w struct {
Data struct {
ID string `json:"guest_boot_id"`
Key string `json:"guest_boot_key"`
} `json:"data"`
ID string `json:"guest_boot_id"`
Key string `json:"guest_boot_key"`
}
if err := json.Unmarshal(body, &w); err != nil {
t.Fatalf("decode: %v (%s)", err, body)
}
if w.Data.ID+w.ID != "boot-1" || w.Data.Key+w.Key != "key-1" {
t.Fatalf("/disks boot tokens: id=%q key=%q, want boot-1 and key-1 (%s)", w.Data.ID+w.ID, w.Data.Key+w.Key, body)
}
}
// The real binder is what main.go hands the server: it must have the method (the interface makes this a build
// error if not — stated here so the reason is on record).
var _ GuestAttacher = (*GuestBinder)(nil)
+51
View File
@@ -3,6 +3,7 @@ package localapi
import (
"context"
"log/slog"
"sort"
"sync"
"time"
@@ -82,6 +83,40 @@ type guestPowerState struct {
attempts int
nextAt time.Time
raised bool // the give-up fault has already been raised for this run
raisedAt time.Time
}
// GuestPowerGiveUp is one guest the watchdog stopped trying to start (R-1383, D78). It rides the host report
// (`guest_power_gave_up`) so the operator hears it: until this, the give-up was one ERROR line in the host's journal
// while the household's apps were off. Memory only — an agent restart forgets it (and tries three times again).
type GuestPowerGiveUp struct {
VMID int `json:"vmid"`
Since string `json:"since"` // RFC3339 UTC, when the watchdog gave up
Attempts int `json:"attempts"`
}
// GuestPowerGaveUpStanza is GuestPowerGaveUp shaped for the host report: an UNTYPED nil when there is none, so the
// report omits the field (a typed nil slice in an `any` would be sent as `null`).
func (s *Server) GuestPowerGaveUpStanza() any {
if g := s.GuestPowerGaveUp(); len(g) > 0 {
return g
}
return nil
}
// GuestPowerGaveUp returns the guests the watchdog has given up on, by vmid. nil when there is none — the report
// stanza is then omitted. Pinned by TestGuestPower_GiveUpIsReportedUntilTheGuestRuns.
func (s *Server) GuestPowerGaveUp() []GuestPowerGiveUp {
guestPowerMu.Lock()
defer guestPowerMu.Unlock()
var out []GuestPowerGiveUp
for vmid, st := range s.guestPower {
if st.raised {
out = append(out, GuestPowerGiveUp{VMID: vmid, Since: st.raisedAt.UTC().Format(time.RFC3339), Attempts: st.attempts})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].VMID < out[j].VMID })
return out
}
// WatchGuestPower runs the guest-power sweep every guestPowerInterval until ctx is done. No-op when
@@ -117,6 +152,19 @@ func (s *Server) GuestPowerTick(ctx context.Context) {
s.logger.Warn("guest-power: guest list unavailable — skipping sweep (ownership unproven)", "err", err)
return
}
// A guest that left the list (destroyed, or re-made under another number) takes its attempt record with it — a
// give-up for a guest that no longer exists would ride every host report until the agent restarts.
present := make(map[int]bool, len(guests))
for _, g := range guests {
present[g.VMID] = true
}
guestPowerMu.Lock()
for vmid := range s.guestPower {
if !present[vmid] {
delete(s.guestPower, vmid)
}
}
guestPowerMu.Unlock()
var stopped int
for _, g := range guests {
if ctx.Err() != nil {
@@ -167,6 +215,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
// SCENARIO B — a deliberately stopped guest is left alone, forever. onboot:0 means the operator
// (or the golden-image provisioning) does not want this guest running.
if !onboot {
s.forgetGuestPower(g.VMID) // not ours to start any more: a give-up for it must not be reported for ever
return
}
@@ -176,6 +225,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
if lock != "" {
s.logger.Info("guest-power: guest is stopped but LOCKED — leaving it to the stale-lock path",
"vmid", g.VMID, "lock", lock)
s.forgetGuestPower(g.VMID) // another operation owns it now; if it ends stopped, the three tries start again
return
}
@@ -260,6 +310,7 @@ func (s *Server) markGuestPowerRaised(vmid int) {
defer guestPowerMu.Unlock()
st := s.guestPower[vmid]
st.raised = true
st.raisedAt = s.now()
s.guestPower[vmid] = st
}
+79
View File
@@ -2,6 +2,7 @@ package localapi
import (
"context"
"encoding/json"
"errors"
"log/slog"
"testing"
@@ -237,3 +238,81 @@ func TestGuestPower_RunningGuestClearsAttemptHistory(t *testing.T) {
t.Error("attempt history survived the guest coming back healthy")
}
}
// R-1383 (D78): the give-up was ONE journal line while the household's apps were off. It now rides the host report:
// nothing before the give-up (and an untyped nil, so the report omits the field), the guest with the time and the
// attempt count after it, and nothing again once the guest runs.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-guest-power.txt).
func TestGuestPower_GiveUpIsReportedUntilTheGuestRuns(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
startErr: errors.New("cannot start: storage offline"),
}
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
s := gpServer(t, ctl, func() time.Time { return now })
if st := s.GuestPowerGaveUpStanza(); st != nil {
t.Fatalf("before any attempt the stanza is %#v, want an untyped nil (the report omits the field)", st)
}
for i := 0; i < guestPowerMaxAttempts; i++ { // the three failed starts
s.GuestPowerTick(context.Background())
if g := s.GuestPowerGaveUp(); len(g) != 0 {
t.Fatalf("after %d failed start(s) the watchdog still tries, but it reports a give-up: %+v", i+1, g)
}
now = now.Add(10 * time.Minute)
}
gaveUpAt := now
s.GuestPowerTick(context.Background()) // the tick that gives up
now = now.Add(time.Hour)
s.GuestPowerTick(context.Background())
g := s.GuestPowerGaveUp()
if len(g) != 1 || g[0].VMID != 9201 || g[0].Attempts != guestPowerMaxAttempts || g[0].Since != gaveUpAt.Format(time.RFC3339) {
t.Fatalf("give-up list = %+v, want guest 9201, %d attempts, since %s (the give-up's time, not a later tick's)", g, guestPowerMaxAttempts, gaveUpAt.Format(time.RFC3339))
}
b, _ := json.Marshal(s.GuestPowerGaveUpStanza())
if string(b) != `[{"vmid":9201,"since":"2026-11-14T02:30:00Z","attempts":3}]` {
t.Fatalf("the stanza on the wire = %s", b)
}
// the operator starts the guest: the next sweep sees it running and the list is empty again
ctl.guests = []proxmox.Guest{{VMID: 9201, Status: "running"}}
s.GuestPowerTick(context.Background())
if st := s.GuestPowerGaveUpStanza(); st != nil {
t.Fatalf("the guest runs again and the stanza is still %#v", st)
}
}
// Found by review, 2026-11-14: the give-up was forgotten only when the guest was seen RUNNING. A guest the operator
// then switches to onboot:0, a guest another operation locks, and a guest that is destroyed would be reported as
// „given up” for ever (the hub's alarm could never clear). Each of the three ends the report.
func TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs(t *testing.T) {
for _, c := range []struct {
name string
change func(ctl *fakeGuestPowerCtl)
}{
{"onboot switched off", func(ctl *fakeGuestPowerCtl) { ctl.onboot = map[int]bool{9201: false} }},
{"locked by another operation", func(ctl *fakeGuestPowerCtl) { ctl.locks = map[int]string{9201: "backup"} }},
{"destroyed", func(ctl *fakeGuestPowerCtl) { ctl.guests = nil }},
} {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
startErr: errors.New("cannot start"),
}
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
s := gpServer(t, ctl, func() time.Time { return now })
for i := 0; i < guestPowerMaxAttempts+1; i++ {
s.GuestPowerTick(context.Background())
now = now.Add(10 * time.Minute)
}
if len(s.GuestPowerGaveUp()) != 1 {
t.Fatalf("%s: setup — no give-up recorded", c.name)
}
c.change(ctl)
s.GuestPowerTick(context.Background())
if g := s.GuestPowerGaveUp(); len(g) != 0 {
t.Errorf("%s: the give-up is still reported: %+v", c.name, g)
}
}
}
+68 -6
View File
@@ -5,6 +5,7 @@ import (
"context"
"fmt"
"os"
"regexp"
"strconv"
"strings"
)
@@ -288,21 +289,82 @@ func (b *GuestBinder) guestInitPID(ctx context.Context, vmid int) string {
// /proc/<pid>/stat — ticks since host boot, unique per process launch) changes on a guest reboot. ""
// on any read error (the controller then keeps its last-seen → no spurious recreate).
func (b *GuestBinder) GuestBootID(ctx context.Context, vmid int) string {
pid := b.guestInitPID(ctx, vmid)
if pid == "" {
return ""
}
start := procStarttime(pid)
start := b.guestInitStart(ctx, vmid)
if start == "" {
return ""
}
bt := hostBtime()
bt := hostBtimeFn()
if bt == "" {
bt = "0"
}
return bt + "-" + start
}
// guestInitStart is the guest init's start time in ticks since the host's boot ("" = the guest does not run, or a
// read error). Both boot tokens end with it.
func (b *GuestBinder) guestInitStart(ctx context.Context, vmid int) string {
pid := b.guestInitPID(ctx, vmid)
if pid == "" {
return ""
}
return procStarttimeFn(pid)
}
// GuestBootKey (R-1395) is GuestBootID with the host's boot TIME replaced by the kernel's boot_id:
// "<boot_id>-<guest-init-starttime>". The kernel computes the boot time (`btime`) from the wall clock, so a STEP of
// the host's clock moves GuestBootID — on the N100 on 2026-11-14 the time service came back, stepped +32.7 s, and the
// controller recreated every drive-backed app as if the guest had booted. The kernel's boot_id is a random value made
// once per boot; no clock can move it. Same meaning as before: it changes on a host reboot (new boot_id) and on a
// guest reboot (new init start), and is stable across an agent or controller restart.
//
// "" when the guest does not run or the boot_id cannot be read — the controller then decides by GuestBootID, which
// is still sent (controllers that do not know the key read only that). Pinned by TestGuestBootKey_*.
func (b *GuestBinder) GuestBootKey(ctx context.Context, vmid int) string {
start := b.guestInitStart(ctx, vmid)
return guestBootKey(hostBootIDFn(), start)
}
// GuestBootTokens returns both tokens from ONE read of the guest's init: the old token and the key always describe
// the same moment. Read separately, one could succeed and the other fail (the guest stops between the two) — the
// controller would then get an old token with no key, read „an older agent”, and recreate every drive-backed app.
func (b *GuestBinder) GuestBootTokens(ctx context.Context, vmid int) (id, key string) {
start := b.guestInitStart(ctx, vmid)
if start == "" {
return "", ""
}
bt := hostBtimeFn()
if bt == "" {
bt = "0"
}
return bt + "-" + start, guestBootKey(hostBootIDFn(), start)
}
// guestBootKey joins the two halves; "" when either is missing or the boot_id is not the kernel's shape.
func guestBootKey(bootID, start string) string {
if start == "" || !kernelBootIDRe.MatchString(bootID) {
return ""
}
return bootID + "-" + start
}
var kernelBootIDRe = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
// The three reads of /proc behind the boot tokens, as seams (the tests move the boot time and keep the boot_id).
var (
hostBtimeFn = hostBtime
hostBootIDFn = hostBootID
procStarttimeFn = procStarttime
)
// hostBootID returns the kernel's boot_id (/proc/sys/kernel/random/boot_id): random, made once per boot. "" on error.
func hostBootID() string {
data, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
if err != nil {
return ""
}
return strings.TrimSpace(string(data))
}
// procStarttime returns field 22 (starttime) of /proc/<pid>/stat. The comm field (2) can contain spaces
// and parentheses, so we split AFTER the last ')': field 22 is index 19 of the post-comm fields
// (field 3 = state is index 0). "" on any error.
+18
View File
@@ -531,6 +531,8 @@ func (l *Leg) runKernelCleanup(ctx context.Context, runID string, vmid int, trig
gained = fmt.Sprintf("%.1f GiB", float64(*res.BytesGained)/(1<<30))
}
rep.Outcome, rep.Healthy = "cleaned", true
rep.KernelCleanup = &CleanupDetail{Removed: capNames(res.Removed, 40), Kept: capNames(sortedKeys(res.Kept), 40),
BytesGained: res.BytesGained, Leftovers: len(res.Leftovers)}
rep.HealthReason = truncate(fmt.Sprintf("removed %d old kernel package(s): %s; kept %s; %s gained; %d left-over path(s)",
len(res.Removed), strings.Join(res.Removed, ", "), strings.Join(sortedKeys(res.Kept), ", "), gained, len(res.Leftovers)), 600)
lg.Info("osupdate: kernel clean-up — old kernels removed", "removed", len(res.Removed), "names", strings.Join(res.Removed, ","),
@@ -564,6 +566,22 @@ func (l *Leg) KernelCleanupDryRun(ctx context.Context, vmid int) (json.RawMessag
return wr.KernelCleanup, nil
}
// capNames bounds a list that goes to the hub: at most n names, each at most 120 bytes; never nil (the hub reads
// "removed: []" as "none", not as "unknown").
func capNames(in []string, n int) []string {
out := make([]string, 0, len(in))
for _, x := range in {
if len(out) == n {
break
}
if len(x) > 120 {
x = x[:120]
}
out = append(out, x)
}
return out
}
func sortedKeys(m map[string][]string) []string {
out := make([]string, 0, len(m))
for k := range m {
+19
View File
@@ -400,6 +400,25 @@ func TestKernelCleanup_NightOrderOnceAndNoNames(t *testing.T) {
if len(lastRep.Kernel) != 0 || kverRE.MatchString(lastRep.ReleaseID) {
t.Fatalf("the clean-up's report names a kernel (the hub reads that as a kernel step): %+v", lastRep)
}
// R-1383 (release 3): the detail the hub's System page shows — removed, kept, gained, left-overs — rides the same
// report as its own object. On the wire it is `kernel_cleanup`, and the report STILL has no `kernel` key.
d := lastRep.KernelCleanup
if d == nil || len(d.Removed) != 1 || d.Removed[0] != "proxmox-kernel-7.0.14-20-pve-signed" ||
strings.Join(d.Kept, ",") != "7.0.14-22-pve,7.0.2-6-pve" || d.BytesGained == nil || *d.BytesGained != 1073741824 || d.Leftovers != 0 {
t.Fatalf("the clean-up's detail did not reach the hub: %+v", d)
}
wire, _ := json.Marshal(lastRep)
var keys map[string]json.RawMessage
_ = json.Unmarshal(wire, &keys)
if _, has := keys["kernel"]; has {
t.Fatalf("the clean-up's report carries a `kernel` view on the wire: %s", wire)
}
if string(keys["kernel_cleanup"]) != `{"removed":["proxmox-kernel-7.0.14-20-pve-signed"],"kept":["7.0.14-22-pve","7.0.2-6-pve"],"bytes_gained":1073741824,"leftovers":0}` {
t.Fatalf("kernel_cleanup on the wire = %s", keys["kernel_cleanup"])
}
if string(keys["release_id"]) != `"kernel-cleanup"` {
t.Fatalf("release_id on the wire = %s (a kernel version there is read as a step for it)", keys["release_id"])
}
// the same night again: the pass is skipped as a whole — never a second clean-up
l.Run(context.Background(), 9201, "night")
if len(cleanupCalls(w)) != 1 {
+24 -3
View File
@@ -139,7 +139,9 @@ type WrapperReport struct {
KernelCleanup json.RawMessage `json:"kernel_cleanup"`
// ProfileRefresh (R-1388, `11` §5.12): mode profile-refresh's result {outcome, kernel, rewritten, restarted, seconds}.
ProfileRefresh json.RawMessage `json:"profile_refresh"`
OutcomeHint string `json:"outcome_hint"`
// PVECACopy (R-1380, felhom.eu `03` §12a): mode pve-ca-copy's result {outcome, path, sha256, not_after, days_left}.
PVECACopy json.RawMessage `json:"pve_ca_copy"`
OutcomeHint string `json:"outcome_hint"`
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
// agent process that started the pass. ReleaseID / VMID were always in the report.
RunID string `json:"run_id"`
@@ -186,10 +188,23 @@ type Report struct {
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
Kernel json.RawMessage `json:"kernel,omitempty"`
// KernelCleanup (R-1383, release 3): ONLY on the mode kernel-cleanup report with outcome "cleaned" — what the
// clean-up removed, kept and gained, for the hub's System page. It is not the `kernel` view: this report still
// carries no kernel view and no kernel as its release_id, or the hub would read it as a kernel STEP for that kernel
// (the hub's kernelOf reads only those two). Pinned by TestKernelCleanup_NightOrderOnceAndNoNames.
KernelCleanup *CleanupDetail `json:"kernel_cleanup,omitempty"`
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
}
// CleanupDetail is the clean-up's result as the hub shows it (hub osupdates.CleanupDetail, field-exact).
type CleanupDetail struct {
Removed []string `json:"removed"` // package names
Kept []string `json:"kept"` // kernel versions that stay
BytesGained *int64 `json:"bytes_gained,omitempty"` // nil = the wrapper could not measure it
Leftovers int `json:"leftovers"` // paths of a removed kernel that are still there
}
// Reporter posts a report to the hub (*hub.Client).
type Reporter interface {
PostOSReport(ctx context.Context, body []byte) error
@@ -209,9 +224,15 @@ type Leg struct {
MinGap time.Duration // between night runs (default 20 h)
Now func() time.Time
Sleep func(context.Context, time.Duration)
// ProfileRecordPath is the root wrapper's record of the last security-profile refresh (0644, read-only for the
// agent); "" = DefaultProfileRecordPath. A seam for the tests.
ProfileRecordPath string
mu sync.Mutex
block *hub.WireOSUpdate
mu sync.Mutex
block *hub.WireOSUpdate
profileAsk string // how the last ask of the profile refresh ended in THIS process ("" = never asked)
profileAskAt time.Time
profileLast ProfileRefreshStatus // the last refresh as the wrapper answered it to this process
}
// planFile / reportFile: the plan the agent writes and the copy of the report the wrapper keeps beside it (R-868).
+117 -1
View File
@@ -5,11 +5,14 @@ package osupdate
// profile-refresh loads every profile from a fresh compile, once per kernel the box starts on, and restarts only a
// failed service whose own program's profile was stale. This file decides WHEN to ask: at every agent start and after
// a config bundle was installed (a new wrapper may know the mode the old one refused) — the wrapper itself answers
// "not needed" when this kernel was already done. Nothing is sent to the hub (no hub field exists for it; R-1383).
// "not needed" when this kernel was already done. No OS report is sent for it (an unknown layer reads as a GUEST step on
// the hub); since release 3 the result rides the host report as the `profile_refresh` stanza (ProfileRefreshStanza).
import (
"context"
"encoding/json"
"os"
"regexp"
"strings"
"time"
)
@@ -27,6 +30,113 @@ type ProfileRefreshResult struct {
Active bool `json:"active"`
} `json:"restarted"`
Seconds float64 `json:"seconds"`
// The wrapper's own record of the refresh BEFORE this ask (it answers them at every ask, also "not needed").
LastRefreshKernel string `json:"last_refresh_kernel"`
LastRefreshAt string `json:"last_refresh_at"`
}
// DefaultProfileRecordPath is where the root wrapper keeps the record of its last refresh (PROFILE_STATE, 0644).
const DefaultProfileRecordPath = "/var/lib/felhom-os-apply/profile-refresh.json"
// ProfileRefreshStatus is the host report's `profile_refresh` stanza (R-1383, release 3; hub sysfacts.ProfileRefresh,
// field-exact): the last refresh the box did — from the root wrapper's own record, so it outlives an agent restart —
// and how the last ask of THIS agent process ended. The refresh restarted a failed service on the N100 and nobody
// outside the box's journal could see it.
type ProfileRefreshStatus struct {
Kernel string `json:"kernel,omitempty"` // the kernel the last refresh ran on
At string `json:"at,omitempty"` // RFC3339, when
Rewritten *int `json:"rewritten,omitempty"` // how many compiled profiles changed; nil = not known to this process
Restarted []string `json:"restarted,omitempty"` // units it restarted (failed, with a stale profile)
LastAsk string `json:"last_ask,omitempty"` // refreshed | not_needed | failed | refused | no_answer
LastAskAt string `json:"last_ask_at,omitempty"`
}
var profileNameRe = regexp.MustCompile(`^[A-Za-z0-9@._:+-]{1,120}$`)
// ProfileRefreshStanza builds the stanza, or an UNTYPED nil when there is nothing to say (never asked and no record: a
// BYO host, or an agent that did not ask yet) — the report then omits the field.
//
// Two sources, the first that knows wins per field. (1) What the wrapper ANSWERED to this process's last ask: a
// refresh gives everything; "not needed" gives the kernel and time of the earlier refresh (the wrapper answers its own
// record at every ask) — so the last refresh is still told after an agent restart, without the detail. (2) The
// wrapper's record file, when this process may read it: on boxes where its folder was first made by a signed job it
// is root-only (0700), so the file is a bonus, never the only source (found by review, 2026-11-14; not read on a box).
// It never calls the wrapper. Names that are not plain unit / kernel names are dropped, the list is capped.
func (l *Leg) ProfileRefreshStanza() any {
l.mu.Lock()
st, ask, at := l.profileLast, l.profileAsk, l.profileAskAt
l.mu.Unlock()
have := ask != ""
if have {
st.LastAsk, st.LastAskAt = ask, at.UTC().Format(time.RFC3339)
}
path := l.ProfileRecordPath
if path == "" {
path = DefaultProfileRecordPath
}
if raw, err := os.ReadFile(path); err == nil && len(raw) < 64<<10 {
var rec struct {
Kernel string `json:"kernel"`
At string `json:"at"`
Rewritten []string `json:"rewritten"`
Restarted []string `json:"restarted"`
}
if json.Unmarshal(raw, &rec) == nil && (rec.Kernel != "" || rec.At != "") {
have = true
fromRec := profileStatusFrom(rec.Kernel, rec.At, rec.Rewritten, rec.Restarted, true)
if st.At == "" || fromRec.At == st.At || st.Rewritten == nil {
fromRec.LastAsk, fromRec.LastAskAt = st.LastAsk, st.LastAskAt
st = fromRec
}
}
}
if !have {
return nil
}
return st
}
// profileStatusFrom checks and bounds what goes to the hub. detail=false: the count and the units are not known.
func profileStatusFrom(kernel, at string, rewritten, restarted []string, detail bool) ProfileRefreshStatus {
var st ProfileRefreshStatus
if profileNameRe.MatchString(kernel) {
st.Kernel = kernel
}
if _, err := time.Parse(time.RFC3339, at); err == nil {
st.At = at
}
if detail {
n := len(rewritten)
st.Rewritten = &n
for _, u := range restarted {
if len(st.Restarted) < 16 && profileNameRe.MatchString(u) {
st.Restarted = append(st.Restarted, u)
}
}
}
return st
}
// noteProfileAsk remembers how an ask ended and, when the wrapper answered, what it said about the last refresh.
func (l *Leg) noteProfileAsk(outcome string, res *ProfileRefreshResult) {
l.mu.Lock()
defer l.mu.Unlock()
l.profileAsk, l.profileAskAt = outcome, l.now()
if res == nil {
return
}
switch outcome {
case "refreshed":
units := make([]string, 0, len(res.Restarted))
for _, u := range res.Restarted {
units = append(units, u.Unit)
}
l.profileLast = profileStatusFrom(res.Kernel, l.now().UTC().Format(time.RFC3339), res.Rewritten, units, true)
case "not_needed":
if res.LastRefreshAt != "" || res.LastRefreshKernel != "" {
l.profileLast = profileStatusFrom(res.LastRefreshKernel, res.LastRefreshAt, nil, nil, false)
}
}
}
// ProfileGate takes the heavy-op gate (a backup, a restore-test or an OS step must not run beside the refresh).
@@ -69,12 +179,15 @@ func (l *Leg) ProfileRefresh(ctx context.Context, why string, gate ProfileGate,
switch {
case err != nil:
lg.Warn("osupdate: profile refresh — the wrapper gave no answer; the next agent start asks again", "err", err, "took", took)
l.noteProfileAsk("no_answer", nil)
return ""
case wr.refused():
lg.Info("osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)", "refused", string(wr.Refused))
l.noteProfileAsk("refused", nil)
return ""
case wr.failed():
lg.Error("osupdate: profile refresh FAILED — the profiles are as before; the next agent start asks again", "failed", string(wr.Failed), "took", took)
l.noteProfileAsk("failed", nil)
return "failed"
}
var res ProfileRefreshResult
@@ -97,5 +210,8 @@ func (l *Leg) ProfileRefresh(ctx context.Context, why string, gate ProfileGate,
default:
lg.Warn("osupdate: profile refresh — the wrapper answered with no outcome this agent knows", "outcome", res.Outcome)
}
if res.Outcome == "refreshed" || res.Outcome == "not_needed" {
l.noteProfileAsk(res.Outcome, &res)
}
return res.Outcome
}
+72
View File
@@ -3,6 +3,9 @@ package osupdate
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -95,3 +98,72 @@ func TestProfileRefresh_RefusedAndFailed(t *testing.T) {
t.Fatalf("failed → %q", got)
}
}
// R-1383 (release 3): the refresh restarted a failed service on the N100 and only the box's journal knew. The stanza
// that rides the host report: nothing before any ask (an UNTYPED nil — the report omits the field); after a refresh,
// everything; after an agent RESTART, the kernel and time of that refresh again — from the wrapper's answer to the new
// process's ask, because the wrapper's record FILE may be root-only on a box (its folder is 0700 where a signed job
// made it first). Where the file can be read, it adds the detail back.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-b3.txt).
func TestProfileRefreshStanza_RecordAndLastAsk(t *testing.T) {
w := &fakeWrapper{t: t, profileRep: refreshed()}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
unreadable := filepath.Join(t.TempDir(), "root-only", "profile-refresh.json") // does not exist: as a 0700 folder reads
l.ProfileRecordPath = unreadable
now := time.Date(2026, 11, 14, 11, 30, 11, 0, time.UTC)
l.Now = func() time.Time { return now }
if st := l.ProfileRefreshStanza(); st != nil {
t.Fatalf("before any ask the stanza is %#v, want an untyped nil", st)
}
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "refreshed" {
t.Fatalf("outcome = %q", got)
}
b, _ := json.Marshal(l.ProfileRefreshStanza())
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":1,"restarted":["chrony.service"],"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"}` {
t.Fatalf("stanza after a refresh, record file unreadable = %s", b)
}
// an agent restart: a new process asks, the wrapper answers "not needed" with ITS record of the last refresh
l2, _ := newLeg(t, &fakeWrapper{t: t, profileRep: &WrapperReport{ProfileRefresh: json.RawMessage(
`{"outcome":"not_needed","reason":"this kernel was done","kernel":"7.0.14-28-pve","last_refresh_kernel":"7.0.14-28-pve","last_refresh_at":"2026-11-14T11:30:11Z"}`)}},
&hub.WireOSUpdate{Ring: 1, Enabled: true})
l2.ProfileRecordPath = unreadable
l2.Now = func() time.Time { return now.Add(3 * time.Hour) }
if st := l2.ProfileRefreshStanza(); st != nil {
t.Fatalf("a new process that has not asked and cannot read the record: %#v, want nil", st)
}
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "not_needed" {
t.Fatalf("outcome = %q", got)
}
b, _ = json.Marshal(l2.ProfileRefreshStanza())
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
t.Fatalf("stanza after an agent restart (the wrapper's answer, no detail) = %s", b)
}
// where the record file CAN be read, the detail is back — and a name that is not a unit name is dropped
rec := filepath.Join(t.TempDir(), "profile-refresh.json")
if err := os.WriteFile(rec, []byte(`{"at":"2026-11-14T11:30:11Z","kernel":"7.0.14-28-pve","restarted":["chrony.service","bad name; rm -rf"],`+
`"rewritten":["a","b"]}`), 0o644); err != nil {
t.Fatal(err)
}
l2.ProfileRecordPath = rec
b, _ = json.Marshal(l2.ProfileRefreshStanza())
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":2,"restarted":["chrony.service"],"last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
t.Fatalf("stanza with a readable record = %s", b)
}
// a failed ask is told, beside the older record
l2.Runner = (&fakeWrapper{t: t, profileRep: &WrapperReport{Failed: json.RawMessage(`{"code":"F1","reason":"apparmor_parser failed"}`)}})
l2.Now = func() time.Time { return now.Add(4 * time.Hour) }
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "failed" {
t.Fatalf("outcome = %q", got)
}
b, _ = json.Marshal(l2.ProfileRefreshStanza())
if !strings.Contains(string(b), `"last_ask":"failed","last_ask_at":"2026-11-14T15:30:11Z"`) || !strings.Contains(string(b), `"kernel":"7.0.14-28-pve"`) {
t.Fatalf("stanza after a failed ask = %s", b)
}
// a record that is not JSON says nothing and breaks nothing
_ = os.WriteFile(rec, []byte("not json"), 0o644)
l3, _ := newLeg(t, &fakeWrapper{t: t}, &hub.WireOSUpdate{Ring: 1, Enabled: true})
l3.ProfileRecordPath = rec
if st := l3.ProfileRefreshStanza(); st != nil {
t.Fatalf("an unreadable record gave the stanza %#v", st)
}
}
+45
View File
@@ -0,0 +1,45 @@
package osupdate
// The root step that keeps a readable copy of Proxmox's root CA for the agent (R-1380, D80, felhom.eu `03` §12a).
// The rule lives in the wrapper (mode pve-ca-copy: one fixed source, one fixed place, a CA certificate only); the plan
// names nothing. internal/pvetrust decides when to ask and what to do with the answer.
import (
"context"
"encoding/json"
"fmt"
)
// PVECACopyPath is where the wrapper puts the copy (felhom-os-apply PVE_CA_COPY) — the agent's `proxmox.tls.ca_file`.
const PVECACopyPath = "/etc/felhom-agent/pve-root-ca.pem"
// PVECACopyResult is the wrapper's pve_ca_copy object.
type PVECACopyResult struct {
Outcome string `json:"outcome"` // written | same
Path string `json:"path"`
SHA256 string `json:"sha256"`
NotAfter string `json:"not_after"`
DaysLeft int `json:"days_left"`
Replaced bool `json:"replaced"`
}
// PVECACopy asks the wrapper to make or refresh the copy. An error names a refusal (T1–T3, or R11 from a wrapper that
// does not know the mode yet) or a wrapper that gave no answer.
func (l *Leg) PVECACopy(ctx context.Context) (PVECACopyResult, error) {
wr, err := l.call(ctx, "pveca"+l.now().UTC().Format("150405.000"), map[string]any{"release_id": "pve-ca-copy",
"layer": LayerHost, "mode": "pve-ca-copy"})
if err != nil {
return PVECACopyResult{}, err
}
if wr.refused() {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy refused: %s", wr.Refused)
}
if wr.failed() {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy failed: %s", wr.Failed)
}
var res PVECACopyResult
if jerr := json.Unmarshal(wr.PVECACopy, &res); jerr != nil || res.Outcome == "" || res.Path == "" {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy: the wrapper gave no result (%v)", jerr)
}
return res, nil
}
+21
View File
@@ -0,0 +1,21 @@
package pbsdr
import (
"os"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
// agent's state folder on a box, and a test must not need it.
func TestMain(m *testing.M) {
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
if err != nil {
panic(err)
}
config.CutWriteDir = dir
code := m.Run()
_ = os.RemoveAll(dir)
os.Exit(code)
}
+8 -17
View File
@@ -27,6 +27,7 @@ import (
"encoding/json"
"errors"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"log/slog"
"net/netip"
"os"
@@ -483,8 +484,10 @@ func (m *Manager) consumedFail(h, msg string, block *hub.WirePBSDR) {
// seedEscrowStorageID sets escrow.pbs_storage_id in agent.json when empty/absent — the bare
// `--selftest=escrow-create` one-liner must find its storage with no flags. A different existing
// value is NEVER clobbered (warn-and-keep). Unknown config keys are preserved verbatim
// (map[string]RawMessage read-modify-write, atomic rename).
// (map[string]RawMessage read-modify-write, in place behind a journal — config.ReplaceInPlace).
func (m *Manager) seedEscrowStorageID(storageID string) error {
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
defer config.FileMu.Unlock()
if m.configPath == "" {
return nil
}
@@ -527,30 +530,18 @@ func (m *Manager) seedEscrowStorageID(storageID string) error {
if err != nil {
return err
}
st, err := os.Stat(m.configPath)
if err != nil {
return err
}
// The config DIRECTORY (/etc/felhom-agent) is root-owned while the FILE is agent-owned —
// tmp+rename is impossible for the non-root agent (rename mutates the directory; live
// finding on the demo). So: park a recovery copy in the agent state dir, then rewrite the
// file IN PLACE (O_TRUNC). The agent is the file's only writer and the content is small;
// a torn write is recoverable from the parked copy.
// finding on the demo). So: park a copy for a person in the agent state dir, then rewrite the
// file IN PLACE through config.ReplaceInPlace, which journals the old bytes first — the next
// start repairs a cut write (R-1380, TestRecoverCutWrite_*).
if err := os.MkdirAll(m.stateDir, 0o700); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(m.stateDir, "agent.json.pre-seed"), raw, 0o600); err != nil {
return fmt.Errorf("parking the pre-seed copy: %w", err)
}
f, err := os.OpenFile(m.configPath, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
return err
}
if _, err := f.Write(out); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
if err := config.ReplaceInPlace(m.configPath, raw, out); err != nil {
return err
}
m.logger.Info("pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)", "storage_id", storageID)
+40 -5
View File
@@ -10,6 +10,7 @@ import (
"net/http"
"net/url"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
@@ -45,6 +46,39 @@ type Client struct {
node string
token string
http doer
// tlsMu guards http: SetTLS swaps the transport while calls run (R-1380 — the move from a pinned certificate to the
// box's own root CA, and a refreshed CA copy, take effect with no agent restart).
tlsMu sync.RWMutex
timeout time.Duration
}
func (c *Client) httpDoer() doer {
c.tlsMu.RLock()
defer c.tlsMu.RUnlock()
return c.http
}
// SetTLS replaces how this client trusts the Proxmox certificate, for every call that starts after it returns. On an
// error (an unreadable CA file, a bad fingerprint) nothing changes.
func (c *Client) SetTLS(t TLSConfig) error {
tlsCfg, err := t.build()
if err != nil {
return err
}
timeout := c.timeout
if timeout == 0 {
timeout = 30 * time.Second
}
hc := &http.Client{Timeout: timeout, Transport: httpx.NewTransport(tlsCfg, 0)}
c.tlsMu.Lock()
old := c.http
c.http = hc
c.tlsMu.Unlock()
if o, ok := old.(*http.Client); ok && o != nil {
o.CloseIdleConnections() // a kept-alive connection made under the old trust must not serve the next call
}
return nil
}
// NewClient builds an API client. It validates required config and constructs the
@@ -74,10 +108,11 @@ func NewClient(cfg Config) (*Client, error) {
Transport: httpx.NewTransport(tlsCfg, 0),
}
return &Client{
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
node: cfg.Node,
token: cfg.Token,
http: hc,
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
node: cfg.Node,
token: cfg.Token,
http: hc,
timeout: timeout,
}, nil
}
@@ -125,7 +160,7 @@ func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader
req.Header.Set("Content-Type", contentType)
}
resp, err := c.http.Do(req)
resp, err := c.httpDoer().Do(req)
if err != nil {
return fmt.Errorf("proxmox: %s %s: %w", method, path, err)
}
+43
View File
@@ -0,0 +1,43 @@
package proxmox
import (
"crypto/x509"
"errors"
"strings"
)
// The causes a call to Proxmox can fail on TRUST (R-1380, felhom.eu `03` §12a). A symptom alarm („the agent channel is
// down") exists; these name the cause.
const (
TrustPinnedCertChanged = "pinned_cert_changed" // the pinned certificate is not the one Proxmox serves any more
TrustCAMismatch = "ca_mismatch" // the CA file did not sign the served certificate
TrustCertExpired = "cert_expired" // the served certificate (or the CA) has ended
TrustCAFileUnreadable = "ca_file_unreadable" // the CA file is missing or holds no certificate
TrustNameMismatch = "name_mismatch" // the certificate does not name the address the agent calls
)
// TrustProblem names why err is a TRUST failure, or "" when it is none (a refused connection, a 401, a timeout).
func TrustProblem(err error) string {
if err == nil {
return ""
}
var ua x509.UnknownAuthorityError
var inv x509.CertificateInvalidError
var hn x509.HostnameError
switch {
case strings.Contains(err.Error(), "TLS pin mismatch"):
return TrustPinnedCertChanged
case errors.As(err, &ua):
return TrustCAMismatch
case errors.As(err, &inv):
if inv.Reason == x509.Expired {
return TrustCertExpired
}
return TrustCAMismatch
case errors.As(err, &hn):
return TrustNameMismatch
case strings.Contains(err.Error(), "reading TLS CA file"), strings.Contains(err.Error(), "contained no usable certificates"):
return TrustCAFileUnreadable
}
return ""
}
+21
View File
@@ -0,0 +1,21 @@
package pvetrust
import (
"os"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
// agent's state folder on a box, and a test must not need it.
func TestMain(m *testing.M) {
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
if err != nil {
panic(err)
}
config.CutWriteDir = dir
code := m.Run()
_ = os.RemoveAll(dir)
os.Exit(code)
}
+406
View File
@@ -0,0 +1,406 @@
// Package pvetrust moves the agent's trust in Proxmox from ONE pinned certificate to the box's own root CA, and names
// the cause when a call fails on trust (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a).
//
// WHY. Proxmox makes a new node certificate by itself, at most every two years. An agent that pins the old one is then
// blind: every call fails, no host report reaches the hub, and the only alarm names a symptom. A copy of the box's own
// root CA verifies every certificate that CA signs.
//
// THE ORDER, and it is the whole point (a box must never be left with neither way working):
//
// 1. the root step makes or refreshes the copy (it refuses anything that is not one CA certificate);
// 2. a REAL call to Proxmox is made with the copy ALONE — while the fingerprint still is the configured trust;
// 3. only after that call worked, ONE write of the agent's config sets `ca_file` and clears the fingerprint;
// 4. the running client takes the new trust (no restart).
//
// If 1 or 2 fails nothing is written and the fingerprint stays. If 3 fails the fingerprint stays, in the file and in
// the running client. `proxmox.tls` keeps its order in code — a set fingerprint wins over a CA file — so the
// fingerprint is cleared LAST, in the same write that sets `ca_file`.
//
// WHO WRITES THE CONFIG. The agent itself: `agent.json` is the agent's own file (0600) in a root-owned folder, so it is
// rewritten in place after a recovery copy was parked in the agent's state folder — the same way the backup-target move
// and the PBS seed write it. No root step touches it.
package pvetrust
import (
"context"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"log/slog"
"os"
"path/filepath"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// EndsSoonDays: the copy's certificate ends within this many days → the problem „ca_copy_ends_soon". Proxmox renews its
// root CA when less than two years are left, so a healthy box never comes near it.
const EndsSoonDays = 30
// Problems this package names beyond proxmox.Trust*.
const (
ProblemCopyUnavailable = "ca_copy_unavailable" // the root step refused or did not answer
ProblemCopyNoProof = "ca_copy_does_not_verify" // a call with the copy alone failed — the fingerprint stays
ProblemCopyEndsSoon = "ca_copy_ends_soon"
ProblemConfigNotSaved = "config_not_written" // the proof worked but the config could not be written — the fingerprint stays
)
// Copy is what the root step answered.
type Copy struct {
Path string
Outcome string // written | same
NotAfter string
DaysLeft int
}
// Status is the stanza that rides the host report (`proxmox_trust`). No secret: a mode, a cause, a date.
type Status struct {
Mode string `json:"mode"` // ca | fingerprint | ca_file_own | insecure | system
Problem string `json:"problem,omitempty"` // "" = none
Detail string `json:"detail,omitempty"`
CANotAfter string `json:"ca_not_after,omitempty"`
CADaysLeft *int `json:"ca_days_left,omitempty"`
MovedAt string `json:"moved_at,omitempty"` // when this box moved from the pinned certificate to the CA copy
CheckedAt string `json:"checked_at,omitempty"`
}
// Manager holds the trust state of one agent process.
type Manager struct {
// EnsureCopy runs the root step (osupdate.Leg.PVECACopy).
EnsureCopy func(ctx context.Context) (Copy, error)
// Probe makes one real call to Proxmox with EXACTLY this trust (a throwaway client).
Probe func(ctx context.Context, t proxmox.TLSConfig) error
// Apply gives the running client the new trust (proxmox.Client.SetTLS).
Apply func(t proxmox.TLSConfig) error
// ConfigPath is agent.json ("" = the config came from the environment: nothing can be saved, so nothing is moved).
ConfigPath string
StateDir string
Logger *slog.Logger
Now func() time.Time
// MinGap between two runs that a failed call asked for (default 10 minutes).
MinGap time.Duration
runMu sync.Mutex // serialises Ensure
mu sync.Mutex
cur proxmox.TLSConfig
status Status
lastRun time.Time
}
// copyStepTimeout bounds the root step that makes the CA copy.
const copyStepTimeout = 90 * time.Second
func (m *Manager) now() time.Time {
if m.Now != nil {
return m.Now()
}
return time.Now()
}
func (m *Manager) log() *slog.Logger {
if m.Logger != nil {
return m.Logger
}
return slog.Default()
}
// Start records the trust the process started with. Call once, before Ensure.
func (m *Manager) Start(t proxmox.TLSConfig) {
m.mu.Lock()
defer m.mu.Unlock()
m.cur = t
m.status = Status{Mode: modeOf(t)}
}
func modeOf(t proxmox.TLSConfig) string {
switch {
case t.InsecureSkipVerify:
return "insecure"
case t.Fingerprint != "":
return "fingerprint"
case t.CAFile != "":
return "ca"
}
return "system"
}
// Status is the current stanza (nil-safe).
func (m *Manager) Status() *Status {
if m == nil {
return nil
}
m.mu.Lock()
defer m.mu.Unlock()
s := m.status
return &s
}
// Moved reports whether the box trusts Proxmox by the CA copy (nothing is left to move).
func (m *Manager) Moved() bool {
m.mu.Lock()
defer m.mu.Unlock()
return m.cur.Fingerprint == "" && m.cur.CAFile != ""
}
// Run asks again on a timer until ctx ends: every `pending` while the box is not moved yet (a wrapper that did not
// know the mode, a Proxmox that did not answer), every `settled` after (a renewed CA refreshes the copy).
func (m *Manager) Run(ctx context.Context, pending, settled time.Duration) {
for {
d := settled
if !m.Moved() {
d = pending
}
select {
case <-ctx.Done():
return
case <-time.After(d):
m.Ensure(ctx, "timer")
}
}
}
// Current is the trust the running client uses.
func (m *Manager) Current() proxmox.TLSConfig {
m.mu.Lock()
defer m.mu.Unlock()
return m.cur
}
// OnCallError is the hook for a failed Proxmox call: when the failure is one of TRUST, it names the cause and runs
// Ensure (at most once per MinGap) — a pinned box whose certificate changed moves to the CA copy right then, and a box
// whose CA copy was refreshed reloads it. A failure of another kind does nothing.
func (m *Manager) OnCallError(ctx context.Context, err error) {
cause := proxmox.TrustProblem(err)
if cause == "" {
return
}
m.mu.Lock()
gap := m.MinGap
if gap <= 0 {
gap = 10 * time.Minute
}
tooSoon := !m.lastRun.IsZero() && m.now().Sub(m.lastRun) < gap
if m.status.Problem == "" {
m.status.Problem, m.status.Detail = cause, causeText(cause)
}
m.mu.Unlock()
m.log().Error("pvetrust: a call to Proxmox failed on TRUST — "+causeText(cause), "cause", cause, "err", err)
if tooSoon {
return
}
m.Ensure(ctx, cause)
}
func causeText(cause string) string {
switch cause {
case proxmox.TrustPinnedCertChanged:
return "the pinned certificate is not the one Proxmox serves any more (Proxmox made a new certificate)"
case proxmox.TrustCAMismatch:
return "the CA copy did not sign the certificate Proxmox serves (the box's root CA changed, or the copy is not this box's)"
case proxmox.TrustCertExpired:
return "the certificate Proxmox serves, or its CA, has ended"
case proxmox.TrustCAFileUnreadable:
return "the CA copy is missing or holds no certificate"
case proxmox.TrustNameMismatch:
return "the certificate Proxmox serves does not name the address the agent calls"
}
return cause
}
// Ensure runs the order of the package comment once. why is a log word ("start", "daily", a trust cause). It returns
// true when the running trust changed.
func (m *Manager) Ensure(ctx context.Context, why string) bool {
// One run at a time. Ensure is called from the start, the timer, a failed collect and after a bundle install; two
// overlapping runs would both read the config, and the second would park the ALREADY-MOVED file as the "before"
// copy (losing the fingerprint from the recovery copy). Found by review, 2026-11-14.
m.runMu.Lock()
defer m.runMu.Unlock()
m.mu.Lock()
cur := m.cur
m.lastRun = m.now()
m.mu.Unlock()
lg := m.log().With("why", why, "mode", modeOf(cur))
set := func(mode, problem, detail string, c *Copy) {
m.mu.Lock()
defer m.mu.Unlock()
moved := m.status.MovedAt
m.status = Status{Mode: mode, Problem: problem, Detail: detail, MovedAt: moved, CheckedAt: m.now().UTC().Format(time.RFC3339)}
if c != nil {
d := c.DaysLeft
m.status.CANotAfter, m.status.CADaysLeft = c.NotAfter, &d
}
}
if cur.InsecureSkipVerify {
lg.Warn("pvetrust: certificate checks are switched off in the config (insecure_skip_verify) — nothing is moved")
set("insecure", "", "", nil)
return false
}
if m.EnsureCopy == nil || m.Probe == nil {
return false
}
// 1. the copy. The root step gets a deadline of its own: Ensure runs on the daemon's start path and inside the
// report loop, and a wrapper that hangs (sudo, a stuck read of /etc/pve) must not stop either.
cctx, ccancel := context.WithTimeout(ctx, copyStepTimeout)
c, err := m.EnsureCopy(cctx)
ccancel()
if err != nil {
lg.Warn("pvetrust: the root step gave no CA copy — the trust stays as it is", "err", err)
set(modeOf(cur), ProblemCopyUnavailable, truncate(err.Error(), 240), nil)
return false
}
ca := proxmox.TLSConfig{CAFile: c.Path}
onCopy := cur.Fingerprint == "" && cur.CAFile == c.Path
if cur.Fingerprint == "" && cur.CAFile != "" && cur.CAFile != c.Path {
// an operator's own CA file: never replaced
lg.Info("pvetrust: the config names its own CA file — left as it is", "ca_file", cur.CAFile)
set("ca_file_own", "", "", &c)
return false
}
if onCopy && c.Outcome != "written" && why != proxmox.TrustCAMismatch && why != proxmox.TrustCAFileUnreadable && why != proxmox.TrustCertExpired {
// the box already trusts by the copy and the copy did not change: nothing to prove again
problem, detail := "", ""
if c.DaysLeft < EndsSoonDays {
problem, detail = ProblemCopyEndsSoon, fmt.Sprintf("the CA copy ends in %d day(s), on %s", c.DaysLeft, c.NotAfter)
lg.Error("pvetrust: "+detail, "days_left", c.DaysLeft)
}
set("ca", problem, detail, &c)
return false
}
// 2. the proof: a real call with the copy ALONE — the fingerprint is still the configured trust
pctx, cancel := context.WithTimeout(ctx, 30*time.Second)
perr := m.Probe(pctx, ca)
cancel()
if perr != nil {
cause := proxmox.TrustProblem(perr)
detail := "a call to Proxmox with the CA copy alone failed"
if cause != "" {
detail += ": " + causeText(cause)
} else {
detail += " (not a trust failure: " + truncate(perr.Error(), 160) + ")"
}
if onCopy {
lg.Error("pvetrust: the box trusts Proxmox by the CA copy and the copy does not verify the served certificate", "err", perr)
set("ca", firstNonEmpty(cause, ProblemCopyNoProof), detail, &c)
return false
}
if cause == "" {
// Proxmox did not answer at all (it restarts, the box just booted): nothing is known about the copy, so no
// cause is named; the next run asks again.
lg.Warn("pvetrust: the proof could not be made now (not a trust failure) — the trust stays as it is; asked again later", "err", perr)
set(modeOf(cur), "", "", &c)
return false
}
lg.Error("pvetrust: the proof with the CA copy FAILED — the fingerprint STAYS, nothing was written", "err", perr)
set(modeOf(cur), ProblemCopyNoProof, detail+" — the fingerprint stays", &c)
return false
}
if onCopy {
// the copy was refreshed (or a trust failure asked): the running client reloads it
if m.Apply != nil {
if aerr := m.Apply(ca); aerr != nil {
set("ca", proxmox.TrustCAFileUnreadable, truncate(aerr.Error(), 200), &c)
return false
}
}
lg.Warn("pvetrust: the CA copy was refreshed and proven — the running client uses it now", "outcome", c.Outcome)
set("ca", "", "", &c)
return true
}
// 3. the config: ONE write sets ca_file and clears the fingerprint — only now, after the proof
if m.ConfigPath == "" {
lg.Warn("pvetrust: the proof worked, but the config did not come from a file — nothing can be saved, so the trust stays as it is")
set(modeOf(cur), ProblemConfigNotSaved, "the config is not a file; the move was not made", &c)
return false
}
if werr := m.writeConfig(c.Path); werr != nil {
lg.Error("pvetrust: the proof worked but the config could NOT be written — the fingerprint stays", "err", werr)
set(modeOf(cur), ProblemConfigNotSaved, truncate(werr.Error(), 200), &c)
return false
}
// 4. the running client
if m.Apply != nil {
if aerr := m.Apply(ca); aerr != nil {
lg.Error("pvetrust: the config is written but the running client could not take the CA copy — it keeps the old trust until the next start", "err", aerr)
}
}
m.mu.Lock()
m.cur = ca
m.mu.Unlock()
lg.Warn("pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy",
"ca_file", c.Path, "ca_not_after", c.NotAfter)
set("ca", "", "", &c)
m.mu.Lock()
m.status.MovedAt = m.now().UTC().Format(time.RFC3339)
m.mu.Unlock()
return true
}
func firstNonEmpty(a, b string) string {
if a != "" {
return a
}
return b
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
// writeConfig sets proxmox.tls.ca_file and clears proxmox.tls.fingerprint in ONE write of agent.json. Every other key is
// kept byte for byte. A recovery copy is parked in the state folder first; the result is read back and checked.
func (m *Manager) writeConfig(caPath string) error {
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
defer config.FileMu.Unlock()
raw, err := os.ReadFile(m.ConfigPath)
if err != nil {
return err
}
var doc map[string]json.RawMessage
if err := json.Unmarshal(raw, &doc); err != nil {
return fmt.Errorf("parse %s: %w", m.ConfigPath, err)
}
px := map[string]json.RawMessage{}
if cur, ok := doc["proxmox"]; ok {
if err := json.Unmarshal(cur, &px); err != nil {
return fmt.Errorf("parse the proxmox section: %w", err)
}
}
tlsSec := map[string]json.RawMessage{}
if cur, ok := px["tls"]; ok {
if err := json.Unmarshal(cur, &tlsSec); err != nil {
return fmt.Errorf("parse the proxmox.tls section: %w", err)
}
}
caJSON, _ := json.Marshal(caPath)
tlsSec["ca_file"] = caJSON
tlsSec["fingerprint"] = json.RawMessage(`""`)
b, _ := json.Marshal(tlsSec)
px["tls"] = b
b, _ = json.Marshal(px)
doc["proxmox"] = b
out, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return err
}
out = append(out, '\n')
if m.StateDir != "" {
if err := os.MkdirAll(m.StateDir, 0o700); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(m.StateDir, "agent.json.pre-pve-trust"), raw, 0o600); err != nil {
return fmt.Errorf("parking the recovery copy: %w", err)
}
}
// In place, not tmp+rename: the folder is root-owned and the file is the agent's. config.ReplaceInPlace keeps the
// old bytes in a journal first, so a power cut in the middle is repaired at the next start (R-1380); it also reads
// the file back and puts the old bytes back when the write fails with the agent alive.
if err := config.ReplaceInPlace(m.ConfigPath, raw, out); err != nil {
return err
}
return nil
}
+489
View File
@@ -0,0 +1,489 @@
package pvetrust
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"math/big"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ---- a small Proxmox: a root CA that signs node certificates, and a server that can change its certificate ----
type pki struct {
t *testing.T
caKey *ecdsa.PrivateKey
caCert *x509.Certificate
caPEM []byte
}
func newPKI(t *testing.T, cn string) *pki {
t.Helper()
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
return (&pki{t: t, caKey: k}).renewCA(cn)
}
// renewCA makes a new CA certificate with the SAME key and name (what Proxmox does when its root CA is renewed).
func (p *pki) renewCA(cn string) *pki {
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: cn},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true,
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign}
der, err := x509.CreateCertificate(rand.Reader, tpl, tpl, &p.caKey.PublicKey, p.caKey)
if err != nil {
p.t.Fatal(err)
}
p.caCert, _ = x509.ParseCertificate(der)
p.caPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
return p
}
func (p *pki) leaf() tls.Certificate {
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: "node"},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(12 * time.Hour), IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}
der, err := x509.CreateCertificate(rand.Reader, tpl, p.caCert, &k.PublicKey, p.caKey)
if err != nil {
p.t.Fatal(err)
}
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: k}
}
func fp(c tls.Certificate) string {
s := sha256.Sum256(c.Certificate[0])
return hex.EncodeToString(s[:])
}
type pve struct {
srv *httptest.Server
mu sync.Mutex
cert tls.Certificate
}
func newPVE(t *testing.T, c tls.Certificate) *pve {
p := &pve{cert: c}
p.srv = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprint(w, `{"data":{"version":"9.2.2","release":"9.2","repoid":"x"}}`)
}))
// GetConfigForClient, not GetCertificate: httptest adds its own certificate, and for a call by IP address (no
// server name) that one would win over GetCertificate.
p.srv.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
p.mu.Lock()
defer p.mu.Unlock()
return &tls.Config{Certificates: []tls.Certificate{p.cert}}, nil
}}
p.srv.StartTLS()
t.Cleanup(p.srv.Close)
return p
}
// newCert is `pveupdate` making a new node certificate: served from the next connection on.
func (p *pve) newCert(c tls.Certificate) {
p.mu.Lock()
p.cert = c
p.mu.Unlock()
p.srv.CloseClientConnections()
}
// ---- the agent's side ----
type box struct {
t *testing.T
m *Manager
px *proxmox.Client
cfgPath string
copyPath string
source []byte // what the root step would copy (nil = the root step refuses)
events []string
applyErr error
}
func cfgJSON(fingerprint, caFile string) string {
return fmt.Sprintf(`{
"hub": {"url": "https://hub.example", "host_id": "box-1", "api_key": "KEEP-ME"},
"proxmox": {"endpoint": "X", "node": "n", "token": "u@pve!t=s", "tls": {"fingerprint": %q, "ca_file": %q, "insecure_skip_verify": false}, "custom": [1, 2, 3]},
"unknown_key": {"stays": true}
}`, fingerprint, caFile)
}
func newBox(t *testing.T, srv *pve, start proxmox.TLSConfig, source []byte) *box {
t.Helper()
dir := t.TempDir()
b := &box{t: t, cfgPath: filepath.Join(dir, "agent.json"), copyPath: filepath.Join(dir, "pve-root-ca.pem"), source: source}
if start.CAFile == "COPY" {
start.CAFile = b.copyPath
if err := os.WriteFile(b.copyPath, source, 0o644); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(b.cfgPath, []byte(cfgJSON(start.Fingerprint, start.CAFile)), 0o600); err != nil {
t.Fatal(err)
}
pcfg := proxmox.Config{Endpoint: srv.srv.URL, Node: "n", Token: "u@pve!t=s", TLS: start, HTTPTimeout: 5 * time.Second}
px, err := proxmox.NewClient(pcfg)
if err != nil {
t.Fatal(err)
}
b.px = px
b.m = &Manager{ConfigPath: b.cfgPath, StateDir: filepath.Join(dir, "state"),
EnsureCopy: func(context.Context) (Copy, error) {
b.events = append(b.events, "copy")
if b.source == nil {
return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3", "reason": "not a CA certificate"}`)
}
outcome := "same"
if cur, _ := os.ReadFile(b.copyPath); string(cur) != string(b.source) {
outcome = "written"
if err := os.WriteFile(b.copyPath, b.source, 0o644); err != nil {
t.Fatal(err)
}
}
return Copy{Path: b.copyPath, Outcome: outcome, NotAfter: "2036-08-17T15:44:42Z", DaysLeft: 3560}, nil
},
Probe: func(ctx context.Context, tl proxmox.TLSConfig) error {
b.events = append(b.events, fmt.Sprintf("probe(fp=%q,ca=%v) while the file says fp=%q", tl.Fingerprint, tl.CAFile != "", b.fileTLS().Fingerprint))
c := pcfg
c.TLS = tl
p, err := proxmox.NewClient(c)
if err != nil {
return err
}
_, err = p.Version(ctx)
return err
},
Apply: func(tl proxmox.TLSConfig) error {
b.events = append(b.events, fmt.Sprintf("apply(fp=%q,ca=%v)", tl.Fingerprint, tl.CAFile != ""))
if b.applyErr != nil {
return b.applyErr
}
return b.px.SetTLS(tl)
},
}
b.m.Start(start)
return b
}
type fileTLS struct {
Fingerprint string `json:"fingerprint"`
CAFile string `json:"ca_file"`
}
func (b *box) fileTLS() fileTLS {
var d struct {
Proxmox struct {
TLS fileTLS `json:"tls"`
} `json:"proxmox"`
}
raw, _ := os.ReadFile(b.cfgPath)
if err := json.Unmarshal(raw, &d); err != nil {
b.t.Fatalf("agent.json is not JSON any more: %v\n%s", err, raw)
}
return d.Proxmox.TLS
}
func (b *box) call() error {
_, err := b.px.Version(context.Background())
return err
}
func (b *box) has(prefix string) bool {
for _, e := range b.events {
if strings.HasPrefix(e, prefix) {
return true
}
}
return false
}
// Scenario: a NEW box — the installer wrote the copy and ca_file, no fingerprint. Nothing is proven again, nothing written.
func TestNewBox_CAOnly_NothingToMove(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("a box that already trusts by the copy changed its trust")
}
after, _ := os.ReadFile(b.cfgPath)
if string(before) != string(after) || b.has("probe") || b.has("apply") {
t.Fatalf("the config was touched or a proof ran: %v", b.events)
}
if st := b.m.Status(); st.Mode != "ca" || st.Problem != "" || st.CADaysLeft == nil || *st.CADaysLeft != 3560 {
t.Fatalf("status = %+v", st)
}
if err := b.call(); err != nil {
t.Fatalf("the call fails: %v", err)
}
}
// Scenario: an OLD box moves. The proof runs with the CA copy ALONE while the file still holds the fingerprint; only
// then ONE write sets ca_file and clears the fingerprint; every other key stays; a recovery copy is parked.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/A/redproof-agent.txt): write the config
// BEFORE the probe → the order check fails; skip the probe → TestProofFails_TheFingerprintStays fails.
func TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
old, _ := os.ReadFile(b.cfgPath)
if !b.m.Ensure(context.Background(), "start") {
t.Fatalf("the box did not move: %+v %v", b.m.Status(), b.events)
}
want := []string{"copy", fmt.Sprintf("probe(fp=\"\",ca=true) while the file says fp=%q", fp(leaf)), "apply(fp=\"\",ca=true)"}
if strings.Join(b.events, " | ") != strings.Join(want, " | ") {
t.Fatalf("order = %v\nwant %v", b.events, want)
}
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
t.Fatalf("the config's tls = %+v", got)
}
raw, _ := os.ReadFile(b.cfgPath)
for _, keep := range []string{`"KEEP-ME"`, `"unknown_key"`, `"stays"`, `"custom"`, `"u@pve!t=s"`, `"insecure_skip_verify"`} {
if !strings.Contains(string(raw), keep) {
t.Fatalf("the write lost %s:\n%s", keep, raw)
}
}
if parked, _ := os.ReadFile(filepath.Join(b.m.StateDir, "agent.json.pre-pve-trust")); string(parked) != string(old) {
t.Fatal("no recovery copy of the old config was parked")
}
if st := b.m.Status(); st.Mode != "ca" || st.MovedAt == "" || st.Problem != "" {
t.Fatalf("status = %+v", st)
}
// and now Proxmox makes a new certificate: the agent goes on, with no restart and no new client
srv.newCert(p.leaf())
if err := b.call(); err != nil {
t.Fatalf("after a new Proxmox certificate the moved box fails: %v", err)
}
}
// Scenario: the proof fails (the copy is another box's CA). The fingerprint stays — in the file, byte for byte, and in
// the running client — and the cause is named.
func TestProofFails_TheFingerprintStays(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
other := newPKI(t, "another box")
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("the trust changed although the proof failed")
}
after, _ := os.ReadFile(b.cfgPath)
if string(before) != string(after) {
t.Fatal("the config was written although the proof failed")
}
if b.has("apply") {
t.Fatalf("the running client was changed: %v", b.events)
}
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemCopyNoProof || !strings.Contains(st.Detail, "did not sign") {
t.Fatalf("status = %+v", st)
}
if err := b.call(); err != nil {
t.Fatalf("the fingerprint no longer works: %v", err)
}
}
// Scenario: Proxmox makes a new certificate on a box that still PINS the old one. The failed call names the cause, the
// box moves to the CA copy right then, and the next call works — no restart.
func TestPinnedBox_NewCertificate_HealsWithNoRestart(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
srv.newCert(p.leaf())
err := b.call()
if proxmox.TrustProblem(err) != proxmox.TrustPinnedCertChanged {
t.Fatalf("fixture: want a pin mismatch, got %v", err)
}
b.m.OnCallError(context.Background(), err)
if err := b.call(); err != nil {
t.Fatalf("the box is still blind after the move: %v (%v)", err, b.events)
}
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
t.Fatalf("the config's tls = %+v", got)
}
// a failure that is NOT one of trust does nothing
n := len(b.events)
b.m.OnCallError(context.Background(), errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused"))
if len(b.events) != n {
t.Fatalf("a refused connection started a trust run: %v", b.events[n:])
}
}
// Scenario: the root CA itself is renewed with the same key. The copy is refreshed, proven, and the running client
// reloads it; the config is not written again.
func TestCARenewedWithTheSameKey_TheCopyIsRefreshedAndReloaded(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
oldCA := string(p.caPEM)
p.renewCA("PVE CA")
srv.newCert(p.leaf())
b.source = p.caPEM
before, _ := os.ReadFile(b.cfgPath)
if !b.m.Ensure(context.Background(), "daily") {
t.Fatalf("a refreshed copy was not taken: %+v %v", b.m.Status(), b.events)
}
if cur, _ := os.ReadFile(b.copyPath); string(cur) == oldCA || string(cur) != string(p.caPEM) {
t.Fatal("the copy was not refreshed")
}
if !b.has("probe") || !b.has("apply") {
t.Fatalf("a refreshed copy must be proven and reloaded: %v", b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written again")
}
if err := b.call(); err != nil {
t.Fatalf("the call fails after the CA's renewal: %v", err)
}
}
// The root step refuses (the source is not a CA certificate): nothing changes, the cause is named.
func TestRootStepRefuses_NothingChanges(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") || b.has("probe") || b.has("apply") {
t.Fatalf("something ran without a copy: %v", b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written")
}
if st := b.m.Status(); st.Problem != ProblemCopyUnavailable || !strings.Contains(st.Detail, "T3") {
t.Fatalf("status = %+v", st)
}
}
// The proof works but the config cannot be written: the fingerprint stays, in the file and in the running client.
func TestConfigCannotBeWritten_TheFingerprintStays(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root writes a read-only file")
}
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
if err := os.Chmod(b.cfgPath, 0o400); err != nil {
t.Fatal(err)
}
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
t.Fatalf("the trust changed without a saved config: %v", b.events)
}
if got := b.fileTLS(); got.Fingerprint != fp(leaf) {
t.Fatalf("the fingerprint is gone from the file: %+v", got)
}
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemConfigNotSaved {
t.Fatalf("status = %+v", st)
}
}
// What is never moved: an operator's own CA file, a box with checks switched off, a config that is not a file.
func TestNeverMoved(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
own := filepath.Join(t.TempDir(), "own-ca.pem")
os.WriteFile(own, p.caPEM, 0o644)
for name, start := range map[string]proxmox.TLSConfig{"own CA file": {CAFile: own}, "insecure": {InsecureSkipVerify: true}} {
b := newBox(t, srv, start, p.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
t.Fatalf("%s: moved (%v)", name, b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatalf("%s: the config was written", name)
}
}
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
b.m.ConfigPath = ""
if b.m.Ensure(context.Background(), "start") || b.has("apply") || b.m.Status().Problem != ProblemConfigNotSaved {
t.Fatalf("a config that is not a file was 'moved': %v %+v", b.events, b.m.Status())
}
}
// The copy ends soon: the cause is named (the box still works).
func TestCopyEndsSoon(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
inner := b.m.EnsureCopy
b.m.EnsureCopy = func(ctx context.Context) (Copy, error) {
c, err := inner(ctx)
c.DaysLeft, c.NotAfter = 12, "2026-11-26T00:00:00Z"
return c, err
}
b.m.Ensure(context.Background(), "daily")
if st := b.m.Status(); st.Problem != ProblemCopyEndsSoon || !strings.Contains(st.Detail, "12 day") {
t.Fatalf("status = %+v", st)
}
}
// A trust failure asks at most once per MinGap.
func TestOnCallError_Throttled(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
other := newPKI(t, "another box")
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
srv.newCert(p.leaf())
err := b.call()
now := time.Date(2026, 11, 15, 3, 0, 0, 0, time.UTC)
b.m.Now = func() time.Time { return now }
b.m.OnCallError(context.Background(), err)
n := len(b.events)
b.m.OnCallError(context.Background(), err)
if len(b.events) != n {
t.Fatalf("a second failure inside the gap ran again: %v", b.events[n:])
}
now = now.Add(11 * time.Minute)
b.m.OnCallError(context.Background(), err)
if len(b.events) == n {
t.Fatal("no new run after the gap")
}
if st := b.m.Status(); st.Problem == "" {
t.Fatalf("a blind box names no cause: %+v", st)
}
}
// Proxmox does not answer at all (it restarts): nothing is known about the copy — no cause is named, nothing is written,
// and the box counts as not moved, so the timer asks again soon.
func TestProofCannotBeMadeNow_NoCauseNamed(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
b.m.Probe = func(context.Context, proxmox.TLSConfig) error {
return errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused")
}
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("moved without a proof")
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written")
}
if st := b.m.Status(); st.Problem != "" || st.Mode != "fingerprint" || b.m.Moved() {
t.Fatalf("status = %+v moved=%v", st, b.m.Moved())
}
}
+101
View File
@@ -0,0 +1,101 @@
package signedjobs
import (
"context"
"errors"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// outcomeQueue is a jobsQueue that also has the optional half (OutcomeSource), as *hub.Client does.
type outcomeQueue struct {
jobsQueue
omu sync.Mutex
outcomes map[string]hub.JobOutcome
}
func (q *outcomeQueue) CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error {
q.omu.Lock()
if q.outcomes == nil {
q.outcomes = map[string]hub.JobOutcome{}
}
q.outcomes[jobID] = o
q.omu.Unlock()
return q.jobsQueue.CompleteJob(ctx, jobID)
}
// R-1383 (D78): a signed job that is refused or fails was cleared from the hub's queue exactly like one that ran —
// „an agent update never arrives” left one journal line on the box. The runner now says how each job ended, through
// the REAL gate and verifier: the stored effect is the outcome the hub receives with the clearing call.
// The two clock reasons are the ones the ruling is about: a box with a wrong clock refuses every job.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-job-outcome.txt).
func TestRunner_TellsTheHubHowEachJobEnded(t *testing.T) {
s := newTestSigner(t)
_, _, exec := newRealGateRunner(t, s)
base, _, _ := newRealGateRunner(t, s)
src := &outcomeQueue{}
r := NewRunner(src, base.gate, exec, testHost, quiet())
now := time.Now().UTC()
src.add(mintJob(t, s, "ok", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
src.add(mintJob(t, s, "expired", testHost, "", "ops-1", wipeParamsJSON, now.Add(-3*time.Hour), now.Add(-2*time.Hour)))
src.add(mintJob(t, s, "early", testHost, "", "ops-1", wipeParamsJSON, now.Add(2*time.Hour), now.Add(3*time.Hour)))
src.add(mintJob(t, s, "other-box", "some-other-host", "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
src.add(corruptSig(t, mintJob(t, s, "forged", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))))
src.add(hub.JobWire{JobID: "garbage", BlobB64: "!!!not-base64!!!"})
if _, err := r.RunOnce(context.Background()); err != nil {
t.Fatal(err)
}
// and a job that is accepted and whose run fails
exec.err = errors.New("the wrapper refused")
src.add(mintJob(t, s, "run-fails", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
if _, err := r.RunOnce(context.Background()); err != nil {
t.Fatal(err)
}
want := map[string]hub.JobOutcome{
"ok": {Outcome: "done", Op: "storage_wipe"},
"expired": {Outcome: "rejected", Reason: "expired", Op: "storage_wipe"},
"early": {Outcome: "rejected", Reason: "not_yet_valid", Op: "storage_wipe"},
"other-box": {Outcome: "rejected", Reason: "target", Op: "storage_wipe"},
"forged": {Outcome: "rejected", Reason: "malformed", Op: "storage_wipe"}, // corruptSig breaks the armour itself
"garbage": {Outcome: "malformed", Reason: "malformed"},
"run-fails": {Outcome: "failed", Reason: "execution_failed", Op: "storage_wipe"},
}
for id, w := range want {
if got, ok := src.outcomes[id]; !ok || got != w {
t.Errorf("job %q: the hub was told %+v (told=%v), want %+v", id, got, ok, w)
}
}
if len(src.outcomes) != len(want) {
t.Errorf("outcomes for %d jobs, want %d: %+v", len(src.outcomes), len(want), src.outcomes)
}
if exec.count() != 2 {
t.Errorf("executor ran %d times, want 2 (ok and run-fails only)", exec.count())
}
}
// Every code reasonCode can return is on the closed list the hub holds a sentence for.
func TestReasonCode_ClosedList(t *testing.T) {
known := map[string]bool{}
for _, c := range ReasonCodes {
known[c] = true
}
for _, d := range []reconcile.Decision{
{}, {Reason: reconcile.ReasonRejected, Err: errors.New("something new")},
{Reason: reconcile.ReasonRoleDenied}, {Reason: reconcile.ReasonBindingMismatch}, {Reason: reconcile.ReasonPendingSignature},
} {
if c := reasonCode(d); !known[c] {
t.Errorf("reasonCode(%+v) = %q, not on ReasonCodes", d, c)
}
}
if reasonCode(reconcile.Decision{Reason: reconcile.ReasonRejected, Err: errors.New("x")}) != "other" {
t.Error("an unknown refusal must be told as „other”, never as the error's own text")
}
if !known["execution_failed"] || !known["malformed"] {
t.Error("the two codes the runner sets itself are not on the list")
}
}
+48 -6
View File
@@ -35,6 +35,42 @@ type JobSource interface {
CompleteJob(ctx context.Context, jobID string) error
}
// OutcomeSource is the optional half of a JobSource: it clears a job AND tells the hub how it ended (R-1383, D78).
// *hub.Client has it. A source without it is cleared as before.
type OutcomeSource interface {
CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error
}
// ReasonCodes is the CLOSED list of reasons the agent sends with a job that did not run to a good end. The hub holds
// one operator sentence per code (felhom.eu hub/internal/api, signedJobReasons); a code the hub does not know is
// told as „other”. Pinned by TestReasonCode_ClosedList.
var ReasonCodes = []string{"expired", "not_yet_valid", "window_too_long", "replay", "bad_signature", "unknown_signer",
"target", "namespace", "malformed", "role_denied", "binding_mismatch", "execution_failed", "other"}
// reasonCode names why the gate refused: the authz error first (the two clock reasons matter most — a box with a
// wrong clock refuses EVERY job), then the gate's own reason.
func reasonCode(dec reconcile.Decision) string {
for _, m := range []struct {
err error
code string
}{
{authz.ErrExpired, "expired"}, {authz.ErrNotYetValid, "not_yet_valid"}, {authz.ErrWindowTooLong, "window_too_long"},
{authz.ErrReplay, "replay"}, {authz.ErrBadSignature, "bad_signature"}, {authz.ErrUnknownSigner, "unknown_signer"},
{authz.ErrTarget, "target"}, {authz.ErrNamespace, "namespace"}, {authz.ErrMalformed, "malformed"},
} {
if errorsIs(dec.Err, m.err) {
return m.code
}
}
switch dec.Reason {
case reconcile.ReasonRoleDenied:
return "role_denied"
case reconcile.ReasonBindingMismatch:
return "binding_mismatch"
}
return "other"
}
// Authorizer is the reversibility gate. Satisfied by *reconcile.Gate.
type Authorizer interface {
Authorize(intent reconcile.Intent, signed *reconcile.SignedOp) reconcile.Decision
@@ -141,7 +177,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
// A malformed/forged envelope is permanently bad — clear it loudly (a queued garbage job
// must not wedge the queue). No nonce was consumed (we never reached Verify).
r.logger.Error("signedjobs: malformed job envelope — clearing", "job", j.JobID)
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "malformed", Reason: "malformed"})
return true
}
@@ -166,7 +202,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
// it as a security event (a compromised hub queuing a forged op lands here).
r.logger.Warn("signedjobs: REJECTED signed op — executor not called",
"job", j.JobID, "op", ob.Op, "reason", dec.Reason, "err", errStr(dec.Err))
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "rejected", Reason: reasonCode(dec), Op: ob.Op})
return true
}
@@ -177,7 +213,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
switch {
case err == nil:
r.logger.Warn("signedjobs: signed op COMPLETED", "job", j.JobID, "op", ob.Op)
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "done", Op: ob.Op})
return true
case errorsIs(err, ErrNoExecutor):
// No executor for this op class in this build — leave it queued for the owning slice.
@@ -189,7 +225,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
// nonce to retry) and log the failure loudly.
r.logger.Error("signedjobs: signed op execution FAILED (nonce spent — clearing)",
"job", j.JobID, "op", ob.Op, "err", err)
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "failed", Reason: "execution_failed", Op: ob.Op})
return true
}
}
@@ -215,8 +251,14 @@ func (r *Runner) decode(j hub.JobWire) (*reconcile.SignedOp, authz.OpBlob, bool)
return &reconcile.SignedOp{Blob: opBlob, Sig: []byte(env.SigArmored)}, ob, true
}
func (r *Runner) complete(ctx context.Context, jobID string) {
if err := r.source.CompleteJob(ctx, jobID); err != nil {
func (r *Runner) complete(ctx context.Context, jobID string, o hub.JobOutcome) {
var err error
if os, ok := r.source.(OutcomeSource); ok {
err = os.CompleteJobWithOutcome(ctx, jobID, o) // R-1383: the hub hears HOW the job ended
} else {
err = r.source.CompleteJob(ctx, jobID)
}
if err != nil {
r.logger.Warn("signedjobs: clearing job failed (will retry next pass)", "job", jobID, "err", err)
}
}
+3 -3
View File
@@ -24,10 +24,10 @@
"readers": [
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
],
"recorded": "2026-11-10",
"recorded_by": "CC 2026-11-10: day0_agent 0.173.0 (operator vouch 2026-11-10 16:29:16 CET, read from the hub log: agent=0.173.0, bundle 51ebc1f4… = the release; the save took 232 ms); earlier 0.172.1 (CC 2026-11-07) (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"recorded": "2026-11-14",
"recorded_by": "CC 2026-11-14: day0_agent 0.174.0 (operator vouch 2026-11-14 15:21:45 CET, read from the hub log and from the hub's System page: agent=0.174.0, bundle 7d87d058… = the release; the save took 428 ms); earlier 0.173.0 (CC 2026-11-10) (operator vouch 2026-11-10 16:29:16 CET, read from the hub log: agent=0.173.0, bundle 51ebc1f4… = the release; the save took 232 ms); earlier 0.172.1 (CC 2026-11-07) (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.173.0"
"day0_agent": "0.174.0"
}
}