Compare commits

...
61 Commits
Author SHA1 Message Date
admin db01ae0f2a agent (unreleased): a cut write of agent.json cannot leave a blind box — a journal of the old bytes, a repair at start (R-1380)
gates / gates (push) Successful in 42s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-15 08:20:49 +01:00
admin 54ac2c2ac9 agent (unreleased): the CA copy keeps its previous root only for the changeover, and is compared exactly (the commit's security review)
gates / gates (push) Successful in 47s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 19:20:07 +01:00
admin ec7e699fd7 agent (unreleased): fixes after an independent review — one lock for the writers of agent.json, the trust run serialised and bounded, the CA copy keeps its previous root, the profile stanza from the wrapper's answer, a stale give-up ends, both boot tokens from one read
gates / gates (push) Successful in 48s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 19:09:21 +01:00
admin 61feb95c06 agent (unreleased): the hub is told what the kernel clean-up did and what the profile refresh did (R-1383)
gates / gates (push) Successful in 58s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 18:06:12 +01:00
admin c0bf97de80 agent: gofmt (formatting only)
gates / gates (push) Successful in 40s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:47:42 +01:00
admin d5ce5c364a agent (unreleased): a boot token that a clock step cannot move (R-1395)
/disks answers guest_boot_key (the kernel's boot_id and the guest init's start) beside the unchanged guest_boot_id.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:43:33 +01:00
admin a1135018c7 agent (unreleased): a refused signed job and a watchdog give-up reach the hub (R-1383, D78)
gates / gates (push) Successful in 42s
The job runner clears a job with its outcome, a closed reason code and the op's name. The guest-power watchdog's
give-up rides the host report until the guest runs.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:29:53 +01:00
admin ede5d0e6c7 agent (unreleased): the report carries the box's clock at the moment of sending (R-1384, D79)
gates / gates (push) Successful in 46s
sent_at is stamped by Client.Report at every send, never by the collector. The hub reads it to tell a wrong box clock.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-14 17:09:12 +01:00
admin 386a0c4cf4 2b (unreleased): the agent moves from a pinned Proxmox certificate to the box's root CA copy — copy, proof, then the config; the cause of a trust failure is named (R-1380)
gates / gates (push) Successful in 38s
2026-11-14 16:35:22 +01:00
admin a9ab47ec71 2b (unreleased): the root step that keeps a copy of Proxmox's root CA (wrapper mode pve-ca-copy, R-1380)
gates / gates (push) Successful in 44s
2026-11-14 16:28:04 +01:00
admin 2706e35198 v0.174.0 RELEASED and VOUCHED: CHANGELOG, the day-0 pin, CONTEXT, REPORT
gates / gates (push) Successful in 40s
2026-11-14 15:23:16 +01:00
admin cc63dd5204 Old kernels removed by the box (R-1368, wrapper mode kernel-cleanup); security profiles refreshed after a kernel change (R-1388, mode profile-refresh); Go 1.27.4 (R-1391)
gates / gates (push) Successful in 40s
2026-11-14 12:11:06 +01:00
admin 91a2924ad0 docs: the tag push takes its token from the secret store; a repository address holds no credential (R-925)
gates / gates (push) Successful in 40s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-11 11:12:25 +01:00
admin 9edaa8c082 v0.173.0 vouched and delivered to the three test boxes; pinned.day0_agent = 0.173.0 (D50)
gates / gates (push) Successful in 43s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 16:30:01 +01:00
admin 5cebb766e3 v0.173.0 released (not vouched): CHANGELOG, REPORT, CONTEXT — the folder clean-up measured on three boxes, the rollback proven on Tester 1
gates / gates (push) Successful in 42s
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 16:16:42 +01:00
admin 776025c824 selfupdate: the staging folder keeps what it needs and no more (R-1368 part 2) — not released
gates / gates (push) Successful in 35s
Each good update left its staged binary in <state dir>/selfupdate for ever
(49 files / 740 MB on demo-hp). After the wrapper's commit returned success,
MaybeCommit now removes every staged binary but the running version's.

Nothing is removed while a pending marker exists or cannot be read, at a
plain start, or after a rollback. Only a regular file with the exact name
Execute writes is removed. The rollback reads pending.json and
felhom-agent.prev, no staged binary: pinned by a test on the wrapper's text
and by a test that runs the wrapper's real commit and rollback verbs in a
sandbox. Red seen three times.

No version bump, no build, no vouch. /usr/local/bin/felhom-agent.bak-* is
not reached (root-owned; R-1368 part 4).

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 14:10:00 +01:00
adminandClaude Opus 5.5 41d5eb8051 v0.172.1 vouched and delivered to the three test boxes (agent + bundle, 77/77); pinned.day0_agent = 0.172.1 (D50)
gates / gates (push) Successful in 44s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 23:29:24 +01:00
adminandClaude Opus 5.5 8bd137787e v0.172.1 released (not vouched): CHANGELOG, REPORT with the operator's PowerShell delivery, CONTEXT
gates / gates (push) Successful in 33s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 22:43:34 +01:00
adminandClaude Opus 5.5 39db03065a os-apply: proxmox-grub joins the boot-chain lane (pve step refused R6 nightly on demo-felhom)
gates / gates (push) Successful in 36s
proxmox-grub 2.12-9+pmx4 depends on grub-efi-amd64 of the same version. HOST_SLOW_RE
anchors `grub` at the start of a name, so proxmox-grub landed in the ring-0 pve plan,
the simulation pulled grub-efi-amd64, and the whole pve step was refused R6 every
night since 2026-11-05. Same class as the 2026-10-09 proxmox-secure-boot-support fix.

Test test_ring0_pending_pve_leaves_proxmox_grub_out: red before the fix, green after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-07 22:41:05 +01:00
admin ba409de4a2 v0.172.0 vouched: pinned.day0_agent = 0.172.0 (D50); CHANGELOG heading; REPORT + CONTEXT
gates / gates (push) Successful in 34s
2026-11-03 10:29:51 +01:00
admin a4f1280399 v0.172.0 released, not yet vouched: CHANGELOG heading (R-1302; agent 2c0222cd, bundle cc34b0bc)
gates / gates (push) Successful in 36s
2026-11-03 09:08:07 +01:00
admin ecbb0cc479 REPORT: 2026-11-02 night, R-1302, commit only
gates / gates (push) Successful in 36s
2026-11-03 04:13:52 +01:00
admin 50eeccc986 R-1302: the first grant of a tier logs INFO granted; a grant that existed and vanished keeps the R-190 ERROR. Commit only.
gates / gates (push) Successful in 36s
2026-11-02 23:16:10 +01:00
admin 808cdb36ad REPORT + CONTEXT: v0.171.0 released and vouched; R-1344 restart proven live
gates / gates (push) Successful in 33s
2026-10-30 11:09:48 +01:00
admin 84726c9f27 v0.171.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.171.0 (D50)
gates / gates (push) Successful in 33s
2026-10-30 10:49:46 +01:00
admin bd18347c1c R-1344 decided from the night's measurement: a controller that runs but does not answer is restarted at 10 unhealthy sweeps (alarm at 6) — busy file + the brake refuse; stanza carries unresponsive_restart_after_s / unresponsive_restart_at
gates / gates (push) Successful in 33s
2026-10-30 09:20:37 +01:00
admin 98bd1b71e9 REPORT + CONTEXT: v0.170.0 released and vouched; reboot_guest proven on the release builds
gates / gates (push) Successful in 39s
2026-10-29 20:24:29 +01:00
admin 6578a8b91e v0.170.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.170.0 (D50)
gates / gates (push) Successful in 36s
2026-10-29 18:59:05 +01:00
admin ca15a31690 D65 = A: reboot_guest joins the agent's closed operator-action list — guards (busy controller file, heavy gate, jobs, lock, parked, 1/hour, the brake), result with both times, GET /guest/operator-reboot for the household line; one read-only sudo vector
gates / gates (push) Successful in 33s
2026-10-29 16:04:19 +01:00
admin bdea4694a4 REPORT + CONTEXT: v0.169.0 released and vouched; restart_controller proven on the release builds
gates / gates (push) Successful in 35s
2026-10-29 15:11:11 +01:00
admin db25eb0e71 v0.169.0 released and vouched (14:06:09 CET): CHANGELOG; D50 pinned.day0_agent = 0.169.0
gates / gates (push) Successful in 42s
2026-10-29 14:09:04 +01:00
admin 637c92c762 Operator actions for the agent: restart_controller from the hub (closed list, the supervisor's guards, the brake counts operator restarts); R-1344: a running-but-unhealthy controller raises unresponsive — alarm only; tests red first
gates / gates (push) Successful in 36s
2026-10-29 12:30:36 +01:00
admin 63e63ee8bb v0.168.0 released and vouched: CONTEXT, REPORT
gates / gates (push) Successful in 37s
2026-10-29 10:48:37 +01:00
admin 6b0b093199 D50: pinned.day0_agent = 0.168.0 (vouched by the operator 2026-10-29 10:21:56 CET); CHANGELOG notes the vouch
gates / gates (push) Successful in 36s
2026-10-29 10:33:58 +01:00
admin f4e63a55da v0.168.0: CHANGELOG — released (agent 1e70196c…, bundle 3415c034…), committed on demo-felhom, Tester 1, demo-hp; not vouched yet
gates / gates (push) Successful in 37s
2026-10-29 10:15:55 +01:00
admin 91bd96c755 REPORT + CONTEXT: R-528 and R-1254 on main, not released (night 2026-10-28)
gates / gates (push) Successful in 35s
2026-10-29 02:51:00 +01:00
admin 6f933d6ee3 R-528: GET /guest/oom-kills — the kernel's per-container OOM counter of the token's guest, read on the host (no sudo)
gates / gates (push) Successful in 34s
Tests TestR528_* (fake cgroup tree), red-proved by mutation. Commit only; the agent release needs the vouch.
2026-10-28 23:22:40 +01:00
admin 5ddd124cfc R-1254 (ceremony half, HUNT-3 H2): after a failed escrow upload, ask the hub which package it holds before wording the result
gates / gates (push) Successful in 44s
The hub holds the new package -> success; another package -> 502 'previous code still works' (now true);
cannot be asked -> 504, no promise. Tests TestR1254_* red before the fix. Commit only; the agent release needs the vouch.
2026-10-28 22:51:08 +01:00
admin 3055bc54c1 D50 pin: day0_agent 0.167.0 (the operator's vouch 2026-10-27, read from the hub)
gates / gates (push) Successful in 1m4s
2026-10-27 20:45:25 +01:00
admin 007199342f v0.167.0 RELEASED (not vouched yet): CHANGELOG header, REPORT
gates / gates (push) Successful in 1m20s
2026-10-27 15:49:30 +01:00
admin 54d3a1057b CHANGELOG: the unreleased header names R-972 (Go 1.27.3)
gates / gates (push) Successful in 59s
2026-10-26 22:21:47 +01:00
admin fbbb168d1f build(R-972): Go 1.27.3 (GO-2026-6809, crypto/x509) — govulncheck 1 → 0 called
gates / gates (push) Successful in 1m22s
2026-10-26 20:34:25 +01:00
admin 3911cc6f52 fix(R-1232): the tier due-check counts only this box's archives (own key) — a reinstalled box's first whole-guest copy is not a week late
gates / gates (push) Successful in 1m20s
Red-proved (the old box's archive was counted). Commit only.
2026-10-26 20:15:39 +01:00
admin ad80ee749a docs: CONTEXT — R-1285 state and Tuesday's line test
gates / gates (push) Successful in 1m15s
2026-10-26 18:11:24 +01:00
admin 0683cc2dab REPORT: the 2026-10-26 afternoon run. Docs only.
gates / gates (push) Successful in 1m48s
2026-10-26 14:38:15 +01:00
admin b1bc63dd32 R-1285: the new agent runs the bundle policy pass once after its self-update commits — in the normal two-job delivery the old process installs the bundle. Test in the real order (commit, then one bundle-policy plan); red-proved. Commit only, not released.
gates / gates (push) Successful in 2m15s
2026-10-26 12:44:25 +01:00
admin e66a48f2b7 REPORT: the 2026-10-26 attended release run. Docs only.
gates / gates (push) Successful in 1m41s
2026-10-26 10:38:03 +01:00
admin b890b57540 v0.166.0 released and vouched by the operator (10:04:55 CET): CHANGELOG header (R-1280 live, R-1285 back to OPEN — the bundle job runs in the old process); D50 day0_agent pin → 0.166.0. Docs only.
gates / gates (push) Successful in 1m52s
2026-10-26 10:07:15 +01:00
admin aa69114d18 REPORT: the 2026-10-25 night — committed fixes, CI by hash, teardown. Docs only.
gates / gates (push) Successful in 1m43s
2026-10-25 23:54:07 +01:00
admin abcc1e6833 R-1285: after every bundle install the agent asks the wrapper now on disk for the appliance-only policy pass, so a BYO host upgraded from an older wrapper loses our SSH drop-in in one update. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:46:38 +01:00
admin d912307bb7 R-1280: a signed job's validity window is at most 24 h — the verifier, the root wrapper and felhom-opsign. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:42:18 +01:00
admin feaaa0e8ff D50 = A (R-1290): the published gate asserts the newest 5 released versions plus the pinned MinAgent (0.158.1) and day-0 agent (0.165.0); a pin with no tag fails; decoys red-proved. No binary change.
gates / gates (push) Successful in 1m41s
2026-10-25 20:20:30 +01:00
admin 6b6ba0c17f v0.165.0 vouched by the operator (18:38:49 CET); REPORT: release, delivery to four hosts, the BYO proof (removal one bundle late, felhom.eu R-1285). Docs only.
gates / gates (push) Failing after 1m49s
2026-10-25 19:02:48 +01:00
admin 4697f8ac9e v0.165.0: CHANGELOG names the release (R-1257 D43 = A, R-1250); released from a clean clone, sha256 e9d04a47…, bundle 8dea1f34…; not vouched
gates / gates (push) Successful in 1m38s
2026-10-25 17:00:02 +01:00
admin 4a1a84941c D43 = A run close-out: REPORT and CONTEXT. Docs only.
gates / gates (push) Successful in 1m40s
2026-10-24 21:04:19 +02:00
admin c8d3c17b13 D43 = A (R-1257): the root-key-only sshd drop-in only on an appliance; on BYO nothing is written and our copy is removed; the in-force check reads the LAN addresses (X3); a failing sshd -t undoes only the SSH step (X4). Commit only, not released.
gates / gates (push) Successful in 1m32s
2026-10-24 20:47:17 +02:00
admin f5c3d77a61 Review round 2: .claude/rules/unprompted-work.md aligned with its four siblings (the missing 'A version number is pushed once' fact paragraph); no code change
gates / gates (push) Successful in 1m32s
2026-10-24 15:50:59 +02:00
admin 5e05627c4e REPORT: R-1250 test on request, live proof on demo-hp
gates / gates (push) Successful in 1m41s
2026-10-24 13:48:07 +02:00
admin 1e087bb158 linetest: waiting_for_quiet needs the full watched window (R-1250)
gates / gates (push) Successful in 1m35s
The live proof found a fresh agent answering waiting before it had watched one minute; a quiet
box's torrent app would be held for nothing.
2026-10-24 13:24:41 +02:00
admin 29bcf4403e linetest: a test on request for a box that seeds all day (R-1250)
gates / gates (push) Successful in 1m37s
GET /host/line-test reports waiting_for_quiet (due, in the window, tunnel up, only the quiet
missing); POST /host/line-test/run runs one measurement with a 5 s settle and a 10 s quiet
check instead of 10 quiet minutes, runs 5 s apart (~55 s). Every other weekly rule holds.
Commit only; the release rides the next train.
2026-10-24 13:18:59 +02:00
admin 08ca67861f CHANGELOG: v0.164.0 released (the weekly upload line test, R-1247), delivered to the three test boxes, vouched by the operator; REPORT
gates / gates (push) Successful in 1m32s
2026-10-24 09:46:00 +02:00
85 changed files with 8952 additions and 196 deletions

No files matched your search

+6
View File
@@ -50,6 +50,12 @@ it **first** in the morning note. A decision you cannot write in that shape is o
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
on all three — not at night unless the night brief says so. **Agent releases still stop for the
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
**A version number is pushed once** — `felhom-controller/controller/build.sh` now refuses a second
push of a tag that exists, and a registry it cannot reach is a refusal too. This is a FACT about
what the build does, not a new rule: a fix for something already pushed is the next patch version,
and a box on a bad image is moved by the normal update path, never by re-tagging. Earned on
2026-10-15, when 0.315.0 was pushed three times from three different commits and a cache kept
serving the first one (`09-update-architecture.md` §0).
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
Prose is not a record.
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
+481 -1
View File
@@ -1,4 +1,484 @@
## Unreleased (next: v0.164.0; 2026-10-24 attended release run) — the weekly upload line test (R-1247)
## Unreleased (2026-11-15) — a cut write of `agent.json` cannot leave a blind box (R-1380; found before the release)
- **`config.ReplaceInPlace` + `config.RecoverCutWrite`** (`internal/config/cutwrite.go`). `agent.json` is the agent's
file in a root-owned folder, so the trust move, the PBS seed and the backup-target move write it in place
(truncate, write). A power cut in between left an empty or half file and an agent that does not start. Now the old
bytes go into a journal in the state folder first (`agent.json.write-in-flight`, durable), the config is written,
synced and read back, and the journal is removed; the daemon's start repairs a config that does not parse from the
journal BEFORE it reads it. A selftest never repairs. No journal folder → no write. Tests: a cut staged at each of
four steps, the wiring in `main`, and a source pin that the three writers use the helper and open no file by hand.
Red-proofs: felhom.eu `documentation/audits/release-2b-3-2026-11-15/part0/`. Not proven: a real power cut.
## Unreleased (2026-11-14 night) — release 3 („the hub tells you”) and the clock-step fix, the agent's parts, built and NOT released
Commit only. Each part can be released alone; none needs release 2b. No `configs/` change in this section.
- **Fixes after an independent review of the night's code (2026-11-14; felhom.eu
`documentation/audits/night-build-2026-11-14/review/`).** (1) `config.FileMu`: the three functions that read
`agent.json` whole and write it whole (the trust move, the PBS seed, the backup-target move) take one lock — two of
them could interleave and the second put back the document it read before the first one's change; pinned from the
source by `TestAgentJSONWritersTakeTheSharedLock`. (2) `pvetrust.Manager.Ensure` runs one at a time (start, timer,
a failed collect and a bundle install could overlap, and the second parked the already-moved file as the „before”
copy), and its root step has a 90-second deadline (it runs on the start path and inside the report loop). (3) **The
CA copy keeps its previous certificate behind the new one** (`configs/felhom-os-apply`, mode `pve-ca-copy`): the
copy was replaced before anything could be proven with the new root, and Proxmox serves its old certificate until
pveproxy is reloaded — an agent that restarted in that window had no root that verified it. **Only for the
changeover:** at every later run the step makes one handshake to the local Proxmox with the NEW root alone, and
drops the previous root as soon as that verifies, or when the previous root is no longer a CA that has not ended.
The copy is compared exactly, so nothing the step did not check stays in it. To drop the old root at once by hand
(a root changed because it leaked): delete the copy. `configs/` changes again, so the release needs both signed jobs.
(4) The profile-refresh stanza is built from the wrapper's ANSWER (its record of the last refresh comes back at
every ask); the record file is a bonus — its folder is root-only on a box where a signed job made it first (read in
source, not on a box). `rewritten` is absent when not known. (5) A watchdog give-up is forgotten when the guest is
no longer the watchdog's (onboot off, locked, destroyed) — it was reported for ever. (6) `/disks` sends both boot
tokens from ONE read of the guest's init (`GuestBootTokens`). 13 mutations seen red, two after a mutant that did
not compile was done again.
- **The hub is told what the kernel clean-up did and what the profile refresh did (R-1383; felhom.eu `11` §5.11.1,
§5.12).** (1) The clean-up's report carries `kernel_cleanup` (`osupdate.CleanupDetail`: removed, kept, bytes_gained,
leftovers). It still has no `kernel` view and `release_id` stays `kernel-cleanup` — the hub reads only those two as
„the kernel this step was for” (the test asserts the wire form). (2) The host report carries `profile_refresh`
(`osupdate.ProfileRefreshStatus`, from `Leg.ProfileRefreshStanza`, wired in `main.go`): the root wrapper's own record
of the last refresh — so it outlives an agent restart — and how this process's last ask ended. One small file read,
no wrapper call, names checked and capped. No OS report is sent for the refresh (an unknown layer reads as a guest
step on the hub). Tests `TestKernelCleanup_NightOrderOnceAndNoNames` (extended), `TestProfileRefreshStanza_RecordAndLastAsk`,
`TestMainWiresPVETrust`; 9 mutations seen red (one mutant did not compile at first and was done again).
- **A boot token that a clock step cannot move (R-1395).** `/disks` answers `guest_boot_key` =
`<kernel boot_id>-<guest init start>` beside the unchanged `guest_boot_id` (`<boot TIME>-<guest init start>`). The
kernel computes the boot time from the wall clock, so a step of the host's clock moved the old token and the
controller recreated every drive-backed app (the N100, 2026-11-14, +32.7 s). `GuestBinder.GuestBootKey`; `""` when
the guest does not run or the boot_id is unreadable. Only a controller that knows the key (built there, not
released) reads it. Tests `TestGuestBootKey_AClockStepDoesNotMoveIt`, `TestDisks_CarriesBothBootTokens`; 6 mutations
seen red (felhom.eu `documentation/audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt`; two mutants did not
compile at first and were done again).
- **A refused signed job and a watchdog give-up reach the hub (R-1383, D78, felhom.eu `09` §3 decision 299; `05`
§16.6).** (1) `signedjobs.Runner` clears a job with `hub.Client.CompleteJobWithOutcome`: `outcome` (done · rejected
· failed · malformed), a `reason` from the closed `signedjobs.ReasonCodes` (the two clock reasons `expired` and
`not_yet_valid` first — a box with a wrong clock refuses every job) and the op's name, in the `DELETE`'s query. No
signature, nonce or job parameter is sent. (2) The guest-power watchdog's give-up — one ERROR line in the journal
until now — rides the host report as `guest_power_gave_up` (`localapi.Server.GuestPowerGaveUpStanza`, wired in
`main.go`); the list empties when the guest runs. Tests `TestRunner_TellsTheHubHowEachJobEnded` (the real gate
and verifier), `TestReasonCode_ClosedList`, `TestCompleteJob_OutcomeRidesTheQuery`,
`TestGuestPower_GiveUpIsReportedUntilTheGuestRuns`, `TestMainWiresPVETrust`; 12 mutations seen red (felhom.eu
`documentation/audits/night-build-2026-11-14/B/redproof-agent-{job-outcome,guest-power}.txt`).
- **The report carries the box's clock at the moment of sending (R-1384, D79, felhom.eu `09` §3 decision 300).**
`HostReport.SentAt` (`sent_at`, RFC3339 with nanoseconds) is stamped by `Client.Report` at every send, never by the
collector. `reported_at` could not serve: it is stamped when the collect starts. The hub (its own unreleased part)
compares it with its clock and alarms above one minute. Test `TestReport_StampsSentAtAtSending`; 3 mutations seen
red (felhom.eu `documentation/audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt`).
## Unreleased (2026-11-14 night) — release 2b, built and NOT released: trust in Proxmox by a copy of the box's own root CA (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a)
Commit only. No box ran any of this; the proof on a box (demo-hp, a forced new certificate, twice) is the next attended
session's. `configs/` changes, so the release needs BOTH signed jobs.
- **The root step that keeps the copy** (`configs/felhom-os-apply`, host-layer mode `pve-ca-copy`). It copies exactly
`/etc/pve/pve-root-ca.pem` to exactly `/etc/felhom-agent/pve-root-ca.pem` (0644 root), only if the source is ONE
certificate with `CA:TRUE` that has not ended; written whole or not at all; „same" when nothing changed. The plan may
name nothing (T1); no source (T2); not one CA certificate (T3). No appliance check: the copy is the agent's own trust
file and changes nothing of a BYO owner's host. Tests `PVECACopy` (9, one with the real openssl through the real
runner); 7 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-wrapper.txt`).
- **The move of a box that exists, in a safe order** (`internal/pvetrust`, new; wired in `cmd/felhom-agent/main.go`
at the agent's start, after a config bundle install, on a timer — hourly until moved, daily after — and on every
call to Proxmox that fails on trust). (1) the root step makes the copy; (2) ONE real call to Proxmox with the copy
ALONE, while the pinned fingerprint is still the configured trust; (3) only then ONE write of `agent.json` sets
`proxmox.tls.ca_file` and clears `fingerprint` (the agent's own file, rewritten in place after a recovery copy
`agent.json.pre-pve-trust` was parked, read back and checked); (4) the RUNNING client takes the new trust
(`proxmox.Client.SetTLS`, new) — no restart. A failed copy, proof or write changes nothing: the fingerprint stays.
**`proxmox.tls` keeps its order (a set fingerprint wins over a CA file), so the fingerprint is cleared LAST.** An
operator's own CA file, a box with checks switched off and a config that is not a file are never moved.
- **A pinned box whose certificate changed heals by itself:** the failed call starts the same order, the box moves to
the CA copy and the next call works. A refreshed copy (a renewed root CA) is proven and reloaded by the running
client.
- **The cause is named** (`proxmox.TrustProblem`, new): the pinned certificate changed; the CA copy did not sign the
served certificate; a certificate ended; the copy is unreadable; the copy ends within 30 days; the root step gave no
copy; the proof failed. An ERROR line in the journal, and the new `proxmox_trust` stanza of the host report
(`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`). **The hub does not read the stanza yet**
(felhom.eu R-1396), and a box that is fully blind sends no host report at all — that case still shows only as
„host stale" on the hub.
- Tests: `internal/pvetrust` (11, against a real TLS server that changes its certificate and renews its CA), the AST
wiring test `TestMainWiresPVETrust`; 12 + 3 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-agent.txt`;
one mutation is not a rule and says so).
## v0.174.0 (2026-11-14) — the box removes its old kernels (R-1368); security profiles are refreshed after a kernel change (R-1388, R-1301); Go 1.27.4 (R-1391) — RELEASED 2026-11-14 (release-agent.sh from the clean working clone at `cc63dd5`, CI run id 2641; agent sha256 6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92, bundle sha256 7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f) and VOUCHED 2026-11-14 15:21:45 CET
Release 2a of the build order (felhom.eu `09` §3 decision 306). Design: felhom.eu `11-os-updates.md` §5.11.1 and §5.12.
`configs/` changed (the wrapper), so a box needs BOTH signed jobs: `agent_update`, then `agent_config_update`.
**Part A — old kernels (`09` §3 decision 286).** Nothing removed old kernels; a ring-0 box got about one a week, 1 GB
each. Now:
- **The wrapper** (`configs/felhom-os-apply`, layer `kernel`, new mode `kernel-cleanup`): the hand-proven rule of
2026-11-10 as a program. Keep the running kernel, the step's `from`, `to` and `default_before`, both defaults, the
flag's kernel, an operator's boot pin and the newest installed; remove every other installed kernel image in ONE
`apt-get purge` by exact name. Refusals P1, P2, P3, K1, N1, S1, S2 (the design's table), C1 (the plan names
something — it may carry only `dry_run`), R9, R12. „Not run" (exit 0): no root-owned ring-0 mark, no kernel step yet,
the newest step did not end `good`, the box does not run its newest kernel, a step within the last 20 hours. K1 reads
the box a second time, by itself. Repair-first. After the purge it proves the default, the pinned file, the two
generators and the state file unchanged; a left-over folder is reported, never deleted. **Rule R4 and every update
plan are unchanged; `apt-get autoremove` is not used.**
- **The agent** (`internal/osupdate/kernel.go`): the clean-up is the night leg's last call, after a healthy host step,
never in a night with a kernel step, never in a debug pass, never on a BYO host. The hub hears `cleaned`, `refused`
or `failed` (a kernel-layer report with NO kernel version — the hub would read that as a kernel step); „nothing to
remove" and „not run" are log lines. `--selftest=kernel-cleanup-dry-run` prints what it would do and changes nothing.
- **No new sudo line.** The wrapper is reached by the one `--plan` line (pinned by `test_cleanup_needs_no_new_sudo_line`).
**Part B — security profiles (spike first: felhom.eu `audits/profile-refresh-spike-2026-11-14/`).** After a kernel
went back and forth a box can load an old compiled AppArmor profile; on the N100 this stopped the time service.
- **The wrapper**, new host-layer mode `profile-refresh` (an appliance only; no package change; the plan names
nothing, F1): once per kernel the box starts on (a root-owned record), `apparmor_parser --replace --skip-read-cache
--write-cache -- /etc/apparmor.d`; then ONE restart for a `failed` service whose own program's cached profile the
rewrite changed (at most 3). Measured on demo-hp: 0.5 s, no guest or container touched.
- **The agent** (`internal/osupdate/profilerefresh.go`) asks at every start and after a config bundle install, under
the heavy-op gate; nothing is sent to the hub.
**Part C — Go 1.27.4** (GO-2026-6859; hygiene: the agent's one `MaxBytesReader` place does not match the advisory).
**Tests, each seen red** (felhom.eu `audits/release-2a-2026-11-14/A/redproof*.txt`, `B/redproof*.txt`): 31 wrapper
cases for the clean-up, one per acceptance scenario and one per kept-kernel kind (14 mutations, all red); 11 for the
profile refresh (10 mutations); 5 + 4 agent tests (3 + 3 mutations). Two existing agent tests list the night's calls
exactly and gained the clean-up call (`TestRing0_OneCallPerLayer`, `TestRing1_NoReleaseIsInventory`).
**Delivered 2026-11-14 to the three test boxes by signed `agent_update` + `agent_config_update` each** (Tester 1,
demo-hp, the N100): update committed, bundle installed (written 1 of 31), capability probe 77/77, the running binary
reads `go1.27.4`. **On boxes** (felhom.eu `audits/release-2a-2026-11-14/D/`):
- **Profile refresh, by the product's own path, on all three:** refused by the OLD wrapper at the agent's start
(R11, logged), then run right after the bundle install: 0.5 s each, nothing restarted on Tester 1 and demo-hp.
**On the N100 it repaired the time service:** the cached chrony profile was rewritten, `chrony.service` (failed
since 2026-10-16) was restarted and is active, the clock stepped +32.7 s; offset to DooPlex −32.55 s → +0.18 s. After
a restart of Tester 1 the agent answered „not needed" (the same kernel).
- **Clean-up:** dry runs through the released agent on all three (demo-hp: nothing to remove; the N100: 6 of 8 will
go tonight; Tester 1: „not run", it has no ring-0 mark). Refusals P1, P2 and P3 by dry run on demo-hp against states
set and put back. **A real removal on Tester 1**, with the ring-0 mark set for the test on the operator's yes and
put back: `proxmox-kernel-7.0.14-20-pve-signed` removed, 1.12 GB gained, 8.5 s, the default, the pinned file, the
generators and the state file unchanged, no left-over; the second call removed nothing; one orderly restart came
back on the same kernel, healthy, a clean stop for the crash guard. **This real run was started by hand** (the
agent user, the one sudo line, the night leg's own plan), not by the night leg.
- **Not proven:** the step by the agent at night (the N100's and demo-hp's own night, 2026-11-15); a box whose running
kernel is not its newest and a self-revert into the kept kernel (tests only); refusal K1 on a box (it guards a wrong
computation; tests only); a restart by the profile refresh of any program but chrony.
- **Found:** a clock step makes the CONTROLLER read a new boot (its boot identity is `/proc/stat` btime): on the N100
it recreated its three drive-backed apps 9 s after the step (felhom.eu R-1395). The crash guard (kernel `boot_id`)
was not fooled.
**Vouched by the operator 2026-11-14 15:21:45 CET** (the hub log: `Artifact manifest set: agent=0.174.0`, the bundle
sha = the release; the hub's System page: „vouched agent: 0.174.0"). `pinned.day0_agent` = 0.174.0 (D50).
## Unreleased (2026-11-11) — docs only: a repository address holds no credential (R-925, Part C of the clock/end-dates run)
- `scripts/release-agent.sh` header: the tag push needs `GITEA_GIT_TOKEN` from the secret store (`sec run`). The origin
address of CC's clone on DooPlex held a login and a token until 2026-11-11; git now asks the helper
`felhom.eu/scripts/git-credential-sec.sh`. No code change, no version change, no release.
## v0.173.0 (2026-11-10) — R-1368 part 2: the self-update folder keeps what it needs and no more — RELEASED and VOUCHED 2026-11-10 (release-agent.sh from a clean clone of `776025c`; agent sha256 752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea, bundle sha256 51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1)
- **The defect (measured 2026-11-10, space list place H12).** `Execute` downloads each new agent to
`<state dir>/selfupdate/felhom-agent-<version>` and removed that file only when the download or its checksum failed.
A good update left it for ever: 49 / 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1.
- **What the folder is needed for (read first: `configs/felhom-selfupdate-guarded`).** The root wrapper reads the staged
file once, copies it to its own root-owned folder and flips from that copy. The rollback needs `pending.json` (in the
folder) and `/usr/local/bin/felhom-agent.prev` (outside it); the commit needs `pending.json`. No staged binary.
- **The fix (`internal/selfupdate/prune.go`).** After the wrapper's `commit` returned success, `MaybeCommit` removes
every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read,
at a plain start, after a rollback (the failed version's binary stays to be read), or when the running version is
not a version this code can name a file for. Only a regular file with exactly the name `Execute` writes is removed,
one `os.Remove` per name; a failed removal is logged and never fails the commit. A download and a prune take turns
(`stagingMu`).
- **Tests `TestPrune_*` (6).** After six updates the folder holds one binary and every bystander (unknown names, a
folder and a symlink that carry a staged name); a pending or unreadable marker refuses; only a healthy commit prunes;
the name rule follows `versionRe`; the wrapper's `commit` and `rollback` blocks name no staged binary; and the
wrapper's REAL `commit` and `rollback` verbs, run from a sandboxed copy, still put the previous binary back after
four pruned updates. **Red seen three times:** no prune call; no pending refusal; no regular-file check.
- **Not reached.** `/usr/local/bin/felhom-agent.bak-*` (place H13): root-owned, in a folder the agent cannot write, made
by a person. R-1368 part 4 stays open.
- **Corrected 2026-11-10 (measured on three boxes).** This entry said: „A box gets the clean-up at its first good update
AFTER the one that brings this code". That was wrong. The commit runs in the NEW binary, so the box cleans the folder
at the commit of the update that brings the code.
- **Delivered 2026-11-10 to the three test boxes by a signed `agent_update` each (no bundle job: `configs/` did not
change since v0.172.1).** The folder before → after, read on each box: demo-hp 49 entries / 722,596 KB → 1 / 16,028 KB
(commit 14:59:19 UTC, `removed=49`); demo-felhom 48 / 706,968 KB → 1 / 16,028 KB (commit 14:51:50 UTC, `removed=48`);
Tester 1 33 / 497,244 KB → 1 / 16,028 KB (commit 15:00:19 UTC, `removed=33`). After each: `pending.json` absent,
`/usr/local/bin/felhom-agent.prev` = 0.172.1, the unit active with no restart.
- **A rollback still works after the clean-up (Tester 1, 2026-11-10 15:14 UTC).** A signed `agent_update` to 0.172.1
(the vouched version; it has no prune code) was installed with its pending marker. One `SEGV` in the 60-second dwell
→ the unit's `OnFailure` → the wrapper's `rollback` → 0.173.0 ran again 9 seconds after the install, `pending.json`
gone, no failed unit. The failed version's staged file stays (by design, until the next good update): the folder
holds 2 files, 32,040 KB. Evidence: felhom.eu `documentation/audits/release-space-fixes-2026-11-10/agent/`.
- **Vouched by the operator 2026-11-10 16:29:16 CET** (read from the hub log: `Artifact manifest set: agent=0.173.0`, the
bundle sha = the release). `pinned.day0_agent` = 0.173.0 (D50).
## v0.172.1 (2026-11-07) — proxmox-grub joins the boot-chain lane: the nightly pve step stops being refused R6 — RELEASED and VOUCHED 2026-11-07 (release-agent.sh from `39db030`; agent sha256 505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8, bundle 3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3)
- **The defect (measured on demo-felhom, 2026-11-05, 11-06, 11-07).** Proxmox published `proxmox-grub` 2.12-9+pmx4. It
depends on `grub-efi-amd64` of the same version. `HOST_SLOW_RE` matches `grub` only at the START of a name, so
`proxmox-grub` landed in the ring-0 pve plan; the simulation then pulled `grub-efi-amd64`, which is not in the plan,
and `felhom-os-apply` refused the WHOLE pve step (R6 „the plan would touch grub-efi-amd64") every night — 34 Proxmox
userspace upgrades held with it, and an `os_update_failed` error mail each morning. The same class as the 2026-10-09
`proxmox-secure-boot-support` fix. Read-only proof on the box: `apt-get -s install --only-upgrade proxmox-grub` →
`Inst grub-efi-amd64 …pmx4`.
- **The fix.** `proxmox-grub` joins `HOST_SLOW_RE`: GRUB's meta-package now waits with the rest of the boot chain (the
slow lane, the operator's signed step), and the pve step runs without it.
- **Test** `test_ring0_pending_pve_leaves_proxmox_grub_out`; red seen before the fix (proxmox-grub entered the plan), green
after; the wrapper suite 149 of 149.
- **Delivered 2026-11-07 (operator, signed jobs):** demo-felhom, demo-hp, Tester 1 — `agent_update` committed
(23:05/23:13/23:13 CET), then the bundle: `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok`, capability probe
77/77 on each. `pinned.day0_agent` = 0.172.1 (D50).
- **Delivery.** `configs/` changed, so a box needs BOTH signed jobs: `agent_update` 0.172.1, then `agent_config_update`
0.172.1 with the bundle sha. Not vouched by CC. Bundle vs 0.170.0 (what the boxes run): no path added or removed, only
`felhom-os-apply` changed — so no step bundle is needed (R-880). Both downloads re-checked by sha.
## v0.172.0 (2026-11-03) — R-1302: the first grant of a tier is no loss — RELEASED and VOUCHED 10:18:10 CET (release-agent.sh from a clean clone of `ecbb0cc`; agent sha256 2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7, config bundle cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce; configs/ unchanged since 0.171.0, so the signed delivery was agent_update alone)
- **R-1302 — the first grant of a tier is no loss.** A fresh install arms the whole-guest tier before its storage exists, so the
first probe found no grant, repaired it and logged ERROR „GRANT WAS MISSING … investigate the loss (R-190)" plus a DEGRADED
capability, on every new box (VM 9375, 2026-10-26). The agent now remembers which tiers have EVER held their grant
(`<state dir>/store-grant-seen.json`, written by a healthy probe). A first grant logs INFO „granted" and reports healthy; a
grant that existed and vanished keeps R-190's ERROR and its one degraded cycle. A doubt keeps the ERROR: an unreadable or
corrupt file, or no file path. A box updated to this agent records its tiers at the first healthy probe, so a loss in the
one probe before that would read as a first grant. Tests `TestR1302_*`; red seen with the first-grant branch removed and on
three more mutants.
## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
chain (demo-hp's PBS vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test, two
controller updates). No failed probe, no 5-s timeout; slowest 303 ms (demo-hp, during the PBS vzdump). The only
„unhealthy" was the controller stopped by hand for a proof. A busy night never came near ONE failed probe.
- **So the alarm becomes a restart.** At `controllerUnresponsiveConfirm` = 6 unhealthy sweeps the alarm is raised, as
before. At `controllerUnresponsiveRestart` = 10 (2 minutes later; ~6.5 minutes after the first failed probe) the agent
restarts the controller's bootstrap unit — unless the controller's busy file is fresh (a backup, a restore, an
update, an export) or cannot be read, or the brake refuses (3 restarts in 15 minutes → the crash-loop pause, which
the hub mails). A swap, a parked guest, a locked guest and a vzdump keep resetting the count, as before. A refusal
keeps the alarm and is logged once per reason. After a restart the count starts again from zero.
- **The stanza** carries `unresponsive_restart_after_s` (120: the hub's alarm text says when the restart comes) and
`unresponsive_restart_at` (equals `last_restart_at` for that act; the hub mails `controller_unresponsive_restart`).
- **Refactor, no behaviour change:** the not-running restart's brake and its record are now `supervisorBrakeAllows`
and `recordSupervisorRestart`, shared by both restarts. The busy-file stale log line no longer says „operator reboot".
- Tests `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`, `…Guards` (now also: never restarted),
`…RestartRefusals` (busy fresh, busy unreadable, the brake, a crash-loop pause), `…StaleBusyFileDoesNotShield` —
red first (build failure, then 3 behaviour failures); 7 mutants, each killed (one survivor — the count reset after a
restart — found and pinned).
## v0.170.0 (2026-10-29) — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list — RELEASED (release-agent.sh from a clean clone of `ca15a31`; agent sha256 da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404, config bundle 03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543; signed agent_config_update + agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-29 18:57:48 CET; `pinned.day0_agent` = 0.170.0)
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
a racing start would fail the reboot's own start half.
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
restarted your box at <time>".
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
answers on the next report, `operator_action_results: [{id, outcome, message}]` (`hub.HostReport`), which it requests
at once through the cap-1 report trigger. CLOSED list: `restart_controller <vmid>` only — anything else refused, nothing
called. The act is the supervisor's (`systemctl restart felhom-controller-bootstrap.service` in the guest, existing
grants — no sudoers change, no config bundle) with its guards: provisioned guest, guest running, the guest list and the
vzdump state readable, not parked, no swap, not locked, no vzdump. The thrash brake counts operator restarts with the
supervisor's: 3 in 15 minutes → the press is refused, and presses stay refused for 30 minutes. The result says how long
the controller took to answer (Docker's health check: healthy), or `failed` when it did not within 3 minutes. One
`audit: operator action` line per decision. Works with the controller stopped. Tests `TestOpActions_*` (closed list,
unknown/bad arg refused, restart + answer time + report trigger + once per id + re-send until dropped, stopped
controller, every guard, the brake across supervisor and operator restarts, no healthy answer = failed) and
`TestMainWiresTheAgentOperatorActions` (AST) — each seen failing by a mutation.
- **R-1344 — a controller that RUNS but does not answer: ALARM ONLY.** Each sweep also reads Docker's health status for a
running controller (the swap's `docker inspect -f` grant); 6 consecutive „unhealthy" sweeps (3 minutes, on top of
Docker's own ~90 s) start an episode: one ERROR line, the stanza's `unresponsive` / `unresponsive_since` (the hub raises
`controller_unresponsive`, operator-only). NOT restarted for it: the brief allows the restart only after a measurement
shows a busy night never reads like this, and that measurement runs through tonight (felhom.eu R-1344). Not counted
during a swap, a parked or locked guest, or a vzdump; healthy / starting / no health check / an unreadable status reset
the count. Tests `TestSupervisor_Unresponsive*` (red by mutation: the vzdump guard, the threshold).
## v0.168.0 (2026-10-29) — R-528: the host's per-container OOM counter (`GET /guest/oom-kills`); R-1254: a lost or late escrow answer is asked again before it is worded — RELEASED (release-agent.sh from a clean clone of `91bd96c`; agent sha256 1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca, bundle 3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8; no configs/ change since v0.167.0 → binary-only signed updates to demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 10:21:56 CET**; Day-0 pin set (D50)
### R-528
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
### R-1254 (the ceremony half, review round 2 HUNT-3 H2)
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
hub's current package opens only with the NEW code.
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
newest. Full `go build/vet/test` green. The controller and hub move the same night.
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
second later — so 0.166.0's pass never ran on the bundle that delivered it.
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
agent-only — holding the bundle job until the commit would need new hub queue logic.
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
removed → commit only, no pass — the demo-hp symptom).
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
0.165.0 box passes even without this fix.
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
„BUNDLE POLICY DONE install_mode=appliance removed=0".
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
drop-in until its next bundle. A change inside the wrapper alone cannot fix the run that delivers it.
- **The fix — the new AGENT asks the new wrapper:** `internal/osupdate/bundle.go` runs a policy pass right after every
bundle install (`mode: "bundle-policy"`, host layer, no envelope); the wrapper on disk is the bundle's own by then.
`configs/felhom-os-apply` `Bundle.policy_pass`: on a root-owned BYO record it removes the appliance-only files WE
wrote (our header), keeps the previous copy beside the bundle's, `sshd -t` then reloads ssh — a failing `sshd -t` puts
the file back. It writes nothing, so it needs no signature; an appliance and the owner's own file are left alone. Never
fatal for the agent: an older wrapper refuses the unknown mode and the next bundle does it. Report key `bundle_policy`.
- Tests: `TestR1285_AnInstalledBundleIsFollowedByThePolicyPass` (a refused bundle runs no pass);
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
usable that long, and the nonce store no longer keeps its record that long.
- `configs/felhom-os-apply`: the root wrapper holds the same 24 h (`MAX_WINDOW_S`, refusal R3 „the signed job's window
is longer than 24 h").
- `cmd/felhom-opsign`: `-ttl` above 24 h (or ≤ 0) is refused before anything is signed.
- Tests: `TestVerify_RejectsPerStage/window_longer_than_24_h` and `…/window_of_exactly_24_h_is_accepted`;
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
(`hub_min_agent` 0.158.1, `day0_agent` 0.165.0 — copies of hub state, because CI holds no hub secret; the session that
records a vouch or a new MinAgent updates them in the same commit). `check-published-versions.py` asserts the window
plus every pin; a pin with no tag FAILS. CI was red since the operator's hand prune of 0.159.0–0.160.0 (job 2334).
- **Decoys** (`test_gate_decoys.py`, via `FELHOM_RETENTION_POLICY`, test only): a pinned version older than the window is
asserted; its 404 convicts; a pin with no tag convicts. Seen failing on the old gate (two LIVE HOLES): felhom.eu
`documentation/audits/night-2026-10-25/redproof-d50.txt`.
## v0.165.0 (2026-10-25) — D43 = A: BYO hosts keep their own SSH settings (R-1257); the line test for a box that seeds all day (R-1250) — released in the attended release run; **VOUCHED by the operator 2026-10-25 18:38:49 CET** (binary `e9d04a47…`, bundle `8dea1f34…`)
**Found at the release (felhom.eu R-1285, P4):** on a BYO host upgraded from 0.164.0 the drop-in is removed only on the bundle run AFTER the one that installs this wrapper — the first run is planned by the old wrapper. Proven on a BYO test host (VM 9371 on demo-hp): the second `agent_config_update` logged `BUNDLE REMOVED … (ours, on a BYO host — D43)`.
### BYO hosts keep their own SSH settings (R-1257, D43 = A) — 2026-10-24 night
- **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an
appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy
is now `appliance`: written only when the ROOT-OWNED install record (`/var/lib/felhom-install/state.json`) says
`appliance`. On a `byo` record nothing is written, and a copy WE wrote (this exact path, beginning with our header
line) is REMOVED, its previous copy kept for the undo, then `sshd -t` and a reload. A box whose record is absent or
agent-writable proves nothing and is left exactly as it is. The bundle report lists `removed`; the record says
`removed` / `skipped`. The installer records the mode before it installs the bundle, so a new appliance still gets it.
- **X3: the „in force" check asks sshd about each of the host's own LAN addresses** (`ip -4 -o addr show scope
global`), not only 192.0.2.1 — a `Match Address <LAN>` block that re-allows root passwords no longer reads as in force.
- **X4: a failing `sshd -t` puts back ONLY the SSH step** (`undo_sshd_step`: the previous copy, or none; sshd is not
reloaded, so it never ran the refused state) and the rest of the bundle stays; it used to refuse and undo the whole
bundle. The report carries `sshd_step_undone`; the record names what is on disk.
- Tests (`configs/test_felhom_config_bundle.py`, class `ByoKeepsItsSSH`, 7 tests; each rule seen failing with it taken
out): a BYO host gets no drop-in; our copy comes off a BYO host; the owner's own file stays; no record → left as it
is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step.
The fake box now carries an appliance record by default (every earlier test unchanged).
### The line test for a box that seeds all day (R-1250) — 2026-10-24
Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0)
starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested
line and qBittorrent never got its 50 % limit. Now:
- **`linetest.Tester.WaitingForQuiet`** — due, inside the window, tries left, tunnel up, and only the quiet missing.
`GET /host/line-test` carries it as `waiting_for_quiet`; a controller with a seeding torrent app acts only on it.
It needs the full watched window (11 minute samples): a freshly started agent is not known to be busy, so it never
invites a hold (found in the live proof, 2026-10-24: the first build said `true` before one minute was watched;
`TestFreshTesterIsNotWaiting`).
- **`linetest.Tester.RunNow`** and **`POST /host/line-test/run`** (synchronous, about 55 s) — a test on request while the
controller holds its app's upload low: every weekly rule holds (due, window, `MaxTriesPerDay`, tunnel, the one heavy-op
gate) except the 10 quiet minutes, which become a 5 s settle and a 10 s quiet check (below 2 Mbit/s, so a request
never measures the family's own upload); the three runs are 5 s apart instead of 30. A refusal answers its code
(`not-due`, `outside-window`, `tries-used`, `no-tunnel`, `heavy-op`, `not-quiet`, `server-busy`); a failed request
spends one of the day's tries. `Tick` and `RunNow` now share one `ready()` check.
- Tests (each seen failing): `TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt` (the sleeps, the 60 s bound),
`TestRunNowRefusesAnUplinkThatIsStillBusy`, `TestRunNowKeepsTheWeeklyRules`, `TestRunNowFailuresSpendTheDaysTries`,
`TestQuietBoxIsNotWaiting`, `TestLineTestRunRoute`; a compile-time assertion that the Tester serves the request half.
**No sudoers or config-bundle change. MinController: none** — an older controller never calls the route.
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST
Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341` (configs unchanged — no bundle job sent). Built from
`f7e1dcb` (CI 2277 green) in a fresh clone by `scripts/release-agent.sh`. Delivered by one signed `agent_update` per box
(07:25Z): Tester 1, demo-hp and demo-felhom — COMPLETED, self-check 76/76, committed after the 1-minute dwell, the line
test started (felhom.eu `audits/line-test-2026-10-24/agent-0164-boxes.txt`).
Operator ruling 2026-10-24 07:26: once a week the box's HOST measures its upload line with a 10-second iperf3 upload to an
iperf3 server on ep0 that answers **only inside the WireGuard tunnel** (felhom.eu `scripts/ep0-iperf3/`; installed the same
+57
View File
@@ -1,5 +1,62 @@
# CONTEXT — felhom-agent working state
> **2026-11-14 (day, attended) — v0.174.0 RELEASED and VOUCHED (15:21:45 CET); on the three test boxes (agent + bundle, 77/77).** sha `6efbb978…`, bundle `7d87d058…`. Two new wrapper modes: `kernel-cleanup` (layer kernel; old kernel images by the hand-proven rule; ring-0 mark only; the plan names nothing; the night leg's last call, never with a kernel step) and `profile-refresh` (layer host; AppArmor profiles from a fresh compile once per kernel; one restart for a failed service whose cached profile was stale). Go 1.27.4. The N100's time service was repaired by the product path (+32.7 s step). A real kernel removal ran on Tester 1 by hand with a temporary ring-0 mark (operator's yes). **Next:** read the N100's and demo-hp's night (the clean-up by the agent at night is not proven yet); a ruling for ring-1 boxes; release 2b (trust in Proxmox by the root CA copy, D80). Report: felhom.eu `REPORT-release-2a-2026-11-14.md`.
> **2026-11-10 (afternoon, attended) — v0.173.0 RELEASED and VOUCHED (16:29:16 CET); on the three test boxes.** sha `752b2727…`, bundle `51ebc1f4…` (`configs/` unchanged since 0.172.1, so no bundle job). Measured: a box cleans its self-update folder at the commit of the update that BRINGS the code (49 → 1, 48 → 1, 33 → 1 entries; about 16 MB stays). The line below („at the first good update AFTER that release") was wrong. A rollback after the clean-up was proven on Tester 1 with a signed update to 0.172.1 and one crash in the dwell. `pinned.day0_agent` = 0.173.0. Open: the `.bak-*` files (R-1368 part 4).
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
> **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
> **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
>
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
> on demo-hp (felhom.eu R-1250, read, not forced).
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
> put back. Release + the operator's vouch: Monday's list (felhom.eu STATUS).
> **2026-10-24 (attended release run) — v0.164.0: the weekly upload line test (R-1247; operator ruling 07:26).**
> `internal/linetest`: the agent's own iperf3 client (operator's pick — nothing installed on hosts) to ep0's tunnel-only
> server `10.77.0.1:5201`, Tuesday 10:00-15:59, after 10 quiet minutes, under the heavy-op gate, best of 3; `line_test`
+26 -31
View File
@@ -1,36 +1,31 @@
# REPORT — day 2026-10-23: agent 0.163.0 released, delivered to the three test boxes, vouched by the operator
# REPORT — v0.174.0 released, vouched and delivered to the three test boxes (2026-11-14, attended)
**What:** the day brief's Part 2. Design read: `03-host-agent.md` (the tunnel, the bundle), `11` §5.4.2 (the step
bundle, R-880).
**Plain words first.** A box now removes its old kernels by itself, and it reloads its security profiles after it
starts on another kernel. Version 0.174.0 is released and runs on Tester 1, demo-hp and the N100. The operator vouched
it at 15:21 CET. The N100's time service runs again; the product repaired it, with no hand step.
## Release
The full report of the run is in `felhom.eu`: `REPORT-release-2a-2026-11-14.md`.
- Code: `1109092` (D4 v2: the sshd drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`,
`PermitRootLogin prohibit-password`, `try-reload-or-restart`, an `sshd -T` read-back; RootKeyOnly 6 tests). CI 2246
success. CHANGELOG `b4721ee` (v0.163.0 header, both sha256), CI 2249 success.
- `scripts/release-agent.sh 0.163.0`: binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`,
config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`; tag `v0.163.0`; verified by
an independent download.
- **Vouched by the operator** 07:58:46 UTC — read back from the hub DB (with its WAL) and from the running hub's
`/configuration` page: agent 0.163.0, both sha256 as above.
## What a reader must know
## Delivery — three signed jobs per box (felhom-opsign, key felhom-op-1)
1. **The clean-up runs on a box with the root-owned ring-0 mark only.** Tester 1 has no mark, so the program answers
„not run" there. For the proof the operator allowed the mark on Tester 1 for some minutes; it is back as before.
2. **The real removal on Tester 1 was started by hand**, with the agent's own command line. The night leg cannot be
started by day: it follows only the night's own backup (R-899). So „the step by the agent at night" is **not proven
yet**; the N100's and demo-hp's night of 2026-11-15 is the proof.
3. **The profile refresh was spiked first** on demo-hp (felhom.eu `audits/profile-refresh-spike-2026-11-14/`). Three
things it found are built in: a stale copy is told by the cached file changing, not by the kernel's hash; one
profile never compiles the same twice; a refresh hides other kernel audit lines for a few seconds.
4. **A clock step makes the controller read a new boot** (felhom.eu R-1395, a controller row): on the N100 it
recreated three apps once.
The D4 drop-in is a NEW bundle path, and the boxes ran bundle 0.160.0 (sha256 `6b0609…`), so: `agent_update 0.163.0`
→ `agent_config_update 0.163.0-step1` (built with `scripts/build-step-bundle.py` from 0.160.0: only
`felhom-os-apply` replaced; sha256 `6cbc0669…`, uploaded and read back) → `agent_config_update 0.163.0`.
| Box | agent | bundle | self-check | swap = RAM (pool guest 9201) | drop-in, `sshd -T` | sshd reloaded | key login | felhom-sshd |
|---|---|---|---|---|---|---|---|---|
| Tester 1 | 0.163.0 | 0.163.0 | ok | 4096 / 4096 | in place, `without-password` | 09:27:38 SIGHUP | ok | active, listening, `sshd -t` ok |
| demo-hp | 0.163.0 | 0.163.0 | ok | 25898 / 25898 | in place, `without-password` | 09:19:44 SIGHUP | ok | active, listening, `sshd -t` ok |
| demo-felhom | 0.163.0 | 0.163.0 | ok | 12288 / 12288 | in place, `without-password` | 09:24:42 SIGHUP | ok | active, listening, `sshd -t` ok |
**Not proven, said plainly:** the real root password over SSH was not tried (CC does not hold it); a dummy password is
refused, but OpenSSH logs that only as „Failed password", so the policy proof is `sshd -T` + the reload. felhom-sshd
(:8822) admits only the operator peer over the tunnel, so a connection from DooPlex times out by design. The console
login is not touched by the drop-in (`passwd -S root` = P on all three); not tried at a physical console.
Installer 1.36.0 (the uninstall removes the drop-in) was published the same day (felhom.eu). R-982 and R-1204 closed.
Evidence: felhom.eu `documentation/audits/day-2026-10-23/07-agent-0163-boxes.txt`, `10-vouch-readback.txt`,
`11-installer-1360.txt`.
| | |
|---|---|
| Version | 0.174.0, sha256 `6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92`, bundle `7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f`, Go 1.27.4 |
| Code commit, CI | `cc63dd5`, run id 2641 success |
| Tests | wrapper 191 (42 new), agent `internal/osupdate` green; 30 mutations, each seen red (felhom.eu `audits/release-2a-2026-11-14/A`, `B`) |
| Tester 1 | 3 kernels → 2, 1.12 GB gained, 8.5 s; second run: nothing; one restart, same kernel, healthy, clean stop |
| demo-hp | nothing to remove; refusals P1, P2, P3 by dry run |
| The N100 | time service repaired (offset −32.55 s → +0.18 s); tonight it removes 6 of 8 kernels |
| Vouch | the operator, 2026-11-14 15:21:45 CET; `pinned.day0_agent` = 0.174.0 |
| Not proven | the step at night by the agent; a fell-back box and a self-revert into the kept kernel on a box; K1 on a box |
+8 -2
View File
@@ -35,7 +35,7 @@
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()`. **The staging folder cleans itself** (`prune.go`, R-1368): after a healthy `commit` only the running version's staged binary stays; a new file kept in `<state dir>/selfupdate/` must NOT be named `felhom-agent-<version>` or the prune removes it |
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
@@ -62,6 +62,7 @@
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
| `GuestBinder.GuestBootKey` (R-1395) | internal/localapi/intermediary.go | `GuestBootKey(ctx, vmid) string` | "is this a new boot of the guest?" — `<kernel boot_id>-<guest init start>` | **`GuestBootID` carries `/proc/stat` `btime`, which the kernel computes from the wall clock: a clock STEP moves it.** Use the key for any „same boot?” question; `""` = unreadable. Seams `hostBtimeFn` / `hostBootIDFn` / `procStarttimeFn` |
| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
@@ -88,7 +89,8 @@
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause (`supervisorBrakeAllows` — every supervisor restart passes it; `recordSupervisorRestart` puts one on the record + R-539's slow counter); R-1344 (2026-10-30): a RUNNING controller whose Docker health reads „unhealthy" is alarmed at 6 sweeps and restarted at 10 (`restartUnresponsive`: the controller's busy file, then the brake); record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller`, `reboot_guest` since D65 — `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
### Proxmox client / hub / PBS / provisioning
@@ -97,6 +99,10 @@
|---|---|---|---|---|
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
| `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. |
| `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. |
| `pvetrust.Manager` (`Ensure`, `OnCallError`, `Status`) + `proxmox.Client.SetTLS` / `proxmox.TrustProblem` + `Leg.PVECACopy` + the wrapper's `pve_ca_copy` | internal/pvetrust/pvetrust.go; internal/proxmox/{client,trust}.go; internal/osupdate/pvecacopy.go; configs/felhom-os-apply (mode `pve-ca-copy`) | `Ensure(ctx, why) (changed bool)`; `TrustProblem(err) string` | R-1380 (felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA; the move from a pinned certificate; the cause of a trust failure | THE ORDER: the copy, a real call with the copy ALONE, then ONE config write that clears the fingerprint LAST (a set fingerprint wins over `ca_file` in `TLSConfig.build`). Never clear the fingerprint anywhere else. The wrapper's plan names nothing (T1). `agent.json` is written by the agent in place, with a parked recovery copy — the same way as the backup-target move. |
| `config.ReplaceInPlace` / `config.RecoverCutWrite` (+ `config.FileMu`) | internal/config/cutwrite.go; internal/config/filelock.go | `ReplaceInPlace(path, prev, next) error`; `RecoverCutWrite(path) (note string, err error)` | R-1380 (felhom.eu `03` §12a item 7): the only way to write `agent.json` — the folder is root-owned, so no rename; the old bytes go to a journal in the state folder first, and the daemon repairs a cut write at start, before it reads the config | A new writer takes `FileMu`, calls `ReplaceInPlace`, opens no file by hand, and is added to the two source pins in `cmd/felhom-agent` (`…TakeTheSharedLock`, `…UseTheJournalledWrite`). A package whose tests write the config sets `config.CutWriteDir` in `TestMain` |
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
+141
View File
@@ -0,0 +1,141 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// A box whose agent.json was cut in the middle of a write must start (R-1380): main repairs the file from the
// journal BEFORE it reads it. The staging is the state a power cut leaves: a durable journal of the old bytes and a
// half-written config.
func TestMainRecoversACutConfigWrite(t *testing.T) {
dir := t.TempDir()
prev := config.CutWriteDir
config.CutWriteDir = filepath.Join(dir, "state")
t.Cleanup(func() { config.CutWriteDir = prev })
if err := os.MkdirAll(config.CutWriteDir, 0o700); err != nil {
t.Fatal(err)
}
cfgPath := filepath.Join(dir, "agent.json")
old := `{"proxmox":{"node":"pve","tls":{"fingerprint":"AA:BB"}}}`
if err := os.WriteFile(filepath.Join(config.CutWriteDir, "agent.json.write-in-flight"), []byte(old), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(cfgPath, []byte(`{"proxmox":{"node":"pv`), 0o600); err != nil {
t.Fatal(err)
}
if _, err := config.Load(cfgPath); err == nil {
t.Fatal("the staging is wrong: the cut config loads")
}
note, err := recoverCutConfigWrite(cfgPath)
if err != nil || note == "" {
t.Fatalf("note=%q err=%v", note, err)
}
cfg, err := config.Load(cfgPath)
if err != nil {
t.Fatalf("the agent does not start after the repair: %v", err)
}
if cfg.Proxmox.TLS.Fingerprint != "AA:BB" {
t.Fatalf("the repaired config lost the fingerprint: %+v", cfg.Proxmox.TLS)
}
}
// main must call the repair before config.Load, or the repair never runs on the box that needs it (a seam that is
// built and not wired). Read from the source: the call's position in main's body.
func TestMainCallsTheConfigRepairBeforeItLoadsTheConfig(t *testing.T) {
src, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
f, err := parser.ParseFile(token.NewFileSet(), "main.go", src, 0)
if err != nil {
t.Fatal(err)
}
var repair, load token.Pos
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != "main" || fn.Body == nil {
continue
}
ast.Inspect(fn.Body, func(n ast.Node) bool {
c, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "recoverCutConfigWrite" && repair == 0 {
repair = c.Pos()
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Load" {
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" && load == 0 {
load = c.Pos()
}
}
return true
})
}
if repair == 0 || load == 0 {
t.Fatalf("main: repair call found=%v, config.Load found=%v", repair != 0, load != 0)
}
if repair > load {
t.Fatal("main reads the config BEFORE it repairs a cut write — a cut config stops the agent at start")
}
}
// Every writer of agent.json goes through config.ReplaceInPlace and none truncates the file itself: a writer that
// opens the config with O_TRUNC by hand is the unprotected write again.
func TestAgentJSONWritersUseTheJournalledWrite(t *testing.T) {
for _, w := range []struct{ file, fn string }{
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
} {
src, err := os.ReadFile(w.file)
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, w.file, src, 0)
if err != nil {
t.Fatal(err)
}
found := false
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
continue
}
found = true
body := string(src[fset.Position(fn.Body.Pos()).Offset:fset.Position(fn.Body.End()).Offset])
calls := 0
ast.Inspect(fn.Body, func(n ast.Node) bool {
if c, ok := n.(*ast.CallExpr); ok {
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "ReplaceInPlace" {
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" {
calls++
}
}
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "OpenFile" {
t.Errorf("%s %s opens a file by hand — the config is written only by config.ReplaceInPlace", w.file, w.fn)
}
}
return true
})
if calls != 1 {
t.Errorf("%s %s: %d call(s) to config.ReplaceInPlace, want 1", w.file, w.fn, calls)
}
if strings.Contains(body, "O_TRUNC") {
t.Errorf("%s %s truncates a file itself", w.file, w.fn)
}
}
if !found {
t.Errorf("%s: function %s not found — the pin reads nothing", w.file, w.fn)
}
}
}
@@ -0,0 +1,62 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// Three functions in this process read agent.json whole, change one section and write it whole: the trust move, the
// PBS seed and the backup-target move. Without one shared lock two of them interleave and the second puts back the
// document it read before the first one's change (found by review, 2026-11-14). Each must take config.FileMu as its
// FIRST statement and release it by defer — asserted from the source, because the race itself is not a thing a unit
// test can stage reliably.
func TestAgentJSONWritersTakeTheSharedLock(t *testing.T) {
for _, w := range []struct{ file, fn string }{
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
} {
f, err := parser.ParseFile(token.NewFileSet(), w.file, nil, 0)
if err != nil {
t.Fatalf("%s: %v", w.file, err)
}
found := false
for _, d := range f.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
continue
}
found = true
isLockCall := func(e ast.Expr, name string) bool {
c, ok := e.(*ast.CallExpr)
if !ok {
return false
}
sel, ok := c.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != name {
return false
}
inner, ok := sel.X.(*ast.SelectorExpr)
if !ok || inner.Sel.Name != "FileMu" {
return false
}
pkg, ok := inner.X.(*ast.Ident)
return ok && pkg.Name == "config"
}
if len(fn.Body.List) < 2 {
t.Errorf("%s %s: too short to hold the lock", w.file, w.fn)
continue
}
first, ok1 := fn.Body.List[0].(*ast.ExprStmt)
second, ok2 := fn.Body.List[1].(*ast.DeferStmt)
if !ok1 || !isLockCall(first.X, "Lock") || !ok2 || !isLockCall(second.Call, "Unlock") {
t.Errorf("%s %s: must start with config.FileMu.Lock() and defer config.FileMu.Unlock()", w.file, w.fn)
}
}
if !found {
t.Errorf("%s: function %s not found — a writer of agent.json was renamed; keep it under the shared lock and update this test", w.file, w.fn)
}
}
}
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
// passes as held → FAILS.
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
body := func(id string) []byte {
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
return b
}
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
if err != nil {
return nil, err
}
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
}
}
ctx := context.Background()
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
}
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
}
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
t.Fatal("a package without an identity blob cannot be compared — an error")
}
}
+8
View File
@@ -5,8 +5,16 @@ import (
"go/parser"
"go/token"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
)
// R-1250: SetLineTest takes a LineTestReader and finds the request half by a type assertion, so a Tester that lost
// WaitingForQuiet or RunNow would compile and silently answer 503 to every seeding box. This line makes it a build
// failure instead.
var _ localapi.LineTestRequester = (*linetest.Tester)(nil)
// R-1247: the weekly line test has the fstrim shape (component + reporter seam + goroutine) plus a local-API reader,
// so its wiring is asserted from the AST. It must share the ONE heavy-op gate (heavyOps) — or it could run beside a
// backup — and take its server from the tunnel manager, never a literal address.
+234 -4
View File
@@ -54,6 +54,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfheal"
@@ -216,6 +217,15 @@ func main() {
return
}
// Only the daemon repairs: a selftest can run beside a live agent that is in the middle of a write.
var cutWriteNote string
var cutWriteErr error
if selftest.mode == "" {
cutWriteNote, cutWriteErr = recoverCutConfigWrite(cfgPath)
if cutWriteErr != nil {
fmt.Fprintln(os.Stderr, "config repair:", cutWriteErr)
}
}
cfg, err := config.Load(cfgPath)
if err != nil {
// A missing default config file is fine if env provides the values; only a
@@ -229,6 +239,11 @@ func main() {
// logRing is the always-DEBUG capture ring (v0.83.0 observability): served by the
// local API's GET /debug/logs and the heartbeat log-pull. Selftests ignore it.
logger, logRing := applog.New(cfg.LogLevel)
if cutWriteErr != nil {
logger.Error("config: the last write of the config was cut and could not be repaired", "err", cutWriteErr)
} else if cutWriteNote != "" {
logger.Warn("config: a write of the config was cut by a stop — repaired at start", "result", cutWriteNote)
}
switch selftest.mode {
case "":
@@ -251,6 +266,8 @@ func main() {
os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
case "os-facts":
os.Exit(runSelftestFacts(context.Background(), cfg, logger, vmid))
case "kernel-cleanup-dry-run":
os.Exit(runSelftestKernelCleanupDryRun(context.Background(), cfg, logger, vmid))
case "live-restore":
os.Exit(runSelftestLiveRestore(context.Background(), cfg, logger, vmid))
case "pbs-verify":
@@ -521,6 +538,66 @@ type storeGrantRepairer struct {
mu sync.Mutex
last map[string]time.Time // target id → last ATTEMPT (success or failure)
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
// seenPath (R-1302) is the file that remembers which tiers have EVER held the grant: only those can LOSE it. "" = no
// memory, so every missing grant reads as a loss — the old, loud behaviour.
seenPath string
}
// everHeld reports whether this tier's grant was ever seen on this box (R-1302). The direction of a doubt is „yes": an
// unreadable or corrupt file keeps the loss ERROR, so a real loss is never quietened by a broken record.
func (r *storeGrantRepairer) everHeld(target string) bool {
if r == nil || r.seenPath == "" {
return true
}
r.mu.Lock()
defer r.mu.Unlock()
seen, err := readGrantSeen(r.seenPath)
return err != nil || seen[target]
}
// noteHeld records that this tier holds the grant. A write that fails is logged and costs one false „first grant" at most
// after a restart — the old file, or none, stands.
func (r *storeGrantRepairer) noteHeld(target string) {
if r == nil || r.seenPath == "" {
return
}
r.mu.Lock()
defer r.mu.Unlock()
seen, err := readGrantSeen(r.seenPath)
if err != nil {
seen = map[string]bool{} // a corrupt file is replaced by what is true now
}
if seen[target] {
return
}
seen[target] = true
b, _ := json.Marshal(seen)
tmp := r.seenPath + ".tmp"
if werr := os.WriteFile(tmp, b, 0o600); werr == nil {
werr = os.Rename(tmp, r.seenPath)
if werr == nil {
return
}
err = werr
} else {
err = werr
}
repairLogger(r).Warn("store-grant: could not record that this tier holds its grant (R-1302)", "target", target, "err", err)
}
func readGrantSeen(path string) (map[string]bool, error) {
b, err := os.ReadFile(path)
if os.IsNotExist(err) {
return map[string]bool{}, nil
}
if err != nil {
return nil, err
}
seen := map[string]bool{}
if err := json.Unmarshal(b, &seen); err != nil {
return nil, err
}
return seen, nil
}
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
@@ -631,6 +708,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
// host-report has certainly carried it. Without this latch the repairing probe may be a
// log-only one and the hub never learns anything happened (measured live, see the window's
// comment).
repair.noteHeld(targetID) // R-1302: a tier that holds its grant can lose it from now on
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
}
@@ -679,8 +757,25 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
// change, no hub change, no new event type. The `Feature` text carries the explanation because
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
return storeGrantAfterRepair(targetID, critical, repair, s)
}
// storeGrantAfterRepair is the verdict once a repair is confirmed (split out for the tests). R-1302: the FIRST grant of a
// tier that never held one is no loss — on a fresh install the whole-guest tier is armed before its storage exists, so
// every new box logged „investigate the loss" and a DEGRADED capability at its first start, which teaches the reader to
// skip the line. That grant logs INFO and reports healthy. A tier seen holding its grant, or one this box cannot tell
// about, keeps R-190's ERROR and its one degraded cycle. Pinned by TestR1302_*.
func storeGrantAfterRepair(targetID string, critical bool, repair *storeGrantRepairer, healthy capability.Status) capability.Status {
if !repair.everHeld(targetID) {
repair.noteHeld(targetID)
repairLogger(repair).Info("store-grant: granted — the first grant of this tier, nothing was lost (R-1302)",
"target", targetID, "privilege", storeGrantRequiredPriv,
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
healthy.Status, healthy.Reason = capability.StatusOK, ""
return healthy
}
repair.noteRepaired(targetID, time.Now())
s = storeGrantRepairedVerdict(targetID, critical)
s := storeGrantRepairedVerdict(targetID, critical)
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
"target", targetID, "privilege", storeGrantRequiredPriv,
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
@@ -853,8 +948,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
// invoked through the privileged path, which prepends sudo itself.
grantRepairer := &storeGrantRepairer{
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger,
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
log: logger,
seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json"), // R-1302
}
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
@@ -907,6 +1003,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
osLeg := newOSLeg(cfg, client, px, logger)
collector.SetSystemReporter(&factsReporter{leg: osLeg, guest: firstGuest(px)}) // R-852: the versions
desiredSyncer.AddConsumer(osLeg)
// R-1380 (D80, felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA. At this start the root
// step makes the copy, a REAL call proves it while the pinned fingerprint is still the trust, and only then ONE
// write of the config sets ca_file and clears the fingerprint. A call that fails on trust later names the cause and
// runs the same order again (a pinned box whose certificate changed moves right then, with no restart).
pveTrust := newPVETrust(cfg, px, osLeg, logger)
collector.SetProxmoxTrustSource(func() any { return pveTrust.Status() })
loop.SetCollectErrorHook(pveTrust.OnCallError)
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
@@ -937,6 +1040,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1380: the trust order at this start (the copy, the proof, then the config) — before the first report cycle, so
// a pinned box whose certificate changed while the agent was down is not blind for a cycle. Then on a timer.
pveTrust.Ensure(ctx, "start")
go pveTrust.Run(ctx, time.Hour, 24*time.Hour)
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
@@ -1205,6 +1312,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
return release, nil
}}
daemonCtx := ctx
profileGate := func() (func(), string, bool) { return heavyOps.TryAcquire("os-profile-refresh") }
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
@@ -1215,7 +1324,20 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
names = append(names, d.Name)
}
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
// R-1388: a new wrapper may know the profile refresh the old one refused — ask again (its own goroutine:
// it may wait for the heavy-op gate, and the signed job must not wait with it).
go osLeg.ProfileRefresh(daemonCtx, "config bundle installed", profileGate, 12, 5*time.Minute)
// R-1380: a new wrapper may know the CA-copy step the old one refused — run the trust order again.
go pveTrust.Ensure(daemonCtx, "config bundle installed")
}}
// R-1388 / R-1301 (`11` §5.12): after a start, load the security profiles from a fresh compile when the box runs
// another kernel than at the last refresh (the wrapper keeps that record and answers "not needed" otherwise).
// R-1383 (release 3): the last refresh, and how this process's last ask ended, ride the host report.
collector.SetProfileRefreshSource(osLeg.ProfileRefreshStanza)
go osLeg.ProfileRefresh(daemonCtx, "agent start", profileGate, 12, 5*time.Minute)
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
// to the local API server built below).
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
@@ -1247,6 +1369,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
// subprocess used, under the same per-host key, before the code is handed out.
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
// anything — the same per-host GET the identity read-back uses.
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
if client == nil {
return false, fmt.Errorf("hub client not configured")
}
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
},
DRConfigured: func() bool {
if drConfigured != nil {
return drConfigured()
@@ -1256,6 +1386,21 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil {
// R-1383 (D78): the guest-power watchdog's give-up rides the host report (it was one journal line).
collector.SetGuestPowerGaveUpSource(localSrv.GuestPowerGaveUpStanza)
}
if localSrv != nil {
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
// which it requests at once through the same cap-1 trigger the storage watchdog uses.
localSrv.SetOperatorActionTrigger(func() {
select {
case storageTrigger <- struct{}{}:
default:
}
})
collector.SetOperatorActionResults(localSrv.OperatorActionResults)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner, localSrv))
backupCapRefusal = localSrv.BackupCapRefusal // R-973
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
@@ -3363,6 +3508,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
// without an identity blob cannot be compared: an error (the caller then promises nothing).
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
var up escrowUploadRequest
if err := json.Unmarshal(pkg, &up); err != nil {
return false, fmt.Errorf("reading the pending package: %w", err)
}
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
if err != nil || len(mine) == 0 {
return false, fmt.Errorf("the pending package carries no identity blob to compare")
}
resp, err := fetch(ctx)
if err != nil {
return false, err
}
if !resp.Present || resp.IdentityEscrowB64 == "" {
return false, nil
}
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
if err != nil {
return false, fmt.Errorf("decoding the hub's package: %w", err)
}
return bytes.Equal(mine, theirs), nil
}
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
@@ -3801,14 +3972,39 @@ func (f *selftestFlag) Set(v string) error {
f.mode = "os-update"
case "os-facts", "live-restore": // agent v0.142.0
f.mode = v
case "kernel-cleanup-dry-run": // R-1368: what the clean-up of old kernels WOULD do; changes nothing
f.mode = v
case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
f.mode = "wgtunnel"
default:
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update|os-facts|live-restore)", v)
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|kernel-cleanup-dry-run|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update|os-facts|live-restore)", v)
}
return nil
}
// newPVETrust wires internal/pvetrust (R-1380): the root step through the OS wrapper, the proof as a throwaway client
// with EXACTLY the trust under test, and the running client's SetTLS.
func newPVETrust(cfg config.Config, px *proxmox.Client, leg *osupdate.Leg, logger *slog.Logger) *pvetrust.Manager {
m := &pvetrust.Manager{ConfigPath: cfg.SourcePath, StateDir: agentStateDir(cfg), Logger: logger,
EnsureCopy: func(ctx context.Context) (pvetrust.Copy, error) {
r, err := leg.PVECACopy(ctx)
return pvetrust.Copy{Path: r.Path, Outcome: r.Outcome, NotAfter: r.NotAfter, DaysLeft: r.DaysLeft}, err
},
Probe: func(ctx context.Context, t proxmox.TLSConfig) error {
p, err := proxmox.NewClient(proxmox.Config{Endpoint: cfg.Proxmox.Endpoint, Node: cfg.Proxmox.Node, Token: cfg.Proxmox.Token, TLS: t})
if err != nil {
return err
}
_, err = p.Version(ctx)
return err
},
Apply: px.SetTLS,
}
m.Start(proxmox.TLSConfig{CAFile: cfg.Proxmox.TLS.CAFile, Fingerprint: cfg.Proxmox.TLS.Fingerprint,
InsecureSkipVerify: cfg.Proxmox.TLS.InsecureSkipVerify})
return m
}
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
// the once-per-night marker live in the agent's own os/ dir.
func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg {
@@ -3925,6 +4121,24 @@ func runSelftestFacts(ctx context.Context, cfg config.Config, logger *slog.Logge
return 0
}
// runSelftestKernelCleanupDryRun asks the root wrapper what the clean-up of old kernels WOULD do on this box (R-1368,
// `11` §5.11.1): the keep set, the removal set, and the refusal or "not run" reason. It changes nothing.
//
// sudo -u felhom-agent felhom-agent --config … --selftest=kernel-cleanup-dry-run
func runSelftestKernelCleanupDryRun(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
px, _ := newProxmoxClient(cfg)
leg := newOSLeg(cfg, nil, px, logger)
raw, err := leg.KernelCleanupDryRun(ctx, vmid)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest=kernel-cleanup-dry-run:", err)
return 1
}
var v any
_ = json.Unmarshal(raw, &v)
printJSON("kernel clean-up — DRY RUN (nothing was changed)", v)
return 0
}
// runSelftestLiveRestore is the ONE-TIME live-restore step (`09` decision 87) as a debug action — the night leg does
// the same before a ring-0 Docker step. It prints the container ids before and after (they must not change).
//
@@ -4040,3 +4254,19 @@ func agentStateDir(cfg config.Config) string {
}
return "/var/lib/felhom-agent"
}
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
}
// recoverCutConfigWrite repairs agent.json BEFORE it is read, when the last write of it was cut by a power loss
// (config.RecoverCutWrite; R-1380). It runs before the logger exists, so main logs what it returns.
// Pinned by TestMainRecoversACutConfigWrite.
func recoverCutConfigWrite(cfgPath string) (string, error) {
if cfgPath == "" {
return "", nil
}
return config.RecoverCutWrite(cfgPath)
}
+51
View File
@@ -0,0 +1,51 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
// the server in the report loop's observer chain, give the collector its results and give the server the report
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
// RED-PROOF: seen failing before the wiring existed (all three false).
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var observed, results, trigger bool
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
fn, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
switch fn.Sel.Name {
case "MultiObserver":
for _, a := range call.Args {
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
observed = true
}
}
case "SetOperatorActionResults":
if len(call.Args) == 1 {
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
results = true
}
}
case "SetOperatorActionTrigger":
trigger = true
}
return true
})
if !observed || !results || !trigger {
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
}
}
+71
View File
@@ -0,0 +1,71 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-1380: the trust move is a component with seams (the root step, the proof, the running client, the report stanza,
// the failed-call hook). A unit-green pvetrust that main.go never starts is the inert-seam defect, so the wiring is
// asserted from the AST: built once, run at start and after a bundle install, on a timer, fed by a failed collect, and
// reported. The same for the profile refresh (R-1388), which had no such test.
func TestMainWiresPVETrust(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
seen := map[string]int{}
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
sel, ok := node.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
x, _ := sel.X.(*ast.Ident)
switch {
case x != nil && x.Name == "pveTrust" && sel.Sel.Name == "Ensure":
seen["ensure"]++
case sel.Sel.Name == "SetProxmoxTrustSource":
seen["stanza"]++
case sel.Sel.Name == "SetProfileRefreshSource" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "ProfileRefreshStanza" {
seen["profile refresh → report"]++ // R-1383
}
case sel.Sel.Name == "SetGuestPowerGaveUpSource" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "GuestPowerGaveUpStanza" {
seen["guest-power give-up → report"]++ // R-1383
}
case sel.Sel.Name == "SetCollectErrorHook" && len(node.Args) == 1:
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "OnCallError" {
seen["hook"]++
}
case x != nil && x.Name == "osLeg" && sel.Sel.Name == "ProfileRefresh":
seen["profile-refresh"]++
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "pveTrust" {
seen["timer"]++
}
}
case *ast.KeyValueExpr:
if k, ok := node.Key.(*ast.Ident); ok && k.Name == "Apply" {
if v, ok := node.Value.(*ast.SelectorExpr); ok && v.Sel.Name == "SetTLS" {
seen["apply=SetTLS"]++
}
}
}
return true
})
want := map[string]int{"ensure": 2, "stanza": 1, "hook": 1, "timer": 1, "apply=SetTLS": 1, "profile-refresh": 2,
"guest-power give-up → report": 1, "profile refresh → report": 1}
for k, n := range want {
if seen[k] < n {
t.Errorf("main.go wires %q %d time(s), want at least %d (the two-call ones: at start and after a bundle install)", k, seen[k], n)
}
}
}
@@ -0,0 +1,91 @@
package main
import (
"context"
"encoding/json"
"io"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
)
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
type orderRunner struct {
mu sync.Mutex
order []string
}
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
var p map[string]any
b, _ := os.ReadFile(args[1])
_ = json.Unmarshal(b, &p)
mode, _ := p["mode"].(string)
r.order = append(r.order, "os-apply:"+mode)
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
}
if len(args) >= 1 {
r.order = append(r.order, filepath.Base(name)+":"+args[0])
}
return []byte("ok"), nil, nil
}
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
t.Helper()
stateDir := t.TempDir()
if marker != nil {
dir := filepath.Join(stateDir, "selfupdate")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(marker)
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
t.Fatal(err)
}
}
r := &orderRunner{}
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
mgr.MaybeCommit(context.Background())
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.order...)
}
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
}
}
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
}
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 0 {
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
}
}
+101
View File
@@ -1,11 +1,14 @@
package main
import (
"bytes"
"context"
"errors"
"go/ast"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
"be missed entirely", storeGrantRepairReportWindow)
}
}
// ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
//
// A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
// logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
//
// RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
// TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
t.Helper()
var buf bytes.Buffer
return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
}
func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
r, buf := r1302Repairer(t, path)
healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
if s.Status != capability.StatusOK || s.Reason != "" {
t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
}
got := buf.String()
if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
t.Fatalf("a first grant must not log the loss ERROR: %s", got)
}
if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
t.Fatalf("a first grant says so at INFO: %s", got)
}
if r.recentlyRepaired("felhom-pbs", time.Now()) {
t.Fatal("a first grant must not arm the loss report's latch")
}
}
func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
first, _ := r1302Repairer(t, path)
first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
}
if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
}
if !r.recentlyRepaired("felhom-pbs", time.Now()) {
t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
}
}
// After a first grant the tier counts as held: the NEXT disappearance is a loss.
func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
r, _ := r1302Repairer(t, path)
storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
if s.Status != capability.StatusDegraded {
t.Fatalf("the second disappearance is a loss; got %q", s.Status)
}
}
// The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
func TestR1302_ADoubtKeepsTheError(t *testing.T) {
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
r, _ := r1302Repairer(t, path)
if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
}
noMemory, _ := r1302Repairer(t, "")
if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("no record must not quieten a loss; got %q", s.Status)
}
if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
}
}
// The wiring, read from the source: production names the file, and a healthy probe records the tier.
func TestR1302_TheMemoryIsWired(t *testing.T) {
b, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
src := string(b)
if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
}
start := strings.Index(src, "func storeGrantStatus(")
if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
}
}
+4
View File
@@ -123,6 +123,10 @@ func run() error {
now := time.Now().UTC()
issued := now
if *ttl <= 0 || *ttl > authz.MaxWindow {
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
os.Exit(2)
}
expires := now.Add(*ttl)
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
+3
View File
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
+634 -20
View File
@@ -97,9 +97,12 @@ OOMCHECK_SETTLE = 2
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
# proxmox-grub is GRUB's Proxmox meta-package (it depends on grub-efi-amd64 of the same version); the `grub` above is
# anchored at the start, so it missed it. Measured 2026-11-05..07 on demo-felhom: its upgrade pulled grub-efi-amd64 and
# the whole pve step was refused R6 every night. Pinned by test_ring0_pending_pve_leaves_proxmox_grub_out.
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
r"proxmox-secure-boot-support)")
r"proxmox-secure-boot-support|proxmox-grub)")
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
@@ -152,9 +155,36 @@ KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pv
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel",
"kernel-cleanup")
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# The clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1). A step that ENDED in one of these phases
# and is not "good" keeps every kernel (the box fell back or reverted); any other phase is "not ended" (P1).
# Trust in Proxmox by a copy of the box's own root CA (R-1380, `09` §3 decision 303, felhom.eu `03` §12a). The agent's
# user cannot read Proxmox's CA file (0640 root:www-data); this root step keeps a readable copy of that ONE public
# certificate where the agent's `proxmox.tls.ca_file` points.
PVE_CA_SOURCE = "/etc/pve/pve-root-ca.pem"
PVE_CA_COPY = "/etc/felhom-agent/pve-root-ca.pem" # 0644 root: a CA certificate is public; the agent only reads it
PVE_CA_MAX_BYTES = 64 * 1024
PVE_LOCAL_API = "127.0.0.1:8006" # pveproxy on this host (the installer's proof uses the same)
PEM_CERT_RE = re.compile(rb"-----BEGIN CERTIFICATE-----\s[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----")
# The security-profile refresh (R-1388, R-1301, `11` §5.12; measured on demo-hp 2026-11-14,
# felhom.eu audits/profile-refresh-spike-2026-11-14/). AppArmor keeps one cache folder per kernel feature set; a
# profile that changed while ANOTHER kernel ran leaves an old compiled copy in this kernel's folder, with a time that
# looks newer than its source — and the kernel loads it at the next start on this kernel.
PROFILE_DIR = "/etc/apparmor.d"
PROFILE_CACHE = "/var/cache/apparmor"
PROFILE_STATE = "/var/lib/felhom-os-apply/profile-refresh.json" # 0644 root: the kernel the last refresh ran on
PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", PROFILE_DIR]
PROFILE_PATH_NAME_RE = re.compile(r"^/[A-Za-z0-9._/+-]+$") # a profile whose name is one plain program path
UNIT_NAME_RE = re.compile(r"^[A-Za-z0-9@:._-]+\.service$")
PROFILE_RESTART_MAX = 3
KERNEL_ENDED_NOT_GOOD = ("self_reverted", "revert_failed", "fell_back", "cancelled")
KERNEL_LEFTOVER_PATHS = ("/lib/modules/{v}", "/boot/vmlinuz-{v}", "/boot/initrd.img-{v}", "/boot/System.map-{v}",
"/boot/config-{v}")
PINNED_KVER_RE = re.compile(r"gnulinux-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)-advanced-")
ANY_KVER_RE = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve")
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
@@ -168,7 +198,10 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
# sshd → `sshd -t -f` on the snippet alone (R-982).
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none);
# appliance → only on a box whose ROOT-OWNED install record says appliance (D43 = A, operator 2026-10-24: a BYO
# host keeps its own settings, `11` §1). On a BYO box nothing is written and a copy WE wrote (this exact path, our
# first line) is removed; a box with no readable record is left exactly as it is (it cannot prove which it is).
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
# R-861 (agent v0.146.0): the signed agent update, verified here (mode agent_update), then the A/B flip.
SELFUPDATE_OP = "agent_update"
@@ -184,9 +217,13 @@ BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies
BUNDLE_KEEP = 3
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
# R-1280: the longest validity window a signed job may carry — the agent's authz.MaxWindow, the same 24 h.
MAX_WINDOW_S = 24 * 3600
OOB_DIR = "/etc/felhom-sshd"
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
# D43 = A (R-1257): the first line every copy of the drop-in we ship carries — what proves a file on a BYO host is ours.
SSHD_DROPIN_HEADER = b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982"
BUNDLE_FILES = [
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
@@ -227,7 +264,8 @@ BUNDLE_FILES = [
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
# R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
# password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "replace"),
# D43 = A (operator 2026-10-24, R-1257): appliances only — a BYO host keeps its own SSH settings (`11` §1).
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "appliance"),
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
]
@@ -483,7 +521,8 @@ class Apply:
def check_plan(self, plan):
mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
"oom-check", "profile-refresh", "pve-ca-copy") \
+ KERNEL_MODES:
raise Refused("R11", f"unknown mode {mode!r}")
if mode == "agent_update":
@@ -494,6 +533,29 @@ class Apply:
if plan.get("layer") != "host":
raise Refused("R11", "bundle is a host-layer mode")
return mode, "host", 0, "bundle"
if mode == "bundle-policy":
if plan.get("layer") != "host":
raise Refused("R11", "bundle-policy is a host-layer mode")
return mode, "host", 0, "bundle-policy"
if mode == "pve-ca-copy":
if plan.get("layer") != "host":
raise Refused("R11", "pve-ca-copy is a host-layer mode")
# R-1380 (D80): ONE fixed source, ONE fixed place — the plan may name NOTHING (no path, no file, no content)
extra = sorted(k for k in plan if k not in ("mode", "layer", "release_id", "run_id", "trigger", "ring"))
if extra:
raise Refused("T1", f"a pve-ca-copy plan carries {', '.join(extra)} — it takes nothing from a plan: "
f"the source is {PVE_CA_SOURCE} and the copy is {PVE_CA_COPY}, both fixed")
return mode, "host", 0, "pve-ca-copy"
if mode == "profile-refresh":
if plan.get("layer") != "host":
raise Refused("R11", "profile-refresh is a host-layer mode")
# it takes NOTHING from the plan but dry_run: no profile name, no unit name, no package (R-1388)
for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"):
if plan.get(k):
raise Refused("F1", f"a profile-refresh plan carries {k!r} — it takes nothing from a plan but dry_run")
if not isinstance(plan.get("dry_run", False), bool):
raise Refused("F1", "dry_run must be true or false")
return mode, "host", 0, "profile-refresh"
layer = plan.get("layer")
if layer not in ("guest", "host", "docker", "pve", "kernel"):
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
@@ -524,6 +586,17 @@ class Apply:
rid = plan.get("release_id", "")
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
raise Refused("R11", f"release_id {rid!r} is not a plain id")
if mode == "kernel-cleanup":
# `11` §5.11.1, rule 4 of the release: the clean-up NEVER takes a package name (or any choice of what to
# remove) from a plan — the wrapper computes the set from the box. Pinned by test_cleanup_plan_names_refused.
for k in ("packages", "allow_new", "signed", "expect_kver", "undo", "snapshot"):
if plan.get(k):
raise Refused("C1", f"a kernel-cleanup plan carries {k!r} — the clean-up takes nothing from a plan "
f"but dry_run; the wrapper computes the set from the box")
if plan.get("select", "listed") != "listed":
raise Refused("C1", "a kernel-cleanup plan carries a select — the wrapper computes the set from the box")
if not isinstance(plan.get("dry_run", False), bool):
raise Refused("C1", "dry_run must be true or false")
if plan.get("allow_new"):
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
select = plan.get("select", "listed")
@@ -655,6 +728,8 @@ class Apply:
iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ"))
except (KeyError, ValueError, TypeError):
raise Refused("R3", "the signed job has no readable time window")
if exp - iss > MAX_WINDOW_S:
raise Refused("R3", "the signed job's window is longer than 24 h (R-1280)")
if now > exp or now < iss - 120:
raise Refused("R3", "the signed job is expired or not yet valid")
nonce = op.get("nonce")
@@ -1064,6 +1139,15 @@ class Apply:
return self.facts()
if self.mode == "bundle":
return Bundle(self).from_plan(plan)
if self.mode == "bundle-policy":
return Bundle(self).policy_pass()
if self.mode == "pve-ca-copy":
# NO appliance check, on purpose (like the bundle): the copy is the agent's OWN trust file in the agent's own
# config folder, and it changes nothing of the host's — a BYO host's agent talks to Proxmox the same way.
return self.pve_ca_copy()
if self.mode == "profile-refresh":
self.check_appliance()
return self.profile_refresh(plan)
if self.mode == "agent_update":
return self.agent_update(plan)
if self.layer in ("host", "pve", "kernel"):
@@ -1129,6 +1213,198 @@ class Apply:
self.report["oom_check"] = self.oom_check()
return 0
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80) ----------
def is_live_ca(self, block):
"""One PEM certificate block: is it a CA certificate that has not ended? (openssl, the bytes on stdin)."""
try:
rc, out, _ = self.r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-checkend", "0"], 30,
stdin=block.decode("ascii") + "\n")
except (UnicodeDecodeError, subprocess.SubprocessError, OSError):
return False
return rc == 0 and re.search(r"\bCA:TRUE\b", out) is not None
def pve_serves_with(self, cafile):
"""Does the local Proxmox API serve a certificate that verifies with this CA file ALONE? One handshake to
PVE_LOCAL_API; nothing is sent. Any doubt (no answer, a timeout) is "no" — the previous root then stays."""
try:
rc, out, err = self.r.host(["openssl", "s_client", "-connect", PVE_LOCAL_API, "-CAfile", cafile, "-no-CApath",
"-verify_return_error"], 20, stdin="")
except (subprocess.SubprocessError, OSError):
return False
return rc == 0 and "Verify return code: 0 (ok)" in (out + err)
def pve_ca_copy(self):
"""Copy exactly PVE_CA_SOURCE to exactly PVE_CA_COPY — only if the source is ONE CA certificate that has not
ended — written whole or not at all (a root-owned temp file beside the target, fsync, rename). "same" when the
copy is exactly the source's bytes. The bytes that are checked are the bytes that are written first; only
during a change of the root the copy also holds its previous certificate behind them (see below)."""
r = self.r
res = {"outcome": None, "path": PVE_CA_COPY}
self.report["pve_ca_copy"] = res
try:
data = r.read_bytes(PVE_CA_SOURCE)
except OSError as e:
raise Refused("T2", f"{PVE_CA_SOURCE} cannot be read ({e.__class__.__name__}) — is this a Proxmox host?")
if not data or len(data) > PVE_CA_MAX_BYTES:
raise Refused("T3", f"{PVE_CA_SOURCE} is {len(data)} bytes — not one certificate")
certs = PEM_CERT_RE.findall(data)
if len(certs) != 1 or b"PRIVATE KEY" in data:
raise Refused("T3", f"{PVE_CA_SOURCE} holds {len(certs)} certificate(s)"
+ (" and a private key" if b"PRIVATE KEY" in data else "") + " — exactly one CA certificate is expected")
try:
text = data.decode("ascii") # PEM is ASCII; the runner hands text to openssl's stdin
except UnicodeDecodeError:
raise Refused("T3", f"{PVE_CA_SOURCE} is not a PEM text file")
rc, out, err = r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-enddate"], 30, stdin=text)
if rc != 0:
raise Refused("T3", f"{PVE_CA_SOURCE} is not a readable certificate: {(out + err).strip()[-160:]}")
if not re.search(r"\bCA:TRUE\b", out):
raise Refused("T3", f"{PVE_CA_SOURCE} is not a CA certificate (no CA:TRUE) — a leaf or any other file is never copied")
m = re.search(r"notAfter=(.+)", out)
try:
import calendar
not_after = calendar.timegm(time.strptime(m.group(1).strip(), "%b %d %H:%M:%S %Y GMT")) # UTC, no DST
except (AttributeError, ValueError):
raise Refused("T3", f"the end date of {PVE_CA_SOURCE} cannot be read")
days = int((not_after - r.now()) // 86400)
if days < 0:
raise Refused("T3", f"{PVE_CA_SOURCE} ended {-days} day(s) ago — an ended CA is never copied")
res.update(sha256=sha256_hex(data), not_after=_iso(not_after), days_left=days)
try:
have = r.read_bytes(PVE_CA_COPY)
except OSError:
have = None
if have == data:
res["outcome"] = "same"
return 0
# The copy the agent trusts by is replaced BEFORE anything can be proven with the new root (Proxmox serves its
# old certificate until pveproxy is reloaded). So at a CHANGE the copy's previous first certificate — written
# by this same step — is kept behind the new one, and the agent trusts both roots across the change, also
# after a restart. That second root is kept ONLY for the changeover: it is dropped as soon as Proxmox serves a
# certificate that verifies with the new root alone, and at once when it is no longer a CA that has not ended.
# The copy is compared EXACTLY: nothing this step did not check is ever left in it. (Review, 2026-11-14.)
# To drop the old root at once by hand (a root that was changed because it leaked): delete the copy.
prev = PEM_CERT_RE.findall(have) if have and b"PRIVATE KEY" not in have else []
if len(prev) == 2 and have == data + prev[1] + b"\n":
over = self.pve_serves_with(PVE_CA_SOURCE)
if not over and self.is_live_ca(prev[1]):
res["outcome"], res["kept_previous"] = "same", True # the changeover is still on
return 0
r.put_file(PVE_CA_COPY, data, 0o644)
res["outcome"], res["replaced"], res["kept_previous"], res["dropped_previous"] = "written", True, False, True
r.log(f"os-apply: PVE CA COPY the previous root is dropped from {PVE_CA_COPY} "
f"({'Proxmox serves a certificate of the new root' if over else 'the previous root is no longer a live CA'})")
return 0
want, kept = data, False
if prev and prev[0].strip() != certs[0].strip() and self.is_live_ca(prev[0]):
want, kept = data + prev[0] + b"\n", True
r.put_file(PVE_CA_COPY, want, 0o644)
res["outcome"], res["replaced"], res["kept_previous"] = "written", have is not None, kept
r.log(f"os-apply: PVE CA COPY written {PVE_CA_COPY} sha256={res['sha256'][:16]} ends={res['not_after']} "
f"({'replaced an older copy' if have is not None else 'first copy'})")
return 0
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
def profile_cache(self):
"""{"<folder>/<file>": sha256} of every cached profile, every kernel's folder."""
out = {}
for d in self.r.list_dir(PROFILE_CACHE):
dp = f"{PROFILE_CACHE}/{d}"
if not self.r.isdir(dp):
continue
for f in self.r.list_dir(dp):
if f.startswith("."):
continue
try:
out[f"{d}/{f}"] = sha256_hex(self.r.read_bytes(f"{dp}/{f}"))
except OSError:
out[f"{d}/{f}"] = "unreadable"
return out
def profile_refresh(self, plan):
"""Load every AppArmor profile from a FRESH compile (the cache ignored and rewritten), once per kernel the box
starts on; then restart ONLY a failed service whose own program's profile had a stale cached copy.
Changes no package and takes no name from the plan. Measured on demo-hp with both guests running: 0.5 s, no
guest or container touched, the same profiles loaded in the same modes.
WHY NOT a "loaded hash changed" check: for a file that defines several profiles the kernel's hash differs
between a cache load and a fresh load of the SAME policy (17 of 119 on demo-hp), and one file (lxc-containers)
compiles to a different binary every time. The only reliable sign of a stale copy is the CACHED FILE changing
when it is rewritten — that is what the restart rule reads."""
r, dry = self.r, plan.get("dry_run") is True
res = {"dry_run": dry, "outcome": None, "rewritten": [], "restarted": [], "not_restarted": []}
self.report["profile_refresh"] = res
rc, out, _ = r.host(["uname", "-r"], 30)
kernel = out.strip() if rc == 0 else ""
try:
rec = json.loads(r.read_file(PROFILE_STATE))
rec = rec if isinstance(rec, dict) else {}
except (OSError, ValueError):
rec = {}
res["kernel"], res["last_refresh_kernel"], res["last_refresh_at"] = kernel or "unknown", rec.get("kernel"), rec.get("at")
if not kernel:
raise Refused("F2", "the running kernel could not be read")
if not r.isdir(PROFILE_DIR) or not r.isdir(PROFILE_CACHE):
res["outcome"] = "not_needed"
res["reason"] = "this box has no AppArmor profile folder or no profile cache"
return 0
needed = rec.get("kernel") != kernel
if not needed:
res["outcome"], res["reason"] = "not_needed", f"the profiles were refreshed on this kernel ({kernel}) at {rec.get('at')}"
return 0
if dry:
res["outcome"], res["would"] = "dry_run", "refresh"
return 0
before = self.profile_cache()
t0 = time.time()
rc, out, err = r.host(list(PROFILE_REFRESH_CMD), 600)
res["seconds"] = round(time.time() - t0, 1)
if rc != 0:
res["outcome"] = "failed"
self.report["failed"] = {"rc": rc, "step": "profile-refresh", "tail": (out + err).strip().splitlines()[-3:]}
r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)")
return 3
after = self.profile_cache()
res["rewritten"] = sorted(k for k in after if before.get(k) != after[k])
# --- the narrow restart rule: a FAILED service, whose ExecStart program IS the name of a profile whose cached
# copy was stale. Never "whatever failed"; never a unit named by the plan. ---
programs = {}
for key in res["rewritten"]:
fname = key.split("/", 1)[1]
rc_n, names, _ = r.host(["apparmor_parser", "-N", f"{PROFILE_DIR}/{fname}"], 60)
if rc_n != 0:
continue
for n in names.split():
if PROFILE_PATH_NAME_RE.match(n):
programs[n] = fname
if programs:
rc_u, units, _ = r.host(["systemctl", "list-units", "--type=service", "--state=failed", "--no-legend", "--plain"], 60)
for line in (units.splitlines() if rc_u == 0 else []):
unit = (line.split() or [""])[0]
if not UNIT_NAME_RE.match(unit):
continue
_, es, _ = r.host(["systemctl", "show", "-p", "ExecStart", "--value", unit], 30)
m = re.search(r"\bpath=(\S+)", es)
prog = m.group(1) if m else ""
if prog not in programs:
continue
if len(res["restarted"]) >= PROFILE_RESTART_MAX:
res["not_restarted"].append({"unit": unit, "program": prog, "reason": f"more than {PROFILE_RESTART_MAX} units match — look at the box"})
continue
rc_r, o2, e2 = r.host(["systemctl", "restart", unit], 120)
_, act, _ = r.host(["systemctl", "is-active", unit], 30)
res["restarted"].append({"unit": unit, "program": prog, "profile_file": programs[prog], "rc": rc_r,
"active": act.strip() == "active"})
r.log(f"os-apply: PROFILE REFRESH restarted {unit} (failed; its program {prog} had a stale cached profile) "
f"rc={rc_r} now={act.strip() or 'unknown'}")
r.put_file(PROFILE_STATE, (json.dumps({"kernel": kernel, "at": _iso(r.now()), "rewritten": res["rewritten"],
"restarted": [x["unit"] for x in res["restarted"]]}, indent=2, sort_keys=True) + "\n").encode(), 0o644)
res["outcome"] = "refreshed"
r.log(f"os-apply: PROFILE REFRESH DONE kernel={kernel} (last {rec.get('kernel') or 'never'}) seconds={res['seconds']} "
f"rewritten={len(res['rewritten'])} ({', '.join(res['rewritten'][:8])}) restarted={[x['unit'] for x in res['restarted']]}")
return 0
# ---------- R-528: the memory-kill check ----------
def oom_check(self):
"""Run one throwaway container over its memory cap in the guest and read what the engine says about it.
@@ -1473,6 +1749,11 @@ class Kernel:
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
kernel-cleanup remove old kernel images by the rule of `11` §5.11.1 (R-1368, decision 286). The plan names nothing
(C1); `dry_run: true` computes and reports, and changes nothing. Its own refusals: P1–P3, K1, N1,
S1, S2 (the design's table) and C1 (the plan names something). "Not run" (exit 0, nothing
changed): no ring-0 mark, no kernel step yet, the newest step did not end good, the box does not
run its newest kernel, inside a step's 20 h window.
kernel-status read only."""
def __init__(self, apply, plan):
@@ -1626,7 +1907,8 @@ class Kernel:
self.report["kernel"] = v
return 0
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel,
"kernel-cleanup": self.cleanup}[mode]
rc = fn()
self.report["kernel"] = self.view()
return rc
@@ -1862,6 +2144,221 @@ class Kernel:
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
return 0
# ---------- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----------
def kernel_images(self):
"""kver -> [installed image package names] (dpkg status ii only)."""
out = {}
for name in sorted(self.a.installed()):
m = KERNEL_IMAGE_RE.match(name)
if m:
out.setdefault(m.group(1), []).append(name)
return out
def kver_newest(self, kvers):
best = None
for k in kvers:
if best is None or self.a.dpkg_cmp(k[:-4], "gt", best[:-4]):
best = k
return best
def read_or_none(self, path):
try:
return self.r.read_file(path)
except OSError:
return None
def cleanup_sources(self):
"""Every place that names a kernel the lane still needs, read from the box ONCE: (kver, why) pairs. This is
the keep set's only input, and K1's only input — K1 reads it again by itself (cleanup_protected)."""
st = self.state()
cfg = self.grub_cfg()
dflt = self.default_kver(cfg)
m = PINNED_KVER_RE.search(self.read_or_none(KERNEL_DEFAULT_CFG) or "")
pinned = m.group(1) if m else None
flag = self.flag() or ""
m = ANY_KVER_RE.search(flag)
m2 = ANY_KVER_RE.search(self.read_or_none(KERNEL_PIN_FILE) or "")
src = [(self.running(), "running"),
(st.get("from"), "the kernel step's `from` (the last kernel that booted well; a self-revert boots it)"),
(st.get("to"), "the kernel step's `to`"),
(st.get("default_before"), "the kernel step's `default_before`"),
(dflt if KVER_RE.match(dflt) else None, "the default in grub.cfg"),
(pinned, "the pinned default file"),
(m.group(0) if m else None, "the one-shot flag"),
(m2.group(0) if m2 else None, "an operator's boot pin")]
return st, dflt, pinned, flag, [(k, why) for k, why in src if isinstance(k, str) and KVER_RE.match(k)]
def cleanup_keep(self, images, sources):
"""The keep set: every kernel a source names, and the newest installed. (cleanup() adds the next lower
installed kernel when no step record names the last good one.)"""
keep = {}
for k, why in sources:
keep.setdefault(k, []).append(why)
newest = self.kver_newest(images)
if newest:
keep.setdefault(newest, []).append("newest installed")
return keep
def cleanup_protected(self):
"""K1's own reading (`11` §5.11.1: "K1 is the rule's own check and must stay its own"): the kernels that must
never reach the purge line, read from the box AGAIN — not taken from the keep set that built the removal set."""
_, _, _, _, sources = self.cleanup_sources()
return {k: why for k, why in sources}
def cleanup_simulate(self, names):
"""apt's simulation of exactly `names`: (rc, removes, installs, last line)."""
rc, out, err = self.a.x(APT_ENV + ["apt-get", "-s", "-q", "purge"] + names)
removes = sorted({l.split()[1] for l in out.splitlines() if l.startswith(("Remv ", "Purg ")) and len(l.split()) > 1})
installs = sorted({l.split()[1] for l in out.splitlines() if l.startswith(("Inst ", "Conf ")) and len(l.split()) > 1})
return rc, removes, installs, ((err or out).strip().splitlines() or ["?"])[-1][:200]
def cleanup(self):
a = self.a
dry = self.plan.get("dry_run") is True
res = {"dry_run": dry, "outcome": None, "not_run": [], "refusals": [], "removed": [], "leftovers": []}
self.report["kernel_cleanup"] = res
not_run, refuse = res["not_run"], res["refusals"]
# --- may this box run the clean-up at all? (ring 0 only in this release; the mark is ROOT-owned) ---
try:
ring0 = a.load_trust().get("ring0_slow_lane") is True
except Refused as e:
ring0 = False
not_run.append({"code": "no-ring0-mark", "reason": f"no usable trust record: {e.reason}"})
if not ring0 and not not_run:
not_run.append({"code": "no-ring0-mark", "reason": "this box has no root-owned ring-0 mark — the clean-up runs on "
"ring 0 only in this release"})
if a.apt_lock_held():
raise Refused("R9", "another apt/dpkg holds the lock on the host")
if not dry and not not_run:
a.repair() # repair first (`11` §5.4.1): a purge a power cut interrupted is made whole, THEN the rule is computed
# --- read the box ---
images = self.kernel_images()
running = self.running()
newest = self.kver_newest(images)
raw_state = self.read_or_none(KERNEL_STATE)
st, dflt, pinned, flag, sources = self.cleanup_sources()
phase = st.get("phase", "none") if raw_state is not None else "none"
res.update(running=running or "unknown", newest=newest, phase=phase, default=dflt, pinned_default=pinned,
installed={k: images[k] for k in sorted(images, key=lambda v: [int(x) for x in re.findall(r"\d+", v)])})
# --- "when it may run" (decision 286): not run, nothing changed ---
if raw_state is None or not st:
not_run.append({"code": "no-step", "reason": "no kernel step yet (no state file) — nothing proves which kernel booted well"})
elif phase in KERNEL_ENDED_NOT_GOOD:
not_run.append({"code": "step-not-good", "reason": f"the newest kernel step ended {phase!r}, not good — every kernel stays"})
elif phase != "good":
refuse.append({"code": "P1", "reason": f"a kernel step is in phase {phase!r} — not ended (or not a phase this wrapper knows)"})
if running and newest and running in images and running != newest:
not_run.append({"code": "not-on-newest", "reason": f"the box runs {running}, not its newest installed kernel {newest} — every kernel stays"})
times = [x for x in (_parse_iso(st.get(k)) for k in ("staged_at", "rebooted_at", "result_at")) if x is not None]
if times and self.r.now() - max(times) < KERNEL_MIN_GAP:
not_run.append({"code": "window", "reason": "a kernel step staged, booted or ended within the last 20 hours — never in its night"})
# --- the refusal table ---
if flag:
refuse.append({"code": "P2", "reason": f"the one-shot flag names {flag!r}"})
if pinned is None or pinned != dflt or pinned not in images:
refuse.append({"code": "P3", "reason": f"the pinned default {pinned!r} and grub.cfg's default {dflt!r} do not name the same installed kernel"})
if not running or running not in images:
refuse.append({"code": "K1", "reason": f"the running kernel {running or 'unknown'!r} is not an installed kernel image package"})
# --- the keep set and the removal set (exact package names) ---
keep = self.cleanup_keep(images, sources)
if running and not (st.get("to") == running and st.get("from") in images):
lower = [k for k in images if a.dpkg_cmp(k[:-4], "lt", running[:-4])] if running in images else []
lg = self.kver_newest(lower)
if lg:
keep.setdefault(lg, []).append("the next lower installed kernel (no step record names the last good one)")
removal = sorted(n for k in images if k not in keep for n in images[k])
res["kept"] = {k: keep[k] for k in sorted(keep) if k in images}
res["removal"] = removal
# K1 — its own reading of the box, against the removal set
by_name = {n: k for k, names in images.items() for n in names}
protected = self.cleanup_protected()
for n in removal:
if n not in by_name:
refuse.append({"code": "N1", "reason": f"{n} is not an installed kernel image package"})
elif by_name[n] in protected or by_name[n] == newest:
refuse.append({"code": "K1", "reason": f"{n} is a keep-kernel ({protected.get(by_name[n], 'newest installed')})"})
# S1 / S2 — apt's simulation of exactly that set
if removal and not [x for x in refuse if x["code"] in ("K1", "N1")]:
rc, removes, installs, tail = self.cleanup_simulate(removal)
res["simulation"] = {"rc": rc, "removes": removes, "installs": installs}
if rc != 0:
refuse.append({"code": "S1", "reason": "the simulation failed: " + tail})
if installs:
refuse.append({"code": "S1", "reason": f"the simulation would install or configure {', '.join(installs[:6])}"})
extra = [p for p in removes if p not in removal]
if extra:
refuse.append({"code": "S2", "reason": f"the simulation would also remove {', '.join(extra[:8])} — not in the removal set"})
missing = [p for p in removal if p not in removes]
if missing and rc == 0:
refuse.append({"code": "S1", "reason": f"the simulation does not remove {', '.join(missing[:6])}"})
self.log(f"os-apply: KERNEL CLEANUP {'DRY RUN ' if dry else ''}running={running or 'unknown'} newest={newest} phase={phase} "
f"installed={len(images)} keep={len(res['kept'])} removal={len(removal)} not_run={[x['code'] for x in not_run]} "
f"refusals={[x['code'] for x in refuse]}")
# --- the verdict ---
if dry:
res["outcome"] = "dry_run"
res["would"] = "not_run" if not_run else "refuse" if refuse else "remove" if removal else "nothing"
return 0
if not_run:
res["outcome"] = "not_run"
return 0
if refuse:
raise Refused(refuse[0]["code"], "; ".join(f"{x['code']} {x['reason']}" for x in refuse)[:600])
if not removal:
res["outcome"] = "nothing"
self.log("os-apply: KERNEL CLEANUP nothing to remove")
return 0
# --- the purge: ONE apt-get purge of the exact names; then prove the boot setup did not move ---
gens_before = {p: (self.r.lexists(p), self.read_or_none(p)) for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES)}
pin_before = self.read_or_none(KERNEL_DEFAULT_CFG)
free_before = a.free_bytes()
t0 = time.time()
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q", "purge"] + removal)
res["seconds"] = round(time.time() - t0, 1)
if rc != 0:
_, aud, _ = a.x(["dpkg", "--audit"])
res["outcome"] = "failed"
self.report["failed"] = {"rc": rc, "step": "purge", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
"tail": (out + err).strip().splitlines()[-3:]}
self.log(f"os-apply: FAILED rc={rc} step=purge (the next run repairs dpkg first, then computes the rule again)")
return 3
after = self.kernel_images()
after_names = {n for names in after.values() for n in names}
res["removed"] = [n for n in removal if n not in after_names]
free_after = a.free_bytes()
res["bytes_gained"] = free_after - free_before if free_before >= 0 and free_after >= 0 else None
problems = []
if len(res["removed"]) != len(removal):
problems.append(f"still installed after the purge: {', '.join(n for n in removal if n in after_names)}")
gone = [k for k in res["kept"] if k not in after]
if gone:
problems.append(f"a KEPT kernel is no longer installed: {', '.join(gone)}")
got = self.default_kver(self.grub_cfg())
if got != dflt:
problems.append(f"the default in grub.cfg moved from {dflt} to {got}")
if self.read_or_none(KERNEL_DEFAULT_CFG) != pin_before:
problems.append("the pinned default file changed")
if self.read_or_none(KERNEL_STATE) != raw_state:
problems.append("the kernel step's state file changed")
for p, (was, body) in gens_before.items():
if self.r.lexists(p) != was or self.read_or_none(p) != body:
problems.append(f"{p} changed")
for k in sorted(set(by_name[n] for n in res["removed"])):
if k in after:
continue # another image package of the same kernel is still installed
res["leftovers"] += [p.format(v=k) for p in KERNEL_LEFTOVER_PATHS if self.r.lexists(p.format(v=k))]
if problems:
# reported LOUDLY, never repaired by hand here (`11` §5.11.1): the operator looks at the box
res["outcome"] = "failed"
self.report["failed"] = {"rc": 3, "step": "verify", "reason": "; ".join(problems)[:600]}
self.log("os-apply: FAILED rc=3 step=verify after the kernel purge — " + "; ".join(problems))
return 3
res["outcome"] = "removed"
self.log(f"os-apply: KERNEL CLEANUP DONE removed={len(res['removed'])} ({', '.join(res['removed'])}) "
f"kept={', '.join(res['kept'])} gained={res['bytes_gained']} B seconds={res['seconds']} "
f"leftovers={len(res['leftovers'])}; default {dflt} (proved from grub.cfg)")
return 0
class Bundle:
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
@@ -1873,6 +2370,7 @@ class Bundle:
self.a, self.r = apply, apply.r
self.report = apply.report
self.sshd_dropin_not_in_force = False
self.sshd_step_undone = None # X4 (R-1257): why the SSH step alone was put back ("" = not), the rest stays
# ---------- reading and checking ----------
def load_file(self, path):
@@ -1982,14 +2480,40 @@ class Bundle:
return b"", None, None # a symlink or a directory: always replaced by the real file
return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid
def install_mode(self):
"""The box's ROOT-OWNED install record's mode ("appliance", "byo", …), or None when it proves nothing (absent,
agent-writable, unreadable) — the same proof check_appliance asks for the host layer (R12)."""
try:
st = self.r.stat(INSTALL_STATE)
if st.st_uid != 0 or (st.st_mode & 0o022):
return None
mode = json.loads(self.r.read_file(INSTALL_STATE)).get("mode")
except (OSError, ValueError, AttributeError):
return None
return mode if isinstance(mode, str) and mode else None
def plan_writes(self, files):
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt)."""
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt; an
appliance-only file on a box that is not one) / remove (an appliance-only file WE wrote, on a BYO box)."""
oob = self.r.isdir(OOB_DIR)
install_mode = self.install_mode()
plan = []
for dest, content, mode, check, policy in files:
if policy == "oob" and not oob:
plan.append((dest, content, mode, "skipped", None, None))
continue
if policy == "appliance" and install_mode != "appliance":
old, old_mode, _ = self.live(dest)
ours = old is not None and old.startswith(SSHD_DROPIN_HEADER)
if install_mode == "byo" and ours:
self.r.log(f"os-apply: BUNDLE {dest}: a BYO host keeps its own settings (D43) — removing the copy we wrote")
plan.append((dest, content, mode, "remove", old, old_mode))
else:
why = "a BYO host keeps its own settings (D43)" if install_mode == "byo" else \
f"the install record proves no appliance (mode={install_mode!r}) — left as it is"
self.r.log(f"os-apply: BUNDLE {dest} skipped: {why}")
plan.append((dest, content, mode, "skipped", None, None))
continue
old, old_mode, old_uid = self.live(dest)
if policy == "if-absent" and old is not None:
plan.append((dest, content, mode, "kept", old, old_mode))
@@ -2027,11 +2551,47 @@ class Bundle:
ver, files = self.parse(data, sha)
return self.install(ver, sha, files, "installer", False)
def policy_pass(self):
"""R-1285 (D43 = A): the appliance-only files' policy, applied by THIS wrapper. A bundle that brings a new wrapper
is planned by the OLD one, which did not know the policy — so on a BYO host upgraded from 0.164.0 our SSH drop-in
stayed until the next bundle. The agent runs this pass right after every bundle install (the wrapper on disk is
then the new one). It only REMOVES a file WE wrote (our header) from a box whose root-owned install record says
byo, keeps the previous copy beside the bundle's, and reloads sshd; it writes nothing, so it needs no signature.
A failing `sshd -t` puts the file back (R-1257 X4's rule). Report key `bundle_policy`."""
log = self.r.log
install_mode = self.install_mode()
removed = []
if install_mode == "byo":
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-policy")
for dest, _src, _mode, _check, policy in BUNDLE_FILES:
if policy != "appliance":
continue
old, old_mode, _ = self.live(dest)
if old is None or not old.startswith(SSHD_DROPIN_HEADER):
continue
self.r.put_file(prev + dest, old, 0o600)
self.r.remove(dest)
removed.append((dest, old, old_mode))
log(f"os-apply: BUNDLE POLICY REMOVED {dest} (ours, on a BYO host — D43, R-1285)")
if any(d == SSHD_DROPIN for d, _, _ in removed):
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
if rc != 0:
for d, old, old_mode in removed:
self.r.put_file(d, old, old_mode)
log(f"os-apply: BUNDLE POLICY WARNING sshd -t failed without our drop-in (rc={rc}) — put back")
removed = []
else:
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
self.report["bundle_policy"] = {"install_mode": install_mode, "removed": [d for d, _, _ in removed]}
log(f"os-apply: BUNDLE POLICY DONE install_mode={install_mode} removed={len(removed)}")
return 0
# ---------- install, self-check, undo ----------
def install(self, ver, sha, files, authority, bootstrap):
log = self.r.log
plan = self.plan_writes(files)
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped")}
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped", "remove")}
log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} "
f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}")
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
@@ -2039,10 +2599,17 @@ class Bundle:
done = [] # (dest, old bytes or None, old mode)
rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev,
"written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"],
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"]}
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"],
"removed": [p[0] for p in plan if p[3] == "remove"]}
self.report["bundle"] = rep
try:
for dest, content, mode, action, old, old_mode in plan:
if action == "remove": # D43 = A: our copy off a BYO host (the previous copy kept, the undo restores it)
self.r.put_file(prev + dest, old, 0o600)
done.append((dest, old, old_mode))
self.r.remove(dest)
log(f"os-apply: BUNDLE REMOVED {dest} (ours, on a BYO host — D43)")
continue
if action != "write":
continue
if old is not None:
@@ -2062,22 +2629,56 @@ class Bundle:
rep["signers_created"] = False
if self.sshd_dropin_not_in_force:
rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
if self.sshd_step_undone:
rep["sshd_step_undone"] = self.sshd_step_undone[:300] # X4: only the SSH step was put back
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
rep["signers_created"] = True
log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}")
record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority,
"installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())),
"files": {p[0]: (sha256_hex(p[1]) if p[3] in ("write", "same") else
(sha256_hex(p[4]) if p[3] == "kept" else "skipped")) for p in plan}}
"files": {p[0]: self.record_entry(p) for p in plan}}
self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644)
self.prune()
log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} "
f"self-check=ok signers-created={rep['signers_created']}")
return 0
def record_entry(self, p):
"""The record's word for one plan entry: the sha it installed, the sha it kept, or what it did instead."""
dest, content, _, action, old, _ = p
if dest == SSHD_DROPIN and self.sshd_step_undone:
return sha256_hex(old) if old is not None else "absent" # X4: the SSH step was put back
if action in ("write", "same"):
return sha256_hex(content)
if action == "kept":
return sha256_hex(old)
return "removed" if action == "remove" else "skipped"
def lan_addresses(self):
"""X3 (R-1257): the host's own global IPv4 addresses — the ones a LAN client reaches sshd on."""
rc, out, _ = self.r.host(["ip", "-4", "-o", "addr", "show", "scope", "global"], 30)
if rc != 0:
return []
return sorted(set(re.findall(r"\binet (\d+\.\d+\.\d+\.\d+)/", out)))
def undo_sshd_step(self, plan, why):
"""X4 (R-1257): `sshd -t` refused the host's whole config with our step in it — put back ONLY that file (the
previous copy, or none) and keep the rest of the bundle. sshd was not reloaded, so it never ran the refused state."""
for dest, _, _, action, old, old_mode in plan:
if dest != SSHD_DROPIN or action not in ("write", "remove"):
continue
if old is None:
self.r.remove(dest)
else:
self.r.put_file(dest, old, old_mode if old_mode is not None else 0o644)
self.sshd_step_undone = why
self.r.log(f"os-apply: BUNDLE WARNING sshd -t refused the host's config with the SSH step — ONLY that step is put "
f"back, the rest of the bundle stays: {why[:200]} (R-1257)")
def after_install(self, plan):
written = {p[0] for p in plan if p[3] == "write"}
removed = {p[0] for p in plan if p[3] == "remove"}
dests = {p[0] for p in plan if p[3] != "skipped"}
if any(d.startswith("/etc/systemd/system/") for d in written):
self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload")
@@ -2098,16 +2699,29 @@ class Bundle:
# felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
# sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
# — that is said loudly in the log and the report, never silently recorded as done (A2).
if SSHD_DROPIN in written:
self.must(["/usr/sbin/sshd", "-t"], "sshd -t (the host's whole sshd config)")
# R-1257 X4: a failing `sshd -t` puts back ONLY the SSH step (undo_sshd_step); it used to refuse and undo the
# whole bundle. X3: the in-force check asks sshd about each of the host's own LAN addresses — a probe from a
# documentation address (192.0.2.1) read a `Match Address <LAN>` block that re-allows passwords as in force.
if SSHD_DROPIN in written or SSHD_DROPIN in removed:
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
if rc != 0:
self.undo_sshd_step(plan, f"sshd -t failed (rc={rc}): {(out + err).strip()[-200:]}")
return
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", "user=root,host=felhom-check,addr=192.0.2.1"], 60)
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
if rc != 0 or eff != ["without-password"]:
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force: sshd -T says "
f"permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier drop-in "
f"or a Match block (R-982)")
self.sshd_dropin_not_in_force = True
if SSHD_DROPIN in written:
addrs = self.lan_addresses()
if not addrs:
self.r.log("os-apply: BUNDLE WARNING no global IPv4 address read for the sshd in-force check — "
"probing 192.0.2.1 only (R-1257)")
addrs = ["192.0.2.1"]
for addr in addrs:
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", f"user=root,host=felhom-check,addr={addr}"], 60)
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
if rc != 0 or eff != ["without-password"]:
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force for {addr}: sshd -T "
f"says permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier "
f"drop-in or a Match block (R-982, R-1257)")
self.sshd_dropin_not_in_force = True
def must(self, argv, what):
rc, out, err = self.r.host(argv, 120)
+154 -1
View File
@@ -45,8 +45,10 @@ class St:
class Box:
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
def __init__(self, bundle_bytes, signed, oob=False, signers=True):
def __init__(self, bundle_bytes, signed, oob=False, signers=True, install_mode="appliance"):
self.files, self.modes, self.uids = {}, {}, {}
if install_mode is not None: # the installer's ROOT-OWNED record (D43 = A reads it for the sshd drop-in)
self.put(osapply.INSTALL_STATE, json.dumps({"completed": [], "mode": install_mode}).encode(), 0o644)
self.dirs = {osapply.OOB_DIR} if oob else set()
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
if signers:
@@ -513,6 +515,157 @@ class RootKeyOnly(unittest.TestCase):
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
class ByoKeepsItsSSH(unittest.TestCase):
"""D43 = A (operator 2026-10-24, R-1257): the root-key-only drop-in goes ONLY to a box whose root-owned install record
says appliance; on a BYO host nothing is written and a copy WE wrote is removed; a box with no record is left as it
is. X3: the in-force check asks sshd about each of the host's own LAN addresses. X4: a failing `sshd -t` puts back
only the SSH step. RED-PROOF (2026-10-24, felhom-agent f5c3d77 + these tests): with the drop-in's policy back to
`replace`, test_a_byo_host_gets_no_dropin and test_our_copy_comes_off_a_byo_host fail; with the probe back to
192.0.2.1 only, test_a_match_block_for_the_lan_is_not_in_force fails; with `sshd -t` back to `must`,
test_a_failing_sshd_t_undoes_only_the_ssh_step fails (rc 3, the whole bundle undone)."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
OURS = (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes()
def test_a_byo_host_gets_no_dropin(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertNotIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertIn(self.DEST, rep["bundle"]["skipped"])
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
self.assertEqual(rec["files"][self.DEST], "skipped")
def test_our_copy_comes_off_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, self.OURS, 0o644) # an earlier bundle wrote it (before D43)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle"]["removed"], [self.DEST])
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files),
"the previous copy must be kept for the undo")
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], "removed")
def test_the_owners_own_file_stays_on_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, b"# my own\nPermitRootLogin yes\n", 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files[self.DEST], b"# my own\nPermitRootLogin yes\n")
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_box_with_no_record_is_left_as_it_is(self):
for setup in ("none", "agent-writable"):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode=None if setup == "none" else "byo")
if setup == "agent-writable":
box.modes[osapply.INSTALL_STATE] = 0o666 # proves nothing
box.put(self.DEST, self.OURS, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS, setup)
self.assertIn(self.DEST, rep["bundle"]["skipped"], setup)
def _fake(self, box, addrs=("192.168.0.50",), match=None, sshd_t_rc=0):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:3] == ["ip", "-4", "-o"]:
box.calls.append(argv)
return 0, "".join(f"2: vmbr0 inet {a}/24 brd x scope global vmbr0\\ valid_lft forever\n" for a in addrs), ""
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
val = "yes" if match and f"addr={match}" in argv[-1] else "without-password"
return 0, f"port 22\npermitrootlogin {val}\n", ""
if argv == ["/usr/sbin/sshd", "-t"]:
box.calls.append(argv)
return (sshd_t_rc, "", "/etc/ssh/sshd_config line 9: Bad configuration option") if sshd_t_rc else (0, "", "")
return orig(argv, timeout, stdin)
box.host = host
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
data, sha = real_bundle()
box = Box(data, None, install_mode=mode)
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
body = content if content is not None else self.OURS
box.put(self.DEST, body, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), body, mode)
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644)
self._fake(box, sshd_t_rc=1)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS)
self.assertEqual(rep["bundle_policy"]["removed"], [])
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_match_block_for_the_lan_is_not_in_force(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"), match="192.168.0.50")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
probed = [c[-1] for c in box.calls if c[:2] == ["/usr/sbin/sshd", "-T"]]
self.assertTrue(any("addr=192.168.0.50" in p for p in probed) and any("addr=10.77.0.4" in p for p in probed), probed)
self.assertFalse(any("addr=192.0.2.1" in p for p in probed), probed)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force for 192.168.0.50" in l for l in box.logs), box.logs)
def test_the_lan_check_passes_when_every_address_is_key_only(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_failing_sshd_t_undoes_only_the_ssh_step(self):
for before in (None, b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982 old\nPermitRootLogin prohibit-password\n"):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
if before is not None:
box.put(self.DEST, before, 0o644)
self._fake(box, sshd_t_rc=255)
rc, rep = run(box)
self.assertEqual(rc, 0, rep) # the bundle stays
self.assertEqual(box.files.get(self.DEST), before, "only the SSH step is put back")
self.assertIn("/usr/local/sbin/felhom-os-apply", box.files, "the rest of the bundle stays")
self.assertIn("sshd -t failed", rep["bundle"].get("sshd_step_undone", ""), rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, "sshd never ran the refused state")
want = "absent" if before is None else hashlib.sha256(before).hexdigest()
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], want)
class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle()
+868
View File
@@ -833,6 +833,14 @@ class DockerLane(unittest.TestCase):
def test_expired_signed_job_is_refused(self):
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
def test_a_window_longer_than_24h_is_refused(self):
# R-1280. RED-PROOF: drop the MAX_WINDOW_S check → the job (current, signed, 24 h + 1 s) runs.
self.refused(docker_fake(signed=signed_job(issued="2026-10-03T11:59:59Z", expires="2026-10-04T12:00:00Z")), "R3")
def test_a_window_of_24h_is_accepted(self):
rc, rep = run(docker_fake(signed=signed_job(issued="2026-10-03T12:30:00Z", expires="2026-10-04T12:30:00Z")))
self.assertNotEqual((rep.get("refused") or {}).get("code"), "R3", rep)
def test_replayed_signed_job_is_refused(self):
f = docker_fake(signed=signed_job())
f.nonces = {"n1": f.clock + 600}
@@ -1486,6 +1494,24 @@ class PVELane(unittest.TestCase):
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
# 2026-11-05..07, demo-felhom (EFI): proxmox-grub 2.12-9+pmx2 -> pmx4 depends on grub-efi-amd64 of the same version;
# HOST_SLOW_RE's `grub` is anchored at the start, so proxmox-grub fell into the pve plan, the simulation pulled
# grub-efi-amd64 and the whole pve step was refused R6 every night. It is a boot-chain package: left out.
# RED-PROOF: drop proxmox-grub from HOST_SLOW_RE -> it is in the plan -> this test fails.
def test_ring0_pending_pve_leaves_proxmox_grub_out(self):
f = pve_fake(ring0=True)
f.plan["select"], f.plan["packages"] = "pending-pve", []
f.installed.update({"proxmox-grub": "2.12-9+pmx2"})
f.pending_sim = [
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
"Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
self.assertEqual(f.installed["proxmox-grub"], "2.12-9+pmx2")
def test_select_pending_pve_needs_the_pve_layer(self):
f = Fake()
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
@@ -2005,5 +2031,847 @@ class KernelLane(unittest.TestCase):
self.assertIn("one-shot", h["kernel_next_boot_source"])
# ---------- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----------
K = ["7.0.2-6-pve", "7.0.14-20-pve", "7.0.14-22-pve", "7.0.14-26-pve", "7.0.14-28-pve"] # five installed kernels
C_FROM, C_RUN = "7.0.14-26-pve", "7.0.14-28-pve"
GB = 1024 ** 3
def img(k):
return f"proxmox-kernel-{k}-signed"
class CFake(KFake):
"""A ring-0 appliance after a good kernel step 26 -> 28: five kernels installed, the box runs its newest, the
default (grub.cfg and the pinned file) is the running kernel, the flag is empty. `apt-get purge` is emulated like
the real one: the image leaves dpkg and /boot, and the kernel's postrm runs update-grub."""
def __init__(self):
super().__init__()
self.running = C_RUN
self.boot = set(K)
self.modules = set(K) # /lib/modules/<kver>
for n in [x for x in self.installed if osapply.KERNEL_IMAGE_RE.match(x)]:
del self.installed[n]
for k in K:
self.installed[img(k)] = k[:-4]
self.installed["proxmox-kernel-7.0"] = C_RUN[:-4]
self.env = {"felhom_next": ""}
self.tree[osapply.KERNEL_DEFAULT_CFG] = (f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-{C_RUN}-advanced-{U}"\n')
self.state = {"phase": "good", "step_id": "ring0-x", "from": C_FROM, "to": C_RUN, "default_before": C_FROM,
"vmid": 9201, "staged_at": "2026-10-01T02:50:00Z", "rebooted_at": "2026-10-01T02:51:00Z",
"result_at": "2026-10-01T03:10:00Z", "self_revert_used": False}
self.write_state()
self.cfg = self.render()
self.purges, self.purge_rc, self.purge_moves_default, self.purge_extra_sim = [], 0, False, []
self.plan = {"release_id": "kernel", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "kernel-cleanup",
"packages": []}
def write_state(self):
if self.state is None:
self.tree.pop(osapply.KERNEL_STATE, None)
else:
self.tree[osapply.KERNEL_STATE] = json.dumps(self.state)
def lexists(self, p):
m = re.match(r"^/lib/modules/(.+)$", p)
if m:
return m.group(1) in self.modules
return super().lexists(p)
def changing(self):
"""Every command that changes the box (the purge, a repair, update-grub, a flag write, a file write)."""
out = []
for c in self.calls:
a = [x for x in c[-1] if not re.match(r"^[A-Z_]+=", x) and x != "env"]
if (a[0] == "apt-get" and "-s" not in a and "--print-uris" not in a) or a[0] in ("update-grub", "dpkg-reconfigure") \
or (a[0] == "dpkg" and a[1] == "--configure") or (a[0] == "grub-editenv" and a[2] != "list") \
or (a[0] == "systemctl" and a[1] == "reboot"):
out.append(a)
return out
def emulate(self, argv):
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
if a[0] == "apt-get" and "purge" in a and "-s" not in a:
names = [x for x in a[a.index("purge") + 1:] if not x.startswith("-")]
self.purges.append(names)
if self.purge_rc:
self.dpkg_journal = ["0001"] # a purge cut in the middle leaves dpkg's update journal
return self.purge_rc, "", "E: Sub-process /usr/bin/dpkg returned an error code (1)"
for n in names:
self.installed.pop(n, None)
m = osapply.KERNEL_IMAGE_RE.match(n)
if m:
self.boot.discard(m.group(1))
self.free += GB
if self.purge_moves_default:
self.tree.pop(osapply.KERNEL_DEFAULT_CFG, None)
self.cfg = self.render() # the kernel's postrm runs update-grub (zz-update-grub)
return 0, "Removing proxmox-kernel ...\n", ""
return super().emulate(argv)
def sim(self, a):
if "purge" in a:
names = [x for x in a[a.index("purge") + 1:] if not x.startswith("-")]
if getattr(self, "purge_sim_rc", 0):
return self.purge_sim_rc, "", "E: Unable to locate package"
out = "".join(f"Purg {n} [{self.installed[n]}]\n" for n in names if n in self.installed)
return 0, out + "".join(l + "\n" for l in self.purge_extra_sim), ""
return super().sim(a)
def cfake(**kw):
f = CFake()
for k, v in kw.items():
setattr(f, k, v)
return f
def cstate(f, **kw):
f.state.update(kw)
f.write_state()
return f
class KernelCleanup(unittest.TestCase):
"""§5 of the release-2a brief: one test per acceptance scenario, each on the stored effect (the exact apt line or
the absence of any changing command, the report, the exit code). Red-proofs (the four mutations the brief names):
felhom.eu audits/release-2a-2026-11-14/A/redproof.txt."""
REMOVE = [img(k) for k in ("7.0.14-20-pve", "7.0.14-22-pve", "7.0.2-6-pve")] # sorted by name
def nothing_changed(self, f):
self.assertEqual(f.purges, [], "no purge")
self.assertEqual(f.changing(), [], "no command that changes the box")
self.assertEqual(sorted(f.boot), sorted(K), "every kernel is still there")
def refused(self, f, code):
before = (f.tree.get(osapply.KERNEL_STATE), f.tree.get(osapply.KERNEL_DEFAULT_CFG), f.cfg, dict(f.env or {}))
rc, rep = run(f)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
self.nothing_changed(f)
self.assertEqual((f.tree.get(osapply.KERNEL_STATE), f.tree.get(osapply.KERNEL_DEFAULT_CFG), f.cfg, dict(f.env or {})), before)
return rep
def not_run(self, f, code):
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
c = rep["kernel_cleanup"]
self.assertEqual(c["outcome"], "not_run", rep)
self.assertIn(code, [x["code"] for x in c["not_run"]], rep)
self.assertIsNone(rep.get("refused"))
self.nothing_changed(f)
return rep
# --- scenario 1: the normal night ---
def test_s1_the_normal_night_purges_exactly_the_old_kernels_in_one_command(self):
f = cfake()
state0, pin0 = f.tree[osapply.KERNEL_STATE], f.tree[osapply.KERNEL_DEFAULT_CFG]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.purges, [self.REMOVE], "ONE apt-get purge of the exact names")
purge = [c[-1] for c in f.calls if "purge" in c[-1] and "-s" not in c[-1]]
self.assertEqual(len(purge), 1)
self.assertEqual(purge[0][purge[0].index("apt-get"):], ["apt-get", "-y", "-q", "purge"] + self.REMOVE)
self.assertNotIn("autoremove", " ".join(" ".join(c[-1]) for c in f.calls), "never apt's own clean-up")
c = rep["kernel_cleanup"]
self.assertEqual(c["outcome"], "removed")
self.assertEqual(c["removed"], self.REMOVE)
self.assertEqual(sorted(c["kept"]), sorted([C_FROM, C_RUN]))
self.assertEqual(c["bytes_gained"], 3 * GB)
self.assertEqual(sorted(f.boot), sorted([C_FROM, C_RUN]))
# the default in grub.cfg, the pinned default file and the step's state file are unchanged
self.assertEqual(osapply.Kernel.default_kver(f.cfg), C_RUN)
self.assertEqual(f.tree[osapply.KERNEL_DEFAULT_CFG], pin0)
self.assertEqual(f.tree[osapply.KERNEL_STATE], state0)
self.assertEqual(f.env, {"felhom_next": ""})
self.assertEqual(f.reboots, [], "the clean-up never reboots")
self.assertEqual(f.update_grubs, 0, "the wrapper itself runs no update-grub (the kernel's postrm did)")
# leftovers are reported, never deleted
self.assertEqual(c["leftovers"], [f"/lib/modules/{k}" for k in ("7.0.14-20-pve", "7.0.14-22-pve", "7.0.2-6-pve")])
self.assertEqual(f.modules, set(K), "a left-over folder is reported, not deleted")
# --- scenario 2: the second night ---
def test_s2_the_second_night_removes_nothing(self):
f = cfake()
self.assertEqual(run(f)[0], 0)
f.calls.clear()
f.purges.clear()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["kernel_cleanup"]["outcome"], "nothing")
self.assertEqual(rep["kernel_cleanup"]["removal"], [])
self.assertEqual(f.purges, [])
self.assertEqual(f.changing(), [], "no apt call that changes anything")
self.assertFalse([c for c in f.calls if "purge" in c[-1]], "not even a simulation: there is nothing to simulate")
# --- scenario 3: a step is not over (P1) ---
def test_s3_a_step_that_is_not_over_refuses(self):
for phase in ("staged", "oneshot", "judging", "reverting", "some-new-phase"):
f = cstate(cfake(), phase=phase)
rep = self.refused(f, "P1")
self.assertIn(phase, rep["refused"]["reason"])
# --- scenario 4: the flag (P2); the pinned default against grub.cfg (P3) ---
def test_s4_the_flag_names_a_kernel_refuses(self):
f = cfake(env={"felhom_next": "7.0.14-26-pve"})
self.refused(f, "P2")
def test_s4_pinned_default_and_grub_cfg_disagree_refuses(self):
f = cfake()
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-{C_FROM}-advanced-{U}"\n'
self.refused(f, "P3") # grub.cfg still says 28, the file says 26
g = cfake()
del g.tree[osapply.KERNEL_DEFAULT_CFG]
self.refused(g, "P3") # no pinned file at all
h = cfake() # both name a kernel that is not installed
h.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-7.0.14-99-pve-advanced-{U}"\n'
h.cfg = h.cfg.replace(f"gnulinux-{C_RUN}-advanced-{U}\"", f"gnulinux-7.0.14-99-pve-advanced-{U}\"", 1)
self.refused(h, "P3")
# --- scenario 5: the box fell back / does not run its newest kernel ---
def test_s5_a_box_that_fell_back_keeps_every_kernel(self):
for phase in ("fell_back", "self_reverted", "revert_failed", "cancelled"):
self.not_run(cstate(cfake(running=C_FROM), phase=phase), "step-not-good")
def test_s5_a_box_that_does_not_run_its_newest_kernel_keeps_every_kernel(self):
f = cfake(running=C_FROM) # the step says good, but the box runs the older kernel
self.not_run(f, "not-on-newest")
# --- scenario 6: the wrong case (K1) — one case per kept-kernel kind ---
def k1(self, kind, prepare):
"""The keep computation is BROKEN on purpose (it forgets `kind`); K1 reads the box by itself and must refuse
before any apt call."""
f = cfake()
victim = prepare(f)
real = osapply.Kernel.cleanup_keep
def broken(self_, images, sources):
keep = real(self_, images, [(k, why) for k, why in sources if k != victim])
keep.pop(victim, None) # … also when it is the newest installed
return keep
osapply.Kernel.cleanup_keep = broken
try:
rc, rep = run(f)
finally:
osapply.Kernel.cleanup_keep = real
self.assertEqual(rc, 2, f"{kind}: {rep}")
self.assertEqual(rep["refused"]["code"], "K1", f"{kind}: {rep}")
self.assertIn(img(victim), rep["refused"]["reason"])
self.assertFalse([c for c in f.calls if "purge" in c[-1]], f"{kind}: refused BEFORE any apt call, also the simulation")
self.nothing_changed(f)
def test_s6_k1_the_running_kernel(self):
self.k1("running", lambda f: C_RUN)
def test_s6_k1_the_steps_from(self):
self.k1("from", lambda f: C_FROM)
def test_s6_k1_the_steps_to(self):
def prep(f): # a step whose `to` is not the running kernel any more
cstate(f, to="7.0.14-22-pve")
return "7.0.14-22-pve"
self.k1("to", prep)
def test_s6_k1_default_before(self):
def prep(f):
cstate(f, default_before="7.0.14-20-pve")
return "7.0.14-20-pve"
self.k1("default_before", prep)
def test_s6_k1_the_grub_cfg_default_and_the_pinned_file(self):
# the default (both places) names 26; the box runs 28 after a hand boot. The keep computation forgets 26.
def prep(f):
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-7.0.14-22-pve-advanced-{U}"\n'
f.cfg = f.render()
return "7.0.14-22-pve"
self.k1("default", prep)
def test_s6_k1_the_flag(self):
# a flag is a refusal by itself (P2, first); with the keep computation broken, K1 is in the SAME refusal
f = cfake(env={"felhom_next": "7.0.14-20-pve"})
real = osapply.Kernel.cleanup_keep
osapply.Kernel.cleanup_keep = lambda s_, images, sources: real(s_, images, [(k, w) for k, w in sources if k != "7.0.14-20-pve"])
try:
rc, rep = run(f)
finally:
osapply.Kernel.cleanup_keep = real
self.assertEqual((rc, rep["refused"]["code"]), (2, "P2"), rep)
self.assertIn("K1 " + img("7.0.14-20-pve") + " is a keep-kernel (the one-shot flag)", rep["refused"]["reason"])
self.assertFalse([c for c in f.calls if "purge" in c[-1]])
self.nothing_changed(f)
def test_s6_k1_an_operators_boot_pin(self):
def prep(f):
f.tree[osapply.KERNEL_PIN_FILE] = "7.0.14-20-pve\n"
return "7.0.14-20-pve"
self.k1("boot pin", prep)
def test_s6_every_kept_kind_is_in_the_keep_set_and_never_in_the_removal_set(self):
f = cfake()
cstate(f, default_before="7.0.14-20-pve")
f.tree[osapply.KERNEL_PIN_FILE] = "7.0.14-22-pve\n"
f.plan["dry_run"] = True
rc, rep = run(f)
c = rep["kernel_cleanup"]
self.assertEqual(sorted(c["kept"]), sorted(["7.0.14-20-pve", "7.0.14-22-pve", C_FROM, C_RUN]))
self.assertEqual(c["removal"], [img("7.0.2-6-pve")])
def test_s6_the_running_kernel_is_not_an_installed_package(self):
f = cfake()
del f.installed[img(C_RUN)] # the box runs a kernel dpkg does not know (an unsigned hand install, say)
f.installed["proxmox-kernel-7.0.14-28-pve-other"] = "7.0.14-28"
self.refused(f, "P3") # P3 is first (the default names a kernel that is not an installed image) …
rc, rep = run(f)
self.assertIn("K1", rep["refused"]["reason"], "… and K1 is in the same refusal")
def test_no_step_record_keeps_the_next_lower_kernel(self):
f = cstate(cfake(), **{"from": None, "to": None, "default_before": None})
f.plan["dry_run"] = True
rc, rep = run(f)
self.assertIn("7.0.14-26-pve", rep["kernel_cleanup"]["kept"])
self.assertIn("next lower", " ".join(rep["kernel_cleanup"]["kept"]["7.0.14-26-pve"]))
# --- scenario 7: apt would do more (S1, S2) ---
def test_s7_the_simulation_would_remove_something_else(self):
f = cfake(purge_extra_sim=["Purg proxmox-kernel-7.0 [7.0.14-28]", "Remv proxmox-ve [9.2.0]"])
rep = self.refused(f, "S2")
self.assertIn("proxmox-ve", rep["refused"]["reason"])
def test_s7_the_simulation_would_install_something(self):
f = cfake(purge_extra_sim=["Inst proxmox-kernel-7.0.14-28-pve (7.0.14-28 Proxmox Debian Repository:stable [amd64])"])
rep = self.refused(f, "S1")
self.assertIn("proxmox-kernel-7.0.14-28-pve", rep["refused"]["reason"])
def test_s7_the_simulation_fails(self):
self.refused(cfake(purge_sim_rc=100), "S1")
# --- scenario 8: not our box ---
def test_s8_a_byo_host_is_refused_by_the_appliance_rule(self):
f = cfake()
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
self.refused(f, "R12")
def test_s8_a_box_without_the_ring0_mark_does_not_run(self):
f = cfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "tester-1-d70be4", "ring0_slow_lane": False})
rep = self.not_run(f, "no-ring0-mark")
self.assertIn("ring 0 only in this release", rep["kernel_cleanup"]["not_run"][0]["reason"])
g = cfake() # an agent-writable trust file proves nothing
g.stats[osapply.TRUST_FILE] = St(mode=statmod.S_IFREG | 0o666, uid=0)
self.not_run(g, "no-ring0-mark")
h = cfake() # the agent's plan cannot claim the ring
h.files[osapply.TRUST_FILE] = json.dumps({"host_id": "x", "ring0_slow_lane": False})
h.plan["ring"] = 0
self.not_run(h, "no-ring0-mark")
# --- scenario 9: a plan that names packages ---
def test_s9_cleanup_plan_names_refused(self):
for extra in ({"packages": [{"name": img("7.0.14-28-pve"), "version": "7.0.14-28", "origin": "Proxmox Debian Repository"}]},
{"select": "pending-kernel"}, {"allow_new": ["x"]}, {"signed": {"blob_b64": "eA==", "sig": "x"}},
{"expect_kver": "7.0.14-20-pve"}, {"dry_run": "yes"}):
f = cfake()
f.plan.update(extra)
self.refused(f, "C1")
# --- scenario 10: no kernel step ever ran ---
def test_s10_no_state_file_does_not_run(self):
f = cfake()
f.state = None
f.write_state()
rep = self.not_run(f, "no-step")
self.assertIn("no kernel step yet", rep["kernel_cleanup"]["not_run"][0]["reason"])
# --- scenario 11: a power cut in the purge ---
def test_s11_a_cut_purge_is_repaired_first_and_the_rule_is_computed_again(self):
f = cfake(purge_rc=100)
rc, rep = run(f)
self.assertEqual(rc, 3, rep)
self.assertEqual(rep["failed"]["step"], "purge")
self.assertEqual(rep["kernel_cleanup"]["outcome"], "failed")
# the cut removed ONE of the three before it stopped
f.installed.pop(self.REMOVE[0])
f.boot.discard("7.0.14-20-pve")
f.purge_rc, f.purges = 0, []
f.calls.clear()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["repair"]["journal_before"], 1, "repair first: dpkg's journal was replayed")
i_repair = [i for i, c in enumerate(f.calls) if c[-1][-3:-1] == ["--configure", "-a"] or "--configure" in c[-1]][0]
i_purge = [i for i, c in enumerate(f.calls) if "purge" in c[-1]][0]
self.assertLess(i_repair, i_purge, "the repair runs BEFORE the rule is computed and before any purge")
self.assertEqual(f.purges, [self.REMOVE[1:]], "the rule was computed again from the box: the two that are left")
self.assertEqual(rep["kernel_cleanup"]["outcome"], "removed")
# --- the 20-hour window, the dry run, after-the-purge proof ---
def test_inside_a_steps_20_hour_window_it_does_not_run(self):
f = cfake()
cstate(f, result_at=osapply._iso(f.clock - 3600)) # the step ended good one hour ago
self.not_run(f, "window")
def test_the_dry_run_reports_and_changes_nothing(self):
f = cfake()
f.plan["dry_run"] = True
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
c = rep["kernel_cleanup"]
self.assertEqual((c["outcome"], c["would"], c["removal"]), ("dry_run", "remove", self.REMOVE))
self.assertEqual(sorted(c["kept"]), sorted([C_FROM, C_RUN]))
self.nothing_changed(f)
self.assertNotIn("repair", rep, "a dry run does not even repair")
# a dry run of a refusal says which one, with exit 0
g = cstate(cfake(), phase="judging")
g.plan["dry_run"] = True
rc, rep = run(g)
self.assertEqual((rc, rep["kernel_cleanup"]["would"]), (0, "refuse"), rep)
self.assertEqual([x["code"] for x in rep["kernel_cleanup"]["refusals"]], ["P1"])
self.nothing_changed(g)
# … and of a box that would not run
h = cfake()
h.files[osapply.TRUST_FILE] = json.dumps({"host_id": "x", "ring0_slow_lane": False})
h.plan["dry_run"] = True
rc, rep = run(h)
self.assertEqual((rc, rep["kernel_cleanup"]["would"], rep["kernel_cleanup"]["removal"]), (0, "not_run", self.REMOVE), rep)
self.nothing_changed(h)
def test_a_purge_that_moves_the_default_is_reported_failed_loudly(self):
f = cfake(purge_moves_default=True) # the pinned file vanished in the purge: 10_linux makes the newest the default
rc, rep = run(f)
self.assertEqual(rc, 3, rep)
self.assertEqual(rep["failed"]["step"], "verify")
self.assertIn("pinned default file changed", rep["failed"]["reason"])
self.assertEqual(f.update_grubs, 0, "never repaired by hand in code")
self.assertTrue(any("FAILED" in l and "verify" in l for l in f.logs))
def test_the_apt_lock_is_respected(self):
self.refused(cfake(lock_held=True), "R9")
def test_both_image_packages_of_one_old_kernel_go_together(self):
f = cfake()
f.installed["proxmox-kernel-7.0.14-20-pve"] = "7.0.14-20" # the unsigned image beside the signed one
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn("proxmox-kernel-7.0.14-20-pve", f.purges[0])
self.assertIn(img("7.0.14-20-pve"), f.purges[0])
def test_cleanup_needs_no_new_sudo_line(self):
"""The wrapper is reached by `--plan` only: the new mode rides the one sudo line the agent already has."""
sudo = (HERE / "felhom-agent.sudoers").read_text()
live = [l.strip().rstrip("\\,").strip() for l in sudo.splitlines() if l.strip() and not l.strip().startswith("#")]
self.assertEqual([l for l in live if "felhom-os-apply" in l and "Cmnd_Alias" not in l and "NOPASSWD" not in l],
[osapply.SELF_CHECK_SUDO_LINE], "exactly the one --plan line reaches the wrapper")
self.assertFalse([l for l in live if "kernel-cleanup" in l or
("apt" in l and ("purge" in l or "remove" in l))], "no sudo line removes a package")
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
class PFake(Fake):
"""A host with two AppArmor cache folders. `stale`: profile files whose cached copy in the running kernel's folder
is an old compile; the refresh command rewrites the cache from `fresh`. lxc-containers compiles differently every
time (measured on demo-hp). `failed`: unit -> ExecStart program."""
def __init__(self, stale=("usr.sbin.chronyd",), failed=None, kernel="7.0.14-28-pve", record=None):
super().__init__()
self.kernel = kernel
self.fresh = {"usr.sbin.chronyd": b"chrony-4.8", "usr.bin.man": b"man", "lxc-containers": b"lxc-0", "nvidia_modprobe": b"nv"}
self.names = {"usr.sbin.chronyd": "/usr/sbin/chronyd", "usr.bin.man": "/usr/bin/man\nman_filter\nman_groff",
"lxc-containers": "lxc-container-default\nlxc-container-default-cgns", "nvidia_modprobe": "nvidia_modprobe\nnvidia_modprobe//kmod"}
self.cache = {"ae1a4050.0": {k: b"old-" + k.encode() for k in self.fresh}, "f6da0254.0": dict(self.fresh)}
for k in stale:
self.cache["f6da0254.0"][k] = b"OLD-COPY-" + k.encode()
self.failed = dict({"chrony.service": "/usr/sbin/chronyd"} if failed is None else failed)
self.restarts, self.parser_runs, self.parser_rc, self.tree = [], 0, 0, {}
if record is not None:
self.tree[osapply.PROFILE_STATE] = json.dumps(record)
self.plan = {"release_id": "profile-refresh", "layer": "host", "mode": "profile-refresh"}
def isdir(self, p):
return p in (osapply.PROFILE_DIR, osapply.PROFILE_CACHE) or p.startswith(osapply.PROFILE_CACHE + "/") and p.count("/") == 4
def list_dir(self, p):
if p == osapply.PROFILE_CACHE:
return sorted(self.cache)
d = p.rsplit("/", 1)[1]
return sorted(list(self.cache.get(d, {})) + [".features"])
def read_bytes(self, p):
d, f = p.split("/")[-2:]
return self.cache[d][f]
def put_file(self, path, data, mode):
self.tree[path] = data.decode()
def read_file(self, p):
if p in self.tree:
return self.tree[p]
return super().read_file(p)
def host(self, argv, timeout=600, stdin=None):
if argv[0] == "uname":
self.calls.append(("host", argv))
return 0, self.kernel + "\n", ""
if argv == osapply.PROFILE_REFRESH_CMD:
self.calls.append(("host", argv))
self.parser_runs += 1
if self.parser_rc:
return self.parser_rc, "", "AppArmor parser error for /etc/apparmor.d/x"
self.fresh["lxc-containers"] = b"lxc-%d" % self.parser_runs # never the same binary twice
self.cache["f6da0254.0"] = dict(self.fresh)
return 0, "", ""
if argv[:2] == ["apparmor_parser", "-N"]:
self.calls.append(("host", argv))
return 0, self.names[argv[2].rsplit("/", 1)[1]] + "\n", ""
if argv[:2] == ["systemctl", "list-units"]:
self.calls.append(("host", argv))
return 0, "".join(f"{u} loaded failed failed x\n" for u in self.failed), ""
if argv[:2] == ["systemctl", "show"]:
self.calls.append(("host", argv))
return 0, f"{{ path={self.failed[argv[-1]]} ; argv[]={self.failed[argv[-1]]} -F 1 ; ignore_errors=no }}\n", ""
if argv[:2] == ["systemctl", "restart"]:
self.calls.append(("host", argv))
self.restarts.append(argv[2])
return 0, "", ""
return super().host(argv, timeout, stdin)
class ProfileRefresh(unittest.TestCase):
"""Red-proofs: felhom.eu audits/release-2a-2026-11-14/B/redproof.txt."""
def test_a_stale_profile_is_refreshed_and_only_its_failed_unit_restarts(self):
f = PFake(failed={"chrony.service": "/usr/sbin/chronyd", "postfix.service": "/usr/sbin/postfix", "other.service": "/usr/bin/man-not"})
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
p = rep["profile_refresh"]
self.assertEqual(p["outcome"], "refreshed")
self.assertEqual(f.parser_runs, 1)
self.assertIn(("host", ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", "/etc/apparmor.d"]), f.calls)
self.assertEqual(f.cache["f6da0254.0"]["usr.sbin.chronyd"], b"chrony-4.8", "the cache holds the fresh compile")
self.assertEqual(p["rewritten"], ["f6da0254.0/lxc-containers", "f6da0254.0/usr.sbin.chronyd"])
self.assertEqual(f.restarts, ["chrony.service"], "ONLY the failed unit whose own program's profile was stale")
self.assertEqual([x["unit"] for x in p["restarted"]], ["chrony.service"])
rec = json.loads(f.tree[osapply.PROFILE_STATE])
self.assertEqual(rec["kernel"], "7.0.14-28-pve")
def test_a_fresh_cache_changes_nothing_and_restarts_nothing(self):
f = PFake(stale=()) # chrony is failed for another reason; its profile is NOT stale
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["profile_refresh"]["rewritten"], ["f6da0254.0/lxc-containers"], "only the file that never compiles the same")
self.assertEqual(f.restarts, [], "a failed unit whose profile was not stale is NOT restarted")
self.assertEqual(f.cache["f6da0254.0"]["usr.sbin.chronyd"], b"chrony-4.8")
def test_a_stale_profile_of_a_running_program_restarts_nothing(self):
f = PFake(failed={}) # the unit is active: the kernel swaps the profile of a running task itself
rc, rep = run(f)
self.assertEqual((rc, f.restarts, rep["profile_refresh"]["outcome"]), (0, [], "refreshed"), rep)
def test_a_failed_unit_of_another_program_is_not_restarted(self):
f = PFake(stale=("usr.bin.man",), failed={"chrony.service": "/usr/sbin/chronyd"})
rc, rep = run(f)
self.assertEqual((rc, f.restarts), (0, []), rep)
g = PFake(stale=("nvidia_modprobe",), failed={"nvidia.service": "nvidia_modprobe"}) # a profile name that is not a path
rc, rep = run(g)
self.assertEqual((rc, g.restarts), (0, []), rep)
def test_once_per_kernel(self):
f = PFake()
self.assertEqual(run(f)[0], 0)
f.calls.clear()
f.cache["f6da0254.0"]["usr.sbin.chronyd"] = b"OLD-COPY" # even with a stale file: this kernel was done
rc, rep = run(f)
self.assertEqual((rc, rep["profile_refresh"]["outcome"], f.parser_runs), (0, "not_needed", 1), rep)
self.assertFalse([c for c in f.calls if c[1][0] in ("apparmor_parser", "systemctl")], "no parser, no systemctl")
f.kernel = "7.0.14-29-pve" # the box starts on another kernel: again
rc, rep = run(f)
self.assertEqual((rc, rep["profile_refresh"]["outcome"], f.parser_runs), (0, "refreshed", 2), rep)
self.assertEqual(json.loads(f.tree[osapply.PROFILE_STATE])["kernel"], "7.0.14-29-pve")
def test_a_failed_parser_writes_no_record_so_the_next_start_tries_again(self):
f = PFake()
f.parser_rc = 1
rc, rep = run(f)
self.assertEqual((rc, rep["failed"]["step"], rep["profile_refresh"]["outcome"]), (3, "profile-refresh", "failed"), rep)
self.assertNotIn(osapply.PROFILE_STATE, f.tree)
self.assertEqual(f.restarts, [])
f.parser_rc = 0
self.assertEqual(run(f)[1]["profile_refresh"]["outcome"], "refreshed")
def test_the_plan_names_nothing(self):
for extra in ({"profiles": ["usr.sbin.chronyd"]}, {"units": ["ssh.service"]}, {"packages": [{"name": "chrony", "version": "4.8", "origin": "Debian"}]},
{"force": True}, {"dry_run": 1}, {"layer": "guest"}):
f = PFake()
f.plan.update(extra)
rc, rep = run(f)
self.assertEqual(rc, 2, rep)
self.assertIn(rep["refused"]["code"], ("F1", "R11"), rep)
self.assertEqual((f.parser_runs, f.restarts), (0, []))
def test_a_byo_host_is_refused(self):
f = PFake()
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"], f.parser_runs), (2, "R12", 0), rep)
def test_the_dry_run_changes_nothing(self):
f = PFake()
f.plan["dry_run"] = True
rc, rep = run(f)
self.assertEqual((rc, rep["profile_refresh"]["outcome"], rep["profile_refresh"]["would"]), (0, "dry_run", "refresh"), rep)
self.assertEqual((f.parser_runs, f.restarts), (0, []))
self.assertNotIn(osapply.PROFILE_STATE, f.tree)
def test_no_more_than_three_restarts(self):
f = PFake(failed={f"chrony{i}.service": "/usr/sbin/chronyd" for i in range(5)})
rc, rep = run(f)
self.assertEqual((rc, len(f.restarts), len(rep["profile_refresh"]["not_restarted"])), (0, 3, 2), rep)
def test_profile_refresh_changes_no_package(self):
f = PFake()
run(f)
self.assertFalse([c for c in f.calls if c[1][0] in ("apt-get", "dpkg", "apt")], "no package command at all")
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80, felhom.eu `03` §12a) ----------
CA_PEM = b"-----BEGIN CERTIFICATE-----\nQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=\n-----END CERTIFICATE-----\n"
CA_PEM_2 = b"-----BEGIN CERTIFICATE-----\nWllYV1ZVVFNSUVBPTk1MS0pJSEdGRURDQkE=\n-----END CERTIFICATE-----\n"
class TFake(Fake):
"""A Proxmox host: the root CA file, the copy (or none), and openssl's answer about the bytes it is given."""
def __init__(self, source=CA_PEM, copy=None, ca=True, ends="Aug 17 15:44:42 2036 GMT"):
super().__init__()
self.source, self.tree, self.ca, self.ends, self.openssl_rc = source, {}, ca, ends, 0
if copy is not None:
self.tree[osapply.PVE_CA_COPY] = copy
self.openssl_stdin, self.puts = [], []
self.serves_new = False # does pveproxy serve a certificate that verifies with the NEW root alone?
self.dead_ca = set() # certificate texts openssl says are not a live CA (ended)
self.handshakes = []
self.plan = {"release_id": "pve-ca", "layer": "host", "mode": "pve-ca-copy"}
def read_bytes(self, p):
if p == osapply.PVE_CA_SOURCE:
if self.source is None:
raise FileNotFoundError(p)
return self.source
if p in self.tree:
return self.tree[p]
raise FileNotFoundError(p)
def put_file(self, path, data, mode):
self.puts.append((path, data, mode))
self.tree[path] = data
def host(self, argv, timeout=600, stdin=None):
if argv[0] == "openssl" and argv[1] == "s_client":
self.handshakes.append(argv)
return (0, "Verify return code: 0 (ok)\n", "") if self.serves_new else (1, "", "Verify return code: 19 (self-signed certificate in certificate chain)\n")
if argv[0] == "openssl" and "-checkend" in argv:
return (1, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will expire\n", "") if stdin.strip() in self.dead_ca else \
(0, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will not expire\n", "")
if argv[0] == "openssl":
self.calls.append(("host", argv))
self.openssl_stdin.append(stdin)
if self.openssl_rc:
return self.openssl_rc, "", "Could not read certificate from <stdin>"
return 0, ("X509v3 Basic Constraints: critical\n CA:%s\nnotAfter=%s\n" % ("TRUE" if self.ca else "FALSE", self.ends)), ""
return super().host(argv, timeout, stdin)
class PVECACopy(unittest.TestCase):
"""Red-proofs: felhom.eu audits/night-build-2026-11-14/A/redproof-wrapper.txt."""
def refused(self, f, code):
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, code), rep)
self.assertEqual(f.puts, [], "nothing is written on a refusal")
return rep
def test_a_new_box_gets_the_copy(self):
f = TFake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.puts, [(osapply.PVE_CA_COPY, CA_PEM, 0o644)], "ONE file, the fixed place, the source's bytes, 0644")
self.assertEqual(f.openssl_stdin, [CA_PEM.decode()], "the bytes that are checked are the bytes that are written")
c = rep["pve_ca_copy"]
self.assertEqual((c["outcome"], c["path"], c["replaced"]), ("written", "/etc/felhom-agent/pve-root-ca.pem", False))
self.assertEqual(c["not_after"], "2036-08-17T15:44:42Z")
self.assertGreater(c["days_left"], 3000)
def test_the_same_copy_is_not_written_again(self):
f = TFake(copy=CA_PEM)
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
def test_a_renewed_ca_refreshes_the_copy(self):
f = TFake(source=CA_PEM_2, copy=CA_PEM)
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["replaced"]), (0, "written", True), rep)
# the new certificate FIRST, and the copy's previous one kept behind it: Proxmox serves its old certificate until
# pveproxy is reloaded, and an agent that restarts in that window must still have a root that verifies it
bundle = f.tree[osapply.PVE_CA_COPY]
self.assertEqual(bundle, CA_PEM_2 + osapply.PEM_CERT_RE.findall(CA_PEM)[0] + b"\n")
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
self.assertEqual(f.openssl_stdin, [CA_PEM_2.decode()], "the SOURCE is checked in full; the kept one by its own check")
def test_the_previous_root_stays_only_for_the_changeover(self):
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
bundle = CA_PEM_2 + old + b"\n"
# Proxmox still serves the OLD certificate: nothing is written, the old root stays
f = TFake(source=CA_PEM_2, copy=bundle)
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
self.assertEqual(len(f.handshakes), 1)
self.assertIn(osapply.PVE_CA_SOURCE, f.handshakes[0], "the handshake is verified with the NEW root alone (the source file)")
self.assertIn("-no-CApath", f.handshakes[0])
self.assertIn("-verify_return_error", f.handshakes[0])
# pveproxy was reloaded and serves a certificate of the NEW root: the old root is dropped
f = TFake(source=CA_PEM_2, copy=bundle)
f.serves_new = True
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2, "after the changeover the copy is the new root alone")
self.assertEqual((rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["dropped_previous"], rep["pve_ca_copy"]["kept_previous"]), ("written", True, False))
# the old root has ENDED while Proxmox still serves the old certificate: it verifies nothing, so it is dropped
f = TFake(source=CA_PEM_2, copy=bundle)
f.dead_ca = {old.decode()}
rc, rep = run(f)
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
# an ended previous root is not kept at the change itself either
f = TFake(source=CA_PEM_2, copy=CA_PEM)
f.dead_ca = {old.decode()}
rc, rep = run(f)
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
def test_a_copy_with_anything_unchecked_in_it_is_rewritten(self):
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
for name, copy in (("text after the source", CA_PEM_2 + b"anything at all\n"),
("a third certificate", CA_PEM_2 + old + b"\n" + old + b"\n"),
("a key in the file", CA_PEM_2 + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"),
("the two in the wrong order", CA_PEM + osapply.PEM_CERT_RE.findall(CA_PEM_2)[0] + b"\n")):
f = TFake(source=CA_PEM_2, copy=copy)
f.serves_new = True
rc, rep = run(f)
self.assertEqual(rc, 0, (name, rep))
got = f.tree[osapply.PVE_CA_COPY]
self.assertTrue(got.startswith(CA_PEM_2), name)
self.assertNotIn(b"anything", got, name)
self.assertNotIn(b"PRIVATE KEY", got, name)
self.assertLessEqual(len(osapply.PEM_CERT_RE.findall(got)), 2, name)
# a first copy keeps nothing behind it, and makes no handshake
f4 = TFake()
rc, rep = run(f4)
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
self.assertEqual((f4.tree[osapply.PVE_CA_COPY], f4.handshakes), (CA_PEM, []))
def test_a_source_that_is_not_a_ca_certificate_is_refused(self):
self.refused(TFake(ca=False), "T3") # a leaf certificate
self.refused(TFake(source=b"hello\n"), "T3") # not a certificate at all
self.refused(TFake(source=CA_PEM + CA_PEM_2), "T3") # two certificates (a chain, a bundle)
self.refused(TFake(source=CA_PEM + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"), "T3")
self.refused(TFake(source=b""), "T3")
self.refused(TFake(source=CA_PEM + b"x" * (osapply.PVE_CA_MAX_BYTES + 1)), "T3")
f = TFake()
f.openssl_rc = 1 # openssl cannot parse it
self.refused(f, "T3")
self.refused(TFake(ends="Aug 17 15:44:42 2020 GMT"), "T3") # an ended CA
self.refused(TFake(ends="garbage"), "T3")
def test_no_source_is_refused(self):
self.refused(TFake(source=None), "T2")
def test_with_the_real_openssl_and_the_real_runner(self):
"""NOT a fake: a CA and a leaf made by openssl in a temp folder, judged through Runner.host as on a box (the
fake cannot show a wrong argument or a bytes/text mix-up). Skipped where openssl is missing."""
import shutil, tempfile
if not shutil.which("openssl"):
self.skipTest("no openssl here")
d = tempfile.mkdtemp()
try:
def sh(*a):
subprocess.run(a, check=True, capture_output=True, cwd=d)
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "ca.key", "-out", "ca.pem", "-days", "30",
"-subj", "/CN=test CA", "-addext", "basicConstraints=critical,CA:TRUE")
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "leaf.key", "-out", "leaf.pem", "-days", "30",
"-subj", "/CN=leaf", "-addext", "basicConstraints=CA:FALSE")
real = osapply.Runner()
for name, want in (("ca.pem", "written"), ("leaf.pem", "T3")):
data = open(os.path.join(d, name), "rb").read()
f = TFake(source=data)
f.host = lambda argv, timeout=600, stdin=None: real.host(argv, timeout, stdin) if argv[0] == "openssl" else Fake.host(f, argv, timeout, stdin)
f.clock = __import__("time").time()
rc, rep = run(f)
got = rep["pve_ca_copy"]["outcome"] if rc == 0 else rep["refused"]["code"]
self.assertEqual(got, want, f"{name}: {rep}")
if want == "written":
self.assertIn(rep["pve_ca_copy"]["days_left"], (29, 30))
# the two checks the changeover uses, with the REAL openssl and a REAL TLS server
ended = True
try: # an ENDED CA needs openssl 3.4+ (-not_after); older: that one check is not made
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "old.key", "-out", "old.pem",
"-not_before", "20190101000000Z", "-not_after", "20200101000000Z",
"-subj", "/CN=ended CA", "-addext", "basicConstraints=critical,CA:TRUE")
except subprocess.CalledProcessError:
ended = False
sh("openssl", "req", "-newkey", "rsa:2048", "-nodes", "-keyout", "srv.key", "-out", "srv.csr", "-subj", "/CN=pve")
sh("openssl", "x509", "-req", "-in", "srv.csr", "-CA", "ca.pem", "-CAkey", "ca.key", "-CAcreateserial", "-out", "srv.pem", "-days", "5")
app = osapply.Apply.__new__(osapply.Apply)
app.r = real
block = lambda n: osapply.PEM_CERT_RE.findall(open(os.path.join(d, n), "rb").read())[0]
self.assertTrue(app.is_live_ca(block("ca.pem")), "a CA that has not ended")
self.assertFalse(app.is_live_ca(block("leaf.pem")), "a leaf is not a CA")
if ended:
self.assertFalse(app.is_live_ca(block("old.pem")), "an ended CA is not a live one")
import socket
s0 = socket.socket(); s0.bind(("127.0.0.1", 0)); port = s0.getsockname()[1]; s0.close()
srv = subprocess.Popen(["openssl", "s_server", "-accept", f"127.0.0.1:{port}", "-cert", "srv.pem", "-key", "srv.key", "-quiet"],
cwd=d, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
saved, osapply.PVE_LOCAL_API = osapply.PVE_LOCAL_API, f"127.0.0.1:{port}"
ok = False
for _ in range(40): # the server needs a moment to listen
if app.pve_serves_with(os.path.join(d, "ca.pem")):
ok = True
break
__import__("time").sleep(0.1)
self.assertTrue(ok, "the server's certificate is signed by ca.pem: the handshake must verify with it alone")
self.assertFalse(app.pve_serves_with(os.path.join(d, "leaf.pem")), "another certificate as the CA file must NOT verify")
osapply.PVE_LOCAL_API = "127.0.0.1:1"
self.assertFalse(app.pve_serves_with(os.path.join(d, "ca.pem")), "no answer is a no")
finally:
osapply.PVE_LOCAL_API = saved
srv.kill(); srv.wait()
finally:
shutil.rmtree(d, ignore_errors=True)
def test_a_plan_that_names_a_path_or_a_file_is_refused(self):
for extra in ({"source": "/etc/shadow"}, {"path": "/etc/sudoers.d/x"}, {"dest": "/root/.ssh/authorized_keys"},
{"content_b64": "QQ=="}, {"packages": []}, {"file": "x"}, {"vmid": 9201}, {"dry_run": True}):
f = TFake()
f.plan.update(extra)
rep = self.refused(f, "T1")
self.assertEqual(f.openssl_stdin, [], "refused before the source is even read")
g = TFake()
g.plan["layer"] = "kernel"
self.refused(g, "R11")
def test_a_refused_source_keeps_an_older_copy(self):
f = TFake(source=b"not a certificate", copy=CA_PEM)
self.refused(f, "T3")
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM, "written whole or not at all")
def test_it_runs_on_a_byo_host_too_and_changes_nothing_else(self):
f = TFake()
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
rc, rep = run(f)
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"]), (0, "written"), rep)
self.assertEqual([c[1][0] for c in f.calls], ["openssl"], "one openssl call; no package, no service, no other file")
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -1,6 +1,6 @@
module gitea.dooplex.hu/admin/felhom-agent
go 1.27.2
go 1.27.4
require (
golang.org/x/crypto v0.58.0
+2
View File
@@ -22,6 +22,8 @@ var (
ErrExpired = errors.New("authz: op expired")
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
ErrNotYetValid = errors.New("authz: op not yet valid")
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
ErrWindowTooLong = errors.New("authz: op validity window too long")
// ErrReplay: the nonce was already recorded in the window.
ErrReplay = errors.New("authz: replay (nonce already seen)")
)
+8
View File
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
// the validity window stays an honest upper bound.
const DefaultClockSkew = 2 * time.Minute
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
const MaxWindow = 24 * time.Hour
// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
type KeyRole string
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
}
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
}
if now.After(op.ExpiresAt) {
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
}
+20
View File
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
rejects(t, err, ErrTarget)
})
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
t.Run("window longer than 24 h", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
_, err := v.Verify(blob, sig)
rejects(t, err, ErrWindowTooLong)
})
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
if _, err := v.Verify(blob, sig); err != nil {
t.Fatalf("a 24 h window must verify: %v", err)
}
})
t.Run("expired", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
@@ -0,0 +1,55 @@
package backup
import (
"context"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
// old box's time.
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{oldBox},
}
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The fresh box has written nothing yet: the old box's archive must not count.
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatal(err)
}
if found {
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
}
// 2. After the box's own first copy, that copy is the answer.
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
api.content = append(api.content, own)
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
if err != nil || !found || when.Unix() != own.CTime {
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
}
}
+12
View File
@@ -609,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
if err != nil {
return time.Time{}, false, err
}
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
if err != nil {
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
}
var best int64 = -1
for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid {
continue
}
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why)
continue
+1
View File
@@ -151,6 +151,7 @@ var manifest = []Capability{
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
+188
View File
@@ -0,0 +1,188 @@
package config
import (
"bytes"
"encoding/json"
"fmt"
"os"
"path/filepath"
)
// A cut write of agent.json must never leave a blind box (R-1380, 2026-11-15).
//
// agent.json is the agent's own file inside a ROOT-owned folder, so the agent cannot write a new file and rename it
// over the old one: every writer truncates the file and writes it again. A power cut between the truncate and the
// sync leaves an empty or half file, the agent does not start, and nothing can repair a box whose agent does not
// start from the hub. So the three writers (the trust move, the PBS seed, the backup-target move) go through
// ReplaceInPlace: the OLD bytes are first made durable in a journal file in the agent's own state folder (a new file
// and a rename, which the agent CAN do there), then the config is written, synced and read back, then the journal is
// removed. RecoverCutWrite runs at every start, before the config is read: a journal beside a config that does not
// parse means the write was cut, and the old bytes go back. A journal beside a config that parses means the cut fell
// before the truncate or after the sync — the file is whole, old or new, and only the journal is removed.
//
// Pinned by TestReplaceInPlace_* and TestRecoverCutWrite_* (this package) and by TestMainRecoversACutConfigWrite
// (cmd/felhom-agent).
// CutWriteDir is where the journal lives. It is NOT read from the config: the recovery runs before the config can be
// read. /var/lib/felhom-agent is the unit's StateDirectory. A test points it at a temp folder.
var CutWriteDir = "/var/lib/felhom-agent"
// cutWriteJournal is the journal's file name: the bytes agent.json held before the write that is in flight.
const cutWriteJournal = "agent.json.write-in-flight"
// cutPoint lets a test stop a write where a power cut would (the names are the steps of ReplaceInPlace).
var cutPoint = func(step string) error { return nil }
// ReplaceInPlace writes next over the config at path, whose present content is prev. The caller holds FileMu.
// With no journal folder there is no write: an unprotected write is the defect this exists to remove.
func ReplaceInPlace(path string, prev, next []byte) error {
if !json.Valid(prev) || !json.Valid(next) {
return fmt.Errorf("config: refusing to write %s: the old or the new content is not JSON", path)
}
st, err := os.Stat(path)
if err != nil {
return err
}
if err := os.MkdirAll(CutWriteDir, 0o700); err != nil {
return fmt.Errorf("config: the journal folder: %w", err)
}
journal := filepath.Join(CutWriteDir, cutWriteJournal)
if err := writeDurable(journal, prev); err != nil {
return fmt.Errorf("config: the journal of the old content: %w", err)
}
if err := cutPoint("journal-written"); err != nil {
return err
}
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
_ = removeDurable(journal)
return err
}
if err := cutPoint("truncated"); err != nil {
f.Close()
return err
}
half := len(next) / 2
_, werr := f.Write(next[:half])
if werr == nil {
if err := cutPoint("half-written"); err != nil {
f.Close()
return err
}
_, werr = f.Write(next[half:])
}
if werr == nil {
werr = f.Sync()
}
if cerr := f.Close(); werr == nil {
werr = cerr
}
if werr == nil {
var back []byte
if back, werr = os.ReadFile(path); werr == nil && !bytes.Equal(back, next) {
werr = fmt.Errorf("the file did not read back as written")
}
}
if werr != nil {
// The write failed with the process alive: put the old bytes back now. If that fails too, the journal stays
// and the next start repairs the file.
if rerr := restoreInPlace(path, prev); rerr != nil {
return fmt.Errorf("config: writing %s failed (%v) and the old content could not be put back (%v) — the next start repairs it from the journal", path, werr, rerr)
}
_ = removeDurable(journal)
return fmt.Errorf("config: writing %s failed, the old content was put back: %w", path, werr)
}
if err := cutPoint("synced"); err != nil {
return err
}
return removeDurable(journal)
}
// RecoverCutWrite repairs a config whose last write was cut. It returns what it did, for the start's log line:
// "" (no journal), "journal removed — the config is whole", or "the old config was put back".
func RecoverCutWrite(path string) (string, error) {
journal := filepath.Join(CutWriteDir, cutWriteJournal)
prev, err := os.ReadFile(journal)
if os.IsNotExist(err) {
return "", nil
}
if err != nil {
return "", fmt.Errorf("config: reading the write journal: %w", err)
}
cur, cerr := os.ReadFile(path)
if cerr == nil && isJSONObject(cur) {
return "journal removed — the config is whole", removeDurable(journal)
}
if !isJSONObject(prev) {
// A journal that is itself broken cannot have been followed by a truncate (it is made durable first), so
// this is not a cut write. Leave both for a person.
return "", fmt.Errorf("config: %s does not parse and the write journal does not parse either", path)
}
if err := restoreInPlace(path, prev); err != nil {
return "", fmt.Errorf("config: putting the old content back into %s: %w", path, err)
}
return "the old config was put back", removeDurable(journal)
}
func isJSONObject(b []byte) bool {
var doc map[string]json.RawMessage
return json.Unmarshal(b, &doc) == nil && doc != nil
}
// restoreInPlace writes content into the EXISTING file (no create: the agent cannot make a file in the config's
// folder), so the file keeps its owner and mode.
func restoreInPlace(path string, content []byte) error {
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, 0)
if err != nil {
return err
}
if _, err := f.Write(content); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
return f.Close()
}
// writeDurable makes path hold content, whole, across a power cut: a new file, synced, renamed, the folder synced.
func writeDurable(path string, content []byte) error {
tmp := path + ".tmp"
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return err
}
if _, err := f.Write(content); err != nil {
f.Close()
return err
}
if err := f.Sync(); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
return err
}
if err := os.Rename(tmp, path); err != nil {
return err
}
return syncDir(filepath.Dir(path))
}
func removeDurable(path string) error {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return err
}
return syncDir(filepath.Dir(path))
}
func syncDir(dir string) error {
d, err := os.Open(dir)
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}
+140
View File
@@ -0,0 +1,140 @@
package config
import (
"errors"
"os"
"path/filepath"
"testing"
)
const (
oldCfg = `{"proxmox":{"tls":{"fingerprint":"AA:BB"}},"keep":"me"}`
newCfg = `{"proxmox":{"tls":{"ca_file":"/etc/felhom-agent/pve-root-ca.pem","fingerprint":""}},"keep":"me"}`
)
// cutBox is a config file and a journal folder in a temp dir, with the write cut at one named step.
func cutBox(t *testing.T) (cfg string) {
t.Helper()
dir := t.TempDir()
cfg = filepath.Join(dir, "etc", "agent.json")
if err := os.MkdirAll(filepath.Dir(cfg), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(cfg, []byte(oldCfg), 0o600); err != nil {
t.Fatal(err)
}
prevDir, prevCut := CutWriteDir, cutPoint
CutWriteDir = filepath.Join(dir, "state")
t.Cleanup(func() { CutWriteDir, cutPoint = prevDir, prevCut })
return cfg
}
func cutAt(step string) {
cutPoint = func(s string) error {
if s == step {
return errors.New("power cut at " + s)
}
return nil
}
}
func read(t *testing.T, p string) string {
t.Helper()
b, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
return string(b)
}
// The consequence that matters: after a power cut at ANY step of the write and the repair at the next start, the
// config is the whole old file or the whole new file — never a broken one — and Load reads it.
func TestRecoverCutWrite_AfterACutAtEveryStepTheConfigIsWholeAndLoads(t *testing.T) {
want := map[string]string{
"journal-written": oldCfg, // nothing was touched yet
"truncated": oldCfg, // the file is EMPTY at the cut
"half-written": oldCfg, // the file is HALF at the cut
"synced": newCfg, // the new file is whole; only the journal is left
}
for step, wantAfter := range want {
t.Run(step, func(t *testing.T) {
cfg := cutBox(t)
cutAt(step)
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
t.Fatal("the staged cut did not stop the write")
}
atCut := read(t, cfg)
if (step == "truncated" || step == "half-written") && isJSONObject([]byte(atCut)) {
t.Fatalf("the staging is wrong: at %s the file must be broken, it is %q", step, atCut)
}
cutPoint = func(string) error { return nil } // the next start: no cut
note, err := RecoverCutWrite(cfg)
if err != nil {
t.Fatalf("repair: %v", err)
}
if note == "" {
t.Fatal("the repair saw no journal — a cut write would stay broken")
}
if got := read(t, cfg); got != wantAfter {
t.Fatalf("after a cut at %s and the repair the config is %q, want %q", step, got, wantAfter)
}
if _, err := Load(cfg); err != nil {
t.Fatalf("the agent does not start with the repaired config: %v", err)
}
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
t.Fatalf("the journal stayed after the repair (%v) — an old journal would undo a later change", err)
}
})
}
}
// A write that is not cut leaves the new file and NO journal: a journal that stayed would put an old config back
// at some later start.
func TestReplaceInPlace_AWholeWriteLeavesTheNewFileAndNoJournal(t *testing.T) {
cfg := cutBox(t)
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err != nil {
t.Fatal(err)
}
if got := read(t, cfg); got != newCfg {
t.Fatalf("config = %q", got)
}
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
t.Fatalf("the journal stayed after a whole write: %v", err)
}
if note, err := RecoverCutWrite(cfg); err != nil || note != "" {
t.Fatalf("a start after a whole write must do nothing: note=%q err=%v", note, err)
}
if got := read(t, cfg); got != newCfg {
t.Fatalf("the start changed a whole config to %q", got)
}
}
// No journal folder, no write: the old file stays as it was.
func TestReplaceInPlace_WithoutAJournalNothingIsWritten(t *testing.T) {
cfg := cutBox(t)
// the journal cannot be made: a folder sits where its new file must be written
if err := os.MkdirAll(filepath.Join(CutWriteDir, cutWriteJournal+".tmp"), 0o700); err != nil {
t.Fatal(err)
}
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
t.Fatal("the write went on with no journal")
}
if got := read(t, cfg); got != oldCfg {
t.Fatalf("the config changed with no journal: %q", got)
}
}
// A config that is broken with NO journal is not a cut write: the repair does not touch it, and says nothing.
func TestRecoverCutWrite_ABrokenConfigWithNoJournalIsLeftAlone(t *testing.T) {
cfg := cutBox(t)
if err := os.WriteFile(cfg, []byte(`{"half":`), 0o600); err != nil {
t.Fatal(err)
}
note, err := RecoverCutWrite(cfg)
if err != nil || note != "" {
t.Fatalf("note=%q err=%v", note, err)
}
if got := read(t, cfg); got != `{"half":` {
t.Fatalf("the repair wrote a config it had no journal for: %q", got)
}
}
+12
View File
@@ -0,0 +1,12 @@
package config
import "sync"
// FileMu is held by EVERY read-modify-write of agent.json inside this process: the trust move (pvetrust), the PBS
// seed (pbsdr) and the backup-target move (localapi). Each of them reads the whole file, changes one section and
// writes the whole file back in place — without one lock two of them could interleave, and the second writer would
// put back the document it read BEFORE the first one's change (a silently reverted `escrow.pbs_storage_id`,
// `backup.local_backup_target` or trust section). Found by review on 2026-11-14, the night the third writer was
// added. Hold it for the whole read … write, never across a call to Proxmox or the root wrapper.
// Pinned by TestAgentJSONWritersTakeTheSharedLock (cmd/felhom-agent).
var FileMu sync.Mutex
+30 -1
View File
@@ -10,6 +10,7 @@ import (
"io"
"log/slog"
"net/http"
"net/url"
"os"
"strings"
"time"
@@ -86,6 +87,7 @@ func (e *HTTPError) Error() string {
// IS the heartbeat (locked decision 1). Errors are typed (transport vs HTTP) and
// never include the bearer token.
func (c *Client) Report(ctx context.Context, r *HostReport) (*ControlEnvelope, error) {
r.SentAt = time.Now().UTC().Format(time.RFC3339Nano) // the box clock at SENDING (R-1384); see HostReport.SentAt
body, err := json.Marshal(r)
if err != nil {
return nil, fmt.Errorf("hub: marshaling report: %w", err)
@@ -285,10 +287,37 @@ func (c *Client) Jobs(ctx context.Context) ([]JobWire, error) {
// /hosts/{id}/jobs/{job_id}, self-scoped. Called after a job is executed OR permanently rejected
// (the nonce is already durably burned on a passing verify, so re-processing is replay-safe).
func (c *Client) CompleteJob(ctx context.Context, jobID string) error {
return c.CompleteJobWithOutcome(ctx, jobID, JobOutcome{})
}
// JobOutcome says HOW a signed job ended when the agent clears it (R-1383, D78): Outcome is done | rejected | failed
// | malformed; Reason is one code of signedjobs.ReasonCodes (empty for done); Op is the job's op class. It rides the
// DELETE's query, so a hub that does not know it ignores it. Until this, „ran” and „refused” looked the same to the
// hub, and a refused agent update left no trace outside the box's journal. No secret: no signature, no nonce, no
// parameter of the job is in it.
type JobOutcome struct {
Outcome string
Reason string
Op string
}
// CompleteJobWithOutcome clears the job and tells the hub how it ended. The zero JobOutcome sends no query.
func (c *Client) CompleteJobWithOutcome(ctx context.Context, jobID string, o JobOutcome) error {
if c.hostID == "" || jobID == "" {
return fmt.Errorf("hub: CompleteJob requires host_id + job_id")
}
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.baseURL+"/api/v1/hosts/"+c.hostID+"/jobs/"+jobID, nil)
u := c.baseURL + "/api/v1/hosts/" + c.hostID + "/jobs/" + jobID
if o.Outcome != "" {
q := url.Values{"outcome": {o.Outcome}}
if o.Reason != "" {
q.Set("reason", o.Reason)
}
if o.Op != "" {
q.Set("op", o.Op)
}
u += "?" + q.Encode()
}
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, u, nil)
if err != nil {
return err
}
+76
View File
@@ -0,0 +1,76 @@
package hub
import (
"context"
"encoding/json"
"io"
"net/http"
"testing"
"time"
)
// R-1384 (D79): the hub tells a wrong box clock by the time stamped at the moment of SENDING. A report whose collect
// started long ago (an old reported_at) must still leave with a fresh sent_at, and a second send of the SAME report
// struct must carry a new one — or a report that waited on the box reads as a clock that is behind.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt).
func TestReport_StampsSentAtAtSending(t *testing.T) {
var bodies []map[string]any
c := testClient(func(r *http.Request) (*http.Response, error) {
raw, _ := io.ReadAll(r.Body)
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatalf("body: %v", err)
}
bodies = append(bodies, m)
return httpResp(200, `{"status":"ok"}`), nil
})
old := time.Now().Add(-40 * time.Minute).UTC()
rep := &HostReport{HostID: "h", ReportedAt: old.Format(time.RFC3339), SentAt: old.Format(time.RFC3339Nano)}
for i := 0; i < 2; i++ {
if _, err := c.Report(context.Background(), rep); err != nil {
t.Fatalf("Report: %v", err)
}
time.Sleep(5 * time.Millisecond)
}
var sent []time.Time
for i, m := range bodies {
s, _ := m["sent_at"].(string)
at, err := time.Parse(time.RFC3339Nano, s)
if err != nil {
t.Fatalf("send %d: sent_at = %q, want an RFC3339 time", i, s)
}
if d := time.Since(at); d < 0 || d > 5*time.Second {
t.Errorf("send %d: sent_at is %s old, want the moment of sending (reported_at was 40 minutes old)", i, d)
}
if m["reported_at"] != old.Format(time.RFC3339) {
t.Errorf("send %d: reported_at changed to %v — the collect time is not the send time's to touch", i, m["reported_at"])
}
sent = append(sent, at)
}
if len(sent) == 2 && !sent[1].After(sent[0]) {
t.Errorf("the second send carries %s, not after the first %s — the stamp is not taken per send", sent[1], sent[0])
}
}
// R-1383 (D78): the clearing call says how the job ended, in the DELETE's query; a plain CompleteJob sends none (an
// older hub, and the „nothing to say” case, see the same request as before).
func TestCompleteJob_OutcomeRidesTheQuery(t *testing.T) {
var urls []string
c := testClient(func(r *http.Request) (*http.Response, error) {
if r.Method != http.MethodDelete {
t.Errorf("method = %s", r.Method)
}
urls = append(urls, r.URL.Path+"?"+r.URL.RawQuery)
return httpResp(200, `{"status":"ok"}`), nil
})
if err := c.CompleteJob(context.Background(), "j1"); err != nil {
t.Fatal(err)
}
if err := c.CompleteJobWithOutcome(context.Background(), "j2", JobOutcome{Outcome: "rejected", Reason: "expired", Op: "agent_update"}); err != nil {
t.Fatal(err)
}
want := []string{"/api/v1/hosts/demo-host-01/jobs/j1?", "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&reason=expired"}
if len(urls) != 2 || urls[0] != want[0] || urls[1] != want[1] {
t.Fatalf("requests = %q, want %q", urls, want)
}
}
+61 -6
View File
@@ -124,10 +124,14 @@ type Collector struct {
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
pveTrust func() any // R-1380: the proxmox_trust stanza (nil = omitted)
guestPowerGaveUp func() any // R-1383: the guest_power_gave_up stanza (nil = omitted)
profileRefresh func() any // R-1383: the profile_refresh stanza (nil = omitted)
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
@@ -208,6 +212,24 @@ func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capabi
return c
}
// SetProxmoxTrustSource wires the `proxmox_trust` stanza (R-1380). Optional; nil = the stanza is omitted.
func (c *Collector) SetProxmoxTrustSource(f func() any) *Collector {
c.pveTrust = f
return c
}
// SetProfileRefreshSource wires the `profile_refresh` stanza (R-1383). Optional; nil, or a nil result, omits it.
func (c *Collector) SetProfileRefreshSource(f func() any) *Collector {
c.profileRefresh = f
return c
}
// SetGuestPowerGaveUpSource wires the `guest_power_gave_up` stanza (R-1383). Optional; nil, or a nil result, omits it.
func (c *Collector) SetGuestPowerGaveUpSource(f func() any) *Collector {
c.guestPowerGaveUp = f
return c
}
// SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub
// watches it for a re-key). Static per process — set once at startup. "" when the local API is
// disabled. Returns the collector for chaining.
@@ -235,6 +257,12 @@ type ControllerSupervisorReporter interface {
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
}
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
c.opResults = f
return c
}
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
c.ctrlSup = r
@@ -384,6 +412,27 @@ func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
// error (no useful report — the cycle skips the POST); a failed per-guest
// GuestConfig degrades that guest to status="unknown" without spec but still sends;
// a cloudflared probe failure yields status="unknown" and is never fatal.
func (c *Collector) profileRefreshStanza() any {
if c.profileRefresh == nil {
return nil
}
return c.profileRefresh()
}
func (c *Collector) guestPowerGaveUpStanza() any {
if c.guestPowerGaveUp == nil {
return nil
}
return c.guestPowerGaveUp()
}
func (c *Collector) proxmoxTrust() any {
if c.pveTrust == nil {
return nil
}
return c.pveTrust()
}
func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
ns, err := c.px.NodeStatus(ctx)
if err != nil {
@@ -409,12 +458,15 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
RestoreTests: c.collectRestoreTests(ctx),
PBSSnapshots: c.collectPBSSnapshots(ctx),
AuditTail: []AuditEntry{},
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
AuditTail: []AuditEntry{},
Cloudflared: c.cloudflared(ctx),
Capabilities: c.capabilities(ctx),
LeafFingerprint: c.leafFP,
ProxmoxTrust: c.proxmoxTrust(),
GuestPowerGaveUp: c.guestPowerGaveUpStanza(),
ProfileRefresh: c.profileRefreshStanza(),
Addresses: c.collectAddresses(),
System: c.systemInfo(ctx, ns),
}
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
@@ -432,6 +484,9 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.ctrlSup != nil {
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
}
if c.opResults != nil {
report.OperatorActionResults = c.opResults()
}
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
if c.guestNet != nil {
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
+9
View File
@@ -70,8 +70,14 @@ type Loop struct {
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
// path waits on it so the hub has the deliberate-off record before the box goes down.
reported func(r *HostReport)
// collectErr (R-1380) sees every failed collect.
collectErr func(ctx context.Context, err error)
}
// SetCollectErrorHook wires an observer of a failed collect (R-1380: a call to Proxmox that fails on TRUST starts the
// move to the CA copy, or a reload of a refreshed copy). Optional; it must not block for long.
func (l *Loop) SetCollectErrorHook(f func(ctx context.Context, err error)) { l.collectErr = f }
// SetReportedHook wires the R-998 after-report observer. Optional.
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
@@ -144,6 +150,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
report, err := l.collector.Collect(ctx)
if err != nil {
l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err)
if l.collectErr != nil {
l.collectErr(ctx, err)
}
return current
}
// Fulfill a pending log-pull: attach the ring tail to THIS report and clear the
+52 -2
View File
@@ -15,8 +15,13 @@ import (
// server-side received_at is the hub's dead-man's-switch liveness signal. There is
// no separate heartbeat endpoint.
type HostReport struct {
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
// SentAt (R-1384, D79) is this box's clock at the moment the report is SENT — stamped by Client.Report, never by
// the collector. The hub compares it with its own clock to tell a wrong box clock. ReportedAt cannot serve: it is
// stamped when the collect STARTS, and a collect that waits on Proxmox makes it old on a box whose clock is right.
// Pinned by TestReport_StampsSentAtAtSending.
SentAt string `json:"sent_at,omitempty"`
AgentVersion string `json:"agent_version"`
// AgentSHA256 is the sha256 of the binary this process is RUNNING (read through /proc/self/exe,
// once per process), R-349. The version string cannot tell a hand-built proof binary from the
@@ -53,6 +58,11 @@ type HostReport struct {
// alert. Not a secret (the fp is public; the token is never reported).
LeafFingerprint string `json:"leaf_fingerprint"`
// ProxmoxTrust (R-1380, felhom.eu `03` §12a): how the agent trusts Proxmox's certificate (`mode`: ca | fingerprint |
// …), and the CAUSE when something about it is wrong (`problem`, `detail`): the proof with the CA copy failed, the
// copy ends soon, the root step gave no copy. Omitted when not wired. No secret: a mode, a cause, a date.
ProxmoxTrust any `json:"proxmox_trust,omitempty"`
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
@@ -172,6 +182,32 @@ type HostReport struct {
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
// ProfileRefresh (R-1383, release 3): the box's last security-profile refresh and how the agent's last ask ended
// (osupdate.ProfileRefreshStatus, field-exact; hub sysfacts.ProfileRefresh). `any` + omitempty: absent when there
// is nothing to say or the source is not wired.
ProfileRefresh any `json:"profile_refresh,omitempty"`
// GuestPowerGaveUp (R-1383, D78): the guests the guest-power watchdog stopped trying to start (localapi
// GuestPowerGiveUp, field-exact: vmid, since, attempts). The hub raises one operator alarm per guest and clears it
// when the guest runs. `any` + omitempty: absent when there is none or the source is not wired.
GuestPowerGaveUp any `json:"guest_power_gave_up,omitempty"`
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
// when there is nothing to answer, so the golden stays byte-stable.
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
}
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
type OperatorAction struct {
ID int64 `json:"id"`
Action string `json:"action"`
Arg string `json:"arg,omitempty"`
}
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
type OperatorActionResult struct {
ID int64 `json:"id"`
Outcome string `json:"outcome"`
Message string `json:"message"`
}
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
@@ -195,6 +231,17 @@ type ControllerSupervisorGuest struct {
Restarts24h int `json:"restarts_24h"`
SlowCrashloop bool `json:"slow_crashloop"`
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
Unresponsive bool `json:"unresponsive"`
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
// R-1344 restart (decided 2026-10-30 from the night's measurement): UnresponsiveRestartAfterS is how long after the
// alarm the agent restarts a controller that still does not answer (0 / absent = an older agent: alarm only — the
// hub's message says which). UnresponsiveRestartAt is the last such restart; the hub keys on it MOVING
// (controller_unresponsive_restart) and it equals LastRestartAt for that act.
UnresponsiveRestartAfterS int `json:"unresponsive_restart_after_s,omitempty"`
UnresponsiveRestartAt string `json:"unresponsive_restart_at,omitempty"` // RFC3339
}
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
@@ -631,6 +678,9 @@ type ControlEnvelope struct {
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
// Absent/false on an old hub → nothing happens.
LogTailRequested bool `json:"log_tail_requested"`
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
// (`04` §6.1). Absent on an older hub.
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
}
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
+113 -11
View File
@@ -19,6 +19,10 @@
// - No tunnel (no operational wg-felhom with a server address) → no test and no value; the controller then keeps its
// passive floor and sets no qBittorrent limit.
// - The last result is persisted, so a restart neither loses it nor re-tests.
// - A box whose torrent app seeds all day is never quiet for 10 minutes (R-1250). For it the controller holds the
// app's upload limit very low and asks for a test on request (RunNow): every rule above except the 10 quiet
// minutes still holds, the quiet is checked over RequestQuietWindow instead, and the runs are RequestRunGap
// apart, so the whole request ends in about 55 s.
package linetest
import (
@@ -56,6 +60,11 @@ const (
TickInterval = time.Minute
FirstTickDelay = 5 * time.Minute
GateName = "line-test"
// A test on request (R-1250): the controller holds the app's limit while it waits, so it is short.
RequestSettle = 5 * time.Second // the held app's last packets leave before the quiet is read
RequestQuietWindow = 10 * time.Second // the uplink must send below QuietMbps over this window
RequestRunGap = 5 * time.Second
)
// ScheduleText is carried on the host report.
@@ -258,30 +267,52 @@ func (t *Tester) Run(ctx context.Context) {
}
}
// Tick is one look.
func (t *Tester) Tick(ctx context.Context) {
// Refusal is why a test on request did not run (RunNow). Code is stable for the controller.
type Refusal struct {
Code string // not-due | outside-window | tries-used | no-tunnel | heavy-op | not-quiet
Msg string
}
func (r *Refusal) Error() string { return "linetest: " + r.Code + ": " + r.Msg }
// ready applies every rule except the quiet one: due this week, inside the window (the slot on the due day), tries
// left today, the tunnel up. It answers the server address and the local day.
func (t *Tester) ready() (addr, day string, refusal *Refusal) {
lt := t.now().In(t.loc)
t.mu.Lock()
st := t.st
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if !Due(st, lt) {
return
return "", "", &Refusal{"not-due", "measured this week"}
}
anchor := weekAnchor(lt)
anchorDay := lt.YearDay() == anchor.YearDay() && lt.Year() == anchor.Year()
if !EligibleAt(lt, t.slot, anchorDay) {
return
return "", "", &Refusal{"outside-window", "a test starts 10:00-15:59 local, from this host's slot on the due day"}
}
day := lt.Format("2006-01-02")
day = lt.Format("2006-01-02")
if st.TriesDay == day && st.TriesToday >= MaxTriesPerDay {
return
return "", "", &Refusal{"tries-used", fmt.Sprintf("%d tries today", st.TriesToday)}
}
ip, ok := t.target()
if !ok {
t.logger.Debug("linetest: no operational tunnel — no test")
return "", "", &Refusal{"no-tunnel", "no operational tunnel"}
}
return netip.AddrPortFrom(ip, ServerPort).String(), day, nil
}
// Tick is one look.
func (t *Tester) Tick(ctx context.Context) {
addr, day, refusal := t.ready()
if refusal != nil {
if refusal.Code == "no-tunnel" {
t.logger.Debug("linetest: no operational tunnel — no test")
}
return
}
t.mu.Lock()
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if q, worst := quiet(samples); !q {
t.logger.Debug("linetest: uplink not quiet — waiting", "worst_minute_mbps", fmt.Sprintf("%.1f", worst), "samples", len(samples))
return
@@ -292,16 +323,80 @@ func (t *Tester) Tick(ctx context.Context) {
return
}
defer release()
t.measure(ctx, netip.AddrPortFrom(ip, ServerPort).String(), day)
t.measure(ctx, addr, day, RunGap)
}
func (t *Tester) measure(ctx context.Context, addr, day string) {
// WaitingForQuiet reports that a measurement is due and could start now but the uplink has not been quiet for
// QuietMinutes — the case a seeding torrent app makes permanent (R-1250). The controller reads it from
// GET /host/line-test and only then holds its app's limit and calls RunNow.
func (t *Tester) WaitingForQuiet() bool {
if _, _, refusal := t.ready(); refusal != nil {
return false
}
t.mu.Lock()
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if len(samples) < QuietMinutes+1 {
return false // not yet watched for 10 minutes (a fresh start): not known to be busy, so no hold
}
q, _ := quiet(samples)
return !q
}
// RunNow is a test on request (POST /host/line-test/run, R-1250): every rule of the weekly test except the 10 quiet
// minutes, which the caller replaces by holding its own upload down. The uplink must still be quiet over
// RequestQuietWindow (after RequestSettle), so a request never measures the family's upload against itself. A
// *Refusal says why nothing ran; any other error is a failed measurement (the same as a failed weekly attempt).
func (t *Tester) RunNow(ctx context.Context) (mbps float64, runs []float64, err error) {
addr, day, refusal := t.ready()
if refusal != nil {
return 0, nil, refusal
}
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
return 0, nil, &Refusal{"heavy-op", busy}
}
defer release()
if t.uplink == nil {
return 0, nil, &Refusal{"not-quiet", "no uplink counter"}
}
if !t.sleep(ctx, RequestSettle) {
return 0, nil, ctx.Err()
}
tx0, err0 := t.uplink()
at0 := t.now()
if !t.sleep(ctx, RequestQuietWindow) {
return 0, nil, ctx.Err()
}
tx1, err1 := t.uplink()
if err0 != nil || err1 != nil || tx1 < tx0 {
return 0, nil, &Refusal{"not-quiet", "uplink counter unreadable or reset"}
}
secs := t.now().Sub(at0).Seconds()
if secs <= 0 {
secs = RequestQuietWindow.Seconds()
}
if m := float64(tx1-tx0) * 8 / secs / 1e6; m >= QuietMbps {
t.logger.Info("linetest: test on request refused — the uplink is not quiet", "mbps", fmt.Sprintf("%.1f", m))
return 0, nil, &Refusal{"not-quiet", fmt.Sprintf("the uplink sent %.1f Mbit/s while held", m)}
}
t.logger.Info("linetest: test on request (a held torrent app, R-1250)", "server", addr)
if err := t.measure(ctx, addr, day, RequestRunGap); err != nil {
return 0, nil, err
}
t.mu.Lock()
defer t.mu.Unlock()
return t.st.Mbps, append([]float64(nil), t.st.RunsMbps...), nil
}
// measure runs the runs, gap apart, and stores the result. It returns nil only when a value was stored.
func (t *Tester) measure(ctx context.Context, addr, day string, gap time.Duration) error {
start := t.now()
var runs []float64
var lastErr error
busy := false
for i := 0; i < RunsPerMeasurement; i++ {
if i > 0 && !t.sleep(ctx, RunGap) {
if i > 0 && !t.sleep(ctx, gap) {
break
}
rctx, cancel := context.WithTimeout(ctx, RunTimeout)
@@ -356,6 +451,13 @@ func (t *Tester) measure(ctx context.Context, addr, day string) {
if err := t.saveLocked(); err != nil {
t.logger.Warn("linetest: state save failed — the result will not survive a restart", "path", t.statePath, "err", err)
}
if len(runs) > 0 {
return nil
}
if lastErr == nil {
lastErr = errors.New("linetest: no run succeeded")
}
return lastErr
}
// Value is the measured line (ok=false: never measured).
+141
View File
@@ -431,3 +431,144 @@ func TestQuietNeedsTenFullMinutes(t *testing.T) {
t.Fatal("a gap in the samples must not read quiet")
}
}
// ── a test on request (R-1250): a torrent app that seeds all day ───────────────────────────────────────────────────
// seedingRig: due and eligible, but every minute of the last ten sent at the line's speed — the weekly Tick never runs.
func seedingRig(t *testing.T, outs ...runOut) *rig {
r := newRig(t, outs...)
for i := 0; i < QuietMinutes+1; i++ {
r.minute(600_000_000) // 80 Mbit/s: a seed at the line's speed
}
return r
}
func TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt(t *testing.T) {
r := seedingRig(t, runOut{mbps: 88}, runOut{mbps: 92}, runOut{mbps: 90})
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("the weekly Tick tested a seeding box")
}
if !r.t.WaitingForQuiet() {
t.Fatal("a due, eligible, seeding box must report waiting_for_quiet")
}
var slept []time.Duration
r.t.sleep = func(_ context.Context, d time.Duration) bool { slept = append(slept, d); return true }
mbps, runs, err := r.t.RunNow(context.Background()) // the controller holds the app: the counter stands still
if err != nil || mbps != 92 || len(runs) != RunsPerMeasurement || r.run.calls != RunsPerMeasurement {
t.Fatalf("RunNow = %.1f %v %v, calls %d; want the best of 3 = 92", mbps, runs, err, r.run.calls)
}
want := []time.Duration{RequestSettle, RequestQuietWindow, RequestRunGap, RequestRunGap}
if len(slept) != len(want) {
t.Fatalf("slept %v, want %v", slept, want)
}
var total time.Duration
for i := range want {
if slept[i] != want[i] {
t.Fatalf("slept %v, want %v", slept, want)
}
total += slept[i]
}
if total+RunsPerMeasurement*TestTime > 60*time.Second {
t.Fatalf("a request holds the app %v + runs; it must end inside 60 s", total)
}
if v, _, ok := r.t.Value(); !ok || v != 92 {
t.Fatalf("the requested measurement was not stored: %.1f %v", v, ok)
}
if r.t.WaitingForQuiet() {
t.Fatal("measured this week: no longer waiting")
}
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-due" {
t.Fatalf("a second request this week: %v, want not-due", err)
}
}
func refusalCode(err error) string {
var ref *Refusal
if errors.As(err, &ref) {
return ref.Code
}
return ""
}
// The held app is not the only uploader: a request still refuses to measure a busy uplink.
func TestRunNowRefusesAnUplinkThatIsStillBusy(t *testing.T) {
r := seedingRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.t.uplink = func() (uint64, error) { r.tx += 20_000_000; return r.tx, nil } // 20 MB per 10 s = 16 Mbit/s
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-quiet" || r.run.calls != 0 {
t.Fatalf("RunNow on a busy uplink: %v, calls %d; want not-quiet and no run", err, r.run.calls)
}
}
// Every other rule of the weekly test holds for a request.
func TestRunNowKeepsTheWeeklyRules(t *testing.T) {
r := seedingRig(t)
r.gate.busy = "backup:felhom-pbs"
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "heavy-op" {
t.Fatalf("beside a backup: %v, want heavy-op", err)
}
r.gate.busy = ""
r.tunnel = false
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "no-tunnel" {
t.Fatalf("no tunnel: %v, want no-tunnel", err)
}
if r.t.WaitingForQuiet() {
t.Fatal("no tunnel: a request could not run, so it must not be invited")
}
r.tunnel = true
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "outside-window" {
t.Fatalf("at night: %v, want outside-window", err)
}
if r.run.calls != 0 {
t.Fatalf("%d runs on refused requests", r.run.calls)
}
}
// A failed request counts as a try, so a controller cannot request more than MaxTriesPerDay tests a day.
func TestRunNowFailuresSpendTheDaysTries(t *testing.T) {
var outs []runOut
for i := 0; i < MaxTriesPerDay*RunsPerMeasurement; i++ {
outs = append(outs, runOut{err: errors.New("connection refused")})
}
r := seedingRig(t, outs...)
for i := 0; i < MaxTriesPerDay; i++ {
if _, _, err := r.t.RunNow(context.Background()); err == nil || refusalCode(err) != "" {
t.Fatalf("try %d: %v, want a failed measurement", i+1, err)
}
}
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "tries-used" {
t.Fatalf("after %d failures: %v, want tries-used", MaxTriesPerDay, err)
}
}
// A quiet box tests itself on the weekly Tick: it must not invite the controller to hold its app.
func TestQuietBoxIsNotWaiting(t *testing.T) {
r := newRig(t)
if r.t.WaitingForQuiet() {
t.Fatal("ten quiet minutes: the weekly Tick runs; waiting_for_quiet must be false")
}
}
// Live proof 2026-10-24 (demo-hp): a freshly started tester answered waiting_for_quiet before it had watched a single
// minute, so a quiet box's torrent app would be held for nothing. Waiting needs the full window, seen busy.
func TestFreshTesterIsNotWaiting(t *testing.T) {
r := &rig{run: &fakeRunner{}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
r.t.loc = time.UTC
r.t.now = func() time.Time { return r.now }
if r.t.WaitingForQuiet() {
t.Fatal("no minute watched yet: waiting_for_quiet must be false")
}
for i := 0; i < QuietMinutes; i++ {
r.minute(600_000_000)
}
if r.t.WaitingForQuiet() {
t.Fatal("10 samples are 9 minutes: still not a full window")
}
r.minute(600_000_000)
if !r.t.WaitingForQuiet() {
t.Fatal("a full busy window: waiting_for_quiet must be true")
}
}
+8 -15
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"net/http"
"os"
"path/filepath"
@@ -145,9 +146,11 @@ func (s *Server) validateBackupTargetMount(ctx context.Context, where string) er
// begin with. A typed round-trip would silently drop any key this build does not know about.
//
// Written IN PLACE (O_TRUNC), not tmp+rename: /etc/felhom-agent is root-owned while agent.json is
// agent-owned 0600, so the non-root agent cannot rename into that directory. A recovery copy is
// parked first, so a torn write is recoverable.
// agent-owned 0600, so the non-root agent cannot rename into that directory. config.ReplaceInPlace
// journals the old bytes first; the next start repairs a cut write (TestRecoverCutWrite_*).
func (s *Server) setConfiguredBackupTarget(id string) error {
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
defer config.FileMu.Unlock()
path := s.configPath
if path == "" {
return fmt.Errorf("no config path known to this agent (env-only config)")
@@ -179,24 +182,14 @@ func (s *Server) setConfiguredBackupTarget(id string) error {
if err != nil {
return err
}
st, err := os.Stat(path)
if err != nil {
return err
}
if s.stateDir != "" {
if err := os.MkdirAll(s.stateDir, 0o700); err == nil {
_ = os.WriteFile(filepath.Join(s.stateDir, "agent.json.pre-backup-target"), raw, 0o600)
}
}
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
return err
}
if _, err := f.Write(out); err != nil {
f.Close()
return err
}
return f.Close()
// config.ReplaceInPlace: the old bytes go into a journal first, so a power cut in the middle of the write is
// repaired at the next start (R-1380).
return config.ReplaceInPlace(path, raw, out)
}
// wrapperReason surfaces the wrapper's own REFUSED line when it produced one — it explains WHY in
+165 -12
View File
@@ -3,6 +3,7 @@ package localapi
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
@@ -83,6 +84,20 @@ const (
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
// the parked marker.
controllerSlowCounterFile = "controller-restarts-24h.json"
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
// the hub (controller_unresponsive). Not counted during a swap, a parked guest, a locked guest or a vzdump.
controllerUnresponsiveConfirm = 6
// controllerUnresponsiveRestart (decided 2026-10-30 from the measurement): at this many consecutive unhealthy sweeps
// (5 minutes after Docker's „unhealthy", ~6.5 minutes after the first failed probe; 2 minutes after the alarm) the
// agent restarts the controller — the not-running restart's guards and brake, plus the controller's busy file. The
// numbers: Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, 4 733 probes through the night chain (a
// PBS and a local vzdump, the DB dumps, tier-2, the off-site push, a restore test, two controller updates): no failed
// probe, slowest 303 ms against the 5 s timeout. A busy night never came near even ONE failed probe; ten sweeps is
// the margin. Pinned by TestSupervisor_UnresponsiveIsAlarmedThenRestarted.
controllerUnresponsiveRestart = 10
)
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
@@ -103,6 +118,18 @@ type controllerSupState struct {
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
restarts24h []time.Time
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
// opBrakeUntil (2026-10-29): an operator restart refused by the brake refuses every press until then (30 min).
opBrakeUntil time.Time
// R-1344: consecutive running-but-unhealthy sweeps, and the current episode (since kept after it ends).
unhealthySeen int
unresponsive bool
unresponsiveSince time.Time
// unresponsiveRestartAt: the last restart for not answering (kept; the hub keys on it MOVING). unrespRefusal: the
// last reason the restart was refused, so a refusal is logged once per change, not every sweep.
unresponsiveRestartAt time.Time
unrespRefusal string
}
type controllerSupervisor struct {
@@ -238,6 +265,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
if !s.provisionedGuest(g.VMID) {
continue
}
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
s.resetNotRunning(g.VMID)
continue
}
evaluated++
if !s.superviseOneController(ctx, g.VMID, g.Status) {
down++
@@ -293,6 +324,7 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
if running {
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.observeControllerHealth(ctx, vmid, parked)
return true
}
st.notRunningSeen++
@@ -333,9 +365,27 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
return false
}
// Backoff.
if !s.supervisorBrakeAllows(vmid, now) {
return false
}
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
"vmid", vmid, "status", status, "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
s.recordSupervisorRestart(vmid, now, reason, nil)
return false
}
// supervisorBrakeAllows is the brake every supervisor restart passes (the not-running one and R-1344's): a crash-loop
// pause in force refuses; controllerCrashloopMax restarts inside controllerCrashloopWindow start one (logged, carried by
// the stanza, mailed by the hub as controller_crashloop) and refuse.
func (s *Server) supervisorBrakeAllows(vmid int, now time.Time) bool {
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st := s.supState(vmid)
if !st.crashloopSince.IsZero() {
if now.Sub(st.crashloopSince) < controllerCrashloopPause {
s.ctrlSup.mu.Unlock()
@@ -358,22 +408,23 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
return false
}
s.ctrlSup.mu.Unlock()
return true
}
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
"vmid", vmid, "status", status, "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
// recordSupervisorRestart puts a restart the supervisor performed on the record: the brake's window, the totals, the
// reason, and R-539's slow counter (persisted; raised at most once per 24 hours — the hub mails on the raise MOVING).
// mark, when set, runs under the same lock (R-1344 stamps its own record there).
func (s *Server) recordSupervisorRestart(vmid int, now time.Time, reason string, mark func(st *controllerSupState)) {
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st := s.supState(vmid)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = reason
st.notRunningSeen = 0
// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING.
if mark != nil {
mark(st)
}
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
n24 := len(st.restarts24h)
raised := false
@@ -389,7 +440,101 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
}
return false
}
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
// the swap's `docker inspect -f` grant. Anything but „unhealthy" — healthy, starting, no health check, a read that
// fails — resets the count; so do a swap, a parked guest, a locked guest and a vzdump (each is someone else's moment).
func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked bool) {
s.swapMu.Lock()
swapping := s.swapInFlight[vmid]
s.swapMu.Unlock()
unhealthy := false
if !parked && !swapping {
if out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer); err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
unhealthy = len(f) == 2 && f[0] == "true" && f[1] == "unhealthy"
}
}
if unhealthy {
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil || lock != "" {
unhealthy = false
} else if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
unhealthy = false
}
}
now := s.clock()
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if !unhealthy {
was := st.unresponsive
st.unhealthySeen, st.unresponsive = 0, false
s.ctrlSup.mu.Unlock()
if was {
s.logger.Info("controller-supervisor: the controller answers its health check again", "vmid", vmid)
}
return
}
st.unhealthySeen++
seen := st.unhealthySeen
start := seen >= controllerUnresponsiveConfirm && !st.unresponsive
if start {
st.unresponsive, st.unresponsiveSince = true, now
}
since := st.unresponsiveSince
s.ctrlSup.mu.Unlock()
if start {
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so",
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String(),
"restart_after_sweeps", controllerUnresponsiveRestart)
}
if seen >= controllerUnresponsiveRestart {
s.restartUnresponsive(ctx, vmid, seen, since, now)
}
}
// restartUnresponsive is R-1344's restart (decided 2026-10-30): the controller's busy file (fresh = a backup, a
// restore, an update or an export is running; unreadable = cannot be ruled out) refuses, then the brake every
// supervisor restart passes. The swap, the parked marker, the lock and the vzdump never reach here — they reset the
// count. A refusal keeps the alarm and is logged once per reason; the next sweep asks again.
func (s *Server) restartUnresponsive(ctx context.Context, vmid, seen int, since, now time.Time) {
refuse := func(why string) {
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
changed := st.unrespRefusal != why
st.unrespRefusal = why
s.ctrlSup.mu.Unlock()
if changed {
s.logger.Warn("controller-supervisor: the controller does not answer, but the restart is refused — the alarm stays",
"vmid", vmid, "why", why, "unhealthy_sweeps", seen)
}
}
busy, what, err := s.controllerBusy(ctx, vmid)
if err != nil {
refuse("the controller's busy file cannot be read: " + err.Error())
return
}
if busy {
refuse("the controller is busy: " + what)
return
}
if !s.supervisorBrakeAllows(vmid, now) {
refuse("the restart brake")
return
}
reason := fmt.Sprintf("the controller ran but did not answer its health check for %d sweeps (since %s)",
seen, since.UTC().Format(time.RFC3339))
s.logger.Warn("controller-supervisor: the controller does not answer — restarting the bootstrap unit",
"vmid", vmid, "unhealthy_sweeps", seen, "since", since.UTC().Format(time.RFC3339), "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
s.recordSupervisorRestart(vmid, now, reason, func(st *controllerSupState) {
st.unresponsiveRestartAt = now
st.unhealthySeen = 0
st.unrespRefusal = ""
})
}
func (s *Server) resetNotRunning(vmid int) {
@@ -446,6 +591,14 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
}
g.Unresponsive = st.unresponsive
if !st.unresponsiveSince.IsZero() {
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
}
g.UnresponsiveRestartAfterS = (controllerUnresponsiveRestart - controllerUnresponsiveConfirm) * int(controllerSupervisorInterval/time.Second)
if !st.unresponsiveRestartAt.IsZero() {
g.UnresponsiveRestartAt = st.unresponsiveRestartAt.UTC().Format(time.RFC3339)
}
out.Guests = append(out.Guests, g)
}
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
+21
View File
@@ -0,0 +1,21 @@
package localapi
import (
"os"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
// agent's state folder on a box, and a test must not need it.
func TestMain(m *testing.M) {
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
if err != nil {
panic(err)
}
config.CutWriteDir = dir
code := m.Run()
_ = os.RemoveAll(dir)
os.Exit(code)
}
+9 -3
View File
@@ -87,6 +87,11 @@ type GuestAttacher interface {
// GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a
// controller-only restart — the deterministic guest-reboot signal the controller recreates apps on.
GuestBootID(ctx context.Context, vmid int) string
// GuestBootKey is the same signal built from the kernel's boot_id, which a step of the host's clock cannot
// move (R-1395). "" = not readable; the controller then reads GuestBootID.
GuestBootKey(ctx context.Context, vmid int) string
// GuestBootTokens returns (GuestBootID, GuestBootKey) from one read — what /disks sends.
GuestBootTokens(ctx context.Context, vmid int) (id, key string)
// AttachBind is the LEGACY per-drive `pct set -mpN` bind (pre-intermediary). Retained for the
// transition; new attaches use AttachDrive.
AttachBind(ctx context.Context, vmid int, mountKey, where string) error
@@ -431,11 +436,12 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// Guest boot-id (intermediary model): changes on every guest boot, stable across controller restarts.
// The controller persists it and deterministically recreates drive-backed apps when it changes.
bootID := ""
// R-1395: guest_boot_key is the clock-proof form (kernel boot_id); guest_boot_id stays for older controllers.
bootID, bootKey := "", ""
if s.guestAttach != nil {
bootID = s.guestAttach.GuestBootID(r.Context(), vmid)
bootID, bootKey = s.guestAttach.GuestBootTokens(r.Context(), vmid)
}
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID})
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID, "guest_boot_key": bootKey})
}
// handleDiskCandidates (Impl-2a) lists the host disks FREE for Felhom to enroll — the raw-device
+4
View File
@@ -466,6 +466,10 @@ func (f *fakeGuestAttacher) AttachDrive(_ context.Context, _ int, where string)
}
func (f *fakeGuestAttacher) GuestSeesMount(_ context.Context, _ int, _ string) bool { return true }
func (f *fakeGuestAttacher) GuestBootID(_ context.Context, _ int) string { return "boot-1" }
func (f *fakeGuestAttacher) GuestBootKey(_ context.Context, _ int) string { return "key-1" }
func (f *fakeGuestAttacher) GuestBootTokens(_ context.Context, _ int) (string, string) {
return "boot-1", "key-1"
}
func (f *fakeGuestAttacher) attachDriveCount() int {
f.mu.Lock()
defer f.mu.Unlock()
+76 -2
View File
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
Upload func(ctx context.Context, pkg []byte) error
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
}
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
// ceremony incl. its upload was measured at ~2.4 s).
const escrowUploadTimeout = 30 * time.Second
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
const escrowHoldTimeout = 15 * time.Second
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
// or late may have been stored all the same.
type uploadOutcome int
const (
uploadStored uploadOutcome = iota // the hub holds the new package
uploadNotStored // the hub answered and holds another package: the previous code still works
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
)
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
// still works", which is false when the lost answer came after the store.
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
if s.escrowCeremony.HubHolds == nil {
return uploadUnknown
}
base := s.baseCtx
if base == nil {
base = context.Background()
}
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
defer cancel()
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
switch {
case err != nil:
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
return uploadUnknown
case held:
return uploadStored
default:
return uploadNotStored
}
}
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
// spike-shaped output; production is runCeremonySubprocess.
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
}
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
switch outcome {
case uploadNotStored:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
}
s.escrowPending = nil
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg)
cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
}
s.escrowMu.Lock()
s.escrowClaiming = false
if err != nil {
switch outcome {
case uploadNotStored:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
}
s.escrowPending = nil
s.wipeEscrowConfirmLocked()
+22 -3
View File
@@ -41,9 +41,13 @@ type fakeEscrowHub struct {
mu sync.Mutex
current []byte
uploads int
err error // non-nil → the upload fails, nothing stored
gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts
err error // non-nil → the upload fails, nothing stored
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
// holdsErr → the hub cannot be asked which package it holds.
lost bool
holdsErr error
gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts
}
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
}
f.current = append([]byte(nil), pkg...)
f.uploads++
if f.lost {
return context.DeadlineExceeded
}
return nil
}
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
// blob the hub serves back).
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.holdsErr != nil {
return false, f.holdsErr
}
return string(f.current) == string(pkg), nil
}
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
func (f *fakeEscrowHub) opensWith(r string) bool {
f.mu.Lock()
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
HubConfigured: true,
DRConfigured: func() bool { return true },
Upload: hub.upload,
HubHolds: hub.holds,
}
srv.ceremonyRun = run
srv.escrowSudoCheck = func(context.Context) error { return nil }
+98
View File
@@ -0,0 +1,98 @@
package localapi
import (
"errors"
"net/http"
"strings"
"testing"
)
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
// - the hub holds the new package → success (the code is handed out / the confirm completes);
// - the hub holds another package → the old sentence, now true;
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
// hub holds the new package → FAILS.
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if !hub.opensWith(firstR) {
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
}
if code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("the household was told the previous code still works: %s", body)
}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
t.Fatalf("the job must read uploaded: %s", st)
}
}
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusGatewayTimeout {
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("an unknown outcome promised the previous code: %s", body)
}
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
t.Fatalf("the code must stay claimable: %s", st)
}
// The hub answers again: the retry finds the new package already stored and hands the code out.
hub.mu.Lock()
hub.holdsErr = nil
hub.mu.Unlock()
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("retry: %d %s", code, body)
}
}
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
}
}
// The same at the typed-back confirm (R-1151).
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
}
hub.mu.Lock()
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
hub.mu.Unlock()
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h2 := srv2.Handler()
startAndWait(t, srv2, h2)
c2 := claimDeferred(t, h2)
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
}
}
+116
View File
@@ -0,0 +1,116 @@
package localapi
import (
"context"
"encoding/json"
"io"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
type initPIDRunner struct{ pid string }
func (r initPIDRunner) Run(_ context.Context, name string, _ ...string) ([]byte, []byte, error) {
if name == "lxc-info" {
return []byte(r.pid + "\n"), nil, nil
}
return nil, nil, nil
}
func (r initPIDRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
// R-1395: on 2026-11-14 the N100's time service came back and stepped the host's clock by +32.7 s. The kernel
// computes the boot TIME from the wall clock, so guest_boot_id moved and the controller recreated every drive-backed
// app. The consequence asserted: the boot KEY does not move when the boot time moves; it does move on a host reboot
// and on a guest reboot; and it is "" (so the controller falls back to the old token) when the boot_id is unreadable.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt).
func TestGuestBootKey_AClockStepDoesNotMoveIt(t *testing.T) {
oldB, oldK, oldS := hostBtimeFn, hostBootIDFn, procStarttimeFn
t.Cleanup(func() { hostBtimeFn, hostBootIDFn, procStarttimeFn = oldB, oldK, oldS })
btime, bootID, start := "1794112518", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
hostBtimeFn = func() string { return btime }
hostBootIDFn = func() string { return bootID }
procStarttimeFn = func(pid string) string {
if pid != "31337" {
t.Errorf("start time read for pid %q, want the guest's init", pid)
}
return start
}
b := NewGuestBinder(initPIDRunner{"31337"}, slog.New(slog.NewTextHandler(io.Discard, nil)))
ctx := context.Background()
id0, key0 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
if id0 != "1794112518-4242" || key0 != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" {
t.Fatalf("tokens = %q / %q", id0, key0)
}
btime = "1794112550" // THE STEP: the wall clock jumped +32 s, so the kernel's btime did too
id1, key1 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
if id1 == id0 {
t.Fatalf("control: the old token did not move with the boot time (%q) — the test does not stage the defect", id1)
}
if key1 != key0 {
t.Fatalf("the boot key moved with the clock: %q → %q — a clock step reads as a new boot", key0, key1)
}
if strings.Contains(key1, "1794") {
t.Errorf("the boot key carries the boot time: %q", key1)
}
bootID = "c0ffee00-1111-2222-3333-444455556666" // a host reboot
if k := b.GuestBootKey(ctx, 9201); k == key0 {
t.Errorf("a host reboot (new boot_id) did not change the key: %q", k)
}
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "991234" // a guest reboot
if k := b.GuestBootKey(ctx, 9201); k == key0 || !strings.HasSuffix(k, "-991234") {
t.Errorf("a guest reboot (new init start) did not change the key: %q", k)
}
for _, bad := range []string{"", "not-a-boot-id", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11\nx"} {
bootID = bad
if k := b.GuestBootKey(ctx, 9201); k != "" {
t.Errorf("boot_id %q gave the key %q, want \"\" (the controller then reads the old token)", bad, k)
}
if id := b.GuestBootID(ctx, 9201); id == "" {
t.Errorf("the old token must still be sent when the boot_id is unreadable")
}
}
// what /disks sends: both tokens from ONE read of the guest's init — the init is asked for once, not twice
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
reads := 0
procStarttimeFn = func(string) string { reads++; return start }
if id, key := b.GuestBootTokens(ctx, 9201); id != "1794112550-4242" || key != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" || reads != 1 {
t.Errorf("GuestBootTokens = %q, %q after %d read(s) of the init — want both tokens from ONE read", id, key, reads)
}
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "" // the guest does not run
if k := b.GuestBootKey(ctx, 9201); k != "" {
t.Errorf("a guest that does not run gave the key %q", k)
}
}
// The key is no use if /disks does not carry it: both tokens are on the wire, the old one unchanged for controllers
// that do not know the key.
func TestDisks_CarriesBothBootTokens(t *testing.T) {
srv := targetRowServer(t, "", []hub.StorageTarget{})
srv.guestAttach = &fakeGuestAttacher{}
body := do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()
var w struct {
Data struct {
ID string `json:"guest_boot_id"`
Key string `json:"guest_boot_key"`
} `json:"data"`
ID string `json:"guest_boot_id"`
Key string `json:"guest_boot_key"`
}
if err := json.Unmarshal(body, &w); err != nil {
t.Fatalf("decode: %v (%s)", err, body)
}
if w.Data.ID+w.ID != "boot-1" || w.Data.Key+w.Key != "key-1" {
t.Fatalf("/disks boot tokens: id=%q key=%q, want boot-1 and key-1 (%s)", w.Data.ID+w.ID, w.Data.Key+w.Key, body)
}
}
// The real binder is what main.go hands the server: it must have the method (the interface makes this a build
// error if not — stated here so the reason is on record).
var _ GuestAttacher = (*GuestBinder)(nil)
+102
View File
@@ -0,0 +1,102 @@
package localapi
import (
"bufio"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
)
// R-528 — the OOM signal from the host side.
//
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
//
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
// cgroup folder (a VM guest, or a container that is not running).
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
if root == "" {
root = "/sys/fs/cgroup"
}
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
return nil, false, nil
}
out := map[string]int64{}
for _, pattern := range []string{
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
filepath.Join(base, "docker", "*", "memory.events"),
} {
files, err := filepath.Glob(pattern)
if err != nil {
return nil, true, err
}
for _, f := range files {
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
if !isContainerID(id) {
continue
}
n, err := readOOMKill(f)
if err != nil {
continue // a container that stopped between the glob and the read
}
out[id] = n
}
}
return out, true, nil
}
func isContainerID(s string) bool {
if len(s) != 64 {
return false
}
for _, c := range s {
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
return false
}
}
return true
}
func readOOMKill(path string) (int64, error) {
f, err := os.Open(path)
if err != nil {
return 0, err
}
defer f.Close()
sc := bufio.NewScanner(f)
for sc.Scan() {
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
if ok && k == "oom_kill" {
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
}
}
if err := sc.Err(); err != nil {
return 0, err
}
return 0, fmt.Errorf("no oom_kill line in %s", path)
}
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
if err != nil {
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
return
}
if kills == nil {
kills = map[string]int64{}
}
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
}
+101
View File
@@ -0,0 +1,101 @@
package localapi
import (
"encoding/json"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
)
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
// serves the asking guest's counters, and only that guest's.
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
t.Helper()
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
t.Fatal(err)
}
}
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
t.Fatal(err)
}
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
if err := os.MkdirAll(dfDir, 0o755); err != nil {
t.Fatal(err)
}
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
got, avail, err := readGuestOOMKills(root, 100)
if err != nil || !avail {
t.Fatalf("read: avail=%v err=%v", avail, err)
}
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
if len(got) != len(want) {
t.Fatalf("got %v, want %v", got, want)
}
for k, v := range want {
if got[k] != v {
t.Fatalf("got %v, want %v", got, want)
}
}
if _, leaked := got[idOther]; leaked {
t.Fatal("another guest's container leaked into this guest's answer")
}
// A guest with no cgroup folder (a VM, or not running): not available, no error.
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
}
}
// Through the route, with the guest's own token.
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.cgroupRoot = root
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
if w.Code != http.StatusOK {
t.Fatalf("route: %d %s", w.Code, w.Body.String())
}
var env struct {
Data struct {
VMID int `json:"vmid"`
Available bool `json:"available"`
Containers map[string]int64 `json:"containers"`
} `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
t.Fatalf("no token: %d", w.Code)
}
}
+56
View File
@@ -3,6 +3,7 @@ package localapi
import (
"context"
"log/slog"
"sort"
"sync"
"time"
@@ -82,6 +83,40 @@ type guestPowerState struct {
attempts int
nextAt time.Time
raised bool // the give-up fault has already been raised for this run
raisedAt time.Time
}
// GuestPowerGiveUp is one guest the watchdog stopped trying to start (R-1383, D78). It rides the host report
// (`guest_power_gave_up`) so the operator hears it: until this, the give-up was one ERROR line in the host's journal
// while the household's apps were off. Memory only — an agent restart forgets it (and tries three times again).
type GuestPowerGiveUp struct {
VMID int `json:"vmid"`
Since string `json:"since"` // RFC3339 UTC, when the watchdog gave up
Attempts int `json:"attempts"`
}
// GuestPowerGaveUpStanza is GuestPowerGaveUp shaped for the host report: an UNTYPED nil when there is none, so the
// report omits the field (a typed nil slice in an `any` would be sent as `null`).
func (s *Server) GuestPowerGaveUpStanza() any {
if g := s.GuestPowerGaveUp(); len(g) > 0 {
return g
}
return nil
}
// GuestPowerGaveUp returns the guests the watchdog has given up on, by vmid. nil when there is none — the report
// stanza is then omitted. Pinned by TestGuestPower_GiveUpIsReportedUntilTheGuestRuns.
func (s *Server) GuestPowerGaveUp() []GuestPowerGiveUp {
guestPowerMu.Lock()
defer guestPowerMu.Unlock()
var out []GuestPowerGiveUp
for vmid, st := range s.guestPower {
if st.raised {
out = append(out, GuestPowerGiveUp{VMID: vmid, Since: st.raisedAt.UTC().Format(time.RFC3339), Attempts: st.attempts})
}
}
sort.Slice(out, func(i, j int) bool { return out[i].VMID < out[j].VMID })
return out
}
// WatchGuestPower runs the guest-power sweep every guestPowerInterval until ctx is done. No-op when
@@ -117,6 +152,19 @@ func (s *Server) GuestPowerTick(ctx context.Context) {
s.logger.Warn("guest-power: guest list unavailable — skipping sweep (ownership unproven)", "err", err)
return
}
// A guest that left the list (destroyed, or re-made under another number) takes its attempt record with it — a
// give-up for a guest that no longer exists would ride every host report until the agent restarts.
present := make(map[int]bool, len(guests))
for _, g := range guests {
present[g.VMID] = true
}
guestPowerMu.Lock()
for vmid := range s.guestPower {
if !present[vmid] {
delete(s.guestPower, vmid)
}
}
guestPowerMu.Unlock()
var stopped int
for _, g := range guests {
if ctx.Err() != nil {
@@ -152,6 +200,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
return
}
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
// would fail the reboot's own start half).
if s.rebooting(g.VMID) {
return
}
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
if err != nil {
@@ -162,6 +215,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
// SCENARIO B — a deliberately stopped guest is left alone, forever. onboot:0 means the operator
// (or the golden-image provisioning) does not want this guest running.
if !onboot {
s.forgetGuestPower(g.VMID) // not ours to start any more: a give-up for it must not be reported for ever
return
}
@@ -171,6 +225,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
if lock != "" {
s.logger.Info("guest-power: guest is stopped but LOCKED — leaving it to the stale-lock path",
"vmid", g.VMID, "lock", lock)
s.forgetGuestPower(g.VMID) // another operation owns it now; if it ends stopped, the three tries start again
return
}
@@ -255,6 +310,7 @@ func (s *Server) markGuestPowerRaised(vmid int) {
defer guestPowerMu.Unlock()
st := s.guestPower[vmid]
st.raised = true
st.raisedAt = s.now()
s.guestPower[vmid] = st
}
+79
View File
@@ -2,6 +2,7 @@ package localapi
import (
"context"
"encoding/json"
"errors"
"log/slog"
"testing"
@@ -237,3 +238,81 @@ func TestGuestPower_RunningGuestClearsAttemptHistory(t *testing.T) {
t.Error("attempt history survived the guest coming back healthy")
}
}
// R-1383 (D78): the give-up was ONE journal line while the household's apps were off. It now rides the host report:
// nothing before the give-up (and an untyped nil, so the report omits the field), the guest with the time and the
// attempt count after it, and nothing again once the guest runs.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-guest-power.txt).
func TestGuestPower_GiveUpIsReportedUntilTheGuestRuns(t *testing.T) {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
startErr: errors.New("cannot start: storage offline"),
}
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
s := gpServer(t, ctl, func() time.Time { return now })
if st := s.GuestPowerGaveUpStanza(); st != nil {
t.Fatalf("before any attempt the stanza is %#v, want an untyped nil (the report omits the field)", st)
}
for i := 0; i < guestPowerMaxAttempts; i++ { // the three failed starts
s.GuestPowerTick(context.Background())
if g := s.GuestPowerGaveUp(); len(g) != 0 {
t.Fatalf("after %d failed start(s) the watchdog still tries, but it reports a give-up: %+v", i+1, g)
}
now = now.Add(10 * time.Minute)
}
gaveUpAt := now
s.GuestPowerTick(context.Background()) // the tick that gives up
now = now.Add(time.Hour)
s.GuestPowerTick(context.Background())
g := s.GuestPowerGaveUp()
if len(g) != 1 || g[0].VMID != 9201 || g[0].Attempts != guestPowerMaxAttempts || g[0].Since != gaveUpAt.Format(time.RFC3339) {
t.Fatalf("give-up list = %+v, want guest 9201, %d attempts, since %s (the give-up's time, not a later tick's)", g, guestPowerMaxAttempts, gaveUpAt.Format(time.RFC3339))
}
b, _ := json.Marshal(s.GuestPowerGaveUpStanza())
if string(b) != `[{"vmid":9201,"since":"2026-11-14T02:30:00Z","attempts":3}]` {
t.Fatalf("the stanza on the wire = %s", b)
}
// the operator starts the guest: the next sweep sees it running and the list is empty again
ctl.guests = []proxmox.Guest{{VMID: 9201, Status: "running"}}
s.GuestPowerTick(context.Background())
if st := s.GuestPowerGaveUpStanza(); st != nil {
t.Fatalf("the guest runs again and the stanza is still %#v", st)
}
}
// Found by review, 2026-11-14: the give-up was forgotten only when the guest was seen RUNNING. A guest the operator
// then switches to onboot:0, a guest another operation locks, and a guest that is destroyed would be reported as
// „given up” for ever (the hub's alarm could never clear). Each of the three ends the report.
func TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs(t *testing.T) {
for _, c := range []struct {
name string
change func(ctl *fakeGuestPowerCtl)
}{
{"onboot switched off", func(ctl *fakeGuestPowerCtl) { ctl.onboot = map[int]bool{9201: false} }},
{"locked by another operation", func(ctl *fakeGuestPowerCtl) { ctl.locks = map[int]string{9201: "backup"} }},
{"destroyed", func(ctl *fakeGuestPowerCtl) { ctl.guests = nil }},
} {
ctl := &fakeGuestPowerCtl{
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
locks: map[int]string{9201: ""},
onboot: map[int]bool{9201: true},
startErr: errors.New("cannot start"),
}
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
s := gpServer(t, ctl, func() time.Time { return now })
for i := 0; i < guestPowerMaxAttempts+1; i++ {
s.GuestPowerTick(context.Background())
now = now.Add(10 * time.Minute)
}
if len(s.GuestPowerGaveUp()) != 1 {
t.Fatalf("%s: setup — no give-up recorded", c.name)
}
c.change(ctl)
s.GuestPowerTick(context.Background())
if g := s.GuestPowerGaveUp(); len(g) != 0 {
t.Errorf("%s: the give-up is still reported: %+v", c.name, g)
}
}
}
+68 -6
View File
@@ -5,6 +5,7 @@ import (
"context"
"fmt"
"os"
"regexp"
"strconv"
"strings"
)
@@ -288,21 +289,82 @@ func (b *GuestBinder) guestInitPID(ctx context.Context, vmid int) string {
// /proc/<pid>/stat — ticks since host boot, unique per process launch) changes on a guest reboot. ""
// on any read error (the controller then keeps its last-seen → no spurious recreate).
func (b *GuestBinder) GuestBootID(ctx context.Context, vmid int) string {
pid := b.guestInitPID(ctx, vmid)
if pid == "" {
return ""
}
start := procStarttime(pid)
start := b.guestInitStart(ctx, vmid)
if start == "" {
return ""
}
bt := hostBtime()
bt := hostBtimeFn()
if bt == "" {
bt = "0"
}
return bt + "-" + start
}
// guestInitStart is the guest init's start time in ticks since the host's boot ("" = the guest does not run, or a
// read error). Both boot tokens end with it.
func (b *GuestBinder) guestInitStart(ctx context.Context, vmid int) string {
pid := b.guestInitPID(ctx, vmid)
if pid == "" {
return ""
}
return procStarttimeFn(pid)
}
// GuestBootKey (R-1395) is GuestBootID with the host's boot TIME replaced by the kernel's boot_id:
// "<boot_id>-<guest-init-starttime>". The kernel computes the boot time (`btime`) from the wall clock, so a STEP of
// the host's clock moves GuestBootID — on the N100 on 2026-11-14 the time service came back, stepped +32.7 s, and the
// controller recreated every drive-backed app as if the guest had booted. The kernel's boot_id is a random value made
// once per boot; no clock can move it. Same meaning as before: it changes on a host reboot (new boot_id) and on a
// guest reboot (new init start), and is stable across an agent or controller restart.
//
// "" when the guest does not run or the boot_id cannot be read — the controller then decides by GuestBootID, which
// is still sent (controllers that do not know the key read only that). Pinned by TestGuestBootKey_*.
func (b *GuestBinder) GuestBootKey(ctx context.Context, vmid int) string {
start := b.guestInitStart(ctx, vmid)
return guestBootKey(hostBootIDFn(), start)
}
// GuestBootTokens returns both tokens from ONE read of the guest's init: the old token and the key always describe
// the same moment. Read separately, one could succeed and the other fail (the guest stops between the two) — the
// controller would then get an old token with no key, read „an older agent”, and recreate every drive-backed app.
func (b *GuestBinder) GuestBootTokens(ctx context.Context, vmid int) (id, key string) {
start := b.guestInitStart(ctx, vmid)
if start == "" {
return "", ""
}
bt := hostBtimeFn()
if bt == "" {
bt = "0"
}
return bt + "-" + start, guestBootKey(hostBootIDFn(), start)
}
// guestBootKey joins the two halves; "" when either is missing or the boot_id is not the kernel's shape.
func guestBootKey(bootID, start string) string {
if start == "" || !kernelBootIDRe.MatchString(bootID) {
return ""
}
return bootID + "-" + start
}
var kernelBootIDRe = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
// The three reads of /proc behind the boot tokens, as seams (the tests move the boot time and keep the boot_id).
var (
hostBtimeFn = hostBtime
hostBootIDFn = hostBootID
procStarttimeFn = procStarttime
)
// hostBootID returns the kernel's boot_id (/proc/sys/kernel/random/boot_id): random, made once per boot. "" on error.
func hostBootID() string {
data, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
if err != nil {
return ""
}
return strings.TrimSpace(string(data))
}
// procStarttime returns field 22 (starttime) of /proc/<pid>/stat. The comm field (2) can contain spaces
// and parentheses, so we split AFTER the last ')': field 22 is index 19 of the post-comm fields
// (field 3 = state is index 0). "" on any error.
+55 -3
View File
@@ -1,8 +1,12 @@
package localapi
import (
"context"
"errors"
"net/http"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
)
// LineTestReader is the R-1247 weekly upload line test (internal/linetest.Tester) as the local API reads it.
@@ -10,6 +14,13 @@ type LineTestReader interface {
Value() (mbps float64, at time.Time, ok bool)
}
// LineTestRequester is the R-1250 half: a due test the uplink never lets start, and a test on request. The Tester
// satisfies it; a reader without it answers waiting_for_quiet:false and 503 on the POST.
type LineTestRequester interface {
WaitingForQuiet() bool
RunNow(ctx context.Context) (mbps float64, runs []float64, err error)
}
// LineTestResponse is GET /host/line-test. Present=false: never measured, or the job is not wired (no tunnel, disabled,
// or an agent before the route answers 404 — the controller reads all three as „unknown").
type LineTestResponse struct {
@@ -17,6 +28,19 @@ type LineTestResponse struct {
Mbps float64 `json:"mbps,omitempty"`
MeasuredAt string `json:"measured_at,omitempty"` // RFC3339 UTC
Method string `json:"method,omitempty"` // "iperf3-tunnel": a 10-second upload to ep0 inside the tunnel
// WaitingForQuiet: a measurement is due and could start now, but the uplink has not been quiet for 10 minutes
// (R-1250). The controller with a seeding torrent app then holds its limit and POSTs /host/line-test/run.
WaitingForQuiet bool `json:"waiting_for_quiet,omitempty"`
}
// LineTestRunResponse is POST /host/line-test/run. OK=false with Refused: nothing ran (Refused is linetest.Refusal's
// code, or "server-busy"); OK=false without it: the runs failed.
type LineTestRunResponse struct {
OK bool `json:"ok"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Refused string `json:"refused,omitempty"`
Error string `json:"error,omitempty"`
}
// SetLineTest wires the weekly line test (main.go builds it after the local API server).
@@ -27,11 +51,39 @@ func (s *Server) handleLineTest(w http.ResponseWriter, r *http.Request, vmid int
writeOK(w, LineTestResponse{Present: false})
return
}
var waiting bool
if q, ok := s.lineTest.(LineTestRequester); ok {
waiting = q.WaitingForQuiet()
}
mbps, at, ok := s.lineTest.Value()
if !ok {
writeOK(w, LineTestResponse{Present: false})
writeOK(w, LineTestResponse{Present: false, WaitingForQuiet: waiting})
return
}
s.logger.Debug("local-api: /host/line-test served", "vmid", vmid, "mbps", mbps, "measured_at", at.UTC().Format(time.RFC3339))
writeOK(w, LineTestResponse{Present: true, Mbps: mbps, MeasuredAt: at.UTC().Format(time.RFC3339), Method: "iperf3-tunnel"})
s.logger.Debug("local-api: /host/line-test served", "vmid", vmid, "mbps", mbps, "measured_at", at.UTC().Format(time.RFC3339), "waiting_for_quiet", waiting)
writeOK(w, LineTestResponse{Present: true, Mbps: mbps, MeasuredAt: at.UTC().Format(time.RFC3339), Method: "iperf3-tunnel", WaitingForQuiet: waiting})
}
// handleLineTestRun is POST /host/line-test/run (R-1250): a test on request, synchronous (about 55 s). The answer is
// always 200 with the outcome in the body, except 503 when no line test is wired.
func (s *Server) handleLineTestRun(w http.ResponseWriter, r *http.Request, vmid int) {
q, ok := s.lineTest.(LineTestRequester)
if s.lineTest == nil || !ok {
writeErr(w, http.StatusServiceUnavailable, "line test not configured")
return
}
s.logger.Info("local-api: line test requested", "vmid", vmid)
mbps, runs, err := q.RunNow(r.Context())
var ref *linetest.Refusal
switch {
case err == nil:
writeOK(w, LineTestRunResponse{OK: true, Mbps: mbps, RunsMbps: runs})
case errors.As(err, &ref):
s.logger.Info("local-api: line test request refused", "vmid", vmid, "reason", ref.Code, "detail", ref.Msg)
writeOK(w, LineTestRunResponse{Refused: ref.Code, Error: ref.Msg})
case errors.Is(err, linetest.ErrBusy):
writeOK(w, LineTestRunResponse{Refused: "server-busy", Error: err.Error()})
default:
writeOK(w, LineTestRunResponse{Error: err.Error()})
}
}
+65
View File
@@ -1,12 +1,16 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"log/slog"
"net/http/httptest"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
)
type fixedLine struct {
@@ -49,3 +53,64 @@ func TestLineTestRoute(t *testing.T) {
t.Fatalf("measured: %+v", got)
}
}
type requestableLine struct {
fixedLine
waiting bool
mbps float64
err error
calls int
}
func (r *requestableLine) WaitingForQuiet() bool { return r.waiting }
func (r *requestableLine) RunNow(context.Context) (float64, []float64, error) {
r.calls++
if r.err != nil {
return 0, nil, r.err
}
return r.mbps, []float64{r.mbps - 1, r.mbps}, nil
}
func lineTestRun(t *testing.T, s *Server) (int, LineTestRunResponse) {
t.Helper()
rec := httptest.NewRecorder()
s.handleLineTestRun(rec, httptest.NewRequest("POST", "/host/line-test/run", nil), 9201)
var env struct {
OK bool `json:"ok"`
Data LineTestRunResponse `json:"data"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &env)
return rec.Code, env.Data
}
// R-1250: the controller acts only on waiting_for_quiet, and reads a refusal by its code — never as a measurement.
func TestLineTestRunRoute(t *testing.T) {
s := &Server{logger: slog.Default()}
if code, _ := lineTestRun(t, s); code != 503 {
t.Fatalf("not wired: HTTP %d, want 503", code)
}
s.SetLineTest(fixedLine{ok: true, mbps: 90})
if code, _ := lineTestRun(t, s); code != 503 {
t.Fatalf("a reader without the request half: HTTP %d, want 503", code)
}
l := &requestableLine{fixedLine: fixedLine{ok: false}, waiting: true, mbps: 93.5}
s.SetLineTest(l)
if got := lineTestBody(t, s); got.Present || !got.WaitingForQuiet {
t.Fatalf("GET while waiting: %+v, want present:false waiting_for_quiet:true", got)
}
if code, got := lineTestRun(t, s); code != 200 || !got.OK || got.Mbps != 93.5 || len(got.RunsMbps) != 2 {
t.Fatalf("a measured request: HTTP %d %+v", code, got)
}
l.err = &linetest.Refusal{Code: "not-quiet", Msg: "the uplink sent 16.0 Mbit/s while held"}
if _, got := lineTestRun(t, s); got.OK || got.Refused != "not-quiet" || got.Mbps != 0 {
t.Fatalf("a refusal: %+v, want ok:false refused:not-quiet", got)
}
l.err = linetest.ErrBusy
if _, got := lineTestRun(t, s); got.OK || got.Refused != "server-busy" {
t.Fatalf("server busy: %+v, want refused:server-busy", got)
}
l.err = errors.New("dial tcp 10.77.0.1:5201: i/o timeout")
if _, got := lineTestRun(t, s); got.OK || got.Refused != "" || got.Error == "" {
t.Fatalf("a failed run: %+v, want ok:false, no refusal, the error", got)
}
}
+258
View File
@@ -0,0 +1,258 @@
package localapi
import (
"context"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// The operator's actions for THIS agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).
//
// THE DOOR IS THE HOST REPORT'S REPLY, the controller's shape: the operator presses a button on the hub's host page, the
// hub stores a row and pokes the box (the poke's out-of-cycle report gets the reply in seconds; without the tunnel the
// 15-minute cycle carries it), the reply lists `operator_actions: [{id, action, arg}]`, this agent acts ONCE per id, and
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
// The result is re-sent until the reply stops listing the id.
//
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
//
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
// the guest over the existing grants, with the supervisor's guards — not during a controller swap, not parked, not on a
// stopped, locked or vzdump-busy guest, not when the guest list or the vzdump state cannot be read — and the thrash brake
// counts these restarts too: 3 restarts (the supervisor's or the operator's) inside 15 minutes refuse the press, and
// presses stay refused for 30 minutes. It works with the controller stopped or stuck: the agent does it, not the
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
const (
// OpRestartController is the agent's first operator verb.
OpRestartController = "restart_controller"
opOutcomeDone = "done"
opOutcomeRefused = "refused"
opOutcomeFailed = "failed"
// opBrakeRefusal is how long presses stay refused after the brake refused one.
opBrakeRefusal = 30 * time.Minute
// opForgetAfter drops a started id the reply no longer lists.
opForgetAfter = 24 * time.Hour
defaultOpHealthPoll = 2 * time.Second
// defaultOpHealthWait covers the bootstrap's pull-less rm/run (seconds), Docker's start period (10 s) and its first
// probe (30 s interval) with room: the controller's own HEALTHCHECK is the answer.
defaultOpHealthWait = 3 * time.Minute
// opHealthTemplate reads running + health over the swap's `docker inspect -f` grant.
opHealthTemplate = "{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}"
)
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
// OperatorActionNames returns the closed list (sorted copy).
func OperatorActionNames() []string {
out := append([]string(nil), operatorActionNames...)
sort.Strings(out)
return out
}
type opActionState struct {
mu sync.Mutex
started map[int64]time.Time
results map[int64]hub.OperatorActionResult
}
// SetOperatorActionTrigger wires the report loop's out-of-band trigger (the result goes at once, not in 15 minutes).
func (s *Server) SetOperatorActionTrigger(f func()) { s.opTrigger = f }
// OnEnvelope is the report loop's hook (hub.EnvelopeObserver): it starts each newly listed action once and forgets
// the ids the reply no longer lists (the hub has their results).
func (s *Server) OnEnvelope(_ context.Context, env *hub.ControlEnvelope) {
if env == nil {
return
}
listed := map[int64]bool{}
var start []hub.OperatorAction
now := time.Now()
s.opActs.mu.Lock()
if s.opActs.started == nil {
s.opActs.started = map[int64]time.Time{}
s.opActs.results = map[int64]hub.OperatorActionResult{}
}
for _, a := range env.OperatorActions {
listed[a.ID] = true
if _, ok := s.opActs.started[a.ID]; ok {
continue
}
s.opActs.started[a.ID] = now
start = append(start, a)
}
for id, at := range s.opActs.started {
if listed[id] {
continue
}
if _, answered := s.opActs.results[id]; answered || now.Sub(at) > opForgetAfter {
delete(s.opActs.started, id)
delete(s.opActs.results, id)
}
}
s.opActs.mu.Unlock()
for _, a := range start {
go s.runOperatorAction(a) // a restart waits for the health answer — never block the report loop
}
}
// OperatorActionResults is the report's answer list, oldest id first.
func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
s.opActs.mu.Lock()
defer s.opActs.mu.Unlock()
out := make([]hub.OperatorActionResult, 0, len(s.opActs.results))
for _, r := range s.opActs.results {
out = append(out, r)
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out
}
func (s *Server) runOperatorAction(a hub.OperatorAction) {
limit := s.opWait() + time.Minute
if a.Action == OpRebootGuest {
limit += opRebootTimeout
}
ctx, cancel := context.WithTimeout(context.Background(), limit)
defer cancel()
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
switch a.Action {
case OpRestartController:
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
case OpRebootGuest:
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
}
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
"outcome", outcome, "message", msg)
s.opActs.mu.Lock()
if _, still := s.opActs.started[a.ID]; still {
s.opActs.results[a.ID] = hub.OperatorActionResult{ID: a.ID, Outcome: outcome, Message: msg}
}
s.opActs.mu.Unlock()
if s.opTrigger != nil {
s.opTrigger()
}
}
func (s *Server) opPoll() time.Duration {
if s.opHealthPoll > 0 {
return s.opHealthPoll
}
return defaultOpHealthPoll
}
func (s *Server) opWait() time.Duration {
if s.opHealthWait > 0 {
return s.opHealthWait
}
return defaultOpHealthWait
}
// operatorRestartController is the press: the supervisor's guards, the brake, the restart, then the wait for the
// controller's healthy answer. Returns the outcome and the sentence the host page shows.
func (s *Server) operatorRestartController(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
if s.staleLock == nil || s.guestExec == nil {
return opOutcomeRefused, "refused: the controller supervisor is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
s.swapMu.Lock()
swapping := s.swapInFlight[vmid]
s.swapMu.Unlock()
if swapping {
return opOutcomeRefused, "refused: a controller swap (update) is in flight — it owns the controller"
}
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
return opOutcomeRefused, "refused: a whole-guest backup (vzdump) may be in flight for the guest"
}
// The brake: every restart counts — the supervisor's and the operator's.
now := s.clock()
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
reason := fmt.Sprintf("the operator via the hub (action #%d)", id)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = reason
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.logger.Warn("controller-supervisor: RESTARTING the controller on the operator's press", "vmid", vmid, "action_id", id, "unit", bootstrapUnit)
began := time.Now()
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
return opOutcomeFailed, fmt.Sprintf("the restart of guest %d's controller failed: %v", vmid, err)
}
for time.Since(began) < s.opWait() {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
secs := int(time.Since(began).Round(time.Second) / time.Second)
return opOutcomeDone, fmt.Sprintf("restarted; the controller answered after %d s (Docker's health check: healthy)", secs)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, "restarted, but the wait for its answer was cut"
case <-time.After(s.opPoll()):
}
}
return opOutcomeFailed, fmt.Sprintf("restarted, but no healthy answer within %s — look at the box", s.opWait().Round(time.Second))
}
func orDash(s string) string {
if s == "" {
return "not listed"
}
return s
}
+222
View File
@@ -0,0 +1,222 @@
package localapi
import (
"context"
"errors"
"os"
"path/filepath"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// Operator brief 2026-10-29, Part 1: „Restart controller" from the hub. The agent's closed list holds one verb; the act
// is the supervisor's own restart with the supervisor's guards, and the brake counts operator restarts too.
// opExec answers the supervisor's fake plus the health read the operator restart waits on.
type opExec struct {
*supExec
health string // what {{.State.Health.Status}} reads after the restart
}
func (f *opExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) >= 5 && args[0] == "docker" && args[1] == "inspect" && args[3] == opHealthTemplate {
f.mu.Lock()
st := f.status[vmid]
f.mu.Unlock()
if st == "" {
return "", errors.New("pct exec: exit status 1: Error: No such object: felhom-controller")
}
return map[bool]string{true: "true", false: "false"}[st == "running"] + "|" + f.health + "\n", nil
}
return f.supExec.GuestExec(ctx, vmid, args...)
}
func opServer(t *testing.T, status string) (*Server, *opExec, *fakeGuestPowerCtl, *supClock, string, *atomic.Int32) {
t.Helper()
ex := &opExec{supExec: &supExec{status: map[int]string{9201: status}, onRestart: "running"}, health: "healthy"}
ctl := runningGuest(9201)
s, clk, dir := supServer(t, nil, ctl, 9201)
s.guestExec = ex
s.opHealthPoll = time.Millisecond
s.opHealthWait = 2 * time.Second
var triggered atomic.Int32
s.SetOperatorActionTrigger(func() { triggered.Add(1) })
return s, ex, ctl, clk, dir, &triggered
}
func env(acts ...hub.OperatorAction) *hub.ControlEnvelope {
return &hub.ControlEnvelope{OperatorActions: acts}
}
// waitResult waits for the action's result (the act runs in the background).
func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
for _, r := range s.OperatorActionResults() {
if r.ID == id {
return r
}
}
time.Sleep(2 * time.Millisecond)
}
t.Fatalf("no result for #%d", id)
return hub.OperatorActionResult{}
}
// RED-PROOF: seen failing before the list existed (build failure, then the names).
func TestOpActions_AgentClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
}
}
// Anything outside the list — and a bad argument — is refused and NOTHING is called.
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
s.OnEnvelope(context.Background(), env(
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
))
for _, id := range []int64{1, 2, 3} {
if r := waitResult(t, s, id); r.Outcome != "refused" {
t.Errorf("#%d: %+v, want refused", id, r)
}
}
if n := ex.count(9201); n != 0 {
t.Fatalf("a refused action restarted the controller (%d)", n)
}
}
// The press restarts ONCE, says how long the controller took to answer, asks for a report at once, and re-sends the
// result until the hub stops listing the id. RED-PROOF: drop the GuestExec restart → restarts 0 → FAILS.
func TestOpActions_RestartReportsTheAnswerTime(t *testing.T) {
s, ex, _, _, _, trig := opServer(t, "running")
a := hub.OperatorAction{ID: 7, Action: "restart_controller", Arg: "9201"}
s.OnEnvelope(context.Background(), env(a))
r := waitResult(t, s, 7)
if r.Outcome != "done" || !strings.Contains(r.Message, "answered after") {
t.Fatalf("result %+v", r)
}
if n := ex.count(9201); n != 1 {
t.Fatalf("restarts = %d, want 1", n)
}
if trig.Load() == 0 {
t.Fatal("the agent did not ask for a report — the result would wait 15 minutes")
}
s.OnEnvelope(context.Background(), env(a)) // the hub still lists it: re-send, never act again
time.Sleep(100 * time.Millisecond) // a wrong second act runs in the background — give it time to show
if n := ex.count(9201); n != 1 {
t.Fatalf("a re-listed id acted again (%d)", n)
}
if len(s.OperatorActionResults()) != 1 {
t.Fatal("the result must be re-sent while listed")
}
s.OnEnvelope(context.Background(), env()) // the hub recorded it
if left := s.OperatorActionResults(); len(left) != 0 {
t.Fatalf("a recorded result is still sent: %+v", left)
}
}
// The point of the agent doing it: a STOPPED controller is restarted too (the controller cannot restart itself).
func TestOpActions_WorksWithTheControllerStopped(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "exited")
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 8, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 8); r.Outcome != "done" || ex.count(9201) != 1 {
t.Fatalf("result %+v, restarts %d", r, ex.count(9201))
}
}
// Each supervisor guard refuses the press, with the reason. RED-PROOF: drop the lock check → the locked case
// restarts → FAILS (each guard seen failing by its own mutation).
func TestOpActions_GuardsRefuse(t *testing.T) {
cases := []struct {
name, want string
setup func(s *Server, ctl *fakeGuestPowerCtl, dir string)
}{
{"parked", "parked", func(_ *Server, _ *fakeGuestPowerCtl, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *fakeGuestPowerCtl, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.locks[9201] = "backup" }},
{"vzdump", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupRun = map[int]bool{9201: true} }},
{"vzdump unknown", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupErr = errors.New("tasks unreadable") }},
{"guest stopped", "not running", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guests[0].Status = "stopped" }},
{"guest list", "guest list", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guestsErr = errors.New("api down") }},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, ctl, _, dir, _ := opServer(t, "running")
tc.setup(s, ctl, dir)
id := int64(100 + i)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
r := waitResult(t, s, id)
if r.Outcome != "refused" || !strings.Contains(strings.ToLower(r.Message), tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if n := ex.count(9201); n != 0 {
t.Fatalf("guard %q did not hold: restarted %d", tc.name, n)
}
})
}
}
// The thrash brake counts operator restarts AND the supervisor's: 3 in 15 minutes → the next press is refused, and
// presses stay refused for 30 minutes. RED-PROOF: drop the brake → the 4th press restarts → FAILS.
func TestOpActions_TheBrakeCountsEveryRestart(t *testing.T) {
s, ex, _, clk, _, _ := opServer(t, "running")
// one restart by the supervisor itself
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
if ex.count(9201) != 1 {
t.Fatalf("setup: the supervisor did not restart (%d)", ex.count(9201))
}
for id := int64(1); id <= 2; id++ {
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("press %d: %+v", id, r)
}
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9201"}))
r := waitResult(t, s, 3)
if r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 3 {
t.Fatalf("restarts = %d, want 3", n)
}
clk.t = clk.t.Add(20 * time.Minute) // the window has passed, but the 30-minute refusal has not
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" {
t.Fatalf("inside the 30-minute refusal: %+v", r)
}
clk.t = clk.t.Add(11 * time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 5, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 5); r.Outcome != "done" {
t.Fatalf("after the refusal: %+v", r)
}
}
// A controller that does not come back healthy is a FAILED press, said plainly.
func TestOpActions_NoHealthyAnswerIsFailed(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
ex.health = "unhealthy"
s.opHealthWait = 30 * time.Millisecond
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 9, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 9); r.Outcome != "failed" || !strings.Contains(r.Message, "no healthy answer") {
t.Fatalf("result %+v", r)
}
}
+329
View File
@@ -0,0 +1,329 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
//
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
//
// THE GUARDS (each refuses with the reason in the result; nothing is called):
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
// - a controller swap (update) is in flight;
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
// alone while the reboot runs (rebootInFlight).
//
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
// one day). Pinned by opreboot_test.go.
const (
// OpRebootGuest is the agent's second operator verb (D65 = A).
OpRebootGuest = "reboot_guest"
// opRebootEvery: at most one operator reboot per guest per hour.
opRebootEvery = time.Hour
// opRebootNoticeFor: how long the household's dashboard shows the line.
opRebootNoticeFor = 24 * time.Hour
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
opRebootTimeout = 5 * time.Minute
operatorRebootFile = "operator-reboot.json"
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
// Docker does not hide it.
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
// stopped writing (stuck or killed) — not a busy one.
controllerBusyFresh = 90 * time.Second
)
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
type operatorRebootRecord struct {
At time.Time `json:"at"`
ActionID int64 `json:"action_id"`
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
ControllerAfterS int `json:"controller_after_s,omitempty"`
}
func (s *Server) operatorRebootPath(vmid int) string {
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
}
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
// must not silently vanish).
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
if errors.Is(err, os.ErrNotExist) {
return operatorRebootRecord{}, false, nil
}
if err != nil {
return operatorRebootRecord{}, false, err
}
var rec operatorRebootRecord
if err := json.Unmarshal(raw, &rec); err != nil {
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
}
return rec, !rec.At.IsZero(), nil
}
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
raw, err := json.Marshal(rec)
if err != nil {
return err
}
p := s.operatorRebootPath(vmid)
tmp := p + ".tmp"
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
return err
}
return os.Rename(tmp, p)
}
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
func (s *Server) rebooting(vmid int) bool {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
return s.rebootInFlight[vmid]
}
func (s *Server) setRebooting(vmid int, on bool) {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
if s.rebootInFlight == nil {
s.rebootInFlight = map[int]bool{}
}
if on {
s.rebootInFlight[vmid] = true
} else {
delete(s.rebootInFlight, vmid)
}
}
func (s *Server) rebooter() interface {
RebootGuest(ctx context.Context, vmid int) error
} {
if s.opRebooter != nil {
return s.opRebooter
}
if s.guestAttach != nil {
return s.guestAttach
}
return nil
}
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
// (the caller refuses — it cannot prove no backup runs). Two callers: the operator's reboot (D65) and R-1344's restart
// of a controller that does not answer (controllersupervisor.go restartUnresponsive).
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
if err != nil {
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
return false, "", nil
}
return false, "", err
}
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
if perr != nil {
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
}
if s.clock().Sub(at) > controllerBusyFresh {
s.logger.Info("controller-busy: the controller's busy file is stale — the controller stopped writing it; not busy",
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
return false, "", nil
}
why := "work in progress"
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
why = strings.TrimSpace(lines[1])
}
return true, why, nil
}
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
rb := s.rebooter()
if s.staleLock == nil || s.guestExec == nil || rb == nil {
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
if s.rebooting(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
// At most one per hour.
now := s.clock()
last, had, err := s.lastOperatorReboot(vmid)
if err != nil {
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
}
if had && now.Sub(last.At) < opRebootEvery {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
}
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
s.ctrlSup.mu.Unlock()
// The heavy-operation gate, held until the guest is up again.
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
if !ok {
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
}
defer release()
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
}
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
} else if cbusy {
return opOutcomeRefused, "refused: the controller is busy — " + why
}
// The record FIRST: the hourly limit and the household's line both read it.
rec := operatorRebootRecord{At: now, ActionID: id}
if err := s.saveOperatorReboot(vmid, rec); err != nil {
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
}
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.setRebooting(vmid, true)
defer s.setRebooting(vmid, false)
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
began := time.Now()
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
err = rb.RebootGuest(rctx, vmid)
cancel()
if err != nil {
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
}
upS := int(time.Since(began).Round(time.Second) / time.Second)
rec.GuestUpAfterS = upS
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
}
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
case <-time.After(s.opPoll()):
}
}
_ = s.saveOperatorReboot(vmid, rec)
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
}
// rebootBlockerSentence names jobBlocker's code in the result's words.
func rebootBlockerSentence(code string) string {
switch code {
case PowerOffBackup:
return "a backup is running"
case PowerOffCtrlUpdate:
return "a controller update is running"
case PowerOffDrive:
return "a drive or network-storage job is running"
case PowerOffEscrow:
return "an escrow ceremony is running"
case PowerOffUnknown:
return "the running jobs cannot be read"
default:
return "the host is busy"
}
}
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
rec, ok, err := s.lastOperatorReboot(vmid)
if err != nil {
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
return
}
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
return
}
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
}
+306
View File
@@ -0,0 +1,306 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
// rbExec adds the controller's busy file to the operator-restart fake.
type rbExec struct {
*opExec
busy string // the file's content; "" = no file
busyErr error // pct exec itself failed
}
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
if f.busyErr != nil {
return "", f.busyErr
}
if f.busy == "" {
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
}
return f.busy, nil
}
return f.opExec.GuestExec(ctx, vmid, args...)
}
type fakeRebooter struct {
mu sync.Mutex
n int
err error
after func()
}
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
r.mu.Lock()
defer r.mu.Unlock()
r.n++
if r.after != nil {
r.after()
}
return r.err
}
func (r *fakeRebooter) count() int {
r.mu.Lock()
defer r.mu.Unlock()
return r.n
}
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
t.Helper()
s, ox, ctl, clk, dir, _ := opServer(t, "running")
ex := &rbExec{opExec: ox}
s.guestExec = ex
rb := &fakeRebooter{}
s.opRebooter = rb
s.inFlight = &backup.InFlight{}
return s, ex, rb, ctl, clk, dir
}
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
return waitResult(t, s, id)
}
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
s, _, rb, _, clk, _ := rbServer(t)
r := pressReboot(t, s, 1)
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
t.Fatalf("result %+v", r)
}
if rb.count() != 1 {
t.Fatalf("reboots = %d, want 1", rb.count())
}
rec, ok, err := s.lastOperatorReboot(9201)
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
}
if s.inFlight.Busy() != "" {
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
}
if s.rebooting(9201) {
t.Fatal("the reboot-in-flight flag outlived the reboot")
}
}
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
func TestOpReboot_GuardsRefuse(t *testing.T) {
fresh := func(clk *supClock) string {
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
}
cases := []struct {
name, want string
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
}{
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.guests[0].Status = "stopped"
}},
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupRun = map[int]bool{9201: true}
}},
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupErr = errors.New("tasks unreadable")
}},
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.inFlight.TryAcquire("backup:felhom-pbs")
}},
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
}},
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
ex.busy = fresh(clk)
}},
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
}},
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
s.ctrlSup.mu.Lock()
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
s.ctrlSup.mu.Unlock()
}},
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
panic(err)
}
}},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, rb, ctl, clk, dir := rbServer(t)
tc.setup(s, ex, ctl, clk, dir)
r := pressReboot(t, s, int64(100+i))
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if rb.count() != 0 {
t.Fatalf("guard %q did not hold: rebooted", tc.name)
}
if _, ok, _ := s.lastOperatorReboot(9201); ok {
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
}
})
}
}
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
s, ex, rb, _, clk, _ := rbServer(t)
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
t.Fatalf("result %+v, reboots %d", r, rb.count())
}
}
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
s, _, rb, ctl, clk, dir := rbServer(t)
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("first press: %+v", r)
}
clk.t = clk.t.Add(59 * time.Minute)
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
t.Fatalf("second press inside the hour: %+v", r)
}
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
s2, ox2, _, _, _, _ := opServer(t, "running")
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
s2.guestExec = &rbExec{opExec: ox2}
rb2 := &fakeRebooter{}
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
t.Fatalf("after an agent restart, inside the hour: %+v", r)
}
clk.t = clk.t.Add(2 * time.Minute)
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
}
}
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
s, ex, _, _, clk, _ := rbServer(t)
for id := int64(1); id <= 2; id++ {
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("restart %d: %+v", id, r)
}
clk.t = clk.t.Add(time.Minute)
}
if r := pressReboot(t, s, 3); r.Outcome != "done" {
t.Fatalf("reboot: %+v", r)
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 2 {
t.Fatalf("restarts = %d, want 2", n)
}
}
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
s, _, rb, _, _, _ := rbServer(t)
rb.err = errors.New("pct reboot 9201: exit status 255")
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
t.Fatalf("result %+v", r)
}
if s.inFlight.Busy() != "" || s.rebooting(9201) {
t.Fatal("the gate or the flag outlived a failed reboot")
}
}
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
s, ex, rb, ctl, _, _ := rbServer(t)
var started, restarted int
rb.after = func() {
ctl.guests[0].Status = "stopped"
s.GuestPowerTick(context.Background())
ctl.guests[0].Status = "running"
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
started, restarted = len(ctl.started), ex.count(9201)
ex.mu.Lock()
ex.status[9201] = "running"
ex.mu.Unlock()
}
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
if started != 0 || restarted != 0 {
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
}
}
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
s, _, _, _, clk, _ := rbServer(t)
read := func() string {
w := httptest.NewRecorder()
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
var body struct {
Data struct {
At string `json:"at"`
} `json:"data"`
At string `json:"at"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("body %q: %v", w.Body.String(), err)
}
if body.Data.At != "" {
return body.Data.At
}
return body.At
}
if got := read(); got != "" {
t.Fatalf("no reboot yet, got %q", got)
}
at := clk.t
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
clk.t = at.Add(23 * time.Hour)
if got := read(); got != at.UTC().Format(time.RFC3339) {
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
}
clk.t = at.Add(25 * time.Hour)
if got := read(); got != "" {
t.Fatalf("after 25 h the line must be gone, got %q", got)
}
}
+22
View File
@@ -411,6 +411,20 @@ type Server struct {
guestsDir string
ctrlSup controllerSupervisor
// 2026-10-29: the operator's actions from the hub's reply (opactions.go). opHealthPoll/opHealthWait bound the wait
// for a restarted controller's healthy answer (tests shorten them; zero → the defaults).
opActs opActionState
opTrigger func()
opHealthPoll time.Duration
opHealthWait time.Duration
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
opRebooter interface {
RebootGuest(ctx context.Context, vmid int) error
}
rebootMu sync.Mutex
rebootInFlight map[int]bool
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
// detached pipeline runs through (tests inject; production defaults set in NewServer).
netVerifyMu sync.Mutex
@@ -459,6 +473,8 @@ type Server struct {
escrowLookPath func(file string) (string, error)
// statFile reports whether a path exists (preflight's staged-secret item; tests inject).
statFile func(path string) bool
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
cgroupRoot string
baseCtx context.Context // for fire-and-forget backups; set in Run
}
@@ -576,7 +592,11 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
// read it. Read-only; never measured (or the job not wired) = present:false.
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
@@ -590,6 +610,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
+4 -2
View File
@@ -256,8 +256,10 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
}
}
}
if callers != 5 {
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
// 2026-10-29: +2 deliberate callers — the operator's restart_controller (opactions.go) and R-1344's unresponsive count
// (controllersupervisor.go observeControllerHealth). Both through s.staleLock, the one helper.
if callers != 7 {
t.Errorf("expected the seven known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup, the operator restart, the unresponsive count), found %d", callers)
}
b, _ := os.ReadFile("stalelock.go")
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
@@ -0,0 +1,156 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// R-1344 (operator brief 2026-10-29, Part 2; decided 2026-10-30 from the night's measurement): a controller that RUNS
// but whose Docker health check reads „unhealthy" on controllerUnresponsiveConfirm consecutive sweeps is reported
// unresponsive (the stanza; the hub raises controller_unresponsive). If it still reads „unhealthy" at
// controllerUnresponsiveRestart sweeps, the agent restarts it — with the not-running restart's guards and brake, plus
// the controller's own busy file. RED-PROOF: the restart tests seen failing before the restart existed.
func unrespServer(t *testing.T, health string) (*Server, *rbExec, *fakeGuestPowerCtl, *supClock) {
t.Helper()
s, ex, ctl, clk, _, _ := opServer(t, "running")
ex.health = health
rb := &rbExec{opExec: ex}
s.guestExec = rb
return s, rb, ctl, clk
}
func sweep(s *Server, n int) {
for i := 0; i < n; i++ {
s.ControllerSupervisorTick(context.Background())
}
}
func TestSupervisor_UnresponsiveIsAlarmedThenRestarted(t *testing.T) {
s, ex, _, _ := unrespServer(t, "unhealthy")
sweep(s, controllerUnresponsiveConfirm-1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
}
sweep(s, 1)
g := s.ControllerSupervisorStatus(context.Background()).Guests
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
}
if g[0].UnresponsiveRestartAfterS != (controllerUnresponsiveRestart-controllerUnresponsiveConfirm)*int(controllerSupervisorInterval/time.Second) {
t.Fatalf("the stanza must tell the hub when the restart comes: %+v", g[0])
}
since := g[0].UnresponsiveSince
sweep(s, controllerUnresponsiveRestart-controllerUnresponsiveConfirm-1)
if n := ex.count(9201); n != 0 {
t.Fatalf("restarted after %d unhealthy sweeps — the restart waits for %d", controllerUnresponsiveRestart-1, controllerUnresponsiveRestart)
}
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
}
sweep(s, 1)
if n := ex.count(9201); n != 1 {
t.Fatalf("after %d unhealthy sweeps: %d restarts, want 1", controllerUnresponsiveRestart, n)
}
g = s.ControllerSupervisorStatus(context.Background()).Guests
if g[0].UnresponsiveRestartAt == "" || g[0].UnresponsiveRestartAt != g[0].LastRestartAt || g[0].Restarts24h != 1 ||
!strings.Contains(g[0].LastReason, "health check") {
t.Fatalf("the restart is not on the record the hub reads: %+v", g[0])
}
// A restart that did not help (still „unhealthy" on the next sweep) is not repeated at once: the count starts again.
sweep(s, 1)
if n := ex.count(9201); n != 1 {
t.Fatalf("restarted again on the sweep after a restart: %d restarts", n)
}
// The restarted controller starts, then answers: the episode ends, its record stays.
ex.health = "starting"
sweep(s, 1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
t.Fatalf("after the restart: %+v (unresponsive must clear; since stays as the record)", g[0])
}
// A second episode needs the whole count again — never a restart on the first unhealthy read after one.
ex.health = "unhealthy"
sweep(s, controllerUnresponsiveRestart-1)
if n := ex.count(9201); n != 1 {
t.Fatalf("a second episode restarted early: %d restarts", n)
}
}
// Slow but answering (healthy / starting), a swap, a vzdump, a locked or a parked guest never counts — and so is never
// restarted.
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
for _, tc := range []struct {
name string
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl)
}{
{"healthy", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
{"starting", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
{"swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
{"vzdump", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
{"locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
} {
t.Run(tc.name, func(t *testing.T) {
s, ex, ctl, _ := unrespServer(t, "unhealthy")
tc.setup(s, ex, ctl)
sweep(s, controllerUnresponsiveRestart+2)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
}
if n := ex.count(9201); n != 0 {
t.Fatalf("%s: restarted %d times", tc.name, n)
}
})
}
}
// At the restart moment: the controller's busy file (fresh, or unreadable) and the brake refuse it; the alarm stays.
// RED-PROOF: each seen failing by deleting its guard.
func TestSupervisor_UnresponsiveRestartRefusals(t *testing.T) {
for _, tc := range []struct {
name string
setup func(s *Server, ex *rbExec, clk *supClock)
crashloop bool
}{
{"busy file fresh", func(_ *Server, ex *rbExec, clk *supClock) {
ex.busy = clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\na backup, a restore or an off-site run is running\n"
}, false},
{"busy file unreadable", func(_ *Server, ex *rbExec, _ *supClock) { ex.busyErr = errors.New("pct exec: exit status 255") }, false},
{"brake: 3 restarts in 15 minutes", func(s *Server, _ *rbExec, clk *supClock) {
st := s.supState(9201)
st.restarts = []time.Time{clk.t.Add(-10 * time.Minute), clk.t.Add(-5 * time.Minute), clk.t.Add(-time.Minute)}
}, true},
{"crash-loop pause in force", func(s *Server, _ *rbExec, clk *supClock) {
s.supState(9201).crashloopSince = clk.t.Add(-5 * time.Minute)
}, true},
} {
t.Run(tc.name, func(t *testing.T) {
s, ex, _, clk := unrespServer(t, "unhealthy")
tc.setup(s, ex, clk)
sweep(s, controllerUnresponsiveRestart+3)
if n := ex.count(9201); n != 0 {
t.Fatalf("%s: restarted %d times", tc.name, n)
}
g := s.ControllerSupervisorStatus(context.Background()).Guests
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveRestartAt != "" {
t.Fatalf("%s: the alarm must stay and no restart be recorded: %+v", tc.name, g)
}
if tc.crashloop && !g[0].Crashloop {
t.Fatalf("%s: the brake did not raise the crash-loop: %+v", tc.name, g[0])
}
})
}
}
// A busy file the controller stopped writing (older than controllerBusyFresh) is not „busy" — a hung controller must
// not shield itself with its last word.
func TestSupervisor_UnresponsiveStaleBusyFileDoesNotShield(t *testing.T) {
s, ex, _, clk := unrespServer(t, "unhealthy")
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\nan app update is running\n"
sweep(s, controllerUnresponsiveRestart)
if n := ex.count(9201); n != 1 {
t.Fatalf("a stale busy file blocked the restart: %d restarts", n)
}
}
+32
View File
@@ -8,6 +8,7 @@ import (
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"path/filepath"
@@ -114,12 +115,43 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
}
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
e.policyPass(ctx, lg, p.AgentVersion)
if e.AfterInstall != nil {
e.AfterInstall(ctx)
}
return nil
}
// policyPass (R-1285) asks the wrapper NOW on disk — after this install, the bundle's own — to apply the
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
// SAME update. Idempotent; on an appliance it removes nothing.
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
if lg == nil {
lg = slog.Default()
}
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
e.policyPass(ctx, lg, ver)
}
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
switch {
case err != nil:
lg.Warn("osupdate: the bundle policy pass did not run", "err", err)
case wr.refused():
lg.Info("osupdate: the bundle policy pass was refused (a wrapper before 0.166.0?) — the next bundle applies it", "refused", string(wr.Refused))
default:
lg.Info("osupdate: bundle policy pass done", "bundle_policy", string(wr.BundlePolicy))
}
}
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
hc := e.HTTPClient
if hc == nil {
+27
View File
@@ -38,6 +38,9 @@ func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]b
var plan map[string]any
_ = json.Unmarshal(raw, &plan)
b.plans = append(b.plans, plan)
if plan["mode"] == "bundle-policy" { // R-1285: the pass after an install carries no bundle
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":[]}}` + "\n"), nil, nil
}
body, _ := os.ReadFile(plan["bundle"].(string))
b.bodies = append(b.bodies, body)
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
@@ -153,3 +156,27 @@ func TestBundleURL(t *testing.T) {
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return b.Run(ctx, name, args...)
}
// R-1285: right after an install the agent asks the wrapper now on disk for the appliance-only policy pass — the bundle
// run itself was planned by the OLD wrapper (on a BYO host upgraded from 0.164.0 our SSH drop-in stayed one bundle
// longer). A refused or failed bundle runs no pass.
// RED-PROOF: drop the policyPass call → one plan only.
func TestR1285_AnInstalledBundleIsFollowedByThePolicyPass(t *testing.T) {
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
srv, sha := serveBundle(t, body)
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/usr/local/sbin/felhom-os-apply"]}}`}
e, _ := bundleExec(t, w, srv.URL)
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
t.Fatal(err)
}
if len(w.plans) != 2 || w.plans[1]["mode"] != "bundle-policy" || w.plans[1]["layer"] != "host" || w.plans[1]["signed"] != nil {
t.Fatalf("after the install the agent must run the policy pass (host layer, no envelope): %v", w.plans)
}
w2 := &bundleWrapper{t: t, rep: `{"mode":"bundle","refused":{"code":"R3","reason":"x"}}`}
e2, _ := bundleExec(t, w2, srv.URL)
_ = e2.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha))
if len(w2.plans) != 1 {
t.Fatalf("a refused bundle must run no policy pass: %v", w2.plans)
}
}
+146
View File
@@ -21,6 +21,8 @@ import (
"fmt"
"log/slog"
"regexp"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
@@ -444,3 +446,147 @@ func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParam
}
return l.finish(ctx, lg, rep)
}
// ---- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----
//
// The rule, its refusals and "may this box do it" live in the root wrapper (mode kernel-cleanup); the plan names NO
// package. This file decides only WHEN: in the night leg, after a healthy host step, never in a night with a kernel
// step. The wrapper adds its own 20-hour window after a step.
// KernelCleanupResult is the wrapper's kernel_cleanup object.
type KernelCleanupResult struct {
Outcome string `json:"outcome"` // removed | nothing | not_run | dry_run | failed
DryRun bool `json:"dry_run"`
Would string `json:"would"` // dry run only: remove | nothing | refuse | not_run
NotRun []struct {
Code string `json:"code"`
Reason string `json:"reason"`
} `json:"not_run"`
Refusals []struct {
Code string `json:"code"`
Reason string `json:"reason"`
} `json:"refusals"`
Kept map[string][]string `json:"kept"`
Removal []string `json:"removal"`
Removed []string `json:"removed"`
BytesGained *int64 `json:"bytes_gained"`
Leftovers []string `json:"leftovers"`
}
// kernelCleanupSkip says why tonight's pass must not call the clean-up ("" = call it).
func kernelCleanupSkip(appliance bool, trigger string, host, pve, kernel Report, blk hub.WireOSUpdate, okStep func(Report) bool) string {
switch {
case trigger != "night":
return "it runs only in the night leg (never a debug pass)"
case !appliance:
return "not an appliance install (a BYO host belongs to its owner)"
case !blk.Enabled:
return "OS updates are switched off for this box"
case host.Layer == "" || !okStep(host):
return "no healthy host step this pass"
case pve.Layer != "" && !okStep(pve):
return "the Proxmox step did not end healthy"
case kernel.Layer != "":
return "a kernel step ran in this night (" + kernel.Outcome + ") — never in the same night"
}
if due, _ := kernelDue(blk, trigger); due {
return "a kernel step is due tonight — never in the same night"
}
return ""
}
// runKernelCleanup calls the wrapper once. The hub hears about it only when there is something to tell: kernels were
// removed (outcome "cleaned"), the wrapper refused, or the purge failed. "Nothing to remove" and "not run" are log
// lines — one report per box per night would bury the kernel step's own line on the hub's System page.
func (l *Leg) runKernelCleanup(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
lg := l.log().With("run", runID, "layer", LayerKernel, "mode", "kernel-cleanup", "vmid", vmid, "trigger", trigger)
start := l.now()
wr, err := l.call(ctx, runID, kernelPlan("kernel-cleanup", vmid, map[string]any{"release_id": "kernel-cleanup",
"run_id": runID, "trigger": trigger, "ring": blk.Ring}))
// NO kernel view and no kernel version in this report: the hub reads a kernel-layer report's `kernel.to` /
// `release_id` as "the kernel this STEP was for" (kernelOf), and a clean-up is not a step for any kernel.
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "kernel-cleanup",
ReleaseID: "kernel-cleanup", PassSeconds: wr.PassSeconds}
var res KernelCleanupResult
_ = json.Unmarshal(wr.KernelCleanup, &res)
took := l.now().Sub(start).Round(time.Second).String()
switch {
case err != nil:
lg.Warn("osupdate: kernel clean-up — the wrapper gave no answer; not retried tonight", "err", err, "took", took)
rep.Outcome, rep.HealthReason = "failed", "kernel clean-up: "+err.Error()
return l.finish(ctx, lg, rep)
case wr.refused():
lg.Warn("osupdate: kernel clean-up REFUSED — nothing was removed; not retried tonight", "refused", string(wr.Refused), "took", took)
rep.Outcome, rep.Refused, rep.HealthReason = "refused", wr.Refused, "the clean-up of old kernels was refused; nothing was removed"
return l.finish(ctx, lg, rep)
case wr.failed():
lg.Error("osupdate: kernel clean-up FAILED — look at the box; not retried tonight", "failed", string(wr.Failed), "took", took)
rep.Outcome, rep.Refused, rep.HealthReason = "failed", wr.Failed, "the clean-up of old kernels failed"
return l.finish(ctx, lg, rep)
}
switch res.Outcome {
case "removed":
gained := "unknown"
if res.BytesGained != nil {
gained = fmt.Sprintf("%.1f GiB", float64(*res.BytesGained)/(1<<30))
}
rep.Outcome, rep.Healthy = "cleaned", true
rep.KernelCleanup = &CleanupDetail{Removed: capNames(res.Removed, 40), Kept: capNames(sortedKeys(res.Kept), 40),
BytesGained: res.BytesGained, Leftovers: len(res.Leftovers)}
rep.HealthReason = truncate(fmt.Sprintf("removed %d old kernel package(s): %s; kept %s; %s gained; %d left-over path(s)",
len(res.Removed), strings.Join(res.Removed, ", "), strings.Join(sortedKeys(res.Kept), ", "), gained, len(res.Leftovers)), 600)
lg.Info("osupdate: kernel clean-up — old kernels removed", "removed", len(res.Removed), "names", strings.Join(res.Removed, ","),
"kept", strings.Join(sortedKeys(res.Kept), ","), "bytes_gained", res.BytesGained, "leftovers", len(res.Leftovers), "took", took)
return l.finish(ctx, lg, rep)
case "nothing":
lg.Info("osupdate: kernel clean-up — nothing to remove", "kept", strings.Join(sortedKeys(res.Kept), ","), "took", took)
case "not_run":
code, why := "", ""
if len(res.NotRun) > 0 {
code, why = res.NotRun[0].Code, res.NotRun[0].Reason
}
lg.Info("osupdate: kernel clean-up — not run on this box tonight", "code", code, "reason", why, "took", took)
default:
lg.Warn("osupdate: kernel clean-up — the wrapper answered with no outcome this agent knows (an older wrapper?)", "outcome", res.Outcome, "took", took)
}
return Report{}
}
// KernelCleanupDryRun asks the wrapper what the clean-up WOULD do, and changes nothing (the debug tool: `--selftest
// kernel-cleanup-dry-run`). The raw kernel_cleanup object, or the refusal.
func (l *Leg) KernelCleanupDryRun(ctx context.Context, vmid int) (json.RawMessage, error) {
wr, err := l.call(ctx, "kcdry"+l.now().UTC().Format("150405"), kernelPlan("kernel-cleanup", vmid,
map[string]any{"release_id": "kernel-cleanup", "dry_run": true}))
if err != nil {
return nil, err
}
if wr.refused() {
return nil, fmt.Errorf("kernel-cleanup dry run refused: %s", wr.Refused)
}
return wr.KernelCleanup, nil
}
// capNames bounds a list that goes to the hub: at most n names, each at most 120 bytes; never nil (the hub reads
// "removed: []" as "none", not as "unknown").
func capNames(in []string, n int) []string {
out := make([]string, 0, len(in))
for _, x := range in {
if len(out) == n {
break
}
if len(x) > 120 {
x = x[:120]
}
out = append(out, x)
}
return out
}
func sortedKeys(m map[string][]string) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
+204 -1
View File
@@ -28,13 +28,23 @@ func tonight(ring int) *hub.WireOSUpdate {
func kernelCalls(w *fakeWrapper) []string {
var m []string
for _, p := range w.plans {
if p["layer"] == LayerKernel {
if p["layer"] == LayerKernel && p["mode"] != "kernel-cleanup" { // the kernel STEP's calls (the clean-up: cleanupCalls)
m = append(m, p["mode"].(string))
}
}
return m
}
func cleanupCalls(w *fakeWrapper) []map[string]any {
var m []map[string]any
for _, p := range w.plans {
if p["layer"] == LayerKernel && p["mode"] == "kernel-cleanup" {
m = append(m, p)
}
}
return m
}
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
@@ -326,3 +336,196 @@ func TestKernelSet(t *testing.T) {
t.Fatal("a non-kernel string must give no set")
}
}
// ---- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----
func cleanupRep(js string) map[string][]WrapperReport {
return map[string][]WrapperReport{"kernel-cleanup": {{KernelCleanup: json.RawMessage(js)}}}
}
// The night: the clean-up is the LAST call, after the host step, exactly once, and its plan names NO package.
// COMPANION RED-PROOF (observed): make kernelCleanupSkip return "" before its checks → the BYO / debug / unhealthy
// sub-cases of TestKernelCleanup_WhenItMayNotRun fail; put the call before runFast → this test's order check fails.
func TestKernelCleanup_NightOrderOnceAndNoNames(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: cleanupRep(`{"outcome":"removed","removed":["proxmox-kernel-7.0.14-20-pve-signed"],` +
`"kept":{"7.0.14-22-pve":["running"],"7.0.2-6-pve":["from"]},"bytes_gained":1073741824,"leftovers":[]}`)}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
cc := cleanupCalls(w)
if len(cc) != 1 {
t.Fatalf("clean-up calls = %d, want exactly one a night", len(cc))
}
if last := w.plans[len(w.plans)-1]; last["mode"] != "kernel-cleanup" {
t.Fatalf("the clean-up must be the night's last call (after the host step), last = %v", last)
}
hostAt, cleanAt := -1, -1
for i, x := range w.plans {
if x["layer"] == LayerHost {
hostAt = i
}
if x["mode"] == "kernel-cleanup" {
cleanAt = i
}
}
if hostAt < 0 || cleanAt < hostAt {
t.Fatalf("the clean-up ran before the host step: host %d clean-up %d", hostAt, cleanAt)
}
plan := cc[0]
if pk, _ := json.Marshal(plan["packages"]); string(pk) != "[]" {
t.Fatalf("the clean-up plan names packages: %s", pk)
}
for _, k := range []string{"select", "signed", "allow_new", "expect_kver", "dry_run"} {
if _, ok := plan[k]; ok {
t.Fatalf("the clean-up plan carries %q: %v", k, plan)
}
}
if plan["lane"] != "slow" || plan["layer"] != LayerKernel {
t.Fatalf("plan = %v", plan)
}
// the hub hears "cleaned" with what was removed, kept and gained — and NO kernel view / version (the hub would read
// it as a kernel step for that kernel)
rep := p.KernelCleanup
if rep.Outcome != "cleaned" || !rep.Healthy || rep.Mode != "kernel-cleanup" || rep.Layer != LayerKernel {
t.Fatalf("report = %+v", rep)
}
for _, want := range []string{"removed 1 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed", "kept 7.0.14-22-pve, 7.0.2-6-pve", "1.0 GiB gained"} {
if !strings.Contains(rep.HealthReason, want) {
t.Fatalf("report reason %q lacks %q", rep.HealthReason, want)
}
}
lastRep := h.reports[len(h.reports)-1]
if lastRep.Mode != "kernel-cleanup" || lastRep.Outcome != "cleaned" {
t.Fatalf("the hub did not hear the clean-up: %+v", lastRep)
}
if len(lastRep.Kernel) != 0 || kverRE.MatchString(lastRep.ReleaseID) {
t.Fatalf("the clean-up's report names a kernel (the hub reads that as a kernel step): %+v", lastRep)
}
// R-1383 (release 3): the detail the hub's System page shows — removed, kept, gained, left-overs — rides the same
// report as its own object. On the wire it is `kernel_cleanup`, and the report STILL has no `kernel` key.
d := lastRep.KernelCleanup
if d == nil || len(d.Removed) != 1 || d.Removed[0] != "proxmox-kernel-7.0.14-20-pve-signed" ||
strings.Join(d.Kept, ",") != "7.0.14-22-pve,7.0.2-6-pve" || d.BytesGained == nil || *d.BytesGained != 1073741824 || d.Leftovers != 0 {
t.Fatalf("the clean-up's detail did not reach the hub: %+v", d)
}
wire, _ := json.Marshal(lastRep)
var keys map[string]json.RawMessage
_ = json.Unmarshal(wire, &keys)
if _, has := keys["kernel"]; has {
t.Fatalf("the clean-up's report carries a `kernel` view on the wire: %s", wire)
}
if string(keys["kernel_cleanup"]) != `{"removed":["proxmox-kernel-7.0.14-20-pve-signed"],"kept":["7.0.14-22-pve","7.0.2-6-pve"],"bytes_gained":1073741824,"leftovers":0}` {
t.Fatalf("kernel_cleanup on the wire = %s", keys["kernel_cleanup"])
}
if string(keys["release_id"]) != `"kernel-cleanup"` {
t.Fatalf("release_id on the wire = %s (a kernel version there is read as a step for it)", keys["release_id"])
}
// the same night again: the pass is skipped as a whole — never a second clean-up
l.Run(context.Background(), 9201, "night")
if len(cleanupCalls(w)) != 1 {
t.Fatalf("a second clean-up in the same night: %d calls", len(cleanupCalls(w)))
}
}
// "Nothing to remove" and "not run" are log lines, never a report (one a night would bury the kernel step's own line).
func TestKernelCleanup_NothingAndNotRunSendNoReport(t *testing.T) {
for name, js := range map[string]string{
"nothing": `{"outcome":"nothing","removal":[],"kept":{"7.0.14-22-pve":["running"]}}`,
"not_run": `{"outcome":"not_run","not_run":[{"code":"no-ring0-mark","reason":"ring 0 only in this release"}]}`,
} {
w := &fakeWrapper{t: t, kernelRep: cleanupRep(js)}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
if len(cleanupCalls(w)) != 1 {
t.Fatalf("%s: the wrapper was not asked", name)
}
if p.KernelCleanup.Layer != "" {
t.Fatalf("%s: a report was made: %+v", name, p.KernelCleanup)
}
for _, r := range h.reports {
if r.Mode == "kernel-cleanup" {
t.Fatalf("%s: the hub got a clean-up report: %+v", name, r)
}
}
}
}
// A refusal and a failure reach the hub (the operator), once, and are not retried.
func TestKernelCleanup_RefusedAndFailedReachTheHub(t *testing.T) {
for name, wr := range map[string]WrapperReport{
"refused": {Refused: json.RawMessage(`{"code":"P3","reason":"the pinned default and grub.cfg disagree"}`)},
"failed": {Failed: json.RawMessage(`{"rc":3,"step":"verify","reason":"the default in grub.cfg moved"}`), KernelCleanup: json.RawMessage(`{"outcome":"failed"}`)},
} {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-cleanup": {wr}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
if p.KernelCleanup.Outcome != name || p.KernelCleanup.Healthy || len(p.KernelCleanup.Refused) == 0 {
t.Fatalf("%s: report = %+v", name, p.KernelCleanup)
}
if last := h.reports[len(h.reports)-1]; last.Mode != "kernel-cleanup" || last.Outcome != name {
t.Fatalf("%s: the hub did not hear it: %+v", name, last)
}
if len(cleanupCalls(w)) != 1 {
t.Fatalf("%s: retried the same night (%d calls)", name, len(cleanupCalls(w)))
}
}
}
// When it may NOT run: never with a kernel step in the night (due, staged, or reported), not in a debug pass, not on a
// BYO host, not with OS updates off, not after an unhealthy host step.
func TestKernelCleanup_WhenItMayNotRun(t *testing.T) {
type tc struct {
blk *hub.WireOSUpdate
trigger string
prep func(*fakeWrapper, *Leg)
}
bad := &Health{DockerOK: false}
cases := map[string]tc{
"a kernel step is due tonight (ring 0)": {tonight(0), "night", func(w *fakeWrapper, _ *Leg) {
w.kernelRep = map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
"kernel-reboot": {{Kernel: kview("oneshot")}}}
}},
"a kernel step is due tonight (ring 1, nothing staged)": {tonight(1), "night", func(w *fakeWrapper, _ *Leg) {
w.kernelRep = map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}
}},
"a debug pass": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "debug", nil},
"a BYO host": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night", func(_ *fakeWrapper, l *Leg) { l.Appliance = false }},
"updates off": {&hub.WireOSUpdate{Ring: 0, Enabled: false}, "night", nil},
"host unhealthy": {&hub.WireOSUpdate{Ring: 0, Enabled: true, HostRelease: &hub.WireOSRelease{ID: "os-host-1", Packages: []hub.WireOSPackage{{Name: "openssl", Version: "u3", Origin: "Debian"}}}}, "night", func(w *fakeWrapper, _ *Leg) {
w.applyRep[LayerHost] = WrapperReport{Upgraded: []Package{{Name: "openssl", Version: "u3"}}, HealthAfter: &Health{HostServices: map[string]string{"pveproxy": "failed"}, GuestRunning: yes(), Guest: bad}}
w.healthSeq[LayerHost] = []*Health{{HostServices: map[string]string{"pveproxy": "failed"}, GuestRunning: yes(), Guest: bad}}
}},
}
for name, c := range cases {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, c.blk)
if c.prep != nil {
c.prep(w, l)
}
l.HealthWait = time.Second
l.Run(context.Background(), 9201, c.trigger)
if n := len(cleanupCalls(w)); n != 0 {
t.Fatalf("%s: the clean-up ran (%d call(s)); calls = %s", name, n, calls(w))
}
}
}
// The dry run is the same mode with dry_run: true — and it sends nothing to the hub.
func TestKernelCleanup_DryRun(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: cleanupRep(`{"outcome":"dry_run","would":"remove","removal":["proxmox-kernel-7.0.14-20-pve-signed"]}`)}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
raw, err := l.KernelCleanupDryRun(context.Background(), 9201)
if err != nil || !strings.Contains(string(raw), `"would":"remove"`) {
t.Fatalf("dry run = %s, %v", raw, err)
}
cc := cleanupCalls(w)
if len(cc) != 1 || cc[0]["dry_run"] != true {
t.Fatalf("the dry run's plan = %v", cc)
}
if pk, _ := json.Marshal(cc[0]["packages"]); string(pk) != "[]" {
t.Fatalf("the dry run's plan names packages: %s", pk)
}
if len(h.reports) != 0 {
t.Fatalf("a dry run reported to the hub: %+v", h.reports)
}
}
+41 -3
View File
@@ -123,7 +123,8 @@ type WrapperReport struct {
Undo bool `json:"undo"`
LiveRestore json.RawMessage `json:"live_restore"`
Facts json.RawMessage `json:"facts"`
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
BundlePolicy json.RawMessage `json:"bundle_policy"` // the appliance-only policy pass (R-1285, mode "bundle-policy")
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
OOMCheck json.RawMessage `json:"oom_check"`
@@ -133,6 +134,13 @@ type WrapperReport struct {
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
Kernel json.RawMessage `json:"kernel"`
KernelEvent string `json:"kernel_event"`
// KernelCleanup (R-1368, `11` §5.11.1): mode kernel-cleanup's result {outcome, dry_run, kept, removal, removed,
// bytes_gained, not_run, refusals, leftovers}.
KernelCleanup json.RawMessage `json:"kernel_cleanup"`
// ProfileRefresh (R-1388, `11` §5.12): mode profile-refresh's result {outcome, kernel, rewritten, restarted, seconds}.
ProfileRefresh json.RawMessage `json:"profile_refresh"`
// PVECACopy (R-1380, felhom.eu `03` §12a): mode pve-ca-copy's result {outcome, path, sha256, not_after, days_left}.
PVECACopy json.RawMessage `json:"pve_ca_copy"`
OutcomeHint string `json:"outcome_hint"`
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
// agent process that started the pass. ReleaseID / VMID were always in the report.
@@ -180,10 +188,23 @@ type Report struct {
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
Kernel json.RawMessage `json:"kernel,omitempty"`
// KernelCleanup (R-1383, release 3): ONLY on the mode kernel-cleanup report with outcome "cleaned" — what the
// clean-up removed, kept and gained, for the hub's System page. It is not the `kernel` view: this report still
// carries no kernel view and no kernel as its release_id, or the hub would read it as a kernel STEP for that kernel
// (the hub's kernelOf reads only those two). Pinned by TestKernelCleanup_NightOrderOnceAndNoNames.
KernelCleanup *CleanupDetail `json:"kernel_cleanup,omitempty"`
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
}
// CleanupDetail is the clean-up's result as the hub shows it (hub osupdates.CleanupDetail, field-exact).
type CleanupDetail struct {
Removed []string `json:"removed"` // package names
Kept []string `json:"kept"` // kernel versions that stay
BytesGained *int64 `json:"bytes_gained,omitempty"` // nil = the wrapper could not measure it
Leftovers int `json:"leftovers"` // paths of a removed kernel that are still there
}
// Reporter posts a report to the hub (*hub.Client).
type Reporter interface {
PostOSReport(ctx context.Context, body []byte) error
@@ -203,9 +224,15 @@ type Leg struct {
MinGap time.Duration // between night runs (default 20 h)
Now func() time.Time
Sleep func(context.Context, time.Duration)
// ProfileRecordPath is the root wrapper's record of the last security-profile refresh (0644, read-only for the
// agent); "" = DefaultProfileRecordPath. A seam for the tests.
ProfileRecordPath string
mu sync.Mutex
block *hub.WireOSUpdate
mu sync.Mutex
block *hub.WireOSUpdate
profileAsk string // how the last ask of the profile refresh ended in THIS process ("" = never asked)
profileAskAt time.Time
profileLast ProfileRefreshStatus // the last refresh as the wrapper answered it to this process
}
// planFile / reportFile: the plan the agent writes and the copy of the report the wrapper keeps beside it (R-868).
@@ -530,6 +557,8 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
// Pass is one leg's reports; an empty Layer means the step did not run.
type Pass struct {
Guest, Host, Docker, PVE, Kernel Report
// KernelCleanup (R-1368): the clean-up of old kernels. Layer "" when it did not run or had nothing to tell the hub.
KernelCleanup Report
}
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
@@ -581,6 +610,15 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
default:
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
}
// R-1368 (`09` §3 decision 286, `11` §5.11.1): the clean-up of old kernels — its own step, at most once a night (this
// pass is the night's only one), after the host step ended healthy, NEVER in a night with a kernel step. Whether
// this box may do it at all is the root wrapper's answer (its root-owned ring-0 mark), never the agent's.
// Pinned by TestKernelCleanup_*.
if why := kernelCleanupSkip(l.Appliance, trigger, h, p.PVE, p.Kernel, blk, okStep); why != "" {
lg.Info("osupdate: kernel clean-up skipped — " + why)
} else {
p.KernelCleanup = l.runKernelCleanup(ctx, g.RunID, vmid, trigger, blk)
}
return p
}
+10 -3
View File
@@ -26,6 +26,7 @@ type fakeWrapper struct {
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
pveGateHeld bool
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
profileRep *WrapperReport // R-1388: the answer to mode profile-refresh
}
func yes() *bool { b := true; return &b }
@@ -71,6 +72,10 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
}
var rep WrapperReport
switch plan["mode"] {
case "profile-refresh":
if f.profileRep != nil {
rep = *f.profileRep
}
case "inventory":
rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthBefore: ok, HealthAfter: ok,
Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
@@ -176,8 +181,9 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
t.Fatalf("guest %+v\nhost %+v", g, ho)
}
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply" {
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step and the pve step", calls(w))
// R-1368: the clean-up of old kernels is the night's last call when no kernel step is due
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply,kernel:kernel-cleanup" {
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step, the pve step, the kernel clean-up", calls(w))
}
for _, p := range w.plans[:2] {
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
@@ -221,7 +227,8 @@ func TestRing1_NoReleaseIsInventory(t *testing.T) {
w := &fakeWrapper{t: t, pending: pend}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
g, ho := run2(l, "night")
if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory" {
// R-1368: a ring-1 box asks the wrapper too — whether it may clean up is the wrapper's answer (its root-owned mark)
if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory,kernel:kernel-cleanup" {
t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
}
}
+217
View File
@@ -0,0 +1,217 @@
package osupdate
// The security-profile refresh (R-1388, R-1301, `11` §5.12). After a kernel went back and forth a box can load an OLD
// compiled AppArmor profile for a program (on the N100 this stopped the time service). The root wrapper's mode
// profile-refresh loads every profile from a fresh compile, once per kernel the box starts on, and restarts only a
// failed service whose own program's profile was stale. This file decides WHEN to ask: at every agent start and after
// a config bundle was installed (a new wrapper may know the mode the old one refused) — the wrapper itself answers
// "not needed" when this kernel was already done. No OS report is sent for it (an unknown layer reads as a GUEST step on
// the hub); since release 3 the result rides the host report as the `profile_refresh` stanza (ProfileRefreshStanza).
import (
"context"
"encoding/json"
"os"
"regexp"
"strings"
"time"
)
// ProfileRefreshResult is the wrapper's profile_refresh object.
type ProfileRefreshResult struct {
Outcome string `json:"outcome"` // refreshed | not_needed | dry_run | failed
Reason string `json:"reason"`
Kernel string `json:"kernel"`
Rewritten []string `json:"rewritten"`
Restarted []struct {
Unit string `json:"unit"`
Program string `json:"program"`
RC int `json:"rc"`
Active bool `json:"active"`
} `json:"restarted"`
Seconds float64 `json:"seconds"`
// The wrapper's own record of the refresh BEFORE this ask (it answers them at every ask, also "not needed").
LastRefreshKernel string `json:"last_refresh_kernel"`
LastRefreshAt string `json:"last_refresh_at"`
}
// DefaultProfileRecordPath is where the root wrapper keeps the record of its last refresh (PROFILE_STATE, 0644).
const DefaultProfileRecordPath = "/var/lib/felhom-os-apply/profile-refresh.json"
// ProfileRefreshStatus is the host report's `profile_refresh` stanza (R-1383, release 3; hub sysfacts.ProfileRefresh,
// field-exact): the last refresh the box did — from the root wrapper's own record, so it outlives an agent restart —
// and how the last ask of THIS agent process ended. The refresh restarted a failed service on the N100 and nobody
// outside the box's journal could see it.
type ProfileRefreshStatus struct {
Kernel string `json:"kernel,omitempty"` // the kernel the last refresh ran on
At string `json:"at,omitempty"` // RFC3339, when
Rewritten *int `json:"rewritten,omitempty"` // how many compiled profiles changed; nil = not known to this process
Restarted []string `json:"restarted,omitempty"` // units it restarted (failed, with a stale profile)
LastAsk string `json:"last_ask,omitempty"` // refreshed | not_needed | failed | refused | no_answer
LastAskAt string `json:"last_ask_at,omitempty"`
}
var profileNameRe = regexp.MustCompile(`^[A-Za-z0-9@._:+-]{1,120}$`)
// ProfileRefreshStanza builds the stanza, or an UNTYPED nil when there is nothing to say (never asked and no record: a
// BYO host, or an agent that did not ask yet) — the report then omits the field.
//
// Two sources, the first that knows wins per field. (1) What the wrapper ANSWERED to this process's last ask: a
// refresh gives everything; "not needed" gives the kernel and time of the earlier refresh (the wrapper answers its own
// record at every ask) — so the last refresh is still told after an agent restart, without the detail. (2) The
// wrapper's record file, when this process may read it: on boxes where its folder was first made by a signed job it
// is root-only (0700), so the file is a bonus, never the only source (found by review, 2026-11-14; not read on a box).
// It never calls the wrapper. Names that are not plain unit / kernel names are dropped, the list is capped.
func (l *Leg) ProfileRefreshStanza() any {
l.mu.Lock()
st, ask, at := l.profileLast, l.profileAsk, l.profileAskAt
l.mu.Unlock()
have := ask != ""
if have {
st.LastAsk, st.LastAskAt = ask, at.UTC().Format(time.RFC3339)
}
path := l.ProfileRecordPath
if path == "" {
path = DefaultProfileRecordPath
}
if raw, err := os.ReadFile(path); err == nil && len(raw) < 64<<10 {
var rec struct {
Kernel string `json:"kernel"`
At string `json:"at"`
Rewritten []string `json:"rewritten"`
Restarted []string `json:"restarted"`
}
if json.Unmarshal(raw, &rec) == nil && (rec.Kernel != "" || rec.At != "") {
have = true
fromRec := profileStatusFrom(rec.Kernel, rec.At, rec.Rewritten, rec.Restarted, true)
if st.At == "" || fromRec.At == st.At || st.Rewritten == nil {
fromRec.LastAsk, fromRec.LastAskAt = st.LastAsk, st.LastAskAt
st = fromRec
}
}
}
if !have {
return nil
}
return st
}
// profileStatusFrom checks and bounds what goes to the hub. detail=false: the count and the units are not known.
func profileStatusFrom(kernel, at string, rewritten, restarted []string, detail bool) ProfileRefreshStatus {
var st ProfileRefreshStatus
if profileNameRe.MatchString(kernel) {
st.Kernel = kernel
}
if _, err := time.Parse(time.RFC3339, at); err == nil {
st.At = at
}
if detail {
n := len(rewritten)
st.Rewritten = &n
for _, u := range restarted {
if len(st.Restarted) < 16 && profileNameRe.MatchString(u) {
st.Restarted = append(st.Restarted, u)
}
}
}
return st
}
// noteProfileAsk remembers how an ask ended and, when the wrapper answered, what it said about the last refresh.
func (l *Leg) noteProfileAsk(outcome string, res *ProfileRefreshResult) {
l.mu.Lock()
defer l.mu.Unlock()
l.profileAsk, l.profileAskAt = outcome, l.now()
if res == nil {
return
}
switch outcome {
case "refreshed":
units := make([]string, 0, len(res.Restarted))
for _, u := range res.Restarted {
units = append(units, u.Unit)
}
l.profileLast = profileStatusFrom(res.Kernel, l.now().UTC().Format(time.RFC3339), res.Rewritten, units, true)
case "not_needed":
if res.LastRefreshAt != "" || res.LastRefreshKernel != "" {
l.profileLast = profileStatusFrom(res.LastRefreshKernel, res.LastRefreshAt, nil, nil, false)
}
}
}
// ProfileGate takes the heavy-op gate (a backup, a restore-test or an OS step must not run beside the refresh).
type ProfileGate func() (release func(), busy string, ok bool)
// ProfileRefresh asks the wrapper once. It returns the outcome ("" when the wrapper could not be asked or refused).
// It waits for the gate: up to `tries` attempts, `wait` apart. why is the log's short reason for this call.
func (l *Leg) ProfileRefresh(ctx context.Context, why string, gate ProfileGate, tries int, wait time.Duration) string {
lg := l.log().With("mode", "profile-refresh", "why", why)
if !l.Appliance {
lg.Info("osupdate: profile refresh skipped — not an appliance install (a BYO host belongs to its owner)")
return ""
}
var release func()
for i := 1; ; i++ {
if gate == nil {
break
}
r, busy, ok := gate()
if ok {
release = r
break
}
if i >= tries || ctx.Err() != nil {
lg.Warn("osupdate: profile refresh NOT run — the box stayed busy; the next agent start asks again", "busy", busy, "tries", i)
return ""
}
lg.Info("osupdate: profile refresh waits — a heavy operation runs", "busy", busy, "try", i)
l.sleep(ctx, wait)
}
if release != nil {
defer release()
}
unlock := l.lockPass(true) // never beside an OS-update pass
defer unlock()
start := l.now()
wr, err := l.call(ctx, "prefresh"+l.now().UTC().Format("150405"), map[string]any{"release_id": "profile-refresh",
"layer": LayerHost, "mode": "profile-refresh"})
took := l.now().Sub(start).Round(100 * time.Millisecond).String()
switch {
case err != nil:
lg.Warn("osupdate: profile refresh — the wrapper gave no answer; the next agent start asks again", "err", err, "took", took)
l.noteProfileAsk("no_answer", nil)
return ""
case wr.refused():
lg.Info("osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)", "refused", string(wr.Refused))
l.noteProfileAsk("refused", nil)
return ""
case wr.failed():
lg.Error("osupdate: profile refresh FAILED — the profiles are as before; the next agent start asks again", "failed", string(wr.Failed), "took", took)
l.noteProfileAsk("failed", nil)
return "failed"
}
var res ProfileRefreshResult
_ = json.Unmarshal(wr.ProfileRefresh, &res)
switch res.Outcome {
case "refreshed":
units := make([]string, 0, len(res.Restarted))
for _, u := range res.Restarted {
units = append(units, u.Unit)
}
if len(units) > 0 {
lg.Warn("osupdate: profile refresh — security profiles reloaded from a fresh compile; a FAILED service whose profile was stale was restarted",
"kernel", res.Kernel, "restarted", strings.Join(units, ","), "rewritten", len(res.Rewritten), "wrapper_seconds", res.Seconds, "took", took)
} else {
lg.Info("osupdate: profile refresh — security profiles reloaded from a fresh compile", "kernel", res.Kernel,
"rewritten", len(res.Rewritten), "wrapper_seconds", res.Seconds, "took", took)
}
case "not_needed":
lg.Info("osupdate: profile refresh not needed", "kernel", res.Kernel, "reason", res.Reason)
default:
lg.Warn("osupdate: profile refresh — the wrapper answered with no outcome this agent knows", "outcome", res.Outcome)
}
if res.Outcome == "refreshed" || res.Outcome == "not_needed" {
l.noteProfileAsk(res.Outcome, &res)
}
return res.Outcome
}
+169
View File
@@ -0,0 +1,169 @@
package osupdate
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ---- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----
func profileCalls(w *fakeWrapper) []map[string]any {
var m []map[string]any
for _, p := range w.plans {
if p["mode"] == "profile-refresh" {
m = append(m, p)
}
}
return m
}
func refreshed() *WrapperReport {
return &WrapperReport{ProfileRefresh: json.RawMessage(`{"outcome":"refreshed","kernel":"7.0.14-28-pve","rewritten":["f6da0254.0/usr.sbin.chronyd"],` +
`"restarted":[{"unit":"chrony.service","program":"/usr/sbin/chronyd","rc":0,"active":true}],"seconds":0.5}`)}
}
// One call, a host-layer plan that names NOTHING, and nothing to the hub.
// COMPANION RED-PROOF (observed): add "units" to the plan → this test fails; drop the Appliance check → the BYO test fails.
func TestProfileRefresh_OneCallNoNamesNoHubReport(t *testing.T) {
w := &fakeWrapper{t: t, profileRep: refreshed()}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "refreshed" {
t.Fatalf("outcome = %q", got)
}
pc := profileCalls(w)
if len(pc) != 1 {
t.Fatalf("calls = %d", len(pc))
}
if len(pc[0]) != 3 || pc[0]["layer"] != LayerHost || pc[0]["release_id"] != "profile-refresh" {
t.Fatalf("the plan carries more than release_id, layer and mode: %v", pc[0])
}
if len(h.reports) != 0 {
t.Fatalf("the hub got a report: %+v", h.reports)
}
}
func TestProfileRefresh_ByoHostIsNeverAsked(t *testing.T) {
w := &fakeWrapper{t: t, profileRep: refreshed()}
l, _ := newLeg(t, w, nil)
l.Appliance = false
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "" || len(profileCalls(w)) != 0 {
t.Fatalf("a BYO host was asked: %q %v", got, profileCalls(w))
}
}
// The heavy-op gate: busy → wait and ask again; still busy after the tries → not run (the next start asks again).
// COMPANION RED-PROOF (observed): ignore `ok` from the gate → the "always busy" half fails.
func TestProfileRefresh_WaitsForTheHeavyOpGate(t *testing.T) {
w := &fakeWrapper{t: t, profileRep: refreshed()}
l, _ := newLeg(t, w, nil)
n, released := 0, 0
gate := func() (func(), string, bool) {
n++
if n < 3 {
return nil, "primary-backup", false
}
return func() { released++ }, "", true
}
if got := l.ProfileRefresh(context.Background(), "agent start", gate, 5, time.Minute); got != "refreshed" {
t.Fatalf("outcome = %q after %d gate tries", got, n)
}
if n != 3 || released != 1 || len(profileCalls(w)) != 1 {
t.Fatalf("gate tries %d, released %d, calls %d", n, released, len(profileCalls(w)))
}
w2 := &fakeWrapper{t: t, profileRep: refreshed()}
l2, _ := newLeg(t, w2, nil)
busy := func() (func(), string, bool) { return nil, "restore-test", false }
if got := l2.ProfileRefresh(context.Background(), "agent start", busy, 3, time.Minute); got != "" || len(profileCalls(w2)) != 0 {
t.Fatalf("ran beside a heavy operation: %q %d call(s)", got, len(profileCalls(w2)))
}
}
// An older wrapper refuses the mode: logged, nothing else. A failure is said as "failed".
func TestProfileRefresh_RefusedAndFailed(t *testing.T) {
w := &fakeWrapper{t: t, profileRep: &WrapperReport{Refused: json.RawMessage(`{"code":"R11","reason":"unknown mode 'profile-refresh'"}`)}}
l, _ := newLeg(t, w, nil)
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "" {
t.Fatalf("refused → %q", got)
}
w2 := &fakeWrapper{t: t, profileRep: &WrapperReport{Failed: json.RawMessage(`{"rc":1,"step":"profile-refresh"}`)}}
l2, _ := newLeg(t, w2, nil)
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "failed" {
t.Fatalf("failed → %q", got)
}
}
// R-1383 (release 3): the refresh restarted a failed service on the N100 and only the box's journal knew. The stanza
// that rides the host report: nothing before any ask (an UNTYPED nil — the report omits the field); after a refresh,
// everything; after an agent RESTART, the kernel and time of that refresh again — from the wrapper's answer to the new
// process's ask, because the wrapper's record FILE may be root-only on a box (its folder is 0700 where a signed job
// made it first). Where the file can be read, it adds the detail back.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-b3.txt).
func TestProfileRefreshStanza_RecordAndLastAsk(t *testing.T) {
w := &fakeWrapper{t: t, profileRep: refreshed()}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
unreadable := filepath.Join(t.TempDir(), "root-only", "profile-refresh.json") // does not exist: as a 0700 folder reads
l.ProfileRecordPath = unreadable
now := time.Date(2026, 11, 14, 11, 30, 11, 0, time.UTC)
l.Now = func() time.Time { return now }
if st := l.ProfileRefreshStanza(); st != nil {
t.Fatalf("before any ask the stanza is %#v, want an untyped nil", st)
}
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "refreshed" {
t.Fatalf("outcome = %q", got)
}
b, _ := json.Marshal(l.ProfileRefreshStanza())
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":1,"restarted":["chrony.service"],"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"}` {
t.Fatalf("stanza after a refresh, record file unreadable = %s", b)
}
// an agent restart: a new process asks, the wrapper answers "not needed" with ITS record of the last refresh
l2, _ := newLeg(t, &fakeWrapper{t: t, profileRep: &WrapperReport{ProfileRefresh: json.RawMessage(
`{"outcome":"not_needed","reason":"this kernel was done","kernel":"7.0.14-28-pve","last_refresh_kernel":"7.0.14-28-pve","last_refresh_at":"2026-11-14T11:30:11Z"}`)}},
&hub.WireOSUpdate{Ring: 1, Enabled: true})
l2.ProfileRecordPath = unreadable
l2.Now = func() time.Time { return now.Add(3 * time.Hour) }
if st := l2.ProfileRefreshStanza(); st != nil {
t.Fatalf("a new process that has not asked and cannot read the record: %#v, want nil", st)
}
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "not_needed" {
t.Fatalf("outcome = %q", got)
}
b, _ = json.Marshal(l2.ProfileRefreshStanza())
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
t.Fatalf("stanza after an agent restart (the wrapper's answer, no detail) = %s", b)
}
// where the record file CAN be read, the detail is back — and a name that is not a unit name is dropped
rec := filepath.Join(t.TempDir(), "profile-refresh.json")
if err := os.WriteFile(rec, []byte(`{"at":"2026-11-14T11:30:11Z","kernel":"7.0.14-28-pve","restarted":["chrony.service","bad name; rm -rf"],`+
`"rewritten":["a","b"]}`), 0o644); err != nil {
t.Fatal(err)
}
l2.ProfileRecordPath = rec
b, _ = json.Marshal(l2.ProfileRefreshStanza())
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":2,"restarted":["chrony.service"],"last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
t.Fatalf("stanza with a readable record = %s", b)
}
// a failed ask is told, beside the older record
l2.Runner = (&fakeWrapper{t: t, profileRep: &WrapperReport{Failed: json.RawMessage(`{"code":"F1","reason":"apparmor_parser failed"}`)}})
l2.Now = func() time.Time { return now.Add(4 * time.Hour) }
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "failed" {
t.Fatalf("outcome = %q", got)
}
b, _ = json.Marshal(l2.ProfileRefreshStanza())
if !strings.Contains(string(b), `"last_ask":"failed","last_ask_at":"2026-11-14T15:30:11Z"`) || !strings.Contains(string(b), `"kernel":"7.0.14-28-pve"`) {
t.Fatalf("stanza after a failed ask = %s", b)
}
// a record that is not JSON says nothing and breaks nothing
_ = os.WriteFile(rec, []byte("not json"), 0o644)
l3, _ := newLeg(t, &fakeWrapper{t: t}, &hub.WireOSUpdate{Ring: 1, Enabled: true})
l3.ProfileRecordPath = rec
if st := l3.ProfileRefreshStanza(); st != nil {
t.Fatalf("an unreadable record gave the stanza %#v", st)
}
}
+45
View File
@@ -0,0 +1,45 @@
package osupdate
// The root step that keeps a readable copy of Proxmox's root CA for the agent (R-1380, D80, felhom.eu `03` §12a).
// The rule lives in the wrapper (mode pve-ca-copy: one fixed source, one fixed place, a CA certificate only); the plan
// names nothing. internal/pvetrust decides when to ask and what to do with the answer.
import (
"context"
"encoding/json"
"fmt"
)
// PVECACopyPath is where the wrapper puts the copy (felhom-os-apply PVE_CA_COPY) — the agent's `proxmox.tls.ca_file`.
const PVECACopyPath = "/etc/felhom-agent/pve-root-ca.pem"
// PVECACopyResult is the wrapper's pve_ca_copy object.
type PVECACopyResult struct {
Outcome string `json:"outcome"` // written | same
Path string `json:"path"`
SHA256 string `json:"sha256"`
NotAfter string `json:"not_after"`
DaysLeft int `json:"days_left"`
Replaced bool `json:"replaced"`
}
// PVECACopy asks the wrapper to make or refresh the copy. An error names a refusal (T1–T3, or R11 from a wrapper that
// does not know the mode yet) or a wrapper that gave no answer.
func (l *Leg) PVECACopy(ctx context.Context) (PVECACopyResult, error) {
wr, err := l.call(ctx, "pveca"+l.now().UTC().Format("150405.000"), map[string]any{"release_id": "pve-ca-copy",
"layer": LayerHost, "mode": "pve-ca-copy"})
if err != nil {
return PVECACopyResult{}, err
}
if wr.refused() {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy refused: %s", wr.Refused)
}
if wr.failed() {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy failed: %s", wr.Failed)
}
var res PVECACopyResult
if jerr := json.Unmarshal(wr.PVECACopy, &res); jerr != nil || res.Outcome == "" || res.Path == "" {
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy: the wrapper gave no result (%v)", jerr)
}
return res, nil
}
+21
View File
@@ -0,0 +1,21 @@
package pbsdr
import (
"os"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
// agent's state folder on a box, and a test must not need it.
func TestMain(m *testing.M) {
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
if err != nil {
panic(err)
}
config.CutWriteDir = dir
code := m.Run()
_ = os.RemoveAll(dir)
os.Exit(code)
}
+8 -17
View File
@@ -27,6 +27,7 @@ import (
"encoding/json"
"errors"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"log/slog"
"net/netip"
"os"
@@ -483,8 +484,10 @@ func (m *Manager) consumedFail(h, msg string, block *hub.WirePBSDR) {
// seedEscrowStorageID sets escrow.pbs_storage_id in agent.json when empty/absent — the bare
// `--selftest=escrow-create` one-liner must find its storage with no flags. A different existing
// value is NEVER clobbered (warn-and-keep). Unknown config keys are preserved verbatim
// (map[string]RawMessage read-modify-write, atomic rename).
// (map[string]RawMessage read-modify-write, in place behind a journal — config.ReplaceInPlace).
func (m *Manager) seedEscrowStorageID(storageID string) error {
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
defer config.FileMu.Unlock()
if m.configPath == "" {
return nil
}
@@ -527,30 +530,18 @@ func (m *Manager) seedEscrowStorageID(storageID string) error {
if err != nil {
return err
}
st, err := os.Stat(m.configPath)
if err != nil {
return err
}
// The config DIRECTORY (/etc/felhom-agent) is root-owned while the FILE is agent-owned —
// tmp+rename is impossible for the non-root agent (rename mutates the directory; live
// finding on the demo). So: park a recovery copy in the agent state dir, then rewrite the
// file IN PLACE (O_TRUNC). The agent is the file's only writer and the content is small;
// a torn write is recoverable from the parked copy.
// finding on the demo). So: park a copy for a person in the agent state dir, then rewrite the
// file IN PLACE through config.ReplaceInPlace, which journals the old bytes first — the next
// start repairs a cut write (R-1380, TestRecoverCutWrite_*).
if err := os.MkdirAll(m.stateDir, 0o700); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(m.stateDir, "agent.json.pre-seed"), raw, 0o600); err != nil {
return fmt.Errorf("parking the pre-seed copy: %w", err)
}
f, err := os.OpenFile(m.configPath, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
if err != nil {
return err
}
if _, err := f.Write(out); err != nil {
f.Close()
return err
}
if err := f.Close(); err != nil {
if err := config.ReplaceInPlace(m.configPath, raw, out); err != nil {
return err
}
m.logger.Info("pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)", "storage_id", storageID)
+40 -5
View File
@@ -10,6 +10,7 @@ import (
"net/http"
"net/url"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
@@ -45,6 +46,39 @@ type Client struct {
node string
token string
http doer
// tlsMu guards http: SetTLS swaps the transport while calls run (R-1380 — the move from a pinned certificate to the
// box's own root CA, and a refreshed CA copy, take effect with no agent restart).
tlsMu sync.RWMutex
timeout time.Duration
}
func (c *Client) httpDoer() doer {
c.tlsMu.RLock()
defer c.tlsMu.RUnlock()
return c.http
}
// SetTLS replaces how this client trusts the Proxmox certificate, for every call that starts after it returns. On an
// error (an unreadable CA file, a bad fingerprint) nothing changes.
func (c *Client) SetTLS(t TLSConfig) error {
tlsCfg, err := t.build()
if err != nil {
return err
}
timeout := c.timeout
if timeout == 0 {
timeout = 30 * time.Second
}
hc := &http.Client{Timeout: timeout, Transport: httpx.NewTransport(tlsCfg, 0)}
c.tlsMu.Lock()
old := c.http
c.http = hc
c.tlsMu.Unlock()
if o, ok := old.(*http.Client); ok && o != nil {
o.CloseIdleConnections() // a kept-alive connection made under the old trust must not serve the next call
}
return nil
}
// NewClient builds an API client. It validates required config and constructs the
@@ -74,10 +108,11 @@ func NewClient(cfg Config) (*Client, error) {
Transport: httpx.NewTransport(tlsCfg, 0),
}
return &Client{
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
node: cfg.Node,
token: cfg.Token,
http: hc,
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
node: cfg.Node,
token: cfg.Token,
http: hc,
timeout: timeout,
}, nil
}
@@ -125,7 +160,7 @@ func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader
req.Header.Set("Content-Type", contentType)
}
resp, err := c.http.Do(req)
resp, err := c.httpDoer().Do(req)
if err != nil {
return fmt.Errorf("proxmox: %s %s: %w", method, path, err)
}
+43
View File
@@ -0,0 +1,43 @@
package proxmox
import (
"crypto/x509"
"errors"
"strings"
)
// The causes a call to Proxmox can fail on TRUST (R-1380, felhom.eu `03` §12a). A symptom alarm („the agent channel is
// down") exists; these name the cause.
const (
TrustPinnedCertChanged = "pinned_cert_changed" // the pinned certificate is not the one Proxmox serves any more
TrustCAMismatch = "ca_mismatch" // the CA file did not sign the served certificate
TrustCertExpired = "cert_expired" // the served certificate (or the CA) has ended
TrustCAFileUnreadable = "ca_file_unreadable" // the CA file is missing or holds no certificate
TrustNameMismatch = "name_mismatch" // the certificate does not name the address the agent calls
)
// TrustProblem names why err is a TRUST failure, or "" when it is none (a refused connection, a 401, a timeout).
func TrustProblem(err error) string {
if err == nil {
return ""
}
var ua x509.UnknownAuthorityError
var inv x509.CertificateInvalidError
var hn x509.HostnameError
switch {
case strings.Contains(err.Error(), "TLS pin mismatch"):
return TrustPinnedCertChanged
case errors.As(err, &ua):
return TrustCAMismatch
case errors.As(err, &inv):
if inv.Reason == x509.Expired {
return TrustCertExpired
}
return TrustCAMismatch
case errors.As(err, &hn):
return TrustNameMismatch
case strings.Contains(err.Error(), "reading TLS CA file"), strings.Contains(err.Error(), "contained no usable certificates"):
return TrustCAFileUnreadable
}
return ""
}
+21
View File
@@ -0,0 +1,21 @@
package pvetrust
import (
"os"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
)
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
// agent's state folder on a box, and a test must not need it.
func TestMain(m *testing.M) {
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
if err != nil {
panic(err)
}
config.CutWriteDir = dir
code := m.Run()
_ = os.RemoveAll(dir)
os.Exit(code)
}
+406
View File
@@ -0,0 +1,406 @@
// Package pvetrust moves the agent's trust in Proxmox from ONE pinned certificate to the box's own root CA, and names
// the cause when a call fails on trust (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a).
//
// WHY. Proxmox makes a new node certificate by itself, at most every two years. An agent that pins the old one is then
// blind: every call fails, no host report reaches the hub, and the only alarm names a symptom. A copy of the box's own
// root CA verifies every certificate that CA signs.
//
// THE ORDER, and it is the whole point (a box must never be left with neither way working):
//
// 1. the root step makes or refreshes the copy (it refuses anything that is not one CA certificate);
// 2. a REAL call to Proxmox is made with the copy ALONE — while the fingerprint still is the configured trust;
// 3. only after that call worked, ONE write of the agent's config sets `ca_file` and clears the fingerprint;
// 4. the running client takes the new trust (no restart).
//
// If 1 or 2 fails nothing is written and the fingerprint stays. If 3 fails the fingerprint stays, in the file and in
// the running client. `proxmox.tls` keeps its order in code — a set fingerprint wins over a CA file — so the
// fingerprint is cleared LAST, in the same write that sets `ca_file`.
//
// WHO WRITES THE CONFIG. The agent itself: `agent.json` is the agent's own file (0600) in a root-owned folder, so it is
// rewritten in place after a recovery copy was parked in the agent's state folder — the same way the backup-target move
// and the PBS seed write it. No root step touches it.
package pvetrust
import (
"context"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"log/slog"
"os"
"path/filepath"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// EndsSoonDays: the copy's certificate ends within this many days → the problem „ca_copy_ends_soon". Proxmox renews its
// root CA when less than two years are left, so a healthy box never comes near it.
const EndsSoonDays = 30
// Problems this package names beyond proxmox.Trust*.
const (
ProblemCopyUnavailable = "ca_copy_unavailable" // the root step refused or did not answer
ProblemCopyNoProof = "ca_copy_does_not_verify" // a call with the copy alone failed — the fingerprint stays
ProblemCopyEndsSoon = "ca_copy_ends_soon"
ProblemConfigNotSaved = "config_not_written" // the proof worked but the config could not be written — the fingerprint stays
)
// Copy is what the root step answered.
type Copy struct {
Path string
Outcome string // written | same
NotAfter string
DaysLeft int
}
// Status is the stanza that rides the host report (`proxmox_trust`). No secret: a mode, a cause, a date.
type Status struct {
Mode string `json:"mode"` // ca | fingerprint | ca_file_own | insecure | system
Problem string `json:"problem,omitempty"` // "" = none
Detail string `json:"detail,omitempty"`
CANotAfter string `json:"ca_not_after,omitempty"`
CADaysLeft *int `json:"ca_days_left,omitempty"`
MovedAt string `json:"moved_at,omitempty"` // when this box moved from the pinned certificate to the CA copy
CheckedAt string `json:"checked_at,omitempty"`
}
// Manager holds the trust state of one agent process.
type Manager struct {
// EnsureCopy runs the root step (osupdate.Leg.PVECACopy).
EnsureCopy func(ctx context.Context) (Copy, error)
// Probe makes one real call to Proxmox with EXACTLY this trust (a throwaway client).
Probe func(ctx context.Context, t proxmox.TLSConfig) error
// Apply gives the running client the new trust (proxmox.Client.SetTLS).
Apply func(t proxmox.TLSConfig) error
// ConfigPath is agent.json ("" = the config came from the environment: nothing can be saved, so nothing is moved).
ConfigPath string
StateDir string
Logger *slog.Logger
Now func() time.Time
// MinGap between two runs that a failed call asked for (default 10 minutes).
MinGap time.Duration
runMu sync.Mutex // serialises Ensure
mu sync.Mutex
cur proxmox.TLSConfig
status Status
lastRun time.Time
}
// copyStepTimeout bounds the root step that makes the CA copy.
const copyStepTimeout = 90 * time.Second
func (m *Manager) now() time.Time {
if m.Now != nil {
return m.Now()
}
return time.Now()
}
func (m *Manager) log() *slog.Logger {
if m.Logger != nil {
return m.Logger
}
return slog.Default()
}
// Start records the trust the process started with. Call once, before Ensure.
func (m *Manager) Start(t proxmox.TLSConfig) {
m.mu.Lock()
defer m.mu.Unlock()
m.cur = t
m.status = Status{Mode: modeOf(t)}
}
func modeOf(t proxmox.TLSConfig) string {
switch {
case t.InsecureSkipVerify:
return "insecure"
case t.Fingerprint != "":
return "fingerprint"
case t.CAFile != "":
return "ca"
}
return "system"
}
// Status is the current stanza (nil-safe).
func (m *Manager) Status() *Status {
if m == nil {
return nil
}
m.mu.Lock()
defer m.mu.Unlock()
s := m.status
return &s
}
// Moved reports whether the box trusts Proxmox by the CA copy (nothing is left to move).
func (m *Manager) Moved() bool {
m.mu.Lock()
defer m.mu.Unlock()
return m.cur.Fingerprint == "" && m.cur.CAFile != ""
}
// Run asks again on a timer until ctx ends: every `pending` while the box is not moved yet (a wrapper that did not
// know the mode, a Proxmox that did not answer), every `settled` after (a renewed CA refreshes the copy).
func (m *Manager) Run(ctx context.Context, pending, settled time.Duration) {
for {
d := settled
if !m.Moved() {
d = pending
}
select {
case <-ctx.Done():
return
case <-time.After(d):
m.Ensure(ctx, "timer")
}
}
}
// Current is the trust the running client uses.
func (m *Manager) Current() proxmox.TLSConfig {
m.mu.Lock()
defer m.mu.Unlock()
return m.cur
}
// OnCallError is the hook for a failed Proxmox call: when the failure is one of TRUST, it names the cause and runs
// Ensure (at most once per MinGap) — a pinned box whose certificate changed moves to the CA copy right then, and a box
// whose CA copy was refreshed reloads it. A failure of another kind does nothing.
func (m *Manager) OnCallError(ctx context.Context, err error) {
cause := proxmox.TrustProblem(err)
if cause == "" {
return
}
m.mu.Lock()
gap := m.MinGap
if gap <= 0 {
gap = 10 * time.Minute
}
tooSoon := !m.lastRun.IsZero() && m.now().Sub(m.lastRun) < gap
if m.status.Problem == "" {
m.status.Problem, m.status.Detail = cause, causeText(cause)
}
m.mu.Unlock()
m.log().Error("pvetrust: a call to Proxmox failed on TRUST — "+causeText(cause), "cause", cause, "err", err)
if tooSoon {
return
}
m.Ensure(ctx, cause)
}
func causeText(cause string) string {
switch cause {
case proxmox.TrustPinnedCertChanged:
return "the pinned certificate is not the one Proxmox serves any more (Proxmox made a new certificate)"
case proxmox.TrustCAMismatch:
return "the CA copy did not sign the certificate Proxmox serves (the box's root CA changed, or the copy is not this box's)"
case proxmox.TrustCertExpired:
return "the certificate Proxmox serves, or its CA, has ended"
case proxmox.TrustCAFileUnreadable:
return "the CA copy is missing or holds no certificate"
case proxmox.TrustNameMismatch:
return "the certificate Proxmox serves does not name the address the agent calls"
}
return cause
}
// Ensure runs the order of the package comment once. why is a log word ("start", "daily", a trust cause). It returns
// true when the running trust changed.
func (m *Manager) Ensure(ctx context.Context, why string) bool {
// One run at a time. Ensure is called from the start, the timer, a failed collect and after a bundle install; two
// overlapping runs would both read the config, and the second would park the ALREADY-MOVED file as the "before"
// copy (losing the fingerprint from the recovery copy). Found by review, 2026-11-14.
m.runMu.Lock()
defer m.runMu.Unlock()
m.mu.Lock()
cur := m.cur
m.lastRun = m.now()
m.mu.Unlock()
lg := m.log().With("why", why, "mode", modeOf(cur))
set := func(mode, problem, detail string, c *Copy) {
m.mu.Lock()
defer m.mu.Unlock()
moved := m.status.MovedAt
m.status = Status{Mode: mode, Problem: problem, Detail: detail, MovedAt: moved, CheckedAt: m.now().UTC().Format(time.RFC3339)}
if c != nil {
d := c.DaysLeft
m.status.CANotAfter, m.status.CADaysLeft = c.NotAfter, &d
}
}
if cur.InsecureSkipVerify {
lg.Warn("pvetrust: certificate checks are switched off in the config (insecure_skip_verify) — nothing is moved")
set("insecure", "", "", nil)
return false
}
if m.EnsureCopy == nil || m.Probe == nil {
return false
}
// 1. the copy. The root step gets a deadline of its own: Ensure runs on the daemon's start path and inside the
// report loop, and a wrapper that hangs (sudo, a stuck read of /etc/pve) must not stop either.
cctx, ccancel := context.WithTimeout(ctx, copyStepTimeout)
c, err := m.EnsureCopy(cctx)
ccancel()
if err != nil {
lg.Warn("pvetrust: the root step gave no CA copy — the trust stays as it is", "err", err)
set(modeOf(cur), ProblemCopyUnavailable, truncate(err.Error(), 240), nil)
return false
}
ca := proxmox.TLSConfig{CAFile: c.Path}
onCopy := cur.Fingerprint == "" && cur.CAFile == c.Path
if cur.Fingerprint == "" && cur.CAFile != "" && cur.CAFile != c.Path {
// an operator's own CA file: never replaced
lg.Info("pvetrust: the config names its own CA file — left as it is", "ca_file", cur.CAFile)
set("ca_file_own", "", "", &c)
return false
}
if onCopy && c.Outcome != "written" && why != proxmox.TrustCAMismatch && why != proxmox.TrustCAFileUnreadable && why != proxmox.TrustCertExpired {
// the box already trusts by the copy and the copy did not change: nothing to prove again
problem, detail := "", ""
if c.DaysLeft < EndsSoonDays {
problem, detail = ProblemCopyEndsSoon, fmt.Sprintf("the CA copy ends in %d day(s), on %s", c.DaysLeft, c.NotAfter)
lg.Error("pvetrust: "+detail, "days_left", c.DaysLeft)
}
set("ca", problem, detail, &c)
return false
}
// 2. the proof: a real call with the copy ALONE — the fingerprint is still the configured trust
pctx, cancel := context.WithTimeout(ctx, 30*time.Second)
perr := m.Probe(pctx, ca)
cancel()
if perr != nil {
cause := proxmox.TrustProblem(perr)
detail := "a call to Proxmox with the CA copy alone failed"
if cause != "" {
detail += ": " + causeText(cause)
} else {
detail += " (not a trust failure: " + truncate(perr.Error(), 160) + ")"
}
if onCopy {
lg.Error("pvetrust: the box trusts Proxmox by the CA copy and the copy does not verify the served certificate", "err", perr)
set("ca", firstNonEmpty(cause, ProblemCopyNoProof), detail, &c)
return false
}
if cause == "" {
// Proxmox did not answer at all (it restarts, the box just booted): nothing is known about the copy, so no
// cause is named; the next run asks again.
lg.Warn("pvetrust: the proof could not be made now (not a trust failure) — the trust stays as it is; asked again later", "err", perr)
set(modeOf(cur), "", "", &c)
return false
}
lg.Error("pvetrust: the proof with the CA copy FAILED — the fingerprint STAYS, nothing was written", "err", perr)
set(modeOf(cur), ProblemCopyNoProof, detail+" — the fingerprint stays", &c)
return false
}
if onCopy {
// the copy was refreshed (or a trust failure asked): the running client reloads it
if m.Apply != nil {
if aerr := m.Apply(ca); aerr != nil {
set("ca", proxmox.TrustCAFileUnreadable, truncate(aerr.Error(), 200), &c)
return false
}
}
lg.Warn("pvetrust: the CA copy was refreshed and proven — the running client uses it now", "outcome", c.Outcome)
set("ca", "", "", &c)
return true
}
// 3. the config: ONE write sets ca_file and clears the fingerprint — only now, after the proof
if m.ConfigPath == "" {
lg.Warn("pvetrust: the proof worked, but the config did not come from a file — nothing can be saved, so the trust stays as it is")
set(modeOf(cur), ProblemConfigNotSaved, "the config is not a file; the move was not made", &c)
return false
}
if werr := m.writeConfig(c.Path); werr != nil {
lg.Error("pvetrust: the proof worked but the config could NOT be written — the fingerprint stays", "err", werr)
set(modeOf(cur), ProblemConfigNotSaved, truncate(werr.Error(), 200), &c)
return false
}
// 4. the running client
if m.Apply != nil {
if aerr := m.Apply(ca); aerr != nil {
lg.Error("pvetrust: the config is written but the running client could not take the CA copy — it keeps the old trust until the next start", "err", aerr)
}
}
m.mu.Lock()
m.cur = ca
m.mu.Unlock()
lg.Warn("pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy",
"ca_file", c.Path, "ca_not_after", c.NotAfter)
set("ca", "", "", &c)
m.mu.Lock()
m.status.MovedAt = m.now().UTC().Format(time.RFC3339)
m.mu.Unlock()
return true
}
func firstNonEmpty(a, b string) string {
if a != "" {
return a
}
return b
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
// writeConfig sets proxmox.tls.ca_file and clears proxmox.tls.fingerprint in ONE write of agent.json. Every other key is
// kept byte for byte. A recovery copy is parked in the state folder first; the result is read back and checked.
func (m *Manager) writeConfig(caPath string) error {
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
defer config.FileMu.Unlock()
raw, err := os.ReadFile(m.ConfigPath)
if err != nil {
return err
}
var doc map[string]json.RawMessage
if err := json.Unmarshal(raw, &doc); err != nil {
return fmt.Errorf("parse %s: %w", m.ConfigPath, err)
}
px := map[string]json.RawMessage{}
if cur, ok := doc["proxmox"]; ok {
if err := json.Unmarshal(cur, &px); err != nil {
return fmt.Errorf("parse the proxmox section: %w", err)
}
}
tlsSec := map[string]json.RawMessage{}
if cur, ok := px["tls"]; ok {
if err := json.Unmarshal(cur, &tlsSec); err != nil {
return fmt.Errorf("parse the proxmox.tls section: %w", err)
}
}
caJSON, _ := json.Marshal(caPath)
tlsSec["ca_file"] = caJSON
tlsSec["fingerprint"] = json.RawMessage(`""`)
b, _ := json.Marshal(tlsSec)
px["tls"] = b
b, _ = json.Marshal(px)
doc["proxmox"] = b
out, err := json.MarshalIndent(doc, "", " ")
if err != nil {
return err
}
out = append(out, '\n')
if m.StateDir != "" {
if err := os.MkdirAll(m.StateDir, 0o700); err != nil {
return err
}
if err := os.WriteFile(filepath.Join(m.StateDir, "agent.json.pre-pve-trust"), raw, 0o600); err != nil {
return fmt.Errorf("parking the recovery copy: %w", err)
}
}
// In place, not tmp+rename: the folder is root-owned and the file is the agent's. config.ReplaceInPlace keeps the
// old bytes in a journal first, so a power cut in the middle is repaired at the next start (R-1380); it also reads
// the file back and puts the old bytes back when the write fails with the agent alive.
if err := config.ReplaceInPlace(m.ConfigPath, raw, out); err != nil {
return err
}
return nil
}
+489
View File
@@ -0,0 +1,489 @@
package pvetrust
import (
"context"
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/hex"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"math/big"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ---- a small Proxmox: a root CA that signs node certificates, and a server that can change its certificate ----
type pki struct {
t *testing.T
caKey *ecdsa.PrivateKey
caCert *x509.Certificate
caPEM []byte
}
func newPKI(t *testing.T, cn string) *pki {
t.Helper()
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
return (&pki{t: t, caKey: k}).renewCA(cn)
}
// renewCA makes a new CA certificate with the SAME key and name (what Proxmox does when its root CA is renewed).
func (p *pki) renewCA(cn string) *pki {
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: cn},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true,
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign}
der, err := x509.CreateCertificate(rand.Reader, tpl, tpl, &p.caKey.PublicKey, p.caKey)
if err != nil {
p.t.Fatal(err)
}
p.caCert, _ = x509.ParseCertificate(der)
p.caPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
return p
}
func (p *pki) leaf() tls.Certificate {
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: "node"},
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(12 * time.Hour), IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}
der, err := x509.CreateCertificate(rand.Reader, tpl, p.caCert, &k.PublicKey, p.caKey)
if err != nil {
p.t.Fatal(err)
}
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: k}
}
func fp(c tls.Certificate) string {
s := sha256.Sum256(c.Certificate[0])
return hex.EncodeToString(s[:])
}
type pve struct {
srv *httptest.Server
mu sync.Mutex
cert tls.Certificate
}
func newPVE(t *testing.T, c tls.Certificate) *pve {
p := &pve{cert: c}
p.srv = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprint(w, `{"data":{"version":"9.2.2","release":"9.2","repoid":"x"}}`)
}))
// GetConfigForClient, not GetCertificate: httptest adds its own certificate, and for a call by IP address (no
// server name) that one would win over GetCertificate.
p.srv.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
p.mu.Lock()
defer p.mu.Unlock()
return &tls.Config{Certificates: []tls.Certificate{p.cert}}, nil
}}
p.srv.StartTLS()
t.Cleanup(p.srv.Close)
return p
}
// newCert is `pveupdate` making a new node certificate: served from the next connection on.
func (p *pve) newCert(c tls.Certificate) {
p.mu.Lock()
p.cert = c
p.mu.Unlock()
p.srv.CloseClientConnections()
}
// ---- the agent's side ----
type box struct {
t *testing.T
m *Manager
px *proxmox.Client
cfgPath string
copyPath string
source []byte // what the root step would copy (nil = the root step refuses)
events []string
applyErr error
}
func cfgJSON(fingerprint, caFile string) string {
return fmt.Sprintf(`{
"hub": {"url": "https://hub.example", "host_id": "box-1", "api_key": "KEEP-ME"},
"proxmox": {"endpoint": "X", "node": "n", "token": "u@pve!t=s", "tls": {"fingerprint": %q, "ca_file": %q, "insecure_skip_verify": false}, "custom": [1, 2, 3]},
"unknown_key": {"stays": true}
}`, fingerprint, caFile)
}
func newBox(t *testing.T, srv *pve, start proxmox.TLSConfig, source []byte) *box {
t.Helper()
dir := t.TempDir()
b := &box{t: t, cfgPath: filepath.Join(dir, "agent.json"), copyPath: filepath.Join(dir, "pve-root-ca.pem"), source: source}
if start.CAFile == "COPY" {
start.CAFile = b.copyPath
if err := os.WriteFile(b.copyPath, source, 0o644); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(b.cfgPath, []byte(cfgJSON(start.Fingerprint, start.CAFile)), 0o600); err != nil {
t.Fatal(err)
}
pcfg := proxmox.Config{Endpoint: srv.srv.URL, Node: "n", Token: "u@pve!t=s", TLS: start, HTTPTimeout: 5 * time.Second}
px, err := proxmox.NewClient(pcfg)
if err != nil {
t.Fatal(err)
}
b.px = px
b.m = &Manager{ConfigPath: b.cfgPath, StateDir: filepath.Join(dir, "state"),
EnsureCopy: func(context.Context) (Copy, error) {
b.events = append(b.events, "copy")
if b.source == nil {
return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3", "reason": "not a CA certificate"}`)
}
outcome := "same"
if cur, _ := os.ReadFile(b.copyPath); string(cur) != string(b.source) {
outcome = "written"
if err := os.WriteFile(b.copyPath, b.source, 0o644); err != nil {
t.Fatal(err)
}
}
return Copy{Path: b.copyPath, Outcome: outcome, NotAfter: "2036-08-17T15:44:42Z", DaysLeft: 3560}, nil
},
Probe: func(ctx context.Context, tl proxmox.TLSConfig) error {
b.events = append(b.events, fmt.Sprintf("probe(fp=%q,ca=%v) while the file says fp=%q", tl.Fingerprint, tl.CAFile != "", b.fileTLS().Fingerprint))
c := pcfg
c.TLS = tl
p, err := proxmox.NewClient(c)
if err != nil {
return err
}
_, err = p.Version(ctx)
return err
},
Apply: func(tl proxmox.TLSConfig) error {
b.events = append(b.events, fmt.Sprintf("apply(fp=%q,ca=%v)", tl.Fingerprint, tl.CAFile != ""))
if b.applyErr != nil {
return b.applyErr
}
return b.px.SetTLS(tl)
},
}
b.m.Start(start)
return b
}
type fileTLS struct {
Fingerprint string `json:"fingerprint"`
CAFile string `json:"ca_file"`
}
func (b *box) fileTLS() fileTLS {
var d struct {
Proxmox struct {
TLS fileTLS `json:"tls"`
} `json:"proxmox"`
}
raw, _ := os.ReadFile(b.cfgPath)
if err := json.Unmarshal(raw, &d); err != nil {
b.t.Fatalf("agent.json is not JSON any more: %v\n%s", err, raw)
}
return d.Proxmox.TLS
}
func (b *box) call() error {
_, err := b.px.Version(context.Background())
return err
}
func (b *box) has(prefix string) bool {
for _, e := range b.events {
if strings.HasPrefix(e, prefix) {
return true
}
}
return false
}
// Scenario: a NEW box — the installer wrote the copy and ca_file, no fingerprint. Nothing is proven again, nothing written.
func TestNewBox_CAOnly_NothingToMove(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("a box that already trusts by the copy changed its trust")
}
after, _ := os.ReadFile(b.cfgPath)
if string(before) != string(after) || b.has("probe") || b.has("apply") {
t.Fatalf("the config was touched or a proof ran: %v", b.events)
}
if st := b.m.Status(); st.Mode != "ca" || st.Problem != "" || st.CADaysLeft == nil || *st.CADaysLeft != 3560 {
t.Fatalf("status = %+v", st)
}
if err := b.call(); err != nil {
t.Fatalf("the call fails: %v", err)
}
}
// Scenario: an OLD box moves. The proof runs with the CA copy ALONE while the file still holds the fingerprint; only
// then ONE write sets ca_file and clears the fingerprint; every other key stays; a recovery copy is parked.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/A/redproof-agent.txt): write the config
// BEFORE the probe → the order check fails; skip the probe → TestProofFails_TheFingerprintStays fails.
func TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
old, _ := os.ReadFile(b.cfgPath)
if !b.m.Ensure(context.Background(), "start") {
t.Fatalf("the box did not move: %+v %v", b.m.Status(), b.events)
}
want := []string{"copy", fmt.Sprintf("probe(fp=\"\",ca=true) while the file says fp=%q", fp(leaf)), "apply(fp=\"\",ca=true)"}
if strings.Join(b.events, " | ") != strings.Join(want, " | ") {
t.Fatalf("order = %v\nwant %v", b.events, want)
}
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
t.Fatalf("the config's tls = %+v", got)
}
raw, _ := os.ReadFile(b.cfgPath)
for _, keep := range []string{`"KEEP-ME"`, `"unknown_key"`, `"stays"`, `"custom"`, `"u@pve!t=s"`, `"insecure_skip_verify"`} {
if !strings.Contains(string(raw), keep) {
t.Fatalf("the write lost %s:\n%s", keep, raw)
}
}
if parked, _ := os.ReadFile(filepath.Join(b.m.StateDir, "agent.json.pre-pve-trust")); string(parked) != string(old) {
t.Fatal("no recovery copy of the old config was parked")
}
if st := b.m.Status(); st.Mode != "ca" || st.MovedAt == "" || st.Problem != "" {
t.Fatalf("status = %+v", st)
}
// and now Proxmox makes a new certificate: the agent goes on, with no restart and no new client
srv.newCert(p.leaf())
if err := b.call(); err != nil {
t.Fatalf("after a new Proxmox certificate the moved box fails: %v", err)
}
}
// Scenario: the proof fails (the copy is another box's CA). The fingerprint stays — in the file, byte for byte, and in
// the running client — and the cause is named.
func TestProofFails_TheFingerprintStays(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
other := newPKI(t, "another box")
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("the trust changed although the proof failed")
}
after, _ := os.ReadFile(b.cfgPath)
if string(before) != string(after) {
t.Fatal("the config was written although the proof failed")
}
if b.has("apply") {
t.Fatalf("the running client was changed: %v", b.events)
}
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemCopyNoProof || !strings.Contains(st.Detail, "did not sign") {
t.Fatalf("status = %+v", st)
}
if err := b.call(); err != nil {
t.Fatalf("the fingerprint no longer works: %v", err)
}
}
// Scenario: Proxmox makes a new certificate on a box that still PINS the old one. The failed call names the cause, the
// box moves to the CA copy right then, and the next call works — no restart.
func TestPinnedBox_NewCertificate_HealsWithNoRestart(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
srv.newCert(p.leaf())
err := b.call()
if proxmox.TrustProblem(err) != proxmox.TrustPinnedCertChanged {
t.Fatalf("fixture: want a pin mismatch, got %v", err)
}
b.m.OnCallError(context.Background(), err)
if err := b.call(); err != nil {
t.Fatalf("the box is still blind after the move: %v (%v)", err, b.events)
}
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
t.Fatalf("the config's tls = %+v", got)
}
// a failure that is NOT one of trust does nothing
n := len(b.events)
b.m.OnCallError(context.Background(), errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused"))
if len(b.events) != n {
t.Fatalf("a refused connection started a trust run: %v", b.events[n:])
}
}
// Scenario: the root CA itself is renewed with the same key. The copy is refreshed, proven, and the running client
// reloads it; the config is not written again.
func TestCARenewedWithTheSameKey_TheCopyIsRefreshedAndReloaded(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
oldCA := string(p.caPEM)
p.renewCA("PVE CA")
srv.newCert(p.leaf())
b.source = p.caPEM
before, _ := os.ReadFile(b.cfgPath)
if !b.m.Ensure(context.Background(), "daily") {
t.Fatalf("a refreshed copy was not taken: %+v %v", b.m.Status(), b.events)
}
if cur, _ := os.ReadFile(b.copyPath); string(cur) == oldCA || string(cur) != string(p.caPEM) {
t.Fatal("the copy was not refreshed")
}
if !b.has("probe") || !b.has("apply") {
t.Fatalf("a refreshed copy must be proven and reloaded: %v", b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written again")
}
if err := b.call(); err != nil {
t.Fatalf("the call fails after the CA's renewal: %v", err)
}
}
// The root step refuses (the source is not a CA certificate): nothing changes, the cause is named.
func TestRootStepRefuses_NothingChanges(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") || b.has("probe") || b.has("apply") {
t.Fatalf("something ran without a copy: %v", b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written")
}
if st := b.m.Status(); st.Problem != ProblemCopyUnavailable || !strings.Contains(st.Detail, "T3") {
t.Fatalf("status = %+v", st)
}
}
// The proof works but the config cannot be written: the fingerprint stays, in the file and in the running client.
func TestConfigCannotBeWritten_TheFingerprintStays(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root writes a read-only file")
}
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
if err := os.Chmod(b.cfgPath, 0o400); err != nil {
t.Fatal(err)
}
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
t.Fatalf("the trust changed without a saved config: %v", b.events)
}
if got := b.fileTLS(); got.Fingerprint != fp(leaf) {
t.Fatalf("the fingerprint is gone from the file: %+v", got)
}
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemConfigNotSaved {
t.Fatalf("status = %+v", st)
}
}
// What is never moved: an operator's own CA file, a box with checks switched off, a config that is not a file.
func TestNeverMoved(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
own := filepath.Join(t.TempDir(), "own-ca.pem")
os.WriteFile(own, p.caPEM, 0o644)
for name, start := range map[string]proxmox.TLSConfig{"own CA file": {CAFile: own}, "insecure": {InsecureSkipVerify: true}} {
b := newBox(t, srv, start, p.caPEM)
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
t.Fatalf("%s: moved (%v)", name, b.events)
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatalf("%s: the config was written", name)
}
}
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
b.m.ConfigPath = ""
if b.m.Ensure(context.Background(), "start") || b.has("apply") || b.m.Status().Problem != ProblemConfigNotSaved {
t.Fatalf("a config that is not a file was 'moved': %v %+v", b.events, b.m.Status())
}
}
// The copy ends soon: the cause is named (the box still works).
func TestCopyEndsSoon(t *testing.T) {
p := newPKI(t, "PVE CA")
srv := newPVE(t, p.leaf())
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
inner := b.m.EnsureCopy
b.m.EnsureCopy = func(ctx context.Context) (Copy, error) {
c, err := inner(ctx)
c.DaysLeft, c.NotAfter = 12, "2026-11-26T00:00:00Z"
return c, err
}
b.m.Ensure(context.Background(), "daily")
if st := b.m.Status(); st.Problem != ProblemCopyEndsSoon || !strings.Contains(st.Detail, "12 day") {
t.Fatalf("status = %+v", st)
}
}
// A trust failure asks at most once per MinGap.
func TestOnCallError_Throttled(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
other := newPKI(t, "another box")
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
srv.newCert(p.leaf())
err := b.call()
now := time.Date(2026, 11, 15, 3, 0, 0, 0, time.UTC)
b.m.Now = func() time.Time { return now }
b.m.OnCallError(context.Background(), err)
n := len(b.events)
b.m.OnCallError(context.Background(), err)
if len(b.events) != n {
t.Fatalf("a second failure inside the gap ran again: %v", b.events[n:])
}
now = now.Add(11 * time.Minute)
b.m.OnCallError(context.Background(), err)
if len(b.events) == n {
t.Fatal("no new run after the gap")
}
if st := b.m.Status(); st.Problem == "" {
t.Fatalf("a blind box names no cause: %+v", st)
}
}
// Proxmox does not answer at all (it restarts): nothing is known about the copy — no cause is named, nothing is written,
// and the box counts as not moved, so the timer asks again soon.
func TestProofCannotBeMadeNow_NoCauseNamed(t *testing.T) {
p := newPKI(t, "PVE CA")
leaf := p.leaf()
srv := newPVE(t, leaf)
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
b.m.Probe = func(context.Context, proxmox.TLSConfig) error {
return errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused")
}
before, _ := os.ReadFile(b.cfgPath)
if b.m.Ensure(context.Background(), "start") {
t.Fatal("moved without a proof")
}
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
t.Fatal("the config was written")
}
if st := b.m.Status(); st.Problem != "" || st.Mode != "fingerprint" || b.m.Moved() {
t.Fatalf("status = %+v moved=%v", st, b.m.Moved())
}
}
+16
View File
@@ -18,8 +18,14 @@ type Manager struct {
logger *slog.Logger
now func() time.Time // injectable for tests
sleep func(context.Context, time.Duration)
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
onCommitted func(context.Context)
}
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
// ManagerConfig wires the commit manager.
type ManagerConfig struct {
StateDir string
@@ -119,4 +125,14 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
return
}
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
if m.onCommitted != nil {
m.onCommitted(ctx)
}
// R-1368 part 2: the update ended healthy — the folder keeps the running version's binary and no other
// (prune.go). A refusal or a failed removal is logged and changes nothing about the commit.
if removed, err := pruneStaged(m.stateDir, m.runningVersion, m.logger); err != nil {
m.logger.Warn("selfupdate: old staged binaries NOT removed after the commit", "err", err)
} else if len(removed) > 0 {
m.logger.Info("selfupdate: old staged binaries removed after the commit", "count", len(removed), "kept", "felhom-agent-"+m.runningVersion)
}
}
+3
View File
@@ -103,6 +103,9 @@ func (e *Executor) Execute(ctx context.Context, op string, params json.RawMessag
}
e.logger.Debug("agent_update: op invariants passed (semver + sha format)", "version", p.Version)
// The staged file is this call's from the download to the hand-off: a prune (prune.go) waits for it.
stagingMu.Lock()
defer stagingMu.Unlock()
dir := stagingDir(e.stateDir)
if err := os.MkdirAll(dir, 0o750); err != nil {
return fmt.Errorf("agent_update: staging dir: %w", err)
+96
View File
@@ -0,0 +1,96 @@
package selfupdate
import (
"fmt"
"log/slog"
"os"
"path/filepath"
"regexp"
"sort"
"sync"
)
// The self-update folder keeps what it needs and no more (R-1368 part 2; space list 2026-11-10, place H12).
//
// WHAT WAS WRONG. Execute downloads each new agent to <StateDir>/selfupdate/felhom-agent-<version> and removes that
// file only when the download or its checksum fails. A good update left it there for ever: measured 2026-11-10, 49 /
// 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1 — one for each version the box ever got.
//
// WHAT THE FOLDER IS NEEDED FOR. The root wrapper (felhom-os-apply, mode agent_update) reads the staged file ONCE,
// copies it to its own root-owned folder and flips from that copy. After that hand-off nothing reads the staged file:
// - the rollback (felhom-selfupdate-guarded `rollback`) needs two files and neither is a staged binary:
// `pending.json` in this folder, and `/usr/local/bin/felhom-agent.prev` outside it;
// - the commit needs `pending.json` only.
//
// So a staged binary is dead weight as soon as its update ended. Pinned by TestPrune_RollbackReadsNoStagedBinary (the
// wrapper's text) and TestPrune_RealRollbackStillWorksAfterNUpdates (the wrapper's rollback verb, run for real).
//
// THE RULE. After an update ended HEALTHY (the wrapper's `commit` returned success), remove every staged binary but
// the one of the running version. Nothing is removed:
// - while a pending marker exists or cannot be read — an update is in flight, and that is the only time a rollback
// can fire;
// - after a rollback, or at a plain start — only the commit path calls this, so the binary of a version that failed
// stays for the operator to look at until the next good update;
// - that is not a regular file with exactly the name Execute gives a staged binary. `pending.json`, a folder, a
// symlink, and any name this code does not know are never touched.
//
// NOT REACHED BY THIS: `/usr/local/bin/felhom-agent.bak-*` (space list H13). Those files are root-owned, in a folder
// the agent cannot write, and no code of the agent makes them (a person's copy before a hand deploy). R-1368 part 4.
// stagedNameRe is the exact name Execute gives a staged binary: "felhom-agent-" + a version that versionRe accepts.
// Kept in step with versionRe by TestPrune_NameShapeFollowsTheVersionRule.
var stagedNameRe = regexp.MustCompile(`^felhom-agent-([0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?)$`)
// stagingMu makes a download and a prune take turns. Without it a prune that runs while Execute downloads the NEXT
// version would remove that half-written file (its name is a staged binary's and it is not the running version).
var stagingMu sync.Mutex
// pruneStaged removes the staged binaries the folder no longer needs and returns the names it removed, sorted. It
// refuses (removes nothing, returns an error) while an update is in flight. A file it cannot remove is logged and
// skipped: a failed clean-up must never fail the commit that came before it.
func pruneStaged(stateDir, runningVersion string, logger *slog.Logger) ([]string, error) {
logger = orDefaultLogger(logger)
stagingMu.Lock()
defer stagingMu.Unlock()
if !versionRe.MatchString(runningVersion) {
return nil, fmt.Errorf("selfupdate prune: refusing — running version %q is not a version this code knows; it could not name the file to keep", runningVersion)
}
pm, err := readPending(stateDir)
if err != nil {
return nil, fmt.Errorf("selfupdate prune: refusing — the pending marker cannot be read: %w", err)
}
if pm != nil {
return nil, fmt.Errorf("selfupdate prune: refusing — an update to %s is pending (a rollback may still fire)", pm.NewVersion)
}
dir := stagingDir(stateDir)
entries, err := os.ReadDir(dir)
if err != nil {
if os.IsNotExist(err) {
return nil, nil
}
return nil, fmt.Errorf("selfupdate prune: read %s: %w", dir, err)
}
keep := "felhom-agent-" + runningVersion
var removed []string
for _, ent := range entries {
name := ent.Name()
if name == keep || !stagedNameRe.MatchString(name) {
continue // the running version's binary, pending.json, or a name this code does not know
}
if !ent.Type().IsRegular() {
logger.Warn("selfupdate prune: a staged-binary name that is not a regular file — left alone", "name", name, "type", ent.Type().String())
continue
}
// name came from ReadDir and matched stagedNameRe, so it holds no path separator: this is one exact file.
if err := os.Remove(filepath.Join(dir, name)); err != nil {
logger.Warn("selfupdate prune: cannot remove an old staged binary — left for the next good update", "name", name, "err", err)
continue
}
logger.Debug("selfupdate prune: removed an old staged binary", "name", name)
removed = append(removed, name)
}
sort.Strings(removed)
logger.Info("selfupdate prune: the self-update folder holds what it needs", "kept", keep, "removed", len(removed))
return removed, nil
}
+373
View File
@@ -0,0 +1,373 @@
package selfupdate
import (
"bytes"
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"testing"
)
// R-1368 part 2 — the self-update folder keeps what it needs and no more (prune.go).
//
// RED-PROOFS (each observed 2026-11-10, then restored):
// - drop the pruneStaged call from MaybeCommit → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with
// "after update 1 (0.170.0) the folder holds [… felhom-agent-0.169.0 felhom-agent-0.170.0 …]", and
// TestPrune_RealRollbackStillWorksAfterNUpdates fails too;
// - drop the pending-marker refusal from pruneStaged → TestPrune_RefusesWhileAnUpdateIsPending fails with
// "removed [...] while an update was pending";
// - drop the IsRegular check → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with "a symlink with a
// staged-binary name was removed".
// committingWrapper is the guarded wrapper's `commit` as far as this package can see it: it clears the pending marker.
type committingWrapper struct {
stateDir string
calls int
}
func (w *committingWrapper) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
if len(args) == 1 && args[0] == "commit" {
w.calls++
if err := os.Remove(pendingPath(w.stateDir)); err != nil {
return nil, []byte(err.Error()), err
}
}
return []byte("ok"), nil, nil
}
func stage(t *testing.T, stateDir, version string) {
t.Helper()
if err := os.MkdirAll(stagingDir(stateDir), 0o750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(stagingDir(stateDir), "felhom-agent-"+version), []byte("binary "+version), 0o755); err != nil {
t.Fatal(err)
}
}
func folder(t *testing.T, stateDir string) []string {
t.Helper()
ents, err := os.ReadDir(stagingDir(stateDir))
if err != nil {
t.Fatal(err)
}
var out []string
for _, e := range ents {
out = append(out, e.Name())
}
sort.Strings(out)
return out
}
// The consequence: after N good updates the folder holds ONE staged binary — the running version's — and every file
// that is not a staged binary is still there.
func TestPrune_AfterNUpdatesTheFolderHoldsOneBinary(t *testing.T) {
stateDir := t.TempDir()
dir := stagingDir(stateDir)
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
// Bystanders: names this code does not know, and two things that carry a staged binary's NAME but are not files.
outside := filepath.Join(t.TempDir(), "precious")
if err := os.WriteFile(outside, []byte("not ours"), 0o644); err != nil {
t.Fatal(err)
}
for _, n := range []string{"notes.txt", "felhom-agent-0.1.0.partial", "felhom-agent.bak-0.139.0"} {
if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
}
if err := os.Mkdir(filepath.Join(dir, "felhom-agent-9.9.9"), 0o750); err != nil {
t.Fatal(err)
}
if err := os.Symlink(outside, filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
t.Fatal(err)
}
bystanders := []string{"felhom-agent-0.1.0.partial", "felhom-agent-8.8.8", "felhom-agent-9.9.9", "felhom-agent.bak-0.139.0", "notes.txt"}
versions := []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1", "0.173.0-rc.1", "0.173.0"}
prev := "0.169.0"
stage(t, stateDir, prev) // what an old agent left behind
for i, v := range versions {
stage(t, stateDir, v) // Execute downloaded it
writePending(t, stateDir, PendingMarker{OldVersion: prev, NewVersion: v, SHA256: "ab", AppliedAt: "t"}) // the wrapper flipped
wrap := &committingWrapper{stateDir: stateDir}
m := newMgr(t, stateDir, v, wrap, nil)
m.MaybeCommit(context.Background())
if wrap.calls != 1 {
t.Fatalf("update %d (%s): commit calls = %d, want 1", i+1, v, wrap.calls)
}
want := append([]string{"felhom-agent-" + v}, bystanders...)
sort.Strings(want)
got := folder(t, stateDir)
if strings.Join(got, " ") != strings.Join(want, " ") {
if _, err := os.Lstat(filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
t.Fatalf("a symlink with a staged-binary name was removed: %v", err)
}
t.Fatalf("after update %d (%s) the folder holds %v, want %v", i+1, v, got, want)
}
prev = v
}
if b, err := os.ReadFile(outside); err != nil || string(b) != "not ours" {
t.Fatalf("the file behind the symlink changed: %q %v", b, err)
}
}
// While an update is pending — the only time a rollback can fire — nothing is removed.
func TestPrune_RefusesWhileAnUpdateIsPending(t *testing.T) {
stateDir := t.TempDir()
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
stage(t, stateDir, v)
}
writePending(t, stateDir, PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0", SHA256: "ab", AppliedAt: "t"})
before := folder(t, stateDir)
removed, err := pruneStaged(stateDir, "0.172.0", nil)
if err == nil || len(removed) != 0 {
t.Fatalf("removed %v while an update was pending (err=%v) — want a refusal", removed, err)
}
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
t.Fatalf("removed %v while an update was pending; folder now %v", before, got)
}
// The same through the commit path: a `commit` that reports success but leaves the marker (fakeWrapper does) must
// not lead to a prune.
m := newMgr(t, stateDir, "0.172.0", &fakeWrapper{}, nil)
m.MaybeCommit(context.Background())
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
t.Fatalf("the commit path pruned with the marker still there; folder now %v", got)
}
// A marker nobody can read is a refusal too.
if err := os.WriteFile(pendingPath(stateDir), []byte("{not json"), 0o644); err != nil {
t.Fatal(err)
}
if removed, err := pruneStaged(stateDir, "0.172.0", nil); err == nil || len(removed) != 0 {
t.Fatalf("an unreadable marker must refuse; removed %v err=%v", removed, err)
}
}
// Only a commit prunes: a plain start, a version mismatch and a failed commit remove nothing — so the binary of a
// version that was rolled back stays for the operator.
func TestPrune_OnlyAHealthyCommitPrunes(t *testing.T) {
cases := []struct {
name string
pending *PendingMarker
running string
wrap WrapperRunner
}{
{"plain start, no marker (also the state after a rollback)", nil, "0.171.0", &fakeWrapper{}},
{"marker names another version", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.171.0", &fakeWrapper{}},
{"commit fails", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.172.0", &fakeWrapper{err: fmt.Errorf("sudo: no")}},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
stateDir := t.TempDir()
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
stage(t, stateDir, v)
}
if c.pending != nil {
writePending(t, stateDir, *c.pending)
}
before := folder(t, stateDir)
newMgr(t, stateDir, c.running, c.wrap, nil).MaybeCommit(context.Background())
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
t.Fatalf("folder changed without a healthy commit: %v → %v", before, got)
}
})
}
// A running version this code cannot name a file for: refuse, do not guess.
stateDir := t.TempDir()
stage(t, stateDir, "0.170.0")
if removed, err := pruneStaged(stateDir, "dev", nil); err == nil || len(removed) != 0 {
t.Fatalf("running version \"dev\" must refuse; removed %v err=%v", removed, err)
}
}
// The name rule and the version rule say the same thing: every name Execute can write is a name the prune knows, and
// nothing else is.
func TestPrune_NameShapeFollowsTheVersionRule(t *testing.T) {
for _, v := range []string{"0.70.1", "0.172.1", "10.0.0", "0.70.2-crash", "0.173.0-rc.1"} {
if !versionRe.MatchString(v) {
t.Fatalf("test input %q is not a version versionRe accepts", v)
}
if !stagedNameRe.MatchString("felhom-agent-" + v) {
t.Errorf("Execute can stage felhom-agent-%s but the prune does not know that name", v)
}
}
for _, n := range []string{"pending.json", "felhom-agent", "felhom-agent-", "felhom-agent-1.2", "felhom-agent-0.1.0.partial",
"felhom-agent.bak-0.139.0", "felhom-agent.prev", "felhom-agent-0.1.0/x", "../felhom-agent-0.1.0", "felhom-agent-0.1.0 ", "xfelhom-agent-0.1.0"} {
if stagedNameRe.MatchString(n) {
t.Errorf("the prune would treat %q as a staged binary", n)
}
}
m := stagedNameRe.FindStringSubmatch("felhom-agent-0.173.0-rc.1")
if m == nil || !versionRe.MatchString(m[1]) {
t.Fatalf("the version part of a staged name must satisfy versionRe; got %v", m)
}
}
func wrapperText(t *testing.T) string {
t.Helper()
b, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-selfupdate-guarded"))
if err != nil {
t.Fatalf("the guarded wrapper is the other half of this proof and must be readable: %v", err)
}
return string(b)
}
// The stated dependency, pinned: the wrapper's `rollback` and `commit` read the pending marker and `.prev`, and no
// staged binary. If either verb starts to read the staging folder, this fails and prune.go's rule must be looked at.
func TestPrune_RollbackReadsNoStagedBinary(t *testing.T) {
s := wrapperText(t)
for _, want := range []string{"\nBIN=/usr/local/bin/felhom-agent\n", "\nPREV=$BIN.prev\n", "\nSTAGING=/var/lib/felhom-agent/selfupdate\n", "\nPENDING=$STAGING/pending.json\n"} {
if !strings.Contains(s, want) {
t.Fatalf("the wrapper no longer holds the line %q — the folder's rule (prune.go) was written against it", strings.TrimSpace(want))
}
}
for _, verb := range []string{"commit", "rollback"} {
i := strings.Index(s, "\n"+verb+")\n")
if i < 0 {
t.Fatalf("no %s) block in the wrapper", verb)
}
block := s[i:]
j := strings.Index(block, "\n ;;\n")
if j < 0 {
t.Fatalf("no end of the %s) block", verb)
}
block = block[:j]
if !strings.Contains(block, "$PENDING") {
t.Fatalf("%s) does not read $PENDING any more", verb)
}
code := regexp.MustCompile(`(?m)^\s*#.*$`).ReplaceAllString(block, "")
for _, bad := range []string{"STAGING", "staged", "felhom-agent-"} {
if strings.Contains(code, bad) {
t.Fatalf("%s) now mentions %q — it may read a staged binary, and the prune removes those", verb, bad)
}
}
}
if stagedNameRe.MatchString("pending.json") || stagedNameRe.MatchString("felhom-agent.prev") {
t.Fatal("the prune would remove a file the rollback needs")
}
}
// The consequence, with the wrapper's own code: after N good updates (each committed by the REAL `commit` verb and
// pruned), update N+1 goes bad and the REAL `rollback` verb puts the previous binary back. The wrapper runs from a copy
// whose four fixed paths point into a temp folder; systemctl, logger and chown are stubs.
func TestPrune_RealRollbackStillWorksAfterNUpdates(t *testing.T) {
sh, err := exec.LookPath("sh")
if err != nil {
t.Skip("no sh on this machine")
}
root := t.TempDir()
stateDir := filepath.Join(root, "state")
binDir := filepath.Join(root, "bin")
stubs := filepath.Join(root, "stubs")
for _, d := range []string{stagingDir(stateDir), binDir, stubs, filepath.Join(root, "rootstaging")} {
if err := os.MkdirAll(d, 0o750); err != nil {
t.Fatal(err)
}
}
live := filepath.Join(binDir, "felhom-agent")
s := wrapperText(t)
for old, repl := range map[string]string{
"\nBIN=/usr/local/bin/felhom-agent\n": "\nBIN=" + live + "\n",
"\nSTAGING=/var/lib/felhom-agent/selfupdate\n": "\nSTAGING=" + stagingDir(stateDir) + "\n",
"\nROOT_STAGING=/var/lib/felhom-os-apply/agent-update\n": "\nROOT_STAGING=" + filepath.Join(root, "rootstaging") + "\n",
} {
if strings.Count(s, old) != 1 {
t.Fatalf("the wrapper's path line %q is not there exactly once — this test cannot sandbox it", strings.TrimSpace(old))
}
s = strings.Replace(s, old, repl, 1)
}
// Safety: no assignment in the copy may still point at a real system path (PREV and PENDING are built from BIN and
// STAGING, so they follow).
for _, line := range strings.Split(s, "\n") {
if regexp.MustCompile(`^[A-Z_]+=/`).MatchString(line) && !strings.Contains(line, root) {
t.Fatalf("the sandboxed wrapper still holds a real path: %q", line)
}
}
script := filepath.Join(root, "wrapper.sh")
if err := os.WriteFile(script, []byte(s), 0o755); err != nil {
t.Fatal(err)
}
stubLog := filepath.Join(root, "stub.log")
for _, n := range []string{"systemctl", "logger", "chown", "systemd-run"} {
body := "#!/bin/sh\necho \"" + n + " $*\" >> '" + stubLog + "'\nexit 0\n"
if err := os.WriteFile(filepath.Join(stubs, n), []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
run := func(verb string) (string, error) {
cmd := exec.Command(sh, script, verb)
cmd.Env = []string{"PATH=" + stubs + ":/usr/bin:/bin"}
out, err := cmd.CombinedOutput()
return string(out), err
}
// applied is what the wrapper's `apply` leaves behind (it needs root, so the test writes the same four things).
applied := func(oldV, newV string) {
cur, err := os.ReadFile(live)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(live+".prev", cur, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(live, []byte("binary "+newV), 0o755); err != nil {
t.Fatal(err)
}
writePending(t, stateDir, PendingMarker{OldVersion: oldV, NewVersion: newV, SHA256: "ab", AppliedAt: "t"})
}
if err := os.WriteFile(live, []byte("binary 0.169.0"), 0o755); err != nil {
t.Fatal(err)
}
prev := "0.169.0"
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1"} {
stage(t, stateDir, v)
applied(prev, v)
m := newMgr(t, stateDir, v, runnerFunc(func(args ...string) ([]byte, []byte, error) {
out, err := run(args[0])
return []byte(out), nil, err
}), nil)
m.MaybeCommit(context.Background()) // the REAL commit verb, then the prune
if got := folder(t, stateDir); len(got) != 1 || got[0] != "felhom-agent-"+v {
t.Fatalf("after the good update to %s the folder holds %v", v, got)
}
prev = v
}
// Update N+1 is applied and goes bad: systemd's OnFailure runs the wrapper's rollback.
stage(t, stateDir, "0.173.0")
applied("0.172.1", "0.173.0")
out, err := run("rollback")
if err != nil {
t.Fatalf("the real rollback verb failed after the prunes: %v\n%s", err, out)
}
if b, _ := os.ReadFile(live); string(b) != "binary 0.172.1" {
t.Fatalf("after the rollback the live binary is %q, want the previous version's", b)
}
if p, _ := readPending(stateDir); p != nil {
t.Fatal("the rollback left the pending marker")
}
if b, _ := os.ReadFile(stubLog); !bytes.Contains(b, []byte("systemctl restart felhom-agent.service")) {
t.Fatalf("the rollback did not restart the agent; stub log:\n%s", b)
}
// The rolled-back agent starts: no marker, so no commit and no prune — the bad version's binary stays to be read.
newMgr(t, stateDir, "0.172.1", &fakeWrapper{}, nil).MaybeCommit(context.Background())
if got := folder(t, stateDir); strings.Join(got, " ") != "felhom-agent-0.172.1 felhom-agent-0.173.0" {
t.Fatalf("after the rollback the folder holds %v, want the running version and the one that failed", got)
}
}
// runnerFunc adapts a function to WrapperRunner; it gets the verb arguments only.
type runnerFunc func(args ...string) ([]byte, []byte, error)
func (f runnerFunc) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
return f(args...)
}
+101
View File
@@ -0,0 +1,101 @@
package signedjobs
import (
"context"
"errors"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// outcomeQueue is a jobsQueue that also has the optional half (OutcomeSource), as *hub.Client does.
type outcomeQueue struct {
jobsQueue
omu sync.Mutex
outcomes map[string]hub.JobOutcome
}
func (q *outcomeQueue) CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error {
q.omu.Lock()
if q.outcomes == nil {
q.outcomes = map[string]hub.JobOutcome{}
}
q.outcomes[jobID] = o
q.omu.Unlock()
return q.jobsQueue.CompleteJob(ctx, jobID)
}
// R-1383 (D78): a signed job that is refused or fails was cleared from the hub's queue exactly like one that ran —
// „an agent update never arrives” left one journal line on the box. The runner now says how each job ended, through
// the REAL gate and verifier: the stored effect is the outcome the hub receives with the clearing call.
// The two clock reasons are the ones the ruling is about: a box with a wrong clock refuses every job.
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-job-outcome.txt).
func TestRunner_TellsTheHubHowEachJobEnded(t *testing.T) {
s := newTestSigner(t)
_, _, exec := newRealGateRunner(t, s)
base, _, _ := newRealGateRunner(t, s)
src := &outcomeQueue{}
r := NewRunner(src, base.gate, exec, testHost, quiet())
now := time.Now().UTC()
src.add(mintJob(t, s, "ok", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
src.add(mintJob(t, s, "expired", testHost, "", "ops-1", wipeParamsJSON, now.Add(-3*time.Hour), now.Add(-2*time.Hour)))
src.add(mintJob(t, s, "early", testHost, "", "ops-1", wipeParamsJSON, now.Add(2*time.Hour), now.Add(3*time.Hour)))
src.add(mintJob(t, s, "other-box", "some-other-host", "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
src.add(corruptSig(t, mintJob(t, s, "forged", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))))
src.add(hub.JobWire{JobID: "garbage", BlobB64: "!!!not-base64!!!"})
if _, err := r.RunOnce(context.Background()); err != nil {
t.Fatal(err)
}
// and a job that is accepted and whose run fails
exec.err = errors.New("the wrapper refused")
src.add(mintJob(t, s, "run-fails", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
if _, err := r.RunOnce(context.Background()); err != nil {
t.Fatal(err)
}
want := map[string]hub.JobOutcome{
"ok": {Outcome: "done", Op: "storage_wipe"},
"expired": {Outcome: "rejected", Reason: "expired", Op: "storage_wipe"},
"early": {Outcome: "rejected", Reason: "not_yet_valid", Op: "storage_wipe"},
"other-box": {Outcome: "rejected", Reason: "target", Op: "storage_wipe"},
"forged": {Outcome: "rejected", Reason: "malformed", Op: "storage_wipe"}, // corruptSig breaks the armour itself
"garbage": {Outcome: "malformed", Reason: "malformed"},
"run-fails": {Outcome: "failed", Reason: "execution_failed", Op: "storage_wipe"},
}
for id, w := range want {
if got, ok := src.outcomes[id]; !ok || got != w {
t.Errorf("job %q: the hub was told %+v (told=%v), want %+v", id, got, ok, w)
}
}
if len(src.outcomes) != len(want) {
t.Errorf("outcomes for %d jobs, want %d: %+v", len(src.outcomes), len(want), src.outcomes)
}
if exec.count() != 2 {
t.Errorf("executor ran %d times, want 2 (ok and run-fails only)", exec.count())
}
}
// Every code reasonCode can return is on the closed list the hub holds a sentence for.
func TestReasonCode_ClosedList(t *testing.T) {
known := map[string]bool{}
for _, c := range ReasonCodes {
known[c] = true
}
for _, d := range []reconcile.Decision{
{}, {Reason: reconcile.ReasonRejected, Err: errors.New("something new")},
{Reason: reconcile.ReasonRoleDenied}, {Reason: reconcile.ReasonBindingMismatch}, {Reason: reconcile.ReasonPendingSignature},
} {
if c := reasonCode(d); !known[c] {
t.Errorf("reasonCode(%+v) = %q, not on ReasonCodes", d, c)
}
}
if reasonCode(reconcile.Decision{Reason: reconcile.ReasonRejected, Err: errors.New("x")}) != "other" {
t.Error("an unknown refusal must be told as „other”, never as the error's own text")
}
if !known["execution_failed"] || !known["malformed"] {
t.Error("the two codes the runner sets itself are not on the list")
}
}
+48 -6
View File
@@ -35,6 +35,42 @@ type JobSource interface {
CompleteJob(ctx context.Context, jobID string) error
}
// OutcomeSource is the optional half of a JobSource: it clears a job AND tells the hub how it ended (R-1383, D78).
// *hub.Client has it. A source without it is cleared as before.
type OutcomeSource interface {
CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error
}
// ReasonCodes is the CLOSED list of reasons the agent sends with a job that did not run to a good end. The hub holds
// one operator sentence per code (felhom.eu hub/internal/api, signedJobReasons); a code the hub does not know is
// told as „other”. Pinned by TestReasonCode_ClosedList.
var ReasonCodes = []string{"expired", "not_yet_valid", "window_too_long", "replay", "bad_signature", "unknown_signer",
"target", "namespace", "malformed", "role_denied", "binding_mismatch", "execution_failed", "other"}
// reasonCode names why the gate refused: the authz error first (the two clock reasons matter most — a box with a
// wrong clock refuses EVERY job), then the gate's own reason.
func reasonCode(dec reconcile.Decision) string {
for _, m := range []struct {
err error
code string
}{
{authz.ErrExpired, "expired"}, {authz.ErrNotYetValid, "not_yet_valid"}, {authz.ErrWindowTooLong, "window_too_long"},
{authz.ErrReplay, "replay"}, {authz.ErrBadSignature, "bad_signature"}, {authz.ErrUnknownSigner, "unknown_signer"},
{authz.ErrTarget, "target"}, {authz.ErrNamespace, "namespace"}, {authz.ErrMalformed, "malformed"},
} {
if errorsIs(dec.Err, m.err) {
return m.code
}
}
switch dec.Reason {
case reconcile.ReasonRoleDenied:
return "role_denied"
case reconcile.ReasonBindingMismatch:
return "binding_mismatch"
}
return "other"
}
// Authorizer is the reversibility gate. Satisfied by *reconcile.Gate.
type Authorizer interface {
Authorize(intent reconcile.Intent, signed *reconcile.SignedOp) reconcile.Decision
@@ -141,7 +177,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
// A malformed/forged envelope is permanently bad — clear it loudly (a queued garbage job
// must not wedge the queue). No nonce was consumed (we never reached Verify).
r.logger.Error("signedjobs: malformed job envelope — clearing", "job", j.JobID)
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "malformed", Reason: "malformed"})
return true
}
@@ -166,7 +202,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
// it as a security event (a compromised hub queuing a forged op lands here).
r.logger.Warn("signedjobs: REJECTED signed op — executor not called",
"job", j.JobID, "op", ob.Op, "reason", dec.Reason, "err", errStr(dec.Err))
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "rejected", Reason: reasonCode(dec), Op: ob.Op})
return true
}
@@ -177,7 +213,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
switch {
case err == nil:
r.logger.Warn("signedjobs: signed op COMPLETED", "job", j.JobID, "op", ob.Op)
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "done", Op: ob.Op})
return true
case errorsIs(err, ErrNoExecutor):
// No executor for this op class in this build — leave it queued for the owning slice.
@@ -189,7 +225,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
// nonce to retry) and log the failure loudly.
r.logger.Error("signedjobs: signed op execution FAILED (nonce spent — clearing)",
"job", j.JobID, "op", ob.Op, "err", err)
r.complete(ctx, j.JobID)
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "failed", Reason: "execution_failed", Op: ob.Op})
return true
}
}
@@ -215,8 +251,14 @@ func (r *Runner) decode(j hub.JobWire) (*reconcile.SignedOp, authz.OpBlob, bool)
return &reconcile.SignedOp{Blob: opBlob, Sig: []byte(env.SigArmored)}, ob, true
}
func (r *Runner) complete(ctx context.Context, jobID string) {
if err := r.source.CompleteJob(ctx, jobID); err != nil {
func (r *Runner) complete(ctx context.Context, jobID string, o hub.JobOutcome) {
var err error
if os, ok := r.source.(OutcomeSource); ok {
err = os.CompleteJobWithOutcome(ctx, jobID, o) // R-1383: the hub hears HOW the job ended
} else {
err = r.source.CompleteJob(ctx, jobID)
}
if err != nil {
r.logger.Warn("signedjobs: clearing job failed (will retry next pass)", "job", jobID, "err", err)
}
}
+28 -3
View File
@@ -98,7 +98,8 @@ TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
# THE retention number, read from the one file that owns it. A check and the policy it enforces
# must read the same number from the same place, or they drift and the drift looks like a defect
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json")
RETENTION_FILE = os.environ.get("FELHOM_RETENTION_POLICY") or \
os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") # the env: decoy tests only
def retention_kept():
@@ -114,6 +115,21 @@ def retention_kept():
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
return n
def retention_pinned():
"""D50 = A: the versions kept besides the newest N — the hub's MinAgent floor and the day-0 agent. Copies of hub
state the vouching session writes into the policy file (CI holds no hub secret). Missing key = no pins."""
with open(RETENTION_FILE, encoding="utf-8") as fh:
pins = json.load(fh).get("pinned") or {}
if not isinstance(pins, dict):
raise ValueError("pinned must be an object, got %r" % (pins,))
out = {}
for name, v in pins.items():
if not isinstance(v, str) or not re.match(r"^\d+\.\d+\.\d+$", v):
raise ValueError("pinned %s must be X.Y.Z, got %r" % (name, v))
out[name] = v
return out
tried = []
@@ -211,6 +227,7 @@ def main():
all_versions = versions
try:
keep = retention_kept()
pins = retention_pinned()
except Exception as e:
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
@@ -218,10 +235,15 @@ def main():
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
def _key(v):
return tuple(int(x) for x in v.split("."))
versions = sorted(all_versions, key=_key)[-keep:]
window = sorted(all_versions, key=_key)[-keep:]
# D50 = A: the pinned versions are asserted wherever they sit. A pin with no tag cannot be installed at all —
# a failure, never a skip.
unpinned_tags = sorted(("%s=%s" % (k, v)) for k, v in pins.items() if v not in all_versions)
versions = sorted(set(window) | set(v for v in pins.values() if v in all_versions), key=_key)
dropped = [v for v in all_versions if v not in versions]
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep))
print(" %d released version(s); retention policy keeps the newest %d plus the pinned %s" % (
len(all_versions), keep, ", ".join("%s=%s" % kv for kv in sorted(pins.items())) or "(none)"))
print(" verifying: %s" % ", ".join(versions))
if dropped:
# NEVER silent. A bounded check that does not say what it stopped covering is how a
@@ -232,6 +254,9 @@ def main():
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
bad = []
for pin in unpinned_tags:
bad.append((pin, ["the pinned version has NO git tag — nothing can install it (D50 = A)"]))
print(" FAIL pinned %s: no git tag v%s" % (pin, pin.split("=")[1]))
for v in versions:
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
+3
View File
@@ -33,6 +33,9 @@
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
# GITEA_BASE / GITEA_OWNER override the defaults.
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
# GITEA_GIT_TOKEN — the tag push (`git push origin <tag>`) needs it since 2026-11-11: the address of a CC clone
# on DooPlex holds no credential, and git takes the token from the secret store through the helper
# felhom.eu/scripts/git-credential-sec.sh. Run the release as `sec run GITEA_GIT_TOKEN,... -- scripts/release-agent.sh <version>`.
set -euo pipefail
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
+23 -31
View File
@@ -1,41 +1,33 @@
{
"_comment": [
"THE retention number for published agent artifacts. One file, read by everything that",
"depends on it, because a check and the policy it enforces must read the same number from the",
"same place or they drift — and the drift looks like a defect in something else.",
"THE retention rule for published agent artifacts. One file, read by everything that depends on it, because a check",
"and the policy it enforces must read the same rule from the same place or they drift — and the drift looks like a",
"defect in something else (R-287, 2026-08-08/09; again R-1290, 2026-10-25).",
"",
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent",
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag",
"still be downloadable. Both rules are individually sensible; together they are impossible.",
"CI went red at a commit whose own run had been green the day before, on a true finding that",
"no one could act on. The red will return at the next publish unless the two read one number.",
"D50 = A (operator ruling 2026-10-25, felhom.eu `09` §3 decision 262): the registry keeps the NEWEST 5 released agent",
"versions, PLUS the hub's MinAgent floor, PLUS the version the hub's day-0 artifact manifest names. The operator prunes",
"by hand to that rule (on 2026-10-25 he removed 0.159.0–0.160.0 and kept 0.158.1, the floor). CI asserts exactly these",
"and never more. Before D50 the number was 10 (the R-267 newest-10 prune, R-287/R-291).",
"",
"HOW THE NUMBER WAS ARRIVED AT. generic_versions_kept is 10 because that is what the registry",
"keeps: the deleter was ESTABLISHED on 2026-08-10 (R-287) — the newest-10 prune of generic packages",
"run under R-267 (felhom-agent and felhom-golden generic held exactly 10 afterwards). Until then this",
"file called the 10 an observed state with no located ruling; that is superseded (corrected",
"2026-10-05, R-291). Container packages are not pruned by that rule.",
"THE PINS ARE COPIES OF HUB STATE, because CI cannot read the hub without a secret it must not hold. The session that",
"records an agent vouch (day0_agent) or declares a new MinAgent with a floor (hub_min_agent) updates `pinned` in the",
"same commit (felhom-build-deploy skill, agent release). A pin that names a version with no tag is a FAILURE, never a",
"skip — a pin nobody can install is the defect the pin exists to catch.",
"",
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
"operator should confirm or replace the number — at which point this file changes and both",
"readers follow it in the same commit.",
"SO THIS FILE IS A FLOOR, NOT A LICENCE: it says what CI may assume is still downloadable. It does not authorise",
"deleting anything else.",
"",
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled",
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an",
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
"the gate does not have today. Filed as the follow-up in R-287.",
"",
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
"version at all — a strictly worse act than an un-downloadable binary."
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config files from",
"raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that version at all."
],
"generic_versions_kept": 10,
"generic_versions_kept": 5,
"readers": [
"scripts/check-published-versions.py — bounds its assertion to the newest N versions"
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
],
"recorded": "2026-08-09",
"recorded_by": "CC 2026-08-09; the number's source (the R-267 newest-10 prune, established 2026-08-10 by R-287) recorded 2026-10-05 (R-291)"
"recorded": "2026-11-14",
"recorded_by": "CC 2026-11-14: day0_agent 0.174.0 (operator vouch 2026-11-14 15:21:45 CET, read from the hub log and from the hub's System page: agent=0.174.0, bundle 7d87d058… = the release; the save took 428 ms); earlier 0.173.0 (CC 2026-11-10) (operator vouch 2026-11-10 16:29:16 CET, read from the hub log: agent=0.173.0, bundle 51ebc1f4… = the release; the save took 232 ms); earlier 0.172.1 (CC 2026-11-07) (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.174.0"
}
}
+31 -1
View File
@@ -179,13 +179,24 @@ def published_cases(base):
TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3
VERS = [t[1:] for t in TAGS]
# D50 = A: the gate also asserts the PINNED versions. The decoys run against their own policy file (the real pins
# name versions the fake Gitea does not hold); PINS is what each case writes there.
pol_dir = tempfile.mkdtemp(prefix="retention-decoy-")
pol = os.path.join(pol_dir, "retention-policy.json")
PINS = {}
def case(name, setup, expect_rc, must=()):
FAKE.reset()
FAKE.tags = list(TAGS)
FAKE.packages = set(VERS)
FAKE.raw = set(VERS)
PINS.clear()
setup()
rc, out = run([sys.executable, gate], ROOT, child_env(base))
with io.open(pol, "w", encoding="utf-8") as fh:
json.dump({"generic_versions_kept": keep, "pinned": dict(PINS)}, fh)
env = child_env(base)
env["FELHOM_RETENTION_POLICY"] = pol
rc, out = run([sys.executable, gate], ROOT, env)
if not FAKE.hits:
fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name)
report("published: " + name, rc, out, expect_rc, must)
@@ -220,6 +231,25 @@ def published_cases(base):
case("BY DESIGN: a version older than the retention window is not asserted", retired, 0,
("NOT ASSERTED", "0.9.0"))
def pinned_old_ok():
FAKE.tags = ["v0.8.0"] + ["v0.10.%d" % i for i in range(keep)]
FAKE.packages = set(t[1:] for t in FAKE.tags)
FAKE.raw = set(t[1:] for t in FAKE.tags)
PINS["hub_min_agent"] = "0.8.0"
case("GENUINE (D50): a pinned version older than the window is asserted and downloads", pinned_old_ok, 0,
("verifying: 0.8.0", "ALL RELEASED VERSIONS INSTALLABLE"))
def pinned_old_gone():
pinned_old_ok()
FAKE.packages.discard("0.8.0")
case("FACT (D50): a pinned version older than the window whose package 404s", pinned_old_gone, 1,
("FAIL v0.8.0", "binary NOT downloadable"))
def pinned_no_tag():
PINS["day0_agent"] = "0.150.9"
case("FACT (D50): a pin that names a version with no tag", pinned_no_tag, 1,
("FAIL pinned day0_agent=0.150.9", "no git tag"))
def five_hundred():
FAKE.tags_status = 500
case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",))