Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
88d5d56b50 | ||
|
|
5ecdd3d521 | ||
|
|
3ea208b8fb | ||
|
|
db01ae0f2a | ||
|
|
54ac2c2ac9 | ||
|
|
ec7e699fd7 | ||
|
|
61feb95c06 | ||
|
|
c0bf97de80 | ||
|
|
d5ce5c364a | ||
|
|
a1135018c7 | ||
|
|
ede5d0e6c7 | ||
|
|
386a0c4cf4 | ||
|
|
a9ab47ec71 | ||
|
|
2706e35198 | ||
|
|
cc63dd5204 | ||
|
|
91a2924ad0 | ||
|
|
9edaa8c082 | ||
|
|
5cebb766e3 | ||
|
|
776025c824 | ||
|
|
41d5eb8051 | ||
|
|
8bd137787e | ||
|
|
39db03065a | ||
|
|
ba409de4a2 | ||
|
|
a4f1280399 | ||
|
|
ecbb0cc479 | ||
|
|
50eeccc986 | ||
|
|
808cdb36ad | ||
|
|
84726c9f27 | ||
|
|
bd18347c1c | ||
|
|
98bd1b71e9 | ||
|
|
6578a8b91e | ||
|
|
ca15a31690 | ||
|
|
bdea4694a4 | ||
|
|
db25eb0e71 | ||
|
|
637c92c762 | ||
|
|
63e63ee8bb | ||
|
|
6b0b093199 | ||
|
|
f4e63a55da | ||
|
|
91bd96c755 | ||
|
|
6f933d6ee3 | ||
|
|
5ddd124cfc | ||
|
|
3055bc54c1 | ||
|
|
007199342f | ||
|
|
54d3a1057b | ||
|
|
fbbb168d1f | ||
|
|
3911cc6f52 | ||
|
|
ad80ee749a | ||
|
|
0683cc2dab | ||
|
|
b1bc63dd32 | ||
|
|
e66a48f2b7 | ||
|
|
b890b57540 | ||
|
|
aa69114d18 | ||
|
|
abcc1e6833 | ||
|
|
d912307bb7 | ||
|
|
feaaa0e8ff | ||
|
|
6b6ba0c17f | ||
|
|
4697f8ac9e | ||
|
|
4a1a84941c | ||
|
|
c8d3c17b13 | ||
|
|
f5c3d77a61 | ||
|
|
5e05627c4e | ||
|
|
1e087bb158 | ||
|
|
29bcf4403e | ||
|
|
08ca67861f | ||
|
|
f7e1dcbb27 | ||
|
|
2d4b1930df | ||
|
|
b4721eee11 |
No files matched your search
@@ -50,6 +50,12 @@ it **first** in the morning note. A decision you cannot write in that shape is o
|
||||
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
|
||||
on all three — not at night unless the night brief says so. **Agent releases still stop for the
|
||||
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
|
||||
**A version number is pushed once** — `felhom-controller/controller/build.sh` now refuses a second
|
||||
push of a tag that exists, and a registry it cannot reach is a refusal too. This is a FACT about
|
||||
what the build does, not a new rule: a fix for something already pushed is the next patch version,
|
||||
and a box on a bad image is moved by the normal update path, never by re-tagging. Earned on
|
||||
2026-10-15, when 0.315.0 was pushed three times from three different commits and a cache kept
|
||||
serving the first one (`09-update-architecture.md` §0).
|
||||
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
|
||||
Prose is not a record.
|
||||
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
|
||||
|
||||
+529
-3
@@ -1,4 +1,530 @@
|
||||
## Unreleased (2026-10-22/23 night) — the tunnel heals itself after a clock that ran ahead (D30, R-1204) — COMMITTED, NOT RELEASED
|
||||
## Unreleased (after v0.175.0, 2026-11-15) — root files older than the agent are not a trust problem
|
||||
|
||||
- **`pvetrust.Ensure`: the wrapper's „unknown mode” refusal on a PINNED box names no problem.** Between the signed
|
||||
agent update and the signed root-files job, the box's wrapper does not know `pve-ca-copy`. 0.175.0 reported
|
||||
`ca_copy_unavailable` for those minutes and the hub mailed the operator (seen on Tester 1, 2026-11-15 10:04). The
|
||||
pin works, so nothing is wrong; the bundle install runs the move. An unpinned box, and every other refusal, still
|
||||
names the problem. Test `TestRootStepOlderThanTheAgent_IsNotATrustProblem`; 3 mutations seen red.
|
||||
|
||||
## v0.175.0 (2026-11-15) — trust in Proxmox by a copy of the box's own root CA (R-1380, release 2b); the agent tells the hub its clock, a refused job, a give-up and what a clean-up did (release 3); a boot key a clock step cannot move (R-1395) — RELEASED 2026-11-15 (release-agent.sh from the clean working clone at `db01ae0`, CI run id 2671; agent sha256 d1a30ed51f6663d96369ad3e88c221194af92773addf4fbb48852ee58be4b19a, bundle sha256 3b4a8b78155fc4385148c98c229da2e61e7c6b02bad9ed9f1fdc4a9ea1370d95); NOT vouched yet
|
||||
|
||||
`configs/` changed (the wrapper's `pve-ca-copy` mode), so a box needs BOTH signed jobs: `agent_update`, then
|
||||
`agent_config_update`. The three sections below were built on 2026-11-14 night and 2026-11-15 morning.
|
||||
|
||||
### A cut write of `agent.json` cannot leave a blind box (R-1380; found before the release, 2026-11-15)
|
||||
|
||||
- **`config.ReplaceInPlace` + `config.RecoverCutWrite`** (`internal/config/cutwrite.go`). `agent.json` is the agent's
|
||||
file in a root-owned folder, so the trust move, the PBS seed and the backup-target move write it in place
|
||||
(truncate, write). A power cut in between left an empty or half file and an agent that does not start. Now the old
|
||||
bytes go into a journal in the state folder first (`agent.json.write-in-flight`, durable), the config is written,
|
||||
synced and read back, and the journal is removed; the daemon's start repairs a config that does not parse from the
|
||||
journal BEFORE it reads it. A selftest never repairs. No journal folder → no write. Tests: a cut staged at each of
|
||||
four steps, the wiring in `main`, and a source pin that the three writers use the helper and open no file by hand.
|
||||
Red-proofs: felhom.eu `documentation/audits/release-2b-3-2026-11-15/part0/`. Not proven: a real power cut.
|
||||
|
||||
### Release 3 („the hub tells you”) and the clock-step fix, the agent's parts (2026-11-14 night)
|
||||
|
||||
Commit only. Each part can be released alone; none needs release 2b. No `configs/` change in this section.
|
||||
|
||||
- **Fixes after an independent review of the night's code (2026-11-14; felhom.eu
|
||||
`documentation/audits/night-build-2026-11-14/review/`).** (1) `config.FileMu`: the three functions that read
|
||||
`agent.json` whole and write it whole (the trust move, the PBS seed, the backup-target move) take one lock — two of
|
||||
them could interleave and the second put back the document it read before the first one's change; pinned from the
|
||||
source by `TestAgentJSONWritersTakeTheSharedLock`. (2) `pvetrust.Manager.Ensure` runs one at a time (start, timer,
|
||||
a failed collect and a bundle install could overlap, and the second parked the already-moved file as the „before”
|
||||
copy), and its root step has a 90-second deadline (it runs on the start path and inside the report loop). (3) **The
|
||||
CA copy keeps its previous certificate behind the new one** (`configs/felhom-os-apply`, mode `pve-ca-copy`): the
|
||||
copy was replaced before anything could be proven with the new root, and Proxmox serves its old certificate until
|
||||
pveproxy is reloaded — an agent that restarted in that window had no root that verified it. **Only for the
|
||||
changeover:** at every later run the step makes one handshake to the local Proxmox with the NEW root alone, and
|
||||
drops the previous root as soon as that verifies, or when the previous root is no longer a CA that has not ended.
|
||||
The copy is compared exactly, so nothing the step did not check stays in it. To drop the old root at once by hand
|
||||
(a root changed because it leaked): delete the copy. `configs/` changes again, so the release needs both signed jobs.
|
||||
(4) The profile-refresh stanza is built from the wrapper's ANSWER (its record of the last refresh comes back at
|
||||
every ask); the record file is a bonus — its folder is root-only on a box where a signed job made it first (read in
|
||||
source, not on a box). `rewritten` is absent when not known. (5) A watchdog give-up is forgotten when the guest is
|
||||
no longer the watchdog's (onboot off, locked, destroyed) — it was reported for ever. (6) `/disks` sends both boot
|
||||
tokens from ONE read of the guest's init (`GuestBootTokens`). 13 mutations seen red, two after a mutant that did
|
||||
not compile was done again.
|
||||
- **The hub is told what the kernel clean-up did and what the profile refresh did (R-1383; felhom.eu `11` §5.11.1,
|
||||
§5.12).** (1) The clean-up's report carries `kernel_cleanup` (`osupdate.CleanupDetail`: removed, kept, bytes_gained,
|
||||
leftovers). It still has no `kernel` view and `release_id` stays `kernel-cleanup` — the hub reads only those two as
|
||||
„the kernel this step was for” (the test asserts the wire form). (2) The host report carries `profile_refresh`
|
||||
(`osupdate.ProfileRefreshStatus`, from `Leg.ProfileRefreshStanza`, wired in `main.go`): the root wrapper's own record
|
||||
of the last refresh — so it outlives an agent restart — and how this process's last ask ended. One small file read,
|
||||
no wrapper call, names checked and capped. No OS report is sent for the refresh (an unknown layer reads as a guest
|
||||
step on the hub). Tests `TestKernelCleanup_NightOrderOnceAndNoNames` (extended), `TestProfileRefreshStanza_RecordAndLastAsk`,
|
||||
`TestMainWiresPVETrust`; 9 mutations seen red (one mutant did not compile at first and was done again).
|
||||
- **A boot token that a clock step cannot move (R-1395).** `/disks` answers `guest_boot_key` =
|
||||
`<kernel boot_id>-<guest init start>` beside the unchanged `guest_boot_id` (`<boot TIME>-<guest init start>`). The
|
||||
kernel computes the boot time from the wall clock, so a step of the host's clock moved the old token and the
|
||||
controller recreated every drive-backed app (the N100, 2026-11-14, +32.7 s). `GuestBinder.GuestBootKey`; `""` when
|
||||
the guest does not run or the boot_id is unreadable. Only a controller that knows the key (built there, not
|
||||
released) reads it. Tests `TestGuestBootKey_AClockStepDoesNotMoveIt`, `TestDisks_CarriesBothBootTokens`; 6 mutations
|
||||
seen red (felhom.eu `documentation/audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt`; two mutants did not
|
||||
compile at first and were done again).
|
||||
- **A refused signed job and a watchdog give-up reach the hub (R-1383, D78, felhom.eu `09` §3 decision 299; `05`
|
||||
§16.6).** (1) `signedjobs.Runner` clears a job with `hub.Client.CompleteJobWithOutcome`: `outcome` (done · rejected
|
||||
· failed · malformed), a `reason` from the closed `signedjobs.ReasonCodes` (the two clock reasons `expired` and
|
||||
`not_yet_valid` first — a box with a wrong clock refuses every job) and the op's name, in the `DELETE`'s query. No
|
||||
signature, nonce or job parameter is sent. (2) The guest-power watchdog's give-up — one ERROR line in the journal
|
||||
until now — rides the host report as `guest_power_gave_up` (`localapi.Server.GuestPowerGaveUpStanza`, wired in
|
||||
`main.go`); the list empties when the guest runs. Tests `TestRunner_TellsTheHubHowEachJobEnded` (the real gate
|
||||
and verifier), `TestReasonCode_ClosedList`, `TestCompleteJob_OutcomeRidesTheQuery`,
|
||||
`TestGuestPower_GiveUpIsReportedUntilTheGuestRuns`, `TestMainWiresPVETrust`; 12 mutations seen red (felhom.eu
|
||||
`documentation/audits/night-build-2026-11-14/B/redproof-agent-{job-outcome,guest-power}.txt`).
|
||||
- **The report carries the box's clock at the moment of sending (R-1384, D79, felhom.eu `09` §3 decision 300).**
|
||||
`HostReport.SentAt` (`sent_at`, RFC3339 with nanoseconds) is stamped by `Client.Report` at every send, never by the
|
||||
collector. `reported_at` could not serve: it is stamped when the collect starts. The hub (its own unreleased part)
|
||||
compares it with its clock and alarms above one minute. Test `TestReport_StampsSentAtAtSending`; 3 mutations seen
|
||||
red (felhom.eu `documentation/audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt`).
|
||||
|
||||
### Release 2b (2026-11-14 night): trust in Proxmox by a copy of the box's own root CA (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a)
|
||||
|
||||
Commit only. No box ran any of this; the proof on a box (demo-hp, a forced new certificate, twice) is the next attended
|
||||
session's. `configs/` changes, so the release needs BOTH signed jobs.
|
||||
|
||||
- **The root step that keeps the copy** (`configs/felhom-os-apply`, host-layer mode `pve-ca-copy`). It copies exactly
|
||||
`/etc/pve/pve-root-ca.pem` to exactly `/etc/felhom-agent/pve-root-ca.pem` (0644 root), only if the source is ONE
|
||||
certificate with `CA:TRUE` that has not ended; written whole or not at all; „same" when nothing changed. The plan may
|
||||
name nothing (T1); no source (T2); not one CA certificate (T3). No appliance check: the copy is the agent's own trust
|
||||
file and changes nothing of a BYO owner's host. Tests `PVECACopy` (9, one with the real openssl through the real
|
||||
runner); 7 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-wrapper.txt`).
|
||||
- **The move of a box that exists, in a safe order** (`internal/pvetrust`, new; wired in `cmd/felhom-agent/main.go`
|
||||
at the agent's start, after a config bundle install, on a timer — hourly until moved, daily after — and on every
|
||||
call to Proxmox that fails on trust). (1) the root step makes the copy; (2) ONE real call to Proxmox with the copy
|
||||
ALONE, while the pinned fingerprint is still the configured trust; (3) only then ONE write of `agent.json` sets
|
||||
`proxmox.tls.ca_file` and clears `fingerprint` (the agent's own file, rewritten in place after a recovery copy
|
||||
`agent.json.pre-pve-trust` was parked, read back and checked); (4) the RUNNING client takes the new trust
|
||||
(`proxmox.Client.SetTLS`, new) — no restart. A failed copy, proof or write changes nothing: the fingerprint stays.
|
||||
**`proxmox.tls` keeps its order (a set fingerprint wins over a CA file), so the fingerprint is cleared LAST.** An
|
||||
operator's own CA file, a box with checks switched off and a config that is not a file are never moved.
|
||||
- **A pinned box whose certificate changed heals by itself:** the failed call starts the same order, the box moves to
|
||||
the CA copy and the next call works. A refreshed copy (a renewed root CA) is proven and reloaded by the running
|
||||
client.
|
||||
- **The cause is named** (`proxmox.TrustProblem`, new): the pinned certificate changed; the CA copy did not sign the
|
||||
served certificate; a certificate ended; the copy is unreadable; the copy ends within 30 days; the root step gave no
|
||||
copy; the proof failed. An ERROR line in the journal, and the new `proxmox_trust` stanza of the host report
|
||||
(`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`). **The hub does not read the stanza yet**
|
||||
(felhom.eu R-1396), and a box that is fully blind sends no host report at all — that case still shows only as
|
||||
„host stale" on the hub.
|
||||
- Tests: `internal/pvetrust` (11, against a real TLS server that changes its certificate and renews its CA), the AST
|
||||
wiring test `TestMainWiresPVETrust`; 12 + 3 mutations seen red (felhom.eu `audits/night-build-2026-11-14/A/redproof-agent.txt`;
|
||||
one mutation is not a rule and says so).
|
||||
|
||||
## v0.174.0 (2026-11-14) — the box removes its old kernels (R-1368); security profiles are refreshed after a kernel change (R-1388, R-1301); Go 1.27.4 (R-1391) — RELEASED 2026-11-14 (release-agent.sh from the clean working clone at `cc63dd5`, CI run id 2641; agent sha256 6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92, bundle sha256 7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f) and VOUCHED 2026-11-14 15:21:45 CET
|
||||
|
||||
Release 2a of the build order (felhom.eu `09` §3 decision 306). Design: felhom.eu `11-os-updates.md` §5.11.1 and §5.12.
|
||||
`configs/` changed (the wrapper), so a box needs BOTH signed jobs: `agent_update`, then `agent_config_update`.
|
||||
|
||||
**Part A — old kernels (`09` §3 decision 286).** Nothing removed old kernels; a ring-0 box got about one a week, 1 GB
|
||||
each. Now:
|
||||
|
||||
- **The wrapper** (`configs/felhom-os-apply`, layer `kernel`, new mode `kernel-cleanup`): the hand-proven rule of
|
||||
2026-11-10 as a program. Keep the running kernel, the step's `from`, `to` and `default_before`, both defaults, the
|
||||
flag's kernel, an operator's boot pin and the newest installed; remove every other installed kernel image in ONE
|
||||
`apt-get purge` by exact name. Refusals P1, P2, P3, K1, N1, S1, S2 (the design's table), C1 (the plan names
|
||||
something — it may carry only `dry_run`), R9, R12. „Not run" (exit 0): no root-owned ring-0 mark, no kernel step yet,
|
||||
the newest step did not end `good`, the box does not run its newest kernel, a step within the last 20 hours. K1 reads
|
||||
the box a second time, by itself. Repair-first. After the purge it proves the default, the pinned file, the two
|
||||
generators and the state file unchanged; a left-over folder is reported, never deleted. **Rule R4 and every update
|
||||
plan are unchanged; `apt-get autoremove` is not used.**
|
||||
- **The agent** (`internal/osupdate/kernel.go`): the clean-up is the night leg's last call, after a healthy host step,
|
||||
never in a night with a kernel step, never in a debug pass, never on a BYO host. The hub hears `cleaned`, `refused`
|
||||
or `failed` (a kernel-layer report with NO kernel version — the hub would read that as a kernel step); „nothing to
|
||||
remove" and „not run" are log lines. `--selftest=kernel-cleanup-dry-run` prints what it would do and changes nothing.
|
||||
- **No new sudo line.** The wrapper is reached by the one `--plan` line (pinned by `test_cleanup_needs_no_new_sudo_line`).
|
||||
|
||||
**Part B — security profiles (spike first: felhom.eu `audits/profile-refresh-spike-2026-11-14/`).** After a kernel
|
||||
went back and forth a box can load an old compiled AppArmor profile; on the N100 this stopped the time service.
|
||||
|
||||
- **The wrapper**, new host-layer mode `profile-refresh` (an appliance only; no package change; the plan names
|
||||
nothing, F1): once per kernel the box starts on (a root-owned record), `apparmor_parser --replace --skip-read-cache
|
||||
--write-cache -- /etc/apparmor.d`; then ONE restart for a `failed` service whose own program's cached profile the
|
||||
rewrite changed (at most 3). Measured on demo-hp: 0.5 s, no guest or container touched.
|
||||
- **The agent** (`internal/osupdate/profilerefresh.go`) asks at every start and after a config bundle install, under
|
||||
the heavy-op gate; nothing is sent to the hub.
|
||||
|
||||
**Part C — Go 1.27.4** (GO-2026-6859; hygiene: the agent's one `MaxBytesReader` place does not match the advisory).
|
||||
|
||||
**Tests, each seen red** (felhom.eu `audits/release-2a-2026-11-14/A/redproof*.txt`, `B/redproof*.txt`): 31 wrapper
|
||||
cases for the clean-up, one per acceptance scenario and one per kept-kernel kind (14 mutations, all red); 11 for the
|
||||
profile refresh (10 mutations); 5 + 4 agent tests (3 + 3 mutations). Two existing agent tests list the night's calls
|
||||
exactly and gained the clean-up call (`TestRing0_OneCallPerLayer`, `TestRing1_NoReleaseIsInventory`).
|
||||
|
||||
**Delivered 2026-11-14 to the three test boxes by signed `agent_update` + `agent_config_update` each** (Tester 1,
|
||||
demo-hp, the N100): update committed, bundle installed (written 1 of 31), capability probe 77/77, the running binary
|
||||
reads `go1.27.4`. **On boxes** (felhom.eu `audits/release-2a-2026-11-14/D/`):
|
||||
|
||||
- **Profile refresh, by the product's own path, on all three:** refused by the OLD wrapper at the agent's start
|
||||
(R11, logged), then run right after the bundle install: 0.5 s each, nothing restarted on Tester 1 and demo-hp.
|
||||
**On the N100 it repaired the time service:** the cached chrony profile was rewritten, `chrony.service` (failed
|
||||
since 2026-10-16) was restarted and is active, the clock stepped +32.7 s; offset to DooPlex −32.55 s → +0.18 s. After
|
||||
a restart of Tester 1 the agent answered „not needed" (the same kernel).
|
||||
- **Clean-up:** dry runs through the released agent on all three (demo-hp: nothing to remove; the N100: 6 of 8 will
|
||||
go tonight; Tester 1: „not run", it has no ring-0 mark). Refusals P1, P2 and P3 by dry run on demo-hp against states
|
||||
set and put back. **A real removal on Tester 1**, with the ring-0 mark set for the test on the operator's yes and
|
||||
put back: `proxmox-kernel-7.0.14-20-pve-signed` removed, 1.12 GB gained, 8.5 s, the default, the pinned file, the
|
||||
generators and the state file unchanged, no left-over; the second call removed nothing; one orderly restart came
|
||||
back on the same kernel, healthy, a clean stop for the crash guard. **This real run was started by hand** (the
|
||||
agent user, the one sudo line, the night leg's own plan), not by the night leg.
|
||||
- **Not proven:** the step by the agent at night (the N100's and demo-hp's own night, 2026-11-15); a box whose running
|
||||
kernel is not its newest and a self-revert into the kept kernel (tests only); refusal K1 on a box (it guards a wrong
|
||||
computation; tests only); a restart by the profile refresh of any program but chrony.
|
||||
- **Found:** a clock step makes the CONTROLLER read a new boot (its boot identity is `/proc/stat` btime): on the N100
|
||||
it recreated its three drive-backed apps 9 s after the step (felhom.eu R-1395). The crash guard (kernel `boot_id`)
|
||||
was not fooled.
|
||||
|
||||
**Vouched by the operator 2026-11-14 15:21:45 CET** (the hub log: `Artifact manifest set: agent=0.174.0`, the bundle
|
||||
sha = the release; the hub's System page: „vouched agent: 0.174.0"). `pinned.day0_agent` = 0.174.0 (D50).
|
||||
|
||||
## Unreleased (2026-11-11) — docs only: a repository address holds no credential (R-925, Part C of the clock/end-dates run)
|
||||
|
||||
- `scripts/release-agent.sh` header: the tag push needs `GITEA_GIT_TOKEN` from the secret store (`sec run`). The origin
|
||||
address of CC's clone on DooPlex held a login and a token until 2026-11-11; git now asks the helper
|
||||
`felhom.eu/scripts/git-credential-sec.sh`. No code change, no version change, no release.
|
||||
|
||||
## v0.173.0 (2026-11-10) — R-1368 part 2: the self-update folder keeps what it needs and no more — RELEASED and VOUCHED 2026-11-10 (release-agent.sh from a clean clone of `776025c`; agent sha256 752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea, bundle sha256 51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1)
|
||||
|
||||
- **The defect (measured 2026-11-10, space list place H12).** `Execute` downloads each new agent to
|
||||
`<state dir>/selfupdate/felhom-agent-<version>` and removed that file only when the download or its checksum failed.
|
||||
A good update left it for ever: 49 / 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1.
|
||||
- **What the folder is needed for (read first: `configs/felhom-selfupdate-guarded`).** The root wrapper reads the staged
|
||||
file once, copies it to its own root-owned folder and flips from that copy. The rollback needs `pending.json` (in the
|
||||
folder) and `/usr/local/bin/felhom-agent.prev` (outside it); the commit needs `pending.json`. No staged binary.
|
||||
- **The fix (`internal/selfupdate/prune.go`).** After the wrapper's `commit` returned success, `MaybeCommit` removes
|
||||
every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read,
|
||||
at a plain start, after a rollback (the failed version's binary stays to be read), or when the running version is
|
||||
not a version this code can name a file for. Only a regular file with exactly the name `Execute` writes is removed,
|
||||
one `os.Remove` per name; a failed removal is logged and never fails the commit. A download and a prune take turns
|
||||
(`stagingMu`).
|
||||
- **Tests `TestPrune_*` (6).** After six updates the folder holds one binary and every bystander (unknown names, a
|
||||
folder and a symlink that carry a staged name); a pending or unreadable marker refuses; only a healthy commit prunes;
|
||||
the name rule follows `versionRe`; the wrapper's `commit` and `rollback` blocks name no staged binary; and the
|
||||
wrapper's REAL `commit` and `rollback` verbs, run from a sandboxed copy, still put the previous binary back after
|
||||
four pruned updates. **Red seen three times:** no prune call; no pending refusal; no regular-file check.
|
||||
- **Not reached.** `/usr/local/bin/felhom-agent.bak-*` (place H13): root-owned, in a folder the agent cannot write, made
|
||||
by a person. R-1368 part 4 stays open.
|
||||
- **Corrected 2026-11-10 (measured on three boxes).** This entry said: „A box gets the clean-up at its first good update
|
||||
AFTER the one that brings this code". That was wrong. The commit runs in the NEW binary, so the box cleans the folder
|
||||
at the commit of the update that brings the code.
|
||||
- **Delivered 2026-11-10 to the three test boxes by a signed `agent_update` each (no bundle job: `configs/` did not
|
||||
change since v0.172.1).** The folder before → after, read on each box: demo-hp 49 entries / 722,596 KB → 1 / 16,028 KB
|
||||
(commit 14:59:19 UTC, `removed=49`); demo-felhom 48 / 706,968 KB → 1 / 16,028 KB (commit 14:51:50 UTC, `removed=48`);
|
||||
Tester 1 33 / 497,244 KB → 1 / 16,028 KB (commit 15:00:19 UTC, `removed=33`). After each: `pending.json` absent,
|
||||
`/usr/local/bin/felhom-agent.prev` = 0.172.1, the unit active with no restart.
|
||||
- **A rollback still works after the clean-up (Tester 1, 2026-11-10 15:14 UTC).** A signed `agent_update` to 0.172.1
|
||||
(the vouched version; it has no prune code) was installed with its pending marker. One `SEGV` in the 60-second dwell
|
||||
→ the unit's `OnFailure` → the wrapper's `rollback` → 0.173.0 ran again 9 seconds after the install, `pending.json`
|
||||
gone, no failed unit. The failed version's staged file stays (by design, until the next good update): the folder
|
||||
holds 2 files, 32,040 KB. Evidence: felhom.eu `documentation/audits/release-space-fixes-2026-11-10/agent/`.
|
||||
- **Vouched by the operator 2026-11-10 16:29:16 CET** (read from the hub log: `Artifact manifest set: agent=0.173.0`, the
|
||||
bundle sha = the release). `pinned.day0_agent` = 0.173.0 (D50).
|
||||
|
||||
## v0.172.1 (2026-11-07) — proxmox-grub joins the boot-chain lane: the nightly pve step stops being refused R6 — RELEASED and VOUCHED 2026-11-07 (release-agent.sh from `39db030`; agent sha256 505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8, bundle 3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3)
|
||||
|
||||
- **The defect (measured on demo-felhom, 2026-11-05, 11-06, 11-07).** Proxmox published `proxmox-grub` 2.12-9+pmx4. It
|
||||
depends on `grub-efi-amd64` of the same version. `HOST_SLOW_RE` matches `grub` only at the START of a name, so
|
||||
`proxmox-grub` landed in the ring-0 pve plan; the simulation then pulled `grub-efi-amd64`, which is not in the plan,
|
||||
and `felhom-os-apply` refused the WHOLE pve step (R6 „the plan would touch grub-efi-amd64") every night — 34 Proxmox
|
||||
userspace upgrades held with it, and an `os_update_failed` error mail each morning. The same class as the 2026-10-09
|
||||
`proxmox-secure-boot-support` fix. Read-only proof on the box: `apt-get -s install --only-upgrade proxmox-grub` →
|
||||
`Inst grub-efi-amd64 …pmx4`.
|
||||
- **The fix.** `proxmox-grub` joins `HOST_SLOW_RE`: GRUB's meta-package now waits with the rest of the boot chain (the
|
||||
slow lane, the operator's signed step), and the pve step runs without it.
|
||||
- **Test** `test_ring0_pending_pve_leaves_proxmox_grub_out`; red seen before the fix (proxmox-grub entered the plan), green
|
||||
after; the wrapper suite 149 of 149.
|
||||
- **Delivered 2026-11-07 (operator, signed jobs):** demo-felhom, demo-hp, Tester 1 — `agent_update` committed
|
||||
(23:05/23:13/23:13 CET), then the bundle: `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok`, capability probe
|
||||
77/77 on each. `pinned.day0_agent` = 0.172.1 (D50).
|
||||
- **Delivery.** `configs/` changed, so a box needs BOTH signed jobs: `agent_update` 0.172.1, then `agent_config_update`
|
||||
0.172.1 with the bundle sha. Not vouched by CC. Bundle vs 0.170.0 (what the boxes run): no path added or removed, only
|
||||
`felhom-os-apply` changed — so no step bundle is needed (R-880). Both downloads re-checked by sha.
|
||||
|
||||
## v0.172.0 (2026-11-03) — R-1302: the first grant of a tier is no loss — RELEASED and VOUCHED 10:18:10 CET (release-agent.sh from a clean clone of `ecbb0cc`; agent sha256 2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7, config bundle cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce; configs/ unchanged since 0.171.0, so the signed delivery was agent_update alone)
|
||||
|
||||
- **R-1302 — the first grant of a tier is no loss.** A fresh install arms the whole-guest tier before its storage exists, so the
|
||||
first probe found no grant, repaired it and logged ERROR „GRANT WAS MISSING … investigate the loss (R-190)" plus a DEGRADED
|
||||
capability, on every new box (VM 9375, 2026-10-26). The agent now remembers which tiers have EVER held their grant
|
||||
(`<state dir>/store-grant-seen.json`, written by a healthy probe). A first grant logs INFO „granted" and reports healthy; a
|
||||
grant that existed and vanished keeps R-190's ERROR and its one degraded cycle. A doubt keeps the ERROR: an unreadable or
|
||||
corrupt file, or no file path. A box updated to this agent records its tiers at the first healthy probe, so a loss in the
|
||||
one probe before that would read as a first grant. Tests `TestR1302_*`; red seen with the first-grant branch removed and on
|
||||
three more mutants.
|
||||
|
||||
## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
|
||||
|
||||
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
|
||||
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
|
||||
chain (demo-hp's PBS vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test, two
|
||||
controller updates). No failed probe, no 5-s timeout; slowest 303 ms (demo-hp, during the PBS vzdump). The only
|
||||
„unhealthy" was the controller stopped by hand for a proof. A busy night never came near ONE failed probe.
|
||||
- **So the alarm becomes a restart.** At `controllerUnresponsiveConfirm` = 6 unhealthy sweeps the alarm is raised, as
|
||||
before. At `controllerUnresponsiveRestart` = 10 (2 minutes later; ~6.5 minutes after the first failed probe) the agent
|
||||
restarts the controller's bootstrap unit — unless the controller's busy file is fresh (a backup, a restore, an
|
||||
update, an export) or cannot be read, or the brake refuses (3 restarts in 15 minutes → the crash-loop pause, which
|
||||
the hub mails). A swap, a parked guest, a locked guest and a vzdump keep resetting the count, as before. A refusal
|
||||
keeps the alarm and is logged once per reason. After a restart the count starts again from zero.
|
||||
- **The stanza** carries `unresponsive_restart_after_s` (120: the hub's alarm text says when the restart comes) and
|
||||
`unresponsive_restart_at` (equals `last_restart_at` for that act; the hub mails `controller_unresponsive_restart`).
|
||||
- **Refactor, no behaviour change:** the not-running restart's brake and its record are now `supervisorBrakeAllows`
|
||||
and `recordSupervisorRestart`, shared by both restarts. The busy-file stale log line no longer says „operator reboot".
|
||||
- Tests `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`, `…Guards` (now also: never restarted),
|
||||
`…RestartRefusals` (busy fresh, busy unreadable, the brake, a crash-loop pause), `…StaleBusyFileDoesNotShield` —
|
||||
red first (build failure, then 3 behaviour failures); 7 mutants, each killed (one survivor — the count reset after a
|
||||
restart — found and pinned).
|
||||
|
||||
## v0.170.0 (2026-10-29) — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list — RELEASED (release-agent.sh from a clean clone of `ca15a31`; agent sha256 da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404, config bundle 03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543; signed agent_config_update + agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-29 18:57:48 CET; `pinned.day0_agent` = 0.170.0)
|
||||
|
||||
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
|
||||
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
|
||||
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
|
||||
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
|
||||
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
|
||||
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
|
||||
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
|
||||
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
|
||||
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
|
||||
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
|
||||
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
|
||||
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
|
||||
a racing start would fail the reboot's own start half.
|
||||
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
|
||||
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
|
||||
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
|
||||
restarted your box at <time>".
|
||||
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
|
||||
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
|
||||
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
|
||||
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
|
||||
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
|
||||
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
|
||||
|
||||
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
|
||||
|
||||
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
|
||||
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
|
||||
answers on the next report, `operator_action_results: [{id, outcome, message}]` (`hub.HostReport`), which it requests
|
||||
at once through the cap-1 report trigger. CLOSED list: `restart_controller <vmid>` only — anything else refused, nothing
|
||||
called. The act is the supervisor's (`systemctl restart felhom-controller-bootstrap.service` in the guest, existing
|
||||
grants — no sudoers change, no config bundle) with its guards: provisioned guest, guest running, the guest list and the
|
||||
vzdump state readable, not parked, no swap, not locked, no vzdump. The thrash brake counts operator restarts with the
|
||||
supervisor's: 3 in 15 minutes → the press is refused, and presses stay refused for 30 minutes. The result says how long
|
||||
the controller took to answer (Docker's health check: healthy), or `failed` when it did not within 3 minutes. One
|
||||
`audit: operator action` line per decision. Works with the controller stopped. Tests `TestOpActions_*` (closed list,
|
||||
unknown/bad arg refused, restart + answer time + report trigger + once per id + re-send until dropped, stopped
|
||||
controller, every guard, the brake across supervisor and operator restarts, no healthy answer = failed) and
|
||||
`TestMainWiresTheAgentOperatorActions` (AST) — each seen failing by a mutation.
|
||||
- **R-1344 — a controller that RUNS but does not answer: ALARM ONLY.** Each sweep also reads Docker's health status for a
|
||||
running controller (the swap's `docker inspect -f` grant); 6 consecutive „unhealthy" sweeps (3 minutes, on top of
|
||||
Docker's own ~90 s) start an episode: one ERROR line, the stanza's `unresponsive` / `unresponsive_since` (the hub raises
|
||||
`controller_unresponsive`, operator-only). NOT restarted for it: the brief allows the restart only after a measurement
|
||||
shows a busy night never reads like this, and that measurement runs through tonight (felhom.eu R-1344). Not counted
|
||||
during a swap, a parked or locked guest, or a vzdump; healthy / starting / no health check / an unreadable status reset
|
||||
the count. Tests `TestSupervisor_Unresponsive*` (red by mutation: the vzdump guard, the threshold).
|
||||
|
||||
## v0.168.0 (2026-10-29) — R-528: the host's per-container OOM counter (`GET /guest/oom-kills`); R-1254: a lost or late escrow answer is asked again before it is worded — RELEASED (release-agent.sh from a clean clone of `91bd96c`; agent sha256 1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca, bundle 3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8; no configs/ change since v0.167.0 → binary-only signed updates to demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 10:21:56 CET**; Day-0 pin set (D50)
|
||||
|
||||
### R-528
|
||||
|
||||
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
|
||||
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
|
||||
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
|
||||
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
|
||||
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
|
||||
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
|
||||
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
|
||||
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
|
||||
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
|
||||
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
|
||||
|
||||
### R-1254 (the ceremony half, review round 2 HUNT-3 H2)
|
||||
|
||||
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
|
||||
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
|
||||
hub's current package opens only with the NEW code.
|
||||
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
|
||||
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
|
||||
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
|
||||
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
|
||||
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
|
||||
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
|
||||
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
|
||||
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
|
||||
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
|
||||
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
|
||||
|
||||
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
|
||||
|
||||
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
|
||||
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
|
||||
newest. Full `go build/vet/test` green. The controller and hub move the same night.
|
||||
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
|
||||
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
|
||||
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
|
||||
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
|
||||
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
|
||||
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
|
||||
|
||||
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
|
||||
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
|
||||
second later — so 0.166.0's pass never ran on the bundle that delivered it.
|
||||
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
|
||||
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
|
||||
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
|
||||
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
|
||||
agent-only — holding the bundle job until the commit would need new hub queue logic.
|
||||
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
|
||||
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
||||
removed → commit only, no pass — the demo-hp symptom).
|
||||
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
|
||||
0.165.0 box passes even without this fix.
|
||||
|
||||
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
|
||||
|
||||
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
|
||||
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
|
||||
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
|
||||
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
|
||||
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
|
||||
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
|
||||
|
||||
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
|
||||
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
|
||||
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
|
||||
„BUNDLE POLICY DONE install_mode=appliance removed=0".
|
||||
|
||||
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
|
||||
|
||||
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
|
||||
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
|
||||
drop-in until its next bundle. A change inside the wrapper alone cannot fix the run that delivers it.
|
||||
- **The fix — the new AGENT asks the new wrapper:** `internal/osupdate/bundle.go` runs a policy pass right after every
|
||||
bundle install (`mode: "bundle-policy"`, host layer, no envelope); the wrapper on disk is the bundle's own by then.
|
||||
`configs/felhom-os-apply` `Bundle.policy_pass`: on a root-owned BYO record it removes the appliance-only files WE
|
||||
wrote (our header), keeps the previous copy beside the bundle's, `sshd -t` then reloads ssh — a failing `sshd -t` puts
|
||||
the file back. It writes nothing, so it needs no signature; an appliance and the owner's own file are left alone. Never
|
||||
fatal for the agent: an older wrapper refuses the unknown mode and the next bundle does it. Report key `bundle_policy`.
|
||||
- Tests: `TestR1285_AnInstalledBundleIsFollowedByThePolicyPass` (a refused bundle runs no pass);
|
||||
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
|
||||
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
|
||||
|
||||
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
|
||||
|
||||
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
|
||||
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
|
||||
usable that long, and the nonce store no longer keeps its record that long.
|
||||
- `configs/felhom-os-apply`: the root wrapper holds the same 24 h (`MAX_WINDOW_S`, refusal R3 „the signed job's window
|
||||
is longer than 24 h").
|
||||
- `cmd/felhom-opsign`: `-ttl` above 24 h (or ≤ 0) is refused before anything is signed.
|
||||
- Tests: `TestVerify_RejectsPerStage/window_longer_than_24_h` and `…/window_of_exactly_24_h_is_accepted`;
|
||||
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
|
||||
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
|
||||
|
||||
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
|
||||
|
||||
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
|
||||
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
|
||||
(`hub_min_agent` 0.158.1, `day0_agent` 0.165.0 — copies of hub state, because CI holds no hub secret; the session that
|
||||
records a vouch or a new MinAgent updates them in the same commit). `check-published-versions.py` asserts the window
|
||||
plus every pin; a pin with no tag FAILS. CI was red since the operator's hand prune of 0.159.0–0.160.0 (job 2334).
|
||||
- **Decoys** (`test_gate_decoys.py`, via `FELHOM_RETENTION_POLICY`, test only): a pinned version older than the window is
|
||||
asserted; its 404 convicts; a pin with no tag convicts. Seen failing on the old gate (two LIVE HOLES): felhom.eu
|
||||
`documentation/audits/night-2026-10-25/redproof-d50.txt`.
|
||||
|
||||
## v0.165.0 (2026-10-25) — D43 = A: BYO hosts keep their own SSH settings (R-1257); the line test for a box that seeds all day (R-1250) — released in the attended release run; **VOUCHED by the operator 2026-10-25 18:38:49 CET** (binary `e9d04a47…`, bundle `8dea1f34…`)
|
||||
|
||||
**Found at the release (felhom.eu R-1285, P4):** on a BYO host upgraded from 0.164.0 the drop-in is removed only on the bundle run AFTER the one that installs this wrapper — the first run is planned by the old wrapper. Proven on a BYO test host (VM 9371 on demo-hp): the second `agent_config_update` logged `BUNDLE REMOVED … (ours, on a BYO host — D43)`.
|
||||
|
||||
### BYO hosts keep their own SSH settings (R-1257, D43 = A) — 2026-10-24 night
|
||||
|
||||
- **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an
|
||||
appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy
|
||||
is now `appliance`: written only when the ROOT-OWNED install record (`/var/lib/felhom-install/state.json`) says
|
||||
`appliance`. On a `byo` record nothing is written, and a copy WE wrote (this exact path, beginning with our header
|
||||
line) is REMOVED, its previous copy kept for the undo, then `sshd -t` and a reload. A box whose record is absent or
|
||||
agent-writable proves nothing and is left exactly as it is. The bundle report lists `removed`; the record says
|
||||
`removed` / `skipped`. The installer records the mode before it installs the bundle, so a new appliance still gets it.
|
||||
- **X3: the „in force" check asks sshd about each of the host's own LAN addresses** (`ip -4 -o addr show scope
|
||||
global`), not only 192.0.2.1 — a `Match Address <LAN>` block that re-allows root passwords no longer reads as in force.
|
||||
- **X4: a failing `sshd -t` puts back ONLY the SSH step** (`undo_sshd_step`: the previous copy, or none; sshd is not
|
||||
reloaded, so it never ran the refused state) and the rest of the bundle stays; it used to refuse and undo the whole
|
||||
bundle. The report carries `sshd_step_undone`; the record names what is on disk.
|
||||
- Tests (`configs/test_felhom_config_bundle.py`, class `ByoKeepsItsSSH`, 7 tests; each rule seen failing with it taken
|
||||
out): a BYO host gets no drop-in; our copy comes off a BYO host; the owner's own file stays; no record → left as it
|
||||
is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step.
|
||||
The fake box now carries an appliance record by default (every earlier test unchanged).
|
||||
|
||||
### The line test for a box that seeds all day (R-1250) — 2026-10-24
|
||||
|
||||
Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0)
|
||||
starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested
|
||||
line and qBittorrent never got its 50 % limit. Now:
|
||||
|
||||
- **`linetest.Tester.WaitingForQuiet`** — due, inside the window, tries left, tunnel up, and only the quiet missing.
|
||||
`GET /host/line-test` carries it as `waiting_for_quiet`; a controller with a seeding torrent app acts only on it.
|
||||
It needs the full watched window (11 minute samples): a freshly started agent is not known to be busy, so it never
|
||||
invites a hold (found in the live proof, 2026-10-24: the first build said `true` before one minute was watched;
|
||||
`TestFreshTesterIsNotWaiting`).
|
||||
- **`linetest.Tester.RunNow`** and **`POST /host/line-test/run`** (synchronous, about 55 s) — a test on request while the
|
||||
controller holds its app's upload low: every weekly rule holds (due, window, `MaxTriesPerDay`, tunnel, the one heavy-op
|
||||
gate) except the 10 quiet minutes, which become a 5 s settle and a 10 s quiet check (below 2 Mbit/s, so a request
|
||||
never measures the family's own upload); the three runs are 5 s apart instead of 30. A refusal answers its code
|
||||
(`not-due`, `outside-window`, `tries-used`, `no-tunnel`, `heavy-op`, `not-quiet`, `server-busy`); a failed request
|
||||
spends one of the day's tries. `Tick` and `RunNow` now share one `ready()` check.
|
||||
- Tests (each seen failing): `TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt` (the sleeps, the 60 s bound),
|
||||
`TestRunNowRefusesAnUplinkThatIsStillBusy`, `TestRunNowKeepsTheWeeklyRules`, `TestRunNowFailuresSpendTheDaysTries`,
|
||||
`TestQuietBoxIsNotWaiting`, `TestLineTestRunRoute`; a compile-time assertion that the Tester serves the request half.
|
||||
|
||||
**No sudoers or config-bundle change. MinController: none** — an older controller never calls the route.
|
||||
|
||||
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
|
||||
|
||||
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
|
||||
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
|
||||
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
|
||||
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
|
||||
|
||||
## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST
|
||||
|
||||
Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256
|
||||
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341` (configs unchanged — no bundle job sent). Built from
|
||||
`f7e1dcb` (CI 2277 green) in a fresh clone by `scripts/release-agent.sh`. Delivered by one signed `agent_update` per box
|
||||
(07:25Z): Tester 1, demo-hp and demo-felhom — COMPLETED, self-check 76/76, committed after the 1-minute dwell, the line
|
||||
test started (felhom.eu `audits/line-test-2026-10-24/agent-0164-boxes.txt`).
|
||||
|
||||
Operator ruling 2026-10-24 07:26: once a week the box's HOST measures its upload line with a 10-second iperf3 upload to an
|
||||
iperf3 server on ep0 that answers **only inside the WireGuard tunnel** (felhom.eu `scripts/ep0-iperf3/`; installed the same
|
||||
day). The agent speaks the iperf3 protocol itself (operator's pick, 2026-10-24: nothing is installed on any host).
|
||||
**No sudoers or config-bundle change.** **MinController: none** — the controller (≥ 0.342.0) reads the new route and
|
||||
treats a 404 as „not measured".
|
||||
|
||||
- **`internal/linetest`** — `Client` (the iperf3 3.x client protocol: cookie, params, N data streams, TEST_END, results
|
||||
exchange; the value is the SERVER's received bytes) and `Tester` (the schedule): due weekly from Tuesday 10:00 local,
|
||||
starts 10:00–15:59 only (never in the night chain), from a per-host slot spread over 10:00–13:59; only after 10 quiet
|
||||
minutes (every one-minute transmit rate of the host's physical uplink below 2 Mbit/s); under the one-heavy-op gate
|
||||
(`line-test`); 3 runs of 10 s with 4 streams, 30 s apart, the highest counts; each run bounded by 25 s; „server busy" →
|
||||
tried again at the next tick the same day, at most 6 tries a day; no operational tunnel → no test, no value. The last
|
||||
result is persisted (`<oob state dir>/line-test.json`).
|
||||
- **Report:** a `line_test` stanza (`schedule`, `last_ok_at`, `mbps`, `runs_mbps`, `server`, `last_attempt_at`,
|
||||
`last_error`). **Local API:** `GET /host/line-test` → `{present, mbps, measured_at, method:"iperf3-tunnel"}`.
|
||||
- `wgtunnel.Manager.LineTestTarget()` — the desired block's `pbs_tunnel_ip`, only while the tunnel is operational
|
||||
(cached, no exec). Config `line_test.disable` (default on).
|
||||
- **Proven live** (felhom.eu `audits/line-test-2026-10-24/`): a probe built from this package on demo-hp's host read
|
||||
**90.7 Mbit/s** in 10.4 s (the push floor said „at least 10"); a second test 2 s later → „server busy" in 0.1 s while the
|
||||
first went on undisturbed (90.6).
|
||||
- Tests, each red-proved by a mutation (15): `TestClientSpeaksTheProtocolAndCountsTheServersBytes` (against a fake
|
||||
iperf3 server), `TestClientBusyServerIsErrBusy`, `TestClientHonoursTheDeadlineAgainstAStalledServer`,
|
||||
`TestEligibleWindowNeverInTheNightChain`, `TestNeverTestsInTheNightOrBeforeTuesday`, `TestBusyServerIsRetriedLaterTheSameDay`,
|
||||
`TestTriesPerDayAreBounded`, `TestNoTunnelNoTest`, `TestBusyUplinkOrHeavyOpWaits`, `TestTickMeasuresTheHighestOfThreeRuns`,
|
||||
`TestStateSurvivesARestartAndReportsIt`, `TestUplinkCountsTheBridgesPhysicalPortsOnly`, `TestLineTestRoute`,
|
||||
`TestLineTestTargetOnlyWhileOperational`, `TestMainWiresTheLineTest`.
|
||||
|
||||
## v0.163.0 — the tunnel heals itself after a clock that ran ahead (D30, R-1204); root SSH from the LAN by key only (D4, R-982) (2026-10-23) — RELEASED on the operator's yes (day brief); the vouch is the operator's
|
||||
|
||||
Binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`; config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`. Built from `1109092` (CI 2246 green). Both sections below are in it.
|
||||
|
||||
### The tunnel re-key (D30) and the D4 bundle drop-in
|
||||
|
||||
- **D30 (operator 2026-10-22: the agent makes a new key and registers it on the existing peer path).** When the latest
|
||||
handshake stays stamped in the future for 15 minutes (the R-1204 sign: ep0 keeps the future timestamp and refuses
|
||||
@@ -32,9 +558,9 @@
|
||||
- **Known gap, unchanged (filed):** after a reboot the kernel forgets the future stamp (latest handshake 0), so neither
|
||||
this re-key nor the hub's alarm sees a box whose clock was corrected and which then restarted (R-1223).
|
||||
|
||||
## Unreleased (2026-10-22 night) — the tunnel's future-stamped handshake is reported (R-1204) — COMMITTED, NOT RELEASED (no agent release tonight); root SSH by key only (R-982, D4) is a READY PATCH, not on main
|
||||
### The future-stamped handshake is reported (R-1204) — first written on the 2026-10-22 night
|
||||
|
||||
- **R-982 / D4 — a ready patch, NOT on main.** Built and proven the same night (felhom.eu
|
||||
- **R-982 / D4 — a ready patch on that night; on `main` since `1109092` and released in v0.163.0 (see above).** Built and proven the same night (felhom.eu
|
||||
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
|
||||
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
|
||||
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
|
||||
|
||||
+63
@@ -1,5 +1,68 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-11-14 (day, attended) — v0.174.0 RELEASED and VOUCHED (15:21:45 CET); on the three test boxes (agent + bundle, 77/77).** sha `6efbb978…`, bundle `7d87d058…`. Two new wrapper modes: `kernel-cleanup` (layer kernel; old kernel images by the hand-proven rule; ring-0 mark only; the plan names nothing; the night leg's last call, never with a kernel step) and `profile-refresh` (layer host; AppArmor profiles from a fresh compile once per kernel; one restart for a failed service whose cached profile was stale). Go 1.27.4. The N100's time service was repaired by the product path (+32.7 s step). A real kernel removal ran on Tester 1 by hand with a temporary ring-0 mark (operator's yes). **Next:** read the N100's and demo-hp's night (the clean-up by the agent at night is not proven yet); a ruling for ring-1 boxes; release 2b (trust in Proxmox by the root CA copy, D80). Report: felhom.eu `REPORT-release-2a-2026-11-14.md`.
|
||||
|
||||
> **2026-11-10 (afternoon, attended) — v0.173.0 RELEASED and VOUCHED (16:29:16 CET); on the three test boxes.** sha `752b2727…`, bundle `51ebc1f4…` (`configs/` unchanged since 0.172.1, so no bundle job). Measured: a box cleans its self-update folder at the commit of the update that BRINGS the code (49 → 1, 48 → 1, 33 → 1 entries; about 16 MB stays). The line below („at the first good update AFTER that release") was wrong. A rollback after the clean-up was proven on Tester 1 with a signed update to 0.172.1 and one crash in the dwell. `pinned.day0_agent` = 0.173.0. Open: the `.bak-*` files (R-1368 part 4).
|
||||
|
||||
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
|
||||
|
||||
> **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
|
||||
|
||||
> **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
|
||||
>
|
||||
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
|
||||
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
|
||||
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
|
||||
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
|
||||
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
|
||||
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
|
||||
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
|
||||
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
|
||||
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
|
||||
|
||||
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
|
||||
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
|
||||
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
|
||||
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
|
||||
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
|
||||
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
|
||||
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
|
||||
|
||||
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
|
||||
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
|
||||
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
|
||||
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
|
||||
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
|
||||
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
|
||||
|
||||
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
|
||||
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
|
||||
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
|
||||
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
|
||||
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
|
||||
|
||||
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
|
||||
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
|
||||
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
|
||||
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
|
||||
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
|
||||
|
||||
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
|
||||
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
|
||||
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
|
||||
> on demo-hp (felhom.eu R-1250, read, not forced).
|
||||
|
||||
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
|
||||
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
|
||||
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
|
||||
> put back. Release + the operator's vouch: Monday's list (felhom.eu STATUS).
|
||||
|
||||
> **2026-10-24 (attended release run) — v0.164.0: the weekly upload line test (R-1247; operator ruling 07:26).**
|
||||
> `internal/linetest`: the agent's own iperf3 client (operator's pick — nothing installed on hosts) to ep0's tunnel-only
|
||||
> server `10.77.0.1:5201`, Tuesday 10:00-15:59, after 10 quiet minutes, under the heavy-op gate, best of 3; `line_test`
|
||||
> stanza + `GET /host/line-test` (the controller's qBittorrent limit, D41 = 50 %). demo-hp measured 90.7 Mbit/s. Release
|
||||
> needs the operator's vouch.
|
||||
|
||||
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
|
||||
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
|
||||
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
|
||||
|
||||
@@ -1,23 +1,31 @@
|
||||
# REPORT — night 2026-10-22 → 23: D30 on `main`, the D4 patches v2, nothing released
|
||||
# REPORT — v0.174.0 released, vouched and delivered to the three test boxes (2026-11-14, attended)
|
||||
|
||||
**What:** the night brief's agent parts. No release, no vouch. Design read: `03-host-agent.md` (the tunnel, the key, the
|
||||
bundle), `06-offsite-connectivity.md` (the full-list push to ep0).
|
||||
**Plain words first.** A box now removes its old kernels by itself, and it reloads its security profiles after it
|
||||
starts on another kernel. Version 0.174.0 is released and runs on Tester 1, demo-hp and the N100. The operator vouched
|
||||
it at 15:21 CET. The N100's time service runs again; the product repaired it, with no hand step.
|
||||
|
||||
- **D30** (`ae69ad4`, hardened `39fb9cb`, `b9b831f`): a handshake stamped in the future for 15 min → a new key pair,
|
||||
registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the key on ep0 — nothing new
|
||||
on ep0). The key file is replaced by an fsynced rename over it (never missing or empty); the first old key stays
|
||||
`private.key.prev`, later ones dated (3 kept); the once-an-hour bound survives a restart; a wait stamp dated after now
|
||||
is dropped. Tests `TestD30_*` (7), each red-proved.
|
||||
- **Live** (felhom.eu `audits/night-2026-10-23/P2-D30/`, test box Tester4-16fdef, fixture `ae69ad4`): with ep0 free to
|
||||
start a handshake, a 2-hour clock jump healed by itself and no re-key ran (correct); with ep0's starts dropped on the
|
||||
box, the tunnel stayed dead, the agent re-keyed at 15 min, the hub took the key (`existed=false generation=3
|
||||
sync=ok`), the box switched at its next desired-state fetch and shook hands with the new key.
|
||||
- **D4 patches v2** (felhom.eu `audits/night-2026-10-23/P3-review/`), after a fresh review (A1–A7): `try-reload-or-restart`,
|
||||
an `sshd -T` read-back reported as `sshd_dropin_not_in_force`, a reload in undo, the right citation; installer: a warning
|
||||
when the reload fails. RootKeyOnly 6/6 (two new, red-proved); installer tests 40/40 with the patch. Both apply to
|
||||
tonight's mains. Live by hand on the test box: root password SSH refused, key works, web password 200.
|
||||
- **Gap filed:** a reboot after the clock was corrected forgets the future stamp (felhom.eu R-1223).
|
||||
The full report of the run is in `felhom.eu`: `REPORT-release-2a-2026-11-14.md`.
|
||||
|
||||
CI by commit: `ae69ad4` 2229, `39fb9cb` 2235 green; `b9b831f` and this docs commit: see felhom.eu
|
||||
`REPORT-night-2026-10-23.md`. Fixture names on the test box: `0.163.0-night.1` (= `ae69ad4`), then `0.162.99`
|
||||
(= `b9b831f`; a plain number, because the controller cannot compare a pre-release string — the off-site unlock refused).
|
||||
## What a reader must know
|
||||
|
||||
1. **The clean-up runs on a box with the root-owned ring-0 mark only.** Tester 1 has no mark, so the program answers
|
||||
„not run" there. For the proof the operator allowed the mark on Tester 1 for some minutes; it is back as before.
|
||||
2. **The real removal on Tester 1 was started by hand**, with the agent's own command line. The night leg cannot be
|
||||
started by day: it follows only the night's own backup (R-899). So „the step by the agent at night" is **not proven
|
||||
yet**; the N100's and demo-hp's night of 2026-11-15 is the proof.
|
||||
3. **The profile refresh was spiked first** on demo-hp (felhom.eu `audits/profile-refresh-spike-2026-11-14/`). Three
|
||||
things it found are built in: a stale copy is told by the cached file changing, not by the kernel's hash; one
|
||||
profile never compiles the same twice; a refresh hides other kernel audit lines for a few seconds.
|
||||
4. **A clock step makes the controller read a new boot** (felhom.eu R-1395, a controller row): on the N100 it
|
||||
recreated three apps once.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Version | 0.174.0, sha256 `6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92`, bundle `7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f`, Go 1.27.4 |
|
||||
| Code commit, CI | `cc63dd5`, run id 2641 success |
|
||||
| Tests | wrapper 191 (42 new), agent `internal/osupdate` green; 30 mutations, each seen red (felhom.eu `audits/release-2a-2026-11-14/A`, `B`) |
|
||||
| Tester 1 | 3 kernels → 2, 1.12 GB gained, 8.5 s; second run: nothing; one restart, same kernel, healthy, clean stop |
|
||||
| demo-hp | nothing to remove; refusals P1, P2, P3 by dry run |
|
||||
| The N100 | time service repaired (offset −32.55 s → +0.18 s); tonight it removes 6 of 8 kernels |
|
||||
| Vouch | the operator, 2026-11-14 15:21:45 CET; `pinned.day0_agent` = 0.174.0 |
|
||||
| Not proven | the step at night by the agent; a fell-back box and a self-revert into the kept kernel on a box; K1 on a box |
|
||||
@@ -35,7 +35,7 @@
|
||||
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
|
||||
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
|
||||
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
|
||||
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
|
||||
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()`. **The staging folder cleans itself** (`prune.go`, R-1368): after a healthy `commit` only the running version's staged binary stays; a new file kept in `<state dir>/selfupdate/` must NOT be named `felhom-agent-<version>` or the prune removes it |
|
||||
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
|
||||
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
|
||||
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
|
||||
@@ -62,6 +62,7 @@
|
||||
| `GuestBinder.EnsureSharedParent` | internal/localapi/intermediary.go | `EnsureSharedParent(ctx) error` | shared-parent bind + boot unit | make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness |
|
||||
| `StablePathForRaw` / `DriveNameFromRaw` | internal/localapi/intermediary.go | `StablePathForRaw("/mnt/<n>") string` | raw host mount → stable guest path | Single source of truth both repos derive from |
|
||||
| `GuestBinder.GuestSeesMount` / `GuestBootID` | internal/localapi/intermediary.go | `GuestSeesMount(ctx, vmid, path) bool` | guest-visible (usable) signal; reboot detection | Host bind present ≠ guest sees it (non-recursive parent bind) |
|
||||
| `GuestBinder.GuestBootKey` (R-1395) | internal/localapi/intermediary.go | `GuestBootKey(ctx, vmid) string` | "is this a new boot of the guest?" — `<kernel boot_id>-<guest init start>` | **`GuestBootID` carries `/proc/stat` `btime`, which the kernel computes from the wall clock: a clock STEP moves it.** Use the key for any „same boot?” question; `""` = unreadable. Seams `hostBtimeFn` / `hostBootIDFn` / `procStarttimeFn` |
|
||||
| `SudoHostOps.EnsureNetworkMount` / `RemoveNetworkMount` / `ListNetworkMounts` | internal/storage/netmount.go | `EnsureNetworkMount(ctx, spec) error` | NAS automount pair | rm glob confined to `mnt-felhom*` units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent) |
|
||||
| `NetworkMountedAt` / `NetworkEndpointReachable` | internal/storage/netmount.go | `NetworkMountedAt(where) bool` | verify mount-truth + the 2 s add pre-probe | /proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8) |
|
||||
| `SudoHostOps.ReassertNetworkAutomounts` + `Server.ReassertNetworkMounts` + `guesthook.PostStartNetworkReassert` | internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go | `ReassertNetworkAutomounts(ctx) []NetReassertResult` | NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests | NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo |
|
||||
@@ -88,7 +89,8 @@
|
||||
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
|
||||
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
|
||||
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
|
||||
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
|
||||
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause (`supervisorBrakeAllows` — every supervisor restart passes it; `recordSupervisorRestart` puts one on the record + R-539's slow counter); R-1344 (2026-10-30): a RUNNING controller whose Docker health reads „unhealthy" is alarmed at 6 sweeps and restarted at 10 (`restartUnresponsive`: the controller's busy file, then the brake); record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
|
||||
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller`, `reboot_guest` since D65 — `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
|
||||
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
|
||||
|
||||
### Proxmox client / hub / PBS / provisioning
|
||||
@@ -97,6 +99,10 @@
|
||||
|---|---|---|---|---|
|
||||
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
|
||||
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
|
||||
| `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. |
|
||||
| `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. |
|
||||
| `pvetrust.Manager` (`Ensure`, `OnCallError`, `Status`) + `proxmox.Client.SetTLS` / `proxmox.TrustProblem` + `Leg.PVECACopy` + the wrapper's `pve_ca_copy` | internal/pvetrust/pvetrust.go; internal/proxmox/{client,trust}.go; internal/osupdate/pvecacopy.go; configs/felhom-os-apply (mode `pve-ca-copy`) | `Ensure(ctx, why) (changed bool)`; `TrustProblem(err) string` | R-1380 (felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA; the move from a pinned certificate; the cause of a trust failure | THE ORDER: the copy, a real call with the copy ALONE, then ONE config write that clears the fingerprint LAST (a set fingerprint wins over `ca_file` in `TLSConfig.build`). Never clear the fingerprint anywhere else. The wrapper's plan names nothing (T1). `agent.json` is written by the agent in place, with a parked recovery copy — the same way as the backup-target move. |
|
||||
| `config.ReplaceInPlace` / `config.RecoverCutWrite` (+ `config.FileMu`) | internal/config/cutwrite.go; internal/config/filelock.go | `ReplaceInPlace(path, prev, next) error`; `RecoverCutWrite(path) (note string, err error)` | R-1380 (felhom.eu `03` §12a item 7): the only way to write `agent.json` — the folder is root-owned, so no rename; the old bytes go to a journal in the state folder first, and the daemon repairs a cut write at start, before it reads the config | A new writer takes `FileMu`, calls `ReplaceInPlace`, opens no file by hand, and is added to the two source pins in `cmd/felhom-agent` (`…TakeTheSharedLock`, `…UseTheJournalledWrite`). A package whose tests write the config sets `config.CutWriteDir` in `TestMain` |
|
||||
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||
@@ -171,6 +177,7 @@
|
||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
||||
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
|
||||
| `linetest.Tester` + `linetest.Client` (R-1247) | internal/linetest/linetest.go · internal/linetest/iperf.go | `New(runner, target, gate, uplink, hostID, statePath, logger)` / `Tick(ctx)` / `Value()` / `LineTestStatus(ctx)`; `Client{Streams, Time}.Run(ctx, addr)` | the weekly upload line test: a 10-second iperf3 upload (the agent speaks the iperf3 protocol — nothing installed on the host) to ep0's tunnel-only server (`wgtunnel.Manager.LineTestTarget`), Tuesday 10:00-15:59 local from a per-host slot, after 10 quiet minutes of the PHYSICAL uplink (`SysUplinkTx`: the bridge's ports with a `device` link — the bridge counter misses bridged guest traffic), under `backup.InFlight`; best of 3; reported as `line_test`, served as GET /host/line-test | **The value is the SERVER's received bytes**, never the client's written ones (the send buffer). `Run` refuses a context without a deadline. iperf3 numbers streams 1, 3, 4, 5 — the server matches results by those ids. „server busy" (ACCESS_DENIED) stops the measurement and is retried at the next tick, at most `MaxTriesPerDay` |
|
||||
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
|
||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
||||
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// A box whose agent.json was cut in the middle of a write must start (R-1380): main repairs the file from the
|
||||
// journal BEFORE it reads it. The staging is the state a power cut leaves: a durable journal of the old bytes and a
|
||||
// half-written config.
|
||||
func TestMainRecoversACutConfigWrite(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
prev := config.CutWriteDir
|
||||
config.CutWriteDir = filepath.Join(dir, "state")
|
||||
t.Cleanup(func() { config.CutWriteDir = prev })
|
||||
if err := os.MkdirAll(config.CutWriteDir, 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfgPath := filepath.Join(dir, "agent.json")
|
||||
old := `{"proxmox":{"node":"pve","tls":{"fingerprint":"AA:BB"}}}`
|
||||
if err := os.WriteFile(filepath.Join(config.CutWriteDir, "agent.json.write-in-flight"), []byte(old), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(cfgPath, []byte(`{"proxmox":{"node":"pv`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := config.Load(cfgPath); err == nil {
|
||||
t.Fatal("the staging is wrong: the cut config loads")
|
||||
}
|
||||
note, err := recoverCutConfigWrite(cfgPath)
|
||||
if err != nil || note == "" {
|
||||
t.Fatalf("note=%q err=%v", note, err)
|
||||
}
|
||||
cfg, err := config.Load(cfgPath)
|
||||
if err != nil {
|
||||
t.Fatalf("the agent does not start after the repair: %v", err)
|
||||
}
|
||||
if cfg.Proxmox.TLS.Fingerprint != "AA:BB" {
|
||||
t.Fatalf("the repaired config lost the fingerprint: %+v", cfg.Proxmox.TLS)
|
||||
}
|
||||
}
|
||||
|
||||
// main must call the repair before config.Load, or the repair never runs on the box that needs it (a seam that is
|
||||
// built and not wired). Read from the source: the call's position in main's body.
|
||||
func TestMainCallsTheConfigRepairBeforeItLoadsTheConfig(t *testing.T) {
|
||||
src, err := os.ReadFile("main.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f, err := parser.ParseFile(token.NewFileSet(), "main.go", src, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var repair, load token.Pos
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != "main" || fn.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fn.Body, func(n ast.Node) bool {
|
||||
c, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
if id, ok := c.Fun.(*ast.Ident); ok && id.Name == "recoverCutConfigWrite" && repair == 0 {
|
||||
repair = c.Pos()
|
||||
}
|
||||
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Load" {
|
||||
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" && load == 0 {
|
||||
load = c.Pos()
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if repair == 0 || load == 0 {
|
||||
t.Fatalf("main: repair call found=%v, config.Load found=%v", repair != 0, load != 0)
|
||||
}
|
||||
if repair > load {
|
||||
t.Fatal("main reads the config BEFORE it repairs a cut write — a cut config stops the agent at start")
|
||||
}
|
||||
}
|
||||
|
||||
// Every writer of agent.json goes through config.ReplaceInPlace and none truncates the file itself: a writer that
|
||||
// opens the config with O_TRUNC by hand is the unprotected write again.
|
||||
func TestAgentJSONWritersUseTheJournalledWrite(t *testing.T) {
|
||||
for _, w := range []struct{ file, fn string }{
|
||||
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
|
||||
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
|
||||
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
|
||||
} {
|
||||
src, err := os.ReadFile(w.file)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, w.file, src, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
found := false
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
body := string(src[fset.Position(fn.Body.Pos()).Offset:fset.Position(fn.Body.End()).Offset])
|
||||
calls := 0
|
||||
ast.Inspect(fn.Body, func(n ast.Node) bool {
|
||||
if c, ok := n.(*ast.CallExpr); ok {
|
||||
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "ReplaceInPlace" {
|
||||
if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "config" {
|
||||
calls++
|
||||
}
|
||||
}
|
||||
if sel, ok := c.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "OpenFile" {
|
||||
t.Errorf("%s %s opens a file by hand — the config is written only by config.ReplaceInPlace", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if calls != 1 {
|
||||
t.Errorf("%s %s: %d call(s) to config.ReplaceInPlace, want 1", w.file, w.fn, calls)
|
||||
}
|
||||
if strings.Contains(body, "O_TRUNC") {
|
||||
t.Errorf("%s %s truncates a file itself", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("%s: function %s not found — the pin reads nothing", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Three functions in this process read agent.json whole, change one section and write it whole: the trust move, the
|
||||
// PBS seed and the backup-target move. Without one shared lock two of them interleave and the second puts back the
|
||||
// document it read before the first one's change (found by review, 2026-11-14). Each must take config.FileMu as its
|
||||
// FIRST statement and release it by defer — asserted from the source, because the race itself is not a thing a unit
|
||||
// test can stage reliably.
|
||||
func TestAgentJSONWritersTakeTheSharedLock(t *testing.T) {
|
||||
for _, w := range []struct{ file, fn string }{
|
||||
{"../../internal/pvetrust/pvetrust.go", "writeConfig"},
|
||||
{"../../internal/pbsdr/manager.go", "seedEscrowStorageID"},
|
||||
{"../../internal/localapi/backup_target.go", "setConfiguredBackupTarget"},
|
||||
} {
|
||||
f, err := parser.ParseFile(token.NewFileSet(), w.file, nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", w.file, err)
|
||||
}
|
||||
found := false
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != w.fn || fn.Body == nil {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
isLockCall := func(e ast.Expr, name string) bool {
|
||||
c, ok := e.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
sel, ok := c.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != name {
|
||||
return false
|
||||
}
|
||||
inner, ok := sel.X.(*ast.SelectorExpr)
|
||||
if !ok || inner.Sel.Name != "FileMu" {
|
||||
return false
|
||||
}
|
||||
pkg, ok := inner.X.(*ast.Ident)
|
||||
return ok && pkg.Name == "config"
|
||||
}
|
||||
if len(fn.Body.List) < 2 {
|
||||
t.Errorf("%s %s: too short to hold the lock", w.file, w.fn)
|
||||
continue
|
||||
}
|
||||
first, ok1 := fn.Body.List[0].(*ast.ExprStmt)
|
||||
second, ok2 := fn.Body.List[1].(*ast.DeferStmt)
|
||||
if !ok1 || !isLockCall(first.X, "Lock") || !ok2 || !isLockCall(second.Call, "Unlock") {
|
||||
t.Errorf("%s %s: must start with config.FileMu.Lock() and defer config.FileMu.Unlock()", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("%s: function %s not found — a writer of agent.json was renamed; keep it under the shared lock and update this test", w.file, w.fn)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
|
||||
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
|
||||
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
|
||||
// passes as held → FAILS.
|
||||
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
|
||||
body := func(id string) []byte {
|
||||
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
|
||||
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
|
||||
return b
|
||||
}
|
||||
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
|
||||
}
|
||||
}
|
||||
ctx := context.Background()
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
|
||||
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
|
||||
}
|
||||
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
|
||||
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
|
||||
}
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
|
||||
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
|
||||
}
|
||||
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
|
||||
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
|
||||
t.Fatal("a package without an identity blob cannot be compared — an error")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||
)
|
||||
|
||||
// R-1250: SetLineTest takes a LineTestReader and finds the request half by a type assertion, so a Tester that lost
|
||||
// WaitingForQuiet or RunNow would compile and silently answer 503 to every seeding box. This line makes it a build
|
||||
// failure instead.
|
||||
var _ localapi.LineTestRequester = (*linetest.Tester)(nil)
|
||||
|
||||
// R-1247: the weekly line test has the fstrim shape (component + reporter seam + goroutine) plus a local-API reader,
|
||||
// so its wiring is asserted from the AST. It must share the ONE heavy-op gate (heavyOps) — or it could run beside a
|
||||
// backup — and take its server from the tunnel manager, never a literal address.
|
||||
func TestMainWiresTheLineTest(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
var withGate, withTunnelTarget, reporterWired, localAPIWired, started bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.CallExpr:
|
||||
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
|
||||
switch fn.Sel.Name {
|
||||
case "New":
|
||||
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "linetest" && len(node.Args) >= 3 {
|
||||
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
|
||||
withGate = true
|
||||
}
|
||||
if sel, ok := node.Args[1].(*ast.SelectorExpr); ok && sel.Sel.Name == "LineTestTarget" {
|
||||
withTunnelTarget = true
|
||||
}
|
||||
}
|
||||
case "SetLineTestReporter":
|
||||
reporterWired = true
|
||||
case "SetLineTest":
|
||||
localAPIWired = true
|
||||
}
|
||||
}
|
||||
case *ast.GoStmt:
|
||||
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
|
||||
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "lineTest" {
|
||||
started = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !withGate {
|
||||
t.Error("main.go never calls linetest.New(..., heavyOps, ...) — no line test, or one outside the heavy-op gate")
|
||||
}
|
||||
if !withTunnelTarget {
|
||||
t.Error("linetest.New does not take its server from wgMgr.LineTestTarget")
|
||||
}
|
||||
if !reporterWired {
|
||||
t.Error("main.go never calls collector.SetLineTestReporter — the line_test stanza never reaches the hub")
|
||||
}
|
||||
if !localAPIWired {
|
||||
t.Error("main.go never calls localSrv.SetLineTest — the controller's GET /host/line-test always reads present:false")
|
||||
}
|
||||
if !started {
|
||||
t.Error("main.go never starts the line test with `go lineTest.Run(ctx)`")
|
||||
}
|
||||
}
|
||||
+253
-4
@@ -43,6 +43,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
|
||||
@@ -53,6 +54,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfheal"
|
||||
@@ -215,6 +217,15 @@ func main() {
|
||||
return
|
||||
}
|
||||
|
||||
// Only the daemon repairs: a selftest can run beside a live agent that is in the middle of a write.
|
||||
var cutWriteNote string
|
||||
var cutWriteErr error
|
||||
if selftest.mode == "" {
|
||||
cutWriteNote, cutWriteErr = recoverCutConfigWrite(cfgPath)
|
||||
if cutWriteErr != nil {
|
||||
fmt.Fprintln(os.Stderr, "config repair:", cutWriteErr)
|
||||
}
|
||||
}
|
||||
cfg, err := config.Load(cfgPath)
|
||||
if err != nil {
|
||||
// A missing default config file is fine if env provides the values; only a
|
||||
@@ -228,6 +239,11 @@ func main() {
|
||||
// logRing is the always-DEBUG capture ring (v0.83.0 observability): served by the
|
||||
// local API's GET /debug/logs and the heartbeat log-pull. Selftests ignore it.
|
||||
logger, logRing := applog.New(cfg.LogLevel)
|
||||
if cutWriteErr != nil {
|
||||
logger.Error("config: the last write of the config was cut and could not be repaired", "err", cutWriteErr)
|
||||
} else if cutWriteNote != "" {
|
||||
logger.Warn("config: a write of the config was cut by a stop — repaired at start", "result", cutWriteNote)
|
||||
}
|
||||
|
||||
switch selftest.mode {
|
||||
case "":
|
||||
@@ -250,6 +266,8 @@ func main() {
|
||||
os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
|
||||
case "os-facts":
|
||||
os.Exit(runSelftestFacts(context.Background(), cfg, logger, vmid))
|
||||
case "kernel-cleanup-dry-run":
|
||||
os.Exit(runSelftestKernelCleanupDryRun(context.Background(), cfg, logger, vmid))
|
||||
case "live-restore":
|
||||
os.Exit(runSelftestLiveRestore(context.Background(), cfg, logger, vmid))
|
||||
case "pbs-verify":
|
||||
@@ -520,6 +538,66 @@ type storeGrantRepairer struct {
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
||||
// seenPath (R-1302) is the file that remembers which tiers have EVER held the grant: only those can LOSE it. "" = no
|
||||
// memory, so every missing grant reads as a loss — the old, loud behaviour.
|
||||
seenPath string
|
||||
}
|
||||
|
||||
// everHeld reports whether this tier's grant was ever seen on this box (R-1302). The direction of a doubt is „yes": an
|
||||
// unreadable or corrupt file keeps the loss ERROR, so a real loss is never quietened by a broken record.
|
||||
func (r *storeGrantRepairer) everHeld(target string) bool {
|
||||
if r == nil || r.seenPath == "" {
|
||||
return true
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
seen, err := readGrantSeen(r.seenPath)
|
||||
return err != nil || seen[target]
|
||||
}
|
||||
|
||||
// noteHeld records that this tier holds the grant. A write that fails is logged and costs one false „first grant" at most
|
||||
// after a restart — the old file, or none, stands.
|
||||
func (r *storeGrantRepairer) noteHeld(target string) {
|
||||
if r == nil || r.seenPath == "" {
|
||||
return
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
seen, err := readGrantSeen(r.seenPath)
|
||||
if err != nil {
|
||||
seen = map[string]bool{} // a corrupt file is replaced by what is true now
|
||||
}
|
||||
if seen[target] {
|
||||
return
|
||||
}
|
||||
seen[target] = true
|
||||
b, _ := json.Marshal(seen)
|
||||
tmp := r.seenPath + ".tmp"
|
||||
if werr := os.WriteFile(tmp, b, 0o600); werr == nil {
|
||||
werr = os.Rename(tmp, r.seenPath)
|
||||
if werr == nil {
|
||||
return
|
||||
}
|
||||
err = werr
|
||||
} else {
|
||||
err = werr
|
||||
}
|
||||
repairLogger(r).Warn("store-grant: could not record that this tier holds its grant (R-1302)", "target", target, "err", err)
|
||||
}
|
||||
|
||||
func readGrantSeen(path string) (map[string]bool, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if os.IsNotExist(err) {
|
||||
return map[string]bool{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
if err := json.Unmarshal(b, &seen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return seen, nil
|
||||
}
|
||||
|
||||
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
||||
@@ -630,6 +708,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
||||
// log-only one and the hub never learns anything happened (measured live, see the window's
|
||||
// comment).
|
||||
repair.noteHeld(targetID) // R-1302: a tier that holds its grant can lose it from now on
|
||||
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
||||
}
|
||||
|
||||
@@ -678,8 +757,25 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
||||
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
||||
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
||||
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
||||
return storeGrantAfterRepair(targetID, critical, repair, s)
|
||||
}
|
||||
|
||||
// storeGrantAfterRepair is the verdict once a repair is confirmed (split out for the tests). R-1302: the FIRST grant of a
|
||||
// tier that never held one is no loss — on a fresh install the whole-guest tier is armed before its storage exists, so
|
||||
// every new box logged „investigate the loss" and a DEGRADED capability at its first start, which teaches the reader to
|
||||
// skip the line. That grant logs INFO and reports healthy. A tier seen holding its grant, or one this box cannot tell
|
||||
// about, keeps R-190's ERROR and its one degraded cycle. Pinned by TestR1302_*.
|
||||
func storeGrantAfterRepair(targetID string, critical bool, repair *storeGrantRepairer, healthy capability.Status) capability.Status {
|
||||
if !repair.everHeld(targetID) {
|
||||
repair.noteHeld(targetID)
|
||||
repairLogger(repair).Info("store-grant: granted — the first grant of this tier, nothing was lost (R-1302)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||
healthy.Status, healthy.Reason = capability.StatusOK, ""
|
||||
return healthy
|
||||
}
|
||||
repair.noteRepaired(targetID, time.Now())
|
||||
s = storeGrantRepairedVerdict(targetID, critical)
|
||||
s := storeGrantRepairedVerdict(targetID, critical)
|
||||
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||
@@ -852,8 +948,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
|
||||
// invoked through the privileged path, which prepends sudo itself.
|
||||
grantRepairer := &storeGrantRepairer{
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json"), // R-1302
|
||||
}
|
||||
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
|
||||
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
|
||||
@@ -906,6 +1003,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
osLeg := newOSLeg(cfg, client, px, logger)
|
||||
collector.SetSystemReporter(&factsReporter{leg: osLeg, guest: firstGuest(px)}) // R-852: the versions
|
||||
desiredSyncer.AddConsumer(osLeg)
|
||||
// R-1380 (D80, felhom.eu `03` §12a): trust in Proxmox by a copy of the box's own root CA. At this start the root
|
||||
// step makes the copy, a REAL call proves it while the pinned fingerprint is still the trust, and only then ONE
|
||||
// write of the config sets ca_file and clears the fingerprint. A call that fails on trust later names the cause and
|
||||
// runs the same order again (a pinned box whose certificate changed moves right then, with no restart).
|
||||
pveTrust := newPVETrust(cfg, px, osLeg, logger)
|
||||
collector.SetProxmoxTrustSource(func() any { return pveTrust.Status() })
|
||||
loop.SetCollectErrorHook(pveTrust.OnCallError)
|
||||
// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
|
||||
// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
|
||||
desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
|
||||
@@ -936,6 +1040,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// network and stays under the hub's 45-minute host_stale (its
|
||||
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
|
||||
// R-1380: the trust order at this start (the copy, the proof, then the config) — before the first report cycle, so
|
||||
// a pinned box whose certificate changed while the agent was down is not blind for a cycle. Then on a timer.
|
||||
pveTrust.Ensure(ctx, "start")
|
||||
go pveTrust.Run(ctx, time.Hour, 24*time.Hour)
|
||||
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
|
||||
// after start so the guest has settled; a failure is logged and the next start retries.
|
||||
go func() {
|
||||
@@ -1204,6 +1312,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
return release, nil
|
||||
}}
|
||||
daemonCtx := ctx
|
||||
profileGate := func() (func(), string, bool) { return heavyOps.TryAcquire("os-profile-refresh") }
|
||||
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
||||
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
||||
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
||||
@@ -1214,7 +1324,20 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
names = append(names, d.Name)
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
// R-1388: a new wrapper may know the profile refresh the old one refused — ask again (its own goroutine:
|
||||
// it may wait for the heavy-op gate, and the signed job must not wait with it).
|
||||
go osLeg.ProfileRefresh(daemonCtx, "config bundle installed", profileGate, 12, 5*time.Minute)
|
||||
// R-1380: a new wrapper may know the CA-copy step the old one refused — run the trust order again.
|
||||
go pveTrust.Ensure(daemonCtx, "config bundle installed")
|
||||
}}
|
||||
// R-1388 / R-1301 (`11` §5.12): after a start, load the security profiles from a fresh compile when the box runs
|
||||
// another kernel than at the last refresh (the wrapper keeps that record and answers "not needed" otherwise).
|
||||
// R-1383 (release 3): the last refresh, and how this process's last ask ended, ride the host report.
|
||||
collector.SetProfileRefreshSource(osLeg.ProfileRefreshStanza)
|
||||
go osLeg.ProfileRefresh(daemonCtx, "agent start", profileGate, 12, 5*time.Minute)
|
||||
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
||||
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
||||
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
|
||||
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
||||
// to the local API server built below).
|
||||
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
||||
@@ -1246,6 +1369,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
|
||||
// subprocess used, under the same per-host key, before the code is handed out.
|
||||
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
|
||||
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
|
||||
// anything — the same per-host GET the identity read-back uses.
|
||||
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
|
||||
if client == nil {
|
||||
return false, fmt.Errorf("hub client not configured")
|
||||
}
|
||||
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
|
||||
},
|
||||
DRConfigured: func() bool {
|
||||
if drConfigured != nil {
|
||||
return drConfigured()
|
||||
@@ -1255,6 +1386,21 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
// R-1383 (D78): the guest-power watchdog's give-up rides the host report (it was one journal line).
|
||||
collector.SetGuestPowerGaveUpSource(localSrv.GuestPowerGaveUpStanza)
|
||||
}
|
||||
if localSrv != nil {
|
||||
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
|
||||
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
|
||||
// which it requests at once through the same cap-1 trigger the storage watchdog uses.
|
||||
localSrv.SetOperatorActionTrigger(func() {
|
||||
select {
|
||||
case storageTrigger <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
})
|
||||
collector.SetOperatorActionResults(localSrv.OperatorActionResults)
|
||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner, localSrv))
|
||||
backupCapRefusal = localSrv.BackupCapRefusal // R-973
|
||||
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||
@@ -1641,6 +1787,24 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
} else {
|
||||
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
|
||||
}
|
||||
// R-1247 (operator ruling 2026-10-24 07:26): the weekly upload line test — a 10-second iperf3 upload from this host
|
||||
// to ep0's tunnel-only server, Tuesday 10:00-15:59 local, after 10 quiet minutes, under the one-heavy-op gate. Only
|
||||
// with the offsite tunnel (the server is reachable nowhere else). Not part of the errc fan-out.
|
||||
switch {
|
||||
case !cfg.LineTest.Enabled():
|
||||
logger.Info("linetest: weekly upload line test disabled by config (line_test.disable)")
|
||||
case wgMgr == nil:
|
||||
logger.Info("linetest: weekly upload line test off — no offsite tunnel on this host")
|
||||
default:
|
||||
lineTest := linetest.New(linetest.Client{Streams: linetest.Streams, Time: linetest.TestTime}, wgMgr.LineTestTarget,
|
||||
heavyOps, linetest.SysUplinkTx("/"), cfg.Hub.HostID,
|
||||
filepath.Join(cfg.OOB.WithDefaults().StateDir, "line-test.json"), logger)
|
||||
collector.SetLineTestReporter(lineTest)
|
||||
if localSrv != nil {
|
||||
localSrv.SetLineTest(lineTest)
|
||||
}
|
||||
go lineTest.Run(ctx)
|
||||
}
|
||||
if lanLoop != nil {
|
||||
lanServers = 1
|
||||
go func() { errc <- lanLoop.Run(ctx) }()
|
||||
@@ -3344,6 +3508,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
|
||||
|
||||
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
|
||||
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
|
||||
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
|
||||
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
|
||||
// without an identity blob cannot be compared: an error (the caller then promises nothing).
|
||||
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
|
||||
var up escrowUploadRequest
|
||||
if err := json.Unmarshal(pkg, &up); err != nil {
|
||||
return false, fmt.Errorf("reading the pending package: %w", err)
|
||||
}
|
||||
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
|
||||
if err != nil || len(mine) == 0 {
|
||||
return false, fmt.Errorf("the pending package carries no identity blob to compare")
|
||||
}
|
||||
resp, err := fetch(ctx)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||
return false, nil
|
||||
}
|
||||
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("decoding the hub's package: %w", err)
|
||||
}
|
||||
return bytes.Equal(mine, theirs), nil
|
||||
}
|
||||
|
||||
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
|
||||
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
||||
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
||||
@@ -3782,14 +3972,39 @@ func (f *selftestFlag) Set(v string) error {
|
||||
f.mode = "os-update"
|
||||
case "os-facts", "live-restore": // agent v0.142.0
|
||||
f.mode = v
|
||||
case "kernel-cleanup-dry-run": // R-1368: what the clean-up of old kernels WOULD do; changes nothing
|
||||
f.mode = v
|
||||
case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
|
||||
f.mode = "wgtunnel"
|
||||
default:
|
||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update|os-facts|live-restore)", v)
|
||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|kernel-cleanup-dry-run|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update|os-facts|live-restore)", v)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// newPVETrust wires internal/pvetrust (R-1380): the root step through the OS wrapper, the proof as a throwaway client
|
||||
// with EXACTLY the trust under test, and the running client's SetTLS.
|
||||
func newPVETrust(cfg config.Config, px *proxmox.Client, leg *osupdate.Leg, logger *slog.Logger) *pvetrust.Manager {
|
||||
m := &pvetrust.Manager{ConfigPath: cfg.SourcePath, StateDir: agentStateDir(cfg), Logger: logger,
|
||||
EnsureCopy: func(ctx context.Context) (pvetrust.Copy, error) {
|
||||
r, err := leg.PVECACopy(ctx)
|
||||
return pvetrust.Copy{Path: r.Path, Outcome: r.Outcome, NotAfter: r.NotAfter, DaysLeft: r.DaysLeft}, err
|
||||
},
|
||||
Probe: func(ctx context.Context, t proxmox.TLSConfig) error {
|
||||
p, err := proxmox.NewClient(proxmox.Config{Endpoint: cfg.Proxmox.Endpoint, Node: cfg.Proxmox.Node, Token: cfg.Proxmox.Token, TLS: t})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = p.Version(ctx)
|
||||
return err
|
||||
},
|
||||
Apply: px.SetTLS,
|
||||
}
|
||||
m.Start(proxmox.TLSConfig{CAFile: cfg.Proxmox.TLS.CAFile, Fingerprint: cfg.Proxmox.TLS.Fingerprint,
|
||||
InsecureSkipVerify: cfg.Proxmox.TLS.InsecureSkipVerify})
|
||||
return m
|
||||
}
|
||||
|
||||
// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
|
||||
// the once-per-night marker live in the agent's own os/ dir.
|
||||
func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg {
|
||||
@@ -3906,6 +4121,24 @@ func runSelftestFacts(ctx context.Context, cfg config.Config, logger *slog.Logge
|
||||
return 0
|
||||
}
|
||||
|
||||
// runSelftestKernelCleanupDryRun asks the root wrapper what the clean-up of old kernels WOULD do on this box (R-1368,
|
||||
// `11` §5.11.1): the keep set, the removal set, and the refusal or "not run" reason. It changes nothing.
|
||||
//
|
||||
// sudo -u felhom-agent felhom-agent --config … --selftest=kernel-cleanup-dry-run
|
||||
func runSelftestKernelCleanupDryRun(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
|
||||
px, _ := newProxmoxClient(cfg)
|
||||
leg := newOSLeg(cfg, nil, px, logger)
|
||||
raw, err := leg.KernelCleanupDryRun(ctx, vmid)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest=kernel-cleanup-dry-run:", err)
|
||||
return 1
|
||||
}
|
||||
var v any
|
||||
_ = json.Unmarshal(raw, &v)
|
||||
printJSON("kernel clean-up — DRY RUN (nothing was changed)", v)
|
||||
return 0
|
||||
}
|
||||
|
||||
// runSelftestLiveRestore is the ONE-TIME live-restore step (`09` decision 87) as a debug action — the night leg does
|
||||
// the same before a ring-0 Docker step. It prints the container ids before and after (they must not change).
|
||||
//
|
||||
@@ -4021,3 +4254,19 @@ func agentStateDir(cfg config.Config) string {
|
||||
}
|
||||
return "/var/lib/felhom-agent"
|
||||
}
|
||||
|
||||
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
|
||||
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
|
||||
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
|
||||
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
|
||||
}
|
||||
|
||||
// recoverCutConfigWrite repairs agent.json BEFORE it is read, when the last write of it was cut by a power loss
|
||||
// (config.RecoverCutWrite; R-1380). It runs before the logger exists, so main logs what it returns.
|
||||
// Pinned by TestMainRecoversACutConfigWrite.
|
||||
func recoverCutConfigWrite(cfgPath string) (string, error) {
|
||||
if cfgPath == "" {
|
||||
return "", nil
|
||||
}
|
||||
return config.RecoverCutWrite(cfgPath)
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
|
||||
// the server in the report loop's observer chain, give the collector its results and give the server the report
|
||||
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
|
||||
// RED-PROOF: seen failing before the wiring existed (all three false).
|
||||
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
var observed, results, trigger bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
fn, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch fn.Sel.Name {
|
||||
case "MultiObserver":
|
||||
for _, a := range call.Args {
|
||||
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
|
||||
observed = true
|
||||
}
|
||||
}
|
||||
case "SetOperatorActionResults":
|
||||
if len(call.Args) == 1 {
|
||||
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
|
||||
results = true
|
||||
}
|
||||
}
|
||||
case "SetOperatorActionTrigger":
|
||||
trigger = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !observed || !results || !trigger {
|
||||
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1380: the trust move is a component with seams (the root step, the proof, the running client, the report stanza,
|
||||
// the failed-call hook). A unit-green pvetrust that main.go never starts is the inert-seam defect, so the wiring is
|
||||
// asserted from the AST: built once, run at start and after a bundle install, on a timer, fed by a failed collect, and
|
||||
// reported. The same for the profile refresh (R-1388), which had no such test.
|
||||
func TestMainWiresPVETrust(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
seen := map[string]int{}
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
switch node := n.(type) {
|
||||
case *ast.CallExpr:
|
||||
sel, ok := node.Fun.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
x, _ := sel.X.(*ast.Ident)
|
||||
switch {
|
||||
case x != nil && x.Name == "pveTrust" && sel.Sel.Name == "Ensure":
|
||||
seen["ensure"]++
|
||||
case sel.Sel.Name == "SetProxmoxTrustSource":
|
||||
seen["stanza"]++
|
||||
case sel.Sel.Name == "SetProfileRefreshSource" && len(node.Args) == 1:
|
||||
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "ProfileRefreshStanza" {
|
||||
seen["profile refresh → report"]++ // R-1383
|
||||
}
|
||||
case sel.Sel.Name == "SetGuestPowerGaveUpSource" && len(node.Args) == 1:
|
||||
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "GuestPowerGaveUpStanza" {
|
||||
seen["guest-power give-up → report"]++ // R-1383
|
||||
}
|
||||
case sel.Sel.Name == "SetCollectErrorHook" && len(node.Args) == 1:
|
||||
if a, ok := node.Args[0].(*ast.SelectorExpr); ok && a.Sel.Name == "OnCallError" {
|
||||
seen["hook"]++
|
||||
}
|
||||
case x != nil && x.Name == "osLeg" && sel.Sel.Name == "ProfileRefresh":
|
||||
seen["profile-refresh"]++
|
||||
}
|
||||
case *ast.GoStmt:
|
||||
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
|
||||
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "pveTrust" {
|
||||
seen["timer"]++
|
||||
}
|
||||
}
|
||||
case *ast.KeyValueExpr:
|
||||
if k, ok := node.Key.(*ast.Ident); ok && k.Name == "Apply" {
|
||||
if v, ok := node.Value.(*ast.SelectorExpr); ok && v.Sel.Name == "SetTLS" {
|
||||
seen["apply=SetTLS"]++
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
want := map[string]int{"ensure": 2, "stanza": 1, "hook": 1, "timer": 1, "apply=SetTLS": 1, "profile-refresh": 2,
|
||||
"guest-power give-up → report": 1, "profile refresh → report": 1}
|
||||
for k, n := range want {
|
||||
if seen[k] < n {
|
||||
t.Errorf("main.go wires %q %d time(s), want at least %d (the two-call ones: at start and after a bundle install)", k, seen[k], n)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
)
|
||||
|
||||
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
|
||||
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
|
||||
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
|
||||
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
|
||||
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
|
||||
|
||||
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
|
||||
type orderRunner struct {
|
||||
mu sync.Mutex
|
||||
order []string
|
||||
}
|
||||
|
||||
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
|
||||
var p map[string]any
|
||||
b, _ := os.ReadFile(args[1])
|
||||
_ = json.Unmarshal(b, &p)
|
||||
mode, _ := p["mode"].(string)
|
||||
r.order = append(r.order, "os-apply:"+mode)
|
||||
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
|
||||
}
|
||||
if len(args) >= 1 {
|
||||
r.order = append(r.order, filepath.Base(name)+":"+args[0])
|
||||
}
|
||||
return []byte("ok"), nil, nil
|
||||
}
|
||||
|
||||
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
|
||||
t.Helper()
|
||||
stateDir := t.TempDir()
|
||||
if marker != nil {
|
||||
dir := filepath.Join(stateDir, "selfupdate")
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := json.Marshal(marker)
|
||||
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
r := &orderRunner{}
|
||||
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
|
||||
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
|
||||
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
|
||||
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
|
||||
mgr.MaybeCommit(context.Background())
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return append([]string(nil), r.order...)
|
||||
}
|
||||
|
||||
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
|
||||
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
|
||||
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
|
||||
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
|
||||
}
|
||||
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,14 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"go/ast"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
|
||||
"be missed entirely", storeGrantRepairReportWindow)
|
||||
}
|
||||
}
|
||||
|
||||
// ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
|
||||
//
|
||||
// A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
|
||||
// logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
|
||||
//
|
||||
// RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
|
||||
// TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
|
||||
func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
|
||||
log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
|
||||
}
|
||||
|
||||
func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
r, buf := r1302Repairer(t, path)
|
||||
healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
|
||||
if s.Status != capability.StatusOK || s.Reason != "" {
|
||||
t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
|
||||
}
|
||||
got := buf.String()
|
||||
if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
|
||||
t.Fatalf("a first grant must not log the loss ERROR: %s", got)
|
||||
}
|
||||
if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
|
||||
t.Fatalf("a first grant says so at INFO: %s", got)
|
||||
}
|
||||
if r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||
t.Fatal("a first grant must not arm the loss report's latch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
first, _ := r1302Repairer(t, path)
|
||||
first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
|
||||
|
||||
r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
|
||||
t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
|
||||
}
|
||||
if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
|
||||
t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
|
||||
}
|
||||
if !r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||
t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
|
||||
}
|
||||
}
|
||||
|
||||
// After a first grant the tier counts as held: the NEXT disappearance is a loss.
|
||||
func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
r, _ := r1302Repairer(t, path)
|
||||
storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||
if s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("the second disappearance is a loss; got %q", s.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
|
||||
func TestR1302_ADoubtKeepsTheError(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, _ := r1302Repairer(t, path)
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
noMemory, _ := r1302Repairer(t, "")
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("no record must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||
t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The wiring, read from the source: production names the file, and a healthy probe records the tier.
|
||||
func TestR1302_TheMemoryIsWired(t *testing.T) {
|
||||
b, err := os.ReadFile("main.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src := string(b)
|
||||
if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
|
||||
t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
|
||||
}
|
||||
start := strings.Index(src, "func storeGrantStatus(")
|
||||
if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
|
||||
t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
|
||||
}
|
||||
}
|
||||
@@ -123,6 +123,10 @@ func run() error {
|
||||
|
||||
now := time.Now().UTC()
|
||||
issued := now
|
||||
if *ttl <= 0 || *ttl > authz.MaxWindow {
|
||||
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
|
||||
os.Exit(2)
|
||||
}
|
||||
expires := now.Add(*ttl)
|
||||
|
||||
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
|
||||
|
||||
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
|
||||
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
||||
# + container across spaces, spike-confirmed)
|
||||
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
||||
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
|
||||
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
|
||||
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
|
||||
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
|
||||
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
|
||||
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
|
||||
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
|
||||
|
||||
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
||||
|
||||
+634
-20
@@ -97,9 +97,12 @@ OOMCHECK_SETTLE = 2
|
||||
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
||||
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
||||
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
||||
# proxmox-grub is GRUB's Proxmox meta-package (it depends on grub-efi-amd64 of the same version); the `grub` above is
|
||||
# anchored at the start, so it missed it. Measured 2026-11-05..07 on demo-felhom: its upgrade pulled grub-efi-amd64 and
|
||||
# the whole pve step was refused R6 every night. Pinned by test_ring0_pending_pve_leaves_proxmox_grub_out.
|
||||
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
||||
r"proxmox-secure-boot-support)")
|
||||
r"proxmox-secure-boot-support|proxmox-grub)")
|
||||
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
||||
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
||||
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
||||
@@ -152,9 +155,36 @@ KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pv
|
||||
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
|
||||
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
|
||||
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
|
||||
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
|
||||
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel",
|
||||
"kernel-cleanup")
|
||||
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
|
||||
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||
# The clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1). A step that ENDED in one of these phases
|
||||
# and is not "good" keeps every kernel (the box fell back or reverted); any other phase is "not ended" (P1).
|
||||
# Trust in Proxmox by a copy of the box's own root CA (R-1380, `09` §3 decision 303, felhom.eu `03` §12a). The agent's
|
||||
# user cannot read Proxmox's CA file (0640 root:www-data); this root step keeps a readable copy of that ONE public
|
||||
# certificate where the agent's `proxmox.tls.ca_file` points.
|
||||
PVE_CA_SOURCE = "/etc/pve/pve-root-ca.pem"
|
||||
PVE_CA_COPY = "/etc/felhom-agent/pve-root-ca.pem" # 0644 root: a CA certificate is public; the agent only reads it
|
||||
PVE_CA_MAX_BYTES = 64 * 1024
|
||||
PVE_LOCAL_API = "127.0.0.1:8006" # pveproxy on this host (the installer's proof uses the same)
|
||||
PEM_CERT_RE = re.compile(rb"-----BEGIN CERTIFICATE-----\s[A-Za-z0-9+/=\s]+-----END CERTIFICATE-----")
|
||||
# The security-profile refresh (R-1388, R-1301, `11` §5.12; measured on demo-hp 2026-11-14,
|
||||
# felhom.eu audits/profile-refresh-spike-2026-11-14/). AppArmor keeps one cache folder per kernel feature set; a
|
||||
# profile that changed while ANOTHER kernel ran leaves an old compiled copy in this kernel's folder, with a time that
|
||||
# looks newer than its source — and the kernel loads it at the next start on this kernel.
|
||||
PROFILE_DIR = "/etc/apparmor.d"
|
||||
PROFILE_CACHE = "/var/cache/apparmor"
|
||||
PROFILE_STATE = "/var/lib/felhom-os-apply/profile-refresh.json" # 0644 root: the kernel the last refresh ran on
|
||||
PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", PROFILE_DIR]
|
||||
PROFILE_PATH_NAME_RE = re.compile(r"^/[A-Za-z0-9._/+-]+$") # a profile whose name is one plain program path
|
||||
UNIT_NAME_RE = re.compile(r"^[A-Za-z0-9@:._-]+\.service$")
|
||||
PROFILE_RESTART_MAX = 3
|
||||
KERNEL_ENDED_NOT_GOOD = ("self_reverted", "revert_failed", "fell_back", "cancelled")
|
||||
KERNEL_LEFTOVER_PATHS = ("/lib/modules/{v}", "/boot/vmlinuz-{v}", "/boot/initrd.img-{v}", "/boot/System.map-{v}",
|
||||
"/boot/config-{v}")
|
||||
PINNED_KVER_RE = re.compile(r"gnulinux-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)-advanced-")
|
||||
ANY_KVER_RE = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve")
|
||||
|
||||
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
||||
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
||||
@@ -168,7 +198,10 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
|
||||
# sshd → `sshd -t -f` on the snippet alone (R-982).
|
||||
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
|
||||
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
|
||||
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none);
|
||||
# appliance → only on a box whose ROOT-OWNED install record says appliance (D43 = A, operator 2026-10-24: a BYO
|
||||
# host keeps its own settings, `11` §1). On a BYO box nothing is written and a copy WE wrote (this exact path, our
|
||||
# first line) is removed; a box with no readable record is left exactly as it is (it cannot prove which it is).
|
||||
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
|
||||
# R-861 (agent v0.146.0): the signed agent update, verified here (mode agent_update), then the A/B flip.
|
||||
SELFUPDATE_OP = "agent_update"
|
||||
@@ -184,9 +217,13 @@ BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies
|
||||
BUNDLE_KEEP = 3
|
||||
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
|
||||
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
|
||||
# R-1280: the longest validity window a signed job may carry — the agent's authz.MaxWindow, the same 24 h.
|
||||
MAX_WINDOW_S = 24 * 3600
|
||||
OOB_DIR = "/etc/felhom-sshd"
|
||||
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
|
||||
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||
# D43 = A (R-1257): the first line every copy of the drop-in we ship carries — what proves a file on a BYO host is ours.
|
||||
SSHD_DROPIN_HEADER = b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982"
|
||||
|
||||
BUNDLE_FILES = [
|
||||
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
|
||||
@@ -227,7 +264,8 @@ BUNDLE_FILES = [
|
||||
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
|
||||
# R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
|
||||
# password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
|
||||
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "replace"),
|
||||
# D43 = A (operator 2026-10-24, R-1257): appliances only — a BYO host keeps its own SSH settings (`11` §1).
|
||||
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "appliance"),
|
||||
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
|
||||
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
|
||||
]
|
||||
@@ -483,7 +521,8 @@ class Apply:
|
||||
|
||||
def check_plan(self, plan):
|
||||
mode = plan.get("mode", "apply")
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
|
||||
"oom-check", "profile-refresh", "pve-ca-copy") \
|
||||
+ KERNEL_MODES:
|
||||
raise Refused("R11", f"unknown mode {mode!r}")
|
||||
if mode == "agent_update":
|
||||
@@ -494,6 +533,29 @@ class Apply:
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "bundle is a host-layer mode")
|
||||
return mode, "host", 0, "bundle"
|
||||
if mode == "bundle-policy":
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "bundle-policy is a host-layer mode")
|
||||
return mode, "host", 0, "bundle-policy"
|
||||
if mode == "pve-ca-copy":
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "pve-ca-copy is a host-layer mode")
|
||||
# R-1380 (D80): ONE fixed source, ONE fixed place — the plan may name NOTHING (no path, no file, no content)
|
||||
extra = sorted(k for k in plan if k not in ("mode", "layer", "release_id", "run_id", "trigger", "ring"))
|
||||
if extra:
|
||||
raise Refused("T1", f"a pve-ca-copy plan carries {', '.join(extra)} — it takes nothing from a plan: "
|
||||
f"the source is {PVE_CA_SOURCE} and the copy is {PVE_CA_COPY}, both fixed")
|
||||
return mode, "host", 0, "pve-ca-copy"
|
||||
if mode == "profile-refresh":
|
||||
if plan.get("layer") != "host":
|
||||
raise Refused("R11", "profile-refresh is a host-layer mode")
|
||||
# it takes NOTHING from the plan but dry_run: no profile name, no unit name, no package (R-1388)
|
||||
for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"):
|
||||
if plan.get(k):
|
||||
raise Refused("F1", f"a profile-refresh plan carries {k!r} — it takes nothing from a plan but dry_run")
|
||||
if not isinstance(plan.get("dry_run", False), bool):
|
||||
raise Refused("F1", "dry_run must be true or false")
|
||||
return mode, "host", 0, "profile-refresh"
|
||||
layer = plan.get("layer")
|
||||
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
||||
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
||||
@@ -524,6 +586,17 @@ class Apply:
|
||||
rid = plan.get("release_id", "")
|
||||
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
||||
raise Refused("R11", f"release_id {rid!r} is not a plain id")
|
||||
if mode == "kernel-cleanup":
|
||||
# `11` §5.11.1, rule 4 of the release: the clean-up NEVER takes a package name (or any choice of what to
|
||||
# remove) from a plan — the wrapper computes the set from the box. Pinned by test_cleanup_plan_names_refused.
|
||||
for k in ("packages", "allow_new", "signed", "expect_kver", "undo", "snapshot"):
|
||||
if plan.get(k):
|
||||
raise Refused("C1", f"a kernel-cleanup plan carries {k!r} — the clean-up takes nothing from a plan "
|
||||
f"but dry_run; the wrapper computes the set from the box")
|
||||
if plan.get("select", "listed") != "listed":
|
||||
raise Refused("C1", "a kernel-cleanup plan carries a select — the wrapper computes the set from the box")
|
||||
if not isinstance(plan.get("dry_run", False), bool):
|
||||
raise Refused("C1", "dry_run must be true or false")
|
||||
if plan.get("allow_new"):
|
||||
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
||||
select = plan.get("select", "listed")
|
||||
@@ -655,6 +728,8 @@ class Apply:
|
||||
iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ"))
|
||||
except (KeyError, ValueError, TypeError):
|
||||
raise Refused("R3", "the signed job has no readable time window")
|
||||
if exp - iss > MAX_WINDOW_S:
|
||||
raise Refused("R3", "the signed job's window is longer than 24 h (R-1280)")
|
||||
if now > exp or now < iss - 120:
|
||||
raise Refused("R3", "the signed job is expired or not yet valid")
|
||||
nonce = op.get("nonce")
|
||||
@@ -1064,6 +1139,15 @@ class Apply:
|
||||
return self.facts()
|
||||
if self.mode == "bundle":
|
||||
return Bundle(self).from_plan(plan)
|
||||
if self.mode == "bundle-policy":
|
||||
return Bundle(self).policy_pass()
|
||||
if self.mode == "pve-ca-copy":
|
||||
# NO appliance check, on purpose (like the bundle): the copy is the agent's OWN trust file in the agent's own
|
||||
# config folder, and it changes nothing of the host's — a BYO host's agent talks to Proxmox the same way.
|
||||
return self.pve_ca_copy()
|
||||
if self.mode == "profile-refresh":
|
||||
self.check_appliance()
|
||||
return self.profile_refresh(plan)
|
||||
if self.mode == "agent_update":
|
||||
return self.agent_update(plan)
|
||||
if self.layer in ("host", "pve", "kernel"):
|
||||
@@ -1129,6 +1213,198 @@ class Apply:
|
||||
self.report["oom_check"] = self.oom_check()
|
||||
return 0
|
||||
|
||||
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80) ----------
|
||||
def is_live_ca(self, block):
|
||||
"""One PEM certificate block: is it a CA certificate that has not ended? (openssl, the bytes on stdin)."""
|
||||
try:
|
||||
rc, out, _ = self.r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-checkend", "0"], 30,
|
||||
stdin=block.decode("ascii") + "\n")
|
||||
except (UnicodeDecodeError, subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
return rc == 0 and re.search(r"\bCA:TRUE\b", out) is not None
|
||||
|
||||
def pve_serves_with(self, cafile):
|
||||
"""Does the local Proxmox API serve a certificate that verifies with this CA file ALONE? One handshake to
|
||||
PVE_LOCAL_API; nothing is sent. Any doubt (no answer, a timeout) is "no" — the previous root then stays."""
|
||||
try:
|
||||
rc, out, err = self.r.host(["openssl", "s_client", "-connect", PVE_LOCAL_API, "-CAfile", cafile, "-no-CApath",
|
||||
"-verify_return_error"], 20, stdin="")
|
||||
except (subprocess.SubprocessError, OSError):
|
||||
return False
|
||||
return rc == 0 and "Verify return code: 0 (ok)" in (out + err)
|
||||
|
||||
def pve_ca_copy(self):
|
||||
"""Copy exactly PVE_CA_SOURCE to exactly PVE_CA_COPY — only if the source is ONE CA certificate that has not
|
||||
ended — written whole or not at all (a root-owned temp file beside the target, fsync, rename). "same" when the
|
||||
copy is exactly the source's bytes. The bytes that are checked are the bytes that are written first; only
|
||||
during a change of the root the copy also holds its previous certificate behind them (see below)."""
|
||||
r = self.r
|
||||
res = {"outcome": None, "path": PVE_CA_COPY}
|
||||
self.report["pve_ca_copy"] = res
|
||||
try:
|
||||
data = r.read_bytes(PVE_CA_SOURCE)
|
||||
except OSError as e:
|
||||
raise Refused("T2", f"{PVE_CA_SOURCE} cannot be read ({e.__class__.__name__}) — is this a Proxmox host?")
|
||||
if not data or len(data) > PVE_CA_MAX_BYTES:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is {len(data)} bytes — not one certificate")
|
||||
certs = PEM_CERT_RE.findall(data)
|
||||
if len(certs) != 1 or b"PRIVATE KEY" in data:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} holds {len(certs)} certificate(s)"
|
||||
+ (" and a private key" if b"PRIVATE KEY" in data else "") + " — exactly one CA certificate is expected")
|
||||
try:
|
||||
text = data.decode("ascii") # PEM is ASCII; the runner hands text to openssl's stdin
|
||||
except UnicodeDecodeError:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is not a PEM text file")
|
||||
rc, out, err = r.host(["openssl", "x509", "-noout", "-ext", "basicConstraints", "-enddate"], 30, stdin=text)
|
||||
if rc != 0:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is not a readable certificate: {(out + err).strip()[-160:]}")
|
||||
if not re.search(r"\bCA:TRUE\b", out):
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} is not a CA certificate (no CA:TRUE) — a leaf or any other file is never copied")
|
||||
m = re.search(r"notAfter=(.+)", out)
|
||||
try:
|
||||
import calendar
|
||||
not_after = calendar.timegm(time.strptime(m.group(1).strip(), "%b %d %H:%M:%S %Y GMT")) # UTC, no DST
|
||||
except (AttributeError, ValueError):
|
||||
raise Refused("T3", f"the end date of {PVE_CA_SOURCE} cannot be read")
|
||||
days = int((not_after - r.now()) // 86400)
|
||||
if days < 0:
|
||||
raise Refused("T3", f"{PVE_CA_SOURCE} ended {-days} day(s) ago — an ended CA is never copied")
|
||||
res.update(sha256=sha256_hex(data), not_after=_iso(not_after), days_left=days)
|
||||
try:
|
||||
have = r.read_bytes(PVE_CA_COPY)
|
||||
except OSError:
|
||||
have = None
|
||||
if have == data:
|
||||
res["outcome"] = "same"
|
||||
return 0
|
||||
# The copy the agent trusts by is replaced BEFORE anything can be proven with the new root (Proxmox serves its
|
||||
# old certificate until pveproxy is reloaded). So at a CHANGE the copy's previous first certificate — written
|
||||
# by this same step — is kept behind the new one, and the agent trusts both roots across the change, also
|
||||
# after a restart. That second root is kept ONLY for the changeover: it is dropped as soon as Proxmox serves a
|
||||
# certificate that verifies with the new root alone, and at once when it is no longer a CA that has not ended.
|
||||
# The copy is compared EXACTLY: nothing this step did not check is ever left in it. (Review, 2026-11-14.)
|
||||
# To drop the old root at once by hand (a root that was changed because it leaked): delete the copy.
|
||||
prev = PEM_CERT_RE.findall(have) if have and b"PRIVATE KEY" not in have else []
|
||||
if len(prev) == 2 and have == data + prev[1] + b"\n":
|
||||
over = self.pve_serves_with(PVE_CA_SOURCE)
|
||||
if not over and self.is_live_ca(prev[1]):
|
||||
res["outcome"], res["kept_previous"] = "same", True # the changeover is still on
|
||||
return 0
|
||||
r.put_file(PVE_CA_COPY, data, 0o644)
|
||||
res["outcome"], res["replaced"], res["kept_previous"], res["dropped_previous"] = "written", True, False, True
|
||||
r.log(f"os-apply: PVE CA COPY the previous root is dropped from {PVE_CA_COPY} "
|
||||
f"({'Proxmox serves a certificate of the new root' if over else 'the previous root is no longer a live CA'})")
|
||||
return 0
|
||||
want, kept = data, False
|
||||
if prev and prev[0].strip() != certs[0].strip() and self.is_live_ca(prev[0]):
|
||||
want, kept = data + prev[0] + b"\n", True
|
||||
r.put_file(PVE_CA_COPY, want, 0o644)
|
||||
res["outcome"], res["replaced"], res["kept_previous"] = "written", have is not None, kept
|
||||
r.log(f"os-apply: PVE CA COPY written {PVE_CA_COPY} sha256={res['sha256'][:16]} ends={res['not_after']} "
|
||||
f"({'replaced an older copy' if have is not None else 'first copy'})")
|
||||
return 0
|
||||
|
||||
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
|
||||
def profile_cache(self):
|
||||
"""{"<folder>/<file>": sha256} of every cached profile, every kernel's folder."""
|
||||
out = {}
|
||||
for d in self.r.list_dir(PROFILE_CACHE):
|
||||
dp = f"{PROFILE_CACHE}/{d}"
|
||||
if not self.r.isdir(dp):
|
||||
continue
|
||||
for f in self.r.list_dir(dp):
|
||||
if f.startswith("."):
|
||||
continue
|
||||
try:
|
||||
out[f"{d}/{f}"] = sha256_hex(self.r.read_bytes(f"{dp}/{f}"))
|
||||
except OSError:
|
||||
out[f"{d}/{f}"] = "unreadable"
|
||||
return out
|
||||
|
||||
def profile_refresh(self, plan):
|
||||
"""Load every AppArmor profile from a FRESH compile (the cache ignored and rewritten), once per kernel the box
|
||||
starts on; then restart ONLY a failed service whose own program's profile had a stale cached copy.
|
||||
|
||||
Changes no package and takes no name from the plan. Measured on demo-hp with both guests running: 0.5 s, no
|
||||
guest or container touched, the same profiles loaded in the same modes.
|
||||
|
||||
WHY NOT a "loaded hash changed" check: for a file that defines several profiles the kernel's hash differs
|
||||
between a cache load and a fresh load of the SAME policy (17 of 119 on demo-hp), and one file (lxc-containers)
|
||||
compiles to a different binary every time. The only reliable sign of a stale copy is the CACHED FILE changing
|
||||
when it is rewritten — that is what the restart rule reads."""
|
||||
r, dry = self.r, plan.get("dry_run") is True
|
||||
res = {"dry_run": dry, "outcome": None, "rewritten": [], "restarted": [], "not_restarted": []}
|
||||
self.report["profile_refresh"] = res
|
||||
rc, out, _ = r.host(["uname", "-r"], 30)
|
||||
kernel = out.strip() if rc == 0 else ""
|
||||
try:
|
||||
rec = json.loads(r.read_file(PROFILE_STATE))
|
||||
rec = rec if isinstance(rec, dict) else {}
|
||||
except (OSError, ValueError):
|
||||
rec = {}
|
||||
res["kernel"], res["last_refresh_kernel"], res["last_refresh_at"] = kernel or "unknown", rec.get("kernel"), rec.get("at")
|
||||
if not kernel:
|
||||
raise Refused("F2", "the running kernel could not be read")
|
||||
if not r.isdir(PROFILE_DIR) or not r.isdir(PROFILE_CACHE):
|
||||
res["outcome"] = "not_needed"
|
||||
res["reason"] = "this box has no AppArmor profile folder or no profile cache"
|
||||
return 0
|
||||
needed = rec.get("kernel") != kernel
|
||||
if not needed:
|
||||
res["outcome"], res["reason"] = "not_needed", f"the profiles were refreshed on this kernel ({kernel}) at {rec.get('at')}"
|
||||
return 0
|
||||
if dry:
|
||||
res["outcome"], res["would"] = "dry_run", "refresh"
|
||||
return 0
|
||||
before = self.profile_cache()
|
||||
t0 = time.time()
|
||||
rc, out, err = r.host(list(PROFILE_REFRESH_CMD), 600)
|
||||
res["seconds"] = round(time.time() - t0, 1)
|
||||
if rc != 0:
|
||||
res["outcome"] = "failed"
|
||||
self.report["failed"] = {"rc": rc, "step": "profile-refresh", "tail": (out + err).strip().splitlines()[-3:]}
|
||||
r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)")
|
||||
return 3
|
||||
after = self.profile_cache()
|
||||
res["rewritten"] = sorted(k for k in after if before.get(k) != after[k])
|
||||
# --- the narrow restart rule: a FAILED service, whose ExecStart program IS the name of a profile whose cached
|
||||
# copy was stale. Never "whatever failed"; never a unit named by the plan. ---
|
||||
programs = {}
|
||||
for key in res["rewritten"]:
|
||||
fname = key.split("/", 1)[1]
|
||||
rc_n, names, _ = r.host(["apparmor_parser", "-N", f"{PROFILE_DIR}/{fname}"], 60)
|
||||
if rc_n != 0:
|
||||
continue
|
||||
for n in names.split():
|
||||
if PROFILE_PATH_NAME_RE.match(n):
|
||||
programs[n] = fname
|
||||
if programs:
|
||||
rc_u, units, _ = r.host(["systemctl", "list-units", "--type=service", "--state=failed", "--no-legend", "--plain"], 60)
|
||||
for line in (units.splitlines() if rc_u == 0 else []):
|
||||
unit = (line.split() or [""])[0]
|
||||
if not UNIT_NAME_RE.match(unit):
|
||||
continue
|
||||
_, es, _ = r.host(["systemctl", "show", "-p", "ExecStart", "--value", unit], 30)
|
||||
m = re.search(r"\bpath=(\S+)", es)
|
||||
prog = m.group(1) if m else ""
|
||||
if prog not in programs:
|
||||
continue
|
||||
if len(res["restarted"]) >= PROFILE_RESTART_MAX:
|
||||
res["not_restarted"].append({"unit": unit, "program": prog, "reason": f"more than {PROFILE_RESTART_MAX} units match — look at the box"})
|
||||
continue
|
||||
rc_r, o2, e2 = r.host(["systemctl", "restart", unit], 120)
|
||||
_, act, _ = r.host(["systemctl", "is-active", unit], 30)
|
||||
res["restarted"].append({"unit": unit, "program": prog, "profile_file": programs[prog], "rc": rc_r,
|
||||
"active": act.strip() == "active"})
|
||||
r.log(f"os-apply: PROFILE REFRESH restarted {unit} (failed; its program {prog} had a stale cached profile) "
|
||||
f"rc={rc_r} now={act.strip() or 'unknown'}")
|
||||
r.put_file(PROFILE_STATE, (json.dumps({"kernel": kernel, "at": _iso(r.now()), "rewritten": res["rewritten"],
|
||||
"restarted": [x["unit"] for x in res["restarted"]]}, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||
res["outcome"] = "refreshed"
|
||||
r.log(f"os-apply: PROFILE REFRESH DONE kernel={kernel} (last {rec.get('kernel') or 'never'}) seconds={res['seconds']} "
|
||||
f"rewritten={len(res['rewritten'])} ({', '.join(res['rewritten'][:8])}) restarted={[x['unit'] for x in res['restarted']]}")
|
||||
return 0
|
||||
|
||||
# ---------- R-528: the memory-kill check ----------
|
||||
def oom_check(self):
|
||||
"""Run one throwaway container over its memory cap in the guest and read what the engine says about it.
|
||||
@@ -1473,6 +1749,11 @@ class Kernel:
|
||||
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
|
||||
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
|
||||
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
|
||||
kernel-cleanup remove old kernel images by the rule of `11` §5.11.1 (R-1368, decision 286). The plan names nothing
|
||||
(C1); `dry_run: true` computes and reports, and changes nothing. Its own refusals: P1–P3, K1, N1,
|
||||
S1, S2 (the design's table) and C1 (the plan names something). "Not run" (exit 0, nothing
|
||||
changed): no ring-0 mark, no kernel step yet, the newest step did not end good, the box does not
|
||||
run its newest kernel, inside a step's 20 h window.
|
||||
kernel-status read only."""
|
||||
|
||||
def __init__(self, apply, plan):
|
||||
@@ -1626,7 +1907,8 @@ class Kernel:
|
||||
self.report["kernel"] = v
|
||||
return 0
|
||||
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
|
||||
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
|
||||
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel,
|
||||
"kernel-cleanup": self.cleanup}[mode]
|
||||
rc = fn()
|
||||
self.report["kernel"] = self.view()
|
||||
return rc
|
||||
@@ -1862,6 +2144,221 @@ class Kernel:
|
||||
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
|
||||
return 0
|
||||
|
||||
# ---------- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----------
|
||||
def kernel_images(self):
|
||||
"""kver -> [installed image package names] (dpkg status ii only)."""
|
||||
out = {}
|
||||
for name in sorted(self.a.installed()):
|
||||
m = KERNEL_IMAGE_RE.match(name)
|
||||
if m:
|
||||
out.setdefault(m.group(1), []).append(name)
|
||||
return out
|
||||
|
||||
def kver_newest(self, kvers):
|
||||
best = None
|
||||
for k in kvers:
|
||||
if best is None or self.a.dpkg_cmp(k[:-4], "gt", best[:-4]):
|
||||
best = k
|
||||
return best
|
||||
|
||||
def read_or_none(self, path):
|
||||
try:
|
||||
return self.r.read_file(path)
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
def cleanup_sources(self):
|
||||
"""Every place that names a kernel the lane still needs, read from the box ONCE: (kver, why) pairs. This is
|
||||
the keep set's only input, and K1's only input — K1 reads it again by itself (cleanup_protected)."""
|
||||
st = self.state()
|
||||
cfg = self.grub_cfg()
|
||||
dflt = self.default_kver(cfg)
|
||||
m = PINNED_KVER_RE.search(self.read_or_none(KERNEL_DEFAULT_CFG) or "")
|
||||
pinned = m.group(1) if m else None
|
||||
flag = self.flag() or ""
|
||||
m = ANY_KVER_RE.search(flag)
|
||||
m2 = ANY_KVER_RE.search(self.read_or_none(KERNEL_PIN_FILE) or "")
|
||||
src = [(self.running(), "running"),
|
||||
(st.get("from"), "the kernel step's `from` (the last kernel that booted well; a self-revert boots it)"),
|
||||
(st.get("to"), "the kernel step's `to`"),
|
||||
(st.get("default_before"), "the kernel step's `default_before`"),
|
||||
(dflt if KVER_RE.match(dflt) else None, "the default in grub.cfg"),
|
||||
(pinned, "the pinned default file"),
|
||||
(m.group(0) if m else None, "the one-shot flag"),
|
||||
(m2.group(0) if m2 else None, "an operator's boot pin")]
|
||||
return st, dflt, pinned, flag, [(k, why) for k, why in src if isinstance(k, str) and KVER_RE.match(k)]
|
||||
|
||||
def cleanup_keep(self, images, sources):
|
||||
"""The keep set: every kernel a source names, and the newest installed. (cleanup() adds the next lower
|
||||
installed kernel when no step record names the last good one.)"""
|
||||
keep = {}
|
||||
for k, why in sources:
|
||||
keep.setdefault(k, []).append(why)
|
||||
newest = self.kver_newest(images)
|
||||
if newest:
|
||||
keep.setdefault(newest, []).append("newest installed")
|
||||
return keep
|
||||
|
||||
def cleanup_protected(self):
|
||||
"""K1's own reading (`11` §5.11.1: "K1 is the rule's own check and must stay its own"): the kernels that must
|
||||
never reach the purge line, read from the box AGAIN — not taken from the keep set that built the removal set."""
|
||||
_, _, _, _, sources = self.cleanup_sources()
|
||||
return {k: why for k, why in sources}
|
||||
|
||||
def cleanup_simulate(self, names):
|
||||
"""apt's simulation of exactly `names`: (rc, removes, installs, last line)."""
|
||||
rc, out, err = self.a.x(APT_ENV + ["apt-get", "-s", "-q", "purge"] + names)
|
||||
removes = sorted({l.split()[1] for l in out.splitlines() if l.startswith(("Remv ", "Purg ")) and len(l.split()) > 1})
|
||||
installs = sorted({l.split()[1] for l in out.splitlines() if l.startswith(("Inst ", "Conf ")) and len(l.split()) > 1})
|
||||
return rc, removes, installs, ((err or out).strip().splitlines() or ["?"])[-1][:200]
|
||||
|
||||
def cleanup(self):
|
||||
a = self.a
|
||||
dry = self.plan.get("dry_run") is True
|
||||
res = {"dry_run": dry, "outcome": None, "not_run": [], "refusals": [], "removed": [], "leftovers": []}
|
||||
self.report["kernel_cleanup"] = res
|
||||
not_run, refuse = res["not_run"], res["refusals"]
|
||||
# --- may this box run the clean-up at all? (ring 0 only in this release; the mark is ROOT-owned) ---
|
||||
try:
|
||||
ring0 = a.load_trust().get("ring0_slow_lane") is True
|
||||
except Refused as e:
|
||||
ring0 = False
|
||||
not_run.append({"code": "no-ring0-mark", "reason": f"no usable trust record: {e.reason}"})
|
||||
if not ring0 and not not_run:
|
||||
not_run.append({"code": "no-ring0-mark", "reason": "this box has no root-owned ring-0 mark — the clean-up runs on "
|
||||
"ring 0 only in this release"})
|
||||
if a.apt_lock_held():
|
||||
raise Refused("R9", "another apt/dpkg holds the lock on the host")
|
||||
if not dry and not not_run:
|
||||
a.repair() # repair first (`11` §5.4.1): a purge a power cut interrupted is made whole, THEN the rule is computed
|
||||
# --- read the box ---
|
||||
images = self.kernel_images()
|
||||
running = self.running()
|
||||
newest = self.kver_newest(images)
|
||||
raw_state = self.read_or_none(KERNEL_STATE)
|
||||
st, dflt, pinned, flag, sources = self.cleanup_sources()
|
||||
phase = st.get("phase", "none") if raw_state is not None else "none"
|
||||
res.update(running=running or "unknown", newest=newest, phase=phase, default=dflt, pinned_default=pinned,
|
||||
installed={k: images[k] for k in sorted(images, key=lambda v: [int(x) for x in re.findall(r"\d+", v)])})
|
||||
# --- "when it may run" (decision 286): not run, nothing changed ---
|
||||
if raw_state is None or not st:
|
||||
not_run.append({"code": "no-step", "reason": "no kernel step yet (no state file) — nothing proves which kernel booted well"})
|
||||
elif phase in KERNEL_ENDED_NOT_GOOD:
|
||||
not_run.append({"code": "step-not-good", "reason": f"the newest kernel step ended {phase!r}, not good — every kernel stays"})
|
||||
elif phase != "good":
|
||||
refuse.append({"code": "P1", "reason": f"a kernel step is in phase {phase!r} — not ended (or not a phase this wrapper knows)"})
|
||||
if running and newest and running in images and running != newest:
|
||||
not_run.append({"code": "not-on-newest", "reason": f"the box runs {running}, not its newest installed kernel {newest} — every kernel stays"})
|
||||
times = [x for x in (_parse_iso(st.get(k)) for k in ("staged_at", "rebooted_at", "result_at")) if x is not None]
|
||||
if times and self.r.now() - max(times) < KERNEL_MIN_GAP:
|
||||
not_run.append({"code": "window", "reason": "a kernel step staged, booted or ended within the last 20 hours — never in its night"})
|
||||
# --- the refusal table ---
|
||||
if flag:
|
||||
refuse.append({"code": "P2", "reason": f"the one-shot flag names {flag!r}"})
|
||||
if pinned is None or pinned != dflt or pinned not in images:
|
||||
refuse.append({"code": "P3", "reason": f"the pinned default {pinned!r} and grub.cfg's default {dflt!r} do not name the same installed kernel"})
|
||||
if not running or running not in images:
|
||||
refuse.append({"code": "K1", "reason": f"the running kernel {running or 'unknown'!r} is not an installed kernel image package"})
|
||||
# --- the keep set and the removal set (exact package names) ---
|
||||
keep = self.cleanup_keep(images, sources)
|
||||
if running and not (st.get("to") == running and st.get("from") in images):
|
||||
lower = [k for k in images if a.dpkg_cmp(k[:-4], "lt", running[:-4])] if running in images else []
|
||||
lg = self.kver_newest(lower)
|
||||
if lg:
|
||||
keep.setdefault(lg, []).append("the next lower installed kernel (no step record names the last good one)")
|
||||
removal = sorted(n for k in images if k not in keep for n in images[k])
|
||||
res["kept"] = {k: keep[k] for k in sorted(keep) if k in images}
|
||||
res["removal"] = removal
|
||||
# K1 — its own reading of the box, against the removal set
|
||||
by_name = {n: k for k, names in images.items() for n in names}
|
||||
protected = self.cleanup_protected()
|
||||
for n in removal:
|
||||
if n not in by_name:
|
||||
refuse.append({"code": "N1", "reason": f"{n} is not an installed kernel image package"})
|
||||
elif by_name[n] in protected or by_name[n] == newest:
|
||||
refuse.append({"code": "K1", "reason": f"{n} is a keep-kernel ({protected.get(by_name[n], 'newest installed')})"})
|
||||
# S1 / S2 — apt's simulation of exactly that set
|
||||
if removal and not [x for x in refuse if x["code"] in ("K1", "N1")]:
|
||||
rc, removes, installs, tail = self.cleanup_simulate(removal)
|
||||
res["simulation"] = {"rc": rc, "removes": removes, "installs": installs}
|
||||
if rc != 0:
|
||||
refuse.append({"code": "S1", "reason": "the simulation failed: " + tail})
|
||||
if installs:
|
||||
refuse.append({"code": "S1", "reason": f"the simulation would install or configure {', '.join(installs[:6])}"})
|
||||
extra = [p for p in removes if p not in removal]
|
||||
if extra:
|
||||
refuse.append({"code": "S2", "reason": f"the simulation would also remove {', '.join(extra[:8])} — not in the removal set"})
|
||||
missing = [p for p in removal if p not in removes]
|
||||
if missing and rc == 0:
|
||||
refuse.append({"code": "S1", "reason": f"the simulation does not remove {', '.join(missing[:6])}"})
|
||||
self.log(f"os-apply: KERNEL CLEANUP {'DRY RUN ' if dry else ''}running={running or 'unknown'} newest={newest} phase={phase} "
|
||||
f"installed={len(images)} keep={len(res['kept'])} removal={len(removal)} not_run={[x['code'] for x in not_run]} "
|
||||
f"refusals={[x['code'] for x in refuse]}")
|
||||
# --- the verdict ---
|
||||
if dry:
|
||||
res["outcome"] = "dry_run"
|
||||
res["would"] = "not_run" if not_run else "refuse" if refuse else "remove" if removal else "nothing"
|
||||
return 0
|
||||
if not_run:
|
||||
res["outcome"] = "not_run"
|
||||
return 0
|
||||
if refuse:
|
||||
raise Refused(refuse[0]["code"], "; ".join(f"{x['code']} {x['reason']}" for x in refuse)[:600])
|
||||
if not removal:
|
||||
res["outcome"] = "nothing"
|
||||
self.log("os-apply: KERNEL CLEANUP nothing to remove")
|
||||
return 0
|
||||
# --- the purge: ONE apt-get purge of the exact names; then prove the boot setup did not move ---
|
||||
gens_before = {p: (self.r.lexists(p), self.read_or_none(p)) for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES)}
|
||||
pin_before = self.read_or_none(KERNEL_DEFAULT_CFG)
|
||||
free_before = a.free_bytes()
|
||||
t0 = time.time()
|
||||
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q", "purge"] + removal)
|
||||
res["seconds"] = round(time.time() - t0, 1)
|
||||
if rc != 0:
|
||||
_, aud, _ = a.x(["dpkg", "--audit"])
|
||||
res["outcome"] = "failed"
|
||||
self.report["failed"] = {"rc": rc, "step": "purge", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
|
||||
"tail": (out + err).strip().splitlines()[-3:]}
|
||||
self.log(f"os-apply: FAILED rc={rc} step=purge (the next run repairs dpkg first, then computes the rule again)")
|
||||
return 3
|
||||
after = self.kernel_images()
|
||||
after_names = {n for names in after.values() for n in names}
|
||||
res["removed"] = [n for n in removal if n not in after_names]
|
||||
free_after = a.free_bytes()
|
||||
res["bytes_gained"] = free_after - free_before if free_before >= 0 and free_after >= 0 else None
|
||||
problems = []
|
||||
if len(res["removed"]) != len(removal):
|
||||
problems.append(f"still installed after the purge: {', '.join(n for n in removal if n in after_names)}")
|
||||
gone = [k for k in res["kept"] if k not in after]
|
||||
if gone:
|
||||
problems.append(f"a KEPT kernel is no longer installed: {', '.join(gone)}")
|
||||
got = self.default_kver(self.grub_cfg())
|
||||
if got != dflt:
|
||||
problems.append(f"the default in grub.cfg moved from {dflt} to {got}")
|
||||
if self.read_or_none(KERNEL_DEFAULT_CFG) != pin_before:
|
||||
problems.append("the pinned default file changed")
|
||||
if self.read_or_none(KERNEL_STATE) != raw_state:
|
||||
problems.append("the kernel step's state file changed")
|
||||
for p, (was, body) in gens_before.items():
|
||||
if self.r.lexists(p) != was or self.read_or_none(p) != body:
|
||||
problems.append(f"{p} changed")
|
||||
for k in sorted(set(by_name[n] for n in res["removed"])):
|
||||
if k in after:
|
||||
continue # another image package of the same kernel is still installed
|
||||
res["leftovers"] += [p.format(v=k) for p in KERNEL_LEFTOVER_PATHS if self.r.lexists(p.format(v=k))]
|
||||
if problems:
|
||||
# reported LOUDLY, never repaired by hand here (`11` §5.11.1): the operator looks at the box
|
||||
res["outcome"] = "failed"
|
||||
self.report["failed"] = {"rc": 3, "step": "verify", "reason": "; ".join(problems)[:600]}
|
||||
self.log("os-apply: FAILED rc=3 step=verify after the kernel purge — " + "; ".join(problems))
|
||||
return 3
|
||||
res["outcome"] = "removed"
|
||||
self.log(f"os-apply: KERNEL CLEANUP DONE removed={len(res['removed'])} ({', '.join(res['removed'])}) "
|
||||
f"kept={', '.join(res['kept'])} gained={res['bytes_gained']} B seconds={res['seconds']} "
|
||||
f"leftovers={len(res['leftovers'])}; default {dflt} (proved from grub.cfg)")
|
||||
return 0
|
||||
|
||||
|
||||
class Bundle:
|
||||
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
||||
@@ -1873,6 +2370,7 @@ class Bundle:
|
||||
self.a, self.r = apply, apply.r
|
||||
self.report = apply.report
|
||||
self.sshd_dropin_not_in_force = False
|
||||
self.sshd_step_undone = None # X4 (R-1257): why the SSH step alone was put back ("" = not), the rest stays
|
||||
|
||||
# ---------- reading and checking ----------
|
||||
def load_file(self, path):
|
||||
@@ -1982,14 +2480,40 @@ class Bundle:
|
||||
return b"", None, None # a symlink or a directory: always replaced by the real file
|
||||
return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid
|
||||
|
||||
def install_mode(self):
|
||||
"""The box's ROOT-OWNED install record's mode ("appliance", "byo", …), or None when it proves nothing (absent,
|
||||
agent-writable, unreadable) — the same proof check_appliance asks for the host layer (R12)."""
|
||||
try:
|
||||
st = self.r.stat(INSTALL_STATE)
|
||||
if st.st_uid != 0 or (st.st_mode & 0o022):
|
||||
return None
|
||||
mode = json.loads(self.r.read_file(INSTALL_STATE)).get("mode")
|
||||
except (OSError, ValueError, AttributeError):
|
||||
return None
|
||||
return mode if isinstance(mode, str) and mode else None
|
||||
|
||||
def plan_writes(self, files):
|
||||
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt)."""
|
||||
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt; an
|
||||
appliance-only file on a box that is not one) / remove (an appliance-only file WE wrote, on a BYO box)."""
|
||||
oob = self.r.isdir(OOB_DIR)
|
||||
install_mode = self.install_mode()
|
||||
plan = []
|
||||
for dest, content, mode, check, policy in files:
|
||||
if policy == "oob" and not oob:
|
||||
plan.append((dest, content, mode, "skipped", None, None))
|
||||
continue
|
||||
if policy == "appliance" and install_mode != "appliance":
|
||||
old, old_mode, _ = self.live(dest)
|
||||
ours = old is not None and old.startswith(SSHD_DROPIN_HEADER)
|
||||
if install_mode == "byo" and ours:
|
||||
self.r.log(f"os-apply: BUNDLE {dest}: a BYO host keeps its own settings (D43) — removing the copy we wrote")
|
||||
plan.append((dest, content, mode, "remove", old, old_mode))
|
||||
else:
|
||||
why = "a BYO host keeps its own settings (D43)" if install_mode == "byo" else \
|
||||
f"the install record proves no appliance (mode={install_mode!r}) — left as it is"
|
||||
self.r.log(f"os-apply: BUNDLE {dest} skipped: {why}")
|
||||
plan.append((dest, content, mode, "skipped", None, None))
|
||||
continue
|
||||
old, old_mode, old_uid = self.live(dest)
|
||||
if policy == "if-absent" and old is not None:
|
||||
plan.append((dest, content, mode, "kept", old, old_mode))
|
||||
@@ -2027,11 +2551,47 @@ class Bundle:
|
||||
ver, files = self.parse(data, sha)
|
||||
return self.install(ver, sha, files, "installer", False)
|
||||
|
||||
def policy_pass(self):
|
||||
"""R-1285 (D43 = A): the appliance-only files' policy, applied by THIS wrapper. A bundle that brings a new wrapper
|
||||
is planned by the OLD one, which did not know the policy — so on a BYO host upgraded from 0.164.0 our SSH drop-in
|
||||
stayed until the next bundle. The agent runs this pass right after every bundle install (the wrapper on disk is
|
||||
then the new one). It only REMOVES a file WE wrote (our header) from a box whose root-owned install record says
|
||||
byo, keeps the previous copy beside the bundle's, and reloads sshd; it writes nothing, so it needs no signature.
|
||||
A failing `sshd -t` puts the file back (R-1257 X4's rule). Report key `bundle_policy`."""
|
||||
log = self.r.log
|
||||
install_mode = self.install_mode()
|
||||
removed = []
|
||||
if install_mode == "byo":
|
||||
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
|
||||
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-policy")
|
||||
for dest, _src, _mode, _check, policy in BUNDLE_FILES:
|
||||
if policy != "appliance":
|
||||
continue
|
||||
old, old_mode, _ = self.live(dest)
|
||||
if old is None or not old.startswith(SSHD_DROPIN_HEADER):
|
||||
continue
|
||||
self.r.put_file(prev + dest, old, 0o600)
|
||||
self.r.remove(dest)
|
||||
removed.append((dest, old, old_mode))
|
||||
log(f"os-apply: BUNDLE POLICY REMOVED {dest} (ours, on a BYO host — D43, R-1285)")
|
||||
if any(d == SSHD_DROPIN for d, _, _ in removed):
|
||||
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
|
||||
if rc != 0:
|
||||
for d, old, old_mode in removed:
|
||||
self.r.put_file(d, old, old_mode)
|
||||
log(f"os-apply: BUNDLE POLICY WARNING sshd -t failed without our drop-in (rc={rc}) — put back")
|
||||
removed = []
|
||||
else:
|
||||
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
|
||||
self.report["bundle_policy"] = {"install_mode": install_mode, "removed": [d for d, _, _ in removed]}
|
||||
log(f"os-apply: BUNDLE POLICY DONE install_mode={install_mode} removed={len(removed)}")
|
||||
return 0
|
||||
|
||||
# ---------- install, self-check, undo ----------
|
||||
def install(self, ver, sha, files, authority, bootstrap):
|
||||
log = self.r.log
|
||||
plan = self.plan_writes(files)
|
||||
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped")}
|
||||
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped", "remove")}
|
||||
log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} "
|
||||
f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}")
|
||||
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
|
||||
@@ -2039,10 +2599,17 @@ class Bundle:
|
||||
done = [] # (dest, old bytes or None, old mode)
|
||||
rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev,
|
||||
"written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"],
|
||||
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"]}
|
||||
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"],
|
||||
"removed": [p[0] for p in plan if p[3] == "remove"]}
|
||||
self.report["bundle"] = rep
|
||||
try:
|
||||
for dest, content, mode, action, old, old_mode in plan:
|
||||
if action == "remove": # D43 = A: our copy off a BYO host (the previous copy kept, the undo restores it)
|
||||
self.r.put_file(prev + dest, old, 0o600)
|
||||
done.append((dest, old, old_mode))
|
||||
self.r.remove(dest)
|
||||
log(f"os-apply: BUNDLE REMOVED {dest} (ours, on a BYO host — D43)")
|
||||
continue
|
||||
if action != "write":
|
||||
continue
|
||||
if old is not None:
|
||||
@@ -2062,22 +2629,56 @@ class Bundle:
|
||||
rep["signers_created"] = False
|
||||
if self.sshd_dropin_not_in_force:
|
||||
rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
|
||||
if self.sshd_step_undone:
|
||||
rep["sshd_step_undone"] = self.sshd_step_undone[:300] # X4: only the SSH step was put back
|
||||
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
|
||||
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
|
||||
rep["signers_created"] = True
|
||||
log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}")
|
||||
record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority,
|
||||
"installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())),
|
||||
"files": {p[0]: (sha256_hex(p[1]) if p[3] in ("write", "same") else
|
||||
(sha256_hex(p[4]) if p[3] == "kept" else "skipped")) for p in plan}}
|
||||
"files": {p[0]: self.record_entry(p) for p in plan}}
|
||||
self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||
self.prune()
|
||||
log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} "
|
||||
f"self-check=ok signers-created={rep['signers_created']}")
|
||||
return 0
|
||||
|
||||
def record_entry(self, p):
|
||||
"""The record's word for one plan entry: the sha it installed, the sha it kept, or what it did instead."""
|
||||
dest, content, _, action, old, _ = p
|
||||
if dest == SSHD_DROPIN and self.sshd_step_undone:
|
||||
return sha256_hex(old) if old is not None else "absent" # X4: the SSH step was put back
|
||||
if action in ("write", "same"):
|
||||
return sha256_hex(content)
|
||||
if action == "kept":
|
||||
return sha256_hex(old)
|
||||
return "removed" if action == "remove" else "skipped"
|
||||
|
||||
def lan_addresses(self):
|
||||
"""X3 (R-1257): the host's own global IPv4 addresses — the ones a LAN client reaches sshd on."""
|
||||
rc, out, _ = self.r.host(["ip", "-4", "-o", "addr", "show", "scope", "global"], 30)
|
||||
if rc != 0:
|
||||
return []
|
||||
return sorted(set(re.findall(r"\binet (\d+\.\d+\.\d+\.\d+)/", out)))
|
||||
|
||||
def undo_sshd_step(self, plan, why):
|
||||
"""X4 (R-1257): `sshd -t` refused the host's whole config with our step in it — put back ONLY that file (the
|
||||
previous copy, or none) and keep the rest of the bundle. sshd was not reloaded, so it never ran the refused state."""
|
||||
for dest, _, _, action, old, old_mode in plan:
|
||||
if dest != SSHD_DROPIN or action not in ("write", "remove"):
|
||||
continue
|
||||
if old is None:
|
||||
self.r.remove(dest)
|
||||
else:
|
||||
self.r.put_file(dest, old, old_mode if old_mode is not None else 0o644)
|
||||
self.sshd_step_undone = why
|
||||
self.r.log(f"os-apply: BUNDLE WARNING sshd -t refused the host's config with the SSH step — ONLY that step is put "
|
||||
f"back, the rest of the bundle stays: {why[:200]} (R-1257)")
|
||||
|
||||
def after_install(self, plan):
|
||||
written = {p[0] for p in plan if p[3] == "write"}
|
||||
removed = {p[0] for p in plan if p[3] == "remove"}
|
||||
dests = {p[0] for p in plan if p[3] != "skipped"}
|
||||
if any(d.startswith("/etc/systemd/system/") for d in written):
|
||||
self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload")
|
||||
@@ -2098,16 +2699,29 @@ class Bundle:
|
||||
# felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
|
||||
# sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
|
||||
# — that is said loudly in the log and the report, never silently recorded as done (A2).
|
||||
if SSHD_DROPIN in written:
|
||||
self.must(["/usr/sbin/sshd", "-t"], "sshd -t (the host's whole sshd config)")
|
||||
# R-1257 X4: a failing `sshd -t` puts back ONLY the SSH step (undo_sshd_step); it used to refuse and undo the
|
||||
# whole bundle. X3: the in-force check asks sshd about each of the host's own LAN addresses — a probe from a
|
||||
# documentation address (192.0.2.1) read a `Match Address <LAN>` block that re-allows passwords as in force.
|
||||
if SSHD_DROPIN in written or SSHD_DROPIN in removed:
|
||||
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
|
||||
if rc != 0:
|
||||
self.undo_sshd_step(plan, f"sshd -t failed (rc={rc}): {(out + err).strip()[-200:]}")
|
||||
return
|
||||
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
|
||||
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", "user=root,host=felhom-check,addr=192.0.2.1"], 60)
|
||||
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
|
||||
if rc != 0 or eff != ["without-password"]:
|
||||
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force: sshd -T says "
|
||||
f"permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier drop-in "
|
||||
f"or a Match block (R-982)")
|
||||
self.sshd_dropin_not_in_force = True
|
||||
if SSHD_DROPIN in written:
|
||||
addrs = self.lan_addresses()
|
||||
if not addrs:
|
||||
self.r.log("os-apply: BUNDLE WARNING no global IPv4 address read for the sshd in-force check — "
|
||||
"probing 192.0.2.1 only (R-1257)")
|
||||
addrs = ["192.0.2.1"]
|
||||
for addr in addrs:
|
||||
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", f"user=root,host=felhom-check,addr={addr}"], 60)
|
||||
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
|
||||
if rc != 0 or eff != ["without-password"]:
|
||||
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force for {addr}: sshd -T "
|
||||
f"says permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier "
|
||||
f"drop-in or a Match block (R-982, R-1257)")
|
||||
self.sshd_dropin_not_in_force = True
|
||||
|
||||
def must(self, argv, what):
|
||||
rc, out, err = self.r.host(argv, 120)
|
||||
|
||||
@@ -45,8 +45,10 @@ class St:
|
||||
class Box:
|
||||
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
|
||||
|
||||
def __init__(self, bundle_bytes, signed, oob=False, signers=True):
|
||||
def __init__(self, bundle_bytes, signed, oob=False, signers=True, install_mode="appliance"):
|
||||
self.files, self.modes, self.uids = {}, {}, {}
|
||||
if install_mode is not None: # the installer's ROOT-OWNED record (D43 = A reads it for the sshd drop-in)
|
||||
self.put(osapply.INSTALL_STATE, json.dumps({"completed": [], "mode": install_mode}).encode(), 0o644)
|
||||
self.dirs = {osapply.OOB_DIR} if oob else set()
|
||||
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
|
||||
if signers:
|
||||
@@ -513,6 +515,157 @@ class RootKeyOnly(unittest.TestCase):
|
||||
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
|
||||
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
|
||||
|
||||
class ByoKeepsItsSSH(unittest.TestCase):
|
||||
"""D43 = A (operator 2026-10-24, R-1257): the root-key-only drop-in goes ONLY to a box whose root-owned install record
|
||||
says appliance; on a BYO host nothing is written and a copy WE wrote is removed; a box with no record is left as it
|
||||
is. X3: the in-force check asks sshd about each of the host's own LAN addresses. X4: a failing `sshd -t` puts back
|
||||
only the SSH step. RED-PROOF (2026-10-24, felhom-agent f5c3d77 + these tests): with the drop-in's policy back to
|
||||
`replace`, test_a_byo_host_gets_no_dropin and test_our_copy_comes_off_a_byo_host fail; with the probe back to
|
||||
192.0.2.1 only, test_a_match_block_for_the_lan_is_not_in_force fails; with `sshd -t` back to `must`,
|
||||
test_a_failing_sshd_t_undoes_only_the_ssh_step fails (rc 3, the whole bundle undone)."""
|
||||
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||
OURS = (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes()
|
||||
|
||||
def test_a_byo_host_gets_no_dropin(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha), install_mode="byo")
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertNotIn(self.DEST, box.files)
|
||||
self.assertNotIn(["/usr/sbin/sshd", "-t"], box.calls)
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
self.assertIn(self.DEST, rep["bundle"]["skipped"])
|
||||
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
|
||||
self.assertEqual(rec["files"][self.DEST], "skipped")
|
||||
|
||||
def test_our_copy_comes_off_a_byo_host(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha), install_mode="byo")
|
||||
box.put(self.DEST, self.OURS, 0o644) # an earlier bundle wrote it (before D43)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertNotIn(self.DEST, box.files)
|
||||
self.assertEqual(rep["bundle"]["removed"], [self.DEST])
|
||||
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files),
|
||||
"the previous copy must be kept for the undo")
|
||||
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], "removed")
|
||||
|
||||
def test_the_owners_own_file_stays_on_a_byo_host(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha), install_mode="byo")
|
||||
box.put(self.DEST, b"# my own\nPermitRootLogin yes\n", 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files[self.DEST], b"# my own\nPermitRootLogin yes\n")
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
|
||||
def test_a_box_with_no_record_is_left_as_it_is(self):
|
||||
for setup in ("none", "agent-writable"):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha), install_mode=None if setup == "none" else "byo")
|
||||
if setup == "agent-writable":
|
||||
box.modes[osapply.INSTALL_STATE] = 0o666 # proves nothing
|
||||
box.put(self.DEST, self.OURS, 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files.get(self.DEST), self.OURS, setup)
|
||||
self.assertIn(self.DEST, rep["bundle"]["skipped"], setup)
|
||||
|
||||
def _fake(self, box, addrs=("192.168.0.50",), match=None, sshd_t_rc=0):
|
||||
orig = box.host
|
||||
def host(argv, timeout=600, stdin=None):
|
||||
if argv[:3] == ["ip", "-4", "-o"]:
|
||||
box.calls.append(argv)
|
||||
return 0, "".join(f"2: vmbr0 inet {a}/24 brd x scope global vmbr0\\ valid_lft forever\n" for a in addrs), ""
|
||||
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
|
||||
box.calls.append(argv)
|
||||
val = "yes" if match and f"addr={match}" in argv[-1] else "without-password"
|
||||
return 0, f"port 22\npermitrootlogin {val}\n", ""
|
||||
if argv == ["/usr/sbin/sshd", "-t"]:
|
||||
box.calls.append(argv)
|
||||
return (sshd_t_rc, "", "/etc/ssh/sshd_config line 9: Bad configuration option") if sshd_t_rc else (0, "", "")
|
||||
return orig(argv, timeout, stdin)
|
||||
box.host = host
|
||||
|
||||
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
|
||||
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
|
||||
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
|
||||
|
||||
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None, install_mode="byo")
|
||||
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertNotIn(self.DEST, box.files)
|
||||
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
|
||||
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
|
||||
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||
|
||||
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
|
||||
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None, install_mode=mode)
|
||||
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||
body = content if content is not None else self.OURS
|
||||
box.put(self.DEST, body, 0o644)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files.get(self.DEST), body, mode)
|
||||
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
|
||||
|
||||
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, None, install_mode="byo")
|
||||
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||
box.put(self.DEST, self.OURS, 0o644)
|
||||
self._fake(box, sshd_t_rc=1)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(box.files.get(self.DEST), self.OURS)
|
||||
self.assertEqual(rep["bundle_policy"]["removed"], [])
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||
|
||||
def test_a_match_block_for_the_lan_is_not_in_force(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"), match="192.168.0.50")
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
probed = [c[-1] for c in box.calls if c[:2] == ["/usr/sbin/sshd", "-T"]]
|
||||
self.assertTrue(any("addr=192.168.0.50" in p for p in probed) and any("addr=10.77.0.4" in p for p in probed), probed)
|
||||
self.assertFalse(any("addr=192.0.2.1" in p for p in probed), probed)
|
||||
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
|
||||
self.assertTrue(any("NOT in force for 192.168.0.50" in l for l in box.logs), box.logs)
|
||||
|
||||
def test_the_lan_check_passes_when_every_address_is_key_only(self):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"))
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
|
||||
|
||||
def test_a_failing_sshd_t_undoes_only_the_ssh_step(self):
|
||||
for before in (None, b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982 old\nPermitRootLogin prohibit-password\n"):
|
||||
data, sha = real_bundle()
|
||||
box = Box(data, signed_job(sha))
|
||||
if before is not None:
|
||||
box.put(self.DEST, before, 0o644)
|
||||
self._fake(box, sshd_t_rc=255)
|
||||
rc, rep = run(box)
|
||||
self.assertEqual(rc, 0, rep) # the bundle stays
|
||||
self.assertEqual(box.files.get(self.DEST), before, "only the SSH step is put back")
|
||||
self.assertIn("/usr/local/sbin/felhom-os-apply", box.files, "the rest of the bundle stays")
|
||||
self.assertIn("sshd -t failed", rep["bundle"].get("sshd_step_undone", ""), rep)
|
||||
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, "sshd never ran the refused state")
|
||||
want = "absent" if before is None else hashlib.sha256(before).hexdigest()
|
||||
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], want)
|
||||
|
||||
|
||||
class TrustBootstrap(unittest.TestCase):
|
||||
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
|
||||
data, sha = real_bundle()
|
||||
|
||||
@@ -833,6 +833,14 @@ class DockerLane(unittest.TestCase):
|
||||
def test_expired_signed_job_is_refused(self):
|
||||
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
|
||||
|
||||
def test_a_window_longer_than_24h_is_refused(self):
|
||||
# R-1280. RED-PROOF: drop the MAX_WINDOW_S check → the job (current, signed, 24 h + 1 s) runs.
|
||||
self.refused(docker_fake(signed=signed_job(issued="2026-10-03T11:59:59Z", expires="2026-10-04T12:00:00Z")), "R3")
|
||||
|
||||
def test_a_window_of_24h_is_accepted(self):
|
||||
rc, rep = run(docker_fake(signed=signed_job(issued="2026-10-03T12:30:00Z", expires="2026-10-04T12:30:00Z")))
|
||||
self.assertNotEqual((rep.get("refused") or {}).get("code"), "R3", rep)
|
||||
|
||||
def test_replayed_signed_job_is_refused(self):
|
||||
f = docker_fake(signed=signed_job())
|
||||
f.nonces = {"n1": f.clock + 600}
|
||||
@@ -1486,6 +1494,24 @@ class PVELane(unittest.TestCase):
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
||||
|
||||
# 2026-11-05..07, demo-felhom (EFI): proxmox-grub 2.12-9+pmx2 -> pmx4 depends on grub-efi-amd64 of the same version;
|
||||
# HOST_SLOW_RE's `grub` is anchored at the start, so proxmox-grub fell into the pve plan, the simulation pulled
|
||||
# grub-efi-amd64 and the whole pve step was refused R6 every night. It is a boot-chain package: left out.
|
||||
# RED-PROOF: drop proxmox-grub from HOST_SLOW_RE -> it is in the plan -> this test fails.
|
||||
def test_ring0_pending_pve_leaves_proxmox_grub_out(self):
|
||||
f = pve_fake(ring0=True)
|
||||
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||
f.installed.update({"proxmox-grub": "2.12-9+pmx2"})
|
||||
f.pending_sim = [
|
||||
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||
]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||
self.assertEqual(f.installed["proxmox-grub"], "2.12-9+pmx2")
|
||||
|
||||
def test_select_pending_pve_needs_the_pve_layer(self):
|
||||
f = Fake()
|
||||
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
||||
@@ -2005,5 +2031,847 @@ class KernelLane(unittest.TestCase):
|
||||
self.assertIn("one-shot", h["kernel_next_boot_source"])
|
||||
|
||||
|
||||
|
||||
# ---------- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----------
|
||||
K = ["7.0.2-6-pve", "7.0.14-20-pve", "7.0.14-22-pve", "7.0.14-26-pve", "7.0.14-28-pve"] # five installed kernels
|
||||
C_FROM, C_RUN = "7.0.14-26-pve", "7.0.14-28-pve"
|
||||
GB = 1024 ** 3
|
||||
|
||||
|
||||
def img(k):
|
||||
return f"proxmox-kernel-{k}-signed"
|
||||
|
||||
|
||||
class CFake(KFake):
|
||||
"""A ring-0 appliance after a good kernel step 26 -> 28: five kernels installed, the box runs its newest, the
|
||||
default (grub.cfg and the pinned file) is the running kernel, the flag is empty. `apt-get purge` is emulated like
|
||||
the real one: the image leaves dpkg and /boot, and the kernel's postrm runs update-grub."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.running = C_RUN
|
||||
self.boot = set(K)
|
||||
self.modules = set(K) # /lib/modules/<kver>
|
||||
for n in [x for x in self.installed if osapply.KERNEL_IMAGE_RE.match(x)]:
|
||||
del self.installed[n]
|
||||
for k in K:
|
||||
self.installed[img(k)] = k[:-4]
|
||||
self.installed["proxmox-kernel-7.0"] = C_RUN[:-4]
|
||||
self.env = {"felhom_next": ""}
|
||||
self.tree[osapply.KERNEL_DEFAULT_CFG] = (f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-{C_RUN}-advanced-{U}"\n')
|
||||
self.state = {"phase": "good", "step_id": "ring0-x", "from": C_FROM, "to": C_RUN, "default_before": C_FROM,
|
||||
"vmid": 9201, "staged_at": "2026-10-01T02:50:00Z", "rebooted_at": "2026-10-01T02:51:00Z",
|
||||
"result_at": "2026-10-01T03:10:00Z", "self_revert_used": False}
|
||||
self.write_state()
|
||||
self.cfg = self.render()
|
||||
self.purges, self.purge_rc, self.purge_moves_default, self.purge_extra_sim = [], 0, False, []
|
||||
self.plan = {"release_id": "kernel", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "kernel-cleanup",
|
||||
"packages": []}
|
||||
|
||||
def write_state(self):
|
||||
if self.state is None:
|
||||
self.tree.pop(osapply.KERNEL_STATE, None)
|
||||
else:
|
||||
self.tree[osapply.KERNEL_STATE] = json.dumps(self.state)
|
||||
|
||||
def lexists(self, p):
|
||||
m = re.match(r"^/lib/modules/(.+)$", p)
|
||||
if m:
|
||||
return m.group(1) in self.modules
|
||||
return super().lexists(p)
|
||||
|
||||
def changing(self):
|
||||
"""Every command that changes the box (the purge, a repair, update-grub, a flag write, a file write)."""
|
||||
out = []
|
||||
for c in self.calls:
|
||||
a = [x for x in c[-1] if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||
if (a[0] == "apt-get" and "-s" not in a and "--print-uris" not in a) or a[0] in ("update-grub", "dpkg-reconfigure") \
|
||||
or (a[0] == "dpkg" and a[1] == "--configure") or (a[0] == "grub-editenv" and a[2] != "list") \
|
||||
or (a[0] == "systemctl" and a[1] == "reboot"):
|
||||
out.append(a)
|
||||
return out
|
||||
|
||||
def emulate(self, argv):
|
||||
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||
if a[0] == "apt-get" and "purge" in a and "-s" not in a:
|
||||
names = [x for x in a[a.index("purge") + 1:] if not x.startswith("-")]
|
||||
self.purges.append(names)
|
||||
if self.purge_rc:
|
||||
self.dpkg_journal = ["0001"] # a purge cut in the middle leaves dpkg's update journal
|
||||
return self.purge_rc, "", "E: Sub-process /usr/bin/dpkg returned an error code (1)"
|
||||
for n in names:
|
||||
self.installed.pop(n, None)
|
||||
m = osapply.KERNEL_IMAGE_RE.match(n)
|
||||
if m:
|
||||
self.boot.discard(m.group(1))
|
||||
self.free += GB
|
||||
if self.purge_moves_default:
|
||||
self.tree.pop(osapply.KERNEL_DEFAULT_CFG, None)
|
||||
self.cfg = self.render() # the kernel's postrm runs update-grub (zz-update-grub)
|
||||
return 0, "Removing proxmox-kernel ...\n", ""
|
||||
return super().emulate(argv)
|
||||
|
||||
def sim(self, a):
|
||||
if "purge" in a:
|
||||
names = [x for x in a[a.index("purge") + 1:] if not x.startswith("-")]
|
||||
if getattr(self, "purge_sim_rc", 0):
|
||||
return self.purge_sim_rc, "", "E: Unable to locate package"
|
||||
out = "".join(f"Purg {n} [{self.installed[n]}]\n" for n in names if n in self.installed)
|
||||
return 0, out + "".join(l + "\n" for l in self.purge_extra_sim), ""
|
||||
return super().sim(a)
|
||||
|
||||
|
||||
def cfake(**kw):
|
||||
f = CFake()
|
||||
for k, v in kw.items():
|
||||
setattr(f, k, v)
|
||||
return f
|
||||
|
||||
|
||||
def cstate(f, **kw):
|
||||
f.state.update(kw)
|
||||
f.write_state()
|
||||
return f
|
||||
|
||||
|
||||
class KernelCleanup(unittest.TestCase):
|
||||
"""§5 of the release-2a brief: one test per acceptance scenario, each on the stored effect (the exact apt line or
|
||||
the absence of any changing command, the report, the exit code). Red-proofs (the four mutations the brief names):
|
||||
felhom.eu audits/release-2a-2026-11-14/A/redproof.txt."""
|
||||
|
||||
REMOVE = [img(k) for k in ("7.0.14-20-pve", "7.0.14-22-pve", "7.0.2-6-pve")] # sorted by name
|
||||
|
||||
def nothing_changed(self, f):
|
||||
self.assertEqual(f.purges, [], "no purge")
|
||||
self.assertEqual(f.changing(), [], "no command that changes the box")
|
||||
self.assertEqual(sorted(f.boot), sorted(K), "every kernel is still there")
|
||||
|
||||
def refused(self, f, code):
|
||||
before = (f.tree.get(osapply.KERNEL_STATE), f.tree.get(osapply.KERNEL_DEFAULT_CFG), f.cfg, dict(f.env or {}))
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 2, rep)
|
||||
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||
self.nothing_changed(f)
|
||||
self.assertEqual((f.tree.get(osapply.KERNEL_STATE), f.tree.get(osapply.KERNEL_DEFAULT_CFG), f.cfg, dict(f.env or {})), before)
|
||||
return rep
|
||||
|
||||
def not_run(self, f, code):
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
c = rep["kernel_cleanup"]
|
||||
self.assertEqual(c["outcome"], "not_run", rep)
|
||||
self.assertIn(code, [x["code"] for x in c["not_run"]], rep)
|
||||
self.assertIsNone(rep.get("refused"))
|
||||
self.nothing_changed(f)
|
||||
return rep
|
||||
|
||||
# --- scenario 1: the normal night ---
|
||||
def test_s1_the_normal_night_purges_exactly_the_old_kernels_in_one_command(self):
|
||||
f = cfake()
|
||||
state0, pin0 = f.tree[osapply.KERNEL_STATE], f.tree[osapply.KERNEL_DEFAULT_CFG]
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.purges, [self.REMOVE], "ONE apt-get purge of the exact names")
|
||||
purge = [c[-1] for c in f.calls if "purge" in c[-1] and "-s" not in c[-1]]
|
||||
self.assertEqual(len(purge), 1)
|
||||
self.assertEqual(purge[0][purge[0].index("apt-get"):], ["apt-get", "-y", "-q", "purge"] + self.REMOVE)
|
||||
self.assertNotIn("autoremove", " ".join(" ".join(c[-1]) for c in f.calls), "never apt's own clean-up")
|
||||
c = rep["kernel_cleanup"]
|
||||
self.assertEqual(c["outcome"], "removed")
|
||||
self.assertEqual(c["removed"], self.REMOVE)
|
||||
self.assertEqual(sorted(c["kept"]), sorted([C_FROM, C_RUN]))
|
||||
self.assertEqual(c["bytes_gained"], 3 * GB)
|
||||
self.assertEqual(sorted(f.boot), sorted([C_FROM, C_RUN]))
|
||||
# the default in grub.cfg, the pinned default file and the step's state file are unchanged
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), C_RUN)
|
||||
self.assertEqual(f.tree[osapply.KERNEL_DEFAULT_CFG], pin0)
|
||||
self.assertEqual(f.tree[osapply.KERNEL_STATE], state0)
|
||||
self.assertEqual(f.env, {"felhom_next": ""})
|
||||
self.assertEqual(f.reboots, [], "the clean-up never reboots")
|
||||
self.assertEqual(f.update_grubs, 0, "the wrapper itself runs no update-grub (the kernel's postrm did)")
|
||||
# leftovers are reported, never deleted
|
||||
self.assertEqual(c["leftovers"], [f"/lib/modules/{k}" for k in ("7.0.14-20-pve", "7.0.14-22-pve", "7.0.2-6-pve")])
|
||||
self.assertEqual(f.modules, set(K), "a left-over folder is reported, not deleted")
|
||||
|
||||
# --- scenario 2: the second night ---
|
||||
def test_s2_the_second_night_removes_nothing(self):
|
||||
f = cfake()
|
||||
self.assertEqual(run(f)[0], 0)
|
||||
f.calls.clear()
|
||||
f.purges.clear()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["kernel_cleanup"]["outcome"], "nothing")
|
||||
self.assertEqual(rep["kernel_cleanup"]["removal"], [])
|
||||
self.assertEqual(f.purges, [])
|
||||
self.assertEqual(f.changing(), [], "no apt call that changes anything")
|
||||
self.assertFalse([c for c in f.calls if "purge" in c[-1]], "not even a simulation: there is nothing to simulate")
|
||||
|
||||
# --- scenario 3: a step is not over (P1) ---
|
||||
def test_s3_a_step_that_is_not_over_refuses(self):
|
||||
for phase in ("staged", "oneshot", "judging", "reverting", "some-new-phase"):
|
||||
f = cstate(cfake(), phase=phase)
|
||||
rep = self.refused(f, "P1")
|
||||
self.assertIn(phase, rep["refused"]["reason"])
|
||||
|
||||
# --- scenario 4: the flag (P2); the pinned default against grub.cfg (P3) ---
|
||||
def test_s4_the_flag_names_a_kernel_refuses(self):
|
||||
f = cfake(env={"felhom_next": "7.0.14-26-pve"})
|
||||
self.refused(f, "P2")
|
||||
|
||||
def test_s4_pinned_default_and_grub_cfg_disagree_refuses(self):
|
||||
f = cfake()
|
||||
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-{C_FROM}-advanced-{U}"\n'
|
||||
self.refused(f, "P3") # grub.cfg still says 28, the file says 26
|
||||
g = cfake()
|
||||
del g.tree[osapply.KERNEL_DEFAULT_CFG]
|
||||
self.refused(g, "P3") # no pinned file at all
|
||||
h = cfake() # both name a kernel that is not installed
|
||||
h.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-7.0.14-99-pve-advanced-{U}"\n'
|
||||
h.cfg = h.cfg.replace(f"gnulinux-{C_RUN}-advanced-{U}\"", f"gnulinux-7.0.14-99-pve-advanced-{U}\"", 1)
|
||||
self.refused(h, "P3")
|
||||
|
||||
# --- scenario 5: the box fell back / does not run its newest kernel ---
|
||||
def test_s5_a_box_that_fell_back_keeps_every_kernel(self):
|
||||
for phase in ("fell_back", "self_reverted", "revert_failed", "cancelled"):
|
||||
self.not_run(cstate(cfake(running=C_FROM), phase=phase), "step-not-good")
|
||||
|
||||
def test_s5_a_box_that_does_not_run_its_newest_kernel_keeps_every_kernel(self):
|
||||
f = cfake(running=C_FROM) # the step says good, but the box runs the older kernel
|
||||
self.not_run(f, "not-on-newest")
|
||||
|
||||
# --- scenario 6: the wrong case (K1) — one case per kept-kernel kind ---
|
||||
def k1(self, kind, prepare):
|
||||
"""The keep computation is BROKEN on purpose (it forgets `kind`); K1 reads the box by itself and must refuse
|
||||
before any apt call."""
|
||||
f = cfake()
|
||||
victim = prepare(f)
|
||||
real = osapply.Kernel.cleanup_keep
|
||||
|
||||
def broken(self_, images, sources):
|
||||
keep = real(self_, images, [(k, why) for k, why in sources if k != victim])
|
||||
keep.pop(victim, None) # … also when it is the newest installed
|
||||
return keep
|
||||
osapply.Kernel.cleanup_keep = broken
|
||||
try:
|
||||
rc, rep = run(f)
|
||||
finally:
|
||||
osapply.Kernel.cleanup_keep = real
|
||||
self.assertEqual(rc, 2, f"{kind}: {rep}")
|
||||
self.assertEqual(rep["refused"]["code"], "K1", f"{kind}: {rep}")
|
||||
self.assertIn(img(victim), rep["refused"]["reason"])
|
||||
self.assertFalse([c for c in f.calls if "purge" in c[-1]], f"{kind}: refused BEFORE any apt call, also the simulation")
|
||||
self.nothing_changed(f)
|
||||
|
||||
def test_s6_k1_the_running_kernel(self):
|
||||
self.k1("running", lambda f: C_RUN)
|
||||
|
||||
def test_s6_k1_the_steps_from(self):
|
||||
self.k1("from", lambda f: C_FROM)
|
||||
|
||||
def test_s6_k1_the_steps_to(self):
|
||||
def prep(f): # a step whose `to` is not the running kernel any more
|
||||
cstate(f, to="7.0.14-22-pve")
|
||||
return "7.0.14-22-pve"
|
||||
self.k1("to", prep)
|
||||
|
||||
def test_s6_k1_default_before(self):
|
||||
def prep(f):
|
||||
cstate(f, default_before="7.0.14-20-pve")
|
||||
return "7.0.14-20-pve"
|
||||
self.k1("default_before", prep)
|
||||
|
||||
def test_s6_k1_the_grub_cfg_default_and_the_pinned_file(self):
|
||||
# the default (both places) names 26; the box runs 28 after a hand boot. The keep computation forgets 26.
|
||||
def prep(f):
|
||||
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-7.0.14-22-pve-advanced-{U}"\n'
|
||||
f.cfg = f.render()
|
||||
return "7.0.14-22-pve"
|
||||
self.k1("default", prep)
|
||||
|
||||
def test_s6_k1_the_flag(self):
|
||||
# a flag is a refusal by itself (P2, first); with the keep computation broken, K1 is in the SAME refusal
|
||||
f = cfake(env={"felhom_next": "7.0.14-20-pve"})
|
||||
real = osapply.Kernel.cleanup_keep
|
||||
osapply.Kernel.cleanup_keep = lambda s_, images, sources: real(s_, images, [(k, w) for k, w in sources if k != "7.0.14-20-pve"])
|
||||
try:
|
||||
rc, rep = run(f)
|
||||
finally:
|
||||
osapply.Kernel.cleanup_keep = real
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "P2"), rep)
|
||||
self.assertIn("K1 " + img("7.0.14-20-pve") + " is a keep-kernel (the one-shot flag)", rep["refused"]["reason"])
|
||||
self.assertFalse([c for c in f.calls if "purge" in c[-1]])
|
||||
self.nothing_changed(f)
|
||||
|
||||
def test_s6_k1_an_operators_boot_pin(self):
|
||||
def prep(f):
|
||||
f.tree[osapply.KERNEL_PIN_FILE] = "7.0.14-20-pve\n"
|
||||
return "7.0.14-20-pve"
|
||||
self.k1("boot pin", prep)
|
||||
|
||||
def test_s6_every_kept_kind_is_in_the_keep_set_and_never_in_the_removal_set(self):
|
||||
f = cfake()
|
||||
cstate(f, default_before="7.0.14-20-pve")
|
||||
f.tree[osapply.KERNEL_PIN_FILE] = "7.0.14-22-pve\n"
|
||||
f.plan["dry_run"] = True
|
||||
rc, rep = run(f)
|
||||
c = rep["kernel_cleanup"]
|
||||
self.assertEqual(sorted(c["kept"]), sorted(["7.0.14-20-pve", "7.0.14-22-pve", C_FROM, C_RUN]))
|
||||
self.assertEqual(c["removal"], [img("7.0.2-6-pve")])
|
||||
|
||||
def test_s6_the_running_kernel_is_not_an_installed_package(self):
|
||||
f = cfake()
|
||||
del f.installed[img(C_RUN)] # the box runs a kernel dpkg does not know (an unsigned hand install, say)
|
||||
f.installed["proxmox-kernel-7.0.14-28-pve-other"] = "7.0.14-28"
|
||||
self.refused(f, "P3") # P3 is first (the default names a kernel that is not an installed image) …
|
||||
rc, rep = run(f)
|
||||
self.assertIn("K1", rep["refused"]["reason"], "… and K1 is in the same refusal")
|
||||
|
||||
def test_no_step_record_keeps_the_next_lower_kernel(self):
|
||||
f = cstate(cfake(), **{"from": None, "to": None, "default_before": None})
|
||||
f.plan["dry_run"] = True
|
||||
rc, rep = run(f)
|
||||
self.assertIn("7.0.14-26-pve", rep["kernel_cleanup"]["kept"])
|
||||
self.assertIn("next lower", " ".join(rep["kernel_cleanup"]["kept"]["7.0.14-26-pve"]))
|
||||
|
||||
# --- scenario 7: apt would do more (S1, S2) ---
|
||||
def test_s7_the_simulation_would_remove_something_else(self):
|
||||
f = cfake(purge_extra_sim=["Purg proxmox-kernel-7.0 [7.0.14-28]", "Remv proxmox-ve [9.2.0]"])
|
||||
rep = self.refused(f, "S2")
|
||||
self.assertIn("proxmox-ve", rep["refused"]["reason"])
|
||||
|
||||
def test_s7_the_simulation_would_install_something(self):
|
||||
f = cfake(purge_extra_sim=["Inst proxmox-kernel-7.0.14-28-pve (7.0.14-28 Proxmox Debian Repository:stable [amd64])"])
|
||||
rep = self.refused(f, "S1")
|
||||
self.assertIn("proxmox-kernel-7.0.14-28-pve", rep["refused"]["reason"])
|
||||
|
||||
def test_s7_the_simulation_fails(self):
|
||||
self.refused(cfake(purge_sim_rc=100), "S1")
|
||||
|
||||
# --- scenario 8: not our box ---
|
||||
def test_s8_a_byo_host_is_refused_by_the_appliance_rule(self):
|
||||
f = cfake()
|
||||
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||
self.refused(f, "R12")
|
||||
|
||||
def test_s8_a_box_without_the_ring0_mark_does_not_run(self):
|
||||
f = cfake()
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "tester-1-d70be4", "ring0_slow_lane": False})
|
||||
rep = self.not_run(f, "no-ring0-mark")
|
||||
self.assertIn("ring 0 only in this release", rep["kernel_cleanup"]["not_run"][0]["reason"])
|
||||
g = cfake() # an agent-writable trust file proves nothing
|
||||
g.stats[osapply.TRUST_FILE] = St(mode=statmod.S_IFREG | 0o666, uid=0)
|
||||
self.not_run(g, "no-ring0-mark")
|
||||
h = cfake() # the agent's plan cannot claim the ring
|
||||
h.files[osapply.TRUST_FILE] = json.dumps({"host_id": "x", "ring0_slow_lane": False})
|
||||
h.plan["ring"] = 0
|
||||
self.not_run(h, "no-ring0-mark")
|
||||
|
||||
# --- scenario 9: a plan that names packages ---
|
||||
def test_s9_cleanup_plan_names_refused(self):
|
||||
for extra in ({"packages": [{"name": img("7.0.14-28-pve"), "version": "7.0.14-28", "origin": "Proxmox Debian Repository"}]},
|
||||
{"select": "pending-kernel"}, {"allow_new": ["x"]}, {"signed": {"blob_b64": "eA==", "sig": "x"}},
|
||||
{"expect_kver": "7.0.14-20-pve"}, {"dry_run": "yes"}):
|
||||
f = cfake()
|
||||
f.plan.update(extra)
|
||||
self.refused(f, "C1")
|
||||
|
||||
# --- scenario 10: no kernel step ever ran ---
|
||||
def test_s10_no_state_file_does_not_run(self):
|
||||
f = cfake()
|
||||
f.state = None
|
||||
f.write_state()
|
||||
rep = self.not_run(f, "no-step")
|
||||
self.assertIn("no kernel step yet", rep["kernel_cleanup"]["not_run"][0]["reason"])
|
||||
|
||||
# --- scenario 11: a power cut in the purge ---
|
||||
def test_s11_a_cut_purge_is_repaired_first_and_the_rule_is_computed_again(self):
|
||||
f = cfake(purge_rc=100)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assertEqual(rep["failed"]["step"], "purge")
|
||||
self.assertEqual(rep["kernel_cleanup"]["outcome"], "failed")
|
||||
# the cut removed ONE of the three before it stopped
|
||||
f.installed.pop(self.REMOVE[0])
|
||||
f.boot.discard("7.0.14-20-pve")
|
||||
f.purge_rc, f.purges = 0, []
|
||||
f.calls.clear()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["repair"]["journal_before"], 1, "repair first: dpkg's journal was replayed")
|
||||
i_repair = [i for i, c in enumerate(f.calls) if c[-1][-3:-1] == ["--configure", "-a"] or "--configure" in c[-1]][0]
|
||||
i_purge = [i for i, c in enumerate(f.calls) if "purge" in c[-1]][0]
|
||||
self.assertLess(i_repair, i_purge, "the repair runs BEFORE the rule is computed and before any purge")
|
||||
self.assertEqual(f.purges, [self.REMOVE[1:]], "the rule was computed again from the box: the two that are left")
|
||||
self.assertEqual(rep["kernel_cleanup"]["outcome"], "removed")
|
||||
|
||||
# --- the 20-hour window, the dry run, after-the-purge proof ---
|
||||
def test_inside_a_steps_20_hour_window_it_does_not_run(self):
|
||||
f = cfake()
|
||||
cstate(f, result_at=osapply._iso(f.clock - 3600)) # the step ended good one hour ago
|
||||
self.not_run(f, "window")
|
||||
|
||||
def test_the_dry_run_reports_and_changes_nothing(self):
|
||||
f = cfake()
|
||||
f.plan["dry_run"] = True
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
c = rep["kernel_cleanup"]
|
||||
self.assertEqual((c["outcome"], c["would"], c["removal"]), ("dry_run", "remove", self.REMOVE))
|
||||
self.assertEqual(sorted(c["kept"]), sorted([C_FROM, C_RUN]))
|
||||
self.nothing_changed(f)
|
||||
self.assertNotIn("repair", rep, "a dry run does not even repair")
|
||||
# a dry run of a refusal says which one, with exit 0
|
||||
g = cstate(cfake(), phase="judging")
|
||||
g.plan["dry_run"] = True
|
||||
rc, rep = run(g)
|
||||
self.assertEqual((rc, rep["kernel_cleanup"]["would"]), (0, "refuse"), rep)
|
||||
self.assertEqual([x["code"] for x in rep["kernel_cleanup"]["refusals"]], ["P1"])
|
||||
self.nothing_changed(g)
|
||||
# … and of a box that would not run
|
||||
h = cfake()
|
||||
h.files[osapply.TRUST_FILE] = json.dumps({"host_id": "x", "ring0_slow_lane": False})
|
||||
h.plan["dry_run"] = True
|
||||
rc, rep = run(h)
|
||||
self.assertEqual((rc, rep["kernel_cleanup"]["would"], rep["kernel_cleanup"]["removal"]), (0, "not_run", self.REMOVE), rep)
|
||||
self.nothing_changed(h)
|
||||
|
||||
def test_a_purge_that_moves_the_default_is_reported_failed_loudly(self):
|
||||
f = cfake(purge_moves_default=True) # the pinned file vanished in the purge: 10_linux makes the newest the default
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assertEqual(rep["failed"]["step"], "verify")
|
||||
self.assertIn("pinned default file changed", rep["failed"]["reason"])
|
||||
self.assertEqual(f.update_grubs, 0, "never repaired by hand in code")
|
||||
self.assertTrue(any("FAILED" in l and "verify" in l for l in f.logs))
|
||||
|
||||
def test_the_apt_lock_is_respected(self):
|
||||
self.refused(cfake(lock_held=True), "R9")
|
||||
|
||||
def test_both_image_packages_of_one_old_kernel_go_together(self):
|
||||
f = cfake()
|
||||
f.installed["proxmox-kernel-7.0.14-20-pve"] = "7.0.14-20" # the unsigned image beside the signed one
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertIn("proxmox-kernel-7.0.14-20-pve", f.purges[0])
|
||||
self.assertIn(img("7.0.14-20-pve"), f.purges[0])
|
||||
|
||||
def test_cleanup_needs_no_new_sudo_line(self):
|
||||
"""The wrapper is reached by `--plan` only: the new mode rides the one sudo line the agent already has."""
|
||||
sudo = (HERE / "felhom-agent.sudoers").read_text()
|
||||
live = [l.strip().rstrip("\\,").strip() for l in sudo.splitlines() if l.strip() and not l.strip().startswith("#")]
|
||||
self.assertEqual([l for l in live if "felhom-os-apply" in l and "Cmnd_Alias" not in l and "NOPASSWD" not in l],
|
||||
[osapply.SELF_CHECK_SUDO_LINE], "exactly the one --plan line reaches the wrapper")
|
||||
self.assertFalse([l for l in live if "kernel-cleanup" in l or
|
||||
("apt" in l and ("purge" in l or "remove" in l))], "no sudo line removes a package")
|
||||
|
||||
|
||||
|
||||
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
|
||||
class PFake(Fake):
|
||||
"""A host with two AppArmor cache folders. `stale`: profile files whose cached copy in the running kernel's folder
|
||||
is an old compile; the refresh command rewrites the cache from `fresh`. lxc-containers compiles differently every
|
||||
time (measured on demo-hp). `failed`: unit -> ExecStart program."""
|
||||
|
||||
def __init__(self, stale=("usr.sbin.chronyd",), failed=None, kernel="7.0.14-28-pve", record=None):
|
||||
super().__init__()
|
||||
self.kernel = kernel
|
||||
self.fresh = {"usr.sbin.chronyd": b"chrony-4.8", "usr.bin.man": b"man", "lxc-containers": b"lxc-0", "nvidia_modprobe": b"nv"}
|
||||
self.names = {"usr.sbin.chronyd": "/usr/sbin/chronyd", "usr.bin.man": "/usr/bin/man\nman_filter\nman_groff",
|
||||
"lxc-containers": "lxc-container-default\nlxc-container-default-cgns", "nvidia_modprobe": "nvidia_modprobe\nnvidia_modprobe//kmod"}
|
||||
self.cache = {"ae1a4050.0": {k: b"old-" + k.encode() for k in self.fresh}, "f6da0254.0": dict(self.fresh)}
|
||||
for k in stale:
|
||||
self.cache["f6da0254.0"][k] = b"OLD-COPY-" + k.encode()
|
||||
self.failed = dict({"chrony.service": "/usr/sbin/chronyd"} if failed is None else failed)
|
||||
self.restarts, self.parser_runs, self.parser_rc, self.tree = [], 0, 0, {}
|
||||
if record is not None:
|
||||
self.tree[osapply.PROFILE_STATE] = json.dumps(record)
|
||||
self.plan = {"release_id": "profile-refresh", "layer": "host", "mode": "profile-refresh"}
|
||||
|
||||
def isdir(self, p):
|
||||
return p in (osapply.PROFILE_DIR, osapply.PROFILE_CACHE) or p.startswith(osapply.PROFILE_CACHE + "/") and p.count("/") == 4
|
||||
|
||||
def list_dir(self, p):
|
||||
if p == osapply.PROFILE_CACHE:
|
||||
return sorted(self.cache)
|
||||
d = p.rsplit("/", 1)[1]
|
||||
return sorted(list(self.cache.get(d, {})) + [".features"])
|
||||
|
||||
def read_bytes(self, p):
|
||||
d, f = p.split("/")[-2:]
|
||||
return self.cache[d][f]
|
||||
|
||||
def put_file(self, path, data, mode):
|
||||
self.tree[path] = data.decode()
|
||||
|
||||
def read_file(self, p):
|
||||
if p in self.tree:
|
||||
return self.tree[p]
|
||||
return super().read_file(p)
|
||||
|
||||
def host(self, argv, timeout=600, stdin=None):
|
||||
if argv[0] == "uname":
|
||||
self.calls.append(("host", argv))
|
||||
return 0, self.kernel + "\n", ""
|
||||
if argv == osapply.PROFILE_REFRESH_CMD:
|
||||
self.calls.append(("host", argv))
|
||||
self.parser_runs += 1
|
||||
if self.parser_rc:
|
||||
return self.parser_rc, "", "AppArmor parser error for /etc/apparmor.d/x"
|
||||
self.fresh["lxc-containers"] = b"lxc-%d" % self.parser_runs # never the same binary twice
|
||||
self.cache["f6da0254.0"] = dict(self.fresh)
|
||||
return 0, "", ""
|
||||
if argv[:2] == ["apparmor_parser", "-N"]:
|
||||
self.calls.append(("host", argv))
|
||||
return 0, self.names[argv[2].rsplit("/", 1)[1]] + "\n", ""
|
||||
if argv[:2] == ["systemctl", "list-units"]:
|
||||
self.calls.append(("host", argv))
|
||||
return 0, "".join(f"{u} loaded failed failed x\n" for u in self.failed), ""
|
||||
if argv[:2] == ["systemctl", "show"]:
|
||||
self.calls.append(("host", argv))
|
||||
return 0, f"{{ path={self.failed[argv[-1]]} ; argv[]={self.failed[argv[-1]]} -F 1 ; ignore_errors=no }}\n", ""
|
||||
if argv[:2] == ["systemctl", "restart"]:
|
||||
self.calls.append(("host", argv))
|
||||
self.restarts.append(argv[2])
|
||||
return 0, "", ""
|
||||
return super().host(argv, timeout, stdin)
|
||||
|
||||
|
||||
class ProfileRefresh(unittest.TestCase):
|
||||
"""Red-proofs: felhom.eu audits/release-2a-2026-11-14/B/redproof.txt."""
|
||||
|
||||
def test_a_stale_profile_is_refreshed_and_only_its_failed_unit_restarts(self):
|
||||
f = PFake(failed={"chrony.service": "/usr/sbin/chronyd", "postfix.service": "/usr/sbin/postfix", "other.service": "/usr/bin/man-not"})
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
p = rep["profile_refresh"]
|
||||
self.assertEqual(p["outcome"], "refreshed")
|
||||
self.assertEqual(f.parser_runs, 1)
|
||||
self.assertIn(("host", ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", "/etc/apparmor.d"]), f.calls)
|
||||
self.assertEqual(f.cache["f6da0254.0"]["usr.sbin.chronyd"], b"chrony-4.8", "the cache holds the fresh compile")
|
||||
self.assertEqual(p["rewritten"], ["f6da0254.0/lxc-containers", "f6da0254.0/usr.sbin.chronyd"])
|
||||
self.assertEqual(f.restarts, ["chrony.service"], "ONLY the failed unit whose own program's profile was stale")
|
||||
self.assertEqual([x["unit"] for x in p["restarted"]], ["chrony.service"])
|
||||
rec = json.loads(f.tree[osapply.PROFILE_STATE])
|
||||
self.assertEqual(rec["kernel"], "7.0.14-28-pve")
|
||||
|
||||
def test_a_fresh_cache_changes_nothing_and_restarts_nothing(self):
|
||||
f = PFake(stale=()) # chrony is failed for another reason; its profile is NOT stale
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["profile_refresh"]["rewritten"], ["f6da0254.0/lxc-containers"], "only the file that never compiles the same")
|
||||
self.assertEqual(f.restarts, [], "a failed unit whose profile was not stale is NOT restarted")
|
||||
self.assertEqual(f.cache["f6da0254.0"]["usr.sbin.chronyd"], b"chrony-4.8")
|
||||
|
||||
def test_a_stale_profile_of_a_running_program_restarts_nothing(self):
|
||||
f = PFake(failed={}) # the unit is active: the kernel swaps the profile of a running task itself
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, f.restarts, rep["profile_refresh"]["outcome"]), (0, [], "refreshed"), rep)
|
||||
|
||||
def test_a_failed_unit_of_another_program_is_not_restarted(self):
|
||||
f = PFake(stale=("usr.bin.man",), failed={"chrony.service": "/usr/sbin/chronyd"})
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, f.restarts), (0, []), rep)
|
||||
g = PFake(stale=("nvidia_modprobe",), failed={"nvidia.service": "nvidia_modprobe"}) # a profile name that is not a path
|
||||
rc, rep = run(g)
|
||||
self.assertEqual((rc, g.restarts), (0, []), rep)
|
||||
|
||||
def test_once_per_kernel(self):
|
||||
f = PFake()
|
||||
self.assertEqual(run(f)[0], 0)
|
||||
f.calls.clear()
|
||||
f.cache["f6da0254.0"]["usr.sbin.chronyd"] = b"OLD-COPY" # even with a stale file: this kernel was done
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["profile_refresh"]["outcome"], f.parser_runs), (0, "not_needed", 1), rep)
|
||||
self.assertFalse([c for c in f.calls if c[1][0] in ("apparmor_parser", "systemctl")], "no parser, no systemctl")
|
||||
f.kernel = "7.0.14-29-pve" # the box starts on another kernel: again
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["profile_refresh"]["outcome"], f.parser_runs), (0, "refreshed", 2), rep)
|
||||
self.assertEqual(json.loads(f.tree[osapply.PROFILE_STATE])["kernel"], "7.0.14-29-pve")
|
||||
|
||||
def test_a_failed_parser_writes_no_record_so_the_next_start_tries_again(self):
|
||||
f = PFake()
|
||||
f.parser_rc = 1
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["failed"]["step"], rep["profile_refresh"]["outcome"]), (3, "profile-refresh", "failed"), rep)
|
||||
self.assertNotIn(osapply.PROFILE_STATE, f.tree)
|
||||
self.assertEqual(f.restarts, [])
|
||||
f.parser_rc = 0
|
||||
self.assertEqual(run(f)[1]["profile_refresh"]["outcome"], "refreshed")
|
||||
|
||||
def test_the_plan_names_nothing(self):
|
||||
for extra in ({"profiles": ["usr.sbin.chronyd"]}, {"units": ["ssh.service"]}, {"packages": [{"name": "chrony", "version": "4.8", "origin": "Debian"}]},
|
||||
{"force": True}, {"dry_run": 1}, {"layer": "guest"}):
|
||||
f = PFake()
|
||||
f.plan.update(extra)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 2, rep)
|
||||
self.assertIn(rep["refused"]["code"], ("F1", "R11"), rep)
|
||||
self.assertEqual((f.parser_runs, f.restarts), (0, []))
|
||||
|
||||
def test_a_byo_host_is_refused(self):
|
||||
f = PFake()
|
||||
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["refused"]["code"], f.parser_runs), (2, "R12", 0), rep)
|
||||
|
||||
def test_the_dry_run_changes_nothing(self):
|
||||
f = PFake()
|
||||
f.plan["dry_run"] = True
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["profile_refresh"]["outcome"], rep["profile_refresh"]["would"]), (0, "dry_run", "refresh"), rep)
|
||||
self.assertEqual((f.parser_runs, f.restarts), (0, []))
|
||||
self.assertNotIn(osapply.PROFILE_STATE, f.tree)
|
||||
|
||||
def test_no_more_than_three_restarts(self):
|
||||
f = PFake(failed={f"chrony{i}.service": "/usr/sbin/chronyd" for i in range(5)})
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, len(f.restarts), len(rep["profile_refresh"]["not_restarted"])), (0, 3, 2), rep)
|
||||
|
||||
def test_profile_refresh_changes_no_package(self):
|
||||
f = PFake()
|
||||
run(f)
|
||||
self.assertFalse([c for c in f.calls if c[1][0] in ("apt-get", "dpkg", "apt")], "no package command at all")
|
||||
|
||||
|
||||
# ---------- trust in Proxmox by a copy of its root CA (R-1380, D80, felhom.eu `03` §12a) ----------
|
||||
CA_PEM = b"-----BEGIN CERTIFICATE-----\nQUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=\n-----END CERTIFICATE-----\n"
|
||||
CA_PEM_2 = b"-----BEGIN CERTIFICATE-----\nWllYV1ZVVFNSUVBPTk1MS0pJSEdGRURDQkE=\n-----END CERTIFICATE-----\n"
|
||||
|
||||
|
||||
class TFake(Fake):
|
||||
"""A Proxmox host: the root CA file, the copy (or none), and openssl's answer about the bytes it is given."""
|
||||
|
||||
def __init__(self, source=CA_PEM, copy=None, ca=True, ends="Aug 17 15:44:42 2036 GMT"):
|
||||
super().__init__()
|
||||
self.source, self.tree, self.ca, self.ends, self.openssl_rc = source, {}, ca, ends, 0
|
||||
if copy is not None:
|
||||
self.tree[osapply.PVE_CA_COPY] = copy
|
||||
self.openssl_stdin, self.puts = [], []
|
||||
self.serves_new = False # does pveproxy serve a certificate that verifies with the NEW root alone?
|
||||
self.dead_ca = set() # certificate texts openssl says are not a live CA (ended)
|
||||
self.handshakes = []
|
||||
self.plan = {"release_id": "pve-ca", "layer": "host", "mode": "pve-ca-copy"}
|
||||
|
||||
def read_bytes(self, p):
|
||||
if p == osapply.PVE_CA_SOURCE:
|
||||
if self.source is None:
|
||||
raise FileNotFoundError(p)
|
||||
return self.source
|
||||
if p in self.tree:
|
||||
return self.tree[p]
|
||||
raise FileNotFoundError(p)
|
||||
|
||||
def put_file(self, path, data, mode):
|
||||
self.puts.append((path, data, mode))
|
||||
self.tree[path] = data
|
||||
|
||||
def host(self, argv, timeout=600, stdin=None):
|
||||
if argv[0] == "openssl" and argv[1] == "s_client":
|
||||
self.handshakes.append(argv)
|
||||
return (0, "Verify return code: 0 (ok)\n", "") if self.serves_new else (1, "", "Verify return code: 19 (self-signed certificate in certificate chain)\n")
|
||||
if argv[0] == "openssl" and "-checkend" in argv:
|
||||
return (1, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will expire\n", "") if stdin.strip() in self.dead_ca else \
|
||||
(0, "X509v3 Basic Constraints: critical\n CA:TRUE\nCertificate will not expire\n", "")
|
||||
if argv[0] == "openssl":
|
||||
self.calls.append(("host", argv))
|
||||
self.openssl_stdin.append(stdin)
|
||||
if self.openssl_rc:
|
||||
return self.openssl_rc, "", "Could not read certificate from <stdin>"
|
||||
return 0, ("X509v3 Basic Constraints: critical\n CA:%s\nnotAfter=%s\n" % ("TRUE" if self.ca else "FALSE", self.ends)), ""
|
||||
return super().host(argv, timeout, stdin)
|
||||
|
||||
|
||||
class PVECACopy(unittest.TestCase):
|
||||
"""Red-proofs: felhom.eu audits/night-build-2026-11-14/A/redproof-wrapper.txt."""
|
||||
|
||||
def refused(self, f, code):
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, code), rep)
|
||||
self.assertEqual(f.puts, [], "nothing is written on a refusal")
|
||||
return rep
|
||||
|
||||
def test_a_new_box_gets_the_copy(self):
|
||||
f = TFake()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.puts, [(osapply.PVE_CA_COPY, CA_PEM, 0o644)], "ONE file, the fixed place, the source's bytes, 0644")
|
||||
self.assertEqual(f.openssl_stdin, [CA_PEM.decode()], "the bytes that are checked are the bytes that are written")
|
||||
c = rep["pve_ca_copy"]
|
||||
self.assertEqual((c["outcome"], c["path"], c["replaced"]), ("written", "/etc/felhom-agent/pve-root-ca.pem", False))
|
||||
self.assertEqual(c["not_after"], "2036-08-17T15:44:42Z")
|
||||
self.assertGreater(c["days_left"], 3000)
|
||||
|
||||
def test_the_same_copy_is_not_written_again(self):
|
||||
f = TFake(copy=CA_PEM)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
|
||||
|
||||
def test_a_renewed_ca_refreshes_the_copy(self):
|
||||
f = TFake(source=CA_PEM_2, copy=CA_PEM)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["replaced"]), (0, "written", True), rep)
|
||||
# the new certificate FIRST, and the copy's previous one kept behind it: Proxmox serves its old certificate until
|
||||
# pveproxy is reloaded, and an agent that restarts in that window must still have a root that verifies it
|
||||
bundle = f.tree[osapply.PVE_CA_COPY]
|
||||
self.assertEqual(bundle, CA_PEM_2 + osapply.PEM_CERT_RE.findall(CA_PEM)[0] + b"\n")
|
||||
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
|
||||
self.assertEqual(f.openssl_stdin, [CA_PEM_2.decode()], "the SOURCE is checked in full; the kept one by its own check")
|
||||
|
||||
def test_the_previous_root_stays_only_for_the_changeover(self):
|
||||
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
|
||||
bundle = CA_PEM_2 + old + b"\n"
|
||||
# Proxmox still serves the OLD certificate: nothing is written, the old root stays
|
||||
f = TFake(source=CA_PEM_2, copy=bundle)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"], f.puts), (0, "same", []), rep)
|
||||
self.assertTrue(rep["pve_ca_copy"]["kept_previous"])
|
||||
self.assertEqual(len(f.handshakes), 1)
|
||||
self.assertIn(osapply.PVE_CA_SOURCE, f.handshakes[0], "the handshake is verified with the NEW root alone (the source file)")
|
||||
self.assertIn("-no-CApath", f.handshakes[0])
|
||||
self.assertIn("-verify_return_error", f.handshakes[0])
|
||||
# pveproxy was reloaded and serves a certificate of the NEW root: the old root is dropped
|
||||
f = TFake(source=CA_PEM_2, copy=bundle)
|
||||
f.serves_new = True
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2, "after the changeover the copy is the new root alone")
|
||||
self.assertEqual((rep["pve_ca_copy"]["outcome"], rep["pve_ca_copy"]["dropped_previous"], rep["pve_ca_copy"]["kept_previous"]), ("written", True, False))
|
||||
# the old root has ENDED while Proxmox still serves the old certificate: it verifies nothing, so it is dropped
|
||||
f = TFake(source=CA_PEM_2, copy=bundle)
|
||||
f.dead_ca = {old.decode()}
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
|
||||
# an ended previous root is not kept at the change itself either
|
||||
f = TFake(source=CA_PEM_2, copy=CA_PEM)
|
||||
f.dead_ca = {old.decode()}
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM_2)
|
||||
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
|
||||
|
||||
def test_a_copy_with_anything_unchecked_in_it_is_rewritten(self):
|
||||
old = osapply.PEM_CERT_RE.findall(CA_PEM)[0]
|
||||
for name, copy in (("text after the source", CA_PEM_2 + b"anything at all\n"),
|
||||
("a third certificate", CA_PEM_2 + old + b"\n" + old + b"\n"),
|
||||
("a key in the file", CA_PEM_2 + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"),
|
||||
("the two in the wrong order", CA_PEM + osapply.PEM_CERT_RE.findall(CA_PEM_2)[0] + b"\n")):
|
||||
f = TFake(source=CA_PEM_2, copy=copy)
|
||||
f.serves_new = True
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, (name, rep))
|
||||
got = f.tree[osapply.PVE_CA_COPY]
|
||||
self.assertTrue(got.startswith(CA_PEM_2), name)
|
||||
self.assertNotIn(b"anything", got, name)
|
||||
self.assertNotIn(b"PRIVATE KEY", got, name)
|
||||
self.assertLessEqual(len(osapply.PEM_CERT_RE.findall(got)), 2, name)
|
||||
# a first copy keeps nothing behind it, and makes no handshake
|
||||
f4 = TFake()
|
||||
rc, rep = run(f4)
|
||||
self.assertFalse(rep["pve_ca_copy"]["kept_previous"])
|
||||
self.assertEqual((f4.tree[osapply.PVE_CA_COPY], f4.handshakes), (CA_PEM, []))
|
||||
|
||||
def test_a_source_that_is_not_a_ca_certificate_is_refused(self):
|
||||
self.refused(TFake(ca=False), "T3") # a leaf certificate
|
||||
self.refused(TFake(source=b"hello\n"), "T3") # not a certificate at all
|
||||
self.refused(TFake(source=CA_PEM + CA_PEM_2), "T3") # two certificates (a chain, a bundle)
|
||||
self.refused(TFake(source=CA_PEM + b"-----BEGIN PRIVATE KEY-----\nAAAA\n-----END PRIVATE KEY-----\n"), "T3")
|
||||
self.refused(TFake(source=b""), "T3")
|
||||
self.refused(TFake(source=CA_PEM + b"x" * (osapply.PVE_CA_MAX_BYTES + 1)), "T3")
|
||||
f = TFake()
|
||||
f.openssl_rc = 1 # openssl cannot parse it
|
||||
self.refused(f, "T3")
|
||||
self.refused(TFake(ends="Aug 17 15:44:42 2020 GMT"), "T3") # an ended CA
|
||||
self.refused(TFake(ends="garbage"), "T3")
|
||||
|
||||
def test_no_source_is_refused(self):
|
||||
self.refused(TFake(source=None), "T2")
|
||||
|
||||
def test_with_the_real_openssl_and_the_real_runner(self):
|
||||
"""NOT a fake: a CA and a leaf made by openssl in a temp folder, judged through Runner.host as on a box (the
|
||||
fake cannot show a wrong argument or a bytes/text mix-up). Skipped where openssl is missing."""
|
||||
import shutil, tempfile
|
||||
if not shutil.which("openssl"):
|
||||
self.skipTest("no openssl here")
|
||||
d = tempfile.mkdtemp()
|
||||
try:
|
||||
def sh(*a):
|
||||
subprocess.run(a, check=True, capture_output=True, cwd=d)
|
||||
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "ca.key", "-out", "ca.pem", "-days", "30",
|
||||
"-subj", "/CN=test CA", "-addext", "basicConstraints=critical,CA:TRUE")
|
||||
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "leaf.key", "-out", "leaf.pem", "-days", "30",
|
||||
"-subj", "/CN=leaf", "-addext", "basicConstraints=CA:FALSE")
|
||||
real = osapply.Runner()
|
||||
for name, want in (("ca.pem", "written"), ("leaf.pem", "T3")):
|
||||
data = open(os.path.join(d, name), "rb").read()
|
||||
f = TFake(source=data)
|
||||
f.host = lambda argv, timeout=600, stdin=None: real.host(argv, timeout, stdin) if argv[0] == "openssl" else Fake.host(f, argv, timeout, stdin)
|
||||
f.clock = __import__("time").time()
|
||||
rc, rep = run(f)
|
||||
got = rep["pve_ca_copy"]["outcome"] if rc == 0 else rep["refused"]["code"]
|
||||
self.assertEqual(got, want, f"{name}: {rep}")
|
||||
if want == "written":
|
||||
self.assertIn(rep["pve_ca_copy"]["days_left"], (29, 30))
|
||||
# the two checks the changeover uses, with the REAL openssl and a REAL TLS server
|
||||
ended = True
|
||||
try: # an ENDED CA needs openssl 3.4+ (-not_after); older: that one check is not made
|
||||
sh("openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", "old.key", "-out", "old.pem",
|
||||
"-not_before", "20190101000000Z", "-not_after", "20200101000000Z",
|
||||
"-subj", "/CN=ended CA", "-addext", "basicConstraints=critical,CA:TRUE")
|
||||
except subprocess.CalledProcessError:
|
||||
ended = False
|
||||
sh("openssl", "req", "-newkey", "rsa:2048", "-nodes", "-keyout", "srv.key", "-out", "srv.csr", "-subj", "/CN=pve")
|
||||
sh("openssl", "x509", "-req", "-in", "srv.csr", "-CA", "ca.pem", "-CAkey", "ca.key", "-CAcreateserial", "-out", "srv.pem", "-days", "5")
|
||||
app = osapply.Apply.__new__(osapply.Apply)
|
||||
app.r = real
|
||||
block = lambda n: osapply.PEM_CERT_RE.findall(open(os.path.join(d, n), "rb").read())[0]
|
||||
self.assertTrue(app.is_live_ca(block("ca.pem")), "a CA that has not ended")
|
||||
self.assertFalse(app.is_live_ca(block("leaf.pem")), "a leaf is not a CA")
|
||||
if ended:
|
||||
self.assertFalse(app.is_live_ca(block("old.pem")), "an ended CA is not a live one")
|
||||
import socket
|
||||
s0 = socket.socket(); s0.bind(("127.0.0.1", 0)); port = s0.getsockname()[1]; s0.close()
|
||||
srv = subprocess.Popen(["openssl", "s_server", "-accept", f"127.0.0.1:{port}", "-cert", "srv.pem", "-key", "srv.key", "-quiet"],
|
||||
cwd=d, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try:
|
||||
saved, osapply.PVE_LOCAL_API = osapply.PVE_LOCAL_API, f"127.0.0.1:{port}"
|
||||
ok = False
|
||||
for _ in range(40): # the server needs a moment to listen
|
||||
if app.pve_serves_with(os.path.join(d, "ca.pem")):
|
||||
ok = True
|
||||
break
|
||||
__import__("time").sleep(0.1)
|
||||
self.assertTrue(ok, "the server's certificate is signed by ca.pem: the handshake must verify with it alone")
|
||||
self.assertFalse(app.pve_serves_with(os.path.join(d, "leaf.pem")), "another certificate as the CA file must NOT verify")
|
||||
osapply.PVE_LOCAL_API = "127.0.0.1:1"
|
||||
self.assertFalse(app.pve_serves_with(os.path.join(d, "ca.pem")), "no answer is a no")
|
||||
finally:
|
||||
osapply.PVE_LOCAL_API = saved
|
||||
srv.kill(); srv.wait()
|
||||
finally:
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
|
||||
def test_a_plan_that_names_a_path_or_a_file_is_refused(self):
|
||||
for extra in ({"source": "/etc/shadow"}, {"path": "/etc/sudoers.d/x"}, {"dest": "/root/.ssh/authorized_keys"},
|
||||
{"content_b64": "QQ=="}, {"packages": []}, {"file": "x"}, {"vmid": 9201}, {"dry_run": True}):
|
||||
f = TFake()
|
||||
f.plan.update(extra)
|
||||
rep = self.refused(f, "T1")
|
||||
self.assertEqual(f.openssl_stdin, [], "refused before the source is even read")
|
||||
g = TFake()
|
||||
g.plan["layer"] = "kernel"
|
||||
self.refused(g, "R11")
|
||||
|
||||
def test_a_refused_source_keeps_an_older_copy(self):
|
||||
f = TFake(source=b"not a certificate", copy=CA_PEM)
|
||||
self.refused(f, "T3")
|
||||
self.assertEqual(f.tree[osapply.PVE_CA_COPY], CA_PEM, "written whole or not at all")
|
||||
|
||||
def test_it_runs_on_a_byo_host_too_and_changes_nothing_else(self):
|
||||
f = TFake()
|
||||
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||
rc, rep = run(f)
|
||||
self.assertEqual((rc, rep["pve_ca_copy"]["outcome"]), (0, "written"), rep)
|
||||
self.assertEqual([c[1][0] for c in f.calls], ["openssl"], "one openssl call; no package, no service, no other file")
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,6 +1,6 @@
|
||||
module gitea.dooplex.hu/admin/felhom-agent
|
||||
|
||||
go 1.27.2
|
||||
go 1.27.4
|
||||
|
||||
require (
|
||||
golang.org/x/crypto v0.58.0
|
||||
|
||||
@@ -22,6 +22,8 @@ var (
|
||||
ErrExpired = errors.New("authz: op expired")
|
||||
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
|
||||
ErrNotYetValid = errors.New("authz: op not yet valid")
|
||||
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
|
||||
ErrWindowTooLong = errors.New("authz: op validity window too long")
|
||||
// ErrReplay: the nonce was already recorded in the window.
|
||||
ErrReplay = errors.New("authz: replay (nonce already seen)")
|
||||
)
|
||||
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
|
||||
// the validity window stays an honest upper bound.
|
||||
const DefaultClockSkew = 2 * time.Minute
|
||||
|
||||
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
|
||||
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
|
||||
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
|
||||
const MaxWindow = 24 * time.Hour
|
||||
|
||||
// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
|
||||
type KeyRole string
|
||||
|
||||
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
|
||||
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
|
||||
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
|
||||
}
|
||||
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
|
||||
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
|
||||
}
|
||||
if now.After(op.ExpiresAt) {
|
||||
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
|
||||
}
|
||||
|
||||
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
|
||||
rejects(t, err, ErrTarget)
|
||||
})
|
||||
|
||||
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
|
||||
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
|
||||
t.Run("window longer than 24 h", func(t *testing.T) {
|
||||
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
|
||||
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
|
||||
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
|
||||
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
|
||||
_, err := v.Verify(blob, sig)
|
||||
rejects(t, err, ErrWindowTooLong)
|
||||
})
|
||||
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
|
||||
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
|
||||
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
|
||||
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
|
||||
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
|
||||
if _, err := v.Verify(blob, sig); err != nil {
|
||||
t.Fatalf("a 24 h window must verify: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("expired", func(t *testing.T) {
|
||||
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
|
||||
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
|
||||
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
|
||||
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
|
||||
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
|
||||
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
|
||||
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
|
||||
// old box's time.
|
||||
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
|
||||
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
|
||||
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
|
||||
content: []proxmox.StorageContent{oldBox},
|
||||
}
|
||||
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
|
||||
// 1. The fresh box has written nothing yet: the old box's archive must not count.
|
||||
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if found {
|
||||
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
|
||||
}
|
||||
// 2. After the box's own first copy, that copy is the answer.
|
||||
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
|
||||
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
|
||||
api.content = append(api.content, own)
|
||||
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
|
||||
if err != nil || !found || when.Unix() != own.CTime {
|
||||
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
|
||||
}
|
||||
}
|
||||
|
||||
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
|
||||
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
|
||||
api := &fakeBackupAPI{
|
||||
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
|
||||
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
|
||||
}
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
|
||||
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
|
||||
}
|
||||
}
|
||||
@@ -609,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
|
||||
if err != nil {
|
||||
return time.Time{}, false, err
|
||||
}
|
||||
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
|
||||
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
|
||||
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
|
||||
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
|
||||
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
|
||||
if err != nil {
|
||||
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
|
||||
}
|
||||
var best int64 = -1
|
||||
for _, e := range contents {
|
||||
if e.Content != "backup" || e.VMID != vmid {
|
||||
continue
|
||||
}
|
||||
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
|
||||
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
|
||||
continue
|
||||
}
|
||||
if ok, why := archivePlausiblyComplete(e); !ok {
|
||||
r.warnRejectedArchiveOnce(e, why)
|
||||
continue
|
||||
|
||||
@@ -151,6 +151,7 @@ var manifest = []Capability{
|
||||
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
|
||||
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
||||
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
||||
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
|
||||
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
||||
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
|
||||
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
|
||||
|
||||
@@ -34,6 +34,7 @@ type Config struct {
|
||||
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
|
||||
GuestNet GuestNetConfig `json:"guest_net"`
|
||||
DiskTrim DiskTrimConfig `json:"disk_trim"`
|
||||
LineTest LineTestConfig `json:"line_test"`
|
||||
OOB OOBConfig `json:"oob"`
|
||||
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
|
||||
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
|
||||
@@ -150,6 +151,16 @@ type DiskTrimConfig struct {
|
||||
// Enabled reports whether the weekly trim should run.
|
||||
func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
|
||||
|
||||
// LineTestConfig configures the R-1247 weekly upload line test (internal/linetest). DEFAULT-ON: it reaches only ep0's
|
||||
// tunnel-only iperf3 server, so it runs only on a box that already has an operational offsite tunnel (operator
|
||||
// ruling 2026-10-24 07:26: every box with a tunnel measures its line weekly).
|
||||
type LineTestConfig struct {
|
||||
Disable bool `json:"disable"`
|
||||
}
|
||||
|
||||
// Enabled reports whether the weekly line test should run.
|
||||
func (l LineTestConfig) Enabled() bool { return !l.Disable }
|
||||
|
||||
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
|
||||
//
|
||||
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// A cut write of agent.json must never leave a blind box (R-1380, 2026-11-15).
|
||||
//
|
||||
// agent.json is the agent's own file inside a ROOT-owned folder, so the agent cannot write a new file and rename it
|
||||
// over the old one: every writer truncates the file and writes it again. A power cut between the truncate and the
|
||||
// sync leaves an empty or half file, the agent does not start, and nothing can repair a box whose agent does not
|
||||
// start from the hub. So the three writers (the trust move, the PBS seed, the backup-target move) go through
|
||||
// ReplaceInPlace: the OLD bytes are first made durable in a journal file in the agent's own state folder (a new file
|
||||
// and a rename, which the agent CAN do there), then the config is written, synced and read back, then the journal is
|
||||
// removed. RecoverCutWrite runs at every start, before the config is read: a journal beside a config that does not
|
||||
// parse means the write was cut, and the old bytes go back. A journal beside a config that parses means the cut fell
|
||||
// before the truncate or after the sync — the file is whole, old or new, and only the journal is removed.
|
||||
//
|
||||
// Pinned by TestReplaceInPlace_* and TestRecoverCutWrite_* (this package) and by TestMainRecoversACutConfigWrite
|
||||
// (cmd/felhom-agent).
|
||||
|
||||
// CutWriteDir is where the journal lives. It is NOT read from the config: the recovery runs before the config can be
|
||||
// read. /var/lib/felhom-agent is the unit's StateDirectory. A test points it at a temp folder.
|
||||
var CutWriteDir = "/var/lib/felhom-agent"
|
||||
|
||||
// cutWriteJournal is the journal's file name: the bytes agent.json held before the write that is in flight.
|
||||
const cutWriteJournal = "agent.json.write-in-flight"
|
||||
|
||||
// cutPoint lets a test stop a write where a power cut would (the names are the steps of ReplaceInPlace).
|
||||
var cutPoint = func(step string) error { return nil }
|
||||
|
||||
// ReplaceInPlace writes next over the config at path, whose present content is prev. The caller holds FileMu.
|
||||
// With no journal folder there is no write: an unprotected write is the defect this exists to remove.
|
||||
func ReplaceInPlace(path string, prev, next []byte) error {
|
||||
if !json.Valid(prev) || !json.Valid(next) {
|
||||
return fmt.Errorf("config: refusing to write %s: the old or the new content is not JSON", path)
|
||||
}
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(CutWriteDir, 0o700); err != nil {
|
||||
return fmt.Errorf("config: the journal folder: %w", err)
|
||||
}
|
||||
journal := filepath.Join(CutWriteDir, cutWriteJournal)
|
||||
if err := writeDurable(journal, prev); err != nil {
|
||||
return fmt.Errorf("config: the journal of the old content: %w", err)
|
||||
}
|
||||
if err := cutPoint("journal-written"); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
|
||||
if err != nil {
|
||||
_ = removeDurable(journal)
|
||||
return err
|
||||
}
|
||||
if err := cutPoint("truncated"); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
half := len(next) / 2
|
||||
_, werr := f.Write(next[:half])
|
||||
if werr == nil {
|
||||
if err := cutPoint("half-written"); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
_, werr = f.Write(next[half:])
|
||||
}
|
||||
if werr == nil {
|
||||
werr = f.Sync()
|
||||
}
|
||||
if cerr := f.Close(); werr == nil {
|
||||
werr = cerr
|
||||
}
|
||||
if werr == nil {
|
||||
var back []byte
|
||||
if back, werr = os.ReadFile(path); werr == nil && !bytes.Equal(back, next) {
|
||||
werr = fmt.Errorf("the file did not read back as written")
|
||||
}
|
||||
}
|
||||
if werr != nil {
|
||||
// The write failed with the process alive: put the old bytes back now. If that fails too, the journal stays
|
||||
// and the next start repairs the file.
|
||||
if rerr := restoreInPlace(path, prev); rerr != nil {
|
||||
return fmt.Errorf("config: writing %s failed (%v) and the old content could not be put back (%v) — the next start repairs it from the journal", path, werr, rerr)
|
||||
}
|
||||
_ = removeDurable(journal)
|
||||
return fmt.Errorf("config: writing %s failed, the old content was put back: %w", path, werr)
|
||||
}
|
||||
if err := cutPoint("synced"); err != nil {
|
||||
return err
|
||||
}
|
||||
return removeDurable(journal)
|
||||
}
|
||||
|
||||
// RecoverCutWrite repairs a config whose last write was cut. It returns what it did, for the start's log line:
|
||||
// "" (no journal), "journal removed — the config is whole", or "the old config was put back".
|
||||
func RecoverCutWrite(path string) (string, error) {
|
||||
journal := filepath.Join(CutWriteDir, cutWriteJournal)
|
||||
prev, err := os.ReadFile(journal)
|
||||
if os.IsNotExist(err) {
|
||||
return "", nil
|
||||
}
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("config: reading the write journal: %w", err)
|
||||
}
|
||||
cur, cerr := os.ReadFile(path)
|
||||
if cerr == nil && isJSONObject(cur) {
|
||||
return "journal removed — the config is whole", removeDurable(journal)
|
||||
}
|
||||
if !isJSONObject(prev) {
|
||||
// A journal that is itself broken cannot have been followed by a truncate (it is made durable first), so
|
||||
// this is not a cut write. Leave both for a person.
|
||||
return "", fmt.Errorf("config: %s does not parse and the write journal does not parse either", path)
|
||||
}
|
||||
if err := restoreInPlace(path, prev); err != nil {
|
||||
return "", fmt.Errorf("config: putting the old content back into %s: %w", path, err)
|
||||
}
|
||||
return "the old config was put back", removeDurable(journal)
|
||||
}
|
||||
|
||||
func isJSONObject(b []byte) bool {
|
||||
var doc map[string]json.RawMessage
|
||||
return json.Unmarshal(b, &doc) == nil && doc != nil
|
||||
}
|
||||
|
||||
// restoreInPlace writes content into the EXISTING file (no create: the agent cannot make a file in the config's
|
||||
// folder), so the file keeps its owner and mode.
|
||||
func restoreInPlace(path string, content []byte) error {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, 0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(content); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Sync(); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
}
|
||||
|
||||
// writeDurable makes path hold content, whole, across a power cut: a new file, synced, renamed, the folder synced.
|
||||
func writeDurable(path string, content []byte) error {
|
||||
tmp := path + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(content); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Sync(); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
return err
|
||||
}
|
||||
return syncDir(filepath.Dir(path))
|
||||
}
|
||||
|
||||
func removeDurable(path string) error {
|
||||
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
|
||||
return err
|
||||
}
|
||||
return syncDir(filepath.Dir(path))
|
||||
}
|
||||
|
||||
func syncDir(dir string) error {
|
||||
d, err := os.Open(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer d.Close()
|
||||
return d.Sync()
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const (
|
||||
oldCfg = `{"proxmox":{"tls":{"fingerprint":"AA:BB"}},"keep":"me"}`
|
||||
newCfg = `{"proxmox":{"tls":{"ca_file":"/etc/felhom-agent/pve-root-ca.pem","fingerprint":""}},"keep":"me"}`
|
||||
)
|
||||
|
||||
// cutBox is a config file and a journal folder in a temp dir, with the write cut at one named step.
|
||||
func cutBox(t *testing.T) (cfg string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
cfg = filepath.Join(dir, "etc", "agent.json")
|
||||
if err := os.MkdirAll(filepath.Dir(cfg), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(cfg, []byte(oldCfg), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prevDir, prevCut := CutWriteDir, cutPoint
|
||||
CutWriteDir = filepath.Join(dir, "state")
|
||||
t.Cleanup(func() { CutWriteDir, cutPoint = prevDir, prevCut })
|
||||
return cfg
|
||||
}
|
||||
|
||||
func cutAt(step string) {
|
||||
cutPoint = func(s string) error {
|
||||
if s == step {
|
||||
return errors.New("power cut at " + s)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func read(t *testing.T, p string) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// The consequence that matters: after a power cut at ANY step of the write and the repair at the next start, the
|
||||
// config is the whole old file or the whole new file — never a broken one — and Load reads it.
|
||||
func TestRecoverCutWrite_AfterACutAtEveryStepTheConfigIsWholeAndLoads(t *testing.T) {
|
||||
want := map[string]string{
|
||||
"journal-written": oldCfg, // nothing was touched yet
|
||||
"truncated": oldCfg, // the file is EMPTY at the cut
|
||||
"half-written": oldCfg, // the file is HALF at the cut
|
||||
"synced": newCfg, // the new file is whole; only the journal is left
|
||||
}
|
||||
for step, wantAfter := range want {
|
||||
t.Run(step, func(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
cutAt(step)
|
||||
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
|
||||
t.Fatal("the staged cut did not stop the write")
|
||||
}
|
||||
atCut := read(t, cfg)
|
||||
if (step == "truncated" || step == "half-written") && isJSONObject([]byte(atCut)) {
|
||||
t.Fatalf("the staging is wrong: at %s the file must be broken, it is %q", step, atCut)
|
||||
}
|
||||
cutPoint = func(string) error { return nil } // the next start: no cut
|
||||
note, err := RecoverCutWrite(cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("repair: %v", err)
|
||||
}
|
||||
if note == "" {
|
||||
t.Fatal("the repair saw no journal — a cut write would stay broken")
|
||||
}
|
||||
if got := read(t, cfg); got != wantAfter {
|
||||
t.Fatalf("after a cut at %s and the repair the config is %q, want %q", step, got, wantAfter)
|
||||
}
|
||||
if _, err := Load(cfg); err != nil {
|
||||
t.Fatalf("the agent does not start with the repaired config: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
|
||||
t.Fatalf("the journal stayed after the repair (%v) — an old journal would undo a later change", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A write that is not cut leaves the new file and NO journal: a journal that stayed would put an old config back
|
||||
// at some later start.
|
||||
func TestReplaceInPlace_AWholeWriteLeavesTheNewFileAndNoJournal(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := read(t, cfg); got != newCfg {
|
||||
t.Fatalf("config = %q", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(CutWriteDir, cutWriteJournal)); !os.IsNotExist(err) {
|
||||
t.Fatalf("the journal stayed after a whole write: %v", err)
|
||||
}
|
||||
if note, err := RecoverCutWrite(cfg); err != nil || note != "" {
|
||||
t.Fatalf("a start after a whole write must do nothing: note=%q err=%v", note, err)
|
||||
}
|
||||
if got := read(t, cfg); got != newCfg {
|
||||
t.Fatalf("the start changed a whole config to %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// No journal folder, no write: the old file stays as it was.
|
||||
func TestReplaceInPlace_WithoutAJournalNothingIsWritten(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
// the journal cannot be made: a folder sits where its new file must be written
|
||||
if err := os.MkdirAll(filepath.Join(CutWriteDir, cutWriteJournal+".tmp"), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := ReplaceInPlace(cfg, []byte(oldCfg), []byte(newCfg)); err == nil {
|
||||
t.Fatal("the write went on with no journal")
|
||||
}
|
||||
if got := read(t, cfg); got != oldCfg {
|
||||
t.Fatalf("the config changed with no journal: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A config that is broken with NO journal is not a cut write: the repair does not touch it, and says nothing.
|
||||
func TestRecoverCutWrite_ABrokenConfigWithNoJournalIsLeftAlone(t *testing.T) {
|
||||
cfg := cutBox(t)
|
||||
if err := os.WriteFile(cfg, []byte(`{"half":`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
note, err := RecoverCutWrite(cfg)
|
||||
if err != nil || note != "" {
|
||||
t.Fatalf("note=%q err=%v", note, err)
|
||||
}
|
||||
if got := read(t, cfg); got != `{"half":` {
|
||||
t.Fatalf("the repair wrote a config it had no journal for: %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package config
|
||||
|
||||
import "sync"
|
||||
|
||||
// FileMu is held by EVERY read-modify-write of agent.json inside this process: the trust move (pvetrust), the PBS
|
||||
// seed (pbsdr) and the backup-target move (localapi). Each of them reads the whole file, changes one section and
|
||||
// writes the whole file back in place — without one lock two of them could interleave, and the second writer would
|
||||
// put back the document it read BEFORE the first one's change (a silently reverted `escrow.pbs_storage_id`,
|
||||
// `backup.local_backup_target` or trust section). Found by review on 2026-11-14, the night the third writer was
|
||||
// added. Hold it for the whole read … write, never across a call to Proxmox or the root wrapper.
|
||||
// Pinned by TestAgentJSONWritersTakeTheSharedLock (cmd/felhom-agent).
|
||||
var FileMu sync.Mutex
|
||||
+30
-1
@@ -10,6 +10,7 @@ import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -86,6 +87,7 @@ func (e *HTTPError) Error() string {
|
||||
// IS the heartbeat (locked decision 1). Errors are typed (transport vs HTTP) and
|
||||
// never include the bearer token.
|
||||
func (c *Client) Report(ctx context.Context, r *HostReport) (*ControlEnvelope, error) {
|
||||
r.SentAt = time.Now().UTC().Format(time.RFC3339Nano) // the box clock at SENDING (R-1384); see HostReport.SentAt
|
||||
body, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("hub: marshaling report: %w", err)
|
||||
@@ -285,10 +287,37 @@ func (c *Client) Jobs(ctx context.Context) ([]JobWire, error) {
|
||||
// /hosts/{id}/jobs/{job_id}, self-scoped. Called after a job is executed OR permanently rejected
|
||||
// (the nonce is already durably burned on a passing verify, so re-processing is replay-safe).
|
||||
func (c *Client) CompleteJob(ctx context.Context, jobID string) error {
|
||||
return c.CompleteJobWithOutcome(ctx, jobID, JobOutcome{})
|
||||
}
|
||||
|
||||
// JobOutcome says HOW a signed job ended when the agent clears it (R-1383, D78): Outcome is done | rejected | failed
|
||||
// | malformed; Reason is one code of signedjobs.ReasonCodes (empty for done); Op is the job's op class. It rides the
|
||||
// DELETE's query, so a hub that does not know it ignores it. Until this, „ran” and „refused” looked the same to the
|
||||
// hub, and a refused agent update left no trace outside the box's journal. No secret: no signature, no nonce, no
|
||||
// parameter of the job is in it.
|
||||
type JobOutcome struct {
|
||||
Outcome string
|
||||
Reason string
|
||||
Op string
|
||||
}
|
||||
|
||||
// CompleteJobWithOutcome clears the job and tells the hub how it ended. The zero JobOutcome sends no query.
|
||||
func (c *Client) CompleteJobWithOutcome(ctx context.Context, jobID string, o JobOutcome) error {
|
||||
if c.hostID == "" || jobID == "" {
|
||||
return fmt.Errorf("hub: CompleteJob requires host_id + job_id")
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.baseURL+"/api/v1/hosts/"+c.hostID+"/jobs/"+jobID, nil)
|
||||
u := c.baseURL + "/api/v1/hosts/" + c.hostID + "/jobs/" + jobID
|
||||
if o.Outcome != "" {
|
||||
q := url.Values{"outcome": {o.Outcome}}
|
||||
if o.Reason != "" {
|
||||
q.Set("reason", o.Reason)
|
||||
}
|
||||
if o.Op != "" {
|
||||
q.Set("op", o.Op)
|
||||
}
|
||||
u += "?" + q.Encode()
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodDelete, u, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
package hub
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1384 (D79): the hub tells a wrong box clock by the time stamped at the moment of SENDING. A report whose collect
|
||||
// started long ago (an old reported_at) must still leave with a fresh sent_at, and a second send of the SAME report
|
||||
// struct must carry a new one — or a report that waited on the box reads as a clock that is behind.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-sent-at.txt).
|
||||
func TestReport_StampsSentAtAtSending(t *testing.T) {
|
||||
var bodies []map[string]any
|
||||
c := testClient(func(r *http.Request) (*http.Response, error) {
|
||||
raw, _ := io.ReadAll(r.Body)
|
||||
var m map[string]any
|
||||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
t.Fatalf("body: %v", err)
|
||||
}
|
||||
bodies = append(bodies, m)
|
||||
return httpResp(200, `{"status":"ok"}`), nil
|
||||
})
|
||||
old := time.Now().Add(-40 * time.Minute).UTC()
|
||||
rep := &HostReport{HostID: "h", ReportedAt: old.Format(time.RFC3339), SentAt: old.Format(time.RFC3339Nano)}
|
||||
for i := 0; i < 2; i++ {
|
||||
if _, err := c.Report(context.Background(), rep); err != nil {
|
||||
t.Fatalf("Report: %v", err)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
var sent []time.Time
|
||||
for i, m := range bodies {
|
||||
s, _ := m["sent_at"].(string)
|
||||
at, err := time.Parse(time.RFC3339Nano, s)
|
||||
if err != nil {
|
||||
t.Fatalf("send %d: sent_at = %q, want an RFC3339 time", i, s)
|
||||
}
|
||||
if d := time.Since(at); d < 0 || d > 5*time.Second {
|
||||
t.Errorf("send %d: sent_at is %s old, want the moment of sending (reported_at was 40 minutes old)", i, d)
|
||||
}
|
||||
if m["reported_at"] != old.Format(time.RFC3339) {
|
||||
t.Errorf("send %d: reported_at changed to %v — the collect time is not the send time's to touch", i, m["reported_at"])
|
||||
}
|
||||
sent = append(sent, at)
|
||||
}
|
||||
if len(sent) == 2 && !sent[1].After(sent[0]) {
|
||||
t.Errorf("the second send carries %s, not after the first %s — the stamp is not taken per send", sent[1], sent[0])
|
||||
}
|
||||
}
|
||||
|
||||
// R-1383 (D78): the clearing call says how the job ended, in the DELETE's query; a plain CompleteJob sends none (an
|
||||
// older hub, and the „nothing to say” case, see the same request as before).
|
||||
func TestCompleteJob_OutcomeRidesTheQuery(t *testing.T) {
|
||||
var urls []string
|
||||
c := testClient(func(r *http.Request) (*http.Response, error) {
|
||||
if r.Method != http.MethodDelete {
|
||||
t.Errorf("method = %s", r.Method)
|
||||
}
|
||||
urls = append(urls, r.URL.Path+"?"+r.URL.RawQuery)
|
||||
return httpResp(200, `{"status":"ok"}`), nil
|
||||
})
|
||||
if err := c.CompleteJob(context.Background(), "j1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.CompleteJobWithOutcome(context.Background(), "j2", JobOutcome{Outcome: "rejected", Reason: "expired", Op: "agent_update"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []string{"/api/v1/hosts/demo-host-01/jobs/j1?", "/api/v1/hosts/demo-host-01/jobs/j2?op=agent_update&outcome=rejected&reason=expired"}
|
||||
if len(urls) != 2 || urls[0] != want[0] || urls[1] != want[1] {
|
||||
t.Fatalf("requests = %q, want %q", urls, want)
|
||||
}
|
||||
}
|
||||
+78
-6
@@ -104,6 +104,12 @@ type GuestDiskTrimReporter interface {
|
||||
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
|
||||
}
|
||||
|
||||
// LineTestReporter is the R-1247 seam the weekly line test plugs into (hub does not import linetest). nil → no
|
||||
// line_test stanza.
|
||||
type LineTestReporter interface {
|
||||
LineTestStatus(ctx context.Context) *LineTestStatus
|
||||
}
|
||||
|
||||
// Collector builds a HostReport from read-only sources. All deps are behind narrow
|
||||
// interfaces for unit testing.
|
||||
type Collector struct {
|
||||
@@ -118,12 +124,17 @@ type Collector struct {
|
||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
|
||||
pveTrust func() any // R-1380: the proxmox_trust stanza (nil = omitted)
|
||||
guestPowerGaveUp func() any // R-1383: the guest_power_gave_up stanza (nil = omitted)
|
||||
profileRefresh func() any // R-1383: the profile_refresh stanza (nil = omitted)
|
||||
addrEnum AddressEnumerator // v0.119.0: host interface enumeration; nil => the REAL one (see collectAddresses)
|
||||
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
|
||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
|
||||
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
|
||||
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
|
||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||
@@ -201,6 +212,24 @@ func (c *Collector) SetCapabilityProber(probe func(ctx context.Context) []capabi
|
||||
return c
|
||||
}
|
||||
|
||||
// SetProxmoxTrustSource wires the `proxmox_trust` stanza (R-1380). Optional; nil = the stanza is omitted.
|
||||
func (c *Collector) SetProxmoxTrustSource(f func() any) *Collector {
|
||||
c.pveTrust = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetProfileRefreshSource wires the `profile_refresh` stanza (R-1383). Optional; nil, or a nil result, omits it.
|
||||
func (c *Collector) SetProfileRefreshSource(f func() any) *Collector {
|
||||
c.profileRefresh = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetGuestPowerGaveUpSource wires the `guest_power_gave_up` stanza (R-1383). Optional; nil, or a nil result, omits it.
|
||||
func (c *Collector) SetGuestPowerGaveUpSource(f func() any) *Collector {
|
||||
c.guestPowerGaveUp = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetLeafFingerprint records the served local-API leaf fp (v0.48.0) to ride every host report (the hub
|
||||
// watches it for a re-key). Static per process — set once at startup. "" when the local API is
|
||||
// disabled. Returns the collector for chaining.
|
||||
@@ -228,6 +257,12 @@ type ControllerSupervisorReporter interface {
|
||||
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
|
||||
}
|
||||
|
||||
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
|
||||
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
|
||||
c.opResults = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
|
||||
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
|
||||
c.ctrlSup = r
|
||||
@@ -259,6 +294,12 @@ func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector
|
||||
return c
|
||||
}
|
||||
|
||||
// SetLineTestReporter wires the R-1247 weekly line test as a report source (nil-safe → stanza omitted).
|
||||
func (c *Collector) SetLineTestReporter(r LineTestReporter) *Collector {
|
||||
c.lineTest = r
|
||||
return c
|
||||
}
|
||||
|
||||
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
|
||||
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
|
||||
c.deliberateOff = f
|
||||
@@ -371,6 +412,27 @@ func (c *Collector) SetOOBReporter(o OOBReporter) *Collector {
|
||||
// error (no useful report — the cycle skips the POST); a failed per-guest
|
||||
// GuestConfig degrades that guest to status="unknown" without spec but still sends;
|
||||
// a cloudflared probe failure yields status="unknown" and is never fatal.
|
||||
func (c *Collector) profileRefreshStanza() any {
|
||||
if c.profileRefresh == nil {
|
||||
return nil
|
||||
}
|
||||
return c.profileRefresh()
|
||||
}
|
||||
|
||||
func (c *Collector) guestPowerGaveUpStanza() any {
|
||||
if c.guestPowerGaveUp == nil {
|
||||
return nil
|
||||
}
|
||||
return c.guestPowerGaveUp()
|
||||
}
|
||||
|
||||
func (c *Collector) proxmoxTrust() any {
|
||||
if c.pveTrust == nil {
|
||||
return nil
|
||||
}
|
||||
return c.pveTrust()
|
||||
}
|
||||
|
||||
func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
ns, err := c.px.NodeStatus(ctx)
|
||||
if err != nil {
|
||||
@@ -396,12 +458,15 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
RestoreTests: c.collectRestoreTests(ctx),
|
||||
PBSSnapshots: c.collectPBSSnapshots(ctx),
|
||||
|
||||
AuditTail: []AuditEntry{},
|
||||
Cloudflared: c.cloudflared(ctx),
|
||||
Capabilities: c.capabilities(ctx),
|
||||
LeafFingerprint: c.leafFP,
|
||||
Addresses: c.collectAddresses(),
|
||||
System: c.systemInfo(ctx, ns),
|
||||
AuditTail: []AuditEntry{},
|
||||
Cloudflared: c.cloudflared(ctx),
|
||||
Capabilities: c.capabilities(ctx),
|
||||
LeafFingerprint: c.leafFP,
|
||||
ProxmoxTrust: c.proxmoxTrust(),
|
||||
GuestPowerGaveUp: c.guestPowerGaveUpStanza(),
|
||||
ProfileRefresh: c.profileRefreshStanza(),
|
||||
Addresses: c.collectAddresses(),
|
||||
System: c.systemInfo(ctx, ns),
|
||||
}
|
||||
// DR recipe host-half — derived from the just-collected guest/storage/PBS facts (no new reads).
|
||||
// Secret-free by construction (identifiers/intents/sizes/coordinates only).
|
||||
@@ -419,6 +484,9 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.ctrlSup != nil {
|
||||
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
|
||||
}
|
||||
if c.opResults != nil {
|
||||
report.OperatorActionResults = c.opResults()
|
||||
}
|
||||
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
|
||||
if c.guestNet != nil {
|
||||
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
|
||||
@@ -427,6 +495,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.diskTrim != nil {
|
||||
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
|
||||
}
|
||||
// R-1247: the last weekly line test (nil reporter = not wired → stanza omitted).
|
||||
if c.lineTest != nil {
|
||||
report.LineTest = c.lineTest.LineTestStatus(ctx)
|
||||
}
|
||||
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
|
||||
if c.foreignKey != nil {
|
||||
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
|
||||
|
||||
@@ -70,8 +70,14 @@ type Loop struct {
|
||||
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
|
||||
// path waits on it so the hub has the deliberate-off record before the box goes down.
|
||||
reported func(r *HostReport)
|
||||
// collectErr (R-1380) sees every failed collect.
|
||||
collectErr func(ctx context.Context, err error)
|
||||
}
|
||||
|
||||
// SetCollectErrorHook wires an observer of a failed collect (R-1380: a call to Proxmox that fails on TRUST starts the
|
||||
// move to the CA copy, or a reload of a refreshed copy). Optional; it must not block for long.
|
||||
func (l *Loop) SetCollectErrorHook(f func(ctx context.Context, err error)) { l.collectErr = f }
|
||||
|
||||
// SetReportedHook wires the R-998 after-report observer. Optional.
|
||||
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
|
||||
|
||||
@@ -144,6 +150,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
|
||||
report, err := l.collector.Collect(ctx)
|
||||
if err != nil {
|
||||
l.logger.Warn("hub: collect failed; skipping this cycle's report", "err", err)
|
||||
if l.collectErr != nil {
|
||||
l.collectErr(ctx, err)
|
||||
}
|
||||
return current
|
||||
}
|
||||
// Fulfill a pending log-pull: attach the ring tail to THIS report and clear the
|
||||
|
||||
+70
-2
@@ -15,8 +15,13 @@ import (
|
||||
// server-side received_at is the hub's dead-man's-switch liveness signal. There is
|
||||
// no separate heartbeat endpoint.
|
||||
type HostReport struct {
|
||||
HostID string `json:"host_id"` // echoes config.Hub.HostID
|
||||
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
|
||||
HostID string `json:"host_id"` // echoes config.Hub.HostID
|
||||
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
|
||||
// SentAt (R-1384, D79) is this box's clock at the moment the report is SENT — stamped by Client.Report, never by
|
||||
// the collector. The hub compares it with its own clock to tell a wrong box clock. ReportedAt cannot serve: it is
|
||||
// stamped when the collect STARTS, and a collect that waits on Proxmox makes it old on a box whose clock is right.
|
||||
// Pinned by TestReport_StampsSentAtAtSending.
|
||||
SentAt string `json:"sent_at,omitempty"`
|
||||
AgentVersion string `json:"agent_version"`
|
||||
// AgentSHA256 is the sha256 of the binary this process is RUNNING (read through /proc/self/exe,
|
||||
// once per process), R-349. The version string cannot tell a hand-built proof binary from the
|
||||
@@ -53,6 +58,11 @@ type HostReport struct {
|
||||
// alert. Not a secret (the fp is public; the token is never reported).
|
||||
LeafFingerprint string `json:"leaf_fingerprint"`
|
||||
|
||||
// ProxmoxTrust (R-1380, felhom.eu `03` §12a): how the agent trusts Proxmox's certificate (`mode`: ca | fingerprint |
|
||||
// …), and the CAUSE when something about it is wrong (`problem`, `detail`): the proof with the CA copy failed, the
|
||||
// copy ends soon, the root step gave no copy. Omitted when not wired. No secret: a mode, a cause, a date.
|
||||
ProxmoxTrust any `json:"proxmox_trust,omitempty"`
|
||||
|
||||
// Addresses are the host's routable addresses, one entry per (interface, address) — the LAN
|
||||
// bridge, the WG tunnel, a tailnet. Added v0.119.0 because the hub could not show a managed
|
||||
// box's IP anywhere: nothing in this report carried one. Non-nil so it marshals as [];
|
||||
@@ -129,6 +139,11 @@ type HostReport struct {
|
||||
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
|
||||
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
|
||||
|
||||
// LineTest is the weekly upload line test stanza (R-1247; operator ruling 2026-10-24): the schedule, the last
|
||||
// measured upload line (best of the runs, Mbit/s, as ep0's iperf3 server counted it) and the last attempt. Present
|
||||
// only when the job is wired. No secret.
|
||||
LineTest *LineTestStatus `json:"line_test,omitempty"`
|
||||
|
||||
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
|
||||
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
|
||||
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
|
||||
@@ -167,6 +182,32 @@ type HostReport struct {
|
||||
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
|
||||
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
|
||||
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
|
||||
// ProfileRefresh (R-1383, release 3): the box's last security-profile refresh and how the agent's last ask ended
|
||||
// (osupdate.ProfileRefreshStatus, field-exact; hub sysfacts.ProfileRefresh). `any` + omitempty: absent when there
|
||||
// is nothing to say or the source is not wired.
|
||||
ProfileRefresh any `json:"profile_refresh,omitempty"`
|
||||
// GuestPowerGaveUp (R-1383, D78): the guests the guest-power watchdog stopped trying to start (localapi
|
||||
// GuestPowerGiveUp, field-exact: vmid, since, attempts). The hub raises one operator alarm per guest and clears it
|
||||
// when the guest runs. `any` + omitempty: absent when there is none or the source is not wired.
|
||||
GuestPowerGaveUp any `json:"guest_power_gave_up,omitempty"`
|
||||
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
|
||||
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
|
||||
// when there is nothing to answer, so the golden stays byte-stable.
|
||||
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
|
||||
}
|
||||
|
||||
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
|
||||
type OperatorAction struct {
|
||||
ID int64 `json:"id"`
|
||||
Action string `json:"action"`
|
||||
Arg string `json:"arg,omitempty"`
|
||||
}
|
||||
|
||||
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
|
||||
type OperatorActionResult struct {
|
||||
ID int64 `json:"id"`
|
||||
Outcome string `json:"outcome"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
|
||||
@@ -190,6 +231,17 @@ type ControllerSupervisorGuest struct {
|
||||
Restarts24h int `json:"restarts_24h"`
|
||||
SlowCrashloop bool `json:"slow_crashloop"`
|
||||
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
|
||||
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
|
||||
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
|
||||
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
|
||||
Unresponsive bool `json:"unresponsive"`
|
||||
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
|
||||
// R-1344 restart (decided 2026-10-30 from the night's measurement): UnresponsiveRestartAfterS is how long after the
|
||||
// alarm the agent restarts a controller that still does not answer (0 / absent = an older agent: alarm only — the
|
||||
// hub's message says which). UnresponsiveRestartAt is the last such restart; the hub keys on it MOVING
|
||||
// (controller_unresponsive_restart) and it equals LastRestartAt for that act.
|
||||
UnresponsiveRestartAfterS int `json:"unresponsive_restart_after_s,omitempty"`
|
||||
UnresponsiveRestartAt string `json:"unresponsive_restart_at,omitempty"` // RFC3339
|
||||
}
|
||||
|
||||
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
||||
@@ -232,6 +284,19 @@ type GuestDiskTrimStatus struct {
|
||||
Guests []GuestDiskTrim `json:"guests,omitempty"`
|
||||
}
|
||||
|
||||
// LineTestStatus is the R-1247 weekly upload line test stanza. `last_ok_at` + `mbps` are the last SUCCESSFUL
|
||||
// measurement ("" / 0 = never); `last_attempt_at` + `last_error` the last attempt, whatever its verdict — never read
|
||||
// the attempt time as a result.
|
||||
type LineTestStatus struct {
|
||||
Schedule string `json:"schedule"`
|
||||
LastOKAt string `json:"last_ok_at,omitempty"`
|
||||
Mbps float64 `json:"mbps,omitempty"`
|
||||
RunsMbps []float64 `json:"runs_mbps,omitempty"`
|
||||
Server string `json:"server,omitempty"`
|
||||
LastAttemptAt string `json:"last_attempt_at,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
|
||||
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
|
||||
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
|
||||
@@ -613,6 +678,9 @@ type ControlEnvelope struct {
|
||||
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
|
||||
// Absent/false on an old hub → nothing happens.
|
||||
LogTailRequested bool `json:"log_tail_requested"`
|
||||
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
|
||||
// (`04` §6.1). Absent on an older hub.
|
||||
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
|
||||
}
|
||||
|
||||
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
|
||||
|
||||
@@ -0,0 +1,294 @@
|
||||
package linetest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"strconv"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
// A minimal iperf3 CLIENT for one TCP upload test (client → server), speaking the iperf3 control protocol (3.x) to the
|
||||
// stock `iperf3 --server` on ep0 (operator ruling 2026-10-24: the agent speaks iperf3 itself, nothing is installed on
|
||||
// the host). The number that counts is the SERVER's: the bytes it received, from its results JSON — bytes the client
|
||||
// handed to its socket include what still sat in the send buffer when the timer ended.
|
||||
//
|
||||
// The protocol, as the server drives it (one state byte at a time on the control connection):
|
||||
//
|
||||
// client → 37-byte cookie
|
||||
// server → PARAM_EXCHANGE(9) client → params JSON (4-byte big-endian length + bytes)
|
||||
// server → CREATE_STREAMS(10) client opens N data connections, each starts with the same cookie
|
||||
// server → TEST_START(1), TEST_RUNNING(2) client sends for the test time, then → TEST_END(4)
|
||||
// server → EXCHANGE_RESULTS(13) client → its results JSON; server → its results JSON
|
||||
// server → DISPLAY_RESULTS(14) client → IPERF_DONE(16)
|
||||
//
|
||||
// A server already running a test answers ACCESS_DENIED(-1) and closes: that is ErrBusy (another box is testing; try
|
||||
// again later the same day). SERVER_ERROR(-3) / SERVER_TERMINATE(-2) end the test with an error.
|
||||
|
||||
const (
|
||||
stParamExchange = 9
|
||||
stCreateStreams = 10
|
||||
stTestStart = 1
|
||||
stTestRunning = 2
|
||||
stTestEnd = 4
|
||||
stExchangeResults = 13
|
||||
stDisplayResults = 14
|
||||
stIperfDone = 16
|
||||
stAccessDenied = -1
|
||||
stServerTerminate = -2
|
||||
stServerError = -3
|
||||
|
||||
cookieSize = 37 // 36 characters + NUL
|
||||
blockSize = 128 * 1024
|
||||
maxJSONBytes = 1 << 20
|
||||
)
|
||||
|
||||
// ErrBusy: the server is running another client's test.
|
||||
var ErrBusy = errors.New("iperf3 server busy")
|
||||
|
||||
// Result is one upload test.
|
||||
type Result struct {
|
||||
Mbps float64 // the server's received bits per second / 1e6
|
||||
ServerBytes int64 // bytes the server received, all streams
|
||||
ClientBytes int64 // bytes the client wrote (diagnostics only)
|
||||
ServerSeconds float64 // the server's measured duration
|
||||
}
|
||||
|
||||
// Dialer opens a TCP connection (net.Dialer in production; a seam for tests).
|
||||
type Dialer interface {
|
||||
DialContext(ctx context.Context, network, address string) (net.Conn, error)
|
||||
}
|
||||
|
||||
// Client runs one test. The whole run is bounded by ctx; Run sets every connection's deadline from it.
|
||||
type Client struct {
|
||||
Dialer Dialer
|
||||
Streams int // parallel streams (-P)
|
||||
Time time.Duration // test time (-t)
|
||||
}
|
||||
|
||||
func (c Client) dialer() Dialer {
|
||||
if c.Dialer != nil {
|
||||
return c.Dialer
|
||||
}
|
||||
return &net.Dialer{Timeout: 5 * time.Second}
|
||||
}
|
||||
|
||||
func newCookie() ([]byte, error) {
|
||||
const alphabet = "abcdefghijklmnopqrstuvwxyz234567"
|
||||
b := make([]byte, cookieSize)
|
||||
if _, err := rand.Read(b[:cookieSize-1]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for i := 0; i < cookieSize-1; i++ {
|
||||
b[i] = alphabet[int(b[i])%len(alphabet)]
|
||||
}
|
||||
b[cookieSize-1] = 0
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func writeJSON(w io.Writer, v any) error {
|
||||
data, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var hdr [4]byte
|
||||
binary.BigEndian.PutUint32(hdr[:], uint32(len(data)))
|
||||
if _, err := w.Write(append(hdr[:], data...)); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func readJSON(r io.Reader, v any) error {
|
||||
var hdr [4]byte
|
||||
if _, err := io.ReadFull(r, hdr[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
n := binary.BigEndian.Uint32(hdr[:])
|
||||
if n == 0 || n > maxJSONBytes {
|
||||
return fmt.Errorf("iperf3: results JSON of %d bytes refused", n)
|
||||
}
|
||||
buf := make([]byte, n)
|
||||
if _, err := io.ReadFull(r, buf); err != nil {
|
||||
return err
|
||||
}
|
||||
return json.Unmarshal(buf, v)
|
||||
}
|
||||
|
||||
type streamResult struct {
|
||||
ID int `json:"id"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Retransmits int64 `json:"retransmits"`
|
||||
Jitter float64 `json:"jitter"`
|
||||
Errors int64 `json:"errors"`
|
||||
OmittedErrors int64 `json:"omitted_errors"`
|
||||
Packets int64 `json:"packets"`
|
||||
OmittedPackets int64 `json:"omitted_packets"`
|
||||
StartTime float64 `json:"start_time"`
|
||||
EndTime float64 `json:"end_time"`
|
||||
}
|
||||
|
||||
type results struct {
|
||||
CPUTotal float64 `json:"cpu_util_total"`
|
||||
CPUUser float64 `json:"cpu_util_user"`
|
||||
CPUSystem float64 `json:"cpu_util_system"`
|
||||
SenderHasRetransmits int `json:"sender_has_retransmits"`
|
||||
CongestionUsed string `json:"congestion_used,omitempty"`
|
||||
Streams []streamResult `json:"streams"`
|
||||
}
|
||||
|
||||
// streamIDs are the ids iperf3 gives its streams: the first is 1, every later one is (streams so far) + 2 — so 1, 3,
|
||||
// 4, 5 … (iperf_add_stream). The server matches our results by these ids.
|
||||
func streamIDs(n int) []int {
|
||||
ids := make([]int, n)
|
||||
for i := range ids {
|
||||
if i == 0 {
|
||||
ids[i] = 1
|
||||
} else {
|
||||
ids[i] = i + 2
|
||||
}
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// Run performs one upload test to addr ("10.77.0.1:5201").
|
||||
func (c Client) Run(ctx context.Context, addr string) (Result, error) {
|
||||
var res Result
|
||||
streams, dur := c.Streams, c.Time
|
||||
if streams < 1 {
|
||||
streams = 1
|
||||
}
|
||||
if dur <= 0 {
|
||||
dur = 10 * time.Second
|
||||
}
|
||||
deadline, ok := ctx.Deadline()
|
||||
if !ok {
|
||||
return res, errors.New("iperf3: Run needs a context with a deadline")
|
||||
}
|
||||
cookie, err := newCookie()
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
ctrl, err := c.dialer().DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("iperf3: connect: %w", err)
|
||||
}
|
||||
defer ctrl.Close()
|
||||
ctrl.SetDeadline(deadline)
|
||||
if _, err := ctrl.Write(cookie); err != nil {
|
||||
return res, fmt.Errorf("iperf3: cookie: %w", err)
|
||||
}
|
||||
|
||||
var data []net.Conn
|
||||
defer func() {
|
||||
for _, d := range data {
|
||||
d.Close()
|
||||
}
|
||||
}()
|
||||
var sent atomic.Int64
|
||||
var server results
|
||||
var wg sync.WaitGroup
|
||||
for {
|
||||
var st [1]byte
|
||||
if _, err := io.ReadFull(ctrl, st[:]); err != nil {
|
||||
return res, fmt.Errorf("iperf3: control read: %w", err)
|
||||
}
|
||||
switch int8(st[0]) {
|
||||
case stParamExchange:
|
||||
params := map[string]any{"tcp": true, "omit": 0, "time": int(dur / time.Second), "num": 0, "blockcount": 0,
|
||||
"parallel": streams, "len": blockSize, "pacing_timer": 1000, "client_version": "3.18"}
|
||||
if err := writeJSON(ctrl, params); err != nil {
|
||||
return res, fmt.Errorf("iperf3: params: %w", err)
|
||||
}
|
||||
case stCreateStreams:
|
||||
for i := 0; i < streams; i++ {
|
||||
d, err := c.dialer().DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("iperf3: data stream %d: %w", i+1, err)
|
||||
}
|
||||
d.SetDeadline(deadline)
|
||||
data = append(data, d)
|
||||
if _, err := d.Write(cookie); err != nil {
|
||||
return res, fmt.Errorf("iperf3: data stream %d cookie: %w", i+1, err)
|
||||
}
|
||||
}
|
||||
case stTestStart:
|
||||
case stTestRunning:
|
||||
stopAt := time.Now().Add(dur)
|
||||
if stopAt.After(deadline) {
|
||||
return res, errors.New("iperf3: no time left for the test inside the deadline")
|
||||
}
|
||||
buf := make([]byte, blockSize)
|
||||
for _, d := range data {
|
||||
d.SetWriteDeadline(stopAt)
|
||||
wg.Add(1)
|
||||
go func(d net.Conn) {
|
||||
defer wg.Done()
|
||||
for {
|
||||
n, err := d.Write(buf)
|
||||
sent.Add(int64(n))
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}(d)
|
||||
}
|
||||
wg.Wait()
|
||||
if _, err := ctrl.Write([]byte{stTestEnd}); err != nil {
|
||||
return res, fmt.Errorf("iperf3: test end: %w", err)
|
||||
}
|
||||
case stExchangeResults:
|
||||
ids := streamIDs(len(data))
|
||||
mine := results{SenderHasRetransmits: -1, CongestionUsed: "cubic"}
|
||||
per := sent.Load() / int64(max(len(data), 1))
|
||||
for _, id := range ids {
|
||||
mine.Streams = append(mine.Streams, streamResult{ID: id, Bytes: per, Retransmits: -1, EndTime: dur.Seconds()})
|
||||
}
|
||||
if err := writeJSON(ctrl, mine); err != nil {
|
||||
return res, fmt.Errorf("iperf3: send results: %w", err)
|
||||
}
|
||||
if err := readJSON(ctrl, &server); err != nil {
|
||||
return res, fmt.Errorf("iperf3: read server results: %w", err)
|
||||
}
|
||||
case stDisplayResults:
|
||||
ctrl.Write([]byte{stIperfDone})
|
||||
return summarize(server, sent.Load(), dur)
|
||||
case stAccessDenied:
|
||||
return res, ErrBusy
|
||||
case stServerTerminate, stServerError:
|
||||
var codes [8]byte
|
||||
io.ReadFull(ctrl, codes[:])
|
||||
return res, fmt.Errorf("iperf3: server error (state %d, iperf errno %d, errno %d)", int8(st[0]),
|
||||
int32(binary.BigEndian.Uint32(codes[:4])), int32(binary.BigEndian.Uint32(codes[4:])))
|
||||
default:
|
||||
return res, fmt.Errorf("iperf3: unexpected state %s", strconv.Itoa(int(int8(st[0]))))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// summarize turns the server's results into a Result. No stream bytes is an error, never a zero line.
|
||||
func summarize(server results, clientBytes int64, dur time.Duration) (Result, error) {
|
||||
res := Result{ClientBytes: clientBytes}
|
||||
for _, s := range server.Streams {
|
||||
res.ServerBytes += s.Bytes
|
||||
if s.EndTime > res.ServerSeconds {
|
||||
res.ServerSeconds = s.EndTime
|
||||
}
|
||||
}
|
||||
if res.ServerBytes <= 0 {
|
||||
return res, errors.New("iperf3: the server reported no received bytes")
|
||||
}
|
||||
secs := res.ServerSeconds
|
||||
if secs <= 0 {
|
||||
secs = dur.Seconds()
|
||||
}
|
||||
res.Mbps = float64(res.ServerBytes) * 8 / secs / 1e6
|
||||
return res, nil
|
||||
}
|
||||
@@ -0,0 +1,541 @@
|
||||
// Package linetest is the weekly upload line-speed test (R-1247; operator ruling 2026-10-24 07:26).
|
||||
//
|
||||
// Why: the controller's upload watch and qBittorrent's default limit (50 % of the line, D41) need the household's
|
||||
// upload line. The passive reading — the nightly off-site push — is only a low floor (2026-10-24 night: demo-hp „at
|
||||
// least 10 Mbit/s"; this test read 90.7 on the same line). So once a week the HOST (the tunnel lives on the host)
|
||||
// runs a 10-second iperf3 upload to ep0's tunnel-only iperf3 server (felhom.eu scripts/ep0-iperf3). No file is left
|
||||
// anywhere and nothing counts against the household's storage.
|
||||
//
|
||||
// The rule, each part pinned by a test in linetest_test.go:
|
||||
// - Weekly: due from Tuesday 10:00 local until one measurement succeeded since then (a box that was off catches up
|
||||
// on a later day, inside the window).
|
||||
// - Daytime only: a test starts only 10:00–15:59 local — never in the night chain's window (01:00–06:59), and each
|
||||
// host starts from its own minute (SlotMinute, spread over 10:00–13:59) so the fleet does not test at once.
|
||||
// - Never beside a backup, a restore-test, an OS update or a trim: the test holds the host-wide one-heavy-op gate.
|
||||
// - Only after QuietMinutes of quiet uplink: every one-minute transmit rate of the host's physical uplink below
|
||||
// QuietMbps — so neither the family's upload nor a restore is measured against itself.
|
||||
// - Up to RunsPerMeasurement runs of TestTime each; the highest counts. A run is bounded by RunTimeout.
|
||||
// - „Server busy" (another box is testing) → tried again at the next tick the same day; at most MaxTriesPerDay.
|
||||
// - No tunnel (no operational wg-felhom with a server address) → no test and no value; the controller then keeps its
|
||||
// passive floor and sets no qBittorrent limit.
|
||||
// - The last result is persisted, so a restart neither loses it nor re-tests.
|
||||
// - A box whose torrent app seeds all day is never quiet for 10 minutes (R-1250). For it the controller holds the
|
||||
// app's upload limit very low and asks for a test on request (RunNow): every rule above except the 10 quiet
|
||||
// minutes still holds, the quiet is checked over RequestQuietWindow instead, and the runs are RequestRunGap
|
||||
// apart, so the whole request ends in about 55 s.
|
||||
package linetest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"hash/fnv"
|
||||
"log/slog"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
const (
|
||||
Weekday = time.Tuesday
|
||||
StartHour = 10 // first eligible local hour (inclusive)
|
||||
EndHour = 16 // first NOT-eligible local hour: the last start is 15:59
|
||||
SlotSpreadMinutes = 240
|
||||
QuietMinutes = 10
|
||||
QuietMbps = 2.0
|
||||
RunsPerMeasurement = 3
|
||||
TestTime = 10 * time.Second
|
||||
Streams = 4
|
||||
RunTimeout = 25 * time.Second
|
||||
RunGap = 30 * time.Second
|
||||
MaxTriesPerDay = 6
|
||||
ServerPort = 5201
|
||||
TickInterval = time.Minute
|
||||
FirstTickDelay = 5 * time.Minute
|
||||
GateName = "line-test"
|
||||
|
||||
// A test on request (R-1250): the controller holds the app's limit while it waits, so it is short.
|
||||
RequestSettle = 5 * time.Second // the held app's last packets leave before the quiet is read
|
||||
RequestQuietWindow = 10 * time.Second // the uplink must send below QuietMbps over this window
|
||||
RequestRunGap = 5 * time.Second
|
||||
)
|
||||
|
||||
// ScheduleText is carried on the host report.
|
||||
const ScheduleText = "weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update"
|
||||
|
||||
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
|
||||
type Gate interface {
|
||||
TryAcquire(what string) (release func(), busy string, ok bool)
|
||||
}
|
||||
|
||||
// Target answers the iperf3 server's tunnel address, ok=false when the tunnel is not operational.
|
||||
type Target func() (netip.Addr, bool)
|
||||
|
||||
// Runner runs one upload test (Client in production).
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, addr string) (Result, error)
|
||||
}
|
||||
|
||||
// UplinkTx reads the host's physical-uplink transmit byte counter.
|
||||
type UplinkTx func() (uint64, error)
|
||||
|
||||
// State is what is persisted (line-test.json).
|
||||
type State struct {
|
||||
LastOKAt time.Time `json:"last_ok_at,omitempty"`
|
||||
Mbps float64 `json:"mbps,omitempty"`
|
||||
RunsMbps []float64 `json:"runs_mbps,omitempty"`
|
||||
Server string `json:"server,omitempty"`
|
||||
LastAttemptAt time.Time `json:"last_attempt_at,omitempty"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
TriesDay string `json:"tries_day,omitempty"` // local date of TriesToday
|
||||
TriesToday int `json:"tries_today,omitempty"`
|
||||
}
|
||||
|
||||
type txSample struct {
|
||||
at time.Time
|
||||
tx uint64
|
||||
}
|
||||
|
||||
// Tester is the weekly job.
|
||||
type Tester struct {
|
||||
runner Runner
|
||||
target Target
|
||||
gate Gate
|
||||
uplink UplinkTx
|
||||
statePath string
|
||||
slot int // minutes after StartHour:00 this host may first start
|
||||
logger *slog.Logger
|
||||
loc *time.Location
|
||||
now func() time.Time
|
||||
sleep func(context.Context, time.Duration) bool
|
||||
|
||||
mu sync.Mutex
|
||||
st State
|
||||
samples []txSample
|
||||
}
|
||||
|
||||
// New builds the job and loads the persisted state (missing = empty; corrupt = logged, empty — the cost is one extra
|
||||
// test, never a lost value elsewhere).
|
||||
func New(runner Runner, target Target, gate Gate, uplink UplinkTx, hostID, statePath string, logger *slog.Logger) *Tester {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
t := &Tester{runner: runner, target: target, gate: gate, uplink: uplink, statePath: statePath,
|
||||
slot: SlotMinute(hostID), logger: logger, loc: time.Local, now: time.Now, sleep: sleepCtx}
|
||||
t.load()
|
||||
return t
|
||||
}
|
||||
|
||||
func sleepCtx(ctx context.Context, d time.Duration) bool {
|
||||
tm := time.NewTimer(d)
|
||||
defer tm.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
case <-tm.C:
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// SlotMinute spreads hosts over the first SlotSpreadMinutes of the window, stable per host id.
|
||||
func SlotMinute(hostID string) int {
|
||||
h := fnv.New32a()
|
||||
h.Write([]byte(hostID))
|
||||
return int(h.Sum32() % SlotSpreadMinutes)
|
||||
}
|
||||
|
||||
func (t *Tester) load() {
|
||||
data, err := os.ReadFile(t.statePath)
|
||||
if err != nil {
|
||||
if !os.IsNotExist(err) {
|
||||
t.logger.Warn("linetest: state read failed — starting empty", "path", t.statePath, "err", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err := json.Unmarshal(data, &t.st); err != nil {
|
||||
t.logger.Warn("linetest: state file corrupt — starting empty", "path", t.statePath, "err", err)
|
||||
t.st = State{}
|
||||
}
|
||||
}
|
||||
|
||||
func (t *Tester) saveLocked() error {
|
||||
data, err := json.MarshalIndent(t.st, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := t.statePath + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
os.Remove(tmp)
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, t.statePath)
|
||||
}
|
||||
|
||||
// EligibleAt reports whether a test may START at local time lt for a host whose slot is slot minutes after
|
||||
// StartHour:00 on the due day. On a catch-up day (anchorDay=false) the slot is not waited for again.
|
||||
func EligibleAt(lt time.Time, slot int, anchorDay bool) bool {
|
||||
h := lt.Hour()
|
||||
if h < StartHour || h >= EndHour {
|
||||
return false
|
||||
}
|
||||
if anchorDay {
|
||||
mins := (h-StartHour)*60 + lt.Minute()
|
||||
return mins >= slot
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// weekAnchor is the most recent Tuesday StartHour:00 at or before lt.
|
||||
func weekAnchor(lt time.Time) time.Time {
|
||||
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
|
||||
d := lt.AddDate(0, 0, -daysBack)
|
||||
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
|
||||
if a.After(lt) {
|
||||
d = d.AddDate(0, 0, -7)
|
||||
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
|
||||
}
|
||||
return a
|
||||
}
|
||||
|
||||
// Due reports whether a measurement is due at lt given the state.
|
||||
func Due(st State, lt time.Time) bool {
|
||||
return st.LastOKAt.IsZero() || st.LastOKAt.In(lt.Location()).Before(weekAnchor(lt))
|
||||
}
|
||||
|
||||
// sample records the uplink counter (once a minute) and keeps QuietMinutes+1 samples.
|
||||
func (t *Tester) sample() {
|
||||
if t.uplink == nil {
|
||||
return
|
||||
}
|
||||
tx, err := t.uplink()
|
||||
if err != nil {
|
||||
t.logger.Debug("linetest: uplink counter unreadable", "err", err)
|
||||
t.mu.Lock()
|
||||
t.samples = nil // a gap is not quiet
|
||||
t.mu.Unlock()
|
||||
return
|
||||
}
|
||||
t.mu.Lock()
|
||||
t.samples = append(t.samples, txSample{at: t.now(), tx: tx})
|
||||
if len(t.samples) > QuietMinutes+1 {
|
||||
t.samples = t.samples[len(t.samples)-(QuietMinutes+1):]
|
||||
}
|
||||
t.mu.Unlock()
|
||||
}
|
||||
|
||||
// quiet reports whether the last QuietMinutes minutes each sent below QuietMbps, and the highest minute seen.
|
||||
func quiet(samples []txSample) (bool, float64) {
|
||||
if len(samples) < QuietMinutes+1 {
|
||||
return false, 0
|
||||
}
|
||||
worst := 0.0
|
||||
for i := 1; i < len(samples); i++ {
|
||||
dt := samples[i].at.Sub(samples[i-1].at).Seconds()
|
||||
if dt <= 0 || dt > 3*TickInterval.Seconds() || samples[i].tx < samples[i-1].tx {
|
||||
return false, worst // a gap or a counter reset is not proof of quiet
|
||||
}
|
||||
mbps := float64(samples[i].tx-samples[i-1].tx) * 8 / dt / 1e6
|
||||
if mbps > worst {
|
||||
worst = mbps
|
||||
}
|
||||
}
|
||||
return worst < QuietMbps, worst
|
||||
}
|
||||
|
||||
// Run ticks every TickInterval until ctx ends. It never returns an error: a failed test is a reported fact.
|
||||
func (t *Tester) Run(ctx context.Context) {
|
||||
t.logger.Info("linetest: weekly upload line test starting", "schedule", ScheduleText, "slot_minute", t.slot)
|
||||
if !t.sleep(ctx, FirstTickDelay) {
|
||||
return
|
||||
}
|
||||
for {
|
||||
t.sample()
|
||||
t.Tick(ctx)
|
||||
if !t.sleep(ctx, TickInterval) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Refusal is why a test on request did not run (RunNow). Code is stable for the controller.
|
||||
type Refusal struct {
|
||||
Code string // not-due | outside-window | tries-used | no-tunnel | heavy-op | not-quiet
|
||||
Msg string
|
||||
}
|
||||
|
||||
func (r *Refusal) Error() string { return "linetest: " + r.Code + ": " + r.Msg }
|
||||
|
||||
// ready applies every rule except the quiet one: due this week, inside the window (the slot on the due day), tries
|
||||
// left today, the tunnel up. It answers the server address and the local day.
|
||||
func (t *Tester) ready() (addr, day string, refusal *Refusal) {
|
||||
lt := t.now().In(t.loc)
|
||||
t.mu.Lock()
|
||||
st := t.st
|
||||
t.mu.Unlock()
|
||||
if !Due(st, lt) {
|
||||
return "", "", &Refusal{"not-due", "measured this week"}
|
||||
}
|
||||
anchor := weekAnchor(lt)
|
||||
anchorDay := lt.YearDay() == anchor.YearDay() && lt.Year() == anchor.Year()
|
||||
if !EligibleAt(lt, t.slot, anchorDay) {
|
||||
return "", "", &Refusal{"outside-window", "a test starts 10:00-15:59 local, from this host's slot on the due day"}
|
||||
}
|
||||
day = lt.Format("2006-01-02")
|
||||
if st.TriesDay == day && st.TriesToday >= MaxTriesPerDay {
|
||||
return "", "", &Refusal{"tries-used", fmt.Sprintf("%d tries today", st.TriesToday)}
|
||||
}
|
||||
ip, ok := t.target()
|
||||
if !ok {
|
||||
return "", "", &Refusal{"no-tunnel", "no operational tunnel"}
|
||||
}
|
||||
return netip.AddrPortFrom(ip, ServerPort).String(), day, nil
|
||||
}
|
||||
|
||||
// Tick is one look.
|
||||
func (t *Tester) Tick(ctx context.Context) {
|
||||
addr, day, refusal := t.ready()
|
||||
if refusal != nil {
|
||||
if refusal.Code == "no-tunnel" {
|
||||
t.logger.Debug("linetest: no operational tunnel — no test")
|
||||
}
|
||||
return
|
||||
}
|
||||
t.mu.Lock()
|
||||
samples := append([]txSample(nil), t.samples...)
|
||||
t.mu.Unlock()
|
||||
if q, worst := quiet(samples); !q {
|
||||
t.logger.Debug("linetest: uplink not quiet — waiting", "worst_minute_mbps", fmt.Sprintf("%.1f", worst), "samples", len(samples))
|
||||
return
|
||||
}
|
||||
release, busy, ok := t.gate.TryAcquire(GateName)
|
||||
if !ok {
|
||||
t.logger.Info("linetest: deferred — a heavy operation is in flight", "busy", busy)
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
t.measure(ctx, addr, day, RunGap)
|
||||
}
|
||||
|
||||
// WaitingForQuiet reports that a measurement is due and could start now but the uplink has not been quiet for
|
||||
// QuietMinutes — the case a seeding torrent app makes permanent (R-1250). The controller reads it from
|
||||
// GET /host/line-test and only then holds its app's limit and calls RunNow.
|
||||
func (t *Tester) WaitingForQuiet() bool {
|
||||
if _, _, refusal := t.ready(); refusal != nil {
|
||||
return false
|
||||
}
|
||||
t.mu.Lock()
|
||||
samples := append([]txSample(nil), t.samples...)
|
||||
t.mu.Unlock()
|
||||
if len(samples) < QuietMinutes+1 {
|
||||
return false // not yet watched for 10 minutes (a fresh start): not known to be busy, so no hold
|
||||
}
|
||||
q, _ := quiet(samples)
|
||||
return !q
|
||||
}
|
||||
|
||||
// RunNow is a test on request (POST /host/line-test/run, R-1250): every rule of the weekly test except the 10 quiet
|
||||
// minutes, which the caller replaces by holding its own upload down. The uplink must still be quiet over
|
||||
// RequestQuietWindow (after RequestSettle), so a request never measures the family's upload against itself. A
|
||||
// *Refusal says why nothing ran; any other error is a failed measurement (the same as a failed weekly attempt).
|
||||
func (t *Tester) RunNow(ctx context.Context) (mbps float64, runs []float64, err error) {
|
||||
addr, day, refusal := t.ready()
|
||||
if refusal != nil {
|
||||
return 0, nil, refusal
|
||||
}
|
||||
release, busy, ok := t.gate.TryAcquire(GateName)
|
||||
if !ok {
|
||||
return 0, nil, &Refusal{"heavy-op", busy}
|
||||
}
|
||||
defer release()
|
||||
if t.uplink == nil {
|
||||
return 0, nil, &Refusal{"not-quiet", "no uplink counter"}
|
||||
}
|
||||
if !t.sleep(ctx, RequestSettle) {
|
||||
return 0, nil, ctx.Err()
|
||||
}
|
||||
tx0, err0 := t.uplink()
|
||||
at0 := t.now()
|
||||
if !t.sleep(ctx, RequestQuietWindow) {
|
||||
return 0, nil, ctx.Err()
|
||||
}
|
||||
tx1, err1 := t.uplink()
|
||||
if err0 != nil || err1 != nil || tx1 < tx0 {
|
||||
return 0, nil, &Refusal{"not-quiet", "uplink counter unreadable or reset"}
|
||||
}
|
||||
secs := t.now().Sub(at0).Seconds()
|
||||
if secs <= 0 {
|
||||
secs = RequestQuietWindow.Seconds()
|
||||
}
|
||||
if m := float64(tx1-tx0) * 8 / secs / 1e6; m >= QuietMbps {
|
||||
t.logger.Info("linetest: test on request refused — the uplink is not quiet", "mbps", fmt.Sprintf("%.1f", m))
|
||||
return 0, nil, &Refusal{"not-quiet", fmt.Sprintf("the uplink sent %.1f Mbit/s while held", m)}
|
||||
}
|
||||
t.logger.Info("linetest: test on request (a held torrent app, R-1250)", "server", addr)
|
||||
if err := t.measure(ctx, addr, day, RequestRunGap); err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
return t.st.Mbps, append([]float64(nil), t.st.RunsMbps...), nil
|
||||
}
|
||||
|
||||
// measure runs the runs, gap apart, and stores the result. It returns nil only when a value was stored.
|
||||
func (t *Tester) measure(ctx context.Context, addr, day string, gap time.Duration) error {
|
||||
start := t.now()
|
||||
var runs []float64
|
||||
var lastErr error
|
||||
busy := false
|
||||
for i := 0; i < RunsPerMeasurement; i++ {
|
||||
if i > 0 && !t.sleep(ctx, gap) {
|
||||
break
|
||||
}
|
||||
rctx, cancel := context.WithTimeout(ctx, RunTimeout)
|
||||
r, err := t.runner.Run(rctx, addr)
|
||||
cancel()
|
||||
if errors.Is(err, ErrBusy) {
|
||||
busy, lastErr = true, err
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
t.logger.Warn("linetest: run failed", "run", i+1, "server", addr, "err", err)
|
||||
continue
|
||||
}
|
||||
runs = append(runs, r.Mbps)
|
||||
t.logger.Info(fmt.Sprintf("linetest: run %d: %.1f Mbit/s", i+1, r.Mbps), "server", addr,
|
||||
"server_bytes", r.ServerBytes, "server_seconds", r.ServerSeconds)
|
||||
}
|
||||
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
if t.st.TriesDay != day {
|
||||
t.st.TriesDay, t.st.TriesToday = day, 0
|
||||
}
|
||||
t.st.TriesToday++
|
||||
t.st.LastAttemptAt = start.UTC()
|
||||
if len(runs) > 0 {
|
||||
best := runs[0]
|
||||
for _, r := range runs {
|
||||
if r > best {
|
||||
best = r
|
||||
}
|
||||
}
|
||||
t.st.LastOKAt, t.st.Mbps, t.st.RunsMbps, t.st.Server, t.st.LastError = start.UTC(), best, runs, addr, ""
|
||||
t.logger.Info(fmt.Sprintf("linetest: upload line measured: %.1f Mbit/s (best of %d)", best, len(runs)),
|
||||
"server", addr, "runs", len(runs), "duration_s", t.now().Sub(start).Seconds())
|
||||
} else {
|
||||
msg := "no run succeeded"
|
||||
if lastErr != nil {
|
||||
msg = lastErr.Error()
|
||||
}
|
||||
if len(msg) > 300 {
|
||||
msg = msg[:300]
|
||||
}
|
||||
t.st.LastError = msg
|
||||
if busy {
|
||||
t.logger.Info("linetest: the test server is busy — trying again later today", "server", addr, "tries_today", t.st.TriesToday)
|
||||
} else {
|
||||
t.logger.Warn("linetest: measurement failed", "server", addr, "tries_today", t.st.TriesToday, "err", msg)
|
||||
}
|
||||
}
|
||||
if err := t.saveLocked(); err != nil {
|
||||
t.logger.Warn("linetest: state save failed — the result will not survive a restart", "path", t.statePath, "err", err)
|
||||
}
|
||||
if len(runs) > 0 {
|
||||
return nil
|
||||
}
|
||||
if lastErr == nil {
|
||||
lastErr = errors.New("linetest: no run succeeded")
|
||||
}
|
||||
return lastErr
|
||||
}
|
||||
|
||||
// Value is the measured line (ok=false: never measured).
|
||||
func (t *Tester) Value() (mbps float64, at time.Time, ok bool) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
if t.st.LastOKAt.IsZero() || t.st.Mbps <= 0 {
|
||||
return 0, time.Time{}, false
|
||||
}
|
||||
return t.st.Mbps, t.st.LastOKAt, true
|
||||
}
|
||||
|
||||
// LineTestStatus implements hub.LineTestReporter: a pure read of the persisted state.
|
||||
func (t *Tester) LineTestStatus(context.Context) *hub.LineTestStatus {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
out := &hub.LineTestStatus{Schedule: ScheduleText, Mbps: t.st.Mbps, RunsMbps: t.st.RunsMbps, Server: t.st.Server,
|
||||
LastError: t.st.LastError}
|
||||
if !t.st.LastOKAt.IsZero() {
|
||||
out.LastOKAt = t.st.LastOKAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
if !t.st.LastAttemptAt.IsZero() {
|
||||
out.LastAttemptAt = t.st.LastAttemptAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── the host's physical uplink counter ──────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// SysUplinkTx reads the transmit bytes of the host's physical uplink under root (normally "/"): the interface of the
|
||||
// default route in /proc/net/route, and — when it is a bridge (vmbr0), as on every Proxmox host — the sum of its
|
||||
// PHYSICAL member ports (those with a `device` link). The bridge's own counter does not see the guests' bridged
|
||||
// traffic, which is the traffic this check exists for.
|
||||
func SysUplinkTx(root string) UplinkTx {
|
||||
return func() (uint64, error) {
|
||||
ifc, err := defaultRouteIface(filepath.Join(root, "proc/net/route"))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
sys := filepath.Join(root, "sys/class/net")
|
||||
ports := []string{ifc}
|
||||
if ents, err := os.ReadDir(filepath.Join(sys, ifc, "brif")); err == nil {
|
||||
ports = ports[:0]
|
||||
for _, e := range ents {
|
||||
if _, err := os.Stat(filepath.Join(sys, e.Name(), "device")); err == nil {
|
||||
ports = append(ports, e.Name())
|
||||
}
|
||||
}
|
||||
if len(ports) == 0 {
|
||||
return 0, fmt.Errorf("linetest: bridge %s has no physical port", ifc)
|
||||
}
|
||||
}
|
||||
var sum uint64
|
||||
for _, p := range ports {
|
||||
b, err := os.ReadFile(filepath.Join(sys, p, "statistics/tx_bytes"))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
n, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
sum += n
|
||||
}
|
||||
return sum, nil
|
||||
}
|
||||
}
|
||||
|
||||
func defaultRouteIface(path string) (string, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, line := range strings.Split(string(b), "\n")[1:] {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 8 && f[1] == "00000000" && f[7] == "00000000" {
|
||||
return f[0], nil
|
||||
}
|
||||
}
|
||||
return "", errors.New("linetest: no default route")
|
||||
}
|
||||
@@ -0,0 +1,574 @@
|
||||
package linetest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// ── a fake iperf3 server (the server half of the protocol, as `iperf3 --server` drives it) ─────────────────────────
|
||||
|
||||
type fakeServer struct {
|
||||
ln net.Listener
|
||||
busy bool // answer ACCESS_DENIED
|
||||
stall bool // accept the cookie, then never answer
|
||||
recvSeconds float64 // what the server reports as its duration
|
||||
gotIDs []int // the stream ids the client reported
|
||||
gotParams map[string]any
|
||||
mu sync.Mutex
|
||||
received int64
|
||||
}
|
||||
|
||||
func newFakeServer(t *testing.T) *fakeServer {
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := &fakeServer{ln: ln, recvSeconds: 1}
|
||||
t.Cleanup(func() { ln.Close() })
|
||||
go s.serve()
|
||||
return s
|
||||
}
|
||||
|
||||
func (s *fakeServer) addr() string { return s.ln.Addr().String() }
|
||||
|
||||
func wjson(w io.Writer, v any) {
|
||||
b, _ := json.Marshal(v)
|
||||
var h [4]byte
|
||||
binary.BigEndian.PutUint32(h[:], uint32(len(b)))
|
||||
w.Write(append(h[:], b...))
|
||||
}
|
||||
|
||||
func (s *fakeServer) serve() {
|
||||
ctrl, err := s.ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer ctrl.Close()
|
||||
cookie := make([]byte, cookieSize)
|
||||
if _, err := io.ReadFull(ctrl, cookie); err != nil {
|
||||
return
|
||||
}
|
||||
if s.busy {
|
||||
ctrl.Write([]byte{0xff})
|
||||
return
|
||||
}
|
||||
if s.stall {
|
||||
time.Sleep(5 * time.Second)
|
||||
return
|
||||
}
|
||||
ctrl.Write([]byte{stParamExchange})
|
||||
var params map[string]any
|
||||
if readJSON(ctrl, ¶ms) != nil {
|
||||
return
|
||||
}
|
||||
s.gotParams = params
|
||||
n := int(params["parallel"].(float64))
|
||||
ctrl.Write([]byte{stCreateStreams})
|
||||
var wg sync.WaitGroup
|
||||
var conns []net.Conn
|
||||
for i := 0; i < n; i++ {
|
||||
d, err := s.ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
c := make([]byte, cookieSize)
|
||||
io.ReadFull(d, c)
|
||||
conns = append(conns, d)
|
||||
wg.Add(1)
|
||||
go func(d net.Conn) {
|
||||
defer wg.Done()
|
||||
k, _ := io.Copy(io.Discard, d)
|
||||
s.mu.Lock()
|
||||
s.received += k
|
||||
s.mu.Unlock()
|
||||
}(d)
|
||||
}
|
||||
ctrl.Write([]byte{stTestStart, stTestRunning})
|
||||
var st [1]byte
|
||||
if _, err := io.ReadFull(ctrl, st[:]); err != nil || st[0] != stTestEnd {
|
||||
return
|
||||
}
|
||||
ctrl.Write([]byte{stExchangeResults})
|
||||
var theirs results
|
||||
if readJSON(ctrl, &theirs) != nil {
|
||||
return
|
||||
}
|
||||
for _, x := range theirs.Streams {
|
||||
s.gotIDs = append(s.gotIDs, x.ID)
|
||||
}
|
||||
// The server counts what it RECEIVED; report a fixed figure so the test can tell it from the client's own count.
|
||||
wjson(ctrl, results{Streams: []streamResult{{ID: 1, Bytes: 12_500_000, EndTime: s.recvSeconds}}})
|
||||
ctrl.Write([]byte{stDisplayResults})
|
||||
io.ReadFull(ctrl, st[:])
|
||||
for _, d := range conns {
|
||||
d.Close()
|
||||
}
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func TestClientSpeaksTheProtocolAndCountsTheServersBytes(t *testing.T) {
|
||||
s := newFakeServer(t)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
r, err := Client{Streams: 4, Time: 300 * time.Millisecond}.Run(ctx, s.addr())
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if r.ServerBytes != 12_500_000 || r.Mbps != 100 {
|
||||
t.Fatalf("result = %+v; want the SERVER's 12.5 MB in 1 s = 100 Mbit/s", r)
|
||||
}
|
||||
if want := []int{1, 3, 4, 5}; len(s.gotIDs) != 4 || s.gotIDs[0] != want[0] || s.gotIDs[1] != want[1] || s.gotIDs[3] != want[3] {
|
||||
t.Fatalf("stream ids sent = %v, want %v (iperf3 numbers its streams 1, 3, 4, 5)", s.gotIDs, want)
|
||||
}
|
||||
if s.gotParams["tcp"] != true || s.gotParams["parallel"].(float64) != 4 {
|
||||
t.Fatalf("params = %v", s.gotParams)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientBusyServerIsErrBusy(t *testing.T) {
|
||||
s := newFakeServer(t)
|
||||
s.busy = true
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if _, err := (Client{Streams: 4, Time: time.Second}).Run(ctx, s.addr()); !errors.Is(err, ErrBusy) {
|
||||
t.Fatalf("err = %v, want ErrBusy", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientHonoursTheDeadlineAgainstAStalledServer(t *testing.T) {
|
||||
s := newFakeServer(t)
|
||||
s.stall = true
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
_, err := Client{Streams: 4, Time: time.Second}.Run(ctx, s.addr())
|
||||
if err == nil {
|
||||
t.Fatal("a stalled server gave no error")
|
||||
}
|
||||
if took := time.Since(start); took > 1500*time.Millisecond {
|
||||
t.Fatalf("Run took %v against a stalled server; the deadline is 400 ms", took)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientRefusesAContextWithoutDeadline(t *testing.T) {
|
||||
if _, err := (Client{}).Run(context.Background(), "127.0.0.1:1"); err == nil {
|
||||
t.Fatal("Run without a deadline must refuse (an unbounded test is the thing RunTimeout exists to prevent)")
|
||||
}
|
||||
}
|
||||
|
||||
// ── the schedule ────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
func TestEligibleWindowNeverInTheNightChain(t *testing.T) {
|
||||
loc := time.FixedZone("CET", 3600)
|
||||
start := time.Date(2026, 10, 19, 0, 0, 0, 0, loc) // a Monday
|
||||
for m := 0; m < 7*24*60; m++ {
|
||||
lt := start.Add(time.Duration(m) * time.Minute)
|
||||
for _, anchorDay := range []bool{true, false} {
|
||||
if !EligibleAt(lt, 0, anchorDay) {
|
||||
continue
|
||||
}
|
||||
if h := lt.Hour(); h < StartHour || h >= EndHour || (h >= 1 && h <= 6) {
|
||||
t.Fatalf("eligible at %s — outside 10:00-15:59 or inside the night chain (01:00-06:59)", lt.Format("Mon 15:04"))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlotDelaysTheFirstStartOnTheDueDayOnly(t *testing.T) {
|
||||
loc := time.UTC
|
||||
tue := time.Date(2026, 10, 20, 10, 30, 0, 0, loc)
|
||||
if EligibleAt(tue, 45, true) {
|
||||
t.Fatal("10:30 with slot 45 must wait until 10:45 on the due day")
|
||||
}
|
||||
if !EligibleAt(tue.Add(15*time.Minute), 45, true) {
|
||||
t.Fatal("10:45 with slot 45 must be eligible")
|
||||
}
|
||||
if !EligibleAt(tue.Add(24*time.Hour), 45, false) {
|
||||
t.Fatal("a catch-up day does not wait for the slot again")
|
||||
}
|
||||
if a, b := SlotMinute("demo-hp-bb76ea"), SlotMinute("demo-felhom-8363b5"); a < 0 || a >= SlotSpreadMinutes || b < 0 || b >= SlotSpreadMinutes {
|
||||
t.Fatalf("slots out of range: %d %d", a, b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDueWeeklyFromTuesday(t *testing.T) {
|
||||
loc := time.UTC
|
||||
mon := time.Date(2026, 10, 19, 12, 0, 0, 0, loc)
|
||||
tue := time.Date(2026, 10, 20, 12, 0, 0, 0, loc)
|
||||
if !Due(State{}, mon) {
|
||||
t.Fatal("never measured is due")
|
||||
}
|
||||
st := State{LastOKAt: time.Date(2026, 10, 14, 11, 0, 0, 0, loc)} // last Wednesday
|
||||
if Due(st, mon) {
|
||||
t.Fatal("measured after last Tuesday 10:00 is not due on Monday")
|
||||
}
|
||||
if !Due(st, tue) {
|
||||
t.Fatal("from Tuesday 10:00 a new measurement is due")
|
||||
}
|
||||
}
|
||||
|
||||
// ── the job ─────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
type fakeRunner struct {
|
||||
mu sync.Mutex
|
||||
calls int
|
||||
out []runOut
|
||||
}
|
||||
type runOut struct {
|
||||
mbps float64
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Run(ctx context.Context, addr string) (Result, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
i := f.calls
|
||||
f.calls++
|
||||
if i >= len(f.out) {
|
||||
return Result{}, errors.New("no more scripted runs")
|
||||
}
|
||||
return Result{Mbps: f.out[i].mbps, ServerBytes: 1}, f.out[i].err
|
||||
}
|
||||
|
||||
type fakeGate struct{ busy string }
|
||||
|
||||
func (g *fakeGate) TryAcquire(what string) (func(), string, bool) {
|
||||
if g.busy != "" {
|
||||
return nil, g.busy, false
|
||||
}
|
||||
return func() {}, "", true
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
t *Tester
|
||||
run *fakeRunner
|
||||
gate *fakeGate
|
||||
now time.Time
|
||||
tx uint64
|
||||
tunnel bool
|
||||
}
|
||||
|
||||
// newRig: Tuesday 12:00 UTC (eligible, any slot), tunnel up, 11 quiet samples taken.
|
||||
func newRig(t *testing.T, outs ...runOut) *rig {
|
||||
r := &rig{run: &fakeRunner{out: outs}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
|
||||
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
|
||||
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
|
||||
r.t.loc = time.UTC
|
||||
r.t.now = func() time.Time { return r.now }
|
||||
r.t.sleep = func(context.Context, time.Duration) bool { return true }
|
||||
r.quietMinutes(QuietMinutes + 1)
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) minute(bytes uint64) {
|
||||
r.now = r.now.Add(time.Minute)
|
||||
r.tx += bytes
|
||||
r.t.sample()
|
||||
}
|
||||
|
||||
func (r *rig) quietMinutes(n int) {
|
||||
for i := 0; i < n; i++ {
|
||||
r.minute(1_000_000) // 1 MB/min = 0.13 Mbit/s
|
||||
}
|
||||
}
|
||||
|
||||
func TestTickMeasuresTheHighestOfThreeRuns(t *testing.T) {
|
||||
r := newRig(t, runOut{mbps: 88}, runOut{mbps: 91.5}, runOut{mbps: 90})
|
||||
r.t.Tick(context.Background())
|
||||
mbps, _, ok := r.t.Value()
|
||||
if !ok || mbps != 91.5 || r.run.calls != RunsPerMeasurement {
|
||||
t.Fatalf("value %.1f ok=%v calls=%d; want the highest of 3 runs = 91.5", mbps, ok, r.run.calls)
|
||||
}
|
||||
r.t.Tick(context.Background())
|
||||
if r.run.calls != RunsPerMeasurement {
|
||||
t.Fatal("a measured week must not test again")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBusyServerIsRetriedLaterTheSameDay(t *testing.T) {
|
||||
r := newRig(t, runOut{err: ErrBusy}, runOut{mbps: 90}, runOut{mbps: 89}, runOut{mbps: 92})
|
||||
r.t.Tick(context.Background())
|
||||
if _, _, ok := r.t.Value(); ok || r.run.calls != 1 {
|
||||
t.Fatalf("busy: ok=%v calls=%d; want no value and the measurement stopped after the busy answer", ok, r.run.calls)
|
||||
}
|
||||
if r.t.st.TriesToday != 1 || r.t.st.LastError == "" {
|
||||
t.Fatalf("busy attempt not recorded: %+v", r.t.st)
|
||||
}
|
||||
r.quietMinutes(1)
|
||||
r.t.Tick(context.Background())
|
||||
if mbps, _, ok := r.t.Value(); !ok || mbps != 92 {
|
||||
t.Fatalf("retry later the same day: value %.1f ok=%v", mbps, ok)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTriesPerDayAreBounded(t *testing.T) {
|
||||
outs := make([]runOut, 20)
|
||||
for i := range outs {
|
||||
outs[i] = runOut{err: ErrBusy}
|
||||
}
|
||||
r := newRig(t, outs...)
|
||||
for i := 0; i < 10; i++ {
|
||||
r.quietMinutes(1)
|
||||
r.t.Tick(context.Background())
|
||||
}
|
||||
if r.run.calls != MaxTriesPerDay {
|
||||
t.Fatalf("calls = %d in one day, want at most %d", r.run.calls, MaxTriesPerDay)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoTunnelNoTest(t *testing.T) {
|
||||
r := newRig(t, runOut{mbps: 90})
|
||||
r.tunnel = false
|
||||
r.t.Tick(context.Background())
|
||||
if r.run.calls != 0 {
|
||||
t.Fatal("tested without an operational tunnel")
|
||||
}
|
||||
if _, _, ok := r.t.Value(); ok {
|
||||
t.Fatal("a value without a test")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBusyUplinkOrHeavyOpWaits(t *testing.T) {
|
||||
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
|
||||
r.minute(60_000_000) // one minute at 8 Mbit/s: the family is uploading
|
||||
r.t.Tick(context.Background())
|
||||
if r.run.calls != 0 {
|
||||
t.Fatal("tested while the uplink was busy in the last 10 minutes")
|
||||
}
|
||||
r.quietMinutes(QuietMinutes)
|
||||
r.gate.busy = "backup:felhom-pbs"
|
||||
r.t.Tick(context.Background())
|
||||
if r.run.calls != 0 {
|
||||
t.Fatal("tested beside a heavy operation")
|
||||
}
|
||||
r.gate.busy = ""
|
||||
r.t.Tick(context.Background())
|
||||
if r.run.calls == 0 {
|
||||
t.Fatal("never tested once the uplink was quiet and the gate free")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNeverTestsInTheNightOrBeforeTuesday(t *testing.T) {
|
||||
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
|
||||
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
|
||||
r.quietMinutes(QuietMinutes + 1)
|
||||
r.t.Tick(context.Background())
|
||||
r.t.st.LastOKAt = time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
|
||||
r.now = time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC) // Monday noon, measured last week
|
||||
r.quietMinutes(QuietMinutes + 1)
|
||||
r.t.Tick(context.Background())
|
||||
if r.run.calls != 0 {
|
||||
t.Fatalf("tested %d times at night or when not due", r.run.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStateSurvivesARestartAndReportsIt(t *testing.T) {
|
||||
r := newRig(t, runOut{mbps: 90.7}, runOut{err: errors.New("reset")}, runOut{mbps: 80})
|
||||
r.t.Tick(context.Background())
|
||||
again := New(r.run, nil, r.gate, nil, "host-x", r.t.statePath, nil)
|
||||
mbps, at, ok := again.Value()
|
||||
if !ok || mbps != 90.7 || at.IsZero() {
|
||||
t.Fatalf("after a restart: %.1f %v %v", mbps, at, ok)
|
||||
}
|
||||
b, _ := json.Marshal(again.LineTestStatus(context.Background()))
|
||||
var back hub.LineTestStatus
|
||||
json.Unmarshal(b, &back)
|
||||
if back.Mbps != 90.7 || back.LastOKAt == "" || len(back.RunsMbps) != 2 || back.Schedule == "" {
|
||||
t.Fatalf("report stanza = %s", b)
|
||||
}
|
||||
}
|
||||
|
||||
// ── the uplink counter ──────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
func TestUplinkCountsTheBridgesPhysicalPortsOnly(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
w := func(p, s string) {
|
||||
os.MkdirAll(filepath.Dir(filepath.Join(root, p)), 0o755)
|
||||
os.WriteFile(filepath.Join(root, p), []byte(s), 0o644)
|
||||
}
|
||||
w("proc/net/route", "Iface\tDestination\tGateway\tFlags\tRefCnt\tUse\tMetric\tMask\n"+
|
||||
"wg-felhom\t0000004D\t00000000\t0001\t0\t0\t0\tFFFFFFFF\n"+
|
||||
"vmbr0\t00000000\t0100A8C0\t0003\t0\t0\t0\t00000000\n")
|
||||
w("sys/class/net/vmbr0/statistics/tx_bytes", "5\n")
|
||||
w("sys/class/net/vmbr0/brif/enp1s0/.keep", "")
|
||||
w("sys/class/net/vmbr0/brif/veth9201i0/.keep", "")
|
||||
w("sys/class/net/enp1s0/device/.keep", "")
|
||||
w("sys/class/net/enp1s0/statistics/tx_bytes", "1000\n")
|
||||
w("sys/class/net/veth9201i0/statistics/tx_bytes", "777777\n")
|
||||
got, err := SysUplinkTx(root)()
|
||||
if err != nil || got != 1000 {
|
||||
t.Fatalf("uplink tx = %d, %v; want 1000 (the physical port only — not the bridge, not the guest's veth)", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQuietNeedsTenFullMinutes(t *testing.T) {
|
||||
base := time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC)
|
||||
var s []txSample
|
||||
for i := 0; i <= QuietMinutes; i++ {
|
||||
s = append(s, txSample{at: base.Add(time.Duration(i) * time.Minute), tx: uint64(i) * 1_000_000})
|
||||
}
|
||||
if q, _ := quiet(s); !q {
|
||||
t.Fatal("ten quiet minutes must read quiet")
|
||||
}
|
||||
if q, _ := quiet(s[1:]); q {
|
||||
t.Fatal("nine minutes of samples must not read quiet")
|
||||
}
|
||||
gap := append([]txSample(nil), s...)
|
||||
gap[5].at = gap[5].at.Add(-10 * time.Minute)
|
||||
if q, _ := quiet(gap); q {
|
||||
t.Fatal("a gap in the samples must not read quiet")
|
||||
}
|
||||
}
|
||||
|
||||
// ── a test on request (R-1250): a torrent app that seeds all day ───────────────────────────────────────────────────
|
||||
|
||||
// seedingRig: due and eligible, but every minute of the last ten sent at the line's speed — the weekly Tick never runs.
|
||||
func seedingRig(t *testing.T, outs ...runOut) *rig {
|
||||
r := newRig(t, outs...)
|
||||
for i := 0; i < QuietMinutes+1; i++ {
|
||||
r.minute(600_000_000) // 80 Mbit/s: a seed at the line's speed
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt(t *testing.T) {
|
||||
r := seedingRig(t, runOut{mbps: 88}, runOut{mbps: 92}, runOut{mbps: 90})
|
||||
r.t.Tick(context.Background())
|
||||
if r.run.calls != 0 {
|
||||
t.Fatal("the weekly Tick tested a seeding box")
|
||||
}
|
||||
if !r.t.WaitingForQuiet() {
|
||||
t.Fatal("a due, eligible, seeding box must report waiting_for_quiet")
|
||||
}
|
||||
var slept []time.Duration
|
||||
r.t.sleep = func(_ context.Context, d time.Duration) bool { slept = append(slept, d); return true }
|
||||
mbps, runs, err := r.t.RunNow(context.Background()) // the controller holds the app: the counter stands still
|
||||
if err != nil || mbps != 92 || len(runs) != RunsPerMeasurement || r.run.calls != RunsPerMeasurement {
|
||||
t.Fatalf("RunNow = %.1f %v %v, calls %d; want the best of 3 = 92", mbps, runs, err, r.run.calls)
|
||||
}
|
||||
want := []time.Duration{RequestSettle, RequestQuietWindow, RequestRunGap, RequestRunGap}
|
||||
if len(slept) != len(want) {
|
||||
t.Fatalf("slept %v, want %v", slept, want)
|
||||
}
|
||||
var total time.Duration
|
||||
for i := range want {
|
||||
if slept[i] != want[i] {
|
||||
t.Fatalf("slept %v, want %v", slept, want)
|
||||
}
|
||||
total += slept[i]
|
||||
}
|
||||
if total+RunsPerMeasurement*TestTime > 60*time.Second {
|
||||
t.Fatalf("a request holds the app %v + runs; it must end inside 60 s", total)
|
||||
}
|
||||
if v, _, ok := r.t.Value(); !ok || v != 92 {
|
||||
t.Fatalf("the requested measurement was not stored: %.1f %v", v, ok)
|
||||
}
|
||||
if r.t.WaitingForQuiet() {
|
||||
t.Fatal("measured this week: no longer waiting")
|
||||
}
|
||||
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-due" {
|
||||
t.Fatalf("a second request this week: %v, want not-due", err)
|
||||
}
|
||||
}
|
||||
|
||||
func refusalCode(err error) string {
|
||||
var ref *Refusal
|
||||
if errors.As(err, &ref) {
|
||||
return ref.Code
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// The held app is not the only uploader: a request still refuses to measure a busy uplink.
|
||||
func TestRunNowRefusesAnUplinkThatIsStillBusy(t *testing.T) {
|
||||
r := seedingRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
|
||||
r.t.uplink = func() (uint64, error) { r.tx += 20_000_000; return r.tx, nil } // 20 MB per 10 s = 16 Mbit/s
|
||||
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-quiet" || r.run.calls != 0 {
|
||||
t.Fatalf("RunNow on a busy uplink: %v, calls %d; want not-quiet and no run", err, r.run.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// Every other rule of the weekly test holds for a request.
|
||||
func TestRunNowKeepsTheWeeklyRules(t *testing.T) {
|
||||
r := seedingRig(t)
|
||||
r.gate.busy = "backup:felhom-pbs"
|
||||
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "heavy-op" {
|
||||
t.Fatalf("beside a backup: %v, want heavy-op", err)
|
||||
}
|
||||
r.gate.busy = ""
|
||||
r.tunnel = false
|
||||
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "no-tunnel" {
|
||||
t.Fatalf("no tunnel: %v, want no-tunnel", err)
|
||||
}
|
||||
if r.t.WaitingForQuiet() {
|
||||
t.Fatal("no tunnel: a request could not run, so it must not be invited")
|
||||
}
|
||||
r.tunnel = true
|
||||
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
|
||||
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "outside-window" {
|
||||
t.Fatalf("at night: %v, want outside-window", err)
|
||||
}
|
||||
if r.run.calls != 0 {
|
||||
t.Fatalf("%d runs on refused requests", r.run.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// A failed request counts as a try, so a controller cannot request more than MaxTriesPerDay tests a day.
|
||||
func TestRunNowFailuresSpendTheDaysTries(t *testing.T) {
|
||||
var outs []runOut
|
||||
for i := 0; i < MaxTriesPerDay*RunsPerMeasurement; i++ {
|
||||
outs = append(outs, runOut{err: errors.New("connection refused")})
|
||||
}
|
||||
r := seedingRig(t, outs...)
|
||||
for i := 0; i < MaxTriesPerDay; i++ {
|
||||
if _, _, err := r.t.RunNow(context.Background()); err == nil || refusalCode(err) != "" {
|
||||
t.Fatalf("try %d: %v, want a failed measurement", i+1, err)
|
||||
}
|
||||
}
|
||||
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "tries-used" {
|
||||
t.Fatalf("after %d failures: %v, want tries-used", MaxTriesPerDay, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A quiet box tests itself on the weekly Tick: it must not invite the controller to hold its app.
|
||||
func TestQuietBoxIsNotWaiting(t *testing.T) {
|
||||
r := newRig(t)
|
||||
if r.t.WaitingForQuiet() {
|
||||
t.Fatal("ten quiet minutes: the weekly Tick runs; waiting_for_quiet must be false")
|
||||
}
|
||||
}
|
||||
|
||||
// Live proof 2026-10-24 (demo-hp): a freshly started tester answered waiting_for_quiet before it had watched a single
|
||||
// minute, so a quiet box's torrent app would be held for nothing. Waiting needs the full window, seen busy.
|
||||
func TestFreshTesterIsNotWaiting(t *testing.T) {
|
||||
r := &rig{run: &fakeRunner{}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
|
||||
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
|
||||
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
|
||||
r.t.loc = time.UTC
|
||||
r.t.now = func() time.Time { return r.now }
|
||||
if r.t.WaitingForQuiet() {
|
||||
t.Fatal("no minute watched yet: waiting_for_quiet must be false")
|
||||
}
|
||||
for i := 0; i < QuietMinutes; i++ {
|
||||
r.minute(600_000_000)
|
||||
}
|
||||
if r.t.WaitingForQuiet() {
|
||||
t.Fatal("10 samples are 9 minutes: still not a full window")
|
||||
}
|
||||
r.minute(600_000_000)
|
||||
if !r.t.WaitingForQuiet() {
|
||||
t.Fatal("a full busy window: waiting_for_quiet must be true")
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -145,9 +146,11 @@ func (s *Server) validateBackupTargetMount(ctx context.Context, where string) er
|
||||
// begin with. A typed round-trip would silently drop any key this build does not know about.
|
||||
//
|
||||
// Written IN PLACE (O_TRUNC), not tmp+rename: /etc/felhom-agent is root-owned while agent.json is
|
||||
// agent-owned 0600, so the non-root agent cannot rename into that directory. A recovery copy is
|
||||
// parked first, so a torn write is recoverable.
|
||||
// agent-owned 0600, so the non-root agent cannot rename into that directory. config.ReplaceInPlace
|
||||
// journals the old bytes first; the next start repairs a cut write (TestRecoverCutWrite_*).
|
||||
func (s *Server) setConfiguredBackupTarget(id string) error {
|
||||
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
|
||||
defer config.FileMu.Unlock()
|
||||
path := s.configPath
|
||||
if path == "" {
|
||||
return fmt.Errorf("no config path known to this agent (env-only config)")
|
||||
@@ -179,24 +182,14 @@ func (s *Server) setConfiguredBackupTarget(id string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
st, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if s.stateDir != "" {
|
||||
if err := os.MkdirAll(s.stateDir, 0o700); err == nil {
|
||||
_ = os.WriteFile(filepath.Join(s.stateDir, "agent.json.pre-backup-target"), raw, 0o600)
|
||||
}
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(out); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
return f.Close()
|
||||
// config.ReplaceInPlace: the old bytes go into a journal first, so a power cut in the middle of the write is
|
||||
// repaired at the next start (R-1380).
|
||||
return config.ReplaceInPlace(path, raw, out)
|
||||
}
|
||||
|
||||
// wrapperReason surfaces the wrapper's own REFUSED line when it produced one — it explains WHY in
|
||||
|
||||
@@ -3,6 +3,7 @@ package localapi
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
@@ -83,6 +84,20 @@ const (
|
||||
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
|
||||
// the parked marker.
|
||||
controllerSlowCounterFile = "controller-restarts-24h.json"
|
||||
|
||||
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
|
||||
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
|
||||
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
|
||||
// the hub (controller_unresponsive). Not counted during a swap, a parked guest, a locked guest or a vzdump.
|
||||
controllerUnresponsiveConfirm = 6
|
||||
// controllerUnresponsiveRestart (decided 2026-10-30 from the measurement): at this many consecutive unhealthy sweeps
|
||||
// (5 minutes after Docker's „unhealthy", ~6.5 minutes after the first failed probe; 2 minutes after the alarm) the
|
||||
// agent restarts the controller — the not-running restart's guards and brake, plus the controller's busy file. The
|
||||
// numbers: Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, 4 733 probes through the night chain (a
|
||||
// PBS and a local vzdump, the DB dumps, tier-2, the off-site push, a restore test, two controller updates): no failed
|
||||
// probe, slowest 303 ms against the 5 s timeout. A busy night never came near even ONE failed probe; ten sweeps is
|
||||
// the margin. Pinned by TestSupervisor_UnresponsiveIsAlarmedThenRestarted.
|
||||
controllerUnresponsiveRestart = 10
|
||||
)
|
||||
|
||||
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
|
||||
@@ -103,6 +118,18 @@ type controllerSupState struct {
|
||||
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
|
||||
restarts24h []time.Time
|
||||
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
|
||||
|
||||
// opBrakeUntil (2026-10-29): an operator restart refused by the brake refuses every press until then (30 min).
|
||||
opBrakeUntil time.Time
|
||||
|
||||
// R-1344: consecutive running-but-unhealthy sweeps, and the current episode (since kept after it ends).
|
||||
unhealthySeen int
|
||||
unresponsive bool
|
||||
unresponsiveSince time.Time
|
||||
// unresponsiveRestartAt: the last restart for not answering (kept; the hub keys on it MOVING). unrespRefusal: the
|
||||
// last reason the restart was refused, so a refusal is logged once per change, not every sweep.
|
||||
unresponsiveRestartAt time.Time
|
||||
unrespRefusal string
|
||||
}
|
||||
|
||||
type controllerSupervisor struct {
|
||||
@@ -238,6 +265,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
|
||||
if !s.provisionedGuest(g.VMID) {
|
||||
continue
|
||||
}
|
||||
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
|
||||
s.resetNotRunning(g.VMID)
|
||||
continue
|
||||
}
|
||||
evaluated++
|
||||
if !s.superviseOneController(ctx, g.VMID, g.Status) {
|
||||
down++
|
||||
@@ -293,6 +324,7 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
if running {
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
s.observeControllerHealth(ctx, vmid, parked)
|
||||
return true
|
||||
}
|
||||
st.notRunningSeen++
|
||||
@@ -333,9 +365,27 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
return false
|
||||
}
|
||||
|
||||
// Backoff.
|
||||
if !s.supervisorBrakeAllows(vmid, now) {
|
||||
return false
|
||||
}
|
||||
|
||||
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
|
||||
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
|
||||
"vmid", vmid, "status", status, "unit", bootstrapUnit)
|
||||
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
|
||||
reason += "; restart FAILED: " + err.Error()
|
||||
}
|
||||
s.recordSupervisorRestart(vmid, now, reason, nil)
|
||||
return false
|
||||
}
|
||||
|
||||
// supervisorBrakeAllows is the brake every supervisor restart passes (the not-running one and R-1344's): a crash-loop
|
||||
// pause in force refuses; controllerCrashloopMax restarts inside controllerCrashloopWindow start one (logged, carried by
|
||||
// the stanza, mailed by the hub as controller_crashloop) and refuse.
|
||||
func (s *Server) supervisorBrakeAllows(vmid int, now time.Time) bool {
|
||||
s.ctrlSup.mu.Lock()
|
||||
st = s.supState(vmid)
|
||||
st := s.supState(vmid)
|
||||
if !st.crashloopSince.IsZero() {
|
||||
if now.Sub(st.crashloopSince) < controllerCrashloopPause {
|
||||
s.ctrlSup.mu.Unlock()
|
||||
@@ -358,22 +408,23 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
return false
|
||||
}
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return true
|
||||
}
|
||||
|
||||
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
|
||||
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
|
||||
"vmid", vmid, "status", status, "unit", bootstrapUnit)
|
||||
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
|
||||
reason += "; restart FAILED: " + err.Error()
|
||||
}
|
||||
// recordSupervisorRestart puts a restart the supervisor performed on the record: the brake's window, the totals, the
|
||||
// reason, and R-539's slow counter (persisted; raised at most once per 24 hours — the hub mails on the raise MOVING).
|
||||
// mark, when set, runs under the same lock (R-1344 stamps its own record there).
|
||||
func (s *Server) recordSupervisorRestart(vmid int, now time.Time, reason string, mark func(st *controllerSupState)) {
|
||||
s.ctrlSup.mu.Lock()
|
||||
st = s.supState(vmid)
|
||||
st := s.supState(vmid)
|
||||
st.restarts = append(st.restarts, now)
|
||||
st.restartsTotal++
|
||||
st.lastRestartAt = now
|
||||
st.lastReason = reason
|
||||
st.notRunningSeen = 0
|
||||
// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING.
|
||||
if mark != nil {
|
||||
mark(st)
|
||||
}
|
||||
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
|
||||
n24 := len(st.restarts24h)
|
||||
raised := false
|
||||
@@ -389,7 +440,101 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
|
||||
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
|
||||
// the swap's `docker inspect -f` grant. Anything but „unhealthy" — healthy, starting, no health check, a read that
|
||||
// fails — resets the count; so do a swap, a parked guest, a locked guest and a vzdump (each is someone else's moment).
|
||||
func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked bool) {
|
||||
s.swapMu.Lock()
|
||||
swapping := s.swapInFlight[vmid]
|
||||
s.swapMu.Unlock()
|
||||
unhealthy := false
|
||||
if !parked && !swapping {
|
||||
if out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer); err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
unhealthy = len(f) == 2 && f[0] == "true" && f[1] == "unhealthy"
|
||||
}
|
||||
}
|
||||
if unhealthy {
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil || lock != "" {
|
||||
unhealthy = false
|
||||
} else if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||
unhealthy = false
|
||||
}
|
||||
}
|
||||
now := s.clock()
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if !unhealthy {
|
||||
was := st.unresponsive
|
||||
st.unhealthySeen, st.unresponsive = 0, false
|
||||
s.ctrlSup.mu.Unlock()
|
||||
if was {
|
||||
s.logger.Info("controller-supervisor: the controller answers its health check again", "vmid", vmid)
|
||||
}
|
||||
return
|
||||
}
|
||||
st.unhealthySeen++
|
||||
seen := st.unhealthySeen
|
||||
start := seen >= controllerUnresponsiveConfirm && !st.unresponsive
|
||||
if start {
|
||||
st.unresponsive, st.unresponsiveSince = true, now
|
||||
}
|
||||
since := st.unresponsiveSince
|
||||
s.ctrlSup.mu.Unlock()
|
||||
if start {
|
||||
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so",
|
||||
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String(),
|
||||
"restart_after_sweeps", controllerUnresponsiveRestart)
|
||||
}
|
||||
if seen >= controllerUnresponsiveRestart {
|
||||
s.restartUnresponsive(ctx, vmid, seen, since, now)
|
||||
}
|
||||
}
|
||||
|
||||
// restartUnresponsive is R-1344's restart (decided 2026-10-30): the controller's busy file (fresh = a backup, a
|
||||
// restore, an update or an export is running; unreadable = cannot be ruled out) refuses, then the brake every
|
||||
// supervisor restart passes. The swap, the parked marker, the lock and the vzdump never reach here — they reset the
|
||||
// count. A refusal keeps the alarm and is logged once per reason; the next sweep asks again.
|
||||
func (s *Server) restartUnresponsive(ctx context.Context, vmid, seen int, since, now time.Time) {
|
||||
refuse := func(why string) {
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
changed := st.unrespRefusal != why
|
||||
st.unrespRefusal = why
|
||||
s.ctrlSup.mu.Unlock()
|
||||
if changed {
|
||||
s.logger.Warn("controller-supervisor: the controller does not answer, but the restart is refused — the alarm stays",
|
||||
"vmid", vmid, "why", why, "unhealthy_sweeps", seen)
|
||||
}
|
||||
}
|
||||
busy, what, err := s.controllerBusy(ctx, vmid)
|
||||
if err != nil {
|
||||
refuse("the controller's busy file cannot be read: " + err.Error())
|
||||
return
|
||||
}
|
||||
if busy {
|
||||
refuse("the controller is busy: " + what)
|
||||
return
|
||||
}
|
||||
if !s.supervisorBrakeAllows(vmid, now) {
|
||||
refuse("the restart brake")
|
||||
return
|
||||
}
|
||||
reason := fmt.Sprintf("the controller ran but did not answer its health check for %d sweeps (since %s)",
|
||||
seen, since.UTC().Format(time.RFC3339))
|
||||
s.logger.Warn("controller-supervisor: the controller does not answer — restarting the bootstrap unit",
|
||||
"vmid", vmid, "unhealthy_sweeps", seen, "since", since.UTC().Format(time.RFC3339), "unit", bootstrapUnit)
|
||||
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
|
||||
reason += "; restart FAILED: " + err.Error()
|
||||
}
|
||||
s.recordSupervisorRestart(vmid, now, reason, func(st *controllerSupState) {
|
||||
st.unresponsiveRestartAt = now
|
||||
st.unhealthySeen = 0
|
||||
st.unrespRefusal = ""
|
||||
})
|
||||
}
|
||||
|
||||
func (s *Server) resetNotRunning(vmid int) {
|
||||
@@ -446,6 +591,14 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
|
||||
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
|
||||
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
|
||||
}
|
||||
g.Unresponsive = st.unresponsive
|
||||
if !st.unresponsiveSince.IsZero() {
|
||||
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
|
||||
}
|
||||
g.UnresponsiveRestartAfterS = (controllerUnresponsiveRestart - controllerUnresponsiveConfirm) * int(controllerSupervisorInterval/time.Second)
|
||||
if !st.unresponsiveRestartAt.IsZero() {
|
||||
g.UnresponsiveRestartAt = st.unresponsiveRestartAt.UTC().Format(time.RFC3339)
|
||||
}
|
||||
out.Guests = append(out.Guests, g)
|
||||
}
|
||||
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
|
||||
// agent's state folder on a box, and a test must not need it.
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
config.CutWriteDir = dir
|
||||
code := m.Run()
|
||||
_ = os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
@@ -87,6 +87,11 @@ type GuestAttacher interface {
|
||||
// GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a
|
||||
// controller-only restart — the deterministic guest-reboot signal the controller recreates apps on.
|
||||
GuestBootID(ctx context.Context, vmid int) string
|
||||
// GuestBootKey is the same signal built from the kernel's boot_id, which a step of the host's clock cannot
|
||||
// move (R-1395). "" = not readable; the controller then reads GuestBootID.
|
||||
GuestBootKey(ctx context.Context, vmid int) string
|
||||
// GuestBootTokens returns (GuestBootID, GuestBootKey) from one read — what /disks sends.
|
||||
GuestBootTokens(ctx context.Context, vmid int) (id, key string)
|
||||
// AttachBind is the LEGACY per-drive `pct set -mpN` bind (pre-intermediary). Retained for the
|
||||
// transition; new attaches use AttachDrive.
|
||||
AttachBind(ctx context.Context, vmid int, mountKey, where string) error
|
||||
@@ -431,11 +436,12 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
|
||||
// Guest boot-id (intermediary model): changes on every guest boot, stable across controller restarts.
|
||||
// The controller persists it and deterministically recreates drive-backed apps when it changes.
|
||||
bootID := ""
|
||||
// R-1395: guest_boot_key is the clock-proof form (kernel boot_id); guest_boot_id stays for older controllers.
|
||||
bootID, bootKey := "", ""
|
||||
if s.guestAttach != nil {
|
||||
bootID = s.guestAttach.GuestBootID(r.Context(), vmid)
|
||||
bootID, bootKey = s.guestAttach.GuestBootTokens(r.Context(), vmid)
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID})
|
||||
writeOK(w, map[string]any{"vmid": vmid, "disks": out, "guest_boot_id": bootID, "guest_boot_key": bootKey})
|
||||
}
|
||||
|
||||
// handleDiskCandidates (Impl-2a) lists the host disks FREE for Felhom to enroll — the raw-device
|
||||
|
||||
@@ -466,6 +466,10 @@ func (f *fakeGuestAttacher) AttachDrive(_ context.Context, _ int, where string)
|
||||
}
|
||||
func (f *fakeGuestAttacher) GuestSeesMount(_ context.Context, _ int, _ string) bool { return true }
|
||||
func (f *fakeGuestAttacher) GuestBootID(_ context.Context, _ int) string { return "boot-1" }
|
||||
func (f *fakeGuestAttacher) GuestBootKey(_ context.Context, _ int) string { return "key-1" }
|
||||
func (f *fakeGuestAttacher) GuestBootTokens(_ context.Context, _ int) (string, string) {
|
||||
return "boot-1", "key-1"
|
||||
}
|
||||
func (f *fakeGuestAttacher) attachDriveCount() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
|
||||
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
|
||||
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
|
||||
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
|
||||
Upload func(ctx context.Context, pkg []byte) error
|
||||
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
|
||||
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
|
||||
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
|
||||
}
|
||||
|
||||
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
|
||||
// ceremony incl. its upload was measured at ~2.4 s).
|
||||
const escrowUploadTimeout = 30 * time.Second
|
||||
|
||||
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
|
||||
const escrowHoldTimeout = 15 * time.Second
|
||||
|
||||
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
|
||||
// or late may have been stored all the same.
|
||||
type uploadOutcome int
|
||||
|
||||
const (
|
||||
uploadStored uploadOutcome = iota // the hub holds the new package
|
||||
uploadNotStored // the hub answered and holds another package: the previous code still works
|
||||
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
|
||||
)
|
||||
|
||||
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
|
||||
// still works", which is false when the lost answer came after the store.
|
||||
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
|
||||
|
||||
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
|
||||
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
|
||||
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
|
||||
if s.escrowCeremony.HubHolds == nil {
|
||||
return uploadUnknown
|
||||
}
|
||||
base := s.baseCtx
|
||||
if base == nil {
|
||||
base = context.Background()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
|
||||
defer cancel()
|
||||
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
|
||||
switch {
|
||||
case err != nil:
|
||||
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
|
||||
return uploadUnknown
|
||||
case held:
|
||||
return uploadStored
|
||||
default:
|
||||
return uploadNotStored
|
||||
}
|
||||
}
|
||||
|
||||
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
|
||||
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
|
||||
// spike-shaped output; production is runCeremonySubprocess.
|
||||
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
|
||||
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
|
||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||
cancel()
|
||||
outcome := uploadStored
|
||||
if err != nil {
|
||||
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
|
||||
}
|
||||
|
||||
s.escrowMu.Lock()
|
||||
s.escrowClaiming = false
|
||||
if err != nil {
|
||||
switch outcome {
|
||||
case uploadNotStored:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||
return
|
||||
case uploadUnknown:
|
||||
s.escrowMu.Unlock()
|
||||
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
|
||||
"vmid", vmid, "job_id", jobID, "err", err)
|
||||
}
|
||||
s.escrowPending = nil
|
||||
s.wipeEscrowConfirmLocked()
|
||||
|
||||
@@ -41,9 +41,13 @@ type fakeEscrowHub struct {
|
||||
mu sync.Mutex
|
||||
current []byte
|
||||
uploads int
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
err error // non-nil → the upload fails, nothing stored
|
||||
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
|
||||
// holdsErr → the hub cannot be asked which package it holds.
|
||||
lost bool
|
||||
holdsErr error
|
||||
gate chan struct{} // non-nil → the upload blocks until closed
|
||||
entered chan struct{} // non-nil → closed when an upload starts
|
||||
}
|
||||
|
||||
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||
}
|
||||
f.current = append([]byte(nil), pkg...)
|
||||
f.uploads++
|
||||
if f.lost {
|
||||
return context.DeadlineExceeded
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
|
||||
// blob the hub serves back).
|
||||
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.holdsErr != nil {
|
||||
return false, f.holdsErr
|
||||
}
|
||||
return string(f.current) == string(pkg), nil
|
||||
}
|
||||
|
||||
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
|
||||
func (f *fakeEscrowHub) opensWith(r string) bool {
|
||||
f.mu.Lock()
|
||||
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
|
||||
HubConfigured: true,
|
||||
DRConfigured: func() bool { return true },
|
||||
Upload: hub.upload,
|
||||
HubHolds: hub.holds,
|
||||
}
|
||||
srv.ceremonyRun = run
|
||||
srv.escrowSudoCheck = func(context.Context) error { return nil }
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
|
||||
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
|
||||
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
|
||||
// - the hub holds the new package → success (the code is handed out / the confirm completes);
|
||||
// - the hub holds another package → the old sentence, now true;
|
||||
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
|
||||
|
||||
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
|
||||
// hub holds the new package → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if !hub.opensWith(firstR) {
|
||||
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
|
||||
}
|
||||
if code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("the household was told the previous code still works: %s", body)
|
||||
}
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
|
||||
t.Fatalf("the job must read uploaded: %s", st)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
|
||||
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
|
||||
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
code, body := claimCode(t, h)
|
||||
if code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
|
||||
}
|
||||
if strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("an unknown outcome promised the previous code: %s", body)
|
||||
}
|
||||
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
|
||||
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
|
||||
t.Fatalf("the code must stay claimable: %s", st)
|
||||
}
|
||||
// The hub answers again: the retry finds the new package already stored and hands the code out.
|
||||
hub.mu.Lock()
|
||||
hub.holdsErr = nil
|
||||
hub.mu.Unlock()
|
||||
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||
t.Fatalf("retry: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
|
||||
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
|
||||
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
|
||||
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The same at the typed-back confirm (R-1151).
|
||||
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
|
||||
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||
hub := &fakeEscrowHub{lost: true}
|
||||
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
|
||||
h := srv.Handler()
|
||||
startAndWait(t, srv, h)
|
||||
c := claimDeferred(t, h)
|
||||
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
|
||||
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
|
||||
}
|
||||
hub.mu.Lock()
|
||||
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
|
||||
hub.mu.Unlock()
|
||||
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||
h2 := srv2.Handler()
|
||||
startAndWait(t, srv2, h2)
|
||||
c2 := claimDeferred(t, h2)
|
||||
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
|
||||
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
type initPIDRunner struct{ pid string }
|
||||
|
||||
func (r initPIDRunner) Run(_ context.Context, name string, _ ...string) ([]byte, []byte, error) {
|
||||
if name == "lxc-info" {
|
||||
return []byte(r.pid + "\n"), nil, nil
|
||||
}
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func (r initPIDRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
// R-1395: on 2026-11-14 the N100's time service came back and stepped the host's clock by +32.7 s. The kernel
|
||||
// computes the boot TIME from the wall clock, so guest_boot_id moved and the controller recreated every drive-backed
|
||||
// app. The consequence asserted: the boot KEY does not move when the boot time moves; it does move on a host reboot
|
||||
// and on a guest reboot; and it is "" (so the controller falls back to the old token) when the boot_id is unreadable.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/C/redproof-agent-boot-key.txt).
|
||||
func TestGuestBootKey_AClockStepDoesNotMoveIt(t *testing.T) {
|
||||
oldB, oldK, oldS := hostBtimeFn, hostBootIDFn, procStarttimeFn
|
||||
t.Cleanup(func() { hostBtimeFn, hostBootIDFn, procStarttimeFn = oldB, oldK, oldS })
|
||||
btime, bootID, start := "1794112518", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
|
||||
hostBtimeFn = func() string { return btime }
|
||||
hostBootIDFn = func() string { return bootID }
|
||||
procStarttimeFn = func(pid string) string {
|
||||
if pid != "31337" {
|
||||
t.Errorf("start time read for pid %q, want the guest's init", pid)
|
||||
}
|
||||
return start
|
||||
}
|
||||
b := NewGuestBinder(initPIDRunner{"31337"}, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
ctx := context.Background()
|
||||
|
||||
id0, key0 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
|
||||
if id0 != "1794112518-4242" || key0 != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" {
|
||||
t.Fatalf("tokens = %q / %q", id0, key0)
|
||||
}
|
||||
btime = "1794112550" // THE STEP: the wall clock jumped +32 s, so the kernel's btime did too
|
||||
id1, key1 := b.GuestBootID(ctx, 9201), b.GuestBootKey(ctx, 9201)
|
||||
if id1 == id0 {
|
||||
t.Fatalf("control: the old token did not move with the boot time (%q) — the test does not stage the defect", id1)
|
||||
}
|
||||
if key1 != key0 {
|
||||
t.Fatalf("the boot key moved with the clock: %q → %q — a clock step reads as a new boot", key0, key1)
|
||||
}
|
||||
if strings.Contains(key1, "1794") {
|
||||
t.Errorf("the boot key carries the boot time: %q", key1)
|
||||
}
|
||||
bootID = "c0ffee00-1111-2222-3333-444455556666" // a host reboot
|
||||
if k := b.GuestBootKey(ctx, 9201); k == key0 {
|
||||
t.Errorf("a host reboot (new boot_id) did not change the key: %q", k)
|
||||
}
|
||||
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "991234" // a guest reboot
|
||||
if k := b.GuestBootKey(ctx, 9201); k == key0 || !strings.HasSuffix(k, "-991234") {
|
||||
t.Errorf("a guest reboot (new init start) did not change the key: %q", k)
|
||||
}
|
||||
for _, bad := range []string{"", "not-a-boot-id", "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11\nx"} {
|
||||
bootID = bad
|
||||
if k := b.GuestBootKey(ctx, 9201); k != "" {
|
||||
t.Errorf("boot_id %q gave the key %q, want \"\" (the controller then reads the old token)", bad, k)
|
||||
}
|
||||
if id := b.GuestBootID(ctx, 9201); id == "" {
|
||||
t.Errorf("the old token must still be sent when the boot_id is unreadable")
|
||||
}
|
||||
}
|
||||
// what /disks sends: both tokens from ONE read of the guest's init — the init is asked for once, not twice
|
||||
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "4242"
|
||||
reads := 0
|
||||
procStarttimeFn = func(string) string { reads++; return start }
|
||||
if id, key := b.GuestBootTokens(ctx, 9201); id != "1794112550-4242" || key != "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11-4242" || reads != 1 {
|
||||
t.Errorf("GuestBootTokens = %q, %q after %d read(s) of the init — want both tokens from ONE read", id, key, reads)
|
||||
}
|
||||
bootID, start = "5b1f6c1e-8f0a-4c56-9d1c-0a8c3b7d2e11", "" // the guest does not run
|
||||
if k := b.GuestBootKey(ctx, 9201); k != "" {
|
||||
t.Errorf("a guest that does not run gave the key %q", k)
|
||||
}
|
||||
}
|
||||
|
||||
// The key is no use if /disks does not carry it: both tokens are on the wire, the old one unchanged for controllers
|
||||
// that do not know the key.
|
||||
func TestDisks_CarriesBothBootTokens(t *testing.T) {
|
||||
srv := targetRowServer(t, "", []hub.StorageTarget{})
|
||||
srv.guestAttach = &fakeGuestAttacher{}
|
||||
body := do(t, srv.Handler(), "GET", "/disks", "A", "").Body.Bytes()
|
||||
var w struct {
|
||||
Data struct {
|
||||
ID string `json:"guest_boot_id"`
|
||||
Key string `json:"guest_boot_key"`
|
||||
} `json:"data"`
|
||||
ID string `json:"guest_boot_id"`
|
||||
Key string `json:"guest_boot_key"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &w); err != nil {
|
||||
t.Fatalf("decode: %v (%s)", err, body)
|
||||
}
|
||||
if w.Data.ID+w.ID != "boot-1" || w.Data.Key+w.Key != "key-1" {
|
||||
t.Fatalf("/disks boot tokens: id=%q key=%q, want boot-1 and key-1 (%s)", w.Data.ID+w.ID, w.Data.Key+w.Key, body)
|
||||
}
|
||||
}
|
||||
|
||||
// The real binder is what main.go hands the server: it must have the method (the interface makes this a build
|
||||
// error if not — stated here so the reason is on record).
|
||||
var _ GuestAttacher = (*GuestBinder)(nil)
|
||||
@@ -0,0 +1,102 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// R-528 — the OOM signal from the host side.
|
||||
//
|
||||
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
|
||||
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
|
||||
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
|
||||
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
|
||||
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
|
||||
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
|
||||
//
|
||||
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
|
||||
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
|
||||
|
||||
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
|
||||
// cgroup folder (a VM guest, or a container that is not running).
|
||||
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
|
||||
if root == "" {
|
||||
root = "/sys/fs/cgroup"
|
||||
}
|
||||
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
|
||||
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
|
||||
return nil, false, nil
|
||||
}
|
||||
out := map[string]int64{}
|
||||
for _, pattern := range []string{
|
||||
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
|
||||
filepath.Join(base, "docker", "*", "memory.events"),
|
||||
} {
|
||||
files, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, true, err
|
||||
}
|
||||
for _, f := range files {
|
||||
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
|
||||
if !isContainerID(id) {
|
||||
continue
|
||||
}
|
||||
n, err := readOOMKill(f)
|
||||
if err != nil {
|
||||
continue // a container that stopped between the glob and the read
|
||||
}
|
||||
out[id] = n
|
||||
}
|
||||
}
|
||||
return out, true, nil
|
||||
}
|
||||
|
||||
func isContainerID(s string) bool {
|
||||
if len(s) != 64 {
|
||||
return false
|
||||
}
|
||||
for _, c := range s {
|
||||
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func readOOMKill(path string) (int64, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
|
||||
if ok && k == "oom_kill" {
|
||||
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return 0, fmt.Errorf("no oom_kill line in %s", path)
|
||||
}
|
||||
|
||||
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
|
||||
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
|
||||
if err != nil {
|
||||
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
|
||||
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
|
||||
return
|
||||
}
|
||||
if kills == nil {
|
||||
kills = map[string]int64{}
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
|
||||
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
|
||||
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
|
||||
// serves the asking guest's counters, and only that guest's.
|
||||
|
||||
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
|
||||
t.Helper()
|
||||
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
|
||||
|
||||
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
|
||||
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
|
||||
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
|
||||
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
|
||||
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
|
||||
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
|
||||
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
|
||||
if err := os.MkdirAll(dfDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
|
||||
|
||||
got, avail, err := readGuestOOMKills(root, 100)
|
||||
if err != nil || !avail {
|
||||
t.Fatalf("read: avail=%v err=%v", avail, err)
|
||||
}
|
||||
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
for k, v := range want {
|
||||
if got[k] != v {
|
||||
t.Fatalf("got %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
if _, leaked := got[idOther]; leaked {
|
||||
t.Fatal("another guest's container leaked into this guest's answer")
|
||||
}
|
||||
// A guest with no cgroup folder (a VM, or not running): not available, no error.
|
||||
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
|
||||
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Through the route, with the guest's own token.
|
||||
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
|
||||
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
|
||||
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||
srv.cgroupRoot = root
|
||||
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("route: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
var env struct {
|
||||
Data struct {
|
||||
VMID int `json:"vmid"`
|
||||
Available bool `json:"available"`
|
||||
Containers map[string]int64 `json:"containers"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
|
||||
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
|
||||
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
|
||||
}
|
||||
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("no token: %d", w.Code)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package localapi
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -82,6 +83,40 @@ type guestPowerState struct {
|
||||
attempts int
|
||||
nextAt time.Time
|
||||
raised bool // the give-up fault has already been raised for this run
|
||||
raisedAt time.Time
|
||||
}
|
||||
|
||||
// GuestPowerGiveUp is one guest the watchdog stopped trying to start (R-1383, D78). It rides the host report
|
||||
// (`guest_power_gave_up`) so the operator hears it: until this, the give-up was one ERROR line in the host's journal
|
||||
// while the household's apps were off. Memory only — an agent restart forgets it (and tries three times again).
|
||||
type GuestPowerGiveUp struct {
|
||||
VMID int `json:"vmid"`
|
||||
Since string `json:"since"` // RFC3339 UTC, when the watchdog gave up
|
||||
Attempts int `json:"attempts"`
|
||||
}
|
||||
|
||||
// GuestPowerGaveUpStanza is GuestPowerGaveUp shaped for the host report: an UNTYPED nil when there is none, so the
|
||||
// report omits the field (a typed nil slice in an `any` would be sent as `null`).
|
||||
func (s *Server) GuestPowerGaveUpStanza() any {
|
||||
if g := s.GuestPowerGaveUp(); len(g) > 0 {
|
||||
return g
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GuestPowerGaveUp returns the guests the watchdog has given up on, by vmid. nil when there is none — the report
|
||||
// stanza is then omitted. Pinned by TestGuestPower_GiveUpIsReportedUntilTheGuestRuns.
|
||||
func (s *Server) GuestPowerGaveUp() []GuestPowerGiveUp {
|
||||
guestPowerMu.Lock()
|
||||
defer guestPowerMu.Unlock()
|
||||
var out []GuestPowerGiveUp
|
||||
for vmid, st := range s.guestPower {
|
||||
if st.raised {
|
||||
out = append(out, GuestPowerGiveUp{VMID: vmid, Since: st.raisedAt.UTC().Format(time.RFC3339), Attempts: st.attempts})
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].VMID < out[j].VMID })
|
||||
return out
|
||||
}
|
||||
|
||||
// WatchGuestPower runs the guest-power sweep every guestPowerInterval until ctx is done. No-op when
|
||||
@@ -117,6 +152,19 @@ func (s *Server) GuestPowerTick(ctx context.Context) {
|
||||
s.logger.Warn("guest-power: guest list unavailable — skipping sweep (ownership unproven)", "err", err)
|
||||
return
|
||||
}
|
||||
// A guest that left the list (destroyed, or re-made under another number) takes its attempt record with it — a
|
||||
// give-up for a guest that no longer exists would ride every host report until the agent restarts.
|
||||
present := make(map[int]bool, len(guests))
|
||||
for _, g := range guests {
|
||||
present[g.VMID] = true
|
||||
}
|
||||
guestPowerMu.Lock()
|
||||
for vmid := range s.guestPower {
|
||||
if !present[vmid] {
|
||||
delete(s.guestPower, vmid)
|
||||
}
|
||||
}
|
||||
guestPowerMu.Unlock()
|
||||
var stopped int
|
||||
for _, g := range guests {
|
||||
if ctx.Err() != nil {
|
||||
@@ -152,6 +200,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
|
||||
return
|
||||
}
|
||||
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
|
||||
// would fail the reboot's own start half).
|
||||
if s.rebooting(g.VMID) {
|
||||
return
|
||||
}
|
||||
|
||||
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
|
||||
if err != nil {
|
||||
@@ -162,6 +215,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
// SCENARIO B — a deliberately stopped guest is left alone, forever. onboot:0 means the operator
|
||||
// (or the golden-image provisioning) does not want this guest running.
|
||||
if !onboot {
|
||||
s.forgetGuestPower(g.VMID) // not ours to start any more: a give-up for it must not be reported for ever
|
||||
return
|
||||
}
|
||||
|
||||
@@ -171,6 +225,7 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
if lock != "" {
|
||||
s.logger.Info("guest-power: guest is stopped but LOCKED — leaving it to the stale-lock path",
|
||||
"vmid", g.VMID, "lock", lock)
|
||||
s.forgetGuestPower(g.VMID) // another operation owns it now; if it ends stopped, the three tries start again
|
||||
return
|
||||
}
|
||||
|
||||
@@ -255,6 +310,7 @@ func (s *Server) markGuestPowerRaised(vmid int) {
|
||||
defer guestPowerMu.Unlock()
|
||||
st := s.guestPower[vmid]
|
||||
st.raised = true
|
||||
st.raisedAt = s.now()
|
||||
s.guestPower[vmid] = st
|
||||
}
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"testing"
|
||||
@@ -237,3 +238,81 @@ func TestGuestPower_RunningGuestClearsAttemptHistory(t *testing.T) {
|
||||
t.Error("attempt history survived the guest coming back healthy")
|
||||
}
|
||||
}
|
||||
|
||||
// R-1383 (D78): the give-up was ONE journal line while the household's apps were off. It now rides the host report:
|
||||
// nothing before the give-up (and an untyped nil, so the report omits the field), the guest with the time and the
|
||||
// attempt count after it, and nothing again once the guest runs.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-guest-power.txt).
|
||||
func TestGuestPower_GiveUpIsReportedUntilTheGuestRuns(t *testing.T) {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
startErr: errors.New("cannot start: storage offline"),
|
||||
}
|
||||
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
|
||||
s := gpServer(t, ctl, func() time.Time { return now })
|
||||
if st := s.GuestPowerGaveUpStanza(); st != nil {
|
||||
t.Fatalf("before any attempt the stanza is %#v, want an untyped nil (the report omits the field)", st)
|
||||
}
|
||||
for i := 0; i < guestPowerMaxAttempts; i++ { // the three failed starts
|
||||
s.GuestPowerTick(context.Background())
|
||||
if g := s.GuestPowerGaveUp(); len(g) != 0 {
|
||||
t.Fatalf("after %d failed start(s) the watchdog still tries, but it reports a give-up: %+v", i+1, g)
|
||||
}
|
||||
now = now.Add(10 * time.Minute)
|
||||
}
|
||||
gaveUpAt := now
|
||||
s.GuestPowerTick(context.Background()) // the tick that gives up
|
||||
now = now.Add(time.Hour)
|
||||
s.GuestPowerTick(context.Background())
|
||||
g := s.GuestPowerGaveUp()
|
||||
if len(g) != 1 || g[0].VMID != 9201 || g[0].Attempts != guestPowerMaxAttempts || g[0].Since != gaveUpAt.Format(time.RFC3339) {
|
||||
t.Fatalf("give-up list = %+v, want guest 9201, %d attempts, since %s (the give-up's time, not a later tick's)", g, guestPowerMaxAttempts, gaveUpAt.Format(time.RFC3339))
|
||||
}
|
||||
b, _ := json.Marshal(s.GuestPowerGaveUpStanza())
|
||||
if string(b) != `[{"vmid":9201,"since":"2026-11-14T02:30:00Z","attempts":3}]` {
|
||||
t.Fatalf("the stanza on the wire = %s", b)
|
||||
}
|
||||
// the operator starts the guest: the next sweep sees it running and the list is empty again
|
||||
ctl.guests = []proxmox.Guest{{VMID: 9201, Status: "running"}}
|
||||
s.GuestPowerTick(context.Background())
|
||||
if st := s.GuestPowerGaveUpStanza(); st != nil {
|
||||
t.Fatalf("the guest runs again and the stanza is still %#v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// Found by review, 2026-11-14: the give-up was forgotten only when the guest was seen RUNNING. A guest the operator
|
||||
// then switches to onboot:0, a guest another operation locks, and a guest that is destroyed would be reported as
|
||||
// „given up” for ever (the hub's alarm could never clear). Each of the three ends the report.
|
||||
func TestGuestPower_GiveUpEndsWhenTheGuestIsNoLongerOurs(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
change func(ctl *fakeGuestPowerCtl)
|
||||
}{
|
||||
{"onboot switched off", func(ctl *fakeGuestPowerCtl) { ctl.onboot = map[int]bool{9201: false} }},
|
||||
{"locked by another operation", func(ctl *fakeGuestPowerCtl) { ctl.locks = map[int]string{9201: "backup"} }},
|
||||
{"destroyed", func(ctl *fakeGuestPowerCtl) { ctl.guests = nil }},
|
||||
} {
|
||||
ctl := &fakeGuestPowerCtl{
|
||||
guests: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
locks: map[int]string{9201: ""},
|
||||
onboot: map[int]bool{9201: true},
|
||||
startErr: errors.New("cannot start"),
|
||||
}
|
||||
now := time.Date(2026, 11, 14, 2, 0, 0, 0, time.UTC)
|
||||
s := gpServer(t, ctl, func() time.Time { return now })
|
||||
for i := 0; i < guestPowerMaxAttempts+1; i++ {
|
||||
s.GuestPowerTick(context.Background())
|
||||
now = now.Add(10 * time.Minute)
|
||||
}
|
||||
if len(s.GuestPowerGaveUp()) != 1 {
|
||||
t.Fatalf("%s: setup — no give-up recorded", c.name)
|
||||
}
|
||||
c.change(ctl)
|
||||
s.GuestPowerTick(context.Background())
|
||||
if g := s.GuestPowerGaveUp(); len(g) != 0 {
|
||||
t.Errorf("%s: the give-up is still reported: %+v", c.name, g)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
@@ -288,21 +289,82 @@ func (b *GuestBinder) guestInitPID(ctx context.Context, vmid int) string {
|
||||
// /proc/<pid>/stat — ticks since host boot, unique per process launch) changes on a guest reboot. ""
|
||||
// on any read error (the controller then keeps its last-seen → no spurious recreate).
|
||||
func (b *GuestBinder) GuestBootID(ctx context.Context, vmid int) string {
|
||||
pid := b.guestInitPID(ctx, vmid)
|
||||
if pid == "" {
|
||||
return ""
|
||||
}
|
||||
start := procStarttime(pid)
|
||||
start := b.guestInitStart(ctx, vmid)
|
||||
if start == "" {
|
||||
return ""
|
||||
}
|
||||
bt := hostBtime()
|
||||
bt := hostBtimeFn()
|
||||
if bt == "" {
|
||||
bt = "0"
|
||||
}
|
||||
return bt + "-" + start
|
||||
}
|
||||
|
||||
// guestInitStart is the guest init's start time in ticks since the host's boot ("" = the guest does not run, or a
|
||||
// read error). Both boot tokens end with it.
|
||||
func (b *GuestBinder) guestInitStart(ctx context.Context, vmid int) string {
|
||||
pid := b.guestInitPID(ctx, vmid)
|
||||
if pid == "" {
|
||||
return ""
|
||||
}
|
||||
return procStarttimeFn(pid)
|
||||
}
|
||||
|
||||
// GuestBootKey (R-1395) is GuestBootID with the host's boot TIME replaced by the kernel's boot_id:
|
||||
// "<boot_id>-<guest-init-starttime>". The kernel computes the boot time (`btime`) from the wall clock, so a STEP of
|
||||
// the host's clock moves GuestBootID — on the N100 on 2026-11-14 the time service came back, stepped +32.7 s, and the
|
||||
// controller recreated every drive-backed app as if the guest had booted. The kernel's boot_id is a random value made
|
||||
// once per boot; no clock can move it. Same meaning as before: it changes on a host reboot (new boot_id) and on a
|
||||
// guest reboot (new init start), and is stable across an agent or controller restart.
|
||||
//
|
||||
// "" when the guest does not run or the boot_id cannot be read — the controller then decides by GuestBootID, which
|
||||
// is still sent (controllers that do not know the key read only that). Pinned by TestGuestBootKey_*.
|
||||
func (b *GuestBinder) GuestBootKey(ctx context.Context, vmid int) string {
|
||||
start := b.guestInitStart(ctx, vmid)
|
||||
return guestBootKey(hostBootIDFn(), start)
|
||||
}
|
||||
|
||||
// GuestBootTokens returns both tokens from ONE read of the guest's init: the old token and the key always describe
|
||||
// the same moment. Read separately, one could succeed and the other fail (the guest stops between the two) — the
|
||||
// controller would then get an old token with no key, read „an older agent”, and recreate every drive-backed app.
|
||||
func (b *GuestBinder) GuestBootTokens(ctx context.Context, vmid int) (id, key string) {
|
||||
start := b.guestInitStart(ctx, vmid)
|
||||
if start == "" {
|
||||
return "", ""
|
||||
}
|
||||
bt := hostBtimeFn()
|
||||
if bt == "" {
|
||||
bt = "0"
|
||||
}
|
||||
return bt + "-" + start, guestBootKey(hostBootIDFn(), start)
|
||||
}
|
||||
|
||||
// guestBootKey joins the two halves; "" when either is missing or the boot_id is not the kernel's shape.
|
||||
func guestBootKey(bootID, start string) string {
|
||||
if start == "" || !kernelBootIDRe.MatchString(bootID) {
|
||||
return ""
|
||||
}
|
||||
return bootID + "-" + start
|
||||
}
|
||||
|
||||
var kernelBootIDRe = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`)
|
||||
|
||||
// The three reads of /proc behind the boot tokens, as seams (the tests move the boot time and keep the boot_id).
|
||||
var (
|
||||
hostBtimeFn = hostBtime
|
||||
hostBootIDFn = hostBootID
|
||||
procStarttimeFn = procStarttime
|
||||
)
|
||||
|
||||
// hostBootID returns the kernel's boot_id (/proc/sys/kernel/random/boot_id): random, made once per boot. "" on error.
|
||||
func hostBootID() string {
|
||||
data, err := os.ReadFile("/proc/sys/kernel/random/boot_id")
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(data))
|
||||
}
|
||||
|
||||
// procStarttime returns field 22 (starttime) of /proc/<pid>/stat. The comm field (2) can contain spaces
|
||||
// and parentheses, so we split AFTER the last ')': field 22 is index 19 of the post-comm fields
|
||||
// (field 3 = state is index 0). "" on any error.
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||
)
|
||||
|
||||
// LineTestReader is the R-1247 weekly upload line test (internal/linetest.Tester) as the local API reads it.
|
||||
type LineTestReader interface {
|
||||
Value() (mbps float64, at time.Time, ok bool)
|
||||
}
|
||||
|
||||
// LineTestRequester is the R-1250 half: a due test the uplink never lets start, and a test on request. The Tester
|
||||
// satisfies it; a reader without it answers waiting_for_quiet:false and 503 on the POST.
|
||||
type LineTestRequester interface {
|
||||
WaitingForQuiet() bool
|
||||
RunNow(ctx context.Context) (mbps float64, runs []float64, err error)
|
||||
}
|
||||
|
||||
// LineTestResponse is GET /host/line-test. Present=false: never measured, or the job is not wired (no tunnel, disabled,
|
||||
// or an agent before the route answers 404 — the controller reads all three as „unknown").
|
||||
type LineTestResponse struct {
|
||||
Present bool `json:"present"`
|
||||
Mbps float64 `json:"mbps,omitempty"`
|
||||
MeasuredAt string `json:"measured_at,omitempty"` // RFC3339 UTC
|
||||
Method string `json:"method,omitempty"` // "iperf3-tunnel": a 10-second upload to ep0 inside the tunnel
|
||||
// WaitingForQuiet: a measurement is due and could start now, but the uplink has not been quiet for 10 minutes
|
||||
// (R-1250). The controller with a seeding torrent app then holds its limit and POSTs /host/line-test/run.
|
||||
WaitingForQuiet bool `json:"waiting_for_quiet,omitempty"`
|
||||
}
|
||||
|
||||
// LineTestRunResponse is POST /host/line-test/run. OK=false with Refused: nothing ran (Refused is linetest.Refusal's
|
||||
// code, or "server-busy"); OK=false without it: the runs failed.
|
||||
type LineTestRunResponse struct {
|
||||
OK bool `json:"ok"`
|
||||
Mbps float64 `json:"mbps,omitempty"`
|
||||
RunsMbps []float64 `json:"runs_mbps,omitempty"`
|
||||
Refused string `json:"refused,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// SetLineTest wires the weekly line test (main.go builds it after the local API server).
|
||||
func (s *Server) SetLineTest(r LineTestReader) { s.lineTest = r }
|
||||
|
||||
func (s *Server) handleLineTest(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
if s.lineTest == nil {
|
||||
writeOK(w, LineTestResponse{Present: false})
|
||||
return
|
||||
}
|
||||
var waiting bool
|
||||
if q, ok := s.lineTest.(LineTestRequester); ok {
|
||||
waiting = q.WaitingForQuiet()
|
||||
}
|
||||
mbps, at, ok := s.lineTest.Value()
|
||||
if !ok {
|
||||
writeOK(w, LineTestResponse{Present: false, WaitingForQuiet: waiting})
|
||||
return
|
||||
}
|
||||
s.logger.Debug("local-api: /host/line-test served", "vmid", vmid, "mbps", mbps, "measured_at", at.UTC().Format(time.RFC3339), "waiting_for_quiet", waiting)
|
||||
writeOK(w, LineTestResponse{Present: true, Mbps: mbps, MeasuredAt: at.UTC().Format(time.RFC3339), Method: "iperf3-tunnel", WaitingForQuiet: waiting})
|
||||
}
|
||||
|
||||
// handleLineTestRun is POST /host/line-test/run (R-1250): a test on request, synchronous (about 55 s). The answer is
|
||||
// always 200 with the outcome in the body, except 503 when no line test is wired.
|
||||
func (s *Server) handleLineTestRun(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||
q, ok := s.lineTest.(LineTestRequester)
|
||||
if s.lineTest == nil || !ok {
|
||||
writeErr(w, http.StatusServiceUnavailable, "line test not configured")
|
||||
return
|
||||
}
|
||||
s.logger.Info("local-api: line test requested", "vmid", vmid)
|
||||
mbps, runs, err := q.RunNow(r.Context())
|
||||
var ref *linetest.Refusal
|
||||
switch {
|
||||
case err == nil:
|
||||
writeOK(w, LineTestRunResponse{OK: true, Mbps: mbps, RunsMbps: runs})
|
||||
case errors.As(err, &ref):
|
||||
s.logger.Info("local-api: line test request refused", "vmid", vmid, "reason", ref.Code, "detail", ref.Msg)
|
||||
writeOK(w, LineTestRunResponse{Refused: ref.Code, Error: ref.Msg})
|
||||
case errors.Is(err, linetest.ErrBusy):
|
||||
writeOK(w, LineTestRunResponse{Refused: "server-busy", Error: err.Error()})
|
||||
default:
|
||||
writeOK(w, LineTestRunResponse{Error: err.Error()})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||
)
|
||||
|
||||
type fixedLine struct {
|
||||
mbps float64
|
||||
ok bool
|
||||
}
|
||||
|
||||
func (f fixedLine) Value() (float64, time.Time, bool) {
|
||||
return f.mbps, time.Date(2026, 10, 24, 6, 31, 0, 0, time.UTC), f.ok
|
||||
}
|
||||
|
||||
func lineTestBody(t *testing.T, s *Server) LineTestResponse {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
s.handleLineTest(rec, httptest.NewRequest("GET", "/host/line-test", nil), 9201)
|
||||
var env struct {
|
||||
OK bool `json:"ok"`
|
||||
Data LineTestResponse `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
|
||||
t.Fatalf("decode %q: %v", rec.Body.String(), err)
|
||||
}
|
||||
return env.Data
|
||||
}
|
||||
|
||||
// R-1247: the controller sets qBittorrent's limit only from a PRESENT value; never measured and not wired must both
|
||||
// read present:false, never a zero line.
|
||||
func TestLineTestRoute(t *testing.T) {
|
||||
s := &Server{logger: slog.Default()}
|
||||
if got := lineTestBody(t, s); got.Present {
|
||||
t.Fatalf("not wired: %+v, want present:false", got)
|
||||
}
|
||||
s.SetLineTest(fixedLine{ok: false})
|
||||
if got := lineTestBody(t, s); got.Present || got.Mbps != 0 {
|
||||
t.Fatalf("never measured: %+v, want present:false", got)
|
||||
}
|
||||
s.SetLineTest(fixedLine{mbps: 90.7, ok: true})
|
||||
got := lineTestBody(t, s)
|
||||
if !got.Present || got.Mbps != 90.7 || got.Method != "iperf3-tunnel" || !strings.HasPrefix(got.MeasuredAt, "2026-10-24T06:31") {
|
||||
t.Fatalf("measured: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
type requestableLine struct {
|
||||
fixedLine
|
||||
waiting bool
|
||||
mbps float64
|
||||
err error
|
||||
calls int
|
||||
}
|
||||
|
||||
func (r *requestableLine) WaitingForQuiet() bool { return r.waiting }
|
||||
func (r *requestableLine) RunNow(context.Context) (float64, []float64, error) {
|
||||
r.calls++
|
||||
if r.err != nil {
|
||||
return 0, nil, r.err
|
||||
}
|
||||
return r.mbps, []float64{r.mbps - 1, r.mbps}, nil
|
||||
}
|
||||
|
||||
func lineTestRun(t *testing.T, s *Server) (int, LineTestRunResponse) {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
s.handleLineTestRun(rec, httptest.NewRequest("POST", "/host/line-test/run", nil), 9201)
|
||||
var env struct {
|
||||
OK bool `json:"ok"`
|
||||
Data LineTestRunResponse `json:"data"`
|
||||
}
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &env)
|
||||
return rec.Code, env.Data
|
||||
}
|
||||
|
||||
// R-1250: the controller acts only on waiting_for_quiet, and reads a refusal by its code — never as a measurement.
|
||||
func TestLineTestRunRoute(t *testing.T) {
|
||||
s := &Server{logger: slog.Default()}
|
||||
if code, _ := lineTestRun(t, s); code != 503 {
|
||||
t.Fatalf("not wired: HTTP %d, want 503", code)
|
||||
}
|
||||
s.SetLineTest(fixedLine{ok: true, mbps: 90})
|
||||
if code, _ := lineTestRun(t, s); code != 503 {
|
||||
t.Fatalf("a reader without the request half: HTTP %d, want 503", code)
|
||||
}
|
||||
l := &requestableLine{fixedLine: fixedLine{ok: false}, waiting: true, mbps: 93.5}
|
||||
s.SetLineTest(l)
|
||||
if got := lineTestBody(t, s); got.Present || !got.WaitingForQuiet {
|
||||
t.Fatalf("GET while waiting: %+v, want present:false waiting_for_quiet:true", got)
|
||||
}
|
||||
if code, got := lineTestRun(t, s); code != 200 || !got.OK || got.Mbps != 93.5 || len(got.RunsMbps) != 2 {
|
||||
t.Fatalf("a measured request: HTTP %d %+v", code, got)
|
||||
}
|
||||
l.err = &linetest.Refusal{Code: "not-quiet", Msg: "the uplink sent 16.0 Mbit/s while held"}
|
||||
if _, got := lineTestRun(t, s); got.OK || got.Refused != "not-quiet" || got.Mbps != 0 {
|
||||
t.Fatalf("a refusal: %+v, want ok:false refused:not-quiet", got)
|
||||
}
|
||||
l.err = linetest.ErrBusy
|
||||
if _, got := lineTestRun(t, s); got.OK || got.Refused != "server-busy" {
|
||||
t.Fatalf("server busy: %+v, want refused:server-busy", got)
|
||||
}
|
||||
l.err = errors.New("dial tcp 10.77.0.1:5201: i/o timeout")
|
||||
if _, got := lineTestRun(t, s); got.OK || got.Refused != "" || got.Error == "" {
|
||||
t.Fatalf("a failed run: %+v, want ok:false, no refusal, the error", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// The operator's actions for THIS agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).
|
||||
//
|
||||
// THE DOOR IS THE HOST REPORT'S REPLY, the controller's shape: the operator presses a button on the hub's host page, the
|
||||
// hub stores a row and pokes the box (the poke's out-of-cycle report gets the reply in seconds; without the tunnel the
|
||||
// 15-minute cycle carries it), the reply lists `operator_actions: [{id, action, arg}]`, this agent acts ONCE per id, and
|
||||
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
|
||||
// The result is re-sent until the reply stops listing the id.
|
||||
//
|
||||
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
|
||||
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
|
||||
//
|
||||
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
|
||||
// the guest over the existing grants, with the supervisor's guards — not during a controller swap, not parked, not on a
|
||||
// stopped, locked or vzdump-busy guest, not when the guest list or the vzdump state cannot be read — and the thrash brake
|
||||
// counts these restarts too: 3 restarts (the supervisor's or the operator's) inside 15 minutes refuse the press, and
|
||||
// presses stay refused for 30 minutes. It works with the controller stopped or stuck: the agent does it, not the
|
||||
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
|
||||
|
||||
const (
|
||||
// OpRestartController is the agent's first operator verb.
|
||||
OpRestartController = "restart_controller"
|
||||
|
||||
opOutcomeDone = "done"
|
||||
opOutcomeRefused = "refused"
|
||||
opOutcomeFailed = "failed"
|
||||
|
||||
// opBrakeRefusal is how long presses stay refused after the brake refused one.
|
||||
opBrakeRefusal = 30 * time.Minute
|
||||
// opForgetAfter drops a started id the reply no longer lists.
|
||||
opForgetAfter = 24 * time.Hour
|
||||
|
||||
defaultOpHealthPoll = 2 * time.Second
|
||||
// defaultOpHealthWait covers the bootstrap's pull-less rm/run (seconds), Docker's start period (10 s) and its first
|
||||
// probe (30 s interval) with room: the controller's own HEALTHCHECK is the answer.
|
||||
defaultOpHealthWait = 3 * time.Minute
|
||||
// opHealthTemplate reads running + health over the swap's `docker inspect -f` grant.
|
||||
opHealthTemplate = "{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}"
|
||||
)
|
||||
|
||||
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
|
||||
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
|
||||
|
||||
// OperatorActionNames returns the closed list (sorted copy).
|
||||
func OperatorActionNames() []string {
|
||||
out := append([]string(nil), operatorActionNames...)
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
type opActionState struct {
|
||||
mu sync.Mutex
|
||||
started map[int64]time.Time
|
||||
results map[int64]hub.OperatorActionResult
|
||||
}
|
||||
|
||||
// SetOperatorActionTrigger wires the report loop's out-of-band trigger (the result goes at once, not in 15 minutes).
|
||||
func (s *Server) SetOperatorActionTrigger(f func()) { s.opTrigger = f }
|
||||
|
||||
// OnEnvelope is the report loop's hook (hub.EnvelopeObserver): it starts each newly listed action once and forgets
|
||||
// the ids the reply no longer lists (the hub has their results).
|
||||
func (s *Server) OnEnvelope(_ context.Context, env *hub.ControlEnvelope) {
|
||||
if env == nil {
|
||||
return
|
||||
}
|
||||
listed := map[int64]bool{}
|
||||
var start []hub.OperatorAction
|
||||
now := time.Now()
|
||||
s.opActs.mu.Lock()
|
||||
if s.opActs.started == nil {
|
||||
s.opActs.started = map[int64]time.Time{}
|
||||
s.opActs.results = map[int64]hub.OperatorActionResult{}
|
||||
}
|
||||
for _, a := range env.OperatorActions {
|
||||
listed[a.ID] = true
|
||||
if _, ok := s.opActs.started[a.ID]; ok {
|
||||
continue
|
||||
}
|
||||
s.opActs.started[a.ID] = now
|
||||
start = append(start, a)
|
||||
}
|
||||
for id, at := range s.opActs.started {
|
||||
if listed[id] {
|
||||
continue
|
||||
}
|
||||
if _, answered := s.opActs.results[id]; answered || now.Sub(at) > opForgetAfter {
|
||||
delete(s.opActs.started, id)
|
||||
delete(s.opActs.results, id)
|
||||
}
|
||||
}
|
||||
s.opActs.mu.Unlock()
|
||||
for _, a := range start {
|
||||
go s.runOperatorAction(a) // a restart waits for the health answer — never block the report loop
|
||||
}
|
||||
}
|
||||
|
||||
// OperatorActionResults is the report's answer list, oldest id first.
|
||||
func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
|
||||
s.opActs.mu.Lock()
|
||||
defer s.opActs.mu.Unlock()
|
||||
out := make([]hub.OperatorActionResult, 0, len(s.opActs.results))
|
||||
for _, r := range s.opActs.results {
|
||||
out = append(out, r)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *Server) runOperatorAction(a hub.OperatorAction) {
|
||||
limit := s.opWait() + time.Minute
|
||||
if a.Action == OpRebootGuest {
|
||||
limit += opRebootTimeout
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), limit)
|
||||
defer cancel()
|
||||
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
|
||||
switch a.Action {
|
||||
case OpRestartController:
|
||||
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
|
||||
case OpRebootGuest:
|
||||
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
|
||||
}
|
||||
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
|
||||
"outcome", outcome, "message", msg)
|
||||
s.opActs.mu.Lock()
|
||||
if _, still := s.opActs.started[a.ID]; still {
|
||||
s.opActs.results[a.ID] = hub.OperatorActionResult{ID: a.ID, Outcome: outcome, Message: msg}
|
||||
}
|
||||
s.opActs.mu.Unlock()
|
||||
if s.opTrigger != nil {
|
||||
s.opTrigger()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) opPoll() time.Duration {
|
||||
if s.opHealthPoll > 0 {
|
||||
return s.opHealthPoll
|
||||
}
|
||||
return defaultOpHealthPoll
|
||||
}
|
||||
|
||||
func (s *Server) opWait() time.Duration {
|
||||
if s.opHealthWait > 0 {
|
||||
return s.opHealthWait
|
||||
}
|
||||
return defaultOpHealthWait
|
||||
}
|
||||
|
||||
// operatorRestartController is the press: the supervisor's guards, the brake, the restart, then the wait for the
|
||||
// controller's healthy answer. Returns the outcome and the sentence the host page shows.
|
||||
func (s *Server) operatorRestartController(ctx context.Context, id int64, arg string) (string, string) {
|
||||
vmid, err := strconv.Atoi(arg)
|
||||
if err != nil || strconv.Itoa(vmid) != arg {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||
}
|
||||
if s.staleLock == nil || s.guestExec == nil {
|
||||
return opOutcomeRefused, "refused: the controller supervisor is not wired on this agent"
|
||||
}
|
||||
if !s.provisionedGuest(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||
}
|
||||
status := ""
|
||||
for _, g := range guests {
|
||||
if g.VMID == vmid {
|
||||
status = g.Status
|
||||
}
|
||||
}
|
||||
if status != "running" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||
}
|
||||
if s.controllerParked(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||
}
|
||||
s.swapMu.Lock()
|
||||
swapping := s.swapInFlight[vmid]
|
||||
s.swapMu.Unlock()
|
||||
if swapping {
|
||||
return opOutcomeRefused, "refused: a controller swap (update) is in flight — it owns the controller"
|
||||
}
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||
} else if lock != "" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||
}
|
||||
if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||
return opOutcomeRefused, "refused: a whole-guest backup (vzdump) may be in flight for the guest"
|
||||
}
|
||||
|
||||
// The brake: every restart counts — the supervisor's and the operator's.
|
||||
now := s.clock()
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if now.Before(st.opBrakeUntil) {
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||
if len(st.restarts) >= controllerCrashloopMax {
|
||||
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
reason := fmt.Sprintf("the operator via the hub (action #%d)", id)
|
||||
st.restarts = append(st.restarts, now)
|
||||
st.restartsTotal++
|
||||
st.lastRestartAt = now
|
||||
st.lastReason = reason
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
|
||||
s.logger.Warn("controller-supervisor: RESTARTING the controller on the operator's press", "vmid", vmid, "action_id", id, "unit", bootstrapUnit)
|
||||
began := time.Now()
|
||||
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||
return opOutcomeFailed, fmt.Sprintf("the restart of guest %d's controller failed: %v", vmid, err)
|
||||
}
|
||||
for time.Since(began) < s.opWait() {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||
if err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||
secs := int(time.Since(began).Round(time.Second) / time.Second)
|
||||
return opOutcomeDone, fmt.Sprintf("restarted; the controller answered after %d s (Docker's health check: healthy)", secs)
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return opOutcomeFailed, "restarted, but the wait for its answer was cut"
|
||||
case <-time.After(s.opPoll()):
|
||||
}
|
||||
}
|
||||
return opOutcomeFailed, fmt.Sprintf("restarted, but no healthy answer within %s — look at the box", s.opWait().Round(time.Second))
|
||||
}
|
||||
|
||||
func orDash(s string) string {
|
||||
if s == "" {
|
||||
return "not listed"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// Operator brief 2026-10-29, Part 1: „Restart controller" from the hub. The agent's closed list holds one verb; the act
|
||||
// is the supervisor's own restart with the supervisor's guards, and the brake counts operator restarts too.
|
||||
|
||||
// opExec answers the supervisor's fake plus the health read the operator restart waits on.
|
||||
type opExec struct {
|
||||
*supExec
|
||||
health string // what {{.State.Health.Status}} reads after the restart
|
||||
}
|
||||
|
||||
func (f *opExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||
if len(args) >= 5 && args[0] == "docker" && args[1] == "inspect" && args[3] == opHealthTemplate {
|
||||
f.mu.Lock()
|
||||
st := f.status[vmid]
|
||||
f.mu.Unlock()
|
||||
if st == "" {
|
||||
return "", errors.New("pct exec: exit status 1: Error: No such object: felhom-controller")
|
||||
}
|
||||
return map[bool]string{true: "true", false: "false"}[st == "running"] + "|" + f.health + "\n", nil
|
||||
}
|
||||
return f.supExec.GuestExec(ctx, vmid, args...)
|
||||
}
|
||||
|
||||
func opServer(t *testing.T, status string) (*Server, *opExec, *fakeGuestPowerCtl, *supClock, string, *atomic.Int32) {
|
||||
t.Helper()
|
||||
ex := &opExec{supExec: &supExec{status: map[int]string{9201: status}, onRestart: "running"}, health: "healthy"}
|
||||
ctl := runningGuest(9201)
|
||||
s, clk, dir := supServer(t, nil, ctl, 9201)
|
||||
s.guestExec = ex
|
||||
s.opHealthPoll = time.Millisecond
|
||||
s.opHealthWait = 2 * time.Second
|
||||
var triggered atomic.Int32
|
||||
s.SetOperatorActionTrigger(func() { triggered.Add(1) })
|
||||
return s, ex, ctl, clk, dir, &triggered
|
||||
}
|
||||
|
||||
func env(acts ...hub.OperatorAction) *hub.ControlEnvelope {
|
||||
return &hub.ControlEnvelope{OperatorActions: acts}
|
||||
}
|
||||
|
||||
// waitResult waits for the action's result (the act runs in the background).
|
||||
func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
for _, r := range s.OperatorActionResults() {
|
||||
if r.ID == id {
|
||||
return r
|
||||
}
|
||||
}
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("no result for #%d", id)
|
||||
return hub.OperatorActionResult{}
|
||||
}
|
||||
|
||||
// RED-PROOF: seen failing before the list existed (build failure, then the names).
|
||||
func TestOpActions_AgentClosedList(t *testing.T) {
|
||||
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Anything outside the list — and a bad argument — is refused and NOTHING is called.
|
||||
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "running")
|
||||
s.OnEnvelope(context.Background(), env(
|
||||
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
|
||||
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
|
||||
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
|
||||
))
|
||||
for _, id := range []int64{1, 2, 3} {
|
||||
if r := waitResult(t, s, id); r.Outcome != "refused" {
|
||||
t.Errorf("#%d: %+v, want refused", id, r)
|
||||
}
|
||||
}
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("a refused action restarted the controller (%d)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The press restarts ONCE, says how long the controller took to answer, asks for a report at once, and re-sends the
|
||||
// result until the hub stops listing the id. RED-PROOF: drop the GuestExec restart → restarts 0 → FAILS.
|
||||
func TestOpActions_RestartReportsTheAnswerTime(t *testing.T) {
|
||||
s, ex, _, _, _, trig := opServer(t, "running")
|
||||
a := hub.OperatorAction{ID: 7, Action: "restart_controller", Arg: "9201"}
|
||||
s.OnEnvelope(context.Background(), env(a))
|
||||
r := waitResult(t, s, 7)
|
||||
if r.Outcome != "done" || !strings.Contains(r.Message, "answered after") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("restarts = %d, want 1", n)
|
||||
}
|
||||
if trig.Load() == 0 {
|
||||
t.Fatal("the agent did not ask for a report — the result would wait 15 minutes")
|
||||
}
|
||||
s.OnEnvelope(context.Background(), env(a)) // the hub still lists it: re-send, never act again
|
||||
time.Sleep(100 * time.Millisecond) // a wrong second act runs in the background — give it time to show
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("a re-listed id acted again (%d)", n)
|
||||
}
|
||||
if len(s.OperatorActionResults()) != 1 {
|
||||
t.Fatal("the result must be re-sent while listed")
|
||||
}
|
||||
s.OnEnvelope(context.Background(), env()) // the hub recorded it
|
||||
if left := s.OperatorActionResults(); len(left) != 0 {
|
||||
t.Fatalf("a recorded result is still sent: %+v", left)
|
||||
}
|
||||
}
|
||||
|
||||
// The point of the agent doing it: a STOPPED controller is restarted too (the controller cannot restart itself).
|
||||
func TestOpActions_WorksWithTheControllerStopped(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "exited")
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 8, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 8); r.Outcome != "done" || ex.count(9201) != 1 {
|
||||
t.Fatalf("result %+v, restarts %d", r, ex.count(9201))
|
||||
}
|
||||
}
|
||||
|
||||
// Each supervisor guard refuses the press, with the reason. RED-PROOF: drop the lock check → the locked case
|
||||
// restarts → FAILS (each guard seen failing by its own mutation).
|
||||
func TestOpActions_GuardsRefuse(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, want string
|
||||
setup func(s *Server, ctl *fakeGuestPowerCtl, dir string)
|
||||
}{
|
||||
{"parked", "parked", func(_ *Server, _ *fakeGuestPowerCtl, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
{"swap", "swap", func(s *Server, _ *fakeGuestPowerCtl, _ string) { s.swapInFlight[9201] = true }},
|
||||
{"locked", "locked", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.locks[9201] = "backup" }},
|
||||
{"vzdump", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||
{"vzdump unknown", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupErr = errors.New("tasks unreadable") }},
|
||||
{"guest stopped", "not running", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guests[0].Status = "stopped" }},
|
||||
{"guest list", "guest list", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guestsErr = errors.New("api down") }},
|
||||
}
|
||||
for i, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, ctl, _, dir, _ := opServer(t, "running")
|
||||
tc.setup(s, ctl, dir)
|
||||
id := int64(100 + i)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
r := waitResult(t, s, id)
|
||||
if r.Outcome != "refused" || !strings.Contains(strings.ToLower(r.Message), tc.want) {
|
||||
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||
}
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("guard %q did not hold: restarted %d", tc.name, n)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The thrash brake counts operator restarts AND the supervisor's: 3 in 15 minutes → the next press is refused, and
|
||||
// presses stay refused for 30 minutes. RED-PROOF: drop the brake → the 4th press restarts → FAILS.
|
||||
func TestOpActions_TheBrakeCountsEveryRestart(t *testing.T) {
|
||||
s, ex, _, clk, _, _ := opServer(t, "running")
|
||||
// one restart by the supervisor itself
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "exited"
|
||||
ex.mu.Unlock()
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
if ex.count(9201) != 1 {
|
||||
t.Fatalf("setup: the supervisor did not restart (%d)", ex.count(9201))
|
||||
}
|
||||
for id := int64(1); id <= 2; id++ {
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||
t.Fatalf("press %d: %+v", id, r)
|
||||
}
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9201"}))
|
||||
r := waitResult(t, s, 3)
|
||||
if r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||
t.Fatalf("the 4th restart in 15 minutes: %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 3 {
|
||||
t.Fatalf("restarts = %d, want 3", n)
|
||||
}
|
||||
clk.t = clk.t.Add(20 * time.Minute) // the window has passed, but the 30-minute refusal has not
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 4); r.Outcome != "refused" {
|
||||
t.Fatalf("inside the 30-minute refusal: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(11 * time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 5, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 5); r.Outcome != "done" {
|
||||
t.Fatalf("after the refusal: %+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
// A controller that does not come back healthy is a FAILED press, said plainly.
|
||||
func TestOpActions_NoHealthyAnswerIsFailed(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "running")
|
||||
ex.health = "unhealthy"
|
||||
s.opHealthWait = 30 * time.Millisecond
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 9, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 9); r.Outcome != "failed" || !strings.Contains(r.Message, "no healthy answer") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,329 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
|
||||
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
|
||||
//
|
||||
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
|
||||
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
|
||||
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
|
||||
//
|
||||
// THE GUARDS (each refuses with the reason in the result; nothing is called):
|
||||
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
|
||||
// - a controller swap (update) is in flight;
|
||||
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
|
||||
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
|
||||
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
|
||||
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
|
||||
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
|
||||
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
|
||||
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
|
||||
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
|
||||
// alone while the reboot runs (rebootInFlight).
|
||||
//
|
||||
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
|
||||
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
|
||||
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
|
||||
// one day). Pinned by opreboot_test.go.
|
||||
|
||||
const (
|
||||
// OpRebootGuest is the agent's second operator verb (D65 = A).
|
||||
OpRebootGuest = "reboot_guest"
|
||||
|
||||
// opRebootEvery: at most one operator reboot per guest per hour.
|
||||
opRebootEvery = time.Hour
|
||||
// opRebootNoticeFor: how long the household's dashboard shows the line.
|
||||
opRebootNoticeFor = 24 * time.Hour
|
||||
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
|
||||
opRebootTimeout = 5 * time.Minute
|
||||
|
||||
operatorRebootFile = "operator-reboot.json"
|
||||
|
||||
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
|
||||
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
|
||||
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
|
||||
// Docker does not hide it.
|
||||
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
|
||||
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
|
||||
// stopped writing (stuck or killed) — not a busy one.
|
||||
controllerBusyFresh = 90 * time.Second
|
||||
)
|
||||
|
||||
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
|
||||
type operatorRebootRecord struct {
|
||||
At time.Time `json:"at"`
|
||||
ActionID int64 `json:"action_id"`
|
||||
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
|
||||
ControllerAfterS int `json:"controller_after_s,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) operatorRebootPath(vmid int) string {
|
||||
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
|
||||
}
|
||||
|
||||
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
|
||||
// must not silently vanish).
|
||||
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
|
||||
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return operatorRebootRecord{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return operatorRebootRecord{}, false, err
|
||||
}
|
||||
var rec operatorRebootRecord
|
||||
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
|
||||
}
|
||||
return rec, !rec.At.IsZero(), nil
|
||||
}
|
||||
|
||||
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
|
||||
raw, err := json.Marshal(rec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p := s.operatorRebootPath(vmid)
|
||||
tmp := p + ".tmp"
|
||||
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, p)
|
||||
}
|
||||
|
||||
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
|
||||
func (s *Server) rebooting(vmid int) bool {
|
||||
s.rebootMu.Lock()
|
||||
defer s.rebootMu.Unlock()
|
||||
return s.rebootInFlight[vmid]
|
||||
}
|
||||
|
||||
func (s *Server) setRebooting(vmid int, on bool) {
|
||||
s.rebootMu.Lock()
|
||||
defer s.rebootMu.Unlock()
|
||||
if s.rebootInFlight == nil {
|
||||
s.rebootInFlight = map[int]bool{}
|
||||
}
|
||||
if on {
|
||||
s.rebootInFlight[vmid] = true
|
||||
} else {
|
||||
delete(s.rebootInFlight, vmid)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) rebooter() interface {
|
||||
RebootGuest(ctx context.Context, vmid int) error
|
||||
} {
|
||||
if s.opRebooter != nil {
|
||||
return s.opRebooter
|
||||
}
|
||||
if s.guestAttach != nil {
|
||||
return s.guestAttach
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
|
||||
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
|
||||
// (the caller refuses — it cannot prove no backup runs). Two callers: the operator's reboot (D65) and R-1344's restart
|
||||
// of a controller that does not answer (controllersupervisor.go restartUnresponsive).
|
||||
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
|
||||
if err != nil {
|
||||
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
|
||||
return false, "", nil
|
||||
}
|
||||
return false, "", err
|
||||
}
|
||||
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
|
||||
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
|
||||
if perr != nil {
|
||||
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
|
||||
}
|
||||
if s.clock().Sub(at) > controllerBusyFresh {
|
||||
s.logger.Info("controller-busy: the controller's busy file is stale — the controller stopped writing it; not busy",
|
||||
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
|
||||
return false, "", nil
|
||||
}
|
||||
why := "work in progress"
|
||||
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
|
||||
why = strings.TrimSpace(lines[1])
|
||||
}
|
||||
return true, why, nil
|
||||
}
|
||||
|
||||
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
|
||||
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
|
||||
vmid, err := strconv.Atoi(arg)
|
||||
if err != nil || strconv.Itoa(vmid) != arg {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||
}
|
||||
rb := s.rebooter()
|
||||
if s.staleLock == nil || s.guestExec == nil || rb == nil {
|
||||
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
|
||||
}
|
||||
if !s.provisionedGuest(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||
}
|
||||
if s.rebooting(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||
}
|
||||
status := ""
|
||||
for _, g := range guests {
|
||||
if g.VMID == vmid {
|
||||
status = g.Status
|
||||
}
|
||||
}
|
||||
if status != "running" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||
}
|
||||
if s.controllerParked(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||
}
|
||||
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||
} else if lock != "" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||
}
|
||||
|
||||
// At most one per hour.
|
||||
now := s.clock()
|
||||
last, had, err := s.lastOperatorReboot(vmid)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
|
||||
}
|
||||
if had && now.Sub(last.At) < opRebootEvery {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
|
||||
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
|
||||
}
|
||||
|
||||
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if now.Before(st.opBrakeUntil) {
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||
if len(st.restarts) >= controllerCrashloopMax {
|
||||
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
s.ctrlSup.mu.Unlock()
|
||||
|
||||
// The heavy-operation gate, held until the guest is up again.
|
||||
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
|
||||
if !ok {
|
||||
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
|
||||
}
|
||||
defer release()
|
||||
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
|
||||
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
|
||||
}
|
||||
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
|
||||
} else if cbusy {
|
||||
return opOutcomeRefused, "refused: the controller is busy — " + why
|
||||
}
|
||||
|
||||
// The record FIRST: the hourly limit and the household's line both read it.
|
||||
rec := operatorRebootRecord{At: now, ActionID: id}
|
||||
if err := s.saveOperatorReboot(vmid, rec); err != nil {
|
||||
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
|
||||
}
|
||||
s.ctrlSup.mu.Lock()
|
||||
st = s.supState(vmid)
|
||||
st.restarts = append(st.restarts, now)
|
||||
st.restartsTotal++
|
||||
st.lastRestartAt = now
|
||||
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
s.setRebooting(vmid, true)
|
||||
defer s.setRebooting(vmid, false)
|
||||
|
||||
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
|
||||
began := time.Now()
|
||||
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
|
||||
err = rb.RebootGuest(rctx, vmid)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
|
||||
}
|
||||
upS := int(time.Since(began).Round(time.Second) / time.Second)
|
||||
rec.GuestUpAfterS = upS
|
||||
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
|
||||
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||
if err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
|
||||
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
|
||||
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
|
||||
}
|
||||
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
|
||||
case <-time.After(s.opPoll()):
|
||||
}
|
||||
}
|
||||
_ = s.saveOperatorReboot(vmid, rec)
|
||||
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
|
||||
}
|
||||
|
||||
// rebootBlockerSentence names jobBlocker's code in the result's words.
|
||||
func rebootBlockerSentence(code string) string {
|
||||
switch code {
|
||||
case PowerOffBackup:
|
||||
return "a backup is running"
|
||||
case PowerOffCtrlUpdate:
|
||||
return "a controller update is running"
|
||||
case PowerOffDrive:
|
||||
return "a drive or network-storage job is running"
|
||||
case PowerOffEscrow:
|
||||
return "an escrow ceremony is running"
|
||||
case PowerOffUnknown:
|
||||
return "the running jobs cannot be read"
|
||||
default:
|
||||
return "the host is busy"
|
||||
}
|
||||
}
|
||||
|
||||
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
|
||||
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
|
||||
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
|
||||
rec, ok, err := s.lastOperatorReboot(vmid)
|
||||
if err != nil {
|
||||
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
|
||||
return
|
||||
}
|
||||
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
|
||||
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
|
||||
return
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
|
||||
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
|
||||
|
||||
// rbExec adds the controller's busy file to the operator-restart fake.
|
||||
type rbExec struct {
|
||||
*opExec
|
||||
busy string // the file's content; "" = no file
|
||||
busyErr error // pct exec itself failed
|
||||
}
|
||||
|
||||
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
|
||||
if f.busyErr != nil {
|
||||
return "", f.busyErr
|
||||
}
|
||||
if f.busy == "" {
|
||||
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
|
||||
}
|
||||
return f.busy, nil
|
||||
}
|
||||
return f.opExec.GuestExec(ctx, vmid, args...)
|
||||
}
|
||||
|
||||
type fakeRebooter struct {
|
||||
mu sync.Mutex
|
||||
n int
|
||||
err error
|
||||
after func()
|
||||
}
|
||||
|
||||
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.n++
|
||||
if r.after != nil {
|
||||
r.after()
|
||||
}
|
||||
return r.err
|
||||
}
|
||||
|
||||
func (r *fakeRebooter) count() int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.n
|
||||
}
|
||||
|
||||
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
|
||||
t.Helper()
|
||||
s, ox, ctl, clk, dir, _ := opServer(t, "running")
|
||||
ex := &rbExec{opExec: ox}
|
||||
s.guestExec = ex
|
||||
rb := &fakeRebooter{}
|
||||
s.opRebooter = rb
|
||||
s.inFlight = &backup.InFlight{}
|
||||
return s, ex, rb, ctl, clk, dir
|
||||
}
|
||||
|
||||
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||
t.Helper()
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
|
||||
return waitResult(t, s, id)
|
||||
}
|
||||
|
||||
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
|
||||
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
|
||||
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
|
||||
s, _, rb, _, clk, _ := rbServer(t)
|
||||
r := pressReboot(t, s, 1)
|
||||
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if rb.count() != 1 {
|
||||
t.Fatalf("reboots = %d, want 1", rb.count())
|
||||
}
|
||||
rec, ok, err := s.lastOperatorReboot(9201)
|
||||
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
|
||||
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
|
||||
}
|
||||
if s.inFlight.Busy() != "" {
|
||||
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
|
||||
}
|
||||
if s.rebooting(9201) {
|
||||
t.Fatal("the reboot-in-flight flag outlived the reboot")
|
||||
}
|
||||
}
|
||||
|
||||
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
|
||||
func TestOpReboot_GuardsRefuse(t *testing.T) {
|
||||
fresh := func(clk *supClock) string {
|
||||
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
|
||||
}
|
||||
cases := []struct {
|
||||
name, want string
|
||||
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
|
||||
}{
|
||||
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
|
||||
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
|
||||
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.guests[0].Status = "stopped"
|
||||
}},
|
||||
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.backupRun = map[int]bool{9201: true}
|
||||
}},
|
||||
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.backupErr = errors.New("tasks unreadable")
|
||||
}},
|
||||
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
s.inFlight.TryAcquire("backup:felhom-pbs")
|
||||
}},
|
||||
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
|
||||
}},
|
||||
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||
ex.busy = fresh(clk)
|
||||
}},
|
||||
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
|
||||
}},
|
||||
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||
s.ctrlSup.mu.Lock()
|
||||
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
|
||||
s.ctrlSup.mu.Unlock()
|
||||
}},
|
||||
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for i, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, rb, ctl, clk, dir := rbServer(t)
|
||||
tc.setup(s, ex, ctl, clk, dir)
|
||||
r := pressReboot(t, s, int64(100+i))
|
||||
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
|
||||
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||
}
|
||||
if rb.count() != 0 {
|
||||
t.Fatalf("guard %q did not hold: rebooted", tc.name)
|
||||
}
|
||||
if _, ok, _ := s.lastOperatorReboot(9201); ok {
|
||||
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
|
||||
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
|
||||
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
|
||||
s, ex, rb, _, clk, _ := rbServer(t)
|
||||
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
|
||||
t.Fatalf("result %+v, reboots %d", r, rb.count())
|
||||
}
|
||||
}
|
||||
|
||||
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
|
||||
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
|
||||
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
|
||||
s, _, rb, ctl, clk, dir := rbServer(t)
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("first press: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(59 * time.Minute)
|
||||
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
|
||||
t.Fatalf("second press inside the hour: %+v", r)
|
||||
}
|
||||
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
|
||||
s2, ox2, _, _, _, _ := opServer(t, "running")
|
||||
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
|
||||
s2.guestExec = &rbExec{opExec: ox2}
|
||||
rb2 := &fakeRebooter{}
|
||||
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
|
||||
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
|
||||
t.Fatalf("after an agent restart, inside the hour: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(2 * time.Minute)
|
||||
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
|
||||
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
|
||||
}
|
||||
}
|
||||
|
||||
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
|
||||
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
|
||||
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
|
||||
s, ex, _, _, clk, _ := rbServer(t)
|
||||
for id := int64(1); id <= 2; id++ {
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||
t.Fatalf("restart %d: %+v", id, r)
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
}
|
||||
if r := pressReboot(t, s, 3); r.Outcome != "done" {
|
||||
t.Fatalf("reboot: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 2 {
|
||||
t.Fatalf("restarts = %d, want 2", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
|
||||
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
|
||||
s, _, rb, _, _, _ := rbServer(t)
|
||||
rb.err = errors.New("pct reboot 9201: exit status 255")
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if s.inFlight.Busy() != "" || s.rebooting(9201) {
|
||||
t.Fatal("the gate or the flag outlived a failed reboot")
|
||||
}
|
||||
}
|
||||
|
||||
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
|
||||
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
|
||||
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
|
||||
s, ex, rb, ctl, _, _ := rbServer(t)
|
||||
var started, restarted int
|
||||
rb.after = func() {
|
||||
ctl.guests[0].Status = "stopped"
|
||||
s.GuestPowerTick(context.Background())
|
||||
ctl.guests[0].Status = "running"
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "exited"
|
||||
ex.mu.Unlock()
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
started, restarted = len(ctl.started), ex.count(9201)
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "running"
|
||||
ex.mu.Unlock()
|
||||
}
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if started != 0 || restarted != 0 {
|
||||
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
|
||||
}
|
||||
}
|
||||
|
||||
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
|
||||
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
|
||||
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
|
||||
s, _, _, _, clk, _ := rbServer(t)
|
||||
read := func() string {
|
||||
w := httptest.NewRecorder()
|
||||
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
|
||||
var body struct {
|
||||
Data struct {
|
||||
At string `json:"at"`
|
||||
} `json:"data"`
|
||||
At string `json:"at"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("body %q: %v", w.Body.String(), err)
|
||||
}
|
||||
if body.Data.At != "" {
|
||||
return body.Data.At
|
||||
}
|
||||
return body.At
|
||||
}
|
||||
if got := read(); got != "" {
|
||||
t.Fatalf("no reboot yet, got %q", got)
|
||||
}
|
||||
at := clk.t
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
clk.t = at.Add(23 * time.Hour)
|
||||
if got := read(); got != at.UTC().Format(time.RFC3339) {
|
||||
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
|
||||
}
|
||||
clk.t = at.Add(25 * time.Hour)
|
||||
if got := read(); got != "" {
|
||||
t.Fatalf("after 25 h the line must be gone, got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -336,6 +336,7 @@ type Server struct {
|
||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
|
||||
lineTest LineTestReader // R-1247 weekly line test (optional; nil → present:false)
|
||||
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
|
||||
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
|
||||
guestPower map[int]guestPowerState
|
||||
@@ -410,6 +411,20 @@ type Server struct {
|
||||
guestsDir string
|
||||
ctrlSup controllerSupervisor
|
||||
|
||||
// 2026-10-29: the operator's actions from the hub's reply (opactions.go). opHealthPoll/opHealthWait bound the wait
|
||||
// for a restarted controller's healthy answer (tests shorten them; zero → the defaults).
|
||||
opActs opActionState
|
||||
opTrigger func()
|
||||
opHealthPoll time.Duration
|
||||
opHealthWait time.Duration
|
||||
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
|
||||
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
|
||||
opRebooter interface {
|
||||
RebootGuest(ctx context.Context, vmid int) error
|
||||
}
|
||||
rebootMu sync.Mutex
|
||||
rebootInFlight map[int]bool
|
||||
|
||||
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
|
||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||
netVerifyMu sync.Mutex
|
||||
@@ -458,6 +473,8 @@ type Server struct {
|
||||
escrowLookPath func(file string) (string, error)
|
||||
// statFile reports whether a path exists (preflight's staged-secret item; tests inject).
|
||||
statFile func(path string) bool
|
||||
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
|
||||
cgroupRoot string
|
||||
|
||||
baseCtx context.Context // for fire-and-forget backups; set in Run
|
||||
}
|
||||
@@ -573,6 +590,13 @@ func (s *Server) Handler() http.Handler {
|
||||
// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
|
||||
// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
|
||||
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
||||
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
|
||||
// read it. Read-only; never measured (or the job not wired) = present:false.
|
||||
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
|
||||
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
|
||||
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
|
||||
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
|
||||
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
|
||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
|
||||
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
|
||||
@@ -586,6 +610,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
||||
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
||||
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
||||
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
|
||||
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
|
||||
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
||||
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
||||
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
||||
|
||||
@@ -256,8 +256,10 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
if callers != 5 {
|
||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
||||
// 2026-10-29: +2 deliberate callers — the operator's restart_controller (opactions.go) and R-1344's unresponsive count
|
||||
// (controllersupervisor.go observeControllerHealth). Both through s.staleLock, the one helper.
|
||||
if callers != 7 {
|
||||
t.Errorf("expected the seven known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup, the operator restart, the unresponsive count), found %d", callers)
|
||||
}
|
||||
b, _ := os.ReadFile("stalelock.go")
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-1344 (operator brief 2026-10-29, Part 2; decided 2026-10-30 from the night's measurement): a controller that RUNS
|
||||
// but whose Docker health check reads „unhealthy" on controllerUnresponsiveConfirm consecutive sweeps is reported
|
||||
// unresponsive (the stanza; the hub raises controller_unresponsive). If it still reads „unhealthy" at
|
||||
// controllerUnresponsiveRestart sweeps, the agent restarts it — with the not-running restart's guards and brake, plus
|
||||
// the controller's own busy file. RED-PROOF: the restart tests seen failing before the restart existed.
|
||||
|
||||
func unrespServer(t *testing.T, health string) (*Server, *rbExec, *fakeGuestPowerCtl, *supClock) {
|
||||
t.Helper()
|
||||
s, ex, ctl, clk, _, _ := opServer(t, "running")
|
||||
ex.health = health
|
||||
rb := &rbExec{opExec: ex}
|
||||
s.guestExec = rb
|
||||
return s, rb, ctl, clk
|
||||
}
|
||||
|
||||
func sweep(s *Server, n int) {
|
||||
for i := 0; i < n; i++ {
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSupervisor_UnresponsiveIsAlarmedThenRestarted(t *testing.T) {
|
||||
s, ex, _, _ := unrespServer(t, "unhealthy")
|
||||
sweep(s, controllerUnresponsiveConfirm-1)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
|
||||
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||
}
|
||||
sweep(s, 1)
|
||||
g := s.ControllerSupervisorStatus(context.Background()).Guests
|
||||
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
|
||||
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||
}
|
||||
if g[0].UnresponsiveRestartAfterS != (controllerUnresponsiveRestart-controllerUnresponsiveConfirm)*int(controllerSupervisorInterval/time.Second) {
|
||||
t.Fatalf("the stanza must tell the hub when the restart comes: %+v", g[0])
|
||||
}
|
||||
since := g[0].UnresponsiveSince
|
||||
sweep(s, controllerUnresponsiveRestart-controllerUnresponsiveConfirm-1)
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("restarted after %d unhealthy sweeps — the restart waits for %d", controllerUnresponsiveRestart-1, controllerUnresponsiveRestart)
|
||||
}
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
|
||||
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
|
||||
}
|
||||
sweep(s, 1)
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("after %d unhealthy sweeps: %d restarts, want 1", controllerUnresponsiveRestart, n)
|
||||
}
|
||||
g = s.ControllerSupervisorStatus(context.Background()).Guests
|
||||
if g[0].UnresponsiveRestartAt == "" || g[0].UnresponsiveRestartAt != g[0].LastRestartAt || g[0].Restarts24h != 1 ||
|
||||
!strings.Contains(g[0].LastReason, "health check") {
|
||||
t.Fatalf("the restart is not on the record the hub reads: %+v", g[0])
|
||||
}
|
||||
// A restart that did not help (still „unhealthy" on the next sweep) is not repeated at once: the count starts again.
|
||||
sweep(s, 1)
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("restarted again on the sweep after a restart: %d restarts", n)
|
||||
}
|
||||
// The restarted controller starts, then answers: the episode ends, its record stays.
|
||||
ex.health = "starting"
|
||||
sweep(s, 1)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
|
||||
t.Fatalf("after the restart: %+v (unresponsive must clear; since stays as the record)", g[0])
|
||||
}
|
||||
// A second episode needs the whole count again — never a restart on the first unhealthy read after one.
|
||||
ex.health = "unhealthy"
|
||||
sweep(s, controllerUnresponsiveRestart-1)
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("a second episode restarted early: %d restarts", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Slow but answering (healthy / starting), a swap, a vzdump, a locked or a parked guest never counts — and so is never
|
||||
// restarted.
|
||||
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl)
|
||||
}{
|
||||
{"healthy", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
|
||||
{"starting", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
|
||||
{"swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
|
||||
{"vzdump", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||
{"locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, ctl, _ := unrespServer(t, "unhealthy")
|
||||
tc.setup(s, ex, ctl)
|
||||
sweep(s, controllerUnresponsiveRestart+2)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
|
||||
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
|
||||
}
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("%s: restarted %d times", tc.name, n)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// At the restart moment: the controller's busy file (fresh, or unreadable) and the brake refuse it; the alarm stays.
|
||||
// RED-PROOF: each seen failing by deleting its guard.
|
||||
func TestSupervisor_UnresponsiveRestartRefusals(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
setup func(s *Server, ex *rbExec, clk *supClock)
|
||||
crashloop bool
|
||||
}{
|
||||
{"busy file fresh", func(_ *Server, ex *rbExec, clk *supClock) {
|
||||
ex.busy = clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\na backup, a restore or an off-site run is running\n"
|
||||
}, false},
|
||||
{"busy file unreadable", func(_ *Server, ex *rbExec, _ *supClock) { ex.busyErr = errors.New("pct exec: exit status 255") }, false},
|
||||
{"brake: 3 restarts in 15 minutes", func(s *Server, _ *rbExec, clk *supClock) {
|
||||
st := s.supState(9201)
|
||||
st.restarts = []time.Time{clk.t.Add(-10 * time.Minute), clk.t.Add(-5 * time.Minute), clk.t.Add(-time.Minute)}
|
||||
}, true},
|
||||
{"crash-loop pause in force", func(s *Server, _ *rbExec, clk *supClock) {
|
||||
s.supState(9201).crashloopSince = clk.t.Add(-5 * time.Minute)
|
||||
}, true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, _, clk := unrespServer(t, "unhealthy")
|
||||
tc.setup(s, ex, clk)
|
||||
sweep(s, controllerUnresponsiveRestart+3)
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("%s: restarted %d times", tc.name, n)
|
||||
}
|
||||
g := s.ControllerSupervisorStatus(context.Background()).Guests
|
||||
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveRestartAt != "" {
|
||||
t.Fatalf("%s: the alarm must stay and no restart be recorded: %+v", tc.name, g)
|
||||
}
|
||||
if tc.crashloop && !g[0].Crashloop {
|
||||
t.Fatalf("%s: the brake did not raise the crash-loop: %+v", tc.name, g[0])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A busy file the controller stopped writing (older than controllerBusyFresh) is not „busy" — a hung controller must
|
||||
// not shield itself with its last word.
|
||||
func TestSupervisor_UnresponsiveStaleBusyFileDoesNotShield(t *testing.T) {
|
||||
s, ex, _, clk := unrespServer(t, "unhealthy")
|
||||
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\nan app update is running\n"
|
||||
sweep(s, controllerUnresponsiveRestart)
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("a stale busy file blocked the restart: %d restarts", n)
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -114,12 +115,43 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
|
||||
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
|
||||
}
|
||||
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
|
||||
e.policyPass(ctx, lg, p.AgentVersion)
|
||||
if e.AfterInstall != nil {
|
||||
e.AfterInstall(ctx)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// policyPass (R-1285) asks the wrapper NOW on disk — after this install, the bundle's own — to apply the
|
||||
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
|
||||
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
|
||||
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
|
||||
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
|
||||
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
|
||||
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
|
||||
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
|
||||
// SAME update. Idempotent; on an appliance it removes nothing.
|
||||
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
if lg == nil {
|
||||
lg = slog.Default()
|
||||
}
|
||||
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
|
||||
e.policyPass(ctx, lg, ver)
|
||||
}
|
||||
|
||||
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
|
||||
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
|
||||
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
|
||||
switch {
|
||||
case err != nil:
|
||||
lg.Warn("osupdate: the bundle policy pass did not run", "err", err)
|
||||
case wr.refused():
|
||||
lg.Info("osupdate: the bundle policy pass was refused (a wrapper before 0.166.0?) — the next bundle applies it", "refused", string(wr.Refused))
|
||||
default:
|
||||
lg.Info("osupdate: bundle policy pass done", "bundle_policy", string(wr.BundlePolicy))
|
||||
}
|
||||
}
|
||||
|
||||
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
|
||||
hc := e.HTTPClient
|
||||
if hc == nil {
|
||||
|
||||
@@ -38,6 +38,9 @@ func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]b
|
||||
var plan map[string]any
|
||||
_ = json.Unmarshal(raw, &plan)
|
||||
b.plans = append(b.plans, plan)
|
||||
if plan["mode"] == "bundle-policy" { // R-1285: the pass after an install carries no bundle
|
||||
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":[]}}` + "\n"), nil, nil
|
||||
}
|
||||
body, _ := os.ReadFile(plan["bundle"].(string))
|
||||
b.bodies = append(b.bodies, body)
|
||||
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
|
||||
@@ -153,3 +156,27 @@ func TestBundleURL(t *testing.T) {
|
||||
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return b.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
// R-1285: right after an install the agent asks the wrapper now on disk for the appliance-only policy pass — the bundle
|
||||
// run itself was planned by the OLD wrapper (on a BYO host upgraded from 0.164.0 our SSH drop-in stayed one bundle
|
||||
// longer). A refused or failed bundle runs no pass.
|
||||
// RED-PROOF: drop the policyPass call → one plan only.
|
||||
func TestR1285_AnInstalledBundleIsFollowedByThePolicyPass(t *testing.T) {
|
||||
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
|
||||
srv, sha := serveBundle(t, body)
|
||||
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/usr/local/sbin/felhom-os-apply"]}}`}
|
||||
e, _ := bundleExec(t, w, srv.URL)
|
||||
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(w.plans) != 2 || w.plans[1]["mode"] != "bundle-policy" || w.plans[1]["layer"] != "host" || w.plans[1]["signed"] != nil {
|
||||
t.Fatalf("after the install the agent must run the policy pass (host layer, no envelope): %v", w.plans)
|
||||
}
|
||||
|
||||
w2 := &bundleWrapper{t: t, rep: `{"mode":"bundle","refused":{"code":"R3","reason":"x"}}`}
|
||||
e2, _ := bundleExec(t, w2, srv.URL)
|
||||
_ = e2.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha))
|
||||
if len(w2.plans) != 1 {
|
||||
t.Fatalf("a refused bundle must run no policy pass: %v", w2.plans)
|
||||
}
|
||||
}
|
||||
@@ -21,6 +21,8 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
@@ -444,3 +446,147 @@ func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParam
|
||||
}
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
|
||||
// ---- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----
|
||||
//
|
||||
// The rule, its refusals and "may this box do it" live in the root wrapper (mode kernel-cleanup); the plan names NO
|
||||
// package. This file decides only WHEN: in the night leg, after a healthy host step, never in a night with a kernel
|
||||
// step. The wrapper adds its own 20-hour window after a step.
|
||||
|
||||
// KernelCleanupResult is the wrapper's kernel_cleanup object.
|
||||
type KernelCleanupResult struct {
|
||||
Outcome string `json:"outcome"` // removed | nothing | not_run | dry_run | failed
|
||||
DryRun bool `json:"dry_run"`
|
||||
Would string `json:"would"` // dry run only: remove | nothing | refuse | not_run
|
||||
NotRun []struct {
|
||||
Code string `json:"code"`
|
||||
Reason string `json:"reason"`
|
||||
} `json:"not_run"`
|
||||
Refusals []struct {
|
||||
Code string `json:"code"`
|
||||
Reason string `json:"reason"`
|
||||
} `json:"refusals"`
|
||||
Kept map[string][]string `json:"kept"`
|
||||
Removal []string `json:"removal"`
|
||||
Removed []string `json:"removed"`
|
||||
BytesGained *int64 `json:"bytes_gained"`
|
||||
Leftovers []string `json:"leftovers"`
|
||||
}
|
||||
|
||||
// kernelCleanupSkip says why tonight's pass must not call the clean-up ("" = call it).
|
||||
func kernelCleanupSkip(appliance bool, trigger string, host, pve, kernel Report, blk hub.WireOSUpdate, okStep func(Report) bool) string {
|
||||
switch {
|
||||
case trigger != "night":
|
||||
return "it runs only in the night leg (never a debug pass)"
|
||||
case !appliance:
|
||||
return "not an appliance install (a BYO host belongs to its owner)"
|
||||
case !blk.Enabled:
|
||||
return "OS updates are switched off for this box"
|
||||
case host.Layer == "" || !okStep(host):
|
||||
return "no healthy host step this pass"
|
||||
case pve.Layer != "" && !okStep(pve):
|
||||
return "the Proxmox step did not end healthy"
|
||||
case kernel.Layer != "":
|
||||
return "a kernel step ran in this night (" + kernel.Outcome + ") — never in the same night"
|
||||
}
|
||||
if due, _ := kernelDue(blk, trigger); due {
|
||||
return "a kernel step is due tonight — never in the same night"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// runKernelCleanup calls the wrapper once. The hub hears about it only when there is something to tell: kernels were
|
||||
// removed (outcome "cleaned"), the wrapper refused, or the purge failed. "Nothing to remove" and "not run" are log
|
||||
// lines — one report per box per night would bury the kernel step's own line on the hub's System page.
|
||||
func (l *Leg) runKernelCleanup(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
|
||||
lg := l.log().With("run", runID, "layer", LayerKernel, "mode", "kernel-cleanup", "vmid", vmid, "trigger", trigger)
|
||||
start := l.now()
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-cleanup", vmid, map[string]any{"release_id": "kernel-cleanup",
|
||||
"run_id": runID, "trigger": trigger, "ring": blk.Ring}))
|
||||
// NO kernel view and no kernel version in this report: the hub reads a kernel-layer report's `kernel.to` /
|
||||
// `release_id` as "the kernel this STEP was for" (kernelOf), and a clean-up is not a step for any kernel.
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "kernel-cleanup",
|
||||
ReleaseID: "kernel-cleanup", PassSeconds: wr.PassSeconds}
|
||||
var res KernelCleanupResult
|
||||
_ = json.Unmarshal(wr.KernelCleanup, &res)
|
||||
took := l.now().Sub(start).Round(time.Second).String()
|
||||
switch {
|
||||
case err != nil:
|
||||
lg.Warn("osupdate: kernel clean-up — the wrapper gave no answer; not retried tonight", "err", err, "took", took)
|
||||
rep.Outcome, rep.HealthReason = "failed", "kernel clean-up: "+err.Error()
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.refused():
|
||||
lg.Warn("osupdate: kernel clean-up REFUSED — nothing was removed; not retried tonight", "refused", string(wr.Refused), "took", took)
|
||||
rep.Outcome, rep.Refused, rep.HealthReason = "refused", wr.Refused, "the clean-up of old kernels was refused; nothing was removed"
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.failed():
|
||||
lg.Error("osupdate: kernel clean-up FAILED — look at the box; not retried tonight", "failed", string(wr.Failed), "took", took)
|
||||
rep.Outcome, rep.Refused, rep.HealthReason = "failed", wr.Failed, "the clean-up of old kernels failed"
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
switch res.Outcome {
|
||||
case "removed":
|
||||
gained := "unknown"
|
||||
if res.BytesGained != nil {
|
||||
gained = fmt.Sprintf("%.1f GiB", float64(*res.BytesGained)/(1<<30))
|
||||
}
|
||||
rep.Outcome, rep.Healthy = "cleaned", true
|
||||
rep.KernelCleanup = &CleanupDetail{Removed: capNames(res.Removed, 40), Kept: capNames(sortedKeys(res.Kept), 40),
|
||||
BytesGained: res.BytesGained, Leftovers: len(res.Leftovers)}
|
||||
rep.HealthReason = truncate(fmt.Sprintf("removed %d old kernel package(s): %s; kept %s; %s gained; %d left-over path(s)",
|
||||
len(res.Removed), strings.Join(res.Removed, ", "), strings.Join(sortedKeys(res.Kept), ", "), gained, len(res.Leftovers)), 600)
|
||||
lg.Info("osupdate: kernel clean-up — old kernels removed", "removed", len(res.Removed), "names", strings.Join(res.Removed, ","),
|
||||
"kept", strings.Join(sortedKeys(res.Kept), ","), "bytes_gained", res.BytesGained, "leftovers", len(res.Leftovers), "took", took)
|
||||
return l.finish(ctx, lg, rep)
|
||||
case "nothing":
|
||||
lg.Info("osupdate: kernel clean-up — nothing to remove", "kept", strings.Join(sortedKeys(res.Kept), ","), "took", took)
|
||||
case "not_run":
|
||||
code, why := "", ""
|
||||
if len(res.NotRun) > 0 {
|
||||
code, why = res.NotRun[0].Code, res.NotRun[0].Reason
|
||||
}
|
||||
lg.Info("osupdate: kernel clean-up — not run on this box tonight", "code", code, "reason", why, "took", took)
|
||||
default:
|
||||
lg.Warn("osupdate: kernel clean-up — the wrapper answered with no outcome this agent knows (an older wrapper?)", "outcome", res.Outcome, "took", took)
|
||||
}
|
||||
return Report{}
|
||||
}
|
||||
|
||||
// KernelCleanupDryRun asks the wrapper what the clean-up WOULD do, and changes nothing (the debug tool: `--selftest
|
||||
// kernel-cleanup-dry-run`). The raw kernel_cleanup object, or the refusal.
|
||||
func (l *Leg) KernelCleanupDryRun(ctx context.Context, vmid int) (json.RawMessage, error) {
|
||||
wr, err := l.call(ctx, "kcdry"+l.now().UTC().Format("150405"), kernelPlan("kernel-cleanup", vmid,
|
||||
map[string]any{"release_id": "kernel-cleanup", "dry_run": true}))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if wr.refused() {
|
||||
return nil, fmt.Errorf("kernel-cleanup dry run refused: %s", wr.Refused)
|
||||
}
|
||||
return wr.KernelCleanup, nil
|
||||
}
|
||||
|
||||
// capNames bounds a list that goes to the hub: at most n names, each at most 120 bytes; never nil (the hub reads
|
||||
// "removed: []" as "none", not as "unknown").
|
||||
func capNames(in []string, n int) []string {
|
||||
out := make([]string, 0, len(in))
|
||||
for _, x := range in {
|
||||
if len(out) == n {
|
||||
break
|
||||
}
|
||||
if len(x) > 120 {
|
||||
x = x[:120]
|
||||
}
|
||||
out = append(out, x)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string][]string) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
@@ -28,13 +28,23 @@ func tonight(ring int) *hub.WireOSUpdate {
|
||||
func kernelCalls(w *fakeWrapper) []string {
|
||||
var m []string
|
||||
for _, p := range w.plans {
|
||||
if p["layer"] == LayerKernel {
|
||||
if p["layer"] == LayerKernel && p["mode"] != "kernel-cleanup" { // the kernel STEP's calls (the clean-up: cleanupCalls)
|
||||
m = append(m, p["mode"].(string))
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func cleanupCalls(w *fakeWrapper) []map[string]any {
|
||||
var m []map[string]any
|
||||
for _, p := range w.plans {
|
||||
if p["layer"] == LayerKernel && p["mode"] == "kernel-cleanup" {
|
||||
m = append(m, p)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
|
||||
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
|
||||
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
|
||||
@@ -326,3 +336,196 @@ func TestKernelSet(t *testing.T) {
|
||||
t.Fatal("a non-kernel string must give no set")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----
|
||||
|
||||
func cleanupRep(js string) map[string][]WrapperReport {
|
||||
return map[string][]WrapperReport{"kernel-cleanup": {{KernelCleanup: json.RawMessage(js)}}}
|
||||
}
|
||||
|
||||
// The night: the clean-up is the LAST call, after the host step, exactly once, and its plan names NO package.
|
||||
// COMPANION RED-PROOF (observed): make kernelCleanupSkip return "" before its checks → the BYO / debug / unhealthy
|
||||
// sub-cases of TestKernelCleanup_WhenItMayNotRun fail; put the call before runFast → this test's order check fails.
|
||||
func TestKernelCleanup_NightOrderOnceAndNoNames(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: cleanupRep(`{"outcome":"removed","removed":["proxmox-kernel-7.0.14-20-pve-signed"],` +
|
||||
`"kept":{"7.0.14-22-pve":["running"],"7.0.2-6-pve":["from"]},"bytes_gained":1073741824,"leftovers":[]}`)}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
cc := cleanupCalls(w)
|
||||
if len(cc) != 1 {
|
||||
t.Fatalf("clean-up calls = %d, want exactly one a night", len(cc))
|
||||
}
|
||||
if last := w.plans[len(w.plans)-1]; last["mode"] != "kernel-cleanup" {
|
||||
t.Fatalf("the clean-up must be the night's last call (after the host step), last = %v", last)
|
||||
}
|
||||
hostAt, cleanAt := -1, -1
|
||||
for i, x := range w.plans {
|
||||
if x["layer"] == LayerHost {
|
||||
hostAt = i
|
||||
}
|
||||
if x["mode"] == "kernel-cleanup" {
|
||||
cleanAt = i
|
||||
}
|
||||
}
|
||||
if hostAt < 0 || cleanAt < hostAt {
|
||||
t.Fatalf("the clean-up ran before the host step: host %d clean-up %d", hostAt, cleanAt)
|
||||
}
|
||||
plan := cc[0]
|
||||
if pk, _ := json.Marshal(plan["packages"]); string(pk) != "[]" {
|
||||
t.Fatalf("the clean-up plan names packages: %s", pk)
|
||||
}
|
||||
for _, k := range []string{"select", "signed", "allow_new", "expect_kver", "dry_run"} {
|
||||
if _, ok := plan[k]; ok {
|
||||
t.Fatalf("the clean-up plan carries %q: %v", k, plan)
|
||||
}
|
||||
}
|
||||
if plan["lane"] != "slow" || plan["layer"] != LayerKernel {
|
||||
t.Fatalf("plan = %v", plan)
|
||||
}
|
||||
// the hub hears "cleaned" with what was removed, kept and gained — and NO kernel view / version (the hub would read
|
||||
// it as a kernel step for that kernel)
|
||||
rep := p.KernelCleanup
|
||||
if rep.Outcome != "cleaned" || !rep.Healthy || rep.Mode != "kernel-cleanup" || rep.Layer != LayerKernel {
|
||||
t.Fatalf("report = %+v", rep)
|
||||
}
|
||||
for _, want := range []string{"removed 1 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed", "kept 7.0.14-22-pve, 7.0.2-6-pve", "1.0 GiB gained"} {
|
||||
if !strings.Contains(rep.HealthReason, want) {
|
||||
t.Fatalf("report reason %q lacks %q", rep.HealthReason, want)
|
||||
}
|
||||
}
|
||||
lastRep := h.reports[len(h.reports)-1]
|
||||
if lastRep.Mode != "kernel-cleanup" || lastRep.Outcome != "cleaned" {
|
||||
t.Fatalf("the hub did not hear the clean-up: %+v", lastRep)
|
||||
}
|
||||
if len(lastRep.Kernel) != 0 || kverRE.MatchString(lastRep.ReleaseID) {
|
||||
t.Fatalf("the clean-up's report names a kernel (the hub reads that as a kernel step): %+v", lastRep)
|
||||
}
|
||||
// R-1383 (release 3): the detail the hub's System page shows — removed, kept, gained, left-overs — rides the same
|
||||
// report as its own object. On the wire it is `kernel_cleanup`, and the report STILL has no `kernel` key.
|
||||
d := lastRep.KernelCleanup
|
||||
if d == nil || len(d.Removed) != 1 || d.Removed[0] != "proxmox-kernel-7.0.14-20-pve-signed" ||
|
||||
strings.Join(d.Kept, ",") != "7.0.14-22-pve,7.0.2-6-pve" || d.BytesGained == nil || *d.BytesGained != 1073741824 || d.Leftovers != 0 {
|
||||
t.Fatalf("the clean-up's detail did not reach the hub: %+v", d)
|
||||
}
|
||||
wire, _ := json.Marshal(lastRep)
|
||||
var keys map[string]json.RawMessage
|
||||
_ = json.Unmarshal(wire, &keys)
|
||||
if _, has := keys["kernel"]; has {
|
||||
t.Fatalf("the clean-up's report carries a `kernel` view on the wire: %s", wire)
|
||||
}
|
||||
if string(keys["kernel_cleanup"]) != `{"removed":["proxmox-kernel-7.0.14-20-pve-signed"],"kept":["7.0.14-22-pve","7.0.2-6-pve"],"bytes_gained":1073741824,"leftovers":0}` {
|
||||
t.Fatalf("kernel_cleanup on the wire = %s", keys["kernel_cleanup"])
|
||||
}
|
||||
if string(keys["release_id"]) != `"kernel-cleanup"` {
|
||||
t.Fatalf("release_id on the wire = %s (a kernel version there is read as a step for it)", keys["release_id"])
|
||||
}
|
||||
// the same night again: the pass is skipped as a whole — never a second clean-up
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
if len(cleanupCalls(w)) != 1 {
|
||||
t.Fatalf("a second clean-up in the same night: %d calls", len(cleanupCalls(w)))
|
||||
}
|
||||
}
|
||||
|
||||
// "Nothing to remove" and "not run" are log lines, never a report (one a night would bury the kernel step's own line).
|
||||
func TestKernelCleanup_NothingAndNotRunSendNoReport(t *testing.T) {
|
||||
for name, js := range map[string]string{
|
||||
"nothing": `{"outcome":"nothing","removal":[],"kept":{"7.0.14-22-pve":["running"]}}`,
|
||||
"not_run": `{"outcome":"not_run","not_run":[{"code":"no-ring0-mark","reason":"ring 0 only in this release"}]}`,
|
||||
} {
|
||||
w := &fakeWrapper{t: t, kernelRep: cleanupRep(js)}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
if len(cleanupCalls(w)) != 1 {
|
||||
t.Fatalf("%s: the wrapper was not asked", name)
|
||||
}
|
||||
if p.KernelCleanup.Layer != "" {
|
||||
t.Fatalf("%s: a report was made: %+v", name, p.KernelCleanup)
|
||||
}
|
||||
for _, r := range h.reports {
|
||||
if r.Mode == "kernel-cleanup" {
|
||||
t.Fatalf("%s: the hub got a clean-up report: %+v", name, r)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A refusal and a failure reach the hub (the operator), once, and are not retried.
|
||||
func TestKernelCleanup_RefusedAndFailedReachTheHub(t *testing.T) {
|
||||
for name, wr := range map[string]WrapperReport{
|
||||
"refused": {Refused: json.RawMessage(`{"code":"P3","reason":"the pinned default and grub.cfg disagree"}`)},
|
||||
"failed": {Failed: json.RawMessage(`{"rc":3,"step":"verify","reason":"the default in grub.cfg moved"}`), KernelCleanup: json.RawMessage(`{"outcome":"failed"}`)},
|
||||
} {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-cleanup": {wr}}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
if p.KernelCleanup.Outcome != name || p.KernelCleanup.Healthy || len(p.KernelCleanup.Refused) == 0 {
|
||||
t.Fatalf("%s: report = %+v", name, p.KernelCleanup)
|
||||
}
|
||||
if last := h.reports[len(h.reports)-1]; last.Mode != "kernel-cleanup" || last.Outcome != name {
|
||||
t.Fatalf("%s: the hub did not hear it: %+v", name, last)
|
||||
}
|
||||
if len(cleanupCalls(w)) != 1 {
|
||||
t.Fatalf("%s: retried the same night (%d calls)", name, len(cleanupCalls(w)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// When it may NOT run: never with a kernel step in the night (due, staged, or reported), not in a debug pass, not on a
|
||||
// BYO host, not with OS updates off, not after an unhealthy host step.
|
||||
func TestKernelCleanup_WhenItMayNotRun(t *testing.T) {
|
||||
type tc struct {
|
||||
blk *hub.WireOSUpdate
|
||||
trigger string
|
||||
prep func(*fakeWrapper, *Leg)
|
||||
}
|
||||
bad := &Health{DockerOK: false}
|
||||
cases := map[string]tc{
|
||||
"a kernel step is due tonight (ring 0)": {tonight(0), "night", func(w *fakeWrapper, _ *Leg) {
|
||||
w.kernelRep = map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
|
||||
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
|
||||
"kernel-reboot": {{Kernel: kview("oneshot")}}}
|
||||
}},
|
||||
"a kernel step is due tonight (ring 1, nothing staged)": {tonight(1), "night", func(w *fakeWrapper, _ *Leg) {
|
||||
w.kernelRep = map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}
|
||||
}},
|
||||
"a debug pass": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "debug", nil},
|
||||
"a BYO host": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night", func(_ *fakeWrapper, l *Leg) { l.Appliance = false }},
|
||||
"updates off": {&hub.WireOSUpdate{Ring: 0, Enabled: false}, "night", nil},
|
||||
"host unhealthy": {&hub.WireOSUpdate{Ring: 0, Enabled: true, HostRelease: &hub.WireOSRelease{ID: "os-host-1", Packages: []hub.WireOSPackage{{Name: "openssl", Version: "u3", Origin: "Debian"}}}}, "night", func(w *fakeWrapper, _ *Leg) {
|
||||
w.applyRep[LayerHost] = WrapperReport{Upgraded: []Package{{Name: "openssl", Version: "u3"}}, HealthAfter: &Health{HostServices: map[string]string{"pveproxy": "failed"}, GuestRunning: yes(), Guest: bad}}
|
||||
w.healthSeq[LayerHost] = []*Health{{HostServices: map[string]string{"pveproxy": "failed"}, GuestRunning: yes(), Guest: bad}}
|
||||
}},
|
||||
}
|
||||
for name, c := range cases {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, c.blk)
|
||||
if c.prep != nil {
|
||||
c.prep(w, l)
|
||||
}
|
||||
l.HealthWait = time.Second
|
||||
l.Run(context.Background(), 9201, c.trigger)
|
||||
if n := len(cleanupCalls(w)); n != 0 {
|
||||
t.Fatalf("%s: the clean-up ran (%d call(s)); calls = %s", name, n, calls(w))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The dry run is the same mode with dry_run: true — and it sends nothing to the hub.
|
||||
func TestKernelCleanup_DryRun(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: cleanupRep(`{"outcome":"dry_run","would":"remove","removal":["proxmox-kernel-7.0.14-20-pve-signed"]}`)}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
raw, err := l.KernelCleanupDryRun(context.Background(), 9201)
|
||||
if err != nil || !strings.Contains(string(raw), `"would":"remove"`) {
|
||||
t.Fatalf("dry run = %s, %v", raw, err)
|
||||
}
|
||||
cc := cleanupCalls(w)
|
||||
if len(cc) != 1 || cc[0]["dry_run"] != true {
|
||||
t.Fatalf("the dry run's plan = %v", cc)
|
||||
}
|
||||
if pk, _ := json.Marshal(cc[0]["packages"]); string(pk) != "[]" {
|
||||
t.Fatalf("the dry run's plan names packages: %s", pk)
|
||||
}
|
||||
if len(h.reports) != 0 {
|
||||
t.Fatalf("a dry run reported to the hub: %+v", h.reports)
|
||||
}
|
||||
}
|
||||
@@ -123,7 +123,8 @@ type WrapperReport struct {
|
||||
Undo bool `json:"undo"`
|
||||
LiveRestore json.RawMessage `json:"live_restore"`
|
||||
Facts json.RawMessage `json:"facts"`
|
||||
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
||||
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
||||
BundlePolicy json.RawMessage `json:"bundle_policy"` // the appliance-only policy pass (R-1285, mode "bundle-policy")
|
||||
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
|
||||
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
|
||||
OOMCheck json.RawMessage `json:"oom_check"`
|
||||
@@ -133,6 +134,13 @@ type WrapperReport struct {
|
||||
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
|
||||
Kernel json.RawMessage `json:"kernel"`
|
||||
KernelEvent string `json:"kernel_event"`
|
||||
// KernelCleanup (R-1368, `11` §5.11.1): mode kernel-cleanup's result {outcome, dry_run, kept, removal, removed,
|
||||
// bytes_gained, not_run, refusals, leftovers}.
|
||||
KernelCleanup json.RawMessage `json:"kernel_cleanup"`
|
||||
// ProfileRefresh (R-1388, `11` §5.12): mode profile-refresh's result {outcome, kernel, rewritten, restarted, seconds}.
|
||||
ProfileRefresh json.RawMessage `json:"profile_refresh"`
|
||||
// PVECACopy (R-1380, felhom.eu `03` §12a): mode pve-ca-copy's result {outcome, path, sha256, not_after, days_left}.
|
||||
PVECACopy json.RawMessage `json:"pve_ca_copy"`
|
||||
OutcomeHint string `json:"outcome_hint"`
|
||||
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
||||
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
||||
@@ -180,10 +188,23 @@ type Report struct {
|
||||
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
|
||||
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
|
||||
Kernel json.RawMessage `json:"kernel,omitempty"`
|
||||
// KernelCleanup (R-1383, release 3): ONLY on the mode kernel-cleanup report with outcome "cleaned" — what the
|
||||
// clean-up removed, kept and gained, for the hub's System page. It is not the `kernel` view: this report still
|
||||
// carries no kernel view and no kernel as its release_id, or the hub would read it as a kernel STEP for that kernel
|
||||
// (the hub's kernelOf reads only those two). Pinned by TestKernelCleanup_NightOrderOnceAndNoNames.
|
||||
KernelCleanup *CleanupDetail `json:"kernel_cleanup,omitempty"`
|
||||
|
||||
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
|
||||
}
|
||||
|
||||
// CleanupDetail is the clean-up's result as the hub shows it (hub osupdates.CleanupDetail, field-exact).
|
||||
type CleanupDetail struct {
|
||||
Removed []string `json:"removed"` // package names
|
||||
Kept []string `json:"kept"` // kernel versions that stay
|
||||
BytesGained *int64 `json:"bytes_gained,omitempty"` // nil = the wrapper could not measure it
|
||||
Leftovers int `json:"leftovers"` // paths of a removed kernel that are still there
|
||||
}
|
||||
|
||||
// Reporter posts a report to the hub (*hub.Client).
|
||||
type Reporter interface {
|
||||
PostOSReport(ctx context.Context, body []byte) error
|
||||
@@ -203,9 +224,15 @@ type Leg struct {
|
||||
MinGap time.Duration // between night runs (default 20 h)
|
||||
Now func() time.Time
|
||||
Sleep func(context.Context, time.Duration)
|
||||
// ProfileRecordPath is the root wrapper's record of the last security-profile refresh (0644, read-only for the
|
||||
// agent); "" = DefaultProfileRecordPath. A seam for the tests.
|
||||
ProfileRecordPath string
|
||||
|
||||
mu sync.Mutex
|
||||
block *hub.WireOSUpdate
|
||||
mu sync.Mutex
|
||||
block *hub.WireOSUpdate
|
||||
profileAsk string // how the last ask of the profile refresh ended in THIS process ("" = never asked)
|
||||
profileAskAt time.Time
|
||||
profileLast ProfileRefreshStatus // the last refresh as the wrapper answered it to this process
|
||||
}
|
||||
|
||||
// planFile / reportFile: the plan the agent writes and the copy of the report the wrapper keeps beside it (R-868).
|
||||
@@ -530,6 +557,8 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
|
||||
// Pass is one leg's reports; an empty Layer means the step did not run.
|
||||
type Pass struct {
|
||||
Guest, Host, Docker, PVE, Kernel Report
|
||||
// KernelCleanup (R-1368): the clean-up of old kernels. Layer "" when it did not run or had nothing to tell the hub.
|
||||
KernelCleanup Report
|
||||
}
|
||||
|
||||
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
||||
@@ -581,6 +610,15 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
|
||||
default:
|
||||
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
|
||||
}
|
||||
// R-1368 (`09` §3 decision 286, `11` §5.11.1): the clean-up of old kernels — its own step, at most once a night (this
|
||||
// pass is the night's only one), after the host step ended healthy, NEVER in a night with a kernel step. Whether
|
||||
// this box may do it at all is the root wrapper's answer (its root-owned ring-0 mark), never the agent's.
|
||||
// Pinned by TestKernelCleanup_*.
|
||||
if why := kernelCleanupSkip(l.Appliance, trigger, h, p.PVE, p.Kernel, blk, okStep); why != "" {
|
||||
lg.Info("osupdate: kernel clean-up skipped — " + why)
|
||||
} else {
|
||||
p.KernelCleanup = l.runKernelCleanup(ctx, g.RunID, vmid, trigger, blk)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ type fakeWrapper struct {
|
||||
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
||||
pveGateHeld bool
|
||||
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
|
||||
profileRep *WrapperReport // R-1388: the answer to mode profile-refresh
|
||||
}
|
||||
|
||||
func yes() *bool { b := true; return &b }
|
||||
@@ -71,6 +72,10 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
|
||||
}
|
||||
var rep WrapperReport
|
||||
switch plan["mode"] {
|
||||
case "profile-refresh":
|
||||
if f.profileRep != nil {
|
||||
rep = *f.profileRep
|
||||
}
|
||||
case "inventory":
|
||||
rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthBefore: ok, HealthAfter: ok,
|
||||
Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
|
||||
@@ -176,8 +181,9 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
|
||||
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
|
||||
t.Fatalf("guest %+v\nhost %+v", g, ho)
|
||||
}
|
||||
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply" {
|
||||
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step and the pve step", calls(w))
|
||||
// R-1368: the clean-up of old kernels is the night's last call when no kernel step is due
|
||||
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply,kernel:kernel-cleanup" {
|
||||
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step, the pve step, the kernel clean-up", calls(w))
|
||||
}
|
||||
for _, p := range w.plans[:2] {
|
||||
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
|
||||
@@ -221,7 +227,8 @@ func TestRing1_NoReleaseIsInventory(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, pending: pend}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
g, ho := run2(l, "night")
|
||||
if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory" {
|
||||
// R-1368: a ring-1 box asks the wrapper too — whether it may clean up is the wrapper's answer (its root-owned mark)
|
||||
if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory,kernel:kernel-cleanup" {
|
||||
t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
package osupdate
|
||||
|
||||
// The security-profile refresh (R-1388, R-1301, `11` §5.12). After a kernel went back and forth a box can load an OLD
|
||||
// compiled AppArmor profile for a program (on the N100 this stopped the time service). The root wrapper's mode
|
||||
// profile-refresh loads every profile from a fresh compile, once per kernel the box starts on, and restarts only a
|
||||
// failed service whose own program's profile was stale. This file decides WHEN to ask: at every agent start and after
|
||||
// a config bundle was installed (a new wrapper may know the mode the old one refused) — the wrapper itself answers
|
||||
// "not needed" when this kernel was already done. No OS report is sent for it (an unknown layer reads as a GUEST step on
|
||||
// the hub); since release 3 the result rides the host report as the `profile_refresh` stanza (ProfileRefreshStanza).
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ProfileRefreshResult is the wrapper's profile_refresh object.
|
||||
type ProfileRefreshResult struct {
|
||||
Outcome string `json:"outcome"` // refreshed | not_needed | dry_run | failed
|
||||
Reason string `json:"reason"`
|
||||
Kernel string `json:"kernel"`
|
||||
Rewritten []string `json:"rewritten"`
|
||||
Restarted []struct {
|
||||
Unit string `json:"unit"`
|
||||
Program string `json:"program"`
|
||||
RC int `json:"rc"`
|
||||
Active bool `json:"active"`
|
||||
} `json:"restarted"`
|
||||
Seconds float64 `json:"seconds"`
|
||||
// The wrapper's own record of the refresh BEFORE this ask (it answers them at every ask, also "not needed").
|
||||
LastRefreshKernel string `json:"last_refresh_kernel"`
|
||||
LastRefreshAt string `json:"last_refresh_at"`
|
||||
}
|
||||
|
||||
// DefaultProfileRecordPath is where the root wrapper keeps the record of its last refresh (PROFILE_STATE, 0644).
|
||||
const DefaultProfileRecordPath = "/var/lib/felhom-os-apply/profile-refresh.json"
|
||||
|
||||
// ProfileRefreshStatus is the host report's `profile_refresh` stanza (R-1383, release 3; hub sysfacts.ProfileRefresh,
|
||||
// field-exact): the last refresh the box did — from the root wrapper's own record, so it outlives an agent restart —
|
||||
// and how the last ask of THIS agent process ended. The refresh restarted a failed service on the N100 and nobody
|
||||
// outside the box's journal could see it.
|
||||
type ProfileRefreshStatus struct {
|
||||
Kernel string `json:"kernel,omitempty"` // the kernel the last refresh ran on
|
||||
At string `json:"at,omitempty"` // RFC3339, when
|
||||
Rewritten *int `json:"rewritten,omitempty"` // how many compiled profiles changed; nil = not known to this process
|
||||
Restarted []string `json:"restarted,omitempty"` // units it restarted (failed, with a stale profile)
|
||||
LastAsk string `json:"last_ask,omitempty"` // refreshed | not_needed | failed | refused | no_answer
|
||||
LastAskAt string `json:"last_ask_at,omitempty"`
|
||||
}
|
||||
|
||||
var profileNameRe = regexp.MustCompile(`^[A-Za-z0-9@._:+-]{1,120}$`)
|
||||
|
||||
// ProfileRefreshStanza builds the stanza, or an UNTYPED nil when there is nothing to say (never asked and no record: a
|
||||
// BYO host, or an agent that did not ask yet) — the report then omits the field.
|
||||
//
|
||||
// Two sources, the first that knows wins per field. (1) What the wrapper ANSWERED to this process's last ask: a
|
||||
// refresh gives everything; "not needed" gives the kernel and time of the earlier refresh (the wrapper answers its own
|
||||
// record at every ask) — so the last refresh is still told after an agent restart, without the detail. (2) The
|
||||
// wrapper's record file, when this process may read it: on boxes where its folder was first made by a signed job it
|
||||
// is root-only (0700), so the file is a bonus, never the only source (found by review, 2026-11-14; not read on a box).
|
||||
// It never calls the wrapper. Names that are not plain unit / kernel names are dropped, the list is capped.
|
||||
func (l *Leg) ProfileRefreshStanza() any {
|
||||
l.mu.Lock()
|
||||
st, ask, at := l.profileLast, l.profileAsk, l.profileAskAt
|
||||
l.mu.Unlock()
|
||||
have := ask != ""
|
||||
if have {
|
||||
st.LastAsk, st.LastAskAt = ask, at.UTC().Format(time.RFC3339)
|
||||
}
|
||||
path := l.ProfileRecordPath
|
||||
if path == "" {
|
||||
path = DefaultProfileRecordPath
|
||||
}
|
||||
if raw, err := os.ReadFile(path); err == nil && len(raw) < 64<<10 {
|
||||
var rec struct {
|
||||
Kernel string `json:"kernel"`
|
||||
At string `json:"at"`
|
||||
Rewritten []string `json:"rewritten"`
|
||||
Restarted []string `json:"restarted"`
|
||||
}
|
||||
if json.Unmarshal(raw, &rec) == nil && (rec.Kernel != "" || rec.At != "") {
|
||||
have = true
|
||||
fromRec := profileStatusFrom(rec.Kernel, rec.At, rec.Rewritten, rec.Restarted, true)
|
||||
if st.At == "" || fromRec.At == st.At || st.Rewritten == nil {
|
||||
fromRec.LastAsk, fromRec.LastAskAt = st.LastAsk, st.LastAskAt
|
||||
st = fromRec
|
||||
}
|
||||
}
|
||||
}
|
||||
if !have {
|
||||
return nil
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// profileStatusFrom checks and bounds what goes to the hub. detail=false: the count and the units are not known.
|
||||
func profileStatusFrom(kernel, at string, rewritten, restarted []string, detail bool) ProfileRefreshStatus {
|
||||
var st ProfileRefreshStatus
|
||||
if profileNameRe.MatchString(kernel) {
|
||||
st.Kernel = kernel
|
||||
}
|
||||
if _, err := time.Parse(time.RFC3339, at); err == nil {
|
||||
st.At = at
|
||||
}
|
||||
if detail {
|
||||
n := len(rewritten)
|
||||
st.Rewritten = &n
|
||||
for _, u := range restarted {
|
||||
if len(st.Restarted) < 16 && profileNameRe.MatchString(u) {
|
||||
st.Restarted = append(st.Restarted, u)
|
||||
}
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// noteProfileAsk remembers how an ask ended and, when the wrapper answered, what it said about the last refresh.
|
||||
func (l *Leg) noteProfileAsk(outcome string, res *ProfileRefreshResult) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.profileAsk, l.profileAskAt = outcome, l.now()
|
||||
if res == nil {
|
||||
return
|
||||
}
|
||||
switch outcome {
|
||||
case "refreshed":
|
||||
units := make([]string, 0, len(res.Restarted))
|
||||
for _, u := range res.Restarted {
|
||||
units = append(units, u.Unit)
|
||||
}
|
||||
l.profileLast = profileStatusFrom(res.Kernel, l.now().UTC().Format(time.RFC3339), res.Rewritten, units, true)
|
||||
case "not_needed":
|
||||
if res.LastRefreshAt != "" || res.LastRefreshKernel != "" {
|
||||
l.profileLast = profileStatusFrom(res.LastRefreshKernel, res.LastRefreshAt, nil, nil, false)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ProfileGate takes the heavy-op gate (a backup, a restore-test or an OS step must not run beside the refresh).
|
||||
type ProfileGate func() (release func(), busy string, ok bool)
|
||||
|
||||
// ProfileRefresh asks the wrapper once. It returns the outcome ("" when the wrapper could not be asked or refused).
|
||||
// It waits for the gate: up to `tries` attempts, `wait` apart. why is the log's short reason for this call.
|
||||
func (l *Leg) ProfileRefresh(ctx context.Context, why string, gate ProfileGate, tries int, wait time.Duration) string {
|
||||
lg := l.log().With("mode", "profile-refresh", "why", why)
|
||||
if !l.Appliance {
|
||||
lg.Info("osupdate: profile refresh skipped — not an appliance install (a BYO host belongs to its owner)")
|
||||
return ""
|
||||
}
|
||||
var release func()
|
||||
for i := 1; ; i++ {
|
||||
if gate == nil {
|
||||
break
|
||||
}
|
||||
r, busy, ok := gate()
|
||||
if ok {
|
||||
release = r
|
||||
break
|
||||
}
|
||||
if i >= tries || ctx.Err() != nil {
|
||||
lg.Warn("osupdate: profile refresh NOT run — the box stayed busy; the next agent start asks again", "busy", busy, "tries", i)
|
||||
return ""
|
||||
}
|
||||
lg.Info("osupdate: profile refresh waits — a heavy operation runs", "busy", busy, "try", i)
|
||||
l.sleep(ctx, wait)
|
||||
}
|
||||
if release != nil {
|
||||
defer release()
|
||||
}
|
||||
unlock := l.lockPass(true) // never beside an OS-update pass
|
||||
defer unlock()
|
||||
start := l.now()
|
||||
wr, err := l.call(ctx, "prefresh"+l.now().UTC().Format("150405"), map[string]any{"release_id": "profile-refresh",
|
||||
"layer": LayerHost, "mode": "profile-refresh"})
|
||||
took := l.now().Sub(start).Round(100 * time.Millisecond).String()
|
||||
switch {
|
||||
case err != nil:
|
||||
lg.Warn("osupdate: profile refresh — the wrapper gave no answer; the next agent start asks again", "err", err, "took", took)
|
||||
l.noteProfileAsk("no_answer", nil)
|
||||
return ""
|
||||
case wr.refused():
|
||||
lg.Info("osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)", "refused", string(wr.Refused))
|
||||
l.noteProfileAsk("refused", nil)
|
||||
return ""
|
||||
case wr.failed():
|
||||
lg.Error("osupdate: profile refresh FAILED — the profiles are as before; the next agent start asks again", "failed", string(wr.Failed), "took", took)
|
||||
l.noteProfileAsk("failed", nil)
|
||||
return "failed"
|
||||
}
|
||||
var res ProfileRefreshResult
|
||||
_ = json.Unmarshal(wr.ProfileRefresh, &res)
|
||||
switch res.Outcome {
|
||||
case "refreshed":
|
||||
units := make([]string, 0, len(res.Restarted))
|
||||
for _, u := range res.Restarted {
|
||||
units = append(units, u.Unit)
|
||||
}
|
||||
if len(units) > 0 {
|
||||
lg.Warn("osupdate: profile refresh — security profiles reloaded from a fresh compile; a FAILED service whose profile was stale was restarted",
|
||||
"kernel", res.Kernel, "restarted", strings.Join(units, ","), "rewritten", len(res.Rewritten), "wrapper_seconds", res.Seconds, "took", took)
|
||||
} else {
|
||||
lg.Info("osupdate: profile refresh — security profiles reloaded from a fresh compile", "kernel", res.Kernel,
|
||||
"rewritten", len(res.Rewritten), "wrapper_seconds", res.Seconds, "took", took)
|
||||
}
|
||||
case "not_needed":
|
||||
lg.Info("osupdate: profile refresh not needed", "kernel", res.Kernel, "reason", res.Reason)
|
||||
default:
|
||||
lg.Warn("osupdate: profile refresh — the wrapper answered with no outcome this agent knows", "outcome", res.Outcome)
|
||||
}
|
||||
if res.Outcome == "refreshed" || res.Outcome == "not_needed" {
|
||||
l.noteProfileAsk(res.Outcome, &res)
|
||||
}
|
||||
return res.Outcome
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// ---- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----
|
||||
|
||||
func profileCalls(w *fakeWrapper) []map[string]any {
|
||||
var m []map[string]any
|
||||
for _, p := range w.plans {
|
||||
if p["mode"] == "profile-refresh" {
|
||||
m = append(m, p)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func refreshed() *WrapperReport {
|
||||
return &WrapperReport{ProfileRefresh: json.RawMessage(`{"outcome":"refreshed","kernel":"7.0.14-28-pve","rewritten":["f6da0254.0/usr.sbin.chronyd"],` +
|
||||
`"restarted":[{"unit":"chrony.service","program":"/usr/sbin/chronyd","rc":0,"active":true}],"seconds":0.5}`)}
|
||||
}
|
||||
|
||||
// One call, a host-layer plan that names NOTHING, and nothing to the hub.
|
||||
// COMPANION RED-PROOF (observed): add "units" to the plan → this test fails; drop the Appliance check → the BYO test fails.
|
||||
func TestProfileRefresh_OneCallNoNamesNoHubReport(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "refreshed" {
|
||||
t.Fatalf("outcome = %q", got)
|
||||
}
|
||||
pc := profileCalls(w)
|
||||
if len(pc) != 1 {
|
||||
t.Fatalf("calls = %d", len(pc))
|
||||
}
|
||||
if len(pc[0]) != 3 || pc[0]["layer"] != LayerHost || pc[0]["release_id"] != "profile-refresh" {
|
||||
t.Fatalf("the plan carries more than release_id, layer and mode: %v", pc[0])
|
||||
}
|
||||
if len(h.reports) != 0 {
|
||||
t.Fatalf("the hub got a report: %+v", h.reports)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileRefresh_ByoHostIsNeverAsked(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||
l, _ := newLeg(t, w, nil)
|
||||
l.Appliance = false
|
||||
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "" || len(profileCalls(w)) != 0 {
|
||||
t.Fatalf("a BYO host was asked: %q %v", got, profileCalls(w))
|
||||
}
|
||||
}
|
||||
|
||||
// The heavy-op gate: busy → wait and ask again; still busy after the tries → not run (the next start asks again).
|
||||
// COMPANION RED-PROOF (observed): ignore `ok` from the gate → the "always busy" half fails.
|
||||
func TestProfileRefresh_WaitsForTheHeavyOpGate(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||
l, _ := newLeg(t, w, nil)
|
||||
n, released := 0, 0
|
||||
gate := func() (func(), string, bool) {
|
||||
n++
|
||||
if n < 3 {
|
||||
return nil, "primary-backup", false
|
||||
}
|
||||
return func() { released++ }, "", true
|
||||
}
|
||||
if got := l.ProfileRefresh(context.Background(), "agent start", gate, 5, time.Minute); got != "refreshed" {
|
||||
t.Fatalf("outcome = %q after %d gate tries", got, n)
|
||||
}
|
||||
if n != 3 || released != 1 || len(profileCalls(w)) != 1 {
|
||||
t.Fatalf("gate tries %d, released %d, calls %d", n, released, len(profileCalls(w)))
|
||||
}
|
||||
w2 := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||
l2, _ := newLeg(t, w2, nil)
|
||||
busy := func() (func(), string, bool) { return nil, "restore-test", false }
|
||||
if got := l2.ProfileRefresh(context.Background(), "agent start", busy, 3, time.Minute); got != "" || len(profileCalls(w2)) != 0 {
|
||||
t.Fatalf("ran beside a heavy operation: %q %d call(s)", got, len(profileCalls(w2)))
|
||||
}
|
||||
}
|
||||
|
||||
// An older wrapper refuses the mode: logged, nothing else. A failure is said as "failed".
|
||||
func TestProfileRefresh_RefusedAndFailed(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, profileRep: &WrapperReport{Refused: json.RawMessage(`{"code":"R11","reason":"unknown mode 'profile-refresh'"}`)}}
|
||||
l, _ := newLeg(t, w, nil)
|
||||
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "" {
|
||||
t.Fatalf("refused → %q", got)
|
||||
}
|
||||
w2 := &fakeWrapper{t: t, profileRep: &WrapperReport{Failed: json.RawMessage(`{"rc":1,"step":"profile-refresh"}`)}}
|
||||
l2, _ := newLeg(t, w2, nil)
|
||||
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "failed" {
|
||||
t.Fatalf("failed → %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// R-1383 (release 3): the refresh restarted a failed service on the N100 and only the box's journal knew. The stanza
|
||||
// that rides the host report: nothing before any ask (an UNTYPED nil — the report omits the field); after a refresh,
|
||||
// everything; after an agent RESTART, the kernel and time of that refresh again — from the wrapper's answer to the new
|
||||
// process's ask, because the wrapper's record FILE may be root-only on a box (its folder is 0700 where a signed job
|
||||
// made it first). Where the file can be read, it adds the detail back.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-b3.txt).
|
||||
func TestProfileRefreshStanza_RecordAndLastAsk(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
unreadable := filepath.Join(t.TempDir(), "root-only", "profile-refresh.json") // does not exist: as a 0700 folder reads
|
||||
l.ProfileRecordPath = unreadable
|
||||
now := time.Date(2026, 11, 14, 11, 30, 11, 0, time.UTC)
|
||||
l.Now = func() time.Time { return now }
|
||||
if st := l.ProfileRefreshStanza(); st != nil {
|
||||
t.Fatalf("before any ask the stanza is %#v, want an untyped nil", st)
|
||||
}
|
||||
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "refreshed" {
|
||||
t.Fatalf("outcome = %q", got)
|
||||
}
|
||||
b, _ := json.Marshal(l.ProfileRefreshStanza())
|
||||
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":1,"restarted":["chrony.service"],"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"}` {
|
||||
t.Fatalf("stanza after a refresh, record file unreadable = %s", b)
|
||||
}
|
||||
// an agent restart: a new process asks, the wrapper answers "not needed" with ITS record of the last refresh
|
||||
l2, _ := newLeg(t, &fakeWrapper{t: t, profileRep: &WrapperReport{ProfileRefresh: json.RawMessage(
|
||||
`{"outcome":"not_needed","reason":"this kernel was done","kernel":"7.0.14-28-pve","last_refresh_kernel":"7.0.14-28-pve","last_refresh_at":"2026-11-14T11:30:11Z"}`)}},
|
||||
&hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
l2.ProfileRecordPath = unreadable
|
||||
l2.Now = func() time.Time { return now.Add(3 * time.Hour) }
|
||||
if st := l2.ProfileRefreshStanza(); st != nil {
|
||||
t.Fatalf("a new process that has not asked and cannot read the record: %#v, want nil", st)
|
||||
}
|
||||
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "not_needed" {
|
||||
t.Fatalf("outcome = %q", got)
|
||||
}
|
||||
b, _ = json.Marshal(l2.ProfileRefreshStanza())
|
||||
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
|
||||
t.Fatalf("stanza after an agent restart (the wrapper's answer, no detail) = %s", b)
|
||||
}
|
||||
// where the record file CAN be read, the detail is back — and a name that is not a unit name is dropped
|
||||
rec := filepath.Join(t.TempDir(), "profile-refresh.json")
|
||||
if err := os.WriteFile(rec, []byte(`{"at":"2026-11-14T11:30:11Z","kernel":"7.0.14-28-pve","restarted":["chrony.service","bad name; rm -rf"],`+
|
||||
`"rewritten":["a","b"]}`), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
l2.ProfileRecordPath = rec
|
||||
b, _ = json.Marshal(l2.ProfileRefreshStanza())
|
||||
if string(b) != `{"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":2,"restarted":["chrony.service"],"last_ask":"not_needed","last_ask_at":"2026-11-14T14:30:11Z"}` {
|
||||
t.Fatalf("stanza with a readable record = %s", b)
|
||||
}
|
||||
// a failed ask is told, beside the older record
|
||||
l2.Runner = (&fakeWrapper{t: t, profileRep: &WrapperReport{Failed: json.RawMessage(`{"code":"F1","reason":"apparmor_parser failed"}`)}})
|
||||
l2.Now = func() time.Time { return now.Add(4 * time.Hour) }
|
||||
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "failed" {
|
||||
t.Fatalf("outcome = %q", got)
|
||||
}
|
||||
b, _ = json.Marshal(l2.ProfileRefreshStanza())
|
||||
if !strings.Contains(string(b), `"last_ask":"failed","last_ask_at":"2026-11-14T15:30:11Z"`) || !strings.Contains(string(b), `"kernel":"7.0.14-28-pve"`) {
|
||||
t.Fatalf("stanza after a failed ask = %s", b)
|
||||
}
|
||||
// a record that is not JSON says nothing and breaks nothing
|
||||
_ = os.WriteFile(rec, []byte("not json"), 0o644)
|
||||
l3, _ := newLeg(t, &fakeWrapper{t: t}, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
l3.ProfileRecordPath = rec
|
||||
if st := l3.ProfileRefreshStanza(); st != nil {
|
||||
t.Fatalf("an unreadable record gave the stanza %#v", st)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package osupdate
|
||||
|
||||
// The root step that keeps a readable copy of Proxmox's root CA for the agent (R-1380, D80, felhom.eu `03` §12a).
|
||||
// The rule lives in the wrapper (mode pve-ca-copy: one fixed source, one fixed place, a CA certificate only); the plan
|
||||
// names nothing. internal/pvetrust decides when to ask and what to do with the answer.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// PVECACopyPath is where the wrapper puts the copy (felhom-os-apply PVE_CA_COPY) — the agent's `proxmox.tls.ca_file`.
|
||||
const PVECACopyPath = "/etc/felhom-agent/pve-root-ca.pem"
|
||||
|
||||
// PVECACopyResult is the wrapper's pve_ca_copy object.
|
||||
type PVECACopyResult struct {
|
||||
Outcome string `json:"outcome"` // written | same
|
||||
Path string `json:"path"`
|
||||
SHA256 string `json:"sha256"`
|
||||
NotAfter string `json:"not_after"`
|
||||
DaysLeft int `json:"days_left"`
|
||||
Replaced bool `json:"replaced"`
|
||||
}
|
||||
|
||||
// PVECACopy asks the wrapper to make or refresh the copy. An error names a refusal (T1–T3, or R11 from a wrapper that
|
||||
// does not know the mode yet) or a wrapper that gave no answer.
|
||||
func (l *Leg) PVECACopy(ctx context.Context) (PVECACopyResult, error) {
|
||||
wr, err := l.call(ctx, "pveca"+l.now().UTC().Format("150405.000"), map[string]any{"release_id": "pve-ca-copy",
|
||||
"layer": LayerHost, "mode": "pve-ca-copy"})
|
||||
if err != nil {
|
||||
return PVECACopyResult{}, err
|
||||
}
|
||||
if wr.refused() {
|
||||
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy refused: %s", wr.Refused)
|
||||
}
|
||||
if wr.failed() {
|
||||
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy failed: %s", wr.Failed)
|
||||
}
|
||||
var res PVECACopyResult
|
||||
if jerr := json.Unmarshal(wr.PVECACopy, &res); jerr != nil || res.Outcome == "" || res.Path == "" {
|
||||
return PVECACopyResult{}, fmt.Errorf("pve-ca-copy: the wrapper gave no result (%v)", jerr)
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package pbsdr
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
|
||||
// agent's state folder on a box, and a test must not need it.
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
config.CutWriteDir = dir
|
||||
code := m.Run()
|
||||
_ = os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
@@ -27,6 +27,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
"log/slog"
|
||||
"net/netip"
|
||||
"os"
|
||||
@@ -483,8 +484,10 @@ func (m *Manager) consumedFail(h, msg string, block *hub.WirePBSDR) {
|
||||
// seedEscrowStorageID sets escrow.pbs_storage_id in agent.json when empty/absent — the bare
|
||||
// `--selftest=escrow-create` one-liner must find its storage with no flags. A different existing
|
||||
// value is NEVER clobbered (warn-and-keep). Unknown config keys are preserved verbatim
|
||||
// (map[string]RawMessage read-modify-write, atomic rename).
|
||||
// (map[string]RawMessage read-modify-write, in place behind a journal — config.ReplaceInPlace).
|
||||
func (m *Manager) seedEscrowStorageID(storageID string) error {
|
||||
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
|
||||
defer config.FileMu.Unlock()
|
||||
if m.configPath == "" {
|
||||
return nil
|
||||
}
|
||||
@@ -527,30 +530,18 @@ func (m *Manager) seedEscrowStorageID(storageID string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
st, err := os.Stat(m.configPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// The config DIRECTORY (/etc/felhom-agent) is root-owned while the FILE is agent-owned —
|
||||
// tmp+rename is impossible for the non-root agent (rename mutates the directory; live
|
||||
// finding on the demo). So: park a recovery copy in the agent state dir, then rewrite the
|
||||
// file IN PLACE (O_TRUNC). The agent is the file's only writer and the content is small;
|
||||
// a torn write is recoverable from the parked copy.
|
||||
// finding on the demo). So: park a copy for a person in the agent state dir, then rewrite the
|
||||
// file IN PLACE through config.ReplaceInPlace, which journals the old bytes first — the next
|
||||
// start repairs a cut write (R-1380, TestRecoverCutWrite_*).
|
||||
if err := os.MkdirAll(m.stateDir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(m.stateDir, "agent.json.pre-seed"), raw, 0o600); err != nil {
|
||||
return fmt.Errorf("parking the pre-seed copy: %w", err)
|
||||
}
|
||||
f, err := os.OpenFile(m.configPath, os.O_WRONLY|os.O_TRUNC, st.Mode().Perm())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := f.Write(out); err != nil {
|
||||
f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
if err := config.ReplaceInPlace(m.configPath, raw, out); err != nil {
|
||||
return err
|
||||
}
|
||||
m.logger.Info("pbsdr: seeded escrow.pbs_storage_id (the ceremony one-liner needs no flags)", "storage_id", storageID)
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/httpx"
|
||||
@@ -45,6 +46,39 @@ type Client struct {
|
||||
node string
|
||||
token string
|
||||
http doer
|
||||
|
||||
// tlsMu guards http: SetTLS swaps the transport while calls run (R-1380 — the move from a pinned certificate to the
|
||||
// box's own root CA, and a refreshed CA copy, take effect with no agent restart).
|
||||
tlsMu sync.RWMutex
|
||||
timeout time.Duration
|
||||
}
|
||||
|
||||
func (c *Client) httpDoer() doer {
|
||||
c.tlsMu.RLock()
|
||||
defer c.tlsMu.RUnlock()
|
||||
return c.http
|
||||
}
|
||||
|
||||
// SetTLS replaces how this client trusts the Proxmox certificate, for every call that starts after it returns. On an
|
||||
// error (an unreadable CA file, a bad fingerprint) nothing changes.
|
||||
func (c *Client) SetTLS(t TLSConfig) error {
|
||||
tlsCfg, err := t.build()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
timeout := c.timeout
|
||||
if timeout == 0 {
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
hc := &http.Client{Timeout: timeout, Transport: httpx.NewTransport(tlsCfg, 0)}
|
||||
c.tlsMu.Lock()
|
||||
old := c.http
|
||||
c.http = hc
|
||||
c.tlsMu.Unlock()
|
||||
if o, ok := old.(*http.Client); ok && o != nil {
|
||||
o.CloseIdleConnections() // a kept-alive connection made under the old trust must not serve the next call
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// NewClient builds an API client. It validates required config and constructs the
|
||||
@@ -74,10 +108,11 @@ func NewClient(cfg Config) (*Client, error) {
|
||||
Transport: httpx.NewTransport(tlsCfg, 0),
|
||||
}
|
||||
return &Client{
|
||||
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
|
||||
node: cfg.Node,
|
||||
token: cfg.Token,
|
||||
http: hc,
|
||||
base: strings.TrimRight(cfg.Endpoint, "/") + "/api2/json",
|
||||
node: cfg.Node,
|
||||
token: cfg.Token,
|
||||
http: hc,
|
||||
timeout: timeout,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -125,7 +160,7 @@ func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader
|
||||
req.Header.Set("Content-Type", contentType)
|
||||
}
|
||||
|
||||
resp, err := c.http.Do(req)
|
||||
resp, err := c.httpDoer().Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("proxmox: %s %s: %w", method, path, err)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
package proxmox
|
||||
|
||||
import (
|
||||
"crypto/x509"
|
||||
"errors"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// The causes a call to Proxmox can fail on TRUST (R-1380, felhom.eu `03` §12a). A symptom alarm („the agent channel is
|
||||
// down") exists; these name the cause.
|
||||
const (
|
||||
TrustPinnedCertChanged = "pinned_cert_changed" // the pinned certificate is not the one Proxmox serves any more
|
||||
TrustCAMismatch = "ca_mismatch" // the CA file did not sign the served certificate
|
||||
TrustCertExpired = "cert_expired" // the served certificate (or the CA) has ended
|
||||
TrustCAFileUnreadable = "ca_file_unreadable" // the CA file is missing or holds no certificate
|
||||
TrustNameMismatch = "name_mismatch" // the certificate does not name the address the agent calls
|
||||
)
|
||||
|
||||
// TrustProblem names why err is a TRUST failure, or "" when it is none (a refused connection, a 401, a timeout).
|
||||
func TrustProblem(err error) string {
|
||||
if err == nil {
|
||||
return ""
|
||||
}
|
||||
var ua x509.UnknownAuthorityError
|
||||
var inv x509.CertificateInvalidError
|
||||
var hn x509.HostnameError
|
||||
switch {
|
||||
case strings.Contains(err.Error(), "TLS pin mismatch"):
|
||||
return TrustPinnedCertChanged
|
||||
case errors.As(err, &ua):
|
||||
return TrustCAMismatch
|
||||
case errors.As(err, &inv):
|
||||
if inv.Reason == x509.Expired {
|
||||
return TrustCertExpired
|
||||
}
|
||||
return TrustCAMismatch
|
||||
case errors.As(err, &hn):
|
||||
return TrustNameMismatch
|
||||
case strings.Contains(err.Error(), "reading TLS CA file"), strings.Contains(err.Error(), "contained no usable certificates"):
|
||||
return TrustCAFileUnreadable
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package pvetrust
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
)
|
||||
|
||||
// TestMain points the config write journal (config.CutWriteDir, R-1380) at a temp folder: the real one is the
|
||||
// agent's state folder on a box, and a test must not need it.
|
||||
func TestMain(m *testing.M) {
|
||||
dir, err := os.MkdirTemp("", "felhom-cutwrite-")
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
config.CutWriteDir = dir
|
||||
code := m.Run()
|
||||
_ = os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
}
|
||||
@@ -0,0 +1,437 @@
|
||||
// Package pvetrust moves the agent's trust in Proxmox from ONE pinned certificate to the box's own root CA, and names
|
||||
// the cause when a call fails on trust (R-1380, D80, felhom.eu `09` §3 decision 303; design `03-host-agent.md` §12a).
|
||||
//
|
||||
// WHY. Proxmox makes a new node certificate by itself, at most every two years. An agent that pins the old one is then
|
||||
// blind: every call fails, no host report reaches the hub, and the only alarm names a symptom. A copy of the box's own
|
||||
// root CA verifies every certificate that CA signs.
|
||||
//
|
||||
// THE ORDER, and it is the whole point (a box must never be left with neither way working):
|
||||
//
|
||||
// 1. the root step makes or refreshes the copy (it refuses anything that is not one CA certificate);
|
||||
// 2. a REAL call to Proxmox is made with the copy ALONE — while the fingerprint still is the configured trust;
|
||||
// 3. only after that call worked, ONE write of the agent's config sets `ca_file` and clears the fingerprint;
|
||||
// 4. the running client takes the new trust (no restart).
|
||||
//
|
||||
// If 1 or 2 fails nothing is written and the fingerprint stays. If 3 fails the fingerprint stays, in the file and in
|
||||
// the running client. `proxmox.tls` keeps its order in code — a set fingerprint wins over a CA file — so the
|
||||
// fingerprint is cleared LAST, in the same write that sets `ca_file`.
|
||||
//
|
||||
// WHO WRITES THE CONFIG. The agent itself: `agent.json` is the agent's own file (0600) in a root-owned folder, so it is
|
||||
// rewritten in place after a recovery copy was parked in the agent's state folder — the same way the backup-target move
|
||||
// and the PBS seed write it. No root step touches it.
|
||||
package pvetrust
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// EndsSoonDays: the copy's certificate ends within this many days → the problem „ca_copy_ends_soon". Proxmox renews its
|
||||
// root CA when less than two years are left, so a healthy box never comes near it.
|
||||
const EndsSoonDays = 30
|
||||
|
||||
// Problems this package names beyond proxmox.Trust*.
|
||||
const (
|
||||
ProblemCopyUnavailable = "ca_copy_unavailable" // the root step refused or did not answer
|
||||
ProblemCopyNoProof = "ca_copy_does_not_verify" // a call with the copy alone failed — the fingerprint stays
|
||||
ProblemCopyEndsSoon = "ca_copy_ends_soon"
|
||||
ProblemConfigNotSaved = "config_not_written" // the proof worked but the config could not be written — the fingerprint stays
|
||||
)
|
||||
|
||||
// Copy is what the root step answered.
|
||||
type Copy struct {
|
||||
Path string
|
||||
Outcome string // written | same
|
||||
NotAfter string
|
||||
DaysLeft int
|
||||
}
|
||||
|
||||
// Status is the stanza that rides the host report (`proxmox_trust`). No secret: a mode, a cause, a date.
|
||||
type Status struct {
|
||||
Mode string `json:"mode"` // ca | fingerprint | ca_file_own | insecure | system
|
||||
Problem string `json:"problem,omitempty"` // "" = none
|
||||
Detail string `json:"detail,omitempty"`
|
||||
CANotAfter string `json:"ca_not_after,omitempty"`
|
||||
CADaysLeft *int `json:"ca_days_left,omitempty"`
|
||||
MovedAt string `json:"moved_at,omitempty"` // when this box moved from the pinned certificate to the CA copy
|
||||
CheckedAt string `json:"checked_at,omitempty"`
|
||||
}
|
||||
|
||||
// Manager holds the trust state of one agent process.
|
||||
type Manager struct {
|
||||
// EnsureCopy runs the root step (osupdate.Leg.PVECACopy).
|
||||
EnsureCopy func(ctx context.Context) (Copy, error)
|
||||
// Probe makes one real call to Proxmox with EXACTLY this trust (a throwaway client).
|
||||
Probe func(ctx context.Context, t proxmox.TLSConfig) error
|
||||
// Apply gives the running client the new trust (proxmox.Client.SetTLS).
|
||||
Apply func(t proxmox.TLSConfig) error
|
||||
// ConfigPath is agent.json ("" = the config came from the environment: nothing can be saved, so nothing is moved).
|
||||
ConfigPath string
|
||||
StateDir string
|
||||
Logger *slog.Logger
|
||||
Now func() time.Time
|
||||
// MinGap between two runs that a failed call asked for (default 10 minutes).
|
||||
MinGap time.Duration
|
||||
|
||||
runMu sync.Mutex // serialises Ensure
|
||||
mu sync.Mutex
|
||||
cur proxmox.TLSConfig
|
||||
status Status
|
||||
lastRun time.Time
|
||||
}
|
||||
|
||||
// copyStepTimeout bounds the root step that makes the CA copy.
|
||||
const copyStepTimeout = 90 * time.Second
|
||||
|
||||
func (m *Manager) now() time.Time {
|
||||
if m.Now != nil {
|
||||
return m.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (m *Manager) log() *slog.Logger {
|
||||
if m.Logger != nil {
|
||||
return m.Logger
|
||||
}
|
||||
return slog.Default()
|
||||
}
|
||||
|
||||
// Start records the trust the process started with. Call once, before Ensure.
|
||||
func (m *Manager) Start(t proxmox.TLSConfig) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.cur = t
|
||||
m.status = Status{Mode: modeOf(t)}
|
||||
}
|
||||
|
||||
func modeOf(t proxmox.TLSConfig) string {
|
||||
switch {
|
||||
case t.InsecureSkipVerify:
|
||||
return "insecure"
|
||||
case t.Fingerprint != "":
|
||||
return "fingerprint"
|
||||
case t.CAFile != "":
|
||||
return "ca"
|
||||
}
|
||||
return "system"
|
||||
}
|
||||
|
||||
// Status is the current stanza (nil-safe).
|
||||
func (m *Manager) Status() *Status {
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
s := m.status
|
||||
return &s
|
||||
}
|
||||
|
||||
// Moved reports whether the box trusts Proxmox by the CA copy (nothing is left to move).
|
||||
func (m *Manager) Moved() bool {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.cur.Fingerprint == "" && m.cur.CAFile != ""
|
||||
}
|
||||
|
||||
// Run asks again on a timer until ctx ends: every `pending` while the box is not moved yet (a wrapper that did not
|
||||
// know the mode, a Proxmox that did not answer), every `settled` after (a renewed CA refreshes the copy).
|
||||
func (m *Manager) Run(ctx context.Context, pending, settled time.Duration) {
|
||||
for {
|
||||
d := settled
|
||||
if !m.Moved() {
|
||||
d = pending
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(d):
|
||||
m.Ensure(ctx, "timer")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Current is the trust the running client uses.
|
||||
func (m *Manager) Current() proxmox.TLSConfig {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.cur
|
||||
}
|
||||
|
||||
// OnCallError is the hook for a failed Proxmox call: when the failure is one of TRUST, it names the cause and runs
|
||||
// Ensure (at most once per MinGap) — a pinned box whose certificate changed moves to the CA copy right then, and a box
|
||||
// whose CA copy was refreshed reloads it. A failure of another kind does nothing.
|
||||
func (m *Manager) OnCallError(ctx context.Context, err error) {
|
||||
cause := proxmox.TrustProblem(err)
|
||||
if cause == "" {
|
||||
return
|
||||
}
|
||||
m.mu.Lock()
|
||||
gap := m.MinGap
|
||||
if gap <= 0 {
|
||||
gap = 10 * time.Minute
|
||||
}
|
||||
tooSoon := !m.lastRun.IsZero() && m.now().Sub(m.lastRun) < gap
|
||||
if m.status.Problem == "" {
|
||||
m.status.Problem, m.status.Detail = cause, causeText(cause)
|
||||
}
|
||||
m.mu.Unlock()
|
||||
m.log().Error("pvetrust: a call to Proxmox failed on TRUST — "+causeText(cause), "cause", cause, "err", err)
|
||||
if tooSoon {
|
||||
return
|
||||
}
|
||||
m.Ensure(ctx, cause)
|
||||
}
|
||||
|
||||
func causeText(cause string) string {
|
||||
switch cause {
|
||||
case proxmox.TrustPinnedCertChanged:
|
||||
return "the pinned certificate is not the one Proxmox serves any more (Proxmox made a new certificate)"
|
||||
case proxmox.TrustCAMismatch:
|
||||
return "the CA copy did not sign the certificate Proxmox serves (the box's root CA changed, or the copy is not this box's)"
|
||||
case proxmox.TrustCertExpired:
|
||||
return "the certificate Proxmox serves, or its CA, has ended"
|
||||
case proxmox.TrustCAFileUnreadable:
|
||||
return "the CA copy is missing or holds no certificate"
|
||||
case proxmox.TrustNameMismatch:
|
||||
return "the certificate Proxmox serves does not name the address the agent calls"
|
||||
}
|
||||
return cause
|
||||
}
|
||||
|
||||
// Ensure runs the order of the package comment once. why is a log word ("start", "daily", a trust cause). It returns
|
||||
// true when the running trust changed.
|
||||
func (m *Manager) Ensure(ctx context.Context, why string) bool {
|
||||
// One run at a time. Ensure is called from the start, the timer, a failed collect and after a bundle install; two
|
||||
// overlapping runs would both read the config, and the second would park the ALREADY-MOVED file as the "before"
|
||||
// copy (losing the fingerprint from the recovery copy). Found by review, 2026-11-14.
|
||||
m.runMu.Lock()
|
||||
defer m.runMu.Unlock()
|
||||
m.mu.Lock()
|
||||
cur := m.cur
|
||||
m.lastRun = m.now()
|
||||
m.mu.Unlock()
|
||||
lg := m.log().With("why", why, "mode", modeOf(cur))
|
||||
set := func(mode, problem, detail string, c *Copy) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
moved := m.status.MovedAt
|
||||
m.status = Status{Mode: mode, Problem: problem, Detail: detail, MovedAt: moved, CheckedAt: m.now().UTC().Format(time.RFC3339)}
|
||||
if c != nil {
|
||||
d := c.DaysLeft
|
||||
m.status.CANotAfter, m.status.CADaysLeft = c.NotAfter, &d
|
||||
}
|
||||
}
|
||||
if cur.InsecureSkipVerify {
|
||||
lg.Warn("pvetrust: certificate checks are switched off in the config (insecure_skip_verify) — nothing is moved")
|
||||
set("insecure", "", "", nil)
|
||||
return false
|
||||
}
|
||||
if m.EnsureCopy == nil || m.Probe == nil {
|
||||
return false
|
||||
}
|
||||
// 1. the copy. The root step gets a deadline of its own: Ensure runs on the daemon's start path and inside the
|
||||
// report loop, and a wrapper that hangs (sudo, a stuck read of /etc/pve) must not stop either.
|
||||
cctx, ccancel := context.WithTimeout(ctx, copyStepTimeout)
|
||||
c, err := m.EnsureCopy(cctx)
|
||||
ccancel()
|
||||
if err != nil {
|
||||
m.mu.Lock()
|
||||
recorded := m.status.Problem
|
||||
m.mu.Unlock()
|
||||
if rootStepTooOld(err) && cur.Fingerprint != "" && !isTrustCause(why) && !isTrustCause(recorded) {
|
||||
// The agent is newer than the box's root files: between the agent update and the root-files job the
|
||||
// wrapper does not know the step yet. The pinned certificate works; this is not a trust problem, and
|
||||
// naming one mailed the operator at every box's update (seen on Tester 1, 2026-11-15). The bundle
|
||||
// install calls Ensure again. NEVER on a run that a failed call started, and never over a recorded
|
||||
// trust failure: then the pin does NOT work, and the box must say so (the commit's security review).
|
||||
lg.Info("pvetrust: the root files are older than this agent (no CA-copy step yet) — the pinned certificate stays; the next root-files update moves the box", "err", err)
|
||||
set(modeOf(cur), "", "", nil)
|
||||
return false
|
||||
}
|
||||
lg.Warn("pvetrust: the root step gave no CA copy — the trust stays as it is", "err", err)
|
||||
set(modeOf(cur), ProblemCopyUnavailable, truncate(err.Error(), 240), nil)
|
||||
return false
|
||||
}
|
||||
ca := proxmox.TLSConfig{CAFile: c.Path}
|
||||
onCopy := cur.Fingerprint == "" && cur.CAFile == c.Path
|
||||
if cur.Fingerprint == "" && cur.CAFile != "" && cur.CAFile != c.Path {
|
||||
// an operator's own CA file: never replaced
|
||||
lg.Info("pvetrust: the config names its own CA file — left as it is", "ca_file", cur.CAFile)
|
||||
set("ca_file_own", "", "", &c)
|
||||
return false
|
||||
}
|
||||
if onCopy && c.Outcome != "written" && why != proxmox.TrustCAMismatch && why != proxmox.TrustCAFileUnreadable && why != proxmox.TrustCertExpired {
|
||||
// the box already trusts by the copy and the copy did not change: nothing to prove again
|
||||
problem, detail := "", ""
|
||||
if c.DaysLeft < EndsSoonDays {
|
||||
problem, detail = ProblemCopyEndsSoon, fmt.Sprintf("the CA copy ends in %d day(s), on %s", c.DaysLeft, c.NotAfter)
|
||||
lg.Error("pvetrust: "+detail, "days_left", c.DaysLeft)
|
||||
}
|
||||
set("ca", problem, detail, &c)
|
||||
return false
|
||||
}
|
||||
// 2. the proof: a real call with the copy ALONE — the fingerprint is still the configured trust
|
||||
pctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
perr := m.Probe(pctx, ca)
|
||||
cancel()
|
||||
if perr != nil {
|
||||
cause := proxmox.TrustProblem(perr)
|
||||
detail := "a call to Proxmox with the CA copy alone failed"
|
||||
if cause != "" {
|
||||
detail += ": " + causeText(cause)
|
||||
} else {
|
||||
detail += " (not a trust failure: " + truncate(perr.Error(), 160) + ")"
|
||||
}
|
||||
if onCopy {
|
||||
lg.Error("pvetrust: the box trusts Proxmox by the CA copy and the copy does not verify the served certificate", "err", perr)
|
||||
set("ca", firstNonEmpty(cause, ProblemCopyNoProof), detail, &c)
|
||||
return false
|
||||
}
|
||||
if cause == "" {
|
||||
// Proxmox did not answer at all (it restarts, the box just booted): nothing is known about the copy, so no
|
||||
// cause is named; the next run asks again.
|
||||
lg.Warn("pvetrust: the proof could not be made now (not a trust failure) — the trust stays as it is; asked again later", "err", perr)
|
||||
set(modeOf(cur), "", "", &c)
|
||||
return false
|
||||
}
|
||||
lg.Error("pvetrust: the proof with the CA copy FAILED — the fingerprint STAYS, nothing was written", "err", perr)
|
||||
set(modeOf(cur), ProblemCopyNoProof, detail+" — the fingerprint stays", &c)
|
||||
return false
|
||||
}
|
||||
if onCopy {
|
||||
// the copy was refreshed (or a trust failure asked): the running client reloads it
|
||||
if m.Apply != nil {
|
||||
if aerr := m.Apply(ca); aerr != nil {
|
||||
set("ca", proxmox.TrustCAFileUnreadable, truncate(aerr.Error(), 200), &c)
|
||||
return false
|
||||
}
|
||||
}
|
||||
lg.Warn("pvetrust: the CA copy was refreshed and proven — the running client uses it now", "outcome", c.Outcome)
|
||||
set("ca", "", "", &c)
|
||||
return true
|
||||
}
|
||||
// 3. the config: ONE write sets ca_file and clears the fingerprint — only now, after the proof
|
||||
if m.ConfigPath == "" {
|
||||
lg.Warn("pvetrust: the proof worked, but the config did not come from a file — nothing can be saved, so the trust stays as it is")
|
||||
set(modeOf(cur), ProblemConfigNotSaved, "the config is not a file; the move was not made", &c)
|
||||
return false
|
||||
}
|
||||
if werr := m.writeConfig(c.Path); werr != nil {
|
||||
lg.Error("pvetrust: the proof worked but the config could NOT be written — the fingerprint stays", "err", werr)
|
||||
set(modeOf(cur), ProblemConfigNotSaved, truncate(werr.Error(), 200), &c)
|
||||
return false
|
||||
}
|
||||
// 4. the running client
|
||||
if m.Apply != nil {
|
||||
if aerr := m.Apply(ca); aerr != nil {
|
||||
lg.Error("pvetrust: the config is written but the running client could not take the CA copy — it keeps the old trust until the next start", "err", aerr)
|
||||
}
|
||||
}
|
||||
m.mu.Lock()
|
||||
m.cur = ca
|
||||
m.mu.Unlock()
|
||||
lg.Warn("pvetrust: MOVED — the agent trusts Proxmox by the box's own root CA now; the pinned fingerprint was cleared AFTER a working call with the copy",
|
||||
"ca_file", c.Path, "ca_not_after", c.NotAfter)
|
||||
set("ca", "", "", &c)
|
||||
m.mu.Lock()
|
||||
m.status.MovedAt = m.now().UTC().Format(time.RFC3339)
|
||||
m.mu.Unlock()
|
||||
return true
|
||||
}
|
||||
|
||||
func firstNonEmpty(a, b string) string {
|
||||
if a != "" {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n]
|
||||
}
|
||||
|
||||
// writeConfig sets proxmox.tls.ca_file and clears proxmox.tls.fingerprint in ONE write of agent.json. Every other key is
|
||||
// kept byte for byte. A recovery copy is parked in the state folder first; the result is read back and checked.
|
||||
func (m *Manager) writeConfig(caPath string) error {
|
||||
config.FileMu.Lock() // one writer of agent.json at a time in this process (config.FileMu)
|
||||
defer config.FileMu.Unlock()
|
||||
raw, err := os.ReadFile(m.ConfigPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var doc map[string]json.RawMessage
|
||||
if err := json.Unmarshal(raw, &doc); err != nil {
|
||||
return fmt.Errorf("parse %s: %w", m.ConfigPath, err)
|
||||
}
|
||||
px := map[string]json.RawMessage{}
|
||||
if cur, ok := doc["proxmox"]; ok {
|
||||
if err := json.Unmarshal(cur, &px); err != nil {
|
||||
return fmt.Errorf("parse the proxmox section: %w", err)
|
||||
}
|
||||
}
|
||||
tlsSec := map[string]json.RawMessage{}
|
||||
if cur, ok := px["tls"]; ok {
|
||||
if err := json.Unmarshal(cur, &tlsSec); err != nil {
|
||||
return fmt.Errorf("parse the proxmox.tls section: %w", err)
|
||||
}
|
||||
}
|
||||
caJSON, _ := json.Marshal(caPath)
|
||||
tlsSec["ca_file"] = caJSON
|
||||
tlsSec["fingerprint"] = json.RawMessage(`""`)
|
||||
b, _ := json.Marshal(tlsSec)
|
||||
px["tls"] = b
|
||||
b, _ = json.Marshal(px)
|
||||
doc["proxmox"] = b
|
||||
out, err := json.MarshalIndent(doc, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out = append(out, '\n')
|
||||
if m.StateDir != "" {
|
||||
if err := os.MkdirAll(m.StateDir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(m.StateDir, "agent.json.pre-pve-trust"), raw, 0o600); err != nil {
|
||||
return fmt.Errorf("parking the recovery copy: %w", err)
|
||||
}
|
||||
}
|
||||
// In place, not tmp+rename: the folder is root-owned and the file is the agent's. config.ReplaceInPlace keeps the
|
||||
// old bytes in a journal first, so a power cut in the middle is repaired at the next start (R-1380); it also reads
|
||||
// the file back and puts the old bytes back when the write fails with the agent alive.
|
||||
if err := config.ReplaceInPlace(m.ConfigPath, raw, out); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// rootStepTooOld says whether the root step refused because it does not know the CA-copy mode at all (the wrapper's
|
||||
// R11 „unknown mode”): the box's root files are older than the agent. Pinned by
|
||||
// TestRootStepOlderThanTheAgent_IsNotATrustProblem.
|
||||
func rootStepTooOld(err error) bool {
|
||||
return err != nil && strings.Contains(err.Error(), `"R11"`) && strings.Contains(err.Error(), "unknown mode")
|
||||
}
|
||||
|
||||
// isTrustCause says whether s is one of the causes proxmox.TrustProblem names for a FAILED call.
|
||||
func isTrustCause(s string) bool {
|
||||
switch s {
|
||||
case proxmox.TrustPinnedCertChanged, proxmox.TrustCAMismatch, proxmox.TrustCertExpired,
|
||||
proxmox.TrustCAFileUnreadable, proxmox.TrustNameMismatch:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,566 @@
|
||||
package pvetrust
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// ---- a small Proxmox: a root CA that signs node certificates, and a server that can change its certificate ----
|
||||
|
||||
type pki struct {
|
||||
t *testing.T
|
||||
caKey *ecdsa.PrivateKey
|
||||
caCert *x509.Certificate
|
||||
caPEM []byte
|
||||
}
|
||||
|
||||
func newPKI(t *testing.T, cn string) *pki {
|
||||
t.Helper()
|
||||
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
return (&pki{t: t, caKey: k}).renewCA(cn)
|
||||
}
|
||||
|
||||
// renewCA makes a new CA certificate with the SAME key and name (what Proxmox does when its root CA is renewed).
|
||||
func (p *pki) renewCA(cn string) *pki {
|
||||
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: cn},
|
||||
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true,
|
||||
KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign}
|
||||
der, err := x509.CreateCertificate(rand.Reader, tpl, tpl, &p.caKey.PublicKey, p.caKey)
|
||||
if err != nil {
|
||||
p.t.Fatal(err)
|
||||
}
|
||||
p.caCert, _ = x509.ParseCertificate(der)
|
||||
p.caPEM = pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *pki) leaf() tls.Certificate {
|
||||
k, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
tpl := &x509.Certificate{SerialNumber: big.NewInt(time.Now().UnixNano()), Subject: pkix.Name{CommonName: "node"},
|
||||
NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(12 * time.Hour), IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
|
||||
KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}}
|
||||
der, err := x509.CreateCertificate(rand.Reader, tpl, p.caCert, &k.PublicKey, p.caKey)
|
||||
if err != nil {
|
||||
p.t.Fatal(err)
|
||||
}
|
||||
return tls.Certificate{Certificate: [][]byte{der}, PrivateKey: k}
|
||||
}
|
||||
|
||||
func fp(c tls.Certificate) string {
|
||||
s := sha256.Sum256(c.Certificate[0])
|
||||
return hex.EncodeToString(s[:])
|
||||
}
|
||||
|
||||
type pve struct {
|
||||
srv *httptest.Server
|
||||
mu sync.Mutex
|
||||
cert tls.Certificate
|
||||
}
|
||||
|
||||
func newPVE(t *testing.T, c tls.Certificate) *pve {
|
||||
p := &pve{cert: c}
|
||||
p.srv = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
fmt.Fprint(w, `{"data":{"version":"9.2.2","release":"9.2","repoid":"x"}}`)
|
||||
}))
|
||||
// GetConfigForClient, not GetCertificate: httptest adds its own certificate, and for a call by IP address (no
|
||||
// server name) that one would win over GetCertificate.
|
||||
p.srv.TLS = &tls.Config{GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
return &tls.Config{Certificates: []tls.Certificate{p.cert}}, nil
|
||||
}}
|
||||
p.srv.StartTLS()
|
||||
t.Cleanup(p.srv.Close)
|
||||
return p
|
||||
}
|
||||
|
||||
// newCert is `pveupdate` making a new node certificate: served from the next connection on.
|
||||
func (p *pve) newCert(c tls.Certificate) {
|
||||
p.mu.Lock()
|
||||
p.cert = c
|
||||
p.mu.Unlock()
|
||||
p.srv.CloseClientConnections()
|
||||
}
|
||||
|
||||
// ---- the agent's side ----
|
||||
|
||||
type box struct {
|
||||
t *testing.T
|
||||
m *Manager
|
||||
px *proxmox.Client
|
||||
cfgPath string
|
||||
copyPath string
|
||||
source []byte // what the root step would copy (nil = the root step refuses)
|
||||
events []string
|
||||
applyErr error
|
||||
}
|
||||
|
||||
func cfgJSON(fingerprint, caFile string) string {
|
||||
return fmt.Sprintf(`{
|
||||
"hub": {"url": "https://hub.example", "host_id": "box-1", "api_key": "KEEP-ME"},
|
||||
"proxmox": {"endpoint": "X", "node": "n", "token": "u@pve!t=s", "tls": {"fingerprint": %q, "ca_file": %q, "insecure_skip_verify": false}, "custom": [1, 2, 3]},
|
||||
"unknown_key": {"stays": true}
|
||||
}`, fingerprint, caFile)
|
||||
}
|
||||
|
||||
func newBox(t *testing.T, srv *pve, start proxmox.TLSConfig, source []byte) *box {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
b := &box{t: t, cfgPath: filepath.Join(dir, "agent.json"), copyPath: filepath.Join(dir, "pve-root-ca.pem"), source: source}
|
||||
if start.CAFile == "COPY" {
|
||||
start.CAFile = b.copyPath
|
||||
if err := os.WriteFile(b.copyPath, source, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(b.cfgPath, []byte(cfgJSON(start.Fingerprint, start.CAFile)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pcfg := proxmox.Config{Endpoint: srv.srv.URL, Node: "n", Token: "u@pve!t=s", TLS: start, HTTPTimeout: 5 * time.Second}
|
||||
px, err := proxmox.NewClient(pcfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b.px = px
|
||||
b.m = &Manager{ConfigPath: b.cfgPath, StateDir: filepath.Join(dir, "state"),
|
||||
EnsureCopy: func(context.Context) (Copy, error) {
|
||||
b.events = append(b.events, "copy")
|
||||
if b.source == nil {
|
||||
return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3", "reason": "not a CA certificate"}`)
|
||||
}
|
||||
outcome := "same"
|
||||
if cur, _ := os.ReadFile(b.copyPath); string(cur) != string(b.source) {
|
||||
outcome = "written"
|
||||
if err := os.WriteFile(b.copyPath, b.source, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return Copy{Path: b.copyPath, Outcome: outcome, NotAfter: "2036-08-17T15:44:42Z", DaysLeft: 3560}, nil
|
||||
},
|
||||
Probe: func(ctx context.Context, tl proxmox.TLSConfig) error {
|
||||
b.events = append(b.events, fmt.Sprintf("probe(fp=%q,ca=%v) while the file says fp=%q", tl.Fingerprint, tl.CAFile != "", b.fileTLS().Fingerprint))
|
||||
c := pcfg
|
||||
c.TLS = tl
|
||||
p, err := proxmox.NewClient(c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = p.Version(ctx)
|
||||
return err
|
||||
},
|
||||
Apply: func(tl proxmox.TLSConfig) error {
|
||||
b.events = append(b.events, fmt.Sprintf("apply(fp=%q,ca=%v)", tl.Fingerprint, tl.CAFile != ""))
|
||||
if b.applyErr != nil {
|
||||
return b.applyErr
|
||||
}
|
||||
return b.px.SetTLS(tl)
|
||||
},
|
||||
}
|
||||
b.m.Start(start)
|
||||
return b
|
||||
}
|
||||
|
||||
type fileTLS struct {
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
CAFile string `json:"ca_file"`
|
||||
}
|
||||
|
||||
func (b *box) fileTLS() fileTLS {
|
||||
var d struct {
|
||||
Proxmox struct {
|
||||
TLS fileTLS `json:"tls"`
|
||||
} `json:"proxmox"`
|
||||
}
|
||||
raw, _ := os.ReadFile(b.cfgPath)
|
||||
if err := json.Unmarshal(raw, &d); err != nil {
|
||||
b.t.Fatalf("agent.json is not JSON any more: %v\n%s", err, raw)
|
||||
}
|
||||
return d.Proxmox.TLS
|
||||
}
|
||||
|
||||
func (b *box) call() error {
|
||||
_, err := b.px.Version(context.Background())
|
||||
return err
|
||||
}
|
||||
|
||||
func (b *box) has(prefix string) bool {
|
||||
for _, e := range b.events {
|
||||
if strings.HasPrefix(e, prefix) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Scenario: a NEW box — the installer wrote the copy and ca_file, no fingerprint. Nothing is proven again, nothing written.
|
||||
func TestNewBox_CAOnly_NothingToMove(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
srv := newPVE(t, p.leaf())
|
||||
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatal("a box that already trusts by the copy changed its trust")
|
||||
}
|
||||
after, _ := os.ReadFile(b.cfgPath)
|
||||
if string(before) != string(after) || b.has("probe") || b.has("apply") {
|
||||
t.Fatalf("the config was touched or a proof ran: %v", b.events)
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "ca" || st.Problem != "" || st.CADaysLeft == nil || *st.CADaysLeft != 3560 {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the call fails: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: an OLD box moves. The proof runs with the CA copy ALONE while the file still holds the fingerprint; only
|
||||
// then ONE write sets ca_file and clears the fingerprint; every other key stays; a recovery copy is parked.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/A/redproof-agent.txt): write the config
|
||||
// BEFORE the probe → the order check fails; skip the probe → TestProofFails_TheFingerprintStays fails.
|
||||
func TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
old, _ := os.ReadFile(b.cfgPath)
|
||||
if !b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatalf("the box did not move: %+v %v", b.m.Status(), b.events)
|
||||
}
|
||||
want := []string{"copy", fmt.Sprintf("probe(fp=\"\",ca=true) while the file says fp=%q", fp(leaf)), "apply(fp=\"\",ca=true)"}
|
||||
if strings.Join(b.events, " | ") != strings.Join(want, " | ") {
|
||||
t.Fatalf("order = %v\nwant %v", b.events, want)
|
||||
}
|
||||
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
|
||||
t.Fatalf("the config's tls = %+v", got)
|
||||
}
|
||||
raw, _ := os.ReadFile(b.cfgPath)
|
||||
for _, keep := range []string{`"KEEP-ME"`, `"unknown_key"`, `"stays"`, `"custom"`, `"u@pve!t=s"`, `"insecure_skip_verify"`} {
|
||||
if !strings.Contains(string(raw), keep) {
|
||||
t.Fatalf("the write lost %s:\n%s", keep, raw)
|
||||
}
|
||||
}
|
||||
if parked, _ := os.ReadFile(filepath.Join(b.m.StateDir, "agent.json.pre-pve-trust")); string(parked) != string(old) {
|
||||
t.Fatal("no recovery copy of the old config was parked")
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "ca" || st.MovedAt == "" || st.Problem != "" {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
// and now Proxmox makes a new certificate: the agent goes on, with no restart and no new client
|
||||
srv.newCert(p.leaf())
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("after a new Proxmox certificate the moved box fails: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: the proof fails (the copy is another box's CA). The fingerprint stays — in the file, byte for byte, and in
|
||||
// the running client — and the cause is named.
|
||||
func TestProofFails_TheFingerprintStays(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
other := newPKI(t, "another box")
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatal("the trust changed although the proof failed")
|
||||
}
|
||||
after, _ := os.ReadFile(b.cfgPath)
|
||||
if string(before) != string(after) {
|
||||
t.Fatal("the config was written although the proof failed")
|
||||
}
|
||||
if b.has("apply") {
|
||||
t.Fatalf("the running client was changed: %v", b.events)
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemCopyNoProof || !strings.Contains(st.Detail, "did not sign") {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the fingerprint no longer works: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: Proxmox makes a new certificate on a box that still PINS the old one. The failed call names the cause, the
|
||||
// box moves to the CA copy right then, and the next call works — no restart.
|
||||
func TestPinnedBox_NewCertificate_HealsWithNoRestart(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
srv.newCert(p.leaf())
|
||||
err := b.call()
|
||||
if proxmox.TrustProblem(err) != proxmox.TrustPinnedCertChanged {
|
||||
t.Fatalf("fixture: want a pin mismatch, got %v", err)
|
||||
}
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the box is still blind after the move: %v (%v)", err, b.events)
|
||||
}
|
||||
if got := b.fileTLS(); got.Fingerprint != "" || got.CAFile != b.copyPath {
|
||||
t.Fatalf("the config's tls = %+v", got)
|
||||
}
|
||||
// a failure that is NOT one of trust does nothing
|
||||
n := len(b.events)
|
||||
b.m.OnCallError(context.Background(), errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused"))
|
||||
if len(b.events) != n {
|
||||
t.Fatalf("a refused connection started a trust run: %v", b.events[n:])
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: the root CA itself is renewed with the same key. The copy is refreshed, proven, and the running client
|
||||
// reloads it; the config is not written again.
|
||||
func TestCARenewedWithTheSameKey_TheCopyIsRefreshedAndReloaded(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
srv := newPVE(t, p.leaf())
|
||||
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
|
||||
oldCA := string(p.caPEM)
|
||||
p.renewCA("PVE CA")
|
||||
srv.newCert(p.leaf())
|
||||
b.source = p.caPEM
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if !b.m.Ensure(context.Background(), "daily") {
|
||||
t.Fatalf("a refreshed copy was not taken: %+v %v", b.m.Status(), b.events)
|
||||
}
|
||||
if cur, _ := os.ReadFile(b.copyPath); string(cur) == oldCA || string(cur) != string(p.caPEM) {
|
||||
t.Fatal("the copy was not refreshed")
|
||||
}
|
||||
if !b.has("probe") || !b.has("apply") {
|
||||
t.Fatalf("a refreshed copy must be proven and reloaded: %v", b.events)
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatal("the config was written again")
|
||||
}
|
||||
if err := b.call(); err != nil {
|
||||
t.Fatalf("the call fails after the CA's renewal: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The root step refuses (the source is not a CA certificate): nothing changes, the cause is named.
|
||||
func TestRootStepRefuses_NothingChanges(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("probe") || b.has("apply") {
|
||||
t.Fatalf("something ran without a copy: %v", b.events)
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatal("the config was written")
|
||||
}
|
||||
if st := b.m.Status(); st.Problem != ProblemCopyUnavailable || !strings.Contains(st.Detail, "T3") {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// The proof works but the config cannot be written: the fingerprint stays, in the file and in the running client.
|
||||
func TestConfigCannotBeWritten_TheFingerprintStays(t *testing.T) {
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("root writes a read-only file")
|
||||
}
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
if err := os.Chmod(b.cfgPath, 0o400); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
|
||||
t.Fatalf("the trust changed without a saved config: %v", b.events)
|
||||
}
|
||||
if got := b.fileTLS(); got.Fingerprint != fp(leaf) {
|
||||
t.Fatalf("the fingerprint is gone from the file: %+v", got)
|
||||
}
|
||||
if st := b.m.Status(); st.Mode != "fingerprint" || st.Problem != ProblemConfigNotSaved {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// What is never moved: an operator's own CA file, a box with checks switched off, a config that is not a file.
|
||||
func TestNeverMoved(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
own := filepath.Join(t.TempDir(), "own-ca.pem")
|
||||
os.WriteFile(own, p.caPEM, 0o644)
|
||||
for name, start := range map[string]proxmox.TLSConfig{"own CA file": {CAFile: own}, "insecure": {InsecureSkipVerify: true}} {
|
||||
b := newBox(t, srv, start, p.caPEM)
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("apply") {
|
||||
t.Fatalf("%s: moved (%v)", name, b.events)
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatalf("%s: the config was written", name)
|
||||
}
|
||||
}
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
b.m.ConfigPath = ""
|
||||
if b.m.Ensure(context.Background(), "start") || b.has("apply") || b.m.Status().Problem != ProblemConfigNotSaved {
|
||||
t.Fatalf("a config that is not a file was 'moved': %v %+v", b.events, b.m.Status())
|
||||
}
|
||||
}
|
||||
|
||||
// The copy ends soon: the cause is named (the box still works).
|
||||
func TestCopyEndsSoon(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
srv := newPVE(t, p.leaf())
|
||||
b := newBox(t, srv, proxmox.TLSConfig{CAFile: "COPY"}, p.caPEM)
|
||||
inner := b.m.EnsureCopy
|
||||
b.m.EnsureCopy = func(ctx context.Context) (Copy, error) {
|
||||
c, err := inner(ctx)
|
||||
c.DaysLeft, c.NotAfter = 12, "2026-11-26T00:00:00Z"
|
||||
return c, err
|
||||
}
|
||||
b.m.Ensure(context.Background(), "daily")
|
||||
if st := b.m.Status(); st.Problem != ProblemCopyEndsSoon || !strings.Contains(st.Detail, "12 day") {
|
||||
t.Fatalf("status = %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// A trust failure asks at most once per MinGap.
|
||||
func TestOnCallError_Throttled(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
other := newPKI(t, "another box")
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, other.caPEM)
|
||||
srv.newCert(p.leaf())
|
||||
err := b.call()
|
||||
now := time.Date(2026, 11, 15, 3, 0, 0, 0, time.UTC)
|
||||
b.m.Now = func() time.Time { return now }
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
n := len(b.events)
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
if len(b.events) != n {
|
||||
t.Fatalf("a second failure inside the gap ran again: %v", b.events[n:])
|
||||
}
|
||||
now = now.Add(11 * time.Minute)
|
||||
b.m.OnCallError(context.Background(), err)
|
||||
if len(b.events) == n {
|
||||
t.Fatal("no new run after the gap")
|
||||
}
|
||||
if st := b.m.Status(); st.Problem == "" {
|
||||
t.Fatalf("a blind box names no cause: %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// Proxmox does not answer at all (it restarts): nothing is known about the copy — no cause is named, nothing is written,
|
||||
// and the box counts as not moved, so the timer asks again soon.
|
||||
func TestProofCannotBeMadeNow_NoCauseNamed(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, p.caPEM)
|
||||
b.m.Probe = func(context.Context, proxmox.TLSConfig) error {
|
||||
return errors.New("proxmox: GET /version: dial tcp 127.0.0.1:8006: connect: connection refused")
|
||||
}
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatal("moved without a proof")
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatal("the config was written")
|
||||
}
|
||||
if st := b.m.Status(); st.Problem != "" || st.Mode != "fingerprint" || b.m.Moved() {
|
||||
t.Fatalf("status = %+v moved=%v", st, b.m.Moved())
|
||||
}
|
||||
}
|
||||
|
||||
// Between the agent update and the root-files job the wrapper does not know the CA-copy step. A pinned box then has
|
||||
// NO trust problem (the pin works): naming one mails the operator at every box's update. A box with NO working pin
|
||||
// that gets the same refusal still names the problem.
|
||||
func TestRootStepOlderThanTheAgent_IsNotATrustProblem(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
old := errors.New(`pve-ca-copy refused: {"code": "R11", "reason": "unknown mode 'pve-ca-copy'"}`)
|
||||
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
|
||||
b.m.EnsureCopy = func(context.Context) (Copy, error) { return Copy{}, old }
|
||||
before, _ := os.ReadFile(b.cfgPath)
|
||||
if b.m.Ensure(context.Background(), "start") {
|
||||
t.Fatal("the trust changed with no copy")
|
||||
}
|
||||
if st := b.m.Status(); st.Problem != "" || st.Mode != "fingerprint" {
|
||||
t.Fatalf("a pinned box with older root files reports a trust problem: %+v", st)
|
||||
}
|
||||
if after, _ := os.ReadFile(b.cfgPath); string(after) != string(before) {
|
||||
t.Fatal("the config was written")
|
||||
}
|
||||
|
||||
// the same refusal on a box that is NOT pinned is still a problem: nothing else holds its trust up
|
||||
b2 := newBox(t, srv, proxmox.TLSConfig{}, nil)
|
||||
b2.m.EnsureCopy = func(context.Context) (Copy, error) { return Copy{}, old }
|
||||
b2.m.Ensure(context.Background(), "start")
|
||||
if st := b2.m.Status(); st.Problem != ProblemCopyUnavailable {
|
||||
t.Fatalf("an unpinned box with no copy step must name the problem: %+v", st)
|
||||
}
|
||||
|
||||
// and another refusal on a pinned box is still named
|
||||
b3 := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
|
||||
b3.m.EnsureCopy = func(context.Context) (Copy, error) { return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3"}`) }
|
||||
b3.m.Ensure(context.Background(), "start")
|
||||
if st := b3.m.Status(); st.Problem != ProblemCopyUnavailable {
|
||||
t.Fatalf("a real refusal lost its name: %+v", st)
|
||||
}
|
||||
|
||||
// a refusal with ANOTHER code whose text happens to hold the words is still named: the code decides
|
||||
b4 := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
|
||||
b4.m.EnsureCopy = func(context.Context) (Copy, error) {
|
||||
return Copy{}, errors.New(`pve-ca-copy refused: {"code": "T3", "reason": "the source has an unknown mode"}`)
|
||||
}
|
||||
b4.m.Ensure(context.Background(), "start")
|
||||
if st := b4.m.Status(); st.Problem != ProblemCopyUnavailable {
|
||||
t.Fatalf("a refusal with another code was read as older root files: %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
// The quiet branch must never hide a box whose pin does NOT work: a run started by a failed call, or a run after a
|
||||
// failed call was recorded, names the problem even when the root files are older than the agent.
|
||||
func TestRootStepOlderThanTheAgent_NeverHidesAFailedPin(t *testing.T) {
|
||||
p := newPKI(t, "PVE CA")
|
||||
leaf := p.leaf()
|
||||
srv := newPVE(t, leaf)
|
||||
old := errors.New(`pve-ca-copy refused: {"code": "R11", "reason": "unknown mode 'pve-ca-copy'"}`)
|
||||
|
||||
// (1) the run was started by a failed call
|
||||
b := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
|
||||
b.m.EnsureCopy = func(context.Context) (Copy, error) { return Copy{}, old }
|
||||
b.m.Ensure(context.Background(), proxmox.TrustPinnedCertChanged)
|
||||
if st := b.m.Status(); st.Problem == "" {
|
||||
t.Fatalf("a run started by a failed pin reports no problem: %+v", st)
|
||||
}
|
||||
|
||||
// (2) a failed call was recorded, then the timer runs
|
||||
b2 := newBox(t, srv, proxmox.TLSConfig{Fingerprint: fp(leaf)}, nil)
|
||||
b2.m.EnsureCopy = func(context.Context) (Copy, error) { return Copy{}, old }
|
||||
b2.m.mu.Lock()
|
||||
b2.m.status.Problem = proxmox.TrustPinnedCertChanged
|
||||
b2.m.mu.Unlock()
|
||||
b2.m.Ensure(context.Background(), "timer")
|
||||
if st := b2.m.Status(); st.Problem == "" {
|
||||
t.Fatalf("the timer cleared a recorded pin failure: %+v", st)
|
||||
}
|
||||
}
|
||||
@@ -18,8 +18,14 @@ type Manager struct {
|
||||
logger *slog.Logger
|
||||
now func() time.Time // injectable for tests
|
||||
sleep func(context.Context, time.Duration)
|
||||
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
|
||||
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
|
||||
onCommitted func(context.Context)
|
||||
}
|
||||
|
||||
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
|
||||
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
|
||||
|
||||
// ManagerConfig wires the commit manager.
|
||||
type ManagerConfig struct {
|
||||
StateDir string
|
||||
@@ -119,4 +125,14 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
|
||||
return
|
||||
}
|
||||
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
|
||||
if m.onCommitted != nil {
|
||||
m.onCommitted(ctx)
|
||||
}
|
||||
// R-1368 part 2: the update ended healthy — the folder keeps the running version's binary and no other
|
||||
// (prune.go). A refusal or a failed removal is logged and changes nothing about the commit.
|
||||
if removed, err := pruneStaged(m.stateDir, m.runningVersion, m.logger); err != nil {
|
||||
m.logger.Warn("selfupdate: old staged binaries NOT removed after the commit", "err", err)
|
||||
} else if len(removed) > 0 {
|
||||
m.logger.Info("selfupdate: old staged binaries removed after the commit", "count", len(removed), "kept", "felhom-agent-"+m.runningVersion)
|
||||
}
|
||||
}
|
||||
@@ -103,6 +103,9 @@ func (e *Executor) Execute(ctx context.Context, op string, params json.RawMessag
|
||||
}
|
||||
e.logger.Debug("agent_update: op invariants passed (semver + sha format)", "version", p.Version)
|
||||
|
||||
// The staged file is this call's from the download to the hand-off: a prune (prune.go) waits for it.
|
||||
stagingMu.Lock()
|
||||
defer stagingMu.Unlock()
|
||||
dir := stagingDir(e.stateDir)
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
return fmt.Errorf("agent_update: staging dir: %w", err)
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// The self-update folder keeps what it needs and no more (R-1368 part 2; space list 2026-11-10, place H12).
|
||||
//
|
||||
// WHAT WAS WRONG. Execute downloads each new agent to <StateDir>/selfupdate/felhom-agent-<version> and removes that
|
||||
// file only when the download or its checksum fails. A good update left it there for ever: measured 2026-11-10, 49 /
|
||||
// 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1 — one for each version the box ever got.
|
||||
//
|
||||
// WHAT THE FOLDER IS NEEDED FOR. The root wrapper (felhom-os-apply, mode agent_update) reads the staged file ONCE,
|
||||
// copies it to its own root-owned folder and flips from that copy. After that hand-off nothing reads the staged file:
|
||||
// - the rollback (felhom-selfupdate-guarded `rollback`) needs two files and neither is a staged binary:
|
||||
// `pending.json` in this folder, and `/usr/local/bin/felhom-agent.prev` outside it;
|
||||
// - the commit needs `pending.json` only.
|
||||
//
|
||||
// So a staged binary is dead weight as soon as its update ended. Pinned by TestPrune_RollbackReadsNoStagedBinary (the
|
||||
// wrapper's text) and TestPrune_RealRollbackStillWorksAfterNUpdates (the wrapper's rollback verb, run for real).
|
||||
//
|
||||
// THE RULE. After an update ended HEALTHY (the wrapper's `commit` returned success), remove every staged binary but
|
||||
// the one of the running version. Nothing is removed:
|
||||
// - while a pending marker exists or cannot be read — an update is in flight, and that is the only time a rollback
|
||||
// can fire;
|
||||
// - after a rollback, or at a plain start — only the commit path calls this, so the binary of a version that failed
|
||||
// stays for the operator to look at until the next good update;
|
||||
// - that is not a regular file with exactly the name Execute gives a staged binary. `pending.json`, a folder, a
|
||||
// symlink, and any name this code does not know are never touched.
|
||||
//
|
||||
// NOT REACHED BY THIS: `/usr/local/bin/felhom-agent.bak-*` (space list H13). Those files are root-owned, in a folder
|
||||
// the agent cannot write, and no code of the agent makes them (a person's copy before a hand deploy). R-1368 part 4.
|
||||
|
||||
// stagedNameRe is the exact name Execute gives a staged binary: "felhom-agent-" + a version that versionRe accepts.
|
||||
// Kept in step with versionRe by TestPrune_NameShapeFollowsTheVersionRule.
|
||||
var stagedNameRe = regexp.MustCompile(`^felhom-agent-([0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?)$`)
|
||||
|
||||
// stagingMu makes a download and a prune take turns. Without it a prune that runs while Execute downloads the NEXT
|
||||
// version would remove that half-written file (its name is a staged binary's and it is not the running version).
|
||||
var stagingMu sync.Mutex
|
||||
|
||||
// pruneStaged removes the staged binaries the folder no longer needs and returns the names it removed, sorted. It
|
||||
// refuses (removes nothing, returns an error) while an update is in flight. A file it cannot remove is logged and
|
||||
// skipped: a failed clean-up must never fail the commit that came before it.
|
||||
func pruneStaged(stateDir, runningVersion string, logger *slog.Logger) ([]string, error) {
|
||||
logger = orDefaultLogger(logger)
|
||||
stagingMu.Lock()
|
||||
defer stagingMu.Unlock()
|
||||
|
||||
if !versionRe.MatchString(runningVersion) {
|
||||
return nil, fmt.Errorf("selfupdate prune: refusing — running version %q is not a version this code knows; it could not name the file to keep", runningVersion)
|
||||
}
|
||||
pm, err := readPending(stateDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("selfupdate prune: refusing — the pending marker cannot be read: %w", err)
|
||||
}
|
||||
if pm != nil {
|
||||
return nil, fmt.Errorf("selfupdate prune: refusing — an update to %s is pending (a rollback may still fire)", pm.NewVersion)
|
||||
}
|
||||
dir := stagingDir(stateDir)
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, fmt.Errorf("selfupdate prune: read %s: %w", dir, err)
|
||||
}
|
||||
keep := "felhom-agent-" + runningVersion
|
||||
var removed []string
|
||||
for _, ent := range entries {
|
||||
name := ent.Name()
|
||||
if name == keep || !stagedNameRe.MatchString(name) {
|
||||
continue // the running version's binary, pending.json, or a name this code does not know
|
||||
}
|
||||
if !ent.Type().IsRegular() {
|
||||
logger.Warn("selfupdate prune: a staged-binary name that is not a regular file — left alone", "name", name, "type", ent.Type().String())
|
||||
continue
|
||||
}
|
||||
// name came from ReadDir and matched stagedNameRe, so it holds no path separator: this is one exact file.
|
||||
if err := os.Remove(filepath.Join(dir, name)); err != nil {
|
||||
logger.Warn("selfupdate prune: cannot remove an old staged binary — left for the next good update", "name", name, "err", err)
|
||||
continue
|
||||
}
|
||||
logger.Debug("selfupdate prune: removed an old staged binary", "name", name)
|
||||
removed = append(removed, name)
|
||||
}
|
||||
sort.Strings(removed)
|
||||
logger.Info("selfupdate prune: the self-update folder holds what it needs", "kept", keep, "removed", len(removed))
|
||||
return removed, nil
|
||||
}
|
||||
@@ -0,0 +1,373 @@
|
||||
package selfupdate
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1368 part 2 — the self-update folder keeps what it needs and no more (prune.go).
|
||||
//
|
||||
// RED-PROOFS (each observed 2026-11-10, then restored):
|
||||
// - drop the pruneStaged call from MaybeCommit → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with
|
||||
// "after update 1 (0.170.0) the folder holds [… felhom-agent-0.169.0 felhom-agent-0.170.0 …]", and
|
||||
// TestPrune_RealRollbackStillWorksAfterNUpdates fails too;
|
||||
// - drop the pending-marker refusal from pruneStaged → TestPrune_RefusesWhileAnUpdateIsPending fails with
|
||||
// "removed [...] while an update was pending";
|
||||
// - drop the IsRegular check → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with "a symlink with a
|
||||
// staged-binary name was removed".
|
||||
|
||||
// committingWrapper is the guarded wrapper's `commit` as far as this package can see it: it clears the pending marker.
|
||||
type committingWrapper struct {
|
||||
stateDir string
|
||||
calls int
|
||||
}
|
||||
|
||||
func (w *committingWrapper) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
|
||||
if len(args) == 1 && args[0] == "commit" {
|
||||
w.calls++
|
||||
if err := os.Remove(pendingPath(w.stateDir)); err != nil {
|
||||
return nil, []byte(err.Error()), err
|
||||
}
|
||||
}
|
||||
return []byte("ok"), nil, nil
|
||||
}
|
||||
|
||||
func stage(t *testing.T, stateDir, version string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(stagingDir(stateDir), 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(stagingDir(stateDir), "felhom-agent-"+version), []byte("binary "+version), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func folder(t *testing.T, stateDir string) []string {
|
||||
t.Helper()
|
||||
ents, err := os.ReadDir(stagingDir(stateDir))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out []string
|
||||
for _, e := range ents {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// The consequence: after N good updates the folder holds ONE staged binary — the running version's — and every file
|
||||
// that is not a staged binary is still there.
|
||||
func TestPrune_AfterNUpdatesTheFolderHoldsOneBinary(t *testing.T) {
|
||||
stateDir := t.TempDir()
|
||||
dir := stagingDir(stateDir)
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Bystanders: names this code does not know, and two things that carry a staged binary's NAME but are not files.
|
||||
outside := filepath.Join(t.TempDir(), "precious")
|
||||
if err := os.WriteFile(outside, []byte("not ours"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, n := range []string{"notes.txt", "felhom-agent-0.1.0.partial", "felhom-agent.bak-0.139.0"} {
|
||||
if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.Mkdir(filepath.Join(dir, "felhom-agent-9.9.9"), 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Symlink(outside, filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
bystanders := []string{"felhom-agent-0.1.0.partial", "felhom-agent-8.8.8", "felhom-agent-9.9.9", "felhom-agent.bak-0.139.0", "notes.txt"}
|
||||
|
||||
versions := []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1", "0.173.0-rc.1", "0.173.0"}
|
||||
prev := "0.169.0"
|
||||
stage(t, stateDir, prev) // what an old agent left behind
|
||||
for i, v := range versions {
|
||||
stage(t, stateDir, v) // Execute downloaded it
|
||||
writePending(t, stateDir, PendingMarker{OldVersion: prev, NewVersion: v, SHA256: "ab", AppliedAt: "t"}) // the wrapper flipped
|
||||
wrap := &committingWrapper{stateDir: stateDir}
|
||||
m := newMgr(t, stateDir, v, wrap, nil)
|
||||
m.MaybeCommit(context.Background())
|
||||
if wrap.calls != 1 {
|
||||
t.Fatalf("update %d (%s): commit calls = %d, want 1", i+1, v, wrap.calls)
|
||||
}
|
||||
want := append([]string{"felhom-agent-" + v}, bystanders...)
|
||||
sort.Strings(want)
|
||||
got := folder(t, stateDir)
|
||||
if strings.Join(got, " ") != strings.Join(want, " ") {
|
||||
if _, err := os.Lstat(filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
|
||||
t.Fatalf("a symlink with a staged-binary name was removed: %v", err)
|
||||
}
|
||||
t.Fatalf("after update %d (%s) the folder holds %v, want %v", i+1, v, got, want)
|
||||
}
|
||||
prev = v
|
||||
}
|
||||
if b, err := os.ReadFile(outside); err != nil || string(b) != "not ours" {
|
||||
t.Fatalf("the file behind the symlink changed: %q %v", b, err)
|
||||
}
|
||||
}
|
||||
|
||||
// While an update is pending — the only time a rollback can fire — nothing is removed.
|
||||
func TestPrune_RefusesWhileAnUpdateIsPending(t *testing.T) {
|
||||
stateDir := t.TempDir()
|
||||
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
|
||||
stage(t, stateDir, v)
|
||||
}
|
||||
writePending(t, stateDir, PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0", SHA256: "ab", AppliedAt: "t"})
|
||||
before := folder(t, stateDir)
|
||||
|
||||
removed, err := pruneStaged(stateDir, "0.172.0", nil)
|
||||
if err == nil || len(removed) != 0 {
|
||||
t.Fatalf("removed %v while an update was pending (err=%v) — want a refusal", removed, err)
|
||||
}
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||
t.Fatalf("removed %v while an update was pending; folder now %v", before, got)
|
||||
}
|
||||
|
||||
// The same through the commit path: a `commit` that reports success but leaves the marker (fakeWrapper does) must
|
||||
// not lead to a prune.
|
||||
m := newMgr(t, stateDir, "0.172.0", &fakeWrapper{}, nil)
|
||||
m.MaybeCommit(context.Background())
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||
t.Fatalf("the commit path pruned with the marker still there; folder now %v", got)
|
||||
}
|
||||
|
||||
// A marker nobody can read is a refusal too.
|
||||
if err := os.WriteFile(pendingPath(stateDir), []byte("{not json"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if removed, err := pruneStaged(stateDir, "0.172.0", nil); err == nil || len(removed) != 0 {
|
||||
t.Fatalf("an unreadable marker must refuse; removed %v err=%v", removed, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Only a commit prunes: a plain start, a version mismatch and a failed commit remove nothing — so the binary of a
|
||||
// version that was rolled back stays for the operator.
|
||||
func TestPrune_OnlyAHealthyCommitPrunes(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
pending *PendingMarker
|
||||
running string
|
||||
wrap WrapperRunner
|
||||
}{
|
||||
{"plain start, no marker (also the state after a rollback)", nil, "0.171.0", &fakeWrapper{}},
|
||||
{"marker names another version", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.171.0", &fakeWrapper{}},
|
||||
{"commit fails", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.172.0", &fakeWrapper{err: fmt.Errorf("sudo: no")}},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
stateDir := t.TempDir()
|
||||
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
|
||||
stage(t, stateDir, v)
|
||||
}
|
||||
if c.pending != nil {
|
||||
writePending(t, stateDir, *c.pending)
|
||||
}
|
||||
before := folder(t, stateDir)
|
||||
newMgr(t, stateDir, c.running, c.wrap, nil).MaybeCommit(context.Background())
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||
t.Fatalf("folder changed without a healthy commit: %v → %v", before, got)
|
||||
}
|
||||
})
|
||||
}
|
||||
// A running version this code cannot name a file for: refuse, do not guess.
|
||||
stateDir := t.TempDir()
|
||||
stage(t, stateDir, "0.170.0")
|
||||
if removed, err := pruneStaged(stateDir, "dev", nil); err == nil || len(removed) != 0 {
|
||||
t.Fatalf("running version \"dev\" must refuse; removed %v err=%v", removed, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The name rule and the version rule say the same thing: every name Execute can write is a name the prune knows, and
|
||||
// nothing else is.
|
||||
func TestPrune_NameShapeFollowsTheVersionRule(t *testing.T) {
|
||||
for _, v := range []string{"0.70.1", "0.172.1", "10.0.0", "0.70.2-crash", "0.173.0-rc.1"} {
|
||||
if !versionRe.MatchString(v) {
|
||||
t.Fatalf("test input %q is not a version versionRe accepts", v)
|
||||
}
|
||||
if !stagedNameRe.MatchString("felhom-agent-" + v) {
|
||||
t.Errorf("Execute can stage felhom-agent-%s but the prune does not know that name", v)
|
||||
}
|
||||
}
|
||||
for _, n := range []string{"pending.json", "felhom-agent", "felhom-agent-", "felhom-agent-1.2", "felhom-agent-0.1.0.partial",
|
||||
"felhom-agent.bak-0.139.0", "felhom-agent.prev", "felhom-agent-0.1.0/x", "../felhom-agent-0.1.0", "felhom-agent-0.1.0 ", "xfelhom-agent-0.1.0"} {
|
||||
if stagedNameRe.MatchString(n) {
|
||||
t.Errorf("the prune would treat %q as a staged binary", n)
|
||||
}
|
||||
}
|
||||
m := stagedNameRe.FindStringSubmatch("felhom-agent-0.173.0-rc.1")
|
||||
if m == nil || !versionRe.MatchString(m[1]) {
|
||||
t.Fatalf("the version part of a staged name must satisfy versionRe; got %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func wrapperText(t *testing.T) string {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-selfupdate-guarded"))
|
||||
if err != nil {
|
||||
t.Fatalf("the guarded wrapper is the other half of this proof and must be readable: %v", err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// The stated dependency, pinned: the wrapper's `rollback` and `commit` read the pending marker and `.prev`, and no
|
||||
// staged binary. If either verb starts to read the staging folder, this fails and prune.go's rule must be looked at.
|
||||
func TestPrune_RollbackReadsNoStagedBinary(t *testing.T) {
|
||||
s := wrapperText(t)
|
||||
for _, want := range []string{"\nBIN=/usr/local/bin/felhom-agent\n", "\nPREV=$BIN.prev\n", "\nSTAGING=/var/lib/felhom-agent/selfupdate\n", "\nPENDING=$STAGING/pending.json\n"} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Fatalf("the wrapper no longer holds the line %q — the folder's rule (prune.go) was written against it", strings.TrimSpace(want))
|
||||
}
|
||||
}
|
||||
for _, verb := range []string{"commit", "rollback"} {
|
||||
i := strings.Index(s, "\n"+verb+")\n")
|
||||
if i < 0 {
|
||||
t.Fatalf("no %s) block in the wrapper", verb)
|
||||
}
|
||||
block := s[i:]
|
||||
j := strings.Index(block, "\n ;;\n")
|
||||
if j < 0 {
|
||||
t.Fatalf("no end of the %s) block", verb)
|
||||
}
|
||||
block = block[:j]
|
||||
if !strings.Contains(block, "$PENDING") {
|
||||
t.Fatalf("%s) does not read $PENDING any more", verb)
|
||||
}
|
||||
code := regexp.MustCompile(`(?m)^\s*#.*$`).ReplaceAllString(block, "")
|
||||
for _, bad := range []string{"STAGING", "staged", "felhom-agent-"} {
|
||||
if strings.Contains(code, bad) {
|
||||
t.Fatalf("%s) now mentions %q — it may read a staged binary, and the prune removes those", verb, bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
if stagedNameRe.MatchString("pending.json") || stagedNameRe.MatchString("felhom-agent.prev") {
|
||||
t.Fatal("the prune would remove a file the rollback needs")
|
||||
}
|
||||
}
|
||||
|
||||
// The consequence, with the wrapper's own code: after N good updates (each committed by the REAL `commit` verb and
|
||||
// pruned), update N+1 goes bad and the REAL `rollback` verb puts the previous binary back. The wrapper runs from a copy
|
||||
// whose four fixed paths point into a temp folder; systemctl, logger and chown are stubs.
|
||||
func TestPrune_RealRollbackStillWorksAfterNUpdates(t *testing.T) {
|
||||
sh, err := exec.LookPath("sh")
|
||||
if err != nil {
|
||||
t.Skip("no sh on this machine")
|
||||
}
|
||||
root := t.TempDir()
|
||||
stateDir := filepath.Join(root, "state")
|
||||
binDir := filepath.Join(root, "bin")
|
||||
stubs := filepath.Join(root, "stubs")
|
||||
for _, d := range []string{stagingDir(stateDir), binDir, stubs, filepath.Join(root, "rootstaging")} {
|
||||
if err := os.MkdirAll(d, 0o750); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
live := filepath.Join(binDir, "felhom-agent")
|
||||
s := wrapperText(t)
|
||||
for old, repl := range map[string]string{
|
||||
"\nBIN=/usr/local/bin/felhom-agent\n": "\nBIN=" + live + "\n",
|
||||
"\nSTAGING=/var/lib/felhom-agent/selfupdate\n": "\nSTAGING=" + stagingDir(stateDir) + "\n",
|
||||
"\nROOT_STAGING=/var/lib/felhom-os-apply/agent-update\n": "\nROOT_STAGING=" + filepath.Join(root, "rootstaging") + "\n",
|
||||
} {
|
||||
if strings.Count(s, old) != 1 {
|
||||
t.Fatalf("the wrapper's path line %q is not there exactly once — this test cannot sandbox it", strings.TrimSpace(old))
|
||||
}
|
||||
s = strings.Replace(s, old, repl, 1)
|
||||
}
|
||||
// Safety: no assignment in the copy may still point at a real system path (PREV and PENDING are built from BIN and
|
||||
// STAGING, so they follow).
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
if regexp.MustCompile(`^[A-Z_]+=/`).MatchString(line) && !strings.Contains(line, root) {
|
||||
t.Fatalf("the sandboxed wrapper still holds a real path: %q", line)
|
||||
}
|
||||
}
|
||||
script := filepath.Join(root, "wrapper.sh")
|
||||
if err := os.WriteFile(script, []byte(s), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stubLog := filepath.Join(root, "stub.log")
|
||||
for _, n := range []string{"systemctl", "logger", "chown", "systemd-run"} {
|
||||
body := "#!/bin/sh\necho \"" + n + " $*\" >> '" + stubLog + "'\nexit 0\n"
|
||||
if err := os.WriteFile(filepath.Join(stubs, n), []byte(body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
run := func(verb string) (string, error) {
|
||||
cmd := exec.Command(sh, script, verb)
|
||||
cmd.Env = []string{"PATH=" + stubs + ":/usr/bin:/bin"}
|
||||
out, err := cmd.CombinedOutput()
|
||||
return string(out), err
|
||||
}
|
||||
// applied is what the wrapper's `apply` leaves behind (it needs root, so the test writes the same four things).
|
||||
applied := func(oldV, newV string) {
|
||||
cur, err := os.ReadFile(live)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(live+".prev", cur, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(live, []byte("binary "+newV), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writePending(t, stateDir, PendingMarker{OldVersion: oldV, NewVersion: newV, SHA256: "ab", AppliedAt: "t"})
|
||||
}
|
||||
if err := os.WriteFile(live, []byte("binary 0.169.0"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
prev := "0.169.0"
|
||||
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1"} {
|
||||
stage(t, stateDir, v)
|
||||
applied(prev, v)
|
||||
m := newMgr(t, stateDir, v, runnerFunc(func(args ...string) ([]byte, []byte, error) {
|
||||
out, err := run(args[0])
|
||||
return []byte(out), nil, err
|
||||
}), nil)
|
||||
m.MaybeCommit(context.Background()) // the REAL commit verb, then the prune
|
||||
if got := folder(t, stateDir); len(got) != 1 || got[0] != "felhom-agent-"+v {
|
||||
t.Fatalf("after the good update to %s the folder holds %v", v, got)
|
||||
}
|
||||
prev = v
|
||||
}
|
||||
|
||||
// Update N+1 is applied and goes bad: systemd's OnFailure runs the wrapper's rollback.
|
||||
stage(t, stateDir, "0.173.0")
|
||||
applied("0.172.1", "0.173.0")
|
||||
out, err := run("rollback")
|
||||
if err != nil {
|
||||
t.Fatalf("the real rollback verb failed after the prunes: %v\n%s", err, out)
|
||||
}
|
||||
if b, _ := os.ReadFile(live); string(b) != "binary 0.172.1" {
|
||||
t.Fatalf("after the rollback the live binary is %q, want the previous version's", b)
|
||||
}
|
||||
if p, _ := readPending(stateDir); p != nil {
|
||||
t.Fatal("the rollback left the pending marker")
|
||||
}
|
||||
if b, _ := os.ReadFile(stubLog); !bytes.Contains(b, []byte("systemctl restart felhom-agent.service")) {
|
||||
t.Fatalf("the rollback did not restart the agent; stub log:\n%s", b)
|
||||
}
|
||||
// The rolled-back agent starts: no marker, so no commit and no prune — the bad version's binary stays to be read.
|
||||
newMgr(t, stateDir, "0.172.1", &fakeWrapper{}, nil).MaybeCommit(context.Background())
|
||||
if got := folder(t, stateDir); strings.Join(got, " ") != "felhom-agent-0.172.1 felhom-agent-0.173.0" {
|
||||
t.Fatalf("after the rollback the folder holds %v, want the running version and the one that failed", got)
|
||||
}
|
||||
}
|
||||
|
||||
// runnerFunc adapts a function to WrapperRunner; it gets the verb arguments only.
|
||||
type runnerFunc func(args ...string) ([]byte, []byte, error)
|
||||
|
||||
func (f runnerFunc) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
|
||||
return f(args...)
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package signedjobs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// outcomeQueue is a jobsQueue that also has the optional half (OutcomeSource), as *hub.Client does.
|
||||
type outcomeQueue struct {
|
||||
jobsQueue
|
||||
omu sync.Mutex
|
||||
outcomes map[string]hub.JobOutcome
|
||||
}
|
||||
|
||||
func (q *outcomeQueue) CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error {
|
||||
q.omu.Lock()
|
||||
if q.outcomes == nil {
|
||||
q.outcomes = map[string]hub.JobOutcome{}
|
||||
}
|
||||
q.outcomes[jobID] = o
|
||||
q.omu.Unlock()
|
||||
return q.jobsQueue.CompleteJob(ctx, jobID)
|
||||
}
|
||||
|
||||
// R-1383 (D78): a signed job that is refused or fails was cleared from the hub's queue exactly like one that ran —
|
||||
// „an agent update never arrives” left one journal line on the box. The runner now says how each job ended, through
|
||||
// the REAL gate and verifier: the stored effect is the outcome the hub receives with the clearing call.
|
||||
// The two clock reasons are the ones the ruling is about: a box with a wrong clock refuses every job.
|
||||
// COMPANION RED-PROOFS (observed, felhom.eu audits/night-build-2026-11-14/B/redproof-agent-job-outcome.txt).
|
||||
func TestRunner_TellsTheHubHowEachJobEnded(t *testing.T) {
|
||||
s := newTestSigner(t)
|
||||
_, _, exec := newRealGateRunner(t, s)
|
||||
base, _, _ := newRealGateRunner(t, s)
|
||||
src := &outcomeQueue{}
|
||||
r := NewRunner(src, base.gate, exec, testHost, quiet())
|
||||
now := time.Now().UTC()
|
||||
|
||||
src.add(mintJob(t, s, "ok", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
|
||||
src.add(mintJob(t, s, "expired", testHost, "", "ops-1", wipeParamsJSON, now.Add(-3*time.Hour), now.Add(-2*time.Hour)))
|
||||
src.add(mintJob(t, s, "early", testHost, "", "ops-1", wipeParamsJSON, now.Add(2*time.Hour), now.Add(3*time.Hour)))
|
||||
src.add(mintJob(t, s, "other-box", "some-other-host", "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
|
||||
src.add(corruptSig(t, mintJob(t, s, "forged", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))))
|
||||
src.add(hub.JobWire{JobID: "garbage", BlobB64: "!!!not-base64!!!"})
|
||||
if _, err := r.RunOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// and a job that is accepted and whose run fails
|
||||
exec.err = errors.New("the wrapper refused")
|
||||
src.add(mintJob(t, s, "run-fails", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)))
|
||||
if _, err := r.RunOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]hub.JobOutcome{
|
||||
"ok": {Outcome: "done", Op: "storage_wipe"},
|
||||
"expired": {Outcome: "rejected", Reason: "expired", Op: "storage_wipe"},
|
||||
"early": {Outcome: "rejected", Reason: "not_yet_valid", Op: "storage_wipe"},
|
||||
"other-box": {Outcome: "rejected", Reason: "target", Op: "storage_wipe"},
|
||||
"forged": {Outcome: "rejected", Reason: "malformed", Op: "storage_wipe"}, // corruptSig breaks the armour itself
|
||||
"garbage": {Outcome: "malformed", Reason: "malformed"},
|
||||
"run-fails": {Outcome: "failed", Reason: "execution_failed", Op: "storage_wipe"},
|
||||
}
|
||||
for id, w := range want {
|
||||
if got, ok := src.outcomes[id]; !ok || got != w {
|
||||
t.Errorf("job %q: the hub was told %+v (told=%v), want %+v", id, got, ok, w)
|
||||
}
|
||||
}
|
||||
if len(src.outcomes) != len(want) {
|
||||
t.Errorf("outcomes for %d jobs, want %d: %+v", len(src.outcomes), len(want), src.outcomes)
|
||||
}
|
||||
if exec.count() != 2 {
|
||||
t.Errorf("executor ran %d times, want 2 (ok and run-fails only)", exec.count())
|
||||
}
|
||||
}
|
||||
|
||||
// Every code reasonCode can return is on the closed list the hub holds a sentence for.
|
||||
func TestReasonCode_ClosedList(t *testing.T) {
|
||||
known := map[string]bool{}
|
||||
for _, c := range ReasonCodes {
|
||||
known[c] = true
|
||||
}
|
||||
for _, d := range []reconcile.Decision{
|
||||
{}, {Reason: reconcile.ReasonRejected, Err: errors.New("something new")},
|
||||
{Reason: reconcile.ReasonRoleDenied}, {Reason: reconcile.ReasonBindingMismatch}, {Reason: reconcile.ReasonPendingSignature},
|
||||
} {
|
||||
if c := reasonCode(d); !known[c] {
|
||||
t.Errorf("reasonCode(%+v) = %q, not on ReasonCodes", d, c)
|
||||
}
|
||||
}
|
||||
if reasonCode(reconcile.Decision{Reason: reconcile.ReasonRejected, Err: errors.New("x")}) != "other" {
|
||||
t.Error("an unknown refusal must be told as „other”, never as the error's own text")
|
||||
}
|
||||
if !known["execution_failed"] || !known["malformed"] {
|
||||
t.Error("the two codes the runner sets itself are not on the list")
|
||||
}
|
||||
}
|
||||
@@ -35,6 +35,42 @@ type JobSource interface {
|
||||
CompleteJob(ctx context.Context, jobID string) error
|
||||
}
|
||||
|
||||
// OutcomeSource is the optional half of a JobSource: it clears a job AND tells the hub how it ended (R-1383, D78).
|
||||
// *hub.Client has it. A source without it is cleared as before.
|
||||
type OutcomeSource interface {
|
||||
CompleteJobWithOutcome(ctx context.Context, jobID string, o hub.JobOutcome) error
|
||||
}
|
||||
|
||||
// ReasonCodes is the CLOSED list of reasons the agent sends with a job that did not run to a good end. The hub holds
|
||||
// one operator sentence per code (felhom.eu hub/internal/api, signedJobReasons); a code the hub does not know is
|
||||
// told as „other”. Pinned by TestReasonCode_ClosedList.
|
||||
var ReasonCodes = []string{"expired", "not_yet_valid", "window_too_long", "replay", "bad_signature", "unknown_signer",
|
||||
"target", "namespace", "malformed", "role_denied", "binding_mismatch", "execution_failed", "other"}
|
||||
|
||||
// reasonCode names why the gate refused: the authz error first (the two clock reasons matter most — a box with a
|
||||
// wrong clock refuses EVERY job), then the gate's own reason.
|
||||
func reasonCode(dec reconcile.Decision) string {
|
||||
for _, m := range []struct {
|
||||
err error
|
||||
code string
|
||||
}{
|
||||
{authz.ErrExpired, "expired"}, {authz.ErrNotYetValid, "not_yet_valid"}, {authz.ErrWindowTooLong, "window_too_long"},
|
||||
{authz.ErrReplay, "replay"}, {authz.ErrBadSignature, "bad_signature"}, {authz.ErrUnknownSigner, "unknown_signer"},
|
||||
{authz.ErrTarget, "target"}, {authz.ErrNamespace, "namespace"}, {authz.ErrMalformed, "malformed"},
|
||||
} {
|
||||
if errorsIs(dec.Err, m.err) {
|
||||
return m.code
|
||||
}
|
||||
}
|
||||
switch dec.Reason {
|
||||
case reconcile.ReasonRoleDenied:
|
||||
return "role_denied"
|
||||
case reconcile.ReasonBindingMismatch:
|
||||
return "binding_mismatch"
|
||||
}
|
||||
return "other"
|
||||
}
|
||||
|
||||
// Authorizer is the reversibility gate. Satisfied by *reconcile.Gate.
|
||||
type Authorizer interface {
|
||||
Authorize(intent reconcile.Intent, signed *reconcile.SignedOp) reconcile.Decision
|
||||
@@ -141,7 +177,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
// A malformed/forged envelope is permanently bad — clear it loudly (a queued garbage job
|
||||
// must not wedge the queue). No nonce was consumed (we never reached Verify).
|
||||
r.logger.Error("signedjobs: malformed job envelope — clearing", "job", j.JobID)
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "malformed", Reason: "malformed"})
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -166,7 +202,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
// it as a security event (a compromised hub queuing a forged op lands here).
|
||||
r.logger.Warn("signedjobs: REJECTED signed op — executor not called",
|
||||
"job", j.JobID, "op", ob.Op, "reason", dec.Reason, "err", errStr(dec.Err))
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "rejected", Reason: reasonCode(dec), Op: ob.Op})
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -177,7 +213,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
switch {
|
||||
case err == nil:
|
||||
r.logger.Warn("signedjobs: signed op COMPLETED", "job", j.JobID, "op", ob.Op)
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "done", Op: ob.Op})
|
||||
return true
|
||||
case errorsIs(err, ErrNoExecutor):
|
||||
// No executor for this op class in this build — leave it queued for the owning slice.
|
||||
@@ -189,7 +225,7 @@ func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool {
|
||||
// nonce to retry) and log the failure loudly.
|
||||
r.logger.Error("signedjobs: signed op execution FAILED (nonce spent — clearing)",
|
||||
"job", j.JobID, "op", ob.Op, "err", err)
|
||||
r.complete(ctx, j.JobID)
|
||||
r.complete(ctx, j.JobID, hub.JobOutcome{Outcome: "failed", Reason: "execution_failed", Op: ob.Op})
|
||||
return true
|
||||
}
|
||||
}
|
||||
@@ -215,8 +251,14 @@ func (r *Runner) decode(j hub.JobWire) (*reconcile.SignedOp, authz.OpBlob, bool)
|
||||
return &reconcile.SignedOp{Blob: opBlob, Sig: []byte(env.SigArmored)}, ob, true
|
||||
}
|
||||
|
||||
func (r *Runner) complete(ctx context.Context, jobID string) {
|
||||
if err := r.source.CompleteJob(ctx, jobID); err != nil {
|
||||
func (r *Runner) complete(ctx context.Context, jobID string, o hub.JobOutcome) {
|
||||
var err error
|
||||
if os, ok := r.source.(OutcomeSource); ok {
|
||||
err = os.CompleteJobWithOutcome(ctx, jobID, o) // R-1383: the hub hears HOW the job ended
|
||||
} else {
|
||||
err = r.source.CompleteJob(ctx, jobID)
|
||||
}
|
||||
if err != nil {
|
||||
r.logger.Warn("signedjobs: clearing job failed (will retry next pass)", "job", jobID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package wgtunnel
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1247: the line test's server is the desired block's pbs_tunnel_ip, and ONLY while the tunnel is operational
|
||||
// (registered + unit active) — „no tunnel → no test" rests on this answer.
|
||||
func TestLineTestTargetOnlyWhileOperational(t *testing.T) {
|
||||
m, _, active, _ := testManager(t, &fakeHub{})
|
||||
ctx := context.Background()
|
||||
if _, ok := m.LineTestTarget(); ok {
|
||||
t.Fatal("a target before any Apply")
|
||||
}
|
||||
m.Apply(ctx, false, nil) // registers
|
||||
pub := localPub(t, m)
|
||||
m.Apply(ctx, true, testBlock(pub)) // installs; the unit is not active yet
|
||||
if _, ok := m.LineTestTarget(); ok {
|
||||
t.Fatal("a target while the unit is not active")
|
||||
}
|
||||
*active = true
|
||||
m.Apply(ctx, true, testBlock(pub))
|
||||
ip, ok := m.LineTestTarget()
|
||||
if !ok || ip.String() != "10.77.0.1" {
|
||||
t.Fatalf("operational tunnel: target %v ok=%v, want 10.77.0.1", ip, ok)
|
||||
}
|
||||
m.Apply(ctx, true, nil) // revoked
|
||||
if _, ok := m.LineTestTarget(); ok {
|
||||
t.Fatal("a target after the hub revoked the tunnel")
|
||||
}
|
||||
}
|
||||
@@ -128,6 +128,7 @@ type Manager struct {
|
||||
type convSnapshot struct {
|
||||
wgBlockDesired bool
|
||||
operational bool
|
||||
serverIP netip.Addr // the desired block's pbs_tunnel_ip (R-1247 line test target); invalid when none
|
||||
}
|
||||
|
||||
// TunnelConvergence returns the cached (desired, operational) snapshot without any exec (fast-tick
|
||||
@@ -139,6 +140,17 @@ func (m *Manager) TunnelConvergence() (desired, operational bool) {
|
||||
return m.conv.wgBlockDesired, m.conv.operational
|
||||
}
|
||||
|
||||
// LineTestTarget is the R-1247 line test's server: ep0's tunnel address, ok only while the tunnel is operational
|
||||
// (registered + unit active, as of the last Apply). Cached — no exec.
|
||||
func (m *Manager) LineTestTarget() (netip.Addr, bool) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if !m.conv.operational || !m.conv.serverIP.IsValid() {
|
||||
return netip.Addr{}, false
|
||||
}
|
||||
return m.conv.serverIP, true
|
||||
}
|
||||
|
||||
// NewManager builds a Manager. stateDir is the agent state dir (default /var/lib/felhom-agent —
|
||||
// note the FELHOM_WG sudoers install entry hard-codes the staged path under it).
|
||||
func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, logger *slog.Logger) *Manager {
|
||||
@@ -348,7 +360,13 @@ func (m *Manager) Apply(ctx context.Context, fetched bool, block *hub.WireWiregu
|
||||
operational = m.isActive(ctx)
|
||||
}
|
||||
}
|
||||
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational}
|
||||
var server netip.Addr
|
||||
if desired {
|
||||
if a, err := netip.ParseAddr(block.Endpoint.PBSTunnelIP); err == nil && a.Is4() {
|
||||
server = a
|
||||
}
|
||||
}
|
||||
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational, serverIP: server}
|
||||
}()
|
||||
|
||||
keyExists := false
|
||||
|
||||
@@ -98,7 +98,8 @@ TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
||||
# THE retention number, read from the one file that owns it. A check and the policy it enforces
|
||||
# must read the same number from the same place, or they drift and the drift looks like a defect
|
||||
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
|
||||
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json")
|
||||
RETENTION_FILE = os.environ.get("FELHOM_RETENTION_POLICY") or \
|
||||
os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") # the env: decoy tests only
|
||||
|
||||
|
||||
def retention_kept():
|
||||
@@ -114,6 +115,21 @@ def retention_kept():
|
||||
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
|
||||
return n
|
||||
|
||||
|
||||
def retention_pinned():
|
||||
"""D50 = A: the versions kept besides the newest N — the hub's MinAgent floor and the day-0 agent. Copies of hub
|
||||
state the vouching session writes into the policy file (CI holds no hub secret). Missing key = no pins."""
|
||||
with open(RETENTION_FILE, encoding="utf-8") as fh:
|
||||
pins = json.load(fh).get("pinned") or {}
|
||||
if not isinstance(pins, dict):
|
||||
raise ValueError("pinned must be an object, got %r" % (pins,))
|
||||
out = {}
|
||||
for name, v in pins.items():
|
||||
if not isinstance(v, str) or not re.match(r"^\d+\.\d+\.\d+$", v):
|
||||
raise ValueError("pinned %s must be X.Y.Z, got %r" % (name, v))
|
||||
out[name] = v
|
||||
return out
|
||||
|
||||
tried = []
|
||||
|
||||
|
||||
@@ -211,6 +227,7 @@ def main():
|
||||
all_versions = versions
|
||||
try:
|
||||
keep = retention_kept()
|
||||
pins = retention_pinned()
|
||||
except Exception as e:
|
||||
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
|
||||
|
||||
@@ -218,10 +235,15 @@ def main():
|
||||
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
|
||||
def _key(v):
|
||||
return tuple(int(x) for x in v.split("."))
|
||||
versions = sorted(all_versions, key=_key)[-keep:]
|
||||
window = sorted(all_versions, key=_key)[-keep:]
|
||||
# D50 = A: the pinned versions are asserted wherever they sit. A pin with no tag cannot be installed at all —
|
||||
# a failure, never a skip.
|
||||
unpinned_tags = sorted(("%s=%s" % (k, v)) for k, v in pins.items() if v not in all_versions)
|
||||
versions = sorted(set(window) | set(v for v in pins.values() if v in all_versions), key=_key)
|
||||
dropped = [v for v in all_versions if v not in versions]
|
||||
|
||||
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep))
|
||||
print(" %d released version(s); retention policy keeps the newest %d plus the pinned %s" % (
|
||||
len(all_versions), keep, ", ".join("%s=%s" % kv for kv in sorted(pins.items())) or "(none)"))
|
||||
print(" verifying: %s" % ", ".join(versions))
|
||||
if dropped:
|
||||
# NEVER silent. A bounded check that does not say what it stopped covering is how a
|
||||
@@ -232,6 +254,9 @@ def main():
|
||||
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
|
||||
|
||||
bad = []
|
||||
for pin in unpinned_tags:
|
||||
bad.append((pin, ["the pinned version has NO git tag — nothing can install it (D50 = A)"]))
|
||||
print(" FAIL pinned %s: no git tag v%s" % (pin, pin.split("=")[1]))
|
||||
for v in versions:
|
||||
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
|
||||
|
||||
@@ -33,6 +33,9 @@
|
||||
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
|
||||
# GITEA_BASE / GITEA_OWNER override the defaults.
|
||||
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
|
||||
# GITEA_GIT_TOKEN — the tag push (`git push origin <tag>`) needs it since 2026-11-11: the address of a CC clone
|
||||
# on DooPlex holds no credential, and git takes the token from the secret store through the helper
|
||||
# felhom.eu/scripts/git-credential-sec.sh. Run the release as `sec run GITEA_GIT_TOKEN,... -- scripts/release-agent.sh <version>`.
|
||||
set -euo pipefail
|
||||
|
||||
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
|
||||
|
||||
@@ -1,41 +1,33 @@
|
||||
{
|
||||
"_comment": [
|
||||
"THE retention number for published agent artifacts. One file, read by everything that",
|
||||
"depends on it, because a check and the policy it enforces must read the same number from the",
|
||||
"same place or they drift — and the drift looks like a defect in something else.",
|
||||
"THE retention rule for published agent artifacts. One file, read by everything that depends on it, because a check",
|
||||
"and the policy it enforces must read the same rule from the same place or they drift — and the drift looks like a",
|
||||
"defect in something else (R-287, 2026-08-08/09; again R-1290, 2026-10-25).",
|
||||
"",
|
||||
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent",
|
||||
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag",
|
||||
"still be downloadable. Both rules are individually sensible; together they are impossible.",
|
||||
"CI went red at a commit whose own run had been green the day before, on a true finding that",
|
||||
"no one could act on. The red will return at the next publish unless the two read one number.",
|
||||
"D50 = A (operator ruling 2026-10-25, felhom.eu `09` §3 decision 262): the registry keeps the NEWEST 5 released agent",
|
||||
"versions, PLUS the hub's MinAgent floor, PLUS the version the hub's day-0 artifact manifest names. The operator prunes",
|
||||
"by hand to that rule (on 2026-10-25 he removed 0.159.0–0.160.0 and kept 0.158.1, the floor). CI asserts exactly these",
|
||||
"and never more. Before D50 the number was 10 (the R-267 newest-10 prune, R-287/R-291).",
|
||||
"",
|
||||
"HOW THE NUMBER WAS ARRIVED AT. generic_versions_kept is 10 because that is what the registry",
|
||||
"keeps: the deleter was ESTABLISHED on 2026-08-10 (R-287) — the newest-10 prune of generic packages",
|
||||
"run under R-267 (felhom-agent and felhom-golden generic held exactly 10 afterwards). Until then this",
|
||||
"file called the 10 an observed state with no located ruling; that is superseded (corrected",
|
||||
"2026-10-05, R-291). Container packages are not pruned by that rule.",
|
||||
"THE PINS ARE COPIES OF HUB STATE, because CI cannot read the hub without a secret it must not hold. The session that",
|
||||
"records an agent vouch (day0_agent) or declares a new MinAgent with a floor (hub_min_agent) updates `pinned` in the",
|
||||
"same commit (felhom-build-deploy skill, agent release). A pin that names a version with no tag is a FAILURE, never a",
|
||||
"skip — a pin nobody can install is the defect the pin exists to catch.",
|
||||
"",
|
||||
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
|
||||
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
|
||||
"operator should confirm or replace the number — at which point this file changes and both",
|
||||
"readers follow it in the same commit.",
|
||||
"SO THIS FILE IS A FLOOR, NOT A LICENCE: it says what CI may assume is still downloadable. It does not authorise",
|
||||
"deleting anything else.",
|
||||
"",
|
||||
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled",
|
||||
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an",
|
||||
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
|
||||
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
|
||||
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
|
||||
"the gate does not have today. Filed as the follow-up in R-287.",
|
||||
"",
|
||||
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
|
||||
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
|
||||
"version at all — a strictly worse act than an un-downloadable binary."
|
||||
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config files from",
|
||||
"raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that version at all."
|
||||
],
|
||||
"generic_versions_kept": 10,
|
||||
"generic_versions_kept": 5,
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — bounds its assertion to the newest N versions"
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-08-09",
|
||||
"recorded_by": "CC 2026-08-09; the number's source (the R-267 newest-10 prune, established 2026-08-10 by R-287) recorded 2026-10-05 (R-291)"
|
||||
"recorded": "2026-11-14",
|
||||
"recorded_by": "CC 2026-11-14: day0_agent 0.174.0 (operator vouch 2026-11-14 15:21:45 CET, read from the hub log and from the hub's System page: agent=0.174.0, bundle 7d87d058… = the release; the save took 428 ms); earlier 0.173.0 (CC 2026-11-10) (operator vouch 2026-11-10 16:29:16 CET, read from the hub log: agent=0.173.0, bundle 51ebc1f4… = the release; the save took 232 ms); earlier 0.172.1 (CC 2026-11-07) (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.174.0"
|
||||
}
|
||||
}
|
||||
@@ -179,13 +179,24 @@ def published_cases(base):
|
||||
TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3
|
||||
VERS = [t[1:] for t in TAGS]
|
||||
|
||||
# D50 = A: the gate also asserts the PINNED versions. The decoys run against their own policy file (the real pins
|
||||
# name versions the fake Gitea does not hold); PINS is what each case writes there.
|
||||
pol_dir = tempfile.mkdtemp(prefix="retention-decoy-")
|
||||
pol = os.path.join(pol_dir, "retention-policy.json")
|
||||
PINS = {}
|
||||
|
||||
def case(name, setup, expect_rc, must=()):
|
||||
FAKE.reset()
|
||||
FAKE.tags = list(TAGS)
|
||||
FAKE.packages = set(VERS)
|
||||
FAKE.raw = set(VERS)
|
||||
PINS.clear()
|
||||
setup()
|
||||
rc, out = run([sys.executable, gate], ROOT, child_env(base))
|
||||
with io.open(pol, "w", encoding="utf-8") as fh:
|
||||
json.dump({"generic_versions_kept": keep, "pinned": dict(PINS)}, fh)
|
||||
env = child_env(base)
|
||||
env["FELHOM_RETENTION_POLICY"] = pol
|
||||
rc, out = run([sys.executable, gate], ROOT, env)
|
||||
if not FAKE.hits:
|
||||
fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name)
|
||||
report("published: " + name, rc, out, expect_rc, must)
|
||||
@@ -220,6 +231,25 @@ def published_cases(base):
|
||||
case("BY DESIGN: a version older than the retention window is not asserted", retired, 0,
|
||||
("NOT ASSERTED", "0.9.0"))
|
||||
|
||||
def pinned_old_ok():
|
||||
FAKE.tags = ["v0.8.0"] + ["v0.10.%d" % i for i in range(keep)]
|
||||
FAKE.packages = set(t[1:] for t in FAKE.tags)
|
||||
FAKE.raw = set(t[1:] for t in FAKE.tags)
|
||||
PINS["hub_min_agent"] = "0.8.0"
|
||||
case("GENUINE (D50): a pinned version older than the window is asserted and downloads", pinned_old_ok, 0,
|
||||
("verifying: 0.8.0", "ALL RELEASED VERSIONS INSTALLABLE"))
|
||||
|
||||
def pinned_old_gone():
|
||||
pinned_old_ok()
|
||||
FAKE.packages.discard("0.8.0")
|
||||
case("FACT (D50): a pinned version older than the window whose package 404s", pinned_old_gone, 1,
|
||||
("FAIL v0.8.0", "binary NOT downloadable"))
|
||||
|
||||
def pinned_no_tag():
|
||||
PINS["day0_agent"] = "0.150.9"
|
||||
case("FACT (D50): a pin that names a version with no tag", pinned_no_tag, 1,
|
||||
("FAIL pinned day0_agent=0.150.9", "no git tag"))
|
||||
|
||||
def five_hundred():
|
||||
FAKE.tags_status = 500
|
||||
case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",))
|
||||
|
||||
Reference in new issue
Block a user