Compare commits
30 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| ad80ee749a | |||
| 0683cc2dab | |||
| b1bc63dd32 | |||
| e66a48f2b7 | |||
| b890b57540 | |||
| aa69114d18 | |||
| abcc1e6833 | |||
| d912307bb7 | |||
| feaaa0e8ff | |||
| 6b6ba0c17f | |||
| 4697f8ac9e | |||
| 4a1a84941c | |||
| c8d3c17b13 | |||
| f5c3d77a61 | |||
| 5e05627c4e | |||
| 1e087bb158 | |||
| 29bcf4403e | |||
| 08ca67861f | |||
| f7e1dcbb27 | |||
| 2d4b1930df | |||
| b4721eee11 | |||
| 1109092def | |||
| b45c47be6c | |||
| b9b831fb90 | |||
| 39fb9cbf6c | |||
| ae69ad4512 | |||
| e803ad2b91 | |||
| 7651461be2 | |||
| a0c88e5183 | |||
| 2d5840ba11 |
@@ -50,6 +50,12 @@ it **first** in the morning note. A decision you cannot write in that shape is o
|
|||||||
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
|
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
|
||||||
on all three — not at night unless the night brief says so. **Agent releases still stop for the
|
on all three — not at night unless the night brief says so. **Agent releases still stop for the
|
||||||
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
|
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
|
||||||
|
**A version number is pushed once** — `felhom-controller/controller/build.sh` now refuses a second
|
||||||
|
push of a tag that exists, and a registry it cannot reach is a refusal too. This is a FACT about
|
||||||
|
what the build does, not a new rule: a fix for something already pushed is the next patch version,
|
||||||
|
and a box on a bad image is moved by the normal update path, never by re-tagging. Earned on
|
||||||
|
2026-10-15, when 0.315.0 was pushed three times from three different commits and a cache kept
|
||||||
|
serving the first one (`09-update-architecture.md` §0).
|
||||||
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
|
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
|
||||||
Prose is not a record.
|
Prose is not a record.
|
||||||
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
|
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
|
||||||
|
|||||||
+221
-1
@@ -1,4 +1,224 @@
|
|||||||
## Unreleased (2026-10-21) — every guest may swap as much as its RAM (R-1192, the 8 GB freeze) — COMMITTED, NOT RELEASED (the release is the operator's word)
|
## Unreleased (next: v0.167.0) — the BYO SSH clean-up in the SAME update (R-1285) (2026-10-26) — committed, not released
|
||||||
|
|
||||||
|
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
|
||||||
|
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
|
||||||
|
second later — so 0.166.0's pass never ran on the bundle that delivered it.
|
||||||
|
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
|
||||||
|
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
|
||||||
|
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
|
||||||
|
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
|
||||||
|
agent-only — holding the bundle job until the commit would need new hub queue logic.
|
||||||
|
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
|
||||||
|
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
||||||
|
removed → commit only, no pass — the demo-hp symptom).
|
||||||
|
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
|
||||||
|
0.165.0 box passes even without this fix.
|
||||||
|
|
||||||
|
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
|
||||||
|
|
||||||
|
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
|
||||||
|
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
|
||||||
|
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
|
||||||
|
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
|
||||||
|
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
|
||||||
|
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
|
||||||
|
|
||||||
|
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
|
||||||
|
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
|
||||||
|
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
|
||||||
|
„BUNDLE POLICY DONE install_mode=appliance removed=0".
|
||||||
|
|
||||||
|
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
|
||||||
|
|
||||||
|
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
|
||||||
|
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
|
||||||
|
drop-in until its next bundle. A change inside the wrapper alone cannot fix the run that delivers it.
|
||||||
|
- **The fix — the new AGENT asks the new wrapper:** `internal/osupdate/bundle.go` runs a policy pass right after every
|
||||||
|
bundle install (`mode: "bundle-policy"`, host layer, no envelope); the wrapper on disk is the bundle's own by then.
|
||||||
|
`configs/felhom-os-apply` `Bundle.policy_pass`: on a root-owned BYO record it removes the appliance-only files WE
|
||||||
|
wrote (our header), keeps the previous copy beside the bundle's, `sshd -t` then reloads ssh — a failing `sshd -t` puts
|
||||||
|
the file back. It writes nothing, so it needs no signature; an appliance and the owner's own file are left alone. Never
|
||||||
|
fatal for the agent: an older wrapper refuses the unknown mode and the next bundle does it. Report key `bundle_policy`.
|
||||||
|
- Tests: `TestR1285_AnInstalledBundleIsFollowedByThePolicyPass` (a refused bundle runs no pass);
|
||||||
|
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
|
||||||
|
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
|
||||||
|
|
||||||
|
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
|
||||||
|
|
||||||
|
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
|
||||||
|
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
|
||||||
|
usable that long, and the nonce store no longer keeps its record that long.
|
||||||
|
- `configs/felhom-os-apply`: the root wrapper holds the same 24 h (`MAX_WINDOW_S`, refusal R3 „the signed job's window
|
||||||
|
is longer than 24 h").
|
||||||
|
- `cmd/felhom-opsign`: `-ttl` above 24 h (or ≤ 0) is refused before anything is signed.
|
||||||
|
- Tests: `TestVerify_RejectsPerStage/window_longer_than_24_h` and `…/window_of_exactly_24_h_is_accepted`;
|
||||||
|
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
|
||||||
|
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
|
||||||
|
|
||||||
|
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
|
||||||
|
|
||||||
|
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
|
||||||
|
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
|
||||||
|
(`hub_min_agent` 0.158.1, `day0_agent` 0.165.0 — copies of hub state, because CI holds no hub secret; the session that
|
||||||
|
records a vouch or a new MinAgent updates them in the same commit). `check-published-versions.py` asserts the window
|
||||||
|
plus every pin; a pin with no tag FAILS. CI was red since the operator's hand prune of 0.159.0–0.160.0 (job 2334).
|
||||||
|
- **Decoys** (`test_gate_decoys.py`, via `FELHOM_RETENTION_POLICY`, test only): a pinned version older than the window is
|
||||||
|
asserted; its 404 convicts; a pin with no tag convicts. Seen failing on the old gate (two LIVE HOLES): felhom.eu
|
||||||
|
`documentation/audits/night-2026-10-25/redproof-d50.txt`.
|
||||||
|
|
||||||
|
## v0.165.0 (2026-10-25) — D43 = A: BYO hosts keep their own SSH settings (R-1257); the line test for a box that seeds all day (R-1250) — released in the attended release run; **VOUCHED by the operator 2026-10-25 18:38:49 CET** (binary `e9d04a47…`, bundle `8dea1f34…`)
|
||||||
|
|
||||||
|
**Found at the release (felhom.eu R-1285, P4):** on a BYO host upgraded from 0.164.0 the drop-in is removed only on the bundle run AFTER the one that installs this wrapper — the first run is planned by the old wrapper. Proven on a BYO test host (VM 9371 on demo-hp): the second `agent_config_update` logged `BUNDLE REMOVED … (ours, on a BYO host — D43)`.
|
||||||
|
|
||||||
|
### BYO hosts keep their own SSH settings (R-1257, D43 = A) — 2026-10-24 night
|
||||||
|
|
||||||
|
- **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an
|
||||||
|
appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy
|
||||||
|
is now `appliance`: written only when the ROOT-OWNED install record (`/var/lib/felhom-install/state.json`) says
|
||||||
|
`appliance`. On a `byo` record nothing is written, and a copy WE wrote (this exact path, beginning with our header
|
||||||
|
line) is REMOVED, its previous copy kept for the undo, then `sshd -t` and a reload. A box whose record is absent or
|
||||||
|
agent-writable proves nothing and is left exactly as it is. The bundle report lists `removed`; the record says
|
||||||
|
`removed` / `skipped`. The installer records the mode before it installs the bundle, so a new appliance still gets it.
|
||||||
|
- **X3: the „in force" check asks sshd about each of the host's own LAN addresses** (`ip -4 -o addr show scope
|
||||||
|
global`), not only 192.0.2.1 — a `Match Address <LAN>` block that re-allows root passwords no longer reads as in force.
|
||||||
|
- **X4: a failing `sshd -t` puts back ONLY the SSH step** (`undo_sshd_step`: the previous copy, or none; sshd is not
|
||||||
|
reloaded, so it never ran the refused state) and the rest of the bundle stays; it used to refuse and undo the whole
|
||||||
|
bundle. The report carries `sshd_step_undone`; the record names what is on disk.
|
||||||
|
- Tests (`configs/test_felhom_config_bundle.py`, class `ByoKeepsItsSSH`, 7 tests; each rule seen failing with it taken
|
||||||
|
out): a BYO host gets no drop-in; our copy comes off a BYO host; the owner's own file stays; no record → left as it
|
||||||
|
is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step.
|
||||||
|
The fake box now carries an appliance record by default (every earlier test unchanged).
|
||||||
|
|
||||||
|
### The line test for a box that seeds all day (R-1250) — 2026-10-24
|
||||||
|
|
||||||
|
Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0)
|
||||||
|
starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested
|
||||||
|
line and qBittorrent never got its 50 % limit. Now:
|
||||||
|
|
||||||
|
- **`linetest.Tester.WaitingForQuiet`** — due, inside the window, tries left, tunnel up, and only the quiet missing.
|
||||||
|
`GET /host/line-test` carries it as `waiting_for_quiet`; a controller with a seeding torrent app acts only on it.
|
||||||
|
It needs the full watched window (11 minute samples): a freshly started agent is not known to be busy, so it never
|
||||||
|
invites a hold (found in the live proof, 2026-10-24: the first build said `true` before one minute was watched;
|
||||||
|
`TestFreshTesterIsNotWaiting`).
|
||||||
|
- **`linetest.Tester.RunNow`** and **`POST /host/line-test/run`** (synchronous, about 55 s) — a test on request while the
|
||||||
|
controller holds its app's upload low: every weekly rule holds (due, window, `MaxTriesPerDay`, tunnel, the one heavy-op
|
||||||
|
gate) except the 10 quiet minutes, which become a 5 s settle and a 10 s quiet check (below 2 Mbit/s, so a request
|
||||||
|
never measures the family's own upload); the three runs are 5 s apart instead of 30. A refusal answers its code
|
||||||
|
(`not-due`, `outside-window`, `tries-used`, `no-tunnel`, `heavy-op`, `not-quiet`, `server-busy`); a failed request
|
||||||
|
spends one of the day's tries. `Tick` and `RunNow` now share one `ready()` check.
|
||||||
|
- Tests (each seen failing): `TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt` (the sleeps, the 60 s bound),
|
||||||
|
`TestRunNowRefusesAnUplinkThatIsStillBusy`, `TestRunNowKeepsTheWeeklyRules`, `TestRunNowFailuresSpendTheDaysTries`,
|
||||||
|
`TestQuietBoxIsNotWaiting`, `TestLineTestRunRoute`; a compile-time assertion that the Tester serves the request half.
|
||||||
|
|
||||||
|
**No sudoers or config-bundle change. MinController: none** — an older controller never calls the route.
|
||||||
|
|
||||||
|
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
|
||||||
|
|
||||||
|
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
|
||||||
|
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
|
||||||
|
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
|
||||||
|
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
|
||||||
|
|
||||||
|
## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST
|
||||||
|
|
||||||
|
Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256
|
||||||
|
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341` (configs unchanged — no bundle job sent). Built from
|
||||||
|
`f7e1dcb` (CI 2277 green) in a fresh clone by `scripts/release-agent.sh`. Delivered by one signed `agent_update` per box
|
||||||
|
(07:25Z): Tester 1, demo-hp and demo-felhom — COMPLETED, self-check 76/76, committed after the 1-minute dwell, the line
|
||||||
|
test started (felhom.eu `audits/line-test-2026-10-24/agent-0164-boxes.txt`).
|
||||||
|
|
||||||
|
Operator ruling 2026-10-24 07:26: once a week the box's HOST measures its upload line with a 10-second iperf3 upload to an
|
||||||
|
iperf3 server on ep0 that answers **only inside the WireGuard tunnel** (felhom.eu `scripts/ep0-iperf3/`; installed the same
|
||||||
|
day). The agent speaks the iperf3 protocol itself (operator's pick, 2026-10-24: nothing is installed on any host).
|
||||||
|
**No sudoers or config-bundle change.** **MinController: none** — the controller (≥ 0.342.0) reads the new route and
|
||||||
|
treats a 404 as „not measured".
|
||||||
|
|
||||||
|
- **`internal/linetest`** — `Client` (the iperf3 3.x client protocol: cookie, params, N data streams, TEST_END, results
|
||||||
|
exchange; the value is the SERVER's received bytes) and `Tester` (the schedule): due weekly from Tuesday 10:00 local,
|
||||||
|
starts 10:00–15:59 only (never in the night chain), from a per-host slot spread over 10:00–13:59; only after 10 quiet
|
||||||
|
minutes (every one-minute transmit rate of the host's physical uplink below 2 Mbit/s); under the one-heavy-op gate
|
||||||
|
(`line-test`); 3 runs of 10 s with 4 streams, 30 s apart, the highest counts; each run bounded by 25 s; „server busy" →
|
||||||
|
tried again at the next tick the same day, at most 6 tries a day; no operational tunnel → no test, no value. The last
|
||||||
|
result is persisted (`<oob state dir>/line-test.json`).
|
||||||
|
- **Report:** a `line_test` stanza (`schedule`, `last_ok_at`, `mbps`, `runs_mbps`, `server`, `last_attempt_at`,
|
||||||
|
`last_error`). **Local API:** `GET /host/line-test` → `{present, mbps, measured_at, method:"iperf3-tunnel"}`.
|
||||||
|
- `wgtunnel.Manager.LineTestTarget()` — the desired block's `pbs_tunnel_ip`, only while the tunnel is operational
|
||||||
|
(cached, no exec). Config `line_test.disable` (default on).
|
||||||
|
- **Proven live** (felhom.eu `audits/line-test-2026-10-24/`): a probe built from this package on demo-hp's host read
|
||||||
|
**90.7 Mbit/s** in 10.4 s (the push floor said „at least 10"); a second test 2 s later → „server busy" in 0.1 s while the
|
||||||
|
first went on undisturbed (90.6).
|
||||||
|
- Tests, each red-proved by a mutation (15): `TestClientSpeaksTheProtocolAndCountsTheServersBytes` (against a fake
|
||||||
|
iperf3 server), `TestClientBusyServerIsErrBusy`, `TestClientHonoursTheDeadlineAgainstAStalledServer`,
|
||||||
|
`TestEligibleWindowNeverInTheNightChain`, `TestNeverTestsInTheNightOrBeforeTuesday`, `TestBusyServerIsRetriedLaterTheSameDay`,
|
||||||
|
`TestTriesPerDayAreBounded`, `TestNoTunnelNoTest`, `TestBusyUplinkOrHeavyOpWaits`, `TestTickMeasuresTheHighestOfThreeRuns`,
|
||||||
|
`TestStateSurvivesARestartAndReportsIt`, `TestUplinkCountsTheBridgesPhysicalPortsOnly`, `TestLineTestRoute`,
|
||||||
|
`TestLineTestTargetOnlyWhileOperational`, `TestMainWiresTheLineTest`.
|
||||||
|
|
||||||
|
## v0.163.0 — the tunnel heals itself after a clock that ran ahead (D30, R-1204); root SSH from the LAN by key only (D4, R-982) (2026-10-23) — RELEASED on the operator's yes (day brief); the vouch is the operator's
|
||||||
|
|
||||||
|
Binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`; config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`. Built from `1109092` (CI 2246 green). Both sections below are in it.
|
||||||
|
|
||||||
|
### The tunnel re-key (D30) and the D4 bundle drop-in
|
||||||
|
|
||||||
|
- **D30 (operator 2026-10-22: the agent makes a new key and registers it on the existing peer path).** When the latest
|
||||||
|
handshake stays stamped in the future for 15 minutes (the R-1204 sign: ep0 keeps the future timestamp and refuses
|
||||||
|
every newer handshake from that key), the agent makes a new WireGuard key pair, keeps the old private key as
|
||||||
|
`wg/private.key.prev` (0600 — the escrow may hold it), and registers the new public key through `POST /hosts/<id>/wg`.
|
||||||
|
The hub re-keys the peer IN PLACE (same /32, generation bump) and its full-list push to ep0 drops the old key and adds
|
||||||
|
the new one — nothing new runs on ep0. A new key is a new peer for ep0, with no stored timestamp, so the next
|
||||||
|
handshake is accepted. At most once an hour; not while the tunnel unit is down; never while a re-register is already
|
||||||
|
in flight. A failed registration is retried by the existing re-key-in-place path. Logs carry public keys only.
|
||||||
|
Tests `TestD30_*` (4), red-proved. A host-loss DR that installs the escrowed (older) key registers it again, and the
|
||||||
|
hub re-keys in place back to it — the same path.
|
||||||
|
- **Review of the night's own code (felhom.eu `REPORT-night-2026-10-23.md`, N6, N7).** The rotation never leaves the box
|
||||||
|
without a readable key: the new key is written and fsynced to a temp file, the current key gets its kept name by a
|
||||||
|
hard link, then the temp file is renamed OVER `private.key` and the directory fsynced (a power cut or a kill between
|
||||||
|
two renames used to leave no key, or an empty one). `private.key.prev` keeps the FIRST old key (the escrow may hold it);
|
||||||
|
later ones are `private.key.prev.<unix>`, the newest three kept. The once-an-hour bound is saved in `wg/last-rekey`, so
|
||||||
|
an agent restart does not reset it; a start or a last re-key dated AFTER now (a clock that jumped back) is dropped,
|
||||||
|
never a block until that date (the commit's own security review). Tests `TestD30_TwoRotationsKeepTheFirstOldKey`,
|
||||||
|
`TestD30_ALastReKeyFromTheFutureDoesNotBlock`,
|
||||||
|
`TestD30_TheHourBoundSurvivesARestart`, red-proved. **Live (felhom.eu `audits/night-2026-10-23/P2-D30/`):** the D30
|
||||||
|
re-key ran on a test box (before these two changes): clock corrected → 15 min → new key registered → the box switched
|
||||||
|
at its next desired-state fetch → a handshake with the new key.
|
||||||
|
- **D4 (operator ruling 2026-10-22, R-982) — root SSH from the LAN by key only, on `main` now (2026-10-23 day brief).** The
|
||||||
|
config bundle carries `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`;
|
||||||
|
content check: exactly that line, `sshd -t -f`). After an install that wrote it: `sshd -t` (the whole config), then
|
||||||
|
`systemctl try-reload-or-restart ssh` (a stopped or socket-activated sshd no longer refuses the bundle), then `sshd -T`
|
||||||
|
is read back and a drop-in that is NOT in force (no Include, an earlier drop-in, a Match block) is said loudly and
|
||||||
|
reported as `sshd_dropin_not_in_force` — never a refusal; an undone bundle reloads sshd again. The console and the
|
||||||
|
web page keep the root password; felhom-sshd (8822) is not touched. Tests `RootKeyOnly` (6), red-proved; the
|
||||||
|
installer's uninstall removes the file (felhom.eu installer 1.36.0). Reviewed 2026-10-23 night (A1–A7).
|
||||||
|
- **Known gap, unchanged (filed):** after a reboot the kernel forgets the future stamp (latest handshake 0), so neither
|
||||||
|
this re-key nor the hub's alarm sees a box whose clock was corrected and which then restarted (R-1223).
|
||||||
|
|
||||||
|
### The future-stamped handshake is reported (R-1204) — first written on the 2026-10-22 night
|
||||||
|
|
||||||
|
- **R-982 / D4 — a ready patch on that night; on `main` since `1109092` and released in v0.163.0 (see above).** Built and proven the same night (felhom.eu
|
||||||
|
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
|
||||||
|
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
|
||||||
|
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
|
||||||
|
nested box from ISO 1.30.0: root password over SSH refused, root key works, the console and the web page still take
|
||||||
|
the password. It was committed (`a0c88e5`) and moved back out (`7651461`): the installer's uninstall must remove every bundle file (felhom.eu `scripts/test_hostinstall.py`), the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110 — a push to `main` alone publishes nothing) — both forbidden tonight. *(Corrected the same night: the first reason written here, „a push of `felhom-host-install.sh` publishes the installer", was wrong — the website serves `/scripts/` from the tag `installer-v1.35.0`, read live in `manifests/webpage.yaml` and the running Deployment.)* Both halves are patches: `felhom-agent-bundle-D4.patch` and `felhom.eu-installer-D4.patch` in that folder; apply them together with the agent release and the next installer tag.
|
||||||
|
- **R-1204 — `wireguard.handshake_future_s`.** A box that ran once with its clock ahead keeps a handshake stamped in the
|
||||||
|
future; ep0 refuses every newer one, and `last_handshake_age_s` (clamped at 0) read „just now" for ever. The heartbeat
|
||||||
|
now carries how many seconds the handshake lies ahead (> 60 s only); the age keeps its clamp. The hub alarms on it
|
||||||
|
(felhom.eu hub `wg_tunnel_stale`). Test `TestR1204_AFutureHandshakeIsReported`, red-proved.
|
||||||
|
|
||||||
|
## v0.162.0 — every guest may swap as much as its RAM (R-1192, the 8 GB freeze); the host's swap is reported and checked; no false „operator access degraded" on a fresh box (R-1194) (2026-10-22) — RELEASED on the operator's yes; the vouch is the operator's
|
||||||
|
|
||||||
|
Released 2026-10-22 07:40 UTC (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at
|
||||||
|
`b310773`: binary sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (the same bytes as the
|
||||||
|
night's candidate), config bundle sha256 `ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5`, tag
|
||||||
|
`v0.162.0`; no `configs/` change since v0.161.0. Delivered by one signed `agent_update` job per box to demo-hp, Tester 1
|
||||||
|
and demo-felhom (07:49–07:58 UTC): each `signed op COMPLETED`, `capabilities self-check ok=76 total=76`, `update
|
||||||
|
committed`, then `guest-swap: raised` for its 9201 (512 MB → its RAM). Evidence felhom.eu
|
||||||
|
`documentation/audits/morning-2026-10-22/release/04-agent-delivery.txt`. **Found at once:** demo-hp (host swap 8191 MB,
|
||||||
|
guest swap 25898 MB) and demo-felhom (8191 MB vs 12288 MB) logged the host-swap WARN and the hub mailed
|
||||||
|
`host_swap_short` — the check compares the host's swap with the guests' whole swap allowance, which a guest larger
|
||||||
|
than the installer's 8 GB host swap always exceeds (felhom.eu register R-1216). Soak (night of 2026-10-21, 8 GB box):
|
||||||
|
11 h, no freeze, the host's swap peaked at 2.7 GB used.
|
||||||
|
|
||||||
|
|
||||||
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
|
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
|
||||||
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
|
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
|
||||||
|
|||||||
+24
@@ -1,5 +1,29 @@
|
|||||||
# CONTEXT — felhom-agent working state
|
# CONTEXT — felhom-agent working state
|
||||||
|
|
||||||
|
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
|
||||||
|
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
|
||||||
|
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
|
||||||
|
> on demo-hp (felhom.eu R-1250, read, not forced).
|
||||||
|
|
||||||
|
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
|
||||||
|
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
|
||||||
|
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
|
||||||
|
> put back. Release + the operator's vouch: Monday's list (felhom.eu STATUS).
|
||||||
|
|
||||||
|
> **2026-10-24 (attended release run) — v0.164.0: the weekly upload line test (R-1247; operator ruling 07:26).**
|
||||||
|
> `internal/linetest`: the agent's own iperf3 client (operator's pick — nothing installed on hosts) to ep0's tunnel-only
|
||||||
|
> server `10.77.0.1:5201`, Tuesday 10:00-15:59, after 10 quiet minutes, under the heavy-op gate, best of 3; `line_test`
|
||||||
|
> stanza + `GET /host/line-test` (the controller's qBittorrent limit, D41 = 50 %). demo-hp measured 90.7 Mbit/s. Release
|
||||||
|
> needs the operator's vouch.
|
||||||
|
|
||||||
|
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
|
||||||
|
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
|
||||||
|
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
|
||||||
|
> key on ep0). Live on a test box (felhom.eu `audits/night-2026-10-23/P2-D30/`): with ep0 free to start a handshake the
|
||||||
|
> tunnel heals by itself (no re-key, correct); with ep0's starts dropped it stays dead and the agent re-keyed at 15 min.
|
||||||
|
> The interface switches at the next desired-state fetch. Gap: a reboot forgets the future stamp (R-1223). The D4 bundle
|
||||||
|
> patch v2 (`P3-review/felhom-agent-bundle-D4-v2.patch`): `try-reload-or-restart`, an `sshd -T` read-back reported as
|
||||||
|
> `sshd_dropin_not_in_force`, a reload in undo. Next agent release = D30 + D4 v2 + R-1204's field, the operator vouches.
|
||||||
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
|
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
|
||||||
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
|
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
|
||||||
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
|
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
|
||||||
|
|||||||
@@ -1,18 +1,21 @@
|
|||||||
# REPORT — agent, the night of 2026-10-21: the host's swap is reported; no false emergency-door alarm on a new box — COMMITTED, NOT RELEASED
|
# REPORT — R-1285: the BYO SSH clean-up in the same update — committed, NOT released (2026-10-26 afternoon)
|
||||||
|
|
||||||
**Brief:** the unattended night brief of 2026-10-21 (Parts 1, 2, 5). Architecture: felhom.eu `03-host-agent.md` §9 (the
|
**Plain words first.** A BYO box coming from 0.164.0 kept our SSH file one release too long: in the normal update the
|
||||||
new swap paragraph). Both join the swap fix (`286a7e3`) in the coming 0.162.0 — the operator's release and vouch (D26 = A).
|
OLD agent installs the config bundle, so the new agent's clean-up pass never ran on it. Now the new agent runs that pass
|
||||||
|
once, right after its self-update commits. Not released today (operator: „not today"); its live proof needs a BYO box on
|
||||||
|
0.164.0, which rides the next agent release.
|
||||||
|
|
||||||
- **Host swap (R-1192 follow-up), `0da792e`:** the host report carries `host.host_swap_mb` (Proxmox node status = the
|
## The fix (`b1bc63d`, CI 2363 ✓)
|
||||||
kernel's SwapTotal) and `host.guest_swap_mb` (the guests' `swap` sum; each guest spec `swap_bytes`); one WARN when the
|
|
||||||
first is below the second, one INFO when covered again; `GET /guest/memory` carries both for the controller's install
|
- `internal/selfupdate/commit.go`: `Manager.SetOnCommitted` — a hook run once after „update committed".
|
||||||
page. The agent never creates host swap (register R-1207). Proven live on the soak box: the hub stored
|
- `internal/osupdate/bundle.go`: `ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's `bundle-policy` mode, host
|
||||||
`host_swap_mb 7935, guest_swap_mb 4096`.
|
layer, no envelope); `cmd/felhom-agent/main.go` `wireBundlePolicyAfterCommit` hooks it on.
|
||||||
- **R-1194, `0da792e`:** no OOB stanza until the felhom-sshd loop's first pass has run — the first report on a fresh box
|
- The smaller of the two fixes: agent-only — holding the bundle job until the commit would need new hub queue logic.
|
||||||
read a config not yet rendered and the hub mailed `oob_degraded` (reproduced 2026-10-21 20:34 CEST on a fresh install).
|
|
||||||
- **The swap fix itself** ran as a hand-placed fixture on the 8 GB soak box: guest swap 512 → 4096 MB one minute after
|
## Proof
|
||||||
start; the 2-hour household load answered every probe (felhom.eu `audits/night-2026-10-21/P1-soak/`).
|
|
||||||
- Release candidate built from a clean clone at `0da792e` with the release flags, version `0.162.0`:
|
- `cmd/felhom-agent/r1285_policy_after_commit_test.go`, in the real order: a pending marker for the running version →
|
||||||
sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (not published).
|
`commit`, then exactly one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
||||||
- Tests (red-proved): `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
|
removed → commit only, no pass — the demo-hp symptom of 2026-10-26).
|
||||||
`TestR1194_NoOOBStanzaBeforeTheFirstPass`. CI job 2176 green.
|
- **Not live:** only a box coming FROM 0.164.0 can show it (0.165.0's wrapper already applies D43 at plan time), which
|
||||||
|
needs a signed downgrade chain on a BYO box.
|
||||||
|
|||||||
@@ -171,6 +171,7 @@
|
|||||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||||
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
|
||||||
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
|
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
|
||||||
|
| `linetest.Tester` + `linetest.Client` (R-1247) | internal/linetest/linetest.go · internal/linetest/iperf.go | `New(runner, target, gate, uplink, hostID, statePath, logger)` / `Tick(ctx)` / `Value()` / `LineTestStatus(ctx)`; `Client{Streams, Time}.Run(ctx, addr)` | the weekly upload line test: a 10-second iperf3 upload (the agent speaks the iperf3 protocol — nothing installed on the host) to ep0's tunnel-only server (`wgtunnel.Manager.LineTestTarget`), Tuesday 10:00-15:59 local from a per-host slot, after 10 quiet minutes of the PHYSICAL uplink (`SysUplinkTx`: the bridge's ports with a `device` link — the bridge counter misses bridged guest traffic), under `backup.InFlight`; best of 3; reported as `line_test`, served as GET /host/line-test | **The value is the SERVER's received bytes**, never the client's written ones (the send buffer). `Run` refuses a context without a deadline. iperf3 numbers streams 1, 3, 4, 5 — the server matches results by those ids. „server busy" (ACCESS_DENIED) stops the measurement and is retried at the next tick, at most `MaxTriesPerDay` |
|
||||||
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
|
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
|
||||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
||||||
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
|
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"go/ast"
|
||||||
|
"go/parser"
|
||||||
|
"go/token"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-1250: SetLineTest takes a LineTestReader and finds the request half by a type assertion, so a Tester that lost
|
||||||
|
// WaitingForQuiet or RunNow would compile and silently answer 503 to every seeding box. This line makes it a build
|
||||||
|
// failure instead.
|
||||||
|
var _ localapi.LineTestRequester = (*linetest.Tester)(nil)
|
||||||
|
|
||||||
|
// R-1247: the weekly line test has the fstrim shape (component + reporter seam + goroutine) plus a local-API reader,
|
||||||
|
// so its wiring is asserted from the AST. It must share the ONE heavy-op gate (heavyOps) — or it could run beside a
|
||||||
|
// backup — and take its server from the tunnel manager, never a literal address.
|
||||||
|
func TestMainWiresTheLineTest(t *testing.T) {
|
||||||
|
fset := token.NewFileSet()
|
||||||
|
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse main.go: %v", err)
|
||||||
|
}
|
||||||
|
var withGate, withTunnelTarget, reporterWired, localAPIWired, started bool
|
||||||
|
ast.Inspect(f, func(n ast.Node) bool {
|
||||||
|
switch node := n.(type) {
|
||||||
|
case *ast.CallExpr:
|
||||||
|
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
|
||||||
|
switch fn.Sel.Name {
|
||||||
|
case "New":
|
||||||
|
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "linetest" && len(node.Args) >= 3 {
|
||||||
|
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
|
||||||
|
withGate = true
|
||||||
|
}
|
||||||
|
if sel, ok := node.Args[1].(*ast.SelectorExpr); ok && sel.Sel.Name == "LineTestTarget" {
|
||||||
|
withTunnelTarget = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "SetLineTestReporter":
|
||||||
|
reporterWired = true
|
||||||
|
case "SetLineTest":
|
||||||
|
localAPIWired = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case *ast.GoStmt:
|
||||||
|
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
|
||||||
|
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "lineTest" {
|
||||||
|
started = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
if !withGate {
|
||||||
|
t.Error("main.go never calls linetest.New(..., heavyOps, ...) — no line test, or one outside the heavy-op gate")
|
||||||
|
}
|
||||||
|
if !withTunnelTarget {
|
||||||
|
t.Error("linetest.New does not take its server from wgMgr.LineTestTarget")
|
||||||
|
}
|
||||||
|
if !reporterWired {
|
||||||
|
t.Error("main.go never calls collector.SetLineTestReporter — the line_test stanza never reaches the hub")
|
||||||
|
}
|
||||||
|
if !localAPIWired {
|
||||||
|
t.Error("main.go never calls localSrv.SetLineTest — the controller's GET /host/line-test always reads present:false")
|
||||||
|
}
|
||||||
|
if !started {
|
||||||
|
t.Error("main.go never starts the line test with `go lineTest.Run(ctx)`")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -43,6 +43,7 @@ import (
|
|||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
|
||||||
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
|
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
|
||||||
@@ -1215,6 +1216,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
}
|
}
|
||||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||||
}}
|
}}
|
||||||
|
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
||||||
|
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
||||||
|
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
|
||||||
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
|
||||||
// to the local API server built below).
|
// to the local API server built below).
|
||||||
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
|
||||||
@@ -1641,6 +1645,24 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
} else {
|
} else {
|
||||||
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
|
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
|
||||||
}
|
}
|
||||||
|
// R-1247 (operator ruling 2026-10-24 07:26): the weekly upload line test — a 10-second iperf3 upload from this host
|
||||||
|
// to ep0's tunnel-only server, Tuesday 10:00-15:59 local, after 10 quiet minutes, under the one-heavy-op gate. Only
|
||||||
|
// with the offsite tunnel (the server is reachable nowhere else). Not part of the errc fan-out.
|
||||||
|
switch {
|
||||||
|
case !cfg.LineTest.Enabled():
|
||||||
|
logger.Info("linetest: weekly upload line test disabled by config (line_test.disable)")
|
||||||
|
case wgMgr == nil:
|
||||||
|
logger.Info("linetest: weekly upload line test off — no offsite tunnel on this host")
|
||||||
|
default:
|
||||||
|
lineTest := linetest.New(linetest.Client{Streams: linetest.Streams, Time: linetest.TestTime}, wgMgr.LineTestTarget,
|
||||||
|
heavyOps, linetest.SysUplinkTx("/"), cfg.Hub.HostID,
|
||||||
|
filepath.Join(cfg.OOB.WithDefaults().StateDir, "line-test.json"), logger)
|
||||||
|
collector.SetLineTestReporter(lineTest)
|
||||||
|
if localSrv != nil {
|
||||||
|
localSrv.SetLineTest(lineTest)
|
||||||
|
}
|
||||||
|
go lineTest.Run(ctx)
|
||||||
|
}
|
||||||
if lanLoop != nil {
|
if lanLoop != nil {
|
||||||
lanServers = 1
|
lanServers = 1
|
||||||
go func() { errc <- lanLoop.Run(ctx) }()
|
go func() { errc <- lanLoop.Run(ctx) }()
|
||||||
@@ -4021,3 +4043,9 @@ func agentStateDir(cfg config.Config) string {
|
|||||||
}
|
}
|
||||||
return "/var/lib/felhom-agent"
|
return "/var/lib/felhom-agent"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
|
||||||
|
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
|
||||||
|
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
|
||||||
|
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,91 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
|
||||||
|
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
|
||||||
|
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
|
||||||
|
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
|
||||||
|
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
|
||||||
|
|
||||||
|
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
|
||||||
|
type orderRunner struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
order []string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
|
||||||
|
var p map[string]any
|
||||||
|
b, _ := os.ReadFile(args[1])
|
||||||
|
_ = json.Unmarshal(b, &p)
|
||||||
|
mode, _ := p["mode"].(string)
|
||||||
|
r.order = append(r.order, "os-apply:"+mode)
|
||||||
|
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
|
||||||
|
}
|
||||||
|
if len(args) >= 1 {
|
||||||
|
r.order = append(r.order, filepath.Base(name)+":"+args[0])
|
||||||
|
}
|
||||||
|
return []byte("ok"), nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||||
|
return r.Run(ctx, name, args...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
|
||||||
|
t.Helper()
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
if marker != nil {
|
||||||
|
dir := filepath.Join(stateDir, "selfupdate")
|
||||||
|
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
b, _ := json.Marshal(marker)
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
r := &orderRunner{}
|
||||||
|
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
|
||||||
|
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
|
||||||
|
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
|
||||||
|
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
|
||||||
|
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
|
||||||
|
mgr.MaybeCommit(context.Background())
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return append([]string(nil), r.order...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
|
||||||
|
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||||
|
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
|
||||||
|
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
|
||||||
|
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
|
||||||
|
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
|
||||||
|
}
|
||||||
|
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
|
||||||
|
if len(got) != 0 {
|
||||||
|
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -123,6 +123,10 @@ func run() error {
|
|||||||
|
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
issued := now
|
issued := now
|
||||||
|
if *ttl <= 0 || *ttl > authz.MaxWindow {
|
||||||
|
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
expires := now.Add(*ttl)
|
expires := now.Add(*ttl)
|
||||||
|
|
||||||
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
|
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
|
||||||
|
|||||||
+182
-10
@@ -165,9 +165,13 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
|||||||
# dest: (source file under configs/, mode, check, policy)
|
# dest: (source file under configs/, mode, check, policy)
|
||||||
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
|
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
|
||||||
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
|
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
|
||||||
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none.
|
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
|
||||||
|
# sshd → `sshd -t -f` on the snippet alone (R-982).
|
||||||
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
|
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
|
||||||
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
|
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none);
|
||||||
|
# appliance → only on a box whose ROOT-OWNED install record says appliance (D43 = A, operator 2026-10-24: a BYO
|
||||||
|
# host keeps its own settings, `11` §1). On a BYO box nothing is written and a copy WE wrote (this exact path, our
|
||||||
|
# first line) is removed; a box with no readable record is left exactly as it is (it cannot prove which it is).
|
||||||
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
|
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
|
||||||
# R-861 (agent v0.146.0): the signed agent update, verified here (mode agent_update), then the A/B flip.
|
# R-861 (agent v0.146.0): the signed agent update, verified here (mode agent_update), then the A/B flip.
|
||||||
SELFUPDATE_OP = "agent_update"
|
SELFUPDATE_OP = "agent_update"
|
||||||
@@ -183,7 +187,14 @@ BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies
|
|||||||
BUNDLE_KEEP = 3
|
BUNDLE_KEEP = 3
|
||||||
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
|
BUNDLE_MAX_BYTES = 4 * 1024 * 1024
|
||||||
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
|
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
|
||||||
|
# R-1280: the longest validity window a signed job may carry — the agent's authz.MaxWindow, the same 24 h.
|
||||||
|
MAX_WINDOW_S = 24 * 3600
|
||||||
OOB_DIR = "/etc/felhom-sshd"
|
OOB_DIR = "/etc/felhom-sshd"
|
||||||
|
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
|
||||||
|
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||||
|
# D43 = A (R-1257): the first line every copy of the drop-in we ship carries — what proves a file on a BYO host is ours.
|
||||||
|
SSHD_DROPIN_HEADER = b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982"
|
||||||
|
|
||||||
BUNDLE_FILES = [
|
BUNDLE_FILES = [
|
||||||
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
|
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
|
||||||
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
|
||||||
@@ -221,6 +232,10 @@ BUNDLE_FILES = [
|
|||||||
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
|
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
|
||||||
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
|
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
|
||||||
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
|
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
|
||||||
|
# R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
|
||||||
|
# password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
|
||||||
|
# D43 = A (operator 2026-10-24, R-1257): appliances only — a BYO host keeps its own SSH settings (`11` §1).
|
||||||
|
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "appliance"),
|
||||||
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
|
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
|
||||||
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
|
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
|
||||||
]
|
]
|
||||||
@@ -434,7 +449,8 @@ class Runner:
|
|||||||
def check_content(self, kind, data):
|
def check_content(self, kind, data):
|
||||||
"""Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
|
"""Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
|
||||||
import tempfile
|
import tempfile
|
||||||
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind]
|
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"],
|
||||||
|
"sshd": ["/usr/sbin/sshd", "-t", "-f"]}[kind]
|
||||||
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
|
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
|
||||||
p = os.path.join(d, "f")
|
p = os.path.join(d, "f")
|
||||||
with open(p, "wb") as f:
|
with open(p, "wb") as f:
|
||||||
@@ -475,7 +491,8 @@ class Apply:
|
|||||||
|
|
||||||
def check_plan(self, plan):
|
def check_plan(self, plan):
|
||||||
mode = plan.get("mode", "apply")
|
mode = plan.get("mode", "apply")
|
||||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
|
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
|
||||||
|
"oom-check") \
|
||||||
+ KERNEL_MODES:
|
+ KERNEL_MODES:
|
||||||
raise Refused("R11", f"unknown mode {mode!r}")
|
raise Refused("R11", f"unknown mode {mode!r}")
|
||||||
if mode == "agent_update":
|
if mode == "agent_update":
|
||||||
@@ -486,6 +503,10 @@ class Apply:
|
|||||||
if plan.get("layer") != "host":
|
if plan.get("layer") != "host":
|
||||||
raise Refused("R11", "bundle is a host-layer mode")
|
raise Refused("R11", "bundle is a host-layer mode")
|
||||||
return mode, "host", 0, "bundle"
|
return mode, "host", 0, "bundle"
|
||||||
|
if mode == "bundle-policy":
|
||||||
|
if plan.get("layer") != "host":
|
||||||
|
raise Refused("R11", "bundle-policy is a host-layer mode")
|
||||||
|
return mode, "host", 0, "bundle-policy"
|
||||||
layer = plan.get("layer")
|
layer = plan.get("layer")
|
||||||
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
||||||
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
||||||
@@ -647,6 +668,8 @@ class Apply:
|
|||||||
iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ"))
|
iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ"))
|
||||||
except (KeyError, ValueError, TypeError):
|
except (KeyError, ValueError, TypeError):
|
||||||
raise Refused("R3", "the signed job has no readable time window")
|
raise Refused("R3", "the signed job has no readable time window")
|
||||||
|
if exp - iss > MAX_WINDOW_S:
|
||||||
|
raise Refused("R3", "the signed job's window is longer than 24 h (R-1280)")
|
||||||
if now > exp or now < iss - 120:
|
if now > exp or now < iss - 120:
|
||||||
raise Refused("R3", "the signed job is expired or not yet valid")
|
raise Refused("R3", "the signed job is expired or not yet valid")
|
||||||
nonce = op.get("nonce")
|
nonce = op.get("nonce")
|
||||||
@@ -1056,6 +1079,8 @@ class Apply:
|
|||||||
return self.facts()
|
return self.facts()
|
||||||
if self.mode == "bundle":
|
if self.mode == "bundle":
|
||||||
return Bundle(self).from_plan(plan)
|
return Bundle(self).from_plan(plan)
|
||||||
|
if self.mode == "bundle-policy":
|
||||||
|
return Bundle(self).policy_pass()
|
||||||
if self.mode == "agent_update":
|
if self.mode == "agent_update":
|
||||||
return self.agent_update(plan)
|
return self.agent_update(plan)
|
||||||
if self.layer in ("host", "pve", "kernel"):
|
if self.layer in ("host", "pve", "kernel"):
|
||||||
@@ -1864,6 +1889,8 @@ class Bundle:
|
|||||||
def __init__(self, apply):
|
def __init__(self, apply):
|
||||||
self.a, self.r = apply, apply.r
|
self.a, self.r = apply, apply.r
|
||||||
self.report = apply.report
|
self.report = apply.report
|
||||||
|
self.sshd_dropin_not_in_force = False
|
||||||
|
self.sshd_step_undone = None # X4 (R-1257): why the SSH step alone was put back ("" = not), the rest stays
|
||||||
|
|
||||||
# ---------- reading and checking ----------
|
# ---------- reading and checking ----------
|
||||||
def load_file(self, path):
|
def load_file(self, path):
|
||||||
@@ -1932,10 +1959,17 @@ class Bundle:
|
|||||||
text = content.decode("utf-8")
|
text = content.decode("utf-8")
|
||||||
except UnicodeDecodeError:
|
except UnicodeDecodeError:
|
||||||
raise Refused("R18", f"{dest} is not UTF-8 text")
|
raise Refused("R18", f"{dest} is not UTF-8 text")
|
||||||
if check in ("sudoers", "sh", "bash", "nft"):
|
if check in ("sudoers", "sh", "bash", "nft", "sshd"):
|
||||||
rc, msg = self.r.check_content(check, content)
|
rc, msg = self.r.check_content(check, content)
|
||||||
if rc != 0:
|
if rc != 0:
|
||||||
raise Refused("R18", f"{dest} fails its {check} check: {msg}")
|
raise Refused("R18", f"{dest} fails its {check} check: {msg}")
|
||||||
|
if check == "sshd":
|
||||||
|
# R-982: the drop-in's PermitRootLogin must be exactly `prohibit-password`. `no` would also shut the root key
|
||||||
|
# out (Proxmox's own node-to-node SSH and the LAN key route), `yes` would undo the ruling — both refused.
|
||||||
|
# Only that line is pinned: the file is signed like every bundle file (review A7).
|
||||||
|
vals = re.findall(r"^\s*PermitRootLogin\s+(\S+)\s*$", text, re.M | re.I)
|
||||||
|
if vals != ["prohibit-password"]:
|
||||||
|
raise Refused("R18", f"{dest} must set exactly `PermitRootLogin prohibit-password` (found {vals})")
|
||||||
elif check == "python":
|
elif check == "python":
|
||||||
try:
|
try:
|
||||||
compile(text, dest, "exec")
|
compile(text, dest, "exec")
|
||||||
@@ -1966,14 +2000,40 @@ class Bundle:
|
|||||||
return b"", None, None # a symlink or a directory: always replaced by the real file
|
return b"", None, None # a symlink or a directory: always replaced by the real file
|
||||||
return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid
|
return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid
|
||||||
|
|
||||||
|
def install_mode(self):
|
||||||
|
"""The box's ROOT-OWNED install record's mode ("appliance", "byo", …), or None when it proves nothing (absent,
|
||||||
|
agent-writable, unreadable) — the same proof check_appliance asks for the host layer (R12)."""
|
||||||
|
try:
|
||||||
|
st = self.r.stat(INSTALL_STATE)
|
||||||
|
if st.st_uid != 0 or (st.st_mode & 0o022):
|
||||||
|
return None
|
||||||
|
mode = json.loads(self.r.read_file(INSTALL_STATE)).get("mode")
|
||||||
|
except (OSError, ValueError, AttributeError):
|
||||||
|
return None
|
||||||
|
return mode if isinstance(mode, str) and mode else None
|
||||||
|
|
||||||
def plan_writes(self, files):
|
def plan_writes(self, files):
|
||||||
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt)."""
|
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt; an
|
||||||
|
appliance-only file on a box that is not one) / remove (an appliance-only file WE wrote, on a BYO box)."""
|
||||||
oob = self.r.isdir(OOB_DIR)
|
oob = self.r.isdir(OOB_DIR)
|
||||||
|
install_mode = self.install_mode()
|
||||||
plan = []
|
plan = []
|
||||||
for dest, content, mode, check, policy in files:
|
for dest, content, mode, check, policy in files:
|
||||||
if policy == "oob" and not oob:
|
if policy == "oob" and not oob:
|
||||||
plan.append((dest, content, mode, "skipped", None, None))
|
plan.append((dest, content, mode, "skipped", None, None))
|
||||||
continue
|
continue
|
||||||
|
if policy == "appliance" and install_mode != "appliance":
|
||||||
|
old, old_mode, _ = self.live(dest)
|
||||||
|
ours = old is not None and old.startswith(SSHD_DROPIN_HEADER)
|
||||||
|
if install_mode == "byo" and ours:
|
||||||
|
self.r.log(f"os-apply: BUNDLE {dest}: a BYO host keeps its own settings (D43) — removing the copy we wrote")
|
||||||
|
plan.append((dest, content, mode, "remove", old, old_mode))
|
||||||
|
else:
|
||||||
|
why = "a BYO host keeps its own settings (D43)" if install_mode == "byo" else \
|
||||||
|
f"the install record proves no appliance (mode={install_mode!r}) — left as it is"
|
||||||
|
self.r.log(f"os-apply: BUNDLE {dest} skipped: {why}")
|
||||||
|
plan.append((dest, content, mode, "skipped", None, None))
|
||||||
|
continue
|
||||||
old, old_mode, old_uid = self.live(dest)
|
old, old_mode, old_uid = self.live(dest)
|
||||||
if policy == "if-absent" and old is not None:
|
if policy == "if-absent" and old is not None:
|
||||||
plan.append((dest, content, mode, "kept", old, old_mode))
|
plan.append((dest, content, mode, "kept", old, old_mode))
|
||||||
@@ -2011,11 +2071,47 @@ class Bundle:
|
|||||||
ver, files = self.parse(data, sha)
|
ver, files = self.parse(data, sha)
|
||||||
return self.install(ver, sha, files, "installer", False)
|
return self.install(ver, sha, files, "installer", False)
|
||||||
|
|
||||||
|
def policy_pass(self):
|
||||||
|
"""R-1285 (D43 = A): the appliance-only files' policy, applied by THIS wrapper. A bundle that brings a new wrapper
|
||||||
|
is planned by the OLD one, which did not know the policy — so on a BYO host upgraded from 0.164.0 our SSH drop-in
|
||||||
|
stayed until the next bundle. The agent runs this pass right after every bundle install (the wrapper on disk is
|
||||||
|
then the new one). It only REMOVES a file WE wrote (our header) from a box whose root-owned install record says
|
||||||
|
byo, keeps the previous copy beside the bundle's, and reloads sshd; it writes nothing, so it needs no signature.
|
||||||
|
A failing `sshd -t` puts the file back (R-1257 X4's rule). Report key `bundle_policy`."""
|
||||||
|
log = self.r.log
|
||||||
|
install_mode = self.install_mode()
|
||||||
|
removed = []
|
||||||
|
if install_mode == "byo":
|
||||||
|
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
|
||||||
|
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-policy")
|
||||||
|
for dest, _src, _mode, _check, policy in BUNDLE_FILES:
|
||||||
|
if policy != "appliance":
|
||||||
|
continue
|
||||||
|
old, old_mode, _ = self.live(dest)
|
||||||
|
if old is None or not old.startswith(SSHD_DROPIN_HEADER):
|
||||||
|
continue
|
||||||
|
self.r.put_file(prev + dest, old, 0o600)
|
||||||
|
self.r.remove(dest)
|
||||||
|
removed.append((dest, old, old_mode))
|
||||||
|
log(f"os-apply: BUNDLE POLICY REMOVED {dest} (ours, on a BYO host — D43, R-1285)")
|
||||||
|
if any(d == SSHD_DROPIN for d, _, _ in removed):
|
||||||
|
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
|
||||||
|
if rc != 0:
|
||||||
|
for d, old, old_mode in removed:
|
||||||
|
self.r.put_file(d, old, old_mode)
|
||||||
|
log(f"os-apply: BUNDLE POLICY WARNING sshd -t failed without our drop-in (rc={rc}) — put back")
|
||||||
|
removed = []
|
||||||
|
else:
|
||||||
|
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
|
||||||
|
self.report["bundle_policy"] = {"install_mode": install_mode, "removed": [d for d, _, _ in removed]}
|
||||||
|
log(f"os-apply: BUNDLE POLICY DONE install_mode={install_mode} removed={len(removed)}")
|
||||||
|
return 0
|
||||||
|
|
||||||
# ---------- install, self-check, undo ----------
|
# ---------- install, self-check, undo ----------
|
||||||
def install(self, ver, sha, files, authority, bootstrap):
|
def install(self, ver, sha, files, authority, bootstrap):
|
||||||
log = self.r.log
|
log = self.r.log
|
||||||
plan = self.plan_writes(files)
|
plan = self.plan_writes(files)
|
||||||
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped")}
|
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped", "remove")}
|
||||||
log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} "
|
log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} "
|
||||||
f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}")
|
f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}")
|
||||||
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
|
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
|
||||||
@@ -2023,10 +2119,17 @@ class Bundle:
|
|||||||
done = [] # (dest, old bytes or None, old mode)
|
done = [] # (dest, old bytes or None, old mode)
|
||||||
rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev,
|
rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev,
|
||||||
"written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"],
|
"written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"],
|
||||||
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"]}
|
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"],
|
||||||
|
"removed": [p[0] for p in plan if p[3] == "remove"]}
|
||||||
self.report["bundle"] = rep
|
self.report["bundle"] = rep
|
||||||
try:
|
try:
|
||||||
for dest, content, mode, action, old, old_mode in plan:
|
for dest, content, mode, action, old, old_mode in plan:
|
||||||
|
if action == "remove": # D43 = A: our copy off a BYO host (the previous copy kept, the undo restores it)
|
||||||
|
self.r.put_file(prev + dest, old, 0o600)
|
||||||
|
done.append((dest, old, old_mode))
|
||||||
|
self.r.remove(dest)
|
||||||
|
log(f"os-apply: BUNDLE REMOVED {dest} (ours, on a BYO host — D43)")
|
||||||
|
continue
|
||||||
if action != "write":
|
if action != "write":
|
||||||
continue
|
continue
|
||||||
if old is not None:
|
if old is not None:
|
||||||
@@ -2044,22 +2147,58 @@ class Bundle:
|
|||||||
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
|
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
|
||||||
return 3
|
return 3
|
||||||
rep["signers_created"] = False
|
rep["signers_created"] = False
|
||||||
|
if self.sshd_dropin_not_in_force:
|
||||||
|
rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
|
||||||
|
if self.sshd_step_undone:
|
||||||
|
rep["sshd_step_undone"] = self.sshd_step_undone[:300] # X4: only the SSH step was put back
|
||||||
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
|
if bootstrap and not self.r.lexists(TRUST_SIGNERS):
|
||||||
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
|
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
|
||||||
rep["signers_created"] = True
|
rep["signers_created"] = True
|
||||||
log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}")
|
log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}")
|
||||||
record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority,
|
record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority,
|
||||||
"installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())),
|
"installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())),
|
||||||
"files": {p[0]: (sha256_hex(p[1]) if p[3] in ("write", "same") else
|
"files": {p[0]: self.record_entry(p) for p in plan}}
|
||||||
(sha256_hex(p[4]) if p[3] == "kept" else "skipped")) for p in plan}}
|
|
||||||
self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||||
self.prune()
|
self.prune()
|
||||||
log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} "
|
log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} "
|
||||||
f"self-check=ok signers-created={rep['signers_created']}")
|
f"self-check=ok signers-created={rep['signers_created']}")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
def record_entry(self, p):
|
||||||
|
"""The record's word for one plan entry: the sha it installed, the sha it kept, or what it did instead."""
|
||||||
|
dest, content, _, action, old, _ = p
|
||||||
|
if dest == SSHD_DROPIN and self.sshd_step_undone:
|
||||||
|
return sha256_hex(old) if old is not None else "absent" # X4: the SSH step was put back
|
||||||
|
if action in ("write", "same"):
|
||||||
|
return sha256_hex(content)
|
||||||
|
if action == "kept":
|
||||||
|
return sha256_hex(old)
|
||||||
|
return "removed" if action == "remove" else "skipped"
|
||||||
|
|
||||||
|
def lan_addresses(self):
|
||||||
|
"""X3 (R-1257): the host's own global IPv4 addresses — the ones a LAN client reaches sshd on."""
|
||||||
|
rc, out, _ = self.r.host(["ip", "-4", "-o", "addr", "show", "scope", "global"], 30)
|
||||||
|
if rc != 0:
|
||||||
|
return []
|
||||||
|
return sorted(set(re.findall(r"\binet (\d+\.\d+\.\d+\.\d+)/", out)))
|
||||||
|
|
||||||
|
def undo_sshd_step(self, plan, why):
|
||||||
|
"""X4 (R-1257): `sshd -t` refused the host's whole config with our step in it — put back ONLY that file (the
|
||||||
|
previous copy, or none) and keep the rest of the bundle. sshd was not reloaded, so it never ran the refused state."""
|
||||||
|
for dest, _, _, action, old, old_mode in plan:
|
||||||
|
if dest != SSHD_DROPIN or action not in ("write", "remove"):
|
||||||
|
continue
|
||||||
|
if old is None:
|
||||||
|
self.r.remove(dest)
|
||||||
|
else:
|
||||||
|
self.r.put_file(dest, old, old_mode if old_mode is not None else 0o644)
|
||||||
|
self.sshd_step_undone = why
|
||||||
|
self.r.log(f"os-apply: BUNDLE WARNING sshd -t refused the host's config with the SSH step — ONLY that step is put "
|
||||||
|
f"back, the rest of the bundle stays: {why[:200]} (R-1257)")
|
||||||
|
|
||||||
def after_install(self, plan):
|
def after_install(self, plan):
|
||||||
written = {p[0] for p in plan if p[3] == "write"}
|
written = {p[0] for p in plan if p[3] == "write"}
|
||||||
|
removed = {p[0] for p in plan if p[3] == "remove"}
|
||||||
dests = {p[0] for p in plan if p[3] != "skipped"}
|
dests = {p[0] for p in plan if p[3] != "skipped"}
|
||||||
if any(d.startswith("/etc/systemd/system/") for d in written):
|
if any(d.startswith("/etc/systemd/system/") for d in written):
|
||||||
self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload")
|
self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload")
|
||||||
@@ -2073,6 +2212,36 @@ class Bundle:
|
|||||||
"enable the crash guard")
|
"enable the crash guard")
|
||||||
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
|
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
|
||||||
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
|
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
|
||||||
|
# R-982: the host's sshd takes the root-key-only drop-in. The WHOLE config is tested first (a failure is a
|
||||||
|
# refusal, and the bundle undo puts the old files back); `try-reload-or-restart` keeps every open session and
|
||||||
|
# does nothing on a box whose ssh.service is not running (a socket-activated or stopped sshd reads the file
|
||||||
|
# at its next start) — a failure there must not freeze the bundle route (review 2026-10-22 night, A1).
|
||||||
|
# felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
|
||||||
|
# sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
|
||||||
|
# — that is said loudly in the log and the report, never silently recorded as done (A2).
|
||||||
|
# R-1257 X4: a failing `sshd -t` puts back ONLY the SSH step (undo_sshd_step); it used to refuse and undo the
|
||||||
|
# whole bundle. X3: the in-force check asks sshd about each of the host's own LAN addresses — a probe from a
|
||||||
|
# documentation address (192.0.2.1) read a `Match Address <LAN>` block that re-allows passwords as in force.
|
||||||
|
if SSHD_DROPIN in written or SSHD_DROPIN in removed:
|
||||||
|
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
|
||||||
|
if rc != 0:
|
||||||
|
self.undo_sshd_step(plan, f"sshd -t failed (rc={rc}): {(out + err).strip()[-200:]}")
|
||||||
|
return
|
||||||
|
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
|
||||||
|
if SSHD_DROPIN in written:
|
||||||
|
addrs = self.lan_addresses()
|
||||||
|
if not addrs:
|
||||||
|
self.r.log("os-apply: BUNDLE WARNING no global IPv4 address read for the sshd in-force check — "
|
||||||
|
"probing 192.0.2.1 only (R-1257)")
|
||||||
|
addrs = ["192.0.2.1"]
|
||||||
|
for addr in addrs:
|
||||||
|
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", f"user=root,host=felhom-check,addr={addr}"], 60)
|
||||||
|
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
|
||||||
|
if rc != 0 or eff != ["without-password"]:
|
||||||
|
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force for {addr}: sshd -T "
|
||||||
|
f"says permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier "
|
||||||
|
f"drop-in or a Match block (R-982, R-1257)")
|
||||||
|
self.sshd_dropin_not_in_force = True
|
||||||
|
|
||||||
def must(self, argv, what):
|
def must(self, argv, what):
|
||||||
rc, out, err = self.r.host(argv, 120)
|
rc, out, err = self.r.host(argv, 120)
|
||||||
@@ -2133,6 +2302,9 @@ class Bundle:
|
|||||||
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
|
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
|
||||||
if any(d.startswith("/etc/systemd/system/") for d in back):
|
if any(d.startswith("/etc/systemd/system/") for d in back):
|
||||||
self.r.host(["systemctl", "daemon-reload"], 120)
|
self.r.host(["systemctl", "daemon-reload"], 120)
|
||||||
|
if SSHD_DROPIN in back:
|
||||||
|
# A3: after_install may already have reloaded sshd with the drop-in; the restored state takes effect now.
|
||||||
|
self.r.host(["systemctl", "try-reload-or-restart", "ssh"], 120)
|
||||||
rc, _, _ = self.r.host(["visudo", "-c"], 60)
|
rc, _, _ = self.r.host(["visudo", "-c"], 60)
|
||||||
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
|
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
|
||||||
return back
|
return back
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
|
||||||
|
# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
|
||||||
|
# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
|
||||||
|
# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
|
||||||
|
# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
|
||||||
|
# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
|
||||||
|
PermitRootLogin prohibit-password
|
||||||
@@ -45,8 +45,10 @@ class St:
|
|||||||
class Box:
|
class Box:
|
||||||
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
|
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
|
||||||
|
|
||||||
def __init__(self, bundle_bytes, signed, oob=False, signers=True):
|
def __init__(self, bundle_bytes, signed, oob=False, signers=True, install_mode="appliance"):
|
||||||
self.files, self.modes, self.uids = {}, {}, {}
|
self.files, self.modes, self.uids = {}, {}, {}
|
||||||
|
if install_mode is not None: # the installer's ROOT-OWNED record (D43 = A reads it for the sshd drop-in)
|
||||||
|
self.put(osapply.INSTALL_STATE, json.dumps({"completed": [], "mode": install_mode}).encode(), 0o644)
|
||||||
self.dirs = {osapply.OOB_DIR} if oob else set()
|
self.dirs = {osapply.OOB_DIR} if oob else set()
|
||||||
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
|
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
|
||||||
if signers:
|
if signers:
|
||||||
@@ -429,6 +431,241 @@ class SelfCheckUndo(unittest.TestCase):
|
|||||||
self.assert_restored(box, before, rep)
|
self.assert_restored(box, before, rep)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
class RootKeyOnly(unittest.TestCase):
|
||||||
|
"""R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
|
||||||
|
whole config and reloads sshd — only when the file was written; a drop-in that says anything but
|
||||||
|
`prohibit-password` is refused before anything is written.
|
||||||
|
RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
|
||||||
|
with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
|
||||||
|
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||||
|
|
||||||
|
def test_fresh_box_takes_the_dropin_and_reloads(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
|
||||||
|
self.assertEqual(box.modes[self.DEST], 0o644)
|
||||||
|
self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
|
||||||
|
self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||||
|
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||||
|
self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
|
||||||
|
|
||||||
|
def test_an_unchanged_dropin_is_not_reloaded(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||||
|
|
||||||
|
def test_a_dropin_that_says_yes_or_no_is_refused(self):
|
||||||
|
for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
|
||||||
|
b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
|
||||||
|
def edit(files, bad=bad):
|
||||||
|
for e in files:
|
||||||
|
if e["path"] == self.DEST:
|
||||||
|
e["content_b64"] = base64.b64encode(bad).decode()
|
||||||
|
data, sha = edited_bundle(edit)
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertNotEqual(rc, 0, bad)
|
||||||
|
self.assertNotIn(self.DEST, box.files, bad)
|
||||||
|
|
||||||
|
# Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
|
||||||
|
# said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
|
||||||
|
# after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
|
||||||
|
# test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
|
||||||
|
# fails — seen failing 2026-10-22 night.
|
||||||
|
def _sshd_T(self, box, value):
|
||||||
|
orig = box.host
|
||||||
|
def host(argv, timeout=600, stdin=None):
|
||||||
|
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
|
||||||
|
box.calls.append(argv)
|
||||||
|
return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
|
||||||
|
return orig(argv, timeout, stdin)
|
||||||
|
box.host = host
|
||||||
|
|
||||||
|
def test_the_effective_value_is_checked(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
self._sshd_T(box, "without-password")
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
|
||||||
|
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
|
||||||
|
|
||||||
|
def test_a_dropin_not_in_force_is_reported(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
|
||||||
|
self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
|
||||||
|
|
||||||
|
def test_an_undone_bundle_reloads_sshd(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
box.visudo_fail = True # the self-check fails AFTER the install → undo
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 3, rep)
|
||||||
|
self.assertNotIn(self.DEST, box.files)
|
||||||
|
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
|
||||||
|
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
|
||||||
|
|
||||||
|
class ByoKeepsItsSSH(unittest.TestCase):
|
||||||
|
"""D43 = A (operator 2026-10-24, R-1257): the root-key-only drop-in goes ONLY to a box whose root-owned install record
|
||||||
|
says appliance; on a BYO host nothing is written and a copy WE wrote is removed; a box with no record is left as it
|
||||||
|
is. X3: the in-force check asks sshd about each of the host's own LAN addresses. X4: a failing `sshd -t` puts back
|
||||||
|
only the SSH step. RED-PROOF (2026-10-24, felhom-agent f5c3d77 + these tests): with the drop-in's policy back to
|
||||||
|
`replace`, test_a_byo_host_gets_no_dropin and test_our_copy_comes_off_a_byo_host fail; with the probe back to
|
||||||
|
192.0.2.1 only, test_a_match_block_for_the_lan_is_not_in_force fails; with `sshd -t` back to `must`,
|
||||||
|
test_a_failing_sshd_t_undoes_only_the_ssh_step fails (rc 3, the whole bundle undone)."""
|
||||||
|
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
|
||||||
|
OURS = (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes()
|
||||||
|
|
||||||
|
def test_a_byo_host_gets_no_dropin(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha), install_mode="byo")
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertNotIn(self.DEST, box.files)
|
||||||
|
self.assertNotIn(["/usr/sbin/sshd", "-t"], box.calls)
|
||||||
|
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||||
|
self.assertIn(self.DEST, rep["bundle"]["skipped"])
|
||||||
|
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
|
||||||
|
self.assertEqual(rec["files"][self.DEST], "skipped")
|
||||||
|
|
||||||
|
def test_our_copy_comes_off_a_byo_host(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha), install_mode="byo")
|
||||||
|
box.put(self.DEST, self.OURS, 0o644) # an earlier bundle wrote it (before D43)
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertNotIn(self.DEST, box.files)
|
||||||
|
self.assertEqual(rep["bundle"]["removed"], [self.DEST])
|
||||||
|
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files),
|
||||||
|
"the previous copy must be kept for the undo")
|
||||||
|
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||||
|
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], "removed")
|
||||||
|
|
||||||
|
def test_the_owners_own_file_stays_on_a_byo_host(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha), install_mode="byo")
|
||||||
|
box.put(self.DEST, b"# my own\nPermitRootLogin yes\n", 0o644)
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(box.files[self.DEST], b"# my own\nPermitRootLogin yes\n")
|
||||||
|
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||||
|
|
||||||
|
def test_a_box_with_no_record_is_left_as_it_is(self):
|
||||||
|
for setup in ("none", "agent-writable"):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha), install_mode=None if setup == "none" else "byo")
|
||||||
|
if setup == "agent-writable":
|
||||||
|
box.modes[osapply.INSTALL_STATE] = 0o666 # proves nothing
|
||||||
|
box.put(self.DEST, self.OURS, 0o644)
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(box.files.get(self.DEST), self.OURS, setup)
|
||||||
|
self.assertIn(self.DEST, rep["bundle"]["skipped"], setup)
|
||||||
|
|
||||||
|
def _fake(self, box, addrs=("192.168.0.50",), match=None, sshd_t_rc=0):
|
||||||
|
orig = box.host
|
||||||
|
def host(argv, timeout=600, stdin=None):
|
||||||
|
if argv[:3] == ["ip", "-4", "-o"]:
|
||||||
|
box.calls.append(argv)
|
||||||
|
return 0, "".join(f"2: vmbr0 inet {a}/24 brd x scope global vmbr0\\ valid_lft forever\n" for a in addrs), ""
|
||||||
|
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
|
||||||
|
box.calls.append(argv)
|
||||||
|
val = "yes" if match and f"addr={match}" in argv[-1] else "without-password"
|
||||||
|
return 0, f"port 22\npermitrootlogin {val}\n", ""
|
||||||
|
if argv == ["/usr/sbin/sshd", "-t"]:
|
||||||
|
box.calls.append(argv)
|
||||||
|
return (sshd_t_rc, "", "/etc/ssh/sshd_config line 9: Bad configuration option") if sshd_t_rc else (0, "", "")
|
||||||
|
return orig(argv, timeout, stdin)
|
||||||
|
box.host = host
|
||||||
|
|
||||||
|
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
|
||||||
|
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
|
||||||
|
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
|
||||||
|
|
||||||
|
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, None, install_mode="byo")
|
||||||
|
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||||
|
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertNotIn(self.DEST, box.files)
|
||||||
|
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
|
||||||
|
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
|
||||||
|
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
|
||||||
|
|
||||||
|
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
|
||||||
|
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, None, install_mode=mode)
|
||||||
|
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||||
|
body = content if content is not None else self.OURS
|
||||||
|
box.put(self.DEST, body, 0o644)
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(box.files.get(self.DEST), body, mode)
|
||||||
|
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
|
||||||
|
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
|
||||||
|
|
||||||
|
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, None, install_mode="byo")
|
||||||
|
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
|
||||||
|
box.put(self.DEST, self.OURS, 0o644)
|
||||||
|
self._fake(box, sshd_t_rc=1)
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(box.files.get(self.DEST), self.OURS)
|
||||||
|
self.assertEqual(rep["bundle_policy"]["removed"], [])
|
||||||
|
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
|
||||||
|
|
||||||
|
def test_a_match_block_for_the_lan_is_not_in_force(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"), match="192.168.0.50")
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
probed = [c[-1] for c in box.calls if c[:2] == ["/usr/sbin/sshd", "-T"]]
|
||||||
|
self.assertTrue(any("addr=192.168.0.50" in p for p in probed) and any("addr=10.77.0.4" in p for p in probed), probed)
|
||||||
|
self.assertFalse(any("addr=192.0.2.1" in p for p in probed), probed)
|
||||||
|
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
|
||||||
|
self.assertTrue(any("NOT in force for 192.168.0.50" in l for l in box.logs), box.logs)
|
||||||
|
|
||||||
|
def test_the_lan_check_passes_when_every_address_is_key_only(self):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"))
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
|
||||||
|
|
||||||
|
def test_a_failing_sshd_t_undoes_only_the_ssh_step(self):
|
||||||
|
for before in (None, b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982 old\nPermitRootLogin prohibit-password\n"):
|
||||||
|
data, sha = real_bundle()
|
||||||
|
box = Box(data, signed_job(sha))
|
||||||
|
if before is not None:
|
||||||
|
box.put(self.DEST, before, 0o644)
|
||||||
|
self._fake(box, sshd_t_rc=255)
|
||||||
|
rc, rep = run(box)
|
||||||
|
self.assertEqual(rc, 0, rep) # the bundle stays
|
||||||
|
self.assertEqual(box.files.get(self.DEST), before, "only the SSH step is put back")
|
||||||
|
self.assertIn("/usr/local/sbin/felhom-os-apply", box.files, "the rest of the bundle stays")
|
||||||
|
self.assertIn("sshd -t failed", rep["bundle"].get("sshd_step_undone", ""), rep)
|
||||||
|
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, "sshd never ran the refused state")
|
||||||
|
want = "absent" if before is None else hashlib.sha256(before).hexdigest()
|
||||||
|
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], want)
|
||||||
|
|
||||||
|
|
||||||
class TrustBootstrap(unittest.TestCase):
|
class TrustBootstrap(unittest.TestCase):
|
||||||
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
|
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
|
||||||
data, sha = real_bundle()
|
data, sha = real_bundle()
|
||||||
|
|||||||
@@ -833,6 +833,14 @@ class DockerLane(unittest.TestCase):
|
|||||||
def test_expired_signed_job_is_refused(self):
|
def test_expired_signed_job_is_refused(self):
|
||||||
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
|
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
|
||||||
|
|
||||||
|
def test_a_window_longer_than_24h_is_refused(self):
|
||||||
|
# R-1280. RED-PROOF: drop the MAX_WINDOW_S check → the job (current, signed, 24 h + 1 s) runs.
|
||||||
|
self.refused(docker_fake(signed=signed_job(issued="2026-10-03T11:59:59Z", expires="2026-10-04T12:00:00Z")), "R3")
|
||||||
|
|
||||||
|
def test_a_window_of_24h_is_accepted(self):
|
||||||
|
rc, rep = run(docker_fake(signed=signed_job(issued="2026-10-03T12:30:00Z", expires="2026-10-04T12:30:00Z")))
|
||||||
|
self.assertNotEqual((rep.get("refused") or {}).get("code"), "R3", rep)
|
||||||
|
|
||||||
def test_replayed_signed_job_is_refused(self):
|
def test_replayed_signed_job_is_refused(self):
|
||||||
f = docker_fake(signed=signed_job())
|
f = docker_fake(signed=signed_job())
|
||||||
f.nonces = {"n1": f.clock + 600}
|
f.nonces = {"n1": f.clock + 600}
|
||||||
|
|||||||
@@ -22,6 +22,8 @@ var (
|
|||||||
ErrExpired = errors.New("authz: op expired")
|
ErrExpired = errors.New("authz: op expired")
|
||||||
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
|
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
|
||||||
ErrNotYetValid = errors.New("authz: op not yet valid")
|
ErrNotYetValid = errors.New("authz: op not yet valid")
|
||||||
|
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
|
||||||
|
ErrWindowTooLong = errors.New("authz: op validity window too long")
|
||||||
// ErrReplay: the nonce was already recorded in the window.
|
// ErrReplay: the nonce was already recorded in the window.
|
||||||
ErrReplay = errors.New("authz: replay (nonce already seen)")
|
ErrReplay = errors.New("authz: replay (nonce already seen)")
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
|
|||||||
// the validity window stays an honest upper bound.
|
// the validity window stays an honest upper bound.
|
||||||
const DefaultClockSkew = 2 * time.Minute
|
const DefaultClockSkew = 2 * time.Minute
|
||||||
|
|
||||||
|
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
|
||||||
|
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
|
||||||
|
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
|
||||||
|
const MaxWindow = 24 * time.Hour
|
||||||
|
|
||||||
// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
|
// KeyRole tags a pinned operator key (doc 04 §3 two-key model).
|
||||||
type KeyRole string
|
type KeyRole string
|
||||||
|
|
||||||
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
|
|||||||
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
|
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
|
||||||
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
|
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
|
||||||
}
|
}
|
||||||
|
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
|
||||||
|
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
|
||||||
|
}
|
||||||
if now.After(op.ExpiresAt) {
|
if now.After(op.ExpiresAt) {
|
||||||
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
|
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
|
|||||||
rejects(t, err, ErrTarget)
|
rejects(t, err, ErrTarget)
|
||||||
})
|
})
|
||||||
|
|
||||||
|
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
|
||||||
|
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
|
||||||
|
t.Run("window longer than 24 h", func(t *testing.T) {
|
||||||
|
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
|
||||||
|
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
|
||||||
|
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
|
||||||
|
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
|
||||||
|
_, err := v.Verify(blob, sig)
|
||||||
|
rejects(t, err, ErrWindowTooLong)
|
||||||
|
})
|
||||||
|
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
|
||||||
|
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
|
||||||
|
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
|
||||||
|
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
|
||||||
|
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
|
||||||
|
if _, err := v.Verify(blob, sig); err != nil {
|
||||||
|
t.Fatalf("a 24 h window must verify: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
t.Run("expired", func(t *testing.T) {
|
t.Run("expired", func(t *testing.T) {
|
||||||
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
|
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
|
||||||
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
|
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
|
||||||
|
|||||||
@@ -34,6 +34,7 @@ type Config struct {
|
|||||||
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
|
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
|
||||||
GuestNet GuestNetConfig `json:"guest_net"`
|
GuestNet GuestNetConfig `json:"guest_net"`
|
||||||
DiskTrim DiskTrimConfig `json:"disk_trim"`
|
DiskTrim DiskTrimConfig `json:"disk_trim"`
|
||||||
|
LineTest LineTestConfig `json:"line_test"`
|
||||||
OOB OOBConfig `json:"oob"`
|
OOB OOBConfig `json:"oob"`
|
||||||
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
|
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
|
||||||
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
|
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
|
||||||
@@ -150,6 +151,16 @@ type DiskTrimConfig struct {
|
|||||||
// Enabled reports whether the weekly trim should run.
|
// Enabled reports whether the weekly trim should run.
|
||||||
func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
|
func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
|
||||||
|
|
||||||
|
// LineTestConfig configures the R-1247 weekly upload line test (internal/linetest). DEFAULT-ON: it reaches only ep0's
|
||||||
|
// tunnel-only iperf3 server, so it runs only on a box that already has an operational offsite tunnel (operator
|
||||||
|
// ruling 2026-10-24 07:26: every box with a tunnel measures its line weekly).
|
||||||
|
type LineTestConfig struct {
|
||||||
|
Disable bool `json:"disable"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Enabled reports whether the weekly line test should run.
|
||||||
|
func (l LineTestConfig) Enabled() bool { return !l.Disable }
|
||||||
|
|
||||||
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
|
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
|
||||||
//
|
//
|
||||||
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
|
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
|
||||||
|
|||||||
@@ -104,6 +104,12 @@ type GuestDiskTrimReporter interface {
|
|||||||
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
|
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// LineTestReporter is the R-1247 seam the weekly line test plugs into (hub does not import linetest). nil → no
|
||||||
|
// line_test stanza.
|
||||||
|
type LineTestReporter interface {
|
||||||
|
LineTestStatus(ctx context.Context) *LineTestStatus
|
||||||
|
}
|
||||||
|
|
||||||
// Collector builds a HostReport from read-only sources. All deps are behind narrow
|
// Collector builds a HostReport from read-only sources. All deps are behind narrow
|
||||||
// interfaces for unit testing.
|
// interfaces for unit testing.
|
||||||
type Collector struct {
|
type Collector struct {
|
||||||
@@ -124,6 +130,7 @@ type Collector struct {
|
|||||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||||
|
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
|
||||||
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
|
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
|
||||||
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
|
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
|
||||||
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
|
||||||
@@ -259,6 +266,12 @@ func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector
|
|||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetLineTestReporter wires the R-1247 weekly line test as a report source (nil-safe → stanza omitted).
|
||||||
|
func (c *Collector) SetLineTestReporter(r LineTestReporter) *Collector {
|
||||||
|
c.lineTest = r
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
|
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
|
||||||
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
|
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
|
||||||
c.deliberateOff = f
|
c.deliberateOff = f
|
||||||
@@ -427,6 +440,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
|||||||
if c.diskTrim != nil {
|
if c.diskTrim != nil {
|
||||||
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
|
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
|
||||||
}
|
}
|
||||||
|
// R-1247: the last weekly line test (nil reporter = not wired → stanza omitted).
|
||||||
|
if c.lineTest != nil {
|
||||||
|
report.LineTest = c.lineTest.LineTestStatus(ctx)
|
||||||
|
}
|
||||||
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
|
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
|
||||||
if c.foreignKey != nil {
|
if c.foreignKey != nil {
|
||||||
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
|
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
|
||||||
|
|||||||
+23
-1
@@ -129,6 +129,11 @@ type HostReport struct {
|
|||||||
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
|
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
|
||||||
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
|
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
|
||||||
|
|
||||||
|
// LineTest is the weekly upload line test stanza (R-1247; operator ruling 2026-10-24): the schedule, the last
|
||||||
|
// measured upload line (best of the runs, Mbit/s, as ep0's iperf3 server counted it) and the last attempt. Present
|
||||||
|
// only when the job is wired. No secret.
|
||||||
|
LineTest *LineTestStatus `json:"line_test,omitempty"`
|
||||||
|
|
||||||
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
|
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
|
||||||
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
|
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
|
||||||
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
|
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
|
||||||
@@ -232,6 +237,19 @@ type GuestDiskTrimStatus struct {
|
|||||||
Guests []GuestDiskTrim `json:"guests,omitempty"`
|
Guests []GuestDiskTrim `json:"guests,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// LineTestStatus is the R-1247 weekly upload line test stanza. `last_ok_at` + `mbps` are the last SUCCESSFUL
|
||||||
|
// measurement ("" / 0 = never); `last_attempt_at` + `last_error` the last attempt, whatever its verdict — never read
|
||||||
|
// the attempt time as a result.
|
||||||
|
type LineTestStatus struct {
|
||||||
|
Schedule string `json:"schedule"`
|
||||||
|
LastOKAt string `json:"last_ok_at,omitempty"`
|
||||||
|
Mbps float64 `json:"mbps,omitempty"`
|
||||||
|
RunsMbps []float64 `json:"runs_mbps,omitempty"`
|
||||||
|
Server string `json:"server,omitempty"`
|
||||||
|
LastAttemptAt string `json:"last_attempt_at,omitempty"`
|
||||||
|
LastError string `json:"last_error,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
|
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
|
||||||
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
|
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
|
||||||
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
|
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
|
||||||
@@ -292,7 +310,11 @@ type WireguardStatus struct {
|
|||||||
Registered bool `json:"registered"` // the registration marker exists
|
Registered bool `json:"registered"` // the registration marker exists
|
||||||
Active bool `json:"active"` // wg-quick@wg-felhom is-active
|
Active bool `json:"active"` // wg-quick@wg-felhom is-active
|
||||||
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
|
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
|
||||||
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
|
// HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
|
||||||
|
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
|
||||||
|
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
|
||||||
|
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
|
||||||
|
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
|
||||||
}
|
}
|
||||||
|
|
||||||
// SystemInfo is the `system` stanza (see HostReport.System).
|
// SystemInfo is the `system` stanza (see HostReport.System).
|
||||||
|
|||||||
@@ -0,0 +1,294 @@
|
|||||||
|
package linetest
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/binary"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"strconv"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A minimal iperf3 CLIENT for one TCP upload test (client → server), speaking the iperf3 control protocol (3.x) to the
|
||||||
|
// stock `iperf3 --server` on ep0 (operator ruling 2026-10-24: the agent speaks iperf3 itself, nothing is installed on
|
||||||
|
// the host). The number that counts is the SERVER's: the bytes it received, from its results JSON — bytes the client
|
||||||
|
// handed to its socket include what still sat in the send buffer when the timer ended.
|
||||||
|
//
|
||||||
|
// The protocol, as the server drives it (one state byte at a time on the control connection):
|
||||||
|
//
|
||||||
|
// client → 37-byte cookie
|
||||||
|
// server → PARAM_EXCHANGE(9) client → params JSON (4-byte big-endian length + bytes)
|
||||||
|
// server → CREATE_STREAMS(10) client opens N data connections, each starts with the same cookie
|
||||||
|
// server → TEST_START(1), TEST_RUNNING(2) client sends for the test time, then → TEST_END(4)
|
||||||
|
// server → EXCHANGE_RESULTS(13) client → its results JSON; server → its results JSON
|
||||||
|
// server → DISPLAY_RESULTS(14) client → IPERF_DONE(16)
|
||||||
|
//
|
||||||
|
// A server already running a test answers ACCESS_DENIED(-1) and closes: that is ErrBusy (another box is testing; try
|
||||||
|
// again later the same day). SERVER_ERROR(-3) / SERVER_TERMINATE(-2) end the test with an error.
|
||||||
|
|
||||||
|
const (
|
||||||
|
stParamExchange = 9
|
||||||
|
stCreateStreams = 10
|
||||||
|
stTestStart = 1
|
||||||
|
stTestRunning = 2
|
||||||
|
stTestEnd = 4
|
||||||
|
stExchangeResults = 13
|
||||||
|
stDisplayResults = 14
|
||||||
|
stIperfDone = 16
|
||||||
|
stAccessDenied = -1
|
||||||
|
stServerTerminate = -2
|
||||||
|
stServerError = -3
|
||||||
|
|
||||||
|
cookieSize = 37 // 36 characters + NUL
|
||||||
|
blockSize = 128 * 1024
|
||||||
|
maxJSONBytes = 1 << 20
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrBusy: the server is running another client's test.
|
||||||
|
var ErrBusy = errors.New("iperf3 server busy")
|
||||||
|
|
||||||
|
// Result is one upload test.
|
||||||
|
type Result struct {
|
||||||
|
Mbps float64 // the server's received bits per second / 1e6
|
||||||
|
ServerBytes int64 // bytes the server received, all streams
|
||||||
|
ClientBytes int64 // bytes the client wrote (diagnostics only)
|
||||||
|
ServerSeconds float64 // the server's measured duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// Dialer opens a TCP connection (net.Dialer in production; a seam for tests).
|
||||||
|
type Dialer interface {
|
||||||
|
DialContext(ctx context.Context, network, address string) (net.Conn, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Client runs one test. The whole run is bounded by ctx; Run sets every connection's deadline from it.
|
||||||
|
type Client struct {
|
||||||
|
Dialer Dialer
|
||||||
|
Streams int // parallel streams (-P)
|
||||||
|
Time time.Duration // test time (-t)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c Client) dialer() Dialer {
|
||||||
|
if c.Dialer != nil {
|
||||||
|
return c.Dialer
|
||||||
|
}
|
||||||
|
return &net.Dialer{Timeout: 5 * time.Second}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newCookie() ([]byte, error) {
|
||||||
|
const alphabet = "abcdefghijklmnopqrstuvwxyz234567"
|
||||||
|
b := make([]byte, cookieSize)
|
||||||
|
if _, err := rand.Read(b[:cookieSize-1]); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
for i := 0; i < cookieSize-1; i++ {
|
||||||
|
b[i] = alphabet[int(b[i])%len(alphabet)]
|
||||||
|
}
|
||||||
|
b[cookieSize-1] = 0
|
||||||
|
return b, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSON(w io.Writer, v any) error {
|
||||||
|
data, err := json.Marshal(v)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
var hdr [4]byte
|
||||||
|
binary.BigEndian.PutUint32(hdr[:], uint32(len(data)))
|
||||||
|
if _, err := w.Write(append(hdr[:], data...)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func readJSON(r io.Reader, v any) error {
|
||||||
|
var hdr [4]byte
|
||||||
|
if _, err := io.ReadFull(r, hdr[:]); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
n := binary.BigEndian.Uint32(hdr[:])
|
||||||
|
if n == 0 || n > maxJSONBytes {
|
||||||
|
return fmt.Errorf("iperf3: results JSON of %d bytes refused", n)
|
||||||
|
}
|
||||||
|
buf := make([]byte, n)
|
||||||
|
if _, err := io.ReadFull(r, buf); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return json.Unmarshal(buf, v)
|
||||||
|
}
|
||||||
|
|
||||||
|
type streamResult struct {
|
||||||
|
ID int `json:"id"`
|
||||||
|
Bytes int64 `json:"bytes"`
|
||||||
|
Retransmits int64 `json:"retransmits"`
|
||||||
|
Jitter float64 `json:"jitter"`
|
||||||
|
Errors int64 `json:"errors"`
|
||||||
|
OmittedErrors int64 `json:"omitted_errors"`
|
||||||
|
Packets int64 `json:"packets"`
|
||||||
|
OmittedPackets int64 `json:"omitted_packets"`
|
||||||
|
StartTime float64 `json:"start_time"`
|
||||||
|
EndTime float64 `json:"end_time"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type results struct {
|
||||||
|
CPUTotal float64 `json:"cpu_util_total"`
|
||||||
|
CPUUser float64 `json:"cpu_util_user"`
|
||||||
|
CPUSystem float64 `json:"cpu_util_system"`
|
||||||
|
SenderHasRetransmits int `json:"sender_has_retransmits"`
|
||||||
|
CongestionUsed string `json:"congestion_used,omitempty"`
|
||||||
|
Streams []streamResult `json:"streams"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// streamIDs are the ids iperf3 gives its streams: the first is 1, every later one is (streams so far) + 2 — so 1, 3,
|
||||||
|
// 4, 5 … (iperf_add_stream). The server matches our results by these ids.
|
||||||
|
func streamIDs(n int) []int {
|
||||||
|
ids := make([]int, n)
|
||||||
|
for i := range ids {
|
||||||
|
if i == 0 {
|
||||||
|
ids[i] = 1
|
||||||
|
} else {
|
||||||
|
ids[i] = i + 2
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run performs one upload test to addr ("10.77.0.1:5201").
|
||||||
|
func (c Client) Run(ctx context.Context, addr string) (Result, error) {
|
||||||
|
var res Result
|
||||||
|
streams, dur := c.Streams, c.Time
|
||||||
|
if streams < 1 {
|
||||||
|
streams = 1
|
||||||
|
}
|
||||||
|
if dur <= 0 {
|
||||||
|
dur = 10 * time.Second
|
||||||
|
}
|
||||||
|
deadline, ok := ctx.Deadline()
|
||||||
|
if !ok {
|
||||||
|
return res, errors.New("iperf3: Run needs a context with a deadline")
|
||||||
|
}
|
||||||
|
cookie, err := newCookie()
|
||||||
|
if err != nil {
|
||||||
|
return res, err
|
||||||
|
}
|
||||||
|
ctrl, err := c.dialer().DialContext(ctx, "tcp", addr)
|
||||||
|
if err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: connect: %w", err)
|
||||||
|
}
|
||||||
|
defer ctrl.Close()
|
||||||
|
ctrl.SetDeadline(deadline)
|
||||||
|
if _, err := ctrl.Write(cookie); err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: cookie: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var data []net.Conn
|
||||||
|
defer func() {
|
||||||
|
for _, d := range data {
|
||||||
|
d.Close()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
var sent atomic.Int64
|
||||||
|
var server results
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for {
|
||||||
|
var st [1]byte
|
||||||
|
if _, err := io.ReadFull(ctrl, st[:]); err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: control read: %w", err)
|
||||||
|
}
|
||||||
|
switch int8(st[0]) {
|
||||||
|
case stParamExchange:
|
||||||
|
params := map[string]any{"tcp": true, "omit": 0, "time": int(dur / time.Second), "num": 0, "blockcount": 0,
|
||||||
|
"parallel": streams, "len": blockSize, "pacing_timer": 1000, "client_version": "3.18"}
|
||||||
|
if err := writeJSON(ctrl, params); err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: params: %w", err)
|
||||||
|
}
|
||||||
|
case stCreateStreams:
|
||||||
|
for i := 0; i < streams; i++ {
|
||||||
|
d, err := c.dialer().DialContext(ctx, "tcp", addr)
|
||||||
|
if err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: data stream %d: %w", i+1, err)
|
||||||
|
}
|
||||||
|
d.SetDeadline(deadline)
|
||||||
|
data = append(data, d)
|
||||||
|
if _, err := d.Write(cookie); err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: data stream %d cookie: %w", i+1, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case stTestStart:
|
||||||
|
case stTestRunning:
|
||||||
|
stopAt := time.Now().Add(dur)
|
||||||
|
if stopAt.After(deadline) {
|
||||||
|
return res, errors.New("iperf3: no time left for the test inside the deadline")
|
||||||
|
}
|
||||||
|
buf := make([]byte, blockSize)
|
||||||
|
for _, d := range data {
|
||||||
|
d.SetWriteDeadline(stopAt)
|
||||||
|
wg.Add(1)
|
||||||
|
go func(d net.Conn) {
|
||||||
|
defer wg.Done()
|
||||||
|
for {
|
||||||
|
n, err := d.Write(buf)
|
||||||
|
sent.Add(int64(n))
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}(d)
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if _, err := ctrl.Write([]byte{stTestEnd}); err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: test end: %w", err)
|
||||||
|
}
|
||||||
|
case stExchangeResults:
|
||||||
|
ids := streamIDs(len(data))
|
||||||
|
mine := results{SenderHasRetransmits: -1, CongestionUsed: "cubic"}
|
||||||
|
per := sent.Load() / int64(max(len(data), 1))
|
||||||
|
for _, id := range ids {
|
||||||
|
mine.Streams = append(mine.Streams, streamResult{ID: id, Bytes: per, Retransmits: -1, EndTime: dur.Seconds()})
|
||||||
|
}
|
||||||
|
if err := writeJSON(ctrl, mine); err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: send results: %w", err)
|
||||||
|
}
|
||||||
|
if err := readJSON(ctrl, &server); err != nil {
|
||||||
|
return res, fmt.Errorf("iperf3: read server results: %w", err)
|
||||||
|
}
|
||||||
|
case stDisplayResults:
|
||||||
|
ctrl.Write([]byte{stIperfDone})
|
||||||
|
return summarize(server, sent.Load(), dur)
|
||||||
|
case stAccessDenied:
|
||||||
|
return res, ErrBusy
|
||||||
|
case stServerTerminate, stServerError:
|
||||||
|
var codes [8]byte
|
||||||
|
io.ReadFull(ctrl, codes[:])
|
||||||
|
return res, fmt.Errorf("iperf3: server error (state %d, iperf errno %d, errno %d)", int8(st[0]),
|
||||||
|
int32(binary.BigEndian.Uint32(codes[:4])), int32(binary.BigEndian.Uint32(codes[4:])))
|
||||||
|
default:
|
||||||
|
return res, fmt.Errorf("iperf3: unexpected state %s", strconv.Itoa(int(int8(st[0]))))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// summarize turns the server's results into a Result. No stream bytes is an error, never a zero line.
|
||||||
|
func summarize(server results, clientBytes int64, dur time.Duration) (Result, error) {
|
||||||
|
res := Result{ClientBytes: clientBytes}
|
||||||
|
for _, s := range server.Streams {
|
||||||
|
res.ServerBytes += s.Bytes
|
||||||
|
if s.EndTime > res.ServerSeconds {
|
||||||
|
res.ServerSeconds = s.EndTime
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if res.ServerBytes <= 0 {
|
||||||
|
return res, errors.New("iperf3: the server reported no received bytes")
|
||||||
|
}
|
||||||
|
secs := res.ServerSeconds
|
||||||
|
if secs <= 0 {
|
||||||
|
secs = dur.Seconds()
|
||||||
|
}
|
||||||
|
res.Mbps = float64(res.ServerBytes) * 8 / secs / 1e6
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,541 @@
|
|||||||
|
// Package linetest is the weekly upload line-speed test (R-1247; operator ruling 2026-10-24 07:26).
|
||||||
|
//
|
||||||
|
// Why: the controller's upload watch and qBittorrent's default limit (50 % of the line, D41) need the household's
|
||||||
|
// upload line. The passive reading — the nightly off-site push — is only a low floor (2026-10-24 night: demo-hp „at
|
||||||
|
// least 10 Mbit/s"; this test read 90.7 on the same line). So once a week the HOST (the tunnel lives on the host)
|
||||||
|
// runs a 10-second iperf3 upload to ep0's tunnel-only iperf3 server (felhom.eu scripts/ep0-iperf3). No file is left
|
||||||
|
// anywhere and nothing counts against the household's storage.
|
||||||
|
//
|
||||||
|
// The rule, each part pinned by a test in linetest_test.go:
|
||||||
|
// - Weekly: due from Tuesday 10:00 local until one measurement succeeded since then (a box that was off catches up
|
||||||
|
// on a later day, inside the window).
|
||||||
|
// - Daytime only: a test starts only 10:00–15:59 local — never in the night chain's window (01:00–06:59), and each
|
||||||
|
// host starts from its own minute (SlotMinute, spread over 10:00–13:59) so the fleet does not test at once.
|
||||||
|
// - Never beside a backup, a restore-test, an OS update or a trim: the test holds the host-wide one-heavy-op gate.
|
||||||
|
// - Only after QuietMinutes of quiet uplink: every one-minute transmit rate of the host's physical uplink below
|
||||||
|
// QuietMbps — so neither the family's upload nor a restore is measured against itself.
|
||||||
|
// - Up to RunsPerMeasurement runs of TestTime each; the highest counts. A run is bounded by RunTimeout.
|
||||||
|
// - „Server busy" (another box is testing) → tried again at the next tick the same day; at most MaxTriesPerDay.
|
||||||
|
// - No tunnel (no operational wg-felhom with a server address) → no test and no value; the controller then keeps its
|
||||||
|
// passive floor and sets no qBittorrent limit.
|
||||||
|
// - The last result is persisted, so a restart neither loses it nor re-tests.
|
||||||
|
// - A box whose torrent app seeds all day is never quiet for 10 minutes (R-1250). For it the controller holds the
|
||||||
|
// app's upload limit very low and asks for a test on request (RunNow): every rule above except the 10 quiet
|
||||||
|
// minutes still holds, the quiet is checked over RequestQuietWindow instead, and the runs are RequestRunGap
|
||||||
|
// apart, so the whole request ends in about 55 s.
|
||||||
|
package linetest
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"hash/fnv"
|
||||||
|
"log/slog"
|
||||||
|
"net/netip"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
Weekday = time.Tuesday
|
||||||
|
StartHour = 10 // first eligible local hour (inclusive)
|
||||||
|
EndHour = 16 // first NOT-eligible local hour: the last start is 15:59
|
||||||
|
SlotSpreadMinutes = 240
|
||||||
|
QuietMinutes = 10
|
||||||
|
QuietMbps = 2.0
|
||||||
|
RunsPerMeasurement = 3
|
||||||
|
TestTime = 10 * time.Second
|
||||||
|
Streams = 4
|
||||||
|
RunTimeout = 25 * time.Second
|
||||||
|
RunGap = 30 * time.Second
|
||||||
|
MaxTriesPerDay = 6
|
||||||
|
ServerPort = 5201
|
||||||
|
TickInterval = time.Minute
|
||||||
|
FirstTickDelay = 5 * time.Minute
|
||||||
|
GateName = "line-test"
|
||||||
|
|
||||||
|
// A test on request (R-1250): the controller holds the app's limit while it waits, so it is short.
|
||||||
|
RequestSettle = 5 * time.Second // the held app's last packets leave before the quiet is read
|
||||||
|
RequestQuietWindow = 10 * time.Second // the uplink must send below QuietMbps over this window
|
||||||
|
RequestRunGap = 5 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// ScheduleText is carried on the host report.
|
||||||
|
const ScheduleText = "weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update"
|
||||||
|
|
||||||
|
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
|
||||||
|
type Gate interface {
|
||||||
|
TryAcquire(what string) (release func(), busy string, ok bool)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Target answers the iperf3 server's tunnel address, ok=false when the tunnel is not operational.
|
||||||
|
type Target func() (netip.Addr, bool)
|
||||||
|
|
||||||
|
// Runner runs one upload test (Client in production).
|
||||||
|
type Runner interface {
|
||||||
|
Run(ctx context.Context, addr string) (Result, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// UplinkTx reads the host's physical-uplink transmit byte counter.
|
||||||
|
type UplinkTx func() (uint64, error)
|
||||||
|
|
||||||
|
// State is what is persisted (line-test.json).
|
||||||
|
type State struct {
|
||||||
|
LastOKAt time.Time `json:"last_ok_at,omitempty"`
|
||||||
|
Mbps float64 `json:"mbps,omitempty"`
|
||||||
|
RunsMbps []float64 `json:"runs_mbps,omitempty"`
|
||||||
|
Server string `json:"server,omitempty"`
|
||||||
|
LastAttemptAt time.Time `json:"last_attempt_at,omitempty"`
|
||||||
|
LastError string `json:"last_error,omitempty"`
|
||||||
|
TriesDay string `json:"tries_day,omitempty"` // local date of TriesToday
|
||||||
|
TriesToday int `json:"tries_today,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type txSample struct {
|
||||||
|
at time.Time
|
||||||
|
tx uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tester is the weekly job.
|
||||||
|
type Tester struct {
|
||||||
|
runner Runner
|
||||||
|
target Target
|
||||||
|
gate Gate
|
||||||
|
uplink UplinkTx
|
||||||
|
statePath string
|
||||||
|
slot int // minutes after StartHour:00 this host may first start
|
||||||
|
logger *slog.Logger
|
||||||
|
loc *time.Location
|
||||||
|
now func() time.Time
|
||||||
|
sleep func(context.Context, time.Duration) bool
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
st State
|
||||||
|
samples []txSample
|
||||||
|
}
|
||||||
|
|
||||||
|
// New builds the job and loads the persisted state (missing = empty; corrupt = logged, empty — the cost is one extra
|
||||||
|
// test, never a lost value elsewhere).
|
||||||
|
func New(runner Runner, target Target, gate Gate, uplink UplinkTx, hostID, statePath string, logger *slog.Logger) *Tester {
|
||||||
|
if logger == nil {
|
||||||
|
logger = slog.Default()
|
||||||
|
}
|
||||||
|
t := &Tester{runner: runner, target: target, gate: gate, uplink: uplink, statePath: statePath,
|
||||||
|
slot: SlotMinute(hostID), logger: logger, loc: time.Local, now: time.Now, sleep: sleepCtx}
|
||||||
|
t.load()
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
func sleepCtx(ctx context.Context, d time.Duration) bool {
|
||||||
|
tm := time.NewTimer(d)
|
||||||
|
defer tm.Stop()
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return false
|
||||||
|
case <-tm.C:
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SlotMinute spreads hosts over the first SlotSpreadMinutes of the window, stable per host id.
|
||||||
|
func SlotMinute(hostID string) int {
|
||||||
|
h := fnv.New32a()
|
||||||
|
h.Write([]byte(hostID))
|
||||||
|
return int(h.Sum32() % SlotSpreadMinutes)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Tester) load() {
|
||||||
|
data, err := os.ReadFile(t.statePath)
|
||||||
|
if err != nil {
|
||||||
|
if !os.IsNotExist(err) {
|
||||||
|
t.logger.Warn("linetest: state read failed — starting empty", "path", t.statePath, "err", err)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, &t.st); err != nil {
|
||||||
|
t.logger.Warn("linetest: state file corrupt — starting empty", "path", t.statePath, "err", err)
|
||||||
|
t.st = State{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Tester) saveLocked() error {
|
||||||
|
data, err := json.MarshalIndent(t.st, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
tmp := t.statePath + ".tmp"
|
||||||
|
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||||
|
os.Remove(tmp)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return os.Rename(tmp, t.statePath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// EligibleAt reports whether a test may START at local time lt for a host whose slot is slot minutes after
|
||||||
|
// StartHour:00 on the due day. On a catch-up day (anchorDay=false) the slot is not waited for again.
|
||||||
|
func EligibleAt(lt time.Time, slot int, anchorDay bool) bool {
|
||||||
|
h := lt.Hour()
|
||||||
|
if h < StartHour || h >= EndHour {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if anchorDay {
|
||||||
|
mins := (h-StartHour)*60 + lt.Minute()
|
||||||
|
return mins >= slot
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// weekAnchor is the most recent Tuesday StartHour:00 at or before lt.
|
||||||
|
func weekAnchor(lt time.Time) time.Time {
|
||||||
|
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
|
||||||
|
d := lt.AddDate(0, 0, -daysBack)
|
||||||
|
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
|
||||||
|
if a.After(lt) {
|
||||||
|
d = d.AddDate(0, 0, -7)
|
||||||
|
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
|
||||||
|
}
|
||||||
|
return a
|
||||||
|
}
|
||||||
|
|
||||||
|
// Due reports whether a measurement is due at lt given the state.
|
||||||
|
func Due(st State, lt time.Time) bool {
|
||||||
|
return st.LastOKAt.IsZero() || st.LastOKAt.In(lt.Location()).Before(weekAnchor(lt))
|
||||||
|
}
|
||||||
|
|
||||||
|
// sample records the uplink counter (once a minute) and keeps QuietMinutes+1 samples.
|
||||||
|
func (t *Tester) sample() {
|
||||||
|
if t.uplink == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
tx, err := t.uplink()
|
||||||
|
if err != nil {
|
||||||
|
t.logger.Debug("linetest: uplink counter unreadable", "err", err)
|
||||||
|
t.mu.Lock()
|
||||||
|
t.samples = nil // a gap is not quiet
|
||||||
|
t.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.mu.Lock()
|
||||||
|
t.samples = append(t.samples, txSample{at: t.now(), tx: tx})
|
||||||
|
if len(t.samples) > QuietMinutes+1 {
|
||||||
|
t.samples = t.samples[len(t.samples)-(QuietMinutes+1):]
|
||||||
|
}
|
||||||
|
t.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// quiet reports whether the last QuietMinutes minutes each sent below QuietMbps, and the highest minute seen.
|
||||||
|
func quiet(samples []txSample) (bool, float64) {
|
||||||
|
if len(samples) < QuietMinutes+1 {
|
||||||
|
return false, 0
|
||||||
|
}
|
||||||
|
worst := 0.0
|
||||||
|
for i := 1; i < len(samples); i++ {
|
||||||
|
dt := samples[i].at.Sub(samples[i-1].at).Seconds()
|
||||||
|
if dt <= 0 || dt > 3*TickInterval.Seconds() || samples[i].tx < samples[i-1].tx {
|
||||||
|
return false, worst // a gap or a counter reset is not proof of quiet
|
||||||
|
}
|
||||||
|
mbps := float64(samples[i].tx-samples[i-1].tx) * 8 / dt / 1e6
|
||||||
|
if mbps > worst {
|
||||||
|
worst = mbps
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return worst < QuietMbps, worst
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run ticks every TickInterval until ctx ends. It never returns an error: a failed test is a reported fact.
|
||||||
|
func (t *Tester) Run(ctx context.Context) {
|
||||||
|
t.logger.Info("linetest: weekly upload line test starting", "schedule", ScheduleText, "slot_minute", t.slot)
|
||||||
|
if !t.sleep(ctx, FirstTickDelay) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for {
|
||||||
|
t.sample()
|
||||||
|
t.Tick(ctx)
|
||||||
|
if !t.sleep(ctx, TickInterval) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Refusal is why a test on request did not run (RunNow). Code is stable for the controller.
|
||||||
|
type Refusal struct {
|
||||||
|
Code string // not-due | outside-window | tries-used | no-tunnel | heavy-op | not-quiet
|
||||||
|
Msg string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Refusal) Error() string { return "linetest: " + r.Code + ": " + r.Msg }
|
||||||
|
|
||||||
|
// ready applies every rule except the quiet one: due this week, inside the window (the slot on the due day), tries
|
||||||
|
// left today, the tunnel up. It answers the server address and the local day.
|
||||||
|
func (t *Tester) ready() (addr, day string, refusal *Refusal) {
|
||||||
|
lt := t.now().In(t.loc)
|
||||||
|
t.mu.Lock()
|
||||||
|
st := t.st
|
||||||
|
t.mu.Unlock()
|
||||||
|
if !Due(st, lt) {
|
||||||
|
return "", "", &Refusal{"not-due", "measured this week"}
|
||||||
|
}
|
||||||
|
anchor := weekAnchor(lt)
|
||||||
|
anchorDay := lt.YearDay() == anchor.YearDay() && lt.Year() == anchor.Year()
|
||||||
|
if !EligibleAt(lt, t.slot, anchorDay) {
|
||||||
|
return "", "", &Refusal{"outside-window", "a test starts 10:00-15:59 local, from this host's slot on the due day"}
|
||||||
|
}
|
||||||
|
day = lt.Format("2006-01-02")
|
||||||
|
if st.TriesDay == day && st.TriesToday >= MaxTriesPerDay {
|
||||||
|
return "", "", &Refusal{"tries-used", fmt.Sprintf("%d tries today", st.TriesToday)}
|
||||||
|
}
|
||||||
|
ip, ok := t.target()
|
||||||
|
if !ok {
|
||||||
|
return "", "", &Refusal{"no-tunnel", "no operational tunnel"}
|
||||||
|
}
|
||||||
|
return netip.AddrPortFrom(ip, ServerPort).String(), day, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tick is one look.
|
||||||
|
func (t *Tester) Tick(ctx context.Context) {
|
||||||
|
addr, day, refusal := t.ready()
|
||||||
|
if refusal != nil {
|
||||||
|
if refusal.Code == "no-tunnel" {
|
||||||
|
t.logger.Debug("linetest: no operational tunnel — no test")
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
t.mu.Lock()
|
||||||
|
samples := append([]txSample(nil), t.samples...)
|
||||||
|
t.mu.Unlock()
|
||||||
|
if q, worst := quiet(samples); !q {
|
||||||
|
t.logger.Debug("linetest: uplink not quiet — waiting", "worst_minute_mbps", fmt.Sprintf("%.1f", worst), "samples", len(samples))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
release, busy, ok := t.gate.TryAcquire(GateName)
|
||||||
|
if !ok {
|
||||||
|
t.logger.Info("linetest: deferred — a heavy operation is in flight", "busy", busy)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
t.measure(ctx, addr, day, RunGap)
|
||||||
|
}
|
||||||
|
|
||||||
|
// WaitingForQuiet reports that a measurement is due and could start now but the uplink has not been quiet for
|
||||||
|
// QuietMinutes — the case a seeding torrent app makes permanent (R-1250). The controller reads it from
|
||||||
|
// GET /host/line-test and only then holds its app's limit and calls RunNow.
|
||||||
|
func (t *Tester) WaitingForQuiet() bool {
|
||||||
|
if _, _, refusal := t.ready(); refusal != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
t.mu.Lock()
|
||||||
|
samples := append([]txSample(nil), t.samples...)
|
||||||
|
t.mu.Unlock()
|
||||||
|
if len(samples) < QuietMinutes+1 {
|
||||||
|
return false // not yet watched for 10 minutes (a fresh start): not known to be busy, so no hold
|
||||||
|
}
|
||||||
|
q, _ := quiet(samples)
|
||||||
|
return !q
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunNow is a test on request (POST /host/line-test/run, R-1250): every rule of the weekly test except the 10 quiet
|
||||||
|
// minutes, which the caller replaces by holding its own upload down. The uplink must still be quiet over
|
||||||
|
// RequestQuietWindow (after RequestSettle), so a request never measures the family's upload against itself. A
|
||||||
|
// *Refusal says why nothing ran; any other error is a failed measurement (the same as a failed weekly attempt).
|
||||||
|
func (t *Tester) RunNow(ctx context.Context) (mbps float64, runs []float64, err error) {
|
||||||
|
addr, day, refusal := t.ready()
|
||||||
|
if refusal != nil {
|
||||||
|
return 0, nil, refusal
|
||||||
|
}
|
||||||
|
release, busy, ok := t.gate.TryAcquire(GateName)
|
||||||
|
if !ok {
|
||||||
|
return 0, nil, &Refusal{"heavy-op", busy}
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
if t.uplink == nil {
|
||||||
|
return 0, nil, &Refusal{"not-quiet", "no uplink counter"}
|
||||||
|
}
|
||||||
|
if !t.sleep(ctx, RequestSettle) {
|
||||||
|
return 0, nil, ctx.Err()
|
||||||
|
}
|
||||||
|
tx0, err0 := t.uplink()
|
||||||
|
at0 := t.now()
|
||||||
|
if !t.sleep(ctx, RequestQuietWindow) {
|
||||||
|
return 0, nil, ctx.Err()
|
||||||
|
}
|
||||||
|
tx1, err1 := t.uplink()
|
||||||
|
if err0 != nil || err1 != nil || tx1 < tx0 {
|
||||||
|
return 0, nil, &Refusal{"not-quiet", "uplink counter unreadable or reset"}
|
||||||
|
}
|
||||||
|
secs := t.now().Sub(at0).Seconds()
|
||||||
|
if secs <= 0 {
|
||||||
|
secs = RequestQuietWindow.Seconds()
|
||||||
|
}
|
||||||
|
if m := float64(tx1-tx0) * 8 / secs / 1e6; m >= QuietMbps {
|
||||||
|
t.logger.Info("linetest: test on request refused — the uplink is not quiet", "mbps", fmt.Sprintf("%.1f", m))
|
||||||
|
return 0, nil, &Refusal{"not-quiet", fmt.Sprintf("the uplink sent %.1f Mbit/s while held", m)}
|
||||||
|
}
|
||||||
|
t.logger.Info("linetest: test on request (a held torrent app, R-1250)", "server", addr)
|
||||||
|
if err := t.measure(ctx, addr, day, RequestRunGap); err != nil {
|
||||||
|
return 0, nil, err
|
||||||
|
}
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
return t.st.Mbps, append([]float64(nil), t.st.RunsMbps...), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// measure runs the runs, gap apart, and stores the result. It returns nil only when a value was stored.
|
||||||
|
func (t *Tester) measure(ctx context.Context, addr, day string, gap time.Duration) error {
|
||||||
|
start := t.now()
|
||||||
|
var runs []float64
|
||||||
|
var lastErr error
|
||||||
|
busy := false
|
||||||
|
for i := 0; i < RunsPerMeasurement; i++ {
|
||||||
|
if i > 0 && !t.sleep(ctx, gap) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
rctx, cancel := context.WithTimeout(ctx, RunTimeout)
|
||||||
|
r, err := t.runner.Run(rctx, addr)
|
||||||
|
cancel()
|
||||||
|
if errors.Is(err, ErrBusy) {
|
||||||
|
busy, lastErr = true, err
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
lastErr = err
|
||||||
|
t.logger.Warn("linetest: run failed", "run", i+1, "server", addr, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
runs = append(runs, r.Mbps)
|
||||||
|
t.logger.Info(fmt.Sprintf("linetest: run %d: %.1f Mbit/s", i+1, r.Mbps), "server", addr,
|
||||||
|
"server_bytes", r.ServerBytes, "server_seconds", r.ServerSeconds)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
if t.st.TriesDay != day {
|
||||||
|
t.st.TriesDay, t.st.TriesToday = day, 0
|
||||||
|
}
|
||||||
|
t.st.TriesToday++
|
||||||
|
t.st.LastAttemptAt = start.UTC()
|
||||||
|
if len(runs) > 0 {
|
||||||
|
best := runs[0]
|
||||||
|
for _, r := range runs {
|
||||||
|
if r > best {
|
||||||
|
best = r
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.st.LastOKAt, t.st.Mbps, t.st.RunsMbps, t.st.Server, t.st.LastError = start.UTC(), best, runs, addr, ""
|
||||||
|
t.logger.Info(fmt.Sprintf("linetest: upload line measured: %.1f Mbit/s (best of %d)", best, len(runs)),
|
||||||
|
"server", addr, "runs", len(runs), "duration_s", t.now().Sub(start).Seconds())
|
||||||
|
} else {
|
||||||
|
msg := "no run succeeded"
|
||||||
|
if lastErr != nil {
|
||||||
|
msg = lastErr.Error()
|
||||||
|
}
|
||||||
|
if len(msg) > 300 {
|
||||||
|
msg = msg[:300]
|
||||||
|
}
|
||||||
|
t.st.LastError = msg
|
||||||
|
if busy {
|
||||||
|
t.logger.Info("linetest: the test server is busy — trying again later today", "server", addr, "tries_today", t.st.TriesToday)
|
||||||
|
} else {
|
||||||
|
t.logger.Warn("linetest: measurement failed", "server", addr, "tries_today", t.st.TriesToday, "err", msg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := t.saveLocked(); err != nil {
|
||||||
|
t.logger.Warn("linetest: state save failed — the result will not survive a restart", "path", t.statePath, "err", err)
|
||||||
|
}
|
||||||
|
if len(runs) > 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if lastErr == nil {
|
||||||
|
lastErr = errors.New("linetest: no run succeeded")
|
||||||
|
}
|
||||||
|
return lastErr
|
||||||
|
}
|
||||||
|
|
||||||
|
// Value is the measured line (ok=false: never measured).
|
||||||
|
func (t *Tester) Value() (mbps float64, at time.Time, ok bool) {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
if t.st.LastOKAt.IsZero() || t.st.Mbps <= 0 {
|
||||||
|
return 0, time.Time{}, false
|
||||||
|
}
|
||||||
|
return t.st.Mbps, t.st.LastOKAt, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// LineTestStatus implements hub.LineTestReporter: a pure read of the persisted state.
|
||||||
|
func (t *Tester) LineTestStatus(context.Context) *hub.LineTestStatus {
|
||||||
|
t.mu.Lock()
|
||||||
|
defer t.mu.Unlock()
|
||||||
|
out := &hub.LineTestStatus{Schedule: ScheduleText, Mbps: t.st.Mbps, RunsMbps: t.st.RunsMbps, Server: t.st.Server,
|
||||||
|
LastError: t.st.LastError}
|
||||||
|
if !t.st.LastOKAt.IsZero() {
|
||||||
|
out.LastOKAt = t.st.LastOKAt.UTC().Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
if !t.st.LastAttemptAt.IsZero() {
|
||||||
|
out.LastAttemptAt = t.st.LastAttemptAt.UTC().Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── the host's physical uplink counter ──────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// SysUplinkTx reads the transmit bytes of the host's physical uplink under root (normally "/"): the interface of the
|
||||||
|
// default route in /proc/net/route, and — when it is a bridge (vmbr0), as on every Proxmox host — the sum of its
|
||||||
|
// PHYSICAL member ports (those with a `device` link). The bridge's own counter does not see the guests' bridged
|
||||||
|
// traffic, which is the traffic this check exists for.
|
||||||
|
func SysUplinkTx(root string) UplinkTx {
|
||||||
|
return func() (uint64, error) {
|
||||||
|
ifc, err := defaultRouteIface(filepath.Join(root, "proc/net/route"))
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
sys := filepath.Join(root, "sys/class/net")
|
||||||
|
ports := []string{ifc}
|
||||||
|
if ents, err := os.ReadDir(filepath.Join(sys, ifc, "brif")); err == nil {
|
||||||
|
ports = ports[:0]
|
||||||
|
for _, e := range ents {
|
||||||
|
if _, err := os.Stat(filepath.Join(sys, e.Name(), "device")); err == nil {
|
||||||
|
ports = append(ports, e.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(ports) == 0 {
|
||||||
|
return 0, fmt.Errorf("linetest: bridge %s has no physical port", ifc)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var sum uint64
|
||||||
|
for _, p := range ports {
|
||||||
|
b, err := os.ReadFile(filepath.Join(sys, p, "statistics/tx_bytes"))
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
n, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
sum += n
|
||||||
|
}
|
||||||
|
return sum, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func defaultRouteIface(path string) (string, error) {
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
for _, line := range strings.Split(string(b), "\n")[1:] {
|
||||||
|
f := strings.Fields(line)
|
||||||
|
if len(f) >= 8 && f[1] == "00000000" && f[7] == "00000000" {
|
||||||
|
return f[0], nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", errors.New("linetest: no default route")
|
||||||
|
}
|
||||||
@@ -0,0 +1,574 @@
|
|||||||
|
package linetest
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/binary"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/netip"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ── a fake iperf3 server (the server half of the protocol, as `iperf3 --server` drives it) ─────────────────────────
|
||||||
|
|
||||||
|
type fakeServer struct {
|
||||||
|
ln net.Listener
|
||||||
|
busy bool // answer ACCESS_DENIED
|
||||||
|
stall bool // accept the cookie, then never answer
|
||||||
|
recvSeconds float64 // what the server reports as its duration
|
||||||
|
gotIDs []int // the stream ids the client reported
|
||||||
|
gotParams map[string]any
|
||||||
|
mu sync.Mutex
|
||||||
|
received int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeServer(t *testing.T) *fakeServer {
|
||||||
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s := &fakeServer{ln: ln, recvSeconds: 1}
|
||||||
|
t.Cleanup(func() { ln.Close() })
|
||||||
|
go s.serve()
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *fakeServer) addr() string { return s.ln.Addr().String() }
|
||||||
|
|
||||||
|
func wjson(w io.Writer, v any) {
|
||||||
|
b, _ := json.Marshal(v)
|
||||||
|
var h [4]byte
|
||||||
|
binary.BigEndian.PutUint32(h[:], uint32(len(b)))
|
||||||
|
w.Write(append(h[:], b...))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *fakeServer) serve() {
|
||||||
|
ctrl, err := s.ln.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer ctrl.Close()
|
||||||
|
cookie := make([]byte, cookieSize)
|
||||||
|
if _, err := io.ReadFull(ctrl, cookie); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.busy {
|
||||||
|
ctrl.Write([]byte{0xff})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.stall {
|
||||||
|
time.Sleep(5 * time.Second)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ctrl.Write([]byte{stParamExchange})
|
||||||
|
var params map[string]any
|
||||||
|
if readJSON(ctrl, ¶ms) != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.gotParams = params
|
||||||
|
n := int(params["parallel"].(float64))
|
||||||
|
ctrl.Write([]byte{stCreateStreams})
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
var conns []net.Conn
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
d, err := s.ln.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
c := make([]byte, cookieSize)
|
||||||
|
io.ReadFull(d, c)
|
||||||
|
conns = append(conns, d)
|
||||||
|
wg.Add(1)
|
||||||
|
go func(d net.Conn) {
|
||||||
|
defer wg.Done()
|
||||||
|
k, _ := io.Copy(io.Discard, d)
|
||||||
|
s.mu.Lock()
|
||||||
|
s.received += k
|
||||||
|
s.mu.Unlock()
|
||||||
|
}(d)
|
||||||
|
}
|
||||||
|
ctrl.Write([]byte{stTestStart, stTestRunning})
|
||||||
|
var st [1]byte
|
||||||
|
if _, err := io.ReadFull(ctrl, st[:]); err != nil || st[0] != stTestEnd {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ctrl.Write([]byte{stExchangeResults})
|
||||||
|
var theirs results
|
||||||
|
if readJSON(ctrl, &theirs) != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, x := range theirs.Streams {
|
||||||
|
s.gotIDs = append(s.gotIDs, x.ID)
|
||||||
|
}
|
||||||
|
// The server counts what it RECEIVED; report a fixed figure so the test can tell it from the client's own count.
|
||||||
|
wjson(ctrl, results{Streams: []streamResult{{ID: 1, Bytes: 12_500_000, EndTime: s.recvSeconds}}})
|
||||||
|
ctrl.Write([]byte{stDisplayResults})
|
||||||
|
io.ReadFull(ctrl, st[:])
|
||||||
|
for _, d := range conns {
|
||||||
|
d.Close()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientSpeaksTheProtocolAndCountsTheServersBytes(t *testing.T) {
|
||||||
|
s := newFakeServer(t)
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
r, err := Client{Streams: 4, Time: 300 * time.Millisecond}.Run(ctx, s.addr())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run: %v", err)
|
||||||
|
}
|
||||||
|
if r.ServerBytes != 12_500_000 || r.Mbps != 100 {
|
||||||
|
t.Fatalf("result = %+v; want the SERVER's 12.5 MB in 1 s = 100 Mbit/s", r)
|
||||||
|
}
|
||||||
|
if want := []int{1, 3, 4, 5}; len(s.gotIDs) != 4 || s.gotIDs[0] != want[0] || s.gotIDs[1] != want[1] || s.gotIDs[3] != want[3] {
|
||||||
|
t.Fatalf("stream ids sent = %v, want %v (iperf3 numbers its streams 1, 3, 4, 5)", s.gotIDs, want)
|
||||||
|
}
|
||||||
|
if s.gotParams["tcp"] != true || s.gotParams["parallel"].(float64) != 4 {
|
||||||
|
t.Fatalf("params = %v", s.gotParams)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientBusyServerIsErrBusy(t *testing.T) {
|
||||||
|
s := newFakeServer(t)
|
||||||
|
s.busy = true
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if _, err := (Client{Streams: 4, Time: time.Second}).Run(ctx, s.addr()); !errors.Is(err, ErrBusy) {
|
||||||
|
t.Fatalf("err = %v, want ErrBusy", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientHonoursTheDeadlineAgainstAStalledServer(t *testing.T) {
|
||||||
|
s := newFakeServer(t)
|
||||||
|
s.stall = true
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
start := time.Now()
|
||||||
|
_, err := Client{Streams: 4, Time: time.Second}.Run(ctx, s.addr())
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("a stalled server gave no error")
|
||||||
|
}
|
||||||
|
if took := time.Since(start); took > 1500*time.Millisecond {
|
||||||
|
t.Fatalf("Run took %v against a stalled server; the deadline is 400 ms", took)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientRefusesAContextWithoutDeadline(t *testing.T) {
|
||||||
|
if _, err := (Client{}).Run(context.Background(), "127.0.0.1:1"); err == nil {
|
||||||
|
t.Fatal("Run without a deadline must refuse (an unbounded test is the thing RunTimeout exists to prevent)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── the schedule ────────────────────────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
func TestEligibleWindowNeverInTheNightChain(t *testing.T) {
|
||||||
|
loc := time.FixedZone("CET", 3600)
|
||||||
|
start := time.Date(2026, 10, 19, 0, 0, 0, 0, loc) // a Monday
|
||||||
|
for m := 0; m < 7*24*60; m++ {
|
||||||
|
lt := start.Add(time.Duration(m) * time.Minute)
|
||||||
|
for _, anchorDay := range []bool{true, false} {
|
||||||
|
if !EligibleAt(lt, 0, anchorDay) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if h := lt.Hour(); h < StartHour || h >= EndHour || (h >= 1 && h <= 6) {
|
||||||
|
t.Fatalf("eligible at %s — outside 10:00-15:59 or inside the night chain (01:00-06:59)", lt.Format("Mon 15:04"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSlotDelaysTheFirstStartOnTheDueDayOnly(t *testing.T) {
|
||||||
|
loc := time.UTC
|
||||||
|
tue := time.Date(2026, 10, 20, 10, 30, 0, 0, loc)
|
||||||
|
if EligibleAt(tue, 45, true) {
|
||||||
|
t.Fatal("10:30 with slot 45 must wait until 10:45 on the due day")
|
||||||
|
}
|
||||||
|
if !EligibleAt(tue.Add(15*time.Minute), 45, true) {
|
||||||
|
t.Fatal("10:45 with slot 45 must be eligible")
|
||||||
|
}
|
||||||
|
if !EligibleAt(tue.Add(24*time.Hour), 45, false) {
|
||||||
|
t.Fatal("a catch-up day does not wait for the slot again")
|
||||||
|
}
|
||||||
|
if a, b := SlotMinute("demo-hp-bb76ea"), SlotMinute("demo-felhom-8363b5"); a < 0 || a >= SlotSpreadMinutes || b < 0 || b >= SlotSpreadMinutes {
|
||||||
|
t.Fatalf("slots out of range: %d %d", a, b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDueWeeklyFromTuesday(t *testing.T) {
|
||||||
|
loc := time.UTC
|
||||||
|
mon := time.Date(2026, 10, 19, 12, 0, 0, 0, loc)
|
||||||
|
tue := time.Date(2026, 10, 20, 12, 0, 0, 0, loc)
|
||||||
|
if !Due(State{}, mon) {
|
||||||
|
t.Fatal("never measured is due")
|
||||||
|
}
|
||||||
|
st := State{LastOKAt: time.Date(2026, 10, 14, 11, 0, 0, 0, loc)} // last Wednesday
|
||||||
|
if Due(st, mon) {
|
||||||
|
t.Fatal("measured after last Tuesday 10:00 is not due on Monday")
|
||||||
|
}
|
||||||
|
if !Due(st, tue) {
|
||||||
|
t.Fatal("from Tuesday 10:00 a new measurement is due")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── the job ─────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
type fakeRunner struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
calls int
|
||||||
|
out []runOut
|
||||||
|
}
|
||||||
|
type runOut struct {
|
||||||
|
mbps float64
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeRunner) Run(ctx context.Context, addr string) (Result, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
i := f.calls
|
||||||
|
f.calls++
|
||||||
|
if i >= len(f.out) {
|
||||||
|
return Result{}, errors.New("no more scripted runs")
|
||||||
|
}
|
||||||
|
return Result{Mbps: f.out[i].mbps, ServerBytes: 1}, f.out[i].err
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeGate struct{ busy string }
|
||||||
|
|
||||||
|
func (g *fakeGate) TryAcquire(what string) (func(), string, bool) {
|
||||||
|
if g.busy != "" {
|
||||||
|
return nil, g.busy, false
|
||||||
|
}
|
||||||
|
return func() {}, "", true
|
||||||
|
}
|
||||||
|
|
||||||
|
type rig struct {
|
||||||
|
t *Tester
|
||||||
|
run *fakeRunner
|
||||||
|
gate *fakeGate
|
||||||
|
now time.Time
|
||||||
|
tx uint64
|
||||||
|
tunnel bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// newRig: Tuesday 12:00 UTC (eligible, any slot), tunnel up, 11 quiet samples taken.
|
||||||
|
func newRig(t *testing.T, outs ...runOut) *rig {
|
||||||
|
r := &rig{run: &fakeRunner{out: outs}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
|
||||||
|
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
|
||||||
|
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
|
||||||
|
r.t.loc = time.UTC
|
||||||
|
r.t.now = func() time.Time { return r.now }
|
||||||
|
r.t.sleep = func(context.Context, time.Duration) bool { return true }
|
||||||
|
r.quietMinutes(QuietMinutes + 1)
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *rig) minute(bytes uint64) {
|
||||||
|
r.now = r.now.Add(time.Minute)
|
||||||
|
r.tx += bytes
|
||||||
|
r.t.sample()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *rig) quietMinutes(n int) {
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
r.minute(1_000_000) // 1 MB/min = 0.13 Mbit/s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTickMeasuresTheHighestOfThreeRuns(t *testing.T) {
|
||||||
|
r := newRig(t, runOut{mbps: 88}, runOut{mbps: 91.5}, runOut{mbps: 90})
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
mbps, _, ok := r.t.Value()
|
||||||
|
if !ok || mbps != 91.5 || r.run.calls != RunsPerMeasurement {
|
||||||
|
t.Fatalf("value %.1f ok=%v calls=%d; want the highest of 3 runs = 91.5", mbps, ok, r.run.calls)
|
||||||
|
}
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if r.run.calls != RunsPerMeasurement {
|
||||||
|
t.Fatal("a measured week must not test again")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBusyServerIsRetriedLaterTheSameDay(t *testing.T) {
|
||||||
|
r := newRig(t, runOut{err: ErrBusy}, runOut{mbps: 90}, runOut{mbps: 89}, runOut{mbps: 92})
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if _, _, ok := r.t.Value(); ok || r.run.calls != 1 {
|
||||||
|
t.Fatalf("busy: ok=%v calls=%d; want no value and the measurement stopped after the busy answer", ok, r.run.calls)
|
||||||
|
}
|
||||||
|
if r.t.st.TriesToday != 1 || r.t.st.LastError == "" {
|
||||||
|
t.Fatalf("busy attempt not recorded: %+v", r.t.st)
|
||||||
|
}
|
||||||
|
r.quietMinutes(1)
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if mbps, _, ok := r.t.Value(); !ok || mbps != 92 {
|
||||||
|
t.Fatalf("retry later the same day: value %.1f ok=%v", mbps, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTriesPerDayAreBounded(t *testing.T) {
|
||||||
|
outs := make([]runOut, 20)
|
||||||
|
for i := range outs {
|
||||||
|
outs[i] = runOut{err: ErrBusy}
|
||||||
|
}
|
||||||
|
r := newRig(t, outs...)
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
r.quietMinutes(1)
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
}
|
||||||
|
if r.run.calls != MaxTriesPerDay {
|
||||||
|
t.Fatalf("calls = %d in one day, want at most %d", r.run.calls, MaxTriesPerDay)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNoTunnelNoTest(t *testing.T) {
|
||||||
|
r := newRig(t, runOut{mbps: 90})
|
||||||
|
r.tunnel = false
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if r.run.calls != 0 {
|
||||||
|
t.Fatal("tested without an operational tunnel")
|
||||||
|
}
|
||||||
|
if _, _, ok := r.t.Value(); ok {
|
||||||
|
t.Fatal("a value without a test")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBusyUplinkOrHeavyOpWaits(t *testing.T) {
|
||||||
|
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
|
||||||
|
r.minute(60_000_000) // one minute at 8 Mbit/s: the family is uploading
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if r.run.calls != 0 {
|
||||||
|
t.Fatal("tested while the uplink was busy in the last 10 minutes")
|
||||||
|
}
|
||||||
|
r.quietMinutes(QuietMinutes)
|
||||||
|
r.gate.busy = "backup:felhom-pbs"
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if r.run.calls != 0 {
|
||||||
|
t.Fatal("tested beside a heavy operation")
|
||||||
|
}
|
||||||
|
r.gate.busy = ""
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if r.run.calls == 0 {
|
||||||
|
t.Fatal("never tested once the uplink was quiet and the gate free")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNeverTestsInTheNightOrBeforeTuesday(t *testing.T) {
|
||||||
|
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
|
||||||
|
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
|
||||||
|
r.quietMinutes(QuietMinutes + 1)
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
r.t.st.LastOKAt = time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
|
||||||
|
r.now = time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC) // Monday noon, measured last week
|
||||||
|
r.quietMinutes(QuietMinutes + 1)
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if r.run.calls != 0 {
|
||||||
|
t.Fatalf("tested %d times at night or when not due", r.run.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStateSurvivesARestartAndReportsIt(t *testing.T) {
|
||||||
|
r := newRig(t, runOut{mbps: 90.7}, runOut{err: errors.New("reset")}, runOut{mbps: 80})
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
again := New(r.run, nil, r.gate, nil, "host-x", r.t.statePath, nil)
|
||||||
|
mbps, at, ok := again.Value()
|
||||||
|
if !ok || mbps != 90.7 || at.IsZero() {
|
||||||
|
t.Fatalf("after a restart: %.1f %v %v", mbps, at, ok)
|
||||||
|
}
|
||||||
|
b, _ := json.Marshal(again.LineTestStatus(context.Background()))
|
||||||
|
var back hub.LineTestStatus
|
||||||
|
json.Unmarshal(b, &back)
|
||||||
|
if back.Mbps != 90.7 || back.LastOKAt == "" || len(back.RunsMbps) != 2 || back.Schedule == "" {
|
||||||
|
t.Fatalf("report stanza = %s", b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── the uplink counter ──────────────────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
func TestUplinkCountsTheBridgesPhysicalPortsOnly(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
w := func(p, s string) {
|
||||||
|
os.MkdirAll(filepath.Dir(filepath.Join(root, p)), 0o755)
|
||||||
|
os.WriteFile(filepath.Join(root, p), []byte(s), 0o644)
|
||||||
|
}
|
||||||
|
w("proc/net/route", "Iface\tDestination\tGateway\tFlags\tRefCnt\tUse\tMetric\tMask\n"+
|
||||||
|
"wg-felhom\t0000004D\t00000000\t0001\t0\t0\t0\tFFFFFFFF\n"+
|
||||||
|
"vmbr0\t00000000\t0100A8C0\t0003\t0\t0\t0\t00000000\n")
|
||||||
|
w("sys/class/net/vmbr0/statistics/tx_bytes", "5\n")
|
||||||
|
w("sys/class/net/vmbr0/brif/enp1s0/.keep", "")
|
||||||
|
w("sys/class/net/vmbr0/brif/veth9201i0/.keep", "")
|
||||||
|
w("sys/class/net/enp1s0/device/.keep", "")
|
||||||
|
w("sys/class/net/enp1s0/statistics/tx_bytes", "1000\n")
|
||||||
|
w("sys/class/net/veth9201i0/statistics/tx_bytes", "777777\n")
|
||||||
|
got, err := SysUplinkTx(root)()
|
||||||
|
if err != nil || got != 1000 {
|
||||||
|
t.Fatalf("uplink tx = %d, %v; want 1000 (the physical port only — not the bridge, not the guest's veth)", got, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestQuietNeedsTenFullMinutes(t *testing.T) {
|
||||||
|
base := time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC)
|
||||||
|
var s []txSample
|
||||||
|
for i := 0; i <= QuietMinutes; i++ {
|
||||||
|
s = append(s, txSample{at: base.Add(time.Duration(i) * time.Minute), tx: uint64(i) * 1_000_000})
|
||||||
|
}
|
||||||
|
if q, _ := quiet(s); !q {
|
||||||
|
t.Fatal("ten quiet minutes must read quiet")
|
||||||
|
}
|
||||||
|
if q, _ := quiet(s[1:]); q {
|
||||||
|
t.Fatal("nine minutes of samples must not read quiet")
|
||||||
|
}
|
||||||
|
gap := append([]txSample(nil), s...)
|
||||||
|
gap[5].at = gap[5].at.Add(-10 * time.Minute)
|
||||||
|
if q, _ := quiet(gap); q {
|
||||||
|
t.Fatal("a gap in the samples must not read quiet")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── a test on request (R-1250): a torrent app that seeds all day ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// seedingRig: due and eligible, but every minute of the last ten sent at the line's speed — the weekly Tick never runs.
|
||||||
|
func seedingRig(t *testing.T, outs ...runOut) *rig {
|
||||||
|
r := newRig(t, outs...)
|
||||||
|
for i := 0; i < QuietMinutes+1; i++ {
|
||||||
|
r.minute(600_000_000) // 80 Mbit/s: a seed at the line's speed
|
||||||
|
}
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt(t *testing.T) {
|
||||||
|
r := seedingRig(t, runOut{mbps: 88}, runOut{mbps: 92}, runOut{mbps: 90})
|
||||||
|
r.t.Tick(context.Background())
|
||||||
|
if r.run.calls != 0 {
|
||||||
|
t.Fatal("the weekly Tick tested a seeding box")
|
||||||
|
}
|
||||||
|
if !r.t.WaitingForQuiet() {
|
||||||
|
t.Fatal("a due, eligible, seeding box must report waiting_for_quiet")
|
||||||
|
}
|
||||||
|
var slept []time.Duration
|
||||||
|
r.t.sleep = func(_ context.Context, d time.Duration) bool { slept = append(slept, d); return true }
|
||||||
|
mbps, runs, err := r.t.RunNow(context.Background()) // the controller holds the app: the counter stands still
|
||||||
|
if err != nil || mbps != 92 || len(runs) != RunsPerMeasurement || r.run.calls != RunsPerMeasurement {
|
||||||
|
t.Fatalf("RunNow = %.1f %v %v, calls %d; want the best of 3 = 92", mbps, runs, err, r.run.calls)
|
||||||
|
}
|
||||||
|
want := []time.Duration{RequestSettle, RequestQuietWindow, RequestRunGap, RequestRunGap}
|
||||||
|
if len(slept) != len(want) {
|
||||||
|
t.Fatalf("slept %v, want %v", slept, want)
|
||||||
|
}
|
||||||
|
var total time.Duration
|
||||||
|
for i := range want {
|
||||||
|
if slept[i] != want[i] {
|
||||||
|
t.Fatalf("slept %v, want %v", slept, want)
|
||||||
|
}
|
||||||
|
total += slept[i]
|
||||||
|
}
|
||||||
|
if total+RunsPerMeasurement*TestTime > 60*time.Second {
|
||||||
|
t.Fatalf("a request holds the app %v + runs; it must end inside 60 s", total)
|
||||||
|
}
|
||||||
|
if v, _, ok := r.t.Value(); !ok || v != 92 {
|
||||||
|
t.Fatalf("the requested measurement was not stored: %.1f %v", v, ok)
|
||||||
|
}
|
||||||
|
if r.t.WaitingForQuiet() {
|
||||||
|
t.Fatal("measured this week: no longer waiting")
|
||||||
|
}
|
||||||
|
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-due" {
|
||||||
|
t.Fatalf("a second request this week: %v, want not-due", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func refusalCode(err error) string {
|
||||||
|
var ref *Refusal
|
||||||
|
if errors.As(err, &ref) {
|
||||||
|
return ref.Code
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// The held app is not the only uploader: a request still refuses to measure a busy uplink.
|
||||||
|
func TestRunNowRefusesAnUplinkThatIsStillBusy(t *testing.T) {
|
||||||
|
r := seedingRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
|
||||||
|
r.t.uplink = func() (uint64, error) { r.tx += 20_000_000; return r.tx, nil } // 20 MB per 10 s = 16 Mbit/s
|
||||||
|
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-quiet" || r.run.calls != 0 {
|
||||||
|
t.Fatalf("RunNow on a busy uplink: %v, calls %d; want not-quiet and no run", err, r.run.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every other rule of the weekly test holds for a request.
|
||||||
|
func TestRunNowKeepsTheWeeklyRules(t *testing.T) {
|
||||||
|
r := seedingRig(t)
|
||||||
|
r.gate.busy = "backup:felhom-pbs"
|
||||||
|
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "heavy-op" {
|
||||||
|
t.Fatalf("beside a backup: %v, want heavy-op", err)
|
||||||
|
}
|
||||||
|
r.gate.busy = ""
|
||||||
|
r.tunnel = false
|
||||||
|
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "no-tunnel" {
|
||||||
|
t.Fatalf("no tunnel: %v, want no-tunnel", err)
|
||||||
|
}
|
||||||
|
if r.t.WaitingForQuiet() {
|
||||||
|
t.Fatal("no tunnel: a request could not run, so it must not be invited")
|
||||||
|
}
|
||||||
|
r.tunnel = true
|
||||||
|
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
|
||||||
|
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "outside-window" {
|
||||||
|
t.Fatalf("at night: %v, want outside-window", err)
|
||||||
|
}
|
||||||
|
if r.run.calls != 0 {
|
||||||
|
t.Fatalf("%d runs on refused requests", r.run.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A failed request counts as a try, so a controller cannot request more than MaxTriesPerDay tests a day.
|
||||||
|
func TestRunNowFailuresSpendTheDaysTries(t *testing.T) {
|
||||||
|
var outs []runOut
|
||||||
|
for i := 0; i < MaxTriesPerDay*RunsPerMeasurement; i++ {
|
||||||
|
outs = append(outs, runOut{err: errors.New("connection refused")})
|
||||||
|
}
|
||||||
|
r := seedingRig(t, outs...)
|
||||||
|
for i := 0; i < MaxTriesPerDay; i++ {
|
||||||
|
if _, _, err := r.t.RunNow(context.Background()); err == nil || refusalCode(err) != "" {
|
||||||
|
t.Fatalf("try %d: %v, want a failed measurement", i+1, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "tries-used" {
|
||||||
|
t.Fatalf("after %d failures: %v, want tries-used", MaxTriesPerDay, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A quiet box tests itself on the weekly Tick: it must not invite the controller to hold its app.
|
||||||
|
func TestQuietBoxIsNotWaiting(t *testing.T) {
|
||||||
|
r := newRig(t)
|
||||||
|
if r.t.WaitingForQuiet() {
|
||||||
|
t.Fatal("ten quiet minutes: the weekly Tick runs; waiting_for_quiet must be false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Live proof 2026-10-24 (demo-hp): a freshly started tester answered waiting_for_quiet before it had watched a single
|
||||||
|
// minute, so a quiet box's torrent app would be held for nothing. Waiting needs the full window, seen busy.
|
||||||
|
func TestFreshTesterIsNotWaiting(t *testing.T) {
|
||||||
|
r := &rig{run: &fakeRunner{}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
|
||||||
|
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
|
||||||
|
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
|
||||||
|
r.t.loc = time.UTC
|
||||||
|
r.t.now = func() time.Time { return r.now }
|
||||||
|
if r.t.WaitingForQuiet() {
|
||||||
|
t.Fatal("no minute watched yet: waiting_for_quiet must be false")
|
||||||
|
}
|
||||||
|
for i := 0; i < QuietMinutes; i++ {
|
||||||
|
r.minute(600_000_000)
|
||||||
|
}
|
||||||
|
if r.t.WaitingForQuiet() {
|
||||||
|
t.Fatal("10 samples are 9 minutes: still not a full window")
|
||||||
|
}
|
||||||
|
r.minute(600_000_000)
|
||||||
|
if !r.t.WaitingForQuiet() {
|
||||||
|
t.Fatal("a full busy window: waiting_for_quiet must be true")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||||
|
)
|
||||||
|
|
||||||
|
// LineTestReader is the R-1247 weekly upload line test (internal/linetest.Tester) as the local API reads it.
|
||||||
|
type LineTestReader interface {
|
||||||
|
Value() (mbps float64, at time.Time, ok bool)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LineTestRequester is the R-1250 half: a due test the uplink never lets start, and a test on request. The Tester
|
||||||
|
// satisfies it; a reader without it answers waiting_for_quiet:false and 503 on the POST.
|
||||||
|
type LineTestRequester interface {
|
||||||
|
WaitingForQuiet() bool
|
||||||
|
RunNow(ctx context.Context) (mbps float64, runs []float64, err error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LineTestResponse is GET /host/line-test. Present=false: never measured, or the job is not wired (no tunnel, disabled,
|
||||||
|
// or an agent before the route answers 404 — the controller reads all three as „unknown").
|
||||||
|
type LineTestResponse struct {
|
||||||
|
Present bool `json:"present"`
|
||||||
|
Mbps float64 `json:"mbps,omitempty"`
|
||||||
|
MeasuredAt string `json:"measured_at,omitempty"` // RFC3339 UTC
|
||||||
|
Method string `json:"method,omitempty"` // "iperf3-tunnel": a 10-second upload to ep0 inside the tunnel
|
||||||
|
// WaitingForQuiet: a measurement is due and could start now, but the uplink has not been quiet for 10 minutes
|
||||||
|
// (R-1250). The controller with a seeding torrent app then holds its limit and POSTs /host/line-test/run.
|
||||||
|
WaitingForQuiet bool `json:"waiting_for_quiet,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// LineTestRunResponse is POST /host/line-test/run. OK=false with Refused: nothing ran (Refused is linetest.Refusal's
|
||||||
|
// code, or "server-busy"); OK=false without it: the runs failed.
|
||||||
|
type LineTestRunResponse struct {
|
||||||
|
OK bool `json:"ok"`
|
||||||
|
Mbps float64 `json:"mbps,omitempty"`
|
||||||
|
RunsMbps []float64 `json:"runs_mbps,omitempty"`
|
||||||
|
Refused string `json:"refused,omitempty"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetLineTest wires the weekly line test (main.go builds it after the local API server).
|
||||||
|
func (s *Server) SetLineTest(r LineTestReader) { s.lineTest = r }
|
||||||
|
|
||||||
|
func (s *Server) handleLineTest(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||||
|
if s.lineTest == nil {
|
||||||
|
writeOK(w, LineTestResponse{Present: false})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var waiting bool
|
||||||
|
if q, ok := s.lineTest.(LineTestRequester); ok {
|
||||||
|
waiting = q.WaitingForQuiet()
|
||||||
|
}
|
||||||
|
mbps, at, ok := s.lineTest.Value()
|
||||||
|
if !ok {
|
||||||
|
writeOK(w, LineTestResponse{Present: false, WaitingForQuiet: waiting})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.logger.Debug("local-api: /host/line-test served", "vmid", vmid, "mbps", mbps, "measured_at", at.UTC().Format(time.RFC3339), "waiting_for_quiet", waiting)
|
||||||
|
writeOK(w, LineTestResponse{Present: true, Mbps: mbps, MeasuredAt: at.UTC().Format(time.RFC3339), Method: "iperf3-tunnel", WaitingForQuiet: waiting})
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleLineTestRun is POST /host/line-test/run (R-1250): a test on request, synchronous (about 55 s). The answer is
|
||||||
|
// always 200 with the outcome in the body, except 503 when no line test is wired.
|
||||||
|
func (s *Server) handleLineTestRun(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||||
|
q, ok := s.lineTest.(LineTestRequester)
|
||||||
|
if s.lineTest == nil || !ok {
|
||||||
|
writeErr(w, http.StatusServiceUnavailable, "line test not configured")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.logger.Info("local-api: line test requested", "vmid", vmid)
|
||||||
|
mbps, runs, err := q.RunNow(r.Context())
|
||||||
|
var ref *linetest.Refusal
|
||||||
|
switch {
|
||||||
|
case err == nil:
|
||||||
|
writeOK(w, LineTestRunResponse{OK: true, Mbps: mbps, RunsMbps: runs})
|
||||||
|
case errors.As(err, &ref):
|
||||||
|
s.logger.Info("local-api: line test request refused", "vmid", vmid, "reason", ref.Code, "detail", ref.Msg)
|
||||||
|
writeOK(w, LineTestRunResponse{Refused: ref.Code, Error: ref.Msg})
|
||||||
|
case errors.Is(err, linetest.ErrBusy):
|
||||||
|
writeOK(w, LineTestRunResponse{Refused: "server-busy", Error: err.Error()})
|
||||||
|
default:
|
||||||
|
writeOK(w, LineTestRunResponse{Error: err.Error()})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"log/slog"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fixedLine struct {
|
||||||
|
mbps float64
|
||||||
|
ok bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f fixedLine) Value() (float64, time.Time, bool) {
|
||||||
|
return f.mbps, time.Date(2026, 10, 24, 6, 31, 0, 0, time.UTC), f.ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func lineTestBody(t *testing.T, s *Server) LineTestResponse {
|
||||||
|
t.Helper()
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.handleLineTest(rec, httptest.NewRequest("GET", "/host/line-test", nil), 9201)
|
||||||
|
var env struct {
|
||||||
|
OK bool `json:"ok"`
|
||||||
|
Data LineTestResponse `json:"data"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
|
||||||
|
t.Fatalf("decode %q: %v", rec.Body.String(), err)
|
||||||
|
}
|
||||||
|
return env.Data
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-1247: the controller sets qBittorrent's limit only from a PRESENT value; never measured and not wired must both
|
||||||
|
// read present:false, never a zero line.
|
||||||
|
func TestLineTestRoute(t *testing.T) {
|
||||||
|
s := &Server{logger: slog.Default()}
|
||||||
|
if got := lineTestBody(t, s); got.Present {
|
||||||
|
t.Fatalf("not wired: %+v, want present:false", got)
|
||||||
|
}
|
||||||
|
s.SetLineTest(fixedLine{ok: false})
|
||||||
|
if got := lineTestBody(t, s); got.Present || got.Mbps != 0 {
|
||||||
|
t.Fatalf("never measured: %+v, want present:false", got)
|
||||||
|
}
|
||||||
|
s.SetLineTest(fixedLine{mbps: 90.7, ok: true})
|
||||||
|
got := lineTestBody(t, s)
|
||||||
|
if !got.Present || got.Mbps != 90.7 || got.Method != "iperf3-tunnel" || !strings.HasPrefix(got.MeasuredAt, "2026-10-24T06:31") {
|
||||||
|
t.Fatalf("measured: %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type requestableLine struct {
|
||||||
|
fixedLine
|
||||||
|
waiting bool
|
||||||
|
mbps float64
|
||||||
|
err error
|
||||||
|
calls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *requestableLine) WaitingForQuiet() bool { return r.waiting }
|
||||||
|
func (r *requestableLine) RunNow(context.Context) (float64, []float64, error) {
|
||||||
|
r.calls++
|
||||||
|
if r.err != nil {
|
||||||
|
return 0, nil, r.err
|
||||||
|
}
|
||||||
|
return r.mbps, []float64{r.mbps - 1, r.mbps}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func lineTestRun(t *testing.T, s *Server) (int, LineTestRunResponse) {
|
||||||
|
t.Helper()
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.handleLineTestRun(rec, httptest.NewRequest("POST", "/host/line-test/run", nil), 9201)
|
||||||
|
var env struct {
|
||||||
|
OK bool `json:"ok"`
|
||||||
|
Data LineTestRunResponse `json:"data"`
|
||||||
|
}
|
||||||
|
_ = json.Unmarshal(rec.Body.Bytes(), &env)
|
||||||
|
return rec.Code, env.Data
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-1250: the controller acts only on waiting_for_quiet, and reads a refusal by its code — never as a measurement.
|
||||||
|
func TestLineTestRunRoute(t *testing.T) {
|
||||||
|
s := &Server{logger: slog.Default()}
|
||||||
|
if code, _ := lineTestRun(t, s); code != 503 {
|
||||||
|
t.Fatalf("not wired: HTTP %d, want 503", code)
|
||||||
|
}
|
||||||
|
s.SetLineTest(fixedLine{ok: true, mbps: 90})
|
||||||
|
if code, _ := lineTestRun(t, s); code != 503 {
|
||||||
|
t.Fatalf("a reader without the request half: HTTP %d, want 503", code)
|
||||||
|
}
|
||||||
|
l := &requestableLine{fixedLine: fixedLine{ok: false}, waiting: true, mbps: 93.5}
|
||||||
|
s.SetLineTest(l)
|
||||||
|
if got := lineTestBody(t, s); got.Present || !got.WaitingForQuiet {
|
||||||
|
t.Fatalf("GET while waiting: %+v, want present:false waiting_for_quiet:true", got)
|
||||||
|
}
|
||||||
|
if code, got := lineTestRun(t, s); code != 200 || !got.OK || got.Mbps != 93.5 || len(got.RunsMbps) != 2 {
|
||||||
|
t.Fatalf("a measured request: HTTP %d %+v", code, got)
|
||||||
|
}
|
||||||
|
l.err = &linetest.Refusal{Code: "not-quiet", Msg: "the uplink sent 16.0 Mbit/s while held"}
|
||||||
|
if _, got := lineTestRun(t, s); got.OK || got.Refused != "not-quiet" || got.Mbps != 0 {
|
||||||
|
t.Fatalf("a refusal: %+v, want ok:false refused:not-quiet", got)
|
||||||
|
}
|
||||||
|
l.err = linetest.ErrBusy
|
||||||
|
if _, got := lineTestRun(t, s); got.OK || got.Refused != "server-busy" {
|
||||||
|
t.Fatalf("server busy: %+v, want refused:server-busy", got)
|
||||||
|
}
|
||||||
|
l.err = errors.New("dial tcp 10.77.0.1:5201: i/o timeout")
|
||||||
|
if _, got := lineTestRun(t, s); got.OK || got.Refused != "" || got.Error == "" {
|
||||||
|
t.Fatalf("a failed run: %+v, want ok:false, no refusal, the error", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -336,6 +336,7 @@ type Server struct {
|
|||||||
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
|
||||||
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
|
||||||
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
|
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
|
||||||
|
lineTest LineTestReader // R-1247 weekly line test (optional; nil → present:false)
|
||||||
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
|
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
|
||||||
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
|
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
|
||||||
guestPower map[int]guestPowerState
|
guestPower map[int]guestPowerState
|
||||||
@@ -573,6 +574,10 @@ func (s *Server) Handler() http.Handler {
|
|||||||
// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
|
// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
|
||||||
// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
|
// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
|
||||||
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
||||||
|
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
|
||||||
|
// read it. Read-only; never measured (or the job not wired) = present:false.
|
||||||
|
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
|
||||||
|
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
|
||||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||||
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
|
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
|
||||||
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
|
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -114,12 +115,43 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
|
|||||||
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
|
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
|
||||||
}
|
}
|
||||||
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
|
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
|
||||||
|
e.policyPass(ctx, lg, p.AgentVersion)
|
||||||
if e.AfterInstall != nil {
|
if e.AfterInstall != nil {
|
||||||
e.AfterInstall(ctx)
|
e.AfterInstall(ctx)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// policyPass (R-1285) asks the wrapper NOW on disk — after this install, the bundle's own — to apply the
|
||||||
|
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
|
||||||
|
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
|
||||||
|
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
|
||||||
|
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
|
||||||
|
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
|
||||||
|
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
|
||||||
|
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
|
||||||
|
// SAME update. Idempotent; on an appliance it removes nothing.
|
||||||
|
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
|
||||||
|
if lg == nil {
|
||||||
|
lg = slog.Default()
|
||||||
|
}
|
||||||
|
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
|
||||||
|
e.policyPass(ctx, lg, ver)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
|
||||||
|
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
|
||||||
|
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
lg.Warn("osupdate: the bundle policy pass did not run", "err", err)
|
||||||
|
case wr.refused():
|
||||||
|
lg.Info("osupdate: the bundle policy pass was refused (a wrapper before 0.166.0?) — the next bundle applies it", "refused", string(wr.Refused))
|
||||||
|
default:
|
||||||
|
lg.Info("osupdate: bundle policy pass done", "bundle_policy", string(wr.BundlePolicy))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
|
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
|
||||||
hc := e.HTTPClient
|
hc := e.HTTPClient
|
||||||
if hc == nil {
|
if hc == nil {
|
||||||
|
|||||||
@@ -38,6 +38,9 @@ func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]b
|
|||||||
var plan map[string]any
|
var plan map[string]any
|
||||||
_ = json.Unmarshal(raw, &plan)
|
_ = json.Unmarshal(raw, &plan)
|
||||||
b.plans = append(b.plans, plan)
|
b.plans = append(b.plans, plan)
|
||||||
|
if plan["mode"] == "bundle-policy" { // R-1285: the pass after an install carries no bundle
|
||||||
|
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":[]}}` + "\n"), nil, nil
|
||||||
|
}
|
||||||
body, _ := os.ReadFile(plan["bundle"].(string))
|
body, _ := os.ReadFile(plan["bundle"].(string))
|
||||||
b.bodies = append(b.bodies, body)
|
b.bodies = append(b.bodies, body)
|
||||||
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
|
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
|
||||||
@@ -153,3 +156,27 @@ func TestBundleURL(t *testing.T) {
|
|||||||
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||||
return b.Run(ctx, name, args...)
|
return b.Run(ctx, name, args...)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-1285: right after an install the agent asks the wrapper now on disk for the appliance-only policy pass — the bundle
|
||||||
|
// run itself was planned by the OLD wrapper (on a BYO host upgraded from 0.164.0 our SSH drop-in stayed one bundle
|
||||||
|
// longer). A refused or failed bundle runs no pass.
|
||||||
|
// RED-PROOF: drop the policyPass call → one plan only.
|
||||||
|
func TestR1285_AnInstalledBundleIsFollowedByThePolicyPass(t *testing.T) {
|
||||||
|
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
|
||||||
|
srv, sha := serveBundle(t, body)
|
||||||
|
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/usr/local/sbin/felhom-os-apply"]}}`}
|
||||||
|
e, _ := bundleExec(t, w, srv.URL)
|
||||||
|
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(w.plans) != 2 || w.plans[1]["mode"] != "bundle-policy" || w.plans[1]["layer"] != "host" || w.plans[1]["signed"] != nil {
|
||||||
|
t.Fatalf("after the install the agent must run the policy pass (host layer, no envelope): %v", w.plans)
|
||||||
|
}
|
||||||
|
|
||||||
|
w2 := &bundleWrapper{t: t, rep: `{"mode":"bundle","refused":{"code":"R3","reason":"x"}}`}
|
||||||
|
e2, _ := bundleExec(t, w2, srv.URL)
|
||||||
|
_ = e2.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha))
|
||||||
|
if len(w2.plans) != 1 {
|
||||||
|
t.Fatalf("a refused bundle must run no policy pass: %v", w2.plans)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -123,7 +123,8 @@ type WrapperReport struct {
|
|||||||
Undo bool `json:"undo"`
|
Undo bool `json:"undo"`
|
||||||
LiveRestore json.RawMessage `json:"live_restore"`
|
LiveRestore json.RawMessage `json:"live_restore"`
|
||||||
Facts json.RawMessage `json:"facts"`
|
Facts json.RawMessage `json:"facts"`
|
||||||
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
|
||||||
|
BundlePolicy json.RawMessage `json:"bundle_policy"` // the appliance-only policy pass (R-1285, mode "bundle-policy")
|
||||||
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
|
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
|
||||||
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
|
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
|
||||||
OOMCheck json.RawMessage `json:"oom_check"`
|
OOMCheck json.RawMessage `json:"oom_check"`
|
||||||
|
|||||||
@@ -18,8 +18,14 @@ type Manager struct {
|
|||||||
logger *slog.Logger
|
logger *slog.Logger
|
||||||
now func() time.Time // injectable for tests
|
now func() time.Time // injectable for tests
|
||||||
sleep func(context.Context, time.Duration)
|
sleep func(context.Context, time.Duration)
|
||||||
|
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
|
||||||
|
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
|
||||||
|
onCommitted func(context.Context)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
|
||||||
|
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
|
||||||
|
|
||||||
// ManagerConfig wires the commit manager.
|
// ManagerConfig wires the commit manager.
|
||||||
type ManagerConfig struct {
|
type ManagerConfig struct {
|
||||||
StateDir string
|
StateDir string
|
||||||
@@ -119,4 +125,7 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
|
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
|
||||||
|
if m.onCommitted != nil {
|
||||||
|
m.onCommitted(ctx)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
package wgtunnel
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-1247: the line test's server is the desired block's pbs_tunnel_ip, and ONLY while the tunnel is operational
|
||||||
|
// (registered + unit active) — „no tunnel → no test" rests on this answer.
|
||||||
|
func TestLineTestTargetOnlyWhileOperational(t *testing.T) {
|
||||||
|
m, _, active, _ := testManager(t, &fakeHub{})
|
||||||
|
ctx := context.Background()
|
||||||
|
if _, ok := m.LineTestTarget(); ok {
|
||||||
|
t.Fatal("a target before any Apply")
|
||||||
|
}
|
||||||
|
m.Apply(ctx, false, nil) // registers
|
||||||
|
pub := localPub(t, m)
|
||||||
|
m.Apply(ctx, true, testBlock(pub)) // installs; the unit is not active yet
|
||||||
|
if _, ok := m.LineTestTarget(); ok {
|
||||||
|
t.Fatal("a target while the unit is not active")
|
||||||
|
}
|
||||||
|
*active = true
|
||||||
|
m.Apply(ctx, true, testBlock(pub))
|
||||||
|
ip, ok := m.LineTestTarget()
|
||||||
|
if !ok || ip.String() != "10.77.0.1" {
|
||||||
|
t.Fatalf("operational tunnel: target %v ok=%v, want 10.77.0.1", ip, ok)
|
||||||
|
}
|
||||||
|
m.Apply(ctx, true, nil) // revoked
|
||||||
|
if _, ok := m.LineTestTarget(); ok {
|
||||||
|
t.Fatal("a target after the hub revoked the tunnel")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -99,6 +99,7 @@ type Manager struct {
|
|||||||
resolver Resolver
|
resolver Resolver
|
||||||
|
|
||||||
staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2)
|
staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2)
|
||||||
|
rekeyAfter time.Duration // D30: how long a future-stamped handshake stays before the re-key (rekey.go)
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
nextRegAt time.Time
|
nextRegAt time.Time
|
||||||
@@ -111,6 +112,10 @@ type Manager struct {
|
|||||||
resolveFailLogged bool // DNS-failure log throttle (reset on the next success)
|
resolveFailLogged bool // DNS-failure log throttle (reset on the next success)
|
||||||
staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP)
|
staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP)
|
||||||
|
|
||||||
|
// D30 (rekey.go): when the latest handshake was first seen stamped in the future, and the last re-key.
|
||||||
|
futureSince time.Time
|
||||||
|
lastRekey time.Time
|
||||||
|
|
||||||
// conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at
|
// conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at
|
||||||
// the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing
|
// the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing
|
||||||
// `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired;
|
// `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired;
|
||||||
@@ -123,6 +128,7 @@ type Manager struct {
|
|||||||
type convSnapshot struct {
|
type convSnapshot struct {
|
||||||
wgBlockDesired bool
|
wgBlockDesired bool
|
||||||
operational bool
|
operational bool
|
||||||
|
serverIP netip.Addr // the desired block's pbs_tunnel_ip (R-1247 line test target); invalid when none
|
||||||
}
|
}
|
||||||
|
|
||||||
// TunnelConvergence returns the cached (desired, operational) snapshot without any exec (fast-tick
|
// TunnelConvergence returns the cached (desired, operational) snapshot without any exec (fast-tick
|
||||||
@@ -134,6 +140,17 @@ func (m *Manager) TunnelConvergence() (desired, operational bool) {
|
|||||||
return m.conv.wgBlockDesired, m.conv.operational
|
return m.conv.wgBlockDesired, m.conv.operational
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// LineTestTarget is the R-1247 line test's server: ep0's tunnel address, ok only while the tunnel is operational
|
||||||
|
// (registered + unit active, as of the last Apply). Cached — no exec.
|
||||||
|
func (m *Manager) LineTestTarget() (netip.Addr, bool) {
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
if !m.conv.operational || !m.conv.serverIP.IsValid() {
|
||||||
|
return netip.Addr{}, false
|
||||||
|
}
|
||||||
|
return m.conv.serverIP, true
|
||||||
|
}
|
||||||
|
|
||||||
// NewManager builds a Manager. stateDir is the agent state dir (default /var/lib/felhom-agent —
|
// NewManager builds a Manager. stateDir is the agent state dir (default /var/lib/felhom-agent —
|
||||||
// note the FELHOM_WG sudoers install entry hard-codes the staged path under it).
|
// note the FELHOM_WG sudoers install entry hard-codes the staged path under it).
|
||||||
func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, logger *slog.Logger) *Manager {
|
func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, logger *slog.Logger) *Manager {
|
||||||
@@ -156,6 +173,7 @@ func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, log
|
|||||||
now: time.Now,
|
now: time.Now,
|
||||||
resolver: netResolver{},
|
resolver: netResolver{},
|
||||||
staleAfter: 180 * time.Second,
|
staleAfter: 180 * time.Second,
|
||||||
|
rekeyAfter: rekeyAfterDefault,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -342,7 +360,13 @@ func (m *Manager) Apply(ctx context.Context, fetched bool, block *hub.WireWiregu
|
|||||||
operational = m.isActive(ctx)
|
operational = m.isActive(ctx)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational}
|
var server netip.Addr
|
||||||
|
if desired {
|
||||||
|
if a, err := netip.ParseAddr(block.Endpoint.PBSTunnelIP); err == nil && a.Is4() {
|
||||||
|
server = a
|
||||||
|
}
|
||||||
|
}
|
||||||
|
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational, serverIP: server}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
keyExists := false
|
keyExists := false
|
||||||
@@ -600,6 +624,10 @@ func (m *Manager) Watchdog(ctx context.Context, fetched bool, block *hub.WireWir
|
|||||||
if !fetched || block == nil || !m.lastResolvedIP.IsValid() {
|
if !fetched || block == nil || !m.lastResolvedIP.IsValid() {
|
||||||
return // nothing applied to watch, or no desired block
|
return // nothing applied to watch, or no desired block
|
||||||
}
|
}
|
||||||
|
// D30: a handshake stamped in the future that stays → a new key on the existing peer path (rekey.go).
|
||||||
|
if m.clockAheadRekeyLocked(ctx, block) {
|
||||||
|
return
|
||||||
|
}
|
||||||
if !m.handshakeStaleLocked(ctx) {
|
if !m.handshakeStaleLocked(ctx) {
|
||||||
return // healthy → no DNS, no action (the load-bearing negative)
|
return // healthy → no DNS, no action (the load-bearing negative)
|
||||||
}
|
}
|
||||||
@@ -661,6 +689,9 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
|
|||||||
if age, ok := parseHandshakeAge(string(out), m.now()); ok {
|
if age, ok := parseHandshakeAge(string(out), m.now()); ok {
|
||||||
st.LastHandshakeAgeS = &age
|
st.LastHandshakeAgeS = &age
|
||||||
}
|
}
|
||||||
|
if fut, ok := handshakeFuture(string(out), m.now()); ok {
|
||||||
|
st.HandshakeFutureS = &fut
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if st.Pubkey == "" && !st.Registered && !st.Active {
|
if st.Pubkey == "" && !st.Registered && !st.Active {
|
||||||
@@ -669,6 +700,28 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
|
|||||||
return st
|
return st
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handshakeFuture (R-1204) reports how many seconds the latest handshake lies in the future, ok only when it does
|
||||||
|
// (by more than a minute — clock jitter is not the case). parseHandshakeAge clamps that case to 0 („just now"), which
|
||||||
|
// is what made a box whose clock had run ahead look healthy while ep0 refused its tunnel. Pinned by
|
||||||
|
// TestR1204_AFutureHandshakeIsReported.
|
||||||
|
func handshakeFuture(out string, now time.Time) (int64, bool) {
|
||||||
|
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
|
||||||
|
f := strings.Fields(line)
|
||||||
|
if len(f) != 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
epoch, err := strconv.ParseInt(f[1], 10, 64)
|
||||||
|
if err != nil || epoch <= 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if fut := epoch - now.Unix(); fut > 60 {
|
||||||
|
return fut, true
|
||||||
|
}
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
|
||||||
// parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>").
|
// parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>").
|
||||||
// epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0).
|
// epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0).
|
||||||
func parseHandshakeAge(out string, now time.Time) (int64, bool) {
|
func parseHandshakeAge(out string, now time.Time) (int64, bool) {
|
||||||
|
|||||||
@@ -696,3 +696,31 @@ func TestInitialResolveFailureNoTeardown(t *testing.T) {
|
|||||||
t.Errorf("did not recover after DNS returned: %v", rr.calls)
|
t.Errorf("did not recover after DNS returned: %v", rr.calls)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-1204: a box that ran once with its clock ahead keeps a handshake stamped in the FUTURE — ep0 then refuses every
|
||||||
|
// newer one. The age stays clamped at 0 (unchanged contract), and the new field says how far ahead it lies; a
|
||||||
|
// handshake in the past (or within a minute of now) never sets it.
|
||||||
|
// RED-PROOF: with Status not setting HandshakeFutureS the first assertion fails — seen failing 2026-10-22.
|
||||||
|
func TestR1204_AFutureHandshakeIsReported(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, _, active, _ := testManager(t, fh)
|
||||||
|
ctx := context.Background()
|
||||||
|
m.Apply(ctx, false, nil)
|
||||||
|
*active = true
|
||||||
|
m.now = func() time.Time { return time.Unix(1783107115-2*24*3600, 0) } // the clock came back two days earlier
|
||||||
|
st := m.Status(ctx)
|
||||||
|
if st.HandshakeFutureS == nil || *st.HandshakeFutureS != 2*24*3600 {
|
||||||
|
t.Fatalf("handshake_future_s = %v, want %d", st.HandshakeFutureS, 2*24*3600)
|
||||||
|
}
|
||||||
|
if st.LastHandshakeAgeS == nil || *st.LastHandshakeAgeS != 0 {
|
||||||
|
t.Fatalf("the age stays clamped at 0: %v", st.LastHandshakeAgeS)
|
||||||
|
}
|
||||||
|
m.now = func() time.Time { return time.Unix(1783107115+100, 0) }
|
||||||
|
if st = m.Status(ctx); st.HandshakeFutureS != nil {
|
||||||
|
t.Fatalf("a past handshake must not set handshake_future_s: %v", *st.HandshakeFutureS)
|
||||||
|
}
|
||||||
|
m.now = func() time.Time { return time.Unix(1783107115-30, 0) } // 30 s of jitter is not the case
|
||||||
|
if st = m.Status(ctx); st.HandshakeFutureS != nil {
|
||||||
|
t.Fatalf("clock jitter must not set handshake_future_s")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,189 @@
|
|||||||
|
package wgtunnel
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/base64"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// D30 (operator 2026-10-22, R-1204): the tunnel heals itself after the box ran with its clock ahead.
|
||||||
|
//
|
||||||
|
// WireGuard refuses a handshake whose timestamp is not newer than the last one it accepted from that key (replay
|
||||||
|
// protection). A box whose clock ran ahead handshook with a future timestamp; ep0 keeps it, and once the clock is
|
||||||
|
// right again ep0 refuses every new handshake from that key — until the box's clock passes the stored stamp. The
|
||||||
|
// sign on the box: its own latest handshake stays stamped in the future (R-1204's handshake_future_s).
|
||||||
|
//
|
||||||
|
// The repair: when that sign stays for rekeyAfter, the agent makes a NEW key pair and registers the new public key on
|
||||||
|
// the existing path (POST /hosts/<id>/wg → the hub re-keys the peer IN PLACE: same /32, new key, generation bump,
|
||||||
|
// and the wgsync push of the FULL peer list to ep0 drops the old key and adds the new one). A new key is a new peer
|
||||||
|
// for ep0, with no stored timestamp — the next handshake is accepted. Nothing new runs on ep0.
|
||||||
|
//
|
||||||
|
// The previous private key is kept as private.key.prev (0600): the escrow ceremony may hold it (a host-loss DR that
|
||||||
|
// installs it registers it again, and the hub re-keys in place back to it — the same path).
|
||||||
|
// Pinned by TestD30_* (rekey_test.go).
|
||||||
|
|
||||||
|
const (
|
||||||
|
// rekeyAfter: how long the latest handshake must stay in the future before the re-key. A working tunnel
|
||||||
|
// handshakes every two minutes, which ends the sign within minutes of the clock being right.
|
||||||
|
rekeyAfterDefault = 15 * time.Minute
|
||||||
|
// rekeyMinGap bounds the re-key to once an hour, whatever the clock does.
|
||||||
|
rekeyMinGap = time.Hour
|
||||||
|
prevKeyName = "private.key.prev"
|
||||||
|
)
|
||||||
|
|
||||||
|
// clockAheadRekeyLocked watches for a handshake stamped in the future and re-keys when it stays. It reports whether
|
||||||
|
// it re-keyed (the caller then skips the rest of its tick). Caller holds m.mu.
|
||||||
|
func (m *Manager) clockAheadRekeyLocked(ctx context.Context, block *hub.WireWireguard) bool {
|
||||||
|
if m.keyBroken || !m.isActive(ctx) {
|
||||||
|
m.futureSince = time.Time{}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
out, _, err := m.runner.Run(ctx, "wg", "show", Iface, "latest-handshakes")
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
now := m.now()
|
||||||
|
fut, ok := handshakeFuture(string(out), now)
|
||||||
|
if !ok {
|
||||||
|
if !m.futureSince.IsZero() {
|
||||||
|
m.logger.Info("wgtunnel: the handshake is no longer stamped in the future — no re-key needed")
|
||||||
|
}
|
||||||
|
m.futureSince = time.Time{}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if m.futureSince.IsZero() {
|
||||||
|
m.futureSince = now
|
||||||
|
m.logger.Warn("wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it stays",
|
||||||
|
"future_s", fut, "rekey_after", m.rekeyAfter.String())
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// A clock that jumps backwards must not stretch either wait (commit security review 2026-10-23 night): a start or a
|
||||||
|
// last re-key stamped AFTER now is from a wrong clock and is dropped — this whole feature exists because clocks jump.
|
||||||
|
if now.Before(m.futureSince) {
|
||||||
|
m.futureSince = now
|
||||||
|
}
|
||||||
|
if now.Sub(m.futureSince) < m.rekeyAfter {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if m.lastRekey.IsZero() {
|
||||||
|
m.lastRekey = m.loadLastRekey()
|
||||||
|
}
|
||||||
|
if now.Before(m.lastRekey) {
|
||||||
|
m.lastRekey = time.Time{}
|
||||||
|
}
|
||||||
|
if !m.lastRekey.IsZero() && now.Sub(m.lastRekey) < rekeyMinGap {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
pub, _, err := EnsureKey(m.stateDir)
|
||||||
|
if err != nil || block.Pubkey != pub {
|
||||||
|
return false // a re-register is already in flight, or the key is not readable — never re-key over that
|
||||||
|
}
|
||||||
|
newPub, err := rotateKey(m.stateDir)
|
||||||
|
if err != nil {
|
||||||
|
m.logger.Error("wgtunnel: re-key after a future handshake FAILED — the old key stays", "err", err)
|
||||||
|
m.lastRekey = now // not again before rekeyMinGap
|
||||||
|
m.saveLastRekey(now)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
m.lastRekey, m.futureSince = now, time.Time{}
|
||||||
|
m.saveLastRekey(now)
|
||||||
|
m.logger.Warn("wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key",
|
||||||
|
"old_pubkey", pub, "new_pubkey", newPub, "future_s", fut)
|
||||||
|
m.nextRegAt = time.Time{} // register now, not after an older backoff
|
||||||
|
m.registerLocked(ctx, newPub)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// rotateKey makes a new key pair, never leaving the box without a readable key (review 2026-10-23 night, N6): the new
|
||||||
|
// key is written and fsynced to a temp file; the current key gets a second name by a HARD LINK (the escrow may hold
|
||||||
|
// it — `private.key.prev` keeps the first such key, later ones `private.key.prev.<unix>`, the newest three kept, N7);
|
||||||
|
// then the temp file is renamed OVER the current name (atomic) and the directory fsynced. At every instant
|
||||||
|
// `private.key` is a complete key. Returns the new PUBLIC key only.
|
||||||
|
func rotateKey(stateDir string) (string, error) {
|
||||||
|
dir := filepath.Join(stateDir, "wg")
|
||||||
|
cur := filepath.Join(dir, keyFileName)
|
||||||
|
if _, err := os.Stat(cur); err != nil {
|
||||||
|
return "", fmt.Errorf("no current key: %w", err)
|
||||||
|
}
|
||||||
|
priv := make([]byte, 32)
|
||||||
|
if _, err := rand.Read(priv); err != nil {
|
||||||
|
return "", fmt.Errorf("entropy: %w", err)
|
||||||
|
}
|
||||||
|
clampPrivateKey(priv)
|
||||||
|
pub, err := derivePublic(priv)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
tmp := filepath.Join(dir, keyFileName+".new")
|
||||||
|
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("writing the new key: %w", err)
|
||||||
|
}
|
||||||
|
_, werr := f.Write([]byte(base64.StdEncoding.EncodeToString(priv) + "\n"))
|
||||||
|
serr := f.Sync()
|
||||||
|
cerr := f.Close()
|
||||||
|
if werr != nil || serr != nil || cerr != nil {
|
||||||
|
os.Remove(tmp)
|
||||||
|
return "", fmt.Errorf("writing the new key: %v %v %v", werr, serr, cerr)
|
||||||
|
}
|
||||||
|
prev := filepath.Join(dir, prevKeyName)
|
||||||
|
if _, err := os.Lstat(prev); err == nil {
|
||||||
|
prev = filepath.Join(dir, fmt.Sprintf("%s.%d", prevKeyName, rekeyNow().Unix()))
|
||||||
|
}
|
||||||
|
if err := os.Link(cur, prev); err != nil {
|
||||||
|
os.Remove(tmp)
|
||||||
|
return "", fmt.Errorf("keeping the old key: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(tmp, cur); err != nil {
|
||||||
|
os.Remove(tmp)
|
||||||
|
return "", fmt.Errorf("installing the new key: %w", err)
|
||||||
|
}
|
||||||
|
if d, err := os.Open(dir); err == nil {
|
||||||
|
d.Sync()
|
||||||
|
d.Close()
|
||||||
|
}
|
||||||
|
prunePrevKeys(dir, 3)
|
||||||
|
return pub, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// rekeyNow is the clock for the kept-key names (a seam).
|
||||||
|
var rekeyNow = time.Now
|
||||||
|
|
||||||
|
// prunePrevKeys keeps the newest `keep` of the `private.key.prev.<unix>` files (`private.key.prev` itself always stays).
|
||||||
|
func prunePrevKeys(dir string, keep int) {
|
||||||
|
m, _ := filepath.Glob(filepath.Join(dir, prevKeyName+".*"))
|
||||||
|
if len(m) <= keep {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
sort.Strings(m) // the same digit count until 2286: lexical = time order
|
||||||
|
for _, p := range m[:len(m)-keep] {
|
||||||
|
os.Remove(p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastRekeyFile records the last re-key's time across agent restarts (N7: the once-an-hour bound was in memory only).
|
||||||
|
const lastRekeyFile = "last-rekey"
|
||||||
|
|
||||||
|
func (m *Manager) loadLastRekey() time.Time {
|
||||||
|
b, err := os.ReadFile(filepath.Join(m.wgDir(), lastRekeyFile))
|
||||||
|
if err != nil {
|
||||||
|
return time.Time{}
|
||||||
|
}
|
||||||
|
var u int64
|
||||||
|
if _, err := fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &u); err != nil {
|
||||||
|
return time.Time{}
|
||||||
|
}
|
||||||
|
return time.Unix(u, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Manager) saveLastRekey(t time.Time) {
|
||||||
|
_ = os.WriteFile(filepath.Join(m.wgDir(), lastRekeyFile), []byte(fmt.Sprintf("%d\n", t.Unix())), 0o600)
|
||||||
|
}
|
||||||
@@ -0,0 +1,227 @@
|
|||||||
|
package wgtunnel
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// D30 (R-1204): a handshake stamped in the future that STAYS for rekeyAfter → a new key pair, registered on the
|
||||||
|
// existing peer path (the hub re-keys in place; its full-list push to ep0 swaps the key). The fake `wg` reports a
|
||||||
|
// handshake at 1783107115; a clock three hours behind that sees it three hours in the future.
|
||||||
|
// RED-PROOF (2026-10-22 night): with the clockAheadRekeyLocked call removed from Watchdog,
|
||||||
|
// TestD30_AFutureHandshakeThatStaysReKeys and …AFailedRegistration… fail (no new key, no registration); with the
|
||||||
|
// rekeyAfter wait removed, …ThatStaysReKeys fails (a re-key at 10 min); with the isActive check removed,
|
||||||
|
// TestD30_NoReKeyWhileTheTunnelIsDown fails.
|
||||||
|
|
||||||
|
const hs = 1783107115
|
||||||
|
|
||||||
|
func at(m *Manager, unix int64) { m.now = func() time.Time { return time.Unix(unix, 0) } }
|
||||||
|
|
||||||
|
func readKey(t *testing.T, m *Manager, name string) string {
|
||||||
|
t.Helper()
|
||||||
|
b, err := os.ReadFile(filepath.Join(m.stateDir, "wg", name))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(string(b))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestD30_AFutureHandshakeThatStaysReKeys(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, _, active, logs := testManager(t, fh)
|
||||||
|
oldPub := establishWG(t, m, active)
|
||||||
|
oldPriv := readKey(t, m, keyFileName)
|
||||||
|
ctx := context.Background()
|
||||||
|
calls := fh.count()
|
||||||
|
|
||||||
|
t0 := int64(hs - 3*3600) // the clock is right again; the last handshake (made while it ran ahead) is 3 h ahead
|
||||||
|
at(m, t0)
|
||||||
|
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||||
|
at(m, t0+10*60)
|
||||||
|
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||||
|
if fh.count() != calls || localPub(t, m) != oldPub {
|
||||||
|
t.Fatal("no re-key before the future stamp has stayed rekeyAfter")
|
||||||
|
}
|
||||||
|
at(m, t0+16*60)
|
||||||
|
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||||
|
newPub := localPub(t, m)
|
||||||
|
if newPub == oldPub {
|
||||||
|
t.Fatal("a future stamp that stayed 16 min must give a new key")
|
||||||
|
}
|
||||||
|
if fh.count() != calls+1 {
|
||||||
|
t.Fatalf("the new key must be registered on the existing peer path (RegisterWG calls %d → %d)", calls, fh.count())
|
||||||
|
}
|
||||||
|
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
|
||||||
|
t.Fatalf("the registration marker must carry the new key: %+v", mk)
|
||||||
|
}
|
||||||
|
if readKey(t, m, prevKeyName) != oldPriv {
|
||||||
|
t.Fatal("the old private key must be kept as private.key.prev (the escrow may hold it)")
|
||||||
|
}
|
||||||
|
if strings.Contains(logs.String(), oldPriv) || strings.Contains(logs.String(), readKey(t, m, keyFileName)) {
|
||||||
|
t.Fatal("a private key reached the log")
|
||||||
|
}
|
||||||
|
// The served block still names the OLD key until the next fetch: no second re-key over that.
|
||||||
|
at(m, t0+17*60)
|
||||||
|
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||||
|
if localPub(t, m) != newPub {
|
||||||
|
t.Fatal("a second re-key ran before the hub served the new key")
|
||||||
|
}
|
||||||
|
// The hub serves the new key → the tunnel is applied with the NEW private key.
|
||||||
|
m.Apply(ctx, true, testBlock(newPub))
|
||||||
|
conf, err := os.ReadFile(m.stagedConfPath())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(conf), readKey(t, m, keyFileName)) || strings.Contains(string(conf), oldPriv) {
|
||||||
|
t.Fatal("the applied tunnel must use the new private key, not the old one")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestD30_AShortFutureSpellDoesNotReKey(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, _, active, _ := testManager(t, fh)
|
||||||
|
pub := establishWG(t, m, active)
|
||||||
|
ctx := context.Background()
|
||||||
|
t0 := int64(hs - 3*3600)
|
||||||
|
at(m, t0)
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
at(m, hs+100) // a handshake now lies in the past: the sign ended (a working tunnel)
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
at(m, t0+20*60) // in the future again: a NEW window starts, it does not continue the old one
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
if localPub(t, m) != pub {
|
||||||
|
t.Fatal("a future stamp that did not STAY for rekeyAfter must not re-key")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestD30_NoReKeyWhileTheTunnelIsDown(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, _, active, _ := testManager(t, fh)
|
||||||
|
pub := establishWG(t, m, active)
|
||||||
|
*active = false
|
||||||
|
ctx := context.Background()
|
||||||
|
t0 := int64(hs - 3*3600)
|
||||||
|
for _, d := range []int64{0, 20 * 60, 40 * 60} {
|
||||||
|
at(m, t0+d)
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
}
|
||||||
|
if localPub(t, m) != pub {
|
||||||
|
t.Fatal("no re-key while the tunnel unit is not active")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestD30_AFailedRegistrationIsRetriedByTheExistingPath(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, _, active, _ := testManager(t, fh)
|
||||||
|
oldPub := establishWG(t, m, active)
|
||||||
|
ctx := context.Background()
|
||||||
|
t0 := int64(hs - 3*3600)
|
||||||
|
at(m, t0)
|
||||||
|
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||||
|
fh.mu.Lock()
|
||||||
|
fh.err = errors.New("hub down")
|
||||||
|
fh.mu.Unlock()
|
||||||
|
at(m, t0+16*60)
|
||||||
|
m.Watchdog(ctx, true, testBlock(oldPub))
|
||||||
|
newPub := localPub(t, m)
|
||||||
|
if newPub == oldPub {
|
||||||
|
t.Fatal("the key rotates before the registration")
|
||||||
|
}
|
||||||
|
if mk := m.loadMarker(); mk == nil || mk.Pubkey != oldPub {
|
||||||
|
t.Fatal("a failed registration leaves the marker on the old key")
|
||||||
|
}
|
||||||
|
fh.mu.Lock()
|
||||||
|
fh.err = nil
|
||||||
|
fh.mu.Unlock()
|
||||||
|
at(m, t0+30*60) // past the backoff: Apply sees the served (old) key ≠ the local key → re-key-in-place registers
|
||||||
|
m.Apply(ctx, true, testBlock(oldPub))
|
||||||
|
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
|
||||||
|
t.Fatalf("the existing re-key-in-place path must register the new key: %+v", mk)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Review 2026-10-23 night. N7: a second rotation keeps the FIRST old key as `private.key.prev` (the escrow may hold it)
|
||||||
|
// and the later one beside it; N6: the current key is replaced by a rename OVER it, so `private.key` is never missing.
|
||||||
|
// RED-PROOF: with the „prev exists → a dated name" branch removed, TwoRotations fails (the first key overwritten) —
|
||||||
|
// seen failing 2026-10-23 night.
|
||||||
|
func TestD30_TwoRotationsKeepTheFirstOldKey(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, _, active, _ := testManager(t, fh)
|
||||||
|
establishWG(t, m, active)
|
||||||
|
first := readKey(t, m, keyFileName)
|
||||||
|
old := rekeyNow
|
||||||
|
defer func() { rekeyNow = old }()
|
||||||
|
rekeyNow = func() time.Time { return time.Unix(hs, 0) }
|
||||||
|
if _, err := rotateKey(m.stateDir); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
second := readKey(t, m, keyFileName)
|
||||||
|
rekeyNow = func() time.Time { return time.Unix(hs+7200, 0) }
|
||||||
|
if _, err := rotateKey(m.stateDir); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if readKey(t, m, prevKeyName) != first {
|
||||||
|
t.Fatal("private.key.prev must keep the FIRST old key")
|
||||||
|
}
|
||||||
|
if readKey(t, m, fmt.Sprintf("%s.%d", prevKeyName, hs+7200)) != second {
|
||||||
|
t.Fatal("the second old key must be kept beside it")
|
||||||
|
}
|
||||||
|
if cur := readKey(t, m, keyFileName); cur == first || cur == second {
|
||||||
|
t.Fatal("the current key must be new")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// N7: the once-an-hour bound survives an agent restart (it was in memory only). RED-PROOF: with loadLastRekey removed
|
||||||
|
// from clockAheadRekeyLocked, this fails — seen failing 2026-10-23 night.
|
||||||
|
func TestD30_TheHourBoundSurvivesARestart(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, rr, active, _ := testManager(t, fh)
|
||||||
|
pub := establishWG(t, m, active)
|
||||||
|
ctx := context.Background()
|
||||||
|
t0 := int64(hs - 3*3600)
|
||||||
|
at(m, t0)
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
at(m, t0+16*60)
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
newPub := localPub(t, m)
|
||||||
|
if newPub == pub {
|
||||||
|
t.Fatal("fixture: the first re-key must happen")
|
||||||
|
}
|
||||||
|
// The agent restarts: a NEW manager on the same state; the served block now names the new key; the sign is back.
|
||||||
|
m2 := NewManager(rr, fh, m.stateDir, m.logger)
|
||||||
|
m2.isActive, m2.wgPresent, m2.resolver = m.isActive, m.wgPresent, m.resolver
|
||||||
|
m2.Apply(ctx, true, testBlock(newPub))
|
||||||
|
at(m2, t0+20*60)
|
||||||
|
m2.Watchdog(ctx, true, testBlock(newPub))
|
||||||
|
at(m2, t0+40*60)
|
||||||
|
m2.Watchdog(ctx, true, testBlock(newPub))
|
||||||
|
if localPub(t, m2) != newPub {
|
||||||
|
t.Fatal("a restart must not reset the once-an-hour bound")
|
||||||
|
}
|
||||||
|
_ = active
|
||||||
|
}
|
||||||
|
|
||||||
|
// Commit security review 2026-10-23 night: a last re-key saved while the clock ran ahead (a later date than now) must
|
||||||
|
// not block the next re-key until that date. RED-PROOF: with the „lastRekey after now → dropped" guard removed, this
|
||||||
|
// fails — seen failing 2026-10-23 night.
|
||||||
|
func TestD30_ALastReKeyFromTheFutureDoesNotBlock(t *testing.T) {
|
||||||
|
fh := &fakeHub{}
|
||||||
|
m, _, active, _ := testManager(t, fh)
|
||||||
|
pub := establishWG(t, m, active)
|
||||||
|
m.saveLastRekey(time.Unix(hs+2*365*86400, 0)) // written while the clock was two years ahead
|
||||||
|
ctx := context.Background()
|
||||||
|
t0 := int64(hs - 3*3600)
|
||||||
|
at(m, t0)
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
at(m, t0+16*60)
|
||||||
|
m.Watchdog(ctx, true, testBlock(pub))
|
||||||
|
if localPub(t, m) == pub {
|
||||||
|
t.Fatal("a last re-key dated in the future must not block the re-key")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -98,7 +98,8 @@ TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
|||||||
# THE retention number, read from the one file that owns it. A check and the policy it enforces
|
# THE retention number, read from the one file that owns it. A check and the policy it enforces
|
||||||
# must read the same number from the same place, or they drift and the drift looks like a defect
|
# must read the same number from the same place, or they drift and the drift looks like a defect
|
||||||
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
|
# in something else — which is exactly what happened on 2026-08-08/09 (R-287).
|
||||||
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json")
|
RETENTION_FILE = os.environ.get("FELHOM_RETENTION_POLICY") or \
|
||||||
|
os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") # the env: decoy tests only
|
||||||
|
|
||||||
|
|
||||||
def retention_kept():
|
def retention_kept():
|
||||||
@@ -114,6 +115,21 @@ def retention_kept():
|
|||||||
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
|
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
|
||||||
return n
|
return n
|
||||||
|
|
||||||
|
|
||||||
|
def retention_pinned():
|
||||||
|
"""D50 = A: the versions kept besides the newest N — the hub's MinAgent floor and the day-0 agent. Copies of hub
|
||||||
|
state the vouching session writes into the policy file (CI holds no hub secret). Missing key = no pins."""
|
||||||
|
with open(RETENTION_FILE, encoding="utf-8") as fh:
|
||||||
|
pins = json.load(fh).get("pinned") or {}
|
||||||
|
if not isinstance(pins, dict):
|
||||||
|
raise ValueError("pinned must be an object, got %r" % (pins,))
|
||||||
|
out = {}
|
||||||
|
for name, v in pins.items():
|
||||||
|
if not isinstance(v, str) or not re.match(r"^\d+\.\d+\.\d+$", v):
|
||||||
|
raise ValueError("pinned %s must be X.Y.Z, got %r" % (name, v))
|
||||||
|
out[name] = v
|
||||||
|
return out
|
||||||
|
|
||||||
tried = []
|
tried = []
|
||||||
|
|
||||||
|
|
||||||
@@ -211,6 +227,7 @@ def main():
|
|||||||
all_versions = versions
|
all_versions = versions
|
||||||
try:
|
try:
|
||||||
keep = retention_kept()
|
keep = retention_kept()
|
||||||
|
pins = retention_pinned()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
|
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
|
||||||
|
|
||||||
@@ -218,10 +235,15 @@ def main():
|
|||||||
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
|
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
|
||||||
def _key(v):
|
def _key(v):
|
||||||
return tuple(int(x) for x in v.split("."))
|
return tuple(int(x) for x in v.split("."))
|
||||||
versions = sorted(all_versions, key=_key)[-keep:]
|
window = sorted(all_versions, key=_key)[-keep:]
|
||||||
|
# D50 = A: the pinned versions are asserted wherever they sit. A pin with no tag cannot be installed at all —
|
||||||
|
# a failure, never a skip.
|
||||||
|
unpinned_tags = sorted(("%s=%s" % (k, v)) for k, v in pins.items() if v not in all_versions)
|
||||||
|
versions = sorted(set(window) | set(v for v in pins.values() if v in all_versions), key=_key)
|
||||||
dropped = [v for v in all_versions if v not in versions]
|
dropped = [v for v in all_versions if v not in versions]
|
||||||
|
|
||||||
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep))
|
print(" %d released version(s); retention policy keeps the newest %d plus the pinned %s" % (
|
||||||
|
len(all_versions), keep, ", ".join("%s=%s" % kv for kv in sorted(pins.items())) or "(none)"))
|
||||||
print(" verifying: %s" % ", ".join(versions))
|
print(" verifying: %s" % ", ".join(versions))
|
||||||
if dropped:
|
if dropped:
|
||||||
# NEVER silent. A bounded check that does not say what it stopped covering is how a
|
# NEVER silent. A bounded check that does not say what it stopped covering is how a
|
||||||
@@ -232,6 +254,9 @@ def main():
|
|||||||
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
|
print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
|
||||||
|
|
||||||
bad = []
|
bad = []
|
||||||
|
for pin in unpinned_tags:
|
||||||
|
bad.append((pin, ["the pinned version has NO git tag — nothing can install it (D50 = A)"]))
|
||||||
|
print(" FAIL pinned %s: no git tag v%s" % (pin, pin.split("=")[1]))
|
||||||
for v in versions:
|
for v in versions:
|
||||||
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||||
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
|
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
|
||||||
|
|||||||
@@ -1,41 +1,33 @@
|
|||||||
{
|
{
|
||||||
"_comment": [
|
"_comment": [
|
||||||
"THE retention number for published agent artifacts. One file, read by everything that",
|
"THE retention rule for published agent artifacts. One file, read by everything that depends on it, because a check",
|
||||||
"depends on it, because a check and the policy it enforces must read the same number from the",
|
"and the policy it enforces must read the same rule from the same place or they drift — and the drift looks like a",
|
||||||
"same place or they drift — and the drift looks like a defect in something else.",
|
"defect in something else (R-287, 2026-08-08/09; again R-1290, 2026-10-25).",
|
||||||
"",
|
"",
|
||||||
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent",
|
"D50 = A (operator ruling 2026-10-25, felhom.eu `09` §3 decision 262): the registry keeps the NEWEST 5 released agent",
|
||||||
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag",
|
"versions, PLUS the hub's MinAgent floor, PLUS the version the hub's day-0 artifact manifest names. The operator prunes",
|
||||||
"still be downloadable. Both rules are individually sensible; together they are impossible.",
|
"by hand to that rule (on 2026-10-25 he removed 0.159.0–0.160.0 and kept 0.158.1, the floor). CI asserts exactly these",
|
||||||
"CI went red at a commit whose own run had been green the day before, on a true finding that",
|
"and never more. Before D50 the number was 10 (the R-267 newest-10 prune, R-287/R-291).",
|
||||||
"no one could act on. The red will return at the next publish unless the two read one number.",
|
|
||||||
"",
|
"",
|
||||||
"HOW THE NUMBER WAS ARRIVED AT. generic_versions_kept is 10 because that is what the registry",
|
"THE PINS ARE COPIES OF HUB STATE, because CI cannot read the hub without a secret it must not hold. The session that",
|
||||||
"keeps: the deleter was ESTABLISHED on 2026-08-10 (R-287) — the newest-10 prune of generic packages",
|
"records an agent vouch (day0_agent) or declares a new MinAgent with a floor (hub_min_agent) updates `pinned` in the",
|
||||||
"run under R-267 (felhom-agent and felhom-golden generic held exactly 10 afterwards). Until then this",
|
"same commit (felhom-build-deploy skill, agent release). A pin that names a version with no tag is a FAILURE, never a",
|
||||||
"file called the 10 an observed state with no located ruling; that is superseded (corrected",
|
"skip — a pin nobody can install is the defect the pin exists to catch.",
|
||||||
"2026-10-05, R-291). Container packages are not pruned by that rule.",
|
|
||||||
"",
|
"",
|
||||||
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
|
"SO THIS FILE IS A FLOOR, NOT A LICENCE: it says what CI may assume is still downloadable. It does not authorise",
|
||||||
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
|
"deleting anything else.",
|
||||||
"operator should confirm or replace the number — at which point this file changes and both",
|
|
||||||
"readers follow it in the same commit.",
|
|
||||||
"",
|
"",
|
||||||
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled",
|
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config files from",
|
||||||
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an",
|
"raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that version at all."
|
||||||
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
|
|
||||||
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
|
|
||||||
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
|
|
||||||
"the gate does not have today. Filed as the follow-up in R-287.",
|
|
||||||
"",
|
|
||||||
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
|
|
||||||
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
|
|
||||||
"version at all — a strictly worse act than an un-downloadable binary."
|
|
||||||
],
|
],
|
||||||
"generic_versions_kept": 10,
|
"generic_versions_kept": 5,
|
||||||
"readers": [
|
"readers": [
|
||||||
"scripts/check-published-versions.py — bounds its assertion to the newest N versions"
|
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||||
],
|
],
|
||||||
"recorded": "2026-08-09",
|
"recorded": "2026-10-26",
|
||||||
"recorded_by": "CC 2026-08-09; the number's source (the R-267 newest-10 prune, established 2026-08-10 by R-287) recorded 2026-10-05 (R-291)"
|
"recorded_by": "CC 2026-10-26: day0_agent 0.166.0 (operator vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||||
|
"pinned": {
|
||||||
|
"hub_min_agent": "0.158.1",
|
||||||
|
"day0_agent": "0.166.0"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -179,13 +179,24 @@ def published_cases(base):
|
|||||||
TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3
|
TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3
|
||||||
VERS = [t[1:] for t in TAGS]
|
VERS = [t[1:] for t in TAGS]
|
||||||
|
|
||||||
|
# D50 = A: the gate also asserts the PINNED versions. The decoys run against their own policy file (the real pins
|
||||||
|
# name versions the fake Gitea does not hold); PINS is what each case writes there.
|
||||||
|
pol_dir = tempfile.mkdtemp(prefix="retention-decoy-")
|
||||||
|
pol = os.path.join(pol_dir, "retention-policy.json")
|
||||||
|
PINS = {}
|
||||||
|
|
||||||
def case(name, setup, expect_rc, must=()):
|
def case(name, setup, expect_rc, must=()):
|
||||||
FAKE.reset()
|
FAKE.reset()
|
||||||
FAKE.tags = list(TAGS)
|
FAKE.tags = list(TAGS)
|
||||||
FAKE.packages = set(VERS)
|
FAKE.packages = set(VERS)
|
||||||
FAKE.raw = set(VERS)
|
FAKE.raw = set(VERS)
|
||||||
|
PINS.clear()
|
||||||
setup()
|
setup()
|
||||||
rc, out = run([sys.executable, gate], ROOT, child_env(base))
|
with io.open(pol, "w", encoding="utf-8") as fh:
|
||||||
|
json.dump({"generic_versions_kept": keep, "pinned": dict(PINS)}, fh)
|
||||||
|
env = child_env(base)
|
||||||
|
env["FELHOM_RETENTION_POLICY"] = pol
|
||||||
|
rc, out = run([sys.executable, gate], ROOT, env)
|
||||||
if not FAKE.hits:
|
if not FAKE.hits:
|
||||||
fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name)
|
fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name)
|
||||||
report("published: " + name, rc, out, expect_rc, must)
|
report("published: " + name, rc, out, expect_rc, must)
|
||||||
@@ -220,6 +231,25 @@ def published_cases(base):
|
|||||||
case("BY DESIGN: a version older than the retention window is not asserted", retired, 0,
|
case("BY DESIGN: a version older than the retention window is not asserted", retired, 0,
|
||||||
("NOT ASSERTED", "0.9.0"))
|
("NOT ASSERTED", "0.9.0"))
|
||||||
|
|
||||||
|
def pinned_old_ok():
|
||||||
|
FAKE.tags = ["v0.8.0"] + ["v0.10.%d" % i for i in range(keep)]
|
||||||
|
FAKE.packages = set(t[1:] for t in FAKE.tags)
|
||||||
|
FAKE.raw = set(t[1:] for t in FAKE.tags)
|
||||||
|
PINS["hub_min_agent"] = "0.8.0"
|
||||||
|
case("GENUINE (D50): a pinned version older than the window is asserted and downloads", pinned_old_ok, 0,
|
||||||
|
("verifying: 0.8.0", "ALL RELEASED VERSIONS INSTALLABLE"))
|
||||||
|
|
||||||
|
def pinned_old_gone():
|
||||||
|
pinned_old_ok()
|
||||||
|
FAKE.packages.discard("0.8.0")
|
||||||
|
case("FACT (D50): a pinned version older than the window whose package 404s", pinned_old_gone, 1,
|
||||||
|
("FAIL v0.8.0", "binary NOT downloadable"))
|
||||||
|
|
||||||
|
def pinned_no_tag():
|
||||||
|
PINS["day0_agent"] = "0.150.9"
|
||||||
|
case("FACT (D50): a pin that names a version with no tag", pinned_no_tag, 1,
|
||||||
|
("FAIL pinned day0_agent=0.150.9", "no git tag"))
|
||||||
|
|
||||||
def five_hundred():
|
def five_hundred():
|
||||||
FAKE.tags_status = 500
|
FAKE.tags_status = 500
|
||||||
case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",))
|
case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",))
|
||||||
|
|||||||
Reference in New Issue
Block a user