Compare commits

...

16 Commits

Author SHA1 Message Date
admin f7e1dcbb27 R-1247: the weekly upload line test — the agent's own iperf3 client to ep0's tunnel-only server, Tuesday 10:00-15:59 after 10 quiet minutes, under the heavy-op gate, best of 3; line_test stanza + GET /host/line-test; 15 tests red-proved; unreleased (next v0.164.0)
gates / gates (push) Successful in 1m35s
2026-10-24 08:59:42 +02:00
admin 2d4b1930df REPORT: day 2026-10-23 — agent 0.163.0 released, delivered (step bundle), vouched by the operator
gates / gates (push) Successful in 1m48s
2026-10-23 11:08:14 +02:00
admin b4721eee11 CHANGELOG: v0.163.0 released (D30, D4 root SSH by key only, R-1204)
gates / gates (push) Successful in 2m5s
2026-10-23 08:44:19 +02:00
admin 1109092def D4 (R-982): root SSH from the LAN by key only — the bundle's sshd drop-in, try-reload-or-restart, an sshd -T read-back reported when not in force, a reload in undo (the reviewed v2 patch); unreleased
gates / gates (push) Successful in 2m9s
2026-10-23 08:22:17 +02:00
admin b45c47be6c REPORT: night 2026-10-22/23 (D30 live, hardened; D4 patches v2)
gates / gates (push) Successful in 2m1s
2026-10-23 00:06:13 +02:00
admin b9b831fb90 D30: a wait stamp dated after now (a clock that jumped back) is dropped, never a block until that date — the commit's security review; red-proved; unreleased
gates / gates (push) Successful in 2m3s
2026-10-22 23:08:25 +02:00
admin 39fb9cbf6c D30 rotation hardened after review: the key file is replaced by an fsynced rename over it (never missing or empty), the first old key stays private.key.prev, later ones dated (3 kept), the once-an-hour bound survives a restart; red-proved; CONTEXT; unreleased
gates / gates (push) Successful in 2m24s
2026-10-22 23:00:05 +02:00
admin ae69ad4512 D30 (R-1204): a handshake stamped in the future for 15 min gives a new WireGuard key, registered on the existing peer path (the hub re-keys in place; its full-list push swaps the key on ep0 — nothing new on ep0); old key kept as private.key.prev; red-proved; unreleased
gates / gates (push) Successful in 2m0s
2026-10-22 20:52:52 +02:00
admin e803ad2b91 night 2026-10-22: REPORT; CHANGELOG corrected — a push of the installer to main publishes nothing (R-110, tag installer-v1.35.0); D4 stays a patch because both halves need a release
gates / gates (push) Successful in 2m24s
2026-10-22 18:13:35 +02:00
admin 7651461be2 D4 (R-982) moved out of main into a ready patch — its installer half would publish the installer (forbidden tonight); R-1204 stays; unreleased
gates / gates (push) Successful in 2m45s
2026-10-22 17:10:46 +02:00
admin a0c88e5183 R-982 (D4): root SSH from the LAN by key only — a bundle drop-in, checked and reloaded by felhom-os-apply; R-1204: the heartbeat reports a future-stamped tunnel handshake; unreleased
gates / gates (push) Successful in 2m23s
2026-10-22 16:58:45 +02:00
admin 2d5840ba11 v0.162.0: released (operator yes 2026-10-22), delivered to the 3 test boxes; CHANGELOG + REPORT
gates / gates (push) Successful in 2m10s
2026-10-22 10:39:06 +02:00
admin b310773dbf REPORT: the night of 2026-10-21 (host swap report, R-1194), unreleased
gates / gates (push) Successful in 2m7s
2026-10-22 00:06:21 +02:00
admin 0da792e670 R-1192 follow-up: report host swap vs the guests' swap allowance (log once, hub alarm, local API); R-1194: no OOB stanza before the first felhom-sshd pass; unreleased
gates / gates (push) Successful in 1m56s
2026-10-21 21:29:07 +02:00
admin 286a7e3ecf R-1192: every guest may swap as much as its RAM (bring-up, RAM resize, once at start) — the measured 8 GB freeze; unreleased
gates / gates (push) Successful in 1m39s
2026-10-21 15:43:18 +02:00
admin 4e1621c650 CHANGELOG + CONTEXT + REPORT: v0.161.0 released (operator yes) and delivered to demo-hp, Tester 1, demo-felhom; not vouched; R-1083 left out
gates / gates (push) Successful in 1m27s
2026-10-20 09:07:30 +02:00
34 changed files with 2630 additions and 29 deletions
+137 -3
View File
@@ -1,7 +1,141 @@
## Unreleased (2026-10-19) — three fixes for the next release (R-921, R-1070, R-1010); two texts made true after 0.160.0 ## Unreleased (next: v0.164.0; 2026-10-24 attended release run) — the weekly upload line test (R-1247)
Not released (the night brief of 2026-10-19: agent code + tests + commit only; the release is the operator's word). No Operator ruling 2026-10-24 07:26: once a week the box's HOST measures its upload line with a 10-second iperf3 upload to an
sudoers, wrapper or config-bundle change. iperf3 server on ep0 that answers **only inside the WireGuard tunnel** (felhom.eu `scripts/ep0-iperf3/`; installed the same
day). The agent speaks the iperf3 protocol itself (operator's pick, 2026-10-24: nothing is installed on any host).
**No sudoers or config-bundle change.** **MinController: none** — the controller (≥ 0.342.0) reads the new route and
treats a 404 as „not measured".
- **`internal/linetest`** — `Client` (the iperf3 3.x client protocol: cookie, params, N data streams, TEST_END, results
exchange; the value is the SERVER's received bytes) and `Tester` (the schedule): due weekly from Tuesday 10:00 local,
starts 10:00–15:59 only (never in the night chain), from a per-host slot spread over 10:00–13:59; only after 10 quiet
minutes (every one-minute transmit rate of the host's physical uplink below 2 Mbit/s); under the one-heavy-op gate
(`line-test`); 3 runs of 10 s with 4 streams, 30 s apart, the highest counts; each run bounded by 25 s; „server busy" →
tried again at the next tick the same day, at most 6 tries a day; no operational tunnel → no test, no value. The last
result is persisted (`<oob state dir>/line-test.json`).
- **Report:** a `line_test` stanza (`schedule`, `last_ok_at`, `mbps`, `runs_mbps`, `server`, `last_attempt_at`,
`last_error`). **Local API:** `GET /host/line-test` → `{present, mbps, measured_at, method:"iperf3-tunnel"}`.
- `wgtunnel.Manager.LineTestTarget()` — the desired block's `pbs_tunnel_ip`, only while the tunnel is operational
(cached, no exec). Config `line_test.disable` (default on).
- **Proven live** (felhom.eu `audits/line-test-2026-10-24/`): a probe built from this package on demo-hp's host read
**90.7 Mbit/s** in 10.4 s (the push floor said „at least 10"); a second test 2 s later → „server busy" in 0.1 s while the
first went on undisturbed (90.6).
- Tests, each red-proved by a mutation (15): `TestClientSpeaksTheProtocolAndCountsTheServersBytes` (against a fake
iperf3 server), `TestClientBusyServerIsErrBusy`, `TestClientHonoursTheDeadlineAgainstAStalledServer`,
`TestEligibleWindowNeverInTheNightChain`, `TestNeverTestsInTheNightOrBeforeTuesday`, `TestBusyServerIsRetriedLaterTheSameDay`,
`TestTriesPerDayAreBounded`, `TestNoTunnelNoTest`, `TestBusyUplinkOrHeavyOpWaits`, `TestTickMeasuresTheHighestOfThreeRuns`,
`TestStateSurvivesARestartAndReportsIt`, `TestUplinkCountsTheBridgesPhysicalPortsOnly`, `TestLineTestRoute`,
`TestLineTestTargetOnlyWhileOperational`, `TestMainWiresTheLineTest`.
## v0.163.0 — the tunnel heals itself after a clock that ran ahead (D30, R-1204); root SSH from the LAN by key only (D4, R-982) (2026-10-23) — RELEASED on the operator's yes (day brief); the vouch is the operator's
Binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`; config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`. Built from `1109092` (CI 2246 green). Both sections below are in it.
### The tunnel re-key (D30) and the D4 bundle drop-in
- **D30 (operator 2026-10-22: the agent makes a new key and registers it on the existing peer path).** When the latest
handshake stays stamped in the future for 15 minutes (the R-1204 sign: ep0 keeps the future timestamp and refuses
every newer handshake from that key), the agent makes a new WireGuard key pair, keeps the old private key as
`wg/private.key.prev` (0600 — the escrow may hold it), and registers the new public key through `POST /hosts/<id>/wg`.
The hub re-keys the peer IN PLACE (same /32, generation bump) and its full-list push to ep0 drops the old key and adds
the new one — nothing new runs on ep0. A new key is a new peer for ep0, with no stored timestamp, so the next
handshake is accepted. At most once an hour; not while the tunnel unit is down; never while a re-register is already
in flight. A failed registration is retried by the existing re-key-in-place path. Logs carry public keys only.
Tests `TestD30_*` (4), red-proved. A host-loss DR that installs the escrowed (older) key registers it again, and the
hub re-keys in place back to it — the same path.
- **Review of the night's own code (felhom.eu `REPORT-night-2026-10-23.md`, N6, N7).** The rotation never leaves the box
without a readable key: the new key is written and fsynced to a temp file, the current key gets its kept name by a
hard link, then the temp file is renamed OVER `private.key` and the directory fsynced (a power cut or a kill between
two renames used to leave no key, or an empty one). `private.key.prev` keeps the FIRST old key (the escrow may hold it);
later ones are `private.key.prev.<unix>`, the newest three kept. The once-an-hour bound is saved in `wg/last-rekey`, so
an agent restart does not reset it; a start or a last re-key dated AFTER now (a clock that jumped back) is dropped,
never a block until that date (the commit's own security review). Tests `TestD30_TwoRotationsKeepTheFirstOldKey`,
`TestD30_ALastReKeyFromTheFutureDoesNotBlock`,
`TestD30_TheHourBoundSurvivesARestart`, red-proved. **Live (felhom.eu `audits/night-2026-10-23/P2-D30/`):** the D30
re-key ran on a test box (before these two changes): clock corrected → 15 min → new key registered → the box switched
at its next desired-state fetch → a handshake with the new key.
- **D4 (operator ruling 2026-10-22, R-982) — root SSH from the LAN by key only, on `main` now (2026-10-23 day brief).** The
config bundle carries `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`;
content check: exactly that line, `sshd -t -f`). After an install that wrote it: `sshd -t` (the whole config), then
`systemctl try-reload-or-restart ssh` (a stopped or socket-activated sshd no longer refuses the bundle), then `sshd -T`
is read back and a drop-in that is NOT in force (no Include, an earlier drop-in, a Match block) is said loudly and
reported as `sshd_dropin_not_in_force` — never a refusal; an undone bundle reloads sshd again. The console and the
web page keep the root password; felhom-sshd (8822) is not touched. Tests `RootKeyOnly` (6), red-proved; the
installer's uninstall removes the file (felhom.eu installer 1.36.0). Reviewed 2026-10-23 night (A1–A7).
- **Known gap, unchanged (filed):** after a reboot the kernel forgets the future stamp (latest handshake 0), so neither
this re-key nor the hub's alarm sees a box whose clock was corrected and which then restarted (R-1223).
### The future-stamped handshake is reported (R-1204) — first written on the 2026-10-22 night
- **R-982 / D4 — a ready patch on that night; on `main` since `1109092` and released in v0.163.0 (see above).** Built and proven the same night (felhom.eu
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
nested box from ISO 1.30.0: root password over SSH refused, root key works, the console and the web page still take
the password. It was committed (`a0c88e5`) and moved back out (`7651461`): the installer's uninstall must remove every bundle file (felhom.eu `scripts/test_hostinstall.py`), the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110 — a push to `main` alone publishes nothing) — both forbidden tonight. *(Corrected the same night: the first reason written here, „a push of `felhom-host-install.sh` publishes the installer", was wrong — the website serves `/scripts/` from the tag `installer-v1.35.0`, read live in `manifests/webpage.yaml` and the running Deployment.)* Both halves are patches: `felhom-agent-bundle-D4.patch` and `felhom.eu-installer-D4.patch` in that folder; apply them together with the agent release and the next installer tag.
- **R-1204 — `wireguard.handshake_future_s`.** A box that ran once with its clock ahead keeps a handshake stamped in the
future; ep0 refuses every newer one, and `last_handshake_age_s` (clamped at 0) read „just now" for ever. The heartbeat
now carries how many seconds the handshake lies ahead (> 60 s only); the age keeps its clamp. The hub alarms on it
(felhom.eu hub `wg_tunnel_stale`). Test `TestR1204_AFutureHandshakeIsReported`, red-proved.
## v0.162.0 — every guest may swap as much as its RAM (R-1192, the 8 GB freeze); the host's swap is reported and checked; no false „operator access degraded" on a fresh box (R-1194) (2026-10-22) — RELEASED on the operator's yes; the vouch is the operator's
Released 2026-10-22 07:40 UTC (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at
`b310773`: binary sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (the same bytes as the
night's candidate), config bundle sha256 `ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5`, tag
`v0.162.0`; no `configs/` change since v0.161.0. Delivered by one signed `agent_update` job per box to demo-hp, Tester 1
and demo-felhom (07:49–07:58 UTC): each `signed op COMPLETED`, `capabilities self-check ok=76 total=76`, `update
committed`, then `guest-swap: raised` for its 9201 (512 MB → its RAM). Evidence felhom.eu
`documentation/audits/morning-2026-10-22/release/04-agent-delivery.txt`. **Found at once:** demo-hp (host swap 8191 MB,
guest swap 25898 MB) and demo-felhom (8191 MB vs 12288 MB) logged the host-swap WARN and the hub mailed
`host_swap_short` — the check compares the host's swap with the guests' whole swap allowance, which a guest larger
than the installer's 8 GB host swap always exceeds (felhom.eu register R-1216). Soak (night of 2026-10-21, 8 GB box):
11 h, no freeze, the host's swap peaked at 2.7 GB used.
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
never set it. On the 8 GB machine (guest 4096 MB) the first uploads made Immich load its analysis models; the guest hit
its limit with the swap full and could evict only program code — 150 000–600 000 refaults and ~1 GB of disk reads per
10 s, memory „full" pressure 92–95 %, load 155, the dashboard dead for 6½ minutes, no app killed. Raising the swap cap
to 4096 MB live ended it within a minute while the uploads ran on.
- `reconcile.buildBringUpConfig`: a new guest gets `swap` = its memory.
- `POST /guest/memory` (RAM resize): `swap` moves with `memory` (`localapi.GuestSwapMB`).
- At agent start (one minute in): `localapi.EnsureGuestSwap` raises every LXC in the `felhom` pool whose swap is below
its RAM to it — a live cgroup change, no restart; never lowers a higher swap; one guest's failure is logged and the
others are still done.
- Tests `TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM`, `TestR1192_BringUpSetsSwapToTheRAM`, the resize test's
new swap assertion — each red-proved (the write removed → `raised 0, want 2`; `swap=""` on both other paths).
- Privilege: `swap` is under `VM.Config.Memory`, which the RAM resize already uses; no role, sudoers or bundle change.
Added the night of 2026-10-21 (still unreleased, the same release):
- **The host's swap is reported and checked (R-1192 follow-up).** The guest fix moves a memory burst onto the HOST's
swap; a host with none gets nothing from it. The host report carries `host.host_swap_mb` (Proxmox node status — the
kernel's SwapTotal, the same number as `/proc/meminfo`) and `host.guest_swap_mb` (the sum of the guests' `swap`; each
guest spec also carries `swap_bytes`). When the first is below the second the agent logs one WARN (and one INFO when
it is covered again); the hub raises `host_swap_short`. `GET /guest/memory` carries `host_swap_mb` and
`guest_swap_mb` for the controller's install page. The agent never creates host swap (a decision — felhom.eu
register). Measured the same night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap.
- **R-1194 — no false „operator access degraded" on a fresh box.** The first host report was sent before the
felhom-sshd loop's first pass, read a config that was never rendered („config invalid, port 0") and the hub mailed
`oob_degraded` — reproduced on a fresh install 2026-10-21 (the report at 20:34:00, the apply 0.4 s later, the daemon
healthy; the check runs `sshd -t` through sudo, so the non-root theory was false). The loop now reports no OOB stanza
until its first pass has run (a failed pass counts, so a real lasting failure still reaches the hub).
- Tests `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass` — each red-proved.
## v0.161.0 — three fixes (R-921, R-1070, R-1010); two texts made true after 0.160.0 (2026-10-20) — RELEASED on the operator's yes, NOT VOUCHED
Released 2026-10-20 (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at `5152123`:
binary sha256 `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle sha256
`485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47`, tag `v0.161.0`. Delivered by one signed
`agent_update` job per box to demo-hp (06:38 UTC), Tester 1 (06:54) and demo-felhom (06:55): each `signed op COMPLETED`,
`capabilities self-check ok=76 total=76`, `update committed`. **The vouch is the operator's.** Live on demo-hp: `host_busy`
answers `{"busy": true, "what": "backup:local"}` during a „Mentés most" whole-guest backup and `{"busy": false}` before
and after (read from inside 9201 with its own token; felhom.eu `audits/night2-2026-10-20/P1/`). R-1083 (a key for an
existing guest) is NOT in this release: see the R-1083 row. No sudoers, wrapper or config-bundle change.
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the - **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
+19
View File
@@ -1,5 +1,24 @@
# CONTEXT — felhom-agent working state # CONTEXT — felhom-agent working state
> **2026-10-24 (attended release run) — v0.164.0: the weekly upload line test (R-1247; operator ruling 07:26).**
> `internal/linetest`: the agent's own iperf3 client (operator's pick — nothing installed on hosts) to ep0's tunnel-only
> server `10.77.0.1:5201`, Tuesday 10:00-15:59, after 10 quiet minutes, under the heavy-op gate, best of 3; `line_test`
> stanza + `GET /host/line-test` (the controller's qBittorrent limit, D41 = 50 %). demo-hp measured 90.7 Mbit/s. Release
> needs the operator's vouch.
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
> key on ep0). Live on a test box (felhom.eu `audits/night-2026-10-23/P2-D30/`): with ep0 free to start a handshake the
> tunnel heals by itself (no re-key, correct); with ep0's starts dropped it stays dead and the agent re-keyed at 15 min.
> The interface switches at the next desired-state fetch. Gap: a reboot forgets the future stamp (R-1223). The D4 bundle
> patch v2 (`P3-review/felhom-agent-bundle-D4-v2.patch`): `try-reload-or-restart`, an `sshd -T` read-back reported as
> `sshd_dropin_not_in_force`, a reload in undo. Next agent release = D30 + D4 v2 + R-1204's field, the operator vouches.
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978, > **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with > R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn > `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
+33 -17
View File
@@ -1,20 +1,36 @@
# REPORT — agent, evening/night brief 2026-10-19 (code + tests + commit only, NOT released) # REPORT — day 2026-10-23: agent 0.163.0 released, delivered to the three test boxes, vouched by the operator
Commit `56aabc7` (CHANGELOG „Unreleased"), scanner wiring `84ef581`. No sudoers, wrapper or config-bundle change. **What:** the day brief's Part 2. Design read: `03-host-agent.md` (the tunnel, the bundle), `11` §5.4.2 (the step
The release is the operator's word (STATUS morning list: „Release agent 0.161.0?"), then the vouch. bundle, R-880).
- **R-921 — `host_busy` on `GET /backup/status`:** whether the host-wide heavy-operation gate is held (`{busy, what}`), ## Release
the state in which `POST /backup` answers 409. Controller 0.335.0 (the floor) reads it before any stop. Test
`TestBackupStatus_R921_HostBusy`, red-proved (assignment removed → `free gate: host_busy = <nil>`).
- **R-1070 — tunnel `absent`:** Docker 29 prints „error: no such object" in lower case; the case-sensitive match read
a missing cloudflared container as `unknown`, which failed the host OS-update rule (Tester4, 2026-10-15). Now a
missing container is `absent`, and the host rule passes it only when the reading from before the step also had no
cloudflared container. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp`, `TestClassifyTunnel` (Docker 29's wording,
measured on demo-hp), red-proved both ways.
- **R-1010 — the janitor's forced snapshot delete:** on „Failed to find logical volume …_vzdump" only, one retry with
the API's `force=1` (`DeleteSnapshotForce`; `pvesh usage … -command delete` lists `--force`). Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows), `TestDeleteSnapshotForce_R1010`, red-proved.
- **R-986 — the security-scan gate** (tooling): the shared `felhom.eu/scripts/security_scan_gate.py` in the fast set;
`.security-accepted` holds 15 accepted test constants; decoys in `test_gate_decoys.py`.
Gates: `go build/vet/test ./...` green; `agent_gates.py --fast` OK; decoys 36 cases OK. CI 2103, 2110 success. - Code: `1109092` (D4 v2: the sshd drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`,
`PermitRootLogin prohibit-password`, `try-reload-or-restart`, an `sshd -T` read-back; RootKeyOnly 6 tests). CI 2246
success. CHANGELOG `b4721ee` (v0.163.0 header, both sha256), CI 2249 success.
- `scripts/release-agent.sh 0.163.0`: binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`,
config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`; tag `v0.163.0`; verified by
an independent download.
- **Vouched by the operator** 07:58:46 UTC — read back from the hub DB (with its WAL) and from the running hub's
`/configuration` page: agent 0.163.0, both sha256 as above.
## Delivery — three signed jobs per box (felhom-opsign, key felhom-op-1)
The D4 drop-in is a NEW bundle path, and the boxes ran bundle 0.160.0 (sha256 `6b0609…`), so: `agent_update 0.163.0`
→ `agent_config_update 0.163.0-step1` (built with `scripts/build-step-bundle.py` from 0.160.0: only
`felhom-os-apply` replaced; sha256 `6cbc0669…`, uploaded and read back) → `agent_config_update 0.163.0`.
| Box | agent | bundle | self-check | swap = RAM (pool guest 9201) | drop-in, `sshd -T` | sshd reloaded | key login | felhom-sshd |
|---|---|---|---|---|---|---|---|---|
| Tester 1 | 0.163.0 | 0.163.0 | ok | 4096 / 4096 | in place, `without-password` | 09:27:38 SIGHUP | ok | active, listening, `sshd -t` ok |
| demo-hp | 0.163.0 | 0.163.0 | ok | 25898 / 25898 | in place, `without-password` | 09:19:44 SIGHUP | ok | active, listening, `sshd -t` ok |
| demo-felhom | 0.163.0 | 0.163.0 | ok | 12288 / 12288 | in place, `without-password` | 09:24:42 SIGHUP | ok | active, listening, `sshd -t` ok |
**Not proven, said plainly:** the real root password over SSH was not tried (CC does not hold it); a dummy password is
refused, but OpenSSH logs that only as „Failed password", so the policy proof is `sshd -T` + the reload. felhom-sshd
(:8822) admits only the operator peer over the tunnel, so a connection from DooPlex times out by design. The console
login is not touched by the drop-in (`passwd -S root` = P on all three); not tried at a physical console.
Installer 1.36.0 (the uninstall removes the drop-in) was published the same day (felhom.eu). R-982 and R-1204 closed.
Evidence: felhom.eu `documentation/audits/day-2026-10-23/07-agent-0163-boxes.txt`, `10-vouch-readback.txt`,
`11-installer-1360.txt`.
+1
View File
@@ -171,6 +171,7 @@
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). | | `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct | | `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
| `linetest.Tester` + `linetest.Client` (R-1247) | internal/linetest/linetest.go · internal/linetest/iperf.go | `New(runner, target, gate, uplink, hostID, statePath, logger)` / `Tick(ctx)` / `Value()` / `LineTestStatus(ctx)`; `Client{Streams, Time}.Run(ctx, addr)` | the weekly upload line test: a 10-second iperf3 upload (the agent speaks the iperf3 protocol — nothing installed on the host) to ep0's tunnel-only server (`wgtunnel.Manager.LineTestTarget`), Tuesday 10:00-15:59 local from a per-host slot, after 10 quiet minutes of the PHYSICAL uplink (`SysUplinkTx`: the bridge's ports with a `device` link — the bridge counter misses bridged guest traffic), under `backup.InFlight`; best of 3; reported as `line_test`, served as GET /host/line-test | **The value is the SERVER's received bytes**, never the client's written ones (the send buffer). `Run` refuses a context without a deadline. iperf3 numbers streams 1, 3, 4, 5 — the server matches results by those ids. „server busy" (ACCESS_DENIED) stops the measurement and is retried at the next tick, at most `MaxTriesPerDay` |
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. | | `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. | | `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. | | `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
+64
View File
@@ -0,0 +1,64 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-1247: the weekly line test has the fstrim shape (component + reporter seam + goroutine) plus a local-API reader,
// so its wiring is asserted from the AST. It must share the ONE heavy-op gate (heavyOps) — or it could run beside a
// backup — and take its server from the tunnel manager, never a literal address.
func TestMainWiresTheLineTest(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var withGate, withTunnelTarget, reporterWired, localAPIWired, started bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
switch fn.Sel.Name {
case "New":
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "linetest" && len(node.Args) >= 3 {
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
withGate = true
}
if sel, ok := node.Args[1].(*ast.SelectorExpr); ok && sel.Sel.Name == "LineTestTarget" {
withTunnelTarget = true
}
}
case "SetLineTestReporter":
reporterWired = true
case "SetLineTest":
localAPIWired = true
}
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "lineTest" {
started = true
}
}
}
return true
})
if !withGate {
t.Error("main.go never calls linetest.New(..., heavyOps, ...) — no line test, or one outside the heavy-op gate")
}
if !withTunnelTarget {
t.Error("linetest.New does not take its server from wgMgr.LineTestTarget")
}
if !reporterWired {
t.Error("main.go never calls collector.SetLineTestReporter — the line_test stanza never reaches the hub")
}
if !localAPIWired {
t.Error("main.go never calls localSrv.SetLineTest — the controller's GET /host/line-test always reads present:false")
}
if !started {
t.Error("main.go never starts the line test with `go lineTest.Run(ctx)`")
}
}
+35
View File
@@ -43,6 +43,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet" "gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver" "gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi" "gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log" applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane" "gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
@@ -936,6 +937,22 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its // network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged. // alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait}) go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
if n, err := localapi.EnsureGuestSwap(sctx, px, reconcile.DefaultPool, logger); err != nil {
logger.Warn("guest-swap: check skipped", "err", err)
} else {
logger.Info("guest-swap: checked", "raised", n)
}
}()
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue // Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving // (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -1625,6 +1642,24 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} else { } else {
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)") logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
} }
// R-1247 (operator ruling 2026-10-24 07:26): the weekly upload line test — a 10-second iperf3 upload from this host
// to ep0's tunnel-only server, Tuesday 10:00-15:59 local, after 10 quiet minutes, under the one-heavy-op gate. Only
// with the offsite tunnel (the server is reachable nowhere else). Not part of the errc fan-out.
switch {
case !cfg.LineTest.Enabled():
logger.Info("linetest: weekly upload line test disabled by config (line_test.disable)")
case wgMgr == nil:
logger.Info("linetest: weekly upload line test off — no offsite tunnel on this host")
default:
lineTest := linetest.New(linetest.Client{Streams: linetest.Streams, Time: linetest.TestTime}, wgMgr.LineTestTarget,
heavyOps, linetest.SysUplinkTx("/"), cfg.Hub.HostID,
filepath.Join(cfg.OOB.WithDefaults().StateDir, "line-test.json"), logger)
collector.SetLineTestReporter(lineTest)
if localSrv != nil {
localSrv.SetLineTest(lineTest)
}
go lineTest.Run(ctx)
}
if lanLoop != nil { if lanLoop != nil {
lanServers = 1 lanServers = 1
go func() { errc <- lanLoop.Run(ctx) }() go func() { errc <- lanLoop.Run(ctx) }()
+41 -3
View File
@@ -165,7 +165,8 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# dest: (source file under configs/, mode, check, policy) # dest: (source file under configs/, mode, check, policy)
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile; # check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident); # unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none. # agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
# sshd → `sshd -t -f` on the snippet alone (R-982).
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may # policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none). # have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none).
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it. # Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
@@ -184,6 +185,9 @@ BUNDLE_KEEP = 3
BUNDLE_MAX_BYTES = 4 * 1024 * 1024 BUNDLE_MAX_BYTES = 4 * 1024 * 1024
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$") BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
OOB_DIR = "/etc/felhom-sshd" OOB_DIR = "/etc/felhom-sshd"
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
BUNDLE_FILES = [ BUNDLE_FILES = [
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"), ("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"), ("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
@@ -221,6 +225,9 @@ BUNDLE_FILES = [
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"), ("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"), ("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"), ("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
# R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
# password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "replace"),
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"), ("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"), ("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
] ]
@@ -434,7 +441,8 @@ class Runner:
def check_content(self, kind, data): def check_content(self, kind, data):
"""Run an external syntax check on a root-only temp copy. Returns (rc, message).""" """Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
import tempfile import tempfile
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind] cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"],
"sshd": ["/usr/sbin/sshd", "-t", "-f"]}[kind]
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d: with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
p = os.path.join(d, "f") p = os.path.join(d, "f")
with open(p, "wb") as f: with open(p, "wb") as f:
@@ -1864,6 +1872,7 @@ class Bundle:
def __init__(self, apply): def __init__(self, apply):
self.a, self.r = apply, apply.r self.a, self.r = apply, apply.r
self.report = apply.report self.report = apply.report
self.sshd_dropin_not_in_force = False
# ---------- reading and checking ---------- # ---------- reading and checking ----------
def load_file(self, path): def load_file(self, path):
@@ -1932,10 +1941,17 @@ class Bundle:
text = content.decode("utf-8") text = content.decode("utf-8")
except UnicodeDecodeError: except UnicodeDecodeError:
raise Refused("R18", f"{dest} is not UTF-8 text") raise Refused("R18", f"{dest} is not UTF-8 text")
if check in ("sudoers", "sh", "bash", "nft"): if check in ("sudoers", "sh", "bash", "nft", "sshd"):
rc, msg = self.r.check_content(check, content) rc, msg = self.r.check_content(check, content)
if rc != 0: if rc != 0:
raise Refused("R18", f"{dest} fails its {check} check: {msg}") raise Refused("R18", f"{dest} fails its {check} check: {msg}")
if check == "sshd":
# R-982: the drop-in's PermitRootLogin must be exactly `prohibit-password`. `no` would also shut the root key
# out (Proxmox's own node-to-node SSH and the LAN key route), `yes` would undo the ruling — both refused.
# Only that line is pinned: the file is signed like every bundle file (review A7).
vals = re.findall(r"^\s*PermitRootLogin\s+(\S+)\s*$", text, re.M | re.I)
if vals != ["prohibit-password"]:
raise Refused("R18", f"{dest} must set exactly `PermitRootLogin prohibit-password` (found {vals})")
elif check == "python": elif check == "python":
try: try:
compile(text, dest, "exec") compile(text, dest, "exec")
@@ -2044,6 +2060,8 @@ class Bundle:
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]} self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
return 3 return 3
rep["signers_created"] = False rep["signers_created"] = False
if self.sshd_dropin_not_in_force:
rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
if bootstrap and not self.r.lexists(TRUST_SIGNERS): if bootstrap and not self.r.lexists(TRUST_SIGNERS):
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644) self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
rep["signers_created"] = True rep["signers_created"] = True
@@ -2073,6 +2091,23 @@ class Bundle:
"enable the crash guard") "enable the crash guard")
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests: if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer") self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
# R-982: the host's sshd takes the root-key-only drop-in. The WHOLE config is tested first (a failure is a
# refusal, and the bundle undo puts the old files back); `try-reload-or-restart` keeps every open session and
# does nothing on a box whose ssh.service is not running (a socket-activated or stopped sshd reads the file
# at its next start) — a failure there must not freeze the bundle route (review 2026-10-22 night, A1).
# felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
# sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
# — that is said loudly in the log and the report, never silently recorded as done (A2).
if SSHD_DROPIN in written:
self.must(["/usr/sbin/sshd", "-t"], "sshd -t (the host's whole sshd config)")
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", "user=root,host=felhom-check,addr=192.0.2.1"], 60)
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
if rc != 0 or eff != ["without-password"]:
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force: sshd -T says "
f"permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier drop-in "
f"or a Match block (R-982)")
self.sshd_dropin_not_in_force = True
def must(self, argv, what): def must(self, argv, what):
rc, out, err = self.r.host(argv, 120) rc, out, err = self.r.host(argv, 120)
@@ -2133,6 +2168,9 @@ class Bundle:
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}") self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
if any(d.startswith("/etc/systemd/system/") for d in back): if any(d.startswith("/etc/systemd/system/") for d in back):
self.r.host(["systemctl", "daemon-reload"], 120) self.r.host(["systemctl", "daemon-reload"], 120)
if SSHD_DROPIN in back:
# A3: after_install may already have reloaded sshd with the drop-in; the restored state takes effect now.
self.r.host(["systemctl", "try-reload-or-restart", "ssh"], 120)
rc, _, _ = self.r.host(["visudo", "-c"], 60) rc, _, _ = self.r.host(["visudo", "-c"], 60)
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}") self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
return back return back
+7
View File
@@ -0,0 +1,7 @@
# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
PermitRootLogin prohibit-password
+84
View File
@@ -429,6 +429,90 @@ class SelfCheckUndo(unittest.TestCase):
self.assert_restored(box, before, rep) self.assert_restored(box, before, rep)
class RootKeyOnly(unittest.TestCase):
"""R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
whole config and reloads sshd — only when the file was written; a drop-in that says anything but
`prohibit-password` is refused before anything is written.
RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
def test_fresh_box_takes_the_dropin_and_reloads(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
self.assertEqual(box.modes[self.DEST], 0o644)
self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
def test_an_unchanged_dropin_is_not_reloaded(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_dropin_that_says_yes_or_no_is_refused(self):
for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
def edit(files, bad=bad):
for e in files:
if e["path"] == self.DEST:
e["content_b64"] = base64.b64encode(bad).decode()
data, sha = edited_bundle(edit)
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertNotEqual(rc, 0, bad)
self.assertNotIn(self.DEST, box.files, bad)
# Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
# said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
# after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
# test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
# fails — seen failing 2026-10-22 night.
def _sshd_T(self, box, value):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
return orig(argv, timeout, stdin)
box.host = host
def test_the_effective_value_is_checked(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "without-password")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_dropin_not_in_force_is_reported(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
def test_an_undone_bundle_reloads_sshd(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.visudo_fail = True # the self-check fails AFTER the install → undo
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assertNotIn(self.DEST, box.files)
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
class TrustBootstrap(unittest.TestCase): class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self): def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle() data, sha = real_bundle()
+11
View File
@@ -34,6 +34,7 @@ type Config struct {
WGTunnel WGTunnelConfig `json:"wg_tunnel"` WGTunnel WGTunnelConfig `json:"wg_tunnel"`
GuestNet GuestNetConfig `json:"guest_net"` GuestNet GuestNetConfig `json:"guest_net"`
DiskTrim DiskTrimConfig `json:"disk_trim"` DiskTrim DiskTrimConfig `json:"disk_trim"`
LineTest LineTestConfig `json:"line_test"`
OOB OOBConfig `json:"oob"` OOB OOBConfig `json:"oob"`
SelfUpdate SelfUpdateConfig `json:"selfupdate"` SelfUpdate SelfUpdateConfig `json:"selfupdate"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info) LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
@@ -150,6 +151,16 @@ type DiskTrimConfig struct {
// Enabled reports whether the weekly trim should run. // Enabled reports whether the weekly trim should run.
func (d DiskTrimConfig) Enabled() bool { return !d.Disable } func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
// LineTestConfig configures the R-1247 weekly upload line test (internal/linetest). DEFAULT-ON: it reaches only ep0's
// tunnel-only iperf3 server, so it runs only on a box that already has an operational offsite tunnel (operator
// ruling 2026-10-24 07:26: every box with a tunnel measures its line weekly).
type LineTestConfig struct {
Disable bool `json:"disable"`
}
// Enabled reports whether the weekly line test should run.
func (l LineTestConfig) Enabled() bool { return !l.Disable }
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet). // GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
// //
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other // **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
+22 -1
View File
@@ -22,6 +22,11 @@ type Loop struct {
mu sync.Mutex mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal) fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard block *hub.WireWireguard
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
passed bool
nudge chan struct{} nudge chan struct{}
} }
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
if !fetched { if !fetched {
applyBlock = nil applyBlock = nil
} }
defer l.markPassed()
port, err := l.mgr.Apply(ctx, applyBlock) port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil { if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick return // Apply logged; a claim/exhaustion or install error — retry next tick
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
return block.OOBPeerIP return block.OOBPeerIP
} }
// OOBStatus implements the hub collector's reporter seam (Part 4). // markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
// is still reported from the next report on.
func (l *Loop) markPassed() {
l.mu.Lock()
l.passed = true
l.mu.Unlock()
}
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
// until the first reconcile pass has run (R-1194).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus { func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
l.mu.Lock()
passed := l.passed
l.mu.Unlock()
if !passed {
return nil
}
_, block := l.snapshot() _, block := l.snapshot()
return l.mgr.Status(ctx, block) return l.mgr.Status(ctx, block)
} }
@@ -0,0 +1,29 @@
package felhomsshd
import (
"context"
"testing"
"time"
)
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
// counts, so a real lasting failure is still reported from the next report on.
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
r := newScriptRunner()
m := NewManager(r, t.TempDir(), nil)
m.isActive = func(context.Context) bool { return false }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
l := NewLoop(m, nil, time.Minute, nil)
if st := l.OOBStatus(context.Background()); st != nil {
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
}
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
if st := l.OOBStatus(context.Background()); st == nil {
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
}
}
+24 -1
View File
@@ -11,6 +11,7 @@ import (
"os" "os"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability" "gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -103,6 +104,12 @@ type GuestDiskTrimReporter interface {
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
} }
// LineTestReporter is the R-1247 seam the weekly line test plugs into (hub does not import linetest). nil → no
// line_test stanza.
type LineTestReporter interface {
LineTestStatus(ctx context.Context) *LineTestStatus
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow // Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing. // interfaces for unit testing.
type Collector struct { type Collector struct {
@@ -123,6 +130,7 @@ type Collector struct {
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted) ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted) guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted) diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted) deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted) foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false) selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
@@ -136,6 +144,8 @@ type Collector struct {
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256 selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
swapShortWarned atomic.Bool
} }
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is // NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -256,6 +266,12 @@ func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector
return c return c
} }
// SetLineTestReporter wires the R-1247 weekly line test as a report source (nil-safe → stanza omitted).
func (c *Collector) SetLineTestReporter(r LineTestReporter) *Collector {
c.lineTest = r
return c
}
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted). // SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector { func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
c.deliberateOff = f c.deliberateOff = f
@@ -375,6 +391,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
} }
host := hostMetrics(c.px.Node(), ns) host := hostMetrics(c.px.Node(), ns)
guests := c.collectGuests(ctx)
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{ report := &HostReport{
@@ -383,7 +401,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
AgentVersion: c.agentVersion, AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(), AgentSHA256: c.agentSHA256(),
Host: host, Host: host,
Guests: c.collectGuests(ctx), Guests: guests,
// storage_targets populated this slice (slice 5) via the observer; the rest stay // storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as []. // defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx), StorageTargets: c.collectStorage(ctx),
@@ -422,6 +440,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.diskTrim != nil { if c.diskTrim != nil {
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx) report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
} }
// R-1247: the last weekly line test (nil reporter = not wired → stanza omitted).
if c.lineTest != nil {
report.LineTest = c.lineTest.LineTestStatus(ctx)
}
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted). // R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
if c.foreignKey != nil { if c.foreignKey != nil {
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx) report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
@@ -567,6 +589,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
Cores: cfg.Cores, Cores: cfg.Cores,
MemoryBytes: g.MaxMem, MemoryBytes: g.MaxMem,
DiskBytes: g.MaxDisk, DiskBytes: g.MaxDisk,
SwapBytes: cfg.Swap * (1 << 20),
} }
} }
guests = append(guests, entry) guests = append(guests, entry)
+39
View File
@@ -0,0 +1,39 @@
package hub
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
// owner set up is a decision (felhom.eu register).
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
// swap allowance (0) is never short.
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
// state once per change.
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
hostMB := ns.Swap.Total / (1 << 20)
var guestMB int64
for _, g := range guests {
if g.Spec != nil {
guestMB += g.Spec.SwapBytes / (1 << 20)
}
}
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
if HostSwapShort(hostMB, guestMB) {
if c.swapShortWarned.CompareAndSwap(false, true) {
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
return
}
if c.swapShortWarned.CompareAndSwap(true, false) {
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
}
+42
View File
@@ -0,0 +1,42 @@
package hub
import (
"bytes"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
var buf bytes.Buffer
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
var ns proxmox.NodeStatus // a host with no swap
for i := 0; i < 3; i++ {
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
}
}
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
}
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
}
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
t.Fatalf("the recovery must be said once:\n%s", buf.String())
}
if HostSwapShort(0, 0) {
t.Fatal("a host whose guests have no swap allowance is never short")
}
}
+29 -1
View File
@@ -129,6 +129,11 @@ type HostReport struct {
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret. // wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"` GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
// LineTest is the weekly upload line test stanza (R-1247; operator ruling 2026-10-24): the schedule, the last
// measured upload line (best of the runs, Mbit/s, as ep0's iperf3 server counted it) and the last attempt. Present
// only when the job is wired. No secret.
LineTest *LineTestStatus `json:"line_test,omitempty"`
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the // ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box // restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since // cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
@@ -232,6 +237,19 @@ type GuestDiskTrimStatus struct {
Guests []GuestDiskTrim `json:"guests,omitempty"` Guests []GuestDiskTrim `json:"guests,omitempty"`
} }
// LineTestStatus is the R-1247 weekly upload line test stanza. `last_ok_at` + `mbps` are the last SUCCESSFUL
// measurement ("" / 0 = never); `last_attempt_at` + `last_error` the last attempt, whatever its verdict — never read
// the attempt time as a result.
type LineTestStatus struct {
Schedule string `json:"schedule"`
LastOKAt string `json:"last_ok_at,omitempty"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Server string `json:"server,omitempty"`
LastAttemptAt string `json:"last_attempt_at,omitempty"`
LastError string `json:"last_error,omitempty"`
}
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never // GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is // read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points. // the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
@@ -292,7 +310,11 @@ type WireguardStatus struct {
Registered bool `json:"registered"` // the registration marker exists Registered bool `json:"registered"` // the registration marker exists
Active bool `json:"active"` // wg-quick@wg-felhom is-active Active bool `json:"active"` // wg-quick@wg-felhom is-active
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"` LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32" // HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
} }
// SystemInfo is the `system` stanza (see HostReport.System). // SystemInfo is the `system` stanza (see HostReport.System).
@@ -338,6 +360,11 @@ type HostMetrics struct {
// //
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift. // Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
WrapperSHA256 string `json:"wrapper_sha256,omitempty"` WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK // Guest is one LXC. The agent reports vmid; the hub derives the guest PK
@@ -355,6 +382,7 @@ type GuestSpec struct {
Cores int `json:"cores"` Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"` MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"` DiskBytes int64 `json:"disk_bytes"`
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
} }
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest. // Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
+294
View File
@@ -0,0 +1,294 @@
package linetest
import (
"context"
"crypto/rand"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"strconv"
"sync"
"sync/atomic"
"time"
)
// A minimal iperf3 CLIENT for one TCP upload test (client → server), speaking the iperf3 control protocol (3.x) to the
// stock `iperf3 --server` on ep0 (operator ruling 2026-10-24: the agent speaks iperf3 itself, nothing is installed on
// the host). The number that counts is the SERVER's: the bytes it received, from its results JSON — bytes the client
// handed to its socket include what still sat in the send buffer when the timer ended.
//
// The protocol, as the server drives it (one state byte at a time on the control connection):
//
// client → 37-byte cookie
// server → PARAM_EXCHANGE(9) client → params JSON (4-byte big-endian length + bytes)
// server → CREATE_STREAMS(10) client opens N data connections, each starts with the same cookie
// server → TEST_START(1), TEST_RUNNING(2) client sends for the test time, then → TEST_END(4)
// server → EXCHANGE_RESULTS(13) client → its results JSON; server → its results JSON
// server → DISPLAY_RESULTS(14) client → IPERF_DONE(16)
//
// A server already running a test answers ACCESS_DENIED(-1) and closes: that is ErrBusy (another box is testing; try
// again later the same day). SERVER_ERROR(-3) / SERVER_TERMINATE(-2) end the test with an error.
const (
stParamExchange = 9
stCreateStreams = 10
stTestStart = 1
stTestRunning = 2
stTestEnd = 4
stExchangeResults = 13
stDisplayResults = 14
stIperfDone = 16
stAccessDenied = -1
stServerTerminate = -2
stServerError = -3
cookieSize = 37 // 36 characters + NUL
blockSize = 128 * 1024
maxJSONBytes = 1 << 20
)
// ErrBusy: the server is running another client's test.
var ErrBusy = errors.New("iperf3 server busy")
// Result is one upload test.
type Result struct {
Mbps float64 // the server's received bits per second / 1e6
ServerBytes int64 // bytes the server received, all streams
ClientBytes int64 // bytes the client wrote (diagnostics only)
ServerSeconds float64 // the server's measured duration
}
// Dialer opens a TCP connection (net.Dialer in production; a seam for tests).
type Dialer interface {
DialContext(ctx context.Context, network, address string) (net.Conn, error)
}
// Client runs one test. The whole run is bounded by ctx; Run sets every connection's deadline from it.
type Client struct {
Dialer Dialer
Streams int // parallel streams (-P)
Time time.Duration // test time (-t)
}
func (c Client) dialer() Dialer {
if c.Dialer != nil {
return c.Dialer
}
return &net.Dialer{Timeout: 5 * time.Second}
}
func newCookie() ([]byte, error) {
const alphabet = "abcdefghijklmnopqrstuvwxyz234567"
b := make([]byte, cookieSize)
if _, err := rand.Read(b[:cookieSize-1]); err != nil {
return nil, err
}
for i := 0; i < cookieSize-1; i++ {
b[i] = alphabet[int(b[i])%len(alphabet)]
}
b[cookieSize-1] = 0
return b, nil
}
func writeJSON(w io.Writer, v any) error {
data, err := json.Marshal(v)
if err != nil {
return err
}
var hdr [4]byte
binary.BigEndian.PutUint32(hdr[:], uint32(len(data)))
if _, err := w.Write(append(hdr[:], data...)); err != nil {
return err
}
return nil
}
func readJSON(r io.Reader, v any) error {
var hdr [4]byte
if _, err := io.ReadFull(r, hdr[:]); err != nil {
return err
}
n := binary.BigEndian.Uint32(hdr[:])
if n == 0 || n > maxJSONBytes {
return fmt.Errorf("iperf3: results JSON of %d bytes refused", n)
}
buf := make([]byte, n)
if _, err := io.ReadFull(r, buf); err != nil {
return err
}
return json.Unmarshal(buf, v)
}
type streamResult struct {
ID int `json:"id"`
Bytes int64 `json:"bytes"`
Retransmits int64 `json:"retransmits"`
Jitter float64 `json:"jitter"`
Errors int64 `json:"errors"`
OmittedErrors int64 `json:"omitted_errors"`
Packets int64 `json:"packets"`
OmittedPackets int64 `json:"omitted_packets"`
StartTime float64 `json:"start_time"`
EndTime float64 `json:"end_time"`
}
type results struct {
CPUTotal float64 `json:"cpu_util_total"`
CPUUser float64 `json:"cpu_util_user"`
CPUSystem float64 `json:"cpu_util_system"`
SenderHasRetransmits int `json:"sender_has_retransmits"`
CongestionUsed string `json:"congestion_used,omitempty"`
Streams []streamResult `json:"streams"`
}
// streamIDs are the ids iperf3 gives its streams: the first is 1, every later one is (streams so far) + 2 — so 1, 3,
// 4, 5 … (iperf_add_stream). The server matches our results by these ids.
func streamIDs(n int) []int {
ids := make([]int, n)
for i := range ids {
if i == 0 {
ids[i] = 1
} else {
ids[i] = i + 2
}
}
return ids
}
// Run performs one upload test to addr ("10.77.0.1:5201").
func (c Client) Run(ctx context.Context, addr string) (Result, error) {
var res Result
streams, dur := c.Streams, c.Time
if streams < 1 {
streams = 1
}
if dur <= 0 {
dur = 10 * time.Second
}
deadline, ok := ctx.Deadline()
if !ok {
return res, errors.New("iperf3: Run needs a context with a deadline")
}
cookie, err := newCookie()
if err != nil {
return res, err
}
ctrl, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: connect: %w", err)
}
defer ctrl.Close()
ctrl.SetDeadline(deadline)
if _, err := ctrl.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: cookie: %w", err)
}
var data []net.Conn
defer func() {
for _, d := range data {
d.Close()
}
}()
var sent atomic.Int64
var server results
var wg sync.WaitGroup
for {
var st [1]byte
if _, err := io.ReadFull(ctrl, st[:]); err != nil {
return res, fmt.Errorf("iperf3: control read: %w", err)
}
switch int8(st[0]) {
case stParamExchange:
params := map[string]any{"tcp": true, "omit": 0, "time": int(dur / time.Second), "num": 0, "blockcount": 0,
"parallel": streams, "len": blockSize, "pacing_timer": 1000, "client_version": "3.18"}
if err := writeJSON(ctrl, params); err != nil {
return res, fmt.Errorf("iperf3: params: %w", err)
}
case stCreateStreams:
for i := 0; i < streams; i++ {
d, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: data stream %d: %w", i+1, err)
}
d.SetDeadline(deadline)
data = append(data, d)
if _, err := d.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: data stream %d cookie: %w", i+1, err)
}
}
case stTestStart:
case stTestRunning:
stopAt := time.Now().Add(dur)
if stopAt.After(deadline) {
return res, errors.New("iperf3: no time left for the test inside the deadline")
}
buf := make([]byte, blockSize)
for _, d := range data {
d.SetWriteDeadline(stopAt)
wg.Add(1)
go func(d net.Conn) {
defer wg.Done()
for {
n, err := d.Write(buf)
sent.Add(int64(n))
if err != nil {
return
}
}
}(d)
}
wg.Wait()
if _, err := ctrl.Write([]byte{stTestEnd}); err != nil {
return res, fmt.Errorf("iperf3: test end: %w", err)
}
case stExchangeResults:
ids := streamIDs(len(data))
mine := results{SenderHasRetransmits: -1, CongestionUsed: "cubic"}
per := sent.Load() / int64(max(len(data), 1))
for _, id := range ids {
mine.Streams = append(mine.Streams, streamResult{ID: id, Bytes: per, Retransmits: -1, EndTime: dur.Seconds()})
}
if err := writeJSON(ctrl, mine); err != nil {
return res, fmt.Errorf("iperf3: send results: %w", err)
}
if err := readJSON(ctrl, &server); err != nil {
return res, fmt.Errorf("iperf3: read server results: %w", err)
}
case stDisplayResults:
ctrl.Write([]byte{stIperfDone})
return summarize(server, sent.Load(), dur)
case stAccessDenied:
return res, ErrBusy
case stServerTerminate, stServerError:
var codes [8]byte
io.ReadFull(ctrl, codes[:])
return res, fmt.Errorf("iperf3: server error (state %d, iperf errno %d, errno %d)", int8(st[0]),
int32(binary.BigEndian.Uint32(codes[:4])), int32(binary.BigEndian.Uint32(codes[4:])))
default:
return res, fmt.Errorf("iperf3: unexpected state %s", strconv.Itoa(int(int8(st[0]))))
}
}
}
// summarize turns the server's results into a Result. No stream bytes is an error, never a zero line.
func summarize(server results, clientBytes int64, dur time.Duration) (Result, error) {
res := Result{ClientBytes: clientBytes}
for _, s := range server.Streams {
res.ServerBytes += s.Bytes
if s.EndTime > res.ServerSeconds {
res.ServerSeconds = s.EndTime
}
}
if res.ServerBytes <= 0 {
return res, errors.New("iperf3: the server reported no received bytes")
}
secs := res.ServerSeconds
if secs <= 0 {
secs = dur.Seconds()
}
res.Mbps = float64(res.ServerBytes) * 8 / secs / 1e6
return res, nil
}
+439
View File
@@ -0,0 +1,439 @@
// Package linetest is the weekly upload line-speed test (R-1247; operator ruling 2026-10-24 07:26).
//
// Why: the controller's upload watch and qBittorrent's default limit (50 % of the line, D41) need the household's
// upload line. The passive reading — the nightly off-site push — is only a low floor (2026-10-24 night: demo-hp „at
// least 10 Mbit/s"; this test read 90.7 on the same line). So once a week the HOST (the tunnel lives on the host)
// runs a 10-second iperf3 upload to ep0's tunnel-only iperf3 server (felhom.eu scripts/ep0-iperf3). No file is left
// anywhere and nothing counts against the household's storage.
//
// The rule, each part pinned by a test in linetest_test.go:
// - Weekly: due from Tuesday 10:00 local until one measurement succeeded since then (a box that was off catches up
// on a later day, inside the window).
// - Daytime only: a test starts only 10:00–15:59 local — never in the night chain's window (01:00–06:59), and each
// host starts from its own minute (SlotMinute, spread over 10:00–13:59) so the fleet does not test at once.
// - Never beside a backup, a restore-test, an OS update or a trim: the test holds the host-wide one-heavy-op gate.
// - Only after QuietMinutes of quiet uplink: every one-minute transmit rate of the host's physical uplink below
// QuietMbps — so neither the family's upload nor a restore is measured against itself.
// - Up to RunsPerMeasurement runs of TestTime each; the highest counts. A run is bounded by RunTimeout.
// - „Server busy" (another box is testing) → tried again at the next tick the same day; at most MaxTriesPerDay.
// - No tunnel (no operational wg-felhom with a server address) → no test and no value; the controller then keeps its
// passive floor and sets no qBittorrent limit.
// - The last result is persisted, so a restart neither loses it nor re-tests.
package linetest
import (
"context"
"encoding/json"
"errors"
"fmt"
"hash/fnv"
"log/slog"
"net/netip"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
const (
Weekday = time.Tuesday
StartHour = 10 // first eligible local hour (inclusive)
EndHour = 16 // first NOT-eligible local hour: the last start is 15:59
SlotSpreadMinutes = 240
QuietMinutes = 10
QuietMbps = 2.0
RunsPerMeasurement = 3
TestTime = 10 * time.Second
Streams = 4
RunTimeout = 25 * time.Second
RunGap = 30 * time.Second
MaxTriesPerDay = 6
ServerPort = 5201
TickInterval = time.Minute
FirstTickDelay = 5 * time.Minute
GateName = "line-test"
)
// ScheduleText is carried on the host report.
const ScheduleText = "weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update"
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
type Gate interface {
TryAcquire(what string) (release func(), busy string, ok bool)
}
// Target answers the iperf3 server's tunnel address, ok=false when the tunnel is not operational.
type Target func() (netip.Addr, bool)
// Runner runs one upload test (Client in production).
type Runner interface {
Run(ctx context.Context, addr string) (Result, error)
}
// UplinkTx reads the host's physical-uplink transmit byte counter.
type UplinkTx func() (uint64, error)
// State is what is persisted (line-test.json).
type State struct {
LastOKAt time.Time `json:"last_ok_at,omitempty"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Server string `json:"server,omitempty"`
LastAttemptAt time.Time `json:"last_attempt_at,omitempty"`
LastError string `json:"last_error,omitempty"`
TriesDay string `json:"tries_day,omitempty"` // local date of TriesToday
TriesToday int `json:"tries_today,omitempty"`
}
type txSample struct {
at time.Time
tx uint64
}
// Tester is the weekly job.
type Tester struct {
runner Runner
target Target
gate Gate
uplink UplinkTx
statePath string
slot int // minutes after StartHour:00 this host may first start
logger *slog.Logger
loc *time.Location
now func() time.Time
sleep func(context.Context, time.Duration) bool
mu sync.Mutex
st State
samples []txSample
}
// New builds the job and loads the persisted state (missing = empty; corrupt = logged, empty — the cost is one extra
// test, never a lost value elsewhere).
func New(runner Runner, target Target, gate Gate, uplink UplinkTx, hostID, statePath string, logger *slog.Logger) *Tester {
if logger == nil {
logger = slog.Default()
}
t := &Tester{runner: runner, target: target, gate: gate, uplink: uplink, statePath: statePath,
slot: SlotMinute(hostID), logger: logger, loc: time.Local, now: time.Now, sleep: sleepCtx}
t.load()
return t
}
func sleepCtx(ctx context.Context, d time.Duration) bool {
tm := time.NewTimer(d)
defer tm.Stop()
select {
case <-ctx.Done():
return false
case <-tm.C:
return true
}
}
// SlotMinute spreads hosts over the first SlotSpreadMinutes of the window, stable per host id.
func SlotMinute(hostID string) int {
h := fnv.New32a()
h.Write([]byte(hostID))
return int(h.Sum32() % SlotSpreadMinutes)
}
func (t *Tester) load() {
data, err := os.ReadFile(t.statePath)
if err != nil {
if !os.IsNotExist(err) {
t.logger.Warn("linetest: state read failed — starting empty", "path", t.statePath, "err", err)
}
return
}
if err := json.Unmarshal(data, &t.st); err != nil {
t.logger.Warn("linetest: state file corrupt — starting empty", "path", t.statePath, "err", err)
t.st = State{}
}
}
func (t *Tester) saveLocked() error {
data, err := json.MarshalIndent(t.st, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
return err
}
tmp := t.statePath + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, t.statePath)
}
// EligibleAt reports whether a test may START at local time lt for a host whose slot is slot minutes after
// StartHour:00 on the due day. On a catch-up day (anchorDay=false) the slot is not waited for again.
func EligibleAt(lt time.Time, slot int, anchorDay bool) bool {
h := lt.Hour()
if h < StartHour || h >= EndHour {
return false
}
if anchorDay {
mins := (h-StartHour)*60 + lt.Minute()
return mins >= slot
}
return true
}
// weekAnchor is the most recent Tuesday StartHour:00 at or before lt.
func weekAnchor(lt time.Time) time.Time {
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
d := lt.AddDate(0, 0, -daysBack)
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
if a.After(lt) {
d = d.AddDate(0, 0, -7)
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
}
return a
}
// Due reports whether a measurement is due at lt given the state.
func Due(st State, lt time.Time) bool {
return st.LastOKAt.IsZero() || st.LastOKAt.In(lt.Location()).Before(weekAnchor(lt))
}
// sample records the uplink counter (once a minute) and keeps QuietMinutes+1 samples.
func (t *Tester) sample() {
if t.uplink == nil {
return
}
tx, err := t.uplink()
if err != nil {
t.logger.Debug("linetest: uplink counter unreadable", "err", err)
t.mu.Lock()
t.samples = nil // a gap is not quiet
t.mu.Unlock()
return
}
t.mu.Lock()
t.samples = append(t.samples, txSample{at: t.now(), tx: tx})
if len(t.samples) > QuietMinutes+1 {
t.samples = t.samples[len(t.samples)-(QuietMinutes+1):]
}
t.mu.Unlock()
}
// quiet reports whether the last QuietMinutes minutes each sent below QuietMbps, and the highest minute seen.
func quiet(samples []txSample) (bool, float64) {
if len(samples) < QuietMinutes+1 {
return false, 0
}
worst := 0.0
for i := 1; i < len(samples); i++ {
dt := samples[i].at.Sub(samples[i-1].at).Seconds()
if dt <= 0 || dt > 3*TickInterval.Seconds() || samples[i].tx < samples[i-1].tx {
return false, worst // a gap or a counter reset is not proof of quiet
}
mbps := float64(samples[i].tx-samples[i-1].tx) * 8 / dt / 1e6
if mbps > worst {
worst = mbps
}
}
return worst < QuietMbps, worst
}
// Run ticks every TickInterval until ctx ends. It never returns an error: a failed test is a reported fact.
func (t *Tester) Run(ctx context.Context) {
t.logger.Info("linetest: weekly upload line test starting", "schedule", ScheduleText, "slot_minute", t.slot)
if !t.sleep(ctx, FirstTickDelay) {
return
}
for {
t.sample()
t.Tick(ctx)
if !t.sleep(ctx, TickInterval) {
return
}
}
}
// Tick is one look.
func (t *Tester) Tick(ctx context.Context) {
lt := t.now().In(t.loc)
t.mu.Lock()
st := t.st
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if !Due(st, lt) {
return
}
anchor := weekAnchor(lt)
anchorDay := lt.YearDay() == anchor.YearDay() && lt.Year() == anchor.Year()
if !EligibleAt(lt, t.slot, anchorDay) {
return
}
day := lt.Format("2006-01-02")
if st.TriesDay == day && st.TriesToday >= MaxTriesPerDay {
return
}
ip, ok := t.target()
if !ok {
t.logger.Debug("linetest: no operational tunnel — no test")
return
}
if q, worst := quiet(samples); !q {
t.logger.Debug("linetest: uplink not quiet — waiting", "worst_minute_mbps", fmt.Sprintf("%.1f", worst), "samples", len(samples))
return
}
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
t.logger.Info("linetest: deferred — a heavy operation is in flight", "busy", busy)
return
}
defer release()
t.measure(ctx, netip.AddrPortFrom(ip, ServerPort).String(), day)
}
func (t *Tester) measure(ctx context.Context, addr, day string) {
start := t.now()
var runs []float64
var lastErr error
busy := false
for i := 0; i < RunsPerMeasurement; i++ {
if i > 0 && !t.sleep(ctx, RunGap) {
break
}
rctx, cancel := context.WithTimeout(ctx, RunTimeout)
r, err := t.runner.Run(rctx, addr)
cancel()
if errors.Is(err, ErrBusy) {
busy, lastErr = true, err
break
}
if err != nil {
lastErr = err
t.logger.Warn("linetest: run failed", "run", i+1, "server", addr, "err", err)
continue
}
runs = append(runs, r.Mbps)
t.logger.Info(fmt.Sprintf("linetest: run %d: %.1f Mbit/s", i+1, r.Mbps), "server", addr,
"server_bytes", r.ServerBytes, "server_seconds", r.ServerSeconds)
}
t.mu.Lock()
defer t.mu.Unlock()
if t.st.TriesDay != day {
t.st.TriesDay, t.st.TriesToday = day, 0
}
t.st.TriesToday++
t.st.LastAttemptAt = start.UTC()
if len(runs) > 0 {
best := runs[0]
for _, r := range runs {
if r > best {
best = r
}
}
t.st.LastOKAt, t.st.Mbps, t.st.RunsMbps, t.st.Server, t.st.LastError = start.UTC(), best, runs, addr, ""
t.logger.Info(fmt.Sprintf("linetest: upload line measured: %.1f Mbit/s (best of %d)", best, len(runs)),
"server", addr, "runs", len(runs), "duration_s", t.now().Sub(start).Seconds())
} else {
msg := "no run succeeded"
if lastErr != nil {
msg = lastErr.Error()
}
if len(msg) > 300 {
msg = msg[:300]
}
t.st.LastError = msg
if busy {
t.logger.Info("linetest: the test server is busy — trying again later today", "server", addr, "tries_today", t.st.TriesToday)
} else {
t.logger.Warn("linetest: measurement failed", "server", addr, "tries_today", t.st.TriesToday, "err", msg)
}
}
if err := t.saveLocked(); err != nil {
t.logger.Warn("linetest: state save failed — the result will not survive a restart", "path", t.statePath, "err", err)
}
}
// Value is the measured line (ok=false: never measured).
func (t *Tester) Value() (mbps float64, at time.Time, ok bool) {
t.mu.Lock()
defer t.mu.Unlock()
if t.st.LastOKAt.IsZero() || t.st.Mbps <= 0 {
return 0, time.Time{}, false
}
return t.st.Mbps, t.st.LastOKAt, true
}
// LineTestStatus implements hub.LineTestReporter: a pure read of the persisted state.
func (t *Tester) LineTestStatus(context.Context) *hub.LineTestStatus {
t.mu.Lock()
defer t.mu.Unlock()
out := &hub.LineTestStatus{Schedule: ScheduleText, Mbps: t.st.Mbps, RunsMbps: t.st.RunsMbps, Server: t.st.Server,
LastError: t.st.LastError}
if !t.st.LastOKAt.IsZero() {
out.LastOKAt = t.st.LastOKAt.UTC().Format(time.RFC3339)
}
if !t.st.LastAttemptAt.IsZero() {
out.LastAttemptAt = t.st.LastAttemptAt.UTC().Format(time.RFC3339)
}
return out
}
// ── the host's physical uplink counter ──────────────────────────────────────────────────────────────────────────
// SysUplinkTx reads the transmit bytes of the host's physical uplink under root (normally "/"): the interface of the
// default route in /proc/net/route, and — when it is a bridge (vmbr0), as on every Proxmox host — the sum of its
// PHYSICAL member ports (those with a `device` link). The bridge's own counter does not see the guests' bridged
// traffic, which is the traffic this check exists for.
func SysUplinkTx(root string) UplinkTx {
return func() (uint64, error) {
ifc, err := defaultRouteIface(filepath.Join(root, "proc/net/route"))
if err != nil {
return 0, err
}
sys := filepath.Join(root, "sys/class/net")
ports := []string{ifc}
if ents, err := os.ReadDir(filepath.Join(sys, ifc, "brif")); err == nil {
ports = ports[:0]
for _, e := range ents {
if _, err := os.Stat(filepath.Join(sys, e.Name(), "device")); err == nil {
ports = append(ports, e.Name())
}
}
if len(ports) == 0 {
return 0, fmt.Errorf("linetest: bridge %s has no physical port", ifc)
}
}
var sum uint64
for _, p := range ports {
b, err := os.ReadFile(filepath.Join(sys, p, "statistics/tx_bytes"))
if err != nil {
return 0, err
}
n, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
if err != nil {
return 0, err
}
sum += n
}
return sum, nil
}
}
func defaultRouteIface(path string) (string, error) {
b, err := os.ReadFile(path)
if err != nil {
return "", err
}
for _, line := range strings.Split(string(b), "\n")[1:] {
f := strings.Fields(line)
if len(f) >= 8 && f[1] == "00000000" && f[7] == "00000000" {
return f[0], nil
}
}
return "", errors.New("linetest: no default route")
}
+433
View File
@@ -0,0 +1,433 @@
package linetest
import (
"context"
"encoding/binary"
"encoding/json"
"errors"
"io"
"net"
"net/netip"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ── a fake iperf3 server (the server half of the protocol, as `iperf3 --server` drives it) ─────────────────────────
type fakeServer struct {
ln net.Listener
busy bool // answer ACCESS_DENIED
stall bool // accept the cookie, then never answer
recvSeconds float64 // what the server reports as its duration
gotIDs []int // the stream ids the client reported
gotParams map[string]any
mu sync.Mutex
received int64
}
func newFakeServer(t *testing.T) *fakeServer {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
s := &fakeServer{ln: ln, recvSeconds: 1}
t.Cleanup(func() { ln.Close() })
go s.serve()
return s
}
func (s *fakeServer) addr() string { return s.ln.Addr().String() }
func wjson(w io.Writer, v any) {
b, _ := json.Marshal(v)
var h [4]byte
binary.BigEndian.PutUint32(h[:], uint32(len(b)))
w.Write(append(h[:], b...))
}
func (s *fakeServer) serve() {
ctrl, err := s.ln.Accept()
if err != nil {
return
}
defer ctrl.Close()
cookie := make([]byte, cookieSize)
if _, err := io.ReadFull(ctrl, cookie); err != nil {
return
}
if s.busy {
ctrl.Write([]byte{0xff})
return
}
if s.stall {
time.Sleep(5 * time.Second)
return
}
ctrl.Write([]byte{stParamExchange})
var params map[string]any
if readJSON(ctrl, &params) != nil {
return
}
s.gotParams = params
n := int(params["parallel"].(float64))
ctrl.Write([]byte{stCreateStreams})
var wg sync.WaitGroup
var conns []net.Conn
for i := 0; i < n; i++ {
d, err := s.ln.Accept()
if err != nil {
return
}
c := make([]byte, cookieSize)
io.ReadFull(d, c)
conns = append(conns, d)
wg.Add(1)
go func(d net.Conn) {
defer wg.Done()
k, _ := io.Copy(io.Discard, d)
s.mu.Lock()
s.received += k
s.mu.Unlock()
}(d)
}
ctrl.Write([]byte{stTestStart, stTestRunning})
var st [1]byte
if _, err := io.ReadFull(ctrl, st[:]); err != nil || st[0] != stTestEnd {
return
}
ctrl.Write([]byte{stExchangeResults})
var theirs results
if readJSON(ctrl, &theirs) != nil {
return
}
for _, x := range theirs.Streams {
s.gotIDs = append(s.gotIDs, x.ID)
}
// The server counts what it RECEIVED; report a fixed figure so the test can tell it from the client's own count.
wjson(ctrl, results{Streams: []streamResult{{ID: 1, Bytes: 12_500_000, EndTime: s.recvSeconds}}})
ctrl.Write([]byte{stDisplayResults})
io.ReadFull(ctrl, st[:])
for _, d := range conns {
d.Close()
}
wg.Wait()
}
func TestClientSpeaksTheProtocolAndCountsTheServersBytes(t *testing.T) {
s := newFakeServer(t)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
r, err := Client{Streams: 4, Time: 300 * time.Millisecond}.Run(ctx, s.addr())
if err != nil {
t.Fatalf("Run: %v", err)
}
if r.ServerBytes != 12_500_000 || r.Mbps != 100 {
t.Fatalf("result = %+v; want the SERVER's 12.5 MB in 1 s = 100 Mbit/s", r)
}
if want := []int{1, 3, 4, 5}; len(s.gotIDs) != 4 || s.gotIDs[0] != want[0] || s.gotIDs[1] != want[1] || s.gotIDs[3] != want[3] {
t.Fatalf("stream ids sent = %v, want %v (iperf3 numbers its streams 1, 3, 4, 5)", s.gotIDs, want)
}
if s.gotParams["tcp"] != true || s.gotParams["parallel"].(float64) != 4 {
t.Fatalf("params = %v", s.gotParams)
}
}
func TestClientBusyServerIsErrBusy(t *testing.T) {
s := newFakeServer(t)
s.busy = true
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if _, err := (Client{Streams: 4, Time: time.Second}).Run(ctx, s.addr()); !errors.Is(err, ErrBusy) {
t.Fatalf("err = %v, want ErrBusy", err)
}
}
func TestClientHonoursTheDeadlineAgainstAStalledServer(t *testing.T) {
s := newFakeServer(t)
s.stall = true
ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond)
defer cancel()
start := time.Now()
_, err := Client{Streams: 4, Time: time.Second}.Run(ctx, s.addr())
if err == nil {
t.Fatal("a stalled server gave no error")
}
if took := time.Since(start); took > 1500*time.Millisecond {
t.Fatalf("Run took %v against a stalled server; the deadline is 400 ms", took)
}
}
func TestClientRefusesAContextWithoutDeadline(t *testing.T) {
if _, err := (Client{}).Run(context.Background(), "127.0.0.1:1"); err == nil {
t.Fatal("Run without a deadline must refuse (an unbounded test is the thing RunTimeout exists to prevent)")
}
}
// ── the schedule ────────────────────────────────────────────────────────────────────────────────────────────────
func TestEligibleWindowNeverInTheNightChain(t *testing.T) {
loc := time.FixedZone("CET", 3600)
start := time.Date(2026, 10, 19, 0, 0, 0, 0, loc) // a Monday
for m := 0; m < 7*24*60; m++ {
lt := start.Add(time.Duration(m) * time.Minute)
for _, anchorDay := range []bool{true, false} {
if !EligibleAt(lt, 0, anchorDay) {
continue
}
if h := lt.Hour(); h < StartHour || h >= EndHour || (h >= 1 && h <= 6) {
t.Fatalf("eligible at %s — outside 10:00-15:59 or inside the night chain (01:00-06:59)", lt.Format("Mon 15:04"))
}
}
}
}
func TestSlotDelaysTheFirstStartOnTheDueDayOnly(t *testing.T) {
loc := time.UTC
tue := time.Date(2026, 10, 20, 10, 30, 0, 0, loc)
if EligibleAt(tue, 45, true) {
t.Fatal("10:30 with slot 45 must wait until 10:45 on the due day")
}
if !EligibleAt(tue.Add(15*time.Minute), 45, true) {
t.Fatal("10:45 with slot 45 must be eligible")
}
if !EligibleAt(tue.Add(24*time.Hour), 45, false) {
t.Fatal("a catch-up day does not wait for the slot again")
}
if a, b := SlotMinute("demo-hp-bb76ea"), SlotMinute("demo-felhom-8363b5"); a < 0 || a >= SlotSpreadMinutes || b < 0 || b >= SlotSpreadMinutes {
t.Fatalf("slots out of range: %d %d", a, b)
}
}
func TestDueWeeklyFromTuesday(t *testing.T) {
loc := time.UTC
mon := time.Date(2026, 10, 19, 12, 0, 0, 0, loc)
tue := time.Date(2026, 10, 20, 12, 0, 0, 0, loc)
if !Due(State{}, mon) {
t.Fatal("never measured is due")
}
st := State{LastOKAt: time.Date(2026, 10, 14, 11, 0, 0, 0, loc)} // last Wednesday
if Due(st, mon) {
t.Fatal("measured after last Tuesday 10:00 is not due on Monday")
}
if !Due(st, tue) {
t.Fatal("from Tuesday 10:00 a new measurement is due")
}
}
// ── the job ─────────────────────────────────────────────────────────────────────────────────────────────────────
type fakeRunner struct {
mu sync.Mutex
calls int
out []runOut
}
type runOut struct {
mbps float64
err error
}
func (f *fakeRunner) Run(ctx context.Context, addr string) (Result, error) {
f.mu.Lock()
defer f.mu.Unlock()
i := f.calls
f.calls++
if i >= len(f.out) {
return Result{}, errors.New("no more scripted runs")
}
return Result{Mbps: f.out[i].mbps, ServerBytes: 1}, f.out[i].err
}
type fakeGate struct{ busy string }
func (g *fakeGate) TryAcquire(what string) (func(), string, bool) {
if g.busy != "" {
return nil, g.busy, false
}
return func() {}, "", true
}
type rig struct {
t *Tester
run *fakeRunner
gate *fakeGate
now time.Time
tx uint64
tunnel bool
}
// newRig: Tuesday 12:00 UTC (eligible, any slot), tunnel up, 11 quiet samples taken.
func newRig(t *testing.T, outs ...runOut) *rig {
r := &rig{run: &fakeRunner{out: outs}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
r.t.loc = time.UTC
r.t.now = func() time.Time { return r.now }
r.t.sleep = func(context.Context, time.Duration) bool { return true }
r.quietMinutes(QuietMinutes + 1)
return r
}
func (r *rig) minute(bytes uint64) {
r.now = r.now.Add(time.Minute)
r.tx += bytes
r.t.sample()
}
func (r *rig) quietMinutes(n int) {
for i := 0; i < n; i++ {
r.minute(1_000_000) // 1 MB/min = 0.13 Mbit/s
}
}
func TestTickMeasuresTheHighestOfThreeRuns(t *testing.T) {
r := newRig(t, runOut{mbps: 88}, runOut{mbps: 91.5}, runOut{mbps: 90})
r.t.Tick(context.Background())
mbps, _, ok := r.t.Value()
if !ok || mbps != 91.5 || r.run.calls != RunsPerMeasurement {
t.Fatalf("value %.1f ok=%v calls=%d; want the highest of 3 runs = 91.5", mbps, ok, r.run.calls)
}
r.t.Tick(context.Background())
if r.run.calls != RunsPerMeasurement {
t.Fatal("a measured week must not test again")
}
}
func TestBusyServerIsRetriedLaterTheSameDay(t *testing.T) {
r := newRig(t, runOut{err: ErrBusy}, runOut{mbps: 90}, runOut{mbps: 89}, runOut{mbps: 92})
r.t.Tick(context.Background())
if _, _, ok := r.t.Value(); ok || r.run.calls != 1 {
t.Fatalf("busy: ok=%v calls=%d; want no value and the measurement stopped after the busy answer", ok, r.run.calls)
}
if r.t.st.TriesToday != 1 || r.t.st.LastError == "" {
t.Fatalf("busy attempt not recorded: %+v", r.t.st)
}
r.quietMinutes(1)
r.t.Tick(context.Background())
if mbps, _, ok := r.t.Value(); !ok || mbps != 92 {
t.Fatalf("retry later the same day: value %.1f ok=%v", mbps, ok)
}
}
func TestTriesPerDayAreBounded(t *testing.T) {
outs := make([]runOut, 20)
for i := range outs {
outs[i] = runOut{err: ErrBusy}
}
r := newRig(t, outs...)
for i := 0; i < 10; i++ {
r.quietMinutes(1)
r.t.Tick(context.Background())
}
if r.run.calls != MaxTriesPerDay {
t.Fatalf("calls = %d in one day, want at most %d", r.run.calls, MaxTriesPerDay)
}
}
func TestNoTunnelNoTest(t *testing.T) {
r := newRig(t, runOut{mbps: 90})
r.tunnel = false
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested without an operational tunnel")
}
if _, _, ok := r.t.Value(); ok {
t.Fatal("a value without a test")
}
}
func TestBusyUplinkOrHeavyOpWaits(t *testing.T) {
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.minute(60_000_000) // one minute at 8 Mbit/s: the family is uploading
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested while the uplink was busy in the last 10 minutes")
}
r.quietMinutes(QuietMinutes)
r.gate.busy = "backup:felhom-pbs"
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested beside a heavy operation")
}
r.gate.busy = ""
r.t.Tick(context.Background())
if r.run.calls == 0 {
t.Fatal("never tested once the uplink was quiet and the gate free")
}
}
func TestNeverTestsInTheNightOrBeforeTuesday(t *testing.T) {
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
r.quietMinutes(QuietMinutes + 1)
r.t.Tick(context.Background())
r.t.st.LastOKAt = time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
r.now = time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC) // Monday noon, measured last week
r.quietMinutes(QuietMinutes + 1)
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatalf("tested %d times at night or when not due", r.run.calls)
}
}
func TestStateSurvivesARestartAndReportsIt(t *testing.T) {
r := newRig(t, runOut{mbps: 90.7}, runOut{err: errors.New("reset")}, runOut{mbps: 80})
r.t.Tick(context.Background())
again := New(r.run, nil, r.gate, nil, "host-x", r.t.statePath, nil)
mbps, at, ok := again.Value()
if !ok || mbps != 90.7 || at.IsZero() {
t.Fatalf("after a restart: %.1f %v %v", mbps, at, ok)
}
b, _ := json.Marshal(again.LineTestStatus(context.Background()))
var back hub.LineTestStatus
json.Unmarshal(b, &back)
if back.Mbps != 90.7 || back.LastOKAt == "" || len(back.RunsMbps) != 2 || back.Schedule == "" {
t.Fatalf("report stanza = %s", b)
}
}
// ── the uplink counter ──────────────────────────────────────────────────────────────────────────────────────────
func TestUplinkCountsTheBridgesPhysicalPortsOnly(t *testing.T) {
root := t.TempDir()
w := func(p, s string) {
os.MkdirAll(filepath.Dir(filepath.Join(root, p)), 0o755)
os.WriteFile(filepath.Join(root, p), []byte(s), 0o644)
}
w("proc/net/route", "Iface\tDestination\tGateway\tFlags\tRefCnt\tUse\tMetric\tMask\n"+
"wg-felhom\t0000004D\t00000000\t0001\t0\t0\t0\tFFFFFFFF\n"+
"vmbr0\t00000000\t0100A8C0\t0003\t0\t0\t0\t00000000\n")
w("sys/class/net/vmbr0/statistics/tx_bytes", "5\n")
w("sys/class/net/vmbr0/brif/enp1s0/.keep", "")
w("sys/class/net/vmbr0/brif/veth9201i0/.keep", "")
w("sys/class/net/enp1s0/device/.keep", "")
w("sys/class/net/enp1s0/statistics/tx_bytes", "1000\n")
w("sys/class/net/veth9201i0/statistics/tx_bytes", "777777\n")
got, err := SysUplinkTx(root)()
if err != nil || got != 1000 {
t.Fatalf("uplink tx = %d, %v; want 1000 (the physical port only — not the bridge, not the guest's veth)", got, err)
}
}
func TestQuietNeedsTenFullMinutes(t *testing.T) {
base := time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC)
var s []txSample
for i := 0; i <= QuietMinutes; i++ {
s = append(s, txSample{at: base.Add(time.Duration(i) * time.Minute), tx: uint64(i) * 1_000_000})
}
if q, _ := quiet(s); !q {
t.Fatal("ten quiet minutes must read quiet")
}
if q, _ := quiet(s[1:]); q {
t.Fatal("nine minutes of samples must not read quiet")
}
gap := append([]txSample(nil), s...)
gap[5].at = gap[5].at.Add(-10 * time.Minute)
if q, _ := quiet(gap); q {
t.Fatal("a gap in the samples must not read quiet")
}
}
+12 -1
View File
@@ -57,6 +57,10 @@ type MemoryInfo struct {
MaxMB int64 `json:"max_mb"` MaxMB int64 `json:"max_mb"`
FloorMB int64 `json:"floor_mb"` FloorMB int64 `json:"floor_mb"`
Running bool `json:"running"` Running bool `json:"running"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up): the host's swap and this guest's swap allowance, so the controller
// can tell the household when the host has too little swap behind the guest (its install page). nil = not read.
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
} }
// memoryResizeRequest is the POST /guest/memory body. // memoryResizeRequest is the POST /guest/memory body.
@@ -94,6 +98,8 @@ type memoryBounds struct {
maxMB int64 maxMB int64
floorMB int64 floorMB int64
running bool running bool
hostSwapMB int64
guestSwapMB int64
} }
func (b memoryBounds) refusal(code string) memoryRefusal { func (b memoryBounds) refusal(code string) memoryRefusal {
@@ -104,7 +110,9 @@ func (b memoryBounds) refusal(code string) memoryRefusal {
} }
func (b memoryBounds) info(vmid int) MemoryInfo { func (b memoryBounds) info(vmid int) MemoryInfo {
hostSwap, guestSwap := b.hostSwapMB, b.guestSwapMB
return MemoryInfo{ return MemoryInfo{
HostSwapMB: &hostSwap, GuestSwapMB: &guestSwap,
VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB, VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB,
MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running, MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running,
} }
@@ -139,6 +147,8 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max) hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
minMB: minGuestMemoryMB, minMB: minGuestMemoryMB,
running: st.Status == "running", running: st.Status == "running",
hostSwapMB: node.Swap.Total / mib, // R-1192 follow-up: the kernel's SwapTotal, as Proxmox reports it
guestSwapMB: cfg.Swap, // PVE config swap is MB
} }
b.maxMB = b.hostTotalMB - hostReserveMB b.maxMB = b.hostTotalMB - hostReserveMB
b.floorMB = b.usageMB + shrinkUsageMarginMB b.floorMB = b.usageMB + shrinkUsageMarginMB
@@ -225,7 +235,8 @@ func (s *Server) handleGuestMemoryResize(w http.ResponseWriter, r *http.Request,
} }
// Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on. // Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target)}) // R-1192: the swap moves with the memory (GuestSwapMB) — a grown guest with the old swap cap thrashes the same way.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target), "swap": fmt.Sprintf("%d", GuestSwapMB(target))})
if err != nil { if err != nil {
s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err) s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err)
writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error()) writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error())
+8 -1
View File
@@ -25,12 +25,14 @@ type fakeMemory struct {
setUPID string setUPID string
setErr error setErr error
applyReflect bool applyReflect bool
hostSwapB int64 // R-1192 follow-up: node swap total
guestSwapMB int64
} }
func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) { func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
f.mu.Lock() f.mu.Lock()
defer f.mu.Unlock() defer f.mu.Unlock()
return proxmox.GuestConfig{Memory: f.allocMB}, nil return proxmox.GuestConfig{Memory: f.allocMB, Swap: f.guestSwapMB}, nil
} }
func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) { func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
f.mu.Lock() f.mu.Lock()
@@ -42,6 +44,7 @@ func (f *fakeMemory) NodeStatus(_ context.Context) (proxmox.NodeStatus, error) {
defer f.mu.Unlock() defer f.mu.Unlock()
var ns proxmox.NodeStatus var ns proxmox.NodeStatus
ns.Memory.Total = f.hostTotalB ns.Memory.Total = f.hostTotalB
ns.Swap.Total = f.hostSwapB
return ns, nil return ns, nil
} }
func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) { func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) {
@@ -151,6 +154,10 @@ func TestGuestMemory_GrowAndShrink(t *testing.T) {
if f.lastParams["memory"] != "12288" { if f.lastParams["memory"] != "12288" {
t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"]) t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"])
} }
// R-1192: the swap moves with the memory.
if f.lastParams["swap"] != "12288" {
t.Errorf("SetConfig swap = %q, want 12288 (swap = RAM)", f.lastParams["swap"])
}
}) })
t.Run("shrink above floor", func(t *testing.T) { t.Run("shrink above floor", func(t *testing.T) {
f := stdFixture() // floor = max(2048, 3000+512) = 3512 f := stdFixture() // floor = max(2048, 3000+512) = 3512
+73
View File
@@ -0,0 +1,73 @@
package localapi
import (
"context"
"fmt"
"log/slog"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ── R-1192 (2026-10-21): the guest may swap as much as its RAM ────────────────────────────────────────────────────
//
// MEASURED (felhom.eu audits/fixes-2026-10-21/P4-freeze/): on the minimum machine (8 GB host, guest 4096 MB) the
// household's first uploads made Immich load its analysis models (+1.15 GB) on top of 8 apps' 1.75 GB; the guest hit
// its 4096 MB limit with its swap capped at Proxmox's LXC default of 512 MB — and full. The kernel could then evict
// only file pages: ~360 MB was left for the program code of ~25 containers, re-read from disk without end (150 000–
// 600 000 refaults and ~1 GB of reads per 10 s; memory „full" pressure 92–95 %, load 155, the dashboard dead, 0 OOM
// kills). Raising the cap to 4096 MB live ended it in under a minute (memory „full" 96 % → 1.7 %, dashboard back
// 42 s later) while the uploads ran on. The host had 7.4 GB of swap free the whole time.
//
// The agent never set `swap` at all, so every guest had the 512 MB default. Now:
// - a new guest is created with swap = its memory (reconcile.bringup);
// - a RAM resize sets swap with the memory (handleGuestMemoryResize);
// - at agent start, every LXC in the felhom pool whose swap is below its memory is raised to it (this file).
// Raising swap is a live cgroup change (memory.swap.max), no restart. It never lowers a swap someone set higher.
// GuestSwapMB is the swap a guest of memoryMB gets: the same as its RAM.
func GuestSwapMB(memoryMB int64) int64 { return memoryMB }
// SwapOps is the Proxmox surface EnsureGuestSwap needs (satisfied by *proxmox.Client).
type SwapOps interface {
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
// EnsureGuestSwap raises the swap of every LXC guest in pool to GuestSwapMB(memory) when it is below. Returns how
// many guests were raised. One guest's failure is logged and the others are still handled.
func EnsureGuestSwap(ctx context.Context, ops SwapOps, pool string, logger *slog.Logger) (int, error) {
p, err := ops.Pool(ctx, pool)
if err != nil {
return 0, fmt.Errorf("guest swap: read pool %s: %w", pool, err)
}
raised := 0
for _, m := range p.Members {
if m.Type != "lxc" || m.VMID <= 0 {
continue
}
cfg, err := ops.GuestConfig(ctx, m.VMID)
if err != nil {
logger.Warn("guest-swap: config read failed (left as it is)", "vmid", m.VMID, "err", err)
continue
}
want := GuestSwapMB(cfg.Memory)
if cfg.Memory <= 0 || cfg.Swap >= want {
logger.Debug("guest-swap: nothing to do", "vmid", m.VMID, "memory_mb", cfg.Memory, "swap_mb", cfg.Swap)
continue
}
upid, err := ops.SetConfig(ctx, m.VMID, map[string]string{"swap": fmt.Sprintf("%d", want)})
if err == nil && upid != "" {
_, err = ops.WaitTask(ctx, upid, proxmox.WaitOptions{})
}
if err != nil {
logger.Warn("guest-swap: raise failed (left as it is)", "vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want, "err", err)
continue
}
raised++
logger.Info("guest-swap: raised (R-1192 — a guest swap cap below its RAM turned a memory burst into a freeze)",
"vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want)
}
return raised, nil
}
+71
View File
@@ -0,0 +1,71 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: the freeze measured on the minimum machine (felhom.eu audits/fixes-2026-10-21/P4-freeze/) — a guest whose
// swap is capped below its RAM evicts program code instead of idle memory. EnsureGuestSwap raises every pool guest's
// swap to its RAM, never lowers one, skips non-LXC members, and survives one guest's failure.
// COMPANION RED-PROOF: return 0 before the SetConfig call → „raised 0, want 2".
type fakeSwapOps struct {
cfg map[int]proxmox.GuestConfig
failSet map[int]bool
set map[int]map[string]string
}
func (f *fakeSwapOps) Pool(context.Context, string) (proxmox.PoolInfo, error) {
return proxmox.PoolInfo{Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}, {VMID: 9202, Type: "lxc"}, {VMID: 300, Type: "qemu"},
{Type: "storage"}, {VMID: 9203, Type: "lxc"}, {VMID: 9204, Type: "lxc"}}}, nil
}
func (f *fakeSwapOps) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
c, ok := f.cfg[vmid]
if !ok {
return c, errors.New("no such guest")
}
return c, nil
}
func (f *fakeSwapOps) SetConfig(_ context.Context, vmid int, p map[string]string) (string, error) {
if f.failSet[vmid] {
return "", errors.New("pve said no")
}
if f.set == nil {
f.set = map[int]map[string]string{}
}
f.set[vmid] = p
return "", nil
}
func (f *fakeSwapOps) WaitTask(context.Context, string, proxmox.WaitOptions) (proxmox.TaskStatus, error) {
return proxmox.TaskStatus{}, nil
}
func TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM(t *testing.T) {
f := &fakeSwapOps{cfg: map[int]proxmox.GuestConfig{
9201: {Memory: 4096, Swap: 512}, // the measured box: raised
9202: {Memory: 2048, Swap: 4096}, // already higher: never lowered
9203: {Memory: 6144, Swap: 0}, // none: raised
9204: {Memory: 4096, Swap: 512}, // the write fails: logged, the others still done
}, failSet: map[int]bool{9204: true}}
n, err := EnsureGuestSwap(context.Background(), f, "felhom", slog.New(slog.NewTextHandler(io.Discard, nil)))
if err != nil {
t.Fatal(err)
}
if n != 2 {
t.Fatalf("raised %d, want 2 (9201, 9203)", n)
}
if f.set[9201]["swap"] != "4096" || f.set[9203]["swap"] != "6144" {
t.Fatalf("writes = %v", f.set)
}
if _, touched := f.set[9202]; touched {
t.Fatal("a swap above the RAM was lowered")
}
if len(f.set[9201]) != 1 {
t.Fatalf("the swap raise wrote more than the swap: %v", f.set[9201])
}
}
+37
View File
@@ -0,0 +1,37 @@
package localapi
import (
"net/http"
"time"
)
// LineTestReader is the R-1247 weekly upload line test (internal/linetest.Tester) as the local API reads it.
type LineTestReader interface {
Value() (mbps float64, at time.Time, ok bool)
}
// LineTestResponse is GET /host/line-test. Present=false: never measured, or the job is not wired (no tunnel, disabled,
// or an agent before the route answers 404 — the controller reads all three as „unknown").
type LineTestResponse struct {
Present bool `json:"present"`
Mbps float64 `json:"mbps,omitempty"`
MeasuredAt string `json:"measured_at,omitempty"` // RFC3339 UTC
Method string `json:"method,omitempty"` // "iperf3-tunnel": a 10-second upload to ep0 inside the tunnel
}
// SetLineTest wires the weekly line test (main.go builds it after the local API server).
func (s *Server) SetLineTest(r LineTestReader) { s.lineTest = r }
func (s *Server) handleLineTest(w http.ResponseWriter, r *http.Request, vmid int) {
if s.lineTest == nil {
writeOK(w, LineTestResponse{Present: false})
return
}
mbps, at, ok := s.lineTest.Value()
if !ok {
writeOK(w, LineTestResponse{Present: false})
return
}
s.logger.Debug("local-api: /host/line-test served", "vmid", vmid, "mbps", mbps, "measured_at", at.UTC().Format(time.RFC3339))
writeOK(w, LineTestResponse{Present: true, Mbps: mbps, MeasuredAt: at.UTC().Format(time.RFC3339), Method: "iperf3-tunnel"})
}
+51
View File
@@ -0,0 +1,51 @@
package localapi
import (
"encoding/json"
"log/slog"
"net/http/httptest"
"strings"
"testing"
"time"
)
type fixedLine struct {
mbps float64
ok bool
}
func (f fixedLine) Value() (float64, time.Time, bool) {
return f.mbps, time.Date(2026, 10, 24, 6, 31, 0, 0, time.UTC), f.ok
}
func lineTestBody(t *testing.T, s *Server) LineTestResponse {
t.Helper()
rec := httptest.NewRecorder()
s.handleLineTest(rec, httptest.NewRequest("GET", "/host/line-test", nil), 9201)
var env struct {
OK bool `json:"ok"`
Data LineTestResponse `json:"data"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
t.Fatalf("decode %q: %v", rec.Body.String(), err)
}
return env.Data
}
// R-1247: the controller sets qBittorrent's limit only from a PRESENT value; never measured and not wired must both
// read present:false, never a zero line.
func TestLineTestRoute(t *testing.T) {
s := &Server{logger: slog.Default()}
if got := lineTestBody(t, s); got.Present {
t.Fatalf("not wired: %+v, want present:false", got)
}
s.SetLineTest(fixedLine{ok: false})
if got := lineTestBody(t, s); got.Present || got.Mbps != 0 {
t.Fatalf("never measured: %+v, want present:false", got)
}
s.SetLineTest(fixedLine{mbps: 90.7, ok: true})
got := lineTestBody(t, s)
if !got.Present || got.Mbps != 90.7 || got.Method != "iperf3-tunnel" || !strings.HasPrefix(got.MeasuredAt, "2026-10-24T06:31") {
t.Fatalf("measured: %+v", got)
}
}
+39
View File
@@ -0,0 +1,39 @@
package localapi
import (
"encoding/json"
"testing"
)
// R-1192 follow-up (night 2026-10-21): GET /guest/memory carries the host's swap and this guest's swap allowance, so
// the controller's install page can say when the host has no swap behind the guest. 0 is a real answer (no swap);
// absent means an older agent.
// RED-PROOF: without the two MemoryInfo fields the keys are missing → fails on „host_swap_mb absent".
func TestR1192_GuestMemoryCarriesHostAndGuestSwap(t *testing.T) {
for _, c := range []struct {
name string
hostB, guestMB int64
wantHost, wantG int64
}{
{"installer host (7.75 GB swap), guest swap = RAM", 7935 * mib, 8192, 7935, 8192},
{"a host with NO swap", 0, 8192, 0, 8192},
} {
f := stdFixture()
f.hostSwapB, f.guestSwapMB = c.hostB, c.guestMB
w := do(t, memServer(t, f).Handler(), "GET", "/guest/memory", "A", "")
var m struct {
Data map[string]any `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &m); err != nil {
t.Fatal(err)
}
hs, ok := m.Data["host_swap_mb"].(float64)
if !ok {
t.Fatalf("%s: host_swap_mb absent: %s", c.name, w.Body.String())
}
gs, _ := m.Data["guest_swap_mb"].(float64)
if int64(hs) != c.wantHost || int64(gs) != c.wantG {
t.Fatalf("%s: host %v guest %v, want %d / %d", c.name, hs, gs, c.wantHost, c.wantG)
}
}
}
+4
View File
@@ -336,6 +336,7 @@ type Server struct {
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional) formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
staleLock StaleLockController // F2-b startup stale-lock recovery (optional) staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional) partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
lineTest LineTestReader // R-1247 weekly line test (optional; nil → present:false)
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog. // guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState). // Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
guestPower map[int]guestPowerState guestPower map[int]guestPowerState
@@ -573,6 +574,9 @@ func (s *Server) Handler() http.Handler {
// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller // R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false. // waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard)) mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
// read it. Read-only; never measured (or the job not wired) = present:false.
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate. // Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks)) mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates)) mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
+3
View File
@@ -645,6 +645,9 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
} }
if spec.MemoryMB > 0 { if spec.MemoryMB > 0 {
params["memory"] = strconv.Itoa(spec.MemoryMB) params["memory"] = strconv.Itoa(spec.MemoryMB)
// R-1192: swap = RAM. Left unset, Proxmox gives an LXC 512 MB, and a guest that fills its RAM can then evict
// only program code — the measured freeze (localapi.GuestSwapMB carries the reasoning and the numbers).
params["swap"] = strconv.Itoa(spec.MemoryMB)
} }
for i, m := range spec.Mounts { for i, m := range spec.Mounts {
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED // backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
+19
View File
@@ -0,0 +1,19 @@
package reconcile
import (
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: a new guest is created with swap = its RAM (left unset, Proxmox gives an LXC 512 MB — the measured freeze,
// felhom.eu audits/fixes-2026-10-21/P4-freeze/). COMPANION RED-PROOF: drop params["swap"] in buildBringUpConfig.
func TestR1192_BringUpSetsSwapToTheRAM(t *testing.T) {
p := buildBringUpConfig(BringUpSpec{MemoryMB: 4096}, proxmox.GuestConfig{})
if p["memory"] != "4096" || p["swap"] != "4096" {
t.Fatalf("bring-up params memory=%q swap=%q, want 4096/4096", p["memory"], p["swap"])
}
if q := buildBringUpConfig(BringUpSpec{}, proxmox.GuestConfig{}); q["swap"] != "" {
t.Fatalf("no memory in the spec → no swap written either, got %q", q["swap"])
}
}
+32
View File
@@ -0,0 +1,32 @@
package wgtunnel
import (
"context"
"testing"
)
// R-1247: the line test's server is the desired block's pbs_tunnel_ip, and ONLY while the tunnel is operational
// (registered + unit active) — „no tunnel → no test" rests on this answer.
func TestLineTestTargetOnlyWhileOperational(t *testing.T) {
m, _, active, _ := testManager(t, &fakeHub{})
ctx := context.Background()
if _, ok := m.LineTestTarget(); ok {
t.Fatal("a target before any Apply")
}
m.Apply(ctx, false, nil) // registers
pub := localPub(t, m)
m.Apply(ctx, true, testBlock(pub)) // installs; the unit is not active yet
if _, ok := m.LineTestTarget(); ok {
t.Fatal("a target while the unit is not active")
}
*active = true
m.Apply(ctx, true, testBlock(pub))
ip, ok := m.LineTestTarget()
if !ok || ip.String() != "10.77.0.1" {
t.Fatalf("operational tunnel: target %v ok=%v, want 10.77.0.1", ip, ok)
}
m.Apply(ctx, true, nil) // revoked
if _, ok := m.LineTestTarget(); ok {
t.Fatal("a target after the hub revoked the tunnel")
}
}
+54 -1
View File
@@ -99,6 +99,7 @@ type Manager struct {
resolver Resolver resolver Resolver
staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2) staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2)
rekeyAfter time.Duration // D30: how long a future-stamped handshake stays before the re-key (rekey.go)
mu sync.Mutex mu sync.Mutex
nextRegAt time.Time nextRegAt time.Time
@@ -111,6 +112,10 @@ type Manager struct {
resolveFailLogged bool // DNS-failure log throttle (reset on the next success) resolveFailLogged bool // DNS-failure log throttle (reset on the next success)
staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP) staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP)
// D30 (rekey.go): when the latest handshake was first seen stamped in the future, and the last re-key.
futureSince time.Time
lastRekey time.Time
// conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at // conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at
// the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing // the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing
// `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired; // `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired;
@@ -123,6 +128,7 @@ type Manager struct {
type convSnapshot struct { type convSnapshot struct {
wgBlockDesired bool wgBlockDesired bool
operational bool operational bool
serverIP netip.Addr // the desired block's pbs_tunnel_ip (R-1247 line test target); invalid when none
} }
// TunnelConvergence returns the cached (desired, operational) snapshot without any exec (fast-tick // TunnelConvergence returns the cached (desired, operational) snapshot without any exec (fast-tick
@@ -134,6 +140,17 @@ func (m *Manager) TunnelConvergence() (desired, operational bool) {
return m.conv.wgBlockDesired, m.conv.operational return m.conv.wgBlockDesired, m.conv.operational
} }
// LineTestTarget is the R-1247 line test's server: ep0's tunnel address, ok only while the tunnel is operational
// (registered + unit active, as of the last Apply). Cached — no exec.
func (m *Manager) LineTestTarget() (netip.Addr, bool) {
m.mu.Lock()
defer m.mu.Unlock()
if !m.conv.operational || !m.conv.serverIP.IsValid() {
return netip.Addr{}, false
}
return m.conv.serverIP, true
}
// NewManager builds a Manager. stateDir is the agent state dir (default /var/lib/felhom-agent — // NewManager builds a Manager. stateDir is the agent state dir (default /var/lib/felhom-agent —
// note the FELHOM_WG sudoers install entry hard-codes the staged path under it). // note the FELHOM_WG sudoers install entry hard-codes the staged path under it).
func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, logger *slog.Logger) *Manager { func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, logger *slog.Logger) *Manager {
@@ -156,6 +173,7 @@ func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, log
now: time.Now, now: time.Now,
resolver: netResolver{}, resolver: netResolver{},
staleAfter: 180 * time.Second, staleAfter: 180 * time.Second,
rekeyAfter: rekeyAfterDefault,
} }
} }
@@ -342,7 +360,13 @@ func (m *Manager) Apply(ctx context.Context, fetched bool, block *hub.WireWiregu
operational = m.isActive(ctx) operational = m.isActive(ctx)
} }
} }
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational} var server netip.Addr
if desired {
if a, err := netip.ParseAddr(block.Endpoint.PBSTunnelIP); err == nil && a.Is4() {
server = a
}
}
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational, serverIP: server}
}() }()
keyExists := false keyExists := false
@@ -600,6 +624,10 @@ func (m *Manager) Watchdog(ctx context.Context, fetched bool, block *hub.WireWir
if !fetched || block == nil || !m.lastResolvedIP.IsValid() { if !fetched || block == nil || !m.lastResolvedIP.IsValid() {
return // nothing applied to watch, or no desired block return // nothing applied to watch, or no desired block
} }
// D30: a handshake stamped in the future that stays → a new key on the existing peer path (rekey.go).
if m.clockAheadRekeyLocked(ctx, block) {
return
}
if !m.handshakeStaleLocked(ctx) { if !m.handshakeStaleLocked(ctx) {
return // healthy → no DNS, no action (the load-bearing negative) return // healthy → no DNS, no action (the load-bearing negative)
} }
@@ -661,6 +689,9 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
if age, ok := parseHandshakeAge(string(out), m.now()); ok { if age, ok := parseHandshakeAge(string(out), m.now()); ok {
st.LastHandshakeAgeS = &age st.LastHandshakeAgeS = &age
} }
if fut, ok := handshakeFuture(string(out), m.now()); ok {
st.HandshakeFutureS = &fut
}
} }
} }
if st.Pubkey == "" && !st.Registered && !st.Active { if st.Pubkey == "" && !st.Registered && !st.Active {
@@ -669,6 +700,28 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
return st return st
} }
// handshakeFuture (R-1204) reports how many seconds the latest handshake lies in the future, ok only when it does
// (by more than a minute — clock jitter is not the case). parseHandshakeAge clamps that case to 0 („just now"), which
// is what made a box whose clock had run ahead look healthy while ep0 refused its tunnel. Pinned by
// TestR1204_AFutureHandshakeIsReported.
func handshakeFuture(out string, now time.Time) (int64, bool) {
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
f := strings.Fields(line)
if len(f) != 2 {
continue
}
epoch, err := strconv.ParseInt(f[1], 10, 64)
if err != nil || epoch <= 0 {
continue
}
if fut := epoch - now.Unix(); fut > 60 {
return fut, true
}
return 0, false
}
return 0, false
}
// parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>"). // parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>").
// epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0). // epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0).
func parseHandshakeAge(out string, now time.Time) (int64, bool) { func parseHandshakeAge(out string, now time.Time) (int64, bool) {
+28
View File
@@ -696,3 +696,31 @@ func TestInitialResolveFailureNoTeardown(t *testing.T) {
t.Errorf("did not recover after DNS returned: %v", rr.calls) t.Errorf("did not recover after DNS returned: %v", rr.calls)
} }
} }
// R-1204: a box that ran once with its clock ahead keeps a handshake stamped in the FUTURE — ep0 then refuses every
// newer one. The age stays clamped at 0 (unchanged contract), and the new field says how far ahead it lies; a
// handshake in the past (or within a minute of now) never sets it.
// RED-PROOF: with Status not setting HandshakeFutureS the first assertion fails — seen failing 2026-10-22.
func TestR1204_AFutureHandshakeIsReported(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
ctx := context.Background()
m.Apply(ctx, false, nil)
*active = true
m.now = func() time.Time { return time.Unix(1783107115-2*24*3600, 0) } // the clock came back two days earlier
st := m.Status(ctx)
if st.HandshakeFutureS == nil || *st.HandshakeFutureS != 2*24*3600 {
t.Fatalf("handshake_future_s = %v, want %d", st.HandshakeFutureS, 2*24*3600)
}
if st.LastHandshakeAgeS == nil || *st.LastHandshakeAgeS != 0 {
t.Fatalf("the age stays clamped at 0: %v", st.LastHandshakeAgeS)
}
m.now = func() time.Time { return time.Unix(1783107115+100, 0) }
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("a past handshake must not set handshake_future_s: %v", *st.HandshakeFutureS)
}
m.now = func() time.Time { return time.Unix(1783107115-30, 0) } // 30 s of jitter is not the case
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("clock jitter must not set handshake_future_s")
}
}
+189
View File
@@ -0,0 +1,189 @@
package wgtunnel
import (
"context"
"crypto/rand"
"encoding/base64"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D30 (operator 2026-10-22, R-1204): the tunnel heals itself after the box ran with its clock ahead.
//
// WireGuard refuses a handshake whose timestamp is not newer than the last one it accepted from that key (replay
// protection). A box whose clock ran ahead handshook with a future timestamp; ep0 keeps it, and once the clock is
// right again ep0 refuses every new handshake from that key — until the box's clock passes the stored stamp. The
// sign on the box: its own latest handshake stays stamped in the future (R-1204's handshake_future_s).
//
// The repair: when that sign stays for rekeyAfter, the agent makes a NEW key pair and registers the new public key on
// the existing path (POST /hosts/<id>/wg → the hub re-keys the peer IN PLACE: same /32, new key, generation bump,
// and the wgsync push of the FULL peer list to ep0 drops the old key and adds the new one). A new key is a new peer
// for ep0, with no stored timestamp — the next handshake is accepted. Nothing new runs on ep0.
//
// The previous private key is kept as private.key.prev (0600): the escrow ceremony may hold it (a host-loss DR that
// installs it registers it again, and the hub re-keys in place back to it — the same path).
// Pinned by TestD30_* (rekey_test.go).
const (
// rekeyAfter: how long the latest handshake must stay in the future before the re-key. A working tunnel
// handshakes every two minutes, which ends the sign within minutes of the clock being right.
rekeyAfterDefault = 15 * time.Minute
// rekeyMinGap bounds the re-key to once an hour, whatever the clock does.
rekeyMinGap = time.Hour
prevKeyName = "private.key.prev"
)
// clockAheadRekeyLocked watches for a handshake stamped in the future and re-keys when it stays. It reports whether
// it re-keyed (the caller then skips the rest of its tick). Caller holds m.mu.
func (m *Manager) clockAheadRekeyLocked(ctx context.Context, block *hub.WireWireguard) bool {
if m.keyBroken || !m.isActive(ctx) {
m.futureSince = time.Time{}
return false
}
out, _, err := m.runner.Run(ctx, "wg", "show", Iface, "latest-handshakes")
if err != nil {
return false
}
now := m.now()
fut, ok := handshakeFuture(string(out), now)
if !ok {
if !m.futureSince.IsZero() {
m.logger.Info("wgtunnel: the handshake is no longer stamped in the future — no re-key needed")
}
m.futureSince = time.Time{}
return false
}
if m.futureSince.IsZero() {
m.futureSince = now
m.logger.Warn("wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it stays",
"future_s", fut, "rekey_after", m.rekeyAfter.String())
return false
}
// A clock that jumps backwards must not stretch either wait (commit security review 2026-10-23 night): a start or a
// last re-key stamped AFTER now is from a wrong clock and is dropped — this whole feature exists because clocks jump.
if now.Before(m.futureSince) {
m.futureSince = now
}
if now.Sub(m.futureSince) < m.rekeyAfter {
return false
}
if m.lastRekey.IsZero() {
m.lastRekey = m.loadLastRekey()
}
if now.Before(m.lastRekey) {
m.lastRekey = time.Time{}
}
if !m.lastRekey.IsZero() && now.Sub(m.lastRekey) < rekeyMinGap {
return false
}
pub, _, err := EnsureKey(m.stateDir)
if err != nil || block.Pubkey != pub {
return false // a re-register is already in flight, or the key is not readable — never re-key over that
}
newPub, err := rotateKey(m.stateDir)
if err != nil {
m.logger.Error("wgtunnel: re-key after a future handshake FAILED — the old key stays", "err", err)
m.lastRekey = now // not again before rekeyMinGap
m.saveLastRekey(now)
return false
}
m.lastRekey, m.futureSince = now, time.Time{}
m.saveLastRekey(now)
m.logger.Warn("wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key",
"old_pubkey", pub, "new_pubkey", newPub, "future_s", fut)
m.nextRegAt = time.Time{} // register now, not after an older backoff
m.registerLocked(ctx, newPub)
return true
}
// rotateKey makes a new key pair, never leaving the box without a readable key (review 2026-10-23 night, N6): the new
// key is written and fsynced to a temp file; the current key gets a second name by a HARD LINK (the escrow may hold
// it — `private.key.prev` keeps the first such key, later ones `private.key.prev.<unix>`, the newest three kept, N7);
// then the temp file is renamed OVER the current name (atomic) and the directory fsynced. At every instant
// `private.key` is a complete key. Returns the new PUBLIC key only.
func rotateKey(stateDir string) (string, error) {
dir := filepath.Join(stateDir, "wg")
cur := filepath.Join(dir, keyFileName)
if _, err := os.Stat(cur); err != nil {
return "", fmt.Errorf("no current key: %w", err)
}
priv := make([]byte, 32)
if _, err := rand.Read(priv); err != nil {
return "", fmt.Errorf("entropy: %w", err)
}
clampPrivateKey(priv)
pub, err := derivePublic(priv)
if err != nil {
return "", err
}
tmp := filepath.Join(dir, keyFileName+".new")
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return "", fmt.Errorf("writing the new key: %w", err)
}
_, werr := f.Write([]byte(base64.StdEncoding.EncodeToString(priv) + "\n"))
serr := f.Sync()
cerr := f.Close()
if werr != nil || serr != nil || cerr != nil {
os.Remove(tmp)
return "", fmt.Errorf("writing the new key: %v %v %v", werr, serr, cerr)
}
prev := filepath.Join(dir, prevKeyName)
if _, err := os.Lstat(prev); err == nil {
prev = filepath.Join(dir, fmt.Sprintf("%s.%d", prevKeyName, rekeyNow().Unix()))
}
if err := os.Link(cur, prev); err != nil {
os.Remove(tmp)
return "", fmt.Errorf("keeping the old key: %w", err)
}
if err := os.Rename(tmp, cur); err != nil {
os.Remove(tmp)
return "", fmt.Errorf("installing the new key: %w", err)
}
if d, err := os.Open(dir); err == nil {
d.Sync()
d.Close()
}
prunePrevKeys(dir, 3)
return pub, nil
}
// rekeyNow is the clock for the kept-key names (a seam).
var rekeyNow = time.Now
// prunePrevKeys keeps the newest `keep` of the `private.key.prev.<unix>` files (`private.key.prev` itself always stays).
func prunePrevKeys(dir string, keep int) {
m, _ := filepath.Glob(filepath.Join(dir, prevKeyName+".*"))
if len(m) <= keep {
return
}
sort.Strings(m) // the same digit count until 2286: lexical = time order
for _, p := range m[:len(m)-keep] {
os.Remove(p)
}
}
// lastRekeyFile records the last re-key's time across agent restarts (N7: the once-an-hour bound was in memory only).
const lastRekeyFile = "last-rekey"
func (m *Manager) loadLastRekey() time.Time {
b, err := os.ReadFile(filepath.Join(m.wgDir(), lastRekeyFile))
if err != nil {
return time.Time{}
}
var u int64
if _, err := fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &u); err != nil {
return time.Time{}
}
return time.Unix(u, 0)
}
func (m *Manager) saveLastRekey(t time.Time) {
_ = os.WriteFile(filepath.Join(m.wgDir(), lastRekeyFile), []byte(fmt.Sprintf("%d\n", t.Unix())), 0o600)
}
+227
View File
@@ -0,0 +1,227 @@
package wgtunnel
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// D30 (R-1204): a handshake stamped in the future that STAYS for rekeyAfter → a new key pair, registered on the
// existing peer path (the hub re-keys in place; its full-list push to ep0 swaps the key). The fake `wg` reports a
// handshake at 1783107115; a clock three hours behind that sees it three hours in the future.
// RED-PROOF (2026-10-22 night): with the clockAheadRekeyLocked call removed from Watchdog,
// TestD30_AFutureHandshakeThatStaysReKeys and …AFailedRegistration… fail (no new key, no registration); with the
// rekeyAfter wait removed, …ThatStaysReKeys fails (a re-key at 10 min); with the isActive check removed,
// TestD30_NoReKeyWhileTheTunnelIsDown fails.
const hs = 1783107115
func at(m *Manager, unix int64) { m.now = func() time.Time { return time.Unix(unix, 0) } }
func readKey(t *testing.T, m *Manager, name string) string {
t.Helper()
b, err := os.ReadFile(filepath.Join(m.stateDir, "wg", name))
if err != nil {
t.Fatal(err)
}
return strings.TrimSpace(string(b))
}
func TestD30_AFutureHandshakeThatStaysReKeys(t *testing.T) {
fh := &fakeHub{}
m, _, active, logs := testManager(t, fh)
oldPub := establishWG(t, m, active)
oldPriv := readKey(t, m, keyFileName)
ctx := context.Background()
calls := fh.count()
t0 := int64(hs - 3*3600) // the clock is right again; the last handshake (made while it ran ahead) is 3 h ahead
at(m, t0)
m.Watchdog(ctx, true, testBlock(oldPub))
at(m, t0+10*60)
m.Watchdog(ctx, true, testBlock(oldPub))
if fh.count() != calls || localPub(t, m) != oldPub {
t.Fatal("no re-key before the future stamp has stayed rekeyAfter")
}
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(oldPub))
newPub := localPub(t, m)
if newPub == oldPub {
t.Fatal("a future stamp that stayed 16 min must give a new key")
}
if fh.count() != calls+1 {
t.Fatalf("the new key must be registered on the existing peer path (RegisterWG calls %d → %d)", calls, fh.count())
}
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
t.Fatalf("the registration marker must carry the new key: %+v", mk)
}
if readKey(t, m, prevKeyName) != oldPriv {
t.Fatal("the old private key must be kept as private.key.prev (the escrow may hold it)")
}
if strings.Contains(logs.String(), oldPriv) || strings.Contains(logs.String(), readKey(t, m, keyFileName)) {
t.Fatal("a private key reached the log")
}
// The served block still names the OLD key until the next fetch: no second re-key over that.
at(m, t0+17*60)
m.Watchdog(ctx, true, testBlock(oldPub))
if localPub(t, m) != newPub {
t.Fatal("a second re-key ran before the hub served the new key")
}
// The hub serves the new key → the tunnel is applied with the NEW private key.
m.Apply(ctx, true, testBlock(newPub))
conf, err := os.ReadFile(m.stagedConfPath())
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(conf), readKey(t, m, keyFileName)) || strings.Contains(string(conf), oldPriv) {
t.Fatal("the applied tunnel must use the new private key, not the old one")
}
}
func TestD30_AShortFutureSpellDoesNotReKey(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, hs+100) // a handshake now lies in the past: the sign ended (a working tunnel)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+20*60) // in the future again: a NEW window starts, it does not continue the old one
m.Watchdog(ctx, true, testBlock(pub))
if localPub(t, m) != pub {
t.Fatal("a future stamp that did not STAY for rekeyAfter must not re-key")
}
}
func TestD30_NoReKeyWhileTheTunnelIsDown(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
*active = false
ctx := context.Background()
t0 := int64(hs - 3*3600)
for _, d := range []int64{0, 20 * 60, 40 * 60} {
at(m, t0+d)
m.Watchdog(ctx, true, testBlock(pub))
}
if localPub(t, m) != pub {
t.Fatal("no re-key while the tunnel unit is not active")
}
}
func TestD30_AFailedRegistrationIsRetriedByTheExistingPath(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
oldPub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(oldPub))
fh.mu.Lock()
fh.err = errors.New("hub down")
fh.mu.Unlock()
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(oldPub))
newPub := localPub(t, m)
if newPub == oldPub {
t.Fatal("the key rotates before the registration")
}
if mk := m.loadMarker(); mk == nil || mk.Pubkey != oldPub {
t.Fatal("a failed registration leaves the marker on the old key")
}
fh.mu.Lock()
fh.err = nil
fh.mu.Unlock()
at(m, t0+30*60) // past the backoff: Apply sees the served (old) key ≠ the local key → re-key-in-place registers
m.Apply(ctx, true, testBlock(oldPub))
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
t.Fatalf("the existing re-key-in-place path must register the new key: %+v", mk)
}
}
// Review 2026-10-23 night. N7: a second rotation keeps the FIRST old key as `private.key.prev` (the escrow may hold it)
// and the later one beside it; N6: the current key is replaced by a rename OVER it, so `private.key` is never missing.
// RED-PROOF: with the „prev exists → a dated name" branch removed, TwoRotations fails (the first key overwritten) —
// seen failing 2026-10-23 night.
func TestD30_TwoRotationsKeepTheFirstOldKey(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
establishWG(t, m, active)
first := readKey(t, m, keyFileName)
old := rekeyNow
defer func() { rekeyNow = old }()
rekeyNow = func() time.Time { return time.Unix(hs, 0) }
if _, err := rotateKey(m.stateDir); err != nil {
t.Fatal(err)
}
second := readKey(t, m, keyFileName)
rekeyNow = func() time.Time { return time.Unix(hs+7200, 0) }
if _, err := rotateKey(m.stateDir); err != nil {
t.Fatal(err)
}
if readKey(t, m, prevKeyName) != first {
t.Fatal("private.key.prev must keep the FIRST old key")
}
if readKey(t, m, fmt.Sprintf("%s.%d", prevKeyName, hs+7200)) != second {
t.Fatal("the second old key must be kept beside it")
}
if cur := readKey(t, m, keyFileName); cur == first || cur == second {
t.Fatal("the current key must be new")
}
}
// N7: the once-an-hour bound survives an agent restart (it was in memory only). RED-PROOF: with loadLastRekey removed
// from clockAheadRekeyLocked, this fails — seen failing 2026-10-23 night.
func TestD30_TheHourBoundSurvivesARestart(t *testing.T) {
fh := &fakeHub{}
m, rr, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(pub))
newPub := localPub(t, m)
if newPub == pub {
t.Fatal("fixture: the first re-key must happen")
}
// The agent restarts: a NEW manager on the same state; the served block now names the new key; the sign is back.
m2 := NewManager(rr, fh, m.stateDir, m.logger)
m2.isActive, m2.wgPresent, m2.resolver = m.isActive, m.wgPresent, m.resolver
m2.Apply(ctx, true, testBlock(newPub))
at(m2, t0+20*60)
m2.Watchdog(ctx, true, testBlock(newPub))
at(m2, t0+40*60)
m2.Watchdog(ctx, true, testBlock(newPub))
if localPub(t, m2) != newPub {
t.Fatal("a restart must not reset the once-an-hour bound")
}
_ = active
}
// Commit security review 2026-10-23 night: a last re-key saved while the clock ran ahead (a later date than now) must
// not block the next re-key until that date. RED-PROOF: with the „lastRekey after now → dropped" guard removed, this
// fails — seen failing 2026-10-23 night.
func TestD30_ALastReKeyFromTheFutureDoesNotBlock(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
m.saveLastRekey(time.Unix(hs+2*365*86400, 0)) // written while the clock was two years ahead
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(pub))
if localPub(t, m) == pub {
t.Fatal("a last re-key dated in the future must not block the re-key")
}
}