Compare commits

..
50 Commits
Author SHA1 Message Date
admin 808cdb36ad REPORT + CONTEXT: v0.171.0 released and vouched; R-1344 restart proven live
gates / gates (push) Successful in 33s
2026-10-30 11:09:48 +01:00
admin 84726c9f27 v0.171.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.171.0 (D50)
gates / gates (push) Successful in 33s
2026-10-30 10:49:46 +01:00
admin bd18347c1c R-1344 decided from the night's measurement: a controller that runs but does not answer is restarted at 10 unhealthy sweeps (alarm at 6) — busy file + the brake refuse; stanza carries unresponsive_restart_after_s / unresponsive_restart_at
gates / gates (push) Successful in 33s
2026-10-30 09:20:37 +01:00
admin 98bd1b71e9 REPORT + CONTEXT: v0.170.0 released and vouched; reboot_guest proven on the release builds
gates / gates (push) Successful in 39s
2026-10-29 20:24:29 +01:00
admin 6578a8b91e v0.170.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.170.0 (D50)
gates / gates (push) Successful in 36s
2026-10-29 18:59:05 +01:00
admin ca15a31690 D65 = A: reboot_guest joins the agent's closed operator-action list — guards (busy controller file, heavy gate, jobs, lock, parked, 1/hour, the brake), result with both times, GET /guest/operator-reboot for the household line; one read-only sudo vector
gates / gates (push) Successful in 33s
2026-10-29 16:04:19 +01:00
admin bdea4694a4 REPORT + CONTEXT: v0.169.0 released and vouched; restart_controller proven on the release builds
gates / gates (push) Successful in 35s
2026-10-29 15:11:11 +01:00
admin db25eb0e71 v0.169.0 released and vouched (14:06:09 CET): CHANGELOG; D50 pinned.day0_agent = 0.169.0
gates / gates (push) Successful in 42s
2026-10-29 14:09:04 +01:00
admin 637c92c762 Operator actions for the agent: restart_controller from the hub (closed list, the supervisor's guards, the brake counts operator restarts); R-1344: a running-but-unhealthy controller raises unresponsive — alarm only; tests red first
gates / gates (push) Successful in 36s
2026-10-29 12:30:36 +01:00
admin 63e63ee8bb v0.168.0 released and vouched: CONTEXT, REPORT
gates / gates (push) Successful in 37s
2026-10-29 10:48:37 +01:00
admin 6b0b093199 D50: pinned.day0_agent = 0.168.0 (vouched by the operator 2026-10-29 10:21:56 CET); CHANGELOG notes the vouch
gates / gates (push) Successful in 36s
2026-10-29 10:33:58 +01:00
admin f4e63a55da v0.168.0: CHANGELOG — released (agent 1e70196c…, bundle 3415c034…), committed on demo-felhom, Tester 1, demo-hp; not vouched yet
gates / gates (push) Successful in 37s
2026-10-29 10:15:55 +01:00
admin 91bd96c755 REPORT + CONTEXT: R-528 and R-1254 on main, not released (night 2026-10-28)
gates / gates (push) Successful in 35s
2026-10-29 02:51:00 +01:00
admin 6f933d6ee3 R-528: GET /guest/oom-kills — the kernel's per-container OOM counter of the token's guest, read on the host (no sudo)
gates / gates (push) Successful in 34s
Tests TestR528_* (fake cgroup tree), red-proved by mutation. Commit only; the agent release needs the vouch.
2026-10-28 23:22:40 +01:00
admin 5ddd124cfc R-1254 (ceremony half, HUNT-3 H2): after a failed escrow upload, ask the hub which package it holds before wording the result
gates / gates (push) Successful in 44s
The hub holds the new package -> success; another package -> 502 'previous code still works' (now true);
cannot be asked -> 504, no promise. Tests TestR1254_* red before the fix. Commit only; the agent release needs the vouch.
2026-10-28 22:51:08 +01:00
admin 3055bc54c1 D50 pin: day0_agent 0.167.0 (the operator's vouch 2026-10-27, read from the hub)
gates / gates (push) Successful in 1m4s
2026-10-27 20:45:25 +01:00
admin 007199342f v0.167.0 RELEASED (not vouched yet): CHANGELOG header, REPORT
gates / gates (push) Successful in 1m20s
2026-10-27 15:49:30 +01:00
admin 54d3a1057b CHANGELOG: the unreleased header names R-972 (Go 1.27.3)
gates / gates (push) Successful in 59s
2026-10-26 22:21:47 +01:00
admin fbbb168d1f build(R-972): Go 1.27.3 (GO-2026-6809, crypto/x509) — govulncheck 1 → 0 called
gates / gates (push) Successful in 1m22s
2026-10-26 20:34:25 +01:00
admin 3911cc6f52 fix(R-1232): the tier due-check counts only this box's archives (own key) — a reinstalled box's first whole-guest copy is not a week late
gates / gates (push) Successful in 1m20s
Red-proved (the old box's archive was counted). Commit only.
2026-10-26 20:15:39 +01:00
admin ad80ee749a docs: CONTEXT — R-1285 state and Tuesday's line test
gates / gates (push) Successful in 1m15s
2026-10-26 18:11:24 +01:00
admin 0683cc2dab REPORT: the 2026-10-26 afternoon run. Docs only.
gates / gates (push) Successful in 1m48s
2026-10-26 14:38:15 +01:00
admin b1bc63dd32 R-1285: the new agent runs the bundle policy pass once after its self-update commits — in the normal two-job delivery the old process installs the bundle. Test in the real order (commit, then one bundle-policy plan); red-proved. Commit only, not released.
gates / gates (push) Successful in 2m15s
2026-10-26 12:44:25 +01:00
admin e66a48f2b7 REPORT: the 2026-10-26 attended release run. Docs only.
gates / gates (push) Successful in 1m41s
2026-10-26 10:38:03 +01:00
admin b890b57540 v0.166.0 released and vouched by the operator (10:04:55 CET): CHANGELOG header (R-1280 live, R-1285 back to OPEN — the bundle job runs in the old process); D50 day0_agent pin → 0.166.0. Docs only.
gates / gates (push) Successful in 1m52s
2026-10-26 10:07:15 +01:00
admin aa69114d18 REPORT: the 2026-10-25 night — committed fixes, CI by hash, teardown. Docs only.
gates / gates (push) Successful in 1m43s
2026-10-25 23:54:07 +01:00
admin abcc1e6833 R-1285: after every bundle install the agent asks the wrapper now on disk for the appliance-only policy pass, so a BYO host upgraded from an older wrapper loses our SSH drop-in in one update. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:46:38 +01:00
admin d912307bb7 R-1280: a signed job's validity window is at most 24 h — the verifier, the root wrapper and felhom-opsign. Red-proved. Commit only, not released.
gates / gates (push) Successful in 1m42s
2026-10-25 21:42:18 +01:00
admin feaaa0e8ff D50 = A (R-1290): the published gate asserts the newest 5 released versions plus the pinned MinAgent (0.158.1) and day-0 agent (0.165.0); a pin with no tag fails; decoys red-proved. No binary change.
gates / gates (push) Successful in 1m41s
2026-10-25 20:20:30 +01:00
admin 6b6ba0c17f v0.165.0 vouched by the operator (18:38:49 CET); REPORT: release, delivery to four hosts, the BYO proof (removal one bundle late, felhom.eu R-1285). Docs only.
gates / gates (push) Failing after 1m49s
2026-10-25 19:02:48 +01:00
admin 4697f8ac9e v0.165.0: CHANGELOG names the release (R-1257 D43 = A, R-1250); released from a clean clone, sha256 e9d04a47…, bundle 8dea1f34…; not vouched
gates / gates (push) Successful in 1m38s
2026-10-25 17:00:02 +01:00
admin 4a1a84941c D43 = A run close-out: REPORT and CONTEXT. Docs only.
gates / gates (push) Successful in 1m40s
2026-10-24 21:04:19 +02:00
admin c8d3c17b13 D43 = A (R-1257): the root-key-only sshd drop-in only on an appliance; on BYO nothing is written and our copy is removed; the in-force check reads the LAN addresses (X3); a failing sshd -t undoes only the SSH step (X4). Commit only, not released.
gates / gates (push) Successful in 1m32s
2026-10-24 20:47:17 +02:00
admin f5c3d77a61 Review round 2: .claude/rules/unprompted-work.md aligned with its four siblings (the missing 'A version number is pushed once' fact paragraph); no code change
gates / gates (push) Successful in 1m32s
2026-10-24 15:50:59 +02:00
admin 5e05627c4e REPORT: R-1250 test on request, live proof on demo-hp
gates / gates (push) Successful in 1m41s
2026-10-24 13:48:07 +02:00
admin 1e087bb158 linetest: waiting_for_quiet needs the full watched window (R-1250)
gates / gates (push) Successful in 1m35s
The live proof found a fresh agent answering waiting before it had watched one minute; a quiet
box's torrent app would be held for nothing.
2026-10-24 13:24:41 +02:00
admin 29bcf4403e linetest: a test on request for a box that seeds all day (R-1250)
gates / gates (push) Successful in 1m37s
GET /host/line-test reports waiting_for_quiet (due, in the window, tunnel up, only the quiet
missing); POST /host/line-test/run runs one measurement with a 5 s settle and a 10 s quiet
check instead of 10 quiet minutes, runs 5 s apart (~55 s). Every other weekly rule holds.
Commit only; the release rides the next train.
2026-10-24 13:18:59 +02:00
admin 08ca67861f CHANGELOG: v0.164.0 released (the weekly upload line test, R-1247), delivered to the three test boxes, vouched by the operator; REPORT
gates / gates (push) Successful in 1m32s
2026-10-24 09:46:00 +02:00
admin f7e1dcbb27 R-1247: the weekly upload line test — the agent's own iperf3 client to ep0's tunnel-only server, Tuesday 10:00-15:59 after 10 quiet minutes, under the heavy-op gate, best of 3; line_test stanza + GET /host/line-test; 15 tests red-proved; unreleased (next v0.164.0)
gates / gates (push) Successful in 1m35s
2026-10-24 08:59:42 +02:00
admin 2d4b1930df REPORT: day 2026-10-23 — agent 0.163.0 released, delivered (step bundle), vouched by the operator
gates / gates (push) Successful in 1m48s
2026-10-23 11:08:14 +02:00
admin b4721eee11 CHANGELOG: v0.163.0 released (D30, D4 root SSH by key only, R-1204)
gates / gates (push) Successful in 2m5s
2026-10-23 08:44:19 +02:00
admin 1109092def D4 (R-982): root SSH from the LAN by key only — the bundle's sshd drop-in, try-reload-or-restart, an sshd -T read-back reported when not in force, a reload in undo (the reviewed v2 patch); unreleased
gates / gates (push) Successful in 2m9s
2026-10-23 08:22:17 +02:00
admin b45c47be6c REPORT: night 2026-10-22/23 (D30 live, hardened; D4 patches v2)
gates / gates (push) Successful in 2m1s
2026-10-23 00:06:13 +02:00
admin b9b831fb90 D30: a wait stamp dated after now (a clock that jumped back) is dropped, never a block until that date — the commit's security review; red-proved; unreleased
gates / gates (push) Successful in 2m3s
2026-10-22 23:08:25 +02:00
admin 39fb9cbf6c D30 rotation hardened after review: the key file is replaced by an fsynced rename over it (never missing or empty), the first old key stays private.key.prev, later ones dated (3 kept), the once-an-hour bound survives a restart; red-proved; CONTEXT; unreleased
gates / gates (push) Successful in 2m24s
2026-10-22 23:00:05 +02:00
admin ae69ad4512 D30 (R-1204): a handshake stamped in the future for 15 min gives a new WireGuard key, registered on the existing peer path (the hub re-keys in place; its full-list push swaps the key on ep0 — nothing new on ep0); old key kept as private.key.prev; red-proved; unreleased
gates / gates (push) Successful in 2m0s
2026-10-22 20:52:52 +02:00
admin e803ad2b91 night 2026-10-22: REPORT; CHANGELOG corrected — a push of the installer to main publishes nothing (R-110, tag installer-v1.35.0); D4 stays a patch because both halves need a release
gates / gates (push) Successful in 2m24s
2026-10-22 18:13:35 +02:00
admin 7651461be2 D4 (R-982) moved out of main into a ready patch — its installer half would publish the installer (forbidden tonight); R-1204 stays; unreleased
gates / gates (push) Successful in 2m45s
2026-10-22 17:10:46 +02:00
admin a0c88e5183 R-982 (D4): root SSH from the LAN by key only — a bundle drop-in, checked and reloaded by felhom-os-apply; R-1204: the heartbeat reports a future-stamped tunnel handshake; unreleased
gates / gates (push) Successful in 2m23s
2026-10-22 16:58:45 +02:00
admin 2d5840ba11 v0.162.0: released (operator yes 2026-10-22), delivered to the 3 test boxes; CHANGELOG + REPORT
gates / gates (push) Successful in 2m10s
2026-10-22 10:39:06 +02:00
57 changed files with 5559 additions and 82 deletions

No files matched your search

+6
View File
@@ -50,6 +50,12 @@ it **first** in the morning note. A decision you cannot write in that shape is o
on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green on Tester 1, demo-hp and demo-felhom through the normal update path and the box walk's checks are green
on all three — not at night unless the night brief says so. **Agent releases still stop for the on all three — not at night unless the night brief says so. **Agent releases still stop for the
operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.** operator's vouch.** The golden is baked weekly and vouched by the operator; CC never vouches. **No `--no-verify`.**
**A version number is pushed once** — `felhom-controller/controller/build.sh` now refuses a second
push of a tag that exists, and a registry it cannot reach is a refusal too. This is a FACT about
what the build does, not a new rule: a fix for something already pushed is the next patch version,
and a box on a bad image is moved by the normal update path, never by re-tagging. Earned on
2026-10-15, when 0.315.0 was pushed three times from three different commits and a cache kept
serving the first one (`09-update-architecture.md` §0).
7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule). 7. **An enumerated gap becomes a row in the same session — or, if it is small, is fixed in it** (the size rule).
Prose is not a record. Prose is not a record.
8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control. 8. **Hungarian text is searched with ASCII fragments**, with a positive and a negative control.
+333 -1
View File
@@ -1,4 +1,336 @@
## Unreleased (2026-10-21) — every guest may swap as much as its RAM (R-1192, the 8 GB freeze) — COMMITTED, NOT RELEASED (the release is the operator's word) ## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
chain (demo-hp's PBS vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test, two
controller updates). No failed probe, no 5-s timeout; slowest 303 ms (demo-hp, during the PBS vzdump). The only
„unhealthy" was the controller stopped by hand for a proof. A busy night never came near ONE failed probe.
- **So the alarm becomes a restart.** At `controllerUnresponsiveConfirm` = 6 unhealthy sweeps the alarm is raised, as
before. At `controllerUnresponsiveRestart` = 10 (2 minutes later; ~6.5 minutes after the first failed probe) the agent
restarts the controller's bootstrap unit — unless the controller's busy file is fresh (a backup, a restore, an
update, an export) or cannot be read, or the brake refuses (3 restarts in 15 minutes → the crash-loop pause, which
the hub mails). A swap, a parked guest, a locked guest and a vzdump keep resetting the count, as before. A refusal
keeps the alarm and is logged once per reason. After a restart the count starts again from zero.
- **The stanza** carries `unresponsive_restart_after_s` (120: the hub's alarm text says when the restart comes) and
`unresponsive_restart_at` (equals `last_restart_at` for that act; the hub mails `controller_unresponsive_restart`).
- **Refactor, no behaviour change:** the not-running restart's brake and its record are now `supervisorBrakeAllows`
and `recordSupervisorRestart`, shared by both restarts. The busy-file stale log line no longer says „operator reboot".
- Tests `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`, `…Guards` (now also: never restarted),
`…RestartRefusals` (busy fresh, busy unreadable, the brake, a crash-loop pause), `…StaleBusyFileDoesNotShield` —
red first (build failure, then 3 behaviour failures); 7 mutants, each killed (one survivor — the count reset after a
restart — found and pinned).
## v0.170.0 (2026-10-29) — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list — RELEASED (release-agent.sh from a clean clone of `ca15a31`; agent sha256 da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404, config bundle 03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543; signed agent_config_update + agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-29 18:57:48 CET; `pinned.day0_agent` = 0.170.0)
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
a racing start would fail the reboot's own start half.
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
restarted your box at <time>".
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
answers on the next report, `operator_action_results: [{id, outcome, message}]` (`hub.HostReport`), which it requests
at once through the cap-1 report trigger. CLOSED list: `restart_controller <vmid>` only — anything else refused, nothing
called. The act is the supervisor's (`systemctl restart felhom-controller-bootstrap.service` in the guest, existing
grants — no sudoers change, no config bundle) with its guards: provisioned guest, guest running, the guest list and the
vzdump state readable, not parked, no swap, not locked, no vzdump. The thrash brake counts operator restarts with the
supervisor's: 3 in 15 minutes → the press is refused, and presses stay refused for 30 minutes. The result says how long
the controller took to answer (Docker's health check: healthy), or `failed` when it did not within 3 minutes. One
`audit: operator action` line per decision. Works with the controller stopped. Tests `TestOpActions_*` (closed list,
unknown/bad arg refused, restart + answer time + report trigger + once per id + re-send until dropped, stopped
controller, every guard, the brake across supervisor and operator restarts, no healthy answer = failed) and
`TestMainWiresTheAgentOperatorActions` (AST) — each seen failing by a mutation.
- **R-1344 — a controller that RUNS but does not answer: ALARM ONLY.** Each sweep also reads Docker's health status for a
running controller (the swap's `docker inspect -f` grant); 6 consecutive „unhealthy" sweeps (3 minutes, on top of
Docker's own ~90 s) start an episode: one ERROR line, the stanza's `unresponsive` / `unresponsive_since` (the hub raises
`controller_unresponsive`, operator-only). NOT restarted for it: the brief allows the restart only after a measurement
shows a busy night never reads like this, and that measurement runs through tonight (felhom.eu R-1344). Not counted
during a swap, a parked or locked guest, or a vzdump; healthy / starting / no health check / an unreadable status reset
the count. Tests `TestSupervisor_Unresponsive*` (red by mutation: the vzdump guard, the threshold).
## v0.168.0 (2026-10-29) — R-528: the host's per-container OOM counter (`GET /guest/oom-kills`); R-1254: a lost or late escrow answer is asked again before it is worded — RELEASED (release-agent.sh from a clean clone of `91bd96c`; agent sha256 1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca, bundle 3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8; no configs/ change since v0.167.0 → binary-only signed updates to demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 10:21:56 CET**; Day-0 pin set (D50)
### R-528
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
### R-1254 (the ceremony half, review round 2 HUNT-3 H2)
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
hub's current package opens only with the NEW code.
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
newest. Full `go build/vet/test` green. The controller and hub move the same night.
- **R-1232 (P2):** `backup.BackupRunner.NewestArchiveTime` — the tier due-check — counts only archives written with THIS
storage's key on an encrypted storage (the R-727 discriminator; the archive carries no host id). Before: a reinstalled
box shares the customer's PBS namespace and guest number with the old box, the old box's archive read as „the last
backup", and the whole-guest tier waited up to its 168-h cadence (the 2026-10-23 test night). A key lookup failure is
an error (the due-check reads UNKNOWN), never a foreign age. Tests `TestR1232_*` (foreign only → none; own → its time;
an unencrypted dir → unfiltered), red-proved (case 1 returned the old box's time).
- **The cause (measured 2026-10-26 on demo-hp, felhom.eu R-1285):** a box gets `agent_update` and `agent_config_update` in
one pass; the bundle job runs in the OLD process (no policy pass in it) and the restart into the new agent comes a
second later — so 0.166.0's pass never ran on the bundle that delivered it.
- **The fix:** `selfupdate.Manager.SetOnCommitted` — a hook run once right after „update committed"; `main.go`
`wireBundlePolicyAfterCommit` hooks `osupdate.ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's
`bundle-policy` mode, host layer, no envelope) onto it. By the commit the wrapper on disk is the bundle's own, so a BYO
host loses our drop-in in the same update. Idempotent; an appliance removes nothing. The smaller of the two fixes:
agent-only — holding the bundle job until the commit would need new hub queue logic.
- **Test in the real order:** `cmd/felhom-agent/r1285_policy_after_commit_test.go` — a pending marker for the running
version → commit, THEN one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
removed → commit only, no pass — the demo-hp symptom).
- **Which box can show it live:** only one coming FROM 0.164.0 — 0.165.0's wrapper already applies D43 at plan time, so a
0.165.0 box passes even without this fix.
## v0.166.0 (2026-10-26) — a signed job's window is at most 24 h (R-1280); the policy pass after each bundle (R-1285); the published gate follows D50 (R-1290) — released in the attended release run; **VOUCHED by the operator 2026-10-26 10:04:55 CET** (binary `3f22c65b…`, bundle `093772c1…`)
Released from a clean clone at `aa69114` (tag `v0.166.0`; binary sha256 `3f22c65b78914632fa476292f6b58e5af3ed9e9900f3a5a7a347a9c0bfea9c5c`,
bundle `093772c1bda8b974e86ed45472f8fadc8992f247563f5739876102a5156003a3`); delivered to Tester 1, demo-hp and demo-felhom
by signed `agent_update` + `agent_config_update` (each: COMPLETED, bundle INSTALLED, self-check 76/76, update committed).
Live: a 25 h signed job REJECTED on demo-hp („op validity window too long"), a 30 min one ran; `felhom-opsign -ttl 25h`
refuses. D50 pin `day0_agent` → 0.166.0 (`scripts/retention-policy.json`). Evidence: felhom.eu
`documentation/audits/release-monday-2026-10-26/agent-0166/`.
**Found at the release (felhom.eu R-1285, back to OPEN):** in the normal two-job delivery the bundle job runs in the OLD
process (0.165.0) before the restart into 0.166.0, so the policy pass does not run on that bundle — a BYO host coming
from 0.164.0 still loses our drop-in one release late. The pass itself is wired: under 0.166.0 a later bundle logged
„BUNDLE POLICY DONE install_mode=appliance removed=0".
### 2026-10-25 (night, unattended) — a BYO host loses our SSH drop-in in ONE update (R-1285) — committed, NOT released
- **The cause, measured 2026-10-25 on a BYO test host:** the bundle that brings a new `felhom-os-apply` is planned by
the OLD one on disk, which did not know D43's appliance-only policy — so a BYO host upgraded from 0.164.0 kept our
drop-in until its next bundle. A change inside the wrapper alone cannot fix the run that delivers it.
- **The fix — the new AGENT asks the new wrapper:** `internal/osupdate/bundle.go` runs a policy pass right after every
bundle install (`mode: "bundle-policy"`, host layer, no envelope); the wrapper on disk is the bundle's own by then.
`configs/felhom-os-apply` `Bundle.policy_pass`: on a root-owned BYO record it removes the appliance-only files WE
wrote (our header), keeps the previous copy beside the bundle's, `sshd -t` then reloads ssh — a failing `sshd -t` puts
the file back. It writes nothing, so it needs no signature; an appliance and the owner's own file are left alone. Never
fatal for the agent: an older wrapper refuses the unknown mode and the next bundle does it. Report key `bundle_policy`.
- Tests: `TestR1285_AnInstalledBundleIsFollowedByThePolicyPass` (a refused bundle runs no pass);
`ByoKeepsItsSSH.test_the_policy_pass_*` (three). Red-proofs: felhom.eu
`documentation/audits/night-2026-10-25/redproof-r1285.txt`.
### 2026-10-25 (night, unattended) — a signed job's window is at most 24 h (R-1280, D44 #22) — committed, NOT released (rides the next agent release; the vouch is the operator's)
- `internal/authz/verifier.go`: `MaxWindow = 24h`; a blob whose `expires_at - issued_at` is longer is refused
(`ErrWindowTooLong`) before the expiry and nonce checks — a job signed with a long window by mistake no longer stays
usable that long, and the nonce store no longer keeps its record that long.
- `configs/felhom-os-apply`: the root wrapper holds the same 24 h (`MAX_WINDOW_S`, refusal R3 „the signed job's window
is longer than 24 h").
- `cmd/felhom-opsign`: `-ttl` above 24 h (or ≤ 0) is refused before anything is signed.
- Tests: `TestVerify_RejectsPerStage/window_longer_than_24_h` and `…/window_of_exactly_24_h_is_accepted`;
`test_felhom_os_apply.DockerLane.test_a_window_longer_than_24h_is_refused` / `…_of_24h_is_accepted`. Both refusals
seen failing with their check switched off: felhom.eu `documentation/audits/night-2026-10-25/redproof-r1280.txt`.
### 2026-10-25 (evening, attended) — the `published` gate follows D50 = A: the newest 5 plus the pinned MinAgent and day-0 agent (R-1290; no binary change)
- **D50 = A** (operator, 2026-10-25): the registry keeps the newest 5 released agent versions, the hub's MinAgent floor and
the day-0 manifest's agent. `scripts/retention-policy.json`: `generic_versions_kept` 10 → 5, new `pinned`
(`hub_min_agent` 0.158.1, `day0_agent` 0.165.0 — copies of hub state, because CI holds no hub secret; the session that
records a vouch or a new MinAgent updates them in the same commit). `check-published-versions.py` asserts the window
plus every pin; a pin with no tag FAILS. CI was red since the operator's hand prune of 0.159.0–0.160.0 (job 2334).
- **Decoys** (`test_gate_decoys.py`, via `FELHOM_RETENTION_POLICY`, test only): a pinned version older than the window is
asserted; its 404 convicts; a pin with no tag convicts. Seen failing on the old gate (two LIVE HOLES): felhom.eu
`documentation/audits/night-2026-10-25/redproof-d50.txt`.
## v0.165.0 (2026-10-25) — D43 = A: BYO hosts keep their own SSH settings (R-1257); the line test for a box that seeds all day (R-1250) — released in the attended release run; **VOUCHED by the operator 2026-10-25 18:38:49 CET** (binary `e9d04a47…`, bundle `8dea1f34…`)
**Found at the release (felhom.eu R-1285, P4):** on a BYO host upgraded from 0.164.0 the drop-in is removed only on the bundle run AFTER the one that installs this wrapper — the first run is planned by the old wrapper. Proven on a BYO test host (VM 9371 on demo-hp): the second `agent_config_update` logged `BUNDLE REMOVED … (ours, on a BYO host — D43)`.
### BYO hosts keep their own SSH settings (R-1257, D43 = A) — 2026-10-24 night
- **The root-key-only sshd drop-in (`/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`, R-982 / D4) goes only to an
appliance** — operator ruling D43 = A (2026-10-24): a BYO host belongs to its owner (`11` §1). The table entry's policy
is now `appliance`: written only when the ROOT-OWNED install record (`/var/lib/felhom-install/state.json`) says
`appliance`. On a `byo` record nothing is written, and a copy WE wrote (this exact path, beginning with our header
line) is REMOVED, its previous copy kept for the undo, then `sshd -t` and a reload. A box whose record is absent or
agent-writable proves nothing and is left exactly as it is. The bundle report lists `removed`; the record says
`removed` / `skipped`. The installer records the mode before it installs the bundle, so a new appliance still gets it.
- **X3: the „in force" check asks sshd about each of the host's own LAN addresses** (`ip -4 -o addr show scope
global`), not only 192.0.2.1 — a `Match Address <LAN>` block that re-allows root passwords no longer reads as in force.
- **X4: a failing `sshd -t` puts back ONLY the SSH step** (`undo_sshd_step`: the previous copy, or none; sshd is not
reloaded, so it never ran the refused state) and the rest of the bundle stays; it used to refuse and undo the whole
bundle. The report carries `sshd_step_undone`; the record names what is on disk.
- Tests (`configs/test_felhom_config_bundle.py`, class `ByoKeepsItsSSH`, 7 tests; each rule seen failing with it taken
out): a BYO host gets no drop-in; our copy comes off a BYO host; the owner's own file stays; no record → left as it
is; a LAN `Match` block is not in force; every address key-only passes; a failing `sshd -t` undoes only the SSH step.
The fake box now carries an appliance record by default (every earlier test unchanged).
### The line test for a box that seeds all day (R-1250) — 2026-10-24
Attended brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). The weekly test (v0.164.0)
starts only after 10 quiet minutes; a torrent app that seeds all day never gives them, so that box never got a tested
line and qBittorrent never got its 50 % limit. Now:
- **`linetest.Tester.WaitingForQuiet`** — due, inside the window, tries left, tunnel up, and only the quiet missing.
`GET /host/line-test` carries it as `waiting_for_quiet`; a controller with a seeding torrent app acts only on it.
It needs the full watched window (11 minute samples): a freshly started agent is not known to be busy, so it never
invites a hold (found in the live proof, 2026-10-24: the first build said `true` before one minute was watched;
`TestFreshTesterIsNotWaiting`).
- **`linetest.Tester.RunNow`** and **`POST /host/line-test/run`** (synchronous, about 55 s) — a test on request while the
controller holds its app's upload low: every weekly rule holds (due, window, `MaxTriesPerDay`, tunnel, the one heavy-op
gate) except the 10 quiet minutes, which become a 5 s settle and a 10 s quiet check (below 2 Mbit/s, so a request
never measures the family's own upload); the three runs are 5 s apart instead of 30. A refusal answers its code
(`not-due`, `outside-window`, `tries-used`, `no-tunnel`, `heavy-op`, `not-quiet`, `server-busy`); a failed request
spends one of the day's tries. `Tick` and `RunNow` now share one `ready()` check.
- Tests (each seen failing): `TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt` (the sleeps, the 60 s bound),
`TestRunNowRefusesAnUplinkThatIsStillBusy`, `TestRunNowKeepsTheWeeklyRules`, `TestRunNowFailuresSpendTheDaysTries`,
`TestQuietBoxIsNotWaiting`, `TestLineTestRunRoute`; a compile-time assertion that the Tester serves the request half.
**No sudoers or config-bundle change. MinController: none** — an older controller never calls the route.
## 2026-10-24 (review round 2, unattended) — `.claude/rules/unprompted-work.md` aligned with its four siblings
- The copy lacked item 6's paragraph „A version number is pushed once …" that the felhom.eu and felhom-controller copies
have carried since 2026-10-15; the file's header says „change all five or none". The paragraph states a fact
(`felhom-controller/controller/build.sh` refuses an existing tag and an unreachable registry). Now byte-identical to
the other four (md5 b1891bd4). No code change. Record: felhom.eu `documentation/audits/review-round-2-2026-10-24/EDITS.md`.
## v0.164.0 — the weekly upload line test (R-1247) (2026-10-24) — RELEASED on the operator's ruling (attended release run); vouched by the operator 09:44:58 CEST
Binary sha256 `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`; config bundle sha256
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341` (configs unchanged — no bundle job sent). Built from
`f7e1dcb` (CI 2277 green) in a fresh clone by `scripts/release-agent.sh`. Delivered by one signed `agent_update` per box
(07:25Z): Tester 1, demo-hp and demo-felhom — COMPLETED, self-check 76/76, committed after the 1-minute dwell, the line
test started (felhom.eu `audits/line-test-2026-10-24/agent-0164-boxes.txt`).
Operator ruling 2026-10-24 07:26: once a week the box's HOST measures its upload line with a 10-second iperf3 upload to an
iperf3 server on ep0 that answers **only inside the WireGuard tunnel** (felhom.eu `scripts/ep0-iperf3/`; installed the same
day). The agent speaks the iperf3 protocol itself (operator's pick, 2026-10-24: nothing is installed on any host).
**No sudoers or config-bundle change.** **MinController: none** — the controller (≥ 0.342.0) reads the new route and
treats a 404 as „not measured".
- **`internal/linetest`** — `Client` (the iperf3 3.x client protocol: cookie, params, N data streams, TEST_END, results
exchange; the value is the SERVER's received bytes) and `Tester` (the schedule): due weekly from Tuesday 10:00 local,
starts 10:00–15:59 only (never in the night chain), from a per-host slot spread over 10:00–13:59; only after 10 quiet
minutes (every one-minute transmit rate of the host's physical uplink below 2 Mbit/s); under the one-heavy-op gate
(`line-test`); 3 runs of 10 s with 4 streams, 30 s apart, the highest counts; each run bounded by 25 s; „server busy" →
tried again at the next tick the same day, at most 6 tries a day; no operational tunnel → no test, no value. The last
result is persisted (`<oob state dir>/line-test.json`).
- **Report:** a `line_test` stanza (`schedule`, `last_ok_at`, `mbps`, `runs_mbps`, `server`, `last_attempt_at`,
`last_error`). **Local API:** `GET /host/line-test` → `{present, mbps, measured_at, method:"iperf3-tunnel"}`.
- `wgtunnel.Manager.LineTestTarget()` — the desired block's `pbs_tunnel_ip`, only while the tunnel is operational
(cached, no exec). Config `line_test.disable` (default on).
- **Proven live** (felhom.eu `audits/line-test-2026-10-24/`): a probe built from this package on demo-hp's host read
**90.7 Mbit/s** in 10.4 s (the push floor said „at least 10"); a second test 2 s later → „server busy" in 0.1 s while the
first went on undisturbed (90.6).
- Tests, each red-proved by a mutation (15): `TestClientSpeaksTheProtocolAndCountsTheServersBytes` (against a fake
iperf3 server), `TestClientBusyServerIsErrBusy`, `TestClientHonoursTheDeadlineAgainstAStalledServer`,
`TestEligibleWindowNeverInTheNightChain`, `TestNeverTestsInTheNightOrBeforeTuesday`, `TestBusyServerIsRetriedLaterTheSameDay`,
`TestTriesPerDayAreBounded`, `TestNoTunnelNoTest`, `TestBusyUplinkOrHeavyOpWaits`, `TestTickMeasuresTheHighestOfThreeRuns`,
`TestStateSurvivesARestartAndReportsIt`, `TestUplinkCountsTheBridgesPhysicalPortsOnly`, `TestLineTestRoute`,
`TestLineTestTargetOnlyWhileOperational`, `TestMainWiresTheLineTest`.
## v0.163.0 — the tunnel heals itself after a clock that ran ahead (D30, R-1204); root SSH from the LAN by key only (D4, R-982) (2026-10-23) — RELEASED on the operator's yes (day brief); the vouch is the operator's
Binary sha256 `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c`; config bundle sha256 `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8`. Built from `1109092` (CI 2246 green). Both sections below are in it.
### The tunnel re-key (D30) and the D4 bundle drop-in
- **D30 (operator 2026-10-22: the agent makes a new key and registers it on the existing peer path).** When the latest
handshake stays stamped in the future for 15 minutes (the R-1204 sign: ep0 keeps the future timestamp and refuses
every newer handshake from that key), the agent makes a new WireGuard key pair, keeps the old private key as
`wg/private.key.prev` (0600 — the escrow may hold it), and registers the new public key through `POST /hosts/<id>/wg`.
The hub re-keys the peer IN PLACE (same /32, generation bump) and its full-list push to ep0 drops the old key and adds
the new one — nothing new runs on ep0. A new key is a new peer for ep0, with no stored timestamp, so the next
handshake is accepted. At most once an hour; not while the tunnel unit is down; never while a re-register is already
in flight. A failed registration is retried by the existing re-key-in-place path. Logs carry public keys only.
Tests `TestD30_*` (4), red-proved. A host-loss DR that installs the escrowed (older) key registers it again, and the
hub re-keys in place back to it — the same path.
- **Review of the night's own code (felhom.eu `REPORT-night-2026-10-23.md`, N6, N7).** The rotation never leaves the box
without a readable key: the new key is written and fsynced to a temp file, the current key gets its kept name by a
hard link, then the temp file is renamed OVER `private.key` and the directory fsynced (a power cut or a kill between
two renames used to leave no key, or an empty one). `private.key.prev` keeps the FIRST old key (the escrow may hold it);
later ones are `private.key.prev.<unix>`, the newest three kept. The once-an-hour bound is saved in `wg/last-rekey`, so
an agent restart does not reset it; a start or a last re-key dated AFTER now (a clock that jumped back) is dropped,
never a block until that date (the commit's own security review). Tests `TestD30_TwoRotationsKeepTheFirstOldKey`,
`TestD30_ALastReKeyFromTheFutureDoesNotBlock`,
`TestD30_TheHourBoundSurvivesARestart`, red-proved. **Live (felhom.eu `audits/night-2026-10-23/P2-D30/`):** the D30
re-key ran on a test box (before these two changes): clock corrected → 15 min → new key registered → the box switched
at its next desired-state fetch → a handshake with the new key.
- **D4 (operator ruling 2026-10-22, R-982) — root SSH from the LAN by key only, on `main` now (2026-10-23 day brief).** The
config bundle carries `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`;
content check: exactly that line, `sshd -t -f`). After an install that wrote it: `sshd -t` (the whole config), then
`systemctl try-reload-or-restart ssh` (a stopped or socket-activated sshd no longer refuses the bundle), then `sshd -T`
is read back and a drop-in that is NOT in force (no Include, an earlier drop-in, a Match block) is said loudly and
reported as `sshd_dropin_not_in_force` — never a refusal; an undone bundle reloads sshd again. The console and the
web page keep the root password; felhom-sshd (8822) is not touched. Tests `RootKeyOnly` (6), red-proved; the
installer's uninstall removes the file (felhom.eu installer 1.36.0). Reviewed 2026-10-23 night (A1–A7).
- **Known gap, unchanged (filed):** after a reboot the kernel forgets the future stamp (latest handshake 0), so neither
this re-key nor the hub's alarm sees a box whose clock was corrected and which then restarted (R-1223).
### The future-stamped handshake is reported (R-1204) — first written on the 2026-10-22 night
- **R-982 / D4 — a ready patch on that night; on `main` since `1109092` and released in v0.163.0 (see above).** Built and proven the same night (felhom.eu
`audits/night-2026-10-22/P2-D4/`): a bundle drop-in `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`
(`PermitRootLogin prohibit-password`), a `sshd` content check (exactly `prohibit-password`; `sshd -t -f`), and after
an install that wrote it `sshd -t` + `systemctl reload ssh`; tests `RootKeyOnly` (3), red-proved. Live on a fresh
nested box from ISO 1.30.0: root password over SSH refused, root key works, the console and the web page still take
the password. It was committed (`a0c88e5`) and moved back out (`7651461`): the installer's uninstall must remove every bundle file (felhom.eu `scripts/test_hostinstall.py`), the bundle half reaches a box only with an agent release, and the installer half only with a new `installer-v…` tag (R-110 — a push to `main` alone publishes nothing) — both forbidden tonight. *(Corrected the same night: the first reason written here, „a push of `felhom-host-install.sh` publishes the installer", was wrong — the website serves `/scripts/` from the tag `installer-v1.35.0`, read live in `manifests/webpage.yaml` and the running Deployment.)* Both halves are patches: `felhom-agent-bundle-D4.patch` and `felhom.eu-installer-D4.patch` in that folder; apply them together with the agent release and the next installer tag.
- **R-1204 — `wireguard.handshake_future_s`.** A box that ran once with its clock ahead keeps a handshake stamped in the
future; ep0 refuses every newer one, and `last_handshake_age_s` (clamped at 0) read „just now" for ever. The heartbeat
now carries how many seconds the handshake lies ahead (> 60 s only); the age keeps its clamp. The hub alarms on it
(felhom.eu hub `wg_tunnel_stale`). Test `TestR1204_AFutureHandshakeIsReported`, red-proved.
## v0.162.0 — every guest may swap as much as its RAM (R-1192, the 8 GB freeze); the host's swap is reported and checked; no false „operator access degraded" on a fresh box (R-1194) (2026-10-22) — RELEASED on the operator's yes; the vouch is the operator's
Released 2026-10-22 07:40 UTC (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at
`b310773`: binary sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (the same bytes as the
night's candidate), config bundle sha256 `ac772967fd4669045e837a70bf5acca239c58d55f7f00212e472a8facd60b5a5`, tag
`v0.162.0`; no `configs/` change since v0.161.0. Delivered by one signed `agent_update` job per box to demo-hp, Tester 1
and demo-felhom (07:49–07:58 UTC): each `signed op COMPLETED`, `capabilities self-check ok=76 total=76`, `update
committed`, then `guest-swap: raised` for its 9201 (512 MB → its RAM). Evidence felhom.eu
`documentation/audits/morning-2026-10-22/release/04-agent-delivery.txt`. **Found at once:** demo-hp (host swap 8191 MB,
guest swap 25898 MB) and demo-felhom (8191 MB vs 12288 MB) logged the host-swap WARN and the hub mailed
`host_swap_short` — the check compares the host's swap with the guests' whole swap allowance, which a guest larger
than the installer's 8 GB host swap always exceeds (felhom.eu register R-1216). Soak (night of 2026-10-21, 8 GB box):
11 h, no freeze, the host's swap peaked at 2.7 GB used.
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent `audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
+61
View File
@@ -1,5 +1,66 @@
# CONTEXT — felhom-agent working state # CONTEXT — felhom-agent working state
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
> on demo-hp (felhom.eu R-1250, read, not forced).
> **2026-10-24 night (unattended) — D43 = A on `main` (`c8d3c17`), NOT released (next v0.165.0, with R-1250).** The sshd
> drop-in's bundle policy is `appliance` (install record `/var/lib/felhom-install/state.json` mode); BYO: nothing written,
> our copy removed; no proof: untouched. X3: `lan_addresses()` for `sshd -T`. X4: `undo_sshd_step` — only the SSH file is
> put back. Release + the operator's vouch: Monday's list (felhom.eu STATUS).
> **2026-10-24 (attended release run) — v0.164.0: the weekly upload line test (R-1247; operator ruling 07:26).**
> `internal/linetest`: the agent's own iperf3 client (operator's pick — nothing installed on hosts) to ep0's tunnel-only
> server `10.77.0.1:5201`, Tuesday 10:00-15:59, after 10 quiet minutes, under the heavy-op gate, best of 3; `line_test`
> stanza + `GET /host/line-test` (the controller's qBittorrent limit, D41 = 50 %). demo-hp measured 90.7 Mbit/s. Release
> needs the operator's vouch.
> **2026-10-22/23 night — D30 on `main` (`ae69ad4`), unreleased; the D4 patches v2 ready.** Operator ruling D30
> (2026-10-22): a handshake stamped in the future for 15 min → a new WG key pair (`wgtunnel/rekey.go`; old key kept as
> `wg/private.key.prev`), registered on `POST /hosts/<id>/wg` (the hub re-keys in place; its full-list push swaps the
> key on ep0). Live on a test box (felhom.eu `audits/night-2026-10-23/P2-D30/`): with ep0 free to start a handshake the
> tunnel heals by itself (no re-key, correct); with ep0's starts dropped it stays dead and the agent re-keyed at 15 min.
> The interface switches at the next desired-state fetch. Gap: a reboot forgets the future stamp (R-1223). The D4 bundle
> patch v2 (`P3-review/felhom-agent-bundle-D4-v2.patch`): `try-reload-or-restart`, an `sshd -T` read-back reported as
> `sshd_dropin_not_in_force`, a reload in undo. Next agent release = D30 + D4 v2 + R-1204's field, the operator vouches.
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all > **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's > committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256 > real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
+26 -16
View File
@@ -1,18 +1,28 @@
# REPORT — agent, the night of 2026-10-21: the host's swap is reported; no false emergency-door alarm on a new box — COMMITTED, NOT RELEASED # REPORT — v0.171.0 released and vouched: a controller that runs but does not answer is restarted (R-1344, 2026-10-30, attended)
**Brief:** the unattended night brief of 2026-10-21 (Parts 1, 2, 5). Architecture: felhom.eu `03-host-agent.md` §9 (the **Plain words first.** The night's health log showed that a busy night (backups, dumps, updates) never makes the
new swap paragraph). Both join the swap fix (`286a7e3`) in the coming 0.162.0 — the operator's release and vouch (D26 = A). controller miss even one health check — the slowest answer was 0.3 s against a 5 s limit. So the agent no longer only
alarms about a controller that runs but does not answer: 2 minutes after the alarm it restarts it, unless the
controller is busy with a backup, a restore or an update, or the restart brake holds.
- **Host swap (R-1192 follow-up), `0da792e`:** the host report carries `host.host_swap_mb` (Proxmox node status = the ## Commits
kernel's SwapTotal) and `host.guest_swap_mb` (the guests' `swap` sum; each guest spec `swap_bytes`); one WARN when the
first is below the second, one INFO when covered again; `GET /guest/memory` carries both for the controller's install | Commit | CI (job id) | What |
page. The agent never creates host swap (register R-1207). Proven live on the soak box: the hub stored |---|---|---|
`host_swap_mb 7935, guest_swap_mb 4096`. | `bd18347` | 2532 green | `controllerUnresponsiveRestart` = 10 sweeps; `restartUnresponsive` (busy file, then the brake); the brake and the restart record shared with the not-running restart (`supervisorBrakeAllows`, `recordSupervisorRestart`); stanza `unresponsive_restart_after_s`, `unresponsive_restart_at`. Tests red first (build failure, then 3 behaviour failures); 7 mutants, one survivor (the count reset after a restart) pinned by a new assertion. |
- **R-1194, `0da792e`:** no OOB stanza until the felhom-sshd loop's first pass has run — the first report on a fresh box | `84726c9` | (read at the close-out) | v0.171.0 heading; `pinned.day0_agent` = 0.171.0. |
read a config not yet rendered and the hub mailed `oob_degraded` (reproduced 2026-10-21 20:34 CEST on a fresh install).
- **The swap fix itself** ran as a hand-placed fixture on the 8 GB soak box: guest swap 512 → 4096 MB one minute after ## Release
start; the 2-hour household load answered every probe (felhom.eu `audits/night-2026-10-21/P1-soak/`).
- Release candidate built from a clean clone at `0da792e` with the release flags, version `0.162.0`: - `release-agent.sh 0.171.0` from a clean clone of `bd18347`: sha256 `c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5`,
sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (not published). bundle `95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b`, tag `v0.171.0`, verified by download.
- Tests (red-proved): `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`, - configs/ unchanged → one signed `agent_update` per box (Tester 1, demo-hp, demo-felhom): all COMPLETED and
`TestR1194_NoOOBStanzaBeforeTheFirstPass`. CI job 2176 green. „update committed" (10:12–10:20 CET); no new error lines.
- Vouched by the operator 2026-10-30 10:47:36 CET (the save took 132 ms — felhom.eu R-1346, hub 0.178.0).
## Live proof (Tester 1, controller 0.349.0)
`docker kill --signal=SIGSTOP felhom-controller` 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the agent's alarm
09:29:21 (6 sweeps) → „the controller does not answer — restarting the bootstrap unit" 09:31:22 (10 sweeps) → healthy
09:31:24. The hub mailed `controller_unresponsive_restart` and `controller_unresponsive` (09:42:56, with the next report)
and not `controller_restarted_by_agent`. Evidence: felhom.eu `documentation/audits/release-2026-10-30/p1-r1344/`.
+3 -1
View File
@@ -88,7 +88,8 @@
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) | | `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure | | `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` | | `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) | | `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause (`supervisorBrakeAllows` — every supervisor restart passes it; `recordSupervisorRestart` puts one on the record + R-539's slow counter); R-1344 (2026-10-30): a RUNNING controller whose Docker health reads „unhealthy" is alarmed at 6 sweeps and restarted at 10 (`restartUnresponsive`: the controller's busy file, then the brake); record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller`, `reboot_guest` since D65 — `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path | | `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
### Proxmox client / hub / PBS / provisioning ### Proxmox client / hub / PBS / provisioning
@@ -171,6 +172,7 @@
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). | | `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct | | `fstrim.Trimmer` (R-444) | internal/fstrim/fstrim.go | `New(runner, guests, gate, statePath, logger)` / `Pass(ctx)` / `GuestDiskTrimStatus(ctx)` / `ParseTrimmed(out)` | the weekly `pct fstrim <vmid>` of owned running guests (Wednesday from 10:00 local, starts 10:00-20:59 only), under `backup.InFlight`; last result per guest persisted and reported as `guest_disk_trim` | A busy gate DEFERS to the next hourly tick, never waits; a failed trim retries at most `MaxAttemptsPerWeek`; the report reads the persisted record, it never runs pct |
| `linetest.Tester` + `linetest.Client` (R-1247) | internal/linetest/linetest.go · internal/linetest/iperf.go | `New(runner, target, gate, uplink, hostID, statePath, logger)` / `Tick(ctx)` / `Value()` / `LineTestStatus(ctx)`; `Client{Streams, Time}.Run(ctx, addr)` | the weekly upload line test: a 10-second iperf3 upload (the agent speaks the iperf3 protocol — nothing installed on the host) to ep0's tunnel-only server (`wgtunnel.Manager.LineTestTarget`), Tuesday 10:00-15:59 local from a per-host slot, after 10 quiet minutes of the PHYSICAL uplink (`SysUplinkTx`: the bridge's ports with a `device` link — the bridge counter misses bridged guest traffic), under `backup.InFlight`; best of 3; reported as `line_test`, served as GET /host/line-test | **The value is the SERVER's received bytes**, never the client's written ones (the send buffer). `Run` refuses a context without a deadline. iperf3 numbers streams 1, 3, 4, 5 — the server matches results by those ids. „server busy" (ACCESS_DENIED) stops the measurement and is retried at the next tick, at most `MaxTriesPerDay` |
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. | | `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. | | `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. | | `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
+48
View File
@@ -0,0 +1,48 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
// passes as held → FAILS.
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
body := func(id string) []byte {
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
return b
}
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
if err != nil {
return nil, err
}
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
}
}
ctx := context.Background()
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
}
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
}
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
}
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
t.Fatal("a package without an identity blob cannot be compared — an error")
}
}
+72
View File
@@ -0,0 +1,72 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
)
// R-1250: SetLineTest takes a LineTestReader and finds the request half by a type assertion, so a Tester that lost
// WaitingForQuiet or RunNow would compile and silently answer 503 to every seeding box. This line makes it a build
// failure instead.
var _ localapi.LineTestRequester = (*linetest.Tester)(nil)
// R-1247: the weekly line test has the fstrim shape (component + reporter seam + goroutine) plus a local-API reader,
// so its wiring is asserted from the AST. It must share the ONE heavy-op gate (heavyOps) — or it could run beside a
// backup — and take its server from the tunnel manager, never a literal address.
func TestMainWiresTheLineTest(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var withGate, withTunnelTarget, reporterWired, localAPIWired, started bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
if fn, ok := node.Fun.(*ast.SelectorExpr); ok {
switch fn.Sel.Name {
case "New":
if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "linetest" && len(node.Args) >= 3 {
if id, ok := node.Args[2].(*ast.Ident); ok && id.Name == "heavyOps" {
withGate = true
}
if sel, ok := node.Args[1].(*ast.SelectorExpr); ok && sel.Sel.Name == "LineTestTarget" {
withTunnelTarget = true
}
}
case "SetLineTestReporter":
reporterWired = true
case "SetLineTest":
localAPIWired = true
}
}
case *ast.GoStmt:
if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Run" {
if id, ok := sel.X.(*ast.Ident); ok && id.Name == "lineTest" {
started = true
}
}
}
return true
})
if !withGate {
t.Error("main.go never calls linetest.New(..., heavyOps, ...) — no line test, or one outside the heavy-op gate")
}
if !withTunnelTarget {
t.Error("linetest.New does not take its server from wgMgr.LineTestTarget")
}
if !reporterWired {
t.Error("main.go never calls collector.SetLineTestReporter — the line_test stanza never reaches the hub")
}
if !localAPIWired {
t.Error("main.go never calls localSrv.SetLineTest — the controller's GET /host/line-test always reads present:false")
}
if !started {
t.Error("main.go never starts the line test with `go lineTest.Run(ctx)`")
}
}
+73
View File
@@ -43,6 +43,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/guestnet" "gitea.dooplex.hu/admin/felhom-agent/internal/guestnet"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver" "gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
"gitea.dooplex.hu/admin/felhom-agent/internal/localapi" "gitea.dooplex.hu/admin/felhom-agent/internal/localapi"
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log" applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
"gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane" "gitea.dooplex.hu/admin/felhom-agent/internal/mgmtplane"
@@ -1215,6 +1216,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ",")) logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}} }}
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
// v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound // v0.158.0 (felhom.eu `16` §9): the internal data place on an old box — the root helper, then the mount (late-bound
// to the local API server built below). // to the local API server built below).
internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner, internalPlaceExec := signedjobs.NewInternalPlaceExecutor(suRunner,
@@ -1246,6 +1250,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root // R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
// subprocess used, under the same per-host key, before the code is handed out. // subprocess used, under the same per-host key, before the code is handed out.
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) }, Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
// anything — the same per-host GET the identity read-back uses.
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
if client == nil {
return false, fmt.Errorf("hub client not configured")
}
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
},
DRConfigured: func() bool { DRConfigured: func() bool {
if drConfigured != nil { if drConfigured != nil {
return drConfigured() return drConfigured()
@@ -1255,6 +1267,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} }
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens) localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil { if localSrv != nil {
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
// which it requests at once through the same cap-1 trigger the storage watchdog uses.
localSrv.SetOperatorActionTrigger(func() {
select {
case storageTrigger <- struct{}{}:
default:
}
})
collector.SetOperatorActionResults(localSrv.OperatorActionResults)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner, localSrv))
backupCapRefusal = localSrv.BackupCapRefusal // R-973 backupCapRefusal = localSrv.BackupCapRefusal // R-973
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace) internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go. // R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
@@ -1641,6 +1664,24 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
} else { } else {
logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)") logger.Info("fstrim: weekly guest disk trim disabled by config (disk_trim.disable)")
} }
// R-1247 (operator ruling 2026-10-24 07:26): the weekly upload line test — a 10-second iperf3 upload from this host
// to ep0's tunnel-only server, Tuesday 10:00-15:59 local, after 10 quiet minutes, under the one-heavy-op gate. Only
// with the offsite tunnel (the server is reachable nowhere else). Not part of the errc fan-out.
switch {
case !cfg.LineTest.Enabled():
logger.Info("linetest: weekly upload line test disabled by config (line_test.disable)")
case wgMgr == nil:
logger.Info("linetest: weekly upload line test off — no offsite tunnel on this host")
default:
lineTest := linetest.New(linetest.Client{Streams: linetest.Streams, Time: linetest.TestTime}, wgMgr.LineTestTarget,
heavyOps, linetest.SysUplinkTx("/"), cfg.Hub.HostID,
filepath.Join(cfg.OOB.WithDefaults().StateDir, "line-test.json"), logger)
collector.SetLineTestReporter(lineTest)
if localSrv != nil {
localSrv.SetLineTest(lineTest)
}
go lineTest.Run(ctx)
}
if lanLoop != nil { if lanLoop != nil {
lanServers = 1 lanServers = 1
go func() { errc <- lanLoop.Run(ctx) }() go func() { errc <- lanLoop.Run(ctx) }()
@@ -3344,6 +3385,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the // putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978). // text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
// without an identity blob cannot be compared: an error (the caller then promises nothing).
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
var up escrowUploadRequest
if err := json.Unmarshal(pkg, &up); err != nil {
return false, fmt.Errorf("reading the pending package: %w", err)
}
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
if err != nil || len(mine) == 0 {
return false, fmt.Errorf("the pending package carries no identity blob to compare")
}
resp, err := fetch(ctx)
if err != nil {
return false, err
}
if !resp.Present || resp.IdentityEscrowB64 == "" {
return false, nil
}
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
if err != nil {
return false, fmt.Errorf("decoding the hub's package: %w", err)
}
return bytes.Equal(mine, theirs), nil
}
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error { func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" { if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)") return fmt.Errorf("hub not configured (url/host_id/api_key)")
@@ -4021,3 +4088,9 @@ func agentStateDir(cfg config.Config) string {
} }
return "/var/lib/felhom-agent" return "/var/lib/felhom-agent"
} }
// wireBundlePolicyAfterCommit hooks the bundle policy pass onto the self-update commit (R-1285). Pinned by
// TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit.
func wireBundlePolicyAfterCommit(m *selfupdate.Manager, e osupdate.ConfigUpdateExecutor, lg *slog.Logger, ver string) {
m.SetOnCommitted(func(ctx context.Context) { e.PolicyPassAfterUpdate(ctx, lg, ver) })
}
+51
View File
@@ -0,0 +1,51 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
// the server in the report loop's observer chain, give the collector its results and give the server the report
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
// RED-PROOF: seen failing before the wiring existed (all three false).
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var observed, results, trigger bool
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
fn, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
switch fn.Sel.Name {
case "MultiObserver":
for _, a := range call.Args {
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
observed = true
}
}
case "SetOperatorActionResults":
if len(call.Args) == 1 {
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
results = true
}
}
case "SetOperatorActionTrigger":
trigger = true
}
return true
})
if !observed || !results || !trigger {
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
}
}
@@ -0,0 +1,91 @@
package main
import (
"context"
"encoding/json"
"io"
"log/slog"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
)
// R-1285, in the REAL order (measured 2026-10-26 on demo-hp): the box gets agent_update and agent_config_update in one
// pass; the OLD process installs the bundle (no policy pass in it), then restarts into the new agent, which dwells and
// commits. The consequence asserted: the NEW agent runs the bundle policy pass exactly once, AFTER its commit — so a
// BYO host loses our drop-in in the same update. No pass without a commit (no marker, a mismatched marker).
// RED-PROOF: drop the wireBundlePolicyAfterCommit call in main → no bundle-policy plan after the commit.
// orderRunner records, in order, the self-update wrapper's verbs and the os-apply plans' modes.
type orderRunner struct {
mu sync.Mutex
order []string
}
func (r *orderRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
if name == osupdate.WrapperPath && len(args) == 2 && args[0] == "--plan" {
var p map[string]any
b, _ := os.ReadFile(args[1])
_ = json.Unmarshal(b, &p)
mode, _ := p["mode"].(string)
r.order = append(r.order, "os-apply:"+mode)
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":["/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"]}}` + "\n"), nil, nil
}
if len(args) >= 1 {
r.order = append(r.order, filepath.Base(name)+":"+args[0])
}
return []byte("ok"), nil, nil
}
func (r *orderRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func r1285Start(t *testing.T, marker *selfupdate.PendingMarker, running string) []string {
t.Helper()
stateDir := t.TempDir()
if marker != nil {
dir := filepath.Join(stateDir, "selfupdate")
if err := os.MkdirAll(dir, 0o750); err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(marker)
if err := os.WriteFile(filepath.Join(dir, "pending.json"), b, 0o644); err != nil {
t.Fatal(err)
}
}
r := &orderRunner{}
lg := slog.New(slog.NewTextHandler(io.Discard, nil))
mgr := selfupdate.NewManager(selfupdate.ManagerConfig{StateDir: stateDir, RunningVersion: running, Dwell: time.Millisecond, Runner: r, Logger: lg})
leg := &osupdate.Leg{Runner: r, Logger: lg, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last")}
exec := osupdate.ConfigUpdateExecutor{Leg: leg}
wireBundlePolicyAfterCommit(mgr, exec, lg, running)
mgr.MaybeCommit(context.Background())
r.mu.Lock()
defer r.mu.Unlock()
return append([]string(nil), r.order...)
}
func TestR1285_TheNewAgentRunsThePolicyPassAfterItsCommit(t *testing.T) {
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.167.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 2 || got[0] != "felhom-selfupdate-guarded:commit" || got[1] != "os-apply:bundle-policy" {
t.Fatalf("the new agent must commit, then run ONE bundle policy pass: %v", got)
}
}
func TestR1285_NoCommitNoPolicyPass(t *testing.T) {
if got := r1285Start(t, nil, "0.167.0"); len(got) != 0 {
t.Fatalf("an ordinary start (no pending update) must run nothing: %v", got)
}
got := r1285Start(t, &selfupdate.PendingMarker{OldVersion: "0.166.0", NewVersion: "0.168.0", SHA256: "ab", AppliedAt: "t"}, "0.167.0")
if len(got) != 0 {
t.Fatalf("a mismatched marker is not committed, so no pass: %v", got)
}
}
+4
View File
@@ -123,6 +123,10 @@ func run() error {
now := time.Now().UTC() now := time.Now().UTC()
issued := now issued := now
if *ttl <= 0 || *ttl > authz.MaxWindow {
fmt.Fprintf(os.Stderr, "felhom-opsign: -ttl %s refused: a box accepts a window of at most %s (R-1280)\n", *ttl, authz.MaxWindow)
os.Exit(2)
}
expires := now.Add(*ttl) expires := now.Add(*ttl)
// Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over. // Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over.
+3
View File
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template # docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed) # + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change) # systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on # felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag # STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a # and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \ /usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \ /usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \ /usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$ /usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a # Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
+182 -10
View File
@@ -165,9 +165,13 @@ KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# dest: (source file under configs/, mode, check, policy) # dest: (source file under configs/, mode, check, policy)
# check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile; # check: the content check BEFORE anything is written (R18): sudoers → visudo -cf; sh / bash → -n; python → compile;
# unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident); # unit → a [Unit]/[Service]/[Timer] section; unit-nort → that, and never RuntimeDirectory= (the G1 incident);
# agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none. # agent-unit → User=felhom-agent; dropin → a [Unit] section (SF-3); nft → nft -c -f; plain → none;
# sshd → `sshd -t -f` on the snippet alone (R-982).
# policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may # policy: replace → always written when it differs; if-absent → only when the box has none (a setting the operator may
# have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none). # have tuned); oob → only on a box with the OOB belt (/etc/felhom-sshd exists — the installer's --no-oob leaves none);
# appliance → only on a box whose ROOT-OWNED install record says appliance (D43 = A, operator 2026-10-24: a BYO
# host keeps its own settings, `11` §1). On a BYO box nothing is written and a copy WE wrote (this exact path, our
# first line) is removed; a box with no readable record is left exactly as it is (it cannot prove which it is).
# Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it. # Order matters: the sudoers files come LAST, so a referenced wrapper is in place before the line that allows it.
# R-861 (agent v0.146.0): the signed agent update, verified here (mode agent_update), then the A/B flip. # R-861 (agent v0.146.0): the signed agent update, verified here (mode agent_update), then the A/B flip.
SELFUPDATE_OP = "agent_update" SELFUPDATE_OP = "agent_update"
@@ -183,7 +187,14 @@ BUNDLE_PREV_DIR = "/var/lib/felhom-os-apply/bundle-prev" # the previous copies
BUNDLE_KEEP = 3 BUNDLE_KEEP = 3
BUNDLE_MAX_BYTES = 4 * 1024 * 1024 BUNDLE_MAX_BYTES = 4 * 1024 * 1024
BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$") BUNDLE_RE = re.compile(r"^bundle-[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?\.json$")
# R-1280: the longest validity window a signed job may carry — the agent's authz.MaxWindow, the same 24 h.
MAX_WINDOW_S = 24 * 3600
OOB_DIR = "/etc/felhom-sshd" OOB_DIR = "/etc/felhom-sshd"
# R-982: the host's own sshd drop-in (after_install reloads sshd for it; undo reloads after putting it back).
SSHD_DROPIN = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
# D43 = A (R-1257): the first line every copy of the drop-in we ship carries — what proves a file on a BYO host is ours.
SSHD_DROPIN_HEADER = b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982"
BUNDLE_FILES = [ BUNDLE_FILES = [
("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"), ("/usr/local/sbin/felhom-mkfs-guarded", "felhom-mkfs-guarded.sh", 0o755, "bash", "replace"),
("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"), ("/usr/local/sbin/felhom-selfupdate-guarded", "felhom-selfupdate-guarded", 0o755, "sh", "replace"),
@@ -221,6 +232,10 @@ BUNDLE_FILES = [
("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"), ("/etc/systemd/system/felhom-sshd.service", "felhom-sshd.service", 0o644, "unit-nort", "oob"),
("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"), ("/etc/felhom-oob.nft", "felhom-oob.nft", 0o644, "nft", "oob"),
("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"), ("/etc/systemd/system/felhom-oob-nft.service", "felhom-oob-nft.service", 0o644, "unit", "oob"),
# R-982, operator ruling D4 (2026-10-22): the host's own sshd (port 22, the LAN): root by key only. The console
# password stays; felhom-sshd (the operator's door) reads nothing from sshd_config.d. Reloaded in after_install.
# D43 = A (operator 2026-10-24, R-1257): appliances only — a BYO host keeps its own SSH settings (`11` §1).
("/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf", "felhom-root-ssh-key-only.conf", 0o644, "sshd", "appliance"),
("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"), ("/etc/sudoers.d/felhom-op", "felhom-op.sudoers", 0o440, "sudoers", "oob"),
("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"), ("/etc/sudoers.d/felhom-agent", "felhom-agent.sudoers", 0o440, "sudoers", "replace"),
] ]
@@ -434,7 +449,8 @@ class Runner:
def check_content(self, kind, data): def check_content(self, kind, data):
"""Run an external syntax check on a root-only temp copy. Returns (rc, message).""" """Run an external syntax check on a root-only temp copy. Returns (rc, message)."""
import tempfile import tempfile
cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"]}[kind] cmd = {"sudoers": ["visudo", "-cf"], "sh": ["sh", "-n"], "bash": ["bash", "-n"], "nft": ["nft", "-c", "-f"],
"sshd": ["/usr/sbin/sshd", "-t", "-f"]}[kind]
with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d: with tempfile.TemporaryDirectory(prefix="felhom-bundle-") as d:
p = os.path.join(d, "f") p = os.path.join(d, "f")
with open(p, "wb") as f: with open(p, "wb") as f:
@@ -475,7 +491,8 @@ class Apply:
def check_plan(self, plan): def check_plan(self, plan):
mode = plan.get("mode", "apply") mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \ if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
"oom-check") \
+ KERNEL_MODES: + KERNEL_MODES:
raise Refused("R11", f"unknown mode {mode!r}") raise Refused("R11", f"unknown mode {mode!r}")
if mode == "agent_update": if mode == "agent_update":
@@ -486,6 +503,10 @@ class Apply:
if plan.get("layer") != "host": if plan.get("layer") != "host":
raise Refused("R11", "bundle is a host-layer mode") raise Refused("R11", "bundle is a host-layer mode")
return mode, "host", 0, "bundle" return mode, "host", 0, "bundle"
if mode == "bundle-policy":
if plan.get("layer") != "host":
raise Refused("R11", "bundle-policy is a host-layer mode")
return mode, "host", 0, "bundle-policy"
layer = plan.get("layer") layer = plan.get("layer")
if layer not in ("guest", "host", "docker", "pve", "kernel"): if layer not in ("guest", "host", "docker", "pve", "kernel"):
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel") raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
@@ -647,6 +668,8 @@ class Apply:
iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ")) iss = calendar.timegm(time.strptime(op["issued_at"], "%Y-%m-%dT%H:%M:%SZ"))
except (KeyError, ValueError, TypeError): except (KeyError, ValueError, TypeError):
raise Refused("R3", "the signed job has no readable time window") raise Refused("R3", "the signed job has no readable time window")
if exp - iss > MAX_WINDOW_S:
raise Refused("R3", "the signed job's window is longer than 24 h (R-1280)")
if now > exp or now < iss - 120: if now > exp or now < iss - 120:
raise Refused("R3", "the signed job is expired or not yet valid") raise Refused("R3", "the signed job is expired or not yet valid")
nonce = op.get("nonce") nonce = op.get("nonce")
@@ -1056,6 +1079,8 @@ class Apply:
return self.facts() return self.facts()
if self.mode == "bundle": if self.mode == "bundle":
return Bundle(self).from_plan(plan) return Bundle(self).from_plan(plan)
if self.mode == "bundle-policy":
return Bundle(self).policy_pass()
if self.mode == "agent_update": if self.mode == "agent_update":
return self.agent_update(plan) return self.agent_update(plan)
if self.layer in ("host", "pve", "kernel"): if self.layer in ("host", "pve", "kernel"):
@@ -1864,6 +1889,8 @@ class Bundle:
def __init__(self, apply): def __init__(self, apply):
self.a, self.r = apply, apply.r self.a, self.r = apply, apply.r
self.report = apply.report self.report = apply.report
self.sshd_dropin_not_in_force = False
self.sshd_step_undone = None # X4 (R-1257): why the SSH step alone was put back ("" = not), the rest stays
# ---------- reading and checking ---------- # ---------- reading and checking ----------
def load_file(self, path): def load_file(self, path):
@@ -1932,10 +1959,17 @@ class Bundle:
text = content.decode("utf-8") text = content.decode("utf-8")
except UnicodeDecodeError: except UnicodeDecodeError:
raise Refused("R18", f"{dest} is not UTF-8 text") raise Refused("R18", f"{dest} is not UTF-8 text")
if check in ("sudoers", "sh", "bash", "nft"): if check in ("sudoers", "sh", "bash", "nft", "sshd"):
rc, msg = self.r.check_content(check, content) rc, msg = self.r.check_content(check, content)
if rc != 0: if rc != 0:
raise Refused("R18", f"{dest} fails its {check} check: {msg}") raise Refused("R18", f"{dest} fails its {check} check: {msg}")
if check == "sshd":
# R-982: the drop-in's PermitRootLogin must be exactly `prohibit-password`. `no` would also shut the root key
# out (Proxmox's own node-to-node SSH and the LAN key route), `yes` would undo the ruling — both refused.
# Only that line is pinned: the file is signed like every bundle file (review A7).
vals = re.findall(r"^\s*PermitRootLogin\s+(\S+)\s*$", text, re.M | re.I)
if vals != ["prohibit-password"]:
raise Refused("R18", f"{dest} must set exactly `PermitRootLogin prohibit-password` (found {vals})")
elif check == "python": elif check == "python":
try: try:
compile(text, dest, "exec") compile(text, dest, "exec")
@@ -1966,14 +2000,40 @@ class Bundle:
return b"", None, None # a symlink or a directory: always replaced by the real file return b"", None, None # a symlink or a directory: always replaced by the real file
return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid return self.r.read_bytes(dest), stat.S_IMODE(st.st_mode), st.st_uid
def install_mode(self):
"""The box's ROOT-OWNED install record's mode ("appliance", "byo", …), or None when it proves nothing (absent,
agent-writable, unreadable) — the same proof check_appliance asks for the host layer (R12)."""
try:
st = self.r.stat(INSTALL_STATE)
if st.st_uid != 0 or (st.st_mode & 0o022):
return None
mode = json.loads(self.r.read_file(INSTALL_STATE)).get("mode")
except (OSError, ValueError, AttributeError):
return None
return mode if isinstance(mode, str) and mode else None
def plan_writes(self, files): def plan_writes(self, files):
"""Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt).""" """Decide per file: write / same / kept (if-absent and present) / skipped (oob on a box without the belt; an
appliance-only file on a box that is not one) / remove (an appliance-only file WE wrote, on a BYO box)."""
oob = self.r.isdir(OOB_DIR) oob = self.r.isdir(OOB_DIR)
install_mode = self.install_mode()
plan = [] plan = []
for dest, content, mode, check, policy in files: for dest, content, mode, check, policy in files:
if policy == "oob" and not oob: if policy == "oob" and not oob:
plan.append((dest, content, mode, "skipped", None, None)) plan.append((dest, content, mode, "skipped", None, None))
continue continue
if policy == "appliance" and install_mode != "appliance":
old, old_mode, _ = self.live(dest)
ours = old is not None and old.startswith(SSHD_DROPIN_HEADER)
if install_mode == "byo" and ours:
self.r.log(f"os-apply: BUNDLE {dest}: a BYO host keeps its own settings (D43) — removing the copy we wrote")
plan.append((dest, content, mode, "remove", old, old_mode))
else:
why = "a BYO host keeps its own settings (D43)" if install_mode == "byo" else \
f"the install record proves no appliance (mode={install_mode!r}) — left as it is"
self.r.log(f"os-apply: BUNDLE {dest} skipped: {why}")
plan.append((dest, content, mode, "skipped", None, None))
continue
old, old_mode, old_uid = self.live(dest) old, old_mode, old_uid = self.live(dest)
if policy == "if-absent" and old is not None: if policy == "if-absent" and old is not None:
plan.append((dest, content, mode, "kept", old, old_mode)) plan.append((dest, content, mode, "kept", old, old_mode))
@@ -2011,11 +2071,47 @@ class Bundle:
ver, files = self.parse(data, sha) ver, files = self.parse(data, sha)
return self.install(ver, sha, files, "installer", False) return self.install(ver, sha, files, "installer", False)
def policy_pass(self):
"""R-1285 (D43 = A): the appliance-only files' policy, applied by THIS wrapper. A bundle that brings a new wrapper
is planned by the OLD one, which did not know the policy — so on a BYO host upgraded from 0.164.0 our SSH drop-in
stayed until the next bundle. The agent runs this pass right after every bundle install (the wrapper on disk is
then the new one). It only REMOVES a file WE wrote (our header) from a box whose root-owned install record says
byo, keeps the previous copy beside the bundle's, and reloads sshd; it writes nothing, so it needs no signature.
A failing `sshd -t` puts the file back (R-1257 X4's rule). Report key `bundle_policy`."""
log = self.r.log
install_mode = self.install_mode()
removed = []
if install_mode == "byo":
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
prev = os.path.join(BUNDLE_PREV_DIR, f"{stamp}-policy")
for dest, _src, _mode, _check, policy in BUNDLE_FILES:
if policy != "appliance":
continue
old, old_mode, _ = self.live(dest)
if old is None or not old.startswith(SSHD_DROPIN_HEADER):
continue
self.r.put_file(prev + dest, old, 0o600)
self.r.remove(dest)
removed.append((dest, old, old_mode))
log(f"os-apply: BUNDLE POLICY REMOVED {dest} (ours, on a BYO host — D43, R-1285)")
if any(d == SSHD_DROPIN for d, _, _ in removed):
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
if rc != 0:
for d, old, old_mode in removed:
self.r.put_file(d, old, old_mode)
log(f"os-apply: BUNDLE POLICY WARNING sshd -t failed without our drop-in (rc={rc}) — put back")
removed = []
else:
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
self.report["bundle_policy"] = {"install_mode": install_mode, "removed": [d for d, _, _ in removed]}
log(f"os-apply: BUNDLE POLICY DONE install_mode={install_mode} removed={len(removed)}")
return 0
# ---------- install, self-check, undo ---------- # ---------- install, self-check, undo ----------
def install(self, ver, sha, files, authority, bootstrap): def install(self, ver, sha, files, authority, bootstrap):
log = self.r.log log = self.r.log
plan = self.plan_writes(files) plan = self.plan_writes(files)
counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped")} counts = {k: sum(1 for p in plan if p[3] == k) for k in ("write", "same", "kept", "skipped", "remove")}
log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} " log(f"os-apply: BUNDLE START agent={ver} sha={sha[:16]} authority={authority} files={len(plan)} "
f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}") f"write={counts['write']} same={counts['same']} kept={counts['kept']} skipped={counts['skipped']}")
stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now())) stamp = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime(self.r.now()))
@@ -2023,10 +2119,17 @@ class Bundle:
done = [] # (dest, old bytes or None, old mode) done = [] # (dest, old bytes or None, old mode)
rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev, rep = {"agent_version": ver, "sha256": sha, "authority": authority, "prev_dir": prev,
"written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"], "written": [p[0] for p in plan if p[3] == "write"], "kept": [p[0] for p in plan if p[3] == "kept"],
"skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"]} "skipped": [p[0] for p in plan if p[3] == "skipped"], "same": counts["same"],
"removed": [p[0] for p in plan if p[3] == "remove"]}
self.report["bundle"] = rep self.report["bundle"] = rep
try: try:
for dest, content, mode, action, old, old_mode in plan: for dest, content, mode, action, old, old_mode in plan:
if action == "remove": # D43 = A: our copy off a BYO host (the previous copy kept, the undo restores it)
self.r.put_file(prev + dest, old, 0o600)
done.append((dest, old, old_mode))
self.r.remove(dest)
log(f"os-apply: BUNDLE REMOVED {dest} (ours, on a BYO host — D43)")
continue
if action != "write": if action != "write":
continue continue
if old is not None: if old is not None:
@@ -2044,22 +2147,58 @@ class Bundle:
self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]} self.report["failed"] = {"rc": 3, "step": "bundle", "reason": reason[:300]}
return 3 return 3
rep["signers_created"] = False rep["signers_created"] = False
if self.sshd_dropin_not_in_force:
rep["sshd_dropin_not_in_force"] = True # R-982 A2: written and reloaded, but sshd -T shows root passwords allowed
if self.sshd_step_undone:
rep["sshd_step_undone"] = self.sshd_step_undone[:300] # X4: only the SSH step was put back
if bootstrap and not self.r.lexists(TRUST_SIGNERS): if bootstrap and not self.r.lexists(TRUST_SIGNERS):
self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644) self.r.put_file(TRUST_SIGNERS, pinned_signers_line().encode(), 0o644)
rep["signers_created"] = True rep["signers_created"] = True
log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}") log(f"os-apply: BUNDLE created {TRUST_SIGNERS} with exactly the pinned key {PINNED_OPERATOR_KEY_ID}")
record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority, record = {"format": BUNDLE_FORMAT, "agent_version": ver, "bundle_sha256": sha, "authority": authority,
"installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())), "installed_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(self.r.now())),
"files": {p[0]: (sha256_hex(p[1]) if p[3] in ("write", "same") else "files": {p[0]: self.record_entry(p) for p in plan}}
(sha256_hex(p[4]) if p[3] == "kept" else "skipped")) for p in plan}}
self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644) self.r.put_file(BUNDLE_RECORD, (json.dumps(record, indent=2, sort_keys=True) + "\n").encode(), 0o644)
self.prune() self.prune()
log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} " log(f"os-apply: BUNDLE DONE agent={ver} written={counts['write']} same={counts['same']} "
f"self-check=ok signers-created={rep['signers_created']}") f"self-check=ok signers-created={rep['signers_created']}")
return 0 return 0
def record_entry(self, p):
"""The record's word for one plan entry: the sha it installed, the sha it kept, or what it did instead."""
dest, content, _, action, old, _ = p
if dest == SSHD_DROPIN and self.sshd_step_undone:
return sha256_hex(old) if old is not None else "absent" # X4: the SSH step was put back
if action in ("write", "same"):
return sha256_hex(content)
if action == "kept":
return sha256_hex(old)
return "removed" if action == "remove" else "skipped"
def lan_addresses(self):
"""X3 (R-1257): the host's own global IPv4 addresses — the ones a LAN client reaches sshd on."""
rc, out, _ = self.r.host(["ip", "-4", "-o", "addr", "show", "scope", "global"], 30)
if rc != 0:
return []
return sorted(set(re.findall(r"\binet (\d+\.\d+\.\d+\.\d+)/", out)))
def undo_sshd_step(self, plan, why):
"""X4 (R-1257): `sshd -t` refused the host's whole config with our step in it — put back ONLY that file (the
previous copy, or none) and keep the rest of the bundle. sshd was not reloaded, so it never ran the refused state."""
for dest, _, _, action, old, old_mode in plan:
if dest != SSHD_DROPIN or action not in ("write", "remove"):
continue
if old is None:
self.r.remove(dest)
else:
self.r.put_file(dest, old, old_mode if old_mode is not None else 0o644)
self.sshd_step_undone = why
self.r.log(f"os-apply: BUNDLE WARNING sshd -t refused the host's config with the SSH step — ONLY that step is put "
f"back, the rest of the bundle stays: {why[:200]} (R-1257)")
def after_install(self, plan): def after_install(self, plan):
written = {p[0] for p in plan if p[3] == "write"} written = {p[0] for p in plan if p[3] == "write"}
removed = {p[0] for p in plan if p[3] == "remove"}
dests = {p[0] for p in plan if p[3] != "skipped"} dests = {p[0] for p in plan if p[3] != "skipped"}
if any(d.startswith("/etc/systemd/system/") for d in written): if any(d.startswith("/etc/systemd/system/") for d in written):
self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload") self.must(["systemctl", "daemon-reload"], "systemctl daemon-reload")
@@ -2073,6 +2212,36 @@ class Bundle:
"enable the crash guard") "enable the crash guard")
if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests: if "/etc/systemd/system/felhom-mgmt-watchdog.timer" in dests:
self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer") self.must(["systemctl", "enable", "--now", "felhom-mgmt-watchdog.timer"], "enable the mgmt watchdog timer")
# R-982: the host's sshd takes the root-key-only drop-in. The WHOLE config is tested first (a failure is a
# refusal, and the bundle undo puts the old files back); `try-reload-or-restart` keeps every open session and
# does nothing on a box whose ssh.service is not running (a socket-activated or stopped sshd reads the file
# at its next start) — a failure there must not freeze the bundle route (review 2026-10-22 night, A1).
# felhom-sshd is not touched. Then the POSITIVE check: sshd's own effective view for root. A box whose
# sshd_config has no Include of sshd_config.d, an earlier drop-in or a Match block can keep the password route
# — that is said loudly in the log and the report, never silently recorded as done (A2).
# R-1257 X4: a failing `sshd -t` puts back ONLY the SSH step (undo_sshd_step); it used to refuse and undo the
# whole bundle. X3: the in-force check asks sshd about each of the host's own LAN addresses — a probe from a
# documentation address (192.0.2.1) read a `Match Address <LAN>` block that re-allows passwords as in force.
if SSHD_DROPIN in written or SSHD_DROPIN in removed:
rc, out, err = self.r.host(["/usr/sbin/sshd", "-t"], 120)
if rc != 0:
self.undo_sshd_step(plan, f"sshd -t failed (rc={rc}): {(out + err).strip()[-200:]}")
return
self.must(["systemctl", "try-reload-or-restart", "ssh"], "systemctl try-reload-or-restart ssh")
if SSHD_DROPIN in written:
addrs = self.lan_addresses()
if not addrs:
self.r.log("os-apply: BUNDLE WARNING no global IPv4 address read for the sshd in-force check — "
"probing 192.0.2.1 only (R-1257)")
addrs = ["192.0.2.1"]
for addr in addrs:
rc, out, err = self.r.host(["/usr/sbin/sshd", "-T", "-C", f"user=root,host=felhom-check,addr={addr}"], 60)
eff = re.findall(r"^permitrootlogin\s+(\S+)", out, re.M | re.I)
if rc != 0 or eff != ["without-password"]:
self.r.log(f"os-apply: BUNDLE WARNING the root-key-only drop-in is NOT in force for {addr}: sshd -T "
f"says permitrootlogin={eff or '?'} (rc={rc}) — check sshd_config's Include, an earlier "
f"drop-in or a Match block (R-982, R-1257)")
self.sshd_dropin_not_in_force = True
def must(self, argv, what): def must(self, argv, what):
rc, out, err = self.r.host(argv, 120) rc, out, err = self.r.host(argv, 120)
@@ -2133,6 +2302,9 @@ class Bundle:
self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}") self.r.log(f"os-apply: BUNDLE UNDO could not restore {dest}: {e}")
if any(d.startswith("/etc/systemd/system/") for d in back): if any(d.startswith("/etc/systemd/system/") for d in back):
self.r.host(["systemctl", "daemon-reload"], 120) self.r.host(["systemctl", "daemon-reload"], 120)
if SSHD_DROPIN in back:
# A3: after_install may already have reloaded sshd with the drop-in; the restored state takes effect now.
self.r.host(["systemctl", "try-reload-or-restart", "ssh"], 120)
rc, _, _ = self.r.host(["visudo", "-c"], 60) rc, _, _ = self.r.host(["visudo", "-c"], 60)
self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}") self.r.log(f"os-apply: BUNDLE UNDO restored={len(back)} visudo-after={'ok' if rc == 0 else 'FAILED'}")
return back return back
+7
View File
@@ -0,0 +1,7 @@
# MANAGED BY FELHOM (the agent's config bundle) — R-982, operator ruling D4 (2026-10-22).
# The Proxmox host's OWN sshd — the listener the household network reaches on port 22: root logs in with a KEY only.
# The root password still works on the console (the break-glass password: felhom.eu runbooks/break-glass.md). The
# operator's door (felhom-sshd, port 8822, /etc/felhom-sshd/sshd_config) is a separate daemon whose config includes
# nothing from here, so it is not changed. Debian's sshd_config reads this folder BEFORE its own settings, and for
# sshd the first value wins; the bundle checks the effective value with `sshd -T` after the reload.
PermitRootLogin prohibit-password
+238 -1
View File
@@ -45,8 +45,10 @@ class St:
class Box: class Box:
"""An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set.""" """An in-memory host. files: path -> bytes; modes/uids per path; dirs: a set."""
def __init__(self, bundle_bytes, signed, oob=False, signers=True): def __init__(self, bundle_bytes, signed, oob=False, signers=True, install_mode="appliance"):
self.files, self.modes, self.uids = {}, {}, {} self.files, self.modes, self.uids = {}, {}, {}
if install_mode is not None: # the installer's ROOT-OWNED record (D43 = A reads it for the sshd drop-in)
self.put(osapply.INSTALL_STATE, json.dumps({"completed": [], "mode": install_mode}).encode(), 0o644)
self.dirs = {osapply.OOB_DIR} if oob else set() self.dirs = {osapply.OOB_DIR} if oob else set()
self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644) self.put(osapply.TRUST_FILE, json.dumps({"host_id": HOST, "ring0_slow_lane": False}).encode(), 0o644)
if signers: if signers:
@@ -429,6 +431,241 @@ class SelfCheckUndo(unittest.TestCase):
self.assert_restored(box, before, rep) self.assert_restored(box, before, rep)
class RootKeyOnly(unittest.TestCase):
"""R-982, operator ruling D4 (2026-10-22): the bundle gives the host's own sshd a root-key-only drop-in, tests the
whole config and reloads sshd — only when the file was written; a drop-in that says anything but
`prohibit-password` is refused before anything is written.
RED-PROOF: with the after_install block removed, test_fresh_box_takes_the_dropin_and_reloads fails (no reload);
with the content rule removed, test_a_dropin_that_says_yes_or_no_is_refused fails — seen failing 2026-10-22."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
def test_fresh_box_takes_the_dropin_and_reloads(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertIn(b"PermitRootLogin prohibit-password", box.files[self.DEST])
self.assertEqual(box.modes[self.DEST], 0o644)
self.assertIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertFalse(any("felhom-sshd" in " ".join(c) for c in box.calls), "the operator's door must not be touched")
def test_an_unchanged_dropin_is_not_reloaded(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.put(self.DEST, (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes(), 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_dropin_that_says_yes_or_no_is_refused(self):
for bad in (b"PermitRootLogin yes\n", b"PermitRootLogin no\n", b"# nothing\n",
b"PermitRootLogin prohibit-password\nPermitRootLogin yes\n"):
def edit(files, bad=bad):
for e in files:
if e["path"] == self.DEST:
e["content_b64"] = base64.b64encode(bad).decode()
data, sha = edited_bundle(edit)
box = Box(data, signed_job(sha))
rc, rep = run(box)
self.assertNotEqual(rc, 0, bad)
self.assertNotIn(self.DEST, box.files, bad)
# Review 2026-10-22 night. A2: the effective value is read back with `sshd -T`; a drop-in that is not in force is
# said in the log and the report (not refused — a refusal would freeze the bundle route). A3: a bundle undone
# after the reload reloads sshd again with the old state. RED-PROOF: with the sshd -T check removed,
# test_a_dropin_not_in_force_is_reported fails; with the undo reload removed, test_an_undone_bundle_reloads_sshd
# fails — seen failing 2026-10-22 night.
def _sshd_T(self, box, value):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
return 0, f"port 22\npermitrootlogin {value}\npasswordauthentication yes\n", ""
return orig(argv, timeout, stdin)
box.host = host
def test_the_effective_value_is_checked(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "without-password")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(any(c[:2] == ["/usr/sbin/sshd", "-T"] for c in box.calls))
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_dropin_not_in_force_is_reported(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._sshd_T(box, "yes") # e.g. no Include of sshd_config.d
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force" in l for l in box.logs), box.logs)
def test_an_undone_bundle_reloads_sshd(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
box.visudo_fail = True # the self-check fails AFTER the install → undo
rc, rep = run(box)
self.assertEqual(rc, 3, rep)
self.assertNotIn(self.DEST, box.files)
reloads = [i for i, c in enumerate(box.calls) if c == ["systemctl", "try-reload-or-restart", "ssh"]]
self.assertEqual(len(reloads), 2, box.calls) # after_install, then the undo
class ByoKeepsItsSSH(unittest.TestCase):
"""D43 = A (operator 2026-10-24, R-1257): the root-key-only drop-in goes ONLY to a box whose root-owned install record
says appliance; on a BYO host nothing is written and a copy WE wrote is removed; a box with no record is left as it
is. X3: the in-force check asks sshd about each of the host's own LAN addresses. X4: a failing `sshd -t` puts back
only the SSH step. RED-PROOF (2026-10-24, felhom-agent f5c3d77 + these tests): with the drop-in's policy back to
`replace`, test_a_byo_host_gets_no_dropin and test_our_copy_comes_off_a_byo_host fail; with the probe back to
192.0.2.1 only, test_a_match_block_for_the_lan_is_not_in_force fails; with `sshd -t` back to `must`,
test_a_failing_sshd_t_undoes_only_the_ssh_step fails (rc 3, the whole bundle undone)."""
DEST = "/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf"
OURS = (REPO / "configs" / "felhom-root-ssh-key-only.conf").read_bytes()
def test_a_byo_host_gets_no_dropin(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertNotIn(["/usr/sbin/sshd", "-t"], box.calls)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
self.assertIn(self.DEST, rep["bundle"]["skipped"])
rec = json.loads(box.files[osapply.BUNDLE_RECORD])
self.assertEqual(rec["files"][self.DEST], "skipped")
def test_our_copy_comes_off_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, self.OURS, 0o644) # an earlier bundle wrote it (before D43)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle"]["removed"], [self.DEST])
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files),
"the previous copy must be kept for the undo")
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], "removed")
def test_the_owners_own_file_stays_on_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode="byo")
box.put(self.DEST, b"# my own\nPermitRootLogin yes\n", 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files[self.DEST], b"# my own\nPermitRootLogin yes\n")
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_box_with_no_record_is_left_as_it_is(self):
for setup in ("none", "agent-writable"):
data, sha = real_bundle()
box = Box(data, signed_job(sha), install_mode=None if setup == "none" else "byo")
if setup == "agent-writable":
box.modes[osapply.INSTALL_STATE] = 0o666 # proves nothing
box.put(self.DEST, self.OURS, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS, setup)
self.assertIn(self.DEST, rep["bundle"]["skipped"], setup)
def _fake(self, box, addrs=("192.168.0.50",), match=None, sshd_t_rc=0):
orig = box.host
def host(argv, timeout=600, stdin=None):
if argv[:3] == ["ip", "-4", "-o"]:
box.calls.append(argv)
return 0, "".join(f"2: vmbr0 inet {a}/24 brd x scope global vmbr0\\ valid_lft forever\n" for a in addrs), ""
if argv[:2] == ["/usr/sbin/sshd", "-T"]:
box.calls.append(argv)
val = "yes" if match and f"addr={match}" in argv[-1] else "without-password"
return 0, f"port 22\npermitrootlogin {val}\n", ""
if argv == ["/usr/sbin/sshd", "-t"]:
box.calls.append(argv)
return (sshd_t_rc, "", "/etc/ssh/sshd_config line 9: Bad configuration option") if sshd_t_rc else (0, "", "")
return orig(argv, timeout, stdin)
box.host = host
# R-1285: the policy pass the agent runs right after every bundle install, with the wrapper NOW on disk. No
# signature (it only removes a file we wrote). RED-PROOF: make policy_pass return before the loop → our copy stays.
POLICY_PLAN = {"release_id": "bundle-policy-0.166.0", "layer": "host", "mode": "bundle-policy"}
def test_the_policy_pass_removes_our_copy_from_a_byo_host(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644) # the old wrapper's bundle run left it
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn(self.DEST, box.files)
self.assertEqual(rep["bundle_policy"], {"install_mode": "byo", "removed": [self.DEST]}, rep)
self.assertTrue(any(k.endswith(self.DEST) and k.startswith(osapply.BUNDLE_PREV_DIR) for k in box.files))
self.assertLess(box.calls.index(["/usr/sbin/sshd", "-t"]), box.calls.index(["systemctl", "try-reload-or-restart", "ssh"]))
def test_the_policy_pass_leaves_an_appliance_and_the_owners_file(self):
for mode, content in (("appliance", None), ("byo", b"# my own\nPermitRootLogin yes\n")):
data, sha = real_bundle()
box = Box(data, None, install_mode=mode)
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
body = content if content is not None else self.OURS
box.put(self.DEST, body, 0o644)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), body, mode)
self.assertEqual(rep["bundle_policy"]["removed"], [], mode)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, mode)
def test_the_policy_pass_puts_our_copy_back_when_sshd_t_fails(self):
data, sha = real_bundle()
box = Box(data, None, install_mode="byo")
box.put(PLAN, json.dumps(self.POLICY_PLAN).encode(), 0o600, uid=999)
box.put(self.DEST, self.OURS, 0o644)
self._fake(box, sshd_t_rc=1)
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertEqual(box.files.get(self.DEST), self.OURS)
self.assertEqual(rep["bundle_policy"]["removed"], [])
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls)
def test_a_match_block_for_the_lan_is_not_in_force(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"), match="192.168.0.50")
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
probed = [c[-1] for c in box.calls if c[:2] == ["/usr/sbin/sshd", "-T"]]
self.assertTrue(any("addr=192.168.0.50" in p for p in probed) and any("addr=10.77.0.4" in p for p in probed), probed)
self.assertFalse(any("addr=192.0.2.1" in p for p in probed), probed)
self.assertTrue(rep["bundle"].get("sshd_dropin_not_in_force"), rep)
self.assertTrue(any("NOT in force for 192.168.0.50" in l for l in box.logs), box.logs)
def test_the_lan_check_passes_when_every_address_is_key_only(self):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
self._fake(box, addrs=("192.168.0.50", "10.77.0.4"))
rc, rep = run(box)
self.assertEqual(rc, 0, rep)
self.assertNotIn("sshd_dropin_not_in_force", rep["bundle"])
def test_a_failing_sshd_t_undoes_only_the_ssh_step(self):
for before in (None, b"# MANAGED BY FELHOM (the agent's config bundle) \xe2\x80\x94 R-982 old\nPermitRootLogin prohibit-password\n"):
data, sha = real_bundle()
box = Box(data, signed_job(sha))
if before is not None:
box.put(self.DEST, before, 0o644)
self._fake(box, sshd_t_rc=255)
rc, rep = run(box)
self.assertEqual(rc, 0, rep) # the bundle stays
self.assertEqual(box.files.get(self.DEST), before, "only the SSH step is put back")
self.assertIn("/usr/local/sbin/felhom-os-apply", box.files, "the rest of the bundle stays")
self.assertIn("sshd -t failed", rep["bundle"].get("sshd_step_undone", ""), rep)
self.assertNotIn(["systemctl", "try-reload-or-restart", "ssh"], box.calls, "sshd never ran the refused state")
want = "absent" if before is None else hashlib.sha256(before).hexdigest()
self.assertEqual(json.loads(box.files[osapply.BUNDLE_RECORD])["files"][self.DEST], want)
class TrustBootstrap(unittest.TestCase): class TrustBootstrap(unittest.TestCase):
def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self): def test_missing_signers_verifies_against_the_pinned_key_and_creates_it(self):
data, sha = real_bundle() data, sha = real_bundle()
+8
View File
@@ -833,6 +833,14 @@ class DockerLane(unittest.TestCase):
def test_expired_signed_job_is_refused(self): def test_expired_signed_job_is_refused(self):
self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3") self.refused(docker_fake(signed=signed_job(expires="2026-10-04T11:55:00Z")), "R3")
def test_a_window_longer_than_24h_is_refused(self):
# R-1280. RED-PROOF: drop the MAX_WINDOW_S check → the job (current, signed, 24 h + 1 s) runs.
self.refused(docker_fake(signed=signed_job(issued="2026-10-03T11:59:59Z", expires="2026-10-04T12:00:00Z")), "R3")
def test_a_window_of_24h_is_accepted(self):
rc, rep = run(docker_fake(signed=signed_job(issued="2026-10-03T12:30:00Z", expires="2026-10-04T12:30:00Z")))
self.assertNotEqual((rep.get("refused") or {}).get("code"), "R3", rep)
def test_replayed_signed_job_is_refused(self): def test_replayed_signed_job_is_refused(self):
f = docker_fake(signed=signed_job()) f = docker_fake(signed=signed_job())
f.nonces = {"n1": f.clock + 600} f.nonces = {"n1": f.clock + 600}
+1 -1
View File
@@ -1,6 +1,6 @@
module gitea.dooplex.hu/admin/felhom-agent module gitea.dooplex.hu/admin/felhom-agent
go 1.27.2 go 1.27.3
require ( require (
golang.org/x/crypto v0.58.0 golang.org/x/crypto v0.58.0
+2
View File
@@ -22,6 +22,8 @@ var (
ErrExpired = errors.New("authz: op expired") ErrExpired = errors.New("authz: op expired")
// ErrNotYetValid: now < issued_at (minus clock-skew tolerance). // ErrNotYetValid: now < issued_at (minus clock-skew tolerance).
ErrNotYetValid = errors.New("authz: op not yet valid") ErrNotYetValid = errors.New("authz: op not yet valid")
// ErrWindowTooLong: expires_at - issued_at > MaxWindow (R-1280).
ErrWindowTooLong = errors.New("authz: op validity window too long")
// ErrReplay: the nonce was already recorded in the window. // ErrReplay: the nonce was already recorded in the window.
ErrReplay = errors.New("authz: replay (nonce already seen)") ErrReplay = errors.New("authz: replay (nonce already seen)")
) )
+8
View File
@@ -20,6 +20,11 @@ const Namespace = "felhom-op-v1"
// the validity window stays an honest upper bound. // the validity window stays an honest upper bound.
const DefaultClockSkew = 2 * time.Minute const DefaultClockSkew = 2 * time.Minute
// MaxWindow is the longest validity window a signed op may carry (R-1280, D44 #22, operator 2026-10-25): a job signed
// with a long window by mistake would stay usable that long, and the nonce store would keep its record as long. The
// root wrapper (configs/felhom-os-apply, MAX_WINDOW_S) and felhom-opsign (-ttl) hold the same 24 h.
const MaxWindow = 24 * time.Hour
// KeyRole tags a pinned operator key (doc 04 §3 two-key model). // KeyRole tags a pinned operator key (doc 04 §3 two-key model).
type KeyRole string type KeyRole string
@@ -150,6 +155,9 @@ func (v *Verifier) Verify(blob, sigArmored []byte) (*VerifiedOp, error) {
if now.Before(op.IssuedAt.Add(-v.ClockSkew)) { if now.Before(op.IssuedAt.Add(-v.ClockSkew)) {
return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now) return nil, fmt.Errorf("%w: issued_at=%s now=%s", ErrNotYetValid, op.IssuedAt, now)
} }
if w := op.ExpiresAt.Sub(op.IssuedAt); w > MaxWindow {
return nil, fmt.Errorf("%w: issued_at=%s expires_at=%s (%s, the most is %s)", ErrWindowTooLong, op.IssuedAt, op.ExpiresAt, w, MaxWindow)
}
if now.After(op.ExpiresAt) { if now.After(op.ExpiresAt) {
return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now) return nil, fmt.Errorf("%w: expires_at=%s now=%s", ErrExpired, op.ExpiresAt, now)
} }
+20
View File
@@ -117,6 +117,26 @@ func TestVerify_RejectsPerStage(t *testing.T) {
rejects(t, err, ErrTarget) rejects(t, err, ErrTarget)
}) })
// R-1280: a window longer than 24 h is refused even while it is current; exactly 24 h is accepted.
// RED-PROOF: drop the MaxWindow check → the 24 h + 1 s case verifies.
t.Run("window longer than 24 h", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-1", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow+time.Second))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
_, err := v.Verify(blob, sig)
rejects(t, err, ErrWindowTooLong)
})
t.Run("window of exactly 24 h is accepted", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-win-2", "{}",
refNow.Add(-time.Minute), refNow.Add(-time.Minute).Add(MaxWindow))
sig := mintArmor(t, pub.Marshal(), Namespace, "sha512", blob, sign)
v := atRefNow(New(signerSet(pub, "op"), NewMemoryNonceStore(), "demo-felhom"))
if _, err := v.Verify(blob, sig); err != nil {
t.Fatalf("a 24 h window must verify: %v", err)
}
})
t.Run("expired", func(t *testing.T) { t.Run("expired", func(t *testing.T) {
blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}", blob := canonicalBlob("guest_destroy", "demo-felhom", "9001", "op", "n-exp-1", "{}",
refNow.Add(-2*time.Hour), refNow.Add(-time.Hour)) refNow.Add(-2*time.Hour), refNow.Add(-time.Hour))
@@ -0,0 +1,55 @@
package backup
import (
"context"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1232 — the 2026-10-23 test night: a reinstalled box (new install, new PBS key, same customer namespace, same guest
// number 9201) found the OLD box's archive (2.6 days old) when the due-check asked „when did this tier last land?", so
// the whole-guest tier read „not due" for up to its 168-h cadence and the ep0 copy stayed the OLD box's. The due-check
// now counts only archives written with THIS storage's key (the R-727 discriminator; the archive carries no host id).
// The CONSEQUENCE asserted: with only a foreign archive the answer is „none found" (the tier is due); with the box's
// own archive, that archive's time. COMPANION RED-PROOF: drop the key filter in NewestArchiveTime → case 1 returns the
// old box's time.
func TestR1232_TheDueCheckCountsOnlyThisBoxsArchives(t *testing.T) {
oldBox := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-20T21:10:00Z", Content: "backup", VMID: 9201,
Size: 3490689830, CTime: time.Date(2026, 10, 20, 21, 10, 0, 0, time.UTC).Unix(), Encrypted: earlierBox1}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbsdr", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{oldBox},
}
r := NewBackupRunner(api, "felhom-pbsdr", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The fresh box has written nothing yet: the old box's archive must not count.
when, found, err := r.NewestArchiveTime(context.Background(), 9201)
if err != nil {
t.Fatal(err)
}
if found {
t.Fatalf("the due-check counted ANOTHER box's archive (%s) as this tier's last backup — the tier would wait a week", when)
}
// 2. After the box's own first copy, that copy is the answer.
own := proxmox.StorageContent{VolID: "felhom-pbsdr:backup/ct/9201/2026-10-24T02:30:00Z", Content: "backup", VMID: 9201,
Size: 3600000000, CTime: time.Date(2026, 10, 24, 2, 30, 0, 0, time.UTC).Unix(), Encrypted: thisBoxKey}
api.content = append(api.content, own)
when, found, err = r.NewestArchiveTime(context.Background(), 9201)
if err != nil || !found || when.Unix() != own.CTime {
t.Fatalf("want this box's own archive time %d, got %v found=%v err=%v", own.CTime, when, found, err)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps: no key filter, the newest counts.
func TestR1232_UnencryptedStorageCountsEveryArchive(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_10_25-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1793050025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, found, err := r.NewestArchiveTime(context.Background(), 9201); err != nil || !found {
t.Fatalf("an unencrypted local archive was not counted: found=%v err=%v", found, err)
}
}
+12
View File
@@ -609,11 +609,23 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
if err != nil { if err != nil {
return time.Time{}, false, err return time.Time{}, false, err
} }
// R-1232: on an ENCRYPTED storage only archives written with THIS storage's key are this box's (the R-727
// discriminator — the archive carries no host id). A reinstalled box shares the customer's namespace and guest
// number with the old box, and the old box's archive made the tier read „not due" for a whole cadence (168 h).
// A key lookup failure is returned: the due-check reads UNKNOWN, never a foreign archive's age.
ownKey, err := r.storageKeyFingerprint(ctx, r.target)
if err != nil {
return time.Time{}, false, fmt.Errorf("reading the key fingerprint of storage %s: %w", r.target, err)
}
var best int64 = -1 var best int64 = -1
for _, e := range contents { for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid { if e.Content != "backup" || e.VMID != vmid {
continue continue
} }
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this tier's last backup (R-1232)", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
if ok, why := archivePlausiblyComplete(e); !ok { if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why) r.warnRejectedArchiveOnce(e, why)
continue continue
+1
View File
@@ -151,6 +151,7 @@ var manifest = []Capability{
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""}, {"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""}, {"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""}, {"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""}, {"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant. // R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""}, {"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
+11
View File
@@ -34,6 +34,7 @@ type Config struct {
WGTunnel WGTunnelConfig `json:"wg_tunnel"` WGTunnel WGTunnelConfig `json:"wg_tunnel"`
GuestNet GuestNetConfig `json:"guest_net"` GuestNet GuestNetConfig `json:"guest_net"`
DiskTrim DiskTrimConfig `json:"disk_trim"` DiskTrim DiskTrimConfig `json:"disk_trim"`
LineTest LineTestConfig `json:"line_test"`
OOB OOBConfig `json:"oob"` OOB OOBConfig `json:"oob"`
SelfUpdate SelfUpdateConfig `json:"selfupdate"` SelfUpdate SelfUpdateConfig `json:"selfupdate"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info) LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
@@ -150,6 +151,16 @@ type DiskTrimConfig struct {
// Enabled reports whether the weekly trim should run. // Enabled reports whether the weekly trim should run.
func (d DiskTrimConfig) Enabled() bool { return !d.Disable } func (d DiskTrimConfig) Enabled() bool { return !d.Disable }
// LineTestConfig configures the R-1247 weekly upload line test (internal/linetest). DEFAULT-ON: it reaches only ep0's
// tunnel-only iperf3 server, so it runs only on a box that already has an operational offsite tunnel (operator
// ruling 2026-10-24 07:26: every box with a tunnel measures its line weekly).
type LineTestConfig struct {
Disable bool `json:"disable"`
}
// Enabled reports whether the weekly line test should run.
func (l LineTestConfig) Enabled() bool { return !l.Disable }
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet). // GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
// //
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other // **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
+27
View File
@@ -104,6 +104,12 @@ type GuestDiskTrimReporter interface {
GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus GuestDiskTrimStatus(ctx context.Context) *GuestDiskTrimStatus
} }
// LineTestReporter is the R-1247 seam the weekly line test plugs into (hub does not import linetest). nil → no
// line_test stanza.
type LineTestReporter interface {
LineTestStatus(ctx context.Context) *LineTestStatus
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow // Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing. // interfaces for unit testing.
type Collector struct { type Collector struct {
@@ -122,8 +128,10 @@ type Collector struct {
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted) wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted) pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted) ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted) guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted) diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted) deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted) foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false) selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
@@ -228,6 +236,12 @@ type ControllerSupervisorReporter interface {
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
} }
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
c.opResults = f
return c
}
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe). // SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector { func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
c.ctrlSup = r c.ctrlSup = r
@@ -259,6 +273,12 @@ func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector
return c return c
} }
// SetLineTestReporter wires the R-1247 weekly line test as a report source (nil-safe → stanza omitted).
func (c *Collector) SetLineTestReporter(r LineTestReporter) *Collector {
c.lineTest = r
return c
}
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted). // SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector { func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
c.deliberateOff = f c.deliberateOff = f
@@ -419,6 +439,9 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.ctrlSup != nil { if c.ctrlSup != nil {
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx) report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
} }
if c.opResults != nil {
report.OperatorActionResults = c.opResults()
}
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted). // R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
if c.guestNet != nil { if c.guestNet != nil {
report.GuestNet = c.guestNet.GuestNetStatus(ctx) report.GuestNet = c.guestNet.GuestNetStatus(ctx)
@@ -427,6 +450,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.diskTrim != nil { if c.diskTrim != nil {
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx) report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
} }
// R-1247: the last weekly line test (nil reporter = not wired → stanza omitted).
if c.lineTest != nil {
report.LineTest = c.lineTest.LineTestStatus(ctx)
}
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted). // R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
if c.foreignKey != nil { if c.foreignKey != nil {
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx) report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
+54
View File
@@ -129,6 +129,11 @@ type HostReport struct {
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret. // wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"` GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
// LineTest is the weekly upload line test stanza (R-1247; operator ruling 2026-10-24): the schedule, the last
// measured upload line (best of the runs, Mbit/s, as ep0's iperf3 server counted it) and the last attempt. Present
// only when the job is wired. No secret.
LineTest *LineTestStatus `json:"line_test,omitempty"`
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the // ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box // restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since // cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
@@ -167,6 +172,24 @@ type HostReport struct {
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go // byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
// against the JSON TestControllerSupervisorStanza_WireShape pins here. // against the JSON TestControllerSupervisorStanza_WireShape pins here.
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"` ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
// when there is nothing to answer, so the golden stays byte-stable.
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
}
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
type OperatorAction struct {
ID int64 `json:"id"`
Action string `json:"action"`
Arg string `json:"arg,omitempty"`
}
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
type OperatorActionResult struct {
ID int64 `json:"id"`
Outcome string `json:"outcome"`
Message string `json:"message"`
} }
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret. // ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
@@ -190,6 +213,17 @@ type ControllerSupervisorGuest struct {
Restarts24h int `json:"restarts_24h"` Restarts24h int `json:"restarts_24h"`
SlowCrashloop bool `json:"slow_crashloop"` SlowCrashloop bool `json:"slow_crashloop"`
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339 SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
Unresponsive bool `json:"unresponsive"`
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
// R-1344 restart (decided 2026-10-30 from the night's measurement): UnresponsiveRestartAfterS is how long after the
// alarm the agent restarts a controller that still does not answer (0 / absent = an older agent: alarm only — the
// hub's message says which). UnresponsiveRestartAt is the last such restart; the hub keys on it MOVING
// (controller_unresponsive_restart) and it equals LastRestartAt for that act.
UnresponsiveRestartAfterS int `json:"unresponsive_restart_after_s,omitempty"`
UnresponsiveRestartAt string `json:"unresponsive_restart_at,omitempty"` // RFC3339
} }
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States: // PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
@@ -232,6 +266,19 @@ type GuestDiskTrimStatus struct {
Guests []GuestDiskTrim `json:"guests,omitempty"` Guests []GuestDiskTrim `json:"guests,omitempty"`
} }
// LineTestStatus is the R-1247 weekly upload line test stanza. `last_ok_at` + `mbps` are the last SUCCESSFUL
// measurement ("" / 0 = never); `last_attempt_at` + `last_error` the last attempt, whatever its verdict — never read
// the attempt time as a result.
type LineTestStatus struct {
Schedule string `json:"schedule"`
LastOKAt string `json:"last_ok_at,omitempty"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Server string `json:"server,omitempty"`
LastAttemptAt string `json:"last_attempt_at,omitempty"`
LastError string `json:"last_error,omitempty"`
}
// GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never // GuestDiskTrim is one guest's LAST trim attempt. `ok` with `last_attempt_at` is the verdict of that attempt — never
// read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is // read the time alone as success; `last_ok_at` is the last attempt that succeeded ("" = never). `bytes_trimmed` is
// the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points. // the sum of the "(N bytes) trimmed" lines `pct fstrim` printed, over `mounts` mount points.
@@ -292,6 +339,10 @@ type WireguardStatus struct {
Registered bool `json:"registered"` // the registration marker exists Registered bool `json:"registered"` // the registration marker exists
Active bool `json:"active"` // wg-quick@wg-felhom is-active Active bool `json:"active"` // wg-quick@wg-felhom is-active
LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"` LastHandshakeAgeS *int64 `json:"last_handshake_age_s,omitempty"`
// HandshakeFutureS (R-1204): seconds the latest handshake lies in the FUTURE by this box's clock — set (> 0) only
// then. A box that ran once with its clock ahead keeps a handshake stamped in the future; WireGuard on ep0 then
// refuses every newer handshake, and LastHandshakeAgeS (clamped at 0) reads „just now" for ever. The hub alarms on it.
HandshakeFutureS *int64 `json:"handshake_future_s,omitempty"`
AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32" AssignedIP string `json:"assigned_ip,omitempty"` // from the marker, e.g. "10.77.0.2/32"
} }
@@ -609,6 +660,9 @@ type ControlEnvelope struct {
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror). // NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
// Absent/false on an old hub → nothing happens. // Absent/false on an old hub → nothing happens.
LogTailRequested bool `json:"log_tail_requested"` LogTailRequested bool `json:"log_tail_requested"`
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
// (`04` §6.1). Absent on an older hub.
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
} }
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail). // LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
+294
View File
@@ -0,0 +1,294 @@
package linetest
import (
"context"
"crypto/rand"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"strconv"
"sync"
"sync/atomic"
"time"
)
// A minimal iperf3 CLIENT for one TCP upload test (client → server), speaking the iperf3 control protocol (3.x) to the
// stock `iperf3 --server` on ep0 (operator ruling 2026-10-24: the agent speaks iperf3 itself, nothing is installed on
// the host). The number that counts is the SERVER's: the bytes it received, from its results JSON — bytes the client
// handed to its socket include what still sat in the send buffer when the timer ended.
//
// The protocol, as the server drives it (one state byte at a time on the control connection):
//
// client → 37-byte cookie
// server → PARAM_EXCHANGE(9) client → params JSON (4-byte big-endian length + bytes)
// server → CREATE_STREAMS(10) client opens N data connections, each starts with the same cookie
// server → TEST_START(1), TEST_RUNNING(2) client sends for the test time, then → TEST_END(4)
// server → EXCHANGE_RESULTS(13) client → its results JSON; server → its results JSON
// server → DISPLAY_RESULTS(14) client → IPERF_DONE(16)
//
// A server already running a test answers ACCESS_DENIED(-1) and closes: that is ErrBusy (another box is testing; try
// again later the same day). SERVER_ERROR(-3) / SERVER_TERMINATE(-2) end the test with an error.
const (
stParamExchange = 9
stCreateStreams = 10
stTestStart = 1
stTestRunning = 2
stTestEnd = 4
stExchangeResults = 13
stDisplayResults = 14
stIperfDone = 16
stAccessDenied = -1
stServerTerminate = -2
stServerError = -3
cookieSize = 37 // 36 characters + NUL
blockSize = 128 * 1024
maxJSONBytes = 1 << 20
)
// ErrBusy: the server is running another client's test.
var ErrBusy = errors.New("iperf3 server busy")
// Result is one upload test.
type Result struct {
Mbps float64 // the server's received bits per second / 1e6
ServerBytes int64 // bytes the server received, all streams
ClientBytes int64 // bytes the client wrote (diagnostics only)
ServerSeconds float64 // the server's measured duration
}
// Dialer opens a TCP connection (net.Dialer in production; a seam for tests).
type Dialer interface {
DialContext(ctx context.Context, network, address string) (net.Conn, error)
}
// Client runs one test. The whole run is bounded by ctx; Run sets every connection's deadline from it.
type Client struct {
Dialer Dialer
Streams int // parallel streams (-P)
Time time.Duration // test time (-t)
}
func (c Client) dialer() Dialer {
if c.Dialer != nil {
return c.Dialer
}
return &net.Dialer{Timeout: 5 * time.Second}
}
func newCookie() ([]byte, error) {
const alphabet = "abcdefghijklmnopqrstuvwxyz234567"
b := make([]byte, cookieSize)
if _, err := rand.Read(b[:cookieSize-1]); err != nil {
return nil, err
}
for i := 0; i < cookieSize-1; i++ {
b[i] = alphabet[int(b[i])%len(alphabet)]
}
b[cookieSize-1] = 0
return b, nil
}
func writeJSON(w io.Writer, v any) error {
data, err := json.Marshal(v)
if err != nil {
return err
}
var hdr [4]byte
binary.BigEndian.PutUint32(hdr[:], uint32(len(data)))
if _, err := w.Write(append(hdr[:], data...)); err != nil {
return err
}
return nil
}
func readJSON(r io.Reader, v any) error {
var hdr [4]byte
if _, err := io.ReadFull(r, hdr[:]); err != nil {
return err
}
n := binary.BigEndian.Uint32(hdr[:])
if n == 0 || n > maxJSONBytes {
return fmt.Errorf("iperf3: results JSON of %d bytes refused", n)
}
buf := make([]byte, n)
if _, err := io.ReadFull(r, buf); err != nil {
return err
}
return json.Unmarshal(buf, v)
}
type streamResult struct {
ID int `json:"id"`
Bytes int64 `json:"bytes"`
Retransmits int64 `json:"retransmits"`
Jitter float64 `json:"jitter"`
Errors int64 `json:"errors"`
OmittedErrors int64 `json:"omitted_errors"`
Packets int64 `json:"packets"`
OmittedPackets int64 `json:"omitted_packets"`
StartTime float64 `json:"start_time"`
EndTime float64 `json:"end_time"`
}
type results struct {
CPUTotal float64 `json:"cpu_util_total"`
CPUUser float64 `json:"cpu_util_user"`
CPUSystem float64 `json:"cpu_util_system"`
SenderHasRetransmits int `json:"sender_has_retransmits"`
CongestionUsed string `json:"congestion_used,omitempty"`
Streams []streamResult `json:"streams"`
}
// streamIDs are the ids iperf3 gives its streams: the first is 1, every later one is (streams so far) + 2 — so 1, 3,
// 4, 5 … (iperf_add_stream). The server matches our results by these ids.
func streamIDs(n int) []int {
ids := make([]int, n)
for i := range ids {
if i == 0 {
ids[i] = 1
} else {
ids[i] = i + 2
}
}
return ids
}
// Run performs one upload test to addr ("10.77.0.1:5201").
func (c Client) Run(ctx context.Context, addr string) (Result, error) {
var res Result
streams, dur := c.Streams, c.Time
if streams < 1 {
streams = 1
}
if dur <= 0 {
dur = 10 * time.Second
}
deadline, ok := ctx.Deadline()
if !ok {
return res, errors.New("iperf3: Run needs a context with a deadline")
}
cookie, err := newCookie()
if err != nil {
return res, err
}
ctrl, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: connect: %w", err)
}
defer ctrl.Close()
ctrl.SetDeadline(deadline)
if _, err := ctrl.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: cookie: %w", err)
}
var data []net.Conn
defer func() {
for _, d := range data {
d.Close()
}
}()
var sent atomic.Int64
var server results
var wg sync.WaitGroup
for {
var st [1]byte
if _, err := io.ReadFull(ctrl, st[:]); err != nil {
return res, fmt.Errorf("iperf3: control read: %w", err)
}
switch int8(st[0]) {
case stParamExchange:
params := map[string]any{"tcp": true, "omit": 0, "time": int(dur / time.Second), "num": 0, "blockcount": 0,
"parallel": streams, "len": blockSize, "pacing_timer": 1000, "client_version": "3.18"}
if err := writeJSON(ctrl, params); err != nil {
return res, fmt.Errorf("iperf3: params: %w", err)
}
case stCreateStreams:
for i := 0; i < streams; i++ {
d, err := c.dialer().DialContext(ctx, "tcp", addr)
if err != nil {
return res, fmt.Errorf("iperf3: data stream %d: %w", i+1, err)
}
d.SetDeadline(deadline)
data = append(data, d)
if _, err := d.Write(cookie); err != nil {
return res, fmt.Errorf("iperf3: data stream %d cookie: %w", i+1, err)
}
}
case stTestStart:
case stTestRunning:
stopAt := time.Now().Add(dur)
if stopAt.After(deadline) {
return res, errors.New("iperf3: no time left for the test inside the deadline")
}
buf := make([]byte, blockSize)
for _, d := range data {
d.SetWriteDeadline(stopAt)
wg.Add(1)
go func(d net.Conn) {
defer wg.Done()
for {
n, err := d.Write(buf)
sent.Add(int64(n))
if err != nil {
return
}
}
}(d)
}
wg.Wait()
if _, err := ctrl.Write([]byte{stTestEnd}); err != nil {
return res, fmt.Errorf("iperf3: test end: %w", err)
}
case stExchangeResults:
ids := streamIDs(len(data))
mine := results{SenderHasRetransmits: -1, CongestionUsed: "cubic"}
per := sent.Load() / int64(max(len(data), 1))
for _, id := range ids {
mine.Streams = append(mine.Streams, streamResult{ID: id, Bytes: per, Retransmits: -1, EndTime: dur.Seconds()})
}
if err := writeJSON(ctrl, mine); err != nil {
return res, fmt.Errorf("iperf3: send results: %w", err)
}
if err := readJSON(ctrl, &server); err != nil {
return res, fmt.Errorf("iperf3: read server results: %w", err)
}
case stDisplayResults:
ctrl.Write([]byte{stIperfDone})
return summarize(server, sent.Load(), dur)
case stAccessDenied:
return res, ErrBusy
case stServerTerminate, stServerError:
var codes [8]byte
io.ReadFull(ctrl, codes[:])
return res, fmt.Errorf("iperf3: server error (state %d, iperf errno %d, errno %d)", int8(st[0]),
int32(binary.BigEndian.Uint32(codes[:4])), int32(binary.BigEndian.Uint32(codes[4:])))
default:
return res, fmt.Errorf("iperf3: unexpected state %s", strconv.Itoa(int(int8(st[0]))))
}
}
}
// summarize turns the server's results into a Result. No stream bytes is an error, never a zero line.
func summarize(server results, clientBytes int64, dur time.Duration) (Result, error) {
res := Result{ClientBytes: clientBytes}
for _, s := range server.Streams {
res.ServerBytes += s.Bytes
if s.EndTime > res.ServerSeconds {
res.ServerSeconds = s.EndTime
}
}
if res.ServerBytes <= 0 {
return res, errors.New("iperf3: the server reported no received bytes")
}
secs := res.ServerSeconds
if secs <= 0 {
secs = dur.Seconds()
}
res.Mbps = float64(res.ServerBytes) * 8 / secs / 1e6
return res, nil
}
+541
View File
@@ -0,0 +1,541 @@
// Package linetest is the weekly upload line-speed test (R-1247; operator ruling 2026-10-24 07:26).
//
// Why: the controller's upload watch and qBittorrent's default limit (50 % of the line, D41) need the household's
// upload line. The passive reading — the nightly off-site push — is only a low floor (2026-10-24 night: demo-hp „at
// least 10 Mbit/s"; this test read 90.7 on the same line). So once a week the HOST (the tunnel lives on the host)
// runs a 10-second iperf3 upload to ep0's tunnel-only iperf3 server (felhom.eu scripts/ep0-iperf3). No file is left
// anywhere and nothing counts against the household's storage.
//
// The rule, each part pinned by a test in linetest_test.go:
// - Weekly: due from Tuesday 10:00 local until one measurement succeeded since then (a box that was off catches up
// on a later day, inside the window).
// - Daytime only: a test starts only 10:00–15:59 local — never in the night chain's window (01:00–06:59), and each
// host starts from its own minute (SlotMinute, spread over 10:00–13:59) so the fleet does not test at once.
// - Never beside a backup, a restore-test, an OS update or a trim: the test holds the host-wide one-heavy-op gate.
// - Only after QuietMinutes of quiet uplink: every one-minute transmit rate of the host's physical uplink below
// QuietMbps — so neither the family's upload nor a restore is measured against itself.
// - Up to RunsPerMeasurement runs of TestTime each; the highest counts. A run is bounded by RunTimeout.
// - „Server busy" (another box is testing) → tried again at the next tick the same day; at most MaxTriesPerDay.
// - No tunnel (no operational wg-felhom with a server address) → no test and no value; the controller then keeps its
// passive floor and sets no qBittorrent limit.
// - The last result is persisted, so a restart neither loses it nor re-tests.
// - A box whose torrent app seeds all day is never quiet for 10 minutes (R-1250). For it the controller holds the
// app's upload limit very low and asks for a test on request (RunNow): every rule above except the 10 quiet
// minutes still holds, the quiet is checked over RequestQuietWindow instead, and the runs are RequestRunGap
// apart, so the whole request ends in about 55 s.
package linetest
import (
"context"
"encoding/json"
"errors"
"fmt"
"hash/fnv"
"log/slog"
"net/netip"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
const (
Weekday = time.Tuesday
StartHour = 10 // first eligible local hour (inclusive)
EndHour = 16 // first NOT-eligible local hour: the last start is 15:59
SlotSpreadMinutes = 240
QuietMinutes = 10
QuietMbps = 2.0
RunsPerMeasurement = 3
TestTime = 10 * time.Second
Streams = 4
RunTimeout = 25 * time.Second
RunGap = 30 * time.Second
MaxTriesPerDay = 6
ServerPort = 5201
TickInterval = time.Minute
FirstTickDelay = 5 * time.Minute
GateName = "line-test"
// A test on request (R-1250): the controller holds the app's limit while it waits, so it is short.
RequestSettle = 5 * time.Second // the held app's last packets leave before the quiet is read
RequestQuietWindow = 10 * time.Second // the uplink must send below QuietMbps over this window
RequestRunGap = 5 * time.Second
)
// ScheduleText is carried on the host report.
const ScheduleText = "weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update"
// Gate is the host-wide one-heavy-operation gate (*backup.InFlight).
type Gate interface {
TryAcquire(what string) (release func(), busy string, ok bool)
}
// Target answers the iperf3 server's tunnel address, ok=false when the tunnel is not operational.
type Target func() (netip.Addr, bool)
// Runner runs one upload test (Client in production).
type Runner interface {
Run(ctx context.Context, addr string) (Result, error)
}
// UplinkTx reads the host's physical-uplink transmit byte counter.
type UplinkTx func() (uint64, error)
// State is what is persisted (line-test.json).
type State struct {
LastOKAt time.Time `json:"last_ok_at,omitempty"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Server string `json:"server,omitempty"`
LastAttemptAt time.Time `json:"last_attempt_at,omitempty"`
LastError string `json:"last_error,omitempty"`
TriesDay string `json:"tries_day,omitempty"` // local date of TriesToday
TriesToday int `json:"tries_today,omitempty"`
}
type txSample struct {
at time.Time
tx uint64
}
// Tester is the weekly job.
type Tester struct {
runner Runner
target Target
gate Gate
uplink UplinkTx
statePath string
slot int // minutes after StartHour:00 this host may first start
logger *slog.Logger
loc *time.Location
now func() time.Time
sleep func(context.Context, time.Duration) bool
mu sync.Mutex
st State
samples []txSample
}
// New builds the job and loads the persisted state (missing = empty; corrupt = logged, empty — the cost is one extra
// test, never a lost value elsewhere).
func New(runner Runner, target Target, gate Gate, uplink UplinkTx, hostID, statePath string, logger *slog.Logger) *Tester {
if logger == nil {
logger = slog.Default()
}
t := &Tester{runner: runner, target: target, gate: gate, uplink: uplink, statePath: statePath,
slot: SlotMinute(hostID), logger: logger, loc: time.Local, now: time.Now, sleep: sleepCtx}
t.load()
return t
}
func sleepCtx(ctx context.Context, d time.Duration) bool {
tm := time.NewTimer(d)
defer tm.Stop()
select {
case <-ctx.Done():
return false
case <-tm.C:
return true
}
}
// SlotMinute spreads hosts over the first SlotSpreadMinutes of the window, stable per host id.
func SlotMinute(hostID string) int {
h := fnv.New32a()
h.Write([]byte(hostID))
return int(h.Sum32() % SlotSpreadMinutes)
}
func (t *Tester) load() {
data, err := os.ReadFile(t.statePath)
if err != nil {
if !os.IsNotExist(err) {
t.logger.Warn("linetest: state read failed — starting empty", "path", t.statePath, "err", err)
}
return
}
if err := json.Unmarshal(data, &t.st); err != nil {
t.logger.Warn("linetest: state file corrupt — starting empty", "path", t.statePath, "err", err)
t.st = State{}
}
}
func (t *Tester) saveLocked() error {
data, err := json.MarshalIndent(t.st, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(t.statePath), 0o755); err != nil {
return err
}
tmp := t.statePath + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, t.statePath)
}
// EligibleAt reports whether a test may START at local time lt for a host whose slot is slot minutes after
// StartHour:00 on the due day. On a catch-up day (anchorDay=false) the slot is not waited for again.
func EligibleAt(lt time.Time, slot int, anchorDay bool) bool {
h := lt.Hour()
if h < StartHour || h >= EndHour {
return false
}
if anchorDay {
mins := (h-StartHour)*60 + lt.Minute()
return mins >= slot
}
return true
}
// weekAnchor is the most recent Tuesday StartHour:00 at or before lt.
func weekAnchor(lt time.Time) time.Time {
daysBack := (int(lt.Weekday()) - int(Weekday) + 7) % 7
d := lt.AddDate(0, 0, -daysBack)
a := time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
if a.After(lt) {
d = d.AddDate(0, 0, -7)
a = time.Date(d.Year(), d.Month(), d.Day(), StartHour, 0, 0, 0, lt.Location())
}
return a
}
// Due reports whether a measurement is due at lt given the state.
func Due(st State, lt time.Time) bool {
return st.LastOKAt.IsZero() || st.LastOKAt.In(lt.Location()).Before(weekAnchor(lt))
}
// sample records the uplink counter (once a minute) and keeps QuietMinutes+1 samples.
func (t *Tester) sample() {
if t.uplink == nil {
return
}
tx, err := t.uplink()
if err != nil {
t.logger.Debug("linetest: uplink counter unreadable", "err", err)
t.mu.Lock()
t.samples = nil // a gap is not quiet
t.mu.Unlock()
return
}
t.mu.Lock()
t.samples = append(t.samples, txSample{at: t.now(), tx: tx})
if len(t.samples) > QuietMinutes+1 {
t.samples = t.samples[len(t.samples)-(QuietMinutes+1):]
}
t.mu.Unlock()
}
// quiet reports whether the last QuietMinutes minutes each sent below QuietMbps, and the highest minute seen.
func quiet(samples []txSample) (bool, float64) {
if len(samples) < QuietMinutes+1 {
return false, 0
}
worst := 0.0
for i := 1; i < len(samples); i++ {
dt := samples[i].at.Sub(samples[i-1].at).Seconds()
if dt <= 0 || dt > 3*TickInterval.Seconds() || samples[i].tx < samples[i-1].tx {
return false, worst // a gap or a counter reset is not proof of quiet
}
mbps := float64(samples[i].tx-samples[i-1].tx) * 8 / dt / 1e6
if mbps > worst {
worst = mbps
}
}
return worst < QuietMbps, worst
}
// Run ticks every TickInterval until ctx ends. It never returns an error: a failed test is a reported fact.
func (t *Tester) Run(ctx context.Context) {
t.logger.Info("linetest: weekly upload line test starting", "schedule", ScheduleText, "slot_minute", t.slot)
if !t.sleep(ctx, FirstTickDelay) {
return
}
for {
t.sample()
t.Tick(ctx)
if !t.sleep(ctx, TickInterval) {
return
}
}
}
// Refusal is why a test on request did not run (RunNow). Code is stable for the controller.
type Refusal struct {
Code string // not-due | outside-window | tries-used | no-tunnel | heavy-op | not-quiet
Msg string
}
func (r *Refusal) Error() string { return "linetest: " + r.Code + ": " + r.Msg }
// ready applies every rule except the quiet one: due this week, inside the window (the slot on the due day), tries
// left today, the tunnel up. It answers the server address and the local day.
func (t *Tester) ready() (addr, day string, refusal *Refusal) {
lt := t.now().In(t.loc)
t.mu.Lock()
st := t.st
t.mu.Unlock()
if !Due(st, lt) {
return "", "", &Refusal{"not-due", "measured this week"}
}
anchor := weekAnchor(lt)
anchorDay := lt.YearDay() == anchor.YearDay() && lt.Year() == anchor.Year()
if !EligibleAt(lt, t.slot, anchorDay) {
return "", "", &Refusal{"outside-window", "a test starts 10:00-15:59 local, from this host's slot on the due day"}
}
day = lt.Format("2006-01-02")
if st.TriesDay == day && st.TriesToday >= MaxTriesPerDay {
return "", "", &Refusal{"tries-used", fmt.Sprintf("%d tries today", st.TriesToday)}
}
ip, ok := t.target()
if !ok {
return "", "", &Refusal{"no-tunnel", "no operational tunnel"}
}
return netip.AddrPortFrom(ip, ServerPort).String(), day, nil
}
// Tick is one look.
func (t *Tester) Tick(ctx context.Context) {
addr, day, refusal := t.ready()
if refusal != nil {
if refusal.Code == "no-tunnel" {
t.logger.Debug("linetest: no operational tunnel — no test")
}
return
}
t.mu.Lock()
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if q, worst := quiet(samples); !q {
t.logger.Debug("linetest: uplink not quiet — waiting", "worst_minute_mbps", fmt.Sprintf("%.1f", worst), "samples", len(samples))
return
}
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
t.logger.Info("linetest: deferred — a heavy operation is in flight", "busy", busy)
return
}
defer release()
t.measure(ctx, addr, day, RunGap)
}
// WaitingForQuiet reports that a measurement is due and could start now but the uplink has not been quiet for
// QuietMinutes — the case a seeding torrent app makes permanent (R-1250). The controller reads it from
// GET /host/line-test and only then holds its app's limit and calls RunNow.
func (t *Tester) WaitingForQuiet() bool {
if _, _, refusal := t.ready(); refusal != nil {
return false
}
t.mu.Lock()
samples := append([]txSample(nil), t.samples...)
t.mu.Unlock()
if len(samples) < QuietMinutes+1 {
return false // not yet watched for 10 minutes (a fresh start): not known to be busy, so no hold
}
q, _ := quiet(samples)
return !q
}
// RunNow is a test on request (POST /host/line-test/run, R-1250): every rule of the weekly test except the 10 quiet
// minutes, which the caller replaces by holding its own upload down. The uplink must still be quiet over
// RequestQuietWindow (after RequestSettle), so a request never measures the family's upload against itself. A
// *Refusal says why nothing ran; any other error is a failed measurement (the same as a failed weekly attempt).
func (t *Tester) RunNow(ctx context.Context) (mbps float64, runs []float64, err error) {
addr, day, refusal := t.ready()
if refusal != nil {
return 0, nil, refusal
}
release, busy, ok := t.gate.TryAcquire(GateName)
if !ok {
return 0, nil, &Refusal{"heavy-op", busy}
}
defer release()
if t.uplink == nil {
return 0, nil, &Refusal{"not-quiet", "no uplink counter"}
}
if !t.sleep(ctx, RequestSettle) {
return 0, nil, ctx.Err()
}
tx0, err0 := t.uplink()
at0 := t.now()
if !t.sleep(ctx, RequestQuietWindow) {
return 0, nil, ctx.Err()
}
tx1, err1 := t.uplink()
if err0 != nil || err1 != nil || tx1 < tx0 {
return 0, nil, &Refusal{"not-quiet", "uplink counter unreadable or reset"}
}
secs := t.now().Sub(at0).Seconds()
if secs <= 0 {
secs = RequestQuietWindow.Seconds()
}
if m := float64(tx1-tx0) * 8 / secs / 1e6; m >= QuietMbps {
t.logger.Info("linetest: test on request refused — the uplink is not quiet", "mbps", fmt.Sprintf("%.1f", m))
return 0, nil, &Refusal{"not-quiet", fmt.Sprintf("the uplink sent %.1f Mbit/s while held", m)}
}
t.logger.Info("linetest: test on request (a held torrent app, R-1250)", "server", addr)
if err := t.measure(ctx, addr, day, RequestRunGap); err != nil {
return 0, nil, err
}
t.mu.Lock()
defer t.mu.Unlock()
return t.st.Mbps, append([]float64(nil), t.st.RunsMbps...), nil
}
// measure runs the runs, gap apart, and stores the result. It returns nil only when a value was stored.
func (t *Tester) measure(ctx context.Context, addr, day string, gap time.Duration) error {
start := t.now()
var runs []float64
var lastErr error
busy := false
for i := 0; i < RunsPerMeasurement; i++ {
if i > 0 && !t.sleep(ctx, gap) {
break
}
rctx, cancel := context.WithTimeout(ctx, RunTimeout)
r, err := t.runner.Run(rctx, addr)
cancel()
if errors.Is(err, ErrBusy) {
busy, lastErr = true, err
break
}
if err != nil {
lastErr = err
t.logger.Warn("linetest: run failed", "run", i+1, "server", addr, "err", err)
continue
}
runs = append(runs, r.Mbps)
t.logger.Info(fmt.Sprintf("linetest: run %d: %.1f Mbit/s", i+1, r.Mbps), "server", addr,
"server_bytes", r.ServerBytes, "server_seconds", r.ServerSeconds)
}
t.mu.Lock()
defer t.mu.Unlock()
if t.st.TriesDay != day {
t.st.TriesDay, t.st.TriesToday = day, 0
}
t.st.TriesToday++
t.st.LastAttemptAt = start.UTC()
if len(runs) > 0 {
best := runs[0]
for _, r := range runs {
if r > best {
best = r
}
}
t.st.LastOKAt, t.st.Mbps, t.st.RunsMbps, t.st.Server, t.st.LastError = start.UTC(), best, runs, addr, ""
t.logger.Info(fmt.Sprintf("linetest: upload line measured: %.1f Mbit/s (best of %d)", best, len(runs)),
"server", addr, "runs", len(runs), "duration_s", t.now().Sub(start).Seconds())
} else {
msg := "no run succeeded"
if lastErr != nil {
msg = lastErr.Error()
}
if len(msg) > 300 {
msg = msg[:300]
}
t.st.LastError = msg
if busy {
t.logger.Info("linetest: the test server is busy — trying again later today", "server", addr, "tries_today", t.st.TriesToday)
} else {
t.logger.Warn("linetest: measurement failed", "server", addr, "tries_today", t.st.TriesToday, "err", msg)
}
}
if err := t.saveLocked(); err != nil {
t.logger.Warn("linetest: state save failed — the result will not survive a restart", "path", t.statePath, "err", err)
}
if len(runs) > 0 {
return nil
}
if lastErr == nil {
lastErr = errors.New("linetest: no run succeeded")
}
return lastErr
}
// Value is the measured line (ok=false: never measured).
func (t *Tester) Value() (mbps float64, at time.Time, ok bool) {
t.mu.Lock()
defer t.mu.Unlock()
if t.st.LastOKAt.IsZero() || t.st.Mbps <= 0 {
return 0, time.Time{}, false
}
return t.st.Mbps, t.st.LastOKAt, true
}
// LineTestStatus implements hub.LineTestReporter: a pure read of the persisted state.
func (t *Tester) LineTestStatus(context.Context) *hub.LineTestStatus {
t.mu.Lock()
defer t.mu.Unlock()
out := &hub.LineTestStatus{Schedule: ScheduleText, Mbps: t.st.Mbps, RunsMbps: t.st.RunsMbps, Server: t.st.Server,
LastError: t.st.LastError}
if !t.st.LastOKAt.IsZero() {
out.LastOKAt = t.st.LastOKAt.UTC().Format(time.RFC3339)
}
if !t.st.LastAttemptAt.IsZero() {
out.LastAttemptAt = t.st.LastAttemptAt.UTC().Format(time.RFC3339)
}
return out
}
// ── the host's physical uplink counter ──────────────────────────────────────────────────────────────────────────
// SysUplinkTx reads the transmit bytes of the host's physical uplink under root (normally "/"): the interface of the
// default route in /proc/net/route, and — when it is a bridge (vmbr0), as on every Proxmox host — the sum of its
// PHYSICAL member ports (those with a `device` link). The bridge's own counter does not see the guests' bridged
// traffic, which is the traffic this check exists for.
func SysUplinkTx(root string) UplinkTx {
return func() (uint64, error) {
ifc, err := defaultRouteIface(filepath.Join(root, "proc/net/route"))
if err != nil {
return 0, err
}
sys := filepath.Join(root, "sys/class/net")
ports := []string{ifc}
if ents, err := os.ReadDir(filepath.Join(sys, ifc, "brif")); err == nil {
ports = ports[:0]
for _, e := range ents {
if _, err := os.Stat(filepath.Join(sys, e.Name(), "device")); err == nil {
ports = append(ports, e.Name())
}
}
if len(ports) == 0 {
return 0, fmt.Errorf("linetest: bridge %s has no physical port", ifc)
}
}
var sum uint64
for _, p := range ports {
b, err := os.ReadFile(filepath.Join(sys, p, "statistics/tx_bytes"))
if err != nil {
return 0, err
}
n, err := strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64)
if err != nil {
return 0, err
}
sum += n
}
return sum, nil
}
}
func defaultRouteIface(path string) (string, error) {
b, err := os.ReadFile(path)
if err != nil {
return "", err
}
for _, line := range strings.Split(string(b), "\n")[1:] {
f := strings.Fields(line)
if len(f) >= 8 && f[1] == "00000000" && f[7] == "00000000" {
return f[0], nil
}
}
return "", errors.New("linetest: no default route")
}
+574
View File
@@ -0,0 +1,574 @@
package linetest
import (
"context"
"encoding/binary"
"encoding/json"
"errors"
"io"
"net"
"net/netip"
"os"
"path/filepath"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ── a fake iperf3 server (the server half of the protocol, as `iperf3 --server` drives it) ─────────────────────────
type fakeServer struct {
ln net.Listener
busy bool // answer ACCESS_DENIED
stall bool // accept the cookie, then never answer
recvSeconds float64 // what the server reports as its duration
gotIDs []int // the stream ids the client reported
gotParams map[string]any
mu sync.Mutex
received int64
}
func newFakeServer(t *testing.T) *fakeServer {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
s := &fakeServer{ln: ln, recvSeconds: 1}
t.Cleanup(func() { ln.Close() })
go s.serve()
return s
}
func (s *fakeServer) addr() string { return s.ln.Addr().String() }
func wjson(w io.Writer, v any) {
b, _ := json.Marshal(v)
var h [4]byte
binary.BigEndian.PutUint32(h[:], uint32(len(b)))
w.Write(append(h[:], b...))
}
func (s *fakeServer) serve() {
ctrl, err := s.ln.Accept()
if err != nil {
return
}
defer ctrl.Close()
cookie := make([]byte, cookieSize)
if _, err := io.ReadFull(ctrl, cookie); err != nil {
return
}
if s.busy {
ctrl.Write([]byte{0xff})
return
}
if s.stall {
time.Sleep(5 * time.Second)
return
}
ctrl.Write([]byte{stParamExchange})
var params map[string]any
if readJSON(ctrl, &params) != nil {
return
}
s.gotParams = params
n := int(params["parallel"].(float64))
ctrl.Write([]byte{stCreateStreams})
var wg sync.WaitGroup
var conns []net.Conn
for i := 0; i < n; i++ {
d, err := s.ln.Accept()
if err != nil {
return
}
c := make([]byte, cookieSize)
io.ReadFull(d, c)
conns = append(conns, d)
wg.Add(1)
go func(d net.Conn) {
defer wg.Done()
k, _ := io.Copy(io.Discard, d)
s.mu.Lock()
s.received += k
s.mu.Unlock()
}(d)
}
ctrl.Write([]byte{stTestStart, stTestRunning})
var st [1]byte
if _, err := io.ReadFull(ctrl, st[:]); err != nil || st[0] != stTestEnd {
return
}
ctrl.Write([]byte{stExchangeResults})
var theirs results
if readJSON(ctrl, &theirs) != nil {
return
}
for _, x := range theirs.Streams {
s.gotIDs = append(s.gotIDs, x.ID)
}
// The server counts what it RECEIVED; report a fixed figure so the test can tell it from the client's own count.
wjson(ctrl, results{Streams: []streamResult{{ID: 1, Bytes: 12_500_000, EndTime: s.recvSeconds}}})
ctrl.Write([]byte{stDisplayResults})
io.ReadFull(ctrl, st[:])
for _, d := range conns {
d.Close()
}
wg.Wait()
}
func TestClientSpeaksTheProtocolAndCountsTheServersBytes(t *testing.T) {
s := newFakeServer(t)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
r, err := Client{Streams: 4, Time: 300 * time.Millisecond}.Run(ctx, s.addr())
if err != nil {
t.Fatalf("Run: %v", err)
}
if r.ServerBytes != 12_500_000 || r.Mbps != 100 {
t.Fatalf("result = %+v; want the SERVER's 12.5 MB in 1 s = 100 Mbit/s", r)
}
if want := []int{1, 3, 4, 5}; len(s.gotIDs) != 4 || s.gotIDs[0] != want[0] || s.gotIDs[1] != want[1] || s.gotIDs[3] != want[3] {
t.Fatalf("stream ids sent = %v, want %v (iperf3 numbers its streams 1, 3, 4, 5)", s.gotIDs, want)
}
if s.gotParams["tcp"] != true || s.gotParams["parallel"].(float64) != 4 {
t.Fatalf("params = %v", s.gotParams)
}
}
func TestClientBusyServerIsErrBusy(t *testing.T) {
s := newFakeServer(t)
s.busy = true
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if _, err := (Client{Streams: 4, Time: time.Second}).Run(ctx, s.addr()); !errors.Is(err, ErrBusy) {
t.Fatalf("err = %v, want ErrBusy", err)
}
}
func TestClientHonoursTheDeadlineAgainstAStalledServer(t *testing.T) {
s := newFakeServer(t)
s.stall = true
ctx, cancel := context.WithTimeout(context.Background(), 400*time.Millisecond)
defer cancel()
start := time.Now()
_, err := Client{Streams: 4, Time: time.Second}.Run(ctx, s.addr())
if err == nil {
t.Fatal("a stalled server gave no error")
}
if took := time.Since(start); took > 1500*time.Millisecond {
t.Fatalf("Run took %v against a stalled server; the deadline is 400 ms", took)
}
}
func TestClientRefusesAContextWithoutDeadline(t *testing.T) {
if _, err := (Client{}).Run(context.Background(), "127.0.0.1:1"); err == nil {
t.Fatal("Run without a deadline must refuse (an unbounded test is the thing RunTimeout exists to prevent)")
}
}
// ── the schedule ────────────────────────────────────────────────────────────────────────────────────────────────
func TestEligibleWindowNeverInTheNightChain(t *testing.T) {
loc := time.FixedZone("CET", 3600)
start := time.Date(2026, 10, 19, 0, 0, 0, 0, loc) // a Monday
for m := 0; m < 7*24*60; m++ {
lt := start.Add(time.Duration(m) * time.Minute)
for _, anchorDay := range []bool{true, false} {
if !EligibleAt(lt, 0, anchorDay) {
continue
}
if h := lt.Hour(); h < StartHour || h >= EndHour || (h >= 1 && h <= 6) {
t.Fatalf("eligible at %s — outside 10:00-15:59 or inside the night chain (01:00-06:59)", lt.Format("Mon 15:04"))
}
}
}
}
func TestSlotDelaysTheFirstStartOnTheDueDayOnly(t *testing.T) {
loc := time.UTC
tue := time.Date(2026, 10, 20, 10, 30, 0, 0, loc)
if EligibleAt(tue, 45, true) {
t.Fatal("10:30 with slot 45 must wait until 10:45 on the due day")
}
if !EligibleAt(tue.Add(15*time.Minute), 45, true) {
t.Fatal("10:45 with slot 45 must be eligible")
}
if !EligibleAt(tue.Add(24*time.Hour), 45, false) {
t.Fatal("a catch-up day does not wait for the slot again")
}
if a, b := SlotMinute("demo-hp-bb76ea"), SlotMinute("demo-felhom-8363b5"); a < 0 || a >= SlotSpreadMinutes || b < 0 || b >= SlotSpreadMinutes {
t.Fatalf("slots out of range: %d %d", a, b)
}
}
func TestDueWeeklyFromTuesday(t *testing.T) {
loc := time.UTC
mon := time.Date(2026, 10, 19, 12, 0, 0, 0, loc)
tue := time.Date(2026, 10, 20, 12, 0, 0, 0, loc)
if !Due(State{}, mon) {
t.Fatal("never measured is due")
}
st := State{LastOKAt: time.Date(2026, 10, 14, 11, 0, 0, 0, loc)} // last Wednesday
if Due(st, mon) {
t.Fatal("measured after last Tuesday 10:00 is not due on Monday")
}
if !Due(st, tue) {
t.Fatal("from Tuesday 10:00 a new measurement is due")
}
}
// ── the job ─────────────────────────────────────────────────────────────────────────────────────────────────────
type fakeRunner struct {
mu sync.Mutex
calls int
out []runOut
}
type runOut struct {
mbps float64
err error
}
func (f *fakeRunner) Run(ctx context.Context, addr string) (Result, error) {
f.mu.Lock()
defer f.mu.Unlock()
i := f.calls
f.calls++
if i >= len(f.out) {
return Result{}, errors.New("no more scripted runs")
}
return Result{Mbps: f.out[i].mbps, ServerBytes: 1}, f.out[i].err
}
type fakeGate struct{ busy string }
func (g *fakeGate) TryAcquire(what string) (func(), string, bool) {
if g.busy != "" {
return nil, g.busy, false
}
return func() {}, "", true
}
type rig struct {
t *Tester
run *fakeRunner
gate *fakeGate
now time.Time
tx uint64
tunnel bool
}
// newRig: Tuesday 12:00 UTC (eligible, any slot), tunnel up, 11 quiet samples taken.
func newRig(t *testing.T, outs ...runOut) *rig {
r := &rig{run: &fakeRunner{out: outs}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
r.t.loc = time.UTC
r.t.now = func() time.Time { return r.now }
r.t.sleep = func(context.Context, time.Duration) bool { return true }
r.quietMinutes(QuietMinutes + 1)
return r
}
func (r *rig) minute(bytes uint64) {
r.now = r.now.Add(time.Minute)
r.tx += bytes
r.t.sample()
}
func (r *rig) quietMinutes(n int) {
for i := 0; i < n; i++ {
r.minute(1_000_000) // 1 MB/min = 0.13 Mbit/s
}
}
func TestTickMeasuresTheHighestOfThreeRuns(t *testing.T) {
r := newRig(t, runOut{mbps: 88}, runOut{mbps: 91.5}, runOut{mbps: 90})
r.t.Tick(context.Background())
mbps, _, ok := r.t.Value()
if !ok || mbps != 91.5 || r.run.calls != RunsPerMeasurement {
t.Fatalf("value %.1f ok=%v calls=%d; want the highest of 3 runs = 91.5", mbps, ok, r.run.calls)
}
r.t.Tick(context.Background())
if r.run.calls != RunsPerMeasurement {
t.Fatal("a measured week must not test again")
}
}
func TestBusyServerIsRetriedLaterTheSameDay(t *testing.T) {
r := newRig(t, runOut{err: ErrBusy}, runOut{mbps: 90}, runOut{mbps: 89}, runOut{mbps: 92})
r.t.Tick(context.Background())
if _, _, ok := r.t.Value(); ok || r.run.calls != 1 {
t.Fatalf("busy: ok=%v calls=%d; want no value and the measurement stopped after the busy answer", ok, r.run.calls)
}
if r.t.st.TriesToday != 1 || r.t.st.LastError == "" {
t.Fatalf("busy attempt not recorded: %+v", r.t.st)
}
r.quietMinutes(1)
r.t.Tick(context.Background())
if mbps, _, ok := r.t.Value(); !ok || mbps != 92 {
t.Fatalf("retry later the same day: value %.1f ok=%v", mbps, ok)
}
}
func TestTriesPerDayAreBounded(t *testing.T) {
outs := make([]runOut, 20)
for i := range outs {
outs[i] = runOut{err: ErrBusy}
}
r := newRig(t, outs...)
for i := 0; i < 10; i++ {
r.quietMinutes(1)
r.t.Tick(context.Background())
}
if r.run.calls != MaxTriesPerDay {
t.Fatalf("calls = %d in one day, want at most %d", r.run.calls, MaxTriesPerDay)
}
}
func TestNoTunnelNoTest(t *testing.T) {
r := newRig(t, runOut{mbps: 90})
r.tunnel = false
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested without an operational tunnel")
}
if _, _, ok := r.t.Value(); ok {
t.Fatal("a value without a test")
}
}
func TestBusyUplinkOrHeavyOpWaits(t *testing.T) {
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.minute(60_000_000) // one minute at 8 Mbit/s: the family is uploading
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested while the uplink was busy in the last 10 minutes")
}
r.quietMinutes(QuietMinutes)
r.gate.busy = "backup:felhom-pbs"
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("tested beside a heavy operation")
}
r.gate.busy = ""
r.t.Tick(context.Background())
if r.run.calls == 0 {
t.Fatal("never tested once the uplink was quiet and the gate free")
}
}
func TestNeverTestsInTheNightOrBeforeTuesday(t *testing.T) {
r := newRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
r.quietMinutes(QuietMinutes + 1)
r.t.Tick(context.Background())
r.t.st.LastOKAt = time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
r.now = time.Date(2026, 10, 19, 12, 0, 0, 0, time.UTC) // Monday noon, measured last week
r.quietMinutes(QuietMinutes + 1)
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatalf("tested %d times at night or when not due", r.run.calls)
}
}
func TestStateSurvivesARestartAndReportsIt(t *testing.T) {
r := newRig(t, runOut{mbps: 90.7}, runOut{err: errors.New("reset")}, runOut{mbps: 80})
r.t.Tick(context.Background())
again := New(r.run, nil, r.gate, nil, "host-x", r.t.statePath, nil)
mbps, at, ok := again.Value()
if !ok || mbps != 90.7 || at.IsZero() {
t.Fatalf("after a restart: %.1f %v %v", mbps, at, ok)
}
b, _ := json.Marshal(again.LineTestStatus(context.Background()))
var back hub.LineTestStatus
json.Unmarshal(b, &back)
if back.Mbps != 90.7 || back.LastOKAt == "" || len(back.RunsMbps) != 2 || back.Schedule == "" {
t.Fatalf("report stanza = %s", b)
}
}
// ── the uplink counter ──────────────────────────────────────────────────────────────────────────────────────────
func TestUplinkCountsTheBridgesPhysicalPortsOnly(t *testing.T) {
root := t.TempDir()
w := func(p, s string) {
os.MkdirAll(filepath.Dir(filepath.Join(root, p)), 0o755)
os.WriteFile(filepath.Join(root, p), []byte(s), 0o644)
}
w("proc/net/route", "Iface\tDestination\tGateway\tFlags\tRefCnt\tUse\tMetric\tMask\n"+
"wg-felhom\t0000004D\t00000000\t0001\t0\t0\t0\tFFFFFFFF\n"+
"vmbr0\t00000000\t0100A8C0\t0003\t0\t0\t0\t00000000\n")
w("sys/class/net/vmbr0/statistics/tx_bytes", "5\n")
w("sys/class/net/vmbr0/brif/enp1s0/.keep", "")
w("sys/class/net/vmbr0/brif/veth9201i0/.keep", "")
w("sys/class/net/enp1s0/device/.keep", "")
w("sys/class/net/enp1s0/statistics/tx_bytes", "1000\n")
w("sys/class/net/veth9201i0/statistics/tx_bytes", "777777\n")
got, err := SysUplinkTx(root)()
if err != nil || got != 1000 {
t.Fatalf("uplink tx = %d, %v; want 1000 (the physical port only — not the bridge, not the guest's veth)", got, err)
}
}
func TestQuietNeedsTenFullMinutes(t *testing.T) {
base := time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC)
var s []txSample
for i := 0; i <= QuietMinutes; i++ {
s = append(s, txSample{at: base.Add(time.Duration(i) * time.Minute), tx: uint64(i) * 1_000_000})
}
if q, _ := quiet(s); !q {
t.Fatal("ten quiet minutes must read quiet")
}
if q, _ := quiet(s[1:]); q {
t.Fatal("nine minutes of samples must not read quiet")
}
gap := append([]txSample(nil), s...)
gap[5].at = gap[5].at.Add(-10 * time.Minute)
if q, _ := quiet(gap); q {
t.Fatal("a gap in the samples must not read quiet")
}
}
// ── a test on request (R-1250): a torrent app that seeds all day ───────────────────────────────────────────────────
// seedingRig: due and eligible, but every minute of the last ten sent at the line's speed — the weekly Tick never runs.
func seedingRig(t *testing.T, outs ...runOut) *rig {
r := newRig(t, outs...)
for i := 0; i < QuietMinutes+1; i++ {
r.minute(600_000_000) // 80 Mbit/s: a seed at the line's speed
}
return r
}
func TestSeedingBoxIsWaitingForQuietAndRunNowMeasuresIt(t *testing.T) {
r := seedingRig(t, runOut{mbps: 88}, runOut{mbps: 92}, runOut{mbps: 90})
r.t.Tick(context.Background())
if r.run.calls != 0 {
t.Fatal("the weekly Tick tested a seeding box")
}
if !r.t.WaitingForQuiet() {
t.Fatal("a due, eligible, seeding box must report waiting_for_quiet")
}
var slept []time.Duration
r.t.sleep = func(_ context.Context, d time.Duration) bool { slept = append(slept, d); return true }
mbps, runs, err := r.t.RunNow(context.Background()) // the controller holds the app: the counter stands still
if err != nil || mbps != 92 || len(runs) != RunsPerMeasurement || r.run.calls != RunsPerMeasurement {
t.Fatalf("RunNow = %.1f %v %v, calls %d; want the best of 3 = 92", mbps, runs, err, r.run.calls)
}
want := []time.Duration{RequestSettle, RequestQuietWindow, RequestRunGap, RequestRunGap}
if len(slept) != len(want) {
t.Fatalf("slept %v, want %v", slept, want)
}
var total time.Duration
for i := range want {
if slept[i] != want[i] {
t.Fatalf("slept %v, want %v", slept, want)
}
total += slept[i]
}
if total+RunsPerMeasurement*TestTime > 60*time.Second {
t.Fatalf("a request holds the app %v + runs; it must end inside 60 s", total)
}
if v, _, ok := r.t.Value(); !ok || v != 92 {
t.Fatalf("the requested measurement was not stored: %.1f %v", v, ok)
}
if r.t.WaitingForQuiet() {
t.Fatal("measured this week: no longer waiting")
}
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-due" {
t.Fatalf("a second request this week: %v, want not-due", err)
}
}
func refusalCode(err error) string {
var ref *Refusal
if errors.As(err, &ref) {
return ref.Code
}
return ""
}
// The held app is not the only uploader: a request still refuses to measure a busy uplink.
func TestRunNowRefusesAnUplinkThatIsStillBusy(t *testing.T) {
r := seedingRig(t, runOut{mbps: 90}, runOut{mbps: 90}, runOut{mbps: 90})
r.t.uplink = func() (uint64, error) { r.tx += 20_000_000; return r.tx, nil } // 20 MB per 10 s = 16 Mbit/s
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "not-quiet" || r.run.calls != 0 {
t.Fatalf("RunNow on a busy uplink: %v, calls %d; want not-quiet and no run", err, r.run.calls)
}
}
// Every other rule of the weekly test holds for a request.
func TestRunNowKeepsTheWeeklyRules(t *testing.T) {
r := seedingRig(t)
r.gate.busy = "backup:felhom-pbs"
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "heavy-op" {
t.Fatalf("beside a backup: %v, want heavy-op", err)
}
r.gate.busy = ""
r.tunnel = false
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "no-tunnel" {
t.Fatalf("no tunnel: %v, want no-tunnel", err)
}
if r.t.WaitingForQuiet() {
t.Fatal("no tunnel: a request could not run, so it must not be invited")
}
r.tunnel = true
r.now = time.Date(2026, 10, 20, 3, 0, 0, 0, time.UTC) // Tuesday 03:00 — the night chain
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "outside-window" {
t.Fatalf("at night: %v, want outside-window", err)
}
if r.run.calls != 0 {
t.Fatalf("%d runs on refused requests", r.run.calls)
}
}
// A failed request counts as a try, so a controller cannot request more than MaxTriesPerDay tests a day.
func TestRunNowFailuresSpendTheDaysTries(t *testing.T) {
var outs []runOut
for i := 0; i < MaxTriesPerDay*RunsPerMeasurement; i++ {
outs = append(outs, runOut{err: errors.New("connection refused")})
}
r := seedingRig(t, outs...)
for i := 0; i < MaxTriesPerDay; i++ {
if _, _, err := r.t.RunNow(context.Background()); err == nil || refusalCode(err) != "" {
t.Fatalf("try %d: %v, want a failed measurement", i+1, err)
}
}
if _, _, err := r.t.RunNow(context.Background()); refusalCode(err) != "tries-used" {
t.Fatalf("after %d failures: %v, want tries-used", MaxTriesPerDay, err)
}
}
// A quiet box tests itself on the weekly Tick: it must not invite the controller to hold its app.
func TestQuietBoxIsNotWaiting(t *testing.T) {
r := newRig(t)
if r.t.WaitingForQuiet() {
t.Fatal("ten quiet minutes: the weekly Tick runs; waiting_for_quiet must be false")
}
}
// Live proof 2026-10-24 (demo-hp): a freshly started tester answered waiting_for_quiet before it had watched a single
// minute, so a quiet box's torrent app would be held for nothing. Waiting needs the full window, seen busy.
func TestFreshTesterIsNotWaiting(t *testing.T) {
r := &rig{run: &fakeRunner{}, gate: &fakeGate{}, now: time.Date(2026, 10, 20, 12, 0, 0, 0, time.UTC), tunnel: true}
r.t = New(r.run, func() (netip.Addr, bool) { return netip.MustParseAddr("10.77.0.1"), r.tunnel }, r.gate,
func() (uint64, error) { return r.tx, nil }, "host-x", filepath.Join(t.TempDir(), "line-test.json"), nil)
r.t.loc = time.UTC
r.t.now = func() time.Time { return r.now }
if r.t.WaitingForQuiet() {
t.Fatal("no minute watched yet: waiting_for_quiet must be false")
}
for i := 0; i < QuietMinutes; i++ {
r.minute(600_000_000)
}
if r.t.WaitingForQuiet() {
t.Fatal("10 samples are 9 minutes: still not a full window")
}
r.minute(600_000_000)
if !r.t.WaitingForQuiet() {
t.Fatal("a full busy window: waiting_for_quiet must be true")
}
}
+165 -12
View File
@@ -3,6 +3,7 @@ package localapi
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
@@ -83,6 +84,20 @@ const (
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside // controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
// the parked marker. // the parked marker.
controllerSlowCounterFile = "controller-restarts-24h.json" controllerSlowCounterFile = "controller-restarts-24h.json"
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
// the hub (controller_unresponsive). Not counted during a swap, a parked guest, a locked guest or a vzdump.
controllerUnresponsiveConfirm = 6
// controllerUnresponsiveRestart (decided 2026-10-30 from the measurement): at this many consecutive unhealthy sweeps
// (5 minutes after Docker's „unhealthy", ~6.5 minutes after the first failed probe; 2 minutes after the alarm) the
// agent restarts the controller — the not-running restart's guards and brake, plus the controller's busy file. The
// numbers: Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, 4 733 probes through the night chain (a
// PBS and a local vzdump, the DB dumps, tier-2, the off-site push, a restore test, two controller updates): no failed
// probe, slowest 303 ms against the 5 s timeout. A busy night never came near even ONE failed probe; ten sweeps is
// the margin. Pinned by TestSupervisor_UnresponsiveIsAlarmedThenRestarted.
controllerUnresponsiveRestart = 10
) )
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power // controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
@@ -103,6 +118,18 @@ type controllerSupState struct {
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too). // on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
restarts24h []time.Time restarts24h []time.Time
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
// opBrakeUntil (2026-10-29): an operator restart refused by the brake refuses every press until then (30 min).
opBrakeUntil time.Time
// R-1344: consecutive running-but-unhealthy sweeps, and the current episode (since kept after it ends).
unhealthySeen int
unresponsive bool
unresponsiveSince time.Time
// unresponsiveRestartAt: the last restart for not answering (kept; the hub keys on it MOVING). unrespRefusal: the
// last reason the restart was refused, so a refusal is logged once per change, not every sweep.
unresponsiveRestartAt time.Time
unrespRefusal string
} }
type controllerSupervisor struct { type controllerSupervisor struct {
@@ -238,6 +265,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
if !s.provisionedGuest(g.VMID) { if !s.provisionedGuest(g.VMID) {
continue continue
} }
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
s.resetNotRunning(g.VMID)
continue
}
evaluated++ evaluated++
if !s.superviseOneController(ctx, g.VMID, g.Status) { if !s.superviseOneController(ctx, g.VMID, g.Status) {
down++ down++
@@ -293,6 +324,7 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
if running { if running {
st.notRunningSeen = 0 st.notRunningSeen = 0
s.ctrlSup.mu.Unlock() s.ctrlSup.mu.Unlock()
s.observeControllerHealth(ctx, vmid, parked)
return true return true
} }
st.notRunningSeen++ st.notRunningSeen++
@@ -333,9 +365,27 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
return false return false
} }
// Backoff. if !s.supervisorBrakeAllows(vmid, now) {
return false
}
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
"vmid", vmid, "status", status, "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
s.recordSupervisorRestart(vmid, now, reason, nil)
return false
}
// supervisorBrakeAllows is the brake every supervisor restart passes (the not-running one and R-1344's): a crash-loop
// pause in force refuses; controllerCrashloopMax restarts inside controllerCrashloopWindow start one (logged, carried by
// the stanza, mailed by the hub as controller_crashloop) and refuse.
func (s *Server) supervisorBrakeAllows(vmid int, now time.Time) bool {
s.ctrlSup.mu.Lock() s.ctrlSup.mu.Lock()
st = s.supState(vmid) st := s.supState(vmid)
if !st.crashloopSince.IsZero() { if !st.crashloopSince.IsZero() {
if now.Sub(st.crashloopSince) < controllerCrashloopPause { if now.Sub(st.crashloopSince) < controllerCrashloopPause {
s.ctrlSup.mu.Unlock() s.ctrlSup.mu.Unlock()
@@ -358,22 +408,23 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
return false return false
} }
s.ctrlSup.mu.Unlock() s.ctrlSup.mu.Unlock()
return true
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
"vmid", vmid, "status", status, "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
} }
// recordSupervisorRestart puts a restart the supervisor performed on the record: the brake's window, the totals, the
// reason, and R-539's slow counter (persisted; raised at most once per 24 hours — the hub mails on the raise MOVING).
// mark, when set, runs under the same lock (R-1344 stamps its own record there).
func (s *Server) recordSupervisorRestart(vmid int, now time.Time, reason string, mark func(st *controllerSupState)) {
s.ctrlSup.mu.Lock() s.ctrlSup.mu.Lock()
st = s.supState(vmid) st := s.supState(vmid)
st.restarts = append(st.restarts, now) st.restarts = append(st.restarts, now)
st.restartsTotal++ st.restartsTotal++
st.lastRestartAt = now st.lastRestartAt = now
st.lastReason = reason st.lastReason = reason
st.notRunningSeen = 0 st.notRunningSeen = 0
// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING. if mark != nil {
mark(st)
}
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now) st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
n24 := len(st.restarts24h) n24 := len(st.restarts24h)
raised := false raised := false
@@ -389,7 +440,101 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop", s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax) "vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
} }
return false }
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
// the swap's `docker inspect -f` grant. Anything but „unhealthy" — healthy, starting, no health check, a read that
// fails — resets the count; so do a swap, a parked guest, a locked guest and a vzdump (each is someone else's moment).
func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked bool) {
s.swapMu.Lock()
swapping := s.swapInFlight[vmid]
s.swapMu.Unlock()
unhealthy := false
if !parked && !swapping {
if out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer); err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
unhealthy = len(f) == 2 && f[0] == "true" && f[1] == "unhealthy"
}
}
if unhealthy {
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil || lock != "" {
unhealthy = false
} else if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
unhealthy = false
}
}
now := s.clock()
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if !unhealthy {
was := st.unresponsive
st.unhealthySeen, st.unresponsive = 0, false
s.ctrlSup.mu.Unlock()
if was {
s.logger.Info("controller-supervisor: the controller answers its health check again", "vmid", vmid)
}
return
}
st.unhealthySeen++
seen := st.unhealthySeen
start := seen >= controllerUnresponsiveConfirm && !st.unresponsive
if start {
st.unresponsive, st.unresponsiveSince = true, now
}
since := st.unresponsiveSince
s.ctrlSup.mu.Unlock()
if start {
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so",
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String(),
"restart_after_sweeps", controllerUnresponsiveRestart)
}
if seen >= controllerUnresponsiveRestart {
s.restartUnresponsive(ctx, vmid, seen, since, now)
}
}
// restartUnresponsive is R-1344's restart (decided 2026-10-30): the controller's busy file (fresh = a backup, a
// restore, an update or an export is running; unreadable = cannot be ruled out) refuses, then the brake every
// supervisor restart passes. The swap, the parked marker, the lock and the vzdump never reach here — they reset the
// count. A refusal keeps the alarm and is logged once per reason; the next sweep asks again.
func (s *Server) restartUnresponsive(ctx context.Context, vmid, seen int, since, now time.Time) {
refuse := func(why string) {
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
changed := st.unrespRefusal != why
st.unrespRefusal = why
s.ctrlSup.mu.Unlock()
if changed {
s.logger.Warn("controller-supervisor: the controller does not answer, but the restart is refused — the alarm stays",
"vmid", vmid, "why", why, "unhealthy_sweeps", seen)
}
}
busy, what, err := s.controllerBusy(ctx, vmid)
if err != nil {
refuse("the controller's busy file cannot be read: " + err.Error())
return
}
if busy {
refuse("the controller is busy: " + what)
return
}
if !s.supervisorBrakeAllows(vmid, now) {
refuse("the restart brake")
return
}
reason := fmt.Sprintf("the controller ran but did not answer its health check for %d sweeps (since %s)",
seen, since.UTC().Format(time.RFC3339))
s.logger.Warn("controller-supervisor: the controller does not answer — restarting the bootstrap unit",
"vmid", vmid, "unhealthy_sweeps", seen, "since", since.UTC().Format(time.RFC3339), "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
s.recordSupervisorRestart(vmid, now, reason, func(st *controllerSupState) {
st.unresponsiveRestartAt = now
st.unhealthySeen = 0
st.unrespRefusal = ""
})
} }
func (s *Server) resetNotRunning(vmid int) { func (s *Server) resetNotRunning(vmid int) {
@@ -446,6 +591,14 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339) g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
} }
g.Unresponsive = st.unresponsive
if !st.unresponsiveSince.IsZero() {
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
}
g.UnresponsiveRestartAfterS = (controllerUnresponsiveRestart - controllerUnresponsiveConfirm) * int(controllerSupervisorInterval/time.Second)
if !st.unresponsiveRestartAt.IsZero() {
g.UnresponsiveRestartAt = st.unresponsiveRestartAt.UTC().Format(time.RFC3339)
}
out.Guests = append(out.Guests, g) out.Guests = append(out.Guests, g)
} }
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID }) sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
+76 -2
View File
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of // host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
// a pending package is refused (503): a code must never be handed out for a package the hub lacks. // a pending package is refused (503): a code must never be handed out for a package the hub lacks.
Upload func(ctx context.Context, pkg []byte) error Upload func(ctx context.Context, pkg []byte) error
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
} }
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole // escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
// ceremony incl. its upload was measured at ~2.4 s). // ceremony incl. its upload was measured at ~2.4 s).
const escrowUploadTimeout = 30 * time.Second const escrowUploadTimeout = 30 * time.Second
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
const escrowHoldTimeout = 15 * time.Second
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
// or late may have been stored all the same.
type uploadOutcome int
const (
uploadStored uploadOutcome = iota // the hub holds the new package
uploadNotStored // the hub answered and holds another package: the previous code still works
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
)
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
// still works", which is false when the lost answer came after the store.
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
if s.escrowCeremony.HubHolds == nil {
return uploadUnknown
}
base := s.baseCtx
if base == nil {
base = context.Background()
}
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
defer cancel()
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
switch {
case err != nil:
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
return uploadUnknown
case held:
return uploadStored
default:
return uploadNotStored
}
}
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until // ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned // parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
// spike-shaped output; production is runCeremonySubprocess. // spike-shaped output; production is runCeremonySubprocess.
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout) uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg) err := s.escrowCeremony.Upload(uctx, pkg)
cancel() cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.escrowClaiming = false s.escrowClaiming = false
if err != nil { switch outcome {
case uploadNotStored:
s.escrowMu.Unlock() s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)", s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds()) "vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again") writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
} }
s.escrowPending = nil s.escrowPending = nil
if s.escrowJob != nil && s.escrowJob.JobID == jobID { if s.escrowJob != nil && s.escrowJob.JobID == jobID {
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout) uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
err := s.escrowCeremony.Upload(uctx, pkg) err := s.escrowCeremony.Upload(uctx, pkg)
cancel() cancel()
outcome := uploadStored
if err != nil {
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
}
s.escrowMu.Lock() s.escrowMu.Lock()
s.escrowClaiming = false s.escrowClaiming = false
if err != nil { switch outcome {
case uploadNotStored:
s.escrowMu.Unlock() s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)", s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds()) "vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again") writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
return return
case uploadUnknown:
s.escrowMu.Unlock()
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
return
}
if err != nil {
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
"vmid", vmid, "job_id", jobID, "err", err)
} }
s.escrowPending = nil s.escrowPending = nil
s.wipeEscrowConfirmLocked() s.wipeEscrowConfirmLocked()
+19
View File
@@ -42,6 +42,10 @@ type fakeEscrowHub struct {
current []byte current []byte
uploads int uploads int
err error // non-nil → the upload fails, nothing stored err error // non-nil → the upload fails, nothing stored
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
// holdsErr → the hub cannot be asked which package it holds.
lost bool
holdsErr error
gate chan struct{} // non-nil → the upload blocks until closed gate chan struct{} // non-nil → the upload blocks until closed
entered chan struct{} // non-nil → closed when an upload starts entered chan struct{} // non-nil → closed when an upload starts
} }
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
} }
f.current = append([]byte(nil), pkg...) f.current = append([]byte(nil), pkg...)
f.uploads++ f.uploads++
if f.lost {
return context.DeadlineExceeded
}
return nil return nil
} }
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
// blob the hub serves back).
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
f.mu.Lock()
defer f.mu.Unlock()
if f.holdsErr != nil {
return false, f.holdsErr
}
return string(f.current) == string(pkg), nil
}
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question. // opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
func (f *fakeEscrowHub) opensWith(r string) bool { func (f *fakeEscrowHub) opensWith(r string) bool {
f.mu.Lock() f.mu.Lock()
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
HubConfigured: true, HubConfigured: true,
DRConfigured: func() bool { return true }, DRConfigured: func() bool { return true },
Upload: hub.upload, Upload: hub.upload,
HubHolds: hub.holds,
} }
srv.ceremonyRun = run srv.ceremonyRun = run
srv.escrowSudoCheck = func(context.Context) error { return nil } srv.escrowSudoCheck = func(context.Context) error { return nil }
+98
View File
@@ -0,0 +1,98 @@
package localapi
import (
"errors"
"net/http"
"strings"
"testing"
)
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
// - the hub holds the new package → success (the code is handed out / the confirm completes);
// - the hub holds another package → the old sentence, now true;
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
// hub holds the new package → FAILS.
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if !hub.opensWith(firstR) {
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
}
if code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("the household was told the previous code still works: %s", body)
}
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
t.Fatalf("the job must read uploaded: %s", st)
}
}
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
code, body := claimCode(t, h)
if code != http.StatusGatewayTimeout {
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
}
if strings.Contains(body, "previous code still works") {
t.Fatalf("an unknown outcome promised the previous code: %s", body)
}
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
t.Fatalf("the code must stay claimable: %s", st)
}
// The hub answers again: the retry finds the new package already stored and hands the code out.
hub.mu.Lock()
hub.holdsErr = nil
hub.mu.Unlock()
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
t.Fatalf("retry: %d %s", code, body)
}
}
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
}
}
// The same at the typed-back confirm (R-1151).
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
hub := &fakeEscrowHub{lost: true}
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
h := srv.Handler()
startAndWait(t, srv, h)
c := claimDeferred(t, h)
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
}
hub.mu.Lock()
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
hub.mu.Unlock()
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
h2 := srv2.Handler()
startAndWait(t, srv2, h2)
c2 := claimDeferred(t, h2)
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
}
}
+102
View File
@@ -0,0 +1,102 @@
package localapi
import (
"bufio"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
)
// R-528 — the OOM signal from the host side.
//
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
//
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
// cgroup folder (a VM guest, or a container that is not running).
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
if root == "" {
root = "/sys/fs/cgroup"
}
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
return nil, false, nil
}
out := map[string]int64{}
for _, pattern := range []string{
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
filepath.Join(base, "docker", "*", "memory.events"),
} {
files, err := filepath.Glob(pattern)
if err != nil {
return nil, true, err
}
for _, f := range files {
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
if !isContainerID(id) {
continue
}
n, err := readOOMKill(f)
if err != nil {
continue // a container that stopped between the glob and the read
}
out[id] = n
}
}
return out, true, nil
}
func isContainerID(s string) bool {
if len(s) != 64 {
return false
}
for _, c := range s {
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
return false
}
}
return true
}
func readOOMKill(path string) (int64, error) {
f, err := os.Open(path)
if err != nil {
return 0, err
}
defer f.Close()
sc := bufio.NewScanner(f)
for sc.Scan() {
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
if ok && k == "oom_kill" {
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
}
}
if err := sc.Err(); err != nil {
return 0, err
}
return 0, fmt.Errorf("no oom_kill line in %s", path)
}
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
if err != nil {
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
return
}
if kills == nil {
kills = map[string]int64{}
}
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
}
+101
View File
@@ -0,0 +1,101 @@
package localapi
import (
"encoding/json"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
)
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
// serves the asking guest's counters, and only that guest's.
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
t.Helper()
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
t.Fatal(err)
}
}
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
t.Fatal(err)
}
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
if err := os.MkdirAll(dfDir, 0o755); err != nil {
t.Fatal(err)
}
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
got, avail, err := readGuestOOMKills(root, 100)
if err != nil || !avail {
t.Fatalf("read: avail=%v err=%v", avail, err)
}
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
if len(got) != len(want) {
t.Fatalf("got %v, want %v", got, want)
}
for k, v := range want {
if got[k] != v {
t.Fatalf("got %v, want %v", got, want)
}
}
if _, leaked := got[idOther]; leaked {
t.Fatal("another guest's container leaked into this guest's answer")
}
// A guest with no cgroup folder (a VM, or not running): not available, no error.
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
}
}
// Through the route, with the guest's own token.
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
root := t.TempDir()
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.cgroupRoot = root
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
if w.Code != http.StatusOK {
t.Fatalf("route: %d %s", w.Code, w.Body.String())
}
var env struct {
Data struct {
VMID int `json:"vmid"`
Available bool `json:"available"`
Containers map[string]int64 `json:"containers"`
} `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
}
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
t.Fatalf("no token: %d", w.Code)
}
}
+5
View File
@@ -152,6 +152,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
return return
} }
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
// would fail the reboot's own start half).
if s.rebooting(g.VMID) {
return
}
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID) lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
if err != nil { if err != nil {
+89
View File
@@ -0,0 +1,89 @@
package localapi
import (
"context"
"errors"
"net/http"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
)
// LineTestReader is the R-1247 weekly upload line test (internal/linetest.Tester) as the local API reads it.
type LineTestReader interface {
Value() (mbps float64, at time.Time, ok bool)
}
// LineTestRequester is the R-1250 half: a due test the uplink never lets start, and a test on request. The Tester
// satisfies it; a reader without it answers waiting_for_quiet:false and 503 on the POST.
type LineTestRequester interface {
WaitingForQuiet() bool
RunNow(ctx context.Context) (mbps float64, runs []float64, err error)
}
// LineTestResponse is GET /host/line-test. Present=false: never measured, or the job is not wired (no tunnel, disabled,
// or an agent before the route answers 404 — the controller reads all three as „unknown").
type LineTestResponse struct {
Present bool `json:"present"`
Mbps float64 `json:"mbps,omitempty"`
MeasuredAt string `json:"measured_at,omitempty"` // RFC3339 UTC
Method string `json:"method,omitempty"` // "iperf3-tunnel": a 10-second upload to ep0 inside the tunnel
// WaitingForQuiet: a measurement is due and could start now, but the uplink has not been quiet for 10 minutes
// (R-1250). The controller with a seeding torrent app then holds its limit and POSTs /host/line-test/run.
WaitingForQuiet bool `json:"waiting_for_quiet,omitempty"`
}
// LineTestRunResponse is POST /host/line-test/run. OK=false with Refused: nothing ran (Refused is linetest.Refusal's
// code, or "server-busy"); OK=false without it: the runs failed.
type LineTestRunResponse struct {
OK bool `json:"ok"`
Mbps float64 `json:"mbps,omitempty"`
RunsMbps []float64 `json:"runs_mbps,omitempty"`
Refused string `json:"refused,omitempty"`
Error string `json:"error,omitempty"`
}
// SetLineTest wires the weekly line test (main.go builds it after the local API server).
func (s *Server) SetLineTest(r LineTestReader) { s.lineTest = r }
func (s *Server) handleLineTest(w http.ResponseWriter, r *http.Request, vmid int) {
if s.lineTest == nil {
writeOK(w, LineTestResponse{Present: false})
return
}
var waiting bool
if q, ok := s.lineTest.(LineTestRequester); ok {
waiting = q.WaitingForQuiet()
}
mbps, at, ok := s.lineTest.Value()
if !ok {
writeOK(w, LineTestResponse{Present: false, WaitingForQuiet: waiting})
return
}
s.logger.Debug("local-api: /host/line-test served", "vmid", vmid, "mbps", mbps, "measured_at", at.UTC().Format(time.RFC3339), "waiting_for_quiet", waiting)
writeOK(w, LineTestResponse{Present: true, Mbps: mbps, MeasuredAt: at.UTC().Format(time.RFC3339), Method: "iperf3-tunnel", WaitingForQuiet: waiting})
}
// handleLineTestRun is POST /host/line-test/run (R-1250): a test on request, synchronous (about 55 s). The answer is
// always 200 with the outcome in the body, except 503 when no line test is wired.
func (s *Server) handleLineTestRun(w http.ResponseWriter, r *http.Request, vmid int) {
q, ok := s.lineTest.(LineTestRequester)
if s.lineTest == nil || !ok {
writeErr(w, http.StatusServiceUnavailable, "line test not configured")
return
}
s.logger.Info("local-api: line test requested", "vmid", vmid)
mbps, runs, err := q.RunNow(r.Context())
var ref *linetest.Refusal
switch {
case err == nil:
writeOK(w, LineTestRunResponse{OK: true, Mbps: mbps, RunsMbps: runs})
case errors.As(err, &ref):
s.logger.Info("local-api: line test request refused", "vmid", vmid, "reason", ref.Code, "detail", ref.Msg)
writeOK(w, LineTestRunResponse{Refused: ref.Code, Error: ref.Msg})
case errors.Is(err, linetest.ErrBusy):
writeOK(w, LineTestRunResponse{Refused: "server-busy", Error: err.Error()})
default:
writeOK(w, LineTestRunResponse{Error: err.Error()})
}
}
+116
View File
@@ -0,0 +1,116 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"log/slog"
"net/http/httptest"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/linetest"
)
type fixedLine struct {
mbps float64
ok bool
}
func (f fixedLine) Value() (float64, time.Time, bool) {
return f.mbps, time.Date(2026, 10, 24, 6, 31, 0, 0, time.UTC), f.ok
}
func lineTestBody(t *testing.T, s *Server) LineTestResponse {
t.Helper()
rec := httptest.NewRecorder()
s.handleLineTest(rec, httptest.NewRequest("GET", "/host/line-test", nil), 9201)
var env struct {
OK bool `json:"ok"`
Data LineTestResponse `json:"data"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
t.Fatalf("decode %q: %v", rec.Body.String(), err)
}
return env.Data
}
// R-1247: the controller sets qBittorrent's limit only from a PRESENT value; never measured and not wired must both
// read present:false, never a zero line.
func TestLineTestRoute(t *testing.T) {
s := &Server{logger: slog.Default()}
if got := lineTestBody(t, s); got.Present {
t.Fatalf("not wired: %+v, want present:false", got)
}
s.SetLineTest(fixedLine{ok: false})
if got := lineTestBody(t, s); got.Present || got.Mbps != 0 {
t.Fatalf("never measured: %+v, want present:false", got)
}
s.SetLineTest(fixedLine{mbps: 90.7, ok: true})
got := lineTestBody(t, s)
if !got.Present || got.Mbps != 90.7 || got.Method != "iperf3-tunnel" || !strings.HasPrefix(got.MeasuredAt, "2026-10-24T06:31") {
t.Fatalf("measured: %+v", got)
}
}
type requestableLine struct {
fixedLine
waiting bool
mbps float64
err error
calls int
}
func (r *requestableLine) WaitingForQuiet() bool { return r.waiting }
func (r *requestableLine) RunNow(context.Context) (float64, []float64, error) {
r.calls++
if r.err != nil {
return 0, nil, r.err
}
return r.mbps, []float64{r.mbps - 1, r.mbps}, nil
}
func lineTestRun(t *testing.T, s *Server) (int, LineTestRunResponse) {
t.Helper()
rec := httptest.NewRecorder()
s.handleLineTestRun(rec, httptest.NewRequest("POST", "/host/line-test/run", nil), 9201)
var env struct {
OK bool `json:"ok"`
Data LineTestRunResponse `json:"data"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &env)
return rec.Code, env.Data
}
// R-1250: the controller acts only on waiting_for_quiet, and reads a refusal by its code — never as a measurement.
func TestLineTestRunRoute(t *testing.T) {
s := &Server{logger: slog.Default()}
if code, _ := lineTestRun(t, s); code != 503 {
t.Fatalf("not wired: HTTP %d, want 503", code)
}
s.SetLineTest(fixedLine{ok: true, mbps: 90})
if code, _ := lineTestRun(t, s); code != 503 {
t.Fatalf("a reader without the request half: HTTP %d, want 503", code)
}
l := &requestableLine{fixedLine: fixedLine{ok: false}, waiting: true, mbps: 93.5}
s.SetLineTest(l)
if got := lineTestBody(t, s); got.Present || !got.WaitingForQuiet {
t.Fatalf("GET while waiting: %+v, want present:false waiting_for_quiet:true", got)
}
if code, got := lineTestRun(t, s); code != 200 || !got.OK || got.Mbps != 93.5 || len(got.RunsMbps) != 2 {
t.Fatalf("a measured request: HTTP %d %+v", code, got)
}
l.err = &linetest.Refusal{Code: "not-quiet", Msg: "the uplink sent 16.0 Mbit/s while held"}
if _, got := lineTestRun(t, s); got.OK || got.Refused != "not-quiet" || got.Mbps != 0 {
t.Fatalf("a refusal: %+v, want ok:false refused:not-quiet", got)
}
l.err = linetest.ErrBusy
if _, got := lineTestRun(t, s); got.OK || got.Refused != "server-busy" {
t.Fatalf("server busy: %+v, want refused:server-busy", got)
}
l.err = errors.New("dial tcp 10.77.0.1:5201: i/o timeout")
if _, got := lineTestRun(t, s); got.OK || got.Refused != "" || got.Error == "" {
t.Fatalf("a failed run: %+v, want ok:false, no refusal, the error", got)
}
}
+258
View File
@@ -0,0 +1,258 @@
package localapi
import (
"context"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// The operator's actions for THIS agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).
//
// THE DOOR IS THE HOST REPORT'S REPLY, the controller's shape: the operator presses a button on the hub's host page, the
// hub stores a row and pokes the box (the poke's out-of-cycle report gets the reply in seconds; without the tunnel the
// 15-minute cycle carries it), the reply lists `operator_actions: [{id, action, arg}]`, this agent acts ONCE per id, and
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
// The result is re-sent until the reply stops listing the id.
//
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
//
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
// the guest over the existing grants, with the supervisor's guards — not during a controller swap, not parked, not on a
// stopped, locked or vzdump-busy guest, not when the guest list or the vzdump state cannot be read — and the thrash brake
// counts these restarts too: 3 restarts (the supervisor's or the operator's) inside 15 minutes refuse the press, and
// presses stay refused for 30 minutes. It works with the controller stopped or stuck: the agent does it, not the
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
const (
// OpRestartController is the agent's first operator verb.
OpRestartController = "restart_controller"
opOutcomeDone = "done"
opOutcomeRefused = "refused"
opOutcomeFailed = "failed"
// opBrakeRefusal is how long presses stay refused after the brake refused one.
opBrakeRefusal = 30 * time.Minute
// opForgetAfter drops a started id the reply no longer lists.
opForgetAfter = 24 * time.Hour
defaultOpHealthPoll = 2 * time.Second
// defaultOpHealthWait covers the bootstrap's pull-less rm/run (seconds), Docker's start period (10 s) and its first
// probe (30 s interval) with room: the controller's own HEALTHCHECK is the answer.
defaultOpHealthWait = 3 * time.Minute
// opHealthTemplate reads running + health over the swap's `docker inspect -f` grant.
opHealthTemplate = "{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}"
)
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
// OperatorActionNames returns the closed list (sorted copy).
func OperatorActionNames() []string {
out := append([]string(nil), operatorActionNames...)
sort.Strings(out)
return out
}
type opActionState struct {
mu sync.Mutex
started map[int64]time.Time
results map[int64]hub.OperatorActionResult
}
// SetOperatorActionTrigger wires the report loop's out-of-band trigger (the result goes at once, not in 15 minutes).
func (s *Server) SetOperatorActionTrigger(f func()) { s.opTrigger = f }
// OnEnvelope is the report loop's hook (hub.EnvelopeObserver): it starts each newly listed action once and forgets
// the ids the reply no longer lists (the hub has their results).
func (s *Server) OnEnvelope(_ context.Context, env *hub.ControlEnvelope) {
if env == nil {
return
}
listed := map[int64]bool{}
var start []hub.OperatorAction
now := time.Now()
s.opActs.mu.Lock()
if s.opActs.started == nil {
s.opActs.started = map[int64]time.Time{}
s.opActs.results = map[int64]hub.OperatorActionResult{}
}
for _, a := range env.OperatorActions {
listed[a.ID] = true
if _, ok := s.opActs.started[a.ID]; ok {
continue
}
s.opActs.started[a.ID] = now
start = append(start, a)
}
for id, at := range s.opActs.started {
if listed[id] {
continue
}
if _, answered := s.opActs.results[id]; answered || now.Sub(at) > opForgetAfter {
delete(s.opActs.started, id)
delete(s.opActs.results, id)
}
}
s.opActs.mu.Unlock()
for _, a := range start {
go s.runOperatorAction(a) // a restart waits for the health answer — never block the report loop
}
}
// OperatorActionResults is the report's answer list, oldest id first.
func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
s.opActs.mu.Lock()
defer s.opActs.mu.Unlock()
out := make([]hub.OperatorActionResult, 0, len(s.opActs.results))
for _, r := range s.opActs.results {
out = append(out, r)
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out
}
func (s *Server) runOperatorAction(a hub.OperatorAction) {
limit := s.opWait() + time.Minute
if a.Action == OpRebootGuest {
limit += opRebootTimeout
}
ctx, cancel := context.WithTimeout(context.Background(), limit)
defer cancel()
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
switch a.Action {
case OpRestartController:
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
case OpRebootGuest:
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
}
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
"outcome", outcome, "message", msg)
s.opActs.mu.Lock()
if _, still := s.opActs.started[a.ID]; still {
s.opActs.results[a.ID] = hub.OperatorActionResult{ID: a.ID, Outcome: outcome, Message: msg}
}
s.opActs.mu.Unlock()
if s.opTrigger != nil {
s.opTrigger()
}
}
func (s *Server) opPoll() time.Duration {
if s.opHealthPoll > 0 {
return s.opHealthPoll
}
return defaultOpHealthPoll
}
func (s *Server) opWait() time.Duration {
if s.opHealthWait > 0 {
return s.opHealthWait
}
return defaultOpHealthWait
}
// operatorRestartController is the press: the supervisor's guards, the brake, the restart, then the wait for the
// controller's healthy answer. Returns the outcome and the sentence the host page shows.
func (s *Server) operatorRestartController(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
if s.staleLock == nil || s.guestExec == nil {
return opOutcomeRefused, "refused: the controller supervisor is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
s.swapMu.Lock()
swapping := s.swapInFlight[vmid]
s.swapMu.Unlock()
if swapping {
return opOutcomeRefused, "refused: a controller swap (update) is in flight — it owns the controller"
}
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
return opOutcomeRefused, "refused: a whole-guest backup (vzdump) may be in flight for the guest"
}
// The brake: every restart counts — the supervisor's and the operator's.
now := s.clock()
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
reason := fmt.Sprintf("the operator via the hub (action #%d)", id)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = reason
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.logger.Warn("controller-supervisor: RESTARTING the controller on the operator's press", "vmid", vmid, "action_id", id, "unit", bootstrapUnit)
began := time.Now()
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
return opOutcomeFailed, fmt.Sprintf("the restart of guest %d's controller failed: %v", vmid, err)
}
for time.Since(began) < s.opWait() {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
secs := int(time.Since(began).Round(time.Second) / time.Second)
return opOutcomeDone, fmt.Sprintf("restarted; the controller answered after %d s (Docker's health check: healthy)", secs)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, "restarted, but the wait for its answer was cut"
case <-time.After(s.opPoll()):
}
}
return opOutcomeFailed, fmt.Sprintf("restarted, but no healthy answer within %s — look at the box", s.opWait().Round(time.Second))
}
func orDash(s string) string {
if s == "" {
return "not listed"
}
return s
}
+222
View File
@@ -0,0 +1,222 @@
package localapi
import (
"context"
"errors"
"os"
"path/filepath"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// Operator brief 2026-10-29, Part 1: „Restart controller" from the hub. The agent's closed list holds one verb; the act
// is the supervisor's own restart with the supervisor's guards, and the brake counts operator restarts too.
// opExec answers the supervisor's fake plus the health read the operator restart waits on.
type opExec struct {
*supExec
health string // what {{.State.Health.Status}} reads after the restart
}
func (f *opExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) >= 5 && args[0] == "docker" && args[1] == "inspect" && args[3] == opHealthTemplate {
f.mu.Lock()
st := f.status[vmid]
f.mu.Unlock()
if st == "" {
return "", errors.New("pct exec: exit status 1: Error: No such object: felhom-controller")
}
return map[bool]string{true: "true", false: "false"}[st == "running"] + "|" + f.health + "\n", nil
}
return f.supExec.GuestExec(ctx, vmid, args...)
}
func opServer(t *testing.T, status string) (*Server, *opExec, *fakeGuestPowerCtl, *supClock, string, *atomic.Int32) {
t.Helper()
ex := &opExec{supExec: &supExec{status: map[int]string{9201: status}, onRestart: "running"}, health: "healthy"}
ctl := runningGuest(9201)
s, clk, dir := supServer(t, nil, ctl, 9201)
s.guestExec = ex
s.opHealthPoll = time.Millisecond
s.opHealthWait = 2 * time.Second
var triggered atomic.Int32
s.SetOperatorActionTrigger(func() { triggered.Add(1) })
return s, ex, ctl, clk, dir, &triggered
}
func env(acts ...hub.OperatorAction) *hub.ControlEnvelope {
return &hub.ControlEnvelope{OperatorActions: acts}
}
// waitResult waits for the action's result (the act runs in the background).
func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
for _, r := range s.OperatorActionResults() {
if r.ID == id {
return r
}
}
time.Sleep(2 * time.Millisecond)
}
t.Fatalf("no result for #%d", id)
return hub.OperatorActionResult{}
}
// RED-PROOF: seen failing before the list existed (build failure, then the names).
func TestOpActions_AgentClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
}
}
// Anything outside the list — and a bad argument — is refused and NOTHING is called.
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
s.OnEnvelope(context.Background(), env(
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
))
for _, id := range []int64{1, 2, 3} {
if r := waitResult(t, s, id); r.Outcome != "refused" {
t.Errorf("#%d: %+v, want refused", id, r)
}
}
if n := ex.count(9201); n != 0 {
t.Fatalf("a refused action restarted the controller (%d)", n)
}
}
// The press restarts ONCE, says how long the controller took to answer, asks for a report at once, and re-sends the
// result until the hub stops listing the id. RED-PROOF: drop the GuestExec restart → restarts 0 → FAILS.
func TestOpActions_RestartReportsTheAnswerTime(t *testing.T) {
s, ex, _, _, _, trig := opServer(t, "running")
a := hub.OperatorAction{ID: 7, Action: "restart_controller", Arg: "9201"}
s.OnEnvelope(context.Background(), env(a))
r := waitResult(t, s, 7)
if r.Outcome != "done" || !strings.Contains(r.Message, "answered after") {
t.Fatalf("result %+v", r)
}
if n := ex.count(9201); n != 1 {
t.Fatalf("restarts = %d, want 1", n)
}
if trig.Load() == 0 {
t.Fatal("the agent did not ask for a report — the result would wait 15 minutes")
}
s.OnEnvelope(context.Background(), env(a)) // the hub still lists it: re-send, never act again
time.Sleep(100 * time.Millisecond) // a wrong second act runs in the background — give it time to show
if n := ex.count(9201); n != 1 {
t.Fatalf("a re-listed id acted again (%d)", n)
}
if len(s.OperatorActionResults()) != 1 {
t.Fatal("the result must be re-sent while listed")
}
s.OnEnvelope(context.Background(), env()) // the hub recorded it
if left := s.OperatorActionResults(); len(left) != 0 {
t.Fatalf("a recorded result is still sent: %+v", left)
}
}
// The point of the agent doing it: a STOPPED controller is restarted too (the controller cannot restart itself).
func TestOpActions_WorksWithTheControllerStopped(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "exited")
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 8, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 8); r.Outcome != "done" || ex.count(9201) != 1 {
t.Fatalf("result %+v, restarts %d", r, ex.count(9201))
}
}
// Each supervisor guard refuses the press, with the reason. RED-PROOF: drop the lock check → the locked case
// restarts → FAILS (each guard seen failing by its own mutation).
func TestOpActions_GuardsRefuse(t *testing.T) {
cases := []struct {
name, want string
setup func(s *Server, ctl *fakeGuestPowerCtl, dir string)
}{
{"parked", "parked", func(_ *Server, _ *fakeGuestPowerCtl, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *fakeGuestPowerCtl, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.locks[9201] = "backup" }},
{"vzdump", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupRun = map[int]bool{9201: true} }},
{"vzdump unknown", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupErr = errors.New("tasks unreadable") }},
{"guest stopped", "not running", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guests[0].Status = "stopped" }},
{"guest list", "guest list", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guestsErr = errors.New("api down") }},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, ctl, _, dir, _ := opServer(t, "running")
tc.setup(s, ctl, dir)
id := int64(100 + i)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
r := waitResult(t, s, id)
if r.Outcome != "refused" || !strings.Contains(strings.ToLower(r.Message), tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if n := ex.count(9201); n != 0 {
t.Fatalf("guard %q did not hold: restarted %d", tc.name, n)
}
})
}
}
// The thrash brake counts operator restarts AND the supervisor's: 3 in 15 minutes → the next press is refused, and
// presses stay refused for 30 minutes. RED-PROOF: drop the brake → the 4th press restarts → FAILS.
func TestOpActions_TheBrakeCountsEveryRestart(t *testing.T) {
s, ex, _, clk, _, _ := opServer(t, "running")
// one restart by the supervisor itself
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
if ex.count(9201) != 1 {
t.Fatalf("setup: the supervisor did not restart (%d)", ex.count(9201))
}
for id := int64(1); id <= 2; id++ {
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("press %d: %+v", id, r)
}
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9201"}))
r := waitResult(t, s, 3)
if r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 3 {
t.Fatalf("restarts = %d, want 3", n)
}
clk.t = clk.t.Add(20 * time.Minute) // the window has passed, but the 30-minute refusal has not
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" {
t.Fatalf("inside the 30-minute refusal: %+v", r)
}
clk.t = clk.t.Add(11 * time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 5, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 5); r.Outcome != "done" {
t.Fatalf("after the refusal: %+v", r)
}
}
// A controller that does not come back healthy is a FAILED press, said plainly.
func TestOpActions_NoHealthyAnswerIsFailed(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
ex.health = "unhealthy"
s.opHealthWait = 30 * time.Millisecond
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 9, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 9); r.Outcome != "failed" || !strings.Contains(r.Message, "no healthy answer") {
t.Fatalf("result %+v", r)
}
}
+329
View File
@@ -0,0 +1,329 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
//
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
//
// THE GUARDS (each refuses with the reason in the result; nothing is called):
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
// - a controller swap (update) is in flight;
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
// alone while the reboot runs (rebootInFlight).
//
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
// one day). Pinned by opreboot_test.go.
const (
// OpRebootGuest is the agent's second operator verb (D65 = A).
OpRebootGuest = "reboot_guest"
// opRebootEvery: at most one operator reboot per guest per hour.
opRebootEvery = time.Hour
// opRebootNoticeFor: how long the household's dashboard shows the line.
opRebootNoticeFor = 24 * time.Hour
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
opRebootTimeout = 5 * time.Minute
operatorRebootFile = "operator-reboot.json"
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
// Docker does not hide it.
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
// stopped writing (stuck or killed) — not a busy one.
controllerBusyFresh = 90 * time.Second
)
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
type operatorRebootRecord struct {
At time.Time `json:"at"`
ActionID int64 `json:"action_id"`
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
ControllerAfterS int `json:"controller_after_s,omitempty"`
}
func (s *Server) operatorRebootPath(vmid int) string {
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
}
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
// must not silently vanish).
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
if errors.Is(err, os.ErrNotExist) {
return operatorRebootRecord{}, false, nil
}
if err != nil {
return operatorRebootRecord{}, false, err
}
var rec operatorRebootRecord
if err := json.Unmarshal(raw, &rec); err != nil {
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
}
return rec, !rec.At.IsZero(), nil
}
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
raw, err := json.Marshal(rec)
if err != nil {
return err
}
p := s.operatorRebootPath(vmid)
tmp := p + ".tmp"
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
return err
}
return os.Rename(tmp, p)
}
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
func (s *Server) rebooting(vmid int) bool {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
return s.rebootInFlight[vmid]
}
func (s *Server) setRebooting(vmid int, on bool) {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
if s.rebootInFlight == nil {
s.rebootInFlight = map[int]bool{}
}
if on {
s.rebootInFlight[vmid] = true
} else {
delete(s.rebootInFlight, vmid)
}
}
func (s *Server) rebooter() interface {
RebootGuest(ctx context.Context, vmid int) error
} {
if s.opRebooter != nil {
return s.opRebooter
}
if s.guestAttach != nil {
return s.guestAttach
}
return nil
}
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
// (the caller refuses — it cannot prove no backup runs). Two callers: the operator's reboot (D65) and R-1344's restart
// of a controller that does not answer (controllersupervisor.go restartUnresponsive).
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
if err != nil {
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
return false, "", nil
}
return false, "", err
}
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
if perr != nil {
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
}
if s.clock().Sub(at) > controllerBusyFresh {
s.logger.Info("controller-busy: the controller's busy file is stale — the controller stopped writing it; not busy",
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
return false, "", nil
}
why := "work in progress"
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
why = strings.TrimSpace(lines[1])
}
return true, why, nil
}
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
rb := s.rebooter()
if s.staleLock == nil || s.guestExec == nil || rb == nil {
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
if s.rebooting(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
// At most one per hour.
now := s.clock()
last, had, err := s.lastOperatorReboot(vmid)
if err != nil {
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
}
if had && now.Sub(last.At) < opRebootEvery {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
}
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
s.ctrlSup.mu.Unlock()
// The heavy-operation gate, held until the guest is up again.
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
if !ok {
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
}
defer release()
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
}
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
} else if cbusy {
return opOutcomeRefused, "refused: the controller is busy — " + why
}
// The record FIRST: the hourly limit and the household's line both read it.
rec := operatorRebootRecord{At: now, ActionID: id}
if err := s.saveOperatorReboot(vmid, rec); err != nil {
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
}
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.setRebooting(vmid, true)
defer s.setRebooting(vmid, false)
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
began := time.Now()
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
err = rb.RebootGuest(rctx, vmid)
cancel()
if err != nil {
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
}
upS := int(time.Since(began).Round(time.Second) / time.Second)
rec.GuestUpAfterS = upS
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
}
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
case <-time.After(s.opPoll()):
}
}
_ = s.saveOperatorReboot(vmid, rec)
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
}
// rebootBlockerSentence names jobBlocker's code in the result's words.
func rebootBlockerSentence(code string) string {
switch code {
case PowerOffBackup:
return "a backup is running"
case PowerOffCtrlUpdate:
return "a controller update is running"
case PowerOffDrive:
return "a drive or network-storage job is running"
case PowerOffEscrow:
return "an escrow ceremony is running"
case PowerOffUnknown:
return "the running jobs cannot be read"
default:
return "the host is busy"
}
}
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
rec, ok, err := s.lastOperatorReboot(vmid)
if err != nil {
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
return
}
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
return
}
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
}
+306
View File
@@ -0,0 +1,306 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
// rbExec adds the controller's busy file to the operator-restart fake.
type rbExec struct {
*opExec
busy string // the file's content; "" = no file
busyErr error // pct exec itself failed
}
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
if f.busyErr != nil {
return "", f.busyErr
}
if f.busy == "" {
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
}
return f.busy, nil
}
return f.opExec.GuestExec(ctx, vmid, args...)
}
type fakeRebooter struct {
mu sync.Mutex
n int
err error
after func()
}
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
r.mu.Lock()
defer r.mu.Unlock()
r.n++
if r.after != nil {
r.after()
}
return r.err
}
func (r *fakeRebooter) count() int {
r.mu.Lock()
defer r.mu.Unlock()
return r.n
}
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
t.Helper()
s, ox, ctl, clk, dir, _ := opServer(t, "running")
ex := &rbExec{opExec: ox}
s.guestExec = ex
rb := &fakeRebooter{}
s.opRebooter = rb
s.inFlight = &backup.InFlight{}
return s, ex, rb, ctl, clk, dir
}
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
return waitResult(t, s, id)
}
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
s, _, rb, _, clk, _ := rbServer(t)
r := pressReboot(t, s, 1)
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
t.Fatalf("result %+v", r)
}
if rb.count() != 1 {
t.Fatalf("reboots = %d, want 1", rb.count())
}
rec, ok, err := s.lastOperatorReboot(9201)
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
}
if s.inFlight.Busy() != "" {
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
}
if s.rebooting(9201) {
t.Fatal("the reboot-in-flight flag outlived the reboot")
}
}
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
func TestOpReboot_GuardsRefuse(t *testing.T) {
fresh := func(clk *supClock) string {
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
}
cases := []struct {
name, want string
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
}{
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.guests[0].Status = "stopped"
}},
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupRun = map[int]bool{9201: true}
}},
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupErr = errors.New("tasks unreadable")
}},
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.inFlight.TryAcquire("backup:felhom-pbs")
}},
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
}},
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
ex.busy = fresh(clk)
}},
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
}},
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
s.ctrlSup.mu.Lock()
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
s.ctrlSup.mu.Unlock()
}},
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
panic(err)
}
}},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, rb, ctl, clk, dir := rbServer(t)
tc.setup(s, ex, ctl, clk, dir)
r := pressReboot(t, s, int64(100+i))
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if rb.count() != 0 {
t.Fatalf("guard %q did not hold: rebooted", tc.name)
}
if _, ok, _ := s.lastOperatorReboot(9201); ok {
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
}
})
}
}
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
s, ex, rb, _, clk, _ := rbServer(t)
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
t.Fatalf("result %+v, reboots %d", r, rb.count())
}
}
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
s, _, rb, ctl, clk, dir := rbServer(t)
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("first press: %+v", r)
}
clk.t = clk.t.Add(59 * time.Minute)
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
t.Fatalf("second press inside the hour: %+v", r)
}
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
s2, ox2, _, _, _, _ := opServer(t, "running")
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
s2.guestExec = &rbExec{opExec: ox2}
rb2 := &fakeRebooter{}
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
t.Fatalf("after an agent restart, inside the hour: %+v", r)
}
clk.t = clk.t.Add(2 * time.Minute)
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
}
}
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
s, ex, _, _, clk, _ := rbServer(t)
for id := int64(1); id <= 2; id++ {
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("restart %d: %+v", id, r)
}
clk.t = clk.t.Add(time.Minute)
}
if r := pressReboot(t, s, 3); r.Outcome != "done" {
t.Fatalf("reboot: %+v", r)
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 2 {
t.Fatalf("restarts = %d, want 2", n)
}
}
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
s, _, rb, _, _, _ := rbServer(t)
rb.err = errors.New("pct reboot 9201: exit status 255")
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
t.Fatalf("result %+v", r)
}
if s.inFlight.Busy() != "" || s.rebooting(9201) {
t.Fatal("the gate or the flag outlived a failed reboot")
}
}
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
s, ex, rb, ctl, _, _ := rbServer(t)
var started, restarted int
rb.after = func() {
ctl.guests[0].Status = "stopped"
s.GuestPowerTick(context.Background())
ctl.guests[0].Status = "running"
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
started, restarted = len(ctl.started), ex.count(9201)
ex.mu.Lock()
ex.status[9201] = "running"
ex.mu.Unlock()
}
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
if started != 0 || restarted != 0 {
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
}
}
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
s, _, _, _, clk, _ := rbServer(t)
read := func() string {
w := httptest.NewRecorder()
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
var body struct {
Data struct {
At string `json:"at"`
} `json:"data"`
At string `json:"at"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("body %q: %v", w.Body.String(), err)
}
if body.Data.At != "" {
return body.Data.At
}
return body.At
}
if got := read(); got != "" {
t.Fatalf("no reboot yet, got %q", got)
}
at := clk.t
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
clk.t = at.Add(23 * time.Hour)
if got := read(); got != at.UTC().Format(time.RFC3339) {
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
}
clk.t = at.Add(25 * time.Hour)
if got := read(); got != "" {
t.Fatalf("after 25 h the line must be gone, got %q", got)
}
}
+26
View File
@@ -336,6 +336,7 @@ type Server struct {
formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional) formatJobs *FormatJobStore // F20-BUG3 detached-format job record (optional)
staleLock StaleLockController // F2-b startup stale-lock recovery (optional) staleLock StaleLockController // F2-b startup stale-lock recovery (optional)
partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional) partialSweep func(ctx context.Context, vmid int) // R-1006 start-up sweep of dead vzdump leftovers (optional)
lineTest LineTestReader // R-1247 weekly line test (optional; nil → present:false)
// guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog. // guestPower (F-REBOOT) is per-guest start-attempt state for the guest-power watchdog.
// Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState). // Guarded by guestPowerMu in guestpower.go; in-memory on purpose (see guestPowerState).
guestPower map[int]guestPowerState guestPower map[int]guestPowerState
@@ -410,6 +411,20 @@ type Server struct {
guestsDir string guestsDir string
ctrlSup controllerSupervisor ctrlSup controllerSupervisor
// 2026-10-29: the operator's actions from the hub's reply (opactions.go). opHealthPoll/opHealthWait bound the wait
// for a restarted controller's healthy answer (tests shorten them; zero → the defaults).
opActs opActionState
opTrigger func()
opHealthPoll time.Duration
opHealthWait time.Duration
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
opRebooter interface {
RebootGuest(ctx context.Context, vmid int) error
}
rebootMu sync.Mutex
rebootInFlight map[int]bool
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the // Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
// detached pipeline runs through (tests inject; production defaults set in NewServer). // detached pipeline runs through (tests inject; production defaults set in NewServer).
netVerifyMu sync.Mutex netVerifyMu sync.Mutex
@@ -458,6 +473,8 @@ type Server struct {
escrowLookPath func(file string) (string, error) escrowLookPath func(file string) (string, error)
// statFile reports whether a path exists (preflight's staged-secret item; tests inject). // statFile reports whether a path exists (preflight's staged-secret item; tests inject).
statFile func(path string) bool statFile func(path string) bool
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
cgroupRoot string
baseCtx context.Context // for fire-and-forget backups; set in Run baseCtx context.Context // for fire-and-forget backups; set in Run
} }
@@ -573,6 +590,13 @@ func (s *Server) Handler() http.Handler {
// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller // R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false. // waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard)) mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
// read it. Read-only; never measured (or the job not wired) = present:false.
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate. // Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks)) mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates)) mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))
@@ -586,6 +610,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach)) mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds. // Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot)) mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go. // Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff)) mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize // Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
+4 -2
View File
@@ -256,8 +256,10 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
} }
} }
} }
if callers != 5 { // 2026-10-29: +2 deliberate callers — the operator's restart_controller (opactions.go) and R-1344's unresponsive count
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers) // (controllersupervisor.go observeControllerHealth). Both through s.staleLock, the one helper.
if callers != 7 {
t.Errorf("expected the seven known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup, the operator restart, the unresponsive count), found %d", callers)
} }
b, _ := os.ReadFile("stalelock.go") b, _ := os.ReadFile("stalelock.go")
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") { if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
@@ -0,0 +1,156 @@
package localapi
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// R-1344 (operator brief 2026-10-29, Part 2; decided 2026-10-30 from the night's measurement): a controller that RUNS
// but whose Docker health check reads „unhealthy" on controllerUnresponsiveConfirm consecutive sweeps is reported
// unresponsive (the stanza; the hub raises controller_unresponsive). If it still reads „unhealthy" at
// controllerUnresponsiveRestart sweeps, the agent restarts it — with the not-running restart's guards and brake, plus
// the controller's own busy file. RED-PROOF: the restart tests seen failing before the restart existed.
func unrespServer(t *testing.T, health string) (*Server, *rbExec, *fakeGuestPowerCtl, *supClock) {
t.Helper()
s, ex, ctl, clk, _, _ := opServer(t, "running")
ex.health = health
rb := &rbExec{opExec: ex}
s.guestExec = rb
return s, rb, ctl, clk
}
func sweep(s *Server, n int) {
for i := 0; i < n; i++ {
s.ControllerSupervisorTick(context.Background())
}
}
func TestSupervisor_UnresponsiveIsAlarmedThenRestarted(t *testing.T) {
s, ex, _, _ := unrespServer(t, "unhealthy")
sweep(s, controllerUnresponsiveConfirm-1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
}
sweep(s, 1)
g := s.ControllerSupervisorStatus(context.Background()).Guests
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
}
if g[0].UnresponsiveRestartAfterS != (controllerUnresponsiveRestart-controllerUnresponsiveConfirm)*int(controllerSupervisorInterval/time.Second) {
t.Fatalf("the stanza must tell the hub when the restart comes: %+v", g[0])
}
since := g[0].UnresponsiveSince
sweep(s, controllerUnresponsiveRestart-controllerUnresponsiveConfirm-1)
if n := ex.count(9201); n != 0 {
t.Fatalf("restarted after %d unhealthy sweeps — the restart waits for %d", controllerUnresponsiveRestart-1, controllerUnresponsiveRestart)
}
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
}
sweep(s, 1)
if n := ex.count(9201); n != 1 {
t.Fatalf("after %d unhealthy sweeps: %d restarts, want 1", controllerUnresponsiveRestart, n)
}
g = s.ControllerSupervisorStatus(context.Background()).Guests
if g[0].UnresponsiveRestartAt == "" || g[0].UnresponsiveRestartAt != g[0].LastRestartAt || g[0].Restarts24h != 1 ||
!strings.Contains(g[0].LastReason, "health check") {
t.Fatalf("the restart is not on the record the hub reads: %+v", g[0])
}
// A restart that did not help (still „unhealthy" on the next sweep) is not repeated at once: the count starts again.
sweep(s, 1)
if n := ex.count(9201); n != 1 {
t.Fatalf("restarted again on the sweep after a restart: %d restarts", n)
}
// The restarted controller starts, then answers: the episode ends, its record stays.
ex.health = "starting"
sweep(s, 1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
t.Fatalf("after the restart: %+v (unresponsive must clear; since stays as the record)", g[0])
}
// A second episode needs the whole count again — never a restart on the first unhealthy read after one.
ex.health = "unhealthy"
sweep(s, controllerUnresponsiveRestart-1)
if n := ex.count(9201); n != 1 {
t.Fatalf("a second episode restarted early: %d restarts", n)
}
}
// Slow but answering (healthy / starting), a swap, a vzdump, a locked or a parked guest never counts — and so is never
// restarted.
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
for _, tc := range []struct {
name string
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl)
}{
{"healthy", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
{"starting", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
{"swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
{"vzdump", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
{"locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
} {
t.Run(tc.name, func(t *testing.T) {
s, ex, ctl, _ := unrespServer(t, "unhealthy")
tc.setup(s, ex, ctl)
sweep(s, controllerUnresponsiveRestart+2)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
}
if n := ex.count(9201); n != 0 {
t.Fatalf("%s: restarted %d times", tc.name, n)
}
})
}
}
// At the restart moment: the controller's busy file (fresh, or unreadable) and the brake refuse it; the alarm stays.
// RED-PROOF: each seen failing by deleting its guard.
func TestSupervisor_UnresponsiveRestartRefusals(t *testing.T) {
for _, tc := range []struct {
name string
setup func(s *Server, ex *rbExec, clk *supClock)
crashloop bool
}{
{"busy file fresh", func(_ *Server, ex *rbExec, clk *supClock) {
ex.busy = clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\na backup, a restore or an off-site run is running\n"
}, false},
{"busy file unreadable", func(_ *Server, ex *rbExec, _ *supClock) { ex.busyErr = errors.New("pct exec: exit status 255") }, false},
{"brake: 3 restarts in 15 minutes", func(s *Server, _ *rbExec, clk *supClock) {
st := s.supState(9201)
st.restarts = []time.Time{clk.t.Add(-10 * time.Minute), clk.t.Add(-5 * time.Minute), clk.t.Add(-time.Minute)}
}, true},
{"crash-loop pause in force", func(s *Server, _ *rbExec, clk *supClock) {
s.supState(9201).crashloopSince = clk.t.Add(-5 * time.Minute)
}, true},
} {
t.Run(tc.name, func(t *testing.T) {
s, ex, _, clk := unrespServer(t, "unhealthy")
tc.setup(s, ex, clk)
sweep(s, controllerUnresponsiveRestart+3)
if n := ex.count(9201); n != 0 {
t.Fatalf("%s: restarted %d times", tc.name, n)
}
g := s.ControllerSupervisorStatus(context.Background()).Guests
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveRestartAt != "" {
t.Fatalf("%s: the alarm must stay and no restart be recorded: %+v", tc.name, g)
}
if tc.crashloop && !g[0].Crashloop {
t.Fatalf("%s: the brake did not raise the crash-loop: %+v", tc.name, g[0])
}
})
}
}
// A busy file the controller stopped writing (older than controllerBusyFresh) is not „busy" — a hung controller must
// not shield itself with its last word.
func TestSupervisor_UnresponsiveStaleBusyFileDoesNotShield(t *testing.T) {
s, ex, _, clk := unrespServer(t, "unhealthy")
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\nan app update is running\n"
sweep(s, controllerUnresponsiveRestart)
if n := ex.count(9201); n != 1 {
t.Fatalf("a stale busy file blocked the restart: %d restarts", n)
}
}
+32
View File
@@ -8,6 +8,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"io" "io"
"log/slog"
"net/http" "net/http"
"os" "os"
"path/filepath" "path/filepath"
@@ -114,12 +115,43 @@ func (e ConfigUpdateExecutor) Execute(ctx context.Context, op string, params jso
return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed) return fmt.Errorf("agent_config_update: failed (previous files restored): %s", wr.Failed)
} }
lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds) lg.Warn("osupdate: config bundle INSTALLED", "bundle", string(wr.Bundle), "pass_seconds", wr.PassSeconds)
e.policyPass(ctx, lg, p.AgentVersion)
if e.AfterInstall != nil { if e.AfterInstall != nil {
e.AfterInstall(ctx) e.AfterInstall(ctx)
} }
return nil return nil
} }
// policyPass (R-1285) asks the wrapper NOW on disk — after this install, the bundle's own — to apply the
// appliance-only files' policy (mode "bundle-policy"): on a BYO host it removes the SSH drop-in we wrote. The bundle run
// itself was planned by the wrapper that was on disk BEFORE it, which may predate the policy (0.164.0). Never fatal:
// the bundle is installed either way; an older wrapper refuses the unknown mode and the next bundle does it.
// PolicyPassAfterUpdate runs the wrapper's bundle-policy pass once after a self-update commits (R-1285). In the normal
// delivery the box gets agent_update and agent_config_update in ONE pass: the bundle job runs in the OLD process, which
// has no policy pass, and the restart into the new agent comes a second later (measured 2026-10-26 on demo-hp). By the
// commit the wrapper on disk is the bundle's own, so the pass removes our appliance-only files from a BYO host in the
// SAME update. Idempotent; on an appliance it removes nothing.
func (e ConfigUpdateExecutor) PolicyPassAfterUpdate(ctx context.Context, lg *slog.Logger, ver string) {
if lg == nil {
lg = slog.Default()
}
lg.Info("osupdate: running the bundle policy pass after the self-update commit (R-1285)", "agent_version", ver)
e.policyPass(ctx, lg, ver)
}
func (e ConfigUpdateExecutor) policyPass(ctx context.Context, lg *slog.Logger, ver string) {
runID := "bundle-policy-" + e.Leg.now().UTC().Format("20060102T150405Z")
wr, err := e.Leg.call(ctx, runID, map[string]any{"release_id": "bundle-policy-" + ver, "layer": LayerHost, "mode": "bundle-policy"})
switch {
case err != nil:
lg.Warn("osupdate: the bundle policy pass did not run", "err", err)
case wr.refused():
lg.Info("osupdate: the bundle policy pass was refused (a wrapper before 0.166.0?) — the next bundle applies it", "refused", string(wr.Refused))
default:
lg.Info("osupdate: bundle policy pass done", "bundle_policy", string(wr.BundlePolicy))
}
}
func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) { func (e ConfigUpdateExecutor) download(ctx context.Context, url, dest string) (string, error) {
hc := e.HTTPClient hc := e.HTTPClient
if hc == nil { if hc == nil {
+27
View File
@@ -38,6 +38,9 @@ func (b *bundleWrapper) Run(_ context.Context, name string, args ...string) ([]b
var plan map[string]any var plan map[string]any
_ = json.Unmarshal(raw, &plan) _ = json.Unmarshal(raw, &plan)
b.plans = append(b.plans, plan) b.plans = append(b.plans, plan)
if plan["mode"] == "bundle-policy" { // R-1285: the pass after an install carries no bundle
return []byte(`OSAPPLY-REPORT {"mode":"bundle-policy","bundle_policy":{"install_mode":"byo","removed":[]}}` + "\n"), nil, nil
}
body, _ := os.ReadFile(plan["bundle"].(string)) body, _ := os.ReadFile(plan["bundle"].(string))
b.bodies = append(b.bodies, body) b.bodies = append(b.bodies, body)
return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil return []byte("OSAPPLY-REPORT " + b.rep + "\n"), nil, nil
@@ -153,3 +156,27 @@ func TestBundleURL(t *testing.T) {
func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) { func (b *bundleWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return b.Run(ctx, name, args...) return b.Run(ctx, name, args...)
} }
// R-1285: right after an install the agent asks the wrapper now on disk for the appliance-only policy pass — the bundle
// run itself was planned by the OLD wrapper (on a BYO host upgraded from 0.164.0 our SSH drop-in stayed one bundle
// longer). A refused or failed bundle runs no pass.
// RED-PROOF: drop the policyPass call → one plan only.
func TestR1285_AnInstalledBundleIsFollowedByThePolicyPass(t *testing.T) {
body := []byte(`{"format":1,"agent_version":"0.143.0","files":[]}`)
srv, sha := serveBundle(t, body)
w := &bundleWrapper{t: t, rep: `{"mode":"bundle","bundle":{"agent_version":"0.143.0","written":["/usr/local/sbin/felhom-os-apply"]}}`}
e, _ := bundleExec(t, w, srv.URL)
if err := e.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha)); err != nil {
t.Fatal(err)
}
if len(w.plans) != 2 || w.plans[1]["mode"] != "bundle-policy" || w.plans[1]["layer"] != "host" || w.plans[1]["signed"] != nil {
t.Fatalf("after the install the agent must run the policy pass (host layer, no envelope): %v", w.plans)
}
w2 := &bundleWrapper{t: t, rep: `{"mode":"bundle","refused":{"code":"R3","reason":"x"}}`}
e2, _ := bundleExec(t, w2, srv.URL)
_ = e2.Execute(signedCtx(), OpConfigUpdate, params("0.143.0", sha))
if len(w2.plans) != 1 {
t.Fatalf("a refused bundle must run no policy pass: %v", w2.plans)
}
}
+1
View File
@@ -124,6 +124,7 @@ type WrapperReport struct {
LiveRestore json.RawMessage `json:"live_restore"` LiveRestore json.RawMessage `json:"live_restore"`
Facts json.RawMessage `json:"facts"` Facts json.RawMessage `json:"facts"`
Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle") Bundle json.RawMessage `json:"bundle"` // the config bundle's result (R-840, mode "bundle")
BundlePolicy json.RawMessage `json:"bundle_policy"` // the appliance-only policy pass (R-1285, mode "bundle-policy")
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event, // OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it. // exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
OOMCheck json.RawMessage `json:"oom_check"` OOMCheck json.RawMessage `json:"oom_check"`
+9
View File
@@ -18,8 +18,14 @@ type Manager struct {
logger *slog.Logger logger *slog.Logger
now func() time.Time // injectable for tests now func() time.Time // injectable for tests
sleep func(context.Context, time.Duration) sleep func(context.Context, time.Duration)
// onCommitted runs once, right after a successful commit (R-1285: the bundle policy pass of the NEW agent — in the
// normal two-job delivery the bundle was installed by the OLD process, which has no such pass). nil = none.
onCommitted func(context.Context)
} }
// SetOnCommitted wires the after-commit hook (R-1285). INIT-ONLY: call it before MaybeCommit's goroutine starts.
func (m *Manager) SetOnCommitted(fn func(context.Context)) { m.onCommitted = fn }
// ManagerConfig wires the commit manager. // ManagerConfig wires the commit manager.
type ManagerConfig struct { type ManagerConfig struct {
StateDir string StateDir string
@@ -119,4 +125,7 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
return return
} }
m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout)) m.logger.Warn("selfupdate: update committed", "version", m.runningVersion, "wrapper", trim(stdout))
if m.onCommitted != nil {
m.onCommitted(ctx)
}
} }
+32
View File
@@ -0,0 +1,32 @@
package wgtunnel
import (
"context"
"testing"
)
// R-1247: the line test's server is the desired block's pbs_tunnel_ip, and ONLY while the tunnel is operational
// (registered + unit active) — „no tunnel → no test" rests on this answer.
func TestLineTestTargetOnlyWhileOperational(t *testing.T) {
m, _, active, _ := testManager(t, &fakeHub{})
ctx := context.Background()
if _, ok := m.LineTestTarget(); ok {
t.Fatal("a target before any Apply")
}
m.Apply(ctx, false, nil) // registers
pub := localPub(t, m)
m.Apply(ctx, true, testBlock(pub)) // installs; the unit is not active yet
if _, ok := m.LineTestTarget(); ok {
t.Fatal("a target while the unit is not active")
}
*active = true
m.Apply(ctx, true, testBlock(pub))
ip, ok := m.LineTestTarget()
if !ok || ip.String() != "10.77.0.1" {
t.Fatalf("operational tunnel: target %v ok=%v, want 10.77.0.1", ip, ok)
}
m.Apply(ctx, true, nil) // revoked
if _, ok := m.LineTestTarget(); ok {
t.Fatal("a target after the hub revoked the tunnel")
}
}
+54 -1
View File
@@ -99,6 +99,7 @@ type Manager struct {
resolver Resolver resolver Resolver
staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2) staleAfter time.Duration // handshake-age threshold for the re-resolve watchdog (doc 06 §4.2)
rekeyAfter time.Duration // D30: how long a future-stamped handshake stays before the re-key (rekey.go)
mu sync.Mutex mu sync.Mutex
nextRegAt time.Time nextRegAt time.Time
@@ -111,6 +112,10 @@ type Manager struct {
resolveFailLogged bool // DNS-failure log throttle (reset on the next success) resolveFailLogged bool // DNS-failure log throttle (reset on the next success)
staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP) staleSameIPLogged bool // "stale but IP unchanged" log throttle (endpoint-down, not re-IP)
// D30 (rekey.go): when the latest handshake was first seen stamped in the future, and the last re-key.
futureSince time.Time
lastRekey time.Time
// conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at // conv is the CACHED tunnel-convergence snapshot (v0.90.0, R-28 fast-tick source), refreshed at
// the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing // the end of every Apply (the tunnel's own cadence) so the fast-tick reads it WITHOUT execing
// `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired; // `wg`/`systemctl` per tick. wgBlockDesired = a wireguard block for THIS key is desired;
@@ -123,6 +128,7 @@ type Manager struct {
type convSnapshot struct { type convSnapshot struct {
wgBlockDesired bool wgBlockDesired bool
operational bool operational bool
serverIP netip.Addr // the desired block's pbs_tunnel_ip (R-1247 line test target); invalid when none
} }
// TunnelConvergence returns the cached (desired, operational) snapshot without any exec (fast-tick // TunnelConvergence returns the cached (desired, operational) snapshot without any exec (fast-tick
@@ -134,6 +140,17 @@ func (m *Manager) TunnelConvergence() (desired, operational bool) {
return m.conv.wgBlockDesired, m.conv.operational return m.conv.wgBlockDesired, m.conv.operational
} }
// LineTestTarget is the R-1247 line test's server: ep0's tunnel address, ok only while the tunnel is operational
// (registered + unit active, as of the last Apply). Cached — no exec.
func (m *Manager) LineTestTarget() (netip.Addr, bool) {
m.mu.Lock()
defer m.mu.Unlock()
if !m.conv.operational || !m.conv.serverIP.IsValid() {
return netip.Addr{}, false
}
return m.conv.serverIP, true
}
// NewManager builds a Manager. stateDir is the agent state dir (default /var/lib/felhom-agent — // NewManager builds a Manager. stateDir is the agent state dir (default /var/lib/felhom-agent —
// note the FELHOM_WG sudoers install entry hard-codes the staged path under it). // note the FELHOM_WG sudoers install entry hard-codes the staged path under it).
func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, logger *slog.Logger) *Manager { func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, logger *slog.Logger) *Manager {
@@ -156,6 +173,7 @@ func NewManager(runner proxmox.Runner, registrar Registrar, stateDir string, log
now: time.Now, now: time.Now,
resolver: netResolver{}, resolver: netResolver{},
staleAfter: 180 * time.Second, staleAfter: 180 * time.Second,
rekeyAfter: rekeyAfterDefault,
} }
} }
@@ -342,7 +360,13 @@ func (m *Manager) Apply(ctx context.Context, fetched bool, block *hub.WireWiregu
operational = m.isActive(ctx) operational = m.isActive(ctx)
} }
} }
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational} var server netip.Addr
if desired {
if a, err := netip.ParseAddr(block.Endpoint.PBSTunnelIP); err == nil && a.Is4() {
server = a
}
}
m.conv = convSnapshot{wgBlockDesired: desired, operational: operational, serverIP: server}
}() }()
keyExists := false keyExists := false
@@ -600,6 +624,10 @@ func (m *Manager) Watchdog(ctx context.Context, fetched bool, block *hub.WireWir
if !fetched || block == nil || !m.lastResolvedIP.IsValid() { if !fetched || block == nil || !m.lastResolvedIP.IsValid() {
return // nothing applied to watch, or no desired block return // nothing applied to watch, or no desired block
} }
// D30: a handshake stamped in the future that stays → a new key on the existing peer path (rekey.go).
if m.clockAheadRekeyLocked(ctx, block) {
return
}
if !m.handshakeStaleLocked(ctx) { if !m.handshakeStaleLocked(ctx) {
return // healthy → no DNS, no action (the load-bearing negative) return // healthy → no DNS, no action (the load-bearing negative)
} }
@@ -661,6 +689,9 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
if age, ok := parseHandshakeAge(string(out), m.now()); ok { if age, ok := parseHandshakeAge(string(out), m.now()); ok {
st.LastHandshakeAgeS = &age st.LastHandshakeAgeS = &age
} }
if fut, ok := handshakeFuture(string(out), m.now()); ok {
st.HandshakeFutureS = &fut
}
} }
} }
if st.Pubkey == "" && !st.Registered && !st.Active { if st.Pubkey == "" && !st.Registered && !st.Active {
@@ -669,6 +700,28 @@ func (m *Manager) Status(ctx context.Context) *hub.WireguardStatus {
return st return st
} }
// handshakeFuture (R-1204) reports how many seconds the latest handshake lies in the future, ok only when it does
// (by more than a minute — clock jitter is not the case). parseHandshakeAge clamps that case to 0 („just now"), which
// is what made a box whose clock had run ahead look healthy while ep0 refused its tunnel. Pinned by
// TestR1204_AFutureHandshakeIsReported.
func handshakeFuture(out string, now time.Time) (int64, bool) {
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
f := strings.Fields(line)
if len(f) != 2 {
continue
}
epoch, err := strconv.ParseInt(f[1], 10, 64)
if err != nil || epoch <= 0 {
continue
}
if fut := epoch - now.Unix(); fut > 60 {
return fut, true
}
return 0, false
}
return 0, false
}
// parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>"). // parseHandshakeAge parses `wg show <if> latest-handshakes` output ("<peer-pub>\t<epoch>").
// epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0). // epoch 0 = no handshake yet → not ok (nil in the report; nil ≠ 0).
func parseHandshakeAge(out string, now time.Time) (int64, bool) { func parseHandshakeAge(out string, now time.Time) (int64, bool) {
+28
View File
@@ -696,3 +696,31 @@ func TestInitialResolveFailureNoTeardown(t *testing.T) {
t.Errorf("did not recover after DNS returned: %v", rr.calls) t.Errorf("did not recover after DNS returned: %v", rr.calls)
} }
} }
// R-1204: a box that ran once with its clock ahead keeps a handshake stamped in the FUTURE — ep0 then refuses every
// newer one. The age stays clamped at 0 (unchanged contract), and the new field says how far ahead it lies; a
// handshake in the past (or within a minute of now) never sets it.
// RED-PROOF: with Status not setting HandshakeFutureS the first assertion fails — seen failing 2026-10-22.
func TestR1204_AFutureHandshakeIsReported(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
ctx := context.Background()
m.Apply(ctx, false, nil)
*active = true
m.now = func() time.Time { return time.Unix(1783107115-2*24*3600, 0) } // the clock came back two days earlier
st := m.Status(ctx)
if st.HandshakeFutureS == nil || *st.HandshakeFutureS != 2*24*3600 {
t.Fatalf("handshake_future_s = %v, want %d", st.HandshakeFutureS, 2*24*3600)
}
if st.LastHandshakeAgeS == nil || *st.LastHandshakeAgeS != 0 {
t.Fatalf("the age stays clamped at 0: %v", st.LastHandshakeAgeS)
}
m.now = func() time.Time { return time.Unix(1783107115+100, 0) }
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("a past handshake must not set handshake_future_s: %v", *st.HandshakeFutureS)
}
m.now = func() time.Time { return time.Unix(1783107115-30, 0) } // 30 s of jitter is not the case
if st = m.Status(ctx); st.HandshakeFutureS != nil {
t.Fatalf("clock jitter must not set handshake_future_s")
}
}
+189
View File
@@ -0,0 +1,189 @@
package wgtunnel
import (
"context"
"crypto/rand"
"encoding/base64"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D30 (operator 2026-10-22, R-1204): the tunnel heals itself after the box ran with its clock ahead.
//
// WireGuard refuses a handshake whose timestamp is not newer than the last one it accepted from that key (replay
// protection). A box whose clock ran ahead handshook with a future timestamp; ep0 keeps it, and once the clock is
// right again ep0 refuses every new handshake from that key — until the box's clock passes the stored stamp. The
// sign on the box: its own latest handshake stays stamped in the future (R-1204's handshake_future_s).
//
// The repair: when that sign stays for rekeyAfter, the agent makes a NEW key pair and registers the new public key on
// the existing path (POST /hosts/<id>/wg → the hub re-keys the peer IN PLACE: same /32, new key, generation bump,
// and the wgsync push of the FULL peer list to ep0 drops the old key and adds the new one). A new key is a new peer
// for ep0, with no stored timestamp — the next handshake is accepted. Nothing new runs on ep0.
//
// The previous private key is kept as private.key.prev (0600): the escrow ceremony may hold it (a host-loss DR that
// installs it registers it again, and the hub re-keys in place back to it — the same path).
// Pinned by TestD30_* (rekey_test.go).
const (
// rekeyAfter: how long the latest handshake must stay in the future before the re-key. A working tunnel
// handshakes every two minutes, which ends the sign within minutes of the clock being right.
rekeyAfterDefault = 15 * time.Minute
// rekeyMinGap bounds the re-key to once an hour, whatever the clock does.
rekeyMinGap = time.Hour
prevKeyName = "private.key.prev"
)
// clockAheadRekeyLocked watches for a handshake stamped in the future and re-keys when it stays. It reports whether
// it re-keyed (the caller then skips the rest of its tick). Caller holds m.mu.
func (m *Manager) clockAheadRekeyLocked(ctx context.Context, block *hub.WireWireguard) bool {
if m.keyBroken || !m.isActive(ctx) {
m.futureSince = time.Time{}
return false
}
out, _, err := m.runner.Run(ctx, "wg", "show", Iface, "latest-handshakes")
if err != nil {
return false
}
now := m.now()
fut, ok := handshakeFuture(string(out), now)
if !ok {
if !m.futureSince.IsZero() {
m.logger.Info("wgtunnel: the handshake is no longer stamped in the future — no re-key needed")
}
m.futureSince = time.Time{}
return false
}
if m.futureSince.IsZero() {
m.futureSince = now
m.logger.Warn("wgtunnel: the latest handshake is stamped in the future — this box's clock ran ahead, and the endpoint refuses its newer handshakes; re-keying if it stays",
"future_s", fut, "rekey_after", m.rekeyAfter.String())
return false
}
// A clock that jumps backwards must not stretch either wait (commit security review 2026-10-23 night): a start or a
// last re-key stamped AFTER now is from a wrong clock and is dropped — this whole feature exists because clocks jump.
if now.Before(m.futureSince) {
m.futureSince = now
}
if now.Sub(m.futureSince) < m.rekeyAfter {
return false
}
if m.lastRekey.IsZero() {
m.lastRekey = m.loadLastRekey()
}
if now.Before(m.lastRekey) {
m.lastRekey = time.Time{}
}
if !m.lastRekey.IsZero() && now.Sub(m.lastRekey) < rekeyMinGap {
return false
}
pub, _, err := EnsureKey(m.stateDir)
if err != nil || block.Pubkey != pub {
return false // a re-register is already in flight, or the key is not readable — never re-key over that
}
newPub, err := rotateKey(m.stateDir)
if err != nil {
m.logger.Error("wgtunnel: re-key after a future handshake FAILED — the old key stays", "err", err)
m.lastRekey = now // not again before rekeyMinGap
m.saveLastRekey(now)
return false
}
m.lastRekey, m.futureSince = now, time.Time{}
m.saveLastRekey(now)
m.logger.Warn("wgtunnel: RE-KEYED after a handshake stamped in the future (D30) — registering the new key; the hub re-keys the peer in place and the endpoint takes the new key",
"old_pubkey", pub, "new_pubkey", newPub, "future_s", fut)
m.nextRegAt = time.Time{} // register now, not after an older backoff
m.registerLocked(ctx, newPub)
return true
}
// rotateKey makes a new key pair, never leaving the box without a readable key (review 2026-10-23 night, N6): the new
// key is written and fsynced to a temp file; the current key gets a second name by a HARD LINK (the escrow may hold
// it — `private.key.prev` keeps the first such key, later ones `private.key.prev.<unix>`, the newest three kept, N7);
// then the temp file is renamed OVER the current name (atomic) and the directory fsynced. At every instant
// `private.key` is a complete key. Returns the new PUBLIC key only.
func rotateKey(stateDir string) (string, error) {
dir := filepath.Join(stateDir, "wg")
cur := filepath.Join(dir, keyFileName)
if _, err := os.Stat(cur); err != nil {
return "", fmt.Errorf("no current key: %w", err)
}
priv := make([]byte, 32)
if _, err := rand.Read(priv); err != nil {
return "", fmt.Errorf("entropy: %w", err)
}
clampPrivateKey(priv)
pub, err := derivePublic(priv)
if err != nil {
return "", err
}
tmp := filepath.Join(dir, keyFileName+".new")
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return "", fmt.Errorf("writing the new key: %w", err)
}
_, werr := f.Write([]byte(base64.StdEncoding.EncodeToString(priv) + "\n"))
serr := f.Sync()
cerr := f.Close()
if werr != nil || serr != nil || cerr != nil {
os.Remove(tmp)
return "", fmt.Errorf("writing the new key: %v %v %v", werr, serr, cerr)
}
prev := filepath.Join(dir, prevKeyName)
if _, err := os.Lstat(prev); err == nil {
prev = filepath.Join(dir, fmt.Sprintf("%s.%d", prevKeyName, rekeyNow().Unix()))
}
if err := os.Link(cur, prev); err != nil {
os.Remove(tmp)
return "", fmt.Errorf("keeping the old key: %w", err)
}
if err := os.Rename(tmp, cur); err != nil {
os.Remove(tmp)
return "", fmt.Errorf("installing the new key: %w", err)
}
if d, err := os.Open(dir); err == nil {
d.Sync()
d.Close()
}
prunePrevKeys(dir, 3)
return pub, nil
}
// rekeyNow is the clock for the kept-key names (a seam).
var rekeyNow = time.Now
// prunePrevKeys keeps the newest `keep` of the `private.key.prev.<unix>` files (`private.key.prev` itself always stays).
func prunePrevKeys(dir string, keep int) {
m, _ := filepath.Glob(filepath.Join(dir, prevKeyName+".*"))
if len(m) <= keep {
return
}
sort.Strings(m) // the same digit count until 2286: lexical = time order
for _, p := range m[:len(m)-keep] {
os.Remove(p)
}
}
// lastRekeyFile records the last re-key's time across agent restarts (N7: the once-an-hour bound was in memory only).
const lastRekeyFile = "last-rekey"
func (m *Manager) loadLastRekey() time.Time {
b, err := os.ReadFile(filepath.Join(m.wgDir(), lastRekeyFile))
if err != nil {
return time.Time{}
}
var u int64
if _, err := fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &u); err != nil {
return time.Time{}
}
return time.Unix(u, 0)
}
func (m *Manager) saveLastRekey(t time.Time) {
_ = os.WriteFile(filepath.Join(m.wgDir(), lastRekeyFile), []byte(fmt.Sprintf("%d\n", t.Unix())), 0o600)
}
+227
View File
@@ -0,0 +1,227 @@
package wgtunnel
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// D30 (R-1204): a handshake stamped in the future that STAYS for rekeyAfter → a new key pair, registered on the
// existing peer path (the hub re-keys in place; its full-list push to ep0 swaps the key). The fake `wg` reports a
// handshake at 1783107115; a clock three hours behind that sees it three hours in the future.
// RED-PROOF (2026-10-22 night): with the clockAheadRekeyLocked call removed from Watchdog,
// TestD30_AFutureHandshakeThatStaysReKeys and …AFailedRegistration… fail (no new key, no registration); with the
// rekeyAfter wait removed, …ThatStaysReKeys fails (a re-key at 10 min); with the isActive check removed,
// TestD30_NoReKeyWhileTheTunnelIsDown fails.
const hs = 1783107115
func at(m *Manager, unix int64) { m.now = func() time.Time { return time.Unix(unix, 0) } }
func readKey(t *testing.T, m *Manager, name string) string {
t.Helper()
b, err := os.ReadFile(filepath.Join(m.stateDir, "wg", name))
if err != nil {
t.Fatal(err)
}
return strings.TrimSpace(string(b))
}
func TestD30_AFutureHandshakeThatStaysReKeys(t *testing.T) {
fh := &fakeHub{}
m, _, active, logs := testManager(t, fh)
oldPub := establishWG(t, m, active)
oldPriv := readKey(t, m, keyFileName)
ctx := context.Background()
calls := fh.count()
t0 := int64(hs - 3*3600) // the clock is right again; the last handshake (made while it ran ahead) is 3 h ahead
at(m, t0)
m.Watchdog(ctx, true, testBlock(oldPub))
at(m, t0+10*60)
m.Watchdog(ctx, true, testBlock(oldPub))
if fh.count() != calls || localPub(t, m) != oldPub {
t.Fatal("no re-key before the future stamp has stayed rekeyAfter")
}
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(oldPub))
newPub := localPub(t, m)
if newPub == oldPub {
t.Fatal("a future stamp that stayed 16 min must give a new key")
}
if fh.count() != calls+1 {
t.Fatalf("the new key must be registered on the existing peer path (RegisterWG calls %d → %d)", calls, fh.count())
}
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
t.Fatalf("the registration marker must carry the new key: %+v", mk)
}
if readKey(t, m, prevKeyName) != oldPriv {
t.Fatal("the old private key must be kept as private.key.prev (the escrow may hold it)")
}
if strings.Contains(logs.String(), oldPriv) || strings.Contains(logs.String(), readKey(t, m, keyFileName)) {
t.Fatal("a private key reached the log")
}
// The served block still names the OLD key until the next fetch: no second re-key over that.
at(m, t0+17*60)
m.Watchdog(ctx, true, testBlock(oldPub))
if localPub(t, m) != newPub {
t.Fatal("a second re-key ran before the hub served the new key")
}
// The hub serves the new key → the tunnel is applied with the NEW private key.
m.Apply(ctx, true, testBlock(newPub))
conf, err := os.ReadFile(m.stagedConfPath())
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(conf), readKey(t, m, keyFileName)) || strings.Contains(string(conf), oldPriv) {
t.Fatal("the applied tunnel must use the new private key, not the old one")
}
}
func TestD30_AShortFutureSpellDoesNotReKey(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, hs+100) // a handshake now lies in the past: the sign ended (a working tunnel)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+20*60) // in the future again: a NEW window starts, it does not continue the old one
m.Watchdog(ctx, true, testBlock(pub))
if localPub(t, m) != pub {
t.Fatal("a future stamp that did not STAY for rekeyAfter must not re-key")
}
}
func TestD30_NoReKeyWhileTheTunnelIsDown(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
*active = false
ctx := context.Background()
t0 := int64(hs - 3*3600)
for _, d := range []int64{0, 20 * 60, 40 * 60} {
at(m, t0+d)
m.Watchdog(ctx, true, testBlock(pub))
}
if localPub(t, m) != pub {
t.Fatal("no re-key while the tunnel unit is not active")
}
}
func TestD30_AFailedRegistrationIsRetriedByTheExistingPath(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
oldPub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(oldPub))
fh.mu.Lock()
fh.err = errors.New("hub down")
fh.mu.Unlock()
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(oldPub))
newPub := localPub(t, m)
if newPub == oldPub {
t.Fatal("the key rotates before the registration")
}
if mk := m.loadMarker(); mk == nil || mk.Pubkey != oldPub {
t.Fatal("a failed registration leaves the marker on the old key")
}
fh.mu.Lock()
fh.err = nil
fh.mu.Unlock()
at(m, t0+30*60) // past the backoff: Apply sees the served (old) key ≠ the local key → re-key-in-place registers
m.Apply(ctx, true, testBlock(oldPub))
if mk := m.loadMarker(); mk == nil || mk.Pubkey != newPub {
t.Fatalf("the existing re-key-in-place path must register the new key: %+v", mk)
}
}
// Review 2026-10-23 night. N7: a second rotation keeps the FIRST old key as `private.key.prev` (the escrow may hold it)
// and the later one beside it; N6: the current key is replaced by a rename OVER it, so `private.key` is never missing.
// RED-PROOF: with the „prev exists → a dated name" branch removed, TwoRotations fails (the first key overwritten) —
// seen failing 2026-10-23 night.
func TestD30_TwoRotationsKeepTheFirstOldKey(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
establishWG(t, m, active)
first := readKey(t, m, keyFileName)
old := rekeyNow
defer func() { rekeyNow = old }()
rekeyNow = func() time.Time { return time.Unix(hs, 0) }
if _, err := rotateKey(m.stateDir); err != nil {
t.Fatal(err)
}
second := readKey(t, m, keyFileName)
rekeyNow = func() time.Time { return time.Unix(hs+7200, 0) }
if _, err := rotateKey(m.stateDir); err != nil {
t.Fatal(err)
}
if readKey(t, m, prevKeyName) != first {
t.Fatal("private.key.prev must keep the FIRST old key")
}
if readKey(t, m, fmt.Sprintf("%s.%d", prevKeyName, hs+7200)) != second {
t.Fatal("the second old key must be kept beside it")
}
if cur := readKey(t, m, keyFileName); cur == first || cur == second {
t.Fatal("the current key must be new")
}
}
// N7: the once-an-hour bound survives an agent restart (it was in memory only). RED-PROOF: with loadLastRekey removed
// from clockAheadRekeyLocked, this fails — seen failing 2026-10-23 night.
func TestD30_TheHourBoundSurvivesARestart(t *testing.T) {
fh := &fakeHub{}
m, rr, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(pub))
newPub := localPub(t, m)
if newPub == pub {
t.Fatal("fixture: the first re-key must happen")
}
// The agent restarts: a NEW manager on the same state; the served block now names the new key; the sign is back.
m2 := NewManager(rr, fh, m.stateDir, m.logger)
m2.isActive, m2.wgPresent, m2.resolver = m.isActive, m.wgPresent, m.resolver
m2.Apply(ctx, true, testBlock(newPub))
at(m2, t0+20*60)
m2.Watchdog(ctx, true, testBlock(newPub))
at(m2, t0+40*60)
m2.Watchdog(ctx, true, testBlock(newPub))
if localPub(t, m2) != newPub {
t.Fatal("a restart must not reset the once-an-hour bound")
}
_ = active
}
// Commit security review 2026-10-23 night: a last re-key saved while the clock ran ahead (a later date than now) must
// not block the next re-key until that date. RED-PROOF: with the „lastRekey after now → dropped" guard removed, this
// fails — seen failing 2026-10-23 night.
func TestD30_ALastReKeyFromTheFutureDoesNotBlock(t *testing.T) {
fh := &fakeHub{}
m, _, active, _ := testManager(t, fh)
pub := establishWG(t, m, active)
m.saveLastRekey(time.Unix(hs+2*365*86400, 0)) // written while the clock was two years ahead
ctx := context.Background()
t0 := int64(hs - 3*3600)
at(m, t0)
m.Watchdog(ctx, true, testBlock(pub))
at(m, t0+16*60)
m.Watchdog(ctx, true, testBlock(pub))
if localPub(t, m) == pub {
t.Fatal("a last re-key dated in the future must not block the re-key")
}
}
+28 -3
View File
@@ -98,7 +98,8 @@ TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
# THE retention number, read from the one file that owns it. A check and the policy it enforces # THE retention number, read from the one file that owns it. A check and the policy it enforces
# must read the same number from the same place, or they drift and the drift looks like a defect # must read the same number from the same place, or they drift and the drift looks like a defect
# in something else — which is exactly what happened on 2026-08-08/09 (R-287). # in something else — which is exactly what happened on 2026-08-08/09 (R-287).
RETENTION_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") RETENTION_FILE = os.environ.get("FELHOM_RETENTION_POLICY") or \
os.path.join(os.path.dirname(os.path.abspath(__file__)), "retention-policy.json") # the env: decoy tests only
def retention_kept(): def retention_kept():
@@ -114,6 +115,21 @@ def retention_kept():
raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,)) raise ValueError("generic_versions_kept must be a positive int, got %r" % (n,))
return n return n
def retention_pinned():
"""D50 = A: the versions kept besides the newest N — the hub's MinAgent floor and the day-0 agent. Copies of hub
state the vouching session writes into the policy file (CI holds no hub secret). Missing key = no pins."""
with open(RETENTION_FILE, encoding="utf-8") as fh:
pins = json.load(fh).get("pinned") or {}
if not isinstance(pins, dict):
raise ValueError("pinned must be an object, got %r" % (pins,))
out = {}
for name, v in pins.items():
if not isinstance(v, str) or not re.match(r"^\d+\.\d+\.\d+$", v):
raise ValueError("pinned %s must be X.Y.Z, got %r" % (name, v))
out[name] = v
return out
tried = [] tried = []
@@ -211,6 +227,7 @@ def main():
all_versions = versions all_versions = versions
try: try:
keep = retention_kept() keep = retention_kept()
pins = retention_pinned()
except Exception as e: except Exception as e:
inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e)) inconclusive("cannot read the retention policy (%s): %s" % (RETENTION_FILE, e))
@@ -218,10 +235,15 @@ def main():
# lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end. # lexically: "0.9.0" > "0.10.0" as strings, and that would silently drop the wrong end.
def _key(v): def _key(v):
return tuple(int(x) for x in v.split(".")) return tuple(int(x) for x in v.split("."))
versions = sorted(all_versions, key=_key)[-keep:] window = sorted(all_versions, key=_key)[-keep:]
# D50 = A: the pinned versions are asserted wherever they sit. A pin with no tag cannot be installed at all —
# a failure, never a skip.
unpinned_tags = sorted(("%s=%s" % (k, v)) for k, v in pins.items() if v not in all_versions)
versions = sorted(set(window) | set(v for v in pins.values() if v in all_versions), key=_key)
dropped = [v for v in all_versions if v not in versions] dropped = [v for v in all_versions if v not in versions]
print(" %d released version(s); retention policy keeps the newest %d" % (len(all_versions), keep)) print(" %d released version(s); retention policy keeps the newest %d plus the pinned %s" % (
len(all_versions), keep, ", ".join("%s=%s" % kv for kv in sorted(pins.items())) or "(none)"))
print(" verifying: %s" % ", ".join(versions)) print(" verifying: %s" % ", ".join(versions))
if dropped: if dropped:
# NEVER silent. A bounded check that does not say what it stopped covering is how a # NEVER silent. A bounded check that does not say what it stopped covering is how a
@@ -232,6 +254,9 @@ def main():
print(" their configs resolve; what is no longer asserted is the BINARY's presence.") print(" their configs resolve; what is no longer asserted is the BINARY's presence.")
bad = [] bad = []
for pin in unpinned_tags:
bad.append((pin, ["the pinned version has NO git tag — nothing can install it (D50 = A)"]))
print(" FAIL pinned %s: no git tag v%s" % (pin, pin.split("=")[1]))
for v in versions: for v in versions:
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG) pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG) raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
+23 -31
View File
@@ -1,41 +1,33 @@
{ {
"_comment": [ "_comment": [
"THE retention number for published agent artifacts. One file, read by everything that", "THE retention rule for published agent artifacts. One file, read by everything that depends on it, because a check",
"depends on it, because a check and the policy it enforces must read the same number from the", "and the policy it enforces must read the same rule from the same place or they drift — and the drift looks like a",
"same place or they drift — and the drift looks like a defect in something else.", "defect in something else (R-287, 2026-08-08/09; again R-1290, 2026-10-25).",
"", "",
"WHAT WENT WRONG WITHOUT IT (2026-08-08/09). The registry stopped serving felhom-agent", "D50 = A (operator ruling 2026-10-25, felhom.eu `09` §3 decision 262): the registry keeps the NEWEST 5 released agent",
"0.120.0 and older, while scripts/check-published-versions.py demanded that EVERY git tag", "versions, PLUS the hub's MinAgent floor, PLUS the version the hub's day-0 artifact manifest names. The operator prunes",
"still be downloadable. Both rules are individually sensible; together they are impossible.", "by hand to that rule (on 2026-10-25 he removed 0.159.0–0.160.0 and kept 0.158.1, the floor). CI asserts exactly these",
"CI went red at a commit whose own run had been green the day before, on a true finding that", "and never more. Before D50 the number was 10 (the R-267 newest-10 prune, R-287/R-291).",
"no one could act on. The red will return at the next publish unless the two read one number.",
"", "",
"HOW THE NUMBER WAS ARRIVED AT. generic_versions_kept is 10 because that is what the registry", "THE PINS ARE COPIES OF HUB STATE, because CI cannot read the hub without a secret it must not hold. The session that",
"keeps: the deleter was ESTABLISHED on 2026-08-10 (R-287) — the newest-10 prune of generic packages", "records an agent vouch (day0_agent) or declares a new MinAgent with a floor (hub_min_agent) updates `pinned` in the",
"run under R-267 (felhom-agent and felhom-golden generic held exactly 10 afterwards). Until then this", "same commit (felhom-build-deploy skill, agent release). A pin that names a version with no tag is a FAILURE, never a",
"file called the 10 an observed state with no located ruling; that is superseded (corrected", "skip — a pin nobody can install is the defect the pin exists to catch.",
"2026-10-05, R-291). Container packages are not pruned by that rule.",
"", "",
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest", "SO THIS FILE IS A FLOOR, NOT A LICENCE: it says what CI may assume is still downloadable. It does not authorise",
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the", "deleting anything else.",
"operator should confirm or replace the number — at which point this file changes and both",
"readers follow it in the same commit.",
"", "",
"THE DEEPER BOUND, recorded so a future session does not have to re-derive it: the principled", "NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config files from",
"limit is the hub's vouched min_agent floor (0.127.0 on 2026-08-09). Nothing can install an", "raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that version at all."
"agent below it — the hub refuses to vouch one and boxes update to the floor — so a released",
"version below the floor being un-downloadable costs nothing real. Bounding on the floor would",
"be better than bounding on a count, and it needs the gate to read the hub, which is network",
"the gate does not have today. Filed as the follow-up in R-287.",
"",
"NEVER retire a git TAG to satisfy this. felhom-host-install.sh fetches an agent's config",
"files from raw/tag/v<version>/configs/, so deleting a tag retires the ability to install that",
"version at all — a strictly worse act than an un-downloadable binary."
], ],
"generic_versions_kept": 10, "generic_versions_kept": 5,
"readers": [ "readers": [
"scripts/check-published-versions.py — bounds its assertion to the newest N versions" "scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
], ],
"recorded": "2026-08-09", "recorded": "2026-10-29",
"recorded_by": "CC 2026-08-09; the number's source (the R-267 newest-10 prune, established 2026-08-10 by R-287) recorded 2026-10-05 (R-291)" "recorded_by": "CC 2026-10-30: day0_agent 0.171.0 (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.171.0"
}
} }
+31 -1
View File
@@ -179,13 +179,24 @@ def published_cases(base):
TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3 TAGS = ["v0.150.%d" % i for i in range(3)] # inside any retention window >= 3
VERS = [t[1:] for t in TAGS] VERS = [t[1:] for t in TAGS]
# D50 = A: the gate also asserts the PINNED versions. The decoys run against their own policy file (the real pins
# name versions the fake Gitea does not hold); PINS is what each case writes there.
pol_dir = tempfile.mkdtemp(prefix="retention-decoy-")
pol = os.path.join(pol_dir, "retention-policy.json")
PINS = {}
def case(name, setup, expect_rc, must=()): def case(name, setup, expect_rc, must=()):
FAKE.reset() FAKE.reset()
FAKE.tags = list(TAGS) FAKE.tags = list(TAGS)
FAKE.packages = set(VERS) FAKE.packages = set(VERS)
FAKE.raw = set(VERS) FAKE.raw = set(VERS)
PINS.clear()
setup() setup()
rc, out = run([sys.executable, gate], ROOT, child_env(base)) with io.open(pol, "w", encoding="utf-8") as fh:
json.dump({"generic_versions_kept": keep, "pinned": dict(PINS)}, fh)
env = child_env(base)
env["FELHOM_RETENTION_POLICY"] = pol
rc, out = run([sys.executable, gate], ROOT, env)
if not FAKE.hits: if not FAKE.hits:
fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name) fails.append("published/%s: the gate never asked the fake Gitea - the seam is not wired" % name)
report("published: " + name, rc, out, expect_rc, must) report("published: " + name, rc, out, expect_rc, must)
@@ -220,6 +231,25 @@ def published_cases(base):
case("BY DESIGN: a version older than the retention window is not asserted", retired, 0, case("BY DESIGN: a version older than the retention window is not asserted", retired, 0,
("NOT ASSERTED", "0.9.0")) ("NOT ASSERTED", "0.9.0"))
def pinned_old_ok():
FAKE.tags = ["v0.8.0"] + ["v0.10.%d" % i for i in range(keep)]
FAKE.packages = set(t[1:] for t in FAKE.tags)
FAKE.raw = set(t[1:] for t in FAKE.tags)
PINS["hub_min_agent"] = "0.8.0"
case("GENUINE (D50): a pinned version older than the window is asserted and downloads", pinned_old_ok, 0,
("verifying: 0.8.0", "ALL RELEASED VERSIONS INSTALLABLE"))
def pinned_old_gone():
pinned_old_ok()
FAKE.packages.discard("0.8.0")
case("FACT (D50): a pinned version older than the window whose package 404s", pinned_old_gone, 1,
("FAIL v0.8.0", "binary NOT downloadable"))
def pinned_no_tag():
PINS["day0_agent"] = "0.150.9"
case("FACT (D50): a pin that names a version with no tag", pinned_no_tag, 1,
("FAIL pinned day0_agent=0.150.9", "no git tag"))
def five_hundred(): def five_hundred():
FAKE.tags_status = 500 FAKE.tags_status = 500
case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",)) case("INCONCLUSIVE: the tags api answers 500", five_hundred, 2, ("INCONCLUSIVE",))