Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cc63dd5204 | ||
|
|
91a2924ad0 | ||
|
|
9edaa8c082 | ||
|
|
5cebb766e3 | ||
|
|
776025c824 | ||
|
|
41d5eb8051 | ||
|
|
8bd137787e | ||
|
|
39db03065a | ||
|
|
ba409de4a2 | ||
|
|
a4f1280399 | ||
|
|
ecbb0cc479 | ||
|
|
50eeccc986 | ||
|
|
808cdb36ad | ||
|
|
84726c9f27 | ||
|
|
bd18347c1c | ||
|
|
98bd1b71e9 | ||
|
|
6578a8b91e | ||
|
|
ca15a31690 | ||
|
|
bdea4694a4 | ||
|
|
db25eb0e71 | ||
|
|
637c92c762 | ||
|
|
63e63ee8bb | ||
|
|
6b0b093199 | ||
|
|
f4e63a55da | ||
|
|
91bd96c755 | ||
|
|
6f933d6ee3 | ||
|
|
5ddd124cfc | ||
|
|
3055bc54c1 | ||
|
|
007199342f |
No files matched your search
+177
-1
@@ -1,4 +1,180 @@
|
|||||||
## Unreleased (next: v0.167.0) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — committed, not released
|
## Unreleased (2026-11-11) — docs only: a repository address holds no credential (R-925, Part C of the clock/end-dates run)
|
||||||
|
|
||||||
|
- `scripts/release-agent.sh` header: the tag push needs `GITEA_GIT_TOKEN` from the secret store (`sec run`). The origin
|
||||||
|
address of CC's clone on DooPlex held a login and a token until 2026-11-11; git now asks the helper
|
||||||
|
`felhom.eu/scripts/git-credential-sec.sh`. No code change, no version change, no release.
|
||||||
|
|
||||||
|
## v0.173.0 (2026-11-10) — R-1368 part 2: the self-update folder keeps what it needs and no more — RELEASED and VOUCHED 2026-11-10 (release-agent.sh from a clean clone of `776025c`; agent sha256 752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea, bundle sha256 51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1)
|
||||||
|
|
||||||
|
- **The defect (measured 2026-11-10, space list place H12).** `Execute` downloads each new agent to
|
||||||
|
`<state dir>/selfupdate/felhom-agent-<version>` and removed that file only when the download or its checksum failed.
|
||||||
|
A good update left it for ever: 49 / 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1.
|
||||||
|
- **What the folder is needed for (read first: `configs/felhom-selfupdate-guarded`).** The root wrapper reads the staged
|
||||||
|
file once, copies it to its own root-owned folder and flips from that copy. The rollback needs `pending.json` (in the
|
||||||
|
folder) and `/usr/local/bin/felhom-agent.prev` (outside it); the commit needs `pending.json`. No staged binary.
|
||||||
|
- **The fix (`internal/selfupdate/prune.go`).** After the wrapper's `commit` returned success, `MaybeCommit` removes
|
||||||
|
every staged binary but the running version's. Nothing is removed while a pending marker exists or cannot be read,
|
||||||
|
at a plain start, after a rollback (the failed version's binary stays to be read), or when the running version is
|
||||||
|
not a version this code can name a file for. Only a regular file with exactly the name `Execute` writes is removed,
|
||||||
|
one `os.Remove` per name; a failed removal is logged and never fails the commit. A download and a prune take turns
|
||||||
|
(`stagingMu`).
|
||||||
|
- **Tests `TestPrune_*` (6).** After six updates the folder holds one binary and every bystander (unknown names, a
|
||||||
|
folder and a symlink that carry a staged name); a pending or unreadable marker refuses; only a healthy commit prunes;
|
||||||
|
the name rule follows `versionRe`; the wrapper's `commit` and `rollback` blocks name no staged binary; and the
|
||||||
|
wrapper's REAL `commit` and `rollback` verbs, run from a sandboxed copy, still put the previous binary back after
|
||||||
|
four pruned updates. **Red seen three times:** no prune call; no pending refusal; no regular-file check.
|
||||||
|
- **Not reached.** `/usr/local/bin/felhom-agent.bak-*` (place H13): root-owned, in a folder the agent cannot write, made
|
||||||
|
by a person. R-1368 part 4 stays open.
|
||||||
|
- **Corrected 2026-11-10 (measured on three boxes).** This entry said: „A box gets the clean-up at its first good update
|
||||||
|
AFTER the one that brings this code". That was wrong. The commit runs in the NEW binary, so the box cleans the folder
|
||||||
|
at the commit of the update that brings the code.
|
||||||
|
- **Delivered 2026-11-10 to the three test boxes by a signed `agent_update` each (no bundle job: `configs/` did not
|
||||||
|
change since v0.172.1).** The folder before → after, read on each box: demo-hp 49 entries / 722,596 KB → 1 / 16,028 KB
|
||||||
|
(commit 14:59:19 UTC, `removed=49`); demo-felhom 48 / 706,968 KB → 1 / 16,028 KB (commit 14:51:50 UTC, `removed=48`);
|
||||||
|
Tester 1 33 / 497,244 KB → 1 / 16,028 KB (commit 15:00:19 UTC, `removed=33`). After each: `pending.json` absent,
|
||||||
|
`/usr/local/bin/felhom-agent.prev` = 0.172.1, the unit active with no restart.
|
||||||
|
- **A rollback still works after the clean-up (Tester 1, 2026-11-10 15:14 UTC).** A signed `agent_update` to 0.172.1
|
||||||
|
(the vouched version; it has no prune code) was installed with its pending marker. One `SEGV` in the 60-second dwell
|
||||||
|
→ the unit's `OnFailure` → the wrapper's `rollback` → 0.173.0 ran again 9 seconds after the install, `pending.json`
|
||||||
|
gone, no failed unit. The failed version's staged file stays (by design, until the next good update): the folder
|
||||||
|
holds 2 files, 32,040 KB. Evidence: felhom.eu `documentation/audits/release-space-fixes-2026-11-10/agent/`.
|
||||||
|
- **Vouched by the operator 2026-11-10 16:29:16 CET** (read from the hub log: `Artifact manifest set: agent=0.173.0`, the
|
||||||
|
bundle sha = the release). `pinned.day0_agent` = 0.173.0 (D50).
|
||||||
|
|
||||||
|
## v0.172.1 (2026-11-07) — proxmox-grub joins the boot-chain lane: the nightly pve step stops being refused R6 — RELEASED and VOUCHED 2026-11-07 (release-agent.sh from `39db030`; agent sha256 505a0594a2fb117f359e329a0b7ae2c7a6902b74c51f52c1883145f3c2ac72f8, bundle 3e7acaee4500893bfaf567b1d427790ef89bf41f08cb6a1419ee528aeb40ddc3)
|
||||||
|
|
||||||
|
- **The defect (measured on demo-felhom, 2026-11-05, 11-06, 11-07).** Proxmox published `proxmox-grub` 2.12-9+pmx4. It
|
||||||
|
depends on `grub-efi-amd64` of the same version. `HOST_SLOW_RE` matches `grub` only at the START of a name, so
|
||||||
|
`proxmox-grub` landed in the ring-0 pve plan; the simulation then pulled `grub-efi-amd64`, which is not in the plan,
|
||||||
|
and `felhom-os-apply` refused the WHOLE pve step (R6 „the plan would touch grub-efi-amd64") every night — 34 Proxmox
|
||||||
|
userspace upgrades held with it, and an `os_update_failed` error mail each morning. The same class as the 2026-10-09
|
||||||
|
`proxmox-secure-boot-support` fix. Read-only proof on the box: `apt-get -s install --only-upgrade proxmox-grub` →
|
||||||
|
`Inst grub-efi-amd64 …pmx4`.
|
||||||
|
- **The fix.** `proxmox-grub` joins `HOST_SLOW_RE`: GRUB's meta-package now waits with the rest of the boot chain (the
|
||||||
|
slow lane, the operator's signed step), and the pve step runs without it.
|
||||||
|
- **Test** `test_ring0_pending_pve_leaves_proxmox_grub_out`; red seen before the fix (proxmox-grub entered the plan), green
|
||||||
|
after; the wrapper suite 149 of 149.
|
||||||
|
- **Delivered 2026-11-07 (operator, signed jobs):** demo-felhom, demo-hp, Tester 1 — `agent_update` committed
|
||||||
|
(23:05/23:13/23:13 CET), then the bundle: `BUNDLE DONE agent=0.172.1 written=1 same=30 self-check=ok`, capability probe
|
||||||
|
77/77 on each. `pinned.day0_agent` = 0.172.1 (D50).
|
||||||
|
- **Delivery.** `configs/` changed, so a box needs BOTH signed jobs: `agent_update` 0.172.1, then `agent_config_update`
|
||||||
|
0.172.1 with the bundle sha. Not vouched by CC. Bundle vs 0.170.0 (what the boxes run): no path added or removed, only
|
||||||
|
`felhom-os-apply` changed — so no step bundle is needed (R-880). Both downloads re-checked by sha.
|
||||||
|
|
||||||
|
## v0.172.0 (2026-11-03) — R-1302: the first grant of a tier is no loss — RELEASED and VOUCHED 10:18:10 CET (release-agent.sh from a clean clone of `ecbb0cc`; agent sha256 2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7, config bundle cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce; configs/ unchanged since 0.171.0, so the signed delivery was agent_update alone)
|
||||||
|
|
||||||
|
- **R-1302 — the first grant of a tier is no loss.** A fresh install arms the whole-guest tier before its storage exists, so the
|
||||||
|
first probe found no grant, repaired it and logged ERROR „GRANT WAS MISSING … investigate the loss (R-190)" plus a DEGRADED
|
||||||
|
capability, on every new box (VM 9375, 2026-10-26). The agent now remembers which tiers have EVER held their grant
|
||||||
|
(`<state dir>/store-grant-seen.json`, written by a healthy probe). A first grant logs INFO „granted" and reports healthy; a
|
||||||
|
grant that existed and vanished keeps R-190's ERROR and its one degraded cycle. A doubt keeps the ERROR: an unreadable or
|
||||||
|
corrupt file, or no file path. A box updated to this agent records its tiers at the first healthy probe, so a loss in the
|
||||||
|
one probe before that would read as a first grant. Tests `TestR1302_*`; red seen with the first-grant branch removed and on
|
||||||
|
three more mutants.
|
||||||
|
|
||||||
|
## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
|
||||||
|
|
||||||
|
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
|
||||||
|
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
|
||||||
|
chain (demo-hp's PBS vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test, two
|
||||||
|
controller updates). No failed probe, no 5-s timeout; slowest 303 ms (demo-hp, during the PBS vzdump). The only
|
||||||
|
„unhealthy" was the controller stopped by hand for a proof. A busy night never came near ONE failed probe.
|
||||||
|
- **So the alarm becomes a restart.** At `controllerUnresponsiveConfirm` = 6 unhealthy sweeps the alarm is raised, as
|
||||||
|
before. At `controllerUnresponsiveRestart` = 10 (2 minutes later; ~6.5 minutes after the first failed probe) the agent
|
||||||
|
restarts the controller's bootstrap unit — unless the controller's busy file is fresh (a backup, a restore, an
|
||||||
|
update, an export) or cannot be read, or the brake refuses (3 restarts in 15 minutes → the crash-loop pause, which
|
||||||
|
the hub mails). A swap, a parked guest, a locked guest and a vzdump keep resetting the count, as before. A refusal
|
||||||
|
keeps the alarm and is logged once per reason. After a restart the count starts again from zero.
|
||||||
|
- **The stanza** carries `unresponsive_restart_after_s` (120: the hub's alarm text says when the restart comes) and
|
||||||
|
`unresponsive_restart_at` (equals `last_restart_at` for that act; the hub mails `controller_unresponsive_restart`).
|
||||||
|
- **Refactor, no behaviour change:** the not-running restart's brake and its record are now `supervisorBrakeAllows`
|
||||||
|
and `recordSupervisorRestart`, shared by both restarts. The busy-file stale log line no longer says „operator reboot".
|
||||||
|
- Tests `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`, `…Guards` (now also: never restarted),
|
||||||
|
`…RestartRefusals` (busy fresh, busy unreadable, the brake, a crash-loop pause), `…StaleBusyFileDoesNotShield` —
|
||||||
|
red first (build failure, then 3 behaviour failures); 7 mutants, each killed (one survivor — the count reset after a
|
||||||
|
restart — found and pinned).
|
||||||
|
|
||||||
|
## v0.170.0 (2026-10-29) — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list — RELEASED (release-agent.sh from a clean clone of `ca15a31`; agent sha256 da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404, config bundle 03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543; signed agent_config_update + agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-29 18:57:48 CET; `pinned.day0_agent` = 0.170.0)
|
||||||
|
|
||||||
|
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
|
||||||
|
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
|
||||||
|
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
|
||||||
|
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
|
||||||
|
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
|
||||||
|
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
|
||||||
|
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
|
||||||
|
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
|
||||||
|
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
|
||||||
|
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
|
||||||
|
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
|
||||||
|
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
|
||||||
|
a racing start would fail the reboot's own start half.
|
||||||
|
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
|
||||||
|
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
|
||||||
|
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
|
||||||
|
restarted your box at <time>".
|
||||||
|
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
|
||||||
|
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
|
||||||
|
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
|
||||||
|
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
|
||||||
|
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
|
||||||
|
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
|
||||||
|
|
||||||
|
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
|
||||||
|
|
||||||
|
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
|
||||||
|
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
|
||||||
|
answers on the next report, `operator_action_results: [{id, outcome, message}]` (`hub.HostReport`), which it requests
|
||||||
|
at once through the cap-1 report trigger. CLOSED list: `restart_controller <vmid>` only — anything else refused, nothing
|
||||||
|
called. The act is the supervisor's (`systemctl restart felhom-controller-bootstrap.service` in the guest, existing
|
||||||
|
grants — no sudoers change, no config bundle) with its guards: provisioned guest, guest running, the guest list and the
|
||||||
|
vzdump state readable, not parked, no swap, not locked, no vzdump. The thrash brake counts operator restarts with the
|
||||||
|
supervisor's: 3 in 15 minutes → the press is refused, and presses stay refused for 30 minutes. The result says how long
|
||||||
|
the controller took to answer (Docker's health check: healthy), or `failed` when it did not within 3 minutes. One
|
||||||
|
`audit: operator action` line per decision. Works with the controller stopped. Tests `TestOpActions_*` (closed list,
|
||||||
|
unknown/bad arg refused, restart + answer time + report trigger + once per id + re-send until dropped, stopped
|
||||||
|
controller, every guard, the brake across supervisor and operator restarts, no healthy answer = failed) and
|
||||||
|
`TestMainWiresTheAgentOperatorActions` (AST) — each seen failing by a mutation.
|
||||||
|
- **R-1344 — a controller that RUNS but does not answer: ALARM ONLY.** Each sweep also reads Docker's health status for a
|
||||||
|
running controller (the swap's `docker inspect -f` grant); 6 consecutive „unhealthy" sweeps (3 minutes, on top of
|
||||||
|
Docker's own ~90 s) start an episode: one ERROR line, the stanza's `unresponsive` / `unresponsive_since` (the hub raises
|
||||||
|
`controller_unresponsive`, operator-only). NOT restarted for it: the brief allows the restart only after a measurement
|
||||||
|
shows a busy night never reads like this, and that measurement runs through tonight (felhom.eu R-1344). Not counted
|
||||||
|
during a swap, a parked or locked guest, or a vzdump; healthy / starting / no health check / an unreadable status reset
|
||||||
|
the count. Tests `TestSupervisor_Unresponsive*` (red by mutation: the vzdump guard, the threshold).
|
||||||
|
|
||||||
|
## v0.168.0 (2026-10-29) — R-528: the host's per-container OOM counter (`GET /guest/oom-kills`); R-1254: a lost or late escrow answer is asked again before it is worded — RELEASED (release-agent.sh from a clean clone of `91bd96c`; agent sha256 1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca, bundle 3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8; no configs/ change since v0.167.0 → binary-only signed updates to demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 10:21:56 CET**; Day-0 pin set (D50)
|
||||||
|
|
||||||
|
### R-528
|
||||||
|
|
||||||
|
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
|
||||||
|
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
|
||||||
|
could not fire. The kernel counts every kill per container cgroup, and the host can read it.
|
||||||
|
- **`internal/localapi/guest_oom.go`:** `GET /guest/oom-kills` (withGuest — the token's guest only; a cross-guest
|
||||||
|
`?vmid=` is 403) reads `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex>.scope/memory.events` (and
|
||||||
|
`ns/docker/<id>/` for the cgroupfs driver), line `oom_kill N`, and answers `{vmid, available, containers: {id: n}}`.
|
||||||
|
A guest with no cgroup folder → `available: false`. Read-only, no sudo: the files are 0444 and the non-root
|
||||||
|
felhom-agent user reads them (measured on Tester 1's host 2026-10-28). A container's scope is new at each start, so the
|
||||||
|
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
|
||||||
|
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
|
||||||
|
|
||||||
|
### R-1254 (the ceremony half, review round 2 HUNT-3 H2)
|
||||||
|
|
||||||
|
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
|
||||||
|
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
|
||||||
|
hub's current package opens only with the NEW code.
|
||||||
|
- **Now (`internal/localapi/escrow_ceremony.go`):** after a failed upload the agent asks the hub (`HubHolds`, 15 s,
|
||||||
|
outside the lock, the slot still marked claiming): the hub holds THIS package → a success (the code is handed out /
|
||||||
|
the confirm completes; a WARN line names the lost answer); the hub holds another package → 502 and the old sentence,
|
||||||
|
now true; the hub cannot be asked → **504** with a sentence that promises nothing about either code, and the wait goes
|
||||||
|
on (a retry decides). `cmd/felhom-agent`: `HubHolds` compares the pending package's identity blob with the one the hub
|
||||||
|
serves back (`GET /api/v1/hosts/<id>/escrow`, the read-back the agent already uses; ciphertext only, nothing logged);
|
||||||
|
a package without an identity blob cannot be compared → unknown. Tests `TestR1254_*` (localapi, red before the fix:
|
||||||
|
502 while the hub held the new package) and `TestR1254_HubHoldsComparesTheIdentityBlob` (red-proved by mutation).
|
||||||
|
- Needs controller ≥ the R-1254 controller commit (2026-10-28 night) to word 504 for the household; an older controller
|
||||||
|
shows its generic „A kód lekérése nem sikerült." (claim) / its 502 sentence (confirm) for it.
|
||||||
|
|
||||||
|
## v0.167.0 (2026-10-27) — the BYO SSH clean-up in the SAME update (R-1285); a reinstalled box's whole-guest tier is due at once (R-1232); Go 1.27.3 (R-972) — RELEASED (release-agent.sh from a clean clone; agent sha256 28bdd282…d826f0, bundle 9d034416…ed1268; signed updates committed on demo-felhom, Tester 1, demo-hp; VOUCHED by the operator 2026-10-27, Day-0 pin set)
|
||||||
|
|
||||||
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
|
- **R-972: Go 1.27.3** (`go.mod`; released 2026-10-19 with a security fix to `crypto/x509`). `govulncheck` v1.8.0 before:
|
||||||
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
|
1 called vulnerability (GO-2026-6809, `crypto/x509@go1.27.2`); after: 0. `golang.org/x/crypto` v0.58.0 is already the
|
||||||
|
|||||||
+45
@@ -1,5 +1,50 @@
|
|||||||
# CONTEXT — felhom-agent working state
|
# CONTEXT — felhom-agent working state
|
||||||
|
|
||||||
|
> **2026-11-10 (afternoon, attended) — v0.173.0 RELEASED and VOUCHED (16:29:16 CET); on the three test boxes.** sha `752b2727…`, bundle `51ebc1f4…` (`configs/` unchanged since 0.172.1, so no bundle job). Measured: a box cleans its self-update folder at the commit of the update that BRINGS the code (49 → 1, 48 → 1, 33 → 1 entries; about 16 MB stays). The line below („at the first good update AFTER that release") was wrong. A rollback after the clean-up was proven on Tester 1 with a signed update to 0.172.1 and one crash in the dwell. `pinned.day0_agent` = 0.173.0. Open: the `.bak-*` files (R-1368 part 4).
|
||||||
|
|
||||||
|
> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: the self-update folder cleans itself (R-1368 part 2).** After a healthy `commit` the agent removes every staged binary but the running version's (`internal/selfupdate/prune.go`; tests `TestPrune_*`, the wrapper's real `rollback` verb run in a sandbox). No version bump, no build, no vouch: it rides the next agent release. A box cleans its folder at the first good update AFTER that release. Not reached: `/usr/local/bin/felhom-agent.bak-*` (root-owned, R-1368 part 4). The old-kernel clean-up is a design only (felhom.eu `11-os-updates.md` §5.11.1, proven by hand on demo-hp); nothing of it is in this repo.
|
||||||
|
|
||||||
|
> **2026-11-07 (attended) — v0.172.1 RELEASED and VOUCHED; on the three test boxes (agent + bundle, 77/77).** `proxmox-grub` joins `HOST_SLOW_RE` (the N100's ring-0 pve step was refused R6 three nights: proxmox-grub pmx4 pulls grub-efi-amd64). sha `505a0594…`, bundle `3e7acaee…` (only `felhom-os-apply` changed vs 0.170.0, no step bundle). Done: vouch, both signed jobs on the three boxes, `pinned.day0_agent` = 0.172.1. Next: the N100's pve step on its next night (positive line: `os-apply: DONE` for `layer=pve:9201`, no R6).
|
||||||
|
|
||||||
|
> **2026-11-03 (day, attended) — v0.172.0 RELEASED and VOUCHED (10:18:10 CET), R-1302.** `2c0222cd…`, bundle `cc34b0bc…`, `pinned.day0_agent` = 0.172.0; on the three test boxes by signed `agent_update`. The appliance path pre-positions the grant, so a plain first start logs neither line; staged live both ways (felhom.eu `audits/release-2026-11-03/part3/`).
|
||||||
|
>
|
||||||
|
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
|
||||||
|
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
|
||||||
|
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
|
||||||
|
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
|
||||||
|
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
|
||||||
|
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
|
||||||
|
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
|
||||||
|
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
|
||||||
|
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
|
||||||
|
|
||||||
|
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
|
||||||
|
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
|
||||||
|
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
|
||||||
|
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
|
||||||
|
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
|
||||||
|
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
|
||||||
|
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
|
||||||
|
|
||||||
|
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
|
||||||
|
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
|
||||||
|
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
|
||||||
|
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
|
||||||
|
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
|
||||||
|
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
|
||||||
|
|
||||||
|
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
|
||||||
|
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
|
||||||
|
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
|
||||||
|
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
|
||||||
|
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
|
||||||
|
|
||||||
|
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
|
||||||
|
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
|
||||||
|
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
|
||||||
|
> build (felhom.eu `audits/night-2026-10-28/r528-oom/`). Escrow ceremony: after a failed upload `HubHolds` compares the
|
||||||
|
> hub's identity blob with the new package (`hubHoldsEscrowPackage`, cmd) — stored / not stored / unknown (504).
|
||||||
|
|
||||||
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
|
> **2026-10-26 — R-1285 on `main` (`b1bc63d`), NOT released (operator: „not today").** The new agent runs the bundle
|
||||||
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
|
> policy pass once after its self-update commits (`selfupdate.Manager.SetOnCommitted` → `PolicyPassAfterUpdate`). Live
|
||||||
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
|
> proof needs a BYO box coming from 0.164.0. Tuesday 2026-10-27: the weekly line test's first run against a seeding box
|
||||||
|
|||||||
@@ -1,21 +1,56 @@
|
|||||||
# REPORT — R-1285: the BYO SSH clean-up in the same update — committed, NOT released (2026-10-26 afternoon)
|
# REPORT — v0.173.0 released, vouched and delivered to the three test boxes (2026-11-10, attended)
|
||||||
|
|
||||||
**Plain words first.** A BYO box coming from 0.164.0 kept our SSH file one release too long: in the normal update the
|
**Plain words first.** Each agent update left its downloaded file on the box for ever. Version 0.173.0 removes the old
|
||||||
OLD agent installs the config bundle, so the new agent's clean-up pass never ran on it. Now the new agent runs that pass
|
files after an update that ended well. It is released and it runs on the three test boxes. The folders went from
|
||||||
once, right after its self-update commits. Not released today (operator: „not today"); its live proof needs a BYO box on
|
49, 48 and 33 files to 1 file each. A rollback still works after the clean-up. The operator vouched it at 16:29 CET.
|
||||||
0.164.0, which rides the next agent release.
|
|
||||||
|
|
||||||
## The fix (`b1bc63d`, CI 2363 ✓)
|
The full report of the run is in `felhom.eu`: `REPORT-release-space-fixes-2026-11-10.md`.
|
||||||
|
|
||||||
- `internal/selfupdate/commit.go`: `Manager.SetOnCommitted` — a hook run once after „update committed".
|
## One claim that was wrong
|
||||||
- `internal/osupdate/bundle.go`: `ConfigUpdateExecutor.PolicyPassAfterUpdate` (the wrapper's `bundle-policy` mode, host
|
|
||||||
layer, no envelope); `cmd/felhom-agent/main.go` `wireBundlePolicyAfterCommit` hooks it on.
|
|
||||||
- The smaller of the two fixes: agent-only — holding the bundle job until the commit would need new hub queue logic.
|
|
||||||
|
|
||||||
## Proof
|
The CHANGELOG entry and the report of the morning said: a box cleans its folder at the first good update AFTER the
|
||||||
|
release. That was wrong. The commit runs in the new binary. So the box cleans the folder at the commit of the update
|
||||||
|
that brings the code. Measured on three boxes; the CHANGELOG entry is corrected.
|
||||||
|
|
||||||
- `cmd/felhom-agent/r1285_policy_after_commit_test.go`, in the real order: a pending marker for the running version →
|
## The release
|
||||||
`commit`, then exactly one `bundle-policy` plan; no marker / a mismatched marker → nothing. Red-proved (the hook call
|
|
||||||
removed → commit only, no pass — the demo-hp symptom of 2026-10-26).
|
| | |
|
||||||
- **Not live:** only a box coming FROM 0.164.0 can show it (0.165.0's wrapper already applies D43 at plan time), which
|
|---|---|
|
||||||
needs a signed downgrade chain on a BYO box.
|
| Version | 0.173.0, `scripts/release-agent.sh` from a clean clone of `776025c8` (tag `v0.173.0`) |
|
||||||
|
| Agent sha256 | `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea` |
|
||||||
|
| Bundle sha256 | `51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` |
|
||||||
|
| CI | `776025c8`: run 2608, `gates #311`, success (read by commit on the public Actions page) |
|
||||||
|
| Delivery | one signed `agent_update` per box. No bundle job: `configs/` did not change since v0.172.1 |
|
||||||
|
| Vouch | the operator, 2026-11-10 16:29:16 CET. Read from the hub log: `Artifact manifest set: agent=0.173.0`, bundle sha = the release. `pinned.day0_agent` = 0.173.0 (D50) |
|
||||||
|
|
||||||
|
## The folder `/var/lib/felhom-agent/selfupdate`, before and after
|
||||||
|
|
||||||
|
| Box | Before (on 0.172.1) | After the commit of 0.173.0 | Commit (UTC) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| demo-hp | 49 entries, 722,596 KB | 1 entry, 16,028 KB | 14:59:19, `removed=49` |
|
||||||
|
| demo-felhom | 48 entries, 706,968 KB | 1 entry, 16,028 KB | 14:51:50, `removed=48` |
|
||||||
|
| Tester 1 | 33 entries, 497,244 KB | 1 entry, 16,028 KB | 15:00:19, `removed=33` |
|
||||||
|
|
||||||
|
After each commit: `pending.json` absent, `/usr/local/bin/felhom-agent.prev` = 0.172.1, the unit active, no restart.
|
||||||
|
|
||||||
|
## The rollback proof (Tester 1)
|
||||||
|
|
||||||
|
1. A signed `agent_update` to 0.172.1 (the vouched version) was uploaded at 15:01:19 UTC. The box took it at its next
|
||||||
|
report, 15:14:31 UTC. The wrapper installed 0.172.1 and wrote the pending marker.
|
||||||
|
2. At 15:14:40 UTC, inside the 60-second dwell, one `systemctl kill -s SEGV` hit the agent's main process.
|
||||||
|
3. The unit's `OnFailure` ran the wrapper's `rollback`. Journal: `rolled back to previous binary and restarted
|
||||||
|
felhom-agent.service`. 0.173.0 started at 15:14:40 UTC.
|
||||||
|
4. Read at 15:15:56 UTC: agent 0.173.0, active, `pending.json` absent, no failed unit. The folder holds 2 files
|
||||||
|
(32,040 KB): the running version's and the failed 0.172.1. The failed file stays by design, until the next good
|
||||||
|
update.
|
||||||
|
|
||||||
|
This is the path the repo already has. No second version was released for the proof.
|
||||||
|
|
||||||
|
## What stays open
|
||||||
|
|
||||||
|
- `/usr/local/bin/felhom-agent.bak-*` (R-1368 part 4): demo-felhom holds 11 such files, demo-hp 1. The agent cannot
|
||||||
|
reach them.
|
||||||
|
- After a rollback, `/usr/local/bin/felhom-agent.prev` holds the same version as the running binary. That is how the
|
||||||
|
wrapper works; it is not new.
|
||||||
|
|
||||||
|
Evidence: felhom.eu `documentation/audits/release-space-fixes-2026-11-10/agent/`.
|
||||||
@@ -35,7 +35,7 @@
|
|||||||
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
|
| `SudoHostOps.ListCandidateDisks` | internal/storage/candidates.go | `ListCandidateDisks(ctx) ([]CandidateDisk, error)` | enroll-candidate discovery | Fail-safe: omits anything not provably unclaimed |
|
||||||
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
|
| `antiRetargetResolveExpect` (+ `antiRetargetResolve`, `antiRetargetResolveBlank`) | internal/localapi/wipe_reresolve.go | `(durableID, expectDataBearing, resolve, derive, inspect) (device, err)` | pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect | AGENT-001 + audit D3; refuses path-only bindings; wired via `Server.reresolveWipe`/`reresolveBlank` (test-injectable) |
|
||||||
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
|
| `signedjobs.WipeExecutor.Execute` | internal/signedjobs/wipe.go | `Execute(ctx, op, params) error` | operator-signed data-bearing wipe | Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets |
|
||||||
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()` |
|
| `selfupdate.Executor` / `selfupdate.Manager` | internal/selfupdate/{executor,commit}.go | `NewExecutor(Config)` / `NewManager(ManagerConfig)` | operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper `apply`; startup dwell → `commit` | sha is the ONLY integrity root; wrapper (`felhom-selfupdate-guarded`) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). `WrapperRunner` seam. Report seam `SelfUpdatePending()`. **The staging folder cleans itself** (`prune.go`, R-1368): after a healthy `commit` only the running version's staged binary stays; a new file kept in `<state dir>/selfupdate/` must NOT be named `felhom-agent-<version>` or the prune removes it |
|
||||||
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
|
| `Gate.AuthorizeStorageWipe` | internal/reconcile/gate.go | `AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision` | tiered wipe authz | user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, `Confirmed` IGNORED by role |
|
||||||
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
|
| `Gate.Authorize` | internal/reconcile/gate.go | `Authorize(Intent, *SignedOp) Decision` | every destructive intent | role-scoping (`roleAuthorizes`) + op-to-action binding; benign passes unsigned; audits every decision |
|
||||||
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
|
| `storage.DeviceDurableID` / `ResolveDurableDevice` | internal/storage/durable_device.go | `DeviceDurableID(device) (string, error)` | WIPE-binding ids (`byid:`/`byuuid:`) | Single seam for /disks list AND gate (F20-BUG2); `ResolveDurableDevice` refuses bare paths |
|
||||||
@@ -88,7 +88,8 @@
|
|||||||
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
|
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
|
||||||
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
|
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
|
||||||
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
|
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
|
||||||
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
|
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause (`supervisorBrakeAllows` — every supervisor restart passes it; `recordSupervisorRestart` puts one on the record + R-539's slow counter); R-1344 (2026-10-30): a RUNNING controller whose Docker health reads „unhealthy" is alarmed at 6 sweeps and restarted at 10 (`restartUnresponsive`: the controller's busy file, then the brake); record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
|
||||||
|
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller`, `reboot_guest` since D65 — `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
|
||||||
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
|
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
|
||||||
|
|
||||||
### Proxmox client / hub / PBS / provisioning
|
### Proxmox client / hub / PBS / provisioning
|
||||||
@@ -97,6 +98,8 @@
|
|||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
|
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
|
||||||
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
|
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
|
||||||
|
| `Leg.runKernelCleanup` / `kernelCleanupSkip` / `Leg.KernelCleanupDryRun` + the wrapper's `Kernel.cleanup` | internal/osupdate/kernel.go; configs/felhom-os-apply (mode `kernel-cleanup`) | `kernelCleanupSkip(appliance, trigger, host, pve, kernel, blk, okStep) string` ("" = call it) | R-1368 (`11` §5.11.1): old kernel images are removed by the WRAPPER's rule; the agent decides only WHEN (the night leg, after a healthy host step, never in a night with a kernel step) | The plan names NO package (C1). Never `apt-get autoremove`. Whether a box may do it is the root-owned ring-0 mark, never the hub's ring. The hub report carries NO kernel view and no kernel version (the hub would read it as a kernel step). `--selftest=kernel-cleanup-dry-run` changes nothing. |
|
||||||
|
| `Leg.ProfileRefresh` + the wrapper's `Apply.profile_refresh` | internal/osupdate/profilerefresh.go; configs/felhom-os-apply (mode `profile-refresh`) | `ProfileRefresh(ctx, why, gate, tries, wait) string` (the outcome) | R-1388 (`11` §5.12): load every AppArmor profile from a fresh compile once per kernel the box starts on; restart only a FAILED service whose own program's cached profile was stale | The plan names nothing (F1). A stale copy is told by the CACHED FILE changing at the rewrite — never by the kernel's loaded hash (it differs between a cache load and a fresh load for multi-profile files; `lxc-containers` never compiles the same twice). Called at agent start and after a bundle install, under the heavy-op gate. |
|
||||||
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
||||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||||
|
|||||||
@@ -0,0 +1,48 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-1254 H2, the production question: does the hub's current package equal the pending upload body? Compared by the
|
||||||
|
// identity blob the hub serves back. A package without an identity, or a hub that cannot answer, is never „held".
|
||||||
|
// RED-PROOF: make hubHoldsEscrowPackage return true when the hub has any package → the „another package" case
|
||||||
|
// passes as held → FAILS.
|
||||||
|
func TestR1254_HubHoldsComparesTheIdentityBlob(t *testing.T) {
|
||||||
|
body := func(id string) []byte {
|
||||||
|
b, _ := json.Marshal(escrowUploadRequest{BlobB64: base64.StdEncoding.EncodeToString([]byte("k")),
|
||||||
|
IdentityBlobB64: base64.StdEncoding.EncodeToString([]byte(id))})
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
hubWith := func(id string, present bool, err error) func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||||
|
return func(context.Context) (*hub.IdentityEscrowResponse, error) {
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &hub.IdentityEscrowResponse{Present: present, IdentityEscrowB64: base64.StdEncoding.EncodeToString([]byte(id))}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
if held, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), body("NEW")); err != nil || !held {
|
||||||
|
t.Fatalf("the hub holds the new package: held=%v err=%v", held, err)
|
||||||
|
}
|
||||||
|
if held, err := hubHoldsEscrowPackage(ctx, hubWith("OLD", true, nil), body("NEW")); err != nil || held {
|
||||||
|
t.Fatalf("the hub holds another package: held=%v err=%v", held, err)
|
||||||
|
}
|
||||||
|
if held, err := hubHoldsEscrowPackage(ctx, hubWith("", false, nil), body("NEW")); err != nil || held {
|
||||||
|
t.Fatalf("the hub holds nothing: held=%v err=%v", held, err)
|
||||||
|
}
|
||||||
|
if _, err := hubHoldsEscrowPackage(ctx, hubWith("", false, errors.New("unreachable")), body("NEW")); err == nil {
|
||||||
|
t.Fatal("a hub that cannot answer must be an error (unknown), never „not held\"")
|
||||||
|
}
|
||||||
|
noID, _ := json.Marshal(escrowUploadRequest{BlobB64: "aw=="})
|
||||||
|
if _, err := hubHoldsEscrowPackage(ctx, hubWith("NEW", true, nil), noID); err == nil {
|
||||||
|
t.Fatal("a package without an identity blob cannot be compared — an error")
|
||||||
|
}
|
||||||
|
}
|
||||||
+158
-4
@@ -251,6 +251,8 @@ func main() {
|
|||||||
os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
|
os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
|
||||||
case "os-facts":
|
case "os-facts":
|
||||||
os.Exit(runSelftestFacts(context.Background(), cfg, logger, vmid))
|
os.Exit(runSelftestFacts(context.Background(), cfg, logger, vmid))
|
||||||
|
case "kernel-cleanup-dry-run":
|
||||||
|
os.Exit(runSelftestKernelCleanupDryRun(context.Background(), cfg, logger, vmid))
|
||||||
case "live-restore":
|
case "live-restore":
|
||||||
os.Exit(runSelftestLiveRestore(context.Background(), cfg, logger, vmid))
|
os.Exit(runSelftestLiveRestore(context.Background(), cfg, logger, vmid))
|
||||||
case "pbs-verify":
|
case "pbs-verify":
|
||||||
@@ -521,6 +523,66 @@ type storeGrantRepairer struct {
|
|||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
last map[string]time.Time // target id → last ATTEMPT (success or failure)
|
||||||
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
repaired map[string]time.Time // target id → last CONFIRMED repair (drives the report latch)
|
||||||
|
// seenPath (R-1302) is the file that remembers which tiers have EVER held the grant: only those can LOSE it. "" = no
|
||||||
|
// memory, so every missing grant reads as a loss — the old, loud behaviour.
|
||||||
|
seenPath string
|
||||||
|
}
|
||||||
|
|
||||||
|
// everHeld reports whether this tier's grant was ever seen on this box (R-1302). The direction of a doubt is „yes": an
|
||||||
|
// unreadable or corrupt file keeps the loss ERROR, so a real loss is never quietened by a broken record.
|
||||||
|
func (r *storeGrantRepairer) everHeld(target string) bool {
|
||||||
|
if r == nil || r.seenPath == "" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
seen, err := readGrantSeen(r.seenPath)
|
||||||
|
return err != nil || seen[target]
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteHeld records that this tier holds the grant. A write that fails is logged and costs one false „first grant" at most
|
||||||
|
// after a restart — the old file, or none, stands.
|
||||||
|
func (r *storeGrantRepairer) noteHeld(target string) {
|
||||||
|
if r == nil || r.seenPath == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
seen, err := readGrantSeen(r.seenPath)
|
||||||
|
if err != nil {
|
||||||
|
seen = map[string]bool{} // a corrupt file is replaced by what is true now
|
||||||
|
}
|
||||||
|
if seen[target] {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
seen[target] = true
|
||||||
|
b, _ := json.Marshal(seen)
|
||||||
|
tmp := r.seenPath + ".tmp"
|
||||||
|
if werr := os.WriteFile(tmp, b, 0o600); werr == nil {
|
||||||
|
werr = os.Rename(tmp, r.seenPath)
|
||||||
|
if werr == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
err = werr
|
||||||
|
} else {
|
||||||
|
err = werr
|
||||||
|
}
|
||||||
|
repairLogger(r).Warn("store-grant: could not record that this tier holds its grant (R-1302)", "target", target, "err", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
func readGrantSeen(path string) (map[string]bool, error) {
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return map[string]bool{}, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
if err := json.Unmarshal(b, &seen); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return seen, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
// noteRepaired latches a confirmed repair so it is reported for storeGrantRepairReportWindow.
|
||||||
@@ -631,6 +693,7 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
|||||||
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
// host-report has certainly carried it. Without this latch the repairing probe may be a
|
||||||
// log-only one and the hub never learns anything happened (measured live, see the window's
|
// log-only one and the hub never learns anything happened (measured live, see the window's
|
||||||
// comment).
|
// comment).
|
||||||
|
repair.noteHeld(targetID) // R-1302: a tier that holds its grant can lose it from now on
|
||||||
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
return storeGrantHealthyVerdict(targetID, critical, s, repair.recentlyRepaired(targetID, time.Now()))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -679,8 +742,25 @@ func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string,
|
|||||||
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
// produces exactly one alert pair and the operator learns of it. NOTHING NEW WAS BUILT: no wire
|
||||||
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
// change, no hub change, no new event type. The `Feature` text carries the explanation because
|
||||||
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
// that is the field the hub puts in the operator's e-mail (the Reason does not travel).
|
||||||
|
return storeGrantAfterRepair(targetID, critical, repair, s)
|
||||||
|
}
|
||||||
|
|
||||||
|
// storeGrantAfterRepair is the verdict once a repair is confirmed (split out for the tests). R-1302: the FIRST grant of a
|
||||||
|
// tier that never held one is no loss — on a fresh install the whole-guest tier is armed before its storage exists, so
|
||||||
|
// every new box logged „investigate the loss" and a DEGRADED capability at its first start, which teaches the reader to
|
||||||
|
// skip the line. That grant logs INFO and reports healthy. A tier seen holding its grant, or one this box cannot tell
|
||||||
|
// about, keeps R-190's ERROR and its one degraded cycle. Pinned by TestR1302_*.
|
||||||
|
func storeGrantAfterRepair(targetID string, critical bool, repair *storeGrantRepairer, healthy capability.Status) capability.Status {
|
||||||
|
if !repair.everHeld(targetID) {
|
||||||
|
repair.noteHeld(targetID)
|
||||||
|
repairLogger(repair).Info("store-grant: granted — the first grant of this tier, nothing was lost (R-1302)",
|
||||||
|
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||||
|
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||||
|
healthy.Status, healthy.Reason = capability.StatusOK, ""
|
||||||
|
return healthy
|
||||||
|
}
|
||||||
repair.noteRepaired(targetID, time.Now())
|
repair.noteRepaired(targetID, time.Now())
|
||||||
s = storeGrantRepairedVerdict(targetID, critical)
|
s := storeGrantRepairedVerdict(targetID, critical)
|
||||||
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
repairLogger(repair).Error("store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)",
|
||||||
"target", targetID, "privilege", storeGrantRequiredPriv,
|
"target", targetID, "privilege", storeGrantRequiredPriv,
|
||||||
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
"action", "felhom-backup-target-apply grant "+targetID, "confirmed_by", "re-read")
|
||||||
@@ -853,8 +933,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
|
// The runner is the DIRECT one for the same reason the sudo prober uses it: the wrapper is
|
||||||
// invoked through the privileged path, which prepends sudo itself.
|
// invoked through the privileged path, which prepends sudo itself.
|
||||||
grantRepairer := &storeGrantRepairer{
|
grantRepairer := &storeGrantRepairer{
|
||||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||||
log: logger,
|
log: logger,
|
||||||
|
seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json"), // R-1302
|
||||||
}
|
}
|
||||||
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
|
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
|
||||||
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
|
// operator-only whole_guest_backup_refused event (hub v0.159.0, R-1150) tells the operator. Late-bound like
|
||||||
@@ -1205,6 +1286,8 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
}
|
}
|
||||||
return release, nil
|
return release, nil
|
||||||
}}
|
}}
|
||||||
|
daemonCtx := ctx
|
||||||
|
profileGate := func() (func(), string, bool) { return heavyOps.TryAcquire("os-profile-refresh") }
|
||||||
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
||||||
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
||||||
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
||||||
@@ -1215,7 +1298,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
names = append(names, d.Name)
|
names = append(names, d.Name)
|
||||||
}
|
}
|
||||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||||
|
// R-1388: a new wrapper may know the profile refresh the old one refused — ask again (its own goroutine:
|
||||||
|
// it may wait for the heavy-op gate, and the signed job must not wait with it).
|
||||||
|
go osLeg.ProfileRefresh(daemonCtx, "config bundle installed", profileGate, 12, 5*time.Minute)
|
||||||
}}
|
}}
|
||||||
|
// R-1388 / R-1301 (`11` §5.12): after a start, load the security profiles from a fresh compile when the box runs
|
||||||
|
// another kernel than at the last refresh (the wrapper keeps that record and answers "not needed" otherwise).
|
||||||
|
go osLeg.ProfileRefresh(daemonCtx, "agent start", profileGate, 12, 5*time.Minute)
|
||||||
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
// R-1285: the NEW agent runs the bundle policy pass once after its self-update commits — in the normal two-job
|
||||||
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
// delivery the bundle was installed by the OLD process, which has no such pass (measured 2026-10-26).
|
||||||
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
|
wireBundlePolicyAfterCommit(selfUpdateMgr, bundleExec, logger, version)
|
||||||
@@ -1250,6 +1339,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
|
// R-978: the ceremony hands its package back; the claim uploads it with the same PUT the root
|
||||||
// subprocess used, under the same per-host key, before the code is handed out.
|
// subprocess used, under the same per-host key, before the code is handed out.
|
||||||
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
|
Upload: func(ctx context.Context, pkg []byte) error { return putEscrowBody(ctx, cfg, pkg) },
|
||||||
|
// R-1254 H2: after a failed upload, the hub is asked which package it holds before the household is told
|
||||||
|
// anything — the same per-host GET the identity read-back uses.
|
||||||
|
HubHolds: func(ctx context.Context, pkg []byte) (bool, error) {
|
||||||
|
if client == nil {
|
||||||
|
return false, fmt.Errorf("hub client not configured")
|
||||||
|
}
|
||||||
|
return hubHoldsEscrowPackage(ctx, client.FetchIdentityEscrow, pkg)
|
||||||
|
},
|
||||||
DRConfigured: func() bool {
|
DRConfigured: func() bool {
|
||||||
if drConfigured != nil {
|
if drConfigured != nil {
|
||||||
return drConfigured()
|
return drConfigured()
|
||||||
@@ -1259,6 +1356,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
|||||||
}
|
}
|
||||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||||
if localSrv != nil {
|
if localSrv != nil {
|
||||||
|
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
|
||||||
|
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
|
||||||
|
// which it requests at once through the same cap-1 trigger the storage watchdog uses.
|
||||||
|
localSrv.SetOperatorActionTrigger(func() {
|
||||||
|
select {
|
||||||
|
case storageTrigger <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
})
|
||||||
|
collector.SetOperatorActionResults(localSrv.OperatorActionResults)
|
||||||
|
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner, localSrv))
|
||||||
backupCapRefusal = localSrv.BackupCapRefusal // R-973
|
backupCapRefusal = localSrv.BackupCapRefusal // R-973
|
||||||
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
||||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||||
@@ -3366,6 +3474,32 @@ func escrowUploadBody(res escrow.CreateResult, resticPwSHA256 string) []byte {
|
|||||||
|
|
||||||
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
|
// putEscrowBody PUTs an escrow upload body to this host's hub record (the per-host key). Used by the
|
||||||
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
|
// text-mode ceremony at once, and by the daemon when the customer claims the code (R-978).
|
||||||
|
// hubHoldsEscrowPackage (R-1254 H2) reports whether the hub's CURRENT package is the upload body pkg, by its identity
|
||||||
|
// blob — the part the hub serves back (GET /api/v1/hosts/<id>/escrow). Ciphertext only; nothing is logged. A package
|
||||||
|
// without an identity blob cannot be compared: an error (the caller then promises nothing).
|
||||||
|
func hubHoldsEscrowPackage(ctx context.Context, fetch func(context.Context) (*hub.IdentityEscrowResponse, error), pkg []byte) (bool, error) {
|
||||||
|
var up escrowUploadRequest
|
||||||
|
if err := json.Unmarshal(pkg, &up); err != nil {
|
||||||
|
return false, fmt.Errorf("reading the pending package: %w", err)
|
||||||
|
}
|
||||||
|
mine, err := base64.StdEncoding.DecodeString(up.IdentityBlobB64)
|
||||||
|
if err != nil || len(mine) == 0 {
|
||||||
|
return false, fmt.Errorf("the pending package carries no identity blob to compare")
|
||||||
|
}
|
||||||
|
resp, err := fetch(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
if !resp.Present || resp.IdentityEscrowB64 == "" {
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
theirs, err := base64.StdEncoding.DecodeString(resp.IdentityEscrowB64)
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("decoding the hub's package: %w", err)
|
||||||
|
}
|
||||||
|
return bytes.Equal(mine, theirs), nil
|
||||||
|
}
|
||||||
|
|
||||||
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
|
func putEscrowBody(ctx context.Context, cfg config.Config, body []byte) error {
|
||||||
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
|
||||||
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
return fmt.Errorf("hub not configured (url/host_id/api_key)")
|
||||||
@@ -3804,10 +3938,12 @@ func (f *selftestFlag) Set(v string) error {
|
|||||||
f.mode = "os-update"
|
f.mode = "os-update"
|
||||||
case "os-facts", "live-restore": // agent v0.142.0
|
case "os-facts", "live-restore": // agent v0.142.0
|
||||||
f.mode = v
|
f.mode = v
|
||||||
|
case "kernel-cleanup-dry-run": // R-1368: what the clean-up of old kernels WOULD do; changes nothing
|
||||||
|
f.mode = v
|
||||||
case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
|
case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
|
||||||
f.mode = "wgtunnel"
|
f.mode = "wgtunnel"
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update|os-facts|live-restore)", v)
|
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|kernel-cleanup-dry-run|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update|os-facts|live-restore)", v)
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -3928,6 +4064,24 @@ func runSelftestFacts(ctx context.Context, cfg config.Config, logger *slog.Logge
|
|||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// runSelftestKernelCleanupDryRun asks the root wrapper what the clean-up of old kernels WOULD do on this box (R-1368,
|
||||||
|
// `11` §5.11.1): the keep set, the removal set, and the refusal or "not run" reason. It changes nothing.
|
||||||
|
//
|
||||||
|
// sudo -u felhom-agent felhom-agent --config … --selftest=kernel-cleanup-dry-run
|
||||||
|
func runSelftestKernelCleanupDryRun(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
|
||||||
|
px, _ := newProxmoxClient(cfg)
|
||||||
|
leg := newOSLeg(cfg, nil, px, logger)
|
||||||
|
raw, err := leg.KernelCleanupDryRun(ctx, vmid)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "selftest=kernel-cleanup-dry-run:", err)
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
var v any
|
||||||
|
_ = json.Unmarshal(raw, &v)
|
||||||
|
printJSON("kernel clean-up — DRY RUN (nothing was changed)", v)
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
// runSelftestLiveRestore is the ONE-TIME live-restore step (`09` decision 87) as a debug action — the night leg does
|
// runSelftestLiveRestore is the ONE-TIME live-restore step (`09` decision 87) as a debug action — the night leg does
|
||||||
// the same before a ring-0 Docker step. It prints the container ids before and after (they must not change).
|
// the same before a ring-0 Docker step. It prints the container ids before and after (they must not change).
|
||||||
//
|
//
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"go/ast"
|
||||||
|
"go/parser"
|
||||||
|
"go/token"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
|
||||||
|
// the server in the report loop's observer chain, give the collector its results and give the server the report
|
||||||
|
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
|
||||||
|
// RED-PROOF: seen failing before the wiring existed (all three false).
|
||||||
|
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
|
||||||
|
fset := token.NewFileSet()
|
||||||
|
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse main.go: %v", err)
|
||||||
|
}
|
||||||
|
var observed, results, trigger bool
|
||||||
|
ast.Inspect(f, func(n ast.Node) bool {
|
||||||
|
call, ok := n.(*ast.CallExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
fn, ok := call.Fun.(*ast.SelectorExpr)
|
||||||
|
if !ok {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
switch fn.Sel.Name {
|
||||||
|
case "MultiObserver":
|
||||||
|
for _, a := range call.Args {
|
||||||
|
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
|
||||||
|
observed = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "SetOperatorActionResults":
|
||||||
|
if len(call.Args) == 1 {
|
||||||
|
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
|
||||||
|
results = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "SetOperatorActionTrigger":
|
||||||
|
trigger = true
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
if !observed || !results || !trigger {
|
||||||
|
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,11 +1,14 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"go/ast"
|
"go/ast"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -415,3 +418,101 @@ func TestGrantRepair_TransitionSurvivesALaterProbe(t *testing.T) {
|
|||||||
"be missed entirely", storeGrantRepairReportWindow)
|
"be missed entirely", storeGrantRepairReportWindow)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── R-1302 — the first grant of a tier is no loss ──────────────────────────────────────
|
||||||
|
//
|
||||||
|
// A fresh install arms the whole-guest tier before its storage exists, so its first probe found no grant, repaired it and
|
||||||
|
// logged „GRANT WAS MISSING … investigate the loss" with a DEGRADED capability — on every new box (VM 9375, 2026-10-26).
|
||||||
|
//
|
||||||
|
// RED-PROOF (2026-11-02): seen failing before storeGrantAfterRepair existed (build), and with its first-grant branch removed
|
||||||
|
// TestR1302_TheFirstGrantIsNoLoss fails on „a first grant must not report a loss".
|
||||||
|
func r1302Repairer(t *testing.T, path string) (*storeGrantRepairer, *bytes.Buffer) {
|
||||||
|
t.Helper()
|
||||||
|
var buf bytes.Buffer
|
||||||
|
return &storeGrantRepairer{run: (&fakeRepairRunner{}).Run, seenPath: path,
|
||||||
|
log: slog.New(slog.NewTextHandler(&buf, nil))}, &buf
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR1302_TheFirstGrantIsNoLoss(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||||
|
r, buf := r1302Repairer(t, path)
|
||||||
|
healthy := capability.Status{Name: "pve:store-grant:felhom-pbs", Status: capability.StatusDegraded, Reason: "missing", Critical: true}
|
||||||
|
s := storeGrantAfterRepair("felhom-pbs", true, r, healthy)
|
||||||
|
if s.Status != capability.StatusOK || s.Reason != "" {
|
||||||
|
t.Fatalf("a first grant must not report a loss: status %q reason %q", s.Status, s.Reason)
|
||||||
|
}
|
||||||
|
got := buf.String()
|
||||||
|
if strings.Contains(got, "investigate the loss") || strings.Contains(got, "level=ERROR") {
|
||||||
|
t.Fatalf("a first grant must not log the loss ERROR: %s", got)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got, "level=INFO") || !strings.Contains(got, "granted") || !strings.Contains(got, "felhom-pbs") {
|
||||||
|
t.Fatalf("a first grant says so at INFO: %s", got)
|
||||||
|
}
|
||||||
|
if r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||||
|
t.Fatal("a first grant must not arm the loss report's latch")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestR1302_AGrantThatExistedAndVanishedKeepsTheError(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||||
|
first, _ := r1302Repairer(t, path)
|
||||||
|
first.noteHeld("felhom-pbs") // a healthy probe, in an earlier run of the agent
|
||||||
|
|
||||||
|
r, buf := r1302Repairer(t, path) // the agent restarted: the memory is the file, not the process
|
||||||
|
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||||
|
if s.Status != capability.StatusDegraded || !strings.Contains(s.Feature, "R-190") {
|
||||||
|
t.Fatalf("a grant that existed and vanished must keep its degraded cycle; got %q / %q", s.Status, s.Feature)
|
||||||
|
}
|
||||||
|
if got := buf.String(); !strings.Contains(got, "level=ERROR") || !strings.Contains(got, "investigate the loss") {
|
||||||
|
t.Fatalf("a grant that existed and vanished must keep its ERROR: %s", got)
|
||||||
|
}
|
||||||
|
if !r.recentlyRepaired("felhom-pbs", time.Now()) {
|
||||||
|
t.Fatal("the loss must arm the report's latch, or the hub may never hear it")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// After a first grant the tier counts as held: the NEXT disappearance is a loss.
|
||||||
|
func TestR1302_AfterTheFirstGrantALossIsALoss(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||||
|
r, _ := r1302Repairer(t, path)
|
||||||
|
storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||||
|
s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{})
|
||||||
|
if s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("the second disappearance is a loss; got %q", s.Status)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The direction of a doubt: a broken record, a repairer with no record, or no repairer at all keep the loud way.
|
||||||
|
func TestR1302_ADoubtKeepsTheError(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "store-grant-seen.json")
|
||||||
|
if err := os.WriteFile(path, []byte("{not json"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r, _ := r1302Repairer(t, path)
|
||||||
|
if s := storeGrantAfterRepair("felhom-pbs", true, r, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("a corrupt record must not quieten a loss; got %q", s.Status)
|
||||||
|
}
|
||||||
|
noMemory, _ := r1302Repairer(t, "")
|
||||||
|
if s := storeGrantAfterRepair("felhom-pbs", true, noMemory, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("no record must not quieten a loss; got %q", s.Status)
|
||||||
|
}
|
||||||
|
if s := storeGrantAfterRepair("felhom-pbs", true, nil, capability.Status{}); s.Status != capability.StatusDegraded {
|
||||||
|
t.Fatalf("no repairer must not quieten a loss; got %q", s.Status)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The wiring, read from the source: production names the file, and a healthy probe records the tier.
|
||||||
|
func TestR1302_TheMemoryIsWired(t *testing.T) {
|
||||||
|
b, err := os.ReadFile("main.go")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
src := string(b)
|
||||||
|
if !strings.Contains(src, `seenPath: filepath.Join(agentStateDir(cfg), "store-grant-seen.json")`) {
|
||||||
|
t.Fatal("main.go must give the repairer its file, or every first grant reads as a loss")
|
||||||
|
}
|
||||||
|
start := strings.Index(src, "func storeGrantStatus(")
|
||||||
|
if start < 0 || !strings.Contains(src[start:start+3000], "repair.noteHeld(targetID)") {
|
||||||
|
t.Fatal("a healthy probe must record the tier, or a real loss would read as a first grant")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
|
|||||||
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
||||||
# + container across spaces, spike-confirmed)
|
# + container across spaces, spike-confirmed)
|
||||||
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
||||||
|
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
|
||||||
|
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
|
||||||
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
|
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
|
||||||
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
|
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
|
||||||
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
|
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
|
||||||
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
|||||||
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
||||||
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
||||||
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
||||||
|
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
|
||||||
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
|
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
|
||||||
|
|
||||||
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
||||||
|
|||||||
+372
-4
@@ -97,9 +97,12 @@ OOMCHECK_SETTLE = 2
|
|||||||
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
|
||||||
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
|
||||||
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
|
||||||
|
# proxmox-grub is GRUB's Proxmox meta-package (it depends on grub-efi-amd64 of the same version); the `grub` above is
|
||||||
|
# anchored at the start, so it missed it. Measured 2026-11-05..07 on demo-felhom: its upgrade pulled grub-efi-amd64 and
|
||||||
|
# the whole pve step was refused R6 every night. Pinned by test_ring0_pending_pve_leaves_proxmox_grub_out.
|
||||||
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
|
||||||
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
|
||||||
r"proxmox-secure-boot-support)")
|
r"proxmox-secure-boot-support|proxmox-grub)")
|
||||||
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
|
||||||
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
|
||||||
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
|
||||||
@@ -152,9 +155,28 @@ KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pv
|
|||||||
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
|
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
|
||||||
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
|
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
|
||||||
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
|
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
|
||||||
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
|
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel",
|
||||||
|
"kernel-cleanup")
|
||||||
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
|
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
|
||||||
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||||
|
# The clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1). A step that ENDED in one of these phases
|
||||||
|
# and is not "good" keeps every kernel (the box fell back or reverted); any other phase is "not ended" (P1).
|
||||||
|
# The security-profile refresh (R-1388, R-1301, `11` §5.12; measured on demo-hp 2026-11-14,
|
||||||
|
# felhom.eu audits/profile-refresh-spike-2026-11-14/). AppArmor keeps one cache folder per kernel feature set; a
|
||||||
|
# profile that changed while ANOTHER kernel ran leaves an old compiled copy in this kernel's folder, with a time that
|
||||||
|
# looks newer than its source — and the kernel loads it at the next start on this kernel.
|
||||||
|
PROFILE_DIR = "/etc/apparmor.d"
|
||||||
|
PROFILE_CACHE = "/var/cache/apparmor"
|
||||||
|
PROFILE_STATE = "/var/lib/felhom-os-apply/profile-refresh.json" # 0644 root: the kernel the last refresh ran on
|
||||||
|
PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", PROFILE_DIR]
|
||||||
|
PROFILE_PATH_NAME_RE = re.compile(r"^/[A-Za-z0-9._/+-]+$") # a profile whose name is one plain program path
|
||||||
|
UNIT_NAME_RE = re.compile(r"^[A-Za-z0-9@:._-]+\.service$")
|
||||||
|
PROFILE_RESTART_MAX = 3
|
||||||
|
KERNEL_ENDED_NOT_GOOD = ("self_reverted", "revert_failed", "fell_back", "cancelled")
|
||||||
|
KERNEL_LEFTOVER_PATHS = ("/lib/modules/{v}", "/boot/vmlinuz-{v}", "/boot/initrd.img-{v}", "/boot/System.map-{v}",
|
||||||
|
"/boot/config-{v}")
|
||||||
|
PINNED_KVER_RE = re.compile(r"gnulinux-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)-advanced-")
|
||||||
|
ANY_KVER_RE = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve")
|
||||||
|
|
||||||
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
||||||
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
||||||
@@ -492,7 +514,7 @@ class Apply:
|
|||||||
def check_plan(self, plan):
|
def check_plan(self, plan):
|
||||||
mode = plan.get("mode", "apply")
|
mode = plan.get("mode", "apply")
|
||||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
|
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "bundle-policy", "agent_update",
|
||||||
"oom-check") \
|
"oom-check", "profile-refresh") \
|
||||||
+ KERNEL_MODES:
|
+ KERNEL_MODES:
|
||||||
raise Refused("R11", f"unknown mode {mode!r}")
|
raise Refused("R11", f"unknown mode {mode!r}")
|
||||||
if mode == "agent_update":
|
if mode == "agent_update":
|
||||||
@@ -507,6 +529,16 @@ class Apply:
|
|||||||
if plan.get("layer") != "host":
|
if plan.get("layer") != "host":
|
||||||
raise Refused("R11", "bundle-policy is a host-layer mode")
|
raise Refused("R11", "bundle-policy is a host-layer mode")
|
||||||
return mode, "host", 0, "bundle-policy"
|
return mode, "host", 0, "bundle-policy"
|
||||||
|
if mode == "profile-refresh":
|
||||||
|
if plan.get("layer") != "host":
|
||||||
|
raise Refused("R11", "profile-refresh is a host-layer mode")
|
||||||
|
# it takes NOTHING from the plan but dry_run: no profile name, no unit name, no package (R-1388)
|
||||||
|
for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"):
|
||||||
|
if plan.get(k):
|
||||||
|
raise Refused("F1", f"a profile-refresh plan carries {k!r} — it takes nothing from a plan but dry_run")
|
||||||
|
if not isinstance(plan.get("dry_run", False), bool):
|
||||||
|
raise Refused("F1", "dry_run must be true or false")
|
||||||
|
return mode, "host", 0, "profile-refresh"
|
||||||
layer = plan.get("layer")
|
layer = plan.get("layer")
|
||||||
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
||||||
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
||||||
@@ -537,6 +569,17 @@ class Apply:
|
|||||||
rid = plan.get("release_id", "")
|
rid = plan.get("release_id", "")
|
||||||
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
||||||
raise Refused("R11", f"release_id {rid!r} is not a plain id")
|
raise Refused("R11", f"release_id {rid!r} is not a plain id")
|
||||||
|
if mode == "kernel-cleanup":
|
||||||
|
# `11` §5.11.1, rule 4 of the release: the clean-up NEVER takes a package name (or any choice of what to
|
||||||
|
# remove) from a plan — the wrapper computes the set from the box. Pinned by test_cleanup_plan_names_refused.
|
||||||
|
for k in ("packages", "allow_new", "signed", "expect_kver", "undo", "snapshot"):
|
||||||
|
if plan.get(k):
|
||||||
|
raise Refused("C1", f"a kernel-cleanup plan carries {k!r} — the clean-up takes nothing from a plan "
|
||||||
|
f"but dry_run; the wrapper computes the set from the box")
|
||||||
|
if plan.get("select", "listed") != "listed":
|
||||||
|
raise Refused("C1", "a kernel-cleanup plan carries a select — the wrapper computes the set from the box")
|
||||||
|
if not isinstance(plan.get("dry_run", False), bool):
|
||||||
|
raise Refused("C1", "dry_run must be true or false")
|
||||||
if plan.get("allow_new"):
|
if plan.get("allow_new"):
|
||||||
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
||||||
select = plan.get("select", "listed")
|
select = plan.get("select", "listed")
|
||||||
@@ -1081,6 +1124,9 @@ class Apply:
|
|||||||
return Bundle(self).from_plan(plan)
|
return Bundle(self).from_plan(plan)
|
||||||
if self.mode == "bundle-policy":
|
if self.mode == "bundle-policy":
|
||||||
return Bundle(self).policy_pass()
|
return Bundle(self).policy_pass()
|
||||||
|
if self.mode == "profile-refresh":
|
||||||
|
self.check_appliance()
|
||||||
|
return self.profile_refresh(plan)
|
||||||
if self.mode == "agent_update":
|
if self.mode == "agent_update":
|
||||||
return self.agent_update(plan)
|
return self.agent_update(plan)
|
||||||
if self.layer in ("host", "pve", "kernel"):
|
if self.layer in ("host", "pve", "kernel"):
|
||||||
@@ -1146,6 +1192,107 @@ class Apply:
|
|||||||
self.report["oom_check"] = self.oom_check()
|
self.report["oom_check"] = self.oom_check()
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
|
||||||
|
def profile_cache(self):
|
||||||
|
"""{"<folder>/<file>": sha256} of every cached profile, every kernel's folder."""
|
||||||
|
out = {}
|
||||||
|
for d in self.r.list_dir(PROFILE_CACHE):
|
||||||
|
dp = f"{PROFILE_CACHE}/{d}"
|
||||||
|
if not self.r.isdir(dp):
|
||||||
|
continue
|
||||||
|
for f in self.r.list_dir(dp):
|
||||||
|
if f.startswith("."):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
out[f"{d}/{f}"] = sha256_hex(self.r.read_bytes(f"{dp}/{f}"))
|
||||||
|
except OSError:
|
||||||
|
out[f"{d}/{f}"] = "unreadable"
|
||||||
|
return out
|
||||||
|
|
||||||
|
def profile_refresh(self, plan):
|
||||||
|
"""Load every AppArmor profile from a FRESH compile (the cache ignored and rewritten), once per kernel the box
|
||||||
|
starts on; then restart ONLY a failed service whose own program's profile had a stale cached copy.
|
||||||
|
|
||||||
|
Changes no package and takes no name from the plan. Measured on demo-hp with both guests running: 0.5 s, no
|
||||||
|
guest or container touched, the same profiles loaded in the same modes.
|
||||||
|
|
||||||
|
WHY NOT a "loaded hash changed" check: for a file that defines several profiles the kernel's hash differs
|
||||||
|
between a cache load and a fresh load of the SAME policy (17 of 119 on demo-hp), and one file (lxc-containers)
|
||||||
|
compiles to a different binary every time. The only reliable sign of a stale copy is the CACHED FILE changing
|
||||||
|
when it is rewritten — that is what the restart rule reads."""
|
||||||
|
r, dry = self.r, plan.get("dry_run") is True
|
||||||
|
res = {"dry_run": dry, "outcome": None, "rewritten": [], "restarted": [], "not_restarted": []}
|
||||||
|
self.report["profile_refresh"] = res
|
||||||
|
rc, out, _ = r.host(["uname", "-r"], 30)
|
||||||
|
kernel = out.strip() if rc == 0 else ""
|
||||||
|
try:
|
||||||
|
rec = json.loads(r.read_file(PROFILE_STATE))
|
||||||
|
rec = rec if isinstance(rec, dict) else {}
|
||||||
|
except (OSError, ValueError):
|
||||||
|
rec = {}
|
||||||
|
res["kernel"], res["last_refresh_kernel"], res["last_refresh_at"] = kernel or "unknown", rec.get("kernel"), rec.get("at")
|
||||||
|
if not kernel:
|
||||||
|
raise Refused("F2", "the running kernel could not be read")
|
||||||
|
if not r.isdir(PROFILE_DIR) or not r.isdir(PROFILE_CACHE):
|
||||||
|
res["outcome"] = "not_needed"
|
||||||
|
res["reason"] = "this box has no AppArmor profile folder or no profile cache"
|
||||||
|
return 0
|
||||||
|
needed = rec.get("kernel") != kernel
|
||||||
|
if not needed:
|
||||||
|
res["outcome"], res["reason"] = "not_needed", f"the profiles were refreshed on this kernel ({kernel}) at {rec.get('at')}"
|
||||||
|
return 0
|
||||||
|
if dry:
|
||||||
|
res["outcome"], res["would"] = "dry_run", "refresh"
|
||||||
|
return 0
|
||||||
|
before = self.profile_cache()
|
||||||
|
t0 = time.time()
|
||||||
|
rc, out, err = r.host(list(PROFILE_REFRESH_CMD), 600)
|
||||||
|
res["seconds"] = round(time.time() - t0, 1)
|
||||||
|
if rc != 0:
|
||||||
|
res["outcome"] = "failed"
|
||||||
|
self.report["failed"] = {"rc": rc, "step": "profile-refresh", "tail": (out + err).strip().splitlines()[-3:]}
|
||||||
|
r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)")
|
||||||
|
return 3
|
||||||
|
after = self.profile_cache()
|
||||||
|
res["rewritten"] = sorted(k for k in after if before.get(k) != after[k])
|
||||||
|
# --- the narrow restart rule: a FAILED service, whose ExecStart program IS the name of a profile whose cached
|
||||||
|
# copy was stale. Never "whatever failed"; never a unit named by the plan. ---
|
||||||
|
programs = {}
|
||||||
|
for key in res["rewritten"]:
|
||||||
|
fname = key.split("/", 1)[1]
|
||||||
|
rc_n, names, _ = r.host(["apparmor_parser", "-N", f"{PROFILE_DIR}/{fname}"], 60)
|
||||||
|
if rc_n != 0:
|
||||||
|
continue
|
||||||
|
for n in names.split():
|
||||||
|
if PROFILE_PATH_NAME_RE.match(n):
|
||||||
|
programs[n] = fname
|
||||||
|
if programs:
|
||||||
|
rc_u, units, _ = r.host(["systemctl", "list-units", "--type=service", "--state=failed", "--no-legend", "--plain"], 60)
|
||||||
|
for line in (units.splitlines() if rc_u == 0 else []):
|
||||||
|
unit = (line.split() or [""])[0]
|
||||||
|
if not UNIT_NAME_RE.match(unit):
|
||||||
|
continue
|
||||||
|
_, es, _ = r.host(["systemctl", "show", "-p", "ExecStart", "--value", unit], 30)
|
||||||
|
m = re.search(r"\bpath=(\S+)", es)
|
||||||
|
prog = m.group(1) if m else ""
|
||||||
|
if prog not in programs:
|
||||||
|
continue
|
||||||
|
if len(res["restarted"]) >= PROFILE_RESTART_MAX:
|
||||||
|
res["not_restarted"].append({"unit": unit, "program": prog, "reason": f"more than {PROFILE_RESTART_MAX} units match — look at the box"})
|
||||||
|
continue
|
||||||
|
rc_r, o2, e2 = r.host(["systemctl", "restart", unit], 120)
|
||||||
|
_, act, _ = r.host(["systemctl", "is-active", unit], 30)
|
||||||
|
res["restarted"].append({"unit": unit, "program": prog, "profile_file": programs[prog], "rc": rc_r,
|
||||||
|
"active": act.strip() == "active"})
|
||||||
|
r.log(f"os-apply: PROFILE REFRESH restarted {unit} (failed; its program {prog} had a stale cached profile) "
|
||||||
|
f"rc={rc_r} now={act.strip() or 'unknown'}")
|
||||||
|
r.put_file(PROFILE_STATE, (json.dumps({"kernel": kernel, "at": _iso(r.now()), "rewritten": res["rewritten"],
|
||||||
|
"restarted": [x["unit"] for x in res["restarted"]]}, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||||
|
res["outcome"] = "refreshed"
|
||||||
|
r.log(f"os-apply: PROFILE REFRESH DONE kernel={kernel} (last {rec.get('kernel') or 'never'}) seconds={res['seconds']} "
|
||||||
|
f"rewritten={len(res['rewritten'])} ({', '.join(res['rewritten'][:8])}) restarted={[x['unit'] for x in res['restarted']]}")
|
||||||
|
return 0
|
||||||
|
|
||||||
# ---------- R-528: the memory-kill check ----------
|
# ---------- R-528: the memory-kill check ----------
|
||||||
def oom_check(self):
|
def oom_check(self):
|
||||||
"""Run one throwaway container over its memory cap in the guest and read what the engine says about it.
|
"""Run one throwaway container over its memory cap in the guest and read what the engine says about it.
|
||||||
@@ -1490,6 +1637,11 @@ class Kernel:
|
|||||||
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
|
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
|
||||||
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
|
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
|
||||||
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
|
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
|
||||||
|
kernel-cleanup remove old kernel images by the rule of `11` §5.11.1 (R-1368, decision 286). The plan names nothing
|
||||||
|
(C1); `dry_run: true` computes and reports, and changes nothing. Its own refusals: P1–P3, K1, N1,
|
||||||
|
S1, S2 (the design's table) and C1 (the plan names something). "Not run" (exit 0, nothing
|
||||||
|
changed): no ring-0 mark, no kernel step yet, the newest step did not end good, the box does not
|
||||||
|
run its newest kernel, inside a step's 20 h window.
|
||||||
kernel-status read only."""
|
kernel-status read only."""
|
||||||
|
|
||||||
def __init__(self, apply, plan):
|
def __init__(self, apply, plan):
|
||||||
@@ -1643,7 +1795,8 @@ class Kernel:
|
|||||||
self.report["kernel"] = v
|
self.report["kernel"] = v
|
||||||
return 0
|
return 0
|
||||||
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
|
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
|
||||||
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
|
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel,
|
||||||
|
"kernel-cleanup": self.cleanup}[mode]
|
||||||
rc = fn()
|
rc = fn()
|
||||||
self.report["kernel"] = self.view()
|
self.report["kernel"] = self.view()
|
||||||
return rc
|
return rc
|
||||||
@@ -1879,6 +2032,221 @@ class Kernel:
|
|||||||
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
|
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
# ---------- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----------
|
||||||
|
def kernel_images(self):
|
||||||
|
"""kver -> [installed image package names] (dpkg status ii only)."""
|
||||||
|
out = {}
|
||||||
|
for name in sorted(self.a.installed()):
|
||||||
|
m = KERNEL_IMAGE_RE.match(name)
|
||||||
|
if m:
|
||||||
|
out.setdefault(m.group(1), []).append(name)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def kver_newest(self, kvers):
|
||||||
|
best = None
|
||||||
|
for k in kvers:
|
||||||
|
if best is None or self.a.dpkg_cmp(k[:-4], "gt", best[:-4]):
|
||||||
|
best = k
|
||||||
|
return best
|
||||||
|
|
||||||
|
def read_or_none(self, path):
|
||||||
|
try:
|
||||||
|
return self.r.read_file(path)
|
||||||
|
except OSError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def cleanup_sources(self):
|
||||||
|
"""Every place that names a kernel the lane still needs, read from the box ONCE: (kver, why) pairs. This is
|
||||||
|
the keep set's only input, and K1's only input — K1 reads it again by itself (cleanup_protected)."""
|
||||||
|
st = self.state()
|
||||||
|
cfg = self.grub_cfg()
|
||||||
|
dflt = self.default_kver(cfg)
|
||||||
|
m = PINNED_KVER_RE.search(self.read_or_none(KERNEL_DEFAULT_CFG) or "")
|
||||||
|
pinned = m.group(1) if m else None
|
||||||
|
flag = self.flag() or ""
|
||||||
|
m = ANY_KVER_RE.search(flag)
|
||||||
|
m2 = ANY_KVER_RE.search(self.read_or_none(KERNEL_PIN_FILE) or "")
|
||||||
|
src = [(self.running(), "running"),
|
||||||
|
(st.get("from"), "the kernel step's `from` (the last kernel that booted well; a self-revert boots it)"),
|
||||||
|
(st.get("to"), "the kernel step's `to`"),
|
||||||
|
(st.get("default_before"), "the kernel step's `default_before`"),
|
||||||
|
(dflt if KVER_RE.match(dflt) else None, "the default in grub.cfg"),
|
||||||
|
(pinned, "the pinned default file"),
|
||||||
|
(m.group(0) if m else None, "the one-shot flag"),
|
||||||
|
(m2.group(0) if m2 else None, "an operator's boot pin")]
|
||||||
|
return st, dflt, pinned, flag, [(k, why) for k, why in src if isinstance(k, str) and KVER_RE.match(k)]
|
||||||
|
|
||||||
|
def cleanup_keep(self, images, sources):
|
||||||
|
"""The keep set: every kernel a source names, and the newest installed. (cleanup() adds the next lower
|
||||||
|
installed kernel when no step record names the last good one.)"""
|
||||||
|
keep = {}
|
||||||
|
for k, why in sources:
|
||||||
|
keep.setdefault(k, []).append(why)
|
||||||
|
newest = self.kver_newest(images)
|
||||||
|
if newest:
|
||||||
|
keep.setdefault(newest, []).append("newest installed")
|
||||||
|
return keep
|
||||||
|
|
||||||
|
def cleanup_protected(self):
|
||||||
|
"""K1's own reading (`11` §5.11.1: "K1 is the rule's own check and must stay its own"): the kernels that must
|
||||||
|
never reach the purge line, read from the box AGAIN — not taken from the keep set that built the removal set."""
|
||||||
|
_, _, _, _, sources = self.cleanup_sources()
|
||||||
|
return {k: why for k, why in sources}
|
||||||
|
|
||||||
|
def cleanup_simulate(self, names):
|
||||||
|
"""apt's simulation of exactly `names`: (rc, removes, installs, last line)."""
|
||||||
|
rc, out, err = self.a.x(APT_ENV + ["apt-get", "-s", "-q", "purge"] + names)
|
||||||
|
removes = sorted({l.split()[1] for l in out.splitlines() if l.startswith(("Remv ", "Purg ")) and len(l.split()) > 1})
|
||||||
|
installs = sorted({l.split()[1] for l in out.splitlines() if l.startswith(("Inst ", "Conf ")) and len(l.split()) > 1})
|
||||||
|
return rc, removes, installs, ((err or out).strip().splitlines() or ["?"])[-1][:200]
|
||||||
|
|
||||||
|
def cleanup(self):
|
||||||
|
a = self.a
|
||||||
|
dry = self.plan.get("dry_run") is True
|
||||||
|
res = {"dry_run": dry, "outcome": None, "not_run": [], "refusals": [], "removed": [], "leftovers": []}
|
||||||
|
self.report["kernel_cleanup"] = res
|
||||||
|
not_run, refuse = res["not_run"], res["refusals"]
|
||||||
|
# --- may this box run the clean-up at all? (ring 0 only in this release; the mark is ROOT-owned) ---
|
||||||
|
try:
|
||||||
|
ring0 = a.load_trust().get("ring0_slow_lane") is True
|
||||||
|
except Refused as e:
|
||||||
|
ring0 = False
|
||||||
|
not_run.append({"code": "no-ring0-mark", "reason": f"no usable trust record: {e.reason}"})
|
||||||
|
if not ring0 and not not_run:
|
||||||
|
not_run.append({"code": "no-ring0-mark", "reason": "this box has no root-owned ring-0 mark — the clean-up runs on "
|
||||||
|
"ring 0 only in this release"})
|
||||||
|
if a.apt_lock_held():
|
||||||
|
raise Refused("R9", "another apt/dpkg holds the lock on the host")
|
||||||
|
if not dry and not not_run:
|
||||||
|
a.repair() # repair first (`11` §5.4.1): a purge a power cut interrupted is made whole, THEN the rule is computed
|
||||||
|
# --- read the box ---
|
||||||
|
images = self.kernel_images()
|
||||||
|
running = self.running()
|
||||||
|
newest = self.kver_newest(images)
|
||||||
|
raw_state = self.read_or_none(KERNEL_STATE)
|
||||||
|
st, dflt, pinned, flag, sources = self.cleanup_sources()
|
||||||
|
phase = st.get("phase", "none") if raw_state is not None else "none"
|
||||||
|
res.update(running=running or "unknown", newest=newest, phase=phase, default=dflt, pinned_default=pinned,
|
||||||
|
installed={k: images[k] for k in sorted(images, key=lambda v: [int(x) for x in re.findall(r"\d+", v)])})
|
||||||
|
# --- "when it may run" (decision 286): not run, nothing changed ---
|
||||||
|
if raw_state is None or not st:
|
||||||
|
not_run.append({"code": "no-step", "reason": "no kernel step yet (no state file) — nothing proves which kernel booted well"})
|
||||||
|
elif phase in KERNEL_ENDED_NOT_GOOD:
|
||||||
|
not_run.append({"code": "step-not-good", "reason": f"the newest kernel step ended {phase!r}, not good — every kernel stays"})
|
||||||
|
elif phase != "good":
|
||||||
|
refuse.append({"code": "P1", "reason": f"a kernel step is in phase {phase!r} — not ended (or not a phase this wrapper knows)"})
|
||||||
|
if running and newest and running in images and running != newest:
|
||||||
|
not_run.append({"code": "not-on-newest", "reason": f"the box runs {running}, not its newest installed kernel {newest} — every kernel stays"})
|
||||||
|
times = [x for x in (_parse_iso(st.get(k)) for k in ("staged_at", "rebooted_at", "result_at")) if x is not None]
|
||||||
|
if times and self.r.now() - max(times) < KERNEL_MIN_GAP:
|
||||||
|
not_run.append({"code": "window", "reason": "a kernel step staged, booted or ended within the last 20 hours — never in its night"})
|
||||||
|
# --- the refusal table ---
|
||||||
|
if flag:
|
||||||
|
refuse.append({"code": "P2", "reason": f"the one-shot flag names {flag!r}"})
|
||||||
|
if pinned is None or pinned != dflt or pinned not in images:
|
||||||
|
refuse.append({"code": "P3", "reason": f"the pinned default {pinned!r} and grub.cfg's default {dflt!r} do not name the same installed kernel"})
|
||||||
|
if not running or running not in images:
|
||||||
|
refuse.append({"code": "K1", "reason": f"the running kernel {running or 'unknown'!r} is not an installed kernel image package"})
|
||||||
|
# --- the keep set and the removal set (exact package names) ---
|
||||||
|
keep = self.cleanup_keep(images, sources)
|
||||||
|
if running and not (st.get("to") == running and st.get("from") in images):
|
||||||
|
lower = [k for k in images if a.dpkg_cmp(k[:-4], "lt", running[:-4])] if running in images else []
|
||||||
|
lg = self.kver_newest(lower)
|
||||||
|
if lg:
|
||||||
|
keep.setdefault(lg, []).append("the next lower installed kernel (no step record names the last good one)")
|
||||||
|
removal = sorted(n for k in images if k not in keep for n in images[k])
|
||||||
|
res["kept"] = {k: keep[k] for k in sorted(keep) if k in images}
|
||||||
|
res["removal"] = removal
|
||||||
|
# K1 — its own reading of the box, against the removal set
|
||||||
|
by_name = {n: k for k, names in images.items() for n in names}
|
||||||
|
protected = self.cleanup_protected()
|
||||||
|
for n in removal:
|
||||||
|
if n not in by_name:
|
||||||
|
refuse.append({"code": "N1", "reason": f"{n} is not an installed kernel image package"})
|
||||||
|
elif by_name[n] in protected or by_name[n] == newest:
|
||||||
|
refuse.append({"code": "K1", "reason": f"{n} is a keep-kernel ({protected.get(by_name[n], 'newest installed')})"})
|
||||||
|
# S1 / S2 — apt's simulation of exactly that set
|
||||||
|
if removal and not [x for x in refuse if x["code"] in ("K1", "N1")]:
|
||||||
|
rc, removes, installs, tail = self.cleanup_simulate(removal)
|
||||||
|
res["simulation"] = {"rc": rc, "removes": removes, "installs": installs}
|
||||||
|
if rc != 0:
|
||||||
|
refuse.append({"code": "S1", "reason": "the simulation failed: " + tail})
|
||||||
|
if installs:
|
||||||
|
refuse.append({"code": "S1", "reason": f"the simulation would install or configure {', '.join(installs[:6])}"})
|
||||||
|
extra = [p for p in removes if p not in removal]
|
||||||
|
if extra:
|
||||||
|
refuse.append({"code": "S2", "reason": f"the simulation would also remove {', '.join(extra[:8])} — not in the removal set"})
|
||||||
|
missing = [p for p in removal if p not in removes]
|
||||||
|
if missing and rc == 0:
|
||||||
|
refuse.append({"code": "S1", "reason": f"the simulation does not remove {', '.join(missing[:6])}"})
|
||||||
|
self.log(f"os-apply: KERNEL CLEANUP {'DRY RUN ' if dry else ''}running={running or 'unknown'} newest={newest} phase={phase} "
|
||||||
|
f"installed={len(images)} keep={len(res['kept'])} removal={len(removal)} not_run={[x['code'] for x in not_run]} "
|
||||||
|
f"refusals={[x['code'] for x in refuse]}")
|
||||||
|
# --- the verdict ---
|
||||||
|
if dry:
|
||||||
|
res["outcome"] = "dry_run"
|
||||||
|
res["would"] = "not_run" if not_run else "refuse" if refuse else "remove" if removal else "nothing"
|
||||||
|
return 0
|
||||||
|
if not_run:
|
||||||
|
res["outcome"] = "not_run"
|
||||||
|
return 0
|
||||||
|
if refuse:
|
||||||
|
raise Refused(refuse[0]["code"], "; ".join(f"{x['code']} {x['reason']}" for x in refuse)[:600])
|
||||||
|
if not removal:
|
||||||
|
res["outcome"] = "nothing"
|
||||||
|
self.log("os-apply: KERNEL CLEANUP nothing to remove")
|
||||||
|
return 0
|
||||||
|
# --- the purge: ONE apt-get purge of the exact names; then prove the boot setup did not move ---
|
||||||
|
gens_before = {p: (self.r.lexists(p), self.read_or_none(p)) for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES)}
|
||||||
|
pin_before = self.read_or_none(KERNEL_DEFAULT_CFG)
|
||||||
|
free_before = a.free_bytes()
|
||||||
|
t0 = time.time()
|
||||||
|
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q", "purge"] + removal)
|
||||||
|
res["seconds"] = round(time.time() - t0, 1)
|
||||||
|
if rc != 0:
|
||||||
|
_, aud, _ = a.x(["dpkg", "--audit"])
|
||||||
|
res["outcome"] = "failed"
|
||||||
|
self.report["failed"] = {"rc": rc, "step": "purge", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
|
||||||
|
"tail": (out + err).strip().splitlines()[-3:]}
|
||||||
|
self.log(f"os-apply: FAILED rc={rc} step=purge (the next run repairs dpkg first, then computes the rule again)")
|
||||||
|
return 3
|
||||||
|
after = self.kernel_images()
|
||||||
|
after_names = {n for names in after.values() for n in names}
|
||||||
|
res["removed"] = [n for n in removal if n not in after_names]
|
||||||
|
free_after = a.free_bytes()
|
||||||
|
res["bytes_gained"] = free_after - free_before if free_before >= 0 and free_after >= 0 else None
|
||||||
|
problems = []
|
||||||
|
if len(res["removed"]) != len(removal):
|
||||||
|
problems.append(f"still installed after the purge: {', '.join(n for n in removal if n in after_names)}")
|
||||||
|
gone = [k for k in res["kept"] if k not in after]
|
||||||
|
if gone:
|
||||||
|
problems.append(f"a KEPT kernel is no longer installed: {', '.join(gone)}")
|
||||||
|
got = self.default_kver(self.grub_cfg())
|
||||||
|
if got != dflt:
|
||||||
|
problems.append(f"the default in grub.cfg moved from {dflt} to {got}")
|
||||||
|
if self.read_or_none(KERNEL_DEFAULT_CFG) != pin_before:
|
||||||
|
problems.append("the pinned default file changed")
|
||||||
|
if self.read_or_none(KERNEL_STATE) != raw_state:
|
||||||
|
problems.append("the kernel step's state file changed")
|
||||||
|
for p, (was, body) in gens_before.items():
|
||||||
|
if self.r.lexists(p) != was or self.read_or_none(p) != body:
|
||||||
|
problems.append(f"{p} changed")
|
||||||
|
for k in sorted(set(by_name[n] for n in res["removed"])):
|
||||||
|
if k in after:
|
||||||
|
continue # another image package of the same kernel is still installed
|
||||||
|
res["leftovers"] += [p.format(v=k) for p in KERNEL_LEFTOVER_PATHS if self.r.lexists(p.format(v=k))]
|
||||||
|
if problems:
|
||||||
|
# reported LOUDLY, never repaired by hand here (`11` §5.11.1): the operator looks at the box
|
||||||
|
res["outcome"] = "failed"
|
||||||
|
self.report["failed"] = {"rc": 3, "step": "verify", "reason": "; ".join(problems)[:600]}
|
||||||
|
self.log("os-apply: FAILED rc=3 step=verify after the kernel purge — " + "; ".join(problems))
|
||||||
|
return 3
|
||||||
|
res["outcome"] = "removed"
|
||||||
|
self.log(f"os-apply: KERNEL CLEANUP DONE removed={len(res['removed'])} ({', '.join(res['removed'])}) "
|
||||||
|
f"kept={', '.join(res['kept'])} gained={res['bytes_gained']} B seconds={res['seconds']} "
|
||||||
|
f"leftovers={len(res['leftovers'])}; default {dflt} (proved from grub.cfg)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
class Bundle:
|
class Bundle:
|
||||||
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
||||||
|
|||||||
@@ -1494,6 +1494,24 @@ class PVELane(unittest.TestCase):
|
|||||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||||
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
|
||||||
|
|
||||||
|
# 2026-11-05..07, demo-felhom (EFI): proxmox-grub 2.12-9+pmx2 -> pmx4 depends on grub-efi-amd64 of the same version;
|
||||||
|
# HOST_SLOW_RE's `grub` is anchored at the start, so proxmox-grub fell into the pve plan, the simulation pulled
|
||||||
|
# grub-efi-amd64 and the whole pve step was refused R6 every night. It is a boot-chain package: left out.
|
||||||
|
# RED-PROOF: drop proxmox-grub from HOST_SLOW_RE -> it is in the plan -> this test fails.
|
||||||
|
def test_ring0_pending_pve_leaves_proxmox_grub_out(self):
|
||||||
|
f = pve_fake(ring0=True)
|
||||||
|
f.plan["select"], f.plan["packages"] = "pending-pve", []
|
||||||
|
f.installed.update({"proxmox-grub": "2.12-9+pmx2"})
|
||||||
|
f.pending_sim = [
|
||||||
|
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst proxmox-grub [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
"Inst grub-efi-amd64 [2.12-9+pmx2] (2.12-9+pmx4 Proxmox Debian Repository:stable [amd64])",
|
||||||
|
]
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
|
||||||
|
self.assertEqual(f.installed["proxmox-grub"], "2.12-9+pmx2")
|
||||||
|
|
||||||
def test_select_pending_pve_needs_the_pve_layer(self):
|
def test_select_pending_pve_needs_the_pve_layer(self):
|
||||||
f = Fake()
|
f = Fake()
|
||||||
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
|
||||||
@@ -2013,5 +2031,607 @@ class KernelLane(unittest.TestCase):
|
|||||||
self.assertIn("one-shot", h["kernel_next_boot_source"])
|
self.assertIn("one-shot", h["kernel_next_boot_source"])
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----------
|
||||||
|
K = ["7.0.2-6-pve", "7.0.14-20-pve", "7.0.14-22-pve", "7.0.14-26-pve", "7.0.14-28-pve"] # five installed kernels
|
||||||
|
C_FROM, C_RUN = "7.0.14-26-pve", "7.0.14-28-pve"
|
||||||
|
GB = 1024 ** 3
|
||||||
|
|
||||||
|
|
||||||
|
def img(k):
|
||||||
|
return f"proxmox-kernel-{k}-signed"
|
||||||
|
|
||||||
|
|
||||||
|
class CFake(KFake):
|
||||||
|
"""A ring-0 appliance after a good kernel step 26 -> 28: five kernels installed, the box runs its newest, the
|
||||||
|
default (grub.cfg and the pinned file) is the running kernel, the flag is empty. `apt-get purge` is emulated like
|
||||||
|
the real one: the image leaves dpkg and /boot, and the kernel's postrm runs update-grub."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self.running = C_RUN
|
||||||
|
self.boot = set(K)
|
||||||
|
self.modules = set(K) # /lib/modules/<kver>
|
||||||
|
for n in [x for x in self.installed if osapply.KERNEL_IMAGE_RE.match(x)]:
|
||||||
|
del self.installed[n]
|
||||||
|
for k in K:
|
||||||
|
self.installed[img(k)] = k[:-4]
|
||||||
|
self.installed["proxmox-kernel-7.0"] = C_RUN[:-4]
|
||||||
|
self.env = {"felhom_next": ""}
|
||||||
|
self.tree[osapply.KERNEL_DEFAULT_CFG] = (f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-{C_RUN}-advanced-{U}"\n')
|
||||||
|
self.state = {"phase": "good", "step_id": "ring0-x", "from": C_FROM, "to": C_RUN, "default_before": C_FROM,
|
||||||
|
"vmid": 9201, "staged_at": "2026-10-01T02:50:00Z", "rebooted_at": "2026-10-01T02:51:00Z",
|
||||||
|
"result_at": "2026-10-01T03:10:00Z", "self_revert_used": False}
|
||||||
|
self.write_state()
|
||||||
|
self.cfg = self.render()
|
||||||
|
self.purges, self.purge_rc, self.purge_moves_default, self.purge_extra_sim = [], 0, False, []
|
||||||
|
self.plan = {"release_id": "kernel", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "kernel-cleanup",
|
||||||
|
"packages": []}
|
||||||
|
|
||||||
|
def write_state(self):
|
||||||
|
if self.state is None:
|
||||||
|
self.tree.pop(osapply.KERNEL_STATE, None)
|
||||||
|
else:
|
||||||
|
self.tree[osapply.KERNEL_STATE] = json.dumps(self.state)
|
||||||
|
|
||||||
|
def lexists(self, p):
|
||||||
|
m = re.match(r"^/lib/modules/(.+)$", p)
|
||||||
|
if m:
|
||||||
|
return m.group(1) in self.modules
|
||||||
|
return super().lexists(p)
|
||||||
|
|
||||||
|
def changing(self):
|
||||||
|
"""Every command that changes the box (the purge, a repair, update-grub, a flag write, a file write)."""
|
||||||
|
out = []
|
||||||
|
for c in self.calls:
|
||||||
|
a = [x for x in c[-1] if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||||
|
if (a[0] == "apt-get" and "-s" not in a and "--print-uris" not in a) or a[0] in ("update-grub", "dpkg-reconfigure") \
|
||||||
|
or (a[0] == "dpkg" and a[1] == "--configure") or (a[0] == "grub-editenv" and a[2] != "list") \
|
||||||
|
or (a[0] == "systemctl" and a[1] == "reboot"):
|
||||||
|
out.append(a)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def emulate(self, argv):
|
||||||
|
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||||
|
if a[0] == "apt-get" and "purge" in a and "-s" not in a:
|
||||||
|
names = [x for x in a[a.index("purge") + 1:] if not x.startswith("-")]
|
||||||
|
self.purges.append(names)
|
||||||
|
if self.purge_rc:
|
||||||
|
self.dpkg_journal = ["0001"] # a purge cut in the middle leaves dpkg's update journal
|
||||||
|
return self.purge_rc, "", "E: Sub-process /usr/bin/dpkg returned an error code (1)"
|
||||||
|
for n in names:
|
||||||
|
self.installed.pop(n, None)
|
||||||
|
m = osapply.KERNEL_IMAGE_RE.match(n)
|
||||||
|
if m:
|
||||||
|
self.boot.discard(m.group(1))
|
||||||
|
self.free += GB
|
||||||
|
if self.purge_moves_default:
|
||||||
|
self.tree.pop(osapply.KERNEL_DEFAULT_CFG, None)
|
||||||
|
self.cfg = self.render() # the kernel's postrm runs update-grub (zz-update-grub)
|
||||||
|
return 0, "Removing proxmox-kernel ...\n", ""
|
||||||
|
return super().emulate(argv)
|
||||||
|
|
||||||
|
def sim(self, a):
|
||||||
|
if "purge" in a:
|
||||||
|
names = [x for x in a[a.index("purge") + 1:] if not x.startswith("-")]
|
||||||
|
if getattr(self, "purge_sim_rc", 0):
|
||||||
|
return self.purge_sim_rc, "", "E: Unable to locate package"
|
||||||
|
out = "".join(f"Purg {n} [{self.installed[n]}]\n" for n in names if n in self.installed)
|
||||||
|
return 0, out + "".join(l + "\n" for l in self.purge_extra_sim), ""
|
||||||
|
return super().sim(a)
|
||||||
|
|
||||||
|
|
||||||
|
def cfake(**kw):
|
||||||
|
f = CFake()
|
||||||
|
for k, v in kw.items():
|
||||||
|
setattr(f, k, v)
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
def cstate(f, **kw):
|
||||||
|
f.state.update(kw)
|
||||||
|
f.write_state()
|
||||||
|
return f
|
||||||
|
|
||||||
|
|
||||||
|
class KernelCleanup(unittest.TestCase):
|
||||||
|
"""§5 of the release-2a brief: one test per acceptance scenario, each on the stored effect (the exact apt line or
|
||||||
|
the absence of any changing command, the report, the exit code). Red-proofs (the four mutations the brief names):
|
||||||
|
felhom.eu audits/release-2a-2026-11-14/A/redproof.txt."""
|
||||||
|
|
||||||
|
REMOVE = [img(k) for k in ("7.0.14-20-pve", "7.0.14-22-pve", "7.0.2-6-pve")] # sorted by name
|
||||||
|
|
||||||
|
def nothing_changed(self, f):
|
||||||
|
self.assertEqual(f.purges, [], "no purge")
|
||||||
|
self.assertEqual(f.changing(), [], "no command that changes the box")
|
||||||
|
self.assertEqual(sorted(f.boot), sorted(K), "every kernel is still there")
|
||||||
|
|
||||||
|
def refused(self, f, code):
|
||||||
|
before = (f.tree.get(osapply.KERNEL_STATE), f.tree.get(osapply.KERNEL_DEFAULT_CFG), f.cfg, dict(f.env or {}))
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 2, rep)
|
||||||
|
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||||
|
self.nothing_changed(f)
|
||||||
|
self.assertEqual((f.tree.get(osapply.KERNEL_STATE), f.tree.get(osapply.KERNEL_DEFAULT_CFG), f.cfg, dict(f.env or {})), before)
|
||||||
|
return rep
|
||||||
|
|
||||||
|
def not_run(self, f, code):
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
c = rep["kernel_cleanup"]
|
||||||
|
self.assertEqual(c["outcome"], "not_run", rep)
|
||||||
|
self.assertIn(code, [x["code"] for x in c["not_run"]], rep)
|
||||||
|
self.assertIsNone(rep.get("refused"))
|
||||||
|
self.nothing_changed(f)
|
||||||
|
return rep
|
||||||
|
|
||||||
|
# --- scenario 1: the normal night ---
|
||||||
|
def test_s1_the_normal_night_purges_exactly_the_old_kernels_in_one_command(self):
|
||||||
|
f = cfake()
|
||||||
|
state0, pin0 = f.tree[osapply.KERNEL_STATE], f.tree[osapply.KERNEL_DEFAULT_CFG]
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(f.purges, [self.REMOVE], "ONE apt-get purge of the exact names")
|
||||||
|
purge = [c[-1] for c in f.calls if "purge" in c[-1] and "-s" not in c[-1]]
|
||||||
|
self.assertEqual(len(purge), 1)
|
||||||
|
self.assertEqual(purge[0][purge[0].index("apt-get"):], ["apt-get", "-y", "-q", "purge"] + self.REMOVE)
|
||||||
|
self.assertNotIn("autoremove", " ".join(" ".join(c[-1]) for c in f.calls), "never apt's own clean-up")
|
||||||
|
c = rep["kernel_cleanup"]
|
||||||
|
self.assertEqual(c["outcome"], "removed")
|
||||||
|
self.assertEqual(c["removed"], self.REMOVE)
|
||||||
|
self.assertEqual(sorted(c["kept"]), sorted([C_FROM, C_RUN]))
|
||||||
|
self.assertEqual(c["bytes_gained"], 3 * GB)
|
||||||
|
self.assertEqual(sorted(f.boot), sorted([C_FROM, C_RUN]))
|
||||||
|
# the default in grub.cfg, the pinned default file and the step's state file are unchanged
|
||||||
|
self.assertEqual(osapply.Kernel.default_kver(f.cfg), C_RUN)
|
||||||
|
self.assertEqual(f.tree[osapply.KERNEL_DEFAULT_CFG], pin0)
|
||||||
|
self.assertEqual(f.tree[osapply.KERNEL_STATE], state0)
|
||||||
|
self.assertEqual(f.env, {"felhom_next": ""})
|
||||||
|
self.assertEqual(f.reboots, [], "the clean-up never reboots")
|
||||||
|
self.assertEqual(f.update_grubs, 0, "the wrapper itself runs no update-grub (the kernel's postrm did)")
|
||||||
|
# leftovers are reported, never deleted
|
||||||
|
self.assertEqual(c["leftovers"], [f"/lib/modules/{k}" for k in ("7.0.14-20-pve", "7.0.14-22-pve", "7.0.2-6-pve")])
|
||||||
|
self.assertEqual(f.modules, set(K), "a left-over folder is reported, not deleted")
|
||||||
|
|
||||||
|
# --- scenario 2: the second night ---
|
||||||
|
def test_s2_the_second_night_removes_nothing(self):
|
||||||
|
f = cfake()
|
||||||
|
self.assertEqual(run(f)[0], 0)
|
||||||
|
f.calls.clear()
|
||||||
|
f.purges.clear()
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["kernel_cleanup"]["outcome"], "nothing")
|
||||||
|
self.assertEqual(rep["kernel_cleanup"]["removal"], [])
|
||||||
|
self.assertEqual(f.purges, [])
|
||||||
|
self.assertEqual(f.changing(), [], "no apt call that changes anything")
|
||||||
|
self.assertFalse([c for c in f.calls if "purge" in c[-1]], "not even a simulation: there is nothing to simulate")
|
||||||
|
|
||||||
|
# --- scenario 3: a step is not over (P1) ---
|
||||||
|
def test_s3_a_step_that_is_not_over_refuses(self):
|
||||||
|
for phase in ("staged", "oneshot", "judging", "reverting", "some-new-phase"):
|
||||||
|
f = cstate(cfake(), phase=phase)
|
||||||
|
rep = self.refused(f, "P1")
|
||||||
|
self.assertIn(phase, rep["refused"]["reason"])
|
||||||
|
|
||||||
|
# --- scenario 4: the flag (P2); the pinned default against grub.cfg (P3) ---
|
||||||
|
def test_s4_the_flag_names_a_kernel_refuses(self):
|
||||||
|
f = cfake(env={"felhom_next": "7.0.14-26-pve"})
|
||||||
|
self.refused(f, "P2")
|
||||||
|
|
||||||
|
def test_s4_pinned_default_and_grub_cfg_disagree_refuses(self):
|
||||||
|
f = cfake()
|
||||||
|
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-{C_FROM}-advanced-{U}"\n'
|
||||||
|
self.refused(f, "P3") # grub.cfg still says 28, the file says 26
|
||||||
|
g = cfake()
|
||||||
|
del g.tree[osapply.KERNEL_DEFAULT_CFG]
|
||||||
|
self.refused(g, "P3") # no pinned file at all
|
||||||
|
h = cfake() # both name a kernel that is not installed
|
||||||
|
h.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-7.0.14-99-pve-advanced-{U}"\n'
|
||||||
|
h.cfg = h.cfg.replace(f"gnulinux-{C_RUN}-advanced-{U}\"", f"gnulinux-7.0.14-99-pve-advanced-{U}\"", 1)
|
||||||
|
self.refused(h, "P3")
|
||||||
|
|
||||||
|
# --- scenario 5: the box fell back / does not run its newest kernel ---
|
||||||
|
def test_s5_a_box_that_fell_back_keeps_every_kernel(self):
|
||||||
|
for phase in ("fell_back", "self_reverted", "revert_failed", "cancelled"):
|
||||||
|
self.not_run(cstate(cfake(running=C_FROM), phase=phase), "step-not-good")
|
||||||
|
|
||||||
|
def test_s5_a_box_that_does_not_run_its_newest_kernel_keeps_every_kernel(self):
|
||||||
|
f = cfake(running=C_FROM) # the step says good, but the box runs the older kernel
|
||||||
|
self.not_run(f, "not-on-newest")
|
||||||
|
|
||||||
|
# --- scenario 6: the wrong case (K1) — one case per kept-kernel kind ---
|
||||||
|
def k1(self, kind, prepare):
|
||||||
|
"""The keep computation is BROKEN on purpose (it forgets `kind`); K1 reads the box by itself and must refuse
|
||||||
|
before any apt call."""
|
||||||
|
f = cfake()
|
||||||
|
victim = prepare(f)
|
||||||
|
real = osapply.Kernel.cleanup_keep
|
||||||
|
|
||||||
|
def broken(self_, images, sources):
|
||||||
|
keep = real(self_, images, [(k, why) for k, why in sources if k != victim])
|
||||||
|
keep.pop(victim, None) # … also when it is the newest installed
|
||||||
|
return keep
|
||||||
|
osapply.Kernel.cleanup_keep = broken
|
||||||
|
try:
|
||||||
|
rc, rep = run(f)
|
||||||
|
finally:
|
||||||
|
osapply.Kernel.cleanup_keep = real
|
||||||
|
self.assertEqual(rc, 2, f"{kind}: {rep}")
|
||||||
|
self.assertEqual(rep["refused"]["code"], "K1", f"{kind}: {rep}")
|
||||||
|
self.assertIn(img(victim), rep["refused"]["reason"])
|
||||||
|
self.assertFalse([c for c in f.calls if "purge" in c[-1]], f"{kind}: refused BEFORE any apt call, also the simulation")
|
||||||
|
self.nothing_changed(f)
|
||||||
|
|
||||||
|
def test_s6_k1_the_running_kernel(self):
|
||||||
|
self.k1("running", lambda f: C_RUN)
|
||||||
|
|
||||||
|
def test_s6_k1_the_steps_from(self):
|
||||||
|
self.k1("from", lambda f: C_FROM)
|
||||||
|
|
||||||
|
def test_s6_k1_the_steps_to(self):
|
||||||
|
def prep(f): # a step whose `to` is not the running kernel any more
|
||||||
|
cstate(f, to="7.0.14-22-pve")
|
||||||
|
return "7.0.14-22-pve"
|
||||||
|
self.k1("to", prep)
|
||||||
|
|
||||||
|
def test_s6_k1_default_before(self):
|
||||||
|
def prep(f):
|
||||||
|
cstate(f, default_before="7.0.14-20-pve")
|
||||||
|
return "7.0.14-20-pve"
|
||||||
|
self.k1("default_before", prep)
|
||||||
|
|
||||||
|
def test_s6_k1_the_grub_cfg_default_and_the_pinned_file(self):
|
||||||
|
# the default (both places) names 26; the box runs 28 after a hand boot. The keep computation forgets 26.
|
||||||
|
def prep(f):
|
||||||
|
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-7.0.14-22-pve-advanced-{U}"\n'
|
||||||
|
f.cfg = f.render()
|
||||||
|
return "7.0.14-22-pve"
|
||||||
|
self.k1("default", prep)
|
||||||
|
|
||||||
|
def test_s6_k1_the_flag(self):
|
||||||
|
# a flag is a refusal by itself (P2, first); with the keep computation broken, K1 is in the SAME refusal
|
||||||
|
f = cfake(env={"felhom_next": "7.0.14-20-pve"})
|
||||||
|
real = osapply.Kernel.cleanup_keep
|
||||||
|
osapply.Kernel.cleanup_keep = lambda s_, images, sources: real(s_, images, [(k, w) for k, w in sources if k != "7.0.14-20-pve"])
|
||||||
|
try:
|
||||||
|
rc, rep = run(f)
|
||||||
|
finally:
|
||||||
|
osapply.Kernel.cleanup_keep = real
|
||||||
|
self.assertEqual((rc, rep["refused"]["code"]), (2, "P2"), rep)
|
||||||
|
self.assertIn("K1 " + img("7.0.14-20-pve") + " is a keep-kernel (the one-shot flag)", rep["refused"]["reason"])
|
||||||
|
self.assertFalse([c for c in f.calls if "purge" in c[-1]])
|
||||||
|
self.nothing_changed(f)
|
||||||
|
|
||||||
|
def test_s6_k1_an_operators_boot_pin(self):
|
||||||
|
def prep(f):
|
||||||
|
f.tree[osapply.KERNEL_PIN_FILE] = "7.0.14-20-pve\n"
|
||||||
|
return "7.0.14-20-pve"
|
||||||
|
self.k1("boot pin", prep)
|
||||||
|
|
||||||
|
def test_s6_every_kept_kind_is_in_the_keep_set_and_never_in_the_removal_set(self):
|
||||||
|
f = cfake()
|
||||||
|
cstate(f, default_before="7.0.14-20-pve")
|
||||||
|
f.tree[osapply.KERNEL_PIN_FILE] = "7.0.14-22-pve\n"
|
||||||
|
f.plan["dry_run"] = True
|
||||||
|
rc, rep = run(f)
|
||||||
|
c = rep["kernel_cleanup"]
|
||||||
|
self.assertEqual(sorted(c["kept"]), sorted(["7.0.14-20-pve", "7.0.14-22-pve", C_FROM, C_RUN]))
|
||||||
|
self.assertEqual(c["removal"], [img("7.0.2-6-pve")])
|
||||||
|
|
||||||
|
def test_s6_the_running_kernel_is_not_an_installed_package(self):
|
||||||
|
f = cfake()
|
||||||
|
del f.installed[img(C_RUN)] # the box runs a kernel dpkg does not know (an unsigned hand install, say)
|
||||||
|
f.installed["proxmox-kernel-7.0.14-28-pve-other"] = "7.0.14-28"
|
||||||
|
self.refused(f, "P3") # P3 is first (the default names a kernel that is not an installed image) …
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertIn("K1", rep["refused"]["reason"], "… and K1 is in the same refusal")
|
||||||
|
|
||||||
|
def test_no_step_record_keeps_the_next_lower_kernel(self):
|
||||||
|
f = cstate(cfake(), **{"from": None, "to": None, "default_before": None})
|
||||||
|
f.plan["dry_run"] = True
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertIn("7.0.14-26-pve", rep["kernel_cleanup"]["kept"])
|
||||||
|
self.assertIn("next lower", " ".join(rep["kernel_cleanup"]["kept"]["7.0.14-26-pve"]))
|
||||||
|
|
||||||
|
# --- scenario 7: apt would do more (S1, S2) ---
|
||||||
|
def test_s7_the_simulation_would_remove_something_else(self):
|
||||||
|
f = cfake(purge_extra_sim=["Purg proxmox-kernel-7.0 [7.0.14-28]", "Remv proxmox-ve [9.2.0]"])
|
||||||
|
rep = self.refused(f, "S2")
|
||||||
|
self.assertIn("proxmox-ve", rep["refused"]["reason"])
|
||||||
|
|
||||||
|
def test_s7_the_simulation_would_install_something(self):
|
||||||
|
f = cfake(purge_extra_sim=["Inst proxmox-kernel-7.0.14-28-pve (7.0.14-28 Proxmox Debian Repository:stable [amd64])"])
|
||||||
|
rep = self.refused(f, "S1")
|
||||||
|
self.assertIn("proxmox-kernel-7.0.14-28-pve", rep["refused"]["reason"])
|
||||||
|
|
||||||
|
def test_s7_the_simulation_fails(self):
|
||||||
|
self.refused(cfake(purge_sim_rc=100), "S1")
|
||||||
|
|
||||||
|
# --- scenario 8: not our box ---
|
||||||
|
def test_s8_a_byo_host_is_refused_by_the_appliance_rule(self):
|
||||||
|
f = cfake()
|
||||||
|
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||||
|
self.refused(f, "R12")
|
||||||
|
|
||||||
|
def test_s8_a_box_without_the_ring0_mark_does_not_run(self):
|
||||||
|
f = cfake()
|
||||||
|
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "tester-1-d70be4", "ring0_slow_lane": False})
|
||||||
|
rep = self.not_run(f, "no-ring0-mark")
|
||||||
|
self.assertIn("ring 0 only in this release", rep["kernel_cleanup"]["not_run"][0]["reason"])
|
||||||
|
g = cfake() # an agent-writable trust file proves nothing
|
||||||
|
g.stats[osapply.TRUST_FILE] = St(mode=statmod.S_IFREG | 0o666, uid=0)
|
||||||
|
self.not_run(g, "no-ring0-mark")
|
||||||
|
h = cfake() # the agent's plan cannot claim the ring
|
||||||
|
h.files[osapply.TRUST_FILE] = json.dumps({"host_id": "x", "ring0_slow_lane": False})
|
||||||
|
h.plan["ring"] = 0
|
||||||
|
self.not_run(h, "no-ring0-mark")
|
||||||
|
|
||||||
|
# --- scenario 9: a plan that names packages ---
|
||||||
|
def test_s9_cleanup_plan_names_refused(self):
|
||||||
|
for extra in ({"packages": [{"name": img("7.0.14-28-pve"), "version": "7.0.14-28", "origin": "Proxmox Debian Repository"}]},
|
||||||
|
{"select": "pending-kernel"}, {"allow_new": ["x"]}, {"signed": {"blob_b64": "eA==", "sig": "x"}},
|
||||||
|
{"expect_kver": "7.0.14-20-pve"}, {"dry_run": "yes"}):
|
||||||
|
f = cfake()
|
||||||
|
f.plan.update(extra)
|
||||||
|
self.refused(f, "C1")
|
||||||
|
|
||||||
|
# --- scenario 10: no kernel step ever ran ---
|
||||||
|
def test_s10_no_state_file_does_not_run(self):
|
||||||
|
f = cfake()
|
||||||
|
f.state = None
|
||||||
|
f.write_state()
|
||||||
|
rep = self.not_run(f, "no-step")
|
||||||
|
self.assertIn("no kernel step yet", rep["kernel_cleanup"]["not_run"][0]["reason"])
|
||||||
|
|
||||||
|
# --- scenario 11: a power cut in the purge ---
|
||||||
|
def test_s11_a_cut_purge_is_repaired_first_and_the_rule_is_computed_again(self):
|
||||||
|
f = cfake(purge_rc=100)
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 3, rep)
|
||||||
|
self.assertEqual(rep["failed"]["step"], "purge")
|
||||||
|
self.assertEqual(rep["kernel_cleanup"]["outcome"], "failed")
|
||||||
|
# the cut removed ONE of the three before it stopped
|
||||||
|
f.installed.pop(self.REMOVE[0])
|
||||||
|
f.boot.discard("7.0.14-20-pve")
|
||||||
|
f.purge_rc, f.purges = 0, []
|
||||||
|
f.calls.clear()
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["repair"]["journal_before"], 1, "repair first: dpkg's journal was replayed")
|
||||||
|
i_repair = [i for i, c in enumerate(f.calls) if c[-1][-3:-1] == ["--configure", "-a"] or "--configure" in c[-1]][0]
|
||||||
|
i_purge = [i for i, c in enumerate(f.calls) if "purge" in c[-1]][0]
|
||||||
|
self.assertLess(i_repair, i_purge, "the repair runs BEFORE the rule is computed and before any purge")
|
||||||
|
self.assertEqual(f.purges, [self.REMOVE[1:]], "the rule was computed again from the box: the two that are left")
|
||||||
|
self.assertEqual(rep["kernel_cleanup"]["outcome"], "removed")
|
||||||
|
|
||||||
|
# --- the 20-hour window, the dry run, after-the-purge proof ---
|
||||||
|
def test_inside_a_steps_20_hour_window_it_does_not_run(self):
|
||||||
|
f = cfake()
|
||||||
|
cstate(f, result_at=osapply._iso(f.clock - 3600)) # the step ended good one hour ago
|
||||||
|
self.not_run(f, "window")
|
||||||
|
|
||||||
|
def test_the_dry_run_reports_and_changes_nothing(self):
|
||||||
|
f = cfake()
|
||||||
|
f.plan["dry_run"] = True
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
c = rep["kernel_cleanup"]
|
||||||
|
self.assertEqual((c["outcome"], c["would"], c["removal"]), ("dry_run", "remove", self.REMOVE))
|
||||||
|
self.assertEqual(sorted(c["kept"]), sorted([C_FROM, C_RUN]))
|
||||||
|
self.nothing_changed(f)
|
||||||
|
self.assertNotIn("repair", rep, "a dry run does not even repair")
|
||||||
|
# a dry run of a refusal says which one, with exit 0
|
||||||
|
g = cstate(cfake(), phase="judging")
|
||||||
|
g.plan["dry_run"] = True
|
||||||
|
rc, rep = run(g)
|
||||||
|
self.assertEqual((rc, rep["kernel_cleanup"]["would"]), (0, "refuse"), rep)
|
||||||
|
self.assertEqual([x["code"] for x in rep["kernel_cleanup"]["refusals"]], ["P1"])
|
||||||
|
self.nothing_changed(g)
|
||||||
|
# … and of a box that would not run
|
||||||
|
h = cfake()
|
||||||
|
h.files[osapply.TRUST_FILE] = json.dumps({"host_id": "x", "ring0_slow_lane": False})
|
||||||
|
h.plan["dry_run"] = True
|
||||||
|
rc, rep = run(h)
|
||||||
|
self.assertEqual((rc, rep["kernel_cleanup"]["would"], rep["kernel_cleanup"]["removal"]), (0, "not_run", self.REMOVE), rep)
|
||||||
|
self.nothing_changed(h)
|
||||||
|
|
||||||
|
def test_a_purge_that_moves_the_default_is_reported_failed_loudly(self):
|
||||||
|
f = cfake(purge_moves_default=True) # the pinned file vanished in the purge: 10_linux makes the newest the default
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 3, rep)
|
||||||
|
self.assertEqual(rep["failed"]["step"], "verify")
|
||||||
|
self.assertIn("pinned default file changed", rep["failed"]["reason"])
|
||||||
|
self.assertEqual(f.update_grubs, 0, "never repaired by hand in code")
|
||||||
|
self.assertTrue(any("FAILED" in l and "verify" in l for l in f.logs))
|
||||||
|
|
||||||
|
def test_the_apt_lock_is_respected(self):
|
||||||
|
self.refused(cfake(lock_held=True), "R9")
|
||||||
|
|
||||||
|
def test_both_image_packages_of_one_old_kernel_go_together(self):
|
||||||
|
f = cfake()
|
||||||
|
f.installed["proxmox-kernel-7.0.14-20-pve"] = "7.0.14-20" # the unsigned image beside the signed one
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertIn("proxmox-kernel-7.0.14-20-pve", f.purges[0])
|
||||||
|
self.assertIn(img("7.0.14-20-pve"), f.purges[0])
|
||||||
|
|
||||||
|
def test_cleanup_needs_no_new_sudo_line(self):
|
||||||
|
"""The wrapper is reached by `--plan` only: the new mode rides the one sudo line the agent already has."""
|
||||||
|
sudo = (HERE / "felhom-agent.sudoers").read_text()
|
||||||
|
live = [l.strip().rstrip("\\,").strip() for l in sudo.splitlines() if l.strip() and not l.strip().startswith("#")]
|
||||||
|
self.assertEqual([l for l in live if "felhom-os-apply" in l and "Cmnd_Alias" not in l and "NOPASSWD" not in l],
|
||||||
|
[osapply.SELF_CHECK_SUDO_LINE], "exactly the one --plan line reaches the wrapper")
|
||||||
|
self.assertFalse([l for l in live if "kernel-cleanup" in l or
|
||||||
|
("apt" in l and ("purge" in l or "remove" in l))], "no sudo line removes a package")
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----------
|
||||||
|
class PFake(Fake):
|
||||||
|
"""A host with two AppArmor cache folders. `stale`: profile files whose cached copy in the running kernel's folder
|
||||||
|
is an old compile; the refresh command rewrites the cache from `fresh`. lxc-containers compiles differently every
|
||||||
|
time (measured on demo-hp). `failed`: unit -> ExecStart program."""
|
||||||
|
|
||||||
|
def __init__(self, stale=("usr.sbin.chronyd",), failed=None, kernel="7.0.14-28-pve", record=None):
|
||||||
|
super().__init__()
|
||||||
|
self.kernel = kernel
|
||||||
|
self.fresh = {"usr.sbin.chronyd": b"chrony-4.8", "usr.bin.man": b"man", "lxc-containers": b"lxc-0", "nvidia_modprobe": b"nv"}
|
||||||
|
self.names = {"usr.sbin.chronyd": "/usr/sbin/chronyd", "usr.bin.man": "/usr/bin/man\nman_filter\nman_groff",
|
||||||
|
"lxc-containers": "lxc-container-default\nlxc-container-default-cgns", "nvidia_modprobe": "nvidia_modprobe\nnvidia_modprobe//kmod"}
|
||||||
|
self.cache = {"ae1a4050.0": {k: b"old-" + k.encode() for k in self.fresh}, "f6da0254.0": dict(self.fresh)}
|
||||||
|
for k in stale:
|
||||||
|
self.cache["f6da0254.0"][k] = b"OLD-COPY-" + k.encode()
|
||||||
|
self.failed = dict({"chrony.service": "/usr/sbin/chronyd"} if failed is None else failed)
|
||||||
|
self.restarts, self.parser_runs, self.parser_rc, self.tree = [], 0, 0, {}
|
||||||
|
if record is not None:
|
||||||
|
self.tree[osapply.PROFILE_STATE] = json.dumps(record)
|
||||||
|
self.plan = {"release_id": "profile-refresh", "layer": "host", "mode": "profile-refresh"}
|
||||||
|
|
||||||
|
def isdir(self, p):
|
||||||
|
return p in (osapply.PROFILE_DIR, osapply.PROFILE_CACHE) or p.startswith(osapply.PROFILE_CACHE + "/") and p.count("/") == 4
|
||||||
|
|
||||||
|
def list_dir(self, p):
|
||||||
|
if p == osapply.PROFILE_CACHE:
|
||||||
|
return sorted(self.cache)
|
||||||
|
d = p.rsplit("/", 1)[1]
|
||||||
|
return sorted(list(self.cache.get(d, {})) + [".features"])
|
||||||
|
|
||||||
|
def read_bytes(self, p):
|
||||||
|
d, f = p.split("/")[-2:]
|
||||||
|
return self.cache[d][f]
|
||||||
|
|
||||||
|
def put_file(self, path, data, mode):
|
||||||
|
self.tree[path] = data.decode()
|
||||||
|
|
||||||
|
def read_file(self, p):
|
||||||
|
if p in self.tree:
|
||||||
|
return self.tree[p]
|
||||||
|
return super().read_file(p)
|
||||||
|
|
||||||
|
def host(self, argv, timeout=600, stdin=None):
|
||||||
|
if argv[0] == "uname":
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
return 0, self.kernel + "\n", ""
|
||||||
|
if argv == osapply.PROFILE_REFRESH_CMD:
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
self.parser_runs += 1
|
||||||
|
if self.parser_rc:
|
||||||
|
return self.parser_rc, "", "AppArmor parser error for /etc/apparmor.d/x"
|
||||||
|
self.fresh["lxc-containers"] = b"lxc-%d" % self.parser_runs # never the same binary twice
|
||||||
|
self.cache["f6da0254.0"] = dict(self.fresh)
|
||||||
|
return 0, "", ""
|
||||||
|
if argv[:2] == ["apparmor_parser", "-N"]:
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
return 0, self.names[argv[2].rsplit("/", 1)[1]] + "\n", ""
|
||||||
|
if argv[:2] == ["systemctl", "list-units"]:
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
return 0, "".join(f"{u} loaded failed failed x\n" for u in self.failed), ""
|
||||||
|
if argv[:2] == ["systemctl", "show"]:
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
return 0, f"{{ path={self.failed[argv[-1]]} ; argv[]={self.failed[argv[-1]]} -F 1 ; ignore_errors=no }}\n", ""
|
||||||
|
if argv[:2] == ["systemctl", "restart"]:
|
||||||
|
self.calls.append(("host", argv))
|
||||||
|
self.restarts.append(argv[2])
|
||||||
|
return 0, "", ""
|
||||||
|
return super().host(argv, timeout, stdin)
|
||||||
|
|
||||||
|
|
||||||
|
class ProfileRefresh(unittest.TestCase):
|
||||||
|
"""Red-proofs: felhom.eu audits/release-2a-2026-11-14/B/redproof.txt."""
|
||||||
|
|
||||||
|
def test_a_stale_profile_is_refreshed_and_only_its_failed_unit_restarts(self):
|
||||||
|
f = PFake(failed={"chrony.service": "/usr/sbin/chronyd", "postfix.service": "/usr/sbin/postfix", "other.service": "/usr/bin/man-not"})
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
p = rep["profile_refresh"]
|
||||||
|
self.assertEqual(p["outcome"], "refreshed")
|
||||||
|
self.assertEqual(f.parser_runs, 1)
|
||||||
|
self.assertIn(("host", ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", "/etc/apparmor.d"]), f.calls)
|
||||||
|
self.assertEqual(f.cache["f6da0254.0"]["usr.sbin.chronyd"], b"chrony-4.8", "the cache holds the fresh compile")
|
||||||
|
self.assertEqual(p["rewritten"], ["f6da0254.0/lxc-containers", "f6da0254.0/usr.sbin.chronyd"])
|
||||||
|
self.assertEqual(f.restarts, ["chrony.service"], "ONLY the failed unit whose own program's profile was stale")
|
||||||
|
self.assertEqual([x["unit"] for x in p["restarted"]], ["chrony.service"])
|
||||||
|
rec = json.loads(f.tree[osapply.PROFILE_STATE])
|
||||||
|
self.assertEqual(rec["kernel"], "7.0.14-28-pve")
|
||||||
|
|
||||||
|
def test_a_fresh_cache_changes_nothing_and_restarts_nothing(self):
|
||||||
|
f = PFake(stale=()) # chrony is failed for another reason; its profile is NOT stale
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 0, rep)
|
||||||
|
self.assertEqual(rep["profile_refresh"]["rewritten"], ["f6da0254.0/lxc-containers"], "only the file that never compiles the same")
|
||||||
|
self.assertEqual(f.restarts, [], "a failed unit whose profile was not stale is NOT restarted")
|
||||||
|
self.assertEqual(f.cache["f6da0254.0"]["usr.sbin.chronyd"], b"chrony-4.8")
|
||||||
|
|
||||||
|
def test_a_stale_profile_of_a_running_program_restarts_nothing(self):
|
||||||
|
f = PFake(failed={}) # the unit is active: the kernel swaps the profile of a running task itself
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, f.restarts, rep["profile_refresh"]["outcome"]), (0, [], "refreshed"), rep)
|
||||||
|
|
||||||
|
def test_a_failed_unit_of_another_program_is_not_restarted(self):
|
||||||
|
f = PFake(stale=("usr.bin.man",), failed={"chrony.service": "/usr/sbin/chronyd"})
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, f.restarts), (0, []), rep)
|
||||||
|
g = PFake(stale=("nvidia_modprobe",), failed={"nvidia.service": "nvidia_modprobe"}) # a profile name that is not a path
|
||||||
|
rc, rep = run(g)
|
||||||
|
self.assertEqual((rc, g.restarts), (0, []), rep)
|
||||||
|
|
||||||
|
def test_once_per_kernel(self):
|
||||||
|
f = PFake()
|
||||||
|
self.assertEqual(run(f)[0], 0)
|
||||||
|
f.calls.clear()
|
||||||
|
f.cache["f6da0254.0"]["usr.sbin.chronyd"] = b"OLD-COPY" # even with a stale file: this kernel was done
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["profile_refresh"]["outcome"], f.parser_runs), (0, "not_needed", 1), rep)
|
||||||
|
self.assertFalse([c for c in f.calls if c[1][0] in ("apparmor_parser", "systemctl")], "no parser, no systemctl")
|
||||||
|
f.kernel = "7.0.14-29-pve" # the box starts on another kernel: again
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["profile_refresh"]["outcome"], f.parser_runs), (0, "refreshed", 2), rep)
|
||||||
|
self.assertEqual(json.loads(f.tree[osapply.PROFILE_STATE])["kernel"], "7.0.14-29-pve")
|
||||||
|
|
||||||
|
def test_a_failed_parser_writes_no_record_so_the_next_start_tries_again(self):
|
||||||
|
f = PFake()
|
||||||
|
f.parser_rc = 1
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["failed"]["step"], rep["profile_refresh"]["outcome"]), (3, "profile-refresh", "failed"), rep)
|
||||||
|
self.assertNotIn(osapply.PROFILE_STATE, f.tree)
|
||||||
|
self.assertEqual(f.restarts, [])
|
||||||
|
f.parser_rc = 0
|
||||||
|
self.assertEqual(run(f)[1]["profile_refresh"]["outcome"], "refreshed")
|
||||||
|
|
||||||
|
def test_the_plan_names_nothing(self):
|
||||||
|
for extra in ({"profiles": ["usr.sbin.chronyd"]}, {"units": ["ssh.service"]}, {"packages": [{"name": "chrony", "version": "4.8", "origin": "Debian"}]},
|
||||||
|
{"force": True}, {"dry_run": 1}, {"layer": "guest"}):
|
||||||
|
f = PFake()
|
||||||
|
f.plan.update(extra)
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual(rc, 2, rep)
|
||||||
|
self.assertIn(rep["refused"]["code"], ("F1", "R11"), rep)
|
||||||
|
self.assertEqual((f.parser_runs, f.restarts), (0, []))
|
||||||
|
|
||||||
|
def test_a_byo_host_is_refused(self):
|
||||||
|
f = PFake()
|
||||||
|
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["refused"]["code"], f.parser_runs), (2, "R12", 0), rep)
|
||||||
|
|
||||||
|
def test_the_dry_run_changes_nothing(self):
|
||||||
|
f = PFake()
|
||||||
|
f.plan["dry_run"] = True
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, rep["profile_refresh"]["outcome"], rep["profile_refresh"]["would"]), (0, "dry_run", "refresh"), rep)
|
||||||
|
self.assertEqual((f.parser_runs, f.restarts), (0, []))
|
||||||
|
self.assertNotIn(osapply.PROFILE_STATE, f.tree)
|
||||||
|
|
||||||
|
def test_no_more_than_three_restarts(self):
|
||||||
|
f = PFake(failed={f"chrony{i}.service": "/usr/sbin/chronyd" for i in range(5)})
|
||||||
|
rc, rep = run(f)
|
||||||
|
self.assertEqual((rc, len(f.restarts), len(rep["profile_refresh"]["not_restarted"])), (0, 3, 2), rep)
|
||||||
|
|
||||||
|
def test_profile_refresh_changes_no_package(self):
|
||||||
|
f = PFake()
|
||||||
|
run(f)
|
||||||
|
self.assertFalse([c for c in f.calls if c[1][0] in ("apt-get", "dpkg", "apt")], "no package command at all")
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
module gitea.dooplex.hu/admin/felhom-agent
|
module gitea.dooplex.hu/admin/felhom-agent
|
||||||
|
|
||||||
go 1.27.3
|
go 1.27.4
|
||||||
|
|
||||||
require (
|
require (
|
||||||
golang.org/x/crypto v0.58.0
|
golang.org/x/crypto v0.58.0
|
||||||
|
|||||||
@@ -151,6 +151,7 @@ var manifest = []Capability{
|
|||||||
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
|
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
|
||||||
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
||||||
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
||||||
|
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
|
||||||
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
||||||
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
|
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
|
||||||
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
|
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
|
||||||
|
|||||||
@@ -128,6 +128,7 @@ type Collector struct {
|
|||||||
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
||||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||||
|
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
|
||||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||||
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
|
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
|
||||||
@@ -235,6 +236,12 @@ type ControllerSupervisorReporter interface {
|
|||||||
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
|
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
|
||||||
|
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
|
||||||
|
c.opResults = f
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
|
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
|
||||||
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
|
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
|
||||||
c.ctrlSup = r
|
c.ctrlSup = r
|
||||||
@@ -432,6 +439,9 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
|||||||
if c.ctrlSup != nil {
|
if c.ctrlSup != nil {
|
||||||
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
|
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
|
||||||
}
|
}
|
||||||
|
if c.opResults != nil {
|
||||||
|
report.OperatorActionResults = c.opResults()
|
||||||
|
}
|
||||||
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
|
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
|
||||||
if c.guestNet != nil {
|
if c.guestNet != nil {
|
||||||
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
|
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
|
||||||
|
|||||||
@@ -172,6 +172,24 @@ type HostReport struct {
|
|||||||
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
|
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
|
||||||
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
|
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
|
||||||
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
|
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
|
||||||
|
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
|
||||||
|
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
|
||||||
|
// when there is nothing to answer, so the golden stays byte-stable.
|
||||||
|
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
|
||||||
|
type OperatorAction struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
Action string `json:"action"`
|
||||||
|
Arg string `json:"arg,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
|
||||||
|
type OperatorActionResult struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
Outcome string `json:"outcome"`
|
||||||
|
Message string `json:"message"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
|
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
|
||||||
@@ -195,6 +213,17 @@ type ControllerSupervisorGuest struct {
|
|||||||
Restarts24h int `json:"restarts_24h"`
|
Restarts24h int `json:"restarts_24h"`
|
||||||
SlowCrashloop bool `json:"slow_crashloop"`
|
SlowCrashloop bool `json:"slow_crashloop"`
|
||||||
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
|
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
|
||||||
|
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
|
||||||
|
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
|
||||||
|
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
|
||||||
|
Unresponsive bool `json:"unresponsive"`
|
||||||
|
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
|
||||||
|
// R-1344 restart (decided 2026-10-30 from the night's measurement): UnresponsiveRestartAfterS is how long after the
|
||||||
|
// alarm the agent restarts a controller that still does not answer (0 / absent = an older agent: alarm only — the
|
||||||
|
// hub's message says which). UnresponsiveRestartAt is the last such restart; the hub keys on it MOVING
|
||||||
|
// (controller_unresponsive_restart) and it equals LastRestartAt for that act.
|
||||||
|
UnresponsiveRestartAfterS int `json:"unresponsive_restart_after_s,omitempty"`
|
||||||
|
UnresponsiveRestartAt string `json:"unresponsive_restart_at,omitempty"` // RFC3339
|
||||||
}
|
}
|
||||||
|
|
||||||
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
||||||
@@ -631,6 +660,9 @@ type ControlEnvelope struct {
|
|||||||
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
|
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
|
||||||
// Absent/false on an old hub → nothing happens.
|
// Absent/false on an old hub → nothing happens.
|
||||||
LogTailRequested bool `json:"log_tail_requested"`
|
LogTailRequested bool `json:"log_tail_requested"`
|
||||||
|
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
|
||||||
|
// (`04` §6.1). Absent on an older hub.
|
||||||
|
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
|
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package localapi
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
@@ -83,6 +84,20 @@ const (
|
|||||||
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
|
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
|
||||||
// the parked marker.
|
// the parked marker.
|
||||||
controllerSlowCounterFile = "controller-restarts-24h.json"
|
controllerSlowCounterFile = "controller-restarts-24h.json"
|
||||||
|
|
||||||
|
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
|
||||||
|
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
|
||||||
|
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
|
||||||
|
// the hub (controller_unresponsive). Not counted during a swap, a parked guest, a locked guest or a vzdump.
|
||||||
|
controllerUnresponsiveConfirm = 6
|
||||||
|
// controllerUnresponsiveRestart (decided 2026-10-30 from the measurement): at this many consecutive unhealthy sweeps
|
||||||
|
// (5 minutes after Docker's „unhealthy", ~6.5 minutes after the first failed probe; 2 minutes after the alarm) the
|
||||||
|
// agent restarts the controller — the not-running restart's guards and brake, plus the controller's busy file. The
|
||||||
|
// numbers: Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, 4 733 probes through the night chain (a
|
||||||
|
// PBS and a local vzdump, the DB dumps, tier-2, the off-site push, a restore test, two controller updates): no failed
|
||||||
|
// probe, slowest 303 ms against the 5 s timeout. A busy night never came near even ONE failed probe; ten sweeps is
|
||||||
|
// the margin. Pinned by TestSupervisor_UnresponsiveIsAlarmedThenRestarted.
|
||||||
|
controllerUnresponsiveRestart = 10
|
||||||
)
|
)
|
||||||
|
|
||||||
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
|
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
|
||||||
@@ -103,6 +118,18 @@ type controllerSupState struct {
|
|||||||
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
|
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
|
||||||
restarts24h []time.Time
|
restarts24h []time.Time
|
||||||
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
|
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
|
||||||
|
|
||||||
|
// opBrakeUntil (2026-10-29): an operator restart refused by the brake refuses every press until then (30 min).
|
||||||
|
opBrakeUntil time.Time
|
||||||
|
|
||||||
|
// R-1344: consecutive running-but-unhealthy sweeps, and the current episode (since kept after it ends).
|
||||||
|
unhealthySeen int
|
||||||
|
unresponsive bool
|
||||||
|
unresponsiveSince time.Time
|
||||||
|
// unresponsiveRestartAt: the last restart for not answering (kept; the hub keys on it MOVING). unrespRefusal: the
|
||||||
|
// last reason the restart was refused, so a refusal is logged once per change, not every sweep.
|
||||||
|
unresponsiveRestartAt time.Time
|
||||||
|
unrespRefusal string
|
||||||
}
|
}
|
||||||
|
|
||||||
type controllerSupervisor struct {
|
type controllerSupervisor struct {
|
||||||
@@ -238,6 +265,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
|
|||||||
if !s.provisionedGuest(g.VMID) {
|
if !s.provisionedGuest(g.VMID) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
|
||||||
|
s.resetNotRunning(g.VMID)
|
||||||
|
continue
|
||||||
|
}
|
||||||
evaluated++
|
evaluated++
|
||||||
if !s.superviseOneController(ctx, g.VMID, g.Status) {
|
if !s.superviseOneController(ctx, g.VMID, g.Status) {
|
||||||
down++
|
down++
|
||||||
@@ -293,6 +324,7 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
|||||||
if running {
|
if running {
|
||||||
st.notRunningSeen = 0
|
st.notRunningSeen = 0
|
||||||
s.ctrlSup.mu.Unlock()
|
s.ctrlSup.mu.Unlock()
|
||||||
|
s.observeControllerHealth(ctx, vmid, parked)
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
st.notRunningSeen++
|
st.notRunningSeen++
|
||||||
@@ -333,9 +365,27 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
// Backoff.
|
if !s.supervisorBrakeAllows(vmid, now) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
|
||||||
|
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
|
||||||
|
"vmid", vmid, "status", status, "unit", bootstrapUnit)
|
||||||
|
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||||
|
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
|
||||||
|
reason += "; restart FAILED: " + err.Error()
|
||||||
|
}
|
||||||
|
s.recordSupervisorRestart(vmid, now, reason, nil)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// supervisorBrakeAllows is the brake every supervisor restart passes (the not-running one and R-1344's): a crash-loop
|
||||||
|
// pause in force refuses; controllerCrashloopMax restarts inside controllerCrashloopWindow start one (logged, carried by
|
||||||
|
// the stanza, mailed by the hub as controller_crashloop) and refuse.
|
||||||
|
func (s *Server) supervisorBrakeAllows(vmid int, now time.Time) bool {
|
||||||
s.ctrlSup.mu.Lock()
|
s.ctrlSup.mu.Lock()
|
||||||
st = s.supState(vmid)
|
st := s.supState(vmid)
|
||||||
if !st.crashloopSince.IsZero() {
|
if !st.crashloopSince.IsZero() {
|
||||||
if now.Sub(st.crashloopSince) < controllerCrashloopPause {
|
if now.Sub(st.crashloopSince) < controllerCrashloopPause {
|
||||||
s.ctrlSup.mu.Unlock()
|
s.ctrlSup.mu.Unlock()
|
||||||
@@ -358,22 +408,23 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
s.ctrlSup.mu.Unlock()
|
s.ctrlSup.mu.Unlock()
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
|
// recordSupervisorRestart puts a restart the supervisor performed on the record: the brake's window, the totals, the
|
||||||
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
|
// reason, and R-539's slow counter (persisted; raised at most once per 24 hours — the hub mails on the raise MOVING).
|
||||||
"vmid", vmid, "status", status, "unit", bootstrapUnit)
|
// mark, when set, runs under the same lock (R-1344 stamps its own record there).
|
||||||
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
func (s *Server) recordSupervisorRestart(vmid int, now time.Time, reason string, mark func(st *controllerSupState)) {
|
||||||
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
|
|
||||||
reason += "; restart FAILED: " + err.Error()
|
|
||||||
}
|
|
||||||
s.ctrlSup.mu.Lock()
|
s.ctrlSup.mu.Lock()
|
||||||
st = s.supState(vmid)
|
st := s.supState(vmid)
|
||||||
st.restarts = append(st.restarts, now)
|
st.restarts = append(st.restarts, now)
|
||||||
st.restartsTotal++
|
st.restartsTotal++
|
||||||
st.lastRestartAt = now
|
st.lastRestartAt = now
|
||||||
st.lastReason = reason
|
st.lastReason = reason
|
||||||
st.notRunningSeen = 0
|
st.notRunningSeen = 0
|
||||||
// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING.
|
if mark != nil {
|
||||||
|
mark(st)
|
||||||
|
}
|
||||||
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
|
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
|
||||||
n24 := len(st.restarts24h)
|
n24 := len(st.restarts24h)
|
||||||
raised := false
|
raised := false
|
||||||
@@ -389,7 +440,101 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
|||||||
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
|
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
|
||||||
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
|
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
|
||||||
}
|
}
|
||||||
return false
|
}
|
||||||
|
|
||||||
|
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
|
||||||
|
// the swap's `docker inspect -f` grant. Anything but „unhealthy" — healthy, starting, no health check, a read that
|
||||||
|
// fails — resets the count; so do a swap, a parked guest, a locked guest and a vzdump (each is someone else's moment).
|
||||||
|
func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked bool) {
|
||||||
|
s.swapMu.Lock()
|
||||||
|
swapping := s.swapInFlight[vmid]
|
||||||
|
s.swapMu.Unlock()
|
||||||
|
unhealthy := false
|
||||||
|
if !parked && !swapping {
|
||||||
|
if out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer); err == nil {
|
||||||
|
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||||
|
unhealthy = len(f) == 2 && f[0] == "true" && f[1] == "unhealthy"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if unhealthy {
|
||||||
|
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil || lock != "" {
|
||||||
|
unhealthy = false
|
||||||
|
} else if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||||
|
unhealthy = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
now := s.clock()
|
||||||
|
s.ctrlSup.mu.Lock()
|
||||||
|
st := s.supState(vmid)
|
||||||
|
if !unhealthy {
|
||||||
|
was := st.unresponsive
|
||||||
|
st.unhealthySeen, st.unresponsive = 0, false
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
if was {
|
||||||
|
s.logger.Info("controller-supervisor: the controller answers its health check again", "vmid", vmid)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
st.unhealthySeen++
|
||||||
|
seen := st.unhealthySeen
|
||||||
|
start := seen >= controllerUnresponsiveConfirm && !st.unresponsive
|
||||||
|
if start {
|
||||||
|
st.unresponsive, st.unresponsiveSince = true, now
|
||||||
|
}
|
||||||
|
since := st.unresponsiveSince
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
if start {
|
||||||
|
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so",
|
||||||
|
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String(),
|
||||||
|
"restart_after_sweeps", controllerUnresponsiveRestart)
|
||||||
|
}
|
||||||
|
if seen >= controllerUnresponsiveRestart {
|
||||||
|
s.restartUnresponsive(ctx, vmid, seen, since, now)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// restartUnresponsive is R-1344's restart (decided 2026-10-30): the controller's busy file (fresh = a backup, a
|
||||||
|
// restore, an update or an export is running; unreadable = cannot be ruled out) refuses, then the brake every
|
||||||
|
// supervisor restart passes. The swap, the parked marker, the lock and the vzdump never reach here — they reset the
|
||||||
|
// count. A refusal keeps the alarm and is logged once per reason; the next sweep asks again.
|
||||||
|
func (s *Server) restartUnresponsive(ctx context.Context, vmid, seen int, since, now time.Time) {
|
||||||
|
refuse := func(why string) {
|
||||||
|
s.ctrlSup.mu.Lock()
|
||||||
|
st := s.supState(vmid)
|
||||||
|
changed := st.unrespRefusal != why
|
||||||
|
st.unrespRefusal = why
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
if changed {
|
||||||
|
s.logger.Warn("controller-supervisor: the controller does not answer, but the restart is refused — the alarm stays",
|
||||||
|
"vmid", vmid, "why", why, "unhealthy_sweeps", seen)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
busy, what, err := s.controllerBusy(ctx, vmid)
|
||||||
|
if err != nil {
|
||||||
|
refuse("the controller's busy file cannot be read: " + err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if busy {
|
||||||
|
refuse("the controller is busy: " + what)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !s.supervisorBrakeAllows(vmid, now) {
|
||||||
|
refuse("the restart brake")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
reason := fmt.Sprintf("the controller ran but did not answer its health check for %d sweeps (since %s)",
|
||||||
|
seen, since.UTC().Format(time.RFC3339))
|
||||||
|
s.logger.Warn("controller-supervisor: the controller does not answer — restarting the bootstrap unit",
|
||||||
|
"vmid", vmid, "unhealthy_sweeps", seen, "since", since.UTC().Format(time.RFC3339), "unit", bootstrapUnit)
|
||||||
|
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||||
|
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
|
||||||
|
reason += "; restart FAILED: " + err.Error()
|
||||||
|
}
|
||||||
|
s.recordSupervisorRestart(vmid, now, reason, func(st *controllerSupState) {
|
||||||
|
st.unresponsiveRestartAt = now
|
||||||
|
st.unhealthySeen = 0
|
||||||
|
st.unrespRefusal = ""
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) resetNotRunning(vmid int) {
|
func (s *Server) resetNotRunning(vmid int) {
|
||||||
@@ -446,6 +591,14 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
|
|||||||
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
|
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
|
||||||
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
|
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
|
||||||
}
|
}
|
||||||
|
g.Unresponsive = st.unresponsive
|
||||||
|
if !st.unresponsiveSince.IsZero() {
|
||||||
|
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
g.UnresponsiveRestartAfterS = (controllerUnresponsiveRestart - controllerUnresponsiveConfirm) * int(controllerSupervisorInterval/time.Second)
|
||||||
|
if !st.unresponsiveRestartAt.IsZero() {
|
||||||
|
g.UnresponsiveRestartAt = st.unresponsiveRestartAt.UTC().Format(time.RFC3339)
|
||||||
|
}
|
||||||
out.Guests = append(out.Guests, g)
|
out.Guests = append(out.Guests, g)
|
||||||
}
|
}
|
||||||
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
|
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
|
||||||
|
|||||||
@@ -72,12 +72,56 @@ type EscrowCeremonyConfig struct {
|
|||||||
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
|
// host's hub record — called by the claim, BEFORE the code is handed out (R-978). nil → a claim of
|
||||||
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
|
// a pending package is refused (503): a code must never be handed out for a package the hub lacks.
|
||||||
Upload func(ctx context.Context, pkg []byte) error
|
Upload func(ctx context.Context, pkg []byte) error
|
||||||
|
// HubHolds (R-1254 H2) asks the hub whether its CURRENT package is exactly pkg — asked after a failed upload,
|
||||||
|
// whose answer may have been lost AFTER the hub stored it. nil or an error → the outcome is unknown.
|
||||||
|
HubHolds func(ctx context.Context, pkg []byte) (bool, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
|
// escrowUploadTimeout bounds the claim-time package upload (one small PUT; the subprocess's whole
|
||||||
// ceremony incl. its upload was measured at ~2.4 s).
|
// ceremony incl. its upload was measured at ~2.4 s).
|
||||||
const escrowUploadTimeout = 30 * time.Second
|
const escrowUploadTimeout = 30 * time.Second
|
||||||
|
|
||||||
|
// escrowHoldTimeout bounds the hub question after a failed upload (R-1254 H2).
|
||||||
|
const escrowHoldTimeout = 15 * time.Second
|
||||||
|
|
||||||
|
// uploadOutcome is what a claim or confirm upload did, as the HUB tells it (R-1254 H2): an upload whose answer was lost
|
||||||
|
// or late may have been stored all the same.
|
||||||
|
type uploadOutcome int
|
||||||
|
|
||||||
|
const (
|
||||||
|
uploadStored uploadOutcome = iota // the hub holds the new package
|
||||||
|
uploadNotStored // the hub answered and holds another package: the previous code still works
|
||||||
|
uploadUnknown // the hub cannot be asked: nothing may be promised about either code
|
||||||
|
)
|
||||||
|
|
||||||
|
// escrowUnknownOutcomeMsg is the answer when the hub cannot say which package it holds — never „the previous code
|
||||||
|
// still works", which is false when the lost answer came after the store.
|
||||||
|
const escrowUnknownOutcomeMsg = "the hub did not answer in time — it is not known whether it stored the new recovery package; try again (until a try succeeds, do not rely on the previous code alone)"
|
||||||
|
|
||||||
|
// outcomeAfterFailedUpload asks the hub whether it holds pkg after Upload returned an error. Called with the slot
|
||||||
|
// marked claiming (escrowClaiming), outside escrowMu — it is a network call.
|
||||||
|
func (s *Server) outcomeAfterFailedUpload(pkg []byte) uploadOutcome {
|
||||||
|
if s.escrowCeremony.HubHolds == nil {
|
||||||
|
return uploadUnknown
|
||||||
|
}
|
||||||
|
base := s.baseCtx
|
||||||
|
if base == nil {
|
||||||
|
base = context.Background()
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(base, escrowHoldTimeout)
|
||||||
|
defer cancel()
|
||||||
|
held, err := s.escrowCeremony.HubHolds(ctx, pkg)
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
s.logger.Warn("local-api: escrow — the hub cannot say which package it holds after a failed upload (R-1254)", "err", err)
|
||||||
|
return uploadUnknown
|
||||||
|
case held:
|
||||||
|
return uploadStored
|
||||||
|
default:
|
||||||
|
return uploadNotStored
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
|
// ceremonyRunner executes the fixed-argv sudo self-invocation. stdout is SECRET-BEARING until
|
||||||
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
|
// parsed (it carries R inside the JSON object); the caller must zero it. Tests inject canned
|
||||||
// spike-shaped output; production is runCeremonySubprocess.
|
// spike-shaped output; production is runCeremonySubprocess.
|
||||||
@@ -524,15 +568,30 @@ func (s *Server) handleEscrowCeremonyClaim(w http.ResponseWriter, r *http.Reques
|
|||||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||||
cancel()
|
cancel()
|
||||||
|
outcome := uploadStored
|
||||||
|
if err != nil {
|
||||||
|
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2: the answer may be lost AFTER the store
|
||||||
|
}
|
||||||
|
|
||||||
s.escrowMu.Lock()
|
s.escrowMu.Lock()
|
||||||
s.escrowClaiming = false
|
s.escrowClaiming = false
|
||||||
if err != nil {
|
switch outcome {
|
||||||
|
case uploadNotStored:
|
||||||
s.escrowMu.Unlock()
|
s.escrowMu.Unlock()
|
||||||
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
|
s.logger.Warn("local-api: escrow claim — package upload to the hub failed; code kept claimable, previous package stays current (R-978)",
|
||||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||||
return
|
return
|
||||||
|
case uploadUnknown:
|
||||||
|
s.escrowMu.Unlock()
|
||||||
|
s.logger.Warn("local-api: escrow claim — upload failed and the hub cannot say which package it holds; code kept claimable (R-1254)",
|
||||||
|
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||||
|
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
s.logger.Warn("local-api: escrow claim — the upload's answer was lost, but the hub holds the new package; handing out the code (R-1254)",
|
||||||
|
"vmid", vmid, "job_id", jobID, "err", err)
|
||||||
}
|
}
|
||||||
s.escrowPending = nil
|
s.escrowPending = nil
|
||||||
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
|
if s.escrowJob != nil && s.escrowJob.JobID == jobID {
|
||||||
@@ -631,15 +690,30 @@ func (s *Server) handleEscrowCeremonyConfirm(w http.ResponseWriter, r *http.Requ
|
|||||||
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
uctx, cancel := context.WithTimeout(base, escrowUploadTimeout)
|
||||||
err := s.escrowCeremony.Upload(uctx, pkg)
|
err := s.escrowCeremony.Upload(uctx, pkg)
|
||||||
cancel()
|
cancel()
|
||||||
|
outcome := uploadStored
|
||||||
|
if err != nil {
|
||||||
|
outcome = s.outcomeAfterFailedUpload(pkg) // R-1254 H2
|
||||||
|
}
|
||||||
|
|
||||||
s.escrowMu.Lock()
|
s.escrowMu.Lock()
|
||||||
s.escrowClaiming = false
|
s.escrowClaiming = false
|
||||||
if err != nil {
|
switch outcome {
|
||||||
|
case uploadNotStored:
|
||||||
s.escrowMu.Unlock()
|
s.escrowMu.Unlock()
|
||||||
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
|
s.logger.Warn("local-api: escrow confirm — package upload to the hub failed; still waiting, previous package stays current (R-1151)",
|
||||||
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||||
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
writeErr(w, http.StatusBadGateway, "the new recovery package could not be stored on the hub — the previous code still works; try again")
|
||||||
return
|
return
|
||||||
|
case uploadUnknown:
|
||||||
|
s.escrowMu.Unlock()
|
||||||
|
s.logger.Warn("local-api: escrow confirm — upload failed and the hub cannot say which package it holds; still waiting (R-1254)",
|
||||||
|
"vmid", vmid, "job_id", jobID, "err", err, "duration_ms", time.Since(start).Milliseconds())
|
||||||
|
writeErr(w, http.StatusGatewayTimeout, escrowUnknownOutcomeMsg)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
s.logger.Warn("local-api: escrow confirm — the upload's answer was lost, but the hub holds the new package (R-1254)",
|
||||||
|
"vmid", vmid, "job_id", jobID, "err", err)
|
||||||
}
|
}
|
||||||
s.escrowPending = nil
|
s.escrowPending = nil
|
||||||
s.wipeEscrowConfirmLocked()
|
s.wipeEscrowConfirmLocked()
|
||||||
|
|||||||
@@ -41,9 +41,13 @@ type fakeEscrowHub struct {
|
|||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
current []byte
|
current []byte
|
||||||
uploads int
|
uploads int
|
||||||
err error // non-nil → the upload fails, nothing stored
|
err error // non-nil → the upload fails, nothing stored
|
||||||
gate chan struct{} // non-nil → the upload blocks until closed
|
// R-1254 H2: lost → the upload STORES the package and then fails (the answer is lost on the way back);
|
||||||
entered chan struct{} // non-nil → closed when an upload starts
|
// holdsErr → the hub cannot be asked which package it holds.
|
||||||
|
lost bool
|
||||||
|
holdsErr error
|
||||||
|
gate chan struct{} // non-nil → the upload blocks until closed
|
||||||
|
entered chan struct{} // non-nil → closed when an upload starts
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
||||||
@@ -61,9 +65,23 @@ func (f *fakeEscrowHub) upload(ctx context.Context, pkg []byte) error {
|
|||||||
}
|
}
|
||||||
f.current = append([]byte(nil), pkg...)
|
f.current = append([]byte(nil), pkg...)
|
||||||
f.uploads++
|
f.uploads++
|
||||||
|
if f.lost {
|
||||||
|
return context.DeadlineExceeded
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// holds answers „is the hub's current package exactly pkg?" — the R-1254 question (production compares the identity
|
||||||
|
// blob the hub serves back).
|
||||||
|
func (f *fakeEscrowHub) holds(_ context.Context, pkg []byte) (bool, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
if f.holdsErr != nil {
|
||||||
|
return false, f.holdsErr
|
||||||
|
}
|
||||||
|
return string(f.current) == string(pkg), nil
|
||||||
|
}
|
||||||
|
|
||||||
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
|
// opensWith reports whether the hub's CURRENT package opens with code r — the customer's question.
|
||||||
func (f *fakeEscrowHub) opensWith(r string) bool {
|
func (f *fakeEscrowHub) opensWith(r string) bool {
|
||||||
f.mu.Lock()
|
f.mu.Lock()
|
||||||
@@ -100,6 +118,7 @@ func newEscrowTestServerHub(t *testing.T, run ceremonyRunner, hub *fakeEscrowHub
|
|||||||
HubConfigured: true,
|
HubConfigured: true,
|
||||||
DRConfigured: func() bool { return true },
|
DRConfigured: func() bool { return true },
|
||||||
Upload: hub.upload,
|
Upload: hub.upload,
|
||||||
|
HubHolds: hub.holds,
|
||||||
}
|
}
|
||||||
srv.ceremonyRun = run
|
srv.ceremonyRun = run
|
||||||
srv.escrowSudoCheck = func(context.Context) error { return nil }
|
srv.escrowSudoCheck = func(context.Context) error { return nil }
|
||||||
|
|||||||
@@ -0,0 +1,98 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-1254 (the ceremony half, review round 2 HUNT-3 H2): the upload's answer can be lost or late (30 s) AFTER the hub
|
||||||
|
// stored the new package. The household was then told „the previous code still works" — false: the hub now holds the
|
||||||
|
// package the NEW code opens. The agent asks the hub which package it holds before it words the result:
|
||||||
|
// - the hub holds the new package → success (the code is handed out / the confirm completes);
|
||||||
|
// - the hub holds another package → the old sentence, now true;
|
||||||
|
// - the hub cannot be asked → 504 and a sentence that does NOT promise the previous code; the wait goes on.
|
||||||
|
|
||||||
|
// RED-PROOF: remove the hub question from the claim's failure branch → 502 „the previous code still works" while the
|
||||||
|
// hub holds the new package → FAILS.
|
||||||
|
func TestR1254_ALostAnswerAtTheClaimIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||||
|
hub := &fakeEscrowHub{lost: true}
|
||||||
|
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||||
|
h := srv.Handler()
|
||||||
|
startAndWait(t, srv, h)
|
||||||
|
code, body := claimCode(t, h)
|
||||||
|
if !hub.opensWith(firstR) {
|
||||||
|
t.Fatal("fixture: the hub must hold the new package (the answer was lost after the store)")
|
||||||
|
}
|
||||||
|
if code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||||
|
t.Fatalf("claim after a lost answer, the hub holding the new package: %d %s — want 200 with the code", code, body)
|
||||||
|
}
|
||||||
|
if strings.Contains(body, "previous code still works") {
|
||||||
|
t.Fatalf("the household was told the previous code still works: %s", body)
|
||||||
|
}
|
||||||
|
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"uploaded":true`) {
|
||||||
|
t.Fatalf("the job must read uploaded: %s", st)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The hub cannot be asked: no promise about the previous code, 504, the code stays claimable (a retry decides).
|
||||||
|
// RED-PROOF: as above → 502 with „the previous code still works" → FAILS.
|
||||||
|
func TestR1254_AnUnknownOutcomeAtTheClaimPromisesNothing(t *testing.T) {
|
||||||
|
hub := &fakeEscrowHub{lost: true, holdsErr: errors.New("hub unreachable")}
|
||||||
|
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||||
|
h := srv.Handler()
|
||||||
|
startAndWait(t, srv, h)
|
||||||
|
code, body := claimCode(t, h)
|
||||||
|
if code != http.StatusGatewayTimeout {
|
||||||
|
t.Fatalf("claim with an unknown outcome: %d %s — want 504", code, body)
|
||||||
|
}
|
||||||
|
if strings.Contains(body, "previous code still works") {
|
||||||
|
t.Fatalf("an unknown outcome promised the previous code: %s", body)
|
||||||
|
}
|
||||||
|
assertNoRIn(t, "unknown-outcome claim payload", body, firstR)
|
||||||
|
if st := do(t, h, "GET", "/escrow/ceremony/status", "A", "").Body.String(); !strings.Contains(st, `"claimable":true`) {
|
||||||
|
t.Fatalf("the code must stay claimable: %s", st)
|
||||||
|
}
|
||||||
|
// The hub answers again: the retry finds the new package already stored and hands the code out.
|
||||||
|
hub.mu.Lock()
|
||||||
|
hub.holdsErr = nil
|
||||||
|
hub.mu.Unlock()
|
||||||
|
if code, body := claimCode(t, h); code != http.StatusOK || !strings.Contains(body, firstR) {
|
||||||
|
t.Fatalf("retry: %d %s", code, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The hub holds ANOTHER package (the upload really failed): the old sentence stays, and is now true.
|
||||||
|
func TestR1254_AFailedUploadTheHubDidNotStoreKeepsTheOldSentence(t *testing.T) {
|
||||||
|
hub := &fakeEscrowHub{err: errors.New("hub returned HTTP 502")}
|
||||||
|
srv, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||||
|
h := srv.Handler()
|
||||||
|
startAndWait(t, srv, h)
|
||||||
|
if code, body := claimCode(t, h); code != http.StatusBadGateway || !strings.Contains(body, "previous code still works") {
|
||||||
|
t.Fatalf("a failed upload: %d %s — want 502 and the previous code named", code, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The same at the typed-back confirm (R-1151).
|
||||||
|
// RED-PROOF: remove the hub question from the confirm's failure branch → 502 → FAILS.
|
||||||
|
func TestR1254_ALostAnswerAtTheConfirmIsASuccessWhenTheHubHoldsTheNewPackage(t *testing.T) {
|
||||||
|
hub := &fakeEscrowHub{lost: true}
|
||||||
|
srv, _ := newEscrowTestServerHub(t, sequenceRunner(secondR), hub)
|
||||||
|
h := srv.Handler()
|
||||||
|
startAndWait(t, srv, h)
|
||||||
|
c := claimDeferred(t, h)
|
||||||
|
if code := confirm(t, h, c.typedBack(t)); code != http.StatusOK || !hub.opensWith(secondR) {
|
||||||
|
t.Fatalf("confirm after a lost answer: %d (opens: %v) — want 200", code, hub.opensWith(secondR))
|
||||||
|
}
|
||||||
|
hub.mu.Lock()
|
||||||
|
hub.lost, hub.holdsErr = true, errors.New("hub unreachable")
|
||||||
|
hub.mu.Unlock()
|
||||||
|
srv2, _ := newEscrowTestServerHub(t, sequenceRunner(firstR), hub)
|
||||||
|
h2 := srv2.Handler()
|
||||||
|
startAndWait(t, srv2, h2)
|
||||||
|
c2 := claimDeferred(t, h2)
|
||||||
|
if code := confirm(t, h2, c2.typedBack(t)); code != http.StatusGatewayTimeout {
|
||||||
|
t.Fatalf("confirm with an unknown outcome: %d — want 504", code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-528 — the OOM signal from the host side.
|
||||||
|
//
|
||||||
|
// MEASURED 2026-09-15 (scratch 9202, Docker 29.8.0) and 2026-09-16 (a fresh install in a nested VM): an app killed by
|
||||||
|
// its memory limit inside a Felhom LXC guest left Docker's State.OOMKilled false and sent no `oom` event, so the
|
||||||
|
// controller's OOM line (R-514) could not fire. The kernel's own counter does count every kill, and the host can read
|
||||||
|
// it: /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events, line `oom_kill N` (the
|
||||||
|
// systemd cgroup driver; `ns/docker/<id>/` under the cgroupfs driver). Read on Tester 1's host 2026-10-28: the files
|
||||||
|
// are mode 0444, readable by the non-root felhom-agent user — no sudo, no new root path.
|
||||||
|
//
|
||||||
|
// A container's scope is new at each start, so the counter is per container RUN: a count above 0 means this run had a
|
||||||
|
// kill. The controller asks every 30 s and raises its existing app_oom line (once per run) for any such container.
|
||||||
|
|
||||||
|
// readGuestOOMKills returns container id → oom_kill for one guest. available=false (no error) when the guest has no
|
||||||
|
// cgroup folder (a VM guest, or a container that is not running).
|
||||||
|
func readGuestOOMKills(root string, vmid int) (map[string]int64, bool, error) {
|
||||||
|
if root == "" {
|
||||||
|
root = "/sys/fs/cgroup"
|
||||||
|
}
|
||||||
|
base := filepath.Join(root, "lxc", strconv.Itoa(vmid), "ns")
|
||||||
|
if fi, err := os.Stat(base); err != nil || !fi.IsDir() {
|
||||||
|
return nil, false, nil
|
||||||
|
}
|
||||||
|
out := map[string]int64{}
|
||||||
|
for _, pattern := range []string{
|
||||||
|
filepath.Join(base, "system.slice", "docker-*.scope", "memory.events"),
|
||||||
|
filepath.Join(base, "docker", "*", "memory.events"),
|
||||||
|
} {
|
||||||
|
files, err := filepath.Glob(pattern)
|
||||||
|
if err != nil {
|
||||||
|
return nil, true, err
|
||||||
|
}
|
||||||
|
for _, f := range files {
|
||||||
|
id := strings.TrimSuffix(strings.TrimPrefix(filepath.Base(filepath.Dir(f)), "docker-"), ".scope")
|
||||||
|
if !isContainerID(id) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
n, err := readOOMKill(f)
|
||||||
|
if err != nil {
|
||||||
|
continue // a container that stopped between the glob and the read
|
||||||
|
}
|
||||||
|
out[id] = n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isContainerID(s string) bool {
|
||||||
|
if len(s) != 64 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, c := range s {
|
||||||
|
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func readOOMKill(path string) (int64, error) {
|
||||||
|
f, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
sc := bufio.NewScanner(f)
|
||||||
|
for sc.Scan() {
|
||||||
|
k, v, ok := strings.Cut(strings.TrimSpace(sc.Text()), " ")
|
||||||
|
if ok && k == "oom_kill" {
|
||||||
|
return strconv.ParseInt(strings.TrimSpace(v), 10, 64)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := sc.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
return 0, fmt.Errorf("no oom_kill line in %s", path)
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleGuestOOMKills is GET /guest/oom-kills: {"vmid", "available", "containers": {id: oom_kill}}.
|
||||||
|
func (s *Server) handleGuestOOMKills(w http.ResponseWriter, r *http.Request, vmid int) {
|
||||||
|
kills, avail, err := readGuestOOMKills(s.cgroupRoot, vmid)
|
||||||
|
if err != nil {
|
||||||
|
s.logger.Warn("local-api: guest oom-kills read failed (R-528)", "vmid", vmid, "err", err)
|
||||||
|
writeErr(w, http.StatusInternalServerError, "the guest's OOM counters could not be read")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if kills == nil {
|
||||||
|
kills = map[string]int64{}
|
||||||
|
}
|
||||||
|
writeOK(w, map[string]any{"vmid": vmid, "available": avail, "containers": kills})
|
||||||
|
}
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-528: Docker does not always report an OOM kill inside a Felhom LXC guest (OOMKilled stays false, no `oom` event —
|
||||||
|
// measured 2026-09-15/16 on two boxes), but the kernel's own counter does count it, and it is readable from the host:
|
||||||
|
// /sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events, line `oom_kill N`. GET /guest/oom-kills
|
||||||
|
// serves the asking guest's counters, and only that guest's.
|
||||||
|
|
||||||
|
func fakeCgroup(t *testing.T, root string, vmid, id, events string) {
|
||||||
|
t.Helper()
|
||||||
|
dir := filepath.Join(root, "lxc", vmid, "ns", "system.slice", "docker-"+id+".scope")
|
||||||
|
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "memory.events"), []byte(events), 0o444); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||||
|
const idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||||
|
const idOther = "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
|
||||||
|
|
||||||
|
// RED-PROOF: serve the counters of every guest under lxc/ (drop the vmid from the glob) → the other guest's container
|
||||||
|
// appears → FAILS; parse `oom ` instead of `oom_kill` → idA reads 9 → FAILS.
|
||||||
|
func TestR528_GuestOOMKillsServesThisGuestsCountersOnly(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
fakeCgroup(t, root, "100", idA, "low 0\nhigh 0\nmax 4\noom 9\noom_kill 3\noom_group_kill 0\n")
|
||||||
|
fakeCgroup(t, root, "100", idB, "low 0\nhigh 0\nmax 0\noom 0\noom_kill 0\n")
|
||||||
|
fakeCgroup(t, root, "200", idOther, "oom_kill 7\n")
|
||||||
|
// a scope that is not a container (no 64-hex id) and a cgroupfs-driver layout container
|
||||||
|
if err := os.MkdirAll(filepath.Join(root, "lxc", "100", "ns", "system.slice", "cron.service"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
dfDir := filepath.Join(root, "lxc", "100", "ns", "docker", idOther[:8]+strings.Repeat("d", 56))
|
||||||
|
if err := os.MkdirAll(dfDir, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = os.WriteFile(filepath.Join(dfDir, "memory.events"), []byte("oom_kill 2\n"), 0o444)
|
||||||
|
|
||||||
|
got, avail, err := readGuestOOMKills(root, 100)
|
||||||
|
if err != nil || !avail {
|
||||||
|
t.Fatalf("read: avail=%v err=%v", avail, err)
|
||||||
|
}
|
||||||
|
want := map[string]int64{idA: 3, idB: 0, idOther[:8] + strings.Repeat("d", 56): 2}
|
||||||
|
if len(got) != len(want) {
|
||||||
|
t.Fatalf("got %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
for k, v := range want {
|
||||||
|
if got[k] != v {
|
||||||
|
t.Fatalf("got %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, leaked := got[idOther]; leaked {
|
||||||
|
t.Fatal("another guest's container leaked into this guest's answer")
|
||||||
|
}
|
||||||
|
// A guest with no cgroup folder (a VM, or not running): not available, no error.
|
||||||
|
if _, avail, err := readGuestOOMKills(root, 300); avail || err != nil {
|
||||||
|
t.Fatalf("a guest without a cgroup folder: avail=%v err=%v", avail, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Through the route, with the guest's own token.
|
||||||
|
func TestR528_TheRouteAnswersTheTokensGuest(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
fakeCgroup(t, root, "8200", idA, "oom_kill 5\n")
|
||||||
|
fakeCgroup(t, root, "9300", idB, "oom_kill 1\n") // token B's guest: never in A's answer
|
||||||
|
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
|
||||||
|
srv.cgroupRoot = root
|
||||||
|
w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "A", "")
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("route: %d %s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
var env struct {
|
||||||
|
Data struct {
|
||||||
|
VMID int `json:"vmid"`
|
||||||
|
Available bool `json:"available"`
|
||||||
|
Containers map[string]int64 `json:"containers"`
|
||||||
|
} `json:"data"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if env.Data.VMID != 8200 || !env.Data.Available || len(env.Data.Containers) != 1 || env.Data.Containers[idA] != 5 {
|
||||||
|
t.Fatalf("answer %s — want guest 8200's one container with 5", w.Body.String())
|
||||||
|
}
|
||||||
|
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("a cross-guest ask: %d, want 403", w.Code)
|
||||||
|
}
|
||||||
|
if w := do(t, srv.Handler(), "GET", "/guest/oom-kills", "", ""); w.Code != http.StatusUnauthorized {
|
||||||
|
t.Fatalf("no token: %d", w.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -152,6 +152,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
|||||||
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
|
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
|
||||||
|
// would fail the reboot's own start half).
|
||||||
|
if s.rebooting(g.VMID) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
|
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,258 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The operator's actions for THIS agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).
|
||||||
|
//
|
||||||
|
// THE DOOR IS THE HOST REPORT'S REPLY, the controller's shape: the operator presses a button on the hub's host page, the
|
||||||
|
// hub stores a row and pokes the box (the poke's out-of-cycle report gets the reply in seconds; without the tunnel the
|
||||||
|
// 15-minute cycle carries it), the reply lists `operator_actions: [{id, action, arg}]`, this agent acts ONCE per id, and
|
||||||
|
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
|
||||||
|
// The result is re-sent until the reply stops listing the id.
|
||||||
|
//
|
||||||
|
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
|
||||||
|
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
|
||||||
|
//
|
||||||
|
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
|
||||||
|
// the guest over the existing grants, with the supervisor's guards — not during a controller swap, not parked, not on a
|
||||||
|
// stopped, locked or vzdump-busy guest, not when the guest list or the vzdump state cannot be read — and the thrash brake
|
||||||
|
// counts these restarts too: 3 restarts (the supervisor's or the operator's) inside 15 minutes refuse the press, and
|
||||||
|
// presses stay refused for 30 minutes. It works with the controller stopped or stuck: the agent does it, not the
|
||||||
|
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
|
||||||
|
|
||||||
|
const (
|
||||||
|
// OpRestartController is the agent's first operator verb.
|
||||||
|
OpRestartController = "restart_controller"
|
||||||
|
|
||||||
|
opOutcomeDone = "done"
|
||||||
|
opOutcomeRefused = "refused"
|
||||||
|
opOutcomeFailed = "failed"
|
||||||
|
|
||||||
|
// opBrakeRefusal is how long presses stay refused after the brake refused one.
|
||||||
|
opBrakeRefusal = 30 * time.Minute
|
||||||
|
// opForgetAfter drops a started id the reply no longer lists.
|
||||||
|
opForgetAfter = 24 * time.Hour
|
||||||
|
|
||||||
|
defaultOpHealthPoll = 2 * time.Second
|
||||||
|
// defaultOpHealthWait covers the bootstrap's pull-less rm/run (seconds), Docker's start period (10 s) and its first
|
||||||
|
// probe (30 s interval) with room: the controller's own HEALTHCHECK is the answer.
|
||||||
|
defaultOpHealthWait = 3 * time.Minute
|
||||||
|
// opHealthTemplate reads running + health over the swap's `docker inspect -f` grant.
|
||||||
|
opHealthTemplate = "{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}"
|
||||||
|
)
|
||||||
|
|
||||||
|
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
|
||||||
|
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
|
||||||
|
|
||||||
|
// OperatorActionNames returns the closed list (sorted copy).
|
||||||
|
func OperatorActionNames() []string {
|
||||||
|
out := append([]string(nil), operatorActionNames...)
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
type opActionState struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
started map[int64]time.Time
|
||||||
|
results map[int64]hub.OperatorActionResult
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetOperatorActionTrigger wires the report loop's out-of-band trigger (the result goes at once, not in 15 minutes).
|
||||||
|
func (s *Server) SetOperatorActionTrigger(f func()) { s.opTrigger = f }
|
||||||
|
|
||||||
|
// OnEnvelope is the report loop's hook (hub.EnvelopeObserver): it starts each newly listed action once and forgets
|
||||||
|
// the ids the reply no longer lists (the hub has their results).
|
||||||
|
func (s *Server) OnEnvelope(_ context.Context, env *hub.ControlEnvelope) {
|
||||||
|
if env == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
listed := map[int64]bool{}
|
||||||
|
var start []hub.OperatorAction
|
||||||
|
now := time.Now()
|
||||||
|
s.opActs.mu.Lock()
|
||||||
|
if s.opActs.started == nil {
|
||||||
|
s.opActs.started = map[int64]time.Time{}
|
||||||
|
s.opActs.results = map[int64]hub.OperatorActionResult{}
|
||||||
|
}
|
||||||
|
for _, a := range env.OperatorActions {
|
||||||
|
listed[a.ID] = true
|
||||||
|
if _, ok := s.opActs.started[a.ID]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
s.opActs.started[a.ID] = now
|
||||||
|
start = append(start, a)
|
||||||
|
}
|
||||||
|
for id, at := range s.opActs.started {
|
||||||
|
if listed[id] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, answered := s.opActs.results[id]; answered || now.Sub(at) > opForgetAfter {
|
||||||
|
delete(s.opActs.started, id)
|
||||||
|
delete(s.opActs.results, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
s.opActs.mu.Unlock()
|
||||||
|
for _, a := range start {
|
||||||
|
go s.runOperatorAction(a) // a restart waits for the health answer — never block the report loop
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// OperatorActionResults is the report's answer list, oldest id first.
|
||||||
|
func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
|
||||||
|
s.opActs.mu.Lock()
|
||||||
|
defer s.opActs.mu.Unlock()
|
||||||
|
out := make([]hub.OperatorActionResult, 0, len(s.opActs.results))
|
||||||
|
for _, r := range s.opActs.results {
|
||||||
|
out = append(out, r)
|
||||||
|
}
|
||||||
|
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) runOperatorAction(a hub.OperatorAction) {
|
||||||
|
limit := s.opWait() + time.Minute
|
||||||
|
if a.Action == OpRebootGuest {
|
||||||
|
limit += opRebootTimeout
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), limit)
|
||||||
|
defer cancel()
|
||||||
|
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
|
||||||
|
switch a.Action {
|
||||||
|
case OpRestartController:
|
||||||
|
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
|
||||||
|
case OpRebootGuest:
|
||||||
|
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
|
||||||
|
}
|
||||||
|
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
|
||||||
|
"outcome", outcome, "message", msg)
|
||||||
|
s.opActs.mu.Lock()
|
||||||
|
if _, still := s.opActs.started[a.ID]; still {
|
||||||
|
s.opActs.results[a.ID] = hub.OperatorActionResult{ID: a.ID, Outcome: outcome, Message: msg}
|
||||||
|
}
|
||||||
|
s.opActs.mu.Unlock()
|
||||||
|
if s.opTrigger != nil {
|
||||||
|
s.opTrigger()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) opPoll() time.Duration {
|
||||||
|
if s.opHealthPoll > 0 {
|
||||||
|
return s.opHealthPoll
|
||||||
|
}
|
||||||
|
return defaultOpHealthPoll
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) opWait() time.Duration {
|
||||||
|
if s.opHealthWait > 0 {
|
||||||
|
return s.opHealthWait
|
||||||
|
}
|
||||||
|
return defaultOpHealthWait
|
||||||
|
}
|
||||||
|
|
||||||
|
// operatorRestartController is the press: the supervisor's guards, the brake, the restart, then the wait for the
|
||||||
|
// controller's healthy answer. Returns the outcome and the sentence the host page shows.
|
||||||
|
func (s *Server) operatorRestartController(ctx context.Context, id int64, arg string) (string, string) {
|
||||||
|
vmid, err := strconv.Atoi(arg)
|
||||||
|
if err != nil || strconv.Itoa(vmid) != arg {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||||
|
}
|
||||||
|
if s.staleLock == nil || s.guestExec == nil {
|
||||||
|
return opOutcomeRefused, "refused: the controller supervisor is not wired on this agent"
|
||||||
|
}
|
||||||
|
if !s.provisionedGuest(vmid) {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||||
|
}
|
||||||
|
guests, err := s.staleLock.Guests(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||||
|
}
|
||||||
|
status := ""
|
||||||
|
for _, g := range guests {
|
||||||
|
if g.VMID == vmid {
|
||||||
|
status = g.Status
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if status != "running" {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||||
|
}
|
||||||
|
if s.controllerParked(vmid) {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||||
|
}
|
||||||
|
s.swapMu.Lock()
|
||||||
|
swapping := s.swapInFlight[vmid]
|
||||||
|
s.swapMu.Unlock()
|
||||||
|
if swapping {
|
||||||
|
return opOutcomeRefused, "refused: a controller swap (update) is in flight — it owns the controller"
|
||||||
|
}
|
||||||
|
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||||
|
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||||
|
} else if lock != "" {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||||
|
}
|
||||||
|
if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||||
|
return opOutcomeRefused, "refused: a whole-guest backup (vzdump) may be in flight for the guest"
|
||||||
|
}
|
||||||
|
|
||||||
|
// The brake: every restart counts — the supervisor's and the operator's.
|
||||||
|
now := s.clock()
|
||||||
|
s.ctrlSup.mu.Lock()
|
||||||
|
st := s.supState(vmid)
|
||||||
|
if now.Before(st.opBrakeUntil) {
|
||||||
|
until := st.opBrakeUntil
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||||
|
}
|
||||||
|
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||||
|
if len(st.restarts) >= controllerCrashloopMax {
|
||||||
|
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||||
|
until := st.opBrakeUntil
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||||
|
}
|
||||||
|
reason := fmt.Sprintf("the operator via the hub (action #%d)", id)
|
||||||
|
st.restarts = append(st.restarts, now)
|
||||||
|
st.restartsTotal++
|
||||||
|
st.lastRestartAt = now
|
||||||
|
st.lastReason = reason
|
||||||
|
st.notRunningSeen = 0
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
|
||||||
|
s.logger.Warn("controller-supervisor: RESTARTING the controller on the operator's press", "vmid", vmid, "action_id", id, "unit", bootstrapUnit)
|
||||||
|
began := time.Now()
|
||||||
|
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||||
|
return opOutcomeFailed, fmt.Sprintf("the restart of guest %d's controller failed: %v", vmid, err)
|
||||||
|
}
|
||||||
|
for time.Since(began) < s.opWait() {
|
||||||
|
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||||
|
if err == nil {
|
||||||
|
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||||
|
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||||
|
secs := int(time.Since(began).Round(time.Second) / time.Second)
|
||||||
|
return opOutcomeDone, fmt.Sprintf("restarted; the controller answered after %d s (Docker's health check: healthy)", secs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return opOutcomeFailed, "restarted, but the wait for its answer was cut"
|
||||||
|
case <-time.After(s.opPoll()):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return opOutcomeFailed, fmt.Sprintf("restarted, but no healthy answer within %s — look at the box", s.opWait().Round(time.Second))
|
||||||
|
}
|
||||||
|
|
||||||
|
func orDash(s string) string {
|
||||||
|
if s == "" {
|
||||||
|
return "not listed"
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
@@ -0,0 +1,222 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Operator brief 2026-10-29, Part 1: „Restart controller" from the hub. The agent's closed list holds one verb; the act
|
||||||
|
// is the supervisor's own restart with the supervisor's guards, and the brake counts operator restarts too.
|
||||||
|
|
||||||
|
// opExec answers the supervisor's fake plus the health read the operator restart waits on.
|
||||||
|
type opExec struct {
|
||||||
|
*supExec
|
||||||
|
health string // what {{.State.Health.Status}} reads after the restart
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *opExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||||
|
if len(args) >= 5 && args[0] == "docker" && args[1] == "inspect" && args[3] == opHealthTemplate {
|
||||||
|
f.mu.Lock()
|
||||||
|
st := f.status[vmid]
|
||||||
|
f.mu.Unlock()
|
||||||
|
if st == "" {
|
||||||
|
return "", errors.New("pct exec: exit status 1: Error: No such object: felhom-controller")
|
||||||
|
}
|
||||||
|
return map[bool]string{true: "true", false: "false"}[st == "running"] + "|" + f.health + "\n", nil
|
||||||
|
}
|
||||||
|
return f.supExec.GuestExec(ctx, vmid, args...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func opServer(t *testing.T, status string) (*Server, *opExec, *fakeGuestPowerCtl, *supClock, string, *atomic.Int32) {
|
||||||
|
t.Helper()
|
||||||
|
ex := &opExec{supExec: &supExec{status: map[int]string{9201: status}, onRestart: "running"}, health: "healthy"}
|
||||||
|
ctl := runningGuest(9201)
|
||||||
|
s, clk, dir := supServer(t, nil, ctl, 9201)
|
||||||
|
s.guestExec = ex
|
||||||
|
s.opHealthPoll = time.Millisecond
|
||||||
|
s.opHealthWait = 2 * time.Second
|
||||||
|
var triggered atomic.Int32
|
||||||
|
s.SetOperatorActionTrigger(func() { triggered.Add(1) })
|
||||||
|
return s, ex, ctl, clk, dir, &triggered
|
||||||
|
}
|
||||||
|
|
||||||
|
func env(acts ...hub.OperatorAction) *hub.ControlEnvelope {
|
||||||
|
return &hub.ControlEnvelope{OperatorActions: acts}
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitResult waits for the action's result (the act runs in the background).
|
||||||
|
func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||||
|
t.Helper()
|
||||||
|
deadline := time.Now().Add(3 * time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
for _, r := range s.OperatorActionResults() {
|
||||||
|
if r.ID == id {
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(2 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatalf("no result for #%d", id)
|
||||||
|
return hub.OperatorActionResult{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RED-PROOF: seen failing before the list existed (build failure, then the names).
|
||||||
|
func TestOpActions_AgentClosedList(t *testing.T) {
|
||||||
|
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Anything outside the list — and a bad argument — is refused and NOTHING is called.
|
||||||
|
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
|
||||||
|
s, ex, _, _, _, _ := opServer(t, "running")
|
||||||
|
s.OnEnvelope(context.Background(), env(
|
||||||
|
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
|
||||||
|
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
|
||||||
|
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
|
||||||
|
))
|
||||||
|
for _, id := range []int64{1, 2, 3} {
|
||||||
|
if r := waitResult(t, s, id); r.Outcome != "refused" {
|
||||||
|
t.Errorf("#%d: %+v, want refused", id, r)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if n := ex.count(9201); n != 0 {
|
||||||
|
t.Fatalf("a refused action restarted the controller (%d)", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The press restarts ONCE, says how long the controller took to answer, asks for a report at once, and re-sends the
|
||||||
|
// result until the hub stops listing the id. RED-PROOF: drop the GuestExec restart → restarts 0 → FAILS.
|
||||||
|
func TestOpActions_RestartReportsTheAnswerTime(t *testing.T) {
|
||||||
|
s, ex, _, _, _, trig := opServer(t, "running")
|
||||||
|
a := hub.OperatorAction{ID: 7, Action: "restart_controller", Arg: "9201"}
|
||||||
|
s.OnEnvelope(context.Background(), env(a))
|
||||||
|
r := waitResult(t, s, 7)
|
||||||
|
if r.Outcome != "done" || !strings.Contains(r.Message, "answered after") {
|
||||||
|
t.Fatalf("result %+v", r)
|
||||||
|
}
|
||||||
|
if n := ex.count(9201); n != 1 {
|
||||||
|
t.Fatalf("restarts = %d, want 1", n)
|
||||||
|
}
|
||||||
|
if trig.Load() == 0 {
|
||||||
|
t.Fatal("the agent did not ask for a report — the result would wait 15 minutes")
|
||||||
|
}
|
||||||
|
s.OnEnvelope(context.Background(), env(a)) // the hub still lists it: re-send, never act again
|
||||||
|
time.Sleep(100 * time.Millisecond) // a wrong second act runs in the background — give it time to show
|
||||||
|
if n := ex.count(9201); n != 1 {
|
||||||
|
t.Fatalf("a re-listed id acted again (%d)", n)
|
||||||
|
}
|
||||||
|
if len(s.OperatorActionResults()) != 1 {
|
||||||
|
t.Fatal("the result must be re-sent while listed")
|
||||||
|
}
|
||||||
|
s.OnEnvelope(context.Background(), env()) // the hub recorded it
|
||||||
|
if left := s.OperatorActionResults(); len(left) != 0 {
|
||||||
|
t.Fatalf("a recorded result is still sent: %+v", left)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The point of the agent doing it: a STOPPED controller is restarted too (the controller cannot restart itself).
|
||||||
|
func TestOpActions_WorksWithTheControllerStopped(t *testing.T) {
|
||||||
|
s, ex, _, _, _, _ := opServer(t, "exited")
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 8, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
if r := waitResult(t, s, 8); r.Outcome != "done" || ex.count(9201) != 1 {
|
||||||
|
t.Fatalf("result %+v, restarts %d", r, ex.count(9201))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each supervisor guard refuses the press, with the reason. RED-PROOF: drop the lock check → the locked case
|
||||||
|
// restarts → FAILS (each guard seen failing by its own mutation).
|
||||||
|
func TestOpActions_GuardsRefuse(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name, want string
|
||||||
|
setup func(s *Server, ctl *fakeGuestPowerCtl, dir string)
|
||||||
|
}{
|
||||||
|
{"parked", "parked", func(_ *Server, _ *fakeGuestPowerCtl, dir string) {
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{"swap", "swap", func(s *Server, _ *fakeGuestPowerCtl, _ string) { s.swapInFlight[9201] = true }},
|
||||||
|
{"locked", "locked", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.locks[9201] = "backup" }},
|
||||||
|
{"vzdump", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||||
|
{"vzdump unknown", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupErr = errors.New("tasks unreadable") }},
|
||||||
|
{"guest stopped", "not running", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guests[0].Status = "stopped" }},
|
||||||
|
{"guest list", "guest list", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guestsErr = errors.New("api down") }},
|
||||||
|
}
|
||||||
|
for i, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
s, ex, ctl, _, dir, _ := opServer(t, "running")
|
||||||
|
tc.setup(s, ctl, dir)
|
||||||
|
id := int64(100 + i)
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
r := waitResult(t, s, id)
|
||||||
|
if r.Outcome != "refused" || !strings.Contains(strings.ToLower(r.Message), tc.want) {
|
||||||
|
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||||
|
}
|
||||||
|
if n := ex.count(9201); n != 0 {
|
||||||
|
t.Fatalf("guard %q did not hold: restarted %d", tc.name, n)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The thrash brake counts operator restarts AND the supervisor's: 3 in 15 minutes → the next press is refused, and
|
||||||
|
// presses stay refused for 30 minutes. RED-PROOF: drop the brake → the 4th press restarts → FAILS.
|
||||||
|
func TestOpActions_TheBrakeCountsEveryRestart(t *testing.T) {
|
||||||
|
s, ex, _, clk, _, _ := opServer(t, "running")
|
||||||
|
// one restart by the supervisor itself
|
||||||
|
ex.mu.Lock()
|
||||||
|
ex.status[9201] = "exited"
|
||||||
|
ex.mu.Unlock()
|
||||||
|
s.ControllerSupervisorTick(context.Background())
|
||||||
|
s.ControllerSupervisorTick(context.Background())
|
||||||
|
if ex.count(9201) != 1 {
|
||||||
|
t.Fatalf("setup: the supervisor did not restart (%d)", ex.count(9201))
|
||||||
|
}
|
||||||
|
for id := int64(1); id <= 2; id++ {
|
||||||
|
clk.t = clk.t.Add(time.Minute)
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||||
|
t.Fatalf("press %d: %+v", id, r)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(time.Minute)
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
r := waitResult(t, s, 3)
|
||||||
|
if r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||||
|
t.Fatalf("the 4th restart in 15 minutes: %+v", r)
|
||||||
|
}
|
||||||
|
if n := ex.count(9201); n != 3 {
|
||||||
|
t.Fatalf("restarts = %d, want 3", n)
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(20 * time.Minute) // the window has passed, but the 30-minute refusal has not
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
if r := waitResult(t, s, 4); r.Outcome != "refused" {
|
||||||
|
t.Fatalf("inside the 30-minute refusal: %+v", r)
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(11 * time.Minute)
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 5, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
if r := waitResult(t, s, 5); r.Outcome != "done" {
|
||||||
|
t.Fatalf("after the refusal: %+v", r)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A controller that does not come back healthy is a FAILED press, said plainly.
|
||||||
|
func TestOpActions_NoHealthyAnswerIsFailed(t *testing.T) {
|
||||||
|
s, ex, _, _, _, _ := opServer(t, "running")
|
||||||
|
ex.health = "unhealthy"
|
||||||
|
s.opHealthWait = 30 * time.Millisecond
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 9, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
if r := waitResult(t, s, 9); r.Outcome != "failed" || !strings.Contains(r.Message, "no healthy answer") {
|
||||||
|
t.Fatalf("result %+v", r)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
|
||||||
|
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
|
||||||
|
//
|
||||||
|
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
|
||||||
|
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
|
||||||
|
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
|
||||||
|
//
|
||||||
|
// THE GUARDS (each refuses with the reason in the result; nothing is called):
|
||||||
|
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
|
||||||
|
// - a controller swap (update) is in flight;
|
||||||
|
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
|
||||||
|
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
|
||||||
|
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
|
||||||
|
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
|
||||||
|
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
|
||||||
|
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
|
||||||
|
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
|
||||||
|
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
|
||||||
|
// alone while the reboot runs (rebootInFlight).
|
||||||
|
//
|
||||||
|
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
|
||||||
|
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
|
||||||
|
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
|
||||||
|
// one day). Pinned by opreboot_test.go.
|
||||||
|
|
||||||
|
const (
|
||||||
|
// OpRebootGuest is the agent's second operator verb (D65 = A).
|
||||||
|
OpRebootGuest = "reboot_guest"
|
||||||
|
|
||||||
|
// opRebootEvery: at most one operator reboot per guest per hour.
|
||||||
|
opRebootEvery = time.Hour
|
||||||
|
// opRebootNoticeFor: how long the household's dashboard shows the line.
|
||||||
|
opRebootNoticeFor = 24 * time.Hour
|
||||||
|
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
|
||||||
|
opRebootTimeout = 5 * time.Minute
|
||||||
|
|
||||||
|
operatorRebootFile = "operator-reboot.json"
|
||||||
|
|
||||||
|
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
|
||||||
|
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
|
||||||
|
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
|
||||||
|
// Docker does not hide it.
|
||||||
|
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
|
||||||
|
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
|
||||||
|
// stopped writing (stuck or killed) — not a busy one.
|
||||||
|
controllerBusyFresh = 90 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
|
||||||
|
type operatorRebootRecord struct {
|
||||||
|
At time.Time `json:"at"`
|
||||||
|
ActionID int64 `json:"action_id"`
|
||||||
|
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
|
||||||
|
ControllerAfterS int `json:"controller_after_s,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) operatorRebootPath(vmid int) string {
|
||||||
|
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
|
||||||
|
// must not silently vanish).
|
||||||
|
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
|
||||||
|
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
|
||||||
|
if errors.Is(err, os.ErrNotExist) {
|
||||||
|
return operatorRebootRecord{}, false, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return operatorRebootRecord{}, false, err
|
||||||
|
}
|
||||||
|
var rec operatorRebootRecord
|
||||||
|
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||||
|
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
|
||||||
|
}
|
||||||
|
return rec, !rec.At.IsZero(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
|
||||||
|
raw, err := json.Marshal(rec)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
p := s.operatorRebootPath(vmid)
|
||||||
|
tmp := p + ".tmp"
|
||||||
|
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return os.Rename(tmp, p)
|
||||||
|
}
|
||||||
|
|
||||||
|
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
|
||||||
|
func (s *Server) rebooting(vmid int) bool {
|
||||||
|
s.rebootMu.Lock()
|
||||||
|
defer s.rebootMu.Unlock()
|
||||||
|
return s.rebootInFlight[vmid]
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) setRebooting(vmid int, on bool) {
|
||||||
|
s.rebootMu.Lock()
|
||||||
|
defer s.rebootMu.Unlock()
|
||||||
|
if s.rebootInFlight == nil {
|
||||||
|
s.rebootInFlight = map[int]bool{}
|
||||||
|
}
|
||||||
|
if on {
|
||||||
|
s.rebootInFlight[vmid] = true
|
||||||
|
} else {
|
||||||
|
delete(s.rebootInFlight, vmid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) rebooter() interface {
|
||||||
|
RebootGuest(ctx context.Context, vmid int) error
|
||||||
|
} {
|
||||||
|
if s.opRebooter != nil {
|
||||||
|
return s.opRebooter
|
||||||
|
}
|
||||||
|
if s.guestAttach != nil {
|
||||||
|
return s.guestAttach
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
|
||||||
|
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
|
||||||
|
// (the caller refuses — it cannot prove no backup runs). Two callers: the operator's reboot (D65) and R-1344's restart
|
||||||
|
// of a controller that does not answer (controllersupervisor.go restartUnresponsive).
|
||||||
|
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
|
||||||
|
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
|
||||||
|
if err != nil {
|
||||||
|
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
|
||||||
|
return false, "", nil
|
||||||
|
}
|
||||||
|
return false, "", err
|
||||||
|
}
|
||||||
|
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
|
||||||
|
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
|
||||||
|
if perr != nil {
|
||||||
|
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
|
||||||
|
}
|
||||||
|
if s.clock().Sub(at) > controllerBusyFresh {
|
||||||
|
s.logger.Info("controller-busy: the controller's busy file is stale — the controller stopped writing it; not busy",
|
||||||
|
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
|
||||||
|
return false, "", nil
|
||||||
|
}
|
||||||
|
why := "work in progress"
|
||||||
|
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
|
||||||
|
why = strings.TrimSpace(lines[1])
|
||||||
|
}
|
||||||
|
return true, why, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
|
||||||
|
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
|
||||||
|
vmid, err := strconv.Atoi(arg)
|
||||||
|
if err != nil || strconv.Itoa(vmid) != arg {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||||
|
}
|
||||||
|
rb := s.rebooter()
|
||||||
|
if s.staleLock == nil || s.guestExec == nil || rb == nil {
|
||||||
|
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
|
||||||
|
}
|
||||||
|
if !s.provisionedGuest(vmid) {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||||
|
}
|
||||||
|
if s.rebooting(vmid) {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
|
||||||
|
}
|
||||||
|
guests, err := s.staleLock.Guests(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||||
|
}
|
||||||
|
status := ""
|
||||||
|
for _, g := range guests {
|
||||||
|
if g.VMID == vmid {
|
||||||
|
status = g.Status
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if status != "running" {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||||
|
}
|
||||||
|
if s.controllerParked(vmid) {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||||
|
}
|
||||||
|
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
|
||||||
|
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||||
|
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||||
|
} else if lock != "" {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||||
|
}
|
||||||
|
|
||||||
|
// At most one per hour.
|
||||||
|
now := s.clock()
|
||||||
|
last, had, err := s.lastOperatorReboot(vmid)
|
||||||
|
if err != nil {
|
||||||
|
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
|
||||||
|
}
|
||||||
|
if had && now.Sub(last.At) < opRebootEvery {
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
|
||||||
|
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
|
||||||
|
s.ctrlSup.mu.Lock()
|
||||||
|
st := s.supState(vmid)
|
||||||
|
if now.Before(st.opBrakeUntil) {
|
||||||
|
until := st.opBrakeUntil
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||||
|
}
|
||||||
|
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||||
|
if len(st.restarts) >= controllerCrashloopMax {
|
||||||
|
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||||
|
until := st.opBrakeUntil
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||||
|
}
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
|
||||||
|
// The heavy-operation gate, held until the guest is up again.
|
||||||
|
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
|
||||||
|
if !ok {
|
||||||
|
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
|
||||||
|
}
|
||||||
|
defer release()
|
||||||
|
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
|
||||||
|
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
|
||||||
|
}
|
||||||
|
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
|
||||||
|
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
|
||||||
|
} else if cbusy {
|
||||||
|
return opOutcomeRefused, "refused: the controller is busy — " + why
|
||||||
|
}
|
||||||
|
|
||||||
|
// The record FIRST: the hourly limit and the household's line both read it.
|
||||||
|
rec := operatorRebootRecord{At: now, ActionID: id}
|
||||||
|
if err := s.saveOperatorReboot(vmid, rec); err != nil {
|
||||||
|
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
|
||||||
|
}
|
||||||
|
s.ctrlSup.mu.Lock()
|
||||||
|
st = s.supState(vmid)
|
||||||
|
st.restarts = append(st.restarts, now)
|
||||||
|
st.restartsTotal++
|
||||||
|
st.lastRestartAt = now
|
||||||
|
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
|
||||||
|
st.notRunningSeen = 0
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
s.setRebooting(vmid, true)
|
||||||
|
defer s.setRebooting(vmid, false)
|
||||||
|
|
||||||
|
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
|
||||||
|
began := time.Now()
|
||||||
|
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
|
||||||
|
err = rb.RebootGuest(rctx, vmid)
|
||||||
|
cancel()
|
||||||
|
if err != nil {
|
||||||
|
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
|
||||||
|
}
|
||||||
|
upS := int(time.Since(began).Round(time.Second) / time.Second)
|
||||||
|
rec.GuestUpAfterS = upS
|
||||||
|
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
|
||||||
|
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
|
||||||
|
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||||
|
if err == nil {
|
||||||
|
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||||
|
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||||
|
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
|
||||||
|
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
|
||||||
|
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
|
||||||
|
}
|
||||||
|
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
|
||||||
|
case <-time.After(s.opPoll()):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ = s.saveOperatorReboot(vmid, rec)
|
||||||
|
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
|
||||||
|
}
|
||||||
|
|
||||||
|
// rebootBlockerSentence names jobBlocker's code in the result's words.
|
||||||
|
func rebootBlockerSentence(code string) string {
|
||||||
|
switch code {
|
||||||
|
case PowerOffBackup:
|
||||||
|
return "a backup is running"
|
||||||
|
case PowerOffCtrlUpdate:
|
||||||
|
return "a controller update is running"
|
||||||
|
case PowerOffDrive:
|
||||||
|
return "a drive or network-storage job is running"
|
||||||
|
case PowerOffEscrow:
|
||||||
|
return "an escrow ceremony is running"
|
||||||
|
case PowerOffUnknown:
|
||||||
|
return "the running jobs cannot be read"
|
||||||
|
default:
|
||||||
|
return "the host is busy"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
|
||||||
|
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
|
||||||
|
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
|
||||||
|
rec, ok, err := s.lastOperatorReboot(vmid)
|
||||||
|
if err != nil {
|
||||||
|
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
|
||||||
|
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
|
||||||
|
}
|
||||||
@@ -0,0 +1,306 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
|
||||||
|
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
|
||||||
|
|
||||||
|
// rbExec adds the controller's busy file to the operator-restart fake.
|
||||||
|
type rbExec struct {
|
||||||
|
*opExec
|
||||||
|
busy string // the file's content; "" = no file
|
||||||
|
busyErr error // pct exec itself failed
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||||
|
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
|
||||||
|
if f.busyErr != nil {
|
||||||
|
return "", f.busyErr
|
||||||
|
}
|
||||||
|
if f.busy == "" {
|
||||||
|
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
|
||||||
|
}
|
||||||
|
return f.busy, nil
|
||||||
|
}
|
||||||
|
return f.opExec.GuestExec(ctx, vmid, args...)
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeRebooter struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
n int
|
||||||
|
err error
|
||||||
|
after func()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.n++
|
||||||
|
if r.after != nil {
|
||||||
|
r.after()
|
||||||
|
}
|
||||||
|
return r.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *fakeRebooter) count() int {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return r.n
|
||||||
|
}
|
||||||
|
|
||||||
|
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
|
||||||
|
t.Helper()
|
||||||
|
s, ox, ctl, clk, dir, _ := opServer(t, "running")
|
||||||
|
ex := &rbExec{opExec: ox}
|
||||||
|
s.guestExec = ex
|
||||||
|
rb := &fakeRebooter{}
|
||||||
|
s.opRebooter = rb
|
||||||
|
s.inFlight = &backup.InFlight{}
|
||||||
|
return s, ex, rb, ctl, clk, dir
|
||||||
|
}
|
||||||
|
|
||||||
|
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||||
|
t.Helper()
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
|
||||||
|
return waitResult(t, s, id)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
|
||||||
|
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
|
||||||
|
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
|
||||||
|
s, _, rb, _, clk, _ := rbServer(t)
|
||||||
|
r := pressReboot(t, s, 1)
|
||||||
|
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
|
||||||
|
t.Fatalf("result %+v", r)
|
||||||
|
}
|
||||||
|
if rb.count() != 1 {
|
||||||
|
t.Fatalf("reboots = %d, want 1", rb.count())
|
||||||
|
}
|
||||||
|
rec, ok, err := s.lastOperatorReboot(9201)
|
||||||
|
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
|
||||||
|
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
|
||||||
|
}
|
||||||
|
if s.inFlight.Busy() != "" {
|
||||||
|
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
|
||||||
|
}
|
||||||
|
if s.rebooting(9201) {
|
||||||
|
t.Fatal("the reboot-in-flight flag outlived the reboot")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
|
||||||
|
func TestOpReboot_GuardsRefuse(t *testing.T) {
|
||||||
|
fresh := func(clk *supClock) string {
|
||||||
|
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
|
||||||
|
}
|
||||||
|
cases := []struct {
|
||||||
|
name, want string
|
||||||
|
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
|
||||||
|
}{
|
||||||
|
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
|
||||||
|
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
|
||||||
|
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||||
|
ctl.guests[0].Status = "stopped"
|
||||||
|
}},
|
||||||
|
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||||
|
ctl.backupRun = map[int]bool{9201: true}
|
||||||
|
}},
|
||||||
|
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||||
|
ctl.backupErr = errors.New("tasks unreadable")
|
||||||
|
}},
|
||||||
|
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||||
|
s.inFlight.TryAcquire("backup:felhom-pbs")
|
||||||
|
}},
|
||||||
|
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||||
|
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
|
||||||
|
}},
|
||||||
|
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||||
|
ex.busy = fresh(clk)
|
||||||
|
}},
|
||||||
|
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||||
|
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
|
||||||
|
}},
|
||||||
|
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||||
|
s.ctrlSup.mu.Lock()
|
||||||
|
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
|
||||||
|
s.ctrlSup.mu.Unlock()
|
||||||
|
}},
|
||||||
|
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||||
|
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
for i, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
s, ex, rb, ctl, clk, dir := rbServer(t)
|
||||||
|
tc.setup(s, ex, ctl, clk, dir)
|
||||||
|
r := pressReboot(t, s, int64(100+i))
|
||||||
|
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
|
||||||
|
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||||
|
}
|
||||||
|
if rb.count() != 0 {
|
||||||
|
t.Fatalf("guard %q did not hold: rebooted", tc.name)
|
||||||
|
}
|
||||||
|
if _, ok, _ := s.lastOperatorReboot(9201); ok {
|
||||||
|
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
|
||||||
|
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
|
||||||
|
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
|
||||||
|
s, ex, rb, _, clk, _ := rbServer(t)
|
||||||
|
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
|
||||||
|
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
|
||||||
|
t.Fatalf("result %+v, reboots %d", r, rb.count())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
|
||||||
|
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
|
||||||
|
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
|
||||||
|
s, _, rb, ctl, clk, dir := rbServer(t)
|
||||||
|
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||||
|
t.Fatalf("first press: %+v", r)
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(59 * time.Minute)
|
||||||
|
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
|
||||||
|
t.Fatalf("second press inside the hour: %+v", r)
|
||||||
|
}
|
||||||
|
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
|
||||||
|
s2, ox2, _, _, _, _ := opServer(t, "running")
|
||||||
|
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
|
||||||
|
s2.guestExec = &rbExec{opExec: ox2}
|
||||||
|
rb2 := &fakeRebooter{}
|
||||||
|
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
|
||||||
|
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
|
||||||
|
t.Fatalf("after an agent restart, inside the hour: %+v", r)
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(2 * time.Minute)
|
||||||
|
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
|
||||||
|
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
|
||||||
|
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
|
||||||
|
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
|
||||||
|
s, ex, _, _, clk, _ := rbServer(t)
|
||||||
|
for id := int64(1); id <= 2; id++ {
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||||
|
t.Fatalf("restart %d: %+v", id, r)
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(time.Minute)
|
||||||
|
}
|
||||||
|
if r := pressReboot(t, s, 3); r.Outcome != "done" {
|
||||||
|
t.Fatalf("reboot: %+v", r)
|
||||||
|
}
|
||||||
|
clk.t = clk.t.Add(time.Minute)
|
||||||
|
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||||
|
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||||
|
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
|
||||||
|
}
|
||||||
|
if n := ex.count(9201); n != 2 {
|
||||||
|
t.Fatalf("restarts = %d, want 2", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
|
||||||
|
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
|
||||||
|
s, _, rb, _, _, _ := rbServer(t)
|
||||||
|
rb.err = errors.New("pct reboot 9201: exit status 255")
|
||||||
|
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
|
||||||
|
t.Fatalf("result %+v", r)
|
||||||
|
}
|
||||||
|
if s.inFlight.Busy() != "" || s.rebooting(9201) {
|
||||||
|
t.Fatal("the gate or the flag outlived a failed reboot")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
|
||||||
|
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
|
||||||
|
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
|
||||||
|
s, ex, rb, ctl, _, _ := rbServer(t)
|
||||||
|
var started, restarted int
|
||||||
|
rb.after = func() {
|
||||||
|
ctl.guests[0].Status = "stopped"
|
||||||
|
s.GuestPowerTick(context.Background())
|
||||||
|
ctl.guests[0].Status = "running"
|
||||||
|
ex.mu.Lock()
|
||||||
|
ex.status[9201] = "exited"
|
||||||
|
ex.mu.Unlock()
|
||||||
|
s.ControllerSupervisorTick(context.Background())
|
||||||
|
s.ControllerSupervisorTick(context.Background())
|
||||||
|
started, restarted = len(ctl.started), ex.count(9201)
|
||||||
|
ex.mu.Lock()
|
||||||
|
ex.status[9201] = "running"
|
||||||
|
ex.mu.Unlock()
|
||||||
|
}
|
||||||
|
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||||
|
t.Fatalf("result %+v", r)
|
||||||
|
}
|
||||||
|
if started != 0 || restarted != 0 {
|
||||||
|
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
|
||||||
|
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
|
||||||
|
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
|
||||||
|
s, _, _, _, clk, _ := rbServer(t)
|
||||||
|
read := func() string {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
|
||||||
|
var body struct {
|
||||||
|
Data struct {
|
||||||
|
At string `json:"at"`
|
||||||
|
} `json:"data"`
|
||||||
|
At string `json:"at"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
|
||||||
|
t.Fatalf("body %q: %v", w.Body.String(), err)
|
||||||
|
}
|
||||||
|
if body.Data.At != "" {
|
||||||
|
return body.Data.At
|
||||||
|
}
|
||||||
|
return body.At
|
||||||
|
}
|
||||||
|
if got := read(); got != "" {
|
||||||
|
t.Fatalf("no reboot yet, got %q", got)
|
||||||
|
}
|
||||||
|
at := clk.t
|
||||||
|
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||||
|
t.Fatalf("result %+v", r)
|
||||||
|
}
|
||||||
|
clk.t = at.Add(23 * time.Hour)
|
||||||
|
if got := read(); got != at.UTC().Format(time.RFC3339) {
|
||||||
|
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
|
||||||
|
}
|
||||||
|
clk.t = at.Add(25 * time.Hour)
|
||||||
|
if got := read(); got != "" {
|
||||||
|
t.Fatalf("after 25 h the line must be gone, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -411,6 +411,20 @@ type Server struct {
|
|||||||
guestsDir string
|
guestsDir string
|
||||||
ctrlSup controllerSupervisor
|
ctrlSup controllerSupervisor
|
||||||
|
|
||||||
|
// 2026-10-29: the operator's actions from the hub's reply (opactions.go). opHealthPoll/opHealthWait bound the wait
|
||||||
|
// for a restarted controller's healthy answer (tests shorten them; zero → the defaults).
|
||||||
|
opActs opActionState
|
||||||
|
opTrigger func()
|
||||||
|
opHealthPoll time.Duration
|
||||||
|
opHealthWait time.Duration
|
||||||
|
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
|
||||||
|
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
|
||||||
|
opRebooter interface {
|
||||||
|
RebootGuest(ctx context.Context, vmid int) error
|
||||||
|
}
|
||||||
|
rebootMu sync.Mutex
|
||||||
|
rebootInFlight map[int]bool
|
||||||
|
|
||||||
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
|
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
|
||||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||||
netVerifyMu sync.Mutex
|
netVerifyMu sync.Mutex
|
||||||
@@ -459,6 +473,8 @@ type Server struct {
|
|||||||
escrowLookPath func(file string) (string, error)
|
escrowLookPath func(file string) (string, error)
|
||||||
// statFile reports whether a path exists (preflight's staged-secret item; tests inject).
|
// statFile reports whether a path exists (preflight's staged-secret item; tests inject).
|
||||||
statFile func(path string) bool
|
statFile func(path string) bool
|
||||||
|
// cgroupRoot is where the host's cgroup tree is mounted (R-528, GET /guest/oom-kills); "" = /sys/fs/cgroup. Tests set it.
|
||||||
|
cgroupRoot string
|
||||||
|
|
||||||
baseCtx context.Context // for fire-and-forget backups; set in Run
|
baseCtx context.Context // for fire-and-forget backups; set in Run
|
||||||
}
|
}
|
||||||
@@ -576,6 +592,9 @@ func (s *Server) Handler() http.Handler {
|
|||||||
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
|
||||||
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
|
// R-1247: the weekly upload line test's last measurement — the controller's qBittorrent limit and upload watch
|
||||||
// read it. Read-only; never measured (or the job not wired) = present:false.
|
// read it. Read-only; never measured (or the job not wired) = present:false.
|
||||||
|
// R-528: the kernel's OOM-kill counter of each of the guest's containers, read on the host (Docker's own flag and
|
||||||
|
// event can stay silent inside an LXC guest). Read-only; the asking guest's containers only.
|
||||||
|
mux.HandleFunc("GET /guest/oom-kills", s.withGuest(s.handleGuestOOMKills))
|
||||||
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
|
mux.HandleFunc("GET /host/line-test", s.withGuest(s.handleLineTest))
|
||||||
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
|
mux.HandleFunc("POST /host/line-test/run", s.withGuest(s.handleLineTestRun)) // R-1250: a test on request
|
||||||
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
|
||||||
@@ -591,6 +610,8 @@ func (s *Server) Handler() http.Handler {
|
|||||||
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
||||||
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
||||||
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
||||||
|
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
|
||||||
|
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
|
||||||
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
||||||
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
||||||
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
||||||
|
|||||||
@@ -256,8 +256,10 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if callers != 5 {
|
// 2026-10-29: +2 deliberate callers — the operator's restart_controller (opactions.go) and R-1344's unresponsive count
|
||||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
// (controllersupervisor.go observeControllerHealth). Both through s.staleLock, the one helper.
|
||||||
|
if callers != 7 {
|
||||||
|
t.Errorf("expected the seven known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup, the operator restart, the unresponsive count), found %d", callers)
|
||||||
}
|
}
|
||||||
b, _ := os.ReadFile("stalelock.go")
|
b, _ := os.ReadFile("stalelock.go")
|
||||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||||
|
|||||||
@@ -0,0 +1,156 @@
|
|||||||
|
package localapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-1344 (operator brief 2026-10-29, Part 2; decided 2026-10-30 from the night's measurement): a controller that RUNS
|
||||||
|
// but whose Docker health check reads „unhealthy" on controllerUnresponsiveConfirm consecutive sweeps is reported
|
||||||
|
// unresponsive (the stanza; the hub raises controller_unresponsive). If it still reads „unhealthy" at
|
||||||
|
// controllerUnresponsiveRestart sweeps, the agent restarts it — with the not-running restart's guards and brake, plus
|
||||||
|
// the controller's own busy file. RED-PROOF: the restart tests seen failing before the restart existed.
|
||||||
|
|
||||||
|
func unrespServer(t *testing.T, health string) (*Server, *rbExec, *fakeGuestPowerCtl, *supClock) {
|
||||||
|
t.Helper()
|
||||||
|
s, ex, ctl, clk, _, _ := opServer(t, "running")
|
||||||
|
ex.health = health
|
||||||
|
rb := &rbExec{opExec: ex}
|
||||||
|
s.guestExec = rb
|
||||||
|
return s, rb, ctl, clk
|
||||||
|
}
|
||||||
|
|
||||||
|
func sweep(s *Server, n int) {
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
s.ControllerSupervisorTick(context.Background())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSupervisor_UnresponsiveIsAlarmedThenRestarted(t *testing.T) {
|
||||||
|
s, ex, _, _ := unrespServer(t, "unhealthy")
|
||||||
|
sweep(s, controllerUnresponsiveConfirm-1)
|
||||||
|
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
|
||||||
|
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||||
|
}
|
||||||
|
sweep(s, 1)
|
||||||
|
g := s.ControllerSupervisorStatus(context.Background()).Guests
|
||||||
|
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
|
||||||
|
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||||
|
}
|
||||||
|
if g[0].UnresponsiveRestartAfterS != (controllerUnresponsiveRestart-controllerUnresponsiveConfirm)*int(controllerSupervisorInterval/time.Second) {
|
||||||
|
t.Fatalf("the stanza must tell the hub when the restart comes: %+v", g[0])
|
||||||
|
}
|
||||||
|
since := g[0].UnresponsiveSince
|
||||||
|
sweep(s, controllerUnresponsiveRestart-controllerUnresponsiveConfirm-1)
|
||||||
|
if n := ex.count(9201); n != 0 {
|
||||||
|
t.Fatalf("restarted after %d unhealthy sweeps — the restart waits for %d", controllerUnresponsiveRestart-1, controllerUnresponsiveRestart)
|
||||||
|
}
|
||||||
|
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
|
||||||
|
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
|
||||||
|
}
|
||||||
|
sweep(s, 1)
|
||||||
|
if n := ex.count(9201); n != 1 {
|
||||||
|
t.Fatalf("after %d unhealthy sweeps: %d restarts, want 1", controllerUnresponsiveRestart, n)
|
||||||
|
}
|
||||||
|
g = s.ControllerSupervisorStatus(context.Background()).Guests
|
||||||
|
if g[0].UnresponsiveRestartAt == "" || g[0].UnresponsiveRestartAt != g[0].LastRestartAt || g[0].Restarts24h != 1 ||
|
||||||
|
!strings.Contains(g[0].LastReason, "health check") {
|
||||||
|
t.Fatalf("the restart is not on the record the hub reads: %+v", g[0])
|
||||||
|
}
|
||||||
|
// A restart that did not help (still „unhealthy" on the next sweep) is not repeated at once: the count starts again.
|
||||||
|
sweep(s, 1)
|
||||||
|
if n := ex.count(9201); n != 1 {
|
||||||
|
t.Fatalf("restarted again on the sweep after a restart: %d restarts", n)
|
||||||
|
}
|
||||||
|
// The restarted controller starts, then answers: the episode ends, its record stays.
|
||||||
|
ex.health = "starting"
|
||||||
|
sweep(s, 1)
|
||||||
|
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
|
||||||
|
t.Fatalf("after the restart: %+v (unresponsive must clear; since stays as the record)", g[0])
|
||||||
|
}
|
||||||
|
// A second episode needs the whole count again — never a restart on the first unhealthy read after one.
|
||||||
|
ex.health = "unhealthy"
|
||||||
|
sweep(s, controllerUnresponsiveRestart-1)
|
||||||
|
if n := ex.count(9201); n != 1 {
|
||||||
|
t.Fatalf("a second episode restarted early: %d restarts", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Slow but answering (healthy / starting), a swap, a vzdump, a locked or a parked guest never counts — and so is never
|
||||||
|
// restarted.
|
||||||
|
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl)
|
||||||
|
}{
|
||||||
|
{"healthy", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
|
||||||
|
{"starting", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
|
||||||
|
{"swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
|
||||||
|
{"vzdump", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||||
|
{"locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
s, ex, ctl, _ := unrespServer(t, "unhealthy")
|
||||||
|
tc.setup(s, ex, ctl)
|
||||||
|
sweep(s, controllerUnresponsiveRestart+2)
|
||||||
|
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
|
||||||
|
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
|
||||||
|
}
|
||||||
|
if n := ex.count(9201); n != 0 {
|
||||||
|
t.Fatalf("%s: restarted %d times", tc.name, n)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// At the restart moment: the controller's busy file (fresh, or unreadable) and the brake refuse it; the alarm stays.
|
||||||
|
// RED-PROOF: each seen failing by deleting its guard.
|
||||||
|
func TestSupervisor_UnresponsiveRestartRefusals(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
setup func(s *Server, ex *rbExec, clk *supClock)
|
||||||
|
crashloop bool
|
||||||
|
}{
|
||||||
|
{"busy file fresh", func(_ *Server, ex *rbExec, clk *supClock) {
|
||||||
|
ex.busy = clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\na backup, a restore or an off-site run is running\n"
|
||||||
|
}, false},
|
||||||
|
{"busy file unreadable", func(_ *Server, ex *rbExec, _ *supClock) { ex.busyErr = errors.New("pct exec: exit status 255") }, false},
|
||||||
|
{"brake: 3 restarts in 15 minutes", func(s *Server, _ *rbExec, clk *supClock) {
|
||||||
|
st := s.supState(9201)
|
||||||
|
st.restarts = []time.Time{clk.t.Add(-10 * time.Minute), clk.t.Add(-5 * time.Minute), clk.t.Add(-time.Minute)}
|
||||||
|
}, true},
|
||||||
|
{"crash-loop pause in force", func(s *Server, _ *rbExec, clk *supClock) {
|
||||||
|
s.supState(9201).crashloopSince = clk.t.Add(-5 * time.Minute)
|
||||||
|
}, true},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
s, ex, _, clk := unrespServer(t, "unhealthy")
|
||||||
|
tc.setup(s, ex, clk)
|
||||||
|
sweep(s, controllerUnresponsiveRestart+3)
|
||||||
|
if n := ex.count(9201); n != 0 {
|
||||||
|
t.Fatalf("%s: restarted %d times", tc.name, n)
|
||||||
|
}
|
||||||
|
g := s.ControllerSupervisorStatus(context.Background()).Guests
|
||||||
|
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveRestartAt != "" {
|
||||||
|
t.Fatalf("%s: the alarm must stay and no restart be recorded: %+v", tc.name, g)
|
||||||
|
}
|
||||||
|
if tc.crashloop && !g[0].Crashloop {
|
||||||
|
t.Fatalf("%s: the brake did not raise the crash-loop: %+v", tc.name, g[0])
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A busy file the controller stopped writing (older than controllerBusyFresh) is not „busy" — a hung controller must
|
||||||
|
// not shield itself with its last word.
|
||||||
|
func TestSupervisor_UnresponsiveStaleBusyFileDoesNotShield(t *testing.T) {
|
||||||
|
s, ex, _, clk := unrespServer(t, "unhealthy")
|
||||||
|
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\nan app update is running\n"
|
||||||
|
sweep(s, controllerUnresponsiveRestart)
|
||||||
|
if n := ex.count(9201); n != 1 {
|
||||||
|
t.Fatalf("a stale busy file blocked the restart: %d restarts", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -21,6 +21,8 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"regexp"
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
@@ -444,3 +446,129 @@ func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParam
|
|||||||
}
|
}
|
||||||
return l.finish(ctx, lg, rep)
|
return l.finish(ctx, lg, rep)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----
|
||||||
|
//
|
||||||
|
// The rule, its refusals and "may this box do it" live in the root wrapper (mode kernel-cleanup); the plan names NO
|
||||||
|
// package. This file decides only WHEN: in the night leg, after a healthy host step, never in a night with a kernel
|
||||||
|
// step. The wrapper adds its own 20-hour window after a step.
|
||||||
|
|
||||||
|
// KernelCleanupResult is the wrapper's kernel_cleanup object.
|
||||||
|
type KernelCleanupResult struct {
|
||||||
|
Outcome string `json:"outcome"` // removed | nothing | not_run | dry_run | failed
|
||||||
|
DryRun bool `json:"dry_run"`
|
||||||
|
Would string `json:"would"` // dry run only: remove | nothing | refuse | not_run
|
||||||
|
NotRun []struct {
|
||||||
|
Code string `json:"code"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
} `json:"not_run"`
|
||||||
|
Refusals []struct {
|
||||||
|
Code string `json:"code"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
} `json:"refusals"`
|
||||||
|
Kept map[string][]string `json:"kept"`
|
||||||
|
Removal []string `json:"removal"`
|
||||||
|
Removed []string `json:"removed"`
|
||||||
|
BytesGained *int64 `json:"bytes_gained"`
|
||||||
|
Leftovers []string `json:"leftovers"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// kernelCleanupSkip says why tonight's pass must not call the clean-up ("" = call it).
|
||||||
|
func kernelCleanupSkip(appliance bool, trigger string, host, pve, kernel Report, blk hub.WireOSUpdate, okStep func(Report) bool) string {
|
||||||
|
switch {
|
||||||
|
case trigger != "night":
|
||||||
|
return "it runs only in the night leg (never a debug pass)"
|
||||||
|
case !appliance:
|
||||||
|
return "not an appliance install (a BYO host belongs to its owner)"
|
||||||
|
case !blk.Enabled:
|
||||||
|
return "OS updates are switched off for this box"
|
||||||
|
case host.Layer == "" || !okStep(host):
|
||||||
|
return "no healthy host step this pass"
|
||||||
|
case pve.Layer != "" && !okStep(pve):
|
||||||
|
return "the Proxmox step did not end healthy"
|
||||||
|
case kernel.Layer != "":
|
||||||
|
return "a kernel step ran in this night (" + kernel.Outcome + ") — never in the same night"
|
||||||
|
}
|
||||||
|
if due, _ := kernelDue(blk, trigger); due {
|
||||||
|
return "a kernel step is due tonight — never in the same night"
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// runKernelCleanup calls the wrapper once. The hub hears about it only when there is something to tell: kernels were
|
||||||
|
// removed (outcome "cleaned"), the wrapper refused, or the purge failed. "Nothing to remove" and "not run" are log
|
||||||
|
// lines — one report per box per night would bury the kernel step's own line on the hub's System page.
|
||||||
|
func (l *Leg) runKernelCleanup(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
|
||||||
|
lg := l.log().With("run", runID, "layer", LayerKernel, "mode", "kernel-cleanup", "vmid", vmid, "trigger", trigger)
|
||||||
|
start := l.now()
|
||||||
|
wr, err := l.call(ctx, runID, kernelPlan("kernel-cleanup", vmid, map[string]any{"release_id": "kernel-cleanup",
|
||||||
|
"run_id": runID, "trigger": trigger, "ring": blk.Ring}))
|
||||||
|
// NO kernel view and no kernel version in this report: the hub reads a kernel-layer report's `kernel.to` /
|
||||||
|
// `release_id` as "the kernel this STEP was for" (kernelOf), and a clean-up is not a step for any kernel.
|
||||||
|
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "kernel-cleanup",
|
||||||
|
ReleaseID: "kernel-cleanup", PassSeconds: wr.PassSeconds}
|
||||||
|
var res KernelCleanupResult
|
||||||
|
_ = json.Unmarshal(wr.KernelCleanup, &res)
|
||||||
|
took := l.now().Sub(start).Round(time.Second).String()
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
lg.Warn("osupdate: kernel clean-up — the wrapper gave no answer; not retried tonight", "err", err, "took", took)
|
||||||
|
rep.Outcome, rep.HealthReason = "failed", "kernel clean-up: "+err.Error()
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case wr.refused():
|
||||||
|
lg.Warn("osupdate: kernel clean-up REFUSED — nothing was removed; not retried tonight", "refused", string(wr.Refused), "took", took)
|
||||||
|
rep.Outcome, rep.Refused, rep.HealthReason = "refused", wr.Refused, "the clean-up of old kernels was refused; nothing was removed"
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case wr.failed():
|
||||||
|
lg.Error("osupdate: kernel clean-up FAILED — look at the box; not retried tonight", "failed", string(wr.Failed), "took", took)
|
||||||
|
rep.Outcome, rep.Refused, rep.HealthReason = "failed", wr.Failed, "the clean-up of old kernels failed"
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
}
|
||||||
|
switch res.Outcome {
|
||||||
|
case "removed":
|
||||||
|
gained := "unknown"
|
||||||
|
if res.BytesGained != nil {
|
||||||
|
gained = fmt.Sprintf("%.1f GiB", float64(*res.BytesGained)/(1<<30))
|
||||||
|
}
|
||||||
|
rep.Outcome, rep.Healthy = "cleaned", true
|
||||||
|
rep.HealthReason = truncate(fmt.Sprintf("removed %d old kernel package(s): %s; kept %s; %s gained; %d left-over path(s)",
|
||||||
|
len(res.Removed), strings.Join(res.Removed, ", "), strings.Join(sortedKeys(res.Kept), ", "), gained, len(res.Leftovers)), 600)
|
||||||
|
lg.Info("osupdate: kernel clean-up — old kernels removed", "removed", len(res.Removed), "names", strings.Join(res.Removed, ","),
|
||||||
|
"kept", strings.Join(sortedKeys(res.Kept), ","), "bytes_gained", res.BytesGained, "leftovers", len(res.Leftovers), "took", took)
|
||||||
|
return l.finish(ctx, lg, rep)
|
||||||
|
case "nothing":
|
||||||
|
lg.Info("osupdate: kernel clean-up — nothing to remove", "kept", strings.Join(sortedKeys(res.Kept), ","), "took", took)
|
||||||
|
case "not_run":
|
||||||
|
code, why := "", ""
|
||||||
|
if len(res.NotRun) > 0 {
|
||||||
|
code, why = res.NotRun[0].Code, res.NotRun[0].Reason
|
||||||
|
}
|
||||||
|
lg.Info("osupdate: kernel clean-up — not run on this box tonight", "code", code, "reason", why, "took", took)
|
||||||
|
default:
|
||||||
|
lg.Warn("osupdate: kernel clean-up — the wrapper answered with no outcome this agent knows (an older wrapper?)", "outcome", res.Outcome, "took", took)
|
||||||
|
}
|
||||||
|
return Report{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// KernelCleanupDryRun asks the wrapper what the clean-up WOULD do, and changes nothing (the debug tool: `--selftest
|
||||||
|
// kernel-cleanup-dry-run`). The raw kernel_cleanup object, or the refusal.
|
||||||
|
func (l *Leg) KernelCleanupDryRun(ctx context.Context, vmid int) (json.RawMessage, error) {
|
||||||
|
wr, err := l.call(ctx, "kcdry"+l.now().UTC().Format("150405"), kernelPlan("kernel-cleanup", vmid,
|
||||||
|
map[string]any{"release_id": "kernel-cleanup", "dry_run": true}))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if wr.refused() {
|
||||||
|
return nil, fmt.Errorf("kernel-cleanup dry run refused: %s", wr.Refused)
|
||||||
|
}
|
||||||
|
return wr.KernelCleanup, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func sortedKeys(m map[string][]string) []string {
|
||||||
|
out := make([]string, 0, len(m))
|
||||||
|
for k := range m {
|
||||||
|
out = append(out, k)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
@@ -28,13 +28,23 @@ func tonight(ring int) *hub.WireOSUpdate {
|
|||||||
func kernelCalls(w *fakeWrapper) []string {
|
func kernelCalls(w *fakeWrapper) []string {
|
||||||
var m []string
|
var m []string
|
||||||
for _, p := range w.plans {
|
for _, p := range w.plans {
|
||||||
if p["layer"] == LayerKernel {
|
if p["layer"] == LayerKernel && p["mode"] != "kernel-cleanup" { // the kernel STEP's calls (the clean-up: cleanupCalls)
|
||||||
m = append(m, p["mode"].(string))
|
m = append(m, p["mode"].(string))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cleanupCalls(w *fakeWrapper) []map[string]any {
|
||||||
|
var m []map[string]any
|
||||||
|
for _, p := range w.plans {
|
||||||
|
if p["layer"] == LayerKernel && p["mode"] == "kernel-cleanup" {
|
||||||
|
m = append(m, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
|
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
|
||||||
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
|
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
|
||||||
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
|
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
|
||||||
@@ -326,3 +336,177 @@ func TestKernelSet(t *testing.T) {
|
|||||||
t.Fatal("a non-kernel string must give no set")
|
t.Fatal("a non-kernel string must give no set")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- the clean-up of old kernels (R-1368, `09` §3 decision 286, `11` §5.11.1) ----
|
||||||
|
|
||||||
|
func cleanupRep(js string) map[string][]WrapperReport {
|
||||||
|
return map[string][]WrapperReport{"kernel-cleanup": {{KernelCleanup: json.RawMessage(js)}}}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The night: the clean-up is the LAST call, after the host step, exactly once, and its plan names NO package.
|
||||||
|
// COMPANION RED-PROOF (observed): make kernelCleanupSkip return "" before its checks → the BYO / debug / unhealthy
|
||||||
|
// sub-cases of TestKernelCleanup_WhenItMayNotRun fail; put the call before runFast → this test's order check fails.
|
||||||
|
func TestKernelCleanup_NightOrderOnceAndNoNames(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: cleanupRep(`{"outcome":"removed","removed":["proxmox-kernel-7.0.14-20-pve-signed"],` +
|
||||||
|
`"kept":{"7.0.14-22-pve":["running"],"7.0.2-6-pve":["from"]},"bytes_gained":1073741824,"leftovers":[]}`)}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
|
cc := cleanupCalls(w)
|
||||||
|
if len(cc) != 1 {
|
||||||
|
t.Fatalf("clean-up calls = %d, want exactly one a night", len(cc))
|
||||||
|
}
|
||||||
|
if last := w.plans[len(w.plans)-1]; last["mode"] != "kernel-cleanup" {
|
||||||
|
t.Fatalf("the clean-up must be the night's last call (after the host step), last = %v", last)
|
||||||
|
}
|
||||||
|
hostAt, cleanAt := -1, -1
|
||||||
|
for i, x := range w.plans {
|
||||||
|
if x["layer"] == LayerHost {
|
||||||
|
hostAt = i
|
||||||
|
}
|
||||||
|
if x["mode"] == "kernel-cleanup" {
|
||||||
|
cleanAt = i
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if hostAt < 0 || cleanAt < hostAt {
|
||||||
|
t.Fatalf("the clean-up ran before the host step: host %d clean-up %d", hostAt, cleanAt)
|
||||||
|
}
|
||||||
|
plan := cc[0]
|
||||||
|
if pk, _ := json.Marshal(plan["packages"]); string(pk) != "[]" {
|
||||||
|
t.Fatalf("the clean-up plan names packages: %s", pk)
|
||||||
|
}
|
||||||
|
for _, k := range []string{"select", "signed", "allow_new", "expect_kver", "dry_run"} {
|
||||||
|
if _, ok := plan[k]; ok {
|
||||||
|
t.Fatalf("the clean-up plan carries %q: %v", k, plan)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if plan["lane"] != "slow" || plan["layer"] != LayerKernel {
|
||||||
|
t.Fatalf("plan = %v", plan)
|
||||||
|
}
|
||||||
|
// the hub hears "cleaned" with what was removed, kept and gained — and NO kernel view / version (the hub would read
|
||||||
|
// it as a kernel step for that kernel)
|
||||||
|
rep := p.KernelCleanup
|
||||||
|
if rep.Outcome != "cleaned" || !rep.Healthy || rep.Mode != "kernel-cleanup" || rep.Layer != LayerKernel {
|
||||||
|
t.Fatalf("report = %+v", rep)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"removed 1 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed", "kept 7.0.14-22-pve, 7.0.2-6-pve", "1.0 GiB gained"} {
|
||||||
|
if !strings.Contains(rep.HealthReason, want) {
|
||||||
|
t.Fatalf("report reason %q lacks %q", rep.HealthReason, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
lastRep := h.reports[len(h.reports)-1]
|
||||||
|
if lastRep.Mode != "kernel-cleanup" || lastRep.Outcome != "cleaned" {
|
||||||
|
t.Fatalf("the hub did not hear the clean-up: %+v", lastRep)
|
||||||
|
}
|
||||||
|
if len(lastRep.Kernel) != 0 || kverRE.MatchString(lastRep.ReleaseID) {
|
||||||
|
t.Fatalf("the clean-up's report names a kernel (the hub reads that as a kernel step): %+v", lastRep)
|
||||||
|
}
|
||||||
|
// the same night again: the pass is skipped as a whole — never a second clean-up
|
||||||
|
l.Run(context.Background(), 9201, "night")
|
||||||
|
if len(cleanupCalls(w)) != 1 {
|
||||||
|
t.Fatalf("a second clean-up in the same night: %d calls", len(cleanupCalls(w)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// "Nothing to remove" and "not run" are log lines, never a report (one a night would bury the kernel step's own line).
|
||||||
|
func TestKernelCleanup_NothingAndNotRunSendNoReport(t *testing.T) {
|
||||||
|
for name, js := range map[string]string{
|
||||||
|
"nothing": `{"outcome":"nothing","removal":[],"kept":{"7.0.14-22-pve":["running"]}}`,
|
||||||
|
"not_run": `{"outcome":"not_run","not_run":[{"code":"no-ring0-mark","reason":"ring 0 only in this release"}]}`,
|
||||||
|
} {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: cleanupRep(js)}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
|
if len(cleanupCalls(w)) != 1 {
|
||||||
|
t.Fatalf("%s: the wrapper was not asked", name)
|
||||||
|
}
|
||||||
|
if p.KernelCleanup.Layer != "" {
|
||||||
|
t.Fatalf("%s: a report was made: %+v", name, p.KernelCleanup)
|
||||||
|
}
|
||||||
|
for _, r := range h.reports {
|
||||||
|
if r.Mode == "kernel-cleanup" {
|
||||||
|
t.Fatalf("%s: the hub got a clean-up report: %+v", name, r)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A refusal and a failure reach the hub (the operator), once, and are not retried.
|
||||||
|
func TestKernelCleanup_RefusedAndFailedReachTheHub(t *testing.T) {
|
||||||
|
for name, wr := range map[string]WrapperReport{
|
||||||
|
"refused": {Refused: json.RawMessage(`{"code":"P3","reason":"the pinned default and grub.cfg disagree"}`)},
|
||||||
|
"failed": {Failed: json.RawMessage(`{"rc":3,"step":"verify","reason":"the default in grub.cfg moved"}`), KernelCleanup: json.RawMessage(`{"outcome":"failed"}`)},
|
||||||
|
} {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-cleanup": {wr}}}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
p := l.Run(context.Background(), 9201, "night")
|
||||||
|
if p.KernelCleanup.Outcome != name || p.KernelCleanup.Healthy || len(p.KernelCleanup.Refused) == 0 {
|
||||||
|
t.Fatalf("%s: report = %+v", name, p.KernelCleanup)
|
||||||
|
}
|
||||||
|
if last := h.reports[len(h.reports)-1]; last.Mode != "kernel-cleanup" || last.Outcome != name {
|
||||||
|
t.Fatalf("%s: the hub did not hear it: %+v", name, last)
|
||||||
|
}
|
||||||
|
if len(cleanupCalls(w)) != 1 {
|
||||||
|
t.Fatalf("%s: retried the same night (%d calls)", name, len(cleanupCalls(w)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// When it may NOT run: never with a kernel step in the night (due, staged, or reported), not in a debug pass, not on a
|
||||||
|
// BYO host, not with OS updates off, not after an unhealthy host step.
|
||||||
|
func TestKernelCleanup_WhenItMayNotRun(t *testing.T) {
|
||||||
|
type tc struct {
|
||||||
|
blk *hub.WireOSUpdate
|
||||||
|
trigger string
|
||||||
|
prep func(*fakeWrapper, *Leg)
|
||||||
|
}
|
||||||
|
bad := &Health{DockerOK: false}
|
||||||
|
cases := map[string]tc{
|
||||||
|
"a kernel step is due tonight (ring 0)": {tonight(0), "night", func(w *fakeWrapper, _ *Leg) {
|
||||||
|
w.kernelRep = map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
|
||||||
|
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
|
||||||
|
"kernel-reboot": {{Kernel: kview("oneshot")}}}
|
||||||
|
}},
|
||||||
|
"a kernel step is due tonight (ring 1, nothing staged)": {tonight(1), "night", func(w *fakeWrapper, _ *Leg) {
|
||||||
|
w.kernelRep = map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}
|
||||||
|
}},
|
||||||
|
"a debug pass": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "debug", nil},
|
||||||
|
"a BYO host": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night", func(_ *fakeWrapper, l *Leg) { l.Appliance = false }},
|
||||||
|
"updates off": {&hub.WireOSUpdate{Ring: 0, Enabled: false}, "night", nil},
|
||||||
|
"host unhealthy": {&hub.WireOSUpdate{Ring: 0, Enabled: true, HostRelease: &hub.WireOSRelease{ID: "os-host-1", Packages: []hub.WireOSPackage{{Name: "openssl", Version: "u3", Origin: "Debian"}}}}, "night", func(w *fakeWrapper, _ *Leg) {
|
||||||
|
w.applyRep[LayerHost] = WrapperReport{Upgraded: []Package{{Name: "openssl", Version: "u3"}}, HealthAfter: &Health{HostServices: map[string]string{"pveproxy": "failed"}, GuestRunning: yes(), Guest: bad}}
|
||||||
|
w.healthSeq[LayerHost] = []*Health{{HostServices: map[string]string{"pveproxy": "failed"}, GuestRunning: yes(), Guest: bad}}
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
for name, c := range cases {
|
||||||
|
w := &fakeWrapper{t: t}
|
||||||
|
l, _ := newLeg(t, w, c.blk)
|
||||||
|
if c.prep != nil {
|
||||||
|
c.prep(w, l)
|
||||||
|
}
|
||||||
|
l.HealthWait = time.Second
|
||||||
|
l.Run(context.Background(), 9201, c.trigger)
|
||||||
|
if n := len(cleanupCalls(w)); n != 0 {
|
||||||
|
t.Fatalf("%s: the clean-up ran (%d call(s)); calls = %s", name, n, calls(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The dry run is the same mode with dry_run: true — and it sends nothing to the hub.
|
||||||
|
func TestKernelCleanup_DryRun(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, kernelRep: cleanupRep(`{"outcome":"dry_run","would":"remove","removal":["proxmox-kernel-7.0.14-20-pve-signed"]}`)}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||||
|
raw, err := l.KernelCleanupDryRun(context.Background(), 9201)
|
||||||
|
if err != nil || !strings.Contains(string(raw), `"would":"remove"`) {
|
||||||
|
t.Fatalf("dry run = %s, %v", raw, err)
|
||||||
|
}
|
||||||
|
cc := cleanupCalls(w)
|
||||||
|
if len(cc) != 1 || cc[0]["dry_run"] != true {
|
||||||
|
t.Fatalf("the dry run's plan = %v", cc)
|
||||||
|
}
|
||||||
|
if pk, _ := json.Marshal(cc[0]["packages"]); string(pk) != "[]" {
|
||||||
|
t.Fatalf("the dry run's plan names packages: %s", pk)
|
||||||
|
}
|
||||||
|
if len(h.reports) != 0 {
|
||||||
|
t.Fatalf("a dry run reported to the hub: %+v", h.reports)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -134,7 +134,12 @@ type WrapperReport struct {
|
|||||||
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
|
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
|
||||||
Kernel json.RawMessage `json:"kernel"`
|
Kernel json.RawMessage `json:"kernel"`
|
||||||
KernelEvent string `json:"kernel_event"`
|
KernelEvent string `json:"kernel_event"`
|
||||||
OutcomeHint string `json:"outcome_hint"`
|
// KernelCleanup (R-1368, `11` §5.11.1): mode kernel-cleanup's result {outcome, dry_run, kept, removal, removed,
|
||||||
|
// bytes_gained, not_run, refusals, leftovers}.
|
||||||
|
KernelCleanup json.RawMessage `json:"kernel_cleanup"`
|
||||||
|
// ProfileRefresh (R-1388, `11` §5.12): mode profile-refresh's result {outcome, kernel, rewritten, restarted, seconds}.
|
||||||
|
ProfileRefresh json.RawMessage `json:"profile_refresh"`
|
||||||
|
OutcomeHint string `json:"outcome_hint"`
|
||||||
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
||||||
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
||||||
RunID string `json:"run_id"`
|
RunID string `json:"run_id"`
|
||||||
@@ -531,6 +536,8 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
|
|||||||
// Pass is one leg's reports; an empty Layer means the step did not run.
|
// Pass is one leg's reports; an empty Layer means the step did not run.
|
||||||
type Pass struct {
|
type Pass struct {
|
||||||
Guest, Host, Docker, PVE, Kernel Report
|
Guest, Host, Docker, PVE, Kernel Report
|
||||||
|
// KernelCleanup (R-1368): the clean-up of old kernels. Layer "" when it did not run or had nothing to tell the hub.
|
||||||
|
KernelCleanup Report
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
||||||
@@ -582,6 +589,15 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
|
|||||||
default:
|
default:
|
||||||
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
|
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
|
||||||
}
|
}
|
||||||
|
// R-1368 (`09` §3 decision 286, `11` §5.11.1): the clean-up of old kernels — its own step, at most once a night (this
|
||||||
|
// pass is the night's only one), after the host step ended healthy, NEVER in a night with a kernel step. Whether
|
||||||
|
// this box may do it at all is the root wrapper's answer (its root-owned ring-0 mark), never the agent's.
|
||||||
|
// Pinned by TestKernelCleanup_*.
|
||||||
|
if why := kernelCleanupSkip(l.Appliance, trigger, h, p.PVE, p.Kernel, blk, okStep); why != "" {
|
||||||
|
lg.Info("osupdate: kernel clean-up skipped — " + why)
|
||||||
|
} else {
|
||||||
|
p.KernelCleanup = l.runKernelCleanup(ctx, g.RunID, vmid, trigger, blk)
|
||||||
|
}
|
||||||
return p
|
return p
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ type fakeWrapper struct {
|
|||||||
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
||||||
pveGateHeld bool
|
pveGateHeld bool
|
||||||
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
|
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
|
||||||
|
profileRep *WrapperReport // R-1388: the answer to mode profile-refresh
|
||||||
}
|
}
|
||||||
|
|
||||||
func yes() *bool { b := true; return &b }
|
func yes() *bool { b := true; return &b }
|
||||||
@@ -71,6 +72,10 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
|
|||||||
}
|
}
|
||||||
var rep WrapperReport
|
var rep WrapperReport
|
||||||
switch plan["mode"] {
|
switch plan["mode"] {
|
||||||
|
case "profile-refresh":
|
||||||
|
if f.profileRep != nil {
|
||||||
|
rep = *f.profileRep
|
||||||
|
}
|
||||||
case "inventory":
|
case "inventory":
|
||||||
rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthBefore: ok, HealthAfter: ok,
|
rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthBefore: ok, HealthAfter: ok,
|
||||||
Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
|
Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
|
||||||
@@ -176,8 +181,9 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
|
|||||||
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
|
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
|
||||||
t.Fatalf("guest %+v\nhost %+v", g, ho)
|
t.Fatalf("guest %+v\nhost %+v", g, ho)
|
||||||
}
|
}
|
||||||
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply" {
|
// R-1368: the clean-up of old kernels is the night's last call when no kernel step is due
|
||||||
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step and the pve step", calls(w))
|
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply,kernel:kernel-cleanup" {
|
||||||
|
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step, the pve step, the kernel clean-up", calls(w))
|
||||||
}
|
}
|
||||||
for _, p := range w.plans[:2] {
|
for _, p := range w.plans[:2] {
|
||||||
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
|
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
|
||||||
@@ -221,7 +227,8 @@ func TestRing1_NoReleaseIsInventory(t *testing.T) {
|
|||||||
w := &fakeWrapper{t: t, pending: pend}
|
w := &fakeWrapper{t: t, pending: pend}
|
||||||
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
g, ho := run2(l, "night")
|
g, ho := run2(l, "night")
|
||||||
if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory" {
|
// R-1368: a ring-1 box asks the wrapper too — whether it may clean up is the wrapper's answer (its root-owned mark)
|
||||||
|
if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory,kernel:kernel-cleanup" {
|
||||||
t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
|
t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,101 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
// The security-profile refresh (R-1388, R-1301, `11` §5.12). After a kernel went back and forth a box can load an OLD
|
||||||
|
// compiled AppArmor profile for a program (on the N100 this stopped the time service). The root wrapper's mode
|
||||||
|
// profile-refresh loads every profile from a fresh compile, once per kernel the box starts on, and restarts only a
|
||||||
|
// failed service whose own program's profile was stale. This file decides WHEN to ask: at every agent start and after
|
||||||
|
// a config bundle was installed (a new wrapper may know the mode the old one refused) — the wrapper itself answers
|
||||||
|
// "not needed" when this kernel was already done. Nothing is sent to the hub (no hub field exists for it; R-1383).
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ProfileRefreshResult is the wrapper's profile_refresh object.
|
||||||
|
type ProfileRefreshResult struct {
|
||||||
|
Outcome string `json:"outcome"` // refreshed | not_needed | dry_run | failed
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
Kernel string `json:"kernel"`
|
||||||
|
Rewritten []string `json:"rewritten"`
|
||||||
|
Restarted []struct {
|
||||||
|
Unit string `json:"unit"`
|
||||||
|
Program string `json:"program"`
|
||||||
|
RC int `json:"rc"`
|
||||||
|
Active bool `json:"active"`
|
||||||
|
} `json:"restarted"`
|
||||||
|
Seconds float64 `json:"seconds"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProfileGate takes the heavy-op gate (a backup, a restore-test or an OS step must not run beside the refresh).
|
||||||
|
type ProfileGate func() (release func(), busy string, ok bool)
|
||||||
|
|
||||||
|
// ProfileRefresh asks the wrapper once. It returns the outcome ("" when the wrapper could not be asked or refused).
|
||||||
|
// It waits for the gate: up to `tries` attempts, `wait` apart. why is the log's short reason for this call.
|
||||||
|
func (l *Leg) ProfileRefresh(ctx context.Context, why string, gate ProfileGate, tries int, wait time.Duration) string {
|
||||||
|
lg := l.log().With("mode", "profile-refresh", "why", why)
|
||||||
|
if !l.Appliance {
|
||||||
|
lg.Info("osupdate: profile refresh skipped — not an appliance install (a BYO host belongs to its owner)")
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var release func()
|
||||||
|
for i := 1; ; i++ {
|
||||||
|
if gate == nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
r, busy, ok := gate()
|
||||||
|
if ok {
|
||||||
|
release = r
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if i >= tries || ctx.Err() != nil {
|
||||||
|
lg.Warn("osupdate: profile refresh NOT run — the box stayed busy; the next agent start asks again", "busy", busy, "tries", i)
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
lg.Info("osupdate: profile refresh waits — a heavy operation runs", "busy", busy, "try", i)
|
||||||
|
l.sleep(ctx, wait)
|
||||||
|
}
|
||||||
|
if release != nil {
|
||||||
|
defer release()
|
||||||
|
}
|
||||||
|
unlock := l.lockPass(true) // never beside an OS-update pass
|
||||||
|
defer unlock()
|
||||||
|
start := l.now()
|
||||||
|
wr, err := l.call(ctx, "prefresh"+l.now().UTC().Format("150405"), map[string]any{"release_id": "profile-refresh",
|
||||||
|
"layer": LayerHost, "mode": "profile-refresh"})
|
||||||
|
took := l.now().Sub(start).Round(100 * time.Millisecond).String()
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
lg.Warn("osupdate: profile refresh — the wrapper gave no answer; the next agent start asks again", "err", err, "took", took)
|
||||||
|
return ""
|
||||||
|
case wr.refused():
|
||||||
|
lg.Info("osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)", "refused", string(wr.Refused))
|
||||||
|
return ""
|
||||||
|
case wr.failed():
|
||||||
|
lg.Error("osupdate: profile refresh FAILED — the profiles are as before; the next agent start asks again", "failed", string(wr.Failed), "took", took)
|
||||||
|
return "failed"
|
||||||
|
}
|
||||||
|
var res ProfileRefreshResult
|
||||||
|
_ = json.Unmarshal(wr.ProfileRefresh, &res)
|
||||||
|
switch res.Outcome {
|
||||||
|
case "refreshed":
|
||||||
|
units := make([]string, 0, len(res.Restarted))
|
||||||
|
for _, u := range res.Restarted {
|
||||||
|
units = append(units, u.Unit)
|
||||||
|
}
|
||||||
|
if len(units) > 0 {
|
||||||
|
lg.Warn("osupdate: profile refresh — security profiles reloaded from a fresh compile; a FAILED service whose profile was stale was restarted",
|
||||||
|
"kernel", res.Kernel, "restarted", strings.Join(units, ","), "rewritten", len(res.Rewritten), "wrapper_seconds", res.Seconds, "took", took)
|
||||||
|
} else {
|
||||||
|
lg.Info("osupdate: profile refresh — security profiles reloaded from a fresh compile", "kernel", res.Kernel,
|
||||||
|
"rewritten", len(res.Rewritten), "wrapper_seconds", res.Seconds, "took", took)
|
||||||
|
}
|
||||||
|
case "not_needed":
|
||||||
|
lg.Info("osupdate: profile refresh not needed", "kernel", res.Kernel, "reason", res.Reason)
|
||||||
|
default:
|
||||||
|
lg.Warn("osupdate: profile refresh — the wrapper answered with no outcome this agent knows", "outcome", res.Outcome)
|
||||||
|
}
|
||||||
|
return res.Outcome
|
||||||
|
}
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
package osupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ---- the security-profile refresh (R-1388, R-1301, `11` §5.12) ----
|
||||||
|
|
||||||
|
func profileCalls(w *fakeWrapper) []map[string]any {
|
||||||
|
var m []map[string]any
|
||||||
|
for _, p := range w.plans {
|
||||||
|
if p["mode"] == "profile-refresh" {
|
||||||
|
m = append(m, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
func refreshed() *WrapperReport {
|
||||||
|
return &WrapperReport{ProfileRefresh: json.RawMessage(`{"outcome":"refreshed","kernel":"7.0.14-28-pve","rewritten":["f6da0254.0/usr.sbin.chronyd"],` +
|
||||||
|
`"restarted":[{"unit":"chrony.service","program":"/usr/sbin/chronyd","rc":0,"active":true}],"seconds":0.5}`)}
|
||||||
|
}
|
||||||
|
|
||||||
|
// One call, a host-layer plan that names NOTHING, and nothing to the hub.
|
||||||
|
// COMPANION RED-PROOF (observed): add "units" to the plan → this test fails; drop the Appliance check → the BYO test fails.
|
||||||
|
func TestProfileRefresh_OneCallNoNamesNoHubReport(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||||
|
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||||
|
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "refreshed" {
|
||||||
|
t.Fatalf("outcome = %q", got)
|
||||||
|
}
|
||||||
|
pc := profileCalls(w)
|
||||||
|
if len(pc) != 1 {
|
||||||
|
t.Fatalf("calls = %d", len(pc))
|
||||||
|
}
|
||||||
|
if len(pc[0]) != 3 || pc[0]["layer"] != LayerHost || pc[0]["release_id"] != "profile-refresh" {
|
||||||
|
t.Fatalf("the plan carries more than release_id, layer and mode: %v", pc[0])
|
||||||
|
}
|
||||||
|
if len(h.reports) != 0 {
|
||||||
|
t.Fatalf("the hub got a report: %+v", h.reports)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProfileRefresh_ByoHostIsNeverAsked(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||||
|
l, _ := newLeg(t, w, nil)
|
||||||
|
l.Appliance = false
|
||||||
|
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "" || len(profileCalls(w)) != 0 {
|
||||||
|
t.Fatalf("a BYO host was asked: %q %v", got, profileCalls(w))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The heavy-op gate: busy → wait and ask again; still busy after the tries → not run (the next start asks again).
|
||||||
|
// COMPANION RED-PROOF (observed): ignore `ok` from the gate → the "always busy" half fails.
|
||||||
|
func TestProfileRefresh_WaitsForTheHeavyOpGate(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||||
|
l, _ := newLeg(t, w, nil)
|
||||||
|
n, released := 0, 0
|
||||||
|
gate := func() (func(), string, bool) {
|
||||||
|
n++
|
||||||
|
if n < 3 {
|
||||||
|
return nil, "primary-backup", false
|
||||||
|
}
|
||||||
|
return func() { released++ }, "", true
|
||||||
|
}
|
||||||
|
if got := l.ProfileRefresh(context.Background(), "agent start", gate, 5, time.Minute); got != "refreshed" {
|
||||||
|
t.Fatalf("outcome = %q after %d gate tries", got, n)
|
||||||
|
}
|
||||||
|
if n != 3 || released != 1 || len(profileCalls(w)) != 1 {
|
||||||
|
t.Fatalf("gate tries %d, released %d, calls %d", n, released, len(profileCalls(w)))
|
||||||
|
}
|
||||||
|
w2 := &fakeWrapper{t: t, profileRep: refreshed()}
|
||||||
|
l2, _ := newLeg(t, w2, nil)
|
||||||
|
busy := func() (func(), string, bool) { return nil, "restore-test", false }
|
||||||
|
if got := l2.ProfileRefresh(context.Background(), "agent start", busy, 3, time.Minute); got != "" || len(profileCalls(w2)) != 0 {
|
||||||
|
t.Fatalf("ran beside a heavy operation: %q %d call(s)", got, len(profileCalls(w2)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An older wrapper refuses the mode: logged, nothing else. A failure is said as "failed".
|
||||||
|
func TestProfileRefresh_RefusedAndFailed(t *testing.T) {
|
||||||
|
w := &fakeWrapper{t: t, profileRep: &WrapperReport{Refused: json.RawMessage(`{"code":"R11","reason":"unknown mode 'profile-refresh'"}`)}}
|
||||||
|
l, _ := newLeg(t, w, nil)
|
||||||
|
if got := l.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "" {
|
||||||
|
t.Fatalf("refused → %q", got)
|
||||||
|
}
|
||||||
|
w2 := &fakeWrapper{t: t, profileRep: &WrapperReport{Failed: json.RawMessage(`{"rc":1,"step":"profile-refresh"}`)}}
|
||||||
|
l2, _ := newLeg(t, w2, nil)
|
||||||
|
if got := l2.ProfileRefresh(context.Background(), "agent start", nil, 1, time.Second); got != "failed" {
|
||||||
|
t.Fatalf("failed → %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -128,4 +128,11 @@ func (m *Manager) MaybeCommit(ctx context.Context) {
|
|||||||
if m.onCommitted != nil {
|
if m.onCommitted != nil {
|
||||||
m.onCommitted(ctx)
|
m.onCommitted(ctx)
|
||||||
}
|
}
|
||||||
|
// R-1368 part 2: the update ended healthy — the folder keeps the running version's binary and no other
|
||||||
|
// (prune.go). A refusal or a failed removal is logged and changes nothing about the commit.
|
||||||
|
if removed, err := pruneStaged(m.stateDir, m.runningVersion, m.logger); err != nil {
|
||||||
|
m.logger.Warn("selfupdate: old staged binaries NOT removed after the commit", "err", err)
|
||||||
|
} else if len(removed) > 0 {
|
||||||
|
m.logger.Info("selfupdate: old staged binaries removed after the commit", "count", len(removed), "kept", "felhom-agent-"+m.runningVersion)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -103,6 +103,9 @@ func (e *Executor) Execute(ctx context.Context, op string, params json.RawMessag
|
|||||||
}
|
}
|
||||||
e.logger.Debug("agent_update: op invariants passed (semver + sha format)", "version", p.Version)
|
e.logger.Debug("agent_update: op invariants passed (semver + sha format)", "version", p.Version)
|
||||||
|
|
||||||
|
// The staged file is this call's from the download to the hand-off: a prune (prune.go) waits for it.
|
||||||
|
stagingMu.Lock()
|
||||||
|
defer stagingMu.Unlock()
|
||||||
dir := stagingDir(e.stateDir)
|
dir := stagingDir(e.stateDir)
|
||||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||||
return fmt.Errorf("agent_update: staging dir: %w", err)
|
return fmt.Errorf("agent_update: staging dir: %w", err)
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
package selfupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The self-update folder keeps what it needs and no more (R-1368 part 2; space list 2026-11-10, place H12).
|
||||||
|
//
|
||||||
|
// WHAT WAS WRONG. Execute downloads each new agent to <StateDir>/selfupdate/felhom-agent-<version> and removes that
|
||||||
|
// file only when the download or its checksum fails. A good update left it there for ever: measured 2026-11-10, 49 /
|
||||||
|
// 48 / 33 binaries = 740 / 724 / 509 MB on demo-hp, demo-felhom and Tester 1 — one for each version the box ever got.
|
||||||
|
//
|
||||||
|
// WHAT THE FOLDER IS NEEDED FOR. The root wrapper (felhom-os-apply, mode agent_update) reads the staged file ONCE,
|
||||||
|
// copies it to its own root-owned folder and flips from that copy. After that hand-off nothing reads the staged file:
|
||||||
|
// - the rollback (felhom-selfupdate-guarded `rollback`) needs two files and neither is a staged binary:
|
||||||
|
// `pending.json` in this folder, and `/usr/local/bin/felhom-agent.prev` outside it;
|
||||||
|
// - the commit needs `pending.json` only.
|
||||||
|
//
|
||||||
|
// So a staged binary is dead weight as soon as its update ended. Pinned by TestPrune_RollbackReadsNoStagedBinary (the
|
||||||
|
// wrapper's text) and TestPrune_RealRollbackStillWorksAfterNUpdates (the wrapper's rollback verb, run for real).
|
||||||
|
//
|
||||||
|
// THE RULE. After an update ended HEALTHY (the wrapper's `commit` returned success), remove every staged binary but
|
||||||
|
// the one of the running version. Nothing is removed:
|
||||||
|
// - while a pending marker exists or cannot be read — an update is in flight, and that is the only time a rollback
|
||||||
|
// can fire;
|
||||||
|
// - after a rollback, or at a plain start — only the commit path calls this, so the binary of a version that failed
|
||||||
|
// stays for the operator to look at until the next good update;
|
||||||
|
// - that is not a regular file with exactly the name Execute gives a staged binary. `pending.json`, a folder, a
|
||||||
|
// symlink, and any name this code does not know are never touched.
|
||||||
|
//
|
||||||
|
// NOT REACHED BY THIS: `/usr/local/bin/felhom-agent.bak-*` (space list H13). Those files are root-owned, in a folder
|
||||||
|
// the agent cannot write, and no code of the agent makes them (a person's copy before a hand deploy). R-1368 part 4.
|
||||||
|
|
||||||
|
// stagedNameRe is the exact name Execute gives a staged binary: "felhom-agent-" + a version that versionRe accepts.
|
||||||
|
// Kept in step with versionRe by TestPrune_NameShapeFollowsTheVersionRule.
|
||||||
|
var stagedNameRe = regexp.MustCompile(`^felhom-agent-([0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?)$`)
|
||||||
|
|
||||||
|
// stagingMu makes a download and a prune take turns. Without it a prune that runs while Execute downloads the NEXT
|
||||||
|
// version would remove that half-written file (its name is a staged binary's and it is not the running version).
|
||||||
|
var stagingMu sync.Mutex
|
||||||
|
|
||||||
|
// pruneStaged removes the staged binaries the folder no longer needs and returns the names it removed, sorted. It
|
||||||
|
// refuses (removes nothing, returns an error) while an update is in flight. A file it cannot remove is logged and
|
||||||
|
// skipped: a failed clean-up must never fail the commit that came before it.
|
||||||
|
func pruneStaged(stateDir, runningVersion string, logger *slog.Logger) ([]string, error) {
|
||||||
|
logger = orDefaultLogger(logger)
|
||||||
|
stagingMu.Lock()
|
||||||
|
defer stagingMu.Unlock()
|
||||||
|
|
||||||
|
if !versionRe.MatchString(runningVersion) {
|
||||||
|
return nil, fmt.Errorf("selfupdate prune: refusing — running version %q is not a version this code knows; it could not name the file to keep", runningVersion)
|
||||||
|
}
|
||||||
|
pm, err := readPending(stateDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("selfupdate prune: refusing — the pending marker cannot be read: %w", err)
|
||||||
|
}
|
||||||
|
if pm != nil {
|
||||||
|
return nil, fmt.Errorf("selfupdate prune: refusing — an update to %s is pending (a rollback may still fire)", pm.NewVersion)
|
||||||
|
}
|
||||||
|
dir := stagingDir(stateDir)
|
||||||
|
entries, err := os.ReadDir(dir)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("selfupdate prune: read %s: %w", dir, err)
|
||||||
|
}
|
||||||
|
keep := "felhom-agent-" + runningVersion
|
||||||
|
var removed []string
|
||||||
|
for _, ent := range entries {
|
||||||
|
name := ent.Name()
|
||||||
|
if name == keep || !stagedNameRe.MatchString(name) {
|
||||||
|
continue // the running version's binary, pending.json, or a name this code does not know
|
||||||
|
}
|
||||||
|
if !ent.Type().IsRegular() {
|
||||||
|
logger.Warn("selfupdate prune: a staged-binary name that is not a regular file — left alone", "name", name, "type", ent.Type().String())
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// name came from ReadDir and matched stagedNameRe, so it holds no path separator: this is one exact file.
|
||||||
|
if err := os.Remove(filepath.Join(dir, name)); err != nil {
|
||||||
|
logger.Warn("selfupdate prune: cannot remove an old staged binary — left for the next good update", "name", name, "err", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
logger.Debug("selfupdate prune: removed an old staged binary", "name", name)
|
||||||
|
removed = append(removed, name)
|
||||||
|
}
|
||||||
|
sort.Strings(removed)
|
||||||
|
logger.Info("selfupdate prune: the self-update folder holds what it needs", "kept", keep, "removed", len(removed))
|
||||||
|
return removed, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,373 @@
|
|||||||
|
package selfupdate
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-1368 part 2 — the self-update folder keeps what it needs and no more (prune.go).
|
||||||
|
//
|
||||||
|
// RED-PROOFS (each observed 2026-11-10, then restored):
|
||||||
|
// - drop the pruneStaged call from MaybeCommit → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with
|
||||||
|
// "after update 1 (0.170.0) the folder holds [… felhom-agent-0.169.0 felhom-agent-0.170.0 …]", and
|
||||||
|
// TestPrune_RealRollbackStillWorksAfterNUpdates fails too;
|
||||||
|
// - drop the pending-marker refusal from pruneStaged → TestPrune_RefusesWhileAnUpdateIsPending fails with
|
||||||
|
// "removed [...] while an update was pending";
|
||||||
|
// - drop the IsRegular check → TestPrune_AfterNUpdatesTheFolderHoldsOneBinary fails with "a symlink with a
|
||||||
|
// staged-binary name was removed".
|
||||||
|
|
||||||
|
// committingWrapper is the guarded wrapper's `commit` as far as this package can see it: it clears the pending marker.
|
||||||
|
type committingWrapper struct {
|
||||||
|
stateDir string
|
||||||
|
calls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *committingWrapper) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
|
||||||
|
if len(args) == 1 && args[0] == "commit" {
|
||||||
|
w.calls++
|
||||||
|
if err := os.Remove(pendingPath(w.stateDir)); err != nil {
|
||||||
|
return nil, []byte(err.Error()), err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return []byte("ok"), nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func stage(t *testing.T, stateDir, version string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.MkdirAll(stagingDir(stateDir), 0o750); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(stagingDir(stateDir), "felhom-agent-"+version), []byte("binary "+version), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func folder(t *testing.T, stateDir string) []string {
|
||||||
|
t.Helper()
|
||||||
|
ents, err := os.ReadDir(stagingDir(stateDir))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var out []string
|
||||||
|
for _, e := range ents {
|
||||||
|
out = append(out, e.Name())
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// The consequence: after N good updates the folder holds ONE staged binary — the running version's — and every file
|
||||||
|
// that is not a staged binary is still there.
|
||||||
|
func TestPrune_AfterNUpdatesTheFolderHoldsOneBinary(t *testing.T) {
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
dir := stagingDir(stateDir)
|
||||||
|
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Bystanders: names this code does not know, and two things that carry a staged binary's NAME but are not files.
|
||||||
|
outside := filepath.Join(t.TempDir(), "precious")
|
||||||
|
if err := os.WriteFile(outside, []byte("not ours"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, n := range []string{"notes.txt", "felhom-agent-0.1.0.partial", "felhom-agent.bak-0.139.0"} {
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := os.Mkdir(filepath.Join(dir, "felhom-agent-9.9.9"), 0o750); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Symlink(outside, filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
bystanders := []string{"felhom-agent-0.1.0.partial", "felhom-agent-8.8.8", "felhom-agent-9.9.9", "felhom-agent.bak-0.139.0", "notes.txt"}
|
||||||
|
|
||||||
|
versions := []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1", "0.173.0-rc.1", "0.173.0"}
|
||||||
|
prev := "0.169.0"
|
||||||
|
stage(t, stateDir, prev) // what an old agent left behind
|
||||||
|
for i, v := range versions {
|
||||||
|
stage(t, stateDir, v) // Execute downloaded it
|
||||||
|
writePending(t, stateDir, PendingMarker{OldVersion: prev, NewVersion: v, SHA256: "ab", AppliedAt: "t"}) // the wrapper flipped
|
||||||
|
wrap := &committingWrapper{stateDir: stateDir}
|
||||||
|
m := newMgr(t, stateDir, v, wrap, nil)
|
||||||
|
m.MaybeCommit(context.Background())
|
||||||
|
if wrap.calls != 1 {
|
||||||
|
t.Fatalf("update %d (%s): commit calls = %d, want 1", i+1, v, wrap.calls)
|
||||||
|
}
|
||||||
|
want := append([]string{"felhom-agent-" + v}, bystanders...)
|
||||||
|
sort.Strings(want)
|
||||||
|
got := folder(t, stateDir)
|
||||||
|
if strings.Join(got, " ") != strings.Join(want, " ") {
|
||||||
|
if _, err := os.Lstat(filepath.Join(dir, "felhom-agent-8.8.8")); err != nil {
|
||||||
|
t.Fatalf("a symlink with a staged-binary name was removed: %v", err)
|
||||||
|
}
|
||||||
|
t.Fatalf("after update %d (%s) the folder holds %v, want %v", i+1, v, got, want)
|
||||||
|
}
|
||||||
|
prev = v
|
||||||
|
}
|
||||||
|
if b, err := os.ReadFile(outside); err != nil || string(b) != "not ours" {
|
||||||
|
t.Fatalf("the file behind the symlink changed: %q %v", b, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// While an update is pending — the only time a rollback can fire — nothing is removed.
|
||||||
|
func TestPrune_RefusesWhileAnUpdateIsPending(t *testing.T) {
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
|
||||||
|
stage(t, stateDir, v)
|
||||||
|
}
|
||||||
|
writePending(t, stateDir, PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0", SHA256: "ab", AppliedAt: "t"})
|
||||||
|
before := folder(t, stateDir)
|
||||||
|
|
||||||
|
removed, err := pruneStaged(stateDir, "0.172.0", nil)
|
||||||
|
if err == nil || len(removed) != 0 {
|
||||||
|
t.Fatalf("removed %v while an update was pending (err=%v) — want a refusal", removed, err)
|
||||||
|
}
|
||||||
|
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||||
|
t.Fatalf("removed %v while an update was pending; folder now %v", before, got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The same through the commit path: a `commit` that reports success but leaves the marker (fakeWrapper does) must
|
||||||
|
// not lead to a prune.
|
||||||
|
m := newMgr(t, stateDir, "0.172.0", &fakeWrapper{}, nil)
|
||||||
|
m.MaybeCommit(context.Background())
|
||||||
|
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||||
|
t.Fatalf("the commit path pruned with the marker still there; folder now %v", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A marker nobody can read is a refusal too.
|
||||||
|
if err := os.WriteFile(pendingPath(stateDir), []byte("{not json"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if removed, err := pruneStaged(stateDir, "0.172.0", nil); err == nil || len(removed) != 0 {
|
||||||
|
t.Fatalf("an unreadable marker must refuse; removed %v err=%v", removed, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only a commit prunes: a plain start, a version mismatch and a failed commit remove nothing — so the binary of a
|
||||||
|
// version that was rolled back stays for the operator.
|
||||||
|
func TestPrune_OnlyAHealthyCommitPrunes(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
pending *PendingMarker
|
||||||
|
running string
|
||||||
|
wrap WrapperRunner
|
||||||
|
}{
|
||||||
|
{"plain start, no marker (also the state after a rollback)", nil, "0.171.0", &fakeWrapper{}},
|
||||||
|
{"marker names another version", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.171.0", &fakeWrapper{}},
|
||||||
|
{"commit fails", &PendingMarker{OldVersion: "0.171.0", NewVersion: "0.172.0"}, "0.172.0", &fakeWrapper{err: fmt.Errorf("sudo: no")}},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0"} {
|
||||||
|
stage(t, stateDir, v)
|
||||||
|
}
|
||||||
|
if c.pending != nil {
|
||||||
|
writePending(t, stateDir, *c.pending)
|
||||||
|
}
|
||||||
|
before := folder(t, stateDir)
|
||||||
|
newMgr(t, stateDir, c.running, c.wrap, nil).MaybeCommit(context.Background())
|
||||||
|
if got := folder(t, stateDir); strings.Join(got, " ") != strings.Join(before, " ") {
|
||||||
|
t.Fatalf("folder changed without a healthy commit: %v → %v", before, got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
// A running version this code cannot name a file for: refuse, do not guess.
|
||||||
|
stateDir := t.TempDir()
|
||||||
|
stage(t, stateDir, "0.170.0")
|
||||||
|
if removed, err := pruneStaged(stateDir, "dev", nil); err == nil || len(removed) != 0 {
|
||||||
|
t.Fatalf("running version \"dev\" must refuse; removed %v err=%v", removed, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The name rule and the version rule say the same thing: every name Execute can write is a name the prune knows, and
|
||||||
|
// nothing else is.
|
||||||
|
func TestPrune_NameShapeFollowsTheVersionRule(t *testing.T) {
|
||||||
|
for _, v := range []string{"0.70.1", "0.172.1", "10.0.0", "0.70.2-crash", "0.173.0-rc.1"} {
|
||||||
|
if !versionRe.MatchString(v) {
|
||||||
|
t.Fatalf("test input %q is not a version versionRe accepts", v)
|
||||||
|
}
|
||||||
|
if !stagedNameRe.MatchString("felhom-agent-" + v) {
|
||||||
|
t.Errorf("Execute can stage felhom-agent-%s but the prune does not know that name", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, n := range []string{"pending.json", "felhom-agent", "felhom-agent-", "felhom-agent-1.2", "felhom-agent-0.1.0.partial",
|
||||||
|
"felhom-agent.bak-0.139.0", "felhom-agent.prev", "felhom-agent-0.1.0/x", "../felhom-agent-0.1.0", "felhom-agent-0.1.0 ", "xfelhom-agent-0.1.0"} {
|
||||||
|
if stagedNameRe.MatchString(n) {
|
||||||
|
t.Errorf("the prune would treat %q as a staged binary", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
m := stagedNameRe.FindStringSubmatch("felhom-agent-0.173.0-rc.1")
|
||||||
|
if m == nil || !versionRe.MatchString(m[1]) {
|
||||||
|
t.Fatalf("the version part of a staged name must satisfy versionRe; got %v", m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func wrapperText(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
b, err := os.ReadFile(filepath.Join("..", "..", "configs", "felhom-selfupdate-guarded"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("the guarded wrapper is the other half of this proof and must be readable: %v", err)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The stated dependency, pinned: the wrapper's `rollback` and `commit` read the pending marker and `.prev`, and no
|
||||||
|
// staged binary. If either verb starts to read the staging folder, this fails and prune.go's rule must be looked at.
|
||||||
|
func TestPrune_RollbackReadsNoStagedBinary(t *testing.T) {
|
||||||
|
s := wrapperText(t)
|
||||||
|
for _, want := range []string{"\nBIN=/usr/local/bin/felhom-agent\n", "\nPREV=$BIN.prev\n", "\nSTAGING=/var/lib/felhom-agent/selfupdate\n", "\nPENDING=$STAGING/pending.json\n"} {
|
||||||
|
if !strings.Contains(s, want) {
|
||||||
|
t.Fatalf("the wrapper no longer holds the line %q — the folder's rule (prune.go) was written against it", strings.TrimSpace(want))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, verb := range []string{"commit", "rollback"} {
|
||||||
|
i := strings.Index(s, "\n"+verb+")\n")
|
||||||
|
if i < 0 {
|
||||||
|
t.Fatalf("no %s) block in the wrapper", verb)
|
||||||
|
}
|
||||||
|
block := s[i:]
|
||||||
|
j := strings.Index(block, "\n ;;\n")
|
||||||
|
if j < 0 {
|
||||||
|
t.Fatalf("no end of the %s) block", verb)
|
||||||
|
}
|
||||||
|
block = block[:j]
|
||||||
|
if !strings.Contains(block, "$PENDING") {
|
||||||
|
t.Fatalf("%s) does not read $PENDING any more", verb)
|
||||||
|
}
|
||||||
|
code := regexp.MustCompile(`(?m)^\s*#.*$`).ReplaceAllString(block, "")
|
||||||
|
for _, bad := range []string{"STAGING", "staged", "felhom-agent-"} {
|
||||||
|
if strings.Contains(code, bad) {
|
||||||
|
t.Fatalf("%s) now mentions %q — it may read a staged binary, and the prune removes those", verb, bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if stagedNameRe.MatchString("pending.json") || stagedNameRe.MatchString("felhom-agent.prev") {
|
||||||
|
t.Fatal("the prune would remove a file the rollback needs")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The consequence, with the wrapper's own code: after N good updates (each committed by the REAL `commit` verb and
|
||||||
|
// pruned), update N+1 goes bad and the REAL `rollback` verb puts the previous binary back. The wrapper runs from a copy
|
||||||
|
// whose four fixed paths point into a temp folder; systemctl, logger and chown are stubs.
|
||||||
|
func TestPrune_RealRollbackStillWorksAfterNUpdates(t *testing.T) {
|
||||||
|
sh, err := exec.LookPath("sh")
|
||||||
|
if err != nil {
|
||||||
|
t.Skip("no sh on this machine")
|
||||||
|
}
|
||||||
|
root := t.TempDir()
|
||||||
|
stateDir := filepath.Join(root, "state")
|
||||||
|
binDir := filepath.Join(root, "bin")
|
||||||
|
stubs := filepath.Join(root, "stubs")
|
||||||
|
for _, d := range []string{stagingDir(stateDir), binDir, stubs, filepath.Join(root, "rootstaging")} {
|
||||||
|
if err := os.MkdirAll(d, 0o750); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
live := filepath.Join(binDir, "felhom-agent")
|
||||||
|
s := wrapperText(t)
|
||||||
|
for old, repl := range map[string]string{
|
||||||
|
"\nBIN=/usr/local/bin/felhom-agent\n": "\nBIN=" + live + "\n",
|
||||||
|
"\nSTAGING=/var/lib/felhom-agent/selfupdate\n": "\nSTAGING=" + stagingDir(stateDir) + "\n",
|
||||||
|
"\nROOT_STAGING=/var/lib/felhom-os-apply/agent-update\n": "\nROOT_STAGING=" + filepath.Join(root, "rootstaging") + "\n",
|
||||||
|
} {
|
||||||
|
if strings.Count(s, old) != 1 {
|
||||||
|
t.Fatalf("the wrapper's path line %q is not there exactly once — this test cannot sandbox it", strings.TrimSpace(old))
|
||||||
|
}
|
||||||
|
s = strings.Replace(s, old, repl, 1)
|
||||||
|
}
|
||||||
|
// Safety: no assignment in the copy may still point at a real system path (PREV and PENDING are built from BIN and
|
||||||
|
// STAGING, so they follow).
|
||||||
|
for _, line := range strings.Split(s, "\n") {
|
||||||
|
if regexp.MustCompile(`^[A-Z_]+=/`).MatchString(line) && !strings.Contains(line, root) {
|
||||||
|
t.Fatalf("the sandboxed wrapper still holds a real path: %q", line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
script := filepath.Join(root, "wrapper.sh")
|
||||||
|
if err := os.WriteFile(script, []byte(s), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
stubLog := filepath.Join(root, "stub.log")
|
||||||
|
for _, n := range []string{"systemctl", "logger", "chown", "systemd-run"} {
|
||||||
|
body := "#!/bin/sh\necho \"" + n + " $*\" >> '" + stubLog + "'\nexit 0\n"
|
||||||
|
if err := os.WriteFile(filepath.Join(stubs, n), []byte(body), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
run := func(verb string) (string, error) {
|
||||||
|
cmd := exec.Command(sh, script, verb)
|
||||||
|
cmd.Env = []string{"PATH=" + stubs + ":/usr/bin:/bin"}
|
||||||
|
out, err := cmd.CombinedOutput()
|
||||||
|
return string(out), err
|
||||||
|
}
|
||||||
|
// applied is what the wrapper's `apply` leaves behind (it needs root, so the test writes the same four things).
|
||||||
|
applied := func(oldV, newV string) {
|
||||||
|
cur, err := os.ReadFile(live)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(live+".prev", cur, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(live, []byte("binary "+newV), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writePending(t, stateDir, PendingMarker{OldVersion: oldV, NewVersion: newV, SHA256: "ab", AppliedAt: "t"})
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(live, []byte("binary 0.169.0"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prev := "0.169.0"
|
||||||
|
for _, v := range []string{"0.170.0", "0.171.0", "0.172.0", "0.172.1"} {
|
||||||
|
stage(t, stateDir, v)
|
||||||
|
applied(prev, v)
|
||||||
|
m := newMgr(t, stateDir, v, runnerFunc(func(args ...string) ([]byte, []byte, error) {
|
||||||
|
out, err := run(args[0])
|
||||||
|
return []byte(out), nil, err
|
||||||
|
}), nil)
|
||||||
|
m.MaybeCommit(context.Background()) // the REAL commit verb, then the prune
|
||||||
|
if got := folder(t, stateDir); len(got) != 1 || got[0] != "felhom-agent-"+v {
|
||||||
|
t.Fatalf("after the good update to %s the folder holds %v", v, got)
|
||||||
|
}
|
||||||
|
prev = v
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update N+1 is applied and goes bad: systemd's OnFailure runs the wrapper's rollback.
|
||||||
|
stage(t, stateDir, "0.173.0")
|
||||||
|
applied("0.172.1", "0.173.0")
|
||||||
|
out, err := run("rollback")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("the real rollback verb failed after the prunes: %v\n%s", err, out)
|
||||||
|
}
|
||||||
|
if b, _ := os.ReadFile(live); string(b) != "binary 0.172.1" {
|
||||||
|
t.Fatalf("after the rollback the live binary is %q, want the previous version's", b)
|
||||||
|
}
|
||||||
|
if p, _ := readPending(stateDir); p != nil {
|
||||||
|
t.Fatal("the rollback left the pending marker")
|
||||||
|
}
|
||||||
|
if b, _ := os.ReadFile(stubLog); !bytes.Contains(b, []byte("systemctl restart felhom-agent.service")) {
|
||||||
|
t.Fatalf("the rollback did not restart the agent; stub log:\n%s", b)
|
||||||
|
}
|
||||||
|
// The rolled-back agent starts: no marker, so no commit and no prune — the bad version's binary stays to be read.
|
||||||
|
newMgr(t, stateDir, "0.172.1", &fakeWrapper{}, nil).MaybeCommit(context.Background())
|
||||||
|
if got := folder(t, stateDir); strings.Join(got, " ") != "felhom-agent-0.172.1 felhom-agent-0.173.0" {
|
||||||
|
t.Fatalf("after the rollback the folder holds %v, want the running version and the one that failed", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// runnerFunc adapts a function to WrapperRunner; it gets the verb arguments only.
|
||||||
|
type runnerFunc func(args ...string) ([]byte, []byte, error)
|
||||||
|
|
||||||
|
func (f runnerFunc) Run(_ context.Context, _ string, args ...string) ([]byte, []byte, error) {
|
||||||
|
return f(args...)
|
||||||
|
}
|
||||||
@@ -33,6 +33,9 @@
|
|||||||
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
|
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
|
||||||
# GITEA_BASE / GITEA_OWNER override the defaults.
|
# GITEA_BASE / GITEA_OWNER override the defaults.
|
||||||
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
|
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
|
||||||
|
# GITEA_GIT_TOKEN — the tag push (`git push origin <tag>`) needs it since 2026-11-11: the address of a CC clone
|
||||||
|
# on DooPlex holds no credential, and git takes the token from the secret store through the helper
|
||||||
|
# felhom.eu/scripts/git-credential-sec.sh. Run the release as `sec run GITEA_GIT_TOKEN,... -- scripts/release-agent.sh <version>`.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
|
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
|
||||||
|
|||||||
@@ -24,10 +24,10 @@
|
|||||||
"readers": [
|
"readers": [
|
||||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||||
],
|
],
|
||||||
"recorded": "2026-10-26",
|
"recorded": "2026-11-10",
|
||||||
"recorded_by": "CC 2026-10-26: day0_agent 0.166.0 (operator vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
"recorded_by": "CC 2026-11-10: day0_agent 0.173.0 (operator vouch 2026-11-10 16:29:16 CET, read from the hub log: agent=0.173.0, bundle 51ebc1f4… = the release; the save took 232 ms); earlier 0.172.1 (CC 2026-11-07) (operator vouch 2026-11-07 ~23:00 CET, read from the hub DB with its WAL: artifact_agent_version=0.172.1, bundle 3e7acaee… = the release); earlier 0.172.0 (CC 2026-11-03) (operator vouch 2026-11-03 10:18:10 CET, read from the hub log and DB: agent=0.172.0, bundle cc34b0bc… = the release); earlier 0.171.0 (CC 2026-10-30) (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||||
"pinned": {
|
"pinned": {
|
||||||
"hub_min_agent": "0.158.1",
|
"hub_min_agent": "0.158.1",
|
||||||
"day0_agent": "0.166.0"
|
"day0_agent": "0.173.0"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Reference in new issue
Block a user