Compare commits

...
8 Commits
Author SHA1 Message Date
admin 808cdb36ad REPORT + CONTEXT: v0.171.0 released and vouched; R-1344 restart proven live
gates / gates (push) Successful in 33s
2026-10-30 11:09:48 +01:00
admin 84726c9f27 v0.171.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.171.0 (D50)
gates / gates (push) Successful in 33s
2026-10-30 10:49:46 +01:00
admin bd18347c1c R-1344 decided from the night's measurement: a controller that runs but does not answer is restarted at 10 unhealthy sweeps (alarm at 6) — busy file + the brake refuse; stanza carries unresponsive_restart_after_s / unresponsive_restart_at
gates / gates (push) Successful in 33s
2026-10-30 09:20:37 +01:00
admin 98bd1b71e9 REPORT + CONTEXT: v0.170.0 released and vouched; reboot_guest proven on the release builds
gates / gates (push) Successful in 39s
2026-10-29 20:24:29 +01:00
admin 6578a8b91e v0.170.0 released and vouched: CHANGELOG heading; pinned.day0_agent = 0.170.0 (D50)
gates / gates (push) Successful in 36s
2026-10-29 18:59:05 +01:00
admin ca15a31690 D65 = A: reboot_guest joins the agent's closed operator-action list — guards (busy controller file, heavy gate, jobs, lock, parked, 1/hour, the brake), result with both times, GET /guest/operator-reboot for the household line; one read-only sudo vector
gates / gates (push) Successful in 33s
2026-10-29 16:04:19 +01:00
admin bdea4694a4 REPORT + CONTEXT: v0.169.0 released and vouched; restart_controller proven on the release builds
gates / gates (push) Successful in 35s
2026-10-29 15:11:11 +01:00
admin db25eb0e71 v0.169.0 released and vouched (14:06:09 CET): CHANGELOG; D50 pinned.day0_agent = 0.169.0
gates / gates (push) Successful in 42s
2026-10-29 14:09:04 +01:00
16 changed files with 985 additions and 68 deletions

No files matched your search

+49 -1
View File
@@ -1,4 +1,52 @@
## Unreleased (2026-10-29, afternoon) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — commit only; the next agent release (needs the operator's vouch)
## v0.171.0 (2026-10-30) — R-1344 decided from the night's measurement: a controller that runs but does not answer is RESTARTED — RELEASED (release-agent.sh from a clean clone of `bd18347`; agent sha256 c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5, config bundle 95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b; binary-only signed agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-30 10:47:36 CET; `pinned.day0_agent` = 0.171.0). Live on Tester 1: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the restart 09:31:22 → healthy 09:31:24; the hub mailed `controller_unresponsive` and `controller_unresponsive_restart` (hub 0.178.0), not `controller_restarted_by_agent`
- **The measurement (operator brief 2026-10-29 Part 2, read 2026-10-30).** Docker's own health record of the controller
on Tester 1 and demo-hp, every probe from 2026-10-29 11:12 to 2026-10-30 07:00 UTC: 4 733 probes through the night
chain (demo-hp's PBS vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test, two
controller updates). No failed probe, no 5-s timeout; slowest 303 ms (demo-hp, during the PBS vzdump). The only
„unhealthy" was the controller stopped by hand for a proof. A busy night never came near ONE failed probe.
- **So the alarm becomes a restart.** At `controllerUnresponsiveConfirm` = 6 unhealthy sweeps the alarm is raised, as
before. At `controllerUnresponsiveRestart` = 10 (2 minutes later; ~6.5 minutes after the first failed probe) the agent
restarts the controller's bootstrap unit — unless the controller's busy file is fresh (a backup, a restore, an
update, an export) or cannot be read, or the brake refuses (3 restarts in 15 minutes → the crash-loop pause, which
the hub mails). A swap, a parked guest, a locked guest and a vzdump keep resetting the count, as before. A refusal
keeps the alarm and is logged once per reason. After a restart the count starts again from zero.
- **The stanza** carries `unresponsive_restart_after_s` (120: the hub's alarm text says when the restart comes) and
`unresponsive_restart_at` (equals `last_restart_at` for that act; the hub mails `controller_unresponsive_restart`).
- **Refactor, no behaviour change:** the not-running restart's brake and its record are now `supervisorBrakeAllows`
and `recordSupervisorRestart`, shared by both restarts. The busy-file stale log line no longer says „operator reboot".
- Tests `TestSupervisor_UnresponsiveIsAlarmedThenRestarted`, `…Guards` (now also: never restarted),
`…RestartRefusals` (busy fresh, busy unreadable, the brake, a crash-loop pause), `…StaleBusyFileDoesNotShield` —
red first (build failure, then 3 behaviour failures); 7 mutants, each killed (one survivor — the count reset after a
restart — found and pinned).
## v0.170.0 (2026-10-29) — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list — RELEASED (release-agent.sh from a clean clone of `ca15a31`; agent sha256 da8d12a05ef607c338b9ba307e7a81b18253a3188721bbd8421ba36658a22404, config bundle 03f2ff6442fc3ca600b1f25f26767ce4da466a770a258f3e362cf95f6e76b543; signed agent_config_update + agent_update on Tester 1, demo-hp, demo-felhom; vouched by the operator 2026-10-29 18:57:48 CET; `pinned.day0_agent` = 0.170.0)
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
a racing start would fail the reboot's own start half.
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
restarted your box at <time>".
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
+25
View File
@@ -1,5 +1,30 @@
# CONTEXT — felhom-agent working state
> **2026-10-30 (day, attended) — v0.171.0 RELEASED and VOUCHED (10:47:36 CET): R-1344 decided — a controller that runs
> but does not answer is RESTARTED.** The night's measurement (4 733 probes, slowest 303 ms, no failure) licensed it:
> alarm at `controllerUnresponsiveConfirm` = 6 unhealthy sweeps, restart at `controllerUnresponsiveRestart` = 10 —
> refused by a fresh/unreadable busy file (`controllerBusy`, shared with D65's reboot) or the brake
> (`supervisorBrakeAllows`, now shared with the not-running restart; `recordSupervisorRestart` records both). The stanza
> carries `unresponsive_restart_after_s` and `unresponsive_restart_at` (hub 0.178.0 mails
> `controller_unresponsive_restart`). Built from `bd18347` (`c888ca7b…`, bundle `95f977f6…`), binary-only signed
> updates on the three test boxes; live on Tester 1 (SIGSTOP → alarm 4.5 min → restart 6.5 min → healthy).
> `pinned.day0_agent` = 0.171.0. felhom.eu `REPORT-release-2026-10-30.md`.
> **2026-10-29 (evening, attended) — v0.170.0 RELEASED and VOUCHED (18:57:48 CET): `reboot_guest` (D65 = A).**
> `da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`, from a clean clone of `ca15a31`; configs/ changed (one new read-only
> sudo vector: `cat` of the controller's `data/ctl/busy`), so per box a signed `agent_config_update` + `agent_update`
> (Tester 1, demo-hp, demo-felhom; capability probe 76/76). The agent's closed list is `restart_controller` +
> `reboot_guest` (`opreboot.go`; guards in felhom.eu `03`/`04` §6.1). Proven on Tester 1: refused during an off-site run
> (the controller's busy file), done in 67 s / 79 s, refused a second time inside the hour. `pinned.day0_agent` = 0.170.0.
> **R-1344 decided 2026-10-30: RESTART** (see the 2026-10-30 entry above).
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
+25 -15
View File
@@ -1,18 +1,28 @@
# REPORT — v0.168.0 released and vouched (2026-10-29, attended)
# REPORT — v0.171.0 released and vouched: a controller that runs but does not answer is restarted (R-1344, 2026-10-30, attended)
**Plain words first.** The agent 0.168.0 is out, on the three test boxes, and vouched by the operator — new boxes install
it. It carries R-528 (the host's per-container OOM count) and R-1254's ceremony half.
**Plain words first.** The night's health log showed that a busy night (backups, dumps, updates) never makes the
controller miss even one health check — the slowest answer was 0.3 s against a 5 s limit. So the agent no longer only
alarms about a controller that runs but does not answer: 2 minutes after the alarm it restarts it, unless the
controller is busy with a backup, a restore or an update, or the restart brake holds.
- **Release:** `release-agent.sh` from a clean clone of `91bd96c` (CI 2479 green): agent sha256
`1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca`, bundle
`3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8`, tag `v0.168.0`, verified by independent download.
- **Delivery:** no configs/ change since v0.167.0 → one signed `agent_update` per box: demo-hp 09:56, Tester 1 09:57,
demo-felhom 10:04 CET — each „signed op COMPLETED", self-check 76/76, „update committed"; the binary's sha256 matches on
all three.
- **Proof:** on Tester 1 (controller 0.347.0) a 128 MB Gokapi had a memory hog killed: host counter 1, one `app_oom`
„seen by docker,cgroup", one operator mail. R-1254: a ceremony ran on 0.168.0 (the happy path); the lost-answer branch
has tests only.
- **Vouch:** the operator, 10:21:56 CET (hub log: agent=0.168.0, the bundle sha above). `pinned.day0_agent` = 0.168.0
(`6b0b093`).
## Commits
Evidence: felhom.eu `documentation/audits/d63-2026-10-29/` (`opsign-agent.txt`, `agent-delivery*.txt`, `run-p4-oom.log`).
| Commit | CI (job id) | What |
|---|---|---|
| `bd18347` | 2532 green | `controllerUnresponsiveRestart` = 10 sweeps; `restartUnresponsive` (busy file, then the brake); the brake and the restart record shared with the not-running restart (`supervisorBrakeAllows`, `recordSupervisorRestart`); stanza `unresponsive_restart_after_s`, `unresponsive_restart_at`. Tests red first (build failure, then 3 behaviour failures); 7 mutants, one survivor (the count reset after a restart) pinned by a new assertion. |
| `84726c9` | (read at the close-out) | v0.171.0 heading; `pinned.day0_agent` = 0.171.0. |
## Release
- `release-agent.sh 0.171.0` from a clean clone of `bd18347`: sha256 `c888ca7ba22a1f689059081575814a725e2edcecbb6bcc56829e9d9de88202a5`,
bundle `95f977f6dc827185f2b5a2aa9e1c829db41a9f425d49eb40f317286d13c83d2b`, tag `v0.171.0`, verified by download.
- configs/ unchanged → one signed `agent_update` per box (Tester 1, demo-hp, demo-felhom): all COMPLETED and
„update committed" (10:12–10:20 CET); no new error lines.
- Vouched by the operator 2026-10-30 10:47:36 CET (the save took 132 ms — felhom.eu R-1346, hub 0.178.0).
## Live proof (Tester 1, controller 0.349.0)
`docker kill --signal=SIGSTOP felhom-controller` 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the agent's alarm
09:29:21 (6 sweeps) → „the controller does not answer — restarting the bootstrap unit" 09:31:22 (10 sweeps) → healthy
09:31:24. The hub mailed `controller_unresponsive_restart` and `controller_unresponsive` (09:42:56, with the next report)
and not `controller_restarted_by_agent`. Evidence: felhom.eu `documentation/audits/release-2026-10-30/p1-r1344/`.
+2 -2
View File
@@ -88,8 +88,8 @@
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller` only, `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause (`supervisorBrakeAllows` — every supervisor restart passes it; `recordSupervisorRestart` puts one on the record + R-539's slow counter); R-1344 (2026-10-30): a RUNNING controller whose Docker health reads „unhealthy" is alarmed at 6 sweeps and restarted at 10 (`restartUnresponsive`: the controller's busy file, then the brake); record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller`, `reboot_guest` since D65 — `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
### Proxmox client / hub / PBS / provisioning
+3
View File
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
+1
View File
@@ -151,6 +151,7 @@ var manifest = []Capability{
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
+6
View File
@@ -218,6 +218,12 @@ type ControllerSupervisorGuest struct {
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
Unresponsive bool `json:"unresponsive"`
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
// R-1344 restart (decided 2026-10-30 from the night's measurement): UnresponsiveRestartAfterS is how long after the
// alarm the agent restarts a controller that still does not answer (0 / absent = an older agent: alarm only — the
// hub's message says which). UnresponsiveRestartAt is the last such restart; the hub keys on it MOVING
// (controller_unresponsive_restart) and it equals LastRestartAt for that act.
UnresponsiveRestartAfterS int `json:"unresponsive_restart_after_s,omitempty"`
UnresponsiveRestartAt string `json:"unresponsive_restart_at,omitempty"` // RFC3339
}
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
+103 -17
View File
@@ -3,6 +3,7 @@ package localapi
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
@@ -87,10 +88,16 @@ const (
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
// the hub (controller_unresponsive). ALARM ONLY — it is NOT restarted for this until the measurement (R-1344) shows a
// busy night (backups, updates) never reads unhealthy that long. Not counted during a swap, a parked guest, a locked
// guest or a vzdump.
// the hub (controller_unresponsive). Not counted during a swap, a parked guest, a locked guest or a vzdump.
controllerUnresponsiveConfirm = 6
// controllerUnresponsiveRestart (decided 2026-10-30 from the measurement): at this many consecutive unhealthy sweeps
// (5 minutes after Docker's „unhealthy", ~6.5 minutes after the first failed probe; 2 minutes after the alarm) the
// agent restarts the controller — the not-running restart's guards and brake, plus the controller's busy file. The
// numbers: Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, 4 733 probes through the night chain (a
// PBS and a local vzdump, the DB dumps, tier-2, the off-site push, a restore test, two controller updates): no failed
// probe, slowest 303 ms against the 5 s timeout. A busy night never came near even ONE failed probe; ten sweeps is
// the margin. Pinned by TestSupervisor_UnresponsiveIsAlarmedThenRestarted.
controllerUnresponsiveRestart = 10
)
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
@@ -119,6 +126,10 @@ type controllerSupState struct {
unhealthySeen int
unresponsive bool
unresponsiveSince time.Time
// unresponsiveRestartAt: the last restart for not answering (kept; the hub keys on it MOVING). unrespRefusal: the
// last reason the restart was refused, so a refusal is logged once per change, not every sweep.
unresponsiveRestartAt time.Time
unrespRefusal string
}
type controllerSupervisor struct {
@@ -254,6 +265,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
if !s.provisionedGuest(g.VMID) {
continue
}
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
s.resetNotRunning(g.VMID)
continue
}
evaluated++
if !s.superviseOneController(ctx, g.VMID, g.Status) {
down++
@@ -350,9 +365,27 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
return false
}
// Backoff.
if !s.supervisorBrakeAllows(vmid, now) {
return false
}
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
"vmid", vmid, "status", status, "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
s.recordSupervisorRestart(vmid, now, reason, nil)
return false
}
// supervisorBrakeAllows is the brake every supervisor restart passes (the not-running one and R-1344's): a crash-loop
// pause in force refuses; controllerCrashloopMax restarts inside controllerCrashloopWindow start one (logged, carried by
// the stanza, mailed by the hub as controller_crashloop) and refuse.
func (s *Server) supervisorBrakeAllows(vmid int, now time.Time) bool {
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st := s.supState(vmid)
if !st.crashloopSince.IsZero() {
if now.Sub(st.crashloopSince) < controllerCrashloopPause {
s.ctrlSup.mu.Unlock()
@@ -375,22 +408,23 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
return false
}
s.ctrlSup.mu.Unlock()
return true
}
reason := "controller container " + status + " on " + strconv.Itoa(controllerSupervisorConfirm) + " consecutive sweeps"
s.logger.Warn("controller-supervisor: controller is NOT running — restarting the bootstrap unit",
"vmid", vmid, "status", status, "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
// recordSupervisorRestart puts a restart the supervisor performed on the record: the brake's window, the totals, the
// reason, and R-539's slow counter (persisted; raised at most once per 24 hours — the hub mails on the raise MOVING).
// mark, when set, runs under the same lock (R-1344 stamps its own record there).
func (s *Server) recordSupervisorRestart(vmid int, now time.Time, reason string, mark func(st *controllerSupState)) {
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st := s.supState(vmid)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = reason
st.notRunningSeen = 0
// R-539: the slow counter. Raise at most once per 24 hours — the hub mails on the raise MOVING.
if mark != nil {
mark(st)
}
st.restarts24h = append(pruneBefore(st.restarts24h, now.Add(-controllerSlowCrashloopWindow)), now)
n24 := len(st.restarts24h)
raised := false
@@ -406,7 +440,6 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
s.logger.Warn("controller-supervisor: SLOW CRASH-LOOP — the controller keeps dying; still restarting it, raising controller_slow_crashloop",
"vmid", vmid, "restarts_24h", n24, "window", controllerSlowCrashloopWindow.String(), "threshold", controllerSlowCrashloopMax)
}
return false
}
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
@@ -448,11 +481,60 @@ func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked b
if start {
st.unresponsive, st.unresponsiveSince = true, now
}
since := st.unresponsiveSince
s.ctrlSup.mu.Unlock()
if start {
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; NOT restarting it (R-1344: alarm only until measured)",
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String())
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; restarting it if it stays so",
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String(),
"restart_after_sweeps", controllerUnresponsiveRestart)
}
if seen >= controllerUnresponsiveRestart {
s.restartUnresponsive(ctx, vmid, seen, since, now)
}
}
// restartUnresponsive is R-1344's restart (decided 2026-10-30): the controller's busy file (fresh = a backup, a
// restore, an update or an export is running; unreadable = cannot be ruled out) refuses, then the brake every
// supervisor restart passes. The swap, the parked marker, the lock and the vzdump never reach here — they reset the
// count. A refusal keeps the alarm and is logged once per reason; the next sweep asks again.
func (s *Server) restartUnresponsive(ctx context.Context, vmid, seen int, since, now time.Time) {
refuse := func(why string) {
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
changed := st.unrespRefusal != why
st.unrespRefusal = why
s.ctrlSup.mu.Unlock()
if changed {
s.logger.Warn("controller-supervisor: the controller does not answer, but the restart is refused — the alarm stays",
"vmid", vmid, "why", why, "unhealthy_sweeps", seen)
}
}
busy, what, err := s.controllerBusy(ctx, vmid)
if err != nil {
refuse("the controller's busy file cannot be read: " + err.Error())
return
}
if busy {
refuse("the controller is busy: " + what)
return
}
if !s.supervisorBrakeAllows(vmid, now) {
refuse("the restart brake")
return
}
reason := fmt.Sprintf("the controller ran but did not answer its health check for %d sweeps (since %s)",
seen, since.UTC().Format(time.RFC3339))
s.logger.Warn("controller-supervisor: the controller does not answer — restarting the bootstrap unit",
"vmid", vmid, "unhealthy_sweeps", seen, "since", since.UTC().Format(time.RFC3339), "unit", bootstrapUnit)
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
s.logger.Error("controller-supervisor: bootstrap restart failed", "vmid", vmid, "err", err)
reason += "; restart FAILED: " + err.Error()
}
s.recordSupervisorRestart(vmid, now, reason, func(st *controllerSupState) {
st.unresponsiveRestartAt = now
st.unhealthySeen = 0
st.unrespRefusal = ""
})
}
func (s *Server) resetNotRunning(vmid int) {
@@ -513,6 +595,10 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
if !st.unresponsiveSince.IsZero() {
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
}
g.UnresponsiveRestartAfterS = (controllerUnresponsiveRestart - controllerUnresponsiveConfirm) * int(controllerSupervisorInterval/time.Second)
if !st.unresponsiveRestartAt.IsZero() {
g.UnresponsiveRestartAt = st.unresponsiveRestartAt.UTC().Format(time.RFC3339)
}
out.Guests = append(out.Guests, g)
}
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
+5
View File
@@ -152,6 +152,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
return
}
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
// would fail the reboot's own start half).
if s.rebooting(g.VMID) {
return
}
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
if err != nil {
+13 -6
View File
@@ -20,7 +20,7 @@ import (
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
// The result is re-sent until the reply stops listing the id.
//
// THE LIST IS CLOSED: one verb, restart_controller <vmid>. Anything else — a verb, a vmid that does not parse, a guest
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
//
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
@@ -31,7 +31,7 @@ import (
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
const (
// OpRestartController is the agent's one operator verb.
// OpRestartController is the agent's first operator verb.
OpRestartController = "restart_controller"
opOutcomeDone = "done"
@@ -52,7 +52,7 @@ const (
)
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
var operatorActionNames = []string{OpRestartController}
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
// OperatorActionNames returns the closed list (sorted copy).
func OperatorActionNames() []string {
@@ -120,11 +120,18 @@ func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
}
func (s *Server) runOperatorAction(a hub.OperatorAction) {
ctx, cancel := context.WithTimeout(context.Background(), s.opWait()+time.Minute)
limit := s.opWait() + time.Minute
if a.Action == OpRebootGuest {
limit += opRebootTimeout
}
ctx, cancel := context.WithTimeout(context.Background(), limit)
defer cancel()
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller only", a.Action)
if a.Action == OpRestartController {
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
switch a.Action {
case OpRestartController:
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
case OpRebootGuest:
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
}
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
"outcome", outcome, "message", msg)
+2 -2
View File
@@ -71,7 +71,7 @@ func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
// RED-PROOF: seen failing before the list existed (build failure, then the names).
func TestOpActions_AgentClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"restart_controller"}; !reflect.DeepEqual(got, want) {
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
}
}
@@ -80,7 +80,7 @@ func TestOpActions_AgentClosedList(t *testing.T) {
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
s.OnEnvelope(context.Background(), env(
hub.OperatorAction{ID: 1, Action: "reboot_guest", Arg: "9201"},
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
))
+329
View File
@@ -0,0 +1,329 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
//
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
//
// THE GUARDS (each refuses with the reason in the result; nothing is called):
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
// - a controller swap (update) is in flight;
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
// alone while the reboot runs (rebootInFlight).
//
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
// one day). Pinned by opreboot_test.go.
const (
// OpRebootGuest is the agent's second operator verb (D65 = A).
OpRebootGuest = "reboot_guest"
// opRebootEvery: at most one operator reboot per guest per hour.
opRebootEvery = time.Hour
// opRebootNoticeFor: how long the household's dashboard shows the line.
opRebootNoticeFor = 24 * time.Hour
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
opRebootTimeout = 5 * time.Minute
operatorRebootFile = "operator-reboot.json"
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
// Docker does not hide it.
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
// stopped writing (stuck or killed) — not a busy one.
controllerBusyFresh = 90 * time.Second
)
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
type operatorRebootRecord struct {
At time.Time `json:"at"`
ActionID int64 `json:"action_id"`
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
ControllerAfterS int `json:"controller_after_s,omitempty"`
}
func (s *Server) operatorRebootPath(vmid int) string {
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
}
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
// must not silently vanish).
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
if errors.Is(err, os.ErrNotExist) {
return operatorRebootRecord{}, false, nil
}
if err != nil {
return operatorRebootRecord{}, false, err
}
var rec operatorRebootRecord
if err := json.Unmarshal(raw, &rec); err != nil {
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
}
return rec, !rec.At.IsZero(), nil
}
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
raw, err := json.Marshal(rec)
if err != nil {
return err
}
p := s.operatorRebootPath(vmid)
tmp := p + ".tmp"
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
return err
}
return os.Rename(tmp, p)
}
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
func (s *Server) rebooting(vmid int) bool {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
return s.rebootInFlight[vmid]
}
func (s *Server) setRebooting(vmid int, on bool) {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
if s.rebootInFlight == nil {
s.rebootInFlight = map[int]bool{}
}
if on {
s.rebootInFlight[vmid] = true
} else {
delete(s.rebootInFlight, vmid)
}
}
func (s *Server) rebooter() interface {
RebootGuest(ctx context.Context, vmid int) error
} {
if s.opRebooter != nil {
return s.opRebooter
}
if s.guestAttach != nil {
return s.guestAttach
}
return nil
}
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
// (the caller refuses — it cannot prove no backup runs). Two callers: the operator's reboot (D65) and R-1344's restart
// of a controller that does not answer (controllersupervisor.go restartUnresponsive).
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
if err != nil {
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
return false, "", nil
}
return false, "", err
}
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
if perr != nil {
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
}
if s.clock().Sub(at) > controllerBusyFresh {
s.logger.Info("controller-busy: the controller's busy file is stale — the controller stopped writing it; not busy",
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
return false, "", nil
}
why := "work in progress"
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
why = strings.TrimSpace(lines[1])
}
return true, why, nil
}
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
rb := s.rebooter()
if s.staleLock == nil || s.guestExec == nil || rb == nil {
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
if s.rebooting(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
// At most one per hour.
now := s.clock()
last, had, err := s.lastOperatorReboot(vmid)
if err != nil {
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
}
if had && now.Sub(last.At) < opRebootEvery {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
}
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
s.ctrlSup.mu.Unlock()
// The heavy-operation gate, held until the guest is up again.
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
if !ok {
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
}
defer release()
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
}
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
} else if cbusy {
return opOutcomeRefused, "refused: the controller is busy — " + why
}
// The record FIRST: the hourly limit and the household's line both read it.
rec := operatorRebootRecord{At: now, ActionID: id}
if err := s.saveOperatorReboot(vmid, rec); err != nil {
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
}
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.setRebooting(vmid, true)
defer s.setRebooting(vmid, false)
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
began := time.Now()
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
err = rb.RebootGuest(rctx, vmid)
cancel()
if err != nil {
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
}
upS := int(time.Since(began).Round(time.Second) / time.Second)
rec.GuestUpAfterS = upS
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
}
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
case <-time.After(s.opPoll()):
}
}
_ = s.saveOperatorReboot(vmid, rec)
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
}
// rebootBlockerSentence names jobBlocker's code in the result's words.
func rebootBlockerSentence(code string) string {
switch code {
case PowerOffBackup:
return "a backup is running"
case PowerOffCtrlUpdate:
return "a controller update is running"
case PowerOffDrive:
return "a drive or network-storage job is running"
case PowerOffEscrow:
return "an escrow ceremony is running"
case PowerOffUnknown:
return "the running jobs cannot be read"
default:
return "the host is busy"
}
}
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
rec, ok, err := s.lastOperatorReboot(vmid)
if err != nil {
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
return
}
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
return
}
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
}
+306
View File
@@ -0,0 +1,306 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
// rbExec adds the controller's busy file to the operator-restart fake.
type rbExec struct {
*opExec
busy string // the file's content; "" = no file
busyErr error // pct exec itself failed
}
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
if f.busyErr != nil {
return "", f.busyErr
}
if f.busy == "" {
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
}
return f.busy, nil
}
return f.opExec.GuestExec(ctx, vmid, args...)
}
type fakeRebooter struct {
mu sync.Mutex
n int
err error
after func()
}
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
r.mu.Lock()
defer r.mu.Unlock()
r.n++
if r.after != nil {
r.after()
}
return r.err
}
func (r *fakeRebooter) count() int {
r.mu.Lock()
defer r.mu.Unlock()
return r.n
}
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
t.Helper()
s, ox, ctl, clk, dir, _ := opServer(t, "running")
ex := &rbExec{opExec: ox}
s.guestExec = ex
rb := &fakeRebooter{}
s.opRebooter = rb
s.inFlight = &backup.InFlight{}
return s, ex, rb, ctl, clk, dir
}
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
return waitResult(t, s, id)
}
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
s, _, rb, _, clk, _ := rbServer(t)
r := pressReboot(t, s, 1)
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
t.Fatalf("result %+v", r)
}
if rb.count() != 1 {
t.Fatalf("reboots = %d, want 1", rb.count())
}
rec, ok, err := s.lastOperatorReboot(9201)
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
}
if s.inFlight.Busy() != "" {
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
}
if s.rebooting(9201) {
t.Fatal("the reboot-in-flight flag outlived the reboot")
}
}
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
func TestOpReboot_GuardsRefuse(t *testing.T) {
fresh := func(clk *supClock) string {
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
}
cases := []struct {
name, want string
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
}{
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.guests[0].Status = "stopped"
}},
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupRun = map[int]bool{9201: true}
}},
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupErr = errors.New("tasks unreadable")
}},
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.inFlight.TryAcquire("backup:felhom-pbs")
}},
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
}},
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
ex.busy = fresh(clk)
}},
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
}},
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
s.ctrlSup.mu.Lock()
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
s.ctrlSup.mu.Unlock()
}},
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
panic(err)
}
}},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, rb, ctl, clk, dir := rbServer(t)
tc.setup(s, ex, ctl, clk, dir)
r := pressReboot(t, s, int64(100+i))
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if rb.count() != 0 {
t.Fatalf("guard %q did not hold: rebooted", tc.name)
}
if _, ok, _ := s.lastOperatorReboot(9201); ok {
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
}
})
}
}
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
s, ex, rb, _, clk, _ := rbServer(t)
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
t.Fatalf("result %+v, reboots %d", r, rb.count())
}
}
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
s, _, rb, ctl, clk, dir := rbServer(t)
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("first press: %+v", r)
}
clk.t = clk.t.Add(59 * time.Minute)
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
t.Fatalf("second press inside the hour: %+v", r)
}
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
s2, ox2, _, _, _, _ := opServer(t, "running")
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
s2.guestExec = &rbExec{opExec: ox2}
rb2 := &fakeRebooter{}
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
t.Fatalf("after an agent restart, inside the hour: %+v", r)
}
clk.t = clk.t.Add(2 * time.Minute)
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
}
}
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
s, ex, _, _, clk, _ := rbServer(t)
for id := int64(1); id <= 2; id++ {
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("restart %d: %+v", id, r)
}
clk.t = clk.t.Add(time.Minute)
}
if r := pressReboot(t, s, 3); r.Outcome != "done" {
t.Fatalf("reboot: %+v", r)
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 2 {
t.Fatalf("restarts = %d, want 2", n)
}
}
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
s, _, rb, _, _, _ := rbServer(t)
rb.err = errors.New("pct reboot 9201: exit status 255")
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
t.Fatalf("result %+v", r)
}
if s.inFlight.Busy() != "" || s.rebooting(9201) {
t.Fatal("the gate or the flag outlived a failed reboot")
}
}
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
s, ex, rb, ctl, _, _ := rbServer(t)
var started, restarted int
rb.after = func() {
ctl.guests[0].Status = "stopped"
s.GuestPowerTick(context.Background())
ctl.guests[0].Status = "running"
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
started, restarted = len(ctl.started), ex.count(9201)
ex.mu.Lock()
ex.status[9201] = "running"
ex.mu.Unlock()
}
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
if started != 0 || restarted != 0 {
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
}
}
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
s, _, _, _, clk, _ := rbServer(t)
read := func() string {
w := httptest.NewRecorder()
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
var body struct {
Data struct {
At string `json:"at"`
} `json:"data"`
At string `json:"at"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("body %q: %v", w.Body.String(), err)
}
if body.Data.At != "" {
return body.Data.At
}
return body.At
}
if got := read(); got != "" {
t.Fatalf("no reboot yet, got %q", got)
}
at := clk.t
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
clk.t = at.Add(23 * time.Hour)
if got := read(); got != at.UTC().Format(time.RFC3339) {
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
}
clk.t = at.Add(25 * time.Hour)
if got := read(); got != "" {
t.Fatalf("after 25 h the line must be gone, got %q", got)
}
}
+9
View File
@@ -417,6 +417,13 @@ type Server struct {
opTrigger func()
opHealthPoll time.Duration
opHealthWait time.Duration
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
opRebooter interface {
RebootGuest(ctx context.Context, vmid int) error
}
rebootMu sync.Mutex
rebootInFlight map[int]bool
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
// detached pipeline runs through (tests inject; production defaults set in NewServer).
@@ -603,6 +610,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
+105 -23
View File
@@ -2,19 +2,25 @@ package localapi
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// R-1344 (operator brief 2026-10-29, Part 2): a controller that RUNS but whose Docker health check reads „unhealthy" on
// controllerUnresponsiveConfirm consecutive sweeps is reported unresponsive (the stanza; the hub raises
// controller_unresponsive). ALARM ONLY: it is never restarted for it until the measurement licenses that.
// RED-PROOF: seen failing before the stanza carried it.
// R-1344 (operator brief 2026-10-29, Part 2; decided 2026-10-30 from the night's measurement): a controller that RUNS
// but whose Docker health check reads „unhealthy" on controllerUnresponsiveConfirm consecutive sweeps is reported
// unresponsive (the stanza; the hub raises controller_unresponsive). If it still reads „unhealthy" at
// controllerUnresponsiveRestart sweeps, the agent restarts it — with the not-running restart's guards and brake, plus
// the controller's own busy file. RED-PROOF: the restart tests seen failing before the restart existed.
func unrespServer(t *testing.T, health string) (*Server, *opExec, *fakeGuestPowerCtl) {
func unrespServer(t *testing.T, health string) (*Server, *rbExec, *fakeGuestPowerCtl, *supClock) {
t.Helper()
s, ex, ctl, _, _, _ := opServer(t, "running")
s, ex, ctl, clk, _, _ := opServer(t, "running")
ex.health = health
return s, ex, ctl
rb := &rbExec{opExec: ex}
s.guestExec = rb
return s, rb, ctl, clk
}
func sweep(s *Server, n int) {
@@ -23,8 +29,8 @@ func sweep(s *Server, n int) {
}
}
func TestSupervisor_UnresponsiveIsAlarmedNeverRestarted(t *testing.T) {
s, ex, _ := unrespServer(t, "unhealthy")
func TestSupervisor_UnresponsiveIsAlarmedThenRestarted(t *testing.T) {
s, ex, _, _ := unrespServer(t, "unhealthy")
sweep(s, controllerUnresponsiveConfirm-1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
@@ -34,41 +40,117 @@ func TestSupervisor_UnresponsiveIsAlarmedNeverRestarted(t *testing.T) {
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
}
if g[0].UnresponsiveRestartAfterS != (controllerUnresponsiveRestart-controllerUnresponsiveConfirm)*int(controllerSupervisorInterval/time.Second) {
t.Fatalf("the stanza must tell the hub when the restart comes: %+v", g[0])
}
since := g[0].UnresponsiveSince
sweep(s, 10)
sweep(s, controllerUnresponsiveRestart-controllerUnresponsiveConfirm-1)
if n := ex.count(9201); n != 0 {
t.Fatalf("an unresponsive controller was RESTARTED (%d) — the restart waits for the measurement", n)
t.Fatalf("restarted after %d unhealthy sweeps — the restart waits for %d", controllerUnresponsiveRestart-1, controllerUnresponsiveRestart)
}
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
}
// it answers again
ex.health = "healthy"
sweep(s, 1)
if n := ex.count(9201); n != 1 {
t.Fatalf("after %d unhealthy sweeps: %d restarts, want 1", controllerUnresponsiveRestart, n)
}
g = s.ControllerSupervisorStatus(context.Background()).Guests
if g[0].UnresponsiveRestartAt == "" || g[0].UnresponsiveRestartAt != g[0].LastRestartAt || g[0].Restarts24h != 1 ||
!strings.Contains(g[0].LastReason, "health check") {
t.Fatalf("the restart is not on the record the hub reads: %+v", g[0])
}
// A restart that did not help (still „unhealthy" on the next sweep) is not repeated at once: the count starts again.
sweep(s, 1)
if n := ex.count(9201); n != 1 {
t.Fatalf("restarted again on the sweep after a restart: %d restarts", n)
}
// The restarted controller starts, then answers: the episode ends, its record stays.
ex.health = "starting"
sweep(s, 1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
t.Fatalf("after it answered: %+v (unresponsive must clear; since stays as the record)", g[0])
t.Fatalf("after the restart: %+v (unresponsive must clear; since stays as the record)", g[0])
}
// A second episode needs the whole count again — never a restart on the first unhealthy read after one.
ex.health = "unhealthy"
sweep(s, controllerUnresponsiveRestart-1)
if n := ex.count(9201); n != 1 {
t.Fatalf("a second episode restarted early: %d restarts", n)
}
}
// Slow but answering (healthy / starting), a swap, a vzdump or a parked guest never counts.
// Slow but answering (healthy / starting), a swap, a vzdump, a locked or a parked guest never counts — and so is never
// restarted.
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
for _, tc := range []struct {
name string
setup func(s *Server, ex *opExec, ctl *fakeGuestPowerCtl)
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl)
}{
{"healthy", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
{"starting", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
{"swap", func(s *Server, _ *opExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
{"vzdump", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
{"locked", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
{"healthy", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
{"starting", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
{"swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
{"vzdump", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
{"locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
} {
t.Run(tc.name, func(t *testing.T) {
s, ex, ctl := unrespServer(t, "unhealthy")
s, ex, ctl, _ := unrespServer(t, "unhealthy")
tc.setup(s, ex, ctl)
sweep(s, controllerUnresponsiveConfirm+2)
sweep(s, controllerUnresponsiveRestart+2)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
}
if n := ex.count(9201); n != 0 {
t.Fatalf("%s: restarted %d times", tc.name, n)
}
})
}
}
// At the restart moment: the controller's busy file (fresh, or unreadable) and the brake refuse it; the alarm stays.
// RED-PROOF: each seen failing by deleting its guard.
func TestSupervisor_UnresponsiveRestartRefusals(t *testing.T) {
for _, tc := range []struct {
name string
setup func(s *Server, ex *rbExec, clk *supClock)
crashloop bool
}{
{"busy file fresh", func(_ *Server, ex *rbExec, clk *supClock) {
ex.busy = clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\na backup, a restore or an off-site run is running\n"
}, false},
{"busy file unreadable", func(_ *Server, ex *rbExec, _ *supClock) { ex.busyErr = errors.New("pct exec: exit status 255") }, false},
{"brake: 3 restarts in 15 minutes", func(s *Server, _ *rbExec, clk *supClock) {
st := s.supState(9201)
st.restarts = []time.Time{clk.t.Add(-10 * time.Minute), clk.t.Add(-5 * time.Minute), clk.t.Add(-time.Minute)}
}, true},
{"crash-loop pause in force", func(s *Server, _ *rbExec, clk *supClock) {
s.supState(9201).crashloopSince = clk.t.Add(-5 * time.Minute)
}, true},
} {
t.Run(tc.name, func(t *testing.T) {
s, ex, _, clk := unrespServer(t, "unhealthy")
tc.setup(s, ex, clk)
sweep(s, controllerUnresponsiveRestart+3)
if n := ex.count(9201); n != 0 {
t.Fatalf("%s: restarted %d times", tc.name, n)
}
g := s.ControllerSupervisorStatus(context.Background()).Guests
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveRestartAt != "" {
t.Fatalf("%s: the alarm must stay and no restart be recorded: %+v", tc.name, g)
}
if tc.crashloop && !g[0].Crashloop {
t.Fatalf("%s: the brake did not raise the crash-loop: %+v", tc.name, g[0])
}
})
}
}
// A busy file the controller stopped writing (older than controllerBusyFresh) is not „busy" — a hung controller must
// not shield itself with its last word.
func TestSupervisor_UnresponsiveStaleBusyFileDoesNotShield(t *testing.T) {
s, ex, _, clk := unrespServer(t, "unhealthy")
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\nan app update is running\n"
sweep(s, controllerUnresponsiveRestart)
if n := ex.count(9201); n != 1 {
t.Fatalf("a stale busy file blocked the restart: %d restarts", n)
}
}
+2 -2
View File
@@ -25,9 +25,9 @@
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
],
"recorded": "2026-10-29",
"recorded_by": "CC 2026-10-29: day0_agent 0.168.0 (operator vouch 2026-10-29 10:21:56 CET, read from the hub log: agent=0.168.0, bundle 3415c034… = the release); earlier 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"recorded_by": "CC 2026-10-30: day0_agent 0.171.0 (operator vouch 2026-10-30 10:47:36 CET, read from the hub log: agent=0.171.0, bundle 95f977f6… = the release; the save took 132 ms with hub 0.178.0, R-1346); earlier 0.170.0 (operator vouch 2026-10-29 18:57:48 CET, read from the hub log: agent=0.170.0, bundle 03f2ff64… = the release; the save took 38.3 s, R-1346); earlier 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.168.0"
"day0_agent": "0.171.0"
}
}