Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ca15a31690 | ||
|
|
bdea4694a4 | ||
|
|
db25eb0e71 | ||
|
|
637c92c762 | ||
|
|
63e63ee8bb | ||
|
|
6b0b093199 | ||
|
|
f4e63a55da |
No files matched your search
+53
-2
@@ -1,4 +1,55 @@
|
||||
## Unreleased (2026-10-28 night) — R-528: GET /guest/oom-kills, the kernel's OOM counter of each of the guest's containers, read on the host — commit only; the next agent release (needs the operator's vouch)
|
||||
## Unreleased — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list
|
||||
|
||||
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
|
||||
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
|
||||
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
|
||||
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
|
||||
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
|
||||
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
|
||||
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
|
||||
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
|
||||
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
|
||||
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
|
||||
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
|
||||
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
|
||||
a racing start would fail the reboot's own start half.
|
||||
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
|
||||
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
|
||||
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
|
||||
restarted your box at <time>".
|
||||
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
|
||||
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
|
||||
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
|
||||
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
|
||||
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
|
||||
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
|
||||
|
||||
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
|
||||
|
||||
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
|
||||
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
|
||||
answers on the next report, `operator_action_results: [{id, outcome, message}]` (`hub.HostReport`), which it requests
|
||||
at once through the cap-1 report trigger. CLOSED list: `restart_controller <vmid>` only — anything else refused, nothing
|
||||
called. The act is the supervisor's (`systemctl restart felhom-controller-bootstrap.service` in the guest, existing
|
||||
grants — no sudoers change, no config bundle) with its guards: provisioned guest, guest running, the guest list and the
|
||||
vzdump state readable, not parked, no swap, not locked, no vzdump. The thrash brake counts operator restarts with the
|
||||
supervisor's: 3 in 15 minutes → the press is refused, and presses stay refused for 30 minutes. The result says how long
|
||||
the controller took to answer (Docker's health check: healthy), or `failed` when it did not within 3 minutes. One
|
||||
`audit: operator action` line per decision. Works with the controller stopped. Tests `TestOpActions_*` (closed list,
|
||||
unknown/bad arg refused, restart + answer time + report trigger + once per id + re-send until dropped, stopped
|
||||
controller, every guard, the brake across supervisor and operator restarts, no healthy answer = failed) and
|
||||
`TestMainWiresTheAgentOperatorActions` (AST) — each seen failing by a mutation.
|
||||
- **R-1344 — a controller that RUNS but does not answer: ALARM ONLY.** Each sweep also reads Docker's health status for a
|
||||
running controller (the swap's `docker inspect -f` grant); 6 consecutive „unhealthy" sweeps (3 minutes, on top of
|
||||
Docker's own ~90 s) start an episode: one ERROR line, the stanza's `unresponsive` / `unresponsive_since` (the hub raises
|
||||
`controller_unresponsive`, operator-only). NOT restarted for it: the brief allows the restart only after a measurement
|
||||
shows a busy night never reads like this, and that measurement runs through tonight (felhom.eu R-1344). Not counted
|
||||
during a swap, a parked or locked guest, or a vzdump; healthy / starting / no health check / an unreadable status reset
|
||||
the count. Tests `TestSupervisor_Unresponsive*` (red by mutation: the vzdump guard, the threshold).
|
||||
|
||||
## v0.168.0 (2026-10-29) — R-528: the host's per-container OOM counter (`GET /guest/oom-kills`); R-1254: a lost or late escrow answer is asked again before it is worded — RELEASED (release-agent.sh from a clean clone of `91bd96c`; agent sha256 1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca, bundle 3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8; no configs/ change since v0.167.0 → binary-only signed updates to demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 10:21:56 CET**; Day-0 pin set (D50)
|
||||
|
||||
### R-528
|
||||
|
||||
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
|
||||
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
|
||||
@@ -11,7 +62,7 @@
|
||||
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
|
||||
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
|
||||
|
||||
## Unreleased (2026-10-28 night) — R-1254 (the ceremony half, review round 2 HUNT-3 H2): the agent asks the hub which package it holds before it words a failed upload — commit only; the next agent release (needs the operator's vouch)
|
||||
### R-1254 (the ceremony half, review round 2 HUNT-3 H2)
|
||||
|
||||
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
|
||||
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
|
||||
|
||||
+13
@@ -1,5 +1,18 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
|
||||
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
|
||||
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
|
||||
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
|
||||
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
|
||||
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
|
||||
|
||||
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
|
||||
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
|
||||
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
|
||||
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
|
||||
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
|
||||
|
||||
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
|
||||
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
|
||||
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
|
||||
|
||||
@@ -1,30 +1,29 @@
|
||||
# REPORT — two agent fixes on main, NOT released (night 2026-10-28 → 29, unattended)
|
||||
# REPORT — v0.169.0 released and vouched: „Restart controller" from the hub (2026-10-29, attended)
|
||||
|
||||
**Plain words first.** Two changes are on main and tested. Neither is released: an agent release needs the operator's
|
||||
vouch, and the night brief allowed no release. Both go on the morning list.
|
||||
**Plain words first.** The operator can now restart a box's controller from the hub's host page. The agent does it, so it
|
||||
works when the controller is stopped or stuck. It uses the supervisor's own guards and brake: 3 restarts in 15 minutes,
|
||||
then presses are refused for 30 minutes. A controller that runs but does not answer now raises an operator alarm
|
||||
(`controller_unresponsive`); it is not restarted by itself yet — that waits for the night's measurement (R-1344).
|
||||
|
||||
## What changed
|
||||
## Commits
|
||||
|
||||
- **R-528 — the OOM counter (`6f933d6`, CI run id 2473 / #284 green).** New local-API route `GET /guest/oom-kills`. It
|
||||
reads the kernel's `oom_kill` count of each Docker container of the token's own guest, on the host, from
|
||||
`/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events` (mode 0444, readable by the non-root
|
||||
agent user). Docker inside an LXC guest does not see these kills (`OOMKilled` stays false). The controller reads the
|
||||
route every 30 s (controller `eed23e7`). Tests `TestR528_*`, red first by mutation (another guest's container served;
|
||||
`oom` parsed for `oom_kill`).
|
||||
- **R-1254 — the escrow ceremony half (`5ddd124`, CI run id 2471 / #283 green).** After an escrow upload that fails, the
|
||||
agent asks the hub which package it holds (the identity blob compared): the new one → success; another one → 502 and
|
||||
„the previous code still works"; no answer → 504, which promises nothing. Tests `TestR1254_*`, red first.
|
||||
| Commit | CI (job id) | What |
|
||||
|---|---|---|
|
||||
| `637c92c` | 2504, 2507 green | The agent's closed list: `restart_controller <vmid>` on the host-report reply (`ControlEnvelope.OperatorActions`), the result on the next host report (`OperatorActionResults`). Guards: provisioned, running, parked, swap, lock, vzdump. The brake counts operator restarts with the supervisor's (`opBrakeUntil`). One audit line, one result per id, re-sent until the hub records it; a report is asked for at once. R-1344: 6 unhealthy sweeps → `unresponsive_since` in the stanza. Tests in `internal/localapi/opactions_test.go`, `supervisor_unresponsive_test.go`, `opactions_wiring_test.go`, each seen failing by its mutation. |
|
||||
| `db25eb0` | 2513 green | v0.169.0 CHANGELOG; `pinned.day0_agent` = 0.169.0 (D50 pin), after the vouch. |
|
||||
|
||||
## Live proof
|
||||
## Release
|
||||
|
||||
- R-528, 2026-10-28 23:27 UTC on Tester 1 with a test build of this agent (put back after: 0.167.0, sha256 `28bdd282…`).
|
||||
A Radicale capped at 128 MB, a memory hog inside it killed: the host counter 0 → 1, one `app_oom` „seen by
|
||||
docker,cgroup", one mail to the operator with `"source":"docker,cgroup"`. Evidence: felhom.eu
|
||||
`documentation/audits/night-2026-10-28/r528-oom/`.
|
||||
- R-1254: no live proof. It needs a hub answer lost on purpose; that is a hub change, and no hub deploy runs at night.
|
||||
- `release-agent.sh 0.169.0` from a clean clone: sha256 `fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61`,
|
||||
bundle `3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b`, tag `v0.169.0`, verified by download.
|
||||
- Signed `agent_update` to the three test boxes; each committed.
|
||||
- Vouched by the operator 14:06:09 CET (the page answered 504; the save went through — R-1346).
|
||||
|
||||
## Next
|
||||
## Live proofs (Tester 1, through the hub endpoint the button posts, CC's `cc-hosts` token)
|
||||
|
||||
The agent release (the operator's vouch), then `pinned.day0_agent` in `scripts/retention-policy.json` (D50).
|
||||
The controller needs no MinAgent change for R-528: an old agent answers 404, and the controller then waits 10 minutes
|
||||
before it asks again.
|
||||
- 13:01–13:02 CET on rc builds: press 1 → the controller answered after 9 s; press 2 with the controller stopped by hand →
|
||||
restarted 5 s after the stop; press 3 done; press 4 refused („3 restarts in 15 minutes").
|
||||
- 14:49 CET on the release builds (agent 0.169.0, controller 0.348.0, hub 0.176.1): one press → „done — restarted; the
|
||||
controller answered after 9 s", on the hub 16 s after the press; the box showed the controller „Up 12 seconds".
|
||||
|
||||
Evidence: felhom.eu `documentation/audits/release-2026-10-29b/` (`p1-restart/`, `final/`).
|
||||
@@ -89,6 +89,7 @@
|
||||
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
|
||||
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
|
||||
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
|
||||
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller` only, `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
|
||||
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
|
||||
|
||||
### Proxmox client / hub / PBS / provisioning
|
||||
|
||||
@@ -1267,6 +1267,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
|
||||
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
|
||||
// which it requests at once through the same cap-1 trigger the storage watchdog uses.
|
||||
localSrv.SetOperatorActionTrigger(func() {
|
||||
select {
|
||||
case storageTrigger <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
})
|
||||
collector.SetOperatorActionResults(localSrv.OperatorActionResults)
|
||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner, localSrv))
|
||||
backupCapRefusal = localSrv.BackupCapRefusal // R-973
|
||||
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
|
||||
// the server in the report loop's observer chain, give the collector its results and give the server the report
|
||||
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
|
||||
// RED-PROOF: seen failing before the wiring existed (all three false).
|
||||
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
var observed, results, trigger bool
|
||||
ast.Inspect(f, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
fn, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
switch fn.Sel.Name {
|
||||
case "MultiObserver":
|
||||
for _, a := range call.Args {
|
||||
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
|
||||
observed = true
|
||||
}
|
||||
}
|
||||
case "SetOperatorActionResults":
|
||||
if len(call.Args) == 1 {
|
||||
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
|
||||
results = true
|
||||
}
|
||||
}
|
||||
case "SetOperatorActionTrigger":
|
||||
trigger = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !observed || !results || !trigger {
|
||||
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
|
||||
}
|
||||
}
|
||||
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
|
||||
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
||||
# + container across spaces, spike-confirmed)
|
||||
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
||||
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
|
||||
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
|
||||
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
|
||||
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
|
||||
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
|
||||
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
|
||||
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
|
||||
|
||||
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
||||
|
||||
@@ -151,6 +151,7 @@ var manifest = []Capability{
|
||||
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
|
||||
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
||||
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
||||
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
|
||||
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
||||
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
|
||||
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
|
||||
|
||||
@@ -128,6 +128,7 @@ type Collector struct {
|
||||
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
|
||||
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
|
||||
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
|
||||
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
|
||||
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
|
||||
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
|
||||
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
|
||||
@@ -235,6 +236,12 @@ type ControllerSupervisorReporter interface {
|
||||
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
|
||||
}
|
||||
|
||||
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
|
||||
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
|
||||
c.opResults = f
|
||||
return c
|
||||
}
|
||||
|
||||
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
|
||||
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
|
||||
c.ctrlSup = r
|
||||
@@ -432,6 +439,9 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
|
||||
if c.ctrlSup != nil {
|
||||
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
|
||||
}
|
||||
if c.opResults != nil {
|
||||
report.OperatorActionResults = c.opResults()
|
||||
}
|
||||
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
|
||||
if c.guestNet != nil {
|
||||
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
|
||||
|
||||
@@ -172,6 +172,24 @@ type HostReport struct {
|
||||
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
|
||||
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
|
||||
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
|
||||
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
|
||||
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
|
||||
// when there is nothing to answer, so the golden stays byte-stable.
|
||||
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
|
||||
}
|
||||
|
||||
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
|
||||
type OperatorAction struct {
|
||||
ID int64 `json:"id"`
|
||||
Action string `json:"action"`
|
||||
Arg string `json:"arg,omitempty"`
|
||||
}
|
||||
|
||||
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
|
||||
type OperatorActionResult struct {
|
||||
ID int64 `json:"id"`
|
||||
Outcome string `json:"outcome"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
|
||||
@@ -195,6 +213,11 @@ type ControllerSupervisorGuest struct {
|
||||
Restarts24h int `json:"restarts_24h"`
|
||||
SlowCrashloop bool `json:"slow_crashloop"`
|
||||
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
|
||||
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
|
||||
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
|
||||
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
|
||||
Unresponsive bool `json:"unresponsive"`
|
||||
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
|
||||
}
|
||||
|
||||
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
|
||||
@@ -631,6 +654,9 @@ type ControlEnvelope struct {
|
||||
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
|
||||
// Absent/false on an old hub → nothing happens.
|
||||
LogTailRequested bool `json:"log_tail_requested"`
|
||||
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
|
||||
// (`04` §6.1). Absent on an older hub.
|
||||
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
|
||||
}
|
||||
|
||||
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
|
||||
|
||||
@@ -83,6 +83,14 @@ const (
|
||||
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
|
||||
// the parked marker.
|
||||
controllerSlowCounterFile = "controller-restarts-24h.json"
|
||||
|
||||
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
|
||||
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
|
||||
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
|
||||
// the hub (controller_unresponsive). ALARM ONLY — it is NOT restarted for this until the measurement (R-1344) shows a
|
||||
// busy night (backups, updates) never reads unhealthy that long. Not counted during a swap, a parked guest, a locked
|
||||
// guest or a vzdump.
|
||||
controllerUnresponsiveConfirm = 6
|
||||
)
|
||||
|
||||
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
|
||||
@@ -103,6 +111,14 @@ type controllerSupState struct {
|
||||
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
|
||||
restarts24h []time.Time
|
||||
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
|
||||
|
||||
// opBrakeUntil (2026-10-29): an operator restart refused by the brake refuses every press until then (30 min).
|
||||
opBrakeUntil time.Time
|
||||
|
||||
// R-1344: consecutive running-but-unhealthy sweeps, and the current episode (since kept after it ends).
|
||||
unhealthySeen int
|
||||
unresponsive bool
|
||||
unresponsiveSince time.Time
|
||||
}
|
||||
|
||||
type controllerSupervisor struct {
|
||||
@@ -238,6 +254,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
|
||||
if !s.provisionedGuest(g.VMID) {
|
||||
continue
|
||||
}
|
||||
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
|
||||
s.resetNotRunning(g.VMID)
|
||||
continue
|
||||
}
|
||||
evaluated++
|
||||
if !s.superviseOneController(ctx, g.VMID, g.Status) {
|
||||
down++
|
||||
@@ -293,6 +313,7 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
if running {
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
s.observeControllerHealth(ctx, vmid, parked)
|
||||
return true
|
||||
}
|
||||
st.notRunningSeen++
|
||||
@@ -392,6 +413,52 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
|
||||
return false
|
||||
}
|
||||
|
||||
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
|
||||
// the swap's `docker inspect -f` grant. Anything but „unhealthy" — healthy, starting, no health check, a read that
|
||||
// fails — resets the count; so do a swap, a parked guest, a locked guest and a vzdump (each is someone else's moment).
|
||||
func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked bool) {
|
||||
s.swapMu.Lock()
|
||||
swapping := s.swapInFlight[vmid]
|
||||
s.swapMu.Unlock()
|
||||
unhealthy := false
|
||||
if !parked && !swapping {
|
||||
if out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer); err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
unhealthy = len(f) == 2 && f[0] == "true" && f[1] == "unhealthy"
|
||||
}
|
||||
}
|
||||
if unhealthy {
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil || lock != "" {
|
||||
unhealthy = false
|
||||
} else if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||
unhealthy = false
|
||||
}
|
||||
}
|
||||
now := s.clock()
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if !unhealthy {
|
||||
was := st.unresponsive
|
||||
st.unhealthySeen, st.unresponsive = 0, false
|
||||
s.ctrlSup.mu.Unlock()
|
||||
if was {
|
||||
s.logger.Info("controller-supervisor: the controller answers its health check again", "vmid", vmid)
|
||||
}
|
||||
return
|
||||
}
|
||||
st.unhealthySeen++
|
||||
seen := st.unhealthySeen
|
||||
start := seen >= controllerUnresponsiveConfirm && !st.unresponsive
|
||||
if start {
|
||||
st.unresponsive, st.unresponsiveSince = true, now
|
||||
}
|
||||
s.ctrlSup.mu.Unlock()
|
||||
if start {
|
||||
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; NOT restarting it (R-1344: alarm only until measured)",
|
||||
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String())
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) resetNotRunning(vmid int) {
|
||||
s.ctrlSup.mu.Lock()
|
||||
defer s.ctrlSup.mu.Unlock()
|
||||
@@ -446,6 +513,10 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
|
||||
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
|
||||
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
|
||||
}
|
||||
g.Unresponsive = st.unresponsive
|
||||
if !st.unresponsiveSince.IsZero() {
|
||||
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
|
||||
}
|
||||
out.Guests = append(out.Guests, g)
|
||||
}
|
||||
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
|
||||
|
||||
@@ -152,6 +152,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
|
||||
return
|
||||
}
|
||||
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
|
||||
// would fail the reboot's own start half).
|
||||
if s.rebooting(g.VMID) {
|
||||
return
|
||||
}
|
||||
|
||||
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,258 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// The operator's actions for THIS agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).
|
||||
//
|
||||
// THE DOOR IS THE HOST REPORT'S REPLY, the controller's shape: the operator presses a button on the hub's host page, the
|
||||
// hub stores a row and pokes the box (the poke's out-of-cycle report gets the reply in seconds; without the tunnel the
|
||||
// 15-minute cycle carries it), the reply lists `operator_actions: [{id, action, arg}]`, this agent acts ONCE per id, and
|
||||
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
|
||||
// The result is re-sent until the reply stops listing the id.
|
||||
//
|
||||
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
|
||||
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
|
||||
//
|
||||
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
|
||||
// the guest over the existing grants, with the supervisor's guards — not during a controller swap, not parked, not on a
|
||||
// stopped, locked or vzdump-busy guest, not when the guest list or the vzdump state cannot be read — and the thrash brake
|
||||
// counts these restarts too: 3 restarts (the supervisor's or the operator's) inside 15 minutes refuse the press, and
|
||||
// presses stay refused for 30 minutes. It works with the controller stopped or stuck: the agent does it, not the
|
||||
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
|
||||
|
||||
const (
|
||||
// OpRestartController is the agent's first operator verb.
|
||||
OpRestartController = "restart_controller"
|
||||
|
||||
opOutcomeDone = "done"
|
||||
opOutcomeRefused = "refused"
|
||||
opOutcomeFailed = "failed"
|
||||
|
||||
// opBrakeRefusal is how long presses stay refused after the brake refused one.
|
||||
opBrakeRefusal = 30 * time.Minute
|
||||
// opForgetAfter drops a started id the reply no longer lists.
|
||||
opForgetAfter = 24 * time.Hour
|
||||
|
||||
defaultOpHealthPoll = 2 * time.Second
|
||||
// defaultOpHealthWait covers the bootstrap's pull-less rm/run (seconds), Docker's start period (10 s) and its first
|
||||
// probe (30 s interval) with room: the controller's own HEALTHCHECK is the answer.
|
||||
defaultOpHealthWait = 3 * time.Minute
|
||||
// opHealthTemplate reads running + health over the swap's `docker inspect -f` grant.
|
||||
opHealthTemplate = "{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}"
|
||||
)
|
||||
|
||||
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
|
||||
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
|
||||
|
||||
// OperatorActionNames returns the closed list (sorted copy).
|
||||
func OperatorActionNames() []string {
|
||||
out := append([]string(nil), operatorActionNames...)
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
type opActionState struct {
|
||||
mu sync.Mutex
|
||||
started map[int64]time.Time
|
||||
results map[int64]hub.OperatorActionResult
|
||||
}
|
||||
|
||||
// SetOperatorActionTrigger wires the report loop's out-of-band trigger (the result goes at once, not in 15 minutes).
|
||||
func (s *Server) SetOperatorActionTrigger(f func()) { s.opTrigger = f }
|
||||
|
||||
// OnEnvelope is the report loop's hook (hub.EnvelopeObserver): it starts each newly listed action once and forgets
|
||||
// the ids the reply no longer lists (the hub has their results).
|
||||
func (s *Server) OnEnvelope(_ context.Context, env *hub.ControlEnvelope) {
|
||||
if env == nil {
|
||||
return
|
||||
}
|
||||
listed := map[int64]bool{}
|
||||
var start []hub.OperatorAction
|
||||
now := time.Now()
|
||||
s.opActs.mu.Lock()
|
||||
if s.opActs.started == nil {
|
||||
s.opActs.started = map[int64]time.Time{}
|
||||
s.opActs.results = map[int64]hub.OperatorActionResult{}
|
||||
}
|
||||
for _, a := range env.OperatorActions {
|
||||
listed[a.ID] = true
|
||||
if _, ok := s.opActs.started[a.ID]; ok {
|
||||
continue
|
||||
}
|
||||
s.opActs.started[a.ID] = now
|
||||
start = append(start, a)
|
||||
}
|
||||
for id, at := range s.opActs.started {
|
||||
if listed[id] {
|
||||
continue
|
||||
}
|
||||
if _, answered := s.opActs.results[id]; answered || now.Sub(at) > opForgetAfter {
|
||||
delete(s.opActs.started, id)
|
||||
delete(s.opActs.results, id)
|
||||
}
|
||||
}
|
||||
s.opActs.mu.Unlock()
|
||||
for _, a := range start {
|
||||
go s.runOperatorAction(a) // a restart waits for the health answer — never block the report loop
|
||||
}
|
||||
}
|
||||
|
||||
// OperatorActionResults is the report's answer list, oldest id first.
|
||||
func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
|
||||
s.opActs.mu.Lock()
|
||||
defer s.opActs.mu.Unlock()
|
||||
out := make([]hub.OperatorActionResult, 0, len(s.opActs.results))
|
||||
for _, r := range s.opActs.results {
|
||||
out = append(out, r)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out
|
||||
}
|
||||
|
||||
func (s *Server) runOperatorAction(a hub.OperatorAction) {
|
||||
limit := s.opWait() + time.Minute
|
||||
if a.Action == OpRebootGuest {
|
||||
limit += opRebootTimeout
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), limit)
|
||||
defer cancel()
|
||||
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
|
||||
switch a.Action {
|
||||
case OpRestartController:
|
||||
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
|
||||
case OpRebootGuest:
|
||||
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
|
||||
}
|
||||
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
|
||||
"outcome", outcome, "message", msg)
|
||||
s.opActs.mu.Lock()
|
||||
if _, still := s.opActs.started[a.ID]; still {
|
||||
s.opActs.results[a.ID] = hub.OperatorActionResult{ID: a.ID, Outcome: outcome, Message: msg}
|
||||
}
|
||||
s.opActs.mu.Unlock()
|
||||
if s.opTrigger != nil {
|
||||
s.opTrigger()
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) opPoll() time.Duration {
|
||||
if s.opHealthPoll > 0 {
|
||||
return s.opHealthPoll
|
||||
}
|
||||
return defaultOpHealthPoll
|
||||
}
|
||||
|
||||
func (s *Server) opWait() time.Duration {
|
||||
if s.opHealthWait > 0 {
|
||||
return s.opHealthWait
|
||||
}
|
||||
return defaultOpHealthWait
|
||||
}
|
||||
|
||||
// operatorRestartController is the press: the supervisor's guards, the brake, the restart, then the wait for the
|
||||
// controller's healthy answer. Returns the outcome and the sentence the host page shows.
|
||||
func (s *Server) operatorRestartController(ctx context.Context, id int64, arg string) (string, string) {
|
||||
vmid, err := strconv.Atoi(arg)
|
||||
if err != nil || strconv.Itoa(vmid) != arg {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||
}
|
||||
if s.staleLock == nil || s.guestExec == nil {
|
||||
return opOutcomeRefused, "refused: the controller supervisor is not wired on this agent"
|
||||
}
|
||||
if !s.provisionedGuest(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||
}
|
||||
status := ""
|
||||
for _, g := range guests {
|
||||
if g.VMID == vmid {
|
||||
status = g.Status
|
||||
}
|
||||
}
|
||||
if status != "running" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||
}
|
||||
if s.controllerParked(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||
}
|
||||
s.swapMu.Lock()
|
||||
swapping := s.swapInFlight[vmid]
|
||||
s.swapMu.Unlock()
|
||||
if swapping {
|
||||
return opOutcomeRefused, "refused: a controller swap (update) is in flight — it owns the controller"
|
||||
}
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||
} else if lock != "" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||
}
|
||||
if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
|
||||
return opOutcomeRefused, "refused: a whole-guest backup (vzdump) may be in flight for the guest"
|
||||
}
|
||||
|
||||
// The brake: every restart counts — the supervisor's and the operator's.
|
||||
now := s.clock()
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if now.Before(st.opBrakeUntil) {
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||
if len(st.restarts) >= controllerCrashloopMax {
|
||||
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
reason := fmt.Sprintf("the operator via the hub (action #%d)", id)
|
||||
st.restarts = append(st.restarts, now)
|
||||
st.restartsTotal++
|
||||
st.lastRestartAt = now
|
||||
st.lastReason = reason
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
|
||||
s.logger.Warn("controller-supervisor: RESTARTING the controller on the operator's press", "vmid", vmid, "action_id", id, "unit", bootstrapUnit)
|
||||
began := time.Now()
|
||||
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
|
||||
return opOutcomeFailed, fmt.Sprintf("the restart of guest %d's controller failed: %v", vmid, err)
|
||||
}
|
||||
for time.Since(began) < s.opWait() {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||
if err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||
secs := int(time.Since(began).Round(time.Second) / time.Second)
|
||||
return opOutcomeDone, fmt.Sprintf("restarted; the controller answered after %d s (Docker's health check: healthy)", secs)
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return opOutcomeFailed, "restarted, but the wait for its answer was cut"
|
||||
case <-time.After(s.opPoll()):
|
||||
}
|
||||
}
|
||||
return opOutcomeFailed, fmt.Sprintf("restarted, but no healthy answer within %s — look at the box", s.opWait().Round(time.Second))
|
||||
}
|
||||
|
||||
func orDash(s string) string {
|
||||
if s == "" {
|
||||
return "not listed"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// Operator brief 2026-10-29, Part 1: „Restart controller" from the hub. The agent's closed list holds one verb; the act
|
||||
// is the supervisor's own restart with the supervisor's guards, and the brake counts operator restarts too.
|
||||
|
||||
// opExec answers the supervisor's fake plus the health read the operator restart waits on.
|
||||
type opExec struct {
|
||||
*supExec
|
||||
health string // what {{.State.Health.Status}} reads after the restart
|
||||
}
|
||||
|
||||
func (f *opExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||
if len(args) >= 5 && args[0] == "docker" && args[1] == "inspect" && args[3] == opHealthTemplate {
|
||||
f.mu.Lock()
|
||||
st := f.status[vmid]
|
||||
f.mu.Unlock()
|
||||
if st == "" {
|
||||
return "", errors.New("pct exec: exit status 1: Error: No such object: felhom-controller")
|
||||
}
|
||||
return map[bool]string{true: "true", false: "false"}[st == "running"] + "|" + f.health + "\n", nil
|
||||
}
|
||||
return f.supExec.GuestExec(ctx, vmid, args...)
|
||||
}
|
||||
|
||||
func opServer(t *testing.T, status string) (*Server, *opExec, *fakeGuestPowerCtl, *supClock, string, *atomic.Int32) {
|
||||
t.Helper()
|
||||
ex := &opExec{supExec: &supExec{status: map[int]string{9201: status}, onRestart: "running"}, health: "healthy"}
|
||||
ctl := runningGuest(9201)
|
||||
s, clk, dir := supServer(t, nil, ctl, 9201)
|
||||
s.guestExec = ex
|
||||
s.opHealthPoll = time.Millisecond
|
||||
s.opHealthWait = 2 * time.Second
|
||||
var triggered atomic.Int32
|
||||
s.SetOperatorActionTrigger(func() { triggered.Add(1) })
|
||||
return s, ex, ctl, clk, dir, &triggered
|
||||
}
|
||||
|
||||
func env(acts ...hub.OperatorAction) *hub.ControlEnvelope {
|
||||
return &hub.ControlEnvelope{OperatorActions: acts}
|
||||
}
|
||||
|
||||
// waitResult waits for the action's result (the act runs in the background).
|
||||
func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
for _, r := range s.OperatorActionResults() {
|
||||
if r.ID == id {
|
||||
return r
|
||||
}
|
||||
}
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("no result for #%d", id)
|
||||
return hub.OperatorActionResult{}
|
||||
}
|
||||
|
||||
// RED-PROOF: seen failing before the list existed (build failure, then the names).
|
||||
func TestOpActions_AgentClosedList(t *testing.T) {
|
||||
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Anything outside the list — and a bad argument — is refused and NOTHING is called.
|
||||
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "running")
|
||||
s.OnEnvelope(context.Background(), env(
|
||||
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
|
||||
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
|
||||
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
|
||||
))
|
||||
for _, id := range []int64{1, 2, 3} {
|
||||
if r := waitResult(t, s, id); r.Outcome != "refused" {
|
||||
t.Errorf("#%d: %+v, want refused", id, r)
|
||||
}
|
||||
}
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("a refused action restarted the controller (%d)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The press restarts ONCE, says how long the controller took to answer, asks for a report at once, and re-sends the
|
||||
// result until the hub stops listing the id. RED-PROOF: drop the GuestExec restart → restarts 0 → FAILS.
|
||||
func TestOpActions_RestartReportsTheAnswerTime(t *testing.T) {
|
||||
s, ex, _, _, _, trig := opServer(t, "running")
|
||||
a := hub.OperatorAction{ID: 7, Action: "restart_controller", Arg: "9201"}
|
||||
s.OnEnvelope(context.Background(), env(a))
|
||||
r := waitResult(t, s, 7)
|
||||
if r.Outcome != "done" || !strings.Contains(r.Message, "answered after") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("restarts = %d, want 1", n)
|
||||
}
|
||||
if trig.Load() == 0 {
|
||||
t.Fatal("the agent did not ask for a report — the result would wait 15 minutes")
|
||||
}
|
||||
s.OnEnvelope(context.Background(), env(a)) // the hub still lists it: re-send, never act again
|
||||
time.Sleep(100 * time.Millisecond) // a wrong second act runs in the background — give it time to show
|
||||
if n := ex.count(9201); n != 1 {
|
||||
t.Fatalf("a re-listed id acted again (%d)", n)
|
||||
}
|
||||
if len(s.OperatorActionResults()) != 1 {
|
||||
t.Fatal("the result must be re-sent while listed")
|
||||
}
|
||||
s.OnEnvelope(context.Background(), env()) // the hub recorded it
|
||||
if left := s.OperatorActionResults(); len(left) != 0 {
|
||||
t.Fatalf("a recorded result is still sent: %+v", left)
|
||||
}
|
||||
}
|
||||
|
||||
// The point of the agent doing it: a STOPPED controller is restarted too (the controller cannot restart itself).
|
||||
func TestOpActions_WorksWithTheControllerStopped(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "exited")
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 8, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 8); r.Outcome != "done" || ex.count(9201) != 1 {
|
||||
t.Fatalf("result %+v, restarts %d", r, ex.count(9201))
|
||||
}
|
||||
}
|
||||
|
||||
// Each supervisor guard refuses the press, with the reason. RED-PROOF: drop the lock check → the locked case
|
||||
// restarts → FAILS (each guard seen failing by its own mutation).
|
||||
func TestOpActions_GuardsRefuse(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, want string
|
||||
setup func(s *Server, ctl *fakeGuestPowerCtl, dir string)
|
||||
}{
|
||||
{"parked", "parked", func(_ *Server, _ *fakeGuestPowerCtl, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
{"swap", "swap", func(s *Server, _ *fakeGuestPowerCtl, _ string) { s.swapInFlight[9201] = true }},
|
||||
{"locked", "locked", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.locks[9201] = "backup" }},
|
||||
{"vzdump", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||
{"vzdump unknown", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupErr = errors.New("tasks unreadable") }},
|
||||
{"guest stopped", "not running", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guests[0].Status = "stopped" }},
|
||||
{"guest list", "guest list", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guestsErr = errors.New("api down") }},
|
||||
}
|
||||
for i, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, ctl, _, dir, _ := opServer(t, "running")
|
||||
tc.setup(s, ctl, dir)
|
||||
id := int64(100 + i)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
r := waitResult(t, s, id)
|
||||
if r.Outcome != "refused" || !strings.Contains(strings.ToLower(r.Message), tc.want) {
|
||||
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||
}
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("guard %q did not hold: restarted %d", tc.name, n)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The thrash brake counts operator restarts AND the supervisor's: 3 in 15 minutes → the next press is refused, and
|
||||
// presses stay refused for 30 minutes. RED-PROOF: drop the brake → the 4th press restarts → FAILS.
|
||||
func TestOpActions_TheBrakeCountsEveryRestart(t *testing.T) {
|
||||
s, ex, _, clk, _, _ := opServer(t, "running")
|
||||
// one restart by the supervisor itself
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "exited"
|
||||
ex.mu.Unlock()
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
if ex.count(9201) != 1 {
|
||||
t.Fatalf("setup: the supervisor did not restart (%d)", ex.count(9201))
|
||||
}
|
||||
for id := int64(1); id <= 2; id++ {
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||
t.Fatalf("press %d: %+v", id, r)
|
||||
}
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9201"}))
|
||||
r := waitResult(t, s, 3)
|
||||
if r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||
t.Fatalf("the 4th restart in 15 minutes: %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 3 {
|
||||
t.Fatalf("restarts = %d, want 3", n)
|
||||
}
|
||||
clk.t = clk.t.Add(20 * time.Minute) // the window has passed, but the 30-minute refusal has not
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 4); r.Outcome != "refused" {
|
||||
t.Fatalf("inside the 30-minute refusal: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(11 * time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 5, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 5); r.Outcome != "done" {
|
||||
t.Fatalf("after the refusal: %+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
// A controller that does not come back healthy is a FAILED press, said plainly.
|
||||
func TestOpActions_NoHealthyAnswerIsFailed(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "running")
|
||||
ex.health = "unhealthy"
|
||||
s.opHealthWait = 30 * time.Millisecond
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 9, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 9); r.Outcome != "failed" || !strings.Contains(r.Message, "no healthy answer") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
|
||||
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
|
||||
//
|
||||
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
|
||||
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
|
||||
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
|
||||
//
|
||||
// THE GUARDS (each refuses with the reason in the result; nothing is called):
|
||||
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
|
||||
// - a controller swap (update) is in flight;
|
||||
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
|
||||
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
|
||||
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
|
||||
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
|
||||
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
|
||||
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
|
||||
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
|
||||
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
|
||||
// alone while the reboot runs (rebootInFlight).
|
||||
//
|
||||
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
|
||||
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
|
||||
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
|
||||
// one day). Pinned by opreboot_test.go.
|
||||
|
||||
const (
|
||||
// OpRebootGuest is the agent's second operator verb (D65 = A).
|
||||
OpRebootGuest = "reboot_guest"
|
||||
|
||||
// opRebootEvery: at most one operator reboot per guest per hour.
|
||||
opRebootEvery = time.Hour
|
||||
// opRebootNoticeFor: how long the household's dashboard shows the line.
|
||||
opRebootNoticeFor = 24 * time.Hour
|
||||
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
|
||||
opRebootTimeout = 5 * time.Minute
|
||||
|
||||
operatorRebootFile = "operator-reboot.json"
|
||||
|
||||
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
|
||||
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
|
||||
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
|
||||
// Docker does not hide it.
|
||||
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
|
||||
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
|
||||
// stopped writing (stuck or killed) — not a busy one.
|
||||
controllerBusyFresh = 90 * time.Second
|
||||
)
|
||||
|
||||
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
|
||||
type operatorRebootRecord struct {
|
||||
At time.Time `json:"at"`
|
||||
ActionID int64 `json:"action_id"`
|
||||
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
|
||||
ControllerAfterS int `json:"controller_after_s,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) operatorRebootPath(vmid int) string {
|
||||
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
|
||||
}
|
||||
|
||||
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
|
||||
// must not silently vanish).
|
||||
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
|
||||
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return operatorRebootRecord{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return operatorRebootRecord{}, false, err
|
||||
}
|
||||
var rec operatorRebootRecord
|
||||
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
|
||||
}
|
||||
return rec, !rec.At.IsZero(), nil
|
||||
}
|
||||
|
||||
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
|
||||
raw, err := json.Marshal(rec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p := s.operatorRebootPath(vmid)
|
||||
tmp := p + ".tmp"
|
||||
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, p)
|
||||
}
|
||||
|
||||
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
|
||||
func (s *Server) rebooting(vmid int) bool {
|
||||
s.rebootMu.Lock()
|
||||
defer s.rebootMu.Unlock()
|
||||
return s.rebootInFlight[vmid]
|
||||
}
|
||||
|
||||
func (s *Server) setRebooting(vmid int, on bool) {
|
||||
s.rebootMu.Lock()
|
||||
defer s.rebootMu.Unlock()
|
||||
if s.rebootInFlight == nil {
|
||||
s.rebootInFlight = map[int]bool{}
|
||||
}
|
||||
if on {
|
||||
s.rebootInFlight[vmid] = true
|
||||
} else {
|
||||
delete(s.rebootInFlight, vmid)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) rebooter() interface {
|
||||
RebootGuest(ctx context.Context, vmid int) error
|
||||
} {
|
||||
if s.opRebooter != nil {
|
||||
return s.opRebooter
|
||||
}
|
||||
if s.guestAttach != nil {
|
||||
return s.guestAttach
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
|
||||
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
|
||||
// (the caller refuses — it cannot prove no backup runs).
|
||||
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
|
||||
if err != nil {
|
||||
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
|
||||
return false, "", nil
|
||||
}
|
||||
return false, "", err
|
||||
}
|
||||
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
|
||||
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
|
||||
if perr != nil {
|
||||
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
|
||||
}
|
||||
if s.clock().Sub(at) > controllerBusyFresh {
|
||||
s.logger.Info("operator reboot: the controller's busy file is stale — the controller stopped writing it; not busy",
|
||||
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
|
||||
return false, "", nil
|
||||
}
|
||||
why := "work in progress"
|
||||
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
|
||||
why = strings.TrimSpace(lines[1])
|
||||
}
|
||||
return true, why, nil
|
||||
}
|
||||
|
||||
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
|
||||
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
|
||||
vmid, err := strconv.Atoi(arg)
|
||||
if err != nil || strconv.Itoa(vmid) != arg {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||
}
|
||||
rb := s.rebooter()
|
||||
if s.staleLock == nil || s.guestExec == nil || rb == nil {
|
||||
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
|
||||
}
|
||||
if !s.provisionedGuest(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||
}
|
||||
if s.rebooting(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||
}
|
||||
status := ""
|
||||
for _, g := range guests {
|
||||
if g.VMID == vmid {
|
||||
status = g.Status
|
||||
}
|
||||
}
|
||||
if status != "running" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||
}
|
||||
if s.controllerParked(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||
}
|
||||
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||
} else if lock != "" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||
}
|
||||
|
||||
// At most one per hour.
|
||||
now := s.clock()
|
||||
last, had, err := s.lastOperatorReboot(vmid)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
|
||||
}
|
||||
if had && now.Sub(last.At) < opRebootEvery {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
|
||||
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
|
||||
}
|
||||
|
||||
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if now.Before(st.opBrakeUntil) {
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||
if len(st.restarts) >= controllerCrashloopMax {
|
||||
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
s.ctrlSup.mu.Unlock()
|
||||
|
||||
// The heavy-operation gate, held until the guest is up again.
|
||||
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
|
||||
if !ok {
|
||||
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
|
||||
}
|
||||
defer release()
|
||||
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
|
||||
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
|
||||
}
|
||||
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
|
||||
} else if cbusy {
|
||||
return opOutcomeRefused, "refused: the controller is busy — " + why
|
||||
}
|
||||
|
||||
// The record FIRST: the hourly limit and the household's line both read it.
|
||||
rec := operatorRebootRecord{At: now, ActionID: id}
|
||||
if err := s.saveOperatorReboot(vmid, rec); err != nil {
|
||||
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
|
||||
}
|
||||
s.ctrlSup.mu.Lock()
|
||||
st = s.supState(vmid)
|
||||
st.restarts = append(st.restarts, now)
|
||||
st.restartsTotal++
|
||||
st.lastRestartAt = now
|
||||
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
s.setRebooting(vmid, true)
|
||||
defer s.setRebooting(vmid, false)
|
||||
|
||||
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
|
||||
began := time.Now()
|
||||
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
|
||||
err = rb.RebootGuest(rctx, vmid)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
|
||||
}
|
||||
upS := int(time.Since(began).Round(time.Second) / time.Second)
|
||||
rec.GuestUpAfterS = upS
|
||||
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
|
||||
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||
if err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
|
||||
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
|
||||
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
|
||||
}
|
||||
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
|
||||
case <-time.After(s.opPoll()):
|
||||
}
|
||||
}
|
||||
_ = s.saveOperatorReboot(vmid, rec)
|
||||
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
|
||||
}
|
||||
|
||||
// rebootBlockerSentence names jobBlocker's code in the result's words.
|
||||
func rebootBlockerSentence(code string) string {
|
||||
switch code {
|
||||
case PowerOffBackup:
|
||||
return "a backup is running"
|
||||
case PowerOffCtrlUpdate:
|
||||
return "a controller update is running"
|
||||
case PowerOffDrive:
|
||||
return "a drive or network-storage job is running"
|
||||
case PowerOffEscrow:
|
||||
return "an escrow ceremony is running"
|
||||
case PowerOffUnknown:
|
||||
return "the running jobs cannot be read"
|
||||
default:
|
||||
return "the host is busy"
|
||||
}
|
||||
}
|
||||
|
||||
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
|
||||
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
|
||||
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
|
||||
rec, ok, err := s.lastOperatorReboot(vmid)
|
||||
if err != nil {
|
||||
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
|
||||
return
|
||||
}
|
||||
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
|
||||
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
|
||||
return
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
|
||||
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
|
||||
|
||||
// rbExec adds the controller's busy file to the operator-restart fake.
|
||||
type rbExec struct {
|
||||
*opExec
|
||||
busy string // the file's content; "" = no file
|
||||
busyErr error // pct exec itself failed
|
||||
}
|
||||
|
||||
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
|
||||
if f.busyErr != nil {
|
||||
return "", f.busyErr
|
||||
}
|
||||
if f.busy == "" {
|
||||
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
|
||||
}
|
||||
return f.busy, nil
|
||||
}
|
||||
return f.opExec.GuestExec(ctx, vmid, args...)
|
||||
}
|
||||
|
||||
type fakeRebooter struct {
|
||||
mu sync.Mutex
|
||||
n int
|
||||
err error
|
||||
after func()
|
||||
}
|
||||
|
||||
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.n++
|
||||
if r.after != nil {
|
||||
r.after()
|
||||
}
|
||||
return r.err
|
||||
}
|
||||
|
||||
func (r *fakeRebooter) count() int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.n
|
||||
}
|
||||
|
||||
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
|
||||
t.Helper()
|
||||
s, ox, ctl, clk, dir, _ := opServer(t, "running")
|
||||
ex := &rbExec{opExec: ox}
|
||||
s.guestExec = ex
|
||||
rb := &fakeRebooter{}
|
||||
s.opRebooter = rb
|
||||
s.inFlight = &backup.InFlight{}
|
||||
return s, ex, rb, ctl, clk, dir
|
||||
}
|
||||
|
||||
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||
t.Helper()
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
|
||||
return waitResult(t, s, id)
|
||||
}
|
||||
|
||||
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
|
||||
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
|
||||
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
|
||||
s, _, rb, _, clk, _ := rbServer(t)
|
||||
r := pressReboot(t, s, 1)
|
||||
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if rb.count() != 1 {
|
||||
t.Fatalf("reboots = %d, want 1", rb.count())
|
||||
}
|
||||
rec, ok, err := s.lastOperatorReboot(9201)
|
||||
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
|
||||
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
|
||||
}
|
||||
if s.inFlight.Busy() != "" {
|
||||
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
|
||||
}
|
||||
if s.rebooting(9201) {
|
||||
t.Fatal("the reboot-in-flight flag outlived the reboot")
|
||||
}
|
||||
}
|
||||
|
||||
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
|
||||
func TestOpReboot_GuardsRefuse(t *testing.T) {
|
||||
fresh := func(clk *supClock) string {
|
||||
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
|
||||
}
|
||||
cases := []struct {
|
||||
name, want string
|
||||
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
|
||||
}{
|
||||
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
|
||||
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
|
||||
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.guests[0].Status = "stopped"
|
||||
}},
|
||||
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.backupRun = map[int]bool{9201: true}
|
||||
}},
|
||||
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.backupErr = errors.New("tasks unreadable")
|
||||
}},
|
||||
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
s.inFlight.TryAcquire("backup:felhom-pbs")
|
||||
}},
|
||||
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
|
||||
}},
|
||||
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||
ex.busy = fresh(clk)
|
||||
}},
|
||||
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
|
||||
}},
|
||||
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||
s.ctrlSup.mu.Lock()
|
||||
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
|
||||
s.ctrlSup.mu.Unlock()
|
||||
}},
|
||||
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for i, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, rb, ctl, clk, dir := rbServer(t)
|
||||
tc.setup(s, ex, ctl, clk, dir)
|
||||
r := pressReboot(t, s, int64(100+i))
|
||||
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
|
||||
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||
}
|
||||
if rb.count() != 0 {
|
||||
t.Fatalf("guard %q did not hold: rebooted", tc.name)
|
||||
}
|
||||
if _, ok, _ := s.lastOperatorReboot(9201); ok {
|
||||
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
|
||||
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
|
||||
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
|
||||
s, ex, rb, _, clk, _ := rbServer(t)
|
||||
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
|
||||
t.Fatalf("result %+v, reboots %d", r, rb.count())
|
||||
}
|
||||
}
|
||||
|
||||
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
|
||||
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
|
||||
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
|
||||
s, _, rb, ctl, clk, dir := rbServer(t)
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("first press: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(59 * time.Minute)
|
||||
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
|
||||
t.Fatalf("second press inside the hour: %+v", r)
|
||||
}
|
||||
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
|
||||
s2, ox2, _, _, _, _ := opServer(t, "running")
|
||||
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
|
||||
s2.guestExec = &rbExec{opExec: ox2}
|
||||
rb2 := &fakeRebooter{}
|
||||
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
|
||||
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
|
||||
t.Fatalf("after an agent restart, inside the hour: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(2 * time.Minute)
|
||||
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
|
||||
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
|
||||
}
|
||||
}
|
||||
|
||||
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
|
||||
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
|
||||
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
|
||||
s, ex, _, _, clk, _ := rbServer(t)
|
||||
for id := int64(1); id <= 2; id++ {
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||
t.Fatalf("restart %d: %+v", id, r)
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
}
|
||||
if r := pressReboot(t, s, 3); r.Outcome != "done" {
|
||||
t.Fatalf("reboot: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 2 {
|
||||
t.Fatalf("restarts = %d, want 2", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
|
||||
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
|
||||
s, _, rb, _, _, _ := rbServer(t)
|
||||
rb.err = errors.New("pct reboot 9201: exit status 255")
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if s.inFlight.Busy() != "" || s.rebooting(9201) {
|
||||
t.Fatal("the gate or the flag outlived a failed reboot")
|
||||
}
|
||||
}
|
||||
|
||||
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
|
||||
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
|
||||
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
|
||||
s, ex, rb, ctl, _, _ := rbServer(t)
|
||||
var started, restarted int
|
||||
rb.after = func() {
|
||||
ctl.guests[0].Status = "stopped"
|
||||
s.GuestPowerTick(context.Background())
|
||||
ctl.guests[0].Status = "running"
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "exited"
|
||||
ex.mu.Unlock()
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
started, restarted = len(ctl.started), ex.count(9201)
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "running"
|
||||
ex.mu.Unlock()
|
||||
}
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if started != 0 || restarted != 0 {
|
||||
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
|
||||
}
|
||||
}
|
||||
|
||||
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
|
||||
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
|
||||
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
|
||||
s, _, _, _, clk, _ := rbServer(t)
|
||||
read := func() string {
|
||||
w := httptest.NewRecorder()
|
||||
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
|
||||
var body struct {
|
||||
Data struct {
|
||||
At string `json:"at"`
|
||||
} `json:"data"`
|
||||
At string `json:"at"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("body %q: %v", w.Body.String(), err)
|
||||
}
|
||||
if body.Data.At != "" {
|
||||
return body.Data.At
|
||||
}
|
||||
return body.At
|
||||
}
|
||||
if got := read(); got != "" {
|
||||
t.Fatalf("no reboot yet, got %q", got)
|
||||
}
|
||||
at := clk.t
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
clk.t = at.Add(23 * time.Hour)
|
||||
if got := read(); got != at.UTC().Format(time.RFC3339) {
|
||||
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
|
||||
}
|
||||
clk.t = at.Add(25 * time.Hour)
|
||||
if got := read(); got != "" {
|
||||
t.Fatalf("after 25 h the line must be gone, got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -411,6 +411,20 @@ type Server struct {
|
||||
guestsDir string
|
||||
ctrlSup controllerSupervisor
|
||||
|
||||
// 2026-10-29: the operator's actions from the hub's reply (opactions.go). opHealthPoll/opHealthWait bound the wait
|
||||
// for a restarted controller's healthy answer (tests shorten them; zero → the defaults).
|
||||
opActs opActionState
|
||||
opTrigger func()
|
||||
opHealthPoll time.Duration
|
||||
opHealthWait time.Duration
|
||||
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
|
||||
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
|
||||
opRebooter interface {
|
||||
RebootGuest(ctx context.Context, vmid int) error
|
||||
}
|
||||
rebootMu sync.Mutex
|
||||
rebootInFlight map[int]bool
|
||||
|
||||
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
|
||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||
netVerifyMu sync.Mutex
|
||||
@@ -596,6 +610,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
||||
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
||||
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
||||
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
|
||||
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
|
||||
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
||||
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
||||
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
||||
|
||||
@@ -256,8 +256,10 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
if callers != 5 {
|
||||
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
|
||||
// 2026-10-29: +2 deliberate callers — the operator's restart_controller (opactions.go) and R-1344's unresponsive count
|
||||
// (controllersupervisor.go observeControllerHealth). Both through s.staleLock, the one helper.
|
||||
if callers != 7 {
|
||||
t.Errorf("expected the seven known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup, the operator restart, the unresponsive count), found %d", callers)
|
||||
}
|
||||
b, _ := os.ReadFile("stalelock.go")
|
||||
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-1344 (operator brief 2026-10-29, Part 2): a controller that RUNS but whose Docker health check reads „unhealthy" on
|
||||
// controllerUnresponsiveConfirm consecutive sweeps is reported unresponsive (the stanza; the hub raises
|
||||
// controller_unresponsive). ALARM ONLY: it is never restarted for it until the measurement licenses that.
|
||||
// RED-PROOF: seen failing before the stanza carried it.
|
||||
|
||||
func unrespServer(t *testing.T, health string) (*Server, *opExec, *fakeGuestPowerCtl) {
|
||||
t.Helper()
|
||||
s, ex, ctl, _, _, _ := opServer(t, "running")
|
||||
ex.health = health
|
||||
return s, ex, ctl
|
||||
}
|
||||
|
||||
func sweep(s *Server, n int) {
|
||||
for i := 0; i < n; i++ {
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSupervisor_UnresponsiveIsAlarmedNeverRestarted(t *testing.T) {
|
||||
s, ex, _ := unrespServer(t, "unhealthy")
|
||||
sweep(s, controllerUnresponsiveConfirm-1)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
|
||||
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||
}
|
||||
sweep(s, 1)
|
||||
g := s.ControllerSupervisorStatus(context.Background()).Guests
|
||||
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
|
||||
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
|
||||
}
|
||||
since := g[0].UnresponsiveSince
|
||||
sweep(s, 10)
|
||||
if n := ex.count(9201); n != 0 {
|
||||
t.Fatalf("an unresponsive controller was RESTARTED (%d) — the restart waits for the measurement", n)
|
||||
}
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
|
||||
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
|
||||
}
|
||||
// it answers again
|
||||
ex.health = "healthy"
|
||||
sweep(s, 1)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
|
||||
t.Fatalf("after it answered: %+v (unresponsive must clear; since stays as the record)", g[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Slow but answering (healthy / starting), a swap, a vzdump or a parked guest never counts.
|
||||
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
setup func(s *Server, ex *opExec, ctl *fakeGuestPowerCtl)
|
||||
}{
|
||||
{"healthy", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
|
||||
{"starting", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
|
||||
{"swap", func(s *Server, _ *opExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
|
||||
{"vzdump", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
|
||||
{"locked", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, ctl := unrespServer(t, "unhealthy")
|
||||
tc.setup(s, ex, ctl)
|
||||
sweep(s, controllerUnresponsiveConfirm+2)
|
||||
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
|
||||
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -24,10 +24,10 @@
|
||||
"readers": [
|
||||
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
|
||||
],
|
||||
"recorded": "2026-10-27",
|
||||
"recorded_by": "CC 2026-10-27: day0_agent 0.167.0 (operator vouch 2026-10-27, read from the hub: version + agent and bundle sha256 match the release); earlier 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"recorded": "2026-10-29",
|
||||
"recorded_by": "CC 2026-10-29: day0_agent 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
|
||||
"pinned": {
|
||||
"hub_min_agent": "0.158.1",
|
||||
"day0_agent": "0.167.0"
|
||||
"day0_agent": "0.169.0"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user