Compare commits

..
7 Commits
Author SHA1 Message Date
admin ca15a31690 D65 = A: reboot_guest joins the agent's closed operator-action list — guards (busy controller file, heavy gate, jobs, lock, parked, 1/hour, the brake), result with both times, GET /guest/operator-reboot for the household line; one read-only sudo vector
gates / gates (push) Successful in 33s
2026-10-29 16:04:19 +01:00
admin bdea4694a4 REPORT + CONTEXT: v0.169.0 released and vouched; restart_controller proven on the release builds
gates / gates (push) Successful in 35s
2026-10-29 15:11:11 +01:00
admin db25eb0e71 v0.169.0 released and vouched (14:06:09 CET): CHANGELOG; D50 pinned.day0_agent = 0.169.0
gates / gates (push) Successful in 42s
2026-10-29 14:09:04 +01:00
admin 637c92c762 Operator actions for the agent: restart_controller from the hub (closed list, the supervisor's guards, the brake counts operator restarts); R-1344: a running-but-unhealthy controller raises unresponsive — alarm only; tests red first
gates / gates (push) Successful in 36s
2026-10-29 12:30:36 +01:00
admin 63e63ee8bb v0.168.0 released and vouched: CONTEXT, REPORT
gates / gates (push) Successful in 37s
2026-10-29 10:48:37 +01:00
admin 6b0b093199 D50: pinned.day0_agent = 0.168.0 (vouched by the operator 2026-10-29 10:21:56 CET); CHANGELOG notes the vouch
gates / gates (push) Successful in 36s
2026-10-29 10:33:58 +01:00
admin f4e63a55da v0.168.0: CHANGELOG — released (agent 1e70196c…, bundle 3415c034…), committed on demo-felhom, Tester 1, demo-hp; not vouched yet
gates / gates (push) Successful in 37s
2026-10-29 10:15:55 +01:00
20 changed files with 1478 additions and 30 deletions

No files matched your search

+53 -2
View File
@@ -1,4 +1,55 @@
## Unreleased (2026-10-28 night) — R-528: GET /guest/oom-kills, the kernel's OOM counter of each of the guest's containers, read on the host — commit only; the next agent release (needs the operator's vouch)
## Unreleased — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
a racing start would fail the reboot's own start half.
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
restarted your box at <time>".
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
answers on the next report, `operator_action_results: [{id, outcome, message}]` (`hub.HostReport`), which it requests
at once through the cap-1 report trigger. CLOSED list: `restart_controller <vmid>` only — anything else refused, nothing
called. The act is the supervisor's (`systemctl restart felhom-controller-bootstrap.service` in the guest, existing
grants — no sudoers change, no config bundle) with its guards: provisioned guest, guest running, the guest list and the
vzdump state readable, not parked, no swap, not locked, no vzdump. The thrash brake counts operator restarts with the
supervisor's: 3 in 15 minutes → the press is refused, and presses stay refused for 30 minutes. The result says how long
the controller took to answer (Docker's health check: healthy), or `failed` when it did not within 3 minutes. One
`audit: operator action` line per decision. Works with the controller stopped. Tests `TestOpActions_*` (closed list,
unknown/bad arg refused, restart + answer time + report trigger + once per id + re-send until dropped, stopped
controller, every guard, the brake across supervisor and operator restarts, no healthy answer = failed) and
`TestMainWiresTheAgentOperatorActions` (AST) — each seen failing by a mutation.
- **R-1344 — a controller that RUNS but does not answer: ALARM ONLY.** Each sweep also reads Docker's health status for a
running controller (the swap's `docker inspect -f` grant); 6 consecutive „unhealthy" sweeps (3 minutes, on top of
Docker's own ~90 s) start an episode: one ERROR line, the stanza's `unresponsive` / `unresponsive_since` (the hub raises
`controller_unresponsive`, operator-only). NOT restarted for it: the brief allows the restart only after a measurement
shows a busy night never reads like this, and that measurement runs through tonight (felhom.eu R-1344). Not counted
during a swap, a parked or locked guest, or a vzdump; healthy / starting / no health check / an unreadable status reset
the count. Tests `TestSupervisor_Unresponsive*` (red by mutation: the vzdump guard, the threshold).
## v0.168.0 (2026-10-29) — R-528: the host's per-container OOM counter (`GET /guest/oom-kills`); R-1254: a lost or late escrow answer is asked again before it is worded — RELEASED (release-agent.sh from a clean clone of `91bd96c`; agent sha256 1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca, bundle 3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8; no configs/ change since v0.167.0 → binary-only signed updates to demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 10:21:56 CET**; Day-0 pin set (D50)
### R-528
- **Why:** inside a Felhom LXC guest Docker can leave `State.OOMKilled` false and send no `oom` event after a
memory-limit kill (measured 2026-09-15 on 9202 and 2026-09-16 on a fresh install), so the controller's OOM line
@@ -11,7 +62,7 @@
count is per container run. Tests `TestR528_*` (a fake cgroup tree; red-proved: a glob over every guest leaks another
guest's container; parsing `oom` instead of `oom_kill` reads the wrong number).
## Unreleased (2026-10-28 night) — R-1254 (the ceremony half, review round 2 HUNT-3 H2): the agent asks the hub which package it holds before it words a failed upload — commit only; the next agent release (needs the operator's vouch)
### R-1254 (the ceremony half, review round 2 HUNT-3 H2)
- **The defect:** the claim's (R-978) and the confirm's (R-1151) package upload can lose its answer, or answer late
(30 s), AFTER the hub stored the new package. The agent then answered 502 „the previous code still works" — false: the
+13
View File
@@ -1,5 +1,18 @@
# CONTEXT — felhom-agent working state
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
> R-528 proven on the release (Tester 1: one `app_oom` „seen by docker,cgroup", host counter 1). R-1254: the happy path ran
> live (a ceremony on 0.168.0 + controller 0.347.0); the lost-answer branch has tests only.
> **2026-10-28 night (unattended) — R-528 + R-1254 on `main` (`6f933d6`, `5ddd124`), NOT released (the next agent
> release, the operator's vouch).** `GET /guest/oom-kills` (`internal/localapi/guest_oom.go`): the host-side `oom_kill`
> of each Docker container of the token's guest (`Server.cgroupRoot` is the test seam); proven on Tester 1 with a test
+22 -23
View File
@@ -1,30 +1,29 @@
# REPORT — two agent fixes on main, NOT released (night 2026-10-28 → 29, unattended)
# REPORT — v0.169.0 released and vouched: „Restart controller" from the hub (2026-10-29, attended)
**Plain words first.** Two changes are on main and tested. Neither is released: an agent release needs the operator's
vouch, and the night brief allowed no release. Both go on the morning list.
**Plain words first.** The operator can now restart a box's controller from the hub's host page. The agent does it, so it
works when the controller is stopped or stuck. It uses the supervisor's own guards and brake: 3 restarts in 15 minutes,
then presses are refused for 30 minutes. A controller that runs but does not answer now raises an operator alarm
(`controller_unresponsive`); it is not restarted by itself yet — that waits for the night's measurement (R-1344).
## What changed
## Commits
- **R-528 — the OOM counter (`6f933d6`, CI run id 2473 / #284 green).** New local-API route `GET /guest/oom-kills`. It
reads the kernel's `oom_kill` count of each Docker container of the token's own guest, on the host, from
`/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events` (mode 0444, readable by the non-root
agent user). Docker inside an LXC guest does not see these kills (`OOMKilled` stays false). The controller reads the
route every 30 s (controller `eed23e7`). Tests `TestR528_*`, red first by mutation (another guest's container served;
`oom` parsed for `oom_kill`).
- **R-1254 — the escrow ceremony half (`5ddd124`, CI run id 2471 / #283 green).** After an escrow upload that fails, the
agent asks the hub which package it holds (the identity blob compared): the new one → success; another one → 502 and
„the previous code still works"; no answer → 504, which promises nothing. Tests `TestR1254_*`, red first.
| Commit | CI (job id) | What |
|---|---|---|
| `637c92c` | 2504, 2507 green | The agent's closed list: `restart_controller <vmid>` on the host-report reply (`ControlEnvelope.OperatorActions`), the result on the next host report (`OperatorActionResults`). Guards: provisioned, running, parked, swap, lock, vzdump. The brake counts operator restarts with the supervisor's (`opBrakeUntil`). One audit line, one result per id, re-sent until the hub records it; a report is asked for at once. R-1344: 6 unhealthy sweeps → `unresponsive_since` in the stanza. Tests in `internal/localapi/opactions_test.go`, `supervisor_unresponsive_test.go`, `opactions_wiring_test.go`, each seen failing by its mutation. |
| `db25eb0` | 2513 green | v0.169.0 CHANGELOG; `pinned.day0_agent` = 0.169.0 (D50 pin), after the vouch. |
## Live proof
## Release
- R-528, 2026-10-28 23:27 UTC on Tester 1 with a test build of this agent (put back after: 0.167.0, sha256 `28bdd282…`).
A Radicale capped at 128 MB, a memory hog inside it killed: the host counter 0 → 1, one `app_oom` „seen by
docker,cgroup", one mail to the operator with `"source":"docker,cgroup"`. Evidence: felhom.eu
`documentation/audits/night-2026-10-28/r528-oom/`.
- R-1254: no live proof. It needs a hub answer lost on purpose; that is a hub change, and no hub deploy runs at night.
- `release-agent.sh 0.169.0` from a clean clone: sha256 `fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61`,
bundle `3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b`, tag `v0.169.0`, verified by download.
- Signed `agent_update` to the three test boxes; each committed.
- Vouched by the operator 14:06:09 CET (the page answered 504; the save went through — R-1346).
## Next
## Live proofs (Tester 1, through the hub endpoint the button posts, CC's `cc-hosts` token)
The agent release (the operator's vouch), then `pinned.day0_agent` in `scripts/retention-policy.json` (D50).
The controller needs no MinAgent change for R-528: an old agent answers 404, and the controller then waits 10 minutes
before it asks again.
- 13:01–13:02 CET on rc builds: press 1 → the controller answered after 9 s; press 2 with the controller stopped by hand →
restarted 5 s after the stop; press 3 done; press 4 refused („3 restarts in 15 minutes").
- 14:49 CET on the release builds (agent 0.169.0, controller 0.348.0, hub 0.176.1): one press → „done — restarted; the
controller answered after 9 s", on the hub 16 s after the press; the box showed the controller „Up 12 seconds".
Evidence: felhom.eu `documentation/audits/release-2026-10-29b/` (`p1-restart/`, `final/`).
+1
View File
@@ -89,6 +89,7 @@
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
| `Server.ControllerSupervisorTick` + `ControllerParkedMarker` | internal/localapi/controllersupervisor.go | `ControllerSupervisorTick(ctx)` | R-523: restart a provisioned guest's not-running controller via its bootstrap unit | Two-sweep confirm; honours swapInFlight, the host-side park marker, guest lock + vzdump; 3 restarts/15 min → 30 min pause; record rides the report as `controller_supervisor` (the hub mints the events — the agent has no event channel) |
| `Server.OnEnvelope` / `OperatorActionResults` (operator actions) | internal/localapi/opactions.go | `OnEnvelope(ctx, env)`, `OperatorActionResults() []hub.OperatorActionResult` | 2026-10-29: the operator's actions for THIS agent from the host-report reply — a CLOSED list (`restart_controller` only, `TestOpActions_AgentClosedList`), once per id, the result re-sent until the reply stops listing it, the report requested at once (`SetOperatorActionTrigger`). The restart is the supervisor's act with its guards; the 3-in-15-min brake counts operator restarts too (30-min refusal, `opBrakeUntil`). A new verb is an operator decision AND a hub `store.hostOperatorActionNames` entry in the same release |
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
### Proxmox client / hub / PBS / provisioning
+11
View File
@@ -1267,6 +1267,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil {
// 2026-10-29 (operator brief, Part 1): the operator's actions for THIS agent ride the host report's reply — the
// closed list (localapi/opactions.go). The server joins the observer chain; its results ride the next report,
// which it requests at once through the same cap-1 trigger the storage watchdog uses.
localSrv.SetOperatorActionTrigger(func() {
select {
case storageTrigger <- struct{}{}:
default:
}
})
collector.SetOperatorActionResults(localSrv.OperatorActionResults)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner, localSrv))
backupCapRefusal = localSrv.BackupCapRefusal // R-973
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
+51
View File
@@ -0,0 +1,51 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// 2026-10-29 (operator brief, Part 1): the agent's operator actions are unit-green in localapi; main.go must also put
// the server in the report loop's observer chain, give the collector its results and give the server the report
// trigger — an inert seam otherwise (the „seam built but never wired" class). Asserted from the AST.
// RED-PROOF: seen failing before the wiring existed (all three false).
func TestMainWiresTheAgentOperatorActions(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var observed, results, trigger bool
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
fn, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
switch fn.Sel.Name {
case "MultiObserver":
for _, a := range call.Args {
if id, ok := a.(*ast.Ident); ok && id.Name == "localSrv" {
observed = true
}
}
case "SetOperatorActionResults":
if len(call.Args) == 1 {
if sel, ok := call.Args[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "OperatorActionResults" {
results = true
}
}
case "SetOperatorActionTrigger":
trigger = true
}
return true
})
if !observed || !results || !trigger {
t.Fatalf("operator actions wiring: observer=%v results=%v trigger=%v — all three must be wired in main.go", observed, results, trigger)
}
}
+3
View File
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
+1
View File
@@ -151,6 +151,7 @@ var manifest = []Capability{
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
+10
View File
@@ -128,6 +128,7 @@ type Collector struct {
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
opResults func() []OperatorActionResult // 2026-10-29: the operator actions' answers (nil → none)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
lineTest LineTestReporter // R-1247: weekly upload line test (nil → stanza omitted)
@@ -235,6 +236,12 @@ type ControllerSupervisorReporter interface {
ControllerSupervisorStatus(ctx context.Context) *ControllerSupervisorStatus
}
// SetOperatorActionResults wires the source of the operator actions' answers (2026-10-29; nil → none sent).
func (c *Collector) SetOperatorActionResults(f func() []OperatorActionResult) *Collector {
c.opResults = f
return c
}
// SetControllerSupervisorReporter wires the R-523 controller supervisor as a report source (nil-safe).
func (c *Collector) SetControllerSupervisorReporter(r ControllerSupervisorReporter) *Collector {
c.ctrlSup = r
@@ -432,6 +439,9 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.ctrlSup != nil {
report.ControllerSupervisor = c.ctrlSup.ControllerSupervisorStatus(ctx)
}
if c.opResults != nil {
report.OperatorActionResults = c.opResults()
}
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
if c.guestNet != nil {
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
+26
View File
@@ -172,6 +172,24 @@ type HostReport struct {
// byte-stable. The hub parser is pinned by hub/internal/monitor/controller_supervisor_test.go
// against the JSON TestControllerSupervisorStanza_WireShape pins here.
ControllerSupervisor *ControllerSupervisorStatus `json:"controller_supervisor,omitempty"`
// OperatorActionResults (2026-10-29, the operator brief's Part 1): the answers to the operator actions the hub's
// replies listed — the controller's shape (`04` §6.1). Re-sent until the reply stops listing the id. omitempty: absent
// when there is nothing to answer, so the golden stays byte-stable.
OperatorActionResults []OperatorActionResult `json:"operator_action_results,omitempty"`
}
// OperatorAction is ONE entry of the reply's operator_actions (hub store.OperatorActionDirective, field-exact).
type OperatorAction struct {
ID int64 `json:"id"`
Action string `json:"action"`
Arg string `json:"arg,omitempty"`
}
// OperatorActionResult is one answer (hub api.hostOperatorActionResult, field-exact): outcome done | refused | failed.
type OperatorActionResult struct {
ID int64 `json:"id"`
Outcome string `json:"outcome"`
Message string `json:"message"`
}
// ControllerSupervisorStatus is the R-523 stanza. Carries no secret.
@@ -195,6 +213,11 @@ type ControllerSupervisorGuest struct {
Restarts24h int `json:"restarts_24h"`
SlowCrashloop bool `json:"slow_crashloop"`
SlowCrashloopSince string `json:"slow_crashloop_since,omitempty"` // RFC3339
// R-1344 (2026-10-29) — the controller RUNS but Docker's health check has read „unhealthy" on several consecutive
// sweeps. ALARM ONLY: the agent does not restart it for this (the measurement must first show a busy night never
// looks like it). UnresponsiveSince is the episode's start; the hub keys on it MOVING (controller_unresponsive).
Unresponsive bool `json:"unresponsive"`
UnresponsiveSince string `json:"unresponsive_since,omitempty"` // RFC3339
}
// PBSDRStatus is the per-heartbeat PBS-DR-tier bridge state (slice 2). States:
@@ -631,6 +654,9 @@ type ControlEnvelope struct {
// NEXT heartbeat carries it in log_tail (the report-channel log_tail_requests mirror).
// Absent/false on an old hub → nothing happens.
LogTailRequested bool `json:"log_tail_requested"`
// OperatorActions (2026-10-29): the operator's pending actions for THIS host's agent — a CLOSED list on both sides
// (`04` §6.1). Absent on an older hub.
OperatorActions []OperatorAction `json:"operator_actions,omitempty"`
}
// LogTail is the heartbeat's on-demand agent log tail (see HostReport.LogTail).
+71
View File
@@ -83,6 +83,14 @@ const (
// controllerSlowCounterFile holds the 24-hour restart times and the last raise, per guest, beside
// the parked marker.
controllerSlowCounterFile = "controller-restarts-24h.json"
// R-1344 (operator brief 2026-10-29, Part 2) — a controller that RUNS but does not answer. Docker's own HEALTHCHECK
// (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) reads „unhealthy" only after 3 failed probes (~90 s);
// controllerUnresponsiveConfirm more sweeps (3 minutes) make it an episode: logged, carried by the stanza, mailed by
// the hub (controller_unresponsive). ALARM ONLY — it is NOT restarted for this until the measurement (R-1344) shows a
// busy night (backups, updates) never reads unhealthy that long. Not counted during a swap, a parked guest, a locked
// guest or a vzdump.
controllerUnresponsiveConfirm = 6
)
// controllerSupState is one guest's supervisor record. In-memory on purpose (the guest-power
@@ -103,6 +111,14 @@ type controllerSupState struct {
// on the other hand, would hide exactly the box it exists for (one whose agent restarts too).
restarts24h []time.Time
slowCrashloopSince time.Time // the last raise; kept after it ages out, the hub keys on it MOVING
// opBrakeUntil (2026-10-29): an operator restart refused by the brake refuses every press until then (30 min).
opBrakeUntil time.Time
// R-1344: consecutive running-but-unhealthy sweeps, and the current episode (since kept after it ends).
unhealthySeen int
unresponsive bool
unresponsiveSince time.Time
}
type controllerSupervisor struct {
@@ -238,6 +254,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
if !s.provisionedGuest(g.VMID) {
continue
}
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
s.resetNotRunning(g.VMID)
continue
}
evaluated++
if !s.superviseOneController(ctx, g.VMID, g.Status) {
down++
@@ -293,6 +313,7 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
if running {
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.observeControllerHealth(ctx, vmid, parked)
return true
}
st.notRunningSeen++
@@ -392,6 +413,52 @@ func (s *Server) superviseOneController(ctx context.Context, vmid int, guestStat
return false
}
// observeControllerHealth is R-1344's count for a RUNNING controller: Docker's health status, one read per sweep over
// the swap's `docker inspect -f` grant. Anything but „unhealthy" — healthy, starting, no health check, a read that
// fails — resets the count; so do a swap, a parked guest, a locked guest and a vzdump (each is someone else's moment).
func (s *Server) observeControllerHealth(ctx context.Context, vmid int, parked bool) {
s.swapMu.Lock()
swapping := s.swapInFlight[vmid]
s.swapMu.Unlock()
unhealthy := false
if !parked && !swapping {
if out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer); err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
unhealthy = len(f) == 2 && f[0] == "true" && f[1] == "unhealthy"
}
}
if unhealthy {
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil || lock != "" {
unhealthy = false
} else if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
unhealthy = false
}
}
now := s.clock()
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if !unhealthy {
was := st.unresponsive
st.unhealthySeen, st.unresponsive = 0, false
s.ctrlSup.mu.Unlock()
if was {
s.logger.Info("controller-supervisor: the controller answers its health check again", "vmid", vmid)
}
return
}
st.unhealthySeen++
seen := st.unhealthySeen
start := seen >= controllerUnresponsiveConfirm && !st.unresponsive
if start {
st.unresponsive, st.unresponsiveSince = true, now
}
s.ctrlSup.mu.Unlock()
if start {
s.logger.Error("controller-supervisor: the controller RUNS but does not answer its health check — raising controller_unresponsive; NOT restarting it (R-1344: alarm only until measured)",
"vmid", vmid, "unhealthy_sweeps", seen, "interval", controllerSupervisorInterval.String())
}
}
func (s *Server) resetNotRunning(vmid int) {
s.ctrlSup.mu.Lock()
defer s.ctrlSup.mu.Unlock()
@@ -446,6 +513,10 @@ func (s *Server) ControllerSupervisorStatus(_ context.Context) *hub.ControllerSu
g.SlowCrashloopSince = st.slowCrashloopSince.UTC().Format(time.RFC3339)
g.SlowCrashloop = now.Sub(st.slowCrashloopSince) < controllerSlowCrashloopWindow
}
g.Unresponsive = st.unresponsive
if !st.unresponsiveSince.IsZero() {
g.UnresponsiveSince = st.unresponsiveSince.UTC().Format(time.RFC3339)
}
out.Guests = append(out.Guests, g)
}
sort.Slice(out.Guests, func(i, j int) bool { return out.Guests[i].VMID < out.Guests[j].VMID })
+5
View File
@@ -152,6 +152,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
return
}
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
// would fail the reboot's own start half).
if s.rebooting(g.VMID) {
return
}
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
if err != nil {
+258
View File
@@ -0,0 +1,258 @@
package localapi
import (
"context"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// The operator's actions for THIS agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).
//
// THE DOOR IS THE HOST REPORT'S REPLY, the controller's shape: the operator presses a button on the hub's host page, the
// hub stores a row and pokes the box (the poke's out-of-cycle report gets the reply in seconds; without the tunnel the
// 15-minute cycle carries it), the reply lists `operator_actions: [{id, action, arg}]`, this agent acts ONCE per id, and
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
// The result is re-sent until the reply stops listing the id.
//
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
//
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
// the guest over the existing grants, with the supervisor's guards — not during a controller swap, not parked, not on a
// stopped, locked or vzdump-busy guest, not when the guest list or the vzdump state cannot be read — and the thrash brake
// counts these restarts too: 3 restarts (the supervisor's or the operator's) inside 15 minutes refuse the press, and
// presses stay refused for 30 minutes. It works with the controller stopped or stuck: the agent does it, not the
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
const (
// OpRestartController is the agent's first operator verb.
OpRestartController = "restart_controller"
opOutcomeDone = "done"
opOutcomeRefused = "refused"
opOutcomeFailed = "failed"
// opBrakeRefusal is how long presses stay refused after the brake refused one.
opBrakeRefusal = 30 * time.Minute
// opForgetAfter drops a started id the reply no longer lists.
opForgetAfter = 24 * time.Hour
defaultOpHealthPoll = 2 * time.Second
// defaultOpHealthWait covers the bootstrap's pull-less rm/run (seconds), Docker's start period (10 s) and its first
// probe (30 s interval) with room: the controller's own HEALTHCHECK is the answer.
defaultOpHealthWait = 3 * time.Minute
// opHealthTemplate reads running + health over the swap's `docker inspect -f` grant.
opHealthTemplate = "{{.State.Running}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}"
)
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
// OperatorActionNames returns the closed list (sorted copy).
func OperatorActionNames() []string {
out := append([]string(nil), operatorActionNames...)
sort.Strings(out)
return out
}
type opActionState struct {
mu sync.Mutex
started map[int64]time.Time
results map[int64]hub.OperatorActionResult
}
// SetOperatorActionTrigger wires the report loop's out-of-band trigger (the result goes at once, not in 15 minutes).
func (s *Server) SetOperatorActionTrigger(f func()) { s.opTrigger = f }
// OnEnvelope is the report loop's hook (hub.EnvelopeObserver): it starts each newly listed action once and forgets
// the ids the reply no longer lists (the hub has their results).
func (s *Server) OnEnvelope(_ context.Context, env *hub.ControlEnvelope) {
if env == nil {
return
}
listed := map[int64]bool{}
var start []hub.OperatorAction
now := time.Now()
s.opActs.mu.Lock()
if s.opActs.started == nil {
s.opActs.started = map[int64]time.Time{}
s.opActs.results = map[int64]hub.OperatorActionResult{}
}
for _, a := range env.OperatorActions {
listed[a.ID] = true
if _, ok := s.opActs.started[a.ID]; ok {
continue
}
s.opActs.started[a.ID] = now
start = append(start, a)
}
for id, at := range s.opActs.started {
if listed[id] {
continue
}
if _, answered := s.opActs.results[id]; answered || now.Sub(at) > opForgetAfter {
delete(s.opActs.started, id)
delete(s.opActs.results, id)
}
}
s.opActs.mu.Unlock()
for _, a := range start {
go s.runOperatorAction(a) // a restart waits for the health answer — never block the report loop
}
}
// OperatorActionResults is the report's answer list, oldest id first.
func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
s.opActs.mu.Lock()
defer s.opActs.mu.Unlock()
out := make([]hub.OperatorActionResult, 0, len(s.opActs.results))
for _, r := range s.opActs.results {
out = append(out, r)
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out
}
func (s *Server) runOperatorAction(a hub.OperatorAction) {
limit := s.opWait() + time.Minute
if a.Action == OpRebootGuest {
limit += opRebootTimeout
}
ctx, cancel := context.WithTimeout(context.Background(), limit)
defer cancel()
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
switch a.Action {
case OpRestartController:
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
case OpRebootGuest:
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
}
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
"outcome", outcome, "message", msg)
s.opActs.mu.Lock()
if _, still := s.opActs.started[a.ID]; still {
s.opActs.results[a.ID] = hub.OperatorActionResult{ID: a.ID, Outcome: outcome, Message: msg}
}
s.opActs.mu.Unlock()
if s.opTrigger != nil {
s.opTrigger()
}
}
func (s *Server) opPoll() time.Duration {
if s.opHealthPoll > 0 {
return s.opHealthPoll
}
return defaultOpHealthPoll
}
func (s *Server) opWait() time.Duration {
if s.opHealthWait > 0 {
return s.opHealthWait
}
return defaultOpHealthWait
}
// operatorRestartController is the press: the supervisor's guards, the brake, the restart, then the wait for the
// controller's healthy answer. Returns the outcome and the sentence the host page shows.
func (s *Server) operatorRestartController(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
if s.staleLock == nil || s.guestExec == nil {
return opOutcomeRefused, "refused: the controller supervisor is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
s.swapMu.Lock()
swapping := s.swapInFlight[vmid]
s.swapMu.Unlock()
if swapping {
return opOutcomeRefused, "refused: a controller swap (update) is in flight — it owns the controller"
}
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
if busy, berr := s.staleLock.BackupRunning(ctx, vmid); berr != nil || busy {
return opOutcomeRefused, "refused: a whole-guest backup (vzdump) may be in flight for the guest"
}
// The brake: every restart counts — the supervisor's and the operator's.
now := s.clock()
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
reason := fmt.Sprintf("the operator via the hub (action #%d)", id)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = reason
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.logger.Warn("controller-supervisor: RESTARTING the controller on the operator's press", "vmid", vmid, "action_id", id, "unit", bootstrapUnit)
began := time.Now()
if _, err := s.guestExec.GuestExec(ctx, vmid, "systemctl", "restart", bootstrapUnit); err != nil {
return opOutcomeFailed, fmt.Sprintf("the restart of guest %d's controller failed: %v", vmid, err)
}
for time.Since(began) < s.opWait() {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
secs := int(time.Since(began).Round(time.Second) / time.Second)
return opOutcomeDone, fmt.Sprintf("restarted; the controller answered after %d s (Docker's health check: healthy)", secs)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, "restarted, but the wait for its answer was cut"
case <-time.After(s.opPoll()):
}
}
return opOutcomeFailed, fmt.Sprintf("restarted, but no healthy answer within %s — look at the box", s.opWait().Round(time.Second))
}
func orDash(s string) string {
if s == "" {
return "not listed"
}
return s
}
+222
View File
@@ -0,0 +1,222 @@
package localapi
import (
"context"
"errors"
"os"
"path/filepath"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// Operator brief 2026-10-29, Part 1: „Restart controller" from the hub. The agent's closed list holds one verb; the act
// is the supervisor's own restart with the supervisor's guards, and the brake counts operator restarts too.
// opExec answers the supervisor's fake plus the health read the operator restart waits on.
type opExec struct {
*supExec
health string // what {{.State.Health.Status}} reads after the restart
}
func (f *opExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) >= 5 && args[0] == "docker" && args[1] == "inspect" && args[3] == opHealthTemplate {
f.mu.Lock()
st := f.status[vmid]
f.mu.Unlock()
if st == "" {
return "", errors.New("pct exec: exit status 1: Error: No such object: felhom-controller")
}
return map[bool]string{true: "true", false: "false"}[st == "running"] + "|" + f.health + "\n", nil
}
return f.supExec.GuestExec(ctx, vmid, args...)
}
func opServer(t *testing.T, status string) (*Server, *opExec, *fakeGuestPowerCtl, *supClock, string, *atomic.Int32) {
t.Helper()
ex := &opExec{supExec: &supExec{status: map[int]string{9201: status}, onRestart: "running"}, health: "healthy"}
ctl := runningGuest(9201)
s, clk, dir := supServer(t, nil, ctl, 9201)
s.guestExec = ex
s.opHealthPoll = time.Millisecond
s.opHealthWait = 2 * time.Second
var triggered atomic.Int32
s.SetOperatorActionTrigger(func() { triggered.Add(1) })
return s, ex, ctl, clk, dir, &triggered
}
func env(acts ...hub.OperatorAction) *hub.ControlEnvelope {
return &hub.ControlEnvelope{OperatorActions: acts}
}
// waitResult waits for the action's result (the act runs in the background).
func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for time.Now().Before(deadline) {
for _, r := range s.OperatorActionResults() {
if r.ID == id {
return r
}
}
time.Sleep(2 * time.Millisecond)
}
t.Fatalf("no result for #%d", id)
return hub.OperatorActionResult{}
}
// RED-PROOF: seen failing before the list existed (build failure, then the names).
func TestOpActions_AgentClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
}
}
// Anything outside the list — and a bad argument — is refused and NOTHING is called.
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
s.OnEnvelope(context.Background(), env(
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
))
for _, id := range []int64{1, 2, 3} {
if r := waitResult(t, s, id); r.Outcome != "refused" {
t.Errorf("#%d: %+v, want refused", id, r)
}
}
if n := ex.count(9201); n != 0 {
t.Fatalf("a refused action restarted the controller (%d)", n)
}
}
// The press restarts ONCE, says how long the controller took to answer, asks for a report at once, and re-sends the
// result until the hub stops listing the id. RED-PROOF: drop the GuestExec restart → restarts 0 → FAILS.
func TestOpActions_RestartReportsTheAnswerTime(t *testing.T) {
s, ex, _, _, _, trig := opServer(t, "running")
a := hub.OperatorAction{ID: 7, Action: "restart_controller", Arg: "9201"}
s.OnEnvelope(context.Background(), env(a))
r := waitResult(t, s, 7)
if r.Outcome != "done" || !strings.Contains(r.Message, "answered after") {
t.Fatalf("result %+v", r)
}
if n := ex.count(9201); n != 1 {
t.Fatalf("restarts = %d, want 1", n)
}
if trig.Load() == 0 {
t.Fatal("the agent did not ask for a report — the result would wait 15 minutes")
}
s.OnEnvelope(context.Background(), env(a)) // the hub still lists it: re-send, never act again
time.Sleep(100 * time.Millisecond) // a wrong second act runs in the background — give it time to show
if n := ex.count(9201); n != 1 {
t.Fatalf("a re-listed id acted again (%d)", n)
}
if len(s.OperatorActionResults()) != 1 {
t.Fatal("the result must be re-sent while listed")
}
s.OnEnvelope(context.Background(), env()) // the hub recorded it
if left := s.OperatorActionResults(); len(left) != 0 {
t.Fatalf("a recorded result is still sent: %+v", left)
}
}
// The point of the agent doing it: a STOPPED controller is restarted too (the controller cannot restart itself).
func TestOpActions_WorksWithTheControllerStopped(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "exited")
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 8, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 8); r.Outcome != "done" || ex.count(9201) != 1 {
t.Fatalf("result %+v, restarts %d", r, ex.count(9201))
}
}
// Each supervisor guard refuses the press, with the reason. RED-PROOF: drop the lock check → the locked case
// restarts → FAILS (each guard seen failing by its own mutation).
func TestOpActions_GuardsRefuse(t *testing.T) {
cases := []struct {
name, want string
setup func(s *Server, ctl *fakeGuestPowerCtl, dir string)
}{
{"parked", "parked", func(_ *Server, _ *fakeGuestPowerCtl, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *fakeGuestPowerCtl, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.locks[9201] = "backup" }},
{"vzdump", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupRun = map[int]bool{9201: true} }},
{"vzdump unknown", "backup", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.backupErr = errors.New("tasks unreadable") }},
{"guest stopped", "not running", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guests[0].Status = "stopped" }},
{"guest list", "guest list", func(_ *Server, ctl *fakeGuestPowerCtl, _ string) { ctl.guestsErr = errors.New("api down") }},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, ctl, _, dir, _ := opServer(t, "running")
tc.setup(s, ctl, dir)
id := int64(100 + i)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
r := waitResult(t, s, id)
if r.Outcome != "refused" || !strings.Contains(strings.ToLower(r.Message), tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if n := ex.count(9201); n != 0 {
t.Fatalf("guard %q did not hold: restarted %d", tc.name, n)
}
})
}
}
// The thrash brake counts operator restarts AND the supervisor's: 3 in 15 minutes → the next press is refused, and
// presses stay refused for 30 minutes. RED-PROOF: drop the brake → the 4th press restarts → FAILS.
func TestOpActions_TheBrakeCountsEveryRestart(t *testing.T) {
s, ex, _, clk, _, _ := opServer(t, "running")
// one restart by the supervisor itself
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
if ex.count(9201) != 1 {
t.Fatalf("setup: the supervisor did not restart (%d)", ex.count(9201))
}
for id := int64(1); id <= 2; id++ {
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("press %d: %+v", id, r)
}
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9201"}))
r := waitResult(t, s, 3)
if r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 3 {
t.Fatalf("restarts = %d, want 3", n)
}
clk.t = clk.t.Add(20 * time.Minute) // the window has passed, but the 30-minute refusal has not
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" {
t.Fatalf("inside the 30-minute refusal: %+v", r)
}
clk.t = clk.t.Add(11 * time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 5, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 5); r.Outcome != "done" {
t.Fatalf("after the refusal: %+v", r)
}
}
// A controller that does not come back healthy is a FAILED press, said plainly.
func TestOpActions_NoHealthyAnswerIsFailed(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
ex.health = "unhealthy"
s.opHealthWait = 30 * time.Millisecond
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 9, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 9); r.Outcome != "failed" || !strings.Contains(r.Message, "no healthy answer") {
t.Fatalf("result %+v", r)
}
}
+328
View File
@@ -0,0 +1,328 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
//
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
//
// THE GUARDS (each refuses with the reason in the result; nothing is called):
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
// - a controller swap (update) is in flight;
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
// alone while the reboot runs (rebootInFlight).
//
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
// one day). Pinned by opreboot_test.go.
const (
// OpRebootGuest is the agent's second operator verb (D65 = A).
OpRebootGuest = "reboot_guest"
// opRebootEvery: at most one operator reboot per guest per hour.
opRebootEvery = time.Hour
// opRebootNoticeFor: how long the household's dashboard shows the line.
opRebootNoticeFor = 24 * time.Hour
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
opRebootTimeout = 5 * time.Minute
operatorRebootFile = "operator-reboot.json"
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
// Docker does not hide it.
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
// stopped writing (stuck or killed) — not a busy one.
controllerBusyFresh = 90 * time.Second
)
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
type operatorRebootRecord struct {
At time.Time `json:"at"`
ActionID int64 `json:"action_id"`
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
ControllerAfterS int `json:"controller_after_s,omitempty"`
}
func (s *Server) operatorRebootPath(vmid int) string {
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
}
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
// must not silently vanish).
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
if errors.Is(err, os.ErrNotExist) {
return operatorRebootRecord{}, false, nil
}
if err != nil {
return operatorRebootRecord{}, false, err
}
var rec operatorRebootRecord
if err := json.Unmarshal(raw, &rec); err != nil {
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
}
return rec, !rec.At.IsZero(), nil
}
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
raw, err := json.Marshal(rec)
if err != nil {
return err
}
p := s.operatorRebootPath(vmid)
tmp := p + ".tmp"
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
return err
}
return os.Rename(tmp, p)
}
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
func (s *Server) rebooting(vmid int) bool {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
return s.rebootInFlight[vmid]
}
func (s *Server) setRebooting(vmid int, on bool) {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
if s.rebootInFlight == nil {
s.rebootInFlight = map[int]bool{}
}
if on {
s.rebootInFlight[vmid] = true
} else {
delete(s.rebootInFlight, vmid)
}
}
func (s *Server) rebooter() interface {
RebootGuest(ctx context.Context, vmid int) error
} {
if s.opRebooter != nil {
return s.opRebooter
}
if s.guestAttach != nil {
return s.guestAttach
}
return nil
}
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
// (the caller refuses — it cannot prove no backup runs).
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
if err != nil {
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
return false, "", nil
}
return false, "", err
}
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
if perr != nil {
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
}
if s.clock().Sub(at) > controllerBusyFresh {
s.logger.Info("operator reboot: the controller's busy file is stale — the controller stopped writing it; not busy",
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
return false, "", nil
}
why := "work in progress"
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
why = strings.TrimSpace(lines[1])
}
return true, why, nil
}
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
rb := s.rebooter()
if s.staleLock == nil || s.guestExec == nil || rb == nil {
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
if s.rebooting(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
// At most one per hour.
now := s.clock()
last, had, err := s.lastOperatorReboot(vmid)
if err != nil {
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
}
if had && now.Sub(last.At) < opRebootEvery {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
}
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
s.ctrlSup.mu.Unlock()
// The heavy-operation gate, held until the guest is up again.
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
if !ok {
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
}
defer release()
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
}
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
} else if cbusy {
return opOutcomeRefused, "refused: the controller is busy — " + why
}
// The record FIRST: the hourly limit and the household's line both read it.
rec := operatorRebootRecord{At: now, ActionID: id}
if err := s.saveOperatorReboot(vmid, rec); err != nil {
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
}
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.setRebooting(vmid, true)
defer s.setRebooting(vmid, false)
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
began := time.Now()
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
err = rb.RebootGuest(rctx, vmid)
cancel()
if err != nil {
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
}
upS := int(time.Since(began).Round(time.Second) / time.Second)
rec.GuestUpAfterS = upS
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
}
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
case <-time.After(s.opPoll()):
}
}
_ = s.saveOperatorReboot(vmid, rec)
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
}
// rebootBlockerSentence names jobBlocker's code in the result's words.
func rebootBlockerSentence(code string) string {
switch code {
case PowerOffBackup:
return "a backup is running"
case PowerOffCtrlUpdate:
return "a controller update is running"
case PowerOffDrive:
return "a drive or network-storage job is running"
case PowerOffEscrow:
return "an escrow ceremony is running"
case PowerOffUnknown:
return "the running jobs cannot be read"
default:
return "the host is busy"
}
}
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
rec, ok, err := s.lastOperatorReboot(vmid)
if err != nil {
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
return
}
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
return
}
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
}
+306
View File
@@ -0,0 +1,306 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
// rbExec adds the controller's busy file to the operator-restart fake.
type rbExec struct {
*opExec
busy string // the file's content; "" = no file
busyErr error // pct exec itself failed
}
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
if f.busyErr != nil {
return "", f.busyErr
}
if f.busy == "" {
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
}
return f.busy, nil
}
return f.opExec.GuestExec(ctx, vmid, args...)
}
type fakeRebooter struct {
mu sync.Mutex
n int
err error
after func()
}
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
r.mu.Lock()
defer r.mu.Unlock()
r.n++
if r.after != nil {
r.after()
}
return r.err
}
func (r *fakeRebooter) count() int {
r.mu.Lock()
defer r.mu.Unlock()
return r.n
}
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
t.Helper()
s, ox, ctl, clk, dir, _ := opServer(t, "running")
ex := &rbExec{opExec: ox}
s.guestExec = ex
rb := &fakeRebooter{}
s.opRebooter = rb
s.inFlight = &backup.InFlight{}
return s, ex, rb, ctl, clk, dir
}
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
return waitResult(t, s, id)
}
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
s, _, rb, _, clk, _ := rbServer(t)
r := pressReboot(t, s, 1)
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
t.Fatalf("result %+v", r)
}
if rb.count() != 1 {
t.Fatalf("reboots = %d, want 1", rb.count())
}
rec, ok, err := s.lastOperatorReboot(9201)
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
}
if s.inFlight.Busy() != "" {
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
}
if s.rebooting(9201) {
t.Fatal("the reboot-in-flight flag outlived the reboot")
}
}
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
func TestOpReboot_GuardsRefuse(t *testing.T) {
fresh := func(clk *supClock) string {
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
}
cases := []struct {
name, want string
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
}{
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.guests[0].Status = "stopped"
}},
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupRun = map[int]bool{9201: true}
}},
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupErr = errors.New("tasks unreadable")
}},
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.inFlight.TryAcquire("backup:felhom-pbs")
}},
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
}},
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
ex.busy = fresh(clk)
}},
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
}},
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
s.ctrlSup.mu.Lock()
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
s.ctrlSup.mu.Unlock()
}},
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
panic(err)
}
}},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, rb, ctl, clk, dir := rbServer(t)
tc.setup(s, ex, ctl, clk, dir)
r := pressReboot(t, s, int64(100+i))
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if rb.count() != 0 {
t.Fatalf("guard %q did not hold: rebooted", tc.name)
}
if _, ok, _ := s.lastOperatorReboot(9201); ok {
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
}
})
}
}
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
s, ex, rb, _, clk, _ := rbServer(t)
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
t.Fatalf("result %+v, reboots %d", r, rb.count())
}
}
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
s, _, rb, ctl, clk, dir := rbServer(t)
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("first press: %+v", r)
}
clk.t = clk.t.Add(59 * time.Minute)
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
t.Fatalf("second press inside the hour: %+v", r)
}
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
s2, ox2, _, _, _, _ := opServer(t, "running")
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
s2.guestExec = &rbExec{opExec: ox2}
rb2 := &fakeRebooter{}
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
t.Fatalf("after an agent restart, inside the hour: %+v", r)
}
clk.t = clk.t.Add(2 * time.Minute)
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
}
}
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
s, ex, _, _, clk, _ := rbServer(t)
for id := int64(1); id <= 2; id++ {
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("restart %d: %+v", id, r)
}
clk.t = clk.t.Add(time.Minute)
}
if r := pressReboot(t, s, 3); r.Outcome != "done" {
t.Fatalf("reboot: %+v", r)
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 2 {
t.Fatalf("restarts = %d, want 2", n)
}
}
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
s, _, rb, _, _, _ := rbServer(t)
rb.err = errors.New("pct reboot 9201: exit status 255")
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
t.Fatalf("result %+v", r)
}
if s.inFlight.Busy() != "" || s.rebooting(9201) {
t.Fatal("the gate or the flag outlived a failed reboot")
}
}
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
s, ex, rb, ctl, _, _ := rbServer(t)
var started, restarted int
rb.after = func() {
ctl.guests[0].Status = "stopped"
s.GuestPowerTick(context.Background())
ctl.guests[0].Status = "running"
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
started, restarted = len(ctl.started), ex.count(9201)
ex.mu.Lock()
ex.status[9201] = "running"
ex.mu.Unlock()
}
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
if started != 0 || restarted != 0 {
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
}
}
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
s, _, _, _, clk, _ := rbServer(t)
read := func() string {
w := httptest.NewRecorder()
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
var body struct {
Data struct {
At string `json:"at"`
} `json:"data"`
At string `json:"at"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("body %q: %v", w.Body.String(), err)
}
if body.Data.At != "" {
return body.Data.At
}
return body.At
}
if got := read(); got != "" {
t.Fatalf("no reboot yet, got %q", got)
}
at := clk.t
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
clk.t = at.Add(23 * time.Hour)
if got := read(); got != at.UTC().Format(time.RFC3339) {
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
}
clk.t = at.Add(25 * time.Hour)
if got := read(); got != "" {
t.Fatalf("after 25 h the line must be gone, got %q", got)
}
}
+16
View File
@@ -411,6 +411,20 @@ type Server struct {
guestsDir string
ctrlSup controllerSupervisor
// 2026-10-29: the operator's actions from the hub's reply (opactions.go). opHealthPoll/opHealthWait bound the wait
// for a restarted controller's healthy answer (tests shorten them; zero → the defaults).
opActs opActionState
opTrigger func()
opHealthPoll time.Duration
opHealthWait time.Duration
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
opRebooter interface {
RebootGuest(ctx context.Context, vmid int) error
}
rebootMu sync.Mutex
rebootInFlight map[int]bool
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
// detached pipeline runs through (tests inject; production defaults set in NewServer).
netVerifyMu sync.Mutex
@@ -596,6 +610,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
+4 -2
View File
@@ -256,8 +256,10 @@ func TestBackupRunning_AllCallersUseTheOneHelper(t *testing.T) {
}
}
}
if callers != 5 {
t.Errorf("expected the five known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup), found %d", callers)
// 2026-10-29: +2 deliberate callers — the operator's restart_controller (opactions.go) and R-1344's unresponsive count
// (controllersupervisor.go observeControllerHealth). Both through s.staleLock, the one helper.
if callers != 7 {
t.Errorf("expected the seven known callers (stale-lock, guest-power, power-off, supervisor, the R-1006 cleanup, the operator restart, the unresponsive count), found %d", callers)
}
b, _ := os.ReadFile("stalelock.go")
if !strings.Contains(string(b), "vzdumpInFlight(tasks, vmid, proc)") {
@@ -0,0 +1,74 @@
package localapi
import (
"context"
"testing"
)
// R-1344 (operator brief 2026-10-29, Part 2): a controller that RUNS but whose Docker health check reads „unhealthy" on
// controllerUnresponsiveConfirm consecutive sweeps is reported unresponsive (the stanza; the hub raises
// controller_unresponsive). ALARM ONLY: it is never restarted for it until the measurement licenses that.
// RED-PROOF: seen failing before the stanza carried it.
func unrespServer(t *testing.T, health string) (*Server, *opExec, *fakeGuestPowerCtl) {
t.Helper()
s, ex, ctl, _, _, _ := opServer(t, "running")
ex.health = health
return s, ex, ctl
}
func sweep(s *Server, n int) {
for i := 0; i < n; i++ {
s.ControllerSupervisorTick(context.Background())
}
}
func TestSupervisor_UnresponsiveIsAlarmedNeverRestarted(t *testing.T) {
s, ex, _ := unrespServer(t, "unhealthy")
sweep(s, controllerUnresponsiveConfirm-1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) != 1 || g[0].Unresponsive {
t.Fatalf("unresponsive before %d sweeps: %+v", controllerUnresponsiveConfirm, g)
}
sweep(s, 1)
g := s.ControllerSupervisorStatus(context.Background()).Guests
if len(g) != 1 || !g[0].Unresponsive || g[0].UnresponsiveSince == "" {
t.Fatalf("after %d unhealthy sweeps: %+v", controllerUnresponsiveConfirm, g)
}
since := g[0].UnresponsiveSince
sweep(s, 10)
if n := ex.count(9201); n != 0 {
t.Fatalf("an unresponsive controller was RESTARTED (%d) — the restart waits for the measurement", n)
}
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].UnresponsiveSince != since {
t.Fatal("unresponsive_since moved while it stayed unresponsive — the hub would mail again")
}
// it answers again
ex.health = "healthy"
sweep(s, 1)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; g[0].Unresponsive || g[0].UnresponsiveSince != since {
t.Fatalf("after it answered: %+v (unresponsive must clear; since stays as the record)", g[0])
}
}
// Slow but answering (healthy / starting), a swap, a vzdump or a parked guest never counts.
func TestSupervisor_UnresponsiveGuards(t *testing.T) {
for _, tc := range []struct {
name string
setup func(s *Server, ex *opExec, ctl *fakeGuestPowerCtl)
}{
{"healthy", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "healthy" }},
{"starting", func(_ *Server, ex *opExec, _ *fakeGuestPowerCtl) { ex.health = "starting" }},
{"swap", func(s *Server, _ *opExec, _ *fakeGuestPowerCtl) { s.swapInFlight[9201] = true }},
{"vzdump", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.backupRun = map[int]bool{9201: true} }},
{"locked", func(_ *Server, _ *opExec, ctl *fakeGuestPowerCtl) { ctl.locks[9201] = "backup" }},
} {
t.Run(tc.name, func(t *testing.T) {
s, ex, ctl := unrespServer(t, "unhealthy")
tc.setup(s, ex, ctl)
sweep(s, controllerUnresponsiveConfirm+2)
if g := s.ControllerSupervisorStatus(context.Background()).Guests; len(g) == 1 && g[0].Unresponsive {
t.Fatalf("%s counted as unresponsive: %+v", tc.name, g[0])
}
})
}
}
+3 -3
View File
@@ -24,10 +24,10 @@
"readers": [
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
],
"recorded": "2026-10-27",
"recorded_by": "CC 2026-10-27: day0_agent 0.167.0 (operator vouch 2026-10-27, read from the hub: version + agent and bundle sha256 match the release); earlier 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"recorded": "2026-10-29",
"recorded_by": "CC 2026-10-29: day0_agent 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.167.0"
"day0_agent": "0.169.0"
}
}