From 3d0c907b5613e8822b00fc73da74485580ae904c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 29 Oct 2026 12:31:33 +0100 Subject: [PATCH] =?UTF-8?q?hub:=20Restart=20controller=20(the=20agent's=20?= =?UTF-8?q?closed=20list=20on=20the=20host-report=20reply)=20and=20Clear?= =?UTF-8?q?=20hold=20(D64=20=3D=20B,=20the=20controller's=20list)=20on=20t?= =?UTF-8?q?he=20host=20page,=20each=20behind=20a=20confirm;=20controller?= =?UTF-8?q?=5Funresponsive=20(R-1344,=20alarm=20only);=2004=20=C2=A76.1=20?= =?UTF-8?q?+=2009=20=C2=A73=20decision=20283;=20wire=20gate=20declares=20b?= =?UTF-8?q?oth=20new=20wires;=20R-1344=20filed=20=E2=80=94=20tests=20red?= =?UTF-8?q?=20first?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../04-control-plane-authorization.md | 12 ++ .../architecture/09-update-architecture.md | 8 ++ documentation/backlog/OPEN-ITEMS.md | 3 +- hub/CHANGELOG.md | 20 +++ hub/internal/api/handler.go | 20 +++ hub/internal/api/host_opactions_test.go | 86 +++++++++++ hub/internal/api/opactions.go | 30 ++++ hub/internal/monitor/controller_supervisor.go | 19 ++- .../monitor/controller_supervisor_test.go | 29 ++++ hub/internal/monitor/held_apps.go | 2 +- hub/internal/notify/dispatcher.go | 2 + hub/internal/store/opactions.go | 135 ++++++++++++++++-- hub/internal/store/opactions_test.go | 84 ++++++++++- hub/internal/web/held_apps.go | 14 +- hub/internal/web/host_opactions_test.go | 94 ++++++++++++ hub/internal/web/hosts_test.go | 14 +- hub/internal/web/opactions.go | 67 +++++++++ .../web/templates/host_detail_body.html | 31 +++- hub/internal/web/templates/style.css | 7 + scripts/wire_contract_gate.py | 6 + 20 files changed, 663 insertions(+), 20 deletions(-) create mode 100644 hub/internal/api/host_opactions_test.go create mode 100644 hub/internal/web/host_opactions_test.go diff --git a/documentation/architecture/04-control-plane-authorization.md b/documentation/architecture/04-control-plane-authorization.md index b5aaf942..79455cee 100644 --- a/documentation/architecture/04-control-plane-authorization.md +++ b/documentation/architecture/04-control-plane-authorization.md @@ -183,6 +183,18 @@ the deletion is the hub's), `run_job` (`fill-watch`, `offsite-integrity`, `offsi Anything else is refused at the hub's POST and again on the box. No action deletes data, starts a countdown or shortens one (`TestOpActions_ClosedList`, `TestOperatorActions_ClosedList`). Unanswered after 24 h: expired. +**`[DESIGN]` 2026-10-29 — D64 = B, and the agent's own list (operator ruling 12:10; `09` §3 decision 283).** The +controller's list gains **`clear_hold `** for every hold reason, with no signature: the RUNNING controller clears the +hold in memory and saves at once (no later settings save can write it back — R-1343). The operator accepted the cost in +words: a compromised hub could release a data-protecting hold and start an app on half-restored data; the hub's confirm +step shows the hold's reason and, for a restore or put-back hold, that warning. **R-379 („only a person clears a hold") +now reads „a person — on the box, or the operator on the hub".** The hub refuses `clear_hold` for an app the box does not +report held now. **The AGENT has its own closed list on the host report's reply — `restart_controller ` only** +(`store.ValidateHostOperatorAction` / agent `localapi.OperatorActionNames`; never a shared verb, `TestHostOperatorActions_ClosedList`): +the supervisor's restart with its guards, the 3-in-15-minutes brake counting operator restarts too (then 30 minutes +refused), the result in the next report (requested at once), a vmid that is not a guest of that host refused at the hub. +Neither verb deletes data, starts a countdown or shortens one. A guest reboot from the hub is NOT on either list. + ### 6.2 Who may press the operator's buttons: session, Basic, or a scoped token [hub v0.152.0, `09` §3 decision 251] The hub's operator surface has three ways in: a browser session, Basic auth with the operator password (scripts, diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index b39cdf86..ca5cf5a8 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -1206,6 +1206,14 @@ Wanderer's Meilisearch went 256 → 384 MB and was measured again (it sizes itse `-tags faultinject` test build — never in the release image (`TestD63_TheReleaseBuildHasNoFaultSwitch`). *Operator, 2026-10-29 08:17.* `07` §6.3. +### 2026-10-29 (12:10, operator) — D64 = B: the hub may clear any hold + +283. **D64 = B — the hub's „Clear hold" works for every hold reason, with no operator signature.** The operator accepted + that a compromised hub could clear a data-protecting hold (an app could start on half-restored data); the confirm + step shows the hold's reason and, for a restore or put-back hold, that warning before the press. R-379 now reads + „a person — on the box, or the operator on the hub". Built as the controller's `clear_hold` operator action + (`04` §6.1), cleared by the running controller (R-1343). *Operator, 2026-10-29 12:10.* + ### 2026-10-28 (day, attended) — RomM kept; the runtime cache fills itself in CI 277. **D61 = keep RomM offered** (R-1330) while no RomM release carries Node ≥ 24.22.1; the first one that does is taken diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 8e1857cc..e38d8d75 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -293,7 +293,7 @@ stopping line that lies. | **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | -## Monitoring & notifications — 24 rows (P2 2, P3 14, P4 8) +## Monitoring & notifications — 25 rows (P2 2, P3 15, P4 8) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -321,6 +321,7 @@ stopping line that lies. | **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC | | **R-1341** | Monitoring & notifications | P4 | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** Measured 2026-10-29 on Tester 1 (controller 0.347.0, agent 0.168.0, R-528's proof): one kill → one `app_oom` and one operator mail (right), and the `[WARN] [deadapp]` line at 09:02:32, :03:02, :03:32, :04:02, :04:32 … — the same as the night's proof (`audits/night-2026-10-28/r528-oom/`). The loop (`controller/cmd/controller/main.go:1100`, since v0.243.0, R-514) logs every container `ScanOOMKilled` returns, every scan; the notifier's cooldown keeps the event to one. About 2 880 lines a day per OOM-marked container until it restarts — noise that hides the next real line. Fix: log once per (container, start time, kill count), with a test that a second scan of the same kill logs nothing and a new kill logs again. Evidence: `audits/d63-2026-10-29/run-p4-oom.log`. | OPEN | — | the fix above, red first | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | +| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC | ## Hub & operator — 12 rows (P3 3, P4 9) diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 858e0025..ab3d5580 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,23 @@ +## Unreleased (2026-10-29, afternoon) — „Restart controller" and „Clear hold" on the host page (D64 = B); controller_unresponsive (R-1344) — commit only; rides the next hub release + +- **The AGENT's operator actions (operator brief 2026-10-29, Part 1; `04` §6.1).** `operator_actions` gains a `host_id` + column: a row with a host id is the AGENT's (the host-report reply lists it as `operator_actions`; the agent answers + `operator_action_results` on its next report, closed per id AND host, one `operator_action` event each); a row without + is the controller's, as before. The agent's CLOSED list is `restart_controller ` only + (`store.ValidateHostOperatorAction`), and a vmid must be a guest of THAT host. The host page's guest table has + „Restart controller" behind a confirm step that names the box; the press pokes the agent (seconds over the tunnel; the + 15-min cycle otherwise). Tests `TestHostOperatorActions_*` (store), `TestHostReport_AgentOperatorActionRoundTrip` (api), + `TestHostPage_RestartAndClearHold*` (web) — red first; the host-row filter and both web guards red by mutation. +- **D64 = B (operator 2026-10-29 12:10) — `clear_hold` joins the controller's closed list**, for every hold reason, no + signature. The Held apps list has „Clear hold" per row behind a confirm step showing the app, the reason and since when; + a restore or put-back hold adds: „This hold protects data from a failed restore. If you clear it, the app may start on + half-restored data. Check the restore copy first." Refused (400, no row) for an app the box does not report held now. + „Who may clear it" now names the operator here too. `TestOperatorActions_ClosedList` updated deliberately. +- **R-1344 — `controller_unresponsive`** (warning, operator-only, hub-generated like the crash-loop pair) when a guest's + `unresponsive_since` moves in the agent's `controller_supervisor` stanza (also at first sight while unresponsive). The + agent only alarms; it does not restart for it yet. Tests `TestControllerSupervisorChecker_Unresponsive`, + `TestControllerUnresponsive_RegisteredAndOperatorOnly` — red first. + ## v0.175.0 (2026-10-29) — R-1255: two escrow edges (review round 2, HUNT-3 H3/H4); D62: the restore-copy delete event — DEPLOYED on the operator's yes (image sha256:642e94b6…, from `bf69bb91`, CI 2487) - **D62 (operator 2026-10-29) — `restore_copy_deleted` is accepted and operator-only.** The controller's „Törlés most" diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 4bf76297..f63ac010 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -809,6 +809,16 @@ type hostReportPayload struct { DeliberateOff *struct { RequestedAt string `json:"requested_at"` } `json:"deliberate_off"` + // OperatorActionResults (2026-10-29, the agent's operator actions): the answers to the previous replies' + // operator_actions — the controller's shape. Absent on an older agent. + OperatorActionResults []hostOperatorActionResult `json:"operator_action_results"` +} + +// hostOperatorActionResult mirrors the agent's hub.OperatorActionResult. +type hostOperatorActionResult struct { + ID int64 `json:"id"` + Outcome string `json:"outcome"` + Message string `json:"message"` } // hostForeignKeyArchives mirrors the agent's hub.ForeignKeyArchives (R-366 slice 2). @@ -1133,6 +1143,14 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) { if pending, err := h.store.PendingLogBundleRequest(hostID, store.LogBundleComponentAgent); err == nil && pending { resp["log_tail_requested"] = true } + // The agent's operator actions (2026-10-29): the results this report carries close their rows (one event each); + // what is still pending rides this reply. A store error drops nothing from the heartbeat. + h.ingestHostOperatorActionResults(hostID, custID, rep.OperatorActionResults) + if acts, err := h.store.PendingHostOperatorActions(hostID); err != nil { + h.logger.Printf("[WARN] host operator actions for %s: %v", hostID, err) + } else if len(acts) > 0 { + resp["operator_actions"] = acts + } w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode(resp) @@ -2400,6 +2418,8 @@ var allowedEventTypes = map[string]bool{ "controller_crashloop": true, // R-539 (hub v0.117.0, agent v0.132.0) — hub-generated from slow_crashloop_since; operator-only. "controller_slow_crashloop": true, + // R-1344 (2026-10-29): hub-generated from the stanza's unresponsive_since; operator-only (notify.operatorOnlyEvents). + "controller_unresponsive": true, // R-550 (hub v0.117.0, controller v0.246.0) — pushed by the controller at startup when it finds a // restore that was in flight when the box stopped. For the household: they can run it again. "restore_interrupted": true, diff --git a/hub/internal/api/host_opactions_test.go b/hub/internal/api/host_opactions_test.go new file mode 100644 index 00000000..ffef716a --- /dev/null +++ b/hub/internal/api/host_opactions_test.go @@ -0,0 +1,86 @@ +package api + +import ( + "encoding/json" + "net/http" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/notify" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// Operator brief 2026-10-29, Part 1: the AGENT's operator actions ride the host-report reply as `operator_actions` +// (the controller's shape) and its answer rides the next host report as `operator_action_results`; the row closes and +// one `operator_action` event is stored. A controller row never reaches the agent. RED-PROOF: seen failing before the +// reply listed the agent rows (no `operator_actions` key). +func TestHostReport_AgentOperatorActionRoundTrip(t *testing.T) { + h, st, _ := newTestHandler(t) + st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"}) + if _, err := st.CreateOperatorAction("c1", "clear_hold", "radicale", "operator"); err != nil { + t.Fatal(err) + } + id, err := st.CreateHostOperatorAction("c1", "h1", "restart_controller", "9201", "operator browser session from 192.168.0.5") + if err != nil { + t.Fatal(err) + } + rr := do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")) + var env struct { + OperatorActions []store.OperatorActionDirective `json:"operator_actions"` + } + if err := json.Unmarshal(rr.Body.Bytes(), &env); err != nil { + t.Fatal(err) + } + if len(env.OperatorActions) != 1 || env.OperatorActions[0].ID != id || env.OperatorActions[0].Action != "restart_controller" || + env.OperatorActions[0].Arg != "9201" { + t.Fatalf("the agent's reply lists %+v, want only the restart #%d (never the controller's clear_hold)", env.OperatorActions, id) + } + body := strings.Replace(validReportBody("h1"), `"audit_tail":[]`, + `"audit_tail":[],"operator_action_results":[{"id":`+i64s(id)+`,"outcome":"done","message":"the controller answered after 14 s"}]`, 1) + if rr := do(h, http.MethodPost, "/host-report", "HKEY", body); rr.Code != http.StatusOK { + t.Fatalf("result heartbeat = %d %s", rr.Code, rr.Body.String()) + } + if left, _ := st.PendingHostOperatorActions("h1"); len(left) != 0 { + t.Fatalf("the row is still pending: %+v", left) + } + rows, _ := st.ListOperatorActions("c1", 10) + var got *store.OperatorActionRow + for i := range rows { + if rows[i].ID == id { + got = &rows[i] + } + } + if got == nil || got.Outcome != "done" || got.Message != "the controller answered after 14 s" { + t.Fatalf("row = %+v", got) + } + evs, _ := st.GetEventsByType("c1", "operator_action", time.Time{}) + n := 0 + for _, e := range evs { + if e.EventType == "operator_action" && strings.Contains(e.Message, "restart_controller 9201") { + n++ + } + } + if n != 1 { + t.Fatalf("operator_action events for the restart = %d, want 1: %+v", n, evs) + } + // once answered, the reply stops listing it + rr = do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1")) + env.OperatorActions = nil + _ = json.Unmarshal(rr.Body.Bytes(), &env) + if len(env.OperatorActions) != 0 { + t.Fatalf("still listed after its result: %+v", env.OperatorActions) + } +} + +func i64s(n int64) string { b, _ := json.Marshal(n); return string(b) } + +// R-1344: the hub-generated controller_unresponsive is listed (the crash-loop pair's rule) and operator-only. +func TestControllerUnresponsive_RegisteredAndOperatorOnly(t *testing.T) { + if !allowedEventTypes["controller_unresponsive"] { + t.Error("controller_unresponsive is not in allowedEventTypes") + } + if !notify.IsOperatorOnly("controller_unresponsive") { + t.Error("controller_unresponsive is not operator-only") + } +} diff --git a/hub/internal/api/opactions.go b/hub/internal/api/opactions.go index a22cc409..251df7de 100644 --- a/hub/internal/api/opactions.go +++ b/hub/internal/api/opactions.go @@ -55,3 +55,33 @@ func argSuffix(arg string) string { } return " " + arg } + +// ingestHostOperatorActionResults closes each answered AGENT row (matched on the id AND this host) and saves one +// hub-minted operator_action event per closed row — the controller's rule, on the agent's channel. +func (h *Handler) ingestHostOperatorActionResults(hostID, customerID string, results []hostOperatorActionResult) { + for _, res := range results { + row, err := h.store.RecordHostOperatorActionResult(hostID, res.ID, res.Outcome, res.Message) + switch { + case err != nil: + h.logger.Printf("[WARN] host operator action #%d result from %s not recorded: %v", res.ID, hostID, err) + continue + case row == nil: + h.logger.Printf("[DEBUG] host operator action #%d result from %s ignored (already closed, or not this host's)", res.ID, hostID) + continue + } + h.logger.Printf("[INFO] host operator action #%d %s%s on %s (pressed by %s): %s — %s", + row.ID, row.Action, argSuffix(row.Arg), hostID, row.RequestedBy, row.Outcome, row.Message) + if customerID == "" { + continue + } + severity := "info" + if row.Outcome != store.OperatorActionDone { + severity = "warning" + } + if _, eerr := h.store.SaveEvent(customerID, "operator_action", severity, + fmt.Sprintf("Operator action %s%s on host %s (pressed by %s): %s — %s", row.Action, argSuffix(row.Arg), hostID, row.RequestedBy, row.Outcome, row.Message), + "", "hub"); eerr != nil { + h.logger.Printf("[WARN] host operator action #%d recorded, but its event could not be saved: %v", row.ID, eerr) + } + } +} diff --git a/hub/internal/monitor/controller_supervisor.go b/hub/internal/monitor/controller_supervisor.go index e28a22a8..cf63c929 100644 --- a/hub/internal/monitor/controller_supervisor.go +++ b/hub/internal/monitor/controller_supervisor.go @@ -42,6 +42,7 @@ type supSeen struct { lastRestartAt string crashloopSince string slowCrashloopSince string + unresponsiveSince string } // ControllerSupervisorGuest mirrors the agent's hub.ControllerSupervisorGuest wire shape. @@ -57,6 +58,10 @@ type ControllerSupervisorGuest struct { Restarts24h int `json:"restarts_24h"` SlowCrashloop bool `json:"slow_crashloop"` SlowCrashloopSince string `json:"slow_crashloop_since"` + // R-1344 (agent ≥ the 2026-10-29 release): the controller runs but Docker's health check has failed for several + // sweeps. ALARM ONLY — no restart until the measurement licenses one. Absent on older agents → never an event. + Unresponsive bool `json:"unresponsive"` + UnresponsiveSince string `json:"unresponsive_since"` } // ParseControllerSupervisor extracts the stanza from a host report body. A missing or malformed @@ -98,7 +103,8 @@ func (c *ControllerSupervisorChecker) observe(hostID, customerID string, guests for _, g := range guests { key := fmt.Sprintf("%s/%d", hostID, g.VMID) prev, known := c.seen[key] - c.seen[key] = supSeen{lastRestartAt: g.LastRestartAt, crashloopSince: g.CrashloopSince, slowCrashloopSince: g.SlowCrashloopSince} + c.seen[key] = supSeen{lastRestartAt: g.LastRestartAt, crashloopSince: g.CrashloopSince, slowCrashloopSince: g.SlowCrashloopSince, + unresponsiveSince: g.UnresponsiveSince} if !known { if g.Crashloop && g.CrashloopSince != "" { c.emit(customerID, hostID, g, "controller_crashloop") @@ -106,6 +112,9 @@ func (c *ControllerSupervisorChecker) observe(hostID, customerID string, guests if g.SlowCrashloop && g.SlowCrashloopSince != "" { c.emit(customerID, hostID, g, "controller_slow_crashloop") } + if g.Unresponsive && g.UnresponsiveSince != "" { + c.emit(customerID, hostID, g, "controller_unresponsive") + } continue } if g.LastRestartAt != "" && g.LastRestartAt != prev.lastRestartAt { @@ -117,6 +126,9 @@ func (c *ControllerSupervisorChecker) observe(hostID, customerID string, guests if g.SlowCrashloopSince != "" && g.SlowCrashloopSince != prev.slowCrashloopSince { c.emit(customerID, hostID, g, "controller_slow_crashloop") } + if g.UnresponsiveSince != "" && g.UnresponsiveSince != prev.unresponsiveSince { + c.emit(customerID, hostID, g, "controller_unresponsive") + } } } @@ -135,6 +147,10 @@ func (c *ControllerSupervisorChecker) emit(customerID, hostID string, g Controll severity = "warning" message = fmt.Sprintf("Host %s guest %d: the controller keeps dying — the agent restarted it %d times in 24 hours (each one too far apart for the 15-minute brake). It is still being restarted; this is the warning that it will not stay up. Last reason: %s", hostID, g.VMID, g.Restarts24h, g.LastReason) + case "controller_unresponsive": + severity = "warning" + message = fmt.Sprintf("Host %s guest %d: the controller runs but does not answer its health check (since %s). The agent does NOT restart it yet (R-1344: the measurement must first show a busy night never looks like this) — use „Restart controller\" on the host page if it stays so.", + hostID, g.VMID, g.UnresponsiveSince) default: return } @@ -143,6 +159,7 @@ func (c *ControllerSupervisorChecker) emit(customerID, hostID string, g Controll "last_restart_at": g.LastRestartAt, "last_reason": g.LastReason, "crashloop_since": g.CrashloopSince, "parked": g.Parked, "restarts_24h": g.Restarts24h, "slow_crashloop_since": g.SlowCrashloopSince, + "unresponsive_since": g.UnresponsiveSince, }) c.logger.Printf("[INFO] Controller supervisor: %s %s/%d (%s)", eventType, hostID, g.VMID, g.LastReason) if _, err := c.store.SaveEvent(customerID, eventType, severity, message, string(details), "hub"); err != nil { diff --git a/hub/internal/monitor/controller_supervisor_test.go b/hub/internal/monitor/controller_supervisor_test.go index 2585be40..fed8d8a3 100644 --- a/hub/internal/monitor/controller_supervisor_test.go +++ b/hub/internal/monitor/controller_supervisor_test.go @@ -146,3 +146,32 @@ func TestControllerSupervisorChecker_SlowCrashloopAtFirstSight(t *testing.T) { t.Fatalf("first sight in a slow crash loop: got %v, want exactly [controller_slow_crashloop/warning]", evs) } } + +// R-1344 (operator brief 2026-10-29, Part 2 — ALARM ONLY until the measurement licenses a restart): a controller that +// runs but does not answer reaches the operator once per new unresponsive_since, as controller_unresponsive (warning), +// also at first sight; an answering one never. RED-PROOF: seen failing before the branch existed. +func TestControllerSupervisorChecker_Unresponsive(t *testing.T) { + st := newCapStore(t) + rec := &evRec{} + c := NewControllerSupervisorChecker(st, rec.fn, log.New(io.Discard, "", 0)) + st.SaveHostReport("h1", "c1", supReport(`[{"vmid":9201,"restarts_total":0,"crashloop":false,"parked":false,"unresponsive":false}]`), store.HostReportDenorm{}) + c.Check() + if evs := rec.take(); len(evs) != 0 { + t.Fatalf("an answering controller emitted %v", evs) + } + st.SaveHostReport("h1", "c1", supReport(`[{"vmid":9201,"restarts_total":0,"crashloop":false,"parked":false,"unresponsive":true,"unresponsive_since":"2026-10-29T13:00:00Z"}]`), store.HostReportDenorm{}) + c.Check() + if evs := rec.take(); len(evs) != 1 || evs[0] != "controller_unresponsive/warning" { + t.Fatalf("got %v, want exactly [controller_unresponsive/warning]", evs) + } + c.Check() + if evs := rec.take(); len(evs) != 0 { + t.Fatalf("the same unresponsive_since emitted again: %v", evs) + } + // a hub restarted while it is unresponsive says so once + c2 := NewControllerSupervisorChecker(st, rec.fn, log.New(io.Discard, "", 0)) + c2.Check() + if evs := rec.take(); len(evs) != 1 || evs[0] != "controller_unresponsive/warning" { + t.Fatalf("first sight while unresponsive: %v", evs) + } +} diff --git a/hub/internal/monitor/held_apps.go b/hub/internal/monitor/held_apps.go index c8d0ba7f..63d9b616 100644 --- a/hub/internal/monitor/held_apps.go +++ b/hub/internal/monitor/held_apps.go @@ -58,7 +58,7 @@ func HeldAppWhoClears(reason string) string { case "putback_pending": return "nobody — it clears itself when the drive with the pre-restore copy comes back" default: - return "a person only (support: the controller's -clear-restore-hold)" + return "a person — support on the box (--clear-restore-hold) or the operator here (Clear hold, D64)" } } diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index 97885f27..05063559 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -744,6 +744,8 @@ var recordOnlyEvents = map[string]bool{ // is customer-grade and would be DISCARDED by a template). Turning the fallback into a gate would // change behaviour well outside this concern. var operatorOnlyEvents = map[string]bool{ + // R-1344 (2026-10-29): the controller runs but does not answer (agent stanza, alarm only) — the operator's. + "controller_unresponsive": true, // R-97a. A customer can take no action on a failed whole-guest backup, and being told it failed // while it is still retrying behind the R-88 breaker is alarming without being actionable. "whole_guest_backup_failed": true, diff --git a/hub/internal/store/opactions.go b/hub/internal/store/opactions.go index e77b5d33..bfa104cb 100644 --- a/hub/internal/store/opactions.go +++ b/hub/internal/store/opactions.go @@ -3,8 +3,10 @@ package store import ( "database/sql" "fmt" + "regexp" "sort" "strconv" + "strings" "time" ) @@ -28,6 +30,14 @@ const ( OperatorActionAbandonStop = "abandon_stop" OperatorActionAbandonExtend = "abandon_extend" OperatorActionRunJob = "run_job" + // OperatorActionClearHold (D64 = B, operator 2026-10-29 12:10; `04` §6.1): the RUNNING controller clears one app's + // hold (any reason) and saves at once. No signature: the operator accepted that a compromised hub could release a + // data-protecting hold; the hub shows the hold's reason and a warning before the press. + OperatorActionClearHold = "clear_hold" + + // HostOperatorActionRestartController (operator brief 2026-10-29, Part 1): the AGENT restarts a guest's controller + // (the supervisor's own act and guards). The agent's list holds this one verb. + HostOperatorActionRestartController = "restart_controller" // Outcomes: the box's three, plus two the hub sets itself. OperatorActionDone = "done" @@ -50,7 +60,35 @@ const ( operatorActionsListed = 10 ) -var operatorActionNames = []string{OperatorActionOffsiteBackupNow, OperatorActionAbandonStop, OperatorActionAbandonExtend, OperatorActionRunJob} +var operatorActionNames = []string{OperatorActionOffsiteBackupNow, OperatorActionAbandonStop, OperatorActionAbandonExtend, OperatorActionRunJob, OperatorActionClearHold} + +// hostOperatorActionNames is the AGENT's closed list (the host report's reply). Pinned by +// TestHostOperatorActions_ClosedList; never shares a verb with the controller's list. +var hostOperatorActionNames = []string{HostOperatorActionRestartController} + +// stackNameRe is an app's stack name as the controller names it (a catalog slug). +var stackNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,63}$`) + +// HostOperatorActionNames returns the agent's closed list (sorted copy). +func HostOperatorActionNames() []string { return sortedStrings(hostOperatorActionNames) } + +// IsHostOperatorAction reports whether the verb belongs to the agent's list. +func IsHostOperatorAction(action string) bool { return inList(hostOperatorActionNames, action) } + +// ValidateHostOperatorAction refuses anything outside the agent's closed list. restart_controller takes the guest's +// vmid (Proxmox: 100–999999999). +func ValidateHostOperatorAction(action, arg string) error { + switch action { + case HostOperatorActionRestartController: + v, err := strconv.Atoi(arg) + if err != nil || strconv.Itoa(v) != arg || v < 100 || v > 999999999 { + return fmt.Errorf("the guest id must be a number 100-999999999") + } + default: + return fmt.Errorf("unknown host action %q", action) + } + return nil +} // operatorJobNames is the fixed set run_job may name — the controller's scheduler job names. var operatorJobNames = []string{"fill-watch", "offsite-integrity", "offsite-proof", "disk-health-check"} @@ -93,6 +131,10 @@ func ValidateOperatorAction(action, arg string) error { if !inList(operatorJobNames, arg) { return fmt.Errorf("unknown job %q", arg) } + case OperatorActionClearHold: + if !stackNameRe.MatchString(arg) { + return fmt.Errorf("not an app name: %q", arg) + } default: return fmt.Errorf("unknown action %q", action) } @@ -112,6 +154,7 @@ type OperatorActionDirective struct { type OperatorActionRow struct { ID int64 CustomerID string + HostID string // "" = a controller action; else the agent of this host acts Action string Arg string RequestedAt time.Time @@ -135,7 +178,83 @@ func (s *Store) migrateOperatorActions() error { message TEXT NOT NULL DEFAULT '' ); CREATE INDEX IF NOT EXISTS idx_operator_actions_customer ON operator_actions(customer_id, done_at);`) - return err + if err != nil { + return err + } + // The agent's rows (2026-10-29): host_id names the host whose agent acts; "" = the controller's row. + if _, err := s.db.Exec(`ALTER TABLE operator_actions ADD COLUMN host_id TEXT NOT NULL DEFAULT ''`); err != nil && + !strings.Contains(err.Error(), "duplicate column") { + return err + } + return nil +} + +// CreateHostOperatorAction validates against the AGENT's closed list and stores a pending row for hostID (listed on the +// customer's host page beside the controller's rows). +func (s *Store) CreateHostOperatorAction(customerID, hostID, action, arg, by string) (int64, error) { + if hostID == "" { + return 0, fmt.Errorf("host operator action: empty host id") + } + if err := ValidateHostOperatorAction(action, arg); err != nil { + return 0, err + } + res, err := s.db.Exec(`INSERT INTO operator_actions (customer_id, host_id, action, arg, requested_at, requested_by) + VALUES (?, ?, ?, ?, ?, ?)`, customerID, hostID, action, arg, time.Now().UTC(), by) + if err != nil { + return 0, err + } + return res.LastInsertId() +} + +// PendingHostOperatorActions is what the next host-report reply lists for hostID's agent, oldest first; rows older +// than OperatorActionTTL are closed as expired first. +func (s *Store) PendingHostOperatorActions(hostID string) ([]OperatorActionDirective, error) { + now := time.Now().UTC() + if _, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ? + WHERE host_id = ? AND done_at IS NULL AND requested_at < ?`, + now, OperatorActionExpired, "the agent did not answer within a day", hostID, now.Add(-OperatorActionTTL)); err != nil { + return nil, err + } + rows, err := s.db.Query(`SELECT id, action, arg FROM operator_actions + WHERE host_id = ? AND done_at IS NULL ORDER BY id LIMIT ?`, hostID, operatorActionsListed) + if err != nil { + return nil, err + } + defer rows.Close() + var out []OperatorActionDirective + for rows.Next() { + var d OperatorActionDirective + if err := rows.Scan(&d.ID, &d.Action, &d.Arg); err != nil { + return nil, err + } + out = append(out, d) + } + return out, rows.Err() +} + +// RecordHostOperatorActionResult closes a pending AGENT row with the agent's result — matched on the id AND the +// reporting host, so another host's (or the controller's) result changes nothing. +func (s *Store) RecordHostOperatorActionResult(hostID string, id int64, outcome, message string) (*OperatorActionRow, error) { + switch outcome { + case OperatorActionDone, OperatorActionRefused, OperatorActionFailed: + default: + return nil, fmt.Errorf("unknown outcome %q", outcome) + } + if hostID == "" { + return nil, nil + } + if r := []rune(message); len(r) > operatorActionMessageMax { + message = string(r[:operatorActionMessageMax]) + } + res, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ? + WHERE id = ? AND host_id = ? AND done_at IS NULL`, time.Now().UTC(), outcome, message, id, hostID) + if err != nil { + return nil, err + } + if n, _ := res.RowsAffected(); n == 0 { + return nil, nil + } + return s.getOperatorAction(id) } // CreateOperatorAction validates against the closed list and stores a pending row. by names who @@ -160,12 +279,12 @@ func (s *Store) CreateOperatorAction(customerID, action, arg, by string) (int64, func (s *Store) PendingOperatorActions(customerID string) ([]OperatorActionDirective, error) { now := time.Now().UTC() if _, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ? - WHERE customer_id = ? AND done_at IS NULL AND requested_at < ?`, + WHERE customer_id = ? AND host_id = '' AND done_at IS NULL AND requested_at < ?`, now, OperatorActionExpired, "the box did not answer within a day", customerID, now.Add(-OperatorActionTTL)); err != nil { return nil, err } rows, err := s.db.Query(`SELECT id, action, arg FROM operator_actions - WHERE customer_id = ? AND done_at IS NULL ORDER BY id LIMIT ?`, customerID, operatorActionsListed) + WHERE customer_id = ? AND host_id = '' AND done_at IS NULL ORDER BY id LIMIT ?`, customerID, operatorActionsListed) if err != nil { return nil, err } @@ -194,7 +313,7 @@ func (s *Store) RecordOperatorActionResult(customerID string, id int64, outcome, message = string(r[:operatorActionMessageMax]) } res, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ? - WHERE id = ? AND customer_id = ? AND done_at IS NULL`, time.Now().UTC(), outcome, message, id, customerID) + WHERE id = ? AND customer_id = ? AND host_id = '' AND done_at IS NULL`, time.Now().UTC(), outcome, message, id, customerID) if err != nil { return nil, err } @@ -205,7 +324,7 @@ func (s *Store) RecordOperatorActionResult(customerID string, id int64, outcome, } func (s *Store) getOperatorAction(id int64) (*OperatorActionRow, error) { - rows, err := s.db.Query(`SELECT id, customer_id, action, arg, requested_at, requested_by, done_at, outcome, message + rows, err := s.db.Query(`SELECT id, customer_id, host_id, action, arg, requested_at, requested_by, done_at, outcome, message FROM operator_actions WHERE id = ?`, id) if err != nil { return nil, err @@ -220,7 +339,7 @@ func (s *Store) getOperatorAction(id int64) (*OperatorActionRow, error) { // ListOperatorActions returns the customer's newest rows (pending and closed), newest first. func (s *Store) ListOperatorActions(customerID string, limit int) ([]OperatorActionRow, error) { - rows, err := s.db.Query(`SELECT id, customer_id, action, arg, requested_at, requested_by, done_at, outcome, message + rows, err := s.db.Query(`SELECT id, customer_id, host_id, action, arg, requested_at, requested_by, done_at, outcome, message FROM operator_actions WHERE customer_id = ? ORDER BY id DESC LIMIT ?`, customerID, limit) if err != nil { return nil, err @@ -234,7 +353,7 @@ func scanOperatorActions(rows *sql.Rows) ([]OperatorActionRow, error) { for rows.Next() { var r OperatorActionRow var done sql.NullTime - if err := rows.Scan(&r.ID, &r.CustomerID, &r.Action, &r.Arg, &r.RequestedAt, &r.RequestedBy, &done, &r.Outcome, &r.Message); err != nil { + if err := rows.Scan(&r.ID, &r.CustomerID, &r.HostID, &r.Action, &r.Arg, &r.RequestedAt, &r.RequestedBy, &done, &r.Outcome, &r.Message); err != nil { return nil, err } if done.Valid { diff --git a/hub/internal/store/opactions_test.go b/hub/internal/store/opactions_test.go index 4bd21323..031cd33d 100644 --- a/hub/internal/store/opactions_test.go +++ b/hub/internal/store/opactions_test.go @@ -11,7 +11,8 @@ import ( // The list is CLOSED, on the hub side too — this fails when an action or a job is added, by design. The // controller pins the same four (internal/report TestOpActions_ClosedList). func TestOperatorActions_ClosedList(t *testing.T) { - if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) { + // D64 = B (operator 2026-10-29 12:10): clear_hold joins the controller's list. + if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "clear_hold", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) { t.Fatalf("operator actions = %v, want exactly %v — a new action needs the operator's word (decision 185)", got, want) } if got, want := OperatorJobNames(), []string{"disk-health-check", "fill-watch", "offsite-integrity", "offsite-proof"}; !reflect.DeepEqual(got, want) { @@ -108,3 +109,84 @@ func TestOperatorActions_ExpireAndResetCancel(t *testing.T) { t.Fatalf("fresh row after reset = %+v, want cancelled (kept as a record)", r) } } + +// The AGENT's closed list (operator brief 2026-10-29, Part 1): one verb. RED-PROOF: seen failing before the list existed. +func TestHostOperatorActions_ClosedList(t *testing.T) { + if got, want := HostOperatorActionNames(), []string{"restart_controller"}; !reflect.DeepEqual(got, want) { + t.Fatalf("host operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want) + } + for _, a := range OperatorActionNames() { + if IsHostOperatorAction(a) { + t.Errorf("%s is on BOTH lists — a controller action must never reach the agent", a) + } + } +} + +// Each list refuses the other's verbs and every bad argument BEFORE storing. +func TestHostOperatorActions_RefusedBeforeStoring(t *testing.T) { + s := newTestStore(t) + bad := []struct{ action, arg string }{ + {"restart_controller", ""}, {"restart_controller", "abc"}, {"restart_controller", "99"}, {"restart_controller", "9201 "}, + {"restart_controller", "-1"}, {"restart_controller", "10000000000"}, {"reboot_guest", "9201"}, {"clear_hold", "radicale"}, + {"offsite_backup_now", ""}, + } + for _, b := range bad { + if _, err := s.CreateHostOperatorAction("c1", "h1", b.action, b.arg, "operator"); err == nil { + t.Errorf("host action %s(%q) was accepted", b.action, b.arg) + } + } + for _, b := range []struct{ action, arg string }{{"clear_hold", ""}, {"clear_hold", "../x"}, {"clear_hold", "Rad icale"}, + {"clear_hold", "a/b"}, {"restart_controller", "9201"}} { + if _, err := s.CreateOperatorAction("c1", b.action, b.arg, "operator"); err == nil { + t.Errorf("controller action %s(%q) was accepted", b.action, b.arg) + } + } + if _, err := s.CreateOperatorAction("c1", "clear_hold", "radicale", "operator"); err != nil { + t.Fatalf("clear_hold radicale refused: %v", err) + } + if _, err := s.CreateHostOperatorAction("c1", "h1", "restart_controller", "9201", "operator"); err != nil { + t.Fatalf("restart_controller 9201 refused: %v", err) + } +} + +// An agent row reaches only its host's envelope, a controller row only the customer's ACK; a result closes only a row of +// the reporting host. RED-PROOF: drop `AND host_id = ”` from PendingOperatorActions → the restart reaches the controller. +func TestHostOperatorActions_EachSideSeesOnlyItsOwn(t *testing.T) { + s := newTestStore(t) + cid, err := s.CreateOperatorAction("c1", "clear_hold", "radicale", "operator") + if err != nil { + t.Fatal(err) + } + hid, err := s.CreateHostOperatorAction("c1", "h1", "restart_controller", "9201", "operator") + if err != nil { + t.Fatal(err) + } + ctl, _ := s.PendingOperatorActions("c1") + if len(ctl) != 1 || ctl[0].ID != cid { + t.Fatalf("the controller ACK lists %+v, want only #%d", ctl, cid) + } + agent, _ := s.PendingHostOperatorActions("h1") + if len(agent) != 1 || agent[0].ID != hid || agent[0].Action != "restart_controller" || agent[0].Arg != "9201" { + t.Fatalf("the agent envelope lists %+v, want only #%d", agent, hid) + } + if other, _ := s.PendingHostOperatorActions("h2"); len(other) != 0 { + t.Fatalf("another host sees %+v", other) + } + if row, _ := s.RecordHostOperatorActionResult("h2", hid, "done", "x"); row != nil { + t.Fatal("another host closed the row") + } + if row, _ := s.RecordOperatorActionResult("c1", hid, "done", "x"); row != nil { + t.Fatal("the controller's result channel closed an agent row") + } + row, err := s.RecordHostOperatorActionResult("h1", hid, "done", "the controller answered after 14 s") + if err != nil || row == nil || row.Outcome != "done" || row.HostID != "h1" { + t.Fatalf("result: %+v %v", row, err) + } + if left, _ := s.PendingHostOperatorActions("h1"); len(left) != 0 { + t.Fatalf("a closed row is still listed: %+v", left) + } + list, _ := s.ListOperatorActions("c1", 10) + if len(list) != 2 { + t.Fatalf("the host page lists %d rows, want both", len(list)) + } +} diff --git a/hub/internal/web/held_apps.go b/hub/internal/web/held_apps.go index ec76039a..e9f1658e 100644 --- a/hub/internal/web/held_apps.go +++ b/hub/internal/web/held_apps.go @@ -18,6 +18,18 @@ type heldAppRow struct { WhoClears string Reminded bool Stale bool // the box has not repeated it for heldAppStale — it may be down + // ProtectsData (D64 = B): a restore or put-back hold — the confirm step warns before „Clear hold" (`07` §8). + ProtectsData bool +} + +// holdProtectsData: the reasons whose hold keeps an app off half-restored data (`07` §8): a failed restore (the +// reason is "" or restore_failed on older rows), a mixed restore, a put-back waiting for its drive. +func holdProtectsData(reason string) bool { + switch reason { + case "", "restore_failed", "restore_mixed", "putback_pending": + return true + } + return false } // heldAppStaleAfter mirrors monitor's heldAppStale: a box repeats app_held daily. @@ -51,7 +63,7 @@ func toHeldAppRows(held []store.HeldApp, hostOf map[string]string, now time.Time } out = append(out, heldAppRow{CustomerID: h.CustomerID, HostID: hostOf[h.CustomerID], Stack: h.Stack, Reason: h.Reason, Since: since, Why: h.Why, WhoClears: monitor.HeldAppWhoClears(h.Reason), Reminded: !h.RemindedAt.IsZero(), - Stale: now.Sub(h.LastSeen) > heldAppStaleAfter}) + Stale: now.Sub(h.LastSeen) > heldAppStaleAfter, ProtectsData: holdProtectsData(h.Reason)}) } return out } diff --git a/hub/internal/web/host_opactions_test.go b/hub/internal/web/host_opactions_test.go new file mode 100644 index 00000000..47a48c71 --- /dev/null +++ b/hub/internal/web/host_opactions_test.go @@ -0,0 +1,94 @@ +package web + +import ( + "net/http" + "net/url" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// Operator brief 2026-10-29: „Restart controller" (the AGENT's closed list) and „Clear hold" (D64 = B, the +// controller's list) on the host page — each behind a confirm step, CSRF like every form. + +func heldFixture(t *testing.T) (*Server, *store.Store) { + t.Helper() + s, st := newTestServer(t) + if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k"}); err != nil { + t.Fatal(err) + } + now := time.Now() + if err := st.UpsertGuestFromReport(&store.Guest{GuestID: "h1-9201", CustomerID: "c1", HostID: "h1", VMID: 9201, DisplayName: "felhom", + Status: "running", ControllerVersion: "0.347.0", LastSeenAt: &now}); err != nil { + t.Fatal(err) + } + if err := st.UpsertHeldApp("c1", "radicale", "restore_mixed", now.Add(-time.Hour), "a restore stopped half-way", now); err != nil { + t.Fatal(err) + } + if err := st.UpsertHeldApp("c1", "gokapi", "unhealthy_stop", now.Add(-time.Hour), "stopped in a crash loop", now); err != nil { + t.Fatal(err) + } + return s, st +} + +// RED-PROOF: seen failing before the forms existed. +func TestHostPage_RestartAndClearHoldRenderBehindAConfirm(t *testing.T) { + s, _ := heldFixture(t) + body := renderHost(t, s, "h1") + for _, want := range []string{ + `data-restart-controller="9201"`, `name="action" value="restart_controller"`, `name="arg" value="9201"`, + "Restart the controller of guest 9201 on h1 (c1)?", + `data-clear-hold="radicale"`, `name="action" value="clear_hold"`, `name="arg" value="radicale"`, + "This hold protects data from a failed restore. If you clear it, the app may start on half-restored data. Check the restore copy first.", + `data-clear-hold="gokapi"`, + } { + if !strings.Contains(body, want) { + t.Errorf("host page lacks %q", want) + } + } + // the warning only on the data-protecting hold + gk := body[strings.Index(body, `data-clear-hold="gokapi"`):] + gk = gk[:strings.Index(gk, "")] + if strings.Contains(gk, "half-restored") { + t.Error("a crash-loop hold shows the restore warning") + } + if !strings.Contains(gk, "unhealthy_stop") { + t.Error("the confirm step does not name the hold's reason") + } +} + +// restart_controller: a row for the AGENT of this host (never the controller's ACK); clear_hold only for an app that +// IS held; every other verb or argument refused with no row. RED-PROOF: seen failing before the handler routed them. +func TestHostPage_RestartAndClearHoldPosts(t *testing.T) { + s, st := heldFixture(t) + if rr := postOperatorAction(t, s, "h1", url.Values{"action": {"restart_controller"}, "arg": {"9201"}}); rr.Code != http.StatusSeeOther { + t.Fatalf("restart: %d %s", rr.Code, rr.Body.String()) + } + if a, _ := st.PendingHostOperatorActions("h1"); len(a) != 1 || a[0].Arg != "9201" { + t.Fatalf("the agent's reply would list %+v", a) + } + if c, _ := st.PendingOperatorActions("c1"); len(c) != 0 { + t.Fatalf("the restart reached the controller's ACK: %+v", c) + } + if rr := postOperatorAction(t, s, "h1", url.Values{"action": {"clear_hold"}, "arg": {"radicale"}}); rr.Code != http.StatusSeeOther { + t.Fatalf("clear_hold: %d %s", rr.Code, rr.Body.String()) + } + if c, _ := st.PendingOperatorActions("c1"); len(c) != 1 || c[0].Action != "clear_hold" || c[0].Arg != "radicale" { + t.Fatalf("the controller's ACK would list %+v", c) + } + for _, f := range []url.Values{ + {"action": {"clear_hold"}, "arg": {"vikunja"}}, // not held + {"action": {"restart_controller"}, "arg": {"9202"}}, // not a guest of this host + {"action": {"restart_controller"}, "arg": {"abc"}}, + {"action": {"reboot_guest"}, "arg": {"9201"}}, + } { + if rr := postOperatorAction(t, s, "h1", f); rr.Code != http.StatusBadRequest { + t.Errorf("%v: status %d, want 400", f, rr.Code) + } + } + if rows, _ := st.ListOperatorActions("c1", 20); len(rows) != 2 { + t.Fatalf("rows = %d, want the two accepted presses only", len(rows)) + } +} diff --git a/hub/internal/web/hosts_test.go b/hub/internal/web/hosts_test.go index ae885535..2efb07b8 100644 --- a/hub/internal/web/hosts_test.go +++ b/hub/internal/web/hosts_test.go @@ -174,11 +174,17 @@ func TestHandleHostDetail(t *testing.T) { // Decision 185 (2026-10-08): plus the four operator-action buttons — a closed list, none of which // deletes or mutates the host (TestOperatorActions_* pin what they may do). Every OTHER button // remains absent: 2 log requests + 4 operator actions, and the 4 all post to /operator-action. - if got := strings.Count(strings.ToLower(body), "{{.VMID}} {{if .DisplayName}}{{.DisplayName}}{{else}}—{{end}} {{if eq .Status "running"}}{{.Status}}{{else if eq .Status "stopped"}}{{.Status}}{{else}}{{.Status}}{{end}} - {{if .ControllerVersion}}{{.ControllerVersion}}{{else}}—{{end}} + {{if .ControllerVersion}}{{.ControllerVersion}}{{else}}—{{end}} + {{if and .ControllerVersion $.CustomerID}}{{/* 2026-10-29: the AGENT restarts it (the supervisor's act and guards; 3 in 15 min, then refused for 30) — works with the controller stopped */}} +
+ Restart controller +
+ + + +

Restart the controller of guest {{.VMID}} on {{$.HostID}} ({{$.CustomerID}})? The box's agent restarts it, even when the controller does not answer. The household's dashboard is away for about a minute. At most 3 restarts in 15 minutes; the result shows under Operator Actions.

+ +
+
{{end}} + {{timeAgoPtr .LastSeenAt}} {{end}} @@ -146,7 +158,8 @@ {{end}} - +

Held apps

{{if .HeldApps}} @@ -167,7 +180,19 @@ {{.Reason}}{{if .Stale}} (not confirmed by the box for 2 days){{end}} {{timeAgo .Since}} {{.Why}} - {{.WhoClears}}{{if .Reminded}} (operator reminded){{end}} + {{.WhoClears}}{{if .Reminded}} (operator reminded){{end}} +
+ Clear hold +
+ + + +

Clear the hold of {{.Stack}} ({{.Reason}}, since {{.Since.Format "2006-01-02 15:04"}})? The box's controller clears it at once; the household can then start the app.

+ {{if .ProtectsData}}

This hold protects data from a failed restore. If you clear it, the app may start on half-restored data. Check the restore copy first.

{{end}} + +
+
+ {{end}} diff --git a/hub/internal/web/templates/style.css b/hub/internal/web/templates/style.css index b7050dc2..e27752c5 100644 --- a/hub/internal/web/templates/style.css +++ b/hub/internal/web/templates/style.css @@ -1031,3 +1031,10 @@ body.js-tabs .tab-panel:not(.tab-panel-active) { display: none; } } .mail-hold-release .text-muted { font-size: 0.8rem; } #mail-hold a { color: var(--text-1); } + +/* 2026-10-29: the confirm fold-out of „Restart controller" and „Clear hold" (no script; the final button is inside). */ +.op-confirm { margin-top: 0.35rem; } +.op-confirm summary { list-style: none; display: inline-block; cursor: pointer; } +.op-confirm form { margin-top: 0.5rem; max-width: 34rem; } +.op-confirm p { font-size: 0.85rem; margin: 0.25rem 0 0.5rem; } +.op-warning { color: var(--warn); } diff --git a/scripts/wire_contract_gate.py b/scripts/wire_contract_gate.py index f217f2b5..8353324a 100644 --- a/scripts/wire_contract_gate.py +++ b/scripts/wire_contract_gate.py @@ -106,6 +106,9 @@ ROOTS = [ # ACK's operator_actions list. Named type for the same reason as R-311's root above. ("hub -> controller (report ACK, `operator_actions` entry)", "hub", "internal/store", "OperatorActionDirective", "controller"), + # 2026-10-29 (operator brief, Part 1): the AGENT's operator actions — the same entry type on the host-report reply. + ("hub -> agent (host-report reply, `operator_actions` entry)", + "hub", "internal/store", "OperatorActionDirective", "agent"), # R-922 (2026-10-09): the notification-prefs push. Declared when the household's deliberate clear # (`email_cleared`) joined it — a flag the hub cannot decode would leave the cleared address stored. ("controller -> hub (POST /preferences)", @@ -124,6 +127,7 @@ MIRRORS = { "hub -> agent (GET /hosts//escrow/retained)": ("internal/hub", "RetainedEscrowResponse"), "hub -> controller (report ACK, `escrow` object)": ("internal/report", "EscrowStatus"), "hub -> controller (report ACK, `operator_actions` entry)": ("internal/report", "OperatorAction"), + "hub -> agent (host-report reply, `operator_actions` entry)": ("internal/hub", "OperatorAction"), } # R-444 (2026-10-06): a SUBTREE of a name-checked root whose receiver decodes it into one named mirror type gets the @@ -136,6 +140,8 @@ MIRRORS = { SUBTREE_MIRRORS = { ("agent -> hub (POST /host-report)", "guest_disk_trim"): ("internal/sysfacts", "DiskTrim"), ("agent -> hub (POST /host-report)", "line_test"): ("internal/sysfacts", "LineTest"), + # 2026-10-29: the agent's answers to the operator's actions (hub api.hostOperatorActionResult). + ("agent -> hub (POST /host-report)", "operator_action_results"): ("internal/api", "hostOperatorActionResult"), } # Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.