hub: Restart controller (the agent's closed list on the host-report reply) and Clear hold (D64 = B, the controller's list) on the host page, each behind a confirm; controller_unresponsive (R-1344, alarm only); 04 §6.1 + 09 §3 decision 283; wire gate declares both new wires; R-1344 filed — tests red first
gates / gates (push) Successful in 3m23s

This commit is contained in:
admin committed 2026-10-29 12:31:33 +01:00
1 parent 70862fa7b3
commit 3d0c907b56
20 files changed
+663 -20

No files matched your search

@@ -183,6 +183,18 @@ the deletion is the hub's), `run_job` (`fill-watch`, `offsite-integrity`, `offsi
Anything else is refused at the hub's POST and again on the box. No action deletes data, starts a countdown or
shortens one (`TestOpActions_ClosedList`, `TestOperatorActions_ClosedList`). Unanswered after 24 h: expired.
**`[DESIGN]` 2026-10-29 — D64 = B, and the agent's own list (operator ruling 12:10; `09` §3 decision 283).** The
controller's list gains **`clear_hold <app>`** for every hold reason, with no signature: the RUNNING controller clears the
hold in memory and saves at once (no later settings save can write it back — R-1343). The operator accepted the cost in
words: a compromised hub could release a data-protecting hold and start an app on half-restored data; the hub's confirm
step shows the hold's reason and, for a restore or put-back hold, that warning. **R-379 („only a person clears a hold")
now reads „a person — on the box, or the operator on the hub".** The hub refuses `clear_hold` for an app the box does not
report held now. **The AGENT has its own closed list on the host report's reply — `restart_controller <vmid>` only**
(`store.ValidateHostOperatorAction` / agent `localapi.OperatorActionNames`; never a shared verb, `TestHostOperatorActions_ClosedList`):
the supervisor's restart with its guards, the 3-in-15-minutes brake counting operator restarts too (then 30 minutes
refused), the result in the next report (requested at once), a vmid that is not a guest of that host refused at the hub.
Neither verb deletes data, starts a countdown or shortens one. A guest reboot from the hub is NOT on either list.
### 6.2 Who may press the operator's buttons: session, Basic, or a scoped token [hub v0.152.0, `09` §3 decision 251]
The hub's operator surface has three ways in: a browser session, Basic auth with the operator password (scripts,
@@ -1206,6 +1206,14 @@ Wanderer's Meilisearch went 256 → 384 MB and was measured again (it sizes itse
`-tags faultinject` test build — never in the release image (`TestD63_TheReleaseBuildHasNoFaultSwitch`). *Operator,
2026-10-29 08:17.* `07` §6.3.
### 2026-10-29 (12:10, operator) — D64 = B: the hub may clear any hold
283. **D64 = B — the hub's „Clear hold" works for every hold reason, with no operator signature.** The operator accepted
that a compromised hub could clear a data-protecting hold (an app could start on half-restored data); the confirm
step shows the hold's reason and, for a restore or put-back hold, that warning before the press. R-379 now reads
„a person — on the box, or the operator on the hub". Built as the controller's `clear_hold` operator action
(`04` §6.1), cleared by the running controller (R-1343). *Operator, 2026-10-29 12:10.*
### 2026-10-28 (day, attended) — RomM kept; the runtime cache fills itself in CI
277. **D61 = keep RomM offered** (R-1330) while no RomM release carries Node ≥ 24.22.1; the first one that does is taken
+2 -1
View File
@@ -293,7 +293,7 @@ stopping line that lies.
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
## Monitoring & notifications — 24 rows (P2 2, P3 14, P4 8)
## Monitoring & notifications — 25 rows (P2 2, P3 15, P4 8)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -321,6 +321,7 @@ stopping line that lies.
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
| **R-1341** | Monitoring & notifications | P4 | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** Measured 2026-10-29 on Tester 1 (controller 0.347.0, agent 0.168.0, R-528's proof): one kill → one `app_oom` and one operator mail (right), and the `[WARN] [deadapp]` line at 09:02:32, :03:02, :03:32, :04:02, :04:32 … — the same as the night's proof (`audits/night-2026-10-28/r528-oom/`). The loop (`controller/cmd/controller/main.go:1100`, since v0.243.0, R-514) logs every container `ScanOOMKilled` returns, every scan; the notifier's cooldown keeps the event to one. About 2 880 lines a day per OOM-marked container until it restarts — noise that hides the next real line. Fix: log once per (container, start time, kill count), with a test that a second scan of the same kill logs nothing and a new kill logs again. Evidence: `audits/d63-2026-10-29/run-p4-oom.log`. | OPEN | — | the fix above, red first | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
## Hub & operator — 12 rows (P3 3, P4 9)
+20
View File
@@ -1,3 +1,23 @@
## Unreleased (2026-10-29, afternoon) — „Restart controller" and „Clear hold" on the host page (D64 = B); controller_unresponsive (R-1344) — commit only; rides the next hub release
- **The AGENT's operator actions (operator brief 2026-10-29, Part 1; `04` §6.1).** `operator_actions` gains a `host_id`
column: a row with a host id is the AGENT's (the host-report reply lists it as `operator_actions`; the agent answers
`operator_action_results` on its next report, closed per id AND host, one `operator_action` event each); a row without
is the controller's, as before. The agent's CLOSED list is `restart_controller <vmid>` only
(`store.ValidateHostOperatorAction`), and a vmid must be a guest of THAT host. The host page's guest table has
„Restart controller" behind a confirm step that names the box; the press pokes the agent (seconds over the tunnel; the
15-min cycle otherwise). Tests `TestHostOperatorActions_*` (store), `TestHostReport_AgentOperatorActionRoundTrip` (api),
`TestHostPage_RestartAndClearHold*` (web) — red first; the host-row filter and both web guards red by mutation.
- **D64 = B (operator 2026-10-29 12:10) — `clear_hold` joins the controller's closed list**, for every hold reason, no
signature. The Held apps list has „Clear hold" per row behind a confirm step showing the app, the reason and since when;
a restore or put-back hold adds: „This hold protects data from a failed restore. If you clear it, the app may start on
half-restored data. Check the restore copy first." Refused (400, no row) for an app the box does not report held now.
„Who may clear it" now names the operator here too. `TestOperatorActions_ClosedList` updated deliberately.
- **R-1344 — `controller_unresponsive`** (warning, operator-only, hub-generated like the crash-loop pair) when a guest's
`unresponsive_since` moves in the agent's `controller_supervisor` stanza (also at first sight while unresponsive). The
agent only alarms; it does not restart for it yet. Tests `TestControllerSupervisorChecker_Unresponsive`,
`TestControllerUnresponsive_RegisteredAndOperatorOnly` — red first.
## v0.175.0 (2026-10-29) — R-1255: two escrow edges (review round 2, HUNT-3 H3/H4); D62: the restore-copy delete event — DEPLOYED on the operator's yes (image sha256:642e94b6…, from `bf69bb91`, CI 2487)
- **D62 (operator 2026-10-29) — `restore_copy_deleted` is accepted and operator-only.** The controller's „Törlés most"
+20
View File
@@ -809,6 +809,16 @@ type hostReportPayload struct {
DeliberateOff *struct {
RequestedAt string `json:"requested_at"`
} `json:"deliberate_off"`
// OperatorActionResults (2026-10-29, the agent's operator actions): the answers to the previous replies'
// operator_actions — the controller's shape. Absent on an older agent.
OperatorActionResults []hostOperatorActionResult `json:"operator_action_results"`
}
// hostOperatorActionResult mirrors the agent's hub.OperatorActionResult.
type hostOperatorActionResult struct {
ID int64 `json:"id"`
Outcome string `json:"outcome"`
Message string `json:"message"`
}
// hostForeignKeyArchives mirrors the agent's hub.ForeignKeyArchives (R-366 slice 2).
@@ -1133,6 +1143,14 @@ func (h *Handler) handleHostReport(w http.ResponseWriter, r *http.Request) {
if pending, err := h.store.PendingLogBundleRequest(hostID, store.LogBundleComponentAgent); err == nil && pending {
resp["log_tail_requested"] = true
}
// The agent's operator actions (2026-10-29): the results this report carries close their rows (one event each);
// what is still pending rides this reply. A store error drops nothing from the heartbeat.
h.ingestHostOperatorActionResults(hostID, custID, rep.OperatorActionResults)
if acts, err := h.store.PendingHostOperatorActions(hostID); err != nil {
h.logger.Printf("[WARN] host operator actions for %s: %v", hostID, err)
} else if len(acts) > 0 {
resp["operator_actions"] = acts
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(resp)
@@ -2400,6 +2418,8 @@ var allowedEventTypes = map[string]bool{
"controller_crashloop": true,
// R-539 (hub v0.117.0, agent v0.132.0) — hub-generated from slow_crashloop_since; operator-only.
"controller_slow_crashloop": true,
// R-1344 (2026-10-29): hub-generated from the stanza's unresponsive_since; operator-only (notify.operatorOnlyEvents).
"controller_unresponsive": true,
// R-550 (hub v0.117.0, controller v0.246.0) — pushed by the controller at startup when it finds a
// restore that was in flight when the box stopped. For the household: they can run it again.
"restore_interrupted": true,
+86
View File
@@ -0,0 +1,86 @@
package api
import (
"encoding/json"
"net/http"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// Operator brief 2026-10-29, Part 1: the AGENT's operator actions ride the host-report reply as `operator_actions`
// (the controller's shape) and its answer rides the next host report as `operator_action_results`; the row closes and
// one `operator_action` event is stored. A controller row never reaches the agent. RED-PROOF: seen failing before the
// reply listed the agent rows (no `operator_actions` key).
func TestHostReport_AgentOperatorActionRoundTrip(t *testing.T) {
h, st, _ := newTestHandler(t)
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
if _, err := st.CreateOperatorAction("c1", "clear_hold", "radicale", "operator"); err != nil {
t.Fatal(err)
}
id, err := st.CreateHostOperatorAction("c1", "h1", "restart_controller", "9201", "operator browser session from 192.168.0.5")
if err != nil {
t.Fatal(err)
}
rr := do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1"))
var env struct {
OperatorActions []store.OperatorActionDirective `json:"operator_actions"`
}
if err := json.Unmarshal(rr.Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if len(env.OperatorActions) != 1 || env.OperatorActions[0].ID != id || env.OperatorActions[0].Action != "restart_controller" ||
env.OperatorActions[0].Arg != "9201" {
t.Fatalf("the agent's reply lists %+v, want only the restart #%d (never the controller's clear_hold)", env.OperatorActions, id)
}
body := strings.Replace(validReportBody("h1"), `"audit_tail":[]`,
`"audit_tail":[],"operator_action_results":[{"id":`+i64s(id)+`,"outcome":"done","message":"the controller answered after 14 s"}]`, 1)
if rr := do(h, http.MethodPost, "/host-report", "HKEY", body); rr.Code != http.StatusOK {
t.Fatalf("result heartbeat = %d %s", rr.Code, rr.Body.String())
}
if left, _ := st.PendingHostOperatorActions("h1"); len(left) != 0 {
t.Fatalf("the row is still pending: %+v", left)
}
rows, _ := st.ListOperatorActions("c1", 10)
var got *store.OperatorActionRow
for i := range rows {
if rows[i].ID == id {
got = &rows[i]
}
}
if got == nil || got.Outcome != "done" || got.Message != "the controller answered after 14 s" {
t.Fatalf("row = %+v", got)
}
evs, _ := st.GetEventsByType("c1", "operator_action", time.Time{})
n := 0
for _, e := range evs {
if e.EventType == "operator_action" && strings.Contains(e.Message, "restart_controller 9201") {
n++
}
}
if n != 1 {
t.Fatalf("operator_action events for the restart = %d, want 1: %+v", n, evs)
}
// once answered, the reply stops listing it
rr = do(h, http.MethodPost, "/host-report", "HKEY", validReportBody("h1"))
env.OperatorActions = nil
_ = json.Unmarshal(rr.Body.Bytes(), &env)
if len(env.OperatorActions) != 0 {
t.Fatalf("still listed after its result: %+v", env.OperatorActions)
}
}
func i64s(n int64) string { b, _ := json.Marshal(n); return string(b) }
// R-1344: the hub-generated controller_unresponsive is listed (the crash-loop pair's rule) and operator-only.
func TestControllerUnresponsive_RegisteredAndOperatorOnly(t *testing.T) {
if !allowedEventTypes["controller_unresponsive"] {
t.Error("controller_unresponsive is not in allowedEventTypes")
}
if !notify.IsOperatorOnly("controller_unresponsive") {
t.Error("controller_unresponsive is not operator-only")
}
}
+30
View File
@@ -55,3 +55,33 @@ func argSuffix(arg string) string {
}
return " " + arg
}
// ingestHostOperatorActionResults closes each answered AGENT row (matched on the id AND this host) and saves one
// hub-minted operator_action event per closed row — the controller's rule, on the agent's channel.
func (h *Handler) ingestHostOperatorActionResults(hostID, customerID string, results []hostOperatorActionResult) {
for _, res := range results {
row, err := h.store.RecordHostOperatorActionResult(hostID, res.ID, res.Outcome, res.Message)
switch {
case err != nil:
h.logger.Printf("[WARN] host operator action #%d result from %s not recorded: %v", res.ID, hostID, err)
continue
case row == nil:
h.logger.Printf("[DEBUG] host operator action #%d result from %s ignored (already closed, or not this host's)", res.ID, hostID)
continue
}
h.logger.Printf("[INFO] host operator action #%d %s%s on %s (pressed by %s): %s — %s",
row.ID, row.Action, argSuffix(row.Arg), hostID, row.RequestedBy, row.Outcome, row.Message)
if customerID == "" {
continue
}
severity := "info"
if row.Outcome != store.OperatorActionDone {
severity = "warning"
}
if _, eerr := h.store.SaveEvent(customerID, "operator_action", severity,
fmt.Sprintf("Operator action %s%s on host %s (pressed by %s): %s — %s", row.Action, argSuffix(row.Arg), hostID, row.RequestedBy, row.Outcome, row.Message),
"", "hub"); eerr != nil {
h.logger.Printf("[WARN] host operator action #%d recorded, but its event could not be saved: %v", row.ID, eerr)
}
}
}
+18 -1
View File
@@ -42,6 +42,7 @@ type supSeen struct {
lastRestartAt string
crashloopSince string
slowCrashloopSince string
unresponsiveSince string
}
// ControllerSupervisorGuest mirrors the agent's hub.ControllerSupervisorGuest wire shape.
@@ -57,6 +58,10 @@ type ControllerSupervisorGuest struct {
Restarts24h int `json:"restarts_24h"`
SlowCrashloop bool `json:"slow_crashloop"`
SlowCrashloopSince string `json:"slow_crashloop_since"`
// R-1344 (agent ≥ the 2026-10-29 release): the controller runs but Docker's health check has failed for several
// sweeps. ALARM ONLY — no restart until the measurement licenses one. Absent on older agents → never an event.
Unresponsive bool `json:"unresponsive"`
UnresponsiveSince string `json:"unresponsive_since"`
}
// ParseControllerSupervisor extracts the stanza from a host report body. A missing or malformed
@@ -98,7 +103,8 @@ func (c *ControllerSupervisorChecker) observe(hostID, customerID string, guests
for _, g := range guests {
key := fmt.Sprintf("%s/%d", hostID, g.VMID)
prev, known := c.seen[key]
c.seen[key] = supSeen{lastRestartAt: g.LastRestartAt, crashloopSince: g.CrashloopSince, slowCrashloopSince: g.SlowCrashloopSince}
c.seen[key] = supSeen{lastRestartAt: g.LastRestartAt, crashloopSince: g.CrashloopSince, slowCrashloopSince: g.SlowCrashloopSince,
unresponsiveSince: g.UnresponsiveSince}
if !known {
if g.Crashloop && g.CrashloopSince != "" {
c.emit(customerID, hostID, g, "controller_crashloop")
@@ -106,6 +112,9 @@ func (c *ControllerSupervisorChecker) observe(hostID, customerID string, guests
if g.SlowCrashloop && g.SlowCrashloopSince != "" {
c.emit(customerID, hostID, g, "controller_slow_crashloop")
}
if g.Unresponsive && g.UnresponsiveSince != "" {
c.emit(customerID, hostID, g, "controller_unresponsive")
}
continue
}
if g.LastRestartAt != "" && g.LastRestartAt != prev.lastRestartAt {
@@ -117,6 +126,9 @@ func (c *ControllerSupervisorChecker) observe(hostID, customerID string, guests
if g.SlowCrashloopSince != "" && g.SlowCrashloopSince != prev.slowCrashloopSince {
c.emit(customerID, hostID, g, "controller_slow_crashloop")
}
if g.UnresponsiveSince != "" && g.UnresponsiveSince != prev.unresponsiveSince {
c.emit(customerID, hostID, g, "controller_unresponsive")
}
}
}
@@ -135,6 +147,10 @@ func (c *ControllerSupervisorChecker) emit(customerID, hostID string, g Controll
severity = "warning"
message = fmt.Sprintf("Host %s guest %d: the controller keeps dying — the agent restarted it %d times in 24 hours (each one too far apart for the 15-minute brake). It is still being restarted; this is the warning that it will not stay up. Last reason: %s",
hostID, g.VMID, g.Restarts24h, g.LastReason)
case "controller_unresponsive":
severity = "warning"
message = fmt.Sprintf("Host %s guest %d: the controller runs but does not answer its health check (since %s). The agent does NOT restart it yet (R-1344: the measurement must first show a busy night never looks like this) — use „Restart controller\" on the host page if it stays so.",
hostID, g.VMID, g.UnresponsiveSince)
default:
return
}
@@ -143,6 +159,7 @@ func (c *ControllerSupervisorChecker) emit(customerID, hostID string, g Controll
"last_restart_at": g.LastRestartAt, "last_reason": g.LastReason,
"crashloop_since": g.CrashloopSince, "parked": g.Parked,
"restarts_24h": g.Restarts24h, "slow_crashloop_since": g.SlowCrashloopSince,
"unresponsive_since": g.UnresponsiveSince,
})
c.logger.Printf("[INFO] Controller supervisor: %s %s/%d (%s)", eventType, hostID, g.VMID, g.LastReason)
if _, err := c.store.SaveEvent(customerID, eventType, severity, message, string(details), "hub"); err != nil {
@@ -146,3 +146,32 @@ func TestControllerSupervisorChecker_SlowCrashloopAtFirstSight(t *testing.T) {
t.Fatalf("first sight in a slow crash loop: got %v, want exactly [controller_slow_crashloop/warning]", evs)
}
}
// R-1344 (operator brief 2026-10-29, Part 2 — ALARM ONLY until the measurement licenses a restart): a controller that
// runs but does not answer reaches the operator once per new unresponsive_since, as controller_unresponsive (warning),
// also at first sight; an answering one never. RED-PROOF: seen failing before the branch existed.
func TestControllerSupervisorChecker_Unresponsive(t *testing.T) {
st := newCapStore(t)
rec := &evRec{}
c := NewControllerSupervisorChecker(st, rec.fn, log.New(io.Discard, "", 0))
st.SaveHostReport("h1", "c1", supReport(`[{"vmid":9201,"restarts_total":0,"crashloop":false,"parked":false,"unresponsive":false}]`), store.HostReportDenorm{})
c.Check()
if evs := rec.take(); len(evs) != 0 {
t.Fatalf("an answering controller emitted %v", evs)
}
st.SaveHostReport("h1", "c1", supReport(`[{"vmid":9201,"restarts_total":0,"crashloop":false,"parked":false,"unresponsive":true,"unresponsive_since":"2026-10-29T13:00:00Z"}]`), store.HostReportDenorm{})
c.Check()
if evs := rec.take(); len(evs) != 1 || evs[0] != "controller_unresponsive/warning" {
t.Fatalf("got %v, want exactly [controller_unresponsive/warning]", evs)
}
c.Check()
if evs := rec.take(); len(evs) != 0 {
t.Fatalf("the same unresponsive_since emitted again: %v", evs)
}
// a hub restarted while it is unresponsive says so once
c2 := NewControllerSupervisorChecker(st, rec.fn, log.New(io.Discard, "", 0))
c2.Check()
if evs := rec.take(); len(evs) != 1 || evs[0] != "controller_unresponsive/warning" {
t.Fatalf("first sight while unresponsive: %v", evs)
}
}
+1 -1
View File
@@ -58,7 +58,7 @@ func HeldAppWhoClears(reason string) string {
case "putback_pending":
return "nobody — it clears itself when the drive with the pre-restore copy comes back"
default:
return "a person only (support: the controller's -clear-restore-hold)"
return "a person — support on the box (--clear-restore-hold) or the operator here (Clear hold, D64)"
}
}
+2
View File
@@ -744,6 +744,8 @@ var recordOnlyEvents = map[string]bool{
// is customer-grade and would be DISCARDED by a template). Turning the fallback into a gate would
// change behaviour well outside this concern.
var operatorOnlyEvents = map[string]bool{
// R-1344 (2026-10-29): the controller runs but does not answer (agent stanza, alarm only) — the operator's.
"controller_unresponsive": true,
// R-97a. A customer can take no action on a failed whole-guest backup, and being told it failed
// while it is still retrying behind the R-88 breaker is alarming without being actionable.
"whole_guest_backup_failed": true,
+127 -8
View File
@@ -3,8 +3,10 @@ package store
import (
"database/sql"
"fmt"
"regexp"
"sort"
"strconv"
"strings"
"time"
)
@@ -28,6 +30,14 @@ const (
OperatorActionAbandonStop = "abandon_stop"
OperatorActionAbandonExtend = "abandon_extend"
OperatorActionRunJob = "run_job"
// OperatorActionClearHold (D64 = B, operator 2026-10-29 12:10; `04` §6.1): the RUNNING controller clears one app's
// hold (any reason) and saves at once. No signature: the operator accepted that a compromised hub could release a
// data-protecting hold; the hub shows the hold's reason and a warning before the press.
OperatorActionClearHold = "clear_hold"
// HostOperatorActionRestartController (operator brief 2026-10-29, Part 1): the AGENT restarts a guest's controller
// (the supervisor's own act and guards). The agent's list holds this one verb.
HostOperatorActionRestartController = "restart_controller"
// Outcomes: the box's three, plus two the hub sets itself.
OperatorActionDone = "done"
@@ -50,7 +60,35 @@ const (
operatorActionsListed = 10
)
var operatorActionNames = []string{OperatorActionOffsiteBackupNow, OperatorActionAbandonStop, OperatorActionAbandonExtend, OperatorActionRunJob}
var operatorActionNames = []string{OperatorActionOffsiteBackupNow, OperatorActionAbandonStop, OperatorActionAbandonExtend, OperatorActionRunJob, OperatorActionClearHold}
// hostOperatorActionNames is the AGENT's closed list (the host report's reply). Pinned by
// TestHostOperatorActions_ClosedList; never shares a verb with the controller's list.
var hostOperatorActionNames = []string{HostOperatorActionRestartController}
// stackNameRe is an app's stack name as the controller names it (a catalog slug).
var stackNameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,63}$`)
// HostOperatorActionNames returns the agent's closed list (sorted copy).
func HostOperatorActionNames() []string { return sortedStrings(hostOperatorActionNames) }
// IsHostOperatorAction reports whether the verb belongs to the agent's list.
func IsHostOperatorAction(action string) bool { return inList(hostOperatorActionNames, action) }
// ValidateHostOperatorAction refuses anything outside the agent's closed list. restart_controller takes the guest's
// vmid (Proxmox: 100–999999999).
func ValidateHostOperatorAction(action, arg string) error {
switch action {
case HostOperatorActionRestartController:
v, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(v) != arg || v < 100 || v > 999999999 {
return fmt.Errorf("the guest id must be a number 100-999999999")
}
default:
return fmt.Errorf("unknown host action %q", action)
}
return nil
}
// operatorJobNames is the fixed set run_job may name — the controller's scheduler job names.
var operatorJobNames = []string{"fill-watch", "offsite-integrity", "offsite-proof", "disk-health-check"}
@@ -93,6 +131,10 @@ func ValidateOperatorAction(action, arg string) error {
if !inList(operatorJobNames, arg) {
return fmt.Errorf("unknown job %q", arg)
}
case OperatorActionClearHold:
if !stackNameRe.MatchString(arg) {
return fmt.Errorf("not an app name: %q", arg)
}
default:
return fmt.Errorf("unknown action %q", action)
}
@@ -112,6 +154,7 @@ type OperatorActionDirective struct {
type OperatorActionRow struct {
ID int64
CustomerID string
HostID string // "" = a controller action; else the agent of this host acts
Action string
Arg string
RequestedAt time.Time
@@ -135,7 +178,83 @@ func (s *Store) migrateOperatorActions() error {
message TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS idx_operator_actions_customer ON operator_actions(customer_id, done_at);`)
return err
if err != nil {
return err
}
// The agent's rows (2026-10-29): host_id names the host whose agent acts; "" = the controller's row.
if _, err := s.db.Exec(`ALTER TABLE operator_actions ADD COLUMN host_id TEXT NOT NULL DEFAULT ''`); err != nil &&
!strings.Contains(err.Error(), "duplicate column") {
return err
}
return nil
}
// CreateHostOperatorAction validates against the AGENT's closed list and stores a pending row for hostID (listed on the
// customer's host page beside the controller's rows).
func (s *Store) CreateHostOperatorAction(customerID, hostID, action, arg, by string) (int64, error) {
if hostID == "" {
return 0, fmt.Errorf("host operator action: empty host id")
}
if err := ValidateHostOperatorAction(action, arg); err != nil {
return 0, err
}
res, err := s.db.Exec(`INSERT INTO operator_actions (customer_id, host_id, action, arg, requested_at, requested_by)
VALUES (?, ?, ?, ?, ?, ?)`, customerID, hostID, action, arg, time.Now().UTC(), by)
if err != nil {
return 0, err
}
return res.LastInsertId()
}
// PendingHostOperatorActions is what the next host-report reply lists for hostID's agent, oldest first; rows older
// than OperatorActionTTL are closed as expired first.
func (s *Store) PendingHostOperatorActions(hostID string) ([]OperatorActionDirective, error) {
now := time.Now().UTC()
if _, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ?
WHERE host_id = ? AND done_at IS NULL AND requested_at < ?`,
now, OperatorActionExpired, "the agent did not answer within a day", hostID, now.Add(-OperatorActionTTL)); err != nil {
return nil, err
}
rows, err := s.db.Query(`SELECT id, action, arg FROM operator_actions
WHERE host_id = ? AND done_at IS NULL ORDER BY id LIMIT ?`, hostID, operatorActionsListed)
if err != nil {
return nil, err
}
defer rows.Close()
var out []OperatorActionDirective
for rows.Next() {
var d OperatorActionDirective
if err := rows.Scan(&d.ID, &d.Action, &d.Arg); err != nil {
return nil, err
}
out = append(out, d)
}
return out, rows.Err()
}
// RecordHostOperatorActionResult closes a pending AGENT row with the agent's result — matched on the id AND the
// reporting host, so another host's (or the controller's) result changes nothing.
func (s *Store) RecordHostOperatorActionResult(hostID string, id int64, outcome, message string) (*OperatorActionRow, error) {
switch outcome {
case OperatorActionDone, OperatorActionRefused, OperatorActionFailed:
default:
return nil, fmt.Errorf("unknown outcome %q", outcome)
}
if hostID == "" {
return nil, nil
}
if r := []rune(message); len(r) > operatorActionMessageMax {
message = string(r[:operatorActionMessageMax])
}
res, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ?
WHERE id = ? AND host_id = ? AND done_at IS NULL`, time.Now().UTC(), outcome, message, id, hostID)
if err != nil {
return nil, err
}
if n, _ := res.RowsAffected(); n == 0 {
return nil, nil
}
return s.getOperatorAction(id)
}
// CreateOperatorAction validates against the closed list and stores a pending row. by names who
@@ -160,12 +279,12 @@ func (s *Store) CreateOperatorAction(customerID, action, arg, by string) (int64,
func (s *Store) PendingOperatorActions(customerID string) ([]OperatorActionDirective, error) {
now := time.Now().UTC()
if _, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ?
WHERE customer_id = ? AND done_at IS NULL AND requested_at < ?`,
WHERE customer_id = ? AND host_id = '' AND done_at IS NULL AND requested_at < ?`,
now, OperatorActionExpired, "the box did not answer within a day", customerID, now.Add(-OperatorActionTTL)); err != nil {
return nil, err
}
rows, err := s.db.Query(`SELECT id, action, arg FROM operator_actions
WHERE customer_id = ? AND done_at IS NULL ORDER BY id LIMIT ?`, customerID, operatorActionsListed)
WHERE customer_id = ? AND host_id = '' AND done_at IS NULL ORDER BY id LIMIT ?`, customerID, operatorActionsListed)
if err != nil {
return nil, err
}
@@ -194,7 +313,7 @@ func (s *Store) RecordOperatorActionResult(customerID string, id int64, outcome,
message = string(r[:operatorActionMessageMax])
}
res, err := s.db.Exec(`UPDATE operator_actions SET done_at = ?, outcome = ?, message = ?
WHERE id = ? AND customer_id = ? AND done_at IS NULL`, time.Now().UTC(), outcome, message, id, customerID)
WHERE id = ? AND customer_id = ? AND host_id = '' AND done_at IS NULL`, time.Now().UTC(), outcome, message, id, customerID)
if err != nil {
return nil, err
}
@@ -205,7 +324,7 @@ func (s *Store) RecordOperatorActionResult(customerID string, id int64, outcome,
}
func (s *Store) getOperatorAction(id int64) (*OperatorActionRow, error) {
rows, err := s.db.Query(`SELECT id, customer_id, action, arg, requested_at, requested_by, done_at, outcome, message
rows, err := s.db.Query(`SELECT id, customer_id, host_id, action, arg, requested_at, requested_by, done_at, outcome, message
FROM operator_actions WHERE id = ?`, id)
if err != nil {
return nil, err
@@ -220,7 +339,7 @@ func (s *Store) getOperatorAction(id int64) (*OperatorActionRow, error) {
// ListOperatorActions returns the customer's newest rows (pending and closed), newest first.
func (s *Store) ListOperatorActions(customerID string, limit int) ([]OperatorActionRow, error) {
rows, err := s.db.Query(`SELECT id, customer_id, action, arg, requested_at, requested_by, done_at, outcome, message
rows, err := s.db.Query(`SELECT id, customer_id, host_id, action, arg, requested_at, requested_by, done_at, outcome, message
FROM operator_actions WHERE customer_id = ? ORDER BY id DESC LIMIT ?`, customerID, limit)
if err != nil {
return nil, err
@@ -234,7 +353,7 @@ func scanOperatorActions(rows *sql.Rows) ([]OperatorActionRow, error) {
for rows.Next() {
var r OperatorActionRow
var done sql.NullTime
if err := rows.Scan(&r.ID, &r.CustomerID, &r.Action, &r.Arg, &r.RequestedAt, &r.RequestedBy, &done, &r.Outcome, &r.Message); err != nil {
if err := rows.Scan(&r.ID, &r.CustomerID, &r.HostID, &r.Action, &r.Arg, &r.RequestedAt, &r.RequestedBy, &done, &r.Outcome, &r.Message); err != nil {
return nil, err
}
if done.Valid {
+83 -1
View File
@@ -11,7 +11,8 @@ import (
// The list is CLOSED, on the hub side too — this fails when an action or a job is added, by design. The
// controller pins the same four (internal/report TestOpActions_ClosedList).
func TestOperatorActions_ClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
// D64 = B (operator 2026-10-29 12:10): clear_hold joins the controller's list.
if got, want := OperatorActionNames(), []string{"abandon_extend", "abandon_stop", "clear_hold", "offsite_backup_now", "run_job"}; !reflect.DeepEqual(got, want) {
t.Fatalf("operator actions = %v, want exactly %v — a new action needs the operator's word (decision 185)", got, want)
}
if got, want := OperatorJobNames(), []string{"disk-health-check", "fill-watch", "offsite-integrity", "offsite-proof"}; !reflect.DeepEqual(got, want) {
@@ -108,3 +109,84 @@ func TestOperatorActions_ExpireAndResetCancel(t *testing.T) {
t.Fatalf("fresh row after reset = %+v, want cancelled (kept as a record)", r)
}
}
// The AGENT's closed list (operator brief 2026-10-29, Part 1): one verb. RED-PROOF: seen failing before the list existed.
func TestHostOperatorActions_ClosedList(t *testing.T) {
if got, want := HostOperatorActionNames(), []string{"restart_controller"}; !reflect.DeepEqual(got, want) {
t.Fatalf("host operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
}
for _, a := range OperatorActionNames() {
if IsHostOperatorAction(a) {
t.Errorf("%s is on BOTH lists — a controller action must never reach the agent", a)
}
}
}
// Each list refuses the other's verbs and every bad argument BEFORE storing.
func TestHostOperatorActions_RefusedBeforeStoring(t *testing.T) {
s := newTestStore(t)
bad := []struct{ action, arg string }{
{"restart_controller", ""}, {"restart_controller", "abc"}, {"restart_controller", "99"}, {"restart_controller", "9201 "},
{"restart_controller", "-1"}, {"restart_controller", "10000000000"}, {"reboot_guest", "9201"}, {"clear_hold", "radicale"},
{"offsite_backup_now", ""},
}
for _, b := range bad {
if _, err := s.CreateHostOperatorAction("c1", "h1", b.action, b.arg, "operator"); err == nil {
t.Errorf("host action %s(%q) was accepted", b.action, b.arg)
}
}
for _, b := range []struct{ action, arg string }{{"clear_hold", ""}, {"clear_hold", "../x"}, {"clear_hold", "Rad icale"},
{"clear_hold", "a/b"}, {"restart_controller", "9201"}} {
if _, err := s.CreateOperatorAction("c1", b.action, b.arg, "operator"); err == nil {
t.Errorf("controller action %s(%q) was accepted", b.action, b.arg)
}
}
if _, err := s.CreateOperatorAction("c1", "clear_hold", "radicale", "operator"); err != nil {
t.Fatalf("clear_hold radicale refused: %v", err)
}
if _, err := s.CreateHostOperatorAction("c1", "h1", "restart_controller", "9201", "operator"); err != nil {
t.Fatalf("restart_controller 9201 refused: %v", err)
}
}
// An agent row reaches only its host's envelope, a controller row only the customer's ACK; a result closes only a row of
// the reporting host. RED-PROOF: drop `AND host_id = ”` from PendingOperatorActions → the restart reaches the controller.
func TestHostOperatorActions_EachSideSeesOnlyItsOwn(t *testing.T) {
s := newTestStore(t)
cid, err := s.CreateOperatorAction("c1", "clear_hold", "radicale", "operator")
if err != nil {
t.Fatal(err)
}
hid, err := s.CreateHostOperatorAction("c1", "h1", "restart_controller", "9201", "operator")
if err != nil {
t.Fatal(err)
}
ctl, _ := s.PendingOperatorActions("c1")
if len(ctl) != 1 || ctl[0].ID != cid {
t.Fatalf("the controller ACK lists %+v, want only #%d", ctl, cid)
}
agent, _ := s.PendingHostOperatorActions("h1")
if len(agent) != 1 || agent[0].ID != hid || agent[0].Action != "restart_controller" || agent[0].Arg != "9201" {
t.Fatalf("the agent envelope lists %+v, want only #%d", agent, hid)
}
if other, _ := s.PendingHostOperatorActions("h2"); len(other) != 0 {
t.Fatalf("another host sees %+v", other)
}
if row, _ := s.RecordHostOperatorActionResult("h2", hid, "done", "x"); row != nil {
t.Fatal("another host closed the row")
}
if row, _ := s.RecordOperatorActionResult("c1", hid, "done", "x"); row != nil {
t.Fatal("the controller's result channel closed an agent row")
}
row, err := s.RecordHostOperatorActionResult("h1", hid, "done", "the controller answered after 14 s")
if err != nil || row == nil || row.Outcome != "done" || row.HostID != "h1" {
t.Fatalf("result: %+v %v", row, err)
}
if left, _ := s.PendingHostOperatorActions("h1"); len(left) != 0 {
t.Fatalf("a closed row is still listed: %+v", left)
}
list, _ := s.ListOperatorActions("c1", 10)
if len(list) != 2 {
t.Fatalf("the host page lists %d rows, want both", len(list))
}
}
+13 -1
View File
@@ -18,6 +18,18 @@ type heldAppRow struct {
WhoClears string
Reminded bool
Stale bool // the box has not repeated it for heldAppStale — it may be down
// ProtectsData (D64 = B): a restore or put-back hold — the confirm step warns before „Clear hold" (`07` §8).
ProtectsData bool
}
// holdProtectsData: the reasons whose hold keeps an app off half-restored data (`07` §8): a failed restore (the
// reason is "" or restore_failed on older rows), a mixed restore, a put-back waiting for its drive.
func holdProtectsData(reason string) bool {
switch reason {
case "", "restore_failed", "restore_mixed", "putback_pending":
return true
}
return false
}
// heldAppStaleAfter mirrors monitor's heldAppStale: a box repeats app_held daily.
@@ -51,7 +63,7 @@ func toHeldAppRows(held []store.HeldApp, hostOf map[string]string, now time.Time
}
out = append(out, heldAppRow{CustomerID: h.CustomerID, HostID: hostOf[h.CustomerID], Stack: h.Stack, Reason: h.Reason,
Since: since, Why: h.Why, WhoClears: monitor.HeldAppWhoClears(h.Reason), Reminded: !h.RemindedAt.IsZero(),
Stale: now.Sub(h.LastSeen) > heldAppStaleAfter})
Stale: now.Sub(h.LastSeen) > heldAppStaleAfter, ProtectsData: holdProtectsData(h.Reason)})
}
return out
}
+94
View File
@@ -0,0 +1,94 @@
package web
import (
"net/http"
"net/url"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// Operator brief 2026-10-29: „Restart controller" (the AGENT's closed list) and „Clear hold" (D64 = B, the
// controller's list) on the host page — each behind a confirm step, CSRF like every form.
func heldFixture(t *testing.T) (*Server, *store.Store) {
t.Helper()
s, st := newTestServer(t)
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k"}); err != nil {
t.Fatal(err)
}
now := time.Now()
if err := st.UpsertGuestFromReport(&store.Guest{GuestID: "h1-9201", CustomerID: "c1", HostID: "h1", VMID: 9201, DisplayName: "felhom",
Status: "running", ControllerVersion: "0.347.0", LastSeenAt: &now}); err != nil {
t.Fatal(err)
}
if err := st.UpsertHeldApp("c1", "radicale", "restore_mixed", now.Add(-time.Hour), "a restore stopped half-way", now); err != nil {
t.Fatal(err)
}
if err := st.UpsertHeldApp("c1", "gokapi", "unhealthy_stop", now.Add(-time.Hour), "stopped in a crash loop", now); err != nil {
t.Fatal(err)
}
return s, st
}
// RED-PROOF: seen failing before the forms existed.
func TestHostPage_RestartAndClearHoldRenderBehindAConfirm(t *testing.T) {
s, _ := heldFixture(t)
body := renderHost(t, s, "h1")
for _, want := range []string{
`data-restart-controller="9201"`, `name="action" value="restart_controller"`, `name="arg" value="9201"`,
"Restart the controller of guest 9201 on h1 (c1)?",
`data-clear-hold="radicale"`, `name="action" value="clear_hold"`, `name="arg" value="radicale"`,
"This hold protects data from a failed restore. If you clear it, the app may start on half-restored data. Check the restore copy first.",
`data-clear-hold="gokapi"`,
} {
if !strings.Contains(body, want) {
t.Errorf("host page lacks %q", want)
}
}
// the warning only on the data-protecting hold
gk := body[strings.Index(body, `data-clear-hold="gokapi"`):]
gk = gk[:strings.Index(gk, "</details>")]
if strings.Contains(gk, "half-restored") {
t.Error("a crash-loop hold shows the restore warning")
}
if !strings.Contains(gk, "unhealthy_stop") {
t.Error("the confirm step does not name the hold's reason")
}
}
// restart_controller: a row for the AGENT of this host (never the controller's ACK); clear_hold only for an app that
// IS held; every other verb or argument refused with no row. RED-PROOF: seen failing before the handler routed them.
func TestHostPage_RestartAndClearHoldPosts(t *testing.T) {
s, st := heldFixture(t)
if rr := postOperatorAction(t, s, "h1", url.Values{"action": {"restart_controller"}, "arg": {"9201"}}); rr.Code != http.StatusSeeOther {
t.Fatalf("restart: %d %s", rr.Code, rr.Body.String())
}
if a, _ := st.PendingHostOperatorActions("h1"); len(a) != 1 || a[0].Arg != "9201" {
t.Fatalf("the agent's reply would list %+v", a)
}
if c, _ := st.PendingOperatorActions("c1"); len(c) != 0 {
t.Fatalf("the restart reached the controller's ACK: %+v", c)
}
if rr := postOperatorAction(t, s, "h1", url.Values{"action": {"clear_hold"}, "arg": {"radicale"}}); rr.Code != http.StatusSeeOther {
t.Fatalf("clear_hold: %d %s", rr.Code, rr.Body.String())
}
if c, _ := st.PendingOperatorActions("c1"); len(c) != 1 || c[0].Action != "clear_hold" || c[0].Arg != "radicale" {
t.Fatalf("the controller's ACK would list %+v", c)
}
for _, f := range []url.Values{
{"action": {"clear_hold"}, "arg": {"vikunja"}}, // not held
{"action": {"restart_controller"}, "arg": {"9202"}}, // not a guest of this host
{"action": {"restart_controller"}, "arg": {"abc"}},
{"action": {"reboot_guest"}, "arg": {"9201"}},
} {
if rr := postOperatorAction(t, s, "h1", f); rr.Code != http.StatusBadRequest {
t.Errorf("%v: status %d, want 400", f, rr.Code)
}
}
if rows, _ := st.ListOperatorActions("c1", 20); len(rows) != 2 {
t.Fatalf("rows = %d, want the two accepted presses only", len(rows))
}
}
+10 -4
View File
@@ -174,11 +174,17 @@ func TestHandleHostDetail(t *testing.T) {
// Decision 185 (2026-10-08): plus the four operator-action buttons — a closed list, none of which
// deletes or mutates the host (TestOperatorActions_* pin what they may do). Every OTHER button
// remains absent: 2 log requests + 4 operator actions, and the 4 all post to /operator-action.
if got := strings.Count(strings.ToLower(body), "<button"); got != 6 {
t.Errorf("host detail has %d buttons, want exactly the 2 log-request + 4 operator-action buttons", got)
// 2026-10-29 (operator brief, Part 1): plus ONE „Restart controller" per guest that runs a controller (the AGENT's
// closed list, behind a confirm step; this fixture has one such guest). „Clear hold" appears only per held app (none
// here). No other button.
if got := strings.Count(strings.ToLower(body), "<button"); got != 7 {
t.Errorf("host detail has %d buttons, want exactly the 2 log-request + 4 operator-action + 1 restart-controller buttons", got)
}
if got := strings.Count(body, `action="/hosts/demo-felhom-01/operator-action"`); got != 4 {
t.Errorf("operator-action forms = %d, want 4", got)
if got := strings.Count(body, `action="/hosts/demo-felhom-01/operator-action"`); got != 5 {
t.Errorf("operator-action forms = %d, want 5", got)
}
if got := strings.Count(body, `value="restart_controller"`); got != 1 {
t.Errorf("restart_controller forms = %d, want 1 (one guest with a controller)", got)
}
if strings.Count(body, `action="/hosts/demo-felhom-01/request-logs"`) != 2 {
t.Error("the request-logs forms are missing — every button must be a log-bundle request")
+67
View File
@@ -2,6 +2,7 @@ package web
import (
"net/http"
"strconv"
"strings"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
@@ -42,11 +43,22 @@ func (s *Server) handleOperatorAction(w http.ResponseWriter, r *http.Request, ho
}
action := strings.TrimSpace(r.FormValue("action"))
arg := strings.TrimSpace(r.FormValue("arg"))
// The AGENT's closed list (2026-10-29): its own channel, the host report's reply.
if store.IsHostOperatorAction(action) {
s.handleHostOperatorAction(w, r, host, action, arg)
return
}
if err := store.ValidateOperatorAction(action, arg); err != nil {
s.logger.Printf("[INFO] operator action refused for host %s: %v", hostID, err)
http.Error(w, "Refused: "+err.Error(), http.StatusBadRequest)
return
}
// D64 = B: Clear hold only for an app the box reports held now — a stale page or a typo stores nothing.
if action == store.OperatorActionClearHold && !s.customerHoldsApp(host.CustomerID, arg) {
s.logger.Printf("[INFO] clear_hold refused for host %s: %s is not held on this box", hostID, arg)
http.Error(w, "Refused: "+arg+" is not held on this box", http.StatusBadRequest)
return
}
by := operatorActor(r)
id, err := s.store.CreateOperatorAction(host.CustomerID, action, arg, by)
if err != nil {
@@ -80,3 +92,58 @@ func (s *Server) hostOperatorActionRows(host *store.Host) []store.OperatorAction
}
return rows
}
// customerHoldsApp reports whether the box's held-apps mirror lists the app now.
func (s *Server) customerHoldsApp(customerID, app string) bool {
held, err := s.store.ListHeldApps(customerID)
if err != nil {
return false
}
for _, h := range held {
if h.Stack == app {
return true
}
}
return false
}
// handleHostOperatorAction stores an AGENT row (the agent's closed list) and pokes the box, so its out-of-cycle
// report carries the action in seconds; the result comes on the report after (the agent requests it at once).
// restart_controller names a guest of THIS host — another vmid is refused before anything is stored.
func (s *Server) handleHostOperatorAction(w http.ResponseWriter, r *http.Request, host *store.Host, action, arg string) {
if err := store.ValidateHostOperatorAction(action, arg); err != nil {
s.logger.Printf("[INFO] host operator action refused for host %s: %v", host.HostID, err)
http.Error(w, "Refused: "+err.Error(), http.StatusBadRequest)
return
}
if !s.hostHasGuest(host.HostID, arg) {
s.logger.Printf("[INFO] host operator action %s refused for host %s: guest %s is not on this host", action, host.HostID, arg)
http.Error(w, "Refused: guest "+arg+" is not on this host", http.StatusBadRequest)
return
}
by := operatorActor(r)
id, err := s.store.CreateHostOperatorAction(host.CustomerID, host.HostID, action, arg, by)
if err != nil {
s.logger.Printf("[ERROR] host operator action %s for %s: %v", action, host.HostID, err)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
s.logger.Printf("[INFO] host operator action #%d %s%s requested for host %s by %s — the agent acts on its next report (poked now)",
id, action, opArgSuffix(arg), host.HostID, by)
s.poke.PokeHost(host.HostID) // nil-safe; without the tunnel the 15-min cycle carries it
http.Redirect(w, r, "/hosts/"+host.HostID+"#operator-actions", http.StatusSeeOther)
}
// hostHasGuest reports whether vmid is one of the host's guests as its reports name them.
func (s *Server) hostHasGuest(hostID, vmid string) bool {
guests, err := s.store.ListGuestsForHost(hostID)
if err != nil {
return false
}
for _, g := range guests {
if strconv.Itoa(g.VMID) == vmid {
return true
}
}
return false
}
@@ -133,7 +133,19 @@
<td>{{.VMID}}</td>
<td>{{if .DisplayName}}{{.DisplayName}}{{else}}—{{end}}</td>
<td>{{if eq .Status "running"}}<span style="color: var(--green)">{{.Status}}</span>{{else if eq .Status "stopped"}}<span style="color: var(--red)">{{.Status}}</span>{{else}}{{.Status}}{{end}}</td>
<td>{{if .ControllerVersion}}<code>{{.ControllerVersion}}</code>{{else}}—{{end}}</td>
<td>{{if .ControllerVersion}}<code>{{.ControllerVersion}}</code>{{else}}—{{end}}
{{if and .ControllerVersion $.CustomerID}}{{/* 2026-10-29: the AGENT restarts it (the supervisor's act and guards; 3 in 15 min, then refused for 30) — works with the controller stopped */}}
<details class="op-confirm" data-restart-controller="{{.VMID}}">
<summary class="btn btn-sm btn-outline">Restart controller</summary>
<form method="POST" action="/hosts/{{$.HostID}}/operator-action">
<input type="hidden" name="_csrf" value="{{$.CSRFToken}}">
<input type="hidden" name="action" value="restart_controller">
<input type="hidden" name="arg" value="{{.VMID}}">
<p>Restart the controller of guest {{.VMID}} on {{$.HostID}} ({{$.CustomerID}})? The box's agent restarts it, even when the controller does not answer. The household's dashboard is away for about a minute. At most 3 restarts in 15 minutes; the result shows under Operator Actions.</p>
<button type="submit" class="btn btn-sm btn-danger">Restart the controller</button>
</form>
</details>{{end}}
</td>
<td>{{timeAgoPtr .LastSeenAt}}</td>
</tr>
{{end}}
@@ -146,7 +158,8 @@
{{end}}
</section>
<!-- Held apps (D53, `08` §10): apps the box keeps stopped until a person acts. The hub never clears one. -->
<!-- Held apps (D53, `08` §10): apps the box keeps stopped until a person acts. D64 = B (operator 2026-10-29): the
operator may clear any hold here; the confirm step shows the reason and, for a data-protecting hold, a warning. -->
<section class="card" id="held-apps" style="padding: 0; overflow: hidden;">
<h2 style="padding: 1.25rem 1.25rem 0.5rem;">Held apps</h2>
{{if .HeldApps}}
@@ -167,7 +180,19 @@
<td><span class="severity-badge severity-warning">{{.Reason}}</span>{{if .Stale}} <span class="text-muted">(not confirmed by the box for 2 days)</span>{{end}}</td>
<td>{{timeAgo .Since}}</td>
<td>{{.Why}}</td>
<td>{{.WhoClears}}{{if .Reminded}} <span class="text-muted">(operator reminded)</span>{{end}}</td>
<td>{{.WhoClears}}{{if .Reminded}} <span class="text-muted">(operator reminded)</span>{{end}}
<details class="op-confirm" data-clear-hold="{{.Stack}}">
<summary class="btn btn-sm btn-outline">Clear hold</summary>
<form method="POST" action="/hosts/{{$.HostID}}/operator-action">
<input type="hidden" name="_csrf" value="{{$.CSRFToken}}">
<input type="hidden" name="action" value="clear_hold">
<input type="hidden" name="arg" value="{{.Stack}}">
<p>Clear the hold of <code>{{.Stack}}</code> ({{.Reason}}, since {{.Since.Format "2006-01-02 15:04"}})? The box's controller clears it at once; the household can then start the app.</p>
{{if .ProtectsData}}<p class="op-warning">This hold protects data from a failed restore. If you clear it, the app may start on half-restored data. Check the restore copy first.</p>{{end}}
<button type="submit" class="btn btn-sm btn-danger">Clear the hold</button>
</form>
</details>
</td>
</tr>
{{end}}
</tbody>
+7
View File
@@ -1031,3 +1031,10 @@ body.js-tabs .tab-panel:not(.tab-panel-active) { display: none; }
}
.mail-hold-release .text-muted { font-size: 0.8rem; }
#mail-hold a { color: var(--text-1); }
/* 2026-10-29: the confirm fold-out of „Restart controller" and „Clear hold" (no script; the final button is inside). */
.op-confirm { margin-top: 0.35rem; }
.op-confirm summary { list-style: none; display: inline-block; cursor: pointer; }
.op-confirm form { margin-top: 0.5rem; max-width: 34rem; }
.op-confirm p { font-size: 0.85rem; margin: 0.25rem 0 0.5rem; }
.op-warning { color: var(--warn); }
+6
View File
@@ -106,6 +106,9 @@ ROOTS = [
# ACK's operator_actions list. Named type for the same reason as R-311's root above.
("hub -> controller (report ACK, `operator_actions` entry)",
"hub", "internal/store", "OperatorActionDirective", "controller"),
# 2026-10-29 (operator brief, Part 1): the AGENT's operator actions — the same entry type on the host-report reply.
("hub -> agent (host-report reply, `operator_actions` entry)",
"hub", "internal/store", "OperatorActionDirective", "agent"),
# R-922 (2026-10-09): the notification-prefs push. Declared when the household's deliberate clear
# (`email_cleared`) joined it — a flag the hub cannot decode would leave the cleared address stored.
("controller -> hub (POST /preferences)",
@@ -124,6 +127,7 @@ MIRRORS = {
"hub -> agent (GET /hosts/<id>/escrow/retained)": ("internal/hub", "RetainedEscrowResponse"),
"hub -> controller (report ACK, `escrow` object)": ("internal/report", "EscrowStatus"),
"hub -> controller (report ACK, `operator_actions` entry)": ("internal/report", "OperatorAction"),
"hub -> agent (host-report reply, `operator_actions` entry)": ("internal/hub", "OperatorAction"),
}
# R-444 (2026-10-06): a SUBTREE of a name-checked root whose receiver decodes it into one named mirror type gets the
@@ -136,6 +140,8 @@ MIRRORS = {
SUBTREE_MIRRORS = {
("agent -> hub (POST /host-report)", "guest_disk_trim"): ("internal/sysfacts", "DiskTrim"),
("agent -> hub (POST /host-report)", "line_test"): ("internal/sysfacts", "LineTest"),
# 2026-10-29: the agent's answers to the operator's actions (hub api.hostOperatorActionResult).
("agent -> hub (POST /host-report)", "operator_action_results"): ("internal/api", "hostOperatorActionResult"),
}
# Tag names whose literal string carries no information in a repo-wide search. NOT CHECKED.