D65 = A: reboot_guest joins the agent's closed operator-action list — guards (busy controller file, heavy gate, jobs, lock, parked, 1/hour, the brake), result with both times, GET /guest/operator-reboot for the household line; one read-only sudo vector
This commit is contained in:
1 parent
bdea4694a4
commit
90ebce11e9
10 files changed
+697
-8
No files matched your search
@@ -1,3 +1,29 @@
|
||||
## Unreleased — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list
|
||||
|
||||
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
|
||||
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
|
||||
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
|
||||
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
|
||||
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
|
||||
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
|
||||
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
|
||||
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
|
||||
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
|
||||
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
|
||||
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
|
||||
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
|
||||
a racing start would fail the reboot's own start half.
|
||||
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
|
||||
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
|
||||
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
|
||||
restarted your box at <time>".
|
||||
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
|
||||
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
|
||||
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
|
||||
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
|
||||
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
|
||||
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
|
||||
|
||||
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
|
||||
|
||||
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
|
||||
|
||||
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
|
||||
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
|
||||
# + container across spaces, spike-confirmed)
|
||||
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
|
||||
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
|
||||
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
|
||||
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
|
||||
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
|
||||
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
|
||||
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
|
||||
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
|
||||
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
|
||||
|
||||
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
|
||||
|
||||
@@ -151,6 +151,7 @@ var manifest = []Capability{
|
||||
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
|
||||
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
|
||||
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
|
||||
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
|
||||
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
|
||||
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
|
||||
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
|
||||
|
||||
@@ -254,6 +254,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
|
||||
if !s.provisionedGuest(g.VMID) {
|
||||
continue
|
||||
}
|
||||
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
|
||||
s.resetNotRunning(g.VMID)
|
||||
continue
|
||||
}
|
||||
evaluated++
|
||||
if !s.superviseOneController(ctx, g.VMID, g.Status) {
|
||||
down++
|
||||
|
||||
@@ -152,6 +152,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
|
||||
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
|
||||
return
|
||||
}
|
||||
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
|
||||
// would fail the reboot's own start half).
|
||||
if s.rebooting(g.VMID) {
|
||||
return
|
||||
}
|
||||
|
||||
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
|
||||
if err != nil {
|
||||
|
||||
@@ -20,7 +20,7 @@ import (
|
||||
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
|
||||
// The result is re-sent until the reply stops listing the id.
|
||||
//
|
||||
// THE LIST IS CLOSED: one verb, restart_controller <vmid>. Anything else — a verb, a vmid that does not parse, a guest
|
||||
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
|
||||
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
|
||||
//
|
||||
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
|
||||
@@ -31,7 +31,7 @@ import (
|
||||
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
|
||||
|
||||
const (
|
||||
// OpRestartController is the agent's one operator verb.
|
||||
// OpRestartController is the agent's first operator verb.
|
||||
OpRestartController = "restart_controller"
|
||||
|
||||
opOutcomeDone = "done"
|
||||
@@ -52,7 +52,7 @@ const (
|
||||
)
|
||||
|
||||
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
|
||||
var operatorActionNames = []string{OpRestartController}
|
||||
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
|
||||
|
||||
// OperatorActionNames returns the closed list (sorted copy).
|
||||
func OperatorActionNames() []string {
|
||||
@@ -120,11 +120,18 @@ func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
|
||||
}
|
||||
|
||||
func (s *Server) runOperatorAction(a hub.OperatorAction) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), s.opWait()+time.Minute)
|
||||
limit := s.opWait() + time.Minute
|
||||
if a.Action == OpRebootGuest {
|
||||
limit += opRebootTimeout
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), limit)
|
||||
defer cancel()
|
||||
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller only", a.Action)
|
||||
if a.Action == OpRestartController {
|
||||
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
|
||||
switch a.Action {
|
||||
case OpRestartController:
|
||||
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
|
||||
case OpRebootGuest:
|
||||
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
|
||||
}
|
||||
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
|
||||
"outcome", outcome, "message", msg)
|
||||
|
||||
@@ -71,7 +71,7 @@ func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||
|
||||
// RED-PROOF: seen failing before the list existed (build failure, then the names).
|
||||
func TestOpActions_AgentClosedList(t *testing.T) {
|
||||
if got, want := OperatorActionNames(), []string{"restart_controller"}; !reflect.DeepEqual(got, want) {
|
||||
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
|
||||
}
|
||||
}
|
||||
@@ -80,7 +80,7 @@ func TestOpActions_AgentClosedList(t *testing.T) {
|
||||
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
|
||||
s, ex, _, _, _, _ := opServer(t, "running")
|
||||
s.OnEnvelope(context.Background(), env(
|
||||
hub.OperatorAction{ID: 1, Action: "reboot_guest", Arg: "9201"},
|
||||
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
|
||||
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
|
||||
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
|
||||
))
|
||||
|
||||
@@ -0,0 +1,328 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
|
||||
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
|
||||
//
|
||||
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
|
||||
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
|
||||
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
|
||||
//
|
||||
// THE GUARDS (each refuses with the reason in the result; nothing is called):
|
||||
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
|
||||
// - a controller swap (update) is in flight;
|
||||
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
|
||||
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
|
||||
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
|
||||
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
|
||||
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
|
||||
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
|
||||
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
|
||||
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
|
||||
// alone while the reboot runs (rebootInFlight).
|
||||
//
|
||||
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
|
||||
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
|
||||
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
|
||||
// one day). Pinned by opreboot_test.go.
|
||||
|
||||
const (
|
||||
// OpRebootGuest is the agent's second operator verb (D65 = A).
|
||||
OpRebootGuest = "reboot_guest"
|
||||
|
||||
// opRebootEvery: at most one operator reboot per guest per hour.
|
||||
opRebootEvery = time.Hour
|
||||
// opRebootNoticeFor: how long the household's dashboard shows the line.
|
||||
opRebootNoticeFor = 24 * time.Hour
|
||||
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
|
||||
opRebootTimeout = 5 * time.Minute
|
||||
|
||||
operatorRebootFile = "operator-reboot.json"
|
||||
|
||||
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
|
||||
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
|
||||
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
|
||||
// Docker does not hide it.
|
||||
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
|
||||
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
|
||||
// stopped writing (stuck or killed) — not a busy one.
|
||||
controllerBusyFresh = 90 * time.Second
|
||||
)
|
||||
|
||||
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
|
||||
type operatorRebootRecord struct {
|
||||
At time.Time `json:"at"`
|
||||
ActionID int64 `json:"action_id"`
|
||||
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
|
||||
ControllerAfterS int `json:"controller_after_s,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) operatorRebootPath(vmid int) string {
|
||||
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
|
||||
}
|
||||
|
||||
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
|
||||
// must not silently vanish).
|
||||
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
|
||||
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return operatorRebootRecord{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return operatorRebootRecord{}, false, err
|
||||
}
|
||||
var rec operatorRebootRecord
|
||||
if err := json.Unmarshal(raw, &rec); err != nil {
|
||||
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
|
||||
}
|
||||
return rec, !rec.At.IsZero(), nil
|
||||
}
|
||||
|
||||
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
|
||||
raw, err := json.Marshal(rec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
p := s.operatorRebootPath(vmid)
|
||||
tmp := p + ".tmp"
|
||||
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, p)
|
||||
}
|
||||
|
||||
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
|
||||
func (s *Server) rebooting(vmid int) bool {
|
||||
s.rebootMu.Lock()
|
||||
defer s.rebootMu.Unlock()
|
||||
return s.rebootInFlight[vmid]
|
||||
}
|
||||
|
||||
func (s *Server) setRebooting(vmid int, on bool) {
|
||||
s.rebootMu.Lock()
|
||||
defer s.rebootMu.Unlock()
|
||||
if s.rebootInFlight == nil {
|
||||
s.rebootInFlight = map[int]bool{}
|
||||
}
|
||||
if on {
|
||||
s.rebootInFlight[vmid] = true
|
||||
} else {
|
||||
delete(s.rebootInFlight, vmid)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) rebooter() interface {
|
||||
RebootGuest(ctx context.Context, vmid int) error
|
||||
} {
|
||||
if s.opRebooter != nil {
|
||||
return s.opRebooter
|
||||
}
|
||||
if s.guestAttach != nil {
|
||||
return s.guestAttach
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
|
||||
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
|
||||
// (the caller refuses — it cannot prove no backup runs).
|
||||
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
|
||||
if err != nil {
|
||||
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
|
||||
return false, "", nil
|
||||
}
|
||||
return false, "", err
|
||||
}
|
||||
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
|
||||
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
|
||||
if perr != nil {
|
||||
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
|
||||
}
|
||||
if s.clock().Sub(at) > controllerBusyFresh {
|
||||
s.logger.Info("operator reboot: the controller's busy file is stale — the controller stopped writing it; not busy",
|
||||
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
|
||||
return false, "", nil
|
||||
}
|
||||
why := "work in progress"
|
||||
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
|
||||
why = strings.TrimSpace(lines[1])
|
||||
}
|
||||
return true, why, nil
|
||||
}
|
||||
|
||||
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
|
||||
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
|
||||
vmid, err := strconv.Atoi(arg)
|
||||
if err != nil || strconv.Itoa(vmid) != arg {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
|
||||
}
|
||||
rb := s.rebooter()
|
||||
if s.staleLock == nil || s.guestExec == nil || rb == nil {
|
||||
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
|
||||
}
|
||||
if !s.provisionedGuest(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
|
||||
}
|
||||
if s.rebooting(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
|
||||
}
|
||||
guests, err := s.staleLock.Guests(ctx)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
|
||||
}
|
||||
status := ""
|
||||
for _, g := range guests {
|
||||
if g.VMID == vmid {
|
||||
status = g.Status
|
||||
}
|
||||
}
|
||||
if status != "running" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
|
||||
}
|
||||
if s.controllerParked(vmid) {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
|
||||
}
|
||||
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
|
||||
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
|
||||
} else if lock != "" {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
|
||||
}
|
||||
|
||||
// At most one per hour.
|
||||
now := s.clock()
|
||||
last, had, err := s.lastOperatorReboot(vmid)
|
||||
if err != nil {
|
||||
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
|
||||
}
|
||||
if had && now.Sub(last.At) < opRebootEvery {
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
|
||||
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
|
||||
}
|
||||
|
||||
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
|
||||
s.ctrlSup.mu.Lock()
|
||||
st := s.supState(vmid)
|
||||
if now.Before(st.opBrakeUntil) {
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
|
||||
if len(st.restarts) >= controllerCrashloopMax {
|
||||
st.opBrakeUntil = now.Add(opBrakeRefusal)
|
||||
until := st.opBrakeUntil
|
||||
s.ctrlSup.mu.Unlock()
|
||||
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
|
||||
}
|
||||
s.ctrlSup.mu.Unlock()
|
||||
|
||||
// The heavy-operation gate, held until the guest is up again.
|
||||
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
|
||||
if !ok {
|
||||
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
|
||||
}
|
||||
defer release()
|
||||
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
|
||||
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
|
||||
}
|
||||
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
|
||||
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
|
||||
} else if cbusy {
|
||||
return opOutcomeRefused, "refused: the controller is busy — " + why
|
||||
}
|
||||
|
||||
// The record FIRST: the hourly limit and the household's line both read it.
|
||||
rec := operatorRebootRecord{At: now, ActionID: id}
|
||||
if err := s.saveOperatorReboot(vmid, rec); err != nil {
|
||||
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
|
||||
}
|
||||
s.ctrlSup.mu.Lock()
|
||||
st = s.supState(vmid)
|
||||
st.restarts = append(st.restarts, now)
|
||||
st.restartsTotal++
|
||||
st.lastRestartAt = now
|
||||
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
|
||||
st.notRunningSeen = 0
|
||||
s.ctrlSup.mu.Unlock()
|
||||
s.setRebooting(vmid, true)
|
||||
defer s.setRebooting(vmid, false)
|
||||
|
||||
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
|
||||
began := time.Now()
|
||||
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
|
||||
err = rb.RebootGuest(rctx, vmid)
|
||||
cancel()
|
||||
if err != nil {
|
||||
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
|
||||
}
|
||||
upS := int(time.Since(began).Round(time.Second) / time.Second)
|
||||
rec.GuestUpAfterS = upS
|
||||
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
|
||||
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
|
||||
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
|
||||
if err == nil {
|
||||
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
|
||||
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
|
||||
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
|
||||
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
|
||||
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
|
||||
}
|
||||
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
|
||||
case <-time.After(s.opPoll()):
|
||||
}
|
||||
}
|
||||
_ = s.saveOperatorReboot(vmid, rec)
|
||||
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
|
||||
}
|
||||
|
||||
// rebootBlockerSentence names jobBlocker's code in the result's words.
|
||||
func rebootBlockerSentence(code string) string {
|
||||
switch code {
|
||||
case PowerOffBackup:
|
||||
return "a backup is running"
|
||||
case PowerOffCtrlUpdate:
|
||||
return "a controller update is running"
|
||||
case PowerOffDrive:
|
||||
return "a drive or network-storage job is running"
|
||||
case PowerOffEscrow:
|
||||
return "an escrow ceremony is running"
|
||||
case PowerOffUnknown:
|
||||
return "the running jobs cannot be read"
|
||||
default:
|
||||
return "the host is busy"
|
||||
}
|
||||
}
|
||||
|
||||
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
|
||||
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
|
||||
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
|
||||
rec, ok, err := s.lastOperatorReboot(vmid)
|
||||
if err != nil {
|
||||
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
|
||||
return
|
||||
}
|
||||
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
|
||||
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
|
||||
return
|
||||
}
|
||||
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
|
||||
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
|
||||
|
||||
// rbExec adds the controller's busy file to the operator-restart fake.
|
||||
type rbExec struct {
|
||||
*opExec
|
||||
busy string // the file's content; "" = no file
|
||||
busyErr error // pct exec itself failed
|
||||
}
|
||||
|
||||
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
|
||||
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
|
||||
if f.busyErr != nil {
|
||||
return "", f.busyErr
|
||||
}
|
||||
if f.busy == "" {
|
||||
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
|
||||
}
|
||||
return f.busy, nil
|
||||
}
|
||||
return f.opExec.GuestExec(ctx, vmid, args...)
|
||||
}
|
||||
|
||||
type fakeRebooter struct {
|
||||
mu sync.Mutex
|
||||
n int
|
||||
err error
|
||||
after func()
|
||||
}
|
||||
|
||||
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
r.n++
|
||||
if r.after != nil {
|
||||
r.after()
|
||||
}
|
||||
return r.err
|
||||
}
|
||||
|
||||
func (r *fakeRebooter) count() int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.n
|
||||
}
|
||||
|
||||
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
|
||||
t.Helper()
|
||||
s, ox, ctl, clk, dir, _ := opServer(t, "running")
|
||||
ex := &rbExec{opExec: ox}
|
||||
s.guestExec = ex
|
||||
rb := &fakeRebooter{}
|
||||
s.opRebooter = rb
|
||||
s.inFlight = &backup.InFlight{}
|
||||
return s, ex, rb, ctl, clk, dir
|
||||
}
|
||||
|
||||
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
|
||||
t.Helper()
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
|
||||
return waitResult(t, s, id)
|
||||
}
|
||||
|
||||
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
|
||||
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
|
||||
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
|
||||
s, _, rb, _, clk, _ := rbServer(t)
|
||||
r := pressReboot(t, s, 1)
|
||||
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if rb.count() != 1 {
|
||||
t.Fatalf("reboots = %d, want 1", rb.count())
|
||||
}
|
||||
rec, ok, err := s.lastOperatorReboot(9201)
|
||||
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
|
||||
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
|
||||
}
|
||||
if s.inFlight.Busy() != "" {
|
||||
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
|
||||
}
|
||||
if s.rebooting(9201) {
|
||||
t.Fatal("the reboot-in-flight flag outlived the reboot")
|
||||
}
|
||||
}
|
||||
|
||||
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
|
||||
func TestOpReboot_GuardsRefuse(t *testing.T) {
|
||||
fresh := func(clk *supClock) string {
|
||||
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
|
||||
}
|
||||
cases := []struct {
|
||||
name, want string
|
||||
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
|
||||
}{
|
||||
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
|
||||
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
|
||||
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.guests[0].Status = "stopped"
|
||||
}},
|
||||
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.backupRun = map[int]bool{9201: true}
|
||||
}},
|
||||
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ctl.backupErr = errors.New("tasks unreadable")
|
||||
}},
|
||||
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
s.inFlight.TryAcquire("backup:felhom-pbs")
|
||||
}},
|
||||
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
|
||||
}},
|
||||
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||
ex.busy = fresh(clk)
|
||||
}},
|
||||
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
|
||||
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
|
||||
}},
|
||||
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
|
||||
s.ctrlSup.mu.Lock()
|
||||
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
|
||||
s.ctrlSup.mu.Unlock()
|
||||
}},
|
||||
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
|
||||
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}},
|
||||
}
|
||||
for i, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s, ex, rb, ctl, clk, dir := rbServer(t)
|
||||
tc.setup(s, ex, ctl, clk, dir)
|
||||
r := pressReboot(t, s, int64(100+i))
|
||||
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
|
||||
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
|
||||
}
|
||||
if rb.count() != 0 {
|
||||
t.Fatalf("guard %q did not hold: rebooted", tc.name)
|
||||
}
|
||||
if _, ok, _ := s.lastOperatorReboot(9201); ok {
|
||||
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
|
||||
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
|
||||
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
|
||||
s, ex, rb, _, clk, _ := rbServer(t)
|
||||
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
|
||||
t.Fatalf("result %+v, reboots %d", r, rb.count())
|
||||
}
|
||||
}
|
||||
|
||||
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
|
||||
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
|
||||
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
|
||||
s, _, rb, ctl, clk, dir := rbServer(t)
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("first press: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(59 * time.Minute)
|
||||
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
|
||||
t.Fatalf("second press inside the hour: %+v", r)
|
||||
}
|
||||
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
|
||||
s2, ox2, _, _, _, _ := opServer(t, "running")
|
||||
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
|
||||
s2.guestExec = &rbExec{opExec: ox2}
|
||||
rb2 := &fakeRebooter{}
|
||||
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
|
||||
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
|
||||
t.Fatalf("after an agent restart, inside the hour: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(2 * time.Minute)
|
||||
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
|
||||
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
|
||||
}
|
||||
}
|
||||
|
||||
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
|
||||
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
|
||||
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
|
||||
s, ex, _, _, clk, _ := rbServer(t)
|
||||
for id := int64(1); id <= 2; id++ {
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, id); r.Outcome != "done" {
|
||||
t.Fatalf("restart %d: %+v", id, r)
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
}
|
||||
if r := pressReboot(t, s, 3); r.Outcome != "done" {
|
||||
t.Fatalf("reboot: %+v", r)
|
||||
}
|
||||
clk.t = clk.t.Add(time.Minute)
|
||||
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
|
||||
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
|
||||
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
|
||||
}
|
||||
if n := ex.count(9201); n != 2 {
|
||||
t.Fatalf("restarts = %d, want 2", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
|
||||
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
|
||||
s, _, rb, _, _, _ := rbServer(t)
|
||||
rb.err = errors.New("pct reboot 9201: exit status 255")
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if s.inFlight.Busy() != "" || s.rebooting(9201) {
|
||||
t.Fatal("the gate or the flag outlived a failed reboot")
|
||||
}
|
||||
}
|
||||
|
||||
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
|
||||
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
|
||||
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
|
||||
s, ex, rb, ctl, _, _ := rbServer(t)
|
||||
var started, restarted int
|
||||
rb.after = func() {
|
||||
ctl.guests[0].Status = "stopped"
|
||||
s.GuestPowerTick(context.Background())
|
||||
ctl.guests[0].Status = "running"
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "exited"
|
||||
ex.mu.Unlock()
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
s.ControllerSupervisorTick(context.Background())
|
||||
started, restarted = len(ctl.started), ex.count(9201)
|
||||
ex.mu.Lock()
|
||||
ex.status[9201] = "running"
|
||||
ex.mu.Unlock()
|
||||
}
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
if started != 0 || restarted != 0 {
|
||||
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
|
||||
}
|
||||
}
|
||||
|
||||
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
|
||||
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
|
||||
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
|
||||
s, _, _, _, clk, _ := rbServer(t)
|
||||
read := func() string {
|
||||
w := httptest.NewRecorder()
|
||||
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
|
||||
var body struct {
|
||||
Data struct {
|
||||
At string `json:"at"`
|
||||
} `json:"data"`
|
||||
At string `json:"at"`
|
||||
}
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("body %q: %v", w.Body.String(), err)
|
||||
}
|
||||
if body.Data.At != "" {
|
||||
return body.Data.At
|
||||
}
|
||||
return body.At
|
||||
}
|
||||
if got := read(); got != "" {
|
||||
t.Fatalf("no reboot yet, got %q", got)
|
||||
}
|
||||
at := clk.t
|
||||
if r := pressReboot(t, s, 1); r.Outcome != "done" {
|
||||
t.Fatalf("result %+v", r)
|
||||
}
|
||||
clk.t = at.Add(23 * time.Hour)
|
||||
if got := read(); got != at.UTC().Format(time.RFC3339) {
|
||||
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
|
||||
}
|
||||
clk.t = at.Add(25 * time.Hour)
|
||||
if got := read(); got != "" {
|
||||
t.Fatalf("after 25 h the line must be gone, got %q", got)
|
||||
}
|
||||
}
|
||||
@@ -417,6 +417,13 @@ type Server struct {
|
||||
opTrigger func()
|
||||
opHealthPoll time.Duration
|
||||
opHealthWait time.Duration
|
||||
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
|
||||
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
|
||||
opRebooter interface {
|
||||
RebootGuest(ctx context.Context, vmid int) error
|
||||
}
|
||||
rebootMu sync.Mutex
|
||||
rebootInFlight map[int]bool
|
||||
|
||||
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
|
||||
// detached pipeline runs through (tests inject; production defaults set in NewServer).
|
||||
@@ -603,6 +610,8 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
|
||||
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
|
||||
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
|
||||
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
|
||||
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
|
||||
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
|
||||
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
|
||||
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
|
||||
|
||||
Reference in new issue
Block a user