Compare commits

..
3 Commits
13 changed files with 733 additions and 26 deletions

No files matched your search

+27 -1
View File
@@ -1,4 +1,30 @@
## Unreleased (2026-10-29, afternoon) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — commit only; the next agent release (needs the operator's vouch)
## Unreleased — „Reboot guest" from the hub (D65 = A): `reboot_guest` joins the agent's closed list
- **D65 = A (operator ruling 2026-10-29 15:50; felhom.eu `04` §6.1, `03`).** The agent's closed operator-action list is
now `restart_controller` and `reboot_guest` (arg: the vmid). No signature: a reboot destroys nothing. The act is the
existing guest-reboot path (`pct reboot <vmid>` of THAT guest over the existing exact grant) — never the host.
- **Guards** (each refuses with the reason; nothing is called, no record is written): a guest this agent did not
provision, not running, locked, or parked; the host-wide heavy-operation gate held (a backup of any tier the agent
runs, a restore test, an OS step — the reboot HOLDS the gate until the guest is up again); `jobBlocker` (an agent
backup job, a controller swap, a drive or NAS job, an escrow ceremony, a vzdump of the guest); the CONTROLLER busy —
its `data/ctl/busy` file (a backup, a restore, an off-site run, an app update), fresh within 90 s; a stale file is a
controller that stopped writing, not a busy one; any read failure but „no such file" refuses (fail closed); at most
ONE reboot per hour per guest (`<guests>/<vmid>/operator-reboot.json`, on disk — it survives an agent restart); the
operator restart's brake (a refusal in force refuses a reboot; a reboot counts as one of the 3 restarts in 15 minutes).
- **During the reboot** the guest-power watchdog and the controller supervisor leave the guest alone (`rebootInFlight`):
a racing start would fail the reboot's own start half.
- **The result** says both times from the press: „rebooted; the guest was up after 41 s, the controller answered after
63 s" — or `failed`, said plainly. The record is written BEFORE the reboot; the new self-scoped
`GET /guest/operator-reboot` gives it to the controller for one day, for the household's line „Felhom support
restarted your box at <time>".
- **One new sudo vector** (configs/ changed → the signed config bundle): `pct exec <vmid> -- cat
/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy` — read-only, one fixed path, in
`FELHOM_CONTROLLERSWAP`; capability `controller-busy-read` (not critical: without it, every reboot press is refused).
- Tests `TestOpReboot_*` (the times + the record + the gate given back; 13 guard cases; a stale busy file; once per hour
across an agent restart; the brake counting a reboot; a failed reboot; the watchdogs leaving the guest alone; the
notice expiring after a day) and `TestOpActions_AgentClosedList` — each seen failing by a mutation (19 mutants).
## v0.169.0 (2026-10-29) — „Restart controller" from the hub (the agent's own closed list); the unresponsive controller ALARM (R-1344) — RELEASED (release-agent.sh from a clean clone of `637c92c`; agent sha256 fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61, bundle 3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b; binary-only signed updates committed on demo-felhom, Tester 1, demo-hp) — **VOUCHED by the operator 2026-10-29 14:06:09 CET**; Day-0 pin set (D50). Live on Tester 1: four presses (one with the controller stopped, the fourth refused by the brake)
- **Operator actions for the agent (operator brief 2026-10-29, Part 1; felhom.eu `04` §6.1).** The host-report reply may
carry `operator_actions: [{id, action, arg}]` (`hub.ControlEnvelope.OperatorActions`); the agent acts ONCE per id and
+7
View File
@@ -1,5 +1,12 @@
# CONTEXT — felhom-agent working state
> **2026-10-29 (afternoon, attended) — v0.169.0 RELEASED and VOUCHED (14:06:09 CET).** `fc066de6…`, bundle `3ca33f9c…`,
> from `637c92c`; `pinned.day0_agent` 0.169.0 (`db25eb0`). The agent's own closed operator-action list:
> `restart_controller <vmid>` (host-report reply → result on the next host report; the supervisor's guards; the brake
> counts operator restarts, then `opBrakeUntil` refuses presses 30 min). R-1344: 6 unhealthy sweeps → `unresponsive_since`
> → the hub's `controller_unresponsive` — ALARM ONLY until the night's health log is read (to 2026-10-30 07:00 UTC).
> Proven live on Tester 1: stopped controller restarted, 4th press refused, one press on the release builds „done" in 16 s.
> **2026-10-29 (attended release run) — v0.168.0 RELEASED and VOUCHED (10:21:56 CET).** `release-agent.sh` from a clean
> clone of `91bd96c` (agent `1e70196c…`, bundle `3415c034…`, tag `v0.168.0`); no configs/ change → binary-only signed
> `agent_update` to demo-hp, Tester 1, demo-felhom, each committed, self-check 76/76; `pinned.day0_agent` = 0.168.0 (D50).
+26 -15
View File
@@ -1,18 +1,29 @@
# REPORT — v0.168.0 released and vouched (2026-10-29, attended)
# REPORT — v0.169.0 released and vouched: „Restart controller" from the hub (2026-10-29, attended)
**Plain words first.** The agent 0.168.0 is out, on the three test boxes, and vouched by the operator — new boxes install
it. It carries R-528 (the host's per-container OOM count) and R-1254's ceremony half.
**Plain words first.** The operator can now restart a box's controller from the hub's host page. The agent does it, so it
works when the controller is stopped or stuck. It uses the supervisor's own guards and brake: 3 restarts in 15 minutes,
then presses are refused for 30 minutes. A controller that runs but does not answer now raises an operator alarm
(`controller_unresponsive`); it is not restarted by itself yet — that waits for the night's measurement (R-1344).
- **Release:** `release-agent.sh` from a clean clone of `91bd96c` (CI 2479 green): agent sha256
`1e70196c51346cf10e3f04f51975de5899b5dc6baa5a45ec1d17bc248830d2ca`, bundle
`3415c034955d46c59ca3f7b284a648abf852b1257b77cc49bd6ea9994a7b8cb8`, tag `v0.168.0`, verified by independent download.
- **Delivery:** no configs/ change since v0.167.0 → one signed `agent_update` per box: demo-hp 09:56, Tester 1 09:57,
demo-felhom 10:04 CET — each „signed op COMPLETED", self-check 76/76, „update committed"; the binary's sha256 matches on
all three.
- **Proof:** on Tester 1 (controller 0.347.0) a 128 MB Gokapi had a memory hog killed: host counter 1, one `app_oom`
„seen by docker,cgroup", one operator mail. R-1254: a ceremony ran on 0.168.0 (the happy path); the lost-answer branch
has tests only.
- **Vouch:** the operator, 10:21:56 CET (hub log: agent=0.168.0, the bundle sha above). `pinned.day0_agent` = 0.168.0
(`6b0b093`).
## Commits
Evidence: felhom.eu `documentation/audits/d63-2026-10-29/` (`opsign-agent.txt`, `agent-delivery*.txt`, `run-p4-oom.log`).
| Commit | CI (job id) | What |
|---|---|---|
| `637c92c` | 2504, 2507 green | The agent's closed list: `restart_controller <vmid>` on the host-report reply (`ControlEnvelope.OperatorActions`), the result on the next host report (`OperatorActionResults`). Guards: provisioned, running, parked, swap, lock, vzdump. The brake counts operator restarts with the supervisor's (`opBrakeUntil`). One audit line, one result per id, re-sent until the hub records it; a report is asked for at once. R-1344: 6 unhealthy sweeps → `unresponsive_since` in the stanza. Tests in `internal/localapi/opactions_test.go`, `supervisor_unresponsive_test.go`, `opactions_wiring_test.go`, each seen failing by its mutation. |
| `db25eb0` | 2513 green | v0.169.0 CHANGELOG; `pinned.day0_agent` = 0.169.0 (D50 pin), after the vouch. |
## Release
- `release-agent.sh 0.169.0` from a clean clone: sha256 `fc066de640007038ecfaed6a23e4301a27fc486f1d7851c36177acdd26c2df61`,
bundle `3ca33f9cbab3bdd4c1b7f8ac90aa21e43b64209f1cb45c77ac72b3405956041b`, tag `v0.169.0`, verified by download.
- Signed `agent_update` to the three test boxes; each committed.
- Vouched by the operator 14:06:09 CET (the page answered 504; the save went through — R-1346).
## Live proofs (Tester 1, through the hub endpoint the button posts, CC's `cc-hosts` token)
- 13:01–13:02 CET on rc builds: press 1 → the controller answered after 9 s; press 2 with the controller stopped by hand →
restarted 5 s after the stop; press 3 done; press 4 refused („3 restarts in 15 minutes").
- 14:49 CET on the release builds (agent 0.169.0, controller 0.348.0, hub 0.176.1): one press → „done — restarted; the
controller answered after 9 s", on the hub 16 s after the press; the box showed the controller „Up 12 seconds".
Evidence: felhom.eu `documentation/audits/release-2026-10-29b/` (`p1-restart/`, `final/`).
+3
View File
@@ -139,6 +139,8 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# cat <the controller's busy file> — D65 (2026-10-29): before an operator reboot of the guest, is the controller
# running a backup, a restore, an off-site run or an app update? (read-only, one path)
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
@@ -149,6 +151,7 @@ Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- cat /var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy$, \
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
+1
View File
@@ -151,6 +151,7 @@ var manifest = []Capability{
{"controllerswap-read", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/etc/felhom-controller-image"}, true, ""},
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controller-busy-read", "operator guest reboot (D65)", "/usr/sbin/pct", []string{"exec", "9201", "--", "cat", "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"}, false, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
@@ -254,6 +254,10 @@ func (s *Server) ControllerSupervisorTick(ctx context.Context) {
if !s.provisionedGuest(g.VMID) {
continue
}
if s.rebooting(g.VMID) { // D65: the operator's reboot owns the guest until it is up again
s.resetNotRunning(g.VMID)
continue
}
evaluated++
if !s.superviseOneController(ctx, g.VMID, g.Status) {
down++
+5
View File
@@ -152,6 +152,11 @@ func (s *Server) recoverOneStoppedGuest(ctx context.Context, g proxmox.Guest) {
s.forgetGuestPower(g.VMID) // healthy again: clear any attempt history
return
}
// D65: a guest the operator's reboot has down for a moment is pct reboot's to start, not ours (a racing start
// would fail the reboot's own start half).
if s.rebooting(g.VMID) {
return
}
lock, onboot, err := s.staleLock.Lock(ctx, g.VMID)
if err != nil {
+13 -6
View File
@@ -20,7 +20,7 @@ import (
// the next report answers `operator_action_results: [{id, outcome, message}]` — the agent asks for that report at once.
// The result is re-sent until the reply stops listing the id.
//
// THE LIST IS CLOSED: one verb, restart_controller <vmid>. Anything else — a verb, a vmid that does not parse, a guest
// THE LIST IS CLOSED: two verbs, restart_controller <vmid> and (D65 = A, 2026-10-29) reboot_guest <vmid> (opreboot.go). Anything else — a verb, a vmid that does not parse, a guest
// this agent did not provision — is refused and NOTHING is called. Pinned by TestOpActions_AgentClosedList.
//
// THE ACT is the supervisor's own (controllersupervisor.go): `systemctl restart felhom-controller-bootstrap.service` in
@@ -31,7 +31,7 @@ import (
// controller. Every decision is one `audit:` line (the customer-visible audit form of reconcile.SlogAudit).
const (
// OpRestartController is the agent's one operator verb.
// OpRestartController is the agent's first operator verb.
OpRestartController = "restart_controller"
opOutcomeDone = "done"
@@ -52,7 +52,7 @@ const (
)
// operatorActionNames is THE agent's closed list. Adding an entry fails TestOpActions_AgentClosedList by design.
var operatorActionNames = []string{OpRestartController}
var operatorActionNames = []string{OpRestartController, OpRebootGuest}
// OperatorActionNames returns the closed list (sorted copy).
func OperatorActionNames() []string {
@@ -120,11 +120,18 @@ func (s *Server) OperatorActionResults() []hub.OperatorActionResult {
}
func (s *Server) runOperatorAction(a hub.OperatorAction) {
ctx, cancel := context.WithTimeout(context.Background(), s.opWait()+time.Minute)
limit := s.opWait() + time.Minute
if a.Action == OpRebootGuest {
limit += opRebootTimeout
}
ctx, cancel := context.WithTimeout(context.Background(), limit)
defer cancel()
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller only", a.Action)
if a.Action == OpRestartController {
outcome, msg := opOutcomeRefused, fmt.Sprintf("unknown action %q — the agent's list holds restart_controller and reboot_guest only", a.Action)
switch a.Action {
case OpRestartController:
outcome, msg = s.operatorRestartController(ctx, a.ID, a.Arg)
case OpRebootGuest:
outcome, msg = s.operatorRebootGuest(ctx, a.ID, a.Arg)
}
s.logger.Warn("audit: operator action", "id", a.ID, "action", a.Action, "arg", a.Arg, "by", "the operator via the hub",
"outcome", outcome, "message", msg)
+2 -2
View File
@@ -71,7 +71,7 @@ func waitResult(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
// RED-PROOF: seen failing before the list existed (build failure, then the names).
func TestOpActions_AgentClosedList(t *testing.T) {
if got, want := OperatorActionNames(), []string{"restart_controller"}; !reflect.DeepEqual(got, want) {
if got, want := OperatorActionNames(), []string{"reboot_guest", "restart_controller"}; !reflect.DeepEqual(got, want) {
t.Fatalf("the agent's operator actions = %v, want exactly %v — a new verb needs the operator's word", got, want)
}
}
@@ -80,7 +80,7 @@ func TestOpActions_AgentClosedList(t *testing.T) {
func TestOpActions_UnknownOrBadArgIsRefused(t *testing.T) {
s, ex, _, _, _, _ := opServer(t, "running")
s.OnEnvelope(context.Background(), env(
hub.OperatorAction{ID: 1, Action: "reboot_guest", Arg: "9201"},
hub.OperatorAction{ID: 1, Action: "reboot_host", Arg: "9201"}, // never on the list: only the guest is rebooted
hub.OperatorAction{ID: 2, Action: "restart_controller", Arg: "abc"},
hub.OperatorAction{ID: 3, Action: "restart_controller", Arg: "9202"}, // not provisioned here
))
+328
View File
@@ -0,0 +1,328 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
// reboot_guest — the operator reboots a guest from the hub (D65 = A, operator ruling 2026-10-29 15:50; felhom.eu `04`
// §6.1, `03`). No signature: a reboot destroys nothing (`03`, „healing … non-destructive by construction").
//
// WHY: a stuck Docker or a stuck network inside the box is out of reach of „Restart controller" — that restarts one
// container. The reboot is the existing guest-reboot path (`pct reboot <vmid>`, the household's „A Felhom
// újraindítása", the same exact sudo vector) — of THAT guest, never the host.
//
// THE GUARDS (each refuses with the reason in the result; nothing is called):
// - the guest is not one this agent provisioned, is not running, is locked, or its controller is parked;
// - a controller swap (update) is in flight;
// - a heavy operation holds the host-wide gate (a backup of any tier the agent runs, a restore test, an OS step),
// or jobBlocker finds one (an agent backup job, a drive or NAS job, an escrow ceremony, a vzdump of the guest);
// - the CONTROLLER says it is busy: a backup, a restore, an off-site run or an app update (its `ctl/busy` file,
// read over one exact `pct exec … cat` grant; controllerBusy below). Fails closed on any read error but „no file";
// - at most ONE reboot per hour per guest (the record below, persisted — it survives an agent restart);
// - the operator restart's brake: a reboot counts as a restart, and the brake's refusal refuses a reboot too.
// The heavy-operation gate is HELD from the checks until the guest is up again, so no backup, restore test or OS
// step of this agent can start in between. The guest-power watchdog and the controller supervisor leave the guest
// alone while the reboot runs (rebootInFlight).
//
// AFTER: the result says when the guest was up and when the controller answered (both from the press). The record
// `<guests>/<vmid>/operator-reboot.json` is written BEFORE the reboot, so the controller that starts after it finds
// it: GET /guest/operator-reboot hands it to the household's dashboard („Felhom support restarted your box at …",
// one day). Pinned by opreboot_test.go.
const (
// OpRebootGuest is the agent's second operator verb (D65 = A).
OpRebootGuest = "reboot_guest"
// opRebootEvery: at most one operator reboot per guest per hour.
opRebootEvery = time.Hour
// opRebootNoticeFor: how long the household's dashboard shows the line.
opRebootNoticeFor = 24 * time.Hour
// opRebootTimeout bounds `pct reboot` (it blocks until the guest is back).
opRebootTimeout = 5 * time.Minute
operatorRebootFile = "operator-reboot.json"
// ControllerBusyMarkerGuestPath is the controller's busy file as the GUEST sees it: the controller data volume
// (`felhom-controller-data`, mounted at /opt/docker/felhom-controller in the container) + data/ctl/busy. One exact
// sudo vector reads it (configs/felhom-agent.sudoers, FELHOM_CONTROLLERSWAP). It reads without Docker, so a stuck
// Docker does not hide it.
ControllerBusyMarkerGuestPath = "/var/lib/docker/volumes/felhom-controller-data/_data/data/ctl/busy"
// controllerBusyFresh: the controller rewrites the file every 15 s while busy; an older file is a controller that
// stopped writing (stuck or killed) — not a busy one.
controllerBusyFresh = 90 * time.Second
)
// operatorRebootRecord is the persisted record of the last operator reboot of a guest.
type operatorRebootRecord struct {
At time.Time `json:"at"`
ActionID int64 `json:"action_id"`
GuestUpAfterS int `json:"guest_up_after_s,omitempty"`
ControllerAfterS int `json:"controller_after_s,omitempty"`
}
func (s *Server) operatorRebootPath(vmid int) string {
return filepath.Join(s.guestsStateDir(), strconv.Itoa(vmid), operatorRebootFile)
}
// lastOperatorReboot reads the record; ok=false when there is none. A corrupt record is an error (the hourly limit
// must not silently vanish).
func (s *Server) lastOperatorReboot(vmid int) (operatorRebootRecord, bool, error) {
raw, err := os.ReadFile(s.operatorRebootPath(vmid))
if errors.Is(err, os.ErrNotExist) {
return operatorRebootRecord{}, false, nil
}
if err != nil {
return operatorRebootRecord{}, false, err
}
var rec operatorRebootRecord
if err := json.Unmarshal(raw, &rec); err != nil {
return operatorRebootRecord{}, false, fmt.Errorf("corrupt %s: %w", operatorRebootFile, err)
}
return rec, !rec.At.IsZero(), nil
}
func (s *Server) saveOperatorReboot(vmid int, rec operatorRebootRecord) error {
raw, err := json.Marshal(rec)
if err != nil {
return err
}
p := s.operatorRebootPath(vmid)
tmp := p + ".tmp"
if err := os.WriteFile(tmp, raw, 0o644); err != nil {
return err
}
return os.Rename(tmp, p)
}
// rebooting reports whether an operator reboot of the guest is in flight (the watchdogs leave it alone).
func (s *Server) rebooting(vmid int) bool {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
return s.rebootInFlight[vmid]
}
func (s *Server) setRebooting(vmid int, on bool) {
s.rebootMu.Lock()
defer s.rebootMu.Unlock()
if s.rebootInFlight == nil {
s.rebootInFlight = map[int]bool{}
}
if on {
s.rebootInFlight[vmid] = true
} else {
delete(s.rebootInFlight, vmid)
}
}
func (s *Server) rebooter() interface {
RebootGuest(ctx context.Context, vmid int) error
} {
if s.opRebooter != nil {
return s.opRebooter
}
if s.guestAttach != nil {
return s.guestAttach
}
return nil
}
// controllerBusy reads the controller's busy file. busy=true with the controller's reason when the file is fresh;
// a missing file is „not busy" (an idle controller, or one older than the file); any other read failure is an error
// (the caller refuses — it cannot prove no backup runs).
func (s *Server) controllerBusy(ctx context.Context, vmid int) (bool, string, error) {
out, err := s.guestExec.GuestExec(ctx, vmid, "cat", ControllerBusyMarkerGuestPath)
if err != nil {
if strings.Contains(strings.ToLower(err.Error()+" "+out), "no such file") {
return false, "", nil
}
return false, "", err
}
lines := strings.SplitN(strings.TrimSpace(out), "\n", 2)
at, perr := time.Parse(time.RFC3339, strings.TrimSpace(lines[0]))
if perr != nil {
return false, "", fmt.Errorf("the controller's busy file does not start with a time: %q", lines[0])
}
if s.clock().Sub(at) > controllerBusyFresh {
s.logger.Info("operator reboot: the controller's busy file is stale — the controller stopped writing it; not busy",
"vmid", vmid, "written", at.UTC().Format(time.RFC3339))
return false, "", nil
}
why := "work in progress"
if len(lines) == 2 && strings.TrimSpace(lines[1]) != "" {
why = strings.TrimSpace(lines[1])
}
return true, why, nil
}
// operatorRebootGuest is the press: every guard, then the reboot, then the wait for the guest and the controller.
func (s *Server) operatorRebootGuest(ctx context.Context, id int64, arg string) (string, string) {
vmid, err := strconv.Atoi(arg)
if err != nil || strconv.Itoa(vmid) != arg {
return opOutcomeRefused, fmt.Sprintf("refused: %q is not a guest id", arg)
}
rb := s.rebooter()
if s.staleLock == nil || s.guestExec == nil || rb == nil {
return opOutcomeRefused, "refused: the guest reboot is not wired on this agent"
}
if !s.provisionedGuest(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not a guest this agent provisioned", vmid)
}
if s.rebooting(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: a reboot of guest %d is already running", vmid)
}
guests, err := s.staleLock.Guests(ctx)
if err != nil {
return opOutcomeRefused, "refused: the guest list cannot be read — ownership unproven, nothing touched"
}
status := ""
for _, g := range guests {
if g.VMID == vmid {
status = g.Status
}
}
if status != "running" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is not running (%s) — the guest-power watchdog owns it", vmid, orDash(status))
}
if s.controllerParked(vmid) {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d's controller is parked (remove the controller-parked marker first)", vmid)
}
// A controller swap (update) is refused by jobBlocker below, with the agent's other jobs.
if lock, _, err := s.staleLock.Lock(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the guest lock cannot be read — nothing touched"
} else if lock != "" {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d is locked (%s) — another operation owns it", vmid, lock)
}
// At most one per hour.
now := s.clock()
last, had, err := s.lastOperatorReboot(vmid)
if err != nil {
return opOutcomeRefused, "refused: the last reboot's record cannot be read — " + err.Error()
}
if had && now.Sub(last.At) < opRebootEvery {
return opOutcomeRefused, fmt.Sprintf("refused: guest %d was rebooted from the hub at %s UTC — at most one reboot per hour (next from %s UTC)",
vmid, last.At.UTC().Format("15:04"), last.At.Add(opRebootEvery).UTC().Format("15:04"))
}
// The brake: a refusal in force refuses a reboot too, and 3 restarts in 15 minutes refuse it.
s.ctrlSup.mu.Lock()
st := s.supState(vmid)
if now.Before(st.opBrakeUntil) {
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
st.restarts = pruneBefore(st.restarts, now.Add(-controllerCrashloopWindow))
if len(st.restarts) >= controllerCrashloopMax {
st.opBrakeUntil = now.Add(opBrakeRefusal)
until := st.opBrakeUntil
s.ctrlSup.mu.Unlock()
return opOutcomeRefused, fmt.Sprintf("refused: 3 restarts in 15 minutes — presses are refused until %s UTC", until.UTC().Format("15:04"))
}
s.ctrlSup.mu.Unlock()
// The heavy-operation gate, held until the guest is up again.
release, busy, ok := s.inFlight.TryAcquire("guest-reboot")
if !ok {
return opOutcomeRefused, "refused: a backup or another heavy operation runs on the host (" + busy + ")"
}
defer release()
if code, detail := s.jobBlocker(ctx, vmid); code != "" {
return opOutcomeRefused, "refused: " + rebootBlockerSentence(code) + " (" + detail + ")"
}
if cbusy, why, err := s.controllerBusy(ctx, vmid); err != nil {
return opOutcomeRefused, "refused: the controller's busy file cannot be read, so a running backup cannot be ruled out — " + err.Error()
} else if cbusy {
return opOutcomeRefused, "refused: the controller is busy — " + why
}
// The record FIRST: the hourly limit and the household's line both read it.
rec := operatorRebootRecord{At: now, ActionID: id}
if err := s.saveOperatorReboot(vmid, rec); err != nil {
return opOutcomeRefused, "refused: the reboot cannot be recorded — " + err.Error()
}
s.ctrlSup.mu.Lock()
st = s.supState(vmid)
st.restarts = append(st.restarts, now)
st.restartsTotal++
st.lastRestartAt = now
st.lastReason = fmt.Sprintf("guest reboot by the operator via the hub (action #%d)", id)
st.notRunningSeen = 0
s.ctrlSup.mu.Unlock()
s.setRebooting(vmid, true)
defer s.setRebooting(vmid, false)
s.logger.Warn("operator reboot: REBOOTING the guest on the operator's press", "vmid", vmid, "action_id", id)
began := time.Now()
rctx, cancel := context.WithTimeout(ctx, opRebootTimeout)
err = rb.RebootGuest(rctx, vmid)
cancel()
if err != nil {
return opOutcomeFailed, fmt.Sprintf("the reboot of guest %d failed: %v — look at the box (the guest-power watchdog starts a stopped guest)", vmid, err)
}
upS := int(time.Since(began).Round(time.Second) / time.Second)
rec.GuestUpAfterS = upS
// The controller's own HEALTHCHECK is the answer, as for a restart; the wait starts when the guest is up.
for deadline := time.Now().Add(s.opWait()); time.Now().Before(deadline); {
out, err := s.guestExec.GuestExec(ctx, vmid, "docker", "inspect", "-f", opHealthTemplate, controllerContainer)
if err == nil {
f := strings.SplitN(strings.TrimSpace(out), "|", 2)
if len(f) == 2 && f[0] == "true" && (f[1] == "healthy" || f[1] == "none") {
rec.ControllerAfterS = int(time.Since(began).Round(time.Second) / time.Second)
if serr := s.saveOperatorReboot(vmid, rec); serr != nil {
s.logger.Warn("operator reboot: the times could not be added to the record", "vmid", vmid, "err", serr)
}
return opOutcomeDone, fmt.Sprintf("rebooted; the guest was up after %d s, the controller answered after %d s", upS, rec.ControllerAfterS)
}
}
select {
case <-ctx.Done():
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the wait for the controller was cut", upS)
case <-time.After(s.opPoll()):
}
}
_ = s.saveOperatorReboot(vmid, rec)
return opOutcomeFailed, fmt.Sprintf("rebooted; the guest was up after %d s, but the controller gave no healthy answer within %s — look at the box", upS, s.opWait().Round(time.Second))
}
// rebootBlockerSentence names jobBlocker's code in the result's words.
func rebootBlockerSentence(code string) string {
switch code {
case PowerOffBackup:
return "a backup is running"
case PowerOffCtrlUpdate:
return "a controller update is running"
case PowerOffDrive:
return "a drive or network-storage job is running"
case PowerOffEscrow:
return "an escrow ceremony is running"
case PowerOffUnknown:
return "the running jobs cannot be read"
default:
return "the host is busy"
}
}
// handleOperatorRebootNotice — GET /guest/operator-reboot (self-scoped): the last operator reboot of THIS guest, when it
// is less than a day old, for the household's dashboard line. {"at": ""} when there is none.
func (s *Server) handleOperatorRebootNotice(w http.ResponseWriter, _ *http.Request, vmid int) {
rec, ok, err := s.lastOperatorReboot(vmid)
if err != nil {
writeErr(w, http.StatusInternalServerError, "the reboot record cannot be read")
return
}
if !ok || s.clock().Sub(rec.At) >= opRebootNoticeFor {
writeOK(w, map[string]any{"vmid": vmid, "at": ""})
return
}
writeOK(w, map[string]any{"vmid": vmid, "at": rec.At.UTC().Format(time.RFC3339), "by": "operator"})
}
+306
View File
@@ -0,0 +1,306 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// D65 = A (operator 2026-10-29 15:50): „Reboot guest" from the hub. The agent's reboot_guest — the existing
// `pct reboot` of THAT guest — with its guards, the hourly limit, the brake, and the record the household's line reads.
// rbExec adds the controller's busy file to the operator-restart fake.
type rbExec struct {
*opExec
busy string // the file's content; "" = no file
busyErr error // pct exec itself failed
}
func (f *rbExec) GuestExec(ctx context.Context, vmid int, args ...string) (string, error) {
if len(args) == 2 && args[0] == "cat" && args[1] == ControllerBusyMarkerGuestPath {
if f.busyErr != nil {
return "", f.busyErr
}
if f.busy == "" {
return "", errors.New("pct exec 9201 [cat ...]: exit status 1: cat: " + ControllerBusyMarkerGuestPath + ": No such file or directory")
}
return f.busy, nil
}
return f.opExec.GuestExec(ctx, vmid, args...)
}
type fakeRebooter struct {
mu sync.Mutex
n int
err error
after func()
}
func (r *fakeRebooter) RebootGuest(context.Context, int) error {
r.mu.Lock()
defer r.mu.Unlock()
r.n++
if r.after != nil {
r.after()
}
return r.err
}
func (r *fakeRebooter) count() int {
r.mu.Lock()
defer r.mu.Unlock()
return r.n
}
func rbServer(t *testing.T) (*Server, *rbExec, *fakeRebooter, *fakeGuestPowerCtl, *supClock, string) {
t.Helper()
s, ox, ctl, clk, dir, _ := opServer(t, "running")
ex := &rbExec{opExec: ox}
s.guestExec = ex
rb := &fakeRebooter{}
s.opRebooter = rb
s.inFlight = &backup.InFlight{}
return s, ex, rb, ctl, clk, dir
}
func pressReboot(t *testing.T, s *Server, id int64) hub.OperatorActionResult {
t.Helper()
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "reboot_guest", Arg: "9201"}))
return waitResult(t, s, id)
}
// The press reboots ONCE and says when the guest was up and when the controller answered; the record is written for
// the household's line. RED-PROOF: drop the RebootGuest call → reboots 0 → FAILS.
func TestOpReboot_RebootsAndReportsTheTimes(t *testing.T) {
s, _, rb, _, clk, _ := rbServer(t)
r := pressReboot(t, s, 1)
if r.Outcome != "done" || !strings.Contains(r.Message, "the guest was up after") || !strings.Contains(r.Message, "the controller answered after") {
t.Fatalf("result %+v", r)
}
if rb.count() != 1 {
t.Fatalf("reboots = %d, want 1", rb.count())
}
rec, ok, err := s.lastOperatorReboot(9201)
if err != nil || !ok || !rec.At.Equal(clk.t) || rec.ActionID != 1 {
t.Fatalf("record %+v ok=%v err=%v", rec, ok, err)
}
if s.inFlight.Busy() != "" {
t.Fatalf("the heavy-operation gate is still held after the reboot: %q", s.inFlight.Busy())
}
if s.rebooting(9201) {
t.Fatal("the reboot-in-flight flag outlived the reboot")
}
}
// Each guard refuses with its reason and NOTHING is rebooted. RED-PROOF: each guard seen failing by deleting it.
func TestOpReboot_GuardsRefuse(t *testing.T) {
fresh := func(clk *supClock) string {
return clk.t.Add(-20*time.Second).Format(time.RFC3339) + "\nan off-site backup is running\n"
}
cases := []struct {
name, want string
setup func(s *Server, ex *rbExec, ctl *fakeGuestPowerCtl, clk *supClock, dir string)
}{
{"parked", "parked", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.WriteFile(filepath.Join(dir, "9201", ControllerParkedMarker), nil, 0o600); err != nil {
panic(err)
}
}},
{"swap", "swap", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) { s.swapInFlight[9201] = true }},
{"locked", "locked", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) { ctl.locks[9201] = "backup" }},
{"stopped", "not running", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.guests[0].Status = "stopped"
}},
{"vzdump", "backup", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupRun = map[int]bool{9201: true}
}},
{"vzdump unknown", "cannot be read", func(_ *Server, _ *rbExec, ctl *fakeGuestPowerCtl, _ *supClock, _ string) {
ctl.backupErr = errors.New("tasks unreadable")
}},
{"heavy gate", "backup:felhom-pbs", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.inFlight.TryAcquire("backup:felhom-pbs")
}},
{"agent backup job", "a backup is running", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
s.jobs = map[backupJobKey]*backupJob{{vmid: 9201, target: "local"}: {JobID: "j1", Phase: PhaseRunning}}
}},
{"controller busy", "off-site backup", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
ex.busy = fresh(clk)
}},
{"busy file unreadable", "cannot be ruled out", func(_ *Server, ex *rbExec, _ *fakeGuestPowerCtl, _ *supClock, _ string) {
ex.busyErr = errors.New("pct exec 9201: exit status 255: lxc-attach failed")
}},
{"brake in force", "3 restarts in 15 minutes", func(s *Server, _ *rbExec, _ *fakeGuestPowerCtl, clk *supClock, _ string) {
s.ctrlSup.mu.Lock()
s.supState(9201).opBrakeUntil = clk.t.Add(10 * time.Minute)
s.ctrlSup.mu.Unlock()
}},
{"not provisioned", "not a guest this agent provisioned", func(_ *Server, _ *rbExec, _ *fakeGuestPowerCtl, _ *supClock, dir string) {
if err := os.RemoveAll(filepath.Join(dir, "9201", "bootstrap")); err != nil {
panic(err)
}
}},
}
for i, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s, ex, rb, ctl, clk, dir := rbServer(t)
tc.setup(s, ex, ctl, clk, dir)
r := pressReboot(t, s, int64(100+i))
if r.Outcome != "refused" || !strings.Contains(r.Message, tc.want) {
t.Fatalf("result %+v, want refused naming %q", r, tc.want)
}
if rb.count() != 0 {
t.Fatalf("guard %q did not hold: rebooted", tc.name)
}
if _, ok, _ := s.lastOperatorReboot(9201); ok {
t.Fatal("a refused press wrote the reboot record (the household would read a reboot that never happened)")
}
})
}
}
// A busy file the controller stopped writing (stuck, killed) is not a busy controller — the reboot exists for exactly
// that box. RED-PROOF: drop the freshness check → refused → FAILS.
func TestOpReboot_StaleBusyFileDoesNotBlock(t *testing.T) {
s, ex, rb, _, clk, _ := rbServer(t)
ex.busy = clk.t.Add(-10*time.Minute).Format(time.RFC3339) + "\na backup is running\n"
if r := pressReboot(t, s, 1); r.Outcome != "done" || rb.count() != 1 {
t.Fatalf("result %+v, reboots %d", r, rb.count())
}
}
// At most one reboot per hour per guest, and the limit survives an agent restart (the record is on disk).
// RED-PROOF: drop the hourly check → the second press reboots → FAILS.
func TestOpReboot_AtMostOncePerHour(t *testing.T) {
s, _, rb, ctl, clk, dir := rbServer(t)
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("first press: %+v", r)
}
clk.t = clk.t.Add(59 * time.Minute)
if r := pressReboot(t, s, 2); r.Outcome != "refused" || !strings.Contains(r.Message, "at most one reboot per hour") {
t.Fatalf("second press inside the hour: %+v", r)
}
// a restarted agent (a fresh in-memory state over the same guests dir) still refuses
s2, ox2, _, _, _, _ := opServer(t, "running")
s2.guestsDir, s2.staleLock, s2.now = dir, ctl, clk.now
s2.guestExec = &rbExec{opExec: ox2}
rb2 := &fakeRebooter{}
s2.opRebooter, s2.inFlight = rb2, &backup.InFlight{}
if r := pressReboot(t, s2, 3); r.Outcome != "refused" || rb2.count() != 0 {
t.Fatalf("after an agent restart, inside the hour: %+v", r)
}
clk.t = clk.t.Add(2 * time.Minute)
if r := pressReboot(t, s, 4); r.Outcome != "done" || rb.count() != 2 {
t.Fatalf("after the hour: %+v (reboots %d)", r, rb.count())
}
}
// The brake counts a reboot as a restart: two operator restarts and one reboot inside 15 minutes refuse the next
// restart press. RED-PROOF: drop the restarts append in operatorRebootGuest → the 4th press restarts → FAILS.
func TestOpReboot_TheBrakeCountsAReboot(t *testing.T) {
s, ex, _, _, clk, _ := rbServer(t)
for id := int64(1); id <= 2; id++ {
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: id, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, id); r.Outcome != "done" {
t.Fatalf("restart %d: %+v", id, r)
}
clk.t = clk.t.Add(time.Minute)
}
if r := pressReboot(t, s, 3); r.Outcome != "done" {
t.Fatalf("reboot: %+v", r)
}
clk.t = clk.t.Add(time.Minute)
s.OnEnvelope(context.Background(), env(hub.OperatorAction{ID: 4, Action: "restart_controller", Arg: "9201"}))
if r := waitResult(t, s, 4); r.Outcome != "refused" || !strings.Contains(r.Message, "3 restarts in 15 minutes") {
t.Fatalf("the 4th restart (2 restarts + 1 reboot) in 15 minutes: %+v", r)
}
if n := ex.count(9201); n != 2 {
t.Fatalf("restarts = %d, want 2", n)
}
}
// A failed `pct reboot` is a FAILED press, said plainly; the gate and the flag are given back.
func TestOpReboot_FailedRebootIsFailed(t *testing.T) {
s, _, rb, _, _, _ := rbServer(t)
rb.err = errors.New("pct reboot 9201: exit status 255")
if r := pressReboot(t, s, 1); r.Outcome != "failed" || !strings.Contains(r.Message, "failed") {
t.Fatalf("result %+v", r)
}
if s.inFlight.Busy() != "" || s.rebooting(9201) {
t.Fatal("the gate or the flag outlived a failed reboot")
}
}
// While the reboot runs, the guest-power watchdog does not start the (momentarily stopped) guest and the supervisor
// does not restart its controller. RED-PROOF: drop either rebooting() check → a start or a restart → FAILS.
func TestOpReboot_TheWatchdogsLeaveARebootingGuestAlone(t *testing.T) {
s, ex, rb, ctl, _, _ := rbServer(t)
var started, restarted int
rb.after = func() {
ctl.guests[0].Status = "stopped"
s.GuestPowerTick(context.Background())
ctl.guests[0].Status = "running"
ex.mu.Lock()
ex.status[9201] = "exited"
ex.mu.Unlock()
s.ControllerSupervisorTick(context.Background())
s.ControllerSupervisorTick(context.Background())
started, restarted = len(ctl.started), ex.count(9201)
ex.mu.Lock()
ex.status[9201] = "running"
ex.mu.Unlock()
}
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
if started != 0 || restarted != 0 {
t.Fatalf("during the reboot: guest-power started %d, the supervisor restarted %d — both must leave it to pct reboot", started, restarted)
}
}
// The household's line: GET /guest/operator-reboot returns the time for a day, then nothing.
// RED-PROOF: drop the 24-hour check → the 25-hour read still returns the time → FAILS.
func TestOpReboot_TheNoticeExpiresAfterADay(t *testing.T) {
s, _, _, _, clk, _ := rbServer(t)
read := func() string {
w := httptest.NewRecorder()
s.handleOperatorRebootNotice(w, httptest.NewRequest("GET", "/guest/operator-reboot", nil), 9201)
var body struct {
Data struct {
At string `json:"at"`
} `json:"data"`
At string `json:"at"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatalf("body %q: %v", w.Body.String(), err)
}
if body.Data.At != "" {
return body.Data.At
}
return body.At
}
if got := read(); got != "" {
t.Fatalf("no reboot yet, got %q", got)
}
at := clk.t
if r := pressReboot(t, s, 1); r.Outcome != "done" {
t.Fatalf("result %+v", r)
}
clk.t = at.Add(23 * time.Hour)
if got := read(); got != at.UTC().Format(time.RFC3339) {
t.Fatalf("after 23 h: %q, want %s", got, at.UTC().Format(time.RFC3339))
}
clk.t = at.Add(25 * time.Hour)
if got := read(); got != "" {
t.Fatalf("after 25 h the line must be gone, got %q", got)
}
}
+9
View File
@@ -417,6 +417,13 @@ type Server struct {
opTrigger func()
opHealthPoll time.Duration
opHealthWait time.Duration
// D65 = A (2026-10-29): the operator's guest reboot (opreboot.go). opRebooter is a test seam (nil → guestAttach);
// rebootInFlight keeps the guest-power watchdog and the controller supervisor off a guest being rebooted.
opRebooter interface {
RebootGuest(ctx context.Context, vmid int) error
}
rebootMu sync.Mutex
rebootInFlight map[int]bool
// Network-storage verify job (SPIKE-nas-verify): the IN-MEMORY single slot + the seams the
// detached pipeline runs through (tests inject; production defaults set in NewServer).
@@ -603,6 +610,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
// D65 = A: the last operator reboot of THIS guest (< 1 day), for the household's dashboard line (opreboot.go).
mux.HandleFunc("GET /guest/operator-reboot", s.withGuest(s.handleOperatorRebootNotice))
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
+2 -2
View File
@@ -25,9 +25,9 @@
"scripts/check-published-versions.py — asserts the newest N versions plus every pinned version"
],
"recorded": "2026-10-29",
"recorded_by": "CC 2026-10-29: day0_agent 0.168.0 (operator vouch 2026-10-29 10:21:56 CET, read from the hub log: agent=0.168.0, bundle 3415c034… = the release); earlier 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"recorded_by": "CC 2026-10-29: day0_agent 0.169.0 (operator vouch 2026-10-29 14:06:09 CET, read from the hub log: agent=0.169.0, bundle 3ca33f9c… = the release; his browser got a 504 but the save went through); earlier 0.168.0 (vouch 10:21:56 CET); 0.167.0 (vouch 2026-10-27); 0.166.0 (vouch 2026-10-26 10:04:55 CET); D50 = A (operator 2026-10-25); earlier history: R-287 (2026-08-09), R-291 (2026-10-05)",
"pinned": {
"hub_min_agent": "0.158.1",
"day0_agent": "0.168.0"
"day0_agent": "0.169.0"
}
}