v0.246.0: an interrupted restore is told; the recovery-code reminder waits until the box can take it
gates / gates (push) Successful in 15s

MinAgent: 0.131.0 (unchanged). Requires hub v0.117.0 for restore_interrupted.

R-550 (operator ruling: fix). A design reversed and recorded: the restore
op-status was in memory by choice. Now restore-status.json in DataDir, written
atomically at both ends of an op. At startup a record still marked running
becomes a failed, interrupted result kept per app until that app's next
restore, shown on /backups/restore and the off-site wizard, and raised once as
restore_interrupted. Cooldowns stay in memory.

R-546. The R-543 reminder bar consults the agent's own preflight ok (every
blocking item, not a copy of pbs_storage_id), cached 60 s, probed only while
paused. /backup/escrow shows a waiting card that polls and reloads instead of
red crosses and English diagnostics. POST /api/escrow/start refuses 409 before
staging or starting - the direct path chaos night used. Unknown readiness keeps
the bar.

Red-proofs (each seen failing): restore record across restart; main() calls
both startup functions; startup helper with loading skipped; restore page card;
bar held back; waiting card; start refusal. go build/vet/test ./... green, 28
packages; controller_gates --fast all OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
adminandClaude Opus 5 committed 2026-09-17 10:45:29 +02:00
1 parent 714d5bce09
commit 0fe315b759
21 files changed
+788 -10

No files matched your search

+5
View File
@@ -53,6 +53,11 @@ func (s *Server) escrowBannerVisible(r *http.Request) bool {
if r == nil || !s.hasAdminSession(r) || !s.escrowPaused() {
return false
}
// R-546: while the agent says the ceremony cannot run yet, do not urge the household into it.
// Unknown readiness keeps the bar (fail loud).
if ready, known := s.escrowReadiness(r.Context(), false); known && !ready {
return false
}
if c, err := r.Cookie(escrowBannerCookie); err == nil && c.Value == "1" {
return false
}
@@ -127,6 +127,15 @@ func (s *Server) escrowWizardPageHandler(w http.ResponseWriter, r *http.Request)
"OffboxConfigured": s.backupMgr != nil && s.backupMgr.OffboxConfigured(),
"AgentSupported": escrowAgentSupported(agentVer),
}
// R-546: a paused box whose agent says the ceremony cannot run yet gets the waiting card, not a red
// checklist and a start form that would refuse. Only for the FIRST ceremony — an escrowed box's
// re-ceremony keeps the checklist, where a red row is a real fault to show.
if !escrowed {
if ready, known := s.escrowReadiness(r.Context(), true); known && !ready {
data["EscrowNotReady"] = true
data["EscrowNotReadyMessage"] = escrowNotReadyMessage
}
}
s.executeTemplate(w, r, "backups_escrow", data)
}
@@ -188,6 +197,16 @@ func (s *Server) escrowStartAPIHandler(w http.ResponseWriter, r *http.Request) {
}
s.clearAuthFailures(ip)
// (1b) R-546: readiness. The page hides the start form while the agent's preflight is red, but a
// direct POST (the path chaos night used) ran the ceremony and returned the agent's raw stderr.
// Refuse here, BEFORE anything is staged or started. Unknown readiness does not refuse: the
// agent-conn and version gates below answer that case, as before.
if ready, known := s.escrowReadiness(r.Context(), true); known && !ready {
s.logger.Printf("[INFO] [web] escrow start refused: the box is not ready yet (agent preflight not ok)")
escrowJSON(w, http.StatusConflict, nil, escrowNotReadyMessage)
return
}
// (2) Re-stage-first (only when offsite is configured — Scenario C boxes skip it).
if s.backupMgr != nil && s.backupMgr.OffboxConfigured() {
if err := s.escrowStage(r.Context()); err != nil {
+104
View File
@@ -0,0 +1,104 @@
package web
import (
"context"
"sort"
"strings"
"sync"
"time"
)
// R-546 — is the box READY to run the escrow ceremony?
//
// Measured 2026-09-16/17 (chaos night Phase 0): after a fresh bind the box has no PBS storage for ~17
// minutes. The agent's preflight is red, the ceremony refuses, and the R-543 reminder bar (v0.245.0)
// was on every page urging the household into it. The escrow page itself already hid its start form
// behind a red checklist — but it said nothing about WAITING, and a direct POST /api/escrow/start
// (the path chaos night used) ran the ceremony and returned the agent's raw "-storage" stderr.
//
// THE DEFINITION IS THE AGENT'S OWN `ok`: every blocking preflight item (pbs_storage_id, dr_tier,
// age_binary, hub_upload, sudo_grant; staged_secret is informational). Not a controller-side copy of
// one of those items — the escrow.pbs_storage_id reading of R-546 was the SYMPTOM that box showed, and
// a copy of one item is a second definition that drifts when the agent grows a sixth.
//
// Three answers, and UNKNOWN keeps today's behaviour (the bar shows): an unreachable agent must not
// silence a reminder that R-543 made loud on purpose.
//
// Cost: the bar hangs off every page render, so the answer is cached for escrowReadyTTL, and a probe
// is only ever made while the box is paused (escrowPaused gates it) — a finished box never asks.
const (
escrowReadyTTL = 60 * time.Second
escrowReadyTimeout = 3 * time.Second
)
type escrowReadinessCache struct {
mu sync.Mutex
ready bool
known bool
checkedAt time.Time
lastState string // for transition-only INFO logging
}
// escrowReadiness returns (ready, known). fresh=true skips the cache (the escrow page and the start
// API, where one request justifies one probe); fresh=false is the bar's cached read.
func (s *Server) escrowReadiness(ctx context.Context, fresh bool) (ready, known bool) {
c := &s.escrowReady
c.mu.Lock()
defer c.mu.Unlock()
if !fresh && !c.checkedAt.IsZero() && time.Since(c.checkedAt) < escrowReadyTTL {
return c.ready, c.known
}
pctx, cancel := context.WithTimeout(ctx, escrowReadyTimeout)
defer cancel()
ready, known, why := s.probeEscrowReadiness(pctx)
c.ready, c.known, c.checkedAt = ready, known, time.Now()
state := "unknown"
if known && ready {
state = "ready"
} else if known {
state = "not-ready"
}
if state != c.lastState {
switch state {
case "ready":
s.logger.Printf("[INFO] [web] escrow readiness: READY (agent preflight ok) — the recovery-code reminder is shown")
case "not-ready":
s.logger.Printf("[INFO] [web] escrow readiness: NOT READY (%s) — reminder held back; the escrow page says the box is still preparing", why)
default:
s.logger.Printf("[INFO] [web] escrow readiness: UNKNOWN (%s) — reminder shown (fail loud)", why)
}
c.lastState = state
} else if s.isDebug() {
s.logger.Printf("[DEBUG] [web] escrow readiness: %s (%s)", state, why)
}
return ready, known
}
// probeEscrowReadiness asks the agent. Never an error to the caller: failure is "unknown".
func (s *Server) probeEscrowReadiness(ctx context.Context) (ready, known bool, why string) {
agent, err := s.escrowAgentConn()
if err != nil {
return false, false, "agent unavailable: " + err.Error()
}
pf, err := agent.EscrowPreflight(ctx)
if err != nil {
return false, false, "preflight failed: " + err.Error()
}
if pf.OK {
return true, true, "preflight ok"
}
var failing []string
for _, it := range pf.Items {
if !it.OK && it.ID != "staged_secret" {
failing = append(failing, it.ID)
}
}
sort.Strings(failing)
return false, true, "failing: " + strings.Join(failing, ", ")
}
// escrowNotReadyMessage is the one Hungarian sentence for "wait" — the page card and the API refusal
// say the same thing.
const escrowNotReadyMessage = "A doboz még készül — a távoli mentés kulcsát pár perc múlva tudod létrehozni. Ez az oldal magától frissül."
@@ -87,6 +87,9 @@ func newEscrowWizardHarness(t *testing.T) *escrowWizardHarness {
version: "0.88.0",
startResp: agentapi.EscrowCeremonyStartResponse{JobID: "escrow-1", Phase: "running"},
startStatus: http.StatusAccepted,
// R-546 (v0.246.0): the start API now asks the agent's preflight first; this harness is a
// READY box. A not-ready box is r546_escrow_readiness_test.go.
pf: agentapi.EscrowPreflightResponse{OK: true},
}
h.s = &Server{cfg: cfg, backupMgr: h.m, settings: sett, logger: lg}
h.s.escrowAgentFn = func() (escrowAgent, error) { return h.agent, nil }
@@ -131,8 +134,8 @@ func TestEscrowStart_StagesBeforeTrigger(t *testing.T) {
if w.Code != http.StatusOK {
t.Fatalf("start: got %d (%s)", w.Code, w.Body.String())
}
if got := strings.Join(h.order, ","); got != "stage,start" {
t.Fatalf("call order = %q, want stage BEFORE start (a ceremony without the staged secret mints a hash-less blob)", got)
if got := strings.Join(h.order, ","); got != "preflight,stage,start" { // R-546: readiness first
t.Fatalf("call order = %q, want preflight, then stage BEFORE start (a ceremony without the staged secret mints a hash-less blob)", got)
}
if !strings.Contains(w.Body.String(), "escrow-1") {
t.Fatalf("response lacks the job id: %s", w.Body.String())
@@ -147,8 +150,8 @@ func TestEscrowStart_ReceremonyStagesToo(t *testing.T) {
if w := postStart(t, h.s, wizardPassword); w.Code != http.StatusOK {
t.Fatalf("re-ceremony start: got %d", w.Code)
}
if got := strings.Join(h.order, ","); got != "stage,start" {
t.Fatalf("re-ceremony call order = %q, want stage,start", got)
if got := strings.Join(h.order, ","); got != "preflight,stage,start" { // R-546: readiness first
t.Fatalf("re-ceremony call order = %q, want preflight,stage,start", got)
}
}
@@ -159,8 +162,8 @@ func TestEscrowStart_NoOffboxSkipsStaging(t *testing.T) {
if w := postStart(t, h.s, wizardPassword); w.Code != http.StatusOK {
t.Fatalf("start: got %d", w.Code)
}
if got := strings.Join(h.order, ","); got != "start" {
t.Fatalf("call order = %q, want start only (no staging without offbox)", got)
if got := strings.Join(h.order, ","); got != "preflight,start" { // R-546: readiness first
t.Fatalf("call order = %q, want preflight then start only (no staging without offbox)", got)
}
}
+4
View File
@@ -1067,6 +1067,10 @@ func (s *Server) backupsAppsHandler(w http.ResponseWriter, r *http.Request) {
// restore-to-verify list and the .fab export/import loop.
func (s *Server) backupsRestoreHandler(w http.ResponseWriter, r *http.Request) {
data := s.backupsCommonData("backups-restore", "Biztonsági mentés — Visszaállítás", r)
// R-550: restores the box was running when it stopped — shown until that app is restored again.
if s.backupMgr != nil {
data["InterruptedRestores"] = s.backupMgr.InterruptedRestores()
}
s.backupsOffboxData(data) // restore-to-verify lists the offbox-toggled apps
// Full-restore two-step reveal (§7.2): after the size+headroom prepare step, offboxRestoreHandler
// redirects here with the app + human size so the confirm section can show the size BEFORE starting.
@@ -0,0 +1,113 @@
package web
import (
"errors"
"net/http"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
)
// ── R-546 — do not send the household to a ceremony the box cannot run yet ────────────────────────
//
// Measured 2026-09-16/17 (chaos night Phase 0): for ~17 minutes after the bind the box has no PBS
// storage, the agent's preflight is red, and the ceremony refuses. Meanwhile the R-543 bar is on every
// page urging the household there. Readiness is the AGENT'S OWN `ok` — every blocking preflight item
// (pbs_storage_id, dr_tier, age_binary, hub_upload, sudo_grant), never a copy of one of them.
const escrowNotReadyMarker = `id="escrow-not-ready"`
func readinessAgent(ok bool) *fakeEscrowAgent {
order := []string{}
pf := agentapi.EscrowPreflightResponse{OK: ok}
if !ok {
pf.Items = []agentapi.EscrowPreflightItem{{ID: "pbs_storage_id", OK: false, Detail: "escrow.pbs_storage_id not configured"}}
}
return &fakeEscrowAgent{order: &order, version: "0.132.0", pf: pf}
}
// RED-PROOF: remove the readiness check from escrowBannerVisible → the bar renders on a box that
// cannot run the ceremony → "the bar urges a ceremony the box cannot run yet".
func TestR546_NotReadyBoxHoldsTheBar(t *testing.T) {
s := escrowServer(t, "pending")
a := readinessAgent(false)
s.escrowAgentFn = func() (escrowAgent, error) { return a, nil }
body := getPage(t, s, "/dashboard").Body.String()
if strings.Contains(body, escrowBarSentence) {
t.Fatal("R-546: the bar urges a ceremony the box cannot run yet (agent preflight is red)")
}
}
// Control: once the agent says ready, the R-543 bar is back.
func TestR546_ReadyBoxShowsTheBar(t *testing.T) {
s := escrowServer(t, "pending")
a := readinessAgent(true)
s.escrowAgentFn = func() (escrowAgent, error) { return a, nil }
if body := getPage(t, s, "/dashboard").Body.String(); !strings.Contains(body, escrowBarSentence) {
t.Fatal("a READY, paused box no longer shows the R-543 bar")
}
}
// Fail loud: readiness UNKNOWN (agent unreachable) keeps the reminder — silence is not a safe default.
func TestR546_UnknownReadinessStillAsks(t *testing.T) {
s := escrowServer(t, "pending")
s.escrowAgentFn = func() (escrowAgent, error) { return nil, errors.New("agent unreachable") }
if body := getPage(t, s, "/dashboard").Body.String(); !strings.Contains(body, escrowBarSentence) {
t.Fatal("an unknown readiness hid the reminder — an unreachable agent must not silence R-543")
}
}
// The page itself: a calm waiting card, no start form, and none of the agent's raw diagnostics.
// RED-PROOF: never set EscrowNotReady → the checklist + start form render → "start form offered".
func TestR546_EscrowPageWaitsWhenNotReady(t *testing.T) {
s := escrowServer(t, "pending")
a := readinessAgent(false)
s.escrowAgentFn = func() (escrowAgent, error) { return a, nil }
rec := getPage(t, s, "/backup/escrow")
if rec.Code != 200 {
t.Fatalf("GET /backup/escrow = %d: %s", rec.Code, rec.Body.String())
}
body := rec.Body.String()
if !strings.Contains(body, escrowNotReadyMarker) || !strings.Contains(body, "A doboz m") {
t.Fatal("R-546: the escrow page does not tell the household the box is still preparing")
}
if strings.Contains(body, `id="start-form"`) {
t.Fatal("R-546: the start form is offered on a box whose ceremony would refuse")
}
// The exact texts chaos night met: the preflight detail, and the ceremony stderr ("…escrow-create
// requires -storage …"). NOT a bare "-storage": the menu carries id="nav-group-storage".
for _, raw := range []string{"pbs_storage_id not configured", "requires -storage"} {
if strings.Contains(body, raw) {
t.Fatalf("R-546: the agent's raw diagnostic %q reached the household's page", raw)
}
}
if strings.Contains(body, "zzzz-not-present") {
t.Fatal("negative control matched")
}
}
// The direct path — the one that actually produced the stderr in chaos night: POST /api/escrow/start
// on a not-ready box is refused BEFORE anything is staged or started, in Hungarian.
// RED-PROOF: remove the readiness gate from escrowStartAPIHandler → 200 and "stage,start" run.
func TestR546_StartRefusedWhenNotReady(t *testing.T) {
h := newEscrowWizardHarness(t)
h.configureOffbox(t, "pending")
h.agent.pf = agentapi.EscrowPreflightResponse{OK: false,
Items: []agentapi.EscrowPreflightItem{{ID: "pbs_storage_id", OK: false}}}
w := postStart(t, h.s, wizardPassword)
if w.Code != http.StatusConflict {
t.Fatalf("start on a not-ready box = %d (%s), want 409", w.Code, w.Body.String())
}
for _, op := range h.order {
if op == "stage" || op == "start" {
t.Fatalf("a not-ready box still ran %q (order %v) — the refusal must come first", op, h.order)
}
}
if !strings.Contains(w.Body.String(), "A doboz m") {
t.Fatalf("the refusal is not the Hungarian waiting sentence: %s", w.Body.String())
}
}
@@ -0,0 +1,57 @@
package web
import (
"path/filepath"
"strings"
"testing"
"time"
)
// ── R-550 — an interrupted restore is SHOWN on the restore page ─────────────────────────────────
//
// Chaos night round 10: a restore interrupted by a hard reset left the page blank. The record now
// survives (internal/backup/restore_record.go); these drive the REAL page through ServeHTTP, so they
// bite on the wiring — a record that exists but that no page renders is the inert-seam shape.
const interruptedCardFragment = "Megszakadt vissza" // ASCII fragment of the card heading
// RED-PROOF: drop `data["InterruptedRestores"]` from backupsRestoreHandler → the card never renders →
// "the restore page does not show the interrupted restore".
func TestR550_RestorePageShowsInterruptedRestore(t *testing.T) {
s := newDashboardServer(t, time.Time{})
if s.backupMgr == nil {
t.Fatal("fixture invalid: no backup manager, so nothing here measures the restore page")
}
s.backupMgr.SetRestoreRecordPath(filepath.Join(t.TempDir(), "restore-status.json"))
s.backupMgr.BeginRestoreOp("restore", "gokapi") // the box stops here …
if s.backupMgr.LoadRestoreRecord() == nil { // … and this is the next start
t.Fatal("fixture invalid: the record did not convert to an interrupted restore")
}
rec := getPage(t, s, "/backups/restore")
if rec.Code != 200 {
t.Fatalf("GET /backups/restore = %d: %s", rec.Code, rec.Body.String())
}
body := rec.Body.String()
if !strings.Contains(body, interruptedCardFragment) || !strings.Contains(body, "megszakadt") {
t.Fatalf("the restore page does not show the interrupted restore — the household is left not knowing, which is R-550")
}
if !strings.Contains(body, "gokapi") {
t.Fatalf("the interrupted card does not name the app")
}
if strings.Contains(body, "zzzz-not-present") {
t.Fatal("negative control matched — the fragment search is broken")
}
}
// The other branch: a box with no interrupted restore shows no such card.
func TestR550_NoInterruptedRestoreNoCard(t *testing.T) {
s := newDashboardServer(t, time.Time{})
rec := getPage(t, s, "/backups/restore")
if rec.Code != 200 {
t.Fatalf("GET /backups/restore = %d", rec.Code)
}
if strings.Contains(rec.Body.String(), interruptedCardFragment) {
t.Fatalf("a box with no interrupted restore shows the interrupted card")
}
}
@@ -289,6 +289,10 @@ func (s *Server) backupsRestoreWizardHandler(w http.ResponseWriter, r *http.Requ
// survives a reload, which the flash does not.
if in.HasRecentResult {
data["LastResult"] = st.Last
} else if ir, ok := s.backupMgr.InterruptedRestore(app); ok {
// R-550: a restore of THIS app the box was running when it stopped — shown past the recency
// window, until the app is restored again.
data["LastResult"] = &ir
}
s.executeTemplate(w, r, "backups_restore_wizard", data)
+2
View File
@@ -101,6 +101,8 @@ type Server struct {
escrowAgentFn func() (escrowAgent, error)
escrowStageFn func(ctx context.Context) error
escrowStaleFn func() bool
// R-546: the cached agent-preflight readiness behind the reminder bar (escrow_readiness.go).
escrowReady escrowReadinessCache
// escrowSealedAtFn (v0.200.0, R-193) reports WHEN the hub's sealed recovery package was created —
// the one non-secret fact the recovery screen may state before a code is entered. Wired via
@@ -13,7 +13,27 @@
<span class="domain-badge">{{.Domain}}</span>
</div>
{{if not .AgentSupported}}
{{if .EscrowNotReady}}
<!-- R-546: the box cannot run the ceremony yet (the agent's own preflight is not ok — typically the
first minutes after a bind, before the PBS storage is applied). No checklist of red crosses, no
start form that would refuse, none of the agent's raw diagnostics: one sentence, and the page
polls and reloads itself when the agent says ready. -->
<div class="settings-card" id="escrow-not-ready">
<div class="alert alert-info">{{.EscrowNotReadyMessage}}</div>
<p class="form-hint">Ehhez nem kell semmit tenned. Ha fél óra múlva is ezt látod, szólj az üzemeltetőnek.</p>
</div>
<script>
(function(){
function check(){
fetch('/api/escrow/preflight',{headers:{'Accept':'application/json'}})
.then(function(r){ return r.json(); })
.then(function(j){ if(j && j.data && j.data.ok){ window.location.reload(); } })
.catch(function(){});
}
setInterval(check, 20000);
})();
</script>
{{else if not .AgentSupported}}
<div class="alert alert-info">A funkcióhoz az ügynök frissítése szükséges — a frissítés automatikusan megérkezik. Próbálja újra később.</div>
{{else}}
@@ -7,6 +7,18 @@
</div>
{{template "backups_flash" .}}
{{with .InterruptedRestores}}
<!-- R-550: a restore the box was running when it stopped. Persisted, so it survives the restart that
interrupted it; shown until that app's next restore. -->
<div class="settings-card">
<h3>Megszakadt visszaállítás</h3>
{{range .}}
<div class="alert alert-error"><strong>{{.Stack}}</strong>: {{.Message}}</div>
{{end}}
<p class="form-hint">A doboz újraindult, miközben a visszaállítás futott, ezért nem tudjuk, hogy befejeződött-e. Indítsd el újra lent, ugyanabból a mentésből.</p>
</div>
{{end}}
{{template "restore_banner" .}}
{{if not .Backup}}
@@ -34,7 +34,7 @@
<div class="settings-card">
<h3>Eredmény</h3>
<div class="alert {{if .OK}}alert-info{{else}}alert-error{{end}}">{{.Message}}</div>
<p class="form-hint">Befejezve: {{fmtTime .FinishedAt}}. Ha szeretnéd, alább újra indíthatsz egy visszaállítást.</p>
<p class="form-hint">{{if .Interrupted}}Észlelve{{else}}Befejezve{{end}}: {{fmtTime .FinishedAt}}. Ha szeretnéd, alább újra indíthatsz egy visszaállítást.</p>
</div>
{{end}}