0fe315b759
gates / gates (push) Successful in 15s
MinAgent: 0.131.0 (unchanged). Requires hub v0.117.0 for restore_interrupted. R-550 (operator ruling: fix). A design reversed and recorded: the restore op-status was in memory by choice. Now restore-status.json in DataDir, written atomically at both ends of an op. At startup a record still marked running becomes a failed, interrupted result kept per app until that app's next restore, shown on /backups/restore and the off-site wizard, and raised once as restore_interrupted. Cooldowns stay in memory. R-546. The R-543 reminder bar consults the agent's own preflight ok (every blocking item, not a copy of pbs_storage_id), cached 60 s, probed only while paused. /backup/escrow shows a waiting card that polls and reloads instead of red crosses and English diagnostics. POST /api/escrow/start refuses 409 before staging or starting - the direct path chaos night used. Unknown readiness keeps the bar. Red-proofs (each seen failing): restore record across restart; main() calls both startup functions; startup helper with loading skipped; restore page card; bar held back; waiting card; start refusal. go build/vet/test ./... green, 28 packages; controller_gates --fast all OK. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
306 lines
15 KiB
Go
306 lines
15 KiB
Go
package web
|
|
|
|
import (
|
|
"net/http"
|
|
"net/url"
|
|
"strings"
|
|
"time"
|
|
|
|
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
|
)
|
|
|
|
// R-48 — the offsite restore wizard.
|
|
//
|
|
// WHY THIS EXISTS. Until v0.153.0 the „Ellenőrző visszaállítás a távoli tárolóból" list rendered up
|
|
// to five inline `<form style="display:inline">` blocks per app row. Two of them —
|
|
// „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" and
|
|
// „Teljes visszaállítás (fájlok + adatbázis)" — sat next to each other as sibling buttons, and the
|
|
// difference between them is whether the customer's data comes back at all. The missing-only merge
|
|
// cannot resurrect deleted content; the reconstitution can. An operator who had read the source
|
|
// still pressed the wrong one (audits/DIAG-immich-restore-round2-2026-07-19.md, finding 1): the
|
|
// controller log shows /backup/offbox/reconstitute was never hit.
|
|
//
|
|
// The rule R-48 establishes, worth stating where it is implemented: TWO ADJACENT CONTROLS WHOSE
|
|
// DIFFERENCE IS "YOUR DATA COMES BACK" VS "YOUR DATA CANNOT COME BACK" MUST NOT BE DISTINGUISHABLE
|
|
// ONLY BY LAYOUT. So the list page now carries ONE entry per app — „Visszaállítás…" — and this
|
|
// wizard makes the intent an explicit, separately-described choice.
|
|
//
|
|
// It is deliberately a THIN surface: no new mutation endpoint, no job registry (that is R-45), no
|
|
// JSON state API. Every step posts to the endpoint the old buttons posted to, with the same field
|
|
// names, and the server re-renders the next step. The only client-side JS is the card reveal the
|
|
// escrow wizard already established plus the EXISTING confirm helper and status poll.
|
|
|
|
// restoreWizardStep is which phase of the wizard the server decided to render. It is derived, never
|
|
// stored and never accepted from the request — a customer cannot navigate to a step whose
|
|
// preconditions do not hold.
|
|
type restoreWizardStep string
|
|
|
|
const (
|
|
// wizStepIntent is the choice: verify / bring back missing files / full restore.
|
|
wizStepIntent restoreWizardStep = "intent"
|
|
// wizStepPrepareConfirm is the size-gate reveal — the full-restore preparation ran, reported a
|
|
// size, and the customer confirms before the download starts.
|
|
wizStepPrepareConfirm restoreWizardStep = "prepare-confirm"
|
|
// wizStepExecution is "something is running" — every mutation form is suppressed SERVER-SIDE,
|
|
// because the backup manager's single-flight would refuse them anyway and offering a control
|
|
// that is guaranteed to fail is exactly the class of dishonesty R-48 is about.
|
|
wizStepExecution restoreWizardStep = "execution"
|
|
)
|
|
|
|
// restoreWizardInput is the complete set of facts the step derivation is allowed to see. Keeping it
|
|
// a plain struct (rather than reading off the Server) is what makes deriveWizardStep a pure,
|
|
// table-testable function — the Scenario-B table in restore_wizard_test.go is this struct's
|
|
// truth table.
|
|
type restoreWizardInput struct {
|
|
// App is the app this wizard page is for.
|
|
App string
|
|
// OpRunning is true when ANY backup/restore op is in flight — not just this app's. The op status
|
|
// is process-wide, so a restore running for app X must suppress app Y's controls.
|
|
//
|
|
// Source it via restoreOpInFlight (the DISPLAY flag), never Manager.IsRunning() — see the note
|
|
// on that helper. Reading the wrong flag makes this field silently always-false for the
|
|
// verification restore, which is the wizard's most-used path.
|
|
OpRunning bool
|
|
// ScratchReady is true when a completed full-restore scratch exists for App. Both the
|
|
// missing-only merge and the true reconstitution require one.
|
|
ScratchReady bool
|
|
// FullPrepApp is the app the size-gate flash binds to (from ?full_prep=). It binds to ONE app:
|
|
// a prepare for X must not reveal a confirm on Y's page.
|
|
FullPrepApp string
|
|
// HasRecentResult is true when THIS app's restore finished a short while ago — see
|
|
// hasRecentRestoreResult. It moves the phase strip to „Eredmény"; it never changes what the
|
|
// customer may do (a finished restore leaves every intent available again).
|
|
HasRecentResult bool
|
|
}
|
|
|
|
// restoreWizardView is what the template renders. The enabled-flags are part of the derivation (not
|
|
// separate template conditionals) so that the whole "what may the customer do right now" decision is
|
|
// one pure function with one test table.
|
|
type restoreWizardView struct {
|
|
Step restoreWizardStep
|
|
// Phase is which label the phase strip highlights. It is NOT the same as Step: a finished restore
|
|
// is back on the intent step (everything is offered again) while the strip rightly says
|
|
// „Eredmény". Keeping them separate is what stopped the strip from having to lie in one direction
|
|
// or the other.
|
|
Phase restoreWizardPhase
|
|
// VerifyEnabled — intent 1: restore into a separate verification folder (mode=unit). Live data
|
|
// is untouched, so this is the only intent available without a prepared scratch.
|
|
VerifyEnabled bool
|
|
// PrepareEnabled — intent 3, first leg: no scratch yet, so the full restore must first be
|
|
// prepared (mode=full, size-gated).
|
|
PrepareEnabled bool
|
|
// PlaceEnabled — intent 2: missing-only merge (/backup/offbox/place). Needs a prepared scratch.
|
|
PlaceEnabled bool
|
|
// RestoreEnabled — intent 3, second leg: the TRUE restore (/backup/offbox/reconstitute). Needs a
|
|
// prepared scratch.
|
|
RestoreEnabled bool
|
|
// CommitPrepareEnabled — the revealed „Teljes visszaállítás indítása (~méret)" confirm
|
|
// (mode=full&confirm=1). Only on the prepare-confirm step.
|
|
CommitPrepareEnabled bool
|
|
}
|
|
|
|
// restoreWizardPhase is the phase-strip highlight. Four labels, all reachable.
|
|
type restoreWizardPhase string
|
|
|
|
const (
|
|
wizPhasePrepare restoreWizardPhase = "elokeszites"
|
|
wizPhaseConfirm restoreWizardPhase = "megerosites"
|
|
wizPhaseExecute restoreWizardPhase = "vegrehajtas"
|
|
wizPhaseResult restoreWizardPhase = "eredmeny"
|
|
)
|
|
|
|
// restoreResultWindow bounds how long after a finished restore the strip still says „Eredmény".
|
|
// Without a bound the last result would light that phase forever — landing on the page a week later
|
|
// would claim you had just finished a restore. Same reasoning as escrowCeremonyGraceWindow; shorter,
|
|
// because this answers "what just happened", not "are we still waiting".
|
|
//
|
|
// R-351: this is now an ALIAS, not a second value. The list page's banner needs the same bound, and
|
|
// the payload carries the verdict (RestoreOpStatus.LastRecent), so the window is defined once in
|
|
// internal/backup beside the status it bounds. Keeping a separate literal here is how the two
|
|
// surfaces would drift.
|
|
const restoreResultWindow = backup.RestoreResultWindow
|
|
|
|
// hasRecentRestoreResult reports whether THIS app has a just-finished restore to show. Pure (the
|
|
// clock is a parameter) so the boundary and the wrong-app case are table-testable.
|
|
//
|
|
// Bound to the app on purpose: the op status is process-wide, so a finished bookstack restore must
|
|
// not light „Eredmény" on immich's wizard and show bookstack's message there.
|
|
func hasRecentRestoreResult(st backup.RestoreOpStatus, app string, now time.Time) bool {
|
|
if st.Running || st.Last == nil || st.Last.Stack != app || st.Last.FinishedAt.IsZero() {
|
|
return false
|
|
}
|
|
d := now.Sub(st.Last.FinishedAt)
|
|
return d >= 0 && d < restoreResultWindow
|
|
}
|
|
|
|
// deriveWizardStep is the Scenario-B truth table: step and available intents are a PURE function of
|
|
// the state, in strict precedence order.
|
|
//
|
|
// 1. An op is running (any app) → execution; nothing is offered.
|
|
// 2. The size-gate flash is for THIS app → prepare-confirm; only the commit is offered.
|
|
// 3. Otherwise → intent. Verification is always available; the two data-touching intents unlock
|
|
// only with a prepared scratch, and without one the full-restore card offers preparation
|
|
// instead.
|
|
//
|
|
// Precedence matters: execution outranks the flash, because a stale ?full_prep= in the URL must
|
|
// never resurrect a commit button while a restore is mid-flight.
|
|
func deriveWizardStep(in restoreWizardInput) restoreWizardView {
|
|
if in.OpRunning {
|
|
return restoreWizardView{Step: wizStepExecution, Phase: wizPhaseExecute}
|
|
}
|
|
if in.FullPrepApp != "" && in.FullPrepApp == in.App {
|
|
return restoreWizardView{Step: wizStepPrepareConfirm, Phase: wizPhaseConfirm, CommitPrepareEnabled: true}
|
|
}
|
|
phase := wizPhasePrepare
|
|
if in.HasRecentResult {
|
|
phase = wizPhaseResult
|
|
}
|
|
return restoreWizardView{
|
|
Step: wizStepIntent,
|
|
Phase: phase,
|
|
VerifyEnabled: true,
|
|
PrepareEnabled: !in.ScratchReady,
|
|
PlaceEnabled: in.ScratchReady,
|
|
RestoreEnabled: in.ScratchReady,
|
|
}
|
|
}
|
|
|
|
// restoreOpInFlight reports whether a restore op is in flight, FOR DISPLAY.
|
|
//
|
|
// **Use this, not `Manager.IsRunning()`.** The Manager carries two different booleans and they are
|
|
// not interchangeable:
|
|
//
|
|
// - `m.running` (read by `IsRunning`) is the CONCURRENCY single-flight. It is acquired *inside*
|
|
// the restore function, on the background goroutine — and `RestoreOffboxScratch` never acquires
|
|
// it at all. So for the verification restore and the full-restore preparation — the wizard's two
|
|
// most-used actions, and the long ones, since they stream from restic — `IsRunning()` is false
|
|
// for the entire operation.
|
|
// - `m.opRunning` (read by `RestoreStatus`) is the DISPLAY flag, set synchronously by
|
|
// `BeginRestoreOp` in the handler *before* the goroutine launches and cleared by `EndRestoreOp`.
|
|
// It covers all four offsite actions with no start-up window.
|
|
//
|
|
// v0.154.0 shipped with `IsRunning()` here, which made the execution step unreachable for
|
|
// `RestoreOffboxScratch`: the page offered all three intents, with live buttons, while a restore was
|
|
// downloading — and the progress banner (which polls the op status) contradicted it on the same
|
|
// screen. Caught by the operator on the first live click-through.
|
|
func restoreOpInFlight(st backup.RestoreOpStatus) bool {
|
|
return st.Running
|
|
}
|
|
|
|
// restoreOpBlocked reports whether a NEW restore must be refused right now, and returns the
|
|
// Hungarian refusal to show. It reads BOTH flags, deliberately:
|
|
//
|
|
// - `RestoreStatus().Running` — the DISPLAY flag, set synchronously by `BeginRestoreOp` in the
|
|
// handler. It is the only one that is true for the WHOLE duration of an off-box restore, which
|
|
// is what makes it the right flag to refuse on.
|
|
// - `IsRunning()` — the CONCURRENCY flag. The nightly backup run holds this one and never calls
|
|
// `BeginRestoreOp`, so dropping it would open a hole the old guard did close. Kept, not replaced.
|
|
//
|
|
// R-351, the measured defect: every restore handler read ONLY `IsRunning()`, which the restore
|
|
// goroutine acquires AFTER the handler has already returned (offbox_reconstitute.go:180,
|
|
// offbox_restore.go:393). A second press inside that window started a second run and was told
|
|
// „…elindult". Pinned by TestRestoreHandlers_SecondPressDoesNotStartASecondRun, which asserts the
|
|
// CONSEQUENCE — that the first restore's identity survives the second press — rather than which
|
|
// flag was read.
|
|
//
|
|
// The refusal names a reason AND a route: the page it redirects to is the wizard, which carries the
|
|
// live status banner, so „ezen az oldalon" is a true instruction and not a gesture.
|
|
func (s *Server) restoreOpBlocked() (string, bool) {
|
|
if s.backupMgr == nil {
|
|
return "", false
|
|
}
|
|
if st := s.backupMgr.RestoreStatus(); restoreOpInFlight(st) {
|
|
subject := "Egy visszaállítási művelet"
|
|
if st.Stack != "" {
|
|
subject = "Egy visszaállítási művelet (" + st.Stack + ")"
|
|
}
|
|
return subject + " már fut, ezért most nem indítható újabb. Az állapotát ezen az oldalon " +
|
|
"követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
|
|
}
|
|
if s.backupMgr.IsRunning() {
|
|
return "Egy mentési művelet már fut, ezért most nem indítható visszaállítás. Az állapotát " +
|
|
"ezen az oldalon követheted; amint befejeződik, újra indíthatsz visszaállítást.", true
|
|
}
|
|
return "", false
|
|
}
|
|
|
|
// backupsRestoreWizardHandler renders GET /backups/restore/app?name=<app> — the single entry the
|
|
// list page now offers per app.
|
|
//
|
|
// Unknown app, an app that is not toggled for offsite backup, or an unconfigured offsite target all
|
|
// redirect back to the list with a Hungarian flash. They must never 500: the URL is customer-visible
|
|
// and survives a bookmark, an app rename and a toggle being switched off.
|
|
func (s *Server) backupsRestoreWizardHandler(w http.ResponseWriter, r *http.Request) {
|
|
if s.backupMgr == nil || !s.backupMgr.OffboxConfigured() {
|
|
offboxRedirectTo(w, r, "/backups/restore", "A távoli mentési cél nincs beállítva.", true)
|
|
return
|
|
}
|
|
app := strings.TrimSpace(r.URL.Query().Get("name"))
|
|
if app == "" {
|
|
offboxRedirectTo(w, r, "/backups/restore", "Hiányzó alkalmazás.", true)
|
|
return
|
|
}
|
|
|
|
// R-237: the gate is "is it in the store", NOT "is it toggled on for future backups". The old
|
|
// resolver required the toggle, which locked a rebuilt box out of its own snapshots — measured
|
|
// live on the R-201 re-walk. A refusal here still never 500s.
|
|
rows, storeState := s.offsiteRestoreRows(r.Context())
|
|
row := resolveOffsiteRestoreApp(rows, app)
|
|
if row == nil {
|
|
msg := "Ehhez az alkalmazáshoz nincs mentés a távoli tárolóban."
|
|
if storeState != offsiteStoreKnown {
|
|
// Never say "there is nothing" when we could not look — R-225's rule, one screen over.
|
|
msg = "Nem tudjuk elolvasni a távoli tárolót, ezért nem tudjuk, van-e benne mentés ehhez az alkalmazáshoz."
|
|
}
|
|
s.logger.Printf("[INFO] [web] restore wizard refused for %q: not restorable (store=%s)", app, storeState)
|
|
offboxRedirectTo(w, r, "/backups/restore", msg, true)
|
|
return
|
|
}
|
|
|
|
data := s.backupsCommonData("backups-restore", "Visszaállítás — "+row.DisplayName, r)
|
|
|
|
st := s.backupMgr.RestoreStatus()
|
|
in := restoreWizardInput{
|
|
App: app,
|
|
OpRunning: restoreOpInFlight(st),
|
|
ScratchReady: s.backupMgr.OffboxFullScratchReady(app),
|
|
FullPrepApp: strings.TrimSpace(r.URL.Query().Get("full_prep")),
|
|
HasRecentResult: hasRecentRestoreResult(st, app, time.Now()),
|
|
}
|
|
view := deriveWizardStep(in)
|
|
|
|
data["App"] = app
|
|
data["AppDisplayName"] = row.DisplayName
|
|
data["AppSlug"] = row.Slug
|
|
data["Wizard"] = view
|
|
data["FullPrepSize"] = strings.TrimSpace(r.URL.Query().Get("full_size"))
|
|
// The pair-honesty panel (R-43): how old the database half is, whether the two halves come from
|
|
// the same run, and whether the dump looks customer-empty. Only meaningful once a scratch exists.
|
|
if in.ScratchReady {
|
|
pair := s.backupMgr.OffsiteScratchPair(app)
|
|
data["Pair"] = pair
|
|
} else {
|
|
data["Pair"] = backup.OffsitePairInfo{}
|
|
}
|
|
// The running op's identity, so the execution card can say WHAT is running rather than a bare
|
|
// "please wait" — including the case where it belongs to a different app.
|
|
data["RunningStack"] = st.Stack
|
|
// The outcome card for the „Eredmény" phase — the same message the redirect flash carried, but it
|
|
// survives a reload, which the flash does not.
|
|
if in.HasRecentResult {
|
|
data["LastResult"] = st.Last
|
|
} else if ir, ok := s.backupMgr.InterruptedRestore(app); ok {
|
|
// R-550: a restore of THIS app the box was running when it stopped — shown past the recency
|
|
// window, until the app is restored again.
|
|
data["LastResult"] = &ir
|
|
}
|
|
|
|
s.executeTemplate(w, r, "backups_restore_wizard", data)
|
|
}
|
|
|
|
// restoreWizardPath builds the wizard URL for an app. Handlers redirect here after an app-scoped
|
|
// mutation so the customer lands back on the surface they acted from, not on the list.
|
|
func restoreWizardPath(app string) string {
|
|
return "/backups/restore/app?name=" + url.QueryEscape(app)
|
|
}
|