R-1251 (P1): a pending put-back blocks every start — a cut restore's copy on an unbound drive waits (a restore hold), is poured back when the drive binds before any start; a failed put-back holds the app (H2); Stop during a restore still puts back (H7). Commit only, unreleased.
gates / gates (push) Successful in 1m18s

This commit is contained in:
2026-10-24 19:07:02 +02:00
parent 2449223f10
commit bb2809dc68
16 changed files with 847 additions and 24 deletions
+29
View File
@@ -22,6 +22,35 @@ brief 2026-10-24, Part 4 (commit only; the release rides the next normal train).
`…OnlyWhenTheAgentWaitsAndAnAppRuns`, `…QBLimitRuleSkipsAHeldLimit`, `…TheTwoAppsReadAndWriteTheirLimit`,
`TestLineTestRunOutlastsTheClientTimeout`, `TestStartUpwatchRecoversTheHoldFirst`, `TestUpwatchHoldGetsBothAgentCalls`.
### A pending put-back blocks every start (R-1251, P1; review round 2 HUNT-1 H1, H2, H7) — 2026-10-24 night, commit only
- **A power cut during a restore of a drive-backed app no longer brings old data back later.** The restore's copy of
the live data lives on the app's drive; at start the drive was often not bound, Recover read the copy as „gone", the
boot recreate then started the app on the cut restore's data, and the next controller start poured the old copy over
everything written since. Now a copy that cannot be reached while its drive is not bound is WAITING: Recover holds
the app (`settings.HoldReasonPutBackPending`, the copy in `SafetyDump`; the household sees „hold.putback_pending")
and starts nothing. The hold is the one store every start path already honours — the household's
Start/Restart/Update, the boot recreate, the drive return, the media restart, the boot sweep, the night chain, the
update leg. When the drive binds, `AppStopGuard.ResumePutBack` pours the copy back FIRST (the boot recreate and the
drive-return restart call it before `appHeld`; the drive-return branch and the start run `ResumeWaitingPutBacks`
for held apps), lifts the hold, and only then may the app start. New `internal/backup/appstop_putback_wait.go`;
main.go wires `SetHolds(sett)` and `SetUndoReachable` (the copy is reachable while the app's `HDD_PATH` is a live
mountpoint) before Recover.
- **A failed crash put-back leaves the app HELD (H2)** — the D20 hold (`HoldReasonRestoreMixed`, „stopped half-way"),
the copy kept, and the marker lets go of the app (the hold owns it) — never stopped-and-unheld, which Start or the
recreate would run and a later start would pour an old copy over. A copy that vanished while its drive is bound
holds the same way.
- **Stop pressed during a restore no longer skips the put-back after a cut (H7):** the data is put back; the app stays
stopped as the household asked.
- A removal clears a waiting put-back (it belongs to the install); a later restore that completes drops it (the
household chose that data; the copy stays on disk).
- Tests (each seen failing with its rule taken out): `TestR1251_DriveCutRestoreWaitsForTheDriveThenPutsBackBeforeAnyStart`
(the review's lean-clone proof turned round), `…TheWaitingHoldRefusesEveryStartPath`, `…AFailedPutBackHoldsTheApp`,
`…AFailedResumeHoldsTheApp`, `…ACopyGoneWhileItsDriveIsBoundHoldsTheApp`, `…StopPressedDuringTheRestoreStillPutsBack`,
`…ADriveThatReturnsLaterIsPutBackThenStarted`, `…ACompletedRestoreDropsTheWaitingPutBack`; the wiring pins
`TestR1251_DriveStartPathsPutBackBeforeTheyStart` (web) and `TestR1251_MainWiresThePutBackWait` (main); the
removal row in `TestR704_ClearUpdateHoldClearsTheInstallsHoldsOnly`.
### Review round 2 (2026-10-24, unattended; commit only, no release) — two small fixes, each red-proved
- **`REUSE.md` (docs only, same day):** D22's row no longer says „unreleased" (v0.338.0); the `recordInstalledImages`
+1
View File
@@ -83,6 +83,7 @@
| `Manager.RestoreFromRecoveryUnitAt` / `RestoreFromRecoveryUnit` + `UnitRestoreResult` (R-353 v0.226.0; R-102 v0.229.0) | controller/internal/backup/restore_unit.go | `(stack, unitDir string) (UnitRestoreResult, error)` / `(stack string) (…)` | THE local recovery-unit restore, and the facts its surface must state | **Returns a RESULT, not just an error** — volumes replayed, DBs replayed, and what the manifest LISTED. The Manifest* counts are load-bearing: zero-replayed has two causes (the backup held no data / the backup listed data that did not come back) and they are opposite news. Pair it with `unitRestoreOutcomeMsg`; do NOT write a new sentence. **A claim about the APP is forbidden on this path** — it has no `SafetyDump` discriminator, unlike the off-site twin (CONTEXT.md ruling, 07-backup-architecture §6.3). `restoreDockerVolumes` now returns `(int, error)`; `restoreDockerVolumesFrom` is unchanged and still the shared implementation. **R-102: `…At` holds the whole body and takes the unit DIRECTORY; the one-argument form is the thin caller naming the primary unit.** THE SOURCE MOVES; THE DESTINATION DOES NOT — `GetAppDrivePath` still resolves where the data lands. The R-47 mutation order, the unit-over-guest secret precedence, the fail-closed data-key gate and the no-unit `RestoreApp` fallback are all pinned and unchanged. The volume leg goes through the R-354 `volumeReplayFrom` seam so the source directory is assertable without Docker |
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
| `CrashPutBack` + `Manager.persistUndoForCrash` + `AppStopGuard.SetUndo` (D22 / R-1195, v0.338.0) | controller/internal/backup/restore_crash_putback.go | `NewCrashPutBack(stop, writeDef, logger)`; `PutBackInterruptedRestore(stack, undoDir) error` | A restore cut by a power loss is put back at start: the copy (volumes + `definition/` + `undo.json`) is recorded in the app-stop marker once whole; `Recover` pours it back BEFORE the restart | Call `persistUndoForCrash` right after `copyLiveVolumes` in any new restore path. `Recover` runs before the backup manager exists, so the put-back is wired in main.go with `stackMgr.StopStack` and the stack adapter's definition writer. A failed put-back = a FAILED recovery (app not started, marker kept). The household reads `note.restore.interrupted_put_back` (`LoadRestoreRecordAfterRecovery`). Pinned by `TestR1195_*` |
| `AppStopGuard.ResumePutBack` / `ResumeWaitingPutBacks` + `HoldReasonPutBackPending` (R-1251, unreleased) | controller/internal/backup/appstop_putback_wait.go | `ResumePutBack(stack) bool` (true = nothing owed, the caller may start); `ResumeWaitingPutBacks() []string` (puts back, then starts through the gated starter) | A cut restore's copy on a drive that is not bound is WAITING: Recover writes the restore hold `putback_pending` (copy in `SafetyDump`) and starts nothing; the put-back runs when the drive binds, before any start | **Any new path that starts an app when its drive arrives calls `ResumePutBack` first** (the boot recreate and `restartStacks` do, in web via `Server.resumePutBack`). Every other start path is covered by the hold store it already honours — do not add a second check. A failed put-back (at start or on resume) writes the D20 hold (`restore_mixed`) and drops the app from the marker. Needs `SetHolds` + `SetUndoReachable` before `Recover` (main.go). Pinned by `r1251_putback_waits_test.go`, `TestR1251_DriveStartPathsPutBackBeforeTheyStart`, `TestR1251_MainWiresThePutBackWait` |
| `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned |
| `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces |
| `offsiteNoSpaceMsgFmt` + `offsiteSizeUnknownMsg` (R-357, v0.226.0) | controller/internal/backup/offbox_restore.go | two consts | EVERY headroom refusal on the off-site restore surface | **All four gates share these** (prepare, scratch, place, and the destructive reconstitute). A customer meeting one wording on one path and a different one on another has to work out whether it is the same problem. **The reconstitute gate uses NO ×1.1 margin** — it copies a measured tree; `OffboxRestorePrepareFull`'s ×1.1 predicts a download. **Fail closed when either probe reads ≤ 0**: `free < need` with `need == 0` is FALSE, so an unmeasurable input sails through — a gate present and inert |
@@ -493,3 +493,19 @@ func assignedIdentsIn(body *ast.BlockStmt) []string {
})
return names
}
// R-1251: the guard's hold store and drive question are wired BEFORE Recover (a waiting put-back would otherwise be
// read as „gone" and nothing would hold the app), and the waiting put-backs are resumed after it. COMPANION
// RED-PROOF (2026-10-24): delete SetHolds or SetUndoReachable, or move either below Recover → this fails.
func TestR1251_MainWiresThePutBackWait(t *testing.T) {
names := callsInMain(t, mainBody(t))
rec := indexOfCall(names, "Recover")
for _, n := range []string{"SetHolds", "SetUndoReachable"} {
if i := indexOfCall(names, n); i < 0 || i > rec {
t.Errorf("func main() must call appStopGuard.%s before Recover (at %d, Recover at %d)", n, i, rec)
}
}
if i := indexOfCall(names, "ResumeWaitingPutBacks"); i < rec {
t.Errorf("func main() must resume the waiting put-backs after Recover (at %d, Recover at %d)", i, rec)
}
}
+19
View File
@@ -466,7 +466,26 @@ func main() {
}
return (&stackAdapter{mgr: stackMgr, encKey: encKey}).RecreateStackDefinitionFromUnit(name, defDir, cfg.Env)
}, logger))
// R-1251: a cut restore's copy on a drive that is not bound yet WAITS (a restore hold every start path honours),
// it is never „gone" — and is poured back before anything starts the app. The drive question is the app's own:
// a copy under its HDD_PATH is reachable only while that path is a live mountpoint.
appStopGuard.SetHolds(sett)
appStopGuard.SetUndoReachable(func(name, dir string) bool {
cfg := stackMgr.LoadAppConfigByName(name)
if cfg == nil {
return false // cannot determine — the safe side is „waiting"
}
hdd := cfg.Env["HDD_PATH"]
if hdd == "" || !strings.HasPrefix(filepath.Clean(dir), filepath.Clean(hdd)+string(filepath.Separator)) {
return true
}
return stackMgr.DriveLive(hdd)
})
appStopRecovery := appStopGuard.Recover()
// R-1251: a put-back that waited for its drive (a hold from an earlier start) runs now if that drive is bound.
if started := appStopGuard.ResumeWaitingPutBacks(); len(started) > 0 {
logger.Printf("[WARN] [appstop] %d app(s) whose interrupted restore waited for its drive are put back and started: %v", len(started), started)
}
// --- R-166: desired-state backfill (running-only) ---
// Converge the apps whose intent is unambiguous — deployed and observed UP — so the fleet stops
+70 -20
View File
@@ -133,6 +133,10 @@ type AppStopGuard struct {
// putBacker (D22, R-1195) pours a restore's undo copy back before Recover restarts the app. nil = no put-back
// (the pre-D22 restart, unchanged).
putBacker InterruptedRestorePutBacker
// holds and undoReachable (R-1251, appstop_putback_wait.go): where a waiting or failed put-back is HELD, and
// whether the drive a copy lives on is bound. nil = the pre-R-1251 behaviour.
holds AppStopHolds
undoReachable func(stack, dir string) bool
// suppressed (R-330) is the in-memory "the alarm must not fire for these, we are holding them"
// set. It is deliberately NOT persisted: after a restart the guard no longer holds anything —
@@ -162,6 +166,11 @@ type AppStopRecovery struct {
// PutBack are apps whose interrupted restore was undone from its copy before the restart (D22, R-1195): their
// data is as it was before the restore.
PutBack []string
// Waiting (R-1251) are apps whose restore copy is on a drive that is not bound: HELD, not started, put back first
// when the drive binds (ResumePutBack). Like Refused, the holder owns them and they do not alarm.
Waiting []string
// Held (R-1251 H2) are apps whose put-back FAILED and that are now held for support (the D20 hold). They alarm.
Held []string
}
// Alarming reports whether this recovery is worth paging an operator about. A recovery that only
@@ -171,7 +180,7 @@ func (r *AppStopRecovery) Alarming() bool {
if r == nil {
return false
}
return len(r.Failed) > 0 || len(r.Restarted) > 0
return len(r.Failed) > 0 || len(r.Held) > 0 || len(r.Restarted) > 0
}
// Message is the operator-facing headline for an interrupted operation.
@@ -179,14 +188,22 @@ func (r *AppStopRecovery) Message() string {
if r == nil {
return ""
}
if len(r.Failed) > 0 {
if len(r.Failed) > 0 || len(r.Held) > 0 {
bad := len(r.Failed) + len(r.Held)
m := fmt.Sprintf("%s was interrupted by a controller restart and %d of %d app(s) could NOT be restarted",
r.Reason.humanReason(), len(r.Failed), len(r.Restarted)+len(r.Failed))
r.Reason.humanReason(), bad, len(r.Restarted)+bad)
if len(r.Held) > 0 {
m += fmt.Sprintf(" (%d could not be put back and are HELD for support)", len(r.Held))
}
if len(r.Refused) > 0 {
m += fmt.Sprintf(" (a further %d are held by an absent drive and are not counted as failures)", len(r.Refused))
}
return m
}
if len(r.Waiting) > 0 && len(r.Restarted) == 0 {
return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) are HELD until their data drive is bound: the copy taken before the change is on it and is put back first",
r.Reason.humanReason(), len(r.Waiting))
}
if len(r.Refused) > 0 && len(r.Restarted) == 0 {
return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) are left stopped and HELD: their data drive is not available, so the drive gate restarts them when it returns",
r.Reason.humanReason(), len(r.Refused))
@@ -215,6 +232,12 @@ func (r *AppStopRecovery) Detail() string {
if len(r.PutBack) > 0 {
d += fmt.Sprintf(" put_back=%v", r.PutBack)
}
if len(r.Waiting) > 0 {
d += fmt.Sprintf(" put_back_waiting_for_drive=%v", r.Waiting)
}
if len(r.Held) > 0 {
d += fmt.Sprintf(" put_back_failed_held=%v", r.Held)
}
return d
}
@@ -352,28 +375,48 @@ func (g *AppStopGuard) Recover() *AppStopRecovery {
res := &AppStopRecovery{Reason: m.Reason, OpID: m.OpID, StartedAt: m.StartedAt}
hs, _ := g.starter.(interface{ WantsStopped(string) bool })
var usedUndo []string // apps whose copy was poured back: the marker must not name it again
for _, name := range m.Stacks {
if hs != nil && hs.WantsStopped(name) {
// R-721 (v0.283.1): the household pressed Stop while the interrupted operation had the app down.
// It is owed nothing; the marker still clears below.
g.logger.Printf("[INFO] [appstop] crash recovery: NOT restarting %s — the household stopped it", name)
continue
}
// R-721 (v0.283.1): the household pressed Stop while the interrupted operation had the app down — it is owed
// no restart. R-1251 H7: it is still owed its put-back, so the stop is honoured AFTER the copy is poured back.
wantsStopped := hs != nil && hs.WantsStopped(name)
if dir := m.Undo[name]; dir != "" && g.putBacker != nil {
// D22 (R-1195): the restore was cut after its copy of the live data was taken — the app may be on the
// backup's data, or on a mix. Pour the copy back first; a put-back that fails is a FAILED recovery
// (the marker is kept, so the next startup tries again) and the app is not started on unknown data.
if _, statErr := os.Stat(dir); statErr != nil {
g.logger.Printf("[WARN] [appstop] crash recovery: %s's restore copy %s is gone (%v) — restarting as before", name, dir, statErr)
} else if err := g.putBacker.PutBackInterruptedRestore(name, dir); err != nil {
g.logger.Printf("[ERROR] [appstop] crash recovery: putting %s back from %s FAILED: %v — NOT starting it", name, dir, err)
res.Failed = append(res.Failed, name)
// backup's data, or on a mix. Pour the copy back first; a put-back that fails is a FAILED recovery and the
// app is not started on unknown data.
state, statErr := g.copyState(name, dir)
switch state {
case undoWaiting:
// R-1251: the copy is on a drive that is not bound yet. It is not „gone": the app is HELD (every start
// path honours the hold) and the copy is poured back when the drive binds (ResumePutBack).
g.logger.Printf("[WARN] [appstop] crash recovery: %s's restore copy %s cannot be reached yet (%v) — NOT starting it; the put-back waits for its drive", name, dir, statErr)
g.holdPutBackPending(name, dir)
res.Waiting = append(res.Waiting, name)
continue
} else {
case undoGone:
g.logger.Printf("[WARN] [appstop] crash recovery: %s's restore copy %s is gone (%v) — restarting as before", name, dir, statErr)
default:
if err := g.putBacker.PutBackInterruptedRestore(name, dir); err != nil {
g.logger.Printf("[ERROR] [appstop] crash recovery: putting %s back from %s FAILED: %v — NOT starting it", name, dir, err)
// R-1251 H2: HELD (the D20 hold), never stopped-and-unheld. Without a hold store the old way stays:
// Failed, and the marker is kept so the next start tries again.
if g.holdPutBackFailed(name, dir, err) {
res.Held = append(res.Held, name)
} else {
res.Failed = append(res.Failed, name)
}
continue
}
g.logger.Printf("[WARN] [appstop] crash recovery: %s's interrupted restore is undone — its data is as before the restore (R-1195)", name)
g.clearPutBackPending(name)
res.PutBack = append(res.PutBack, name)
usedUndo = append(usedUndo, name)
}
}
if wantsStopped {
g.logger.Printf("[INFO] [appstop] crash recovery: NOT restarting %s — the household stopped it", name)
continue
}
if err := g.starter.StartStack(name); err != nil {
// R-174: a REFUSAL is not a failure. The starter's gate has said this app must not be
// started (an absent data drive), so the app is left down deliberately and the holder
@@ -395,6 +438,8 @@ func (g *AppStopGuard) Recover() *AppStopRecovery {
sort.Strings(res.Refused)
sort.Strings(res.Restarted)
sort.Strings(res.PutBack)
sort.Strings(res.Waiting)
sort.Strings(res.Held)
// The marker is kept for BOTH unfinished outcomes, for the same reason and with different
// urgency: a failed restart is retried next startup, and a refused one is genuinely unfinished
@@ -403,11 +448,16 @@ func (g *AppStopGuard) Recover() *AppStopRecovery {
if len(res.Failed) > 0 {
g.logger.Printf("[ERROR] [appstop] crash recovery: %d app(s) could not be restarted — KEEPING the marker so the next startup retries; the dead-app alarm owns them meanwhile: %v",
len(res.Failed), res.Failed)
g.dropFromMarker(res.Held...)
g.forgetUndo(usedUndo...)
return res
}
if len(res.Refused) > 0 {
g.logger.Printf("[WARN] [appstop] crash recovery: %d app(s) were deliberately NOT restarted (drive absent) — KEEPING the marker; this is the gate working, not a fault: %v",
len(res.Refused), res.Refused)
if len(res.Refused) > 0 || len(res.Waiting) > 0 {
g.logger.Printf("[WARN] [appstop] crash recovery: %d app(s) were deliberately NOT restarted (drive absent: %v; put-back waiting for its drive: %v) — KEEPING the marker; this is the gate working, not a fault",
len(res.Refused)+len(res.Waiting), res.Refused, res.Waiting)
// The held (H2) apps and the used copies leave the marker; the rest of it stays for the next start.
g.dropFromMarker(res.Held...)
g.forgetUndo(usedUndo...)
return res
}
g.End()
@@ -0,0 +1,285 @@
package backup
import (
"fmt"
"os"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── R-1251: a pending put-back blocks every start (review round 2, HUNT-1 H1) ─────────────────────────────────────
//
// D22 pours a cut restore's copy back at start (restore_crash_putback.go). For a DRIVE-backed app the copy lives on
// the app's drive (restoreUndoDir under the drive's namespace), and at start the drive is often not bound yet. Until
// this file Recover read the missing folder as „gone", went on to the gated start (refused: no drive) and kept the
// marker; the drive then bound, the boot recreate started the app on the cut restore's data (it asked only „held?"),
// the household wrote, and the NEXT controller start found the copy and poured it over everything written since.
//
// Now:
// - a copy that cannot be seen while its drive is not bound is WAITING, never gone. Recover writes a restore hold
// (settings.HoldReasonPutBackPending, the copy's folder in SafetyDump) and starts nothing. The hold is the ONE
// storage every start path already honours — the household's Start/Restart/Update, the boot recreate, the drive
// return, the media restart, the boot sweep, the night chain, the update leg — so no path needs its own check;
// - when the drive binds, ResumePutBack pours the copy back FIRST and lifts the hold; only then may the app start;
// - a put-back that FAILS (at start or on resume) leaves the app HELD with the D20 hold (HoldReasonRestoreMixed, the
// household's „stopped half-way" sentence) and the copy on disk — never stopped-and-unheld, which Start or the boot
// recreate would run and a later start would pour an old copy over (H2). The marker then lets go of that app: the
// hold owns it, exactly as holdAppAfterMixedRestore does in-process;
// - Stop pressed during the restore no longer skips the put-back after a cut (H7): the data is put back, the app
// stays stopped as the household asked.
//
// The hold — not the marker — is the durable record of a waiting put-back, because the marker has one slot and the
// next operation's Begin replaces it (R-1253).
//
// Pinned by r1251_putback_waits_test.go.
// AppStopHolds is the hold store the guard writes (settings.Settings in production).
type AppStopHolds interface {
SetRestoreHold(settings.RestoreHold) error
GetRestoreHold(stack string) (settings.RestoreHold, bool)
ClearRestoreHold(stack string) (bool, error)
ListRestoreHolds() []settings.RestoreHold
}
// SetHolds wires the hold store. INIT-ONLY, before Recover. nil = the pre-R-1251 behaviour (no hold is written).
func (g *AppStopGuard) SetHolds(h AppStopHolds) {
if g != nil {
g.holds = h
}
}
// SetUndoReachable wires the question „is the drive this app's restore copy lives on bound right now?" (main.go:
// the app's HDD_PATH is a live mountpoint). nil = every folder counts as reachable, so a missing copy is „gone".
func (g *AppStopGuard) SetUndoReachable(f func(stack, dir string) bool) {
if g != nil {
g.undoReachable = f
}
}
type undoCopyState int
const (
undoPresent undoCopyState = iota
undoGone // its drive is bound and the folder is not there
undoWaiting // its drive is not bound (or the folder cannot be read): it may come back
)
func (s undoCopyState) String() string {
switch s {
case undoPresent:
return "present"
case undoGone:
return "gone"
default:
return "waiting"
}
}
// copyState decides present / gone / waiting for one recorded copy.
func (g *AppStopGuard) copyState(stack, dir string) (undoCopyState, error) {
_, err := os.Stat(dir)
if err == nil {
return undoPresent, nil
}
if !os.IsNotExist(err) {
return undoWaiting, err // an unreadable folder is not proof that it is gone
}
if g.undoReachable != nil && !g.undoReachable(stack, dir) {
return undoWaiting, err
}
return undoGone, err
}
// holdPutBackPending writes the „waiting for the drive" hold. Returns whether the app is now held by SOME hold.
func (g *AppStopGuard) holdPutBackPending(stack, dir string) bool {
if g.holds == nil {
g.logger.Printf("[ERROR] [appstop] %s: its restore copy %s waits for its drive, but no hold store is wired — nothing durable refuses a start (R-1251)", stack, dir)
return false
}
if h, ok := g.holds.GetRestoreHold(stack); ok && h.Reason != settings.HoldReasonPutBackPending {
// Another hold already refuses every start, and its sentence names the harder problem. The marker keeps the
// copy's folder, so the next start puts it back once the drive is there.
g.logger.Printf("[WARN] [appstop] %s: its restore copy %s waits for its drive; the app is already held (%q, set %s) — that hold stays", stack, dir, h.Reason, h.At)
return true
}
h := settings.RestoreHold{Stack: stack, At: g.now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonPutBackPending, SafetyDump: dir}
if err := g.holds.SetRestoreHold(h); err != nil {
g.logger.Printf("[ERROR] [appstop] %s: the waiting put-back could not be recorded as a hold: %v — the marker is kept", stack, err)
return false
}
g.logger.Printf("[WARN] [appstop] %s: the cut restore's copy %s is on a drive that is not bound — the app is HELD until the copy is put back (R-1251)", stack, dir)
return true
}
// holdPutBackFailed writes the D20 hold for a put-back that could not be done (H2). Returns whether it was written.
func (g *AppStopGuard) holdPutBackFailed(stack, dir string, cause error) bool {
if g.holds == nil {
return false
}
h := settings.RestoreHold{Stack: stack, At: g.now().UTC().Format(time.RFC3339), Reason: settings.HoldReasonRestoreMixed,
ReplayError: "interrupted restore, put-back failed: " + cause.Error(), SafetyDump: dir}
if err := g.holds.SetRestoreHold(h); err != nil {
g.logger.Printf("[ERROR] [appstop] %s: the failed put-back could not be recorded as a hold: %v — the marker is kept", stack, err)
return false
}
g.logger.Printf("[ERROR] [appstop] %s: the interrupted restore could NOT be put back (%v) — the app is HELD for support, the copy stays at %s (R-1251 H2)", stack, cause, dir)
return true
}
// clearPutBackPending lifts this app's waiting-put-back hold, and only that kind.
func (g *AppStopGuard) clearPutBackPending(stack string) {
if g.holds == nil {
return
}
if h, ok := g.holds.GetRestoreHold(stack); !ok || h.Reason != settings.HoldReasonPutBackPending {
return
}
if _, err := g.holds.ClearRestoreHold(stack); err != nil {
g.logger.Printf("[ERROR] [appstop] %s: the put-back ran, but its hold could not be cleared: %v — the app stays held", stack, err)
}
}
// dropFromMarker lets the active marker forget these apps (and their copies): a hold now owns them, or their put-back
// is done. A marker left with no app is cleared. A marker that belongs to another operation (it does not hold them)
// is not touched.
func (g *AppStopGuard) dropFromMarker(names ...string) {
if g == nil || g.path == "" || len(names) == 0 {
return
}
m, ok := g.read()
if !ok || !m.Active {
return
}
drop := map[string]bool{}
for _, n := range names {
drop[n] = true
}
var keep []string
for _, s := range m.Stacks {
if !drop[s] {
keep = append(keep, s)
}
}
if len(keep) == len(m.Stacks) {
return
}
if len(keep) == 0 {
if err := os.Remove(g.path); err != nil && !os.IsNotExist(err) {
g.logger.Printf("[ERROR] [appstop] could not clear the app-stop marker at %s: %v", g.path, err)
}
return
}
m.Stacks = keep
for n := range drop {
delete(m.Undo, n)
}
if err := g.write(m); err != nil {
g.logger.Printf("[ERROR] [appstop] could not rewrite the app-stop marker without %v: %v", names, err)
}
}
// forgetUndo removes these apps' copy folders from the active marker (their put-back is done; the copy is removed),
// so a later start never tries to pour back a folder that no longer holds the app's data.
func (g *AppStopGuard) forgetUndo(names ...string) {
if g == nil || g.path == "" || len(names) == 0 {
return
}
m, ok := g.read()
if !ok || !m.Active || len(m.Undo) == 0 {
return
}
changed := false
for _, n := range names {
if _, ok := m.Undo[n]; ok {
delete(m.Undo, n)
changed = true
}
}
if !changed {
return
}
if err := g.write(m); err != nil {
g.logger.Printf("[ERROR] [appstop] could not rewrite the app-stop marker without the used copies of %v: %v", names, err)
}
}
// resumeMu serialises ResumePutBack: the boot recreate and the drive-return sweep can reach the same app at once, and
// a copy must be poured back exactly once.
var resumeMu sync.Mutex
// ResumePutBack runs a put-back that was WAITING for its drive. It returns true when nothing is owed for stack any
// more — there was no waiting put-back, or it ran now — and the caller may go on to start the app (its other holds
// still apply). false: the app stays held (the drive is still not bound, or the put-back failed and the app is now
// held for support). Nil-safe.
func (g *AppStopGuard) ResumePutBack(stack string) bool {
if g == nil || g.holds == nil {
return true
}
resumeMu.Lock()
defer resumeMu.Unlock()
h, ok := g.holds.GetRestoreHold(stack)
if !ok || h.Reason != settings.HoldReasonPutBackPending {
return true
}
dir := h.SafetyDump
if m, ok := g.read(); ok && m.Active && m.Undo[stack] != "" {
dir = m.Undo[stack]
}
if dir == "" || g.putBacker == nil {
g.logger.Printf("[ERROR] [appstop] %s: a put-back waits, but its copy is not named (or no put-back is wired) — the app stays HELD", stack)
return false
}
state, serr := g.copyState(stack, dir)
switch state {
case undoWaiting:
g.logger.Printf("[INFO] [appstop] %s: the restore copy %s is still not reachable (%v) — the app stays held until its drive is bound", stack, dir, serr)
return false
case undoGone:
g.holdPutBackFailed(stack, dir, fmt.Errorf("the restore copy %s is gone while its drive is bound", dir))
g.dropFromMarker(stack)
return false
}
if err := g.putBacker.PutBackInterruptedRestore(stack, dir); err != nil {
if g.holdPutBackFailed(stack, dir, err) {
g.dropFromMarker(stack)
}
return false
}
g.clearPutBackPending(stack)
g.dropFromMarker(stack)
g.logger.Printf("[WARN] [appstop] %s: the drive is bound — the interrupted restore is put back from %s BEFORE the app starts; its data is as before the restore (R-1251)", stack, dir)
return true
}
// ResumeWaitingPutBacks runs every waiting put-back whose drive is bound now, and starts each app afterwards through
// the guard's gated starter — unless the household stopped it. Called at start (after Recover) and when a drive
// returns. Returns the apps started.
func (g *AppStopGuard) ResumeWaitingPutBacks() []string {
if g == nil || g.holds == nil {
return nil
}
var started []string
for _, h := range g.holds.ListRestoreHolds() {
if h.Reason != settings.HoldReasonPutBackPending {
continue
}
if !g.ResumePutBack(h.Stack) {
continue
}
if ws, ok := g.starter.(interface{ WantsStopped(string) bool }); ok && ws.WantsStopped(h.Stack) {
g.logger.Printf("[INFO] [appstop] %s: put back; NOT starting it — the household stopped it", h.Stack)
continue
}
if g.starter == nil {
continue
}
if err := g.starter.StartStack(h.Stack); err != nil {
g.logger.Printf("[WARN] [appstop] %s: put back, but the start was not done: %v", h.Stack, err)
continue
}
started = append(started, h.Stack)
}
return started
}
@@ -369,6 +369,9 @@ func (m *Manager) RestoreHoldForLang(stack, lang string) (bool, string) {
if t, err := time.Parse(time.RFC3339, h.At); err == nil {
when = t.Format("2006-01-02 15:04")
}
if h.Reason == settings.HoldReasonPutBackPending { // R-1251: the put-back waits for the copy's drive
return true, util.Text(lang, "hold.putback_pending", stack)
}
if h.Reason == settings.HoldReasonRestoreMixed { // R-893
return true, util.Text(lang, "note.reconstitute.held_mixed", stack, when)
}
@@ -0,0 +1,320 @@
package backup
// R-1251 (review round 2, HUNT-1 H1, H2, H7): a pending put-back blocks every start. The first test is the review's
// lean-clone proof (audits/review-round-2-2026-10-24/proofs/H1-drive-cut-restore/) turned round: the same cut, the same
// unbound drive, and now the app is HELD until the copy is poured back, and a later start pours nothing.
//
// RED-PROOF (2026-10-24, each seen failing on main 2449223 before appstop_putback_wait.go):
// - TestR1251_DriveCutRestoreWaitsForTheDriveThenPutsBackBeforeAnyStart: „boot 1 must HOLD the app as waiting" —
// Recover read the copy as gone and only REFUSED the start, with no hold;
// - TestR1251_AFailedPutBackHoldsTheApp: „a failed put-back must leave a restore_mixed hold" — none was written;
// - TestR1251_StopPressedDuringTheRestoreStillPutsBack: „the put-back must run" — R-721's skip came first.
import (
"errors"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// r1251Rig is one app (immich) whose restore was cut after its copy was recorded, the copy on the app's drive.
type r1251Rig struct {
marker, drive, undo string
sett *settings.Settings
puts []string
starts []string
stopped map[string]bool
putErr error
}
func newR1251Rig(t *testing.T) *r1251Rig {
t.Helper()
root := t.TempDir()
r := &r1251Rig{
marker: filepath.Join(root, "appstop-state.json"),
drive: filepath.Join(root, "drive"),
stopped: map[string]bool{},
}
r.undo = filepath.Join(r.drive, "felhom-data", "backups", "restore-undo", "immich", "20261024T120000Z")
mustWrite(t, filepath.Join(r.undo, "undo.json"), `{"stack":"immich","copied":["immich_pgdata"]}`)
sett, err := settings.Load(filepath.Join(root, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
r.sett = sett
g := NewAppStopGuard(r.marker, log.New(io.Discard, "", 0))
if err := g.Begin("unit-restore:immich", ReasonUnitRestore, []string{"immich"}); err != nil {
t.Fatal(err)
}
if err := g.SetUndo("immich", r.undo); err != nil {
t.Fatal(err)
}
return r // the power cut: no End
}
func (r *r1251Rig) unbind(t *testing.T) {
t.Helper()
if err := os.Rename(r.drive, r.drive+".unbound"); err != nil {
t.Fatal(err)
}
}
func (r *r1251Rig) bind(t *testing.T) {
t.Helper()
if err := os.Rename(r.drive+".unbound", r.drive); err != nil {
t.Fatal(err)
}
}
func (r *r1251Rig) driveBound() bool { _, err := os.Stat(r.drive); return err == nil }
// r1251Starter is the gated starter: it refuses while the drive is absent or a hold is set (as driveStartGate does).
type r1251Starter struct{ r *r1251Rig }
func (s r1251Starter) StartStack(name string) error {
if _, held := s.r.sett.GetRestoreHold(name); held {
return ErrStartRefused
}
if !s.r.driveBound() {
return ErrStartRefused
}
s.r.starts = append(s.r.starts, name)
return nil
}
func (s r1251Starter) WantsStopped(name string) bool { return s.r.stopped[name] }
type r1251PutBacker struct{ r *r1251Rig }
func (p r1251PutBacker) PutBackInterruptedRestore(stack, dir string) error {
if p.r.putErr != nil {
return p.r.putErr
}
p.r.puts = append(p.r.puts, stack)
return os.RemoveAll(dir) // the real put-back removes the used copy
}
// guard is a fresh controller start over the rig's disk, wired as main.go wires it.
func (r *r1251Rig) guard() *AppStopGuard {
g := NewAppStopGuard(r.marker, log.New(io.Discard, "", 0))
g.SetStarter(r1251Starter{r})
g.SetPutBacker(r1251PutBacker{r})
g.SetHolds(r.sett)
g.SetUndoReachable(func(string, string) bool { return r.driveBound() })
return g
}
func (r *r1251Rig) markerActive() bool {
b, err := os.ReadFile(r.marker)
return err == nil && strings.Contains(string(b), `"active": true`)
}
func TestR1251_DriveCutRestoreWaitsForTheDriveThenPutsBackBeforeAnyStart(t *testing.T) {
r := newR1251Rig(t)
// boot 1: the drive is not bound yet
r.unbind(t)
g1 := r.guard()
rec := g1.Recover()
h, held := r.sett.GetRestoreHold("immich")
if rec == nil || len(rec.Waiting) != 1 || !held || h.Reason != settings.HoldReasonPutBackPending {
t.Fatalf("boot 1 must HOLD the app as waiting for its drive: rec=%+v hold=%+v held=%v", rec, h, held)
}
if h.SafetyDump != r.undo {
t.Fatalf("the hold must name the copy, got %q", h.SafetyDump)
}
if len(r.starts) != 0 || len(r.puts) != 0 || rec.Alarming() {
t.Fatalf("nothing may start or be poured while the drive is absent: starts=%v puts=%v alarming=%v", r.starts, r.puts, rec.Alarming())
}
if !r.markerActive() {
t.Fatal("the marker must be kept while the put-back waits")
}
// a resume while the drive is still absent does nothing and keeps the hold
if g1.ResumePutBack("immich") {
t.Fatal("ResumePutBack must say still-held while the copy cannot be reached")
}
if _, still := r.sett.GetRestoreHold("immich"); !still {
t.Fatal("the hold must stay while the drive is absent")
}
// the drive binds: the boot recreate (and the drive-return sweep) call ResumePutBack before they start the app
r.bind(t)
if !g1.ResumePutBack("immich") {
t.Fatal("with the drive bound the put-back must run and allow the start")
}
if strings.Join(r.puts, ",") != "immich" {
t.Fatalf("the copy must be poured back exactly once, puts=%v", r.puts)
}
if _, still := r.sett.GetRestoreHold("immich"); still {
t.Fatal("the waiting hold must be lifted after the put-back")
}
if r.markerActive() {
t.Fatal("the marker must let go of the app after its put-back")
}
if g1.ResumePutBack("immich") != true || len(r.puts) != 1 {
t.Fatalf("a second resume must pour nothing, puts=%v", r.puts)
}
// the household writes on the drive, then any later controller start: nothing is poured back
householdWrite := filepath.Join(r.drive, "felhom-data", "new-photo-after-the-put-back")
mustWrite(t, householdWrite, "x")
if rec2 := r.guard().Recover(); rec2 != nil {
t.Fatalf("a later start must find nothing to recover, got %+v", rec2)
}
if len(r.puts) != 1 {
t.Fatalf("a later start poured the copy again over the household's writes: puts=%v", r.puts)
}
if _, err := os.Stat(householdWrite); err != nil {
t.Fatal("the household's newest file must still be there")
}
}
// Every start path is refused while the put-back waits: they all read the ONE hold store. The night chain and the
// household's buttons read it through RestoreHoldFor (isHeld, the router's restoreHoldFor, the update guards); the
// boot sweep and Recover through driveStartGate's GetRestoreHold; the web paths (boot recreate, drive return, media
// restart) through appHeld. The web and main.go wiring is pinned in their own packages.
func TestR1251_TheWaitingHoldRefusesEveryStartPath(t *testing.T) {
r := newR1251Rig(t)
r.unbind(t)
r.guard().Recover()
m := &Manager{settings: r.sett, logger: log.New(io.Discard, "", 0)}
held, why := m.RestoreHoldForLang("immich", "hu")
if !held || !strings.Contains(why, "meghajt") || !strings.Contains(why, "immich") {
t.Fatalf("the household's sentence must name the app and the drive: held=%v %q", held, why)
}
if _, en := m.RestoreHoldForLang("immich", "en"); !strings.Contains(en, "drive is back") {
t.Fatalf("the English sentence: %q", en)
}
if !m.isHeld("immich") {
t.Fatal("the night chain's isHeld must see the waiting put-back")
}
// the gated starter (the boot sweep's and Recover's path) refuses too
if err := (r1251Starter{r}).StartStack("immich"); !errors.Is(err, ErrStartRefused) {
t.Fatalf("the gated start must be refused while held, got %v", err)
}
// a removal ends the install the copy belongs to: the waiting hold goes with it (a later install must not wait)
if cleared, _ := r.sett.ClearUpdateHold("immich"); !cleared {
t.Fatal("a removal must clear the waiting put-back hold")
}
}
// H2: a put-back that fails at start leaves the app HELD (the D20 hold, „stopped half-way"), never stopped-and-unheld;
// the copy stays and the marker lets go (the hold owns the app).
func TestR1251_AFailedPutBackHoldsTheApp(t *testing.T) {
r := newR1251Rig(t)
r.putErr = errors.New("tar: short read")
rec := r.guard().Recover()
h, held := r.sett.GetRestoreHold("immich")
if !held || h.Reason != settings.HoldReasonRestoreMixed {
t.Fatalf("a failed put-back must leave a restore_mixed hold, got %+v held=%v", h, held)
}
if rec == nil || len(rec.Held) != 1 || !rec.Alarming() || len(r.starts) != 0 {
t.Fatalf("held, alarming, not started: rec=%+v starts=%v", rec, r.starts)
}
if !strings.Contains(h.ReplayError, "short read") || h.SafetyDump != r.undo {
t.Fatalf("the hold must carry the cause and the copy: %+v", h)
}
if _, err := os.Stat(r.undo); err != nil {
t.Fatal("the copy must stay on disk for support")
}
if r.markerActive() {
t.Fatal("the hold owns the app now — the marker must let go, so no later start pours an old copy over it")
}
// the household's Start, the boot recreate and the update are all refused by that hold
m := &Manager{settings: r.sett, logger: log.New(io.Discard, "", 0)}
if held, why := m.RestoreHoldForLang("immich", "hu"); !held || !strings.Contains(why, "immich") {
t.Fatalf("the D20 sentence: held=%v %q", held, why)
}
}
// A failed resume (the drive came back, the put-back broke) holds the app the same way.
func TestR1251_AFailedResumeHoldsTheApp(t *testing.T) {
r := newR1251Rig(t)
r.unbind(t)
g := r.guard()
g.Recover()
r.bind(t)
r.putErr = errors.New("disk full")
if g.ResumePutBack("immich") {
t.Fatal("a failed put-back must not allow the start")
}
if h, _ := r.sett.GetRestoreHold("immich"); h.Reason != settings.HoldReasonRestoreMixed {
t.Fatalf("a failed resume must hold for support, got %+v", h)
}
if r.markerActive() {
t.Fatal("the marker must let go after the hold took over")
}
}
// A copy that vanished while its drive is BOUND, during a waiting put-back, cannot be poured: held for support.
func TestR1251_ACopyGoneWhileItsDriveIsBoundHoldsTheApp(t *testing.T) {
r := newR1251Rig(t)
r.unbind(t)
g := r.guard()
g.Recover()
r.bind(t)
if err := os.RemoveAll(r.undo); err != nil {
t.Fatal(err)
}
if g.ResumePutBack("immich") {
t.Fatal("a vanished copy must not allow the start")
}
if h, _ := r.sett.GetRestoreHold("immich"); h.Reason != settings.HoldReasonRestoreMixed {
t.Fatalf("held for support, got %+v", h)
}
}
// H7: Stop pressed during the restore — the copy is still poured back after the cut; only the start is skipped.
func TestR1251_StopPressedDuringTheRestoreStillPutsBack(t *testing.T) {
r := newR1251Rig(t)
r.stopped["immich"] = true
rec := r.guard().Recover()
if strings.Join(r.puts, ",") != "immich" {
t.Fatalf("the put-back must run even though the household pressed Stop, puts=%v rec=%+v", r.puts, rec)
}
if len(r.starts) != 0 {
t.Fatalf("the household stopped it — it must not start, starts=%v", r.starts)
}
if r.markerActive() {
t.Fatal("nothing is owed any more — the marker must clear")
}
}
// The drive returns later (not at boot): the sweep puts back, then starts through the gated starter — unless the
// household stopped the app.
func TestR1251_ADriveThatReturnsLaterIsPutBackThenStarted(t *testing.T) {
for _, stopped := range []bool{false, true} {
r := newR1251Rig(t)
r.unbind(t)
g := r.guard()
g.Recover()
if got := g.ResumeWaitingPutBacks(); len(got) != 0 || len(r.puts) != 0 {
t.Fatalf("drive still absent: nothing may happen, started=%v puts=%v", got, r.puts)
}
r.bind(t)
r.stopped["immich"] = stopped
got := g.ResumeWaitingPutBacks()
if len(r.puts) != 1 {
t.Fatalf("stopped=%v: the put-back must run once, puts=%v", stopped, r.puts)
}
if want := !stopped; (len(got) == 1) != want || (len(r.starts) == 1) != want {
t.Fatalf("stopped=%v: started=%v starts=%v", stopped, got, r.starts)
}
}
}
// A later restore that completes drops the waiting put-back: the household chose that restore's data.
func TestR1251_ACompletedRestoreDropsTheWaitingPutBack(t *testing.T) {
r := newR1251Rig(t)
r.unbind(t)
r.guard().Recover()
m := &Manager{settings: r.sett, logger: log.New(io.Discard, "", 0)}
m.clearUpdateHoldAfterRestore("immich")
if _, held := r.sett.GetRestoreHold("immich"); held {
t.Fatal("a completed restore must drop the waiting put-back hold")
}
}
@@ -653,6 +653,16 @@ func (m *Manager) clearUpdateHoldAfterRestore(stackName string) {
return
}
h, ok := m.settings.GetRestoreHold(stackName)
if ok && h.Reason == settings.HoldReasonPutBackPending {
// R-1251: a restore that completed replaced the data a waiting put-back would have poured back — the household
// chose this restore, so the old copy must never be poured over it when its drive returns. The copy stays.
if _, err := m.settings.ClearRestoreHold(stackName); err != nil {
m.logger.Printf("[ERROR] [backup] %s was restored, but its waiting put-back hold could not be cleared: %v", stackName, err)
return
}
m.logger.Printf("[INFO] [backup] %s: restore completed — the waiting put-back (copy %s, set %s) is DROPPED; the copy stays on disk", stackName, h.SafetyDump, h.At)
return
}
if !ok || h.Reason != settings.HoldReasonUpdateFailed {
return
}
+1
View File
@@ -1653,6 +1653,7 @@
"hold.copy_holds.files": "holds the settings, the database and the files",
"hold.copy_holds.volumes": "holds the settings, the database and the data volumes",
"hold.unhealthy_stop.repeat": "We stopped the app again because it kept crashing (or ran out of memory). Felhom support has been told. Press Start to try again.",
"hold.putback_pending": "Restoring %s was interrupted. Its data from before the restore is on a drive that is not available now. When the drive is back, the box puts that data back and starts the app. Until then the app does not start.",
"hold.unhealthy_stop": "We stopped the app because it kept crashing (or ran out of memory). Press Start to try again.",
"hold.update.no_whole_copy": "The update of %s did not work, and neither did the automatic undo. This box has no copy that can bring the app back together with its files. Felhom support has been told — until then, do not restart the app or delete its data.",
"hold.update.sentence": "The update of %s at %s did not succeed, and the app did not start on the new version. The app stays stopped for safety, so that its data is not damaged. It can be restored on the Backups page from this backup: %s, %s — this copy %s.",
+1
View File
@@ -1641,6 +1641,7 @@
"hold.copy_holds.files": "a beállításokat, az adatbázist és a fájlokat tartalmazza",
"hold.copy_holds.volumes": "a beállításokat, az adatbázist és az adatköteteket tartalmazza",
"hold.unhealthy_stop.repeat": "Az alkalmazást ismét leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). A Felhom ügyfélszolgálatát értesítettük. Az Indítás gombbal újra megpróbálhatod.",
"hold.putback_pending": "A(z) %s visszaállítása megszakadt. A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el. Amikor a meghajtó újra elérhető, a doboz visszateszi ezeket az adatokat, és elindítja az alkalmazást. Addig az alkalmazás nem indul el.",
"hold.unhealthy_stop": "Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.",
"hold.update.no_whole_copy": "A(z) %s frissítése nem sikerült, és az automatikus visszaállítás sem. Ezen a dobozon nincs olyan másolat, amely az alkalmazást a fájljaival együtt vissza tudná hozni. A Felhom ügyfélszolgálatát értesítettük — addig ne indítsd újra, és ne töröld az adatait.",
"hold.update.sentence": "A(z) %s frissítése %s-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: %s, %s — ez a másolat %s.",
@@ -18,7 +18,8 @@ func TestR704_ClearUpdateHoldClearsTheInstallsHoldsOnly(t *testing.T) {
for _, c := range []struct {
reason string
want bool
}{{HoldReasonUpdateFailed, true}, {HoldReasonUnhealthyStop, true}, {HoldReasonRestoreFailed, false}} {
}{{HoldReasonUpdateFailed, true}, {HoldReasonUnhealthyStop, true}, {HoldReasonRestoreFailed, false},
{HoldReasonPutBackPending, true}, {HoldReasonRestoreMixed, false}} { // R-1251: a waiting put-back ends with the install
if err := s.SetRestoreHold(RestoreHold{Stack: "app", At: "2026-09-13T19:56:32Z", Reason: c.reason}); err != nil {
t.Fatal(err)
}
+7 -1
View File
@@ -1988,6 +1988,10 @@ const (
// rolled back, the live definition written back, and the app is held: its files and volumes are the
// snapshot's, its database the pre-restore one. Cleared like a restore hold — by the operator.
HoldReasonRestoreMixed = "restore_mixed"
// HoldReasonPutBackPending (R-1251): a restore was cut after its copy of the live data was taken, and the copy
// is on a drive that was not bound when the box came back. Nothing may start the app until the copy is poured
// back (backup.AppStopGuard.ResumePutBack); SafetyDump holds the copy's folder. Cleared by the put-back.
HoldReasonPutBackPending = "putback_pending"
)
// SetRestoreHold records a hold. Modelled on SetDisconnected: a condition, plus what it is holding.
@@ -2044,11 +2048,13 @@ func (s *Settings) ClearRestoreHold(stack string) (bool, error) {
// stays: that one means a database was left in an unknown state and is cleared by the operator
// (`-clear-restore-hold`), never by a removal or an install. Returns whether a hold was cleared.
// Pinned by TestR704_ClearUpdateHoldClearsTheInstallsHoldsOnly.
// R-1251: a put-back still waiting for its drive (HoldReasonPutBackPending) belongs to the install too — its copy is
// of the removed install's volumes, and a later install of the same name must not wait for it.
func (s *Settings) ClearUpdateHold(stack string) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
h, ok := s.RestoreHolds[stack]
if !ok || (h.Reason != HoldReasonUpdateFailed && h.Reason != HoldReasonUnhealthyStop) {
if !ok || (h.Reason != HoldReasonUpdateFailed && h.Reason != HoldReasonUnhealthyStop && h.Reason != HoldReasonPutBackPending) {
return false, nil
}
delete(s.RestoreHolds, stack)
+20 -2
View File
@@ -222,7 +222,7 @@ func (s *Server) restartStacks(names []string) {
// Slice 4 / R-379: the drive-return gate starts apps UNATTENDED, so it must honour a hold
// exactly as the customer's button does. Until v0.237.0 it did not — a held app whose drive
// blinked would have been started again. "A hold that only one path honours is not a hold."
if s.appHeld(name) {
if !s.resumePutBack(name) || s.appHeld(name) {
continue
}
if err := s.stackMgr.StartStack(name); err != nil {
@@ -346,6 +346,13 @@ func (s *Server) ReconcileDriveGates() {
}
}
s.restartStacks(stopped)
// R-1251: a put-back that waited for a drive runs when one returns (the app was held, not gate-stopped,
// so it is not in `stopped`); the guard starts it afterwards through the gated starter.
if s.backupMgr != nil {
if started := s.backupMgr.AppStopGuard().ResumeWaitingPutBacks(); len(started) > 0 {
s.logger.Printf("[WARN] [gate] drive RETURNED %s — %d interrupted restore(s) put back, then started: %v", a.Path, len(started), started)
}
}
if err := s.settings.ClearDisconnected(a.Path); err != nil {
s.logger.Printf("[WARN] [gate] clear disconnected %s: %v", a.Path, err)
}
@@ -469,7 +476,9 @@ func (s *Server) processGuestBootChange() {
}
recreate := func(bs bootStack) {
s.logger.Printf("[INFO] [gate] boot %s: live bind confirmed — recreating drive-backed app %s (state=%s) onto %s", resp.GuestBootID, bs.name, bs.state, bs.hdd)
if s.appHeld(bs.name) {
// R-1251: a cut restore whose copy waited for THIS drive is poured back first — the recreate used to start
// the app on the cut restore's data, and a later start poured the old copy over what the household wrote.
if !s.resumePutBack(bs.name) || s.appHeld(bs.name) {
return
}
_ = s.stackMgr.StopStack(bs.name)
@@ -713,6 +722,15 @@ func (s *Server) notifyDriveReturned(path string, isTarget map[string]bool) {
s.notifier.NotifyStorageReconnected(label)
}
// resumePutBack (R-1251) pours back a cut restore's copy that waited for this app's drive, BEFORE any start. false =
// the app stays held (the copy is still unreachable, or the put-back failed and the app is held for support). Nil-safe.
func (s *Server) resumePutBack(name string) bool {
if s.backupMgr == nil {
return true
}
return s.backupMgr.AppStopGuard().ResumePutBack(name)
}
// appHeld reports whether an app carries a hold (failed update or failed restore) and logs the skip.
// Nil-safe: no backup manager means no hold store, so nothing is held.
func (s *Server) appHeld(name string) bool {
@@ -0,0 +1,62 @@
package web
import (
"go/ast"
"go/parser"
"go/token"
"os"
"strings"
"testing"
)
// R-1251 (review round 2, HUNT-1 H1): the two web paths that START a drive-backed app when its drive is there — the
// boot recreate and the drive-return restart — must pour a waiting put-back back FIRST (resumePutBack), before
// appHeld and before StartStack; and the drive-return branch must run the sweep for held apps it did not stop. A
// source pin: the decision is tested in internal/backup (r1251_putback_waits_test.go); this pins the WIRING.
// COMPANION RED-PROOF (2026-10-24): drop `!s.resumePutBack(...) ||` from either path, or the sweep from the Return
// branch → this fails.
func TestR1251_DriveStartPathsPutBackBeforeTheyStart(t *testing.T) {
src, err := os.ReadFile("intermediary.go")
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "intermediary.go", src, 0)
if err != nil {
t.Fatal(err)
}
body := map[string]string{}
for _, d := range f.Decls {
if fn, ok := d.(*ast.FuncDecl); ok && fn.Body != nil {
body[fn.Name.Name] = string(src[fset.Position(fn.Body.Pos()).Offset:fset.Position(fn.Body.End()).Offset])
}
}
order := func(fn, first, then string) {
t.Helper()
b, ok := body[fn]
if !ok {
t.Fatalf("function %s not found in intermediary.go — the pin is stale", fn)
}
i, j := strings.Index(b, first), strings.Index(b, then)
if i < 0 || j < 0 || i > j {
t.Errorf("%s: %q must come before %q (found at %d, %d) — the app would start on the cut restore's data", fn, first, then, i, j)
}
}
order("restartStacks", "!s.resumePutBack(name)", "s.stackMgr.StartStack(name)")
order("restartStacks", "!s.resumePutBack(name)", "s.appHeld(name)")
// the boot recreate is a closure inside the boot-id handler; find the function that holds it
var recreateFn string
for name, b := range body {
if strings.Contains(b, "live bind confirmed — recreating drive-backed app") {
recreateFn = name
}
}
if recreateFn == "" {
t.Fatal("the boot recreate closure was not found — the pin is stale")
}
order(recreateFn, "!s.resumePutBack(bs.name)", "s.stackMgr.StartStack(bs.name)")
order("ReconcileDriveGates", "s.restartStacks(stopped)", "ResumeWaitingPutBacks()")
if !strings.Contains(body["resumePutBack"], "ResumePutBack(name)") {
t.Error("resumePutBack no longer asks the app-stop guard")
}
}
+1
View File
@@ -10,6 +10,7 @@
"recovery_info.body": "BORN AS A KEY (R-554, 2026-10-12, `09` §3 decision 222) -- the NEW text of recovery-info.txt, pointing at the dashboard's Restore page and the recovery code instead of the deleted setup wizard. The operator approves the Hungarian before release. Pinned by internal/recovery TestRecoveryInfo_PointsAtDashboardNotWizard.",
"err.backup.db_restore_failed_held_mixed": "BORN AS A KEY (R-893, 2026-10-08) -- a NEW sentence for a failed off-site replay whose database rollback worked but whose files/volumes/version had already moved; the app is held. Never a Go literal; pinned by internal/backup/r893_mixed_restore_test.go.",
"err.backup.restore_failed_held_mixed": "BORN AS A KEY (R-893, 2026-10-08, security review) -- a restore that stopped half-way (placement, volume replay or DB start failed) after files/volumes/version had moved; the app is HELD. Pinned by internal/backup/r893_mixed_restore_test.go.",
"hold.putback_pending": "BORN AS A KEY (R-1251, 2026-10-24) -- the hold sentence for an app whose cut restore waits for its drive before its pre-restore data is put back (HoldReasonPutBackPending). Never a Go literal; pinned by internal/backup/r1251_putback_waits_test.go.",
"note.reconstitute.held_mixed": "BORN AS A KEY (R-893, 2026-10-08) -- the hold sentence for that held app (HoldReasonRestoreMixed). Never a Go literal; pinned by internal/backup/r893_mixed_restore_test.go.",
"recovery.older_unchecked": "BORN AS A KEY (R-304, 2026-10-08) -- a NEW sentence of the recovery screen for the agent's 424 (earlier sealed packages not all checked), never a Go literal; pinned by TestR304_OlderUnchecked_IsNotAWrongCode.",
"banner.missed_backup.never": "BORN AS A KEY, v0.295.0 (R-871, `09` decision 110) -- a NEW sentence of the missed-backup banner, never a Go literal; pinned in Hungarian by TestR871_BannerShownThenClosedUntilTheNextMiss and the parity fixture launcher_missed_backup.",