R-359 + R-397: the off-site store gets checked, and the advertised check becomes real

Nothing ever verified that the off-site copies are still readable. The
whole-guest tier has verify jobs; the tier holding the customer's documents and
photos had none -- the complete set of restic verbs this controller used
contained no `check`. We would have found out at restore time, with a customer
waiting. On 2026-08-21 a deliberately damaged pack was caught at once by plain
`restic check`; we had never run it.

R-397: NotifyIntegrityOK/NotifyIntegrityFailed existed with no caller, the hub
allowlists both event types and carries the Hungarian text for both, the
settings checkbox exists, and the debug button posts to /api/debug/backup/
integrity. Everything was built except the part that runs. SIXTH instance of
that shape in this project.

THE HAZARD SHAPES THE WHOLE DESIGN. resticStep self-heals a crash lock by
running `unlock --remove-all` and retrying, and its own comment records why that
is safe: every caller holds the in-process single-flight mutex, so any lock it
meets is stale. A check that did not take that flag could meet a LIVE prune's
lock from this same box, remove it, and retry over the top of it. So the check
TAKES THE FLAG and SKIPS rather than waits -- waiting would pin the nightly
backup behind it, and a skip costs nothing because due-ness makes tomorrow try
again. TestR359_SkipsWhenRunningFlagHeld asserts the NON-EFFECTS: restic never
invoked, `unlock` never in any argv. Its red-proof prints the real thing --
restic running `check` while the flag was held.

DUE-NESS, NOT A WEEKDAY. Daily job, weekly behaviour: "is the last successful
check older than 7 days?" not "is it Sunday?". R-341 is exactly the other shape,
a dated check quietly missed and never caught up. No Weekly primitive added.

THREE OUTCOMES, NOT TWO. Skipped, Unreachable and failed are different facts.
"I could not look" is not "I looked and it is broken" -- R-339 already owns
reachability, and a second alarm for the same fact trains the operator to
discount the one alarm that means the backups are damaged. A timeout is
unreachable, never damage. A failure advances due-ness (a broken store must not
be re-checked nightly); a skip and an unreachable store do not.

Success is severity `info`, which severityNotifies DROPS -- it mails NOBODY, by
design. A weekly success e-mail is how people stop reading their alerts.

The customer gets a SENTENCE; restic's words go to the log, truncated (R-379:
615 bytes of raw database text reached a customer once). read-data-subset ships
OFF and a malformed value is refused at read time rather than handed to restic,
where one typo would fail the whole check.

Published on OffboxReportStatus, NOT on report.BackupReport's IntegrityOK --
those were retired by R-331 YESTERDAY and TestBackupReport_DeadFieldsStayZero
still passes unmodified.

Also: the monitoring page stopped promising a Sunday job that never existed, and
the debug button got its dispatch case.

PART 0 WAS NOT BUILT, AND R-398 WAS MY OWN MISTAKE. The seam it asked for
already exists: offboxRunner/SetOffboxRunner/m.runner() has been injectable
since the off-site tier shipped, and other tests drive restic-backed paths
through it. A resticStepFn seam would have been WORSE here -- it would replace
the `unlock --remove-all` escalation and hide it from the assertions that must
see it. R-358's AST ordering test is converted to a real execution test instead,
which immediately surfaced something the AST walk could not: unlockStale
legitimately runs before the restore.

Four red-proofs, each printing the pre-fix behaviour. Green gate: 28 packages,
rc 0. All 12 controller gates OK.

controller/internal/backup/r358_scratch_marker_test.go

@@ -2,9 +2,8 @@ package backup
 
 import (
 	"bytes"
-	"go/ast"
-	"go/parser"
-	"go/token"
+	"context"
+	"encoding/json"
 	"log"
 	"os"
 	"path/filepath"
@@ -186,63 +185,123 @@ func TestR358_MarkerIsNeverPlaced(t *testing.T) {
 	}
 }
 
-// TestR358_MarkerIsClearedBeforeResticAndWrittenAfter walks the AST of RestoreOffboxScratch.
+// TestR358_MarkerOrderingIsExecuted — R-398, and the row that prompted it was WRONG in a way worth
+// recording rather than quietly fixing.
 //
-// It exists because `resticStep` is not a seam — a test cannot run the real download without restic,
-// so the ORDER of the three calls cannot be proven by execution here. Order is the entire safety
-// property: a marker written before restic certifies a download that has not happened, and a clear
-// that runs after it leaves a stale certificate covering a fresh part-copy. A substring search would
-// not do: a commented-out call satisfies strings.Contains, which a sibling test in this project
-// records paying for.
-func TestR358_MarkerIsClearedBeforeResticAndWrittenAfter(t *testing.T) {
-	fset := token.NewFileSet()
-	f, err := parser.ParseFile(fset, "offbox_restore.go", nil, 0)
-	if err != nil {
-		t.Fatalf("parse offbox_restore.go: %v", err)
-	}
-	var body *ast.BlockStmt
-	for _, d := range f.Decls {
-		if fn, ok := d.(*ast.FuncDecl); ok && fn.Name.Name == "RestoreOffboxScratch" && fn.Body != nil {
-			body = fn.Body
-		}
-	}
-	if body == nil {
-		t.Fatal("RestoreOffboxScratch not found")
+// This test used to walk the AST of RestoreOffboxScratch, on the stated ground that "`resticStep` is
+// not a seam, so the ORDER cannot be proven by execution". **The first half is true and the conclusion
+// was false.** `resticStep` is not overridable, but the layer it calls — `offboxRunner`, injected by
+// `SetOffboxRunner` — has been a seam since the off-site tier shipped, and other tests in this package
+// have been driving restic-backed paths through it all along. R-398 was filed off my own mistaken
+// reading; it is corrected, not closed.
+//
+// Executing it is strictly stronger than the AST walk, and not only because it runs the real function:
+// the runner sees EVERY argv, including the `unlock --remove-all` escalation that `resticStep` performs
+// on a lock error. A `resticStepFn` seam — which R-398 proposed and this test's existence argued for —
+// would have REPLACED that escalation and hidden it from exactly the assertions that need to see it.
+//
+// The order is the whole safety property: a marker written before restic certifies a download that has
+// not happened, and a clear that runs after it leaves a stale certificate over a fresh part-copy.
+func TestR358_MarkerOrderingIsExecuted(t *testing.T) {
+	m, scratch := newR358Manager(t)
+	// A stale marker from a previous run, so "cleared before" is observable as a state change rather
+	// than as an absence that was always absent.
+	if err := m.writeScratchMarker(scratch, "snap-OLD", true); err != nil {
+		t.Fatal(err)
 	}
 
-	var order []string
-	ast.Inspect(body, func(n ast.Node) bool {
-		call, ok := n.(*ast.CallExpr)
-		if !ok {
-			return true
-		}
-		if sel, ok := call.Fun.(*ast.SelectorExpr); ok {
-			switch sel.Sel.Name {
-			case "clearScratchMarker", "resticStep", "writeScratchMarker":
-				order = append(order, sel.Sel.Name)
+	var events []string
+	markerAtRestic := "unset"
+	m.SetOffboxLatestSnapshotFn(func(context.Context, string) (string, []string, error) {
+		return "snap-NEW", []string{"/mnt/old/backups/primary/kimai"}, nil
+	})
+	m.SetOffboxFreeFn(func(string) int64 { return 100 << 30 })
+	m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) {
+		switch {
+		case containsArg(args, "cat") && containsArg(args, "config"):
+			return []byte(`{"version":2}`), nil
+		case containsArg(args, "stats"):
+			return []byte(`{"total_size":1024}`), nil
+		case containsArg(args, "restore"):
+			events = append(events, "restic-restore")
+			// THE OBSERVATION: what does the marker look like at the moment restic runs? If the clear
+			// happened first it is gone; if the write happened first it is already there, certifying a
+			// download that has not finished.
+			if _, err := os.Stat(filepath.Join(scratch, scratchMarkerName)); err == nil {
+				markerAtRestic = "present"
+			} else {
+				markerAtRestic = "absent"
 			}
+			return nil, nil
+		case containsArg(args, "unlock"):
+			// TWO DIFFERENT UNLOCKS, and only one of them is dangerous. Plain `unlock` is restic's
+			// stale-ONLY sweep, run as pre-run hygiene by unlockStale before every restore.
+			// `unlock --remove-all` is resticStep's crash-lock escalation, and it is the act that must
+			// never happen beside a live sibling operation. Recording them separately here is what
+			// lets the R-359 lock-safety tests assert the second is absent without tripping over the
+			// first — a distinction the AST walk this replaced could not have surfaced at all.
+			if containsArg(args, "--remove-all") {
+				events = append(events, "unlock--remove-all")
+			} else {
+				events = append(events, "unlock-stale")
+			}
+			return nil, nil
 		}
-		return true
+		return nil, nil
 	})
 
-	idx := func(name string) int {
-		for i, n := range order {
-			if n == name {
-				return i
-			}
-		}
-		return -1
+	if err := m.RestoreOffboxScratch(context.Background(), "kimai", true); err != nil {
+		t.Fatalf("RestoreOffboxScratch: %v", err)
 	}
-	clear, restic, write := idx("clearScratchMarker"), idx("resticStep"), idx("writeScratchMarker")
-	if clear < 0 || restic < 0 || write < 0 {
-		t.Fatalf("RestoreOffboxScratch does not call all three (order seen: %v) — the marker is not wired", order)
+
+	if markerAtRestic != "absent" {
+		t.Fatalf("the completion marker was %s while restic was still downloading — a marker that "+
+			"predates the download certifies a copy that does not exist yet (and here it was the "+
+			"PREVIOUS run's marker, covering a fresh part-copy)", markerAtRestic)
 	}
-	if !(clear < restic) {
-		t.Errorf("the stale marker is cleared AFTER restic runs (order %v) — a previous run's "+
-			"certificate would cover this run's part-copy", order)
+	// The restore must actually have run, or the ordering assertion above proves nothing. It is NOT
+	// asserted to be first: `unlockStale` legitimately precedes it as pre-run hygiene, which this
+	// execution test surfaced on its first run and the AST walk could never have shown.
+	if !containsEvent(events, "restic-restore") {
+		t.Fatalf("restic never ran, so this test proves nothing about ordering (events=%v)", events)
 	}
-	if !(restic < write) {
-		t.Errorf("the marker is written BEFORE restic returns (order %v) — that certifies a download "+
-			"that has not happened, which is the defect with an extra step", order)
+	// And the DANGEROUS unlock never appeared: nothing here met a lock, so resticStep never escalated.
+	if containsEvent(events, "unlock--remove-all") {
+		t.Fatalf("`unlock --remove-all` ran during an ordinary restore (events=%v) — that escalation "+
+			"is only safe because the single-writer flag proves no sibling is live", events)
+	}
+	// And after a successful run the marker is there, carrying THIS run's snapshot, not the old one.
+	data, err := os.ReadFile(filepath.Join(scratch, scratchMarkerName))
+	if err != nil {
+		t.Fatalf("no marker after a successful restore: %v", err)
+	}
+	var mk scratchMarker
+	if err := json.Unmarshal(data, &mk); err != nil {
+		t.Fatal(err)
+	}
+	if mk.SnapshotID != "snap-NEW" || !mk.Full {
+		t.Fatalf("marker = %+v, want snapshot snap-NEW and full=true — the stale one survived", mk)
+	}
+	if !m.OffboxFullScratchReady("kimai") {
+		t.Fatal("a completed full restore did not read as ready")
 	}
 }
+
+func containsEvent(ev []string, want string) bool {
+	for _, e := range ev {
+		if e == want {
+			return true
+		}
+	}
+	return false
+}
+
+// containsArg is a local helper so this file does not depend on another test file's ordering.
+func containsArg(args []string, want string) bool {
+	for _, a := range args {
+		if a == want {
+			return true
+		}
+	}
+	return false
+}