R-353/R-357/R-358/R-360: the restore tells the truth (v0.226.0)

Four defects on the restore surface, all proven on demo-hp during the 2026-08-21
backup-truth drill, all still in shipped code. They share one acceptance idea: a
restore surface must state what it actually did, and must refuse what it cannot
do.

VERSION NOTE. The task specifying this targeted v0.224.0 against baseline
f8c9390. Both were consumed earlier the same day by R-330 (0.224.0) and R-331
(0.225.0). Drift re-confirmed against live Gitea before the first edit, operator
authorised proceeding, every symbol the spec named re-verified present at the
real baseline e5eee50.

R-353 -- a restore that gave back nothing still said it worked.
RestoreFromRecoveryUnit returned only error, so the surface printed
"<app> visszaallitva (<snapshot>)." -- equally true of a run that returned an
entire dataset and one that returned nothing. The count already existed and was
discarded one line deep: restoreDockerVolumesFrom always returned it, the
wrapper threw it away. Now (UnitRestoreResult, error), carrying replayed counts
AND what the manifest LISTED, because zero-replayed has two causes that are
opposite news. Three cases, three sentences, and EVERY one is a claim about the
BACKUP, never about the app -- this path has no SafetyDump discriminator, and
07-backup-architecture 6.3 records that an absent dump says nothing about the
app (R-361 destroyed canonical .sql files for four months).

R-357 -- the destructive restore had no free-space gate. offbox_reconstitute.go
contained ZERO references to offboxFree; all three existing gates guard
non-destructive paths. The gate now sits before mapOffsiteRestorePaths,
writeSafetyDump and StopStack, so a refusal costs nothing. Position IS the fix,
which is why the test asserts StopStack was never called. No headroom multiplier
(matches PlaceOffsiteRestore; the x1.1 elsewhere predicts a download). Fail
closed on either probe <= 0 -- otherwise `free < need` with need==0 is FALSE and
an unmeasurable scratch sails through: a gate present and inert.

R-358 -- a failed download was offered as a good one. The gate answered "the
directory exists and is non-empty", which is exactly what a part-way restic run
leaves. Now a completion marker written 0600 atomically AFTER restic returns
nil, with any stale one cleared BEFORE it starts; both orders pinned by an AST
test because resticStep is not a seam. Both handlers refuse server-side: the
wizard flags control a button, and a hidden button is not a guard.

SCENARIO F ANSWERED, and worse than the question assumed: a unit-only scratch IS
reachable through the real flow, by the most ordinary route. "Ellenorzo
visszaallitas" (mode=unit, advertised non-destructive) writes the SAME directory
-- offboxRestoreScratchDir ignores `full` and --include limits what restic
extracts, never where -- so a customer who ran the SAFE restore was then offered
the destructive one over a unit-only copy. Filed R-396; the marker closes it.

R-360 -- the delete refused only while a BACKUP ran. IsRunning() is FALSE for the
whole of a verification restore; the five sibling handlers all use
restoreOpBlocked(). Its doc comment claimed it already did this, which is why
nobody looked -- corrected in place.

Red-proofs, each printing the pre-fix behaviour, in CHANGELOG and REPORT. The
first R-357 red-proof exposed a hollow test OF MY OWN and it is recorded rather
than quietly fixed: the fixture refused earlier at the placement stat pre-pass,
so `stops == 0` passed against the pre-fix code. Fixture corrected, assertions
reordered so a removed gate reports the outage rather than "no error returned".

Green gate clean: 28 packages, rc 0. All 12 controller gates OK.

controller/internal/backup/offbox_restore.go

@@ -30,6 +30,20 @@ const (
 // SetOffboxFreeFn overrides the restore free-space probe (tests; the Windows go-test host has no df).
 func (m *Manager) SetOffboxFreeFn(fn func(path string) int64) { m.offboxFreeFn = fn }
 
+// WriteScratchMarkerForTest exposes the marker writer to the web package's flow test. Test-only by
+// name so a production caller reads as obviously wrong: only RestoreOffboxScratch may certify a
+// scratch, because only it knows whether the download finished.
+func (m *Manager) WriteScratchMarkerForTest(scratch, snapshotID string, full bool) error {
+	return m.writeScratchMarker(scratch, snapshotID, full)
+}
+
+// SetOffboxLatestSnapshotFn overrides the restic snapshot lookup (tests; no restic needed). See the
+// field comment on Manager.offboxLatestSnapFn for why this seam exists rather than a code-reading
+// argument that the R-357 gate sits early enough.
+func (m *Manager) SetOffboxLatestSnapshotFn(fn func(ctx context.Context, stack string) (string, []string, error)) {
+	m.offboxLatestSnapFn = fn
+}
+
 // SetOffboxFullPlaceCopier overrides the FULL-restore overwrite copier (tests; no rsync needed).
 func (m *Manager) SetOffboxFullPlaceCopier(fn func(src, dst string) (int, error)) {
 	m.offboxFullPlaceCopier = fn
@@ -88,6 +102,9 @@ func offboxUnitPathOf(paths []string, stack string) string {
 // `snapshots latest --tag <stack> --json`. When the tag spans more than one group (old unit-only shape
 // + new enlarged shape), it returns the newest by time.
 func (m *Manager) offboxLatestSnapshot(ctx context.Context, stack string) (id string, paths []string, err error) {
+	if m.offboxLatestSnapFn != nil {
+		return m.offboxLatestSnapFn(ctx, stack)
+	}
 	t := m.settings.GetOffboxTarget()
 	base, env := m.offboxBaseArgs(t)
 	sctx, cancel := context.WithTimeout(ctx, offboxProbeTimeout)
@@ -239,14 +256,14 @@ func (m *Manager) RestoreOffboxScratch(ctx context.Context, stack string, full b
 		size, serr := m.offboxSnapshotSize(ctx, id)
 		if serr != nil {
 			// SizeUnknown never renders as fits — fail closed.
-			return fmt.Errorf("A mentés mérete nem állapítható meg — a teljes visszaállítás biztonsági okból nem indítható.")
+			return fmt.Errorf(offsiteSizeUnknownMsg)
 		}
 		need := size + size/10 // ×1.1
 		if free < need {
-			return fmt.Errorf("Nincs elég szabad hely a visszaállításhoz (%s szükséges, %s szabad).", humanizeBytes(need), humanizeBytes(free))
+			return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(need), humanizeBytes(free))
 		}
 	} else if free < offboxUnitOnlyFreeFloor {
-		return fmt.Errorf("Nincs elég szabad hely a visszaállításhoz (%s szükséges, %s szabad).", humanizeBytes(offboxUnitOnlyFreeFloor), humanizeBytes(free))
+		return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(offboxUnitOnlyFreeFloor), humanizeBytes(free))
 	}
 	// F-A1 hygiene: drop the legacy rootfs scratch (DataDir/offbox-restore/<app>) best-effort.
 	legacy := filepath.Join(m.cfg.Paths.DataDir, "offbox-restore", stack)
@@ -260,6 +277,11 @@ func (m *Manager) RestoreOffboxScratch(ctx context.Context, stack string, full b
 	if err := os.MkdirAll(scratch, 0o755); err != nil {
 		return fmt.Errorf("restore dir: %w", err)
 	}
+	// R-358: a marker from a PREVIOUS run must never certify this one. Cleared here, before restic
+	// touches anything, so the window in which a stale certificate could vouch for a part-copy does not
+	// exist. If this run fails, the scratch is left with files and NO marker — which is precisely the
+	// state OffboxFullScratchReady must read as "not ready".
+	m.clearScratchMarker(scratch)
 	t := m.settings.GetOffboxTarget()
 	base, env := m.offboxBaseArgs(t)
 	rctx, cancel := context.WithTimeout(ctx, offboxBackupTimeout)
@@ -274,9 +296,94 @@ func (m *Manager) RestoreOffboxScratch(ctx context.Context, stack string, full b
 		return fmt.Errorf("offbox restore %s: %w: %s", stack, rerr, truncate(out))
 	}
 	m.logger.Printf("[INFO] [offbox] restored %s (%s, full=%v) → %s", stack, id, full, scratch)
+	// R-358: the completion certificate, written ONLY now — after restic returned nil. Writing it
+	// earlier would certify a download that has not happened, which is the defect with an extra step.
+	// Written for full=false runs too: the `full` field inside it, not its presence, is what
+	// distinguishes a unit-only scratch from a complete one.
+	if err := m.writeScratchMarker(scratch, id, full); err != nil {
+		// The restore itself succeeded, so this is not an error to fail the operation on — but it is
+		// NOT silent, and the consequence is stated: without the marker the scratch reads as not-ready,
+		// which is the fail-closed direction. Better a re-run than a placement over an uncertified copy.
+		m.logger.Printf("[ERROR] [offbox] %s: restore succeeded but the completion marker could not be written: %v — the scratch will read as NOT ready and the download must be re-run", stack, err)
+	}
 	return nil
 }
 
+// --- R-358: the scratch completion marker ------------------------------------------------------
+//
+// THE DEFECT. `OffboxFullScratchReady` used to answer "the directory exists and is non-empty". A restic
+// download that failed part-way leaves exactly that: a directory with files in it. So the product
+// offered „Teljes visszaállítás indítása" over a part-copy, and pressing it reported success —
+// observed on demo-hp 2026-08-21. A non-empty directory is evidence that something was written, never
+// that everything was.
+//
+// The marker is the missing fact: not "are there files" but "did the run that wrote them FINISH, and
+// was it the full one". Only the run itself can know that, so only the run writes it.
+//
+// It lives at the scratch ROOT, which is safe from placement for a reason worth stating rather than
+// assuming: `mapOffsiteRestorePaths` builds placements from the SNAPSHOT's own path list, not from a
+// directory walk, so a file that exists only locally is invisible to it. That is pinned by
+// TestR358_MarkerIsNeverPlaced rather than left as a comment.
+const scratchMarkerName = ".felhom-restore-complete.json"
+
+// scratchMarker is the on-disk completion certificate. `Schema` is carried so a future format change
+// is a refusal rather than a misreading — an unrecognised schema fails closed like every other
+// unreadable marker.
+type scratchMarker struct {
+	Schema     int    `json:"schema"`
+	SnapshotID string `json:"snapshot_id"`
+	Full       bool   `json:"full"`
+	FinishedAt string `json:"finished_at"`
+}
+
+const scratchMarkerSchema = 1
+
+// clearScratchMarker removes any existing marker, best-effort. A failure to remove is logged and NOT
+// returned: the caller is about to overwrite the scratch anyway, and refusing a restore because a stale
+// certificate would not delete trades a real capability for a bookkeeping problem.
+func (m *Manager) clearScratchMarker(scratch string) {
+	if err := os.Remove(filepath.Join(scratch, scratchMarkerName)); err != nil && !os.IsNotExist(err) {
+		m.logger.Printf("[WARN] [offbox] could not clear the stale scratch marker in %s: %v", scratch, err)
+	}
+}
+
+// writeScratchMarker writes the certificate atomically (tmp + fsync + rename) at mode 0600. Atomic
+// because a torn marker read as valid is the one failure this whole mechanism cannot tolerate — it
+// would certify a part-copy, which is the original defect wearing a new hat.
+func (m *Manager) writeScratchMarker(scratch, snapshotID string, full bool) error {
+	data, err := json.Marshal(scratchMarker{
+		Schema:     scratchMarkerSchema,
+		SnapshotID: snapshotID,
+		Full:       full,
+		FinishedAt: time.Now().UTC().Format(time.RFC3339),
+	})
+	if err != nil {
+		return err
+	}
+	final := filepath.Join(scratch, scratchMarkerName)
+	tmp := final + ".tmp"
+	f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
+	if err != nil {
+		return err
+	}
+	if _, err := f.Write(data); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return err
+	}
+	if err := f.Sync(); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return err
+	}
+	if err := f.Close(); err != nil {
+		os.Remove(tmp)
+		return err
+	}
+	return os.Rename(tmp, final)
+}
+
+
 // OffboxRestorePrepareFull resolves the latest snapshot's restore-size and verifies scratch headroom
 // for a FULL restore WITHOUT starting it (the two-step size-first gate). Returns the human size on
 // success, or a Hungarian error to flash on refusal (size unknown / no headroom — fail-closed).
@@ -293,7 +400,7 @@ func (m *Manager) OffboxRestorePrepareFull(ctx context.Context, stack string) (s
 	}
 	size, serr := m.offboxSnapshotSize(ctx, id)
 	if serr != nil {
-		return "", fmt.Errorf("A mentés mérete nem állapítható meg — a teljes visszaállítás biztonsági okból nem indítható.")
+		return "", fmt.Errorf(offsiteSizeUnknownMsg)
 	}
 	_, nsRoot, derr := m.offboxRestoreScratchDir(stack)
 	if derr != nil {
@@ -301,13 +408,37 @@ func (m *Manager) OffboxRestorePrepareFull(ctx context.Context, stack string) (s
 	}
 	need := size + size/10
 	if free := m.offboxFree()(nsRoot); free < need {
-		return "", fmt.Errorf("Nincs elég szabad hely a visszaállításhoz (%s szükséges, %s szabad).", humanizeBytes(need), humanizeBytes(free))
+		return "", fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(need), humanizeBytes(free))
 	}
 	return humanizeBytes(size), nil
 }
 
-// OffboxFullScratchReady reports whether a (non-empty) full-restore scratch exists for stack — the gate
-// for showing the place-to-live action. PlaceOffsiteRestore re-validates per-path completeness.
+// R-357 customer-facing refusal strings, shared by every headroom gate on the off-site restore
+// surface. Named constants because a test asserts them verbatim and because the destructive gate added
+// in v0.226.0 MUST read identically to the two non-destructive ones that predate it — a customer who
+// meets this refusal on one path and a differently-worded one on another has to work out whether they
+// are the same problem.
+const (
+	offsiteNoSpaceMsgFmt  = "Nincs elég szabad hely a visszaállításhoz (%s szükséges, %s szabad)."
+	offsiteSizeUnknownMsg = "A mentés mérete nem állapítható meg — a teljes visszaállítás biztonsági okból nem indítható."
+)
+
+// OffboxFullScratchReady reports whether a COMPLETED FULL restore scratch exists for stack — the gate
+// for the place-to-live and reconstitute actions.
+//
+// R-358 — WHAT THIS USED TO ANSWER, AND WHY IT WAS THE WRONG QUESTION. It used to be "the directory
+// exists and is non-empty", and its doc comment reassured the reader that
+// `PlaceOffsiteRestore re-validates per-path completeness`. That sentence is what made the weak gate
+// look adequate, and it is not true in the way it reads: PlaceOffsiteRestore stats the top-level
+// PLACEMENTS, not the files inside them, so a placement directory that exists but was only half
+// downloaded passes it. A restic run that died part-way leaves a non-empty directory, so the product
+// offered „Teljes visszaállítás indítása" over a part-copy and reported success on it (demo-hp,
+// 2026-08-21).
+//
+// It now asks the only question that distinguishes them: did the run that wrote this scratch FINISH,
+// and was it the full one. Every other answer — no marker, unreadable marker, wrong schema, full=false
+// — is FALSE, and says at WARN which one it was. **Fail closed: an unreadable marker is not a
+// completion certificate.**
 func (m *Manager) OffboxFullScratchReady(stack string) bool {
 	if !isSafeStackName(stack) {
 		return false
@@ -319,8 +450,27 @@ func (m *Manager) OffboxFullScratchReady(stack string) bool {
 	if fi, sErr := os.Stat(scratch); sErr != nil || !fi.IsDir() {
 		return false
 	}
-	entries, _ := os.ReadDir(scratch)
-	return len(entries) > 0
+	data, rErr := os.ReadFile(filepath.Join(scratch, scratchMarkerName))
+	if rErr != nil {
+		if !os.IsNotExist(rErr) {
+			m.logger.Printf("[WARN] [offbox] %s: scratch completion marker unreadable (%v) — treating the copy as INCOMPLETE", stack, rErr)
+		}
+		return false
+	}
+	var mk scratchMarker
+	if uErr := json.Unmarshal(data, &mk); uErr != nil {
+		m.logger.Printf("[WARN] [offbox] %s: scratch completion marker does not parse (%v) — treating the copy as INCOMPLETE", stack, uErr)
+		return false
+	}
+	if mk.Schema != scratchMarkerSchema {
+		m.logger.Printf("[WARN] [offbox] %s: scratch completion marker has schema %d, expected %d — treating the copy as INCOMPLETE", stack, mk.Schema, scratchMarkerSchema)
+		return false
+	}
+	if !mk.Full {
+		m.logger.Printf("[INFO] [offbox] %s: scratch holds a UNIT-ONLY restore (snapshot %s) — not a full copy, so place-to-live stays closed", stack, mk.SnapshotID)
+		return false
+	}
+	return true
 }
 
 // placement is one source→dest pair for place-to-live: src is the reconstructed absolute path under the
@@ -432,7 +582,7 @@ func (m *Manager) PlaceOffsiteRestore(ctx context.Context, stack string) error {
 	// F-3a-1b: headroom gate — a missing-only merge copies at most the scratch size; refuse before any
 	// copy if the live drive lacks that (conservative — scratch and live often share a drive).
 	if free, need := m.offboxFree()(liveNs), m.offboxSize()(scratch); free < need {
-		return fmt.Errorf("Nincs elég szabad hely a visszaállításhoz (%s szükséges, %s szabad).", humanizeBytes(need), humanizeBytes(free))
+		return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(need), humanizeBytes(free))
 	}
 	placements, err := mapOffsiteRestorePaths(paths, stack, scratch, liveNs)
 	if err != nil {