R-353/R-357/R-358/R-360: the restore tells the truth (v0.226.0)
gates / gates (push) Successful in 11s
gates / gates (push) Successful in 11s
Four defects on the restore surface, all proven on demo-hp during the 2026-08-21 backup-truth drill, all still in shipped code. They share one acceptance idea: a restore surface must state what it actually did, and must refuse what it cannot do. VERSION NOTE. The task specifying this targeted v0.224.0 against baselinef8c9390. Both were consumed earlier the same day by R-330 (0.224.0) and R-331 (0.225.0). Drift re-confirmed against live Gitea before the first edit, operator authorised proceeding, every symbol the spec named re-verified present at the real baselinee5eee50. R-353 -- a restore that gave back nothing still said it worked. RestoreFromRecoveryUnit returned only error, so the surface printed "<app> visszaallitva (<snapshot>)." -- equally true of a run that returned an entire dataset and one that returned nothing. The count already existed and was discarded one line deep: restoreDockerVolumesFrom always returned it, the wrapper threw it away. Now (UnitRestoreResult, error), carrying replayed counts AND what the manifest LISTED, because zero-replayed has two causes that are opposite news. Three cases, three sentences, and EVERY one is a claim about the BACKUP, never about the app -- this path has no SafetyDump discriminator, and 07-backup-architecture 6.3 records that an absent dump says nothing about the app (R-361 destroyed canonical .sql files for four months). R-357 -- the destructive restore had no free-space gate. offbox_reconstitute.go contained ZERO references to offboxFree; all three existing gates guard non-destructive paths. The gate now sits before mapOffsiteRestorePaths, writeSafetyDump and StopStack, so a refusal costs nothing. Position IS the fix, which is why the test asserts StopStack was never called. No headroom multiplier (matches PlaceOffsiteRestore; the x1.1 elsewhere predicts a download). Fail closed on either probe <= 0 -- otherwise `free < need` with need==0 is FALSE and an unmeasurable scratch sails through: a gate present and inert. R-358 -- a failed download was offered as a good one. The gate answered "the directory exists and is non-empty", which is exactly what a part-way restic run leaves. Now a completion marker written 0600 atomically AFTER restic returns nil, with any stale one cleared BEFORE it starts; both orders pinned by an AST test because resticStep is not a seam. Both handlers refuse server-side: the wizard flags control a button, and a hidden button is not a guard. SCENARIO F ANSWERED, and worse than the question assumed: a unit-only scratch IS reachable through the real flow, by the most ordinary route. "Ellenorzo visszaallitas" (mode=unit, advertised non-destructive) writes the SAME directory -- offboxRestoreScratchDir ignores `full` and --include limits what restic extracts, never where -- so a customer who ran the SAFE restore was then offered the destructive one over a unit-only copy. Filed R-396; the marker closes it. R-360 -- the delete refused only while a BACKUP ran. IsRunning() is FALSE for the whole of a verification restore; the five sibling handlers all use restoreOpBlocked(). Its doc comment claimed it already did this, which is why nobody looked -- corrected in place. Red-proofs, each printing the pre-fix behaviour, in CHANGELOG and REPORT. The first R-357 red-proof exposed a hollow test OF MY OWN and it is recorded rather than quietly fixed: the fixture refused earlier at the placement stat pre-pass, so `stops == 0` passed against the pre-fix code. Fixture corrected, assertions reordered so a removed gate reports the outage rather than "no error returned". Green gate clean: 28 packages, rc 0. All 12 controller gates OK.
This commit is contained in:
@@ -547,6 +547,44 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
}
|
||||
liveNs := m.namespaceRoot(hdd)
|
||||
|
||||
// --- R-357: FREE SPACE, BEFORE ANYTHING IS TOUCHED ------------------------------------------
|
||||
//
|
||||
// This file contained ZERO references to offboxFree until now. The three headroom gates that
|
||||
// existed all guarded NON-destructive paths (offbox_restore.go: the download sizer, the prepare
|
||||
// gate, and PlaceOffsiteRestore's missing-only merge). The one path that stops the customer's app
|
||||
// and overwrites their live data had none.
|
||||
//
|
||||
// Measured on demo-hp 2026-08-21: it stopped the app, ran out of disk part-way, left 2 of 5 planted
|
||||
// items in place and restarted the app — a half-restored dataset presented as a completed restore.
|
||||
//
|
||||
// POSITION IS THE WHOLE FIX. This sits before mapOffsiteRestorePaths, before writeSafetyDump and
|
||||
// well before StopStack, so a refusal costs the customer nothing at all — the app never goes down.
|
||||
// A gate after StopStack would turn a refusal into an outage, which is the shape it exists to
|
||||
// prevent. Scenario D asserts the non-effect (StopStack call count 0), not the error string.
|
||||
//
|
||||
// NO HEADROOM MULTIPLIER, deliberately, and stated so the next reader does not "fix" it:
|
||||
// OffboxRestorePrepareFull uses ×1.1 because it is sizing a DOWNLOAD whose final size it is
|
||||
// predicting. This is a local copy of a tree that already exists on disk, so its size is known
|
||||
// exactly — the same reasoning PlaceOffsiteRestore's gate uses, and this matches it.
|
||||
free, need := m.offboxFree()(liveNs), m.offboxSize()(scratch)
|
||||
switch {
|
||||
case need <= 0:
|
||||
// FAIL CLOSED. Without this the comparison below is `free < 0`, which is false, and an
|
||||
// unmeasurable scratch would sail straight through into the destructive phase — the gate
|
||||
// present and inert, which is worse than no gate because it reads as protection.
|
||||
m.logger.Printf("[ERROR] [offbox] %s: REFUSING the destructive restore — the scratch size could not be measured (scratch=%s)", stack, scratch)
|
||||
return res, fmt.Errorf(offsiteSizeUnknownMsg)
|
||||
case free <= 0:
|
||||
// Same direction for the other probe. The customer sentence is shared with the case above
|
||||
// (the operator asked for one wording); the LOG line above and below is what distinguishes
|
||||
// which probe failed.
|
||||
m.logger.Printf("[ERROR] [offbox] %s: REFUSING the destructive restore — free space on the live namespace could not be measured (liveNs=%s)", stack, liveNs)
|
||||
return res, fmt.Errorf(offsiteSizeUnknownMsg)
|
||||
case free < need:
|
||||
m.logger.Printf("[WARN] [offbox] %s: REFUSING the destructive restore — need %d B, free %d B on %s; the app was NOT stopped", stack, need, free, liveNs)
|
||||
return res, fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(need), humanizeBytes(free))
|
||||
}
|
||||
|
||||
placements, err := mapOffsiteRestorePaths(paths, stack, scratch, liveNs)
|
||||
if err != nil {
|
||||
return res, err // whole-placement refusal (no partial writes)
|
||||
|
||||
Reference in New Issue
Block a user