b8af72764d
gates / gates (push) Successful in 11s
Four defects on the restore surface, all proven on demo-hp during the 2026-08-21 backup-truth drill, all still in shipped code. They share one acceptance idea: a restore surface must state what it actually did, and must refuse what it cannot do. VERSION NOTE. The task specifying this targeted v0.224.0 against baselinef8c9390. Both were consumed earlier the same day by R-330 (0.224.0) and R-331 (0.225.0). Drift re-confirmed against live Gitea before the first edit, operator authorised proceeding, every symbol the spec named re-verified present at the real baselinee5eee50. R-353 -- a restore that gave back nothing still said it worked. RestoreFromRecoveryUnit returned only error, so the surface printed "<app> visszaallitva (<snapshot>)." -- equally true of a run that returned an entire dataset and one that returned nothing. The count already existed and was discarded one line deep: restoreDockerVolumesFrom always returned it, the wrapper threw it away. Now (UnitRestoreResult, error), carrying replayed counts AND what the manifest LISTED, because zero-replayed has two causes that are opposite news. Three cases, three sentences, and EVERY one is a claim about the BACKUP, never about the app -- this path has no SafetyDump discriminator, and 07-backup-architecture 6.3 records that an absent dump says nothing about the app (R-361 destroyed canonical .sql files for four months). R-357 -- the destructive restore had no free-space gate. offbox_reconstitute.go contained ZERO references to offboxFree; all three existing gates guard non-destructive paths. The gate now sits before mapOffsiteRestorePaths, writeSafetyDump and StopStack, so a refusal costs nothing. Position IS the fix, which is why the test asserts StopStack was never called. No headroom multiplier (matches PlaceOffsiteRestore; the x1.1 elsewhere predicts a download). Fail closed on either probe <= 0 -- otherwise `free < need` with need==0 is FALSE and an unmeasurable scratch sails through: a gate present and inert. R-358 -- a failed download was offered as a good one. The gate answered "the directory exists and is non-empty", which is exactly what a part-way restic run leaves. Now a completion marker written 0600 atomically AFTER restic returns nil, with any stale one cleared BEFORE it starts; both orders pinned by an AST test because resticStep is not a seam. Both handlers refuse server-side: the wizard flags control a button, and a hidden button is not a guard. SCENARIO F ANSWERED, and worse than the question assumed: a unit-only scratch IS reachable through the real flow, by the most ordinary route. "Ellenorzo visszaallitas" (mode=unit, advertised non-destructive) writes the SAME directory -- offboxRestoreScratchDir ignores `full` and --include limits what restic extracts, never where -- so a customer who ran the SAFE restore was then offered the destructive one over a unit-only copy. Filed R-396; the marker closes it. R-360 -- the delete refused only while a BACKUP ran. IsRunning() is FALSE for the whole of a verification restore; the five sibling handlers all use restoreOpBlocked(). Its doc comment claimed it already did this, which is why nobody looked -- corrected in place. Red-proofs, each printing the pre-fix behaviour, in CHANGELOG and REPORT. The first R-357 red-proof exposed a hollow test OF MY OWN and it is recorded rather than quietly fixed: the fixture refused earlier at the placement stat pre-pass, so `stops == 0` passed against the pre-fix code. Fixture corrected, assertions reordered so a removed gate reports the outage rather than "no error returned". Green gate clean: 28 packages, rc 0. All 12 controller gates OK.
177 lines
8.2 KiB
Go
177 lines
8.2 KiB
Go
package backup
|
|
|
|
import (
|
|
"context"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"sync/atomic"
|
|
"testing"
|
|
|
|
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
|
)
|
|
|
|
// ── R-357 — the ONE restore that deletes and replaces data had no free-space gate ────────────────
|
|
//
|
|
// The two gentler paths both check (offbox_restore.go: the prepare gate and PlaceOffsiteRestore's
|
|
// missing-only merge). `offbox_reconstitute.go` contained ZERO references to offboxFree.
|
|
//
|
|
// Measured on demo-hp 2026-08-21: the destructive restore stopped the app, ran out of disk part-way,
|
|
// left 2 of 5 planted items in place and restarted the app — a half-restored dataset presented as a
|
|
// completed restore.
|
|
//
|
|
// WHAT THESE ASSERT IS THE NON-EFFECT, NOT THE ERROR STRING. The value of the fix is that the app is
|
|
// never stopped: a gate placed after StopStack would turn a refusal into an outage and would still
|
|
// return the right sentence. So `stops == 0` is the assertion that can fail on a wrong-but-plausible
|
|
// implementation, and the message is checked second.
|
|
|
|
// r357Provider counts StopStack so the non-effect is measurable, and reports the app as deployed so
|
|
// the reconstitute reaches the gate rather than refusing earlier for an unrelated reason.
|
|
type r357Provider struct {
|
|
hdd string
|
|
stops int32
|
|
}
|
|
|
|
func (p *r357Provider) GetStackComposePath(string) (string, bool) { return "", false }
|
|
func (p *r357Provider) ListDeployedStacks() []StackSummary {
|
|
return []StackSummary{{Name: "paperless-ngx"}}
|
|
}
|
|
func (p *r357Provider) GetStackHDDMounts(string) []string { return nil }
|
|
func (p *r357Provider) GetStackHDDPath(string) string { return p.hdd }
|
|
func (p *r357Provider) GetImportRoot() string { return "" }
|
|
func (p *r357Provider) GetDockerVolumes(string) []string { return nil }
|
|
func (p *r357Provider) StopStack(string) error { atomic.AddInt32(&p.stops, 1); return nil }
|
|
func (p *r357Provider) StartStack(string) error { return nil }
|
|
func (p *r357Provider) RefreshAndIsRunning(string) bool { return true }
|
|
func (p *r357Provider) GetStackRecoveryInfo(string) (RecoveryInfo, bool) {
|
|
return RecoveryInfo{}, false
|
|
}
|
|
func (p *r357Provider) RecoverStackSecrets(string, []string) map[string]string { return nil }
|
|
func (p *r357Provider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error {
|
|
return nil
|
|
}
|
|
func (p *r357Provider) StartStackServices(string, []string) error { return nil }
|
|
func (p *r357Provider) GetStackClassifiedBinds(string) ([]ClassifiedBind, bool) {
|
|
return nil, false
|
|
}
|
|
|
|
// newR357Manager builds a manager whose reconstitute reaches the headroom gate: a real scratch on
|
|
// disk, a stubbed snapshot lookup (no restic), and the app reported deployed.
|
|
func newR357Manager(t *testing.T) (*Manager, *r357Provider) {
|
|
t.Helper()
|
|
m, sett := newOffboxManager(t)
|
|
drive := t.TempDir()
|
|
if err := sett.AddStoragePath(settings.StoragePath{Path: drive, Label: "drive", Schedulable: true}); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
prov := &r357Provider{hdd: drive}
|
|
m.SetStackProvider(prov)
|
|
|
|
scratch, _, err := m.offboxRestoreScratchDir("paperless-ngx")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
|
|
// THE FIXTURE MUST REACH StopStack WHEN THE GATE IS REMOVED, or `stops == 0` proves nothing.
|
|
// The first draft of this test did not: without the gate the run refused earlier, at the stat
|
|
// pre-pass over the placements, so the assertion passed against the pre-fix code. That is a hollow
|
|
// test, and the red-proof is what exposed it — recorded here because the near-miss is the lesson.
|
|
//
|
|
// So the scratch is populated the way a real completed download leaves it: `mapOffsiteRestorePaths`
|
|
// builds each src as filepath.Join(scratch, <full snapshot path>), so the snapshot's own absolute
|
|
// path is mirrored underneath the scratch.
|
|
const oldNs = "/mnt/old"
|
|
snapPaths := []string{oldNs + "/backups/primary/paperless-ngx", oldNs + "/appdata/paperless-ngx"}
|
|
for _, sp := range snapPaths {
|
|
if err := os.MkdirAll(filepath.Join(scratch, sp), 0o755); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
m.SetOffboxLatestSnapshotFn(func(context.Context, string) (string, []string, error) {
|
|
return "snap-1", snapPaths, nil
|
|
})
|
|
// No Docker in a unit test: the undo copy is seamed out. It runs BEFORE StopStack, so leaving it
|
|
// real would make the fixture fail for a reason that has nothing to do with the gate.
|
|
m.SetSafetyDumpFn(func(context.Context, DiscoveredDB, string) DumpResult { return DumpResult{} })
|
|
return m, prov
|
|
}
|
|
|
|
func TestR357_DestructiveRestoreRefusesWithoutHeadroom(t *testing.T) {
|
|
m, prov := newR357Manager(t)
|
|
m.SetOffboxSizer(func(string) int64 { return 1024 * 1024 }) // the scratch is 1 MB
|
|
m.SetOffboxFreeFn(func(string) int64 { return 300 * 1024 }) // 300 KB free
|
|
|
|
_, err := m.ReconstituteFromOffsite(context.Background(), "paperless-ngx", false)
|
|
|
|
// THE ASSERTION THAT MATTERS, AND IT IS CHECKED FIRST ON PURPOSE. On 2026-08-21 the app went down
|
|
// and came back over a half-written dataset. A gate placed after StopStack would return the right
|
|
// sentence and still take the outage, so the error text cannot be the primary assertion — and if
|
|
// this is checked second, a removed gate reports "no error returned" instead of naming the outage.
|
|
if n := atomic.LoadInt32(&prov.stops); n != 0 {
|
|
t.Fatalf("THE APP WAS STOPPED (%d call(s)) for a restore with 300 KB free for a 1 MB copy — "+
|
|
"the whole point of this gate is that the customer's app never goes down for a restore "+
|
|
"that cannot run (err=%v)", n, err)
|
|
}
|
|
if err == nil {
|
|
t.Fatal("the destructive restore proceeded with 300 KB free for a 1 MB copy")
|
|
}
|
|
for _, want := range []string{"Nincs elég szabad hely", "szükséges", "szabad"} {
|
|
if !strings.Contains(err.Error(), want) {
|
|
t.Errorf("the refusal must name need and free like its two siblings; missing %q in %q", want, err.Error())
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestR357_UnknownSizeFailsClosed(t *testing.T) {
|
|
// The fail-open hole this closes is subtle: with need = 0 the comparison `free < need` is FALSE,
|
|
// so an unmeasurable scratch sailed straight into the destructive phase. A gate that is present
|
|
// and inert is worse than no gate, because it reads as protection.
|
|
m, prov := newR357Manager(t)
|
|
m.SetOffboxSizer(func(string) int64 { return 0 })
|
|
m.SetOffboxFreeFn(func(string) int64 { return 100 << 30 }) // plenty free — irrelevant
|
|
|
|
_, err := m.ReconstituteFromOffsite(context.Background(), "paperless-ngx", false)
|
|
if n := atomic.LoadInt32(&prov.stops); n != 0 {
|
|
t.Fatalf("THE APP WAS STOPPED (%d call(s)) on an unknown size — fail-closed means refuse "+
|
|
"BEFORE the stop, not report an error after it (err=%v)", n, err)
|
|
}
|
|
if err == nil {
|
|
t.Fatal("an unmeasurable scratch was allowed into the destructive restore")
|
|
}
|
|
if !strings.Contains(err.Error(), "nem állapítható meg") {
|
|
t.Errorf("want the size-unknown wording already used by OffboxRestorePrepareFull; got %q", err.Error())
|
|
}
|
|
}
|
|
|
|
func TestR357_UnknownFreeSpaceFailsClosed(t *testing.T) {
|
|
// The mirror hole: free = 0 and need = 0 also compares false. Both probes fail closed.
|
|
m, prov := newR357Manager(t)
|
|
m.SetOffboxSizer(func(string) int64 { return 1024 })
|
|
m.SetOffboxFreeFn(func(string) int64 { return 0 })
|
|
|
|
_, err := m.ReconstituteFromOffsite(context.Background(), "paperless-ngx", false)
|
|
if n := atomic.LoadInt32(&prov.stops); n != 0 {
|
|
t.Fatalf("THE APP WAS STOPPED (%d call(s)) on an unknown free reading (err=%v)", n, err)
|
|
}
|
|
if err == nil {
|
|
t.Fatal("an unmeasurable free-space reading was allowed into the destructive restore")
|
|
}
|
|
}
|
|
|
|
func TestR357_AmpleSpaceIsUnchanged(t *testing.T) {
|
|
// The gate must not become a new way to fail an ordinary restore. With room to spare it does not
|
|
// fire, and the run proceeds past it — which here means it fails LATER, for its own unrelated
|
|
// reasons, never with a headroom sentence.
|
|
m, _ := newR357Manager(t)
|
|
m.SetOffboxSizer(func(string) int64 { return 1024 })
|
|
m.SetOffboxFreeFn(func(string) int64 { return 100 << 30 })
|
|
|
|
_, err := m.ReconstituteFromOffsite(context.Background(), "paperless-ngx", false)
|
|
if err != nil && strings.Contains(err.Error(), "Nincs elég szabad hely") {
|
|
t.Fatalf("the headroom gate fired with 100 GiB free for a 1 KiB copy: %v", err)
|
|
}
|
|
if err != nil && strings.Contains(err.Error(), "nem állapítható meg") {
|
|
t.Fatalf("the size-unknown branch fired on a measurable scratch: %v", err)
|
|
}
|
|
}
|