R-359 + R-397: the off-site store gets checked, and the advertised check becomes real
gates / gates (push) Successful in 12s
gates / gates (push) Successful in 12s
Nothing ever verified that the off-site copies are still readable. The whole-guest tier has verify jobs; the tier holding the customer's documents and photos had none -- the complete set of restic verbs this controller used contained no `check`. We would have found out at restore time, with a customer waiting. On 2026-08-21 a deliberately damaged pack was caught at once by plain `restic check`; we had never run it. R-397: NotifyIntegrityOK/NotifyIntegrityFailed existed with no caller, the hub allowlists both event types and carries the Hungarian text for both, the settings checkbox exists, and the debug button posts to /api/debug/backup/ integrity. Everything was built except the part that runs. SIXTH instance of that shape in this project. THE HAZARD SHAPES THE WHOLE DESIGN. resticStep self-heals a crash lock by running `unlock --remove-all` and retrying, and its own comment records why that is safe: every caller holds the in-process single-flight mutex, so any lock it meets is stale. A check that did not take that flag could meet a LIVE prune's lock from this same box, remove it, and retry over the top of it. So the check TAKES THE FLAG and SKIPS rather than waits -- waiting would pin the nightly backup behind it, and a skip costs nothing because due-ness makes tomorrow try again. TestR359_SkipsWhenRunningFlagHeld asserts the NON-EFFECTS: restic never invoked, `unlock` never in any argv. Its red-proof prints the real thing -- restic running `check` while the flag was held. DUE-NESS, NOT A WEEKDAY. Daily job, weekly behaviour: "is the last successful check older than 7 days?" not "is it Sunday?". R-341 is exactly the other shape, a dated check quietly missed and never caught up. No Weekly primitive added. THREE OUTCOMES, NOT TWO. Skipped, Unreachable and failed are different facts. "I could not look" is not "I looked and it is broken" -- R-339 already owns reachability, and a second alarm for the same fact trains the operator to discount the one alarm that means the backups are damaged. A timeout is unreachable, never damage. A failure advances due-ness (a broken store must not be re-checked nightly); a skip and an unreachable store do not. Success is severity `info`, which severityNotifies DROPS -- it mails NOBODY, by design. A weekly success e-mail is how people stop reading their alerts. The customer gets a SENTENCE; restic's words go to the log, truncated (R-379: 615 bytes of raw database text reached a customer once). read-data-subset ships OFF and a malformed value is refused at read time rather than handed to restic, where one typo would fail the whole check. Published on OffboxReportStatus, NOT on report.BackupReport's IntegrityOK -- those were retired by R-331 YESTERDAY and TestBackupReport_DeadFieldsStayZero still passes unmodified. Also: the monitoring page stopped promising a Sunday job that never existed, and the debug button got its dispatch case. PART 0 WAS NOT BUILT, AND R-398 WAS MY OWN MISTAKE. The seam it asked for already exists: offboxRunner/SetOffboxRunner/m.runner() has been injectable since the off-site tier shipped, and other tests drive restic-backed paths through it. A resticStepFn seam would have been WORSE here -- it would replace the `unlock --remove-all` escalation and hide it from the assertions that must see it. R-358's AST ordering test is converted to a real execution test instead, which immediately surfaced something the AST walk could not: unlockStale legitimately runs before the restore. Four red-proofs, each printing the pre-fix behaviour. Green gate: 28 packages, rc 0. All 12 controller gates OK.
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// ── THE CONTROL — R-359's single most important property ─────────────────────────────────────────
|
||||
//
|
||||
// `resticStep` self-heals a crash lock: on "repository is already locked" it runs
|
||||
// **`unlock --remove-all`** and retries. Its own doc comment states why that is safe — *"the in-process
|
||||
// single-flight mutex (held by every caller of this method) proves no sibling operation is live"*.
|
||||
//
|
||||
// **An integrity check that did not take that flag would break the invariant.** It could meet the lock
|
||||
// of a `forget --prune` running from this same box, remove it, and retry over the top of a live prune —
|
||||
// on the tier holding the customer's documents and photos.
|
||||
//
|
||||
// So these assert NON-EFFECTS, which is the only way to test "it did not do the dangerous thing":
|
||||
// restic was never invoked at all, and `unlock` never appeared in any argument list. A test that only
|
||||
// checked `Skipped == true` would pass against an implementation that skipped AFTER running the check.
|
||||
|
||||
func TestR359_SkipsWhenRunningFlagHeld(t *testing.T) {
|
||||
m, cap := newIntegrityManager(t, okRepo(nil))
|
||||
|
||||
// A sibling operation is live — exactly the state a nightly off-site backup produces.
|
||||
if err := m.AcquireRunningForTest(); err != nil {
|
||||
t.Fatalf("fixture: %v", err)
|
||||
}
|
||||
defer m.ReleaseRunningForTest()
|
||||
|
||||
res := m.CheckOffboxIntegrity(context.Background())
|
||||
|
||||
// THE TWO NON-EFFECTS. These are the assertions that fail against a missing guard; the verdict
|
||||
// fields below would not.
|
||||
if len(cap.argvs) != 0 {
|
||||
t.Fatalf("RESTIC WAS INVOKED while another operation held the single-writer flag: %v — this is "+
|
||||
"the hazard: the check can meet a live prune's lock, and resticStep escalates to "+
|
||||
"`unlock --remove-all` and retries over the top of it", cap.argvs)
|
||||
}
|
||||
if cap.sawVerb("unlock") {
|
||||
t.Fatal("`unlock` was invoked beside a live sibling operation — the exact act that must never happen")
|
||||
}
|
||||
|
||||
if !res.Skipped {
|
||||
t.Fatalf("the check did not report a skip: %+v", res)
|
||||
}
|
||||
if res.OK {
|
||||
t.Fatal("a skip was reported as a passing check — nothing was checked, and recording it as a " +
|
||||
"pass would let a store go unverified while the record says otherwise")
|
||||
}
|
||||
// Due-ness must NOT advance: tomorrow has to try again.
|
||||
if tgt := m.settings.GetOffboxTarget(); tgt != nil && tgt.LastIntegrityCheck != "" {
|
||||
t.Fatalf("a SKIPPED check advanced due-ness (%q) — the store would then wait a full period "+
|
||||
"before anything looked at it again, which is R-341's failure with extra steps", tgt.LastIntegrityCheck)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR359_ReleasesTheFlagOnEveryPath(t *testing.T) {
|
||||
// A check that leaks the flag stops every backup on the box until restart. Each outcome is walked.
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
reply func(args []string) ([]byte, error)
|
||||
}{
|
||||
{"ok", okRepo(nil)},
|
||||
{"damaged", okRepo(func([]string) ([]byte, error) { return []byte("repository contains errors"), errFake })},
|
||||
{"unreachable", func([]string) ([]byte, error) { return []byte("connection refused"), errFake }},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m, _ := newIntegrityManager(t, tc.reply)
|
||||
m.CheckOffboxIntegrity(context.Background())
|
||||
// If the flag leaked, this acquire fails.
|
||||
if err := m.AcquireRunningForTest(); err != nil {
|
||||
t.Fatalf("the single-writer flag was NOT released after a %s outcome (%v) — every "+
|
||||
"backup and restore on this box would refuse until it restarts", tc.name, err)
|
||||
}
|
||||
m.ReleaseRunningForTest()
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestR359_DoesNotBlockAConcurrentBackup(t *testing.T) {
|
||||
// The complement: a check that skipped must leave the flag free for the operation it yielded to.
|
||||
m, _ := newIntegrityManager(t, okRepo(nil))
|
||||
if err := m.AcquireRunningForTest(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.CheckOffboxIntegrity(context.Background()) // skips
|
||||
m.ReleaseRunningForTest() // the sibling finishes
|
||||
|
||||
if err := m.AcquireRunningForTest(); err != nil {
|
||||
t.Fatalf("after a skip the flag could not be acquired: %v — the check held something it "+
|
||||
"never took", err)
|
||||
}
|
||||
m.ReleaseRunningForTest()
|
||||
}
|
||||
|
||||
func TestR359_NeverWritesToTheRepository(t *testing.T) {
|
||||
// `check` is a read verb. This pins that the check's argv carries no verb that could modify the
|
||||
// store — the tier holds real customer data and the box's own credential can already delete from
|
||||
// it (R-95, open and ranked).
|
||||
m, cap := newIntegrityManager(t, okRepo(nil))
|
||||
m.CheckOffboxIntegrity(context.Background())
|
||||
|
||||
for _, verb := range []string{"forget", "prune", "backup", "init", "unlock", "restore"} {
|
||||
if cap.sawVerb(verb) {
|
||||
t.Fatalf("the integrity check invoked `%s` — it must only ever READ (argv=%v)", verb, cap.argvs)
|
||||
}
|
||||
}
|
||||
argv := strings.Join(cap.checkArgv(), " ")
|
||||
if !strings.Contains(argv, "check") {
|
||||
t.Fatalf("no check verb in %q", argv)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user