v0.231.0 - the box proves its own off-site copy still holds something (R-87)
gates / gates (push) Successful in 11s
gates / gates (push) Successful in 11s
R-87 re-scoped by its own spike and built as Option C. MinAgent 0.129.0 unchanged. THE QUESTION NOTHING ASKED. The weekly check proves the stored bytes are the bytes we stored; it cannot tell us we stored the WRONG thing. A hollow recovery unit backs up cleanly, checks cleanly at 100 percent depth, restores cleanly and gives the customer nothing back - measured on demo-hp 2026-08-31, 120082104 B to 7036 B in one nightly run recorded as a success (R-403). No tier and no cadence asked it. Now offsite-proof does, nightly, on one app. IT DOES NOT prove a restore puts data back into a running app. That stays drill work and 07 section 8 matrix row 4 is NOT moved. THE ACCEPTANCE RULE HAS TWO PARTS AND THE OBVIOUS ONE IS A TRAP. "Check the unit against its own packing list" PASSES a hollow unit, because a hollow unit declares nothing. So: (1) everything declared is present, AND (2) the manifest declares what the app is supposed to have. Part 2 is the whole value. RED-PROOFED: the naive rule makes the hollow-unit test read verdict "pass". THE EXPECTATION COMES FROM INSIDE THE UNIT, never the live box - the snapshot may predate the app's shape, and GetDockerVolumes describes the running app. Database half is DBServiceNames, the same discriminator RestoreFromRecoveryUnit uses. Volume half is ParseComposeNamedVolumes as an EXISTENCE check, not a name match: tars are <project>_<volume>.tar and ResolveDockerVolumeNames derives the project from the compose file's parent dir, which inside a unit is the literal string "compose". Measured on all eight real units on demo-hp the counts match exactly and the naming held every time - but "held on eight" is not "derivable" (R-355). Half a rule that is true beats a whole rule that is invented. THREE OUTCOMES: pass, fail (readable and empty), cannot judge. An app that legitimately has neither a database nor volumes PASSES. RED-PROOFED: alarming on any empty unit makes that test read verdict "fail". IT NEVER WRITES TO THE REPOSITORY and that is asserted on the ARGV as a non-effect: --no-lock, no unlockStale, and m.runner() rather than resticStep so the unlock --remove-all escalation is unreachable. RED-PROOFED: routing it the customer path's way makes the test fail on "unlock" appearing in the argv. IT TAKES acquireRunning ITSELF and skips rather than waits, because RestoreOffboxScratch does not take it (R-408) while offbox_integrity.go states that invariant as universal. DUE-NESS IS PER SNAPSHOT (R-86's model), never per clock. RED-PROOFED: recording a timestamp fails the stored-value test AND breaks the rotation - night 2 re-picks night 1's app. ITS SCRATCH IS A SEPARATE ROOT (backups/offsite-proof) and that is a safety decision, not tidiness: the job deletes its copy on every path, and sharing backups/offsite-restore/<app> would mean a nightly background job deleting the verification copy a CUSTOMER is looking at. It is also invisible to placement, so a proof copy can never be pushed into a live app. SHARED RATHER THAN FORKED: offboxScratchDirIn parameterises the scratch resolver on its ROOT builder, and unitOnlyHeadroom extracts the free-space gate, so the customer path and the proof refuse at the same floor with the same Hungarian sentence. RestoreOffboxScratch's behaviour is unchanged. NEW EVENT offsite_proof_empty, severity error, operator-only - deliberately NOT backup_integrity_failed, whose hub template says the store is DAMAGED. Here the store is sound and the content is absent: different cause, different action. The hub half shipped FIRST, in felhom.eu 1aeaa30 (hub v0.110.0, live and verified), because an unallowlisted type is 400'd and vanishes. 33 new tests, all groups green; full suite 1689 tests, 28 packages, rc=0. All 13 controller gates OK. Five red-proofs run and recorded in REPORT.md. A golden carrying 0.231.0 is OWED - the fleet is on 0.230.0. Viktor's call (R-242).
This commit is contained in:
@@ -27,6 +27,20 @@ const (
|
||||
offboxUnitOnlyFreeFloor = int64(2) << 30
|
||||
)
|
||||
|
||||
// unitOnlyHeadroom is THE unit-only free-space gate, shared by the customer's scratch restore and the
|
||||
// R-87 nightly proof so the two can never disagree about how much room a unit restore needs or about
|
||||
// what the customer is told when there is not enough (REUSE.md's `offsiteNoSpaceMsgFmt` rule).
|
||||
//
|
||||
// FAIL-CLOSED ON AN UNMEASURABLE PROBE. `offboxFree` returns 0 when it cannot read the filesystem,
|
||||
// and `0 < floor` is true, so an unreadable drive REFUSES rather than sailing through — the inverse
|
||||
// of the R-357 shape where `free < need` with `need == 0` made a gate inert.
|
||||
func unitOnlyHeadroom(free int64) error {
|
||||
if free < offboxUnitOnlyFreeFloor {
|
||||
return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(offboxUnitOnlyFreeFloor), humanizeBytes(free))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetOffboxFreeFn overrides the restore free-space probe (tests; the Windows go-test host has no df).
|
||||
func (m *Manager) SetOffboxFreeFn(fn func(path string) int64) { m.offboxFreeFn = fn }
|
||||
|
||||
@@ -163,8 +177,29 @@ func (m *Manager) offboxSnapshotSize(ctx context.Context, id string) (int64, err
|
||||
func (m *Manager) offboxRestoreScratchDir(stack string) (scratch, nsRoot string, err error) {
|
||||
// offsiteRestoreRootFor is THE place `backups/offsite-restore` is spelled (offbox_verify_copies.go)
|
||||
// — the listing/delete surface must resolve byte-identical paths to the ones written here.
|
||||
return m.offboxScratchDirIn(stack, m.offsiteRestoreRootFor)
|
||||
}
|
||||
|
||||
// offboxProofScratchDir is the R-87 nightly proof's scratch, resolved by the SAME drive-preference
|
||||
// rules and into a DIFFERENT root (`backups/offsite-proof`).
|
||||
//
|
||||
// THE SEPARATE ROOT IS NOT TIDINESS, IT PREVENTS A DELETE. The proof removes its scratch on every
|
||||
// path, including failure. Sharing `backups/offsite-restore/<app>` would mean a nightly background
|
||||
// job deleting the verification copy a CUSTOMER made and is looking at — a poorer actor destroying a
|
||||
// richer one, which is R-403's shape wearing different clothes. A separate root also keeps the proof
|
||||
// copy invisible to `DeleteOffsiteRestoreCopy`, the copy listing and `OffboxFullScratchReady`, so it
|
||||
// can never be offered for placement into a live app.
|
||||
func (m *Manager) offboxProofScratchDir(stack string) (scratch, nsRoot string, err error) {
|
||||
return m.offboxScratchDirIn(stack, m.offsiteProofRootFor)
|
||||
}
|
||||
|
||||
// offboxScratchDirIn holds the drive-preference rules once. `rootFor` chooses WHICH root under the
|
||||
// namespace the scratch lands in; everything else — the network-storage refusal, the ordering, the
|
||||
// R-252 wording — is shared, so the proof path can never drift from the customer path on the parts
|
||||
// that must not differ.
|
||||
func (m *Manager) offboxScratchDirIn(stack string, rootFor func(string) string) (scratch, nsRoot string, err error) {
|
||||
scratchFor := func(root string) (string, string) {
|
||||
return filepath.Join(m.offsiteRestoreRootFor(root), stack), m.namespaceRoot(root)
|
||||
return filepath.Join(rootFor(root), stack), m.namespaceRoot(root)
|
||||
}
|
||||
isNet := func(path string) bool { return m.settings != nil && m.settings.IsNetworkStoragePath(path) }
|
||||
// (1) the app's own drive — preferred, but ONLY if it is not NETWORK storage (F-3afix-1). restic
|
||||
@@ -262,8 +297,8 @@ func (m *Manager) RestoreOffboxScratch(ctx context.Context, stack string, full b
|
||||
if free < need {
|
||||
return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(need), humanizeBytes(free))
|
||||
}
|
||||
} else if free < offboxUnitOnlyFreeFloor {
|
||||
return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(offboxUnitOnlyFreeFloor), humanizeBytes(free))
|
||||
} else if herr := unitOnlyHeadroom(free); herr != nil {
|
||||
return herr
|
||||
}
|
||||
// F-A1 hygiene: drop the legacy rootfs scratch (DataDir/offbox-restore/<app>) best-effort.
|
||||
legacy := filepath.Join(m.cfg.Paths.DataDir, "offbox-restore", stack)
|
||||
@@ -383,7 +418,6 @@ func (m *Manager) writeScratchMarker(scratch, snapshotID string, full bool) erro
|
||||
return os.Rename(tmp, final)
|
||||
}
|
||||
|
||||
|
||||
// OffboxRestorePrepareFull resolves the latest snapshot's restore-size and verifies scratch headroom
|
||||
// for a FULL restore WITHOUT starting it (the two-step size-first gate). Returns the human size on
|
||||
// success, or a Hungarian error to flash on refusal (size unknown / no headroom — fail-closed).
|
||||
|
||||
Reference in New Issue
Block a user