v0.231.0 - the box proves its own off-site copy still holds something (R-87)
gates / gates (push) Successful in 11s

R-87 re-scoped by its own spike and built as Option C. MinAgent 0.129.0 unchanged.

THE QUESTION NOTHING ASKED. The weekly check proves the stored bytes are the bytes we
stored; it cannot tell us we stored the WRONG thing. A hollow recovery unit backs up
cleanly, checks cleanly at 100 percent depth, restores cleanly and gives the customer
nothing back - measured on demo-hp 2026-08-31, 120082104 B to 7036 B in one nightly run
recorded as a success (R-403). No tier and no cadence asked it. Now offsite-proof does,
nightly, on one app.

IT DOES NOT prove a restore puts data back into a running app. That stays drill work and
07 section 8 matrix row 4 is NOT moved.

THE ACCEPTANCE RULE HAS TWO PARTS AND THE OBVIOUS ONE IS A TRAP. "Check the unit against
its own packing list" PASSES a hollow unit, because a hollow unit declares nothing. So:
(1) everything declared is present, AND (2) the manifest declares what the app is supposed
to have. Part 2 is the whole value. RED-PROOFED: the naive rule makes the hollow-unit test
read verdict "pass".

THE EXPECTATION COMES FROM INSIDE THE UNIT, never the live box - the snapshot may predate
the app's shape, and GetDockerVolumes describes the running app. Database half is
DBServiceNames, the same discriminator RestoreFromRecoveryUnit uses. Volume half is
ParseComposeNamedVolumes as an EXISTENCE check, not a name match: tars are
<project>_<volume>.tar and ResolveDockerVolumeNames derives the project from the compose
file's parent dir, which inside a unit is the literal string "compose". Measured on all
eight real units on demo-hp the counts match exactly and the naming held every time - but
"held on eight" is not "derivable" (R-355). Half a rule that is true beats a whole rule
that is invented.

THREE OUTCOMES: pass, fail (readable and empty), cannot judge. An app that legitimately
has neither a database nor volumes PASSES. RED-PROOFED: alarming on any empty unit makes
that test read verdict "fail".

IT NEVER WRITES TO THE REPOSITORY and that is asserted on the ARGV as a non-effect:
--no-lock, no unlockStale, and m.runner() rather than resticStep so the unlock --remove-all
escalation is unreachable. RED-PROOFED: routing it the customer path's way makes the test
fail on "unlock" appearing in the argv.

IT TAKES acquireRunning ITSELF and skips rather than waits, because RestoreOffboxScratch
does not take it (R-408) while offbox_integrity.go states that invariant as universal.

DUE-NESS IS PER SNAPSHOT (R-86's model), never per clock. RED-PROOFED: recording a
timestamp fails the stored-value test AND breaks the rotation - night 2 re-picks night 1's
app.

ITS SCRATCH IS A SEPARATE ROOT (backups/offsite-proof) and that is a safety decision, not
tidiness: the job deletes its copy on every path, and sharing backups/offsite-restore/<app>
would mean a nightly background job deleting the verification copy a CUSTOMER is looking
at. It is also invisible to placement, so a proof copy can never be pushed into a live app.

SHARED RATHER THAN FORKED: offboxScratchDirIn parameterises the scratch resolver on its
ROOT builder, and unitOnlyHeadroom extracts the free-space gate, so the customer path and
the proof refuse at the same floor with the same Hungarian sentence. RestoreOffboxScratch's
behaviour is unchanged.

NEW EVENT offsite_proof_empty, severity error, operator-only - deliberately NOT
backup_integrity_failed, whose hub template says the store is DAMAGED. Here the store is
sound and the content is absent: different cause, different action. The hub half shipped
FIRST, in felhom.eu 1aeaa30 (hub v0.110.0, live and verified), because an unallowlisted
type is 400'd and vanishes.

33 new tests, all groups green; full suite 1689 tests, 28 packages, rc=0. All 13 controller
gates OK. Five red-proofs run and recorded in REPORT.md.

A golden carrying 0.231.0 is OWED - the fleet is on 0.230.0. Viktor's call (R-242).
This commit is contained in:
2026-08-31 20:55:34 +02:00
parent 2d802d75e8
commit e43b5ec07d
16 changed files with 1973 additions and 33 deletions
+38 -4
View File
@@ -27,6 +27,20 @@ const (
offboxUnitOnlyFreeFloor = int64(2) << 30
)
// unitOnlyHeadroom is THE unit-only free-space gate, shared by the customer's scratch restore and the
// R-87 nightly proof so the two can never disagree about how much room a unit restore needs or about
// what the customer is told when there is not enough (REUSE.md's `offsiteNoSpaceMsgFmt` rule).
//
// FAIL-CLOSED ON AN UNMEASURABLE PROBE. `offboxFree` returns 0 when it cannot read the filesystem,
// and `0 < floor` is true, so an unreadable drive REFUSES rather than sailing through — the inverse
// of the R-357 shape where `free < need` with `need == 0` made a gate inert.
func unitOnlyHeadroom(free int64) error {
if free < offboxUnitOnlyFreeFloor {
return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(offboxUnitOnlyFreeFloor), humanizeBytes(free))
}
return nil
}
// SetOffboxFreeFn overrides the restore free-space probe (tests; the Windows go-test host has no df).
func (m *Manager) SetOffboxFreeFn(fn func(path string) int64) { m.offboxFreeFn = fn }
@@ -163,8 +177,29 @@ func (m *Manager) offboxSnapshotSize(ctx context.Context, id string) (int64, err
func (m *Manager) offboxRestoreScratchDir(stack string) (scratch, nsRoot string, err error) {
// offsiteRestoreRootFor is THE place `backups/offsite-restore` is spelled (offbox_verify_copies.go)
// — the listing/delete surface must resolve byte-identical paths to the ones written here.
return m.offboxScratchDirIn(stack, m.offsiteRestoreRootFor)
}
// offboxProofScratchDir is the R-87 nightly proof's scratch, resolved by the SAME drive-preference
// rules and into a DIFFERENT root (`backups/offsite-proof`).
//
// THE SEPARATE ROOT IS NOT TIDINESS, IT PREVENTS A DELETE. The proof removes its scratch on every
// path, including failure. Sharing `backups/offsite-restore/<app>` would mean a nightly background
// job deleting the verification copy a CUSTOMER made and is looking at — a poorer actor destroying a
// richer one, which is R-403's shape wearing different clothes. A separate root also keeps the proof
// copy invisible to `DeleteOffsiteRestoreCopy`, the copy listing and `OffboxFullScratchReady`, so it
// can never be offered for placement into a live app.
func (m *Manager) offboxProofScratchDir(stack string) (scratch, nsRoot string, err error) {
return m.offboxScratchDirIn(stack, m.offsiteProofRootFor)
}
// offboxScratchDirIn holds the drive-preference rules once. `rootFor` chooses WHICH root under the
// namespace the scratch lands in; everything else — the network-storage refusal, the ordering, the
// R-252 wording — is shared, so the proof path can never drift from the customer path on the parts
// that must not differ.
func (m *Manager) offboxScratchDirIn(stack string, rootFor func(string) string) (scratch, nsRoot string, err error) {
scratchFor := func(root string) (string, string) {
return filepath.Join(m.offsiteRestoreRootFor(root), stack), m.namespaceRoot(root)
return filepath.Join(rootFor(root), stack), m.namespaceRoot(root)
}
isNet := func(path string) bool { return m.settings != nil && m.settings.IsNetworkStoragePath(path) }
// (1) the app's own drive — preferred, but ONLY if it is not NETWORK storage (F-3afix-1). restic
@@ -262,8 +297,8 @@ func (m *Manager) RestoreOffboxScratch(ctx context.Context, stack string, full b
if free < need {
return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(need), humanizeBytes(free))
}
} else if free < offboxUnitOnlyFreeFloor {
return fmt.Errorf(offsiteNoSpaceMsgFmt, humanizeBytes(offboxUnitOnlyFreeFloor), humanizeBytes(free))
} else if herr := unitOnlyHeadroom(free); herr != nil {
return herr
}
// F-A1 hygiene: drop the legacy rootfs scratch (DataDir/offbox-restore/<app>) best-effort.
legacy := filepath.Join(m.cfg.Paths.DataDir, "offbox-restore", stack)
@@ -383,7 +418,6 @@ func (m *Manager) writeScratchMarker(scratch, snapshotID string, full bool) erro
return os.Rename(tmp, final)
}
// OffboxRestorePrepareFull resolves the latest snapshot's restore-size and verifies scratch headroom
// for a FULL restore WITHOUT starting it (the two-step size-first gate). Returns the human size on
// success, or a Hungarian error to flash on refusal (size unknown / no headroom — fail-closed).