v0.231.0 - the box proves its own off-site copy still holds something (R-87)
gates / gates (push) Successful in 11s

R-87 re-scoped by its own spike and built as Option C. MinAgent 0.129.0 unchanged.

THE QUESTION NOTHING ASKED. The weekly check proves the stored bytes are the bytes we
stored; it cannot tell us we stored the WRONG thing. A hollow recovery unit backs up
cleanly, checks cleanly at 100 percent depth, restores cleanly and gives the customer
nothing back - measured on demo-hp 2026-08-31, 120082104 B to 7036 B in one nightly run
recorded as a success (R-403). No tier and no cadence asked it. Now offsite-proof does,
nightly, on one app.

IT DOES NOT prove a restore puts data back into a running app. That stays drill work and
07 section 8 matrix row 4 is NOT moved.

THE ACCEPTANCE RULE HAS TWO PARTS AND THE OBVIOUS ONE IS A TRAP. "Check the unit against
its own packing list" PASSES a hollow unit, because a hollow unit declares nothing. So:
(1) everything declared is present, AND (2) the manifest declares what the app is supposed
to have. Part 2 is the whole value. RED-PROOFED: the naive rule makes the hollow-unit test
read verdict "pass".

THE EXPECTATION COMES FROM INSIDE THE UNIT, never the live box - the snapshot may predate
the app's shape, and GetDockerVolumes describes the running app. Database half is
DBServiceNames, the same discriminator RestoreFromRecoveryUnit uses. Volume half is
ParseComposeNamedVolumes as an EXISTENCE check, not a name match: tars are
<project>_<volume>.tar and ResolveDockerVolumeNames derives the project from the compose
file's parent dir, which inside a unit is the literal string "compose". Measured on all
eight real units on demo-hp the counts match exactly and the naming held every time - but
"held on eight" is not "derivable" (R-355). Half a rule that is true beats a whole rule
that is invented.

THREE OUTCOMES: pass, fail (readable and empty), cannot judge. An app that legitimately
has neither a database nor volumes PASSES. RED-PROOFED: alarming on any empty unit makes
that test read verdict "fail".

IT NEVER WRITES TO THE REPOSITORY and that is asserted on the ARGV as a non-effect:
--no-lock, no unlockStale, and m.runner() rather than resticStep so the unlock --remove-all
escalation is unreachable. RED-PROOFED: routing it the customer path's way makes the test
fail on "unlock" appearing in the argv.

IT TAKES acquireRunning ITSELF and skips rather than waits, because RestoreOffboxScratch
does not take it (R-408) while offbox_integrity.go states that invariant as universal.

DUE-NESS IS PER SNAPSHOT (R-86's model), never per clock. RED-PROOFED: recording a
timestamp fails the stored-value test AND breaks the rotation - night 2 re-picks night 1's
app.

ITS SCRATCH IS A SEPARATE ROOT (backups/offsite-proof) and that is a safety decision, not
tidiness: the job deletes its copy on every path, and sharing backups/offsite-restore/<app>
would mean a nightly background job deleting the verification copy a CUSTOMER is looking
at. It is also invisible to placement, so a proof copy can never be pushed into a live app.

SHARED RATHER THAN FORKED: offboxScratchDirIn parameterises the scratch resolver on its
ROOT builder, and unitOnlyHeadroom extracts the free-space gate, so the customer path and
the proof refuse at the same floor with the same Hungarian sentence. RestoreOffboxScratch's
behaviour is unchanged.

NEW EVENT offsite_proof_empty, severity error, operator-only - deliberately NOT
backup_integrity_failed, whose hub template says the store is DAMAGED. Here the store is
sound and the content is absent: different cause, different action. The hub half shipped
FIRST, in felhom.eu 1aeaa30 (hub v0.110.0, live and verified), because an unallowlisted
type is 400'd and vanishes.

33 new tests, all groups green; full suite 1689 tests, 28 packages, rc=0. All 13 controller
gates OK. Five red-proofs run and recorded in REPORT.md.

A golden carrying 0.231.0 is OWED - the fleet is on 0.230.0. Viktor's call (R-242).
This commit is contained in:
2026-08-31 20:55:34 +02:00
parent 2d802d75e8
commit e43b5ec07d
16 changed files with 1973 additions and 33 deletions
@@ -0,0 +1,168 @@
package backup
import (
"go/ast"
"go/parser"
"go/token"
"strings"
"testing"
)
// R-87 Group D — the WIRING.
//
// THE FAILURE SHAPE THIS EXISTS FOR is the project's most-repeated one: **built and never wired.**
// R-397 was the sixth instance — `NotifyIntegrityOK`/`NotifyIntegrityFailed` existed, the hub
// allowlisted both, the Hungarian text existed, the settings checkbox existed, the debug button
// existed, and the CALLER did not. The product advertised a weekly check that never ran.
//
// `ProveOffboxUnit` is exactly that shape again: a method nobody has to call. It compiles unwired,
// every test in this package passes unwired, and the nightly proof would simply never happen.
//
// It walks the AST rather than grepping, and parses with comments DROPPED, so a commented-out
// registration cannot satisfy it — the string is still in the file either way.
const r87MainPath = "../../cmd/controller/main.go"
func parseMainForR87(t *testing.T) *ast.File {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, r87MainPath, nil, 0) // comments dropped on purpose
if err != nil {
t.Fatalf("parse %s: %v — the R-87 wiring is now unasserted", r87MainPath, err)
}
return f
}
// TestR87_JobIsRegisteredInMain — D1. The scheduler entry must exist, name the job, and call the
// runner. All three, because any one of them alone is satisfiable by an inert line.
func TestR87_JobIsRegisteredInMain(t *testing.T) {
f := parseMainForR87(t)
var sawDailyCall, sawJobName, sawRunnerCall bool
var scheduledAt string
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "Daily" || len(call.Args) < 3 {
return true
}
name, ok := call.Args[0].(*ast.BasicLit)
if !ok || strings.Trim(name.Value, `"`) != "offsite-proof" {
return true
}
sawDailyCall, sawJobName = true, true
if at, ok := call.Args[1].(*ast.BasicLit); ok {
scheduledAt = strings.Trim(at.Value, `"`)
}
// The closure must actually call the runner. A `Daily("offsite-proof", ...)` whose body does
// nothing is precisely the R-397 shape one level in.
ast.Inspect(call.Args[2], func(inner ast.Node) bool {
ic, ok := inner.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := ic.Fun.(*ast.Ident); ok && id.Name == "runOffsiteProof" {
sawRunnerCall = true
}
return true
})
return true
})
if !sawDailyCall || !sawJobName {
t.Fatal(`main.go must register sched.Daily("offsite-proof", ...) — without it the whole of R-87 is inert`)
}
if !sawRunnerCall {
t.Fatal("the offsite-proof job is registered but its closure never calls runOffsiteProof — registered and inert is the R-397 shape")
}
if scheduledAt == "" || !strings.Contains(scheduledAt, ":") {
t.Fatalf("the job must be scheduled at an HH:MM time; got %q", scheduledAt)
}
// It must not collide with the sibling off-site jobs' own slots. Those are read from the live box
// and recorded beside the registration; this pins that they stay distinct.
for _, taken := range []string{"04:15", "05:10", "06:00"} {
if scheduledAt == taken {
t.Fatalf("offsite-proof is scheduled at %s, which is another off-site job's slot — they share the single-writer flag and one would skip every night", scheduledAt)
}
}
}
// TestR87_RunnerAlarmsOnlyOnFail — D1's other half, and Scenario C's job-level guarantee.
//
// `runOffsiteProof` must call the notifier ONLY inside a branch testing for the failing verdict.
// Asserted structurally because the alternative — running the job and watching for silence — proves
// nothing when the job is inert for an unrelated reason.
func TestR87_RunnerAlarmsOnlyOnFail(t *testing.T) {
f := parseMainForR87(t)
var fn *ast.FuncDecl
ast.Inspect(f, func(n ast.Node) bool {
d, ok := n.(*ast.FuncDecl)
if ok && d.Name.Name == "runOffsiteProof" {
fn = d
}
return true
})
if fn == nil {
t.Fatal("runOffsiteProof is missing from main.go — the job has no caller")
}
// Every NotifyOffsiteProofEmpty call must sit inside an if-statement whose condition names the
// FAIL verdict. A bare call at function level would alarm on every outcome, including a pass.
var calls, guarded int
var guardText []string
ast.Inspect(fn, func(n ast.Node) bool {
ifs, ok := n.(*ast.IfStmt)
if !ok {
return true
}
cond := exprText(ifs.Cond)
ast.Inspect(ifs.Body, func(inner ast.Node) bool {
if sel, ok := inner.(*ast.SelectorExpr); ok && sel.Sel.Name == "NotifyOffsiteProofEmpty" {
guarded++
guardText = append(guardText, cond)
}
return true
})
return true
})
ast.Inspect(fn, func(n ast.Node) bool {
if sel, ok := n.(*ast.SelectorExpr); ok && sel.Sel.Name == "NotifyOffsiteProofEmpty" {
calls++
}
return true
})
if calls == 0 {
t.Fatal("runOffsiteProof never calls NotifyOffsiteProofEmpty — a failing proof would be silent, which is R-397's shape exactly")
}
if calls != 1 {
t.Fatalf("exactly ONE alarm call, or a failing proof mails twice; found %d", calls)
}
if guarded != 1 {
t.Fatalf("the alarm must be guarded by an if — an unguarded call alarms on a PASS too; guarded=%d", guarded)
}
if !strings.Contains(guardText[0], "UnitProofFail") {
t.Fatalf("the alarm's guard must test for the FAIL verdict, not merely for 'not a pass' — cannot-judge must never alarm; guard is %q", guardText[0])
}
}
// exprText renders an expression back to source-ish text for an assertion message.
func exprText(e ast.Expr) string {
switch v := e.(type) {
case *ast.BinaryExpr:
return exprText(v.X) + " " + v.Op.String() + " " + exprText(v.Y)
case *ast.SelectorExpr:
return exprText(v.X) + "." + v.Sel.Name
case *ast.Ident:
return v.Name
case *ast.CallExpr:
return exprText(v.Fun) + "(...)"
case *ast.BasicLit:
return v.Value
}
return "?"
}