Files
felhom-controller/controller/internal/backup/r87_judgement_test.go
T
admin e43b5ec07d
gates / gates (push) Successful in 11s
v0.231.0 - the box proves its own off-site copy still holds something (R-87)
R-87 re-scoped by its own spike and built as Option C. MinAgent 0.129.0 unchanged.

THE QUESTION NOTHING ASKED. The weekly check proves the stored bytes are the bytes we
stored; it cannot tell us we stored the WRONG thing. A hollow recovery unit backs up
cleanly, checks cleanly at 100 percent depth, restores cleanly and gives the customer
nothing back - measured on demo-hp 2026-08-31, 120082104 B to 7036 B in one nightly run
recorded as a success (R-403). No tier and no cadence asked it. Now offsite-proof does,
nightly, on one app.

IT DOES NOT prove a restore puts data back into a running app. That stays drill work and
07 section 8 matrix row 4 is NOT moved.

THE ACCEPTANCE RULE HAS TWO PARTS AND THE OBVIOUS ONE IS A TRAP. "Check the unit against
its own packing list" PASSES a hollow unit, because a hollow unit declares nothing. So:
(1) everything declared is present, AND (2) the manifest declares what the app is supposed
to have. Part 2 is the whole value. RED-PROOFED: the naive rule makes the hollow-unit test
read verdict "pass".

THE EXPECTATION COMES FROM INSIDE THE UNIT, never the live box - the snapshot may predate
the app's shape, and GetDockerVolumes describes the running app. Database half is
DBServiceNames, the same discriminator RestoreFromRecoveryUnit uses. Volume half is
ParseComposeNamedVolumes as an EXISTENCE check, not a name match: tars are
<project>_<volume>.tar and ResolveDockerVolumeNames derives the project from the compose
file's parent dir, which inside a unit is the literal string "compose". Measured on all
eight real units on demo-hp the counts match exactly and the naming held every time - but
"held on eight" is not "derivable" (R-355). Half a rule that is true beats a whole rule
that is invented.

THREE OUTCOMES: pass, fail (readable and empty), cannot judge. An app that legitimately
has neither a database nor volumes PASSES. RED-PROOFED: alarming on any empty unit makes
that test read verdict "fail".

IT NEVER WRITES TO THE REPOSITORY and that is asserted on the ARGV as a non-effect:
--no-lock, no unlockStale, and m.runner() rather than resticStep so the unlock --remove-all
escalation is unreachable. RED-PROOFED: routing it the customer path's way makes the test
fail on "unlock" appearing in the argv.

IT TAKES acquireRunning ITSELF and skips rather than waits, because RestoreOffboxScratch
does not take it (R-408) while offbox_integrity.go states that invariant as universal.

DUE-NESS IS PER SNAPSHOT (R-86's model), never per clock. RED-PROOFED: recording a
timestamp fails the stored-value test AND breaks the rotation - night 2 re-picks night 1's
app.

ITS SCRATCH IS A SEPARATE ROOT (backups/offsite-proof) and that is a safety decision, not
tidiness: the job deletes its copy on every path, and sharing backups/offsite-restore/<app>
would mean a nightly background job deleting the verification copy a CUSTOMER is looking
at. It is also invisible to placement, so a proof copy can never be pushed into a live app.

SHARED RATHER THAN FORKED: offboxScratchDirIn parameterises the scratch resolver on its
ROOT builder, and unitOnlyHeadroom extracts the free-space gate, so the customer path and
the proof refuse at the same floor with the same Hungarian sentence. RestoreOffboxScratch's
behaviour is unchanged.

NEW EVENT offsite_proof_empty, severity error, operator-only - deliberately NOT
backup_integrity_failed, whose hub template says the store is DAMAGED. Here the store is
sound and the content is absent: different cause, different action. The hub half shipped
FIRST, in felhom.eu 1aeaa30 (hub v0.110.0, live and verified), because an unallowlisted
type is 400'd and vanishes.

33 new tests, all groups green; full suite 1689 tests, 28 packages, rc=0. All 13 controller
gates OK. Five red-proofs run and recorded in REPORT.md.

A golden carrying 0.231.0 is OWED - the fleet is on 0.230.0. Viktor's call (R-242).
2026-08-31 20:55:34 +02:00

298 lines
12 KiB
Go

package backup
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// R-87 Group A — the JUDGEMENT.
//
// Every test here builds a real recovery-unit directory on disk and asks `JudgeRestoredUnit` about
// it. Nothing is mocked, because there is nothing to mock: the predicate is pure file reads, and a
// fake manifest reader would test the fake.
// unitFixture builds a recovery-unit directory. Empty slices mean "declared nothing"; the files are
// written only for what is declared, so the fixture cannot accidentally satisfy part 1 of the rule.
type unitFixture struct {
compose string // contents of compose/docker-compose.yml; "" = do not write it
dbDumps []string // declared AND written under db-dumps/
volumeDumps []string // declared AND written under volume-dumps/
configFiles []string // declared AND written under compose/
declareOnly bool // declare everything above but write NONE of it (the R-358-adjacent shape)
rawManifest string // if set, written verbatim instead of a marshalled manifest
noManifest bool
}
func buildUnit(t *testing.T, f unitFixture) string {
t.Helper()
dir := t.TempDir()
for _, sub := range []string{"compose", "db-dumps", "volume-dumps"} {
if err := os.MkdirAll(filepath.Join(dir, sub), 0o755); err != nil {
t.Fatalf("mkdir %s: %v", sub, err)
}
}
if f.compose != "" {
if err := os.WriteFile(filepath.Join(dir, "compose", "docker-compose.yml"), []byte(f.compose), 0o644); err != nil {
t.Fatalf("write compose: %v", err)
}
}
if !f.declareOnly {
write := func(sub string, names []string) {
for _, n := range names {
p := filepath.Join(dir, sub, n)
// Never clobber the compose written above: `docker-compose.yml` is BOTH a declared
// config file and the expectation source, and overwriting it with placeholder bytes
// made three tests read `compose_unparseable` — the fixture testing the fixture.
if _, err := os.Stat(p); err == nil {
continue
}
if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
t.Fatalf("write %s/%s: %v", sub, n, err)
}
}
}
write("db-dumps", f.dbDumps)
write("volume-dumps", f.volumeDumps)
write("compose", f.configFiles)
}
switch {
case f.noManifest:
// nothing
case f.rawManifest != "":
if err := os.WriteFile(filepath.Join(dir, "manifest.json"), []byte(f.rawManifest), 0o644); err != nil {
t.Fatalf("write raw manifest: %v", err)
}
default:
m := RecoveryManifest{DBDumps: f.dbDumps, VolumeDumps: f.volumeDumps, ConfigFiles: f.configFiles}
b, err := json.Marshal(m)
if err != nil {
t.Fatalf("marshal manifest: %v", err)
}
if err := os.WriteFile(filepath.Join(dir, "manifest.json"), b, 0o644); err != nil {
t.Fatalf("write manifest: %v", err)
}
}
return dir
}
// composeWithDB is the shape every app with a database has: a service whose image names an engine
// `dbTypeForImage` recognises, plus two named volumes.
const composeWithDB = `services:
app:
image: kimai/kimai2:apache-2.57.0
db:
image: mariadb:11.6
volumes:
kimai_db_data:
kimai_var:
`
// composeNoDBWithVolume — the opengist/privatebin/calibre-web shape measured on demo-hp: no database
// service, one named volume.
const composeNoDBWithVolume = `services:
app:
image: ghcr.io/thomiceli/opengist:1.10
volumes:
opengist_data:
`
// composeNothing — an app that legitimately has neither a database nor a named volume.
const composeNothing = `services:
app:
image: ghcr.io/example/static:1.0
`
func TestR87_UnitWithDumpsAndComposeDBPasses(t *testing.T) {
dir := buildUnit(t, unitFixture{
compose: composeWithDB,
dbDumps: []string{"kimai-mariadb.sql"},
volumeDumps: []string{"kimai_kimai_db_data.tar", "kimai_kimai_var.tar"},
configFiles: []string{"docker-compose.yml"},
})
got := JudgeRestoredUnit(dir)
if !got.OK() {
t.Fatalf("a healthy unit must PASS; got verdict=%q reason=%q missing=%v", got.Verdict, got.Reason, got.Missing)
}
}
// TestR87_HollowUnitForAnAppWithADatabaseFAILS is Scenario B and the whole point of the job.
//
// The unit is INTACT — its manifest parses, and everything it declares is present, because it
// declares nothing. That is exactly the R-403 shape measured on demo-hp on 2026-08-31.
//
// RED-PROOF (run 2026-08-31, recorded in REPORT.md): replacing the rule with "every declared file is
// present" — i.e. returning UnitProofPass as soon as the `missing` list is empty — makes this test
// fail with `verdict "pass", want "fail"`. That is the trap §5.1 names, and this test is what stands
// between the job and it.
func TestR87_HollowUnitForAnAppWithADatabaseFAILS(t *testing.T) {
dir := buildUnit(t, unitFixture{compose: composeWithDB}) // declares nothing at all
// The unit really is internally consistent — prove that, so the failure below cannot be
// mistaken for a corrupt fixture.
if man := readManifest(UnitManifestFile(dir)); man == nil {
t.Fatal("fixture is wrong: the manifest must parse, or this is not the R-403 shape")
}
got := JudgeRestoredUnit(dir)
if got.Verdict != UnitProofFail {
t.Fatalf("a hollow unit for an app WITH a database must FAIL; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
if got.Reason != ProofReasonNoDatabaseDump {
t.Fatalf("reason must name the database expectation; got %q", got.Reason)
}
if len(got.Missing) == 0 || got.Missing[0] != "db" {
t.Fatalf("Missing must name the compose SERVICE that proves the expectation; got %v", got.Missing)
}
}
// TestR87_HollowUnitWithVolumesOnlyFAILS — the second half of the expectation, on an app that has no
// database but does have named volumes. Without this, the rule would only ever fire on database apps
// and would pass opengist, privatebin and calibre-web hollow.
func TestR87_HollowUnitWithVolumesOnlyFAILS(t *testing.T) {
dir := buildUnit(t, unitFixture{compose: composeNoDBWithVolume})
got := JudgeRestoredUnit(dir)
if got.Verdict != UnitProofFail || got.Reason != ProofReasonNoVolumeDump {
t.Fatalf("a hollow unit for an app with named volumes must FAIL on the volume half; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
if len(got.Missing) != 1 || got.Missing[0] != "opengist_data" {
t.Fatalf("Missing must name the compose volume; got %v", got.Missing)
}
}
// TestR87_DatabaseAppWithVolumesButNoDumpFails — the case a coarse "does it carry any data" predicate
// would pass: the unit is NOT hollow (it has tars) and the database is still missing.
//
// This is why `unitCarriesData` alone is not the acceptance rule.
func TestR87_DatabaseAppWithVolumesButNoDumpFails(t *testing.T) {
dir := buildUnit(t, unitFixture{
compose: composeWithDB,
volumeDumps: []string{"kimai_kimai_var.tar"},
})
if unitIsHollow(dir) {
t.Fatal("fixture is wrong: this unit DOES carry data, which is the point of the test")
}
got := JudgeRestoredUnit(dir)
if got.Verdict != UnitProofFail || got.Reason != ProofReasonNoDatabaseDump {
t.Fatalf("a unit with tars but no dump, for an app WITH a database, must FAIL; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
}
// TestR87_AppWithNoDatabaseAndNoVolumesPasses is Scenario C.
//
// RED-PROOF (run 2026-08-31): alarming on any empty unit — i.e. returning UnitProofFail whenever
// `unitIsHollow(dir)` — makes this test fail with `verdict "fail", want "pass"`. A warning that fires
// on healthy things is as bad as a comforting lie, which is the mistake the R-403 work caught in
// itself.
func TestR87_AppWithNoDatabaseAndNoVolumesPasses(t *testing.T) {
dir := buildUnit(t, unitFixture{compose: composeNothing, configFiles: []string{"docker-compose.yml"}})
if !unitIsHollow(dir) {
t.Fatal("fixture is wrong: this unit must be HOLLOW by the coarse predicate, or the test proves nothing")
}
got := JudgeRestoredUnit(dir)
if !got.OK() {
t.Fatalf("an app that legitimately has nothing must PASS; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
}
func TestR87_MissingComposeIsCannotJudgeNotAPass(t *testing.T) {
dir := buildUnit(t, unitFixture{ /* no compose written */ })
got := JudgeRestoredUnit(dir)
if got.Verdict != UnitProofCannotJudge || got.Reason != ProofReasonComposeMissing {
t.Fatalf("a unit with no compose must be CANNOT JUDGE; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
if got.OK() {
t.Fatal("cannot-judge must never read as a pass")
}
}
func TestR87_UnparseableComposeIsCannotJudge(t *testing.T) {
dir := buildUnit(t, unitFixture{compose: "services: [this is not: valid: yaml\n - {"})
got := JudgeRestoredUnit(dir)
if got.Verdict != UnitProofCannotJudge || got.Reason != ProofReasonComposeUnparseable {
t.Fatalf("an unparseable compose must be CANNOT JUDGE, never 'no database'; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
}
// TestR87_UnparseableManifestFails — fail closed. A unit whose manifest cannot be read cannot be
// vouched for, and the direction matters: treating it as sound would let an unreadable backup pass as
// a proved one.
func TestR87_UnparseableManifestFails(t *testing.T) {
for name, fx := range map[string]unitFixture{
"absent": {compose: composeWithDB, noManifest: true},
"not json": {compose: composeWithDB, rawManifest: "{this is not json"},
"truncated": {compose: composeWithDB, rawManifest: `{"db_dumps": [`},
"not object": {compose: composeWithDB, rawManifest: `["a","b"]`},
} {
t.Run(name, func(t *testing.T) {
got := JudgeRestoredUnit(buildUnit(t, fx))
if got.Verdict != UnitProofFail || got.Reason != ProofReasonManifestUnreadable {
t.Fatalf("an unreadable manifest must FAIL CLOSED; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
})
}
}
// TestR87_DeclaredFileAbsentFromScratchFails — part 1 of the rule still holds. A manifest that
// declares a dump the restore did not produce is a failed proof, not a pass.
func TestR87_DeclaredFileAbsentFromScratchFails(t *testing.T) {
dir := buildUnit(t, unitFixture{
compose: composeWithDB,
dbDumps: []string{"kimai-mariadb.sql"},
volumeDumps: []string{"kimai_kimai_db_data.tar"},
declareOnly: true, // declared, never written
})
got := JudgeRestoredUnit(dir)
if got.Verdict != UnitProofFail || got.Reason != ProofReasonDeclaredFileMissing {
t.Fatalf("a declared-but-absent file must FAIL; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
joined := strings.Join(got.Missing, " ")
for _, want := range []string{"db-dumps/kimai-mariadb.sql", "volume-dumps/kimai_kimai_db_data.tar"} {
if !strings.Contains(joined, want) {
t.Fatalf("Missing must name every absent file; %q not in %v", want, got.Missing)
}
}
}
// TestR87_SizeIsNeverConsulted — the R-403 rule, carried forward. A one-byte dump for a tiny app is
// healthy; a fat compose tree with no dumps is the dangerous shape. Size answers "how big", and the
// question here has never been that.
func TestR87_SizeIsNeverConsulted(t *testing.T) {
tiny := buildUnit(t, unitFixture{
compose: composeWithDB,
dbDumps: []string{"kimai-mariadb.sql"}, volumeDumps: []string{"a.tar", "b.tar"},
})
// Make every declared file zero bytes — the smallest a unit can possibly be while still holding
// everything it should.
for _, p := range []string{"db-dumps/kimai-mariadb.sql", "volume-dumps/a.tar", "volume-dumps/b.tar"} {
if err := os.WriteFile(filepath.Join(tiny, p), nil, 0o644); err != nil {
t.Fatalf("truncate %s: %v", p, err)
}
}
if got := JudgeRestoredUnit(tiny); !got.OK() {
t.Fatalf("a zero-byte-but-complete unit must PASS — size is not the question; got %q/%q", got.Verdict, got.Reason)
}
// And the inverse: a LARGE unit that declares nothing must still fail.
fat := buildUnit(t, unitFixture{compose: composeWithDB, configFiles: []string{"docker-compose.yml"}})
if err := os.WriteFile(filepath.Join(fat, "compose", "big.bin"), make([]byte, 1<<20), 0o644); err != nil {
t.Fatalf("write big file: %v", err)
}
if got := JudgeRestoredUnit(fat); got.Verdict != UnitProofFail {
t.Fatalf("a 1 MB unit that declares no data must still FAIL; got %q/%q", got.Verdict, got.Reason)
}
}
// TestR87_ManifestCannotNameAFileOutsideTheUnit — the manifest travels inside the snapshot and a
// restore writes it from the store, so it is not a trusted input. A traversal must read as absent
// (and therefore fail), never as present because something happens to exist up the tree.
func TestR87_ManifestCannotNameAFileOutsideTheUnit(t *testing.T) {
dir := buildUnit(t, unitFixture{compose: composeWithDB, rawManifest: `{"db_dumps":["../../../etc/hostname"]}`})
got := JudgeRestoredUnit(dir)
if got.Verdict != UnitProofFail || got.Reason != ProofReasonDeclaredFileMissing {
t.Fatalf("a traversing manifest entry must read as ABSENT and fail; got verdict=%q reason=%q", got.Verdict, got.Reason)
}
}