v0.228.0 — the off-site check reads the data; the debug page stops lying (R-399 + R-400)
gates / gates (push) Successful in 12s

R-399: monitoring.integrity.read_data_subset defaults to 100%. A pack damaged
without changing its size made plain `restic check` report "no errors were found"
on demo-hp 2026-08-30; every read-data form caught it. Cost on that 134 MB store:
35.0s structure vs 39.2s at 100%. "off" (any case) is the off token; empty means
not-configured, therefore the default; a malformed value falls back to the DEFAULT,
never to structure. A completed check over 5 minutes logs a WARN naming the
duration, the depth and R-401 — operator log only, no hub event, no depth change.
The depth is now recorded with the verdict (LastIntegrityDepth; empty = NOT
RECORDED, never "structure").

R-400: 24 debug-page references, 17 dispatched, 7 dead — three of which fetched on
page LOAD, so those panels were permanently blank. backup/crossdrive implemented;
backup/infra, hub/infra-push, dr/infra-status, storage/watchdog-status and both
storage/simulate-* deleted with their panels and JavaScript.
scripts/debug_route_gate.py fails in both directions and is registered after the
seven were resolved. 18 referenced, 18 dispatched, none orphaned.

Corrections: the dead-field warning in report/types.go said the controller runs no
integrity check and the notifiers are called from nowhere — both false since
v0.227.0. controller.yaml.example gains its missing integrity: block.
integrityCheckTimeout's "ships OFF" comment rewritten.
This commit is contained in:
2026-08-31 10:24:29 +02:00
parent 300d7e87d7
commit 3c49dc8ea4
23 changed files with 1144 additions and 175 deletions
+7
View File
@@ -1469,6 +1469,12 @@ type OffboxReportStatus struct {
// timestamp is the field that says whether the bool means anything at all.
LastIntegrityCheck string `json:"last_integrity_check,omitempty"` // RFC3339
LastIntegrityOK bool `json:"last_integrity_ok,omitempty"`
// LastIntegrityDepth (R-399) is how deep that verdict looked — "structure", or the subset that was
// re-read ("100%"). THIRD field, same argument as the two above and as StatsKnown: a hub that shows
// "checked, OK" without the depth shows the same words for a check that re-read every byte and one
// that only read the index, and those are different news. Absent = the box cannot answer (a
// controller older than v0.228.0), never "structure".
LastIntegrityDepth string `json:"last_integrity_depth,omitempty"`
}
// OffsiteStateNeedsCredential is the ONE declared state (v0.199.0, R-204 item 4 / R-193): this box
@@ -1662,6 +1668,7 @@ func (m *Manager) OffboxReportStatus() *OffboxReportStatus {
StatsKnown: t.StatsKnown, // R-331 — without it the hub cannot tell "empty" from "unmeasured"
LastIntegrityCheck: t.LastIntegrityCheck, // R-359
LastIntegrityOK: t.LastIntegrityOK,
LastIntegrityDepth: t.LastIntegrityDepth, // R-399
AbandonPurgeRequested: t.AbandonPurgeRequested, // R-241: declared until the hub drops the package
}
}
+123 -13
View File
@@ -41,14 +41,64 @@ import (
// any plausible structure check and far below "forever", so the failure mode of a wedged SFTP mount is
// a released flag and a retry tomorrow, never a box whose backups stop because a check never returned.
//
// It is deliberately NOT sized for a `--read-data-subset` run, which downloads pack data and can take
// hours. That option ships OFF (R-399); whoever turns it on must revisit this number, and this comment
// is the note that says so.
// R-399 CHANGED WHAT THIS NUMBER HAS TO COVER, and this paragraph replaces the one that said the
// opposite. Read-data now ships ON at 100% (see defaultIntegrityReadDataSubset below), so a check
// downloads and re-hashes the whole store every week. Measured on demo-hp 2026-08-30: 39.2 s at 100%
// against a 134 MB store, versus 35.0 s at structure depth. 30 minutes is still ~46x the only
// full-depth number that exists, so it is not resized here on the strength of one measurement — but
// it is now the number a LARGE store will meet first, and `integritySlowNoticeThreshold` exists to
// tell the operator long before that happens. R-401 owns the revisit.
const integrityCheckTimeout = 30 * time.Minute
// integritySlowNoticeThreshold is the point at which a COMPLETED check has started costing real time
// and the depth setting needs revisiting (R-401).
//
// CHOSEN, and deliberately imprecise, from the single data point that exists: demo-hp's 134 MB store,
// 2026-08-30, 35.0 s at structure depth and 39.2 s at 100%. 5 minutes is ~7.6x the only full-depth
// number we have, so it cannot fire on anything resembling today's fleet; and it is well under
// integrityCheckTimeout, so the operator hears "this is getting slow" long before a check is killed
// for running too long. A notice changes no behaviour, so an imprecise number is cheap here — whereas
// a precise-looking threshold invented from one measurement on one small store would be the exact
// shape of the four production designs this project has already specced against nothing.
// A var, not a const, for ONE reason: the notice cannot otherwise be proven to fire through the real
// CheckOffboxIntegrity path — no test can make a check take five minutes. Tests lower it and restore
// it with defer. Nothing in production writes it.
var integritySlowNoticeThreshold = 5 * time.Minute
// defaultIntegrityMaxAgeDays is the max age of a SUCCESSFUL check before one is due again.
const defaultIntegrityMaxAgeDays = 7
// defaultIntegrityReadDataSubset is how deep an unconfigured box checks: ALL of it (R-399, Viktor's
// ruling of 2026-08-31).
//
// THE FACT THE DEFAULT RESTS ON, because it is the thing that stops someone turning it back down to
// save four seconds: **the structure check does not detect a size-preserving pack corruption.** On
// 2026-08-30 a pack in demo-hp's store was damaged WITHOUT changing its size; plain `restic check`
// reported `no errors were found` and exited clean, and every read-data form caught it. A store that
// is verified only structurally is a store whose rot is discovered at restore time, with a customer
// waiting.
//
// THE COST, measured the same day on the same store (140 829 678 B / 2 651 blobs / 67 snapshots):
// structure 35.0 s, 10% 35.9 s, 50% 37.3 s, 100% 39.2 s. Four seconds.
//
// WHAT IS NOT ESTABLISHED: how any of that behaves on a store one or two orders of magnitude larger.
// There is exactly ONE data point. That is why this is a constant and a notice (see
// integritySlowNoticeThreshold) rather than a rotation schedule, a size threshold or a bandwidth
// budget — every one of those would be a number invented from a single measurement. R-401.
//
// This default lives HERE and not in config.applyDefaults, deliberately, and defaultIntegrityMaxAgeDays
// beside it is the precedent: both integrity defaults are resolved in this package, in one accessor
// each, next to the reasoning that justifies them. Symmetry with the other Monitoring defaults is
// worth less than having the number and its argument in the same place.
const defaultIntegrityReadDataSubset = "100%"
// integrityOffToken switches the deep check back off without a code change.
//
// A setting with no off switch is not a setting. Without this token there would be no way to return a
// box to structure depth: an EMPTY value means "not configured" and therefore the default (§8), so
// emptiness cannot also mean "off". Matched case-insensitively.
const integrityOffToken = "off"
// readDataSubsetRe accepts the forms restic documents for --read-data-subset: "n/m", a percentage
// like "5%", or a size like "50M". Anything else is refused at read time rather than passed through —
// a typo must not fail the whole check, which is what handing restic an unparsed value would do.
@@ -80,21 +130,64 @@ type IntegrityResult struct {
// integrityReadDataSubset resolves the configured subset spec, refusing anything malformed.
//
// Fail-safe direction: an unrecognised value becomes "" (structure check only) with a WARN, never a
// passthrough. Handing restic `--read-data-subset=banana` fails the entire check, which would turn a
// typo in a config file into a store that silently stops being verified.
// FOUR inputs, three outcomes (§8's table):
// - absent or empty -> defaultIntegrityReadDataSubset. Empty is "not configured", never "off".
// - "off" (any case) -> "" , the structure-and-index check only. The one way to switch it back.
// - a form restic accepts -> itself, unchanged. An explicit value always wins.
// - anything else -> defaultIntegrityReadDataSubset, with a WARN naming the bad value.
//
// THE MALFORMED CASE FALLS BACK TO THE DEFAULT, NOT TO STRUCTURE, and the direction is the point.
// Handing restic `--read-data-subset=banana` fails the whole check, so a typo must not be passed
// through — but downgrading to structure depth on a typo would ALSO silently remove the protection
// R-399 exists to add, which is R-357's shape exactly: a guard that opens quietly. Falling back to the
// default keeps the protection and still says loudly that the config is wrong.
func (m *Manager) integrityReadDataSubset() string {
spec := strings.TrimSpace(m.cfg.Monitoring.Integrity.ReadDataSubset)
if spec == "" {
return defaultIntegrityReadDataSubset
}
if strings.EqualFold(spec, integrityOffToken) {
return ""
}
if !readDataSubsetRe.MatchString(spec) {
m.logger.Printf("[WARN] [offbox] integrity: read_data_subset %q is not a form restic accepts (n/m, N%%, or a size like 50M) — running the STRUCTURE check only", spec)
return ""
m.logger.Printf("[WARN] [offbox] integrity: read_data_subset %q is not a form restic accepts (n/m, N%%, a size like 50M, or %q) — falling back to the DEFAULT depth %q, not to a structure-only check, so a typo cannot quietly remove the protection",
spec, integrityOffToken, defaultIntegrityReadDataSubset)
return defaultIntegrityReadDataSubset
}
return spec
}
// IntegrityDepthCode is the depth as a short RECORDED value, for the persisted verdict and the wire.
//
// "" is reserved to mean NOT RECORDED — a box older than v0.228.0, whose stored verdict cannot say how
// deep it looked. That follows the StatsKnown precedent on the same object: absence means "cannot
// answer", never an answer. So structure depth is written as the word "structure", not as "".
func IntegrityDepthCode(subset string) string {
if subset == "" {
return "structure"
}
return subset
}
// noticeIfSlow logs an operator WARN when a COMPLETED check has started costing real time (R-401).
//
// COMPLETED ONLY. A skip has no duration to judge, and an unreachable store is "I could not look",
// which is not "I looked and it was slow" (§8). Pass and fail BOTH qualify: the notice and the failure
// alarm are independent facts and neither suppresses the other.
//
// It is a log line and NOTHING else — no hub event, no customer alarm. An event type costs the
// severity contract, the grain table and three registers, all to say "this took a while"; 08 §6.2's
// coarse-by-default rule points the other way. And it does NOT change the depth by itself: a notice
// that silently reconfigures the box would be a behaviour change wearing a notice's clothes.
func (m *Manager) noticeIfSlow(res IntegrityResult) {
if res.Skipped || res.Unreachable || res.Duration < integritySlowNoticeThreshold {
return
}
m.logger.Printf("[WARN] [offbox] integrity: the check took %s at depth %s (%s), over the %s notice threshold — R-401: the depth setting needs revisiting for a store this size. Nothing was changed automatically.",
res.Duration.Round(time.Second), IntegrityDepthCode(res.ReadDataSubset),
integrityDepthLabel(res.ReadDataSubset), integritySlowNoticeThreshold)
}
// integrityMaxAge returns the configured max age of a successful check, defaulting to 7 days.
func (m *Manager) integrityMaxAge() time.Duration {
d := m.cfg.Monitoring.Integrity.MaxAgeDays
@@ -133,16 +226,29 @@ func (m *Manager) IntegrityDue(now time.Time) (due bool, last time.Time) {
// the hourly operator cooldown already governs the mail, and the failure is already recorded where a
// surface can read it. A skip or an unreachable repository does NOT reach here, so tomorrow tries
// again.
// RecordIntegrityOutcome is the exported entry point; the caller in main.go owns the decision of WHEN
// a verdict counts, because only it knows whether the run was forced or scheduled.
func (m *Manager) RecordIntegrityOutcome(at time.Time, ok bool) { m.recordIntegrityOutcome(at, ok) }
// RecordIntegrityVerdict is the PRODUCTION entry point: it persists the verdict AND the depth it was
// reached at, in one write. The caller in main.go owns the decision of WHEN a verdict counts, because
// only it knows whether the run was forced or scheduled.
//
// The depth travels with the verdict because a stored result that does not say how deep it looked
// cannot be judged later: "checked, OK" means two different things at structure depth and at 100%,
// and the whole of R-399 is that difference.
func (m *Manager) RecordIntegrityVerdict(res IntegrityResult) {
m.recordIntegrityOutcome(res.RanAt, res.OK, IntegrityDepthCode(res.ReadDataSubset))
}
func (m *Manager) recordIntegrityOutcome(at time.Time, ok bool) {
// RecordIntegrityOutcome records a verdict whose depth is not stated. It writes "" to the depth field,
// which reads as NOT RECORDED rather than as structure depth — see integrityDepthCode. Kept as the
// due-ness surface the R-359 tests drive; production goes through RecordIntegrityVerdict above.
func (m *Manager) RecordIntegrityOutcome(at time.Time, ok bool) { m.recordIntegrityOutcome(at, ok, "") }
func (m *Manager) recordIntegrityOutcome(at time.Time, ok bool, depth string) {
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.LastIntegrityCheck = at.UTC().Format(time.RFC3339)
o.LastIntegrityOK = ok
o.LastIntegrityDepth = depth
}); err != nil {
m.logger.Printf("[ERROR] [offbox] integrity: could not persist the check outcome: %v — the check RAN and its verdict was ok=%v, but due-ness did not advance, so it will run again tomorrow", err, ok)
m.logger.Printf("[ERROR] [offbox] integrity: could not persist the check outcome: %v — the check RAN and its verdict was ok=%v at depth %q, but due-ness did not advance, so it will run again tomorrow", err, ok, depth)
}
}
@@ -198,6 +304,7 @@ func (m *Manager) CheckOffboxIntegrity(ctx context.Context) IntegrityResult {
if err == nil {
res.OK = true
m.logger.Printf("[INFO] [offbox] integrity: check PASSED in %s (%s)", res.Duration.Round(time.Second), integrityDepthLabel(res.ReadDataSubset))
m.noticeIfSlow(res)
return res
}
@@ -221,6 +328,9 @@ func (m *Manager) CheckOffboxIntegrity(ctx context.Context) IntegrityResult {
}
m.logger.Printf("[ERROR] [offbox] integrity: check FAILED after %s — restic reported: %s", res.Duration.Round(time.Second), res.Output)
// A slow FAILING check gets the notice too. The two facts are independent and suppressing one
// because the other fired is how the second fact stops existing.
m.noticeIfSlow(res)
return res
}
@@ -157,21 +157,17 @@ func TestR359_TimeoutIsNotDamage(t *testing.T) {
}
}
func TestR359_StructureCheckPassesNoReadDataFlag(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.CheckOffboxIntegrity(context.Background())
argv := cap.checkArgv()
if argv == nil {
t.Fatal("no check ran")
}
for _, a := range argv {
if strings.HasPrefix(a, "--read-data") {
t.Fatalf("the DEFAULT check downloaded pack data (%q) — that is a bandwidth cost nobody "+
"chose, and R-399 exists precisely so it is not chosen here", a)
}
}
}
// TestR359_StructureCheckPassesNoReadDataFlag was DELETED on 2026-08-31, superseded by R-399.
//
// It asserted that an unconfigured box passes NO --read-data flag. That was the correct contract on
// 2026-08-30, when nothing had measured the cost of a deeper check. The next day a size-preserving
// pack corruption was shown to PASS that structure-only check on real hardware, and Viktor ruled the
// default to full depth. The test is not weakened, it is inverted: its replacement is
// TestR399_AbsentConfigRunsFullDepth in r399_depth_test.go, and the off token it left room for is
// pinned by TestR399_OffTokenRunsStructureOnly.
//
// Recorded here rather than removed silently, so a later reader does not re-derive the old ruling
// from its absence.
func TestR359_ReadDataSubsetIsPassedWhenConfigured(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
@@ -192,25 +188,12 @@ func TestR359_ReadDataSubsetIsPassedWhenConfigured(t *testing.T) {
}
}
func TestR359_MalformedReadDataSubsetIsTreatedAsOff(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = "banana"
res := m.CheckOffboxIntegrity(context.Background())
for _, a := range cap.checkArgv() {
if strings.HasPrefix(a, "--read-data") {
t.Fatalf("a malformed value was handed to restic (%q) — restic rejects it and the WHOLE "+
"check fails, so one typo silently stops the store being verified at all", a)
}
}
if res.ReadDataSubset != "" {
t.Errorf("a refused value was still recorded as the depth: %q", res.ReadDataSubset)
}
if !strings.Contains(cap.logBuf.String(), "WARN") {
t.Error("a refused config value must say so — silence makes a typo indistinguishable from a " +
"deliberate structure-only setting")
}
}
// TestR359_MalformedReadDataSubsetIsTreatedAsOff was DELETED on 2026-08-31, superseded by R-399.
//
// Its NAME was the defect. Treating a typo as "off" downgrades the check silently, which is R-357's
// shape — a guard that opens quietly. The half of it that still holds (a malformed value never
// reaches restic, and it WARNs) is asserted by TestR399_MalformedFallsBackToTheDefault, which also
// pins the new direction: the fallback is the DEFAULT depth, never structure.
func TestR359_MessageNeverCarriesResticOutputOrCredentials(t *testing.T) {
// R-379: 615 bytes of raw database text reached a customer once. And offboxBaseArgs builds the repo
@@ -0,0 +1,193 @@
package backup
import (
"context"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// ── R-399 — the off-site check reads the DATA, not just the catalogue ────────────────────────────
//
// THE MEASUREMENT THESE TESTS DEFEND. On demo-hp, 2026-08-30, a pack in a real 134 MB store was
// damaged WITHOUT changing its size. Plain `restic check` — the structure-and-index check that every
// box in the fleet ran — reported `no errors were found` and exited clean. Every `--read-data*` form
// caught it. Cost of the deeper run on that store: 39.2 s versus 35.0 s.
//
// So the assertions below are on the ARGUMENT LIST, never on the absence of an error. "The check
// passed" is exactly what the broken configuration produced; only the argv can tell the two apart.
// readDataArg returns the --read-data* argument the check passed to restic, or "" when it passed none.
func readDataArg(argv []string) string {
for _, a := range argv {
if strings.HasPrefix(a, "--read-data") {
return a
}
}
return ""
}
// A1 — the fleet's actual configuration: no integrity block at all.
func TestR399_AbsentConfigRunsFullDepth(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
// Deliberately touch nothing: this is a box whose controller.yaml has no `integrity:` key.
res := m.CheckOffboxIntegrity(context.Background())
argv := cap.checkArgv()
if argv == nil {
t.Fatal("no check ran")
}
if got := readDataArg(argv); got != "--read-data-subset=100%" {
t.Fatalf("an unconfigured box ran at depth %q, want --read-data-subset=100%%; argv=%v\n"+
"A structure-only run is what every box did before R-399, and it PASSED a size-preserving "+
"pack corruption on real hardware — the store's rot would be found at restore time",
got, argv)
}
if res.ReadDataSubset != "100%" {
t.Errorf("the result did not record the depth it actually ran at: %+v", res)
}
}
// A2 — an empty string is NOT the off switch. Empty means "not configured", so it means the default.
func TestR399_EmptyStringIsNotOff(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = ""
m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "--read-data-subset=100%" {
t.Fatalf("an empty value was treated as OFF (%q) — emptiness must mean 'not configured', or "+
"there is no way to distinguish an unset key from a deliberate downgrade", got)
}
}
// A3 — the off switch. A setting with no off switch is not a setting.
func TestR399_OffTokenRunsStructureOnly(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = "off"
res := m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "" {
t.Fatalf("the off token still downloaded pack data (%q) — there would be no way to switch the "+
"deep check off without editing code", got)
}
if res.ReadDataSubset != "" {
t.Errorf("a structure-only run recorded a subset: %q", res.ReadDataSubset)
}
if code := IntegrityDepthCode(res.ReadDataSubset); code != "structure" {
t.Errorf("structure depth must be RECORDED as %q, not as an empty string a reader has to "+
"interpret; got %q", "structure", code)
}
}
// A4 — an operator writing "OFF" or "Off" in a YAML file means the same thing.
func TestR399_OffTokenIsCaseInsensitive(t *testing.T) {
for _, tok := range []string{"OFF", "Off", " oFf "} {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = tok
m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "" {
t.Errorf("%q was not recognised as the off token (argv carried %q)", tok, got)
}
}
}
// A5 — an explicit value wins over both the default and the off token.
func TestR399_ExplicitValueWins(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = "10%"
res := m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "--read-data-subset=10%" {
t.Fatalf("an explicit 10%% ran as %q — a chosen value must not be overridden by a default", got)
}
if res.ReadDataSubset != "10%" {
t.Errorf("result recorded %q, want 10%%", res.ReadDataSubset)
}
}
// A6 — a typo falls back to the DEFAULT, not to structure depth.
//
// The direction is the whole point. Passing "banana" through fails the entire check; downgrading to
// structure would silently remove the protection R-399 exists to add, which is R-357's shape exactly —
// a guard that opens quietly. Falling back to the default keeps the protection and still says loudly
// that the config is wrong.
func TestR399_MalformedFallsBackToTheDefault(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = "banana"
res := m.CheckOffboxIntegrity(context.Background())
argv := cap.checkArgv()
for _, a := range argv {
if strings.Contains(a, "banana") {
t.Fatalf("a malformed value was handed to restic (%q) — restic rejects it and the WHOLE "+
"check fails, so one typo would stop the store being verified at all", a)
}
}
if got := readDataArg(argv); got != "--read-data-subset=100%" {
t.Fatalf("a typo downgraded the check to %q instead of falling back to the default 100%% — "+
"a guard that opens quietly on a typo is R-357's failure", got)
}
if res.ReadDataSubset != "100%" {
t.Errorf("result recorded %q after a refused value, want the default", res.ReadDataSubset)
}
log := cap.logBuf.String()
if !strings.Contains(log, "WARN") || !strings.Contains(log, "banana") {
t.Errorf("a refused config value must WARN and NAME the bad value; log was:\n%s", log)
}
}
// A7 — the depth is stated in the outcome, or the result cannot be judged afterwards.
func TestR399_DepthIsStatedInTheOutcome(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
res := m.CheckOffboxIntegrity(context.Background())
if !strings.Contains(cap.logBuf.String(), "100%") {
t.Errorf("the PASSED log line does not say how deep the check looked:\n%s", cap.logBuf.String())
}
// And it is PERSISTED with the verdict, so a later reader of the stored state can judge it too.
m.RecordIntegrityVerdict(res)
tgt := m.settings.GetOffboxTarget()
if tgt == nil || tgt.LastIntegrityDepth != "100%" {
t.Fatalf("the stored verdict does not carry its depth: %+v — 'checked, OK' means two different "+
"things at structure depth and at 100%%", tgt)
}
}
// A8 — THE SEAM TEST. Everything above sets the config field directly; this one proves the default
// survives the PRODUCTION resolution path: real YAML bytes -> config.LoadFromBytes -> the accessor ->
// the argv. A default that only works when a test hand-builds the struct is the inert-seam failure
// this project has shipped four times.
func TestR399_DefaultResolvesThroughTheRealConfigPath(t *testing.T) {
// A minimal but VALID controller.yaml — LoadFromBytes validates, and a config that fails
// validation would never reach a running box, so a fixture that skipped the required keys would
// not be the production path.
const yaml = `
customer:
id: "demo-hp"
domain: "felhom.example.hu"
monitoring:
enabled: true
thresholds:
disk_warn_percent: 80
`
loaded, err := config.LoadFromBytes([]byte(yaml))
if err != nil {
t.Fatalf("the fixture config does not parse: %v", err)
}
if loaded.Monitoring.Integrity.ReadDataSubset != "" {
t.Fatalf("fixture wrong: the parsed config already carries a subset %q, so this test would "+
"prove nothing about the ABSENT case", loaded.Monitoring.Integrity.ReadDataSubset)
}
m, cap := newIntegrityManager(t, okRepo(nil))
// Only the parsed Monitoring block is transplanted; Paths must stay pointing at the test's temp
// dir. The seam under test is config-parse -> Monitoring.Integrity -> integrityReadDataSubset.
m.cfg.Monitoring = loaded.Monitoring
m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "--read-data-subset=100%" {
t.Fatalf("a real controller.yaml with no `integrity:` block resolved to depth %q, want "+
"--read-data-subset=100%% — that is every box in the fleet today", got)
}
}
@@ -0,0 +1,139 @@
package backup
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// ── R-399 / R-401 — the check tells the operator when it starts costing real time ────────────────
//
// The 100% default rests on ONE measurement, on ONE 134 MB store: 39.2 s. It will not stay true. The
// notice is what makes that fact reach a person from the product rather than from a customer.
//
// It is a LOG LINE and nothing else — no hub event, no customer alarm, and it never changes the depth
// by itself. TestR399_SlownessRaisesNoHubEvent in cmd/controller pins the first of those; the other
// two are pinned here.
// slowNoticeFragment is the ASCII-only fingerprint of the notice, chosen so no other line in this
// file's logs can match it.
const slowNoticeFragment = "R-401: the depth setting needs revisiting"
// withSlowThreshold lowers the notice threshold for one test and restores it.
//
// No test can make a check take five minutes, so the threshold is the seam. Everything else runs
// through the real CheckOffboxIntegrity.
func withSlowThreshold(t *testing.T, d time.Duration) {
t.Helper()
prev := integritySlowNoticeThreshold
integritySlowNoticeThreshold = d
t.Cleanup(func() { integritySlowNoticeThreshold = prev })
}
// slowReply makes the `check` call take measurable time so a lowered threshold is genuinely exceeded
// rather than merely equalled.
func slowReply(out []byte, err error) func(args []string) ([]byte, error) {
return okRepo(func(args []string) ([]byte, error) {
time.Sleep(3 * time.Millisecond)
return out, err
})
}
// B1 — a slow check warns, and the warning names the duration and the depth.
func TestR399_SlowCheckWarns(t *testing.T) {
withSlowThreshold(t, time.Millisecond)
m, cap := newIntegrityManager(t, slowReply(nil, nil))
res := m.CheckOffboxIntegrity(context.Background())
if !res.OK {
t.Fatalf("fixture: this run should pass; got %+v", res)
}
log := cap.logBuf.String()
if !strings.Contains(log, slowNoticeFragment) {
t.Fatalf("a check over the threshold produced no notice — the operator would keep spending a "+
"customer's bandwidth every week and hear about it from the customer. Log:\n%s", log)
}
if !strings.Contains(log, "WARN") {
t.Errorf("the notice is not at WARN level:\n%s", log)
}
// It must name BOTH facts: how long, and how deep. Either alone is unactionable.
if !strings.Contains(log, "100%") {
t.Errorf("the notice does not name the depth that was slow:\n%s", log)
}
if !strings.Contains(log, "the check took") {
t.Errorf("the notice does not name the duration:\n%s", log)
}
// And it changes NOTHING by itself. A notice that silently reconfigured the box would be a
// behaviour change wearing a notice's clothes.
if m.integrityReadDataSubset() != "100%" {
t.Error("the notice altered the configured depth — it must only report")
}
}
// B2 — a fast check is silent. A notice that fires every week is not a notice.
func TestR399_FastCheckIsSilent(t *testing.T) {
withSlowThreshold(t, time.Hour)
m, cap := newIntegrityManager(t, okRepo(nil))
m.CheckOffboxIntegrity(context.Background())
if strings.Contains(cap.logBuf.String(), slowNoticeFragment) {
t.Fatalf("a check under the threshold warned anyway:\n%s", cap.logBuf.String())
}
}
// B3 — a SKIP has no duration to judge.
func TestR399_SkipNeverWarns(t *testing.T) {
withSlowThreshold(t, time.Nanosecond) // every non-zero duration would qualify
m, cap := newIntegrityManager(t, okRepo(nil))
if err := m.AcquireRunningForTest(); err != nil {
t.Fatalf("fixture: %v", err)
}
defer m.ReleaseRunningForTest()
res := m.CheckOffboxIntegrity(context.Background())
if !res.Skipped {
t.Fatalf("fixture: expected a skip, got %+v", res)
}
if strings.Contains(cap.logBuf.String(), slowNoticeFragment) {
t.Fatalf("a skipped check produced a slowness notice — it never ran, so there is no duration "+
"to judge:\n%s", cap.logBuf.String())
}
}
// B4 — "I could not look" is not "I looked and it was slow".
func TestR399_UnreachableNeverWarns(t *testing.T) {
withSlowThreshold(t, time.Nanosecond)
m, cap := newIntegrityManager(t, func(args []string) ([]byte, error) {
time.Sleep(3 * time.Millisecond)
return []byte("ssh: connect to host nas.local port 22: Connection refused"), errors.New("exit 1")
})
res := m.CheckOffboxIntegrity(context.Background())
if !res.Unreachable {
t.Fatalf("fixture: expected unreachable, got %+v", res)
}
if strings.Contains(cap.logBuf.String(), slowNoticeFragment) {
t.Fatalf("an unreachable store produced a slowness notice:\n%s", cap.logBuf.String())
}
}
// B5 — the notice and the failure verdict are INDEPENDENT facts. Neither suppresses the other.
func TestR399_SlowAndFailedProducesBoth(t *testing.T) {
withSlowThreshold(t, time.Millisecond)
const damaged = "pack 5b1f2c3d: not found in index\nrepository contains errors"
m, cap := newIntegrityManager(t, slowReply([]byte(damaged), errFake))
res := m.CheckOffboxIntegrity(context.Background())
if res.OK || res.Unreachable {
t.Fatalf("fixture: expected a readable-and-damaged verdict, got %+v", res)
}
log := cap.logBuf.String()
if !strings.Contains(log, "check FAILED") {
t.Fatalf("the failure was not reported:\n%s", log)
}
if !strings.Contains(log, slowNoticeFragment) {
t.Fatalf("a slow FAILING check lost its slowness notice — suppressing one fact because the "+
"other fired is how the second fact stops existing:\n%s", log)
}
}