v0.122.0: three ways the signals lied about themselves (R-189, R-188, R-186)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.
R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.
R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.
R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
This commit is contained in:
+96
-5
@@ -47,6 +47,22 @@ type RestoreTestReporter interface {
|
||||
RestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// ProvenRestoreTestReporter is the DURABLE half of the restore-test signal (R-189).
|
||||
//
|
||||
// RestoreTestReporter above is backed by an in-memory store whose own comment used to read "lost on
|
||||
// restart; the cadence re-populates". That was true while a timer re-tested every tier daily. It
|
||||
// stopped being true on 2026-08-03: under per-archive due-ness the agent will not re-test an archive
|
||||
// it has already proven, so a proof lost to a restart is not repeated for a whole archive generation
|
||||
// — a week on the offsite tier — and the hub reports the tier unproven the entire time.
|
||||
//
|
||||
// Observed, not predicted: a real 14.5 GB offsite restore passed at 15:25:14, the agent was restarted
|
||||
// 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two reports.
|
||||
//
|
||||
// (*backup.RestoreTestState).ProvenRestoreTests satisfies this. nil → the merge is a no-op.
|
||||
type ProvenRestoreTestReporter interface {
|
||||
ProvenRestoreTests(ctx context.Context) []RestoreTest
|
||||
}
|
||||
|
||||
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
|
||||
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
|
||||
type PBSReporter interface {
|
||||
@@ -79,6 +95,7 @@ type Collector struct {
|
||||
storage StorageObserver
|
||||
backups BackupReporter
|
||||
restoreTests RestoreTestReporter
|
||||
provenTests ProvenRestoreTestReporter
|
||||
pbs PBSReporter
|
||||
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
|
||||
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
|
||||
@@ -427,16 +444,90 @@ func (c *Collector) collectBackups(ctx context.Context) []Backup {
|
||||
return []Backup{}
|
||||
}
|
||||
|
||||
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
|
||||
//
|
||||
// The rule is ONE ENTRY PER TIER, NEWEST WINS, and it falls out of what each source means rather
|
||||
// than from a preference between them:
|
||||
//
|
||||
// - the in-memory store holds this process's latest run, pass OR fail. A failure exists nowhere
|
||||
// else and must always reach the hub — a failing tier is retried at the next evaluation, so its
|
||||
// record is short-lived by design;
|
||||
// - the persisted state holds the last SUCCESS per tier and survives a restart.
|
||||
//
|
||||
// Comparing by TestedAt gives the right answer in every case without special-casing: a fresh failure
|
||||
// beats an older stored success (the failure is the news), a stored success beats a stale in-memory
|
||||
// entry after a restart, and a tier proved twice never appears twice — two entries for one tier would
|
||||
// read at the hub as two tests.
|
||||
//
|
||||
// A tier with no usable persisted proof contributes NOTHING. Reporting an unproven tier as proven
|
||||
// would be a worse defect than the one this closes.
|
||||
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
|
||||
if c.restoreTests == nil {
|
||||
return []RestoreTest{}
|
||||
out := []RestoreTest{}
|
||||
if c.restoreTests != nil {
|
||||
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
||||
out = append(out, r...)
|
||||
}
|
||||
}
|
||||
if r := c.restoreTests.RestoreTests(ctx); r != nil {
|
||||
return r
|
||||
if c.provenTests == nil {
|
||||
return out
|
||||
}
|
||||
return []RestoreTest{}
|
||||
|
||||
// Index what we already have by tier, keeping the newest per tier.
|
||||
best := map[string]int{} // tier → index into out
|
||||
for i, rt := range out {
|
||||
if rt.SourceTier == "" {
|
||||
continue // untiered entry: never deduped, never overwritten — we cannot say what it is
|
||||
}
|
||||
if j, seen := best[rt.SourceTier]; !seen || newerRestoreTest(rt, out[j]) {
|
||||
best[rt.SourceTier] = i
|
||||
}
|
||||
}
|
||||
for _, p := range c.provenTests.ProvenRestoreTests(ctx) {
|
||||
if p.SourceTier == "" {
|
||||
continue // not usable as a per-tier proof; the state layer already filters these
|
||||
}
|
||||
i, seen := best[p.SourceTier]
|
||||
if !seen {
|
||||
out = append(out, p)
|
||||
best[p.SourceTier] = len(out) - 1
|
||||
continue
|
||||
}
|
||||
if newerRestoreTest(p, out[i]) {
|
||||
out[i] = p
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// newerRestoreTest reports whether a was tested after b. An unparseable or absent timestamp is
|
||||
// treated as OLDER, so a malformed entry can never displace a good one.
|
||||
func newerRestoreTest(a, b RestoreTest) bool {
|
||||
ta, aok := parseRestoreTestedAt(a.TestedAt)
|
||||
tb, bok := parseRestoreTestedAt(b.TestedAt)
|
||||
if !aok {
|
||||
return false
|
||||
}
|
||||
if !bok {
|
||||
return true
|
||||
}
|
||||
return ta.After(tb)
|
||||
}
|
||||
|
||||
func parseRestoreTestedAt(s string) (time.Time, bool) {
|
||||
t, err := time.Parse(time.RFC3339, s)
|
||||
if err != nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
return t.UTC(), true
|
||||
}
|
||||
|
||||
// SetProvenRestoreTests wires the durable proof source. It is a setter rather than a constructor
|
||||
// argument because the persisted state is opened later in main() than the collector is built; the
|
||||
// same shape as the other late-wired seams here. **The wiring is asserted by an AST test** — the
|
||||
// method it feeds carried a doc comment naming a "host-report gauge" for weeks with no caller at
|
||||
// all, and this fix must not become the next instance of that.
|
||||
func (c *Collector) SetProvenRestoreTests(p ProvenRestoreTestReporter) { c.provenTests = p }
|
||||
|
||||
// collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty).
|
||||
func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
|
||||
if c.pbs == nil {
|
||||
|
||||
Reference in New Issue
Block a user