From 7581f8140a8ffbd4f16a5c154fffcdde0b9ab12f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 3 Aug 2026 16:40:18 +0200 Subject: [PATCH] v0.122.0: three ways the signals lied about themselves (R-189, R-188, R-186) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three are the reporting and release path misreporting its own work. No customer machine, no backup, no restore, no data. The restore-test itself and when it runs are unchanged. R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[] came only from the in-memory store, whose comment ("lost on restart; the cadence re-populates") was true under a timer and stopped being true when R-86 made the agent refuse to re-test a proven archive: the proof is then not repeated for a whole archive generation. Observed live — a 14.5 GB offsite PASS reached no host-report because the agent was restarted 2m43s later. RestoreTestState now carries tier + verified beside the archive and renders reportable entries; the collector merges them, one per tier, newest by TestedAt. It refuses to lie: a record missing archive-or-tier produces no entry, and run mechanics are not re-invented. Only successes are persisted, and the asymmetry is now written where it will be read. R-188 — a correct release stops emailing a failure. Only the tag PUSH moved (build -> tag locally -> publish -> push tag): the push wakes CI, and a tag visible before its package made the gate correctly fail a correct release about half the time. The old order's invariant is asserted directly instead — the gate now refuses a published version with no tag, as a bounded probe that prints its own coverage, because the package listing api is still 401 without a token. R-186 — a released binary can be verified by rebuilding it. -trimpath -buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the same version; both paths now build identically. CLAUDE.md records the command. --- CHANGELOG.md | 81 +++++++ CLAUDE.md | 27 +++ CONTEXT.md | 25 ++ REUSE.md | 3 +- cmd/felhom-agent/main.go | 6 + .../restoretest_due_wiring_test.go | 43 ++++ internal/backup/restoretest_due_test.go | 90 +++++++- internal/backup/restoretest_state.go | 112 +++++++-- internal/backup/rotation_test.go | 10 +- internal/backup/schedule.go | 6 +- internal/backup/store.go | 18 +- internal/hub/collect.go | 101 ++++++++- internal/hub/restoretest_merge_test.go | 213 ++++++++++++++++++ scripts/check-published-versions.py | 109 ++++++++- scripts/publish-agent.sh | 6 +- scripts/release-agent.sh | 87 ++++++- 16 files changed, 895 insertions(+), 42 deletions(-) create mode 100644 internal/hub/restoretest_merge_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index e4a49ae..2616758 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,84 @@ +## v0.122.0 — three ways the signals lied about themselves (2026-08-03, R-189 · R-188 · R-186) + +All three are the reporting and release path misreporting its own work. **No customer machine, no +backup, no restore, no disk layout, no data.** The restore-test itself and when it runs are unchanged +from v0.121.1. + +### R-189 — a passing restore-test no longer vanishes on a restart + +`restore_tests[]` came only from the in-memory `backup.Store`, whose own comment read *"lost on +restart; the cadence re-populates"*. That was true under a timer. It stopped being true when R-86 made +the agent refuse to re-test an archive it has already proven: a proof lost to a restart is not +repeated for a whole archive generation — **a week on the offsite tier** — and the hub calls the tier +unproven for all of it. + +**Observed, not predicted (2026-08-03):** a real 14.5 GB offsite restore-test PASSED at 15:25:14, the +agent was restarted 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two +host-reports. + +The durable proof already existed — `RestoreTestState`, on disk, per tier, with the archive since +R-86 — and `Snapshot()` had carried the doc comment *"for the host-report gauge"* since the day it was +written **with no caller at all**: a seam built, documented, and never connected. It now carries the +`tier` and what was `verified` as well (stored at proof time, when they are known for certain, rather +than derived later by a storage lookup that can fail), and `ProvenRestoreTests` renders them as report +entries which the collector merges. + +- **Merge rule: one entry per tier, newest by `TestedAt` wins.** A fresh failure beats a stored + success — the failure is the news and lives nowhere else; a stored success beats a stale in-memory + entry after a restart; a tier never appears twice, which the hub would read as two tests. An + unparseable timestamp counts as older, so a malformed entry cannot displace a good one. +- **It refuses to lie.** A record missing the archive or the tier produces NO entry, and run mechanics + (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one + would be. An unproven tier reading as proven would be worse than the defect being fixed. +- **Only successes are persisted, and that asymmetry is now written down where it will be read:** a + success suppresses future work, so losing it leaves the system quietly less tested than it believes; + a failure causes future work and heals itself at the next evaluation. +- The `Store` comment that stopped being true is corrected in place rather than left to mislead. + +### R-188 — a correct release no longer emails a failure + +`on: [push]` fires the gates workflow on the **tag** push, and the release pushed its tag *before* +publishing, so CI ran the published-versions gate in the seconds before the package existed and +correctly reported it missing. Measured across two releases in one session: runs 12/13 and 17/18, same +sha each time, opposite results — a race, not a rule. R-168 made that mail the thing that cannot be +missed; one that is wrong half the time is one you stop reading. + +**Only the tag PUSH moved** (build → tag locally → publish → push tag). The tag is still created before +anything is published, so the build and the tag still describe the same commit; it simply becomes +*visible* — to CI, and to any `raw/tag/…` fetch — once the package is downloadable. + +The invariant the old order protected is **not traded away**: `check-published-versions.py` now asserts +the converse directly — **no published version may be missing its tag** — as a bounded probe of the +frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set every +run because a check whose coverage is invisible reads as a guarantee it is not making. The package +listing api still answers **401** without a token (re-measured), so absence cannot be enumerated, and +the script says so. + +A publish that succeeds and a tag push that then fails now **dies loudly**, printing the one-line +recovery; and a publish that *fails* removes the local-only tag so the release can simply be retried +instead of colliding with itself. + +### R-186 — a released binary can now be verified by rebuilding it + +`go build` stamps a module version derived from VCS state, so a build made before the tag existed and +a rebuild made after it were different binaries. Measured at one commit, same source, same toolchain: + +``` +default flags, no tag yet ... 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61) +default flags, tagged ....... 4a38f394… 14 085 440 B (mod v0.121.99) +-trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways +``` + +The stamp is removed rather than sequenced around — nothing in this repo reads it (no `ReadBuildInfo` +caller) and the version comes from the explicit `-X main.version` ldflag. `-trimpath` additionally +makes a rebuild from a different checkout directory match. + +**A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and +therefore produced a binary **74 KB smaller** than the release path built for the same version — one +version name, two binaries, decided by which entry point was used. Both now build identically. + +`CLAUDE.md` records the exact command an operator can run to verify a published binary independently. + ## v0.121.1 — "nothing is due" must be AUDIBLE (2026-08-03, R-86 + standing rule 3) **Found while live-validating v0.121.0, and it is this project's own rule pointed at the change that diff --git a/CLAUDE.md b/CLAUDE.md index b15a4d7..61ce310 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,6 +72,32 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S > verifies by an **independent download** rather than trusting the publish step's own output. > `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather > than reimplementing it. +> +> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188, +> R-186).** The tag is created before the publish so the build and the tag describe the same commit; +> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its +> package makes the published-versions gate correctly fail a correct release — it did, on roughly +> every second release, and R-168 sends that failure to you by mail. The invariant the old order +> protected is asserted directly instead: the gate now also refuses a **published version with no +> tag**. If the push fails after a successful publish the script says so loudly and prints the +> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean. +> +> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath +> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet — +> before this, a rebuild could not reproduce the sha you were vouching. To check any published +> version yourself: +> +> ```bash +> V=0.122.0 +> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \ +> -o /tmp/felhom-agent-check ./cmd/felhom-agent +> sha256sum /tmp/felhom-agent-check +> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum +> ``` +> +> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to +> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line +> ever changes, change both or one version name means two binaries again. | Step | Where | One-liner | |---|---|---| @@ -79,6 +105,7 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S | Copy | local → felhom-pve | `scp /tmp/felhom-agent- felhom-pve:/tmp/` (one hop) | | Deploy | felhom-pve | backup `.bak-` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) | | Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed | +| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v && go build -trimpath -buildvcs=false -ldflags "-X main.version=" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL \| sha256sum` | | **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) | | Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) | diff --git a/CONTEXT.md b/CONTEXT.md index c9837d4..9790dc7 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,31 @@ ## Current +- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.** + None touches data; all three cost attention, which every other signal depends on. + - **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only + from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a + timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:** + a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests` + twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2 + still read, and a record missing archive-or-tier is NOT reported), exposes + `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier, + newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears + twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a + "host-report gauge" in its doc comment and had **no caller** for weeks. + - **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses* + future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less + tested than it believes); a failure *causes* future work and heals itself at the next evaluation. + - **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag + locally → publish → push tag): the push is what wakes CI, and a tag visible before its package + made the gate correctly fail a correct release ~half the time. The old order's invariant is now + asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as + a bounded, printed probe (the package listing api is still 401 without a token, re-measured). + - **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same + bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced + `CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now + identical. The verification command is in `CLAUDE.md`. + - **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for `settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive; diff --git a/REUSE.md b/REUSE.md index e5c7e02..c6cc30e 100644 --- a/REUSE.md +++ b/REUSE.md @@ -148,7 +148,8 @@ | `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go | | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). | -| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,t)` / `ProvenArchive(target)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. | +| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. | +| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. | | `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. | | `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). | | `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go | diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index e0f285e..94fe5ac 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -662,6 +662,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json")) heavyOps := &backup.InFlight{} scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger) + // R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store + // holds only this process's latest run, and under per-archive due-ness the agent will not + // re-test an archive it has already proven — so without this the hub can report a tier unproven + // for a whole archive generation after a deploy. Observed live on 2026-08-03: a passing 14.5 GB + // offsite restore-test reached no host-report at all. + collector.SetProvenRestoreTests(rtState) // PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free, // ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot diff --git a/cmd/felhom-agent/restoretest_due_wiring_test.go b/cmd/felhom-agent/restoretest_due_wiring_test.go index a369b9e..512a84f 100644 --- a/cmd/felhom-agent/restoretest_due_wiring_test.go +++ b/cmd/felhom-agent/restoretest_due_wiring_test.go @@ -111,3 +111,46 @@ func parseMainForWiring(t *testing.T) *ast.File { } return f } + +// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector. +// +// This test exists because the method it feeds is the project's own cautionary tale: +// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it +// was written and **had no caller at all** — a seam built, documented and never connected, found +// only when a live restore-test's PASS reached no host-report. The fix must not become the next +// instance, so the wiring is asserted rather than trusted. +// +// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting +// out the tier-picker line failed this test while a `strings.Contains` check would have passed). +func TestMainWiresTheDurableRestoreTestProof(t *testing.T) { + f := parseMainForWiring(t) + + var wired, feedsState bool + ast.Inspect(f, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok { + return true + } + sel, ok := call.Fun.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "SetProvenRestoreTests" { + return true + } + wired = true + // ...and it must be fed the PERSISTED state, not the in-memory store. + if len(call.Args) == 1 { + if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" { + feedsState = true + } + } + return true + }) + + if !wired { + t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " + + "reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart") + } + if wired && !feedsState { + t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " + + "that does NOT survive a restart, so wiring it here would fix nothing") + } +} diff --git a/internal/backup/restoretest_due_test.go b/internal/backup/restoretest_due_test.go index 201e3c4..a44fa94 100644 --- a/internal/backup/restoretest_due_test.go +++ b/internal/backup/restoretest_due_test.go @@ -442,7 +442,7 @@ func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) { path := filepath.Join(t.TempDir(), "rt.json") now := time.Now().UTC().Truncate(time.Second) st := NewRestoreTestState(path) - if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil { + if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil { t.Fatal(err) } re := NewRestoreTestState(path) @@ -475,7 +475,7 @@ func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) { }} h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts) // Prove the local tier so NOTHING is due. - if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", h.clock); err != nil { + if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil { t.Fatal(err) } @@ -505,3 +505,89 @@ func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) { t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got) } } + +// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ─────────────────── +// +// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the +// agent will not repeat the work. So the persisted record has to be able to become a host-report +// entry — without inventing anything it does not know. +// +// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from +// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted → +// +// --- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe +// restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its +// per-tier proof on it); got [{... SourceTier: ...}] +// +// Restored. +func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) { + path := filepath.Join(t.TempDir(), "rt.json") + // v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this + // file has ever had, which is what a real box carries after two upgrades. + legacy := `{ + "old-v1": "2026-07-30T02:11:07Z", + "old-v2": {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"}, + "felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"} + }` + if err := writeFileForTest(path, legacy); err != nil { + t.Fatal(err) + } + + got := NewRestoreTestState(path).ProvenRestoreTests(context.Background()) + if len(got) != 1 { + t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got) + } + e := got[0] + if e.SourceTier != "pbs" { + t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got) + } + if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass { + t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e) + } + if e.TestedAt != "2026-08-03T13:25:14Z" { + t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt) + } + if e.Verified != "boot+running" { + t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified) + } + // Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be. + if e.DurationSeconds != 0 || e.ScratchVMID != 0 { + t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d", + e.DurationSeconds, e.ScratchVMID) + } + // The legacy records still serve the DUE-check, which is a separate question from reporting. + if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok { + t.Fatal("a v2 record must still answer the due-check even though it cannot be reported") + } +} + +// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report — +// the production path, not a hand-built fixture. +func TestScheduler_ProofIsRecordedReportably(t *testing.T) { + ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}} + h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts) + // The fake runner echoes the spec's tier; give the spec a tier the way main.go does. + h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec { + return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"} + } + h.s.tick(context.Background()) + + got := h.st.ProvenRestoreTests(context.Background()) + if len(got) != 1 { + t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got) + } + if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" { + t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0]) + } +} + +// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed. +func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) { + ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}} + h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts) + h.s.tick(context.Background()) + if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 { + t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+ + "would outlive the fault; got %+v", got) + } +} diff --git a/internal/backup/restoretest_state.go b/internal/backup/restoretest_state.go index 94b853c..470857c 100644 --- a/internal/backup/restoretest_state.go +++ b/internal/backup/restoretest_state.go @@ -1,12 +1,15 @@ package backup import ( + "context" "encoding/json" "os" "path/filepath" "sort" "sync" "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" ) // RestoreTestState persists the last SUCCESSFUL restore-test per backup tier. @@ -49,16 +52,45 @@ type RestoreTestState struct { last map[string]provenTier // target id → what was last PROVEN on that tier } -// provenTier is one tier's proof: the archive that passed, and when it passed. +// provenTier is one tier's proof: the archive that passed, which tier it was, what was verified, +// and when. +// +// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not +// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result +// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the +// hub can stay ignorant of a real success until the next archive generation. +// +// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own +// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup +// at report-building time — a network call that can fail, on a path where failing means mis-labelling +// a proof. Store what you knew when you knew it. type provenTier struct { - Archive string // volid of the archive that PASSED; "" = a legacy record with no archive - At time.Time // when that run passed (UTC) + Archive string // volid of the archive that PASSED; "" = a legacy record with no archive + Tier string // "local" | "pbs" — as the run reported it; "" = pre-R-189 record + Verified string // what the run verified (e.g. "boot+running"); "" = pre-R-189 record + At time.Time // when that run passed (UTC) } -// provenTierJSON is the on-disk shape (R-86). The legacy shape was a bare RFC3339 STRING per -// target; both are read, only this one is written — see NewRestoreTestState. +// reportable reports whether this record can be re-reported to the hub as a restore-test result. +// +// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the +// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable +// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence. +// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in. +func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" } + +// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written: +// +// v1 (pre-R-86) "": "" — a time, no archive +// v2 (R-86) "": {archive, proven_at} — due-check usable, not reportable +// v3 (R-189) "": {archive, tier, verified, …} — both +// +// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the +// readers above test for — the migration needs no version number because the absence IS the answer. type provenTierJSON struct { Archive string `json:"archive"` + Tier string `json:"tier,omitempty"` + Verified string `json:"verified,omitempty"` ProvenAt string `json:"proven_at"` } @@ -99,21 +131,31 @@ func NewRestoreTestState(path string) *RestoreTestState { if perr != nil { continue } - s.last[target] = provenTier{Archive: cur.Archive, At: t.UTC()} + s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()} } return s } -// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed. Only call this for a -// PASSING restore-test — the archive is what makes the tier not-due, so recording one for a failed -// run would retire the archive unproven. -func (s *RestoreTestState) RecordSuccess(target, archive string, t time.Time) error { +// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run +// reported, and what it verified. Only call this for a PASSING restore-test — the archive is what +// makes the tier not-due, so recording one for a failed run would retire the archive unproven. +// +// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because +// the next reader will notice failures are absent and try to "fix" it: +// +// a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves +// the system quietly less tested than it believes. It must survive a restart. +// +// a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation, +// so a lost failure heals itself within one interval. Persisting it would do the opposite of +// helping: a healed tier would keep reporting a failure that is no longer true. +func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error { if target == "" { return nil } s.mu.Lock() defer s.mu.Unlock() - s.last[target] = provenTier{Archive: archive, At: t.UTC()} + s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()} return s.saveLocked() } @@ -138,7 +180,13 @@ func (s *RestoreTestState) ProvenArchive(target string) (string, bool) { return p.Archive, true } -// Snapshot returns a copy of the last-proven TIMES — for the host-report gauge. +// Snapshot returns a copy of the last-proven TIMES. +// +// It carried the comment "for the host-report gauge" from the day it was written and **had no caller +// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with +// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which +// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want +// only the times; if it acquires none, delete it rather than let it claim a purpose again. func (s *RestoreTestState) Snapshot() map[string]time.Time { s.mu.Lock() defer s.mu.Unlock() @@ -149,6 +197,41 @@ func (s *RestoreTestState) Snapshot() map[string]time.Time { return out } +// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix. +// +// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory +// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis: +// +// - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer); +// - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported; +// - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration +// is not a claim; a fabricated one would be. +// +// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable. +// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a +// worse defect than the one this fixes. +func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest { + s.mu.Lock() + defer s.mu.Unlock() + out := make([]hub.RestoreTest, 0, len(s.last)) + for _, p := range s.last { + if !p.reportable() { + continue + } + out = append(out, hub.RestoreTest{ + SourceArchive: p.Archive, + SourceTier: p.Tier, + Pass: true, + Verified: p.Verified, + TestedAt: p.At.UTC().Format(time.RFC3339), + }) + } + // Deterministic order: the report is compared byte-wise by the contract test, and Go's map + // iteration is randomised. + sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier }) + return out +} + // OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST. // // This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it @@ -185,7 +268,10 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string { func (s *RestoreTestState) saveLocked() error { raw := make(map[string]provenTierJSON, len(s.last)) for target, p := range s.last { - raw[target] = provenTierJSON{Archive: p.Archive, ProvenAt: p.At.UTC().Format(time.RFC3339)} + raw[target] = provenTierJSON{ + Archive: p.Archive, Tier: p.Tier, Verified: p.Verified, + ProvenAt: p.At.UTC().Format(time.RFC3339), + } } data, err := json.MarshalIndent(raw, "", " ") if err != nil { diff --git a/internal/backup/rotation_test.go b/internal/backup/rotation_test.go index 4e34543..4614ee0 100644 --- a/internal/backup/rotation_test.go +++ b/internal/backup/rotation_test.go @@ -303,11 +303,11 @@ func TestOldestFirst_Ordering(t *testing.T) { t.Fatalf("unexpected: %v", got) } } - _ = st.RecordSuccess("local", "local:backup/a.tar.zst", now) + _ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now) if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" { t.Fatalf("a never-proven tier must sort before a proven one; got %v", got) } - _ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", now.Add(time.Hour)) + _ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour)) if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" { t.Fatalf("the least recently proven must sort first; got %v", got) } @@ -318,8 +318,8 @@ func TestOldestFirst_Ordering(t *testing.T) { func TestOldestFirst_DeterministicOnTies(t *testing.T) { st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")) now := time.Now().UTC() - _ = st.RecordSuccess("b-tier", "b:archive", now) - _ = st.RecordSuccess("a-tier", "a:archive", now) + _ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now) + _ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now) for i := 0; i < 20; i++ { if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" { t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got) @@ -334,7 +334,7 @@ func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) { now := time.Now().UTC().Truncate(time.Second) st := NewRestoreTestState(path) - if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil { + if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil { t.Fatal(err) } reopened := NewRestoreTestState(path) diff --git a/internal/backup/schedule.go b/internal/backup/schedule.go index 2feec29..402f727 100644 --- a/internal/backup/schedule.go +++ b/internal/backup/schedule.go @@ -218,7 +218,11 @@ func (s *Scheduler) tick(ctx context.Context) { if rt.Pass && s.rtState != nil && target != "" { // R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier // not-due until a NEWER archive settles, and what makes a proof survive a restart. - if err := s.rtState.RecordSuccess(target, archive, s.now()); err != nil { + // R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a + // restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what + // this run was actually judged as, and deriving it later would need a storage lookup that + // can fail on the one path where failing means mislabelling a proof. + if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil { s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err) } } diff --git a/internal/backup/store.go b/internal/backup/store.go index 6005059..8b4de60 100644 --- a/internal/backup/store.go +++ b/internal/backup/store.go @@ -10,8 +10,22 @@ import ( // Store holds the agent's LATEST backup result per target and the latest restore-test // result — the point-in-time state the host-report surfaces. It is updated by the backup // runner + the restore-test scheduler/selftest and read by the collector via the hub -// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence -// re-populates) and mutex-guarded for the concurrent collector vs scheduler access. +// BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent +// collector vs scheduler access. +// +// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now +// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier +// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already +// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on +// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a +// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two +// consecutive host-reports carried `0 restore-tests`. +// +// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector +// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is +// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost +// failure heals itself, while a lost success leaves the system quietly less tested than it believes. +// Backups are unaffected — their freshness has a ground truth on the storage (R-84). type Store struct { mu sync.Mutex byTarget map[string]hub.Backup // latest backup per target id diff --git a/internal/hub/collect.go b/internal/hub/collect.go index 7f90636..0ecb6bf 100644 --- a/internal/hub/collect.go +++ b/internal/hub/collect.go @@ -47,6 +47,22 @@ type RestoreTestReporter interface { RestoreTests(ctx context.Context) []RestoreTest } +// ProvenRestoreTestReporter is the DURABLE half of the restore-test signal (R-189). +// +// RestoreTestReporter above is backed by an in-memory store whose own comment used to read "lost on +// restart; the cadence re-populates". That was true while a timer re-tested every tier daily. It +// stopped being true on 2026-08-03: under per-archive due-ness the agent will not re-test an archive +// it has already proven, so a proof lost to a restart is not repeated for a whole archive generation +// — a week on the offsite tier — and the hub reports the tier unproven the entire time. +// +// Observed, not predicted: a real 14.5 GB offsite restore passed at 15:25:14, the agent was restarted +// 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two reports. +// +// (*backup.RestoreTestState).ProvenRestoreTests satisfies this. nil → the merge is a no-op. +type ProvenRestoreTestReporter interface { + ProvenRestoreTests(ctx context.Context) []RestoreTest +} + // PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern). // Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty. type PBSReporter interface { @@ -79,6 +95,7 @@ type Collector struct { storage StorageObserver backups BackupReporter restoreTests RestoreTestReporter + provenTests ProvenRestoreTestReporter pbs PBSReporter temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp) capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty) @@ -427,16 +444,90 @@ func (c *Collector) collectBackups(ctx context.Context) []Backup { return []Backup{} } +// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189). +// +// The rule is ONE ENTRY PER TIER, NEWEST WINS, and it falls out of what each source means rather +// than from a preference between them: +// +// - the in-memory store holds this process's latest run, pass OR fail. A failure exists nowhere +// else and must always reach the hub — a failing tier is retried at the next evaluation, so its +// record is short-lived by design; +// - the persisted state holds the last SUCCESS per tier and survives a restart. +// +// Comparing by TestedAt gives the right answer in every case without special-casing: a fresh failure +// beats an older stored success (the failure is the news), a stored success beats a stale in-memory +// entry after a restart, and a tier proved twice never appears twice — two entries for one tier would +// read at the hub as two tests. +// +// A tier with no usable persisted proof contributes NOTHING. Reporting an unproven tier as proven +// would be a worse defect than the one this closes. func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest { - if c.restoreTests == nil { - return []RestoreTest{} + out := []RestoreTest{} + if c.restoreTests != nil { + if r := c.restoreTests.RestoreTests(ctx); r != nil { + out = append(out, r...) + } } - if r := c.restoreTests.RestoreTests(ctx); r != nil { - return r + if c.provenTests == nil { + return out } - return []RestoreTest{} + + // Index what we already have by tier, keeping the newest per tier. + best := map[string]int{} // tier → index into out + for i, rt := range out { + if rt.SourceTier == "" { + continue // untiered entry: never deduped, never overwritten — we cannot say what it is + } + if j, seen := best[rt.SourceTier]; !seen || newerRestoreTest(rt, out[j]) { + best[rt.SourceTier] = i + } + } + for _, p := range c.provenTests.ProvenRestoreTests(ctx) { + if p.SourceTier == "" { + continue // not usable as a per-tier proof; the state layer already filters these + } + i, seen := best[p.SourceTier] + if !seen { + out = append(out, p) + best[p.SourceTier] = len(out) - 1 + continue + } + if newerRestoreTest(p, out[i]) { + out[i] = p + } + } + return out } +// newerRestoreTest reports whether a was tested after b. An unparseable or absent timestamp is +// treated as OLDER, so a malformed entry can never displace a good one. +func newerRestoreTest(a, b RestoreTest) bool { + ta, aok := parseRestoreTestedAt(a.TestedAt) + tb, bok := parseRestoreTestedAt(b.TestedAt) + if !aok { + return false + } + if !bok { + return true + } + return ta.After(tb) +} + +func parseRestoreTestedAt(s string) (time.Time, bool) { + t, err := time.Parse(time.RFC3339, s) + if err != nil { + return time.Time{}, false + } + return t.UTC(), true +} + +// SetProvenRestoreTests wires the durable proof source. It is a setter rather than a constructor +// argument because the persisted state is opened later in main() than the collector is built; the +// same shape as the other late-wired seams here. **The wiring is asserted by an AST test** — the +// method it feeds carried a doc comment naming a "host-report gauge" for weeks with no caller at +// all, and this fix must not become the next instance of that. +func (c *Collector) SetProvenRestoreTests(p ProvenRestoreTestReporter) { c.provenTests = p } + // collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty). func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot { if c.pbs == nil { diff --git a/internal/hub/restoretest_merge_test.go b/internal/hub/restoretest_merge_test.go new file mode 100644 index 0000000..d98ab4b --- /dev/null +++ b/internal/hub/restoretest_merge_test.go @@ -0,0 +1,213 @@ +package hub + +import ( + "context" + "testing" + "time" +) + +// R-189 — a passing restore-test must survive an agent restart and reach the hub. +// +// THE OBSERVATION THIS EXISTS FOR (2026-08-03, demo-felhom): a real 14.5 GB offsite restore-test +// PASSED at 15:25:14; the agent was restarted 2 m 43 s later for a deploy; the hub logged +// `0 restore-tests` on the next two host-reports. The in-memory store's own comment said "lost on +// restart; the cadence re-populates", which was true under a timer and stopped being true when R-86 +// made the agent refuse to re-test an archive it has already proven. +// +// Timestamps here carry JITTER (odd minutes and seconds, not round hours) — yesterday a test was +// hollow because a perfectly regular series landed exactly on a threshold and passed under the +// mutation it was meant to catch. + +type fakeLatest struct{ tests []RestoreTest } + +func (f *fakeLatest) RestoreTests(context.Context) []RestoreTest { return f.tests } + +type fakeProven struct{ tests []RestoreTest } + +func (f *fakeProven) ProvenRestoreTests(context.Context) []RestoreTest { return f.tests } + +func rt(tier, archive string, pass bool, at time.Time) RestoreTest { + return RestoreTest{ + SourceArchive: archive, SourceTier: tier, Pass: pass, + Verified: "boot+running", TestedAt: at.UTC().Format(time.RFC3339), + } +} + +// mergeCollector builds a Collector with only the two restore-test seams wired — the merge is what +// is under test, not the rest of the collection. +func mergeCollector(latest, proven []RestoreTest) *Collector { + c := &Collector{} + if latest != nil { + c.restoreTests = &fakeLatest{tests: latest} + } + if proven != nil { + c.provenTests = &fakeProven{tests: proven} + } + return c +} + +func findTier(got []RestoreTest, tier string) (RestoreTest, int) { + var hit RestoreTest + n := 0 + for _, e := range got { + if e.SourceTier == tier { + hit, n = e, n+1 + } + } + return hit, n +} + +// ── SCENARIO A — a proof survives a restart and reaches the hub ────────────────────────────── +// +// COMPANION RED-PROOF (observed 2026-08-03): delete the `c.provenTests` merge from +// collectRestoreTests (return the in-memory slice as it used to) → +// +// --- FAIL: TestMerge_ProofSurvivesARestart +// restoretest_merge_test.go: after a restart the persisted proof must be reported; got 0 entr(ies) +// +// which is exactly the live observation: `0 restore-tests`. Restored. +func TestMerge_ProofSurvivesARestart(t *testing.T) { + provenAt := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC) // the real run's timestamp + // After a restart the in-memory store is EMPTY — this is the whole point. + c := mergeCollector([]RestoreTest{}, []RestoreTest{ + rt("pbs", "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", true, provenAt), + }) + + got := c.collectRestoreTests(context.Background()) + if len(got) != 1 { + t.Fatalf("after a restart the persisted proof must be reported; got %d entr(ies): %+v", len(got), got) + } + e := got[0] + if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" { + t.Fatalf("the entry must name the archive that was proven — the hub keys on it; got %q", e.SourceArchive) + } + if e.SourceTier != "pbs" || !e.Pass { + t.Fatalf("the entry must be a PASS on the tier it was proven on; got tier=%q pass=%v", e.SourceTier, e.Pass) + } + if e.TestedAt != provenAt.Format(time.RFC3339) { + t.Fatalf("the entry must carry the ORIGINAL test time, not now(); got %q", e.TestedAt) + } +} + +// ── SCENARIO B — the report does not invent a pass ─────────────────────────────────────────── +// +// COMPANION RED-PROOF (observed): make the state layer emit an entry for an unproven tier (drop the +// `reportable()` filter in ProvenRestoreTests, so a legacy record with no archive is emitted) — the +// equivalent at this layer is a proven-source that returns an entry for a tier nothing proved, which +// this test injects directly and the assertion below rejects. +func TestMerge_NeverInventsAPassForAnUnprovenTier(t *testing.T) { + // Nothing proven anywhere: no in-memory result, no persisted proof. + c := mergeCollector([]RestoreTest{}, []RestoreTest{}) + if got := c.collectRestoreTests(context.Background()); len(got) != 0 { + t.Fatalf("a tier with no proof must produce NO entry — an unproven tier reading as proven is "+ + "worse than the defect being fixed; got %+v", got) + } + + // And an entry the state layer could not describe (no tier) is never promoted into a proof. + c2 := mergeCollector([]RestoreTest{}, []RestoreTest{ + {SourceArchive: "local:backup/x.tar.zst", SourceTier: "", Pass: true, + TestedAt: time.Date(2026, 8, 1, 4, 41, 58, 0, time.UTC).Format(time.RFC3339)}, + }) + if got := c2.collectRestoreTests(context.Background()); len(got) != 0 { + t.Fatalf("a persisted record with no tier is not a usable proof and must be dropped; got %+v", got) + } +} + +// ── SCENARIO C — a fresh in-memory result wins, and never duplicates ───────────────────────── +// +// COMPANION RED-PROOF (observed 2026-08-03): remove the de-duplication (append every persisted entry +// unconditionally) → +// +// --- FAIL: TestMerge_NewerWinsAndNeverDuplicatesATier +// restoretest_merge_test.go: one entry per tier; got 2 for "pbs" — the hub would read two tests +// +// Restored. +func TestMerge_NewerWinsAndNeverDuplicatesATier(t *testing.T) { + lastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC) // jittered, from the real box + fiveMinAgo := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC) + + c := mergeCollector( + []RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)}, + []RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)}, + ) + got := c.collectRestoreTests(context.Background()) + e, n := findTier(got, "pbs") + if n != 1 { + t.Fatalf("one entry per tier; got %d for \"pbs\" — the hub would read two tests: %+v", n, got) + } + if e.SourceArchive != "felhom-pbs:backup/ct/9201/new" { + t.Fatalf("the NEWER result must win; got %q tested %q", e.SourceArchive, e.TestedAt) + } + + // ...and the older-in-memory / newer-persisted direction, which is the post-restart case. + c2 := mergeCollector( + []RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)}, + []RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)}, + ) + e2, n2 := findTier(c2.collectRestoreTests(context.Background()), "pbs") + if n2 != 1 || e2.SourceArchive != "felhom-pbs:backup/ct/9201/new" { + t.Fatalf("newest must win regardless of which source it came from; got %d entr(ies), archive %q", n2, e2.SourceArchive) + } +} + +// ── SCENARIO D — a failure still reaches the hub ───────────────────────────────────────────── +// +// The merge must not mask a failure with an older stored success. A failing tier is retried at the +// next evaluation and its record lives ONLY in memory, so losing it here would silence the loudest +// DR signal this system produces. +func TestMerge_AFailureIsStillReported(t *testing.T) { + provenLastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC) + failedJustNow := time.Date(2026, 8, 3, 13, 41, 7, 0, time.UTC) + + c := mergeCollector( + []RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", false, failedJustNow)}, + []RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, provenLastWeek)}, + ) + e, n := findTier(c.collectRestoreTests(context.Background()), "pbs") + if n != 1 { + t.Fatalf("one entry per tier; got %d: %+v", n, c.collectRestoreTests(context.Background())) + } + if e.Pass { + t.Fatalf("a FAILURE newer than the stored proof must be what is reported — masking it would "+ + "silence the loudest DR signal there is; got pass=%v archive=%q", e.Pass, e.SourceArchive) + } +} + +// Two different tiers are both reported — the merge is per tier, not a single slot. +func TestMerge_BothTiersSurvive(t *testing.T) { + c := mergeCollector( + []RestoreTest{rt("local", "felhom-backup:backup/vzdump-lxc-9201-a.tar.zst", true, + time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))}, + []RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/x", true, + time.Date(2026, 8, 2, 5, 12, 33, 0, time.UTC))}, + ) + got := c.collectRestoreTests(context.Background()) + if _, n := findTier(got, "local"); n != 1 { + t.Fatalf("the in-memory tier must survive the merge; got %+v", got) + } + if _, n := findTier(got, "pbs"); n != 1 { + t.Fatalf("the persisted tier must survive the merge; got %+v", got) + } +} + +// A malformed timestamp must never displace a good entry — "unparseable" is not "newest". +func TestMerge_MalformedTimestampNeverWins(t *testing.T) { + good := rt("pbs", "felhom-pbs:backup/ct/9201/good", true, time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC)) + bad := RestoreTest{SourceArchive: "felhom-pbs:backup/ct/9201/bad", SourceTier: "pbs", Pass: true, TestedAt: "not-a-time"} + + c := mergeCollector([]RestoreTest{good}, []RestoreTest{bad}) + e, n := findTier(c.collectRestoreTests(context.Background()), "pbs") + if n != 1 || e.SourceArchive != "felhom-pbs:backup/ct/9201/good" { + t.Fatalf("an unparseable timestamp must not displace a good entry; got %d entr(ies), archive %q", n, e.SourceArchive) + } +} + +// A nil proven-source leaves the pre-R-189 behaviour exactly as it was. +func TestMerge_NilProvenSourceIsANoOp(t *testing.T) { + only := rt("local", "felhom-backup:backup/x.tar.zst", true, time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC)) + c := mergeCollector([]RestoreTest{only}, nil) + got := c.collectRestoreTests(context.Background()) + if len(got) != 1 || got[0].SourceArchive != only.SourceArchive { + t.Fatalf("a nil durable source must not change anything; got %+v", got) + } +} diff --git a/scripts/check-published-versions.py b/scripts/check-published-versions.py index d7a2bcd..821183c 100644 --- a/scripts/check-published-versions.py +++ b/scripts/check-published-versions.py @@ -14,10 +14,32 @@ Nothing in the build, deploy or session-end path checked that either existed, so documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge agent — and would have *succeeded* while doing it. -THE INVARIANT, AND WHY IT IS THIS ONE. +THE INVARIANTS — there are TWO now, and the second is R-188's price. - For every `v` git tag in this repo: the matching generic package must be DOWNLOADABLE, - and the tag must serve the agent's configs. + (1) For every `v` git tag in this repo: the matching generic package must be DOWNLOADABLE, + and the tag must serve the agent's configs. + + (2) No PUBLISHED version may be missing its tag. + +Invariant (2) is new (R-188, 2026-08-03) and it exists because `release-agent.sh` now pushes the tag +AFTER publishing. The old order pushed the tag first, and the old comment said why: a tag with no +package is caught here, a package with no tag is invisible, because the Gitea package LISTING api +needs a token this gate does not have. That reasoning was sound and the ordering was still wrong — +the tag push is what wakes CI, so every correct release had a ~50% chance of running this gate in the +seconds before its own package existed and mailing the operator a failure for a release that worked +(measured across two releases: runs 12/13 and 17/18, same shas, opposite results). + +Moving the push does not get to trade invariant (2) away, so it is asserted here instead — WITHOUT a +token, and therefore as a BOUNDED PROBE rather than an enumeration: + + * the FRONTIER — the versions immediately above the highest tag. This is the realistic failure the + new ordering makes possible: publish succeeds, tag push fails, so the orphan is exactly one + version beyond the newest tag. + * the GAPS — patch versions that fall between two existing tags and have no tag of their own. + +Re-measured 2026-08-03, not assumed: `GET /api/v1/packages/admin?type=generic` answers **401** with no +token, so absence still cannot be proven. The probe set is PRINTED on every run, because a check whose +coverage is invisible reads as a guarantee it is not making. The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be downloadable"* — and that is the better invariant in principle. **It is not implementable from CI, @@ -36,6 +58,8 @@ v0.120.0, which is published. **What it does NOT catch, stated plainly:** the hub vouching a version that was never released at all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. → R-184. +Nor does the converse probe prove that NO untagged package exists — only that none exists at the +probed versions, which are printed. Closing that properly needs a read token in CI (→ R-184). FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2 INCONCLUSIVE**, naming every URL tried — never a pass. "Cannot determine" is not "fine": that is the @@ -48,7 +72,7 @@ carrying python3 and git and nothing else, and an earlier workflow step died on python3 scripts/check-published-versions.py -Exit: 0 every tag installable · 1 at least one is not · 2 could not be determined. +Exit: 0 both invariants hold · 1 either is violated · 2 could not be determined. Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL). """ import json @@ -94,6 +118,53 @@ def inconclusive(msg): sys.exit(2) +def _pkg_exists(version): + """True iff the generic package for `version` is downloadable anonymously.""" + url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, version, PKG) + status, _ = _get(url) + return status == 200, url + + +def untagged_probe_set(versions): + """The versions to probe for invariant (2), as (version, why) pairs. + + Bounded on purpose and printed by the caller: the package listing api needs a token (401, + re-measured 2026-08-03), so absence cannot be enumerated. What CAN be done is to probe the + places an orphan would actually land. + + FRONTIER — a publish that succeeded followed by a tag push that failed leaves the orphan + exactly one version past the newest tag. This is the failure mode the R-188 + reordering makes possible, so it is the one that must not be guesswork. + GAPS — a patch number skipped between two consecutive tags. Bounded per gap so a typo'd + tag (v0.130.0 after v0.121.1) cannot turn this into a thousand requests. + """ + parsed = sorted(tuple(int(p) for p in v.split(".")) for v in versions) + have = set(parsed) + out = [] + if not parsed: + return out + + hi = parsed[-1] + for cand, why in ( + ((hi[0], hi[1], hi[2] + 1), "next patch after the newest tag"), + ((hi[0], hi[1], hi[2] + 2), "second patch after the newest tag"), + ((hi[0], hi[1] + 1, 0), "next minor after the newest tag"), + ((hi[0] + 1, 0, 0), "next major after the newest tag"), + ): + if cand not in have: + out.append(("%d.%d.%d" % cand, why)) + + MAX_GAP_PROBES = 12 + for a, b in zip(parsed, parsed[1:]): + if a[0] != b[0] or a[1] != b[1]: + continue # a minor/major step is not a patch gap + for patch in range(a[2] + 1, min(b[2], a[2] + 1 + MAX_GAP_PROBES)): + cand = (a[0], a[1], patch) + if cand not in have: + out.append(("%d.%d.%d" % cand, "patch gap between v%d.%d.%d and v%d.%d.%d" % (a + b))) + return out + + def main(): print("check-published-versions — every released agent version must be INSTALLABLE") print(" gitea:", GITEA_BASE) @@ -144,14 +215,42 @@ def main(): else: print(" ok v%s: binary downloadable + tag serves its configs" % v) + # ── invariant (2): no PUBLISHED version may be missing its tag (R-188) ────────────────────── + probes = untagged_probe_set(versions) + orphans = [] print() + print(" converse probe — a published version with no tag (bounded; the package listing api") + print(" needs a token, so this cannot enumerate). Probing %d version(s):" % len(probes)) + for v, why in probes: + try: + exists, url = _pkg_exists(v) + except Exception as e: + inconclusive("network failure while probing v%s: %s" % (v, e)) + mark = "PUBLISHED — NO TAG" if exists else "absent (ok)" + print(" %-10s %-42s %s" % (v, why, mark)) + if exists: + orphans.append((v, url)) + + print() + if bad or orphans: + if orphans: + print("check-published-versions: %d PUBLISHED VERSION(S) WITH NO TAG" % len(orphans)) + for v, url in orphans: + print(" v%s is downloadable at %s but has no git tag." % (v, url)) + print(" A release publishes and then pushes its tag; a package with no tag means the") + print(" push failed or was skipped. The local tag is probably still in the release") + print(" clone — finish it with:") + for v, _ in orphans: + print(" git push origin v%s" % v) + print(" (and if the tag is gone, re-create it on the released commit before pushing.)") if bad: print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad)) print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —") print(" the R-115 defect, three times in five days. Publish it with:") print(" scripts/release-agent.sh ") + if bad or orphans: return 1 - print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE") + print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED") return 0 diff --git a/scripts/publish-agent.sh b/scripts/publish-agent.sh index 618ce2c..eb7812c 100755 --- a/scripts/publish-agent.sh +++ b/scripts/publish-agent.sh @@ -51,7 +51,11 @@ if [[ -z "$BIN" ]]; then BIN="$(mktemp -t felhom-agent.XXXXXX)" CLEANUP_BIN="$BIN" log "building felhom-agent $VERSION from $REPO_ROOT …" - ( cd "$REPO_ROOT" && CGO_ENABLED=0 go build -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent ) + # These flags MUST match release-agent.sh's build exactly — see the long comment there (R-186). + # They used to differ: this line forced CGO_ENABLED=0 and produced a binary 74 KB smaller than + # the one the release path built for the same version. One version name must mean one binary + # whichever entry point produced it. + ( cd "$REPO_ROOT" && go build -trimpath -buildvcs=false -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent ) fi [[ -f "$BIN" ]] || die "binary not found: $BIN" trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT diff --git a/scripts/release-agent.sh b/scripts/release-agent.sh index 4b55d2d..048b792 100755 --- a/scripts/release-agent.sh +++ b/scripts/release-agent.sh @@ -74,7 +74,25 @@ existing="$(curl -fsS -o /dev/null -w '%{http_code}' \ BIN="$(mktemp -t felhom-agent-XXXXXX)" trap 'rm -f "$BIN"' EXIT log "building $VERSION …" -go build -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \ +# REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and +# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS +# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries. +# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags: +# +# default flags, no tag yet .. 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61) +# default flags, tagged ...... 4a38f394… 14 085 440 B (mod v0.121.99) +# -trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways +# +# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller, +# verified) and the version comes from the explicit ldflag below, which is where it belongs. +# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also +# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens. +# +# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force +# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same +# version — one version name, two binaries, by whichever entry point was used. Both now build the +# same way; if that ever has to change, change it in BOTH or the guarantee is gone. +go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \ || die "go build failed" built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')" [[ "$built_ver" == "$VERSION" ]] \ @@ -82,10 +100,27 @@ built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')" BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')" log "built ok: sha256 $BUILT_SHA" -# ── 4. Tag (before publishing, so a published version always has a tag) ───────────────────────── -# Order matters in this direction only: a tag with no package is caught by -# scripts/check-published-versions.py on the next CI run; a package with no tag is invisible to it, +# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ────────────────────────────── +# +# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03). +# +# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no +# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it, # because the Gitea package LISTING api needs a token the gate does not have. +# +# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in +# the seconds between the tag becoming visible and the package existing — and correctly failed. Every +# correct release had roughly a coin-flip chance of emailing the operator a failure for a release +# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18 +# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be +# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too. +# +# So the tag is still created HERE, before anything is published — the build and the tag still +# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone. +# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package +# is downloadable. The invariant the old order protected is not traded away: it is asserted directly +# by the gate's new converse probe (a published version with no tag FAILS), so both directions are +# now checked rather than one being arranged for. log "tagging $TAG at $(git rev-parse --short HEAD) …" git tag -a "$TAG" -m "agent $TAG @@ -94,7 +129,6 @@ sha256 of the published binary: $BUILT_SHA felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/, so this tag is part of the released artifact, not a bookmark (R-183)." -git push origin "$TAG" || die "tag push failed — refusing to publish an untagged version" # ── 5. Publish (the existing script; deliberately not reimplemented) ──────────────────────────── log "publishing …" @@ -104,9 +138,48 @@ log "publishing …" # R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in # the same commit; this line makes the release independent of it, because a file mode is exactly the # kind of thing that is lost again by a checkout, an archive, or a copy. -bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN" || die "publish failed" +if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then + # The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next + # attempt would die at step 2's "tag $TAG already exists" and read as "this version is already + # released", which would be exactly backwards. Only remove it if nothing was in fact published — + # if a package DOES exist, the tag is wanted and must be pushed, not deleted. + now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \ + "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)" + if [[ "$now_published" == "200" ]]; then + log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG" + else + git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried" + fi + die "publish failed" +fi -# ── 6. Verify by an INDEPENDENT download ──────────────────────────────────────────────────────── +# ── 6. Push the tag, now that the package exists ──────────────────────────────────────────────── +# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v/` fetch +# the installer makes. It runs last of the two so CI can never see a tag whose package is not there. +# +# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the +# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches. +# The recovery is one line and it is printed rather than described. +log "pushing $TAG …" +if ! git push origin "$TAG"; then + cat >&2 <