v0.122.0: three ways the signals lied about themselves (R-189, R-188, R-186)

All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.

R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.

R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.

R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
This commit is contained in:
2026-08-03 16:40:18 +02:00
parent 3d0a1d615d
commit ed0cb7f20c
16 changed files with 895 additions and 42 deletions
+81
View File
@@ -1,3 +1,84 @@
## v0.122.0 — three ways the signals lied about themselves (2026-08-03, R-189 · R-188 · R-186)
All three are the reporting and release path misreporting its own work. **No customer machine, no
backup, no restore, no disk layout, no data.** The restore-test itself and when it runs are unchanged
from v0.121.1.
### R-189 — a passing restore-test no longer vanishes on a restart
`restore_tests[]` came only from the in-memory `backup.Store`, whose own comment read *"lost on
restart; the cadence re-populates"*. That was true under a timer. It stopped being true when R-86 made
the agent refuse to re-test an archive it has already proven: a proof lost to a restart is not
repeated for a whole archive generation — **a week on the offsite tier** — and the hub calls the tier
unproven for all of it.
**Observed, not predicted (2026-08-03):** a real 14.5 GB offsite restore-test PASSED at 15:25:14, the
agent was restarted 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two
host-reports.
The durable proof already existed — `RestoreTestState`, on disk, per tier, with the archive since
R-86 — and `Snapshot()` had carried the doc comment *"for the host-report gauge"* since the day it was
written **with no caller at all**: a seam built, documented, and never connected. It now carries the
`tier` and what was `verified` as well (stored at proof time, when they are known for certain, rather
than derived later by a storage lookup that can fail), and `ProvenRestoreTests` renders them as report
entries which the collector merges.
- **Merge rule: one entry per tier, newest by `TestedAt` wins.** A fresh failure beats a stored
success — the failure is the news and lives nowhere else; a stored success beats a stale in-memory
entry after a restart; a tier never appears twice, which the hub would read as two tests. An
unparseable timestamp counts as older, so a malformed entry cannot displace a good one.
- **It refuses to lie.** A record missing the archive or the tier produces NO entry, and run mechanics
(scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one
would be. An unproven tier reading as proven would be worse than the defect being fixed.
- **Only successes are persisted, and that asymmetry is now written down where it will be read:** a
success suppresses future work, so losing it leaves the system quietly less tested than it believes;
a failure causes future work and heals itself at the next evaluation.
- The `Store` comment that stopped being true is corrected in place rather than left to mislead.
### R-188 — a correct release no longer emails a failure
`on: [push]` fires the gates workflow on the **tag** push, and the release pushed its tag *before*
publishing, so CI ran the published-versions gate in the seconds before the package existed and
correctly reported it missing. Measured across two releases in one session: runs 12/13 and 17/18, same
sha each time, opposite results — a race, not a rule. R-168 made that mail the thing that cannot be
missed; one that is wrong half the time is one you stop reading.
**Only the tag PUSH moved** (build → tag locally → publish → push tag). The tag is still created before
anything is published, so the build and the tag still describe the same commit; it simply becomes
*visible* — to CI, and to any `raw/tag/…` fetch — once the package is downloadable.
The invariant the old order protected is **not traded away**: `check-published-versions.py` now asserts
the converse directly — **no published version may be missing its tag** — as a bounded probe of the
frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set every
run because a check whose coverage is invisible reads as a guarantee it is not making. The package
listing api still answers **401** without a token (re-measured), so absence cannot be enumerated, and
the script says so.
A publish that succeeds and a tag push that then fails now **dies loudly**, printing the one-line
recovery; and a publish that *fails* removes the local-only tag so the release can simply be retried
instead of colliding with itself.
### R-186 — a released binary can now be verified by rebuilding it
`go build` stamps a module version derived from VCS state, so a build made before the tag existed and
a rebuild made after it were different binaries. Measured at one commit, same source, same toolchain:
```
default flags, no tag yet ... 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
default flags, tagged ....... 4a38f394… 14 085 440 B (mod v0.121.99)
-trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
```
The stamp is removed rather than sequenced around — nothing in this repo reads it (no `ReadBuildInfo`
caller) and the version comes from the explicit `-X main.version` ldflag. `-trimpath` additionally
makes a rebuild from a different checkout directory match.
**A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and
therefore produced a binary **74 KB smaller** than the release path built for the same version — one
version name, two binaries, decided by which entry point was used. Both now build identically.
`CLAUDE.md` records the exact command an operator can run to verify a published binary independently.
## v0.121.1 — "nothing is due" must be AUDIBLE (2026-08-03, R-86 + standing rule 3) ## v0.121.1 — "nothing is due" must be AUDIBLE (2026-08-03, R-86 + standing rule 3)
**Found while live-validating v0.121.0, and it is this project's own rule pointed at the change that **Found while live-validating v0.121.0, and it is this project's own rule pointed at the change that
+27
View File
@@ -72,6 +72,32 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
> verifies by an **independent download** rather than trusting the publish step's own output. > verifies by an **independent download** rather than trusting the publish step's own output.
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather > `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
> than reimplementing it. > than reimplementing it.
>
> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188,
> R-186).** The tag is created before the publish so the build and the tag describe the same commit;
> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
> package makes the published-versions gate correctly fail a correct release — it did, on roughly
> every second release, and R-168 sends that failure to you by mail. The invariant the old order
> protected is asserted directly instead: the gate now also refuses a **published version with no
> tag**. If the push fails after a successful publish the script says so loudly and prints the
> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean.
>
> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath
> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet —
> before this, a rebuild could not reproduce the sha you were vouching. To check any published
> version yourself:
>
> ```bash
> V=0.122.0
> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
> -o /tmp/felhom-agent-check ./cmd/felhom-agent
> sha256sum /tmp/felhom-agent-check
> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
> ```
>
> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to
> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line
> ever changes, change both or one version name means two binaries again.
| Step | Where | One-liner | | Step | Where | One-liner |
|---|---|---| |---|---|---|
@@ -79,6 +105,7 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) | | Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
| Deploy | felhom-pve | backup `.bak-<old>``install -m0755``systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) | | Deploy | felhom-pve | backup `.bak-<old>``install -m0755``systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed | | Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) | | **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) | | Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
+25
View File
@@ -5,6 +5,31 @@
## Current ## Current
- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
None touches data; all three cost attention, which every other signal depends on.
- **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
still read, and a record missing archive-or-tier is NOT reported), exposes
`ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
"host-report gauge" in its doc comment and had **no caller** for weeks.
- **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
- **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
made the gate correctly fail a correct release ~half the time. The old order's invariant is now
asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
- **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
`CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
identical. The verification command is in `CLAUDE.md`.
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is - **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive; `settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
+2 -1
View File
@@ -148,7 +148,8 @@
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go | | `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). | | `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,t)` / `ProvenArchive(target)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. | | `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)`**`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. | | `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)`**`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). | | `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go | | `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
+6
View File
@@ -662,6 +662,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json")) rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
heavyOps := &backup.InFlight{} heavyOps := &backup.InFlight{}
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger) scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
// holds only this process's latest run, and under per-archive due-ness the agent will not
// re-test an archive it has already proven — so without this the hub can report a tier unproven
// for a whole archive generation after a deploy. Observed live on 2026-08-03: a passing 14.5 GB
// offsite restore-test reached no host-report at all.
collector.SetProvenRestoreTests(rtState)
// PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free, // PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free,
// ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot // ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot
@@ -111,3 +111,46 @@ func parseMainForWiring(t *testing.T) *ast.File {
} }
return f return f
} }
// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector.
//
// This test exists because the method it feeds is the project's own cautionary tale:
// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it
// was written and **had no caller at all** — a seam built, documented and never connected, found
// only when a live restore-test's PASS reached no host-report. The fix must not become the next
// instance, so the wiring is asserted rather than trusted.
//
// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting
// out the tier-picker line failed this test while a `strings.Contains` check would have passed).
func TestMainWiresTheDurableRestoreTestProof(t *testing.T) {
f := parseMainForWiring(t)
var wired, feedsState bool
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "SetProvenRestoreTests" {
return true
}
wired = true
// ...and it must be fed the PERSISTED state, not the in-memory store.
if len(call.Args) == 1 {
if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" {
feedsState = true
}
}
return true
})
if !wired {
t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " +
"reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart")
}
if wired && !feedsState {
t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " +
"that does NOT survive a restart, so wiring it here would fix nothing")
}
}
+88 -2
View File
@@ -442,7 +442,7 @@ func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json") path := filepath.Join(t.TempDir(), "rt.json")
now := time.Now().UTC().Truncate(time.Second) now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path) st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil { if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
t.Fatal(err) t.Fatal(err)
} }
re := NewRestoreTestState(path) re := NewRestoreTestState(path)
@@ -475,7 +475,7 @@ func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) {
}} }}
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts) h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
// Prove the local tier so NOTHING is due. // Prove the local tier so NOTHING is due.
if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", h.clock); err != nil { if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -505,3 +505,89 @@ func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) {
t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got) t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got)
} }
} }
// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ───────────────────
//
// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the
// agent will not repeat the work. So the persisted record has to be able to become a host-report
// entry — without inventing anything it does not know.
//
// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from
// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted →
//
// --- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe
// restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its
// per-tier proof on it); got [{... SourceTier: ...}]
//
// Restored.
func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
// v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this
// file has ever had, which is what a real box carries after two upgrades.
legacy := `{
"old-v1": "2026-07-30T02:11:07Z",
"old-v2": {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"},
"felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"}
}`
if err := writeFileForTest(path, legacy); err != nil {
t.Fatal(err)
}
got := NewRestoreTestState(path).ProvenRestoreTests(context.Background())
if len(got) != 1 {
t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got)
}
e := got[0]
if e.SourceTier != "pbs" {
t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got)
}
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass {
t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e)
}
if e.TestedAt != "2026-08-03T13:25:14Z" {
t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt)
}
if e.Verified != "boot+running" {
t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified)
}
// Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be.
if e.DurationSeconds != 0 || e.ScratchVMID != 0 {
t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d",
e.DurationSeconds, e.ScratchVMID)
}
// The legacy records still serve the DUE-check, which is a separate question from reporting.
if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok {
t.Fatal("a v2 record must still answer the due-check even though it cannot be reported")
}
}
// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report —
// the production path, not a hand-built fixture.
func TestScheduler_ProofIsRecordedReportably(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
// The fake runner echoes the spec's tier; give the spec a tier the way main.go does.
h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"}
}
h.s.tick(context.Background())
got := h.st.ProvenRestoreTests(context.Background())
if len(got) != 1 {
t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got)
}
if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" {
t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0])
}
}
// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed.
func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts)
h.s.tick(context.Background())
if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 {
t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+
"would outlive the fault; got %+v", got)
}
}
+99 -13
View File
@@ -1,12 +1,15 @@
package backup package backup
import ( import (
"context"
"encoding/json" "encoding/json"
"os" "os"
"path/filepath" "path/filepath"
"sort" "sort"
"sync" "sync"
"time" "time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
) )
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier. // RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
@@ -49,16 +52,45 @@ type RestoreTestState struct {
last map[string]provenTier // target id → what was last PROVEN on that tier last map[string]provenTier // target id → what was last PROVEN on that tier
} }
// provenTier is one tier's proof: the archive that passed, and when it passed. // provenTier is one tier's proof: the archive that passed, which tier it was, what was verified,
// and when.
//
// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not
// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result
// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the
// hub can stay ignorant of a real success until the next archive generation.
//
// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own
// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup
// at report-building time — a network call that can fail, on a path where failing means mis-labelling
// a proof. Store what you knew when you knew it.
type provenTier struct { type provenTier struct {
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
At time.Time // when that run passed (UTC) Tier string // "local" | "pbs" — as the run reported it; "" = pre-R-189 record
Verified string // what the run verified (e.g. "boot+running"); "" = pre-R-189 record
At time.Time // when that run passed (UTC)
} }
// provenTierJSON is the on-disk shape (R-86). The legacy shape was a bare RFC3339 STRING per // reportable reports whether this record can be re-reported to the hub as a restore-test result.
// target; both are read, only this one is written — see NewRestoreTestState. //
// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the
// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable
// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence.
// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in.
func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" }
// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written:
//
// v1 (pre-R-86) "<target>": "<RFC3339>" — a time, no archive
// v2 (R-86) "<target>": {archive, proven_at} — due-check usable, not reportable
// v3 (R-189) "<target>": {archive, tier, verified, …} — both
//
// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the
// readers above test for — the migration needs no version number because the absence IS the answer.
type provenTierJSON struct { type provenTierJSON struct {
Archive string `json:"archive"` Archive string `json:"archive"`
Tier string `json:"tier,omitempty"`
Verified string `json:"verified,omitempty"`
ProvenAt string `json:"proven_at"` ProvenAt string `json:"proven_at"`
} }
@@ -99,21 +131,31 @@ func NewRestoreTestState(path string) *RestoreTestState {
if perr != nil { if perr != nil {
continue continue
} }
s.last[target] = provenTier{Archive: cur.Archive, At: t.UTC()} s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()}
} }
return s return s
} }
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed. Only call this for a // RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run
// PASSING restore-test — the archive is what makes the tier not-due, so recording one for a failed // reported, and what it verified. Only call this for a PASSING restore-test — the archive is what
// run would retire the archive unproven. // makes the tier not-due, so recording one for a failed run would retire the archive unproven.
func (s *RestoreTestState) RecordSuccess(target, archive string, t time.Time) error { //
// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because
// the next reader will notice failures are absent and try to "fix" it:
//
// a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves
// the system quietly less tested than it believes. It must survive a restart.
//
// a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation,
// so a lost failure heals itself within one interval. Persisting it would do the opposite of
// helping: a healed tier would keep reporting a failure that is no longer true.
func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error {
if target == "" { if target == "" {
return nil return nil
} }
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
s.last[target] = provenTier{Archive: archive, At: t.UTC()} s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()}
return s.saveLocked() return s.saveLocked()
} }
@@ -138,7 +180,13 @@ func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
return p.Archive, true return p.Archive, true
} }
// Snapshot returns a copy of the last-proven TIMES — for the host-report gauge. // Snapshot returns a copy of the last-proven TIMES.
//
// It carried the comment "for the host-report gauge" from the day it was written and **had no caller
// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with
// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which
// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want
// only the times; if it acquires none, delete it rather than let it claim a purpose again.
func (s *RestoreTestState) Snapshot() map[string]time.Time { func (s *RestoreTestState) Snapshot() map[string]time.Time {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
@@ -149,6 +197,41 @@ func (s *RestoreTestState) Snapshot() map[string]time.Time {
return out return out
} }
// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix.
//
// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory
// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis:
//
// - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer);
// - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported;
// - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration
// is not a claim; a fabricated one would be.
//
// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable.
// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a
// worse defect than the one this fixes.
func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest {
s.mu.Lock()
defer s.mu.Unlock()
out := make([]hub.RestoreTest, 0, len(s.last))
for _, p := range s.last {
if !p.reportable() {
continue
}
out = append(out, hub.RestoreTest{
SourceArchive: p.Archive,
SourceTier: p.Tier,
Pass: true,
Verified: p.Verified,
TestedAt: p.At.UTC().Format(time.RFC3339),
})
}
// Deterministic order: the report is compared byte-wise by the contract test, and Go's map
// iteration is randomised.
sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier })
return out
}
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST. // OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
// //
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it // This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
@@ -185,7 +268,10 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
func (s *RestoreTestState) saveLocked() error { func (s *RestoreTestState) saveLocked() error {
raw := make(map[string]provenTierJSON, len(s.last)) raw := make(map[string]provenTierJSON, len(s.last))
for target, p := range s.last { for target, p := range s.last {
raw[target] = provenTierJSON{Archive: p.Archive, ProvenAt: p.At.UTC().Format(time.RFC3339)} raw[target] = provenTierJSON{
Archive: p.Archive, Tier: p.Tier, Verified: p.Verified,
ProvenAt: p.At.UTC().Format(time.RFC3339),
}
} }
data, err := json.MarshalIndent(raw, "", " ") data, err := json.MarshalIndent(raw, "", " ")
if err != nil { if err != nil {
+5 -5
View File
@@ -303,11 +303,11 @@ func TestOldestFirst_Ordering(t *testing.T) {
t.Fatalf("unexpected: %v", got) t.Fatalf("unexpected: %v", got)
} }
} }
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", now) _ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now)
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" { if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got) t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
} }
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", now.Add(time.Hour)) _ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour))
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" { if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
t.Fatalf("the least recently proven must sort first; got %v", got) t.Fatalf("the least recently proven must sort first; got %v", got)
} }
@@ -318,8 +318,8 @@ func TestOldestFirst_Ordering(t *testing.T) {
func TestOldestFirst_DeterministicOnTies(t *testing.T) { func TestOldestFirst_DeterministicOnTies(t *testing.T) {
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json")) st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
now := time.Now().UTC() now := time.Now().UTC()
_ = st.RecordSuccess("b-tier", "b:archive", now) _ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now)
_ = st.RecordSuccess("a-tier", "a:archive", now) _ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now)
for i := 0; i < 20; i++ { for i := 0; i < 20; i++ {
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" { if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got) t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
@@ -334,7 +334,7 @@ func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
now := time.Now().UTC().Truncate(time.Second) now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path) st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", now); err != nil { if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
t.Fatal(err) t.Fatal(err)
} }
reopened := NewRestoreTestState(path) reopened := NewRestoreTestState(path)
+5 -1
View File
@@ -218,7 +218,11 @@ func (s *Scheduler) tick(ctx context.Context) {
if rt.Pass && s.rtState != nil && target != "" { if rt.Pass && s.rtState != nil && target != "" {
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier // R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
// not-due until a NEWER archive settles, and what makes a proof survive a restart. // not-due until a NEWER archive settles, and what makes a proof survive a restart.
if err := s.rtState.RecordSuccess(target, archive, s.now()); err != nil { // R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a
// restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what
// this run was actually judged as, and deriving it later would need a storage lookup that
// can fail on the one path where failing means mislabelling a proof.
if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil {
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err) s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
} }
} }
+16 -2
View File
@@ -10,8 +10,22 @@ import (
// Store holds the agent's LATEST backup result per target and the latest restore-test // Store holds the agent's LATEST backup result per target and the latest restore-test
// result — the point-in-time state the host-report surfaces. It is updated by the backup // result — the point-in-time state the host-report surfaces. It is updated by the backup
// runner + the restore-test scheduler/selftest and read by the collector via the hub // runner + the restore-test scheduler/selftest and read by the collector via the hub
// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence // BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent
// re-populates) and mutex-guarded for the concurrent collector vs scheduler access. // collector vs scheduler access.
//
// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now
// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier
// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already
// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on
// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a
// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two
// consecutive host-reports carried `0 restore-tests`.
//
// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
type Store struct { type Store struct {
mu sync.Mutex mu sync.Mutex
byTarget map[string]hub.Backup // latest backup per target id byTarget map[string]hub.Backup // latest backup per target id
+96 -5
View File
@@ -47,6 +47,22 @@ type RestoreTestReporter interface {
RestoreTests(ctx context.Context) []RestoreTest RestoreTests(ctx context.Context) []RestoreTest
} }
// ProvenRestoreTestReporter is the DURABLE half of the restore-test signal (R-189).
//
// RestoreTestReporter above is backed by an in-memory store whose own comment used to read "lost on
// restart; the cadence re-populates". That was true while a timer re-tested every tier daily. It
// stopped being true on 2026-08-03: under per-archive due-ness the agent will not re-test an archive
// it has already proven, so a proof lost to a restart is not repeated for a whole archive generation
// — a week on the offsite tier — and the hub reports the tier unproven the entire time.
//
// Observed, not predicted: a real 14.5 GB offsite restore passed at 15:25:14, the agent was restarted
// 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two reports.
//
// (*backup.RestoreTestState).ProvenRestoreTests satisfies this. nil → the merge is a no-op.
type ProvenRestoreTestReporter interface {
ProvenRestoreTests(ctx context.Context) []RestoreTest
}
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern). // PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty. // Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
type PBSReporter interface { type PBSReporter interface {
@@ -79,6 +95,7 @@ type Collector struct {
storage StorageObserver storage StorageObserver
backups BackupReporter backups BackupReporter
restoreTests RestoreTestReporter restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
pbs PBSReporter pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp) temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty) capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
@@ -427,16 +444,90 @@ func (c *Collector) collectBackups(ctx context.Context) []Backup {
return []Backup{} return []Backup{}
} }
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
//
// The rule is ONE ENTRY PER TIER, NEWEST WINS, and it falls out of what each source means rather
// than from a preference between them:
//
// - the in-memory store holds this process's latest run, pass OR fail. A failure exists nowhere
// else and must always reach the hub — a failing tier is retried at the next evaluation, so its
// record is short-lived by design;
// - the persisted state holds the last SUCCESS per tier and survives a restart.
//
// Comparing by TestedAt gives the right answer in every case without special-casing: a fresh failure
// beats an older stored success (the failure is the news), a stored success beats a stale in-memory
// entry after a restart, and a tier proved twice never appears twice — two entries for one tier would
// read at the hub as two tests.
//
// A tier with no usable persisted proof contributes NOTHING. Reporting an unproven tier as proven
// would be a worse defect than the one this closes.
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest { func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
if c.restoreTests == nil { out := []RestoreTest{}
return []RestoreTest{} if c.restoreTests != nil {
if r := c.restoreTests.RestoreTests(ctx); r != nil {
out = append(out, r...)
}
} }
if r := c.restoreTests.RestoreTests(ctx); r != nil { if c.provenTests == nil {
return r return out
} }
return []RestoreTest{}
// Index what we already have by tier, keeping the newest per tier.
best := map[string]int{} // tier → index into out
for i, rt := range out {
if rt.SourceTier == "" {
continue // untiered entry: never deduped, never overwritten — we cannot say what it is
}
if j, seen := best[rt.SourceTier]; !seen || newerRestoreTest(rt, out[j]) {
best[rt.SourceTier] = i
}
}
for _, p := range c.provenTests.ProvenRestoreTests(ctx) {
if p.SourceTier == "" {
continue // not usable as a per-tier proof; the state layer already filters these
}
i, seen := best[p.SourceTier]
if !seen {
out = append(out, p)
best[p.SourceTier] = len(out) - 1
continue
}
if newerRestoreTest(p, out[i]) {
out[i] = p
}
}
return out
} }
// newerRestoreTest reports whether a was tested after b. An unparseable or absent timestamp is
// treated as OLDER, so a malformed entry can never displace a good one.
func newerRestoreTest(a, b RestoreTest) bool {
ta, aok := parseRestoreTestedAt(a.TestedAt)
tb, bok := parseRestoreTestedAt(b.TestedAt)
if !aok {
return false
}
if !bok {
return true
}
return ta.After(tb)
}
func parseRestoreTestedAt(s string) (time.Time, bool) {
t, err := time.Parse(time.RFC3339, s)
if err != nil {
return time.Time{}, false
}
return t.UTC(), true
}
// SetProvenRestoreTests wires the durable proof source. It is a setter rather than a constructor
// argument because the persisted state is opened later in main() than the collector is built; the
// same shape as the other late-wired seams here. **The wiring is asserted by an AST test** — the
// method it feeds carried a doc comment naming a "host-report gauge" for weeks with no caller at
// all, and this fix must not become the next instance of that.
func (c *Collector) SetProvenRestoreTests(p ProvenRestoreTestReporter) { c.provenTests = p }
// collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty). // collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty).
func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot { func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
if c.pbs == nil { if c.pbs == nil {
+213
View File
@@ -0,0 +1,213 @@
package hub
import (
"context"
"testing"
"time"
)
// R-189 — a passing restore-test must survive an agent restart and reach the hub.
//
// THE OBSERVATION THIS EXISTS FOR (2026-08-03, demo-felhom): a real 14.5 GB offsite restore-test
// PASSED at 15:25:14; the agent was restarted 2 m 43 s later for a deploy; the hub logged
// `0 restore-tests` on the next two host-reports. The in-memory store's own comment said "lost on
// restart; the cadence re-populates", which was true under a timer and stopped being true when R-86
// made the agent refuse to re-test an archive it has already proven.
//
// Timestamps here carry JITTER (odd minutes and seconds, not round hours) — yesterday a test was
// hollow because a perfectly regular series landed exactly on a threshold and passed under the
// mutation it was meant to catch.
type fakeLatest struct{ tests []RestoreTest }
func (f *fakeLatest) RestoreTests(context.Context) []RestoreTest { return f.tests }
type fakeProven struct{ tests []RestoreTest }
func (f *fakeProven) ProvenRestoreTests(context.Context) []RestoreTest { return f.tests }
func rt(tier, archive string, pass bool, at time.Time) RestoreTest {
return RestoreTest{
SourceArchive: archive, SourceTier: tier, Pass: pass,
Verified: "boot+running", TestedAt: at.UTC().Format(time.RFC3339),
}
}
// mergeCollector builds a Collector with only the two restore-test seams wired — the merge is what
// is under test, not the rest of the collection.
func mergeCollector(latest, proven []RestoreTest) *Collector {
c := &Collector{}
if latest != nil {
c.restoreTests = &fakeLatest{tests: latest}
}
if proven != nil {
c.provenTests = &fakeProven{tests: proven}
}
return c
}
func findTier(got []RestoreTest, tier string) (RestoreTest, int) {
var hit RestoreTest
n := 0
for _, e := range got {
if e.SourceTier == tier {
hit, n = e, n+1
}
}
return hit, n
}
// ── SCENARIO A — a proof survives a restart and reaches the hub ──────────────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): delete the `c.provenTests` merge from
// collectRestoreTests (return the in-memory slice as it used to) →
//
// --- FAIL: TestMerge_ProofSurvivesARestart
// restoretest_merge_test.go: after a restart the persisted proof must be reported; got 0 entr(ies)
//
// which is exactly the live observation: `0 restore-tests`. Restored.
func TestMerge_ProofSurvivesARestart(t *testing.T) {
provenAt := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC) // the real run's timestamp
// After a restart the in-memory store is EMPTY — this is the whole point.
c := mergeCollector([]RestoreTest{}, []RestoreTest{
rt("pbs", "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", true, provenAt),
})
got := c.collectRestoreTests(context.Background())
if len(got) != 1 {
t.Fatalf("after a restart the persisted proof must be reported; got %d entr(ies): %+v", len(got), got)
}
e := got[0]
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" {
t.Fatalf("the entry must name the archive that was proven — the hub keys on it; got %q", e.SourceArchive)
}
if e.SourceTier != "pbs" || !e.Pass {
t.Fatalf("the entry must be a PASS on the tier it was proven on; got tier=%q pass=%v", e.SourceTier, e.Pass)
}
if e.TestedAt != provenAt.Format(time.RFC3339) {
t.Fatalf("the entry must carry the ORIGINAL test time, not now(); got %q", e.TestedAt)
}
}
// ── SCENARIO B — the report does not invent a pass ───────────────────────────────────────────
//
// COMPANION RED-PROOF (observed): make the state layer emit an entry for an unproven tier (drop the
// `reportable()` filter in ProvenRestoreTests, so a legacy record with no archive is emitted) — the
// equivalent at this layer is a proven-source that returns an entry for a tier nothing proved, which
// this test injects directly and the assertion below rejects.
func TestMerge_NeverInventsAPassForAnUnprovenTier(t *testing.T) {
// Nothing proven anywhere: no in-memory result, no persisted proof.
c := mergeCollector([]RestoreTest{}, []RestoreTest{})
if got := c.collectRestoreTests(context.Background()); len(got) != 0 {
t.Fatalf("a tier with no proof must produce NO entry — an unproven tier reading as proven is "+
"worse than the defect being fixed; got %+v", got)
}
// And an entry the state layer could not describe (no tier) is never promoted into a proof.
c2 := mergeCollector([]RestoreTest{}, []RestoreTest{
{SourceArchive: "local:backup/x.tar.zst", SourceTier: "", Pass: true,
TestedAt: time.Date(2026, 8, 1, 4, 41, 58, 0, time.UTC).Format(time.RFC3339)},
})
if got := c2.collectRestoreTests(context.Background()); len(got) != 0 {
t.Fatalf("a persisted record with no tier is not a usable proof and must be dropped; got %+v", got)
}
}
// ── SCENARIO C — a fresh in-memory result wins, and never duplicates ─────────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): remove the de-duplication (append every persisted entry
// unconditionally) →
//
// --- FAIL: TestMerge_NewerWinsAndNeverDuplicatesATier
// restoretest_merge_test.go: one entry per tier; got 2 for "pbs" — the hub would read two tests
//
// Restored.
func TestMerge_NewerWinsAndNeverDuplicatesATier(t *testing.T) {
lastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC) // jittered, from the real box
fiveMinAgo := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC)
c := mergeCollector(
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
)
got := c.collectRestoreTests(context.Background())
e, n := findTier(got, "pbs")
if n != 1 {
t.Fatalf("one entry per tier; got %d for \"pbs\" — the hub would read two tests: %+v", n, got)
}
if e.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
t.Fatalf("the NEWER result must win; got %q tested %q", e.SourceArchive, e.TestedAt)
}
// ...and the older-in-memory / newer-persisted direction, which is the post-restart case.
c2 := mergeCollector(
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
)
e2, n2 := findTier(c2.collectRestoreTests(context.Background()), "pbs")
if n2 != 1 || e2.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
t.Fatalf("newest must win regardless of which source it came from; got %d entr(ies), archive %q", n2, e2.SourceArchive)
}
}
// ── SCENARIO D — a failure still reaches the hub ─────────────────────────────────────────────
//
// The merge must not mask a failure with an older stored success. A failing tier is retried at the
// next evaluation and its record lives ONLY in memory, so losing it here would silence the loudest
// DR signal this system produces.
func TestMerge_AFailureIsStillReported(t *testing.T) {
provenLastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC)
failedJustNow := time.Date(2026, 8, 3, 13, 41, 7, 0, time.UTC)
c := mergeCollector(
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", false, failedJustNow)},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, provenLastWeek)},
)
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
if n != 1 {
t.Fatalf("one entry per tier; got %d: %+v", n, c.collectRestoreTests(context.Background()))
}
if e.Pass {
t.Fatalf("a FAILURE newer than the stored proof must be what is reported — masking it would "+
"silence the loudest DR signal there is; got pass=%v archive=%q", e.Pass, e.SourceArchive)
}
}
// Two different tiers are both reported — the merge is per tier, not a single slot.
func TestMerge_BothTiersSurvive(t *testing.T) {
c := mergeCollector(
[]RestoreTest{rt("local", "felhom-backup:backup/vzdump-lxc-9201-a.tar.zst", true,
time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/x", true,
time.Date(2026, 8, 2, 5, 12, 33, 0, time.UTC))},
)
got := c.collectRestoreTests(context.Background())
if _, n := findTier(got, "local"); n != 1 {
t.Fatalf("the in-memory tier must survive the merge; got %+v", got)
}
if _, n := findTier(got, "pbs"); n != 1 {
t.Fatalf("the persisted tier must survive the merge; got %+v", got)
}
}
// A malformed timestamp must never displace a good entry — "unparseable" is not "newest".
func TestMerge_MalformedTimestampNeverWins(t *testing.T) {
good := rt("pbs", "felhom-pbs:backup/ct/9201/good", true, time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC))
bad := RestoreTest{SourceArchive: "felhom-pbs:backup/ct/9201/bad", SourceTier: "pbs", Pass: true, TestedAt: "not-a-time"}
c := mergeCollector([]RestoreTest{good}, []RestoreTest{bad})
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
if n != 1 || e.SourceArchive != "felhom-pbs:backup/ct/9201/good" {
t.Fatalf("an unparseable timestamp must not displace a good entry; got %d entr(ies), archive %q", n, e.SourceArchive)
}
}
// A nil proven-source leaves the pre-R-189 behaviour exactly as it was.
func TestMerge_NilProvenSourceIsANoOp(t *testing.T) {
only := rt("local", "felhom-backup:backup/x.tar.zst", true, time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))
c := mergeCollector([]RestoreTest{only}, nil)
got := c.collectRestoreTests(context.Background())
if len(got) != 1 || got[0].SourceArchive != only.SourceArchive {
t.Fatalf("a nil durable source must not change anything; got %+v", got)
}
}
+104 -5
View File
@@ -14,10 +14,32 @@ Nothing in the build, deploy or session-end path checked that either existed, so
documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
agent and would have *succeeded* while doing it. agent and would have *succeeded* while doing it.
THE INVARIANT, AND WHY IT IS THIS ONE. THE INVARIANTS there are TWO now, and the second is R-188's price.
For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE, (1) For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
and the tag must serve the agent's configs. and the tag must serve the agent's configs.
(2) No PUBLISHED version may be missing its tag.
Invariant (2) is new (R-188, 2026-08-03) and it exists because `release-agent.sh` now pushes the tag
AFTER publishing. The old order pushed the tag first, and the old comment said why: a tag with no
package is caught here, a package with no tag is invisible, because the Gitea package LISTING api
needs a token this gate does not have. That reasoning was sound and the ordering was still wrong
the tag push is what wakes CI, so every correct release had a ~50% chance of running this gate in the
seconds before its own package existed and mailing the operator a failure for a release that worked
(measured across two releases: runs 12/13 and 17/18, same shas, opposite results).
Moving the push does not get to trade invariant (2) away, so it is asserted here instead WITHOUT a
token, and therefore as a BOUNDED PROBE rather than an enumeration:
* the FRONTIER the versions immediately above the highest tag. This is the realistic failure the
new ordering makes possible: publish succeeds, tag push fails, so the orphan is exactly one
version beyond the newest tag.
* the GAPS patch versions that fall between two existing tags and have no tag of their own.
Re-measured 2026-08-03, not assumed: `GET /api/v1/packages/admin?type=generic` answers **401** with no
token, so absence still cannot be proven. The probe set is PRINTED on every run, because a check whose
coverage is invisible reads as a guarantee it is not making.
The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
downloadable"* — and that is the better invariant in principle. **It is not implementable from CI, downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
@@ -36,6 +58,8 @@ v0.120.0, which is published.
**What it does NOT catch, stated plainly:** the hub vouching a version that was never released at **What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. R-184. all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. R-184.
Nor does the converse probe prove that NO untagged package exists only that none exists at the
probed versions, which are printed. Closing that properly needs a read token in CI ( R-184).
FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2 FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
INCONCLUSIVE**, naming every URL tried never a pass. "Cannot determine" is not "fine": that is the INCONCLUSIVE**, naming every URL tried never a pass. "Cannot determine" is not "fine": that is the
@@ -48,7 +72,7 @@ carrying python3 and git and nothing else, and an earlier workflow step died on
python3 scripts/check-published-versions.py python3 scripts/check-published-versions.py
Exit: 0 every tag installable · 1 at least one is not · 2 could not be determined. Exit: 0 both invariants hold · 1 either is violated · 2 could not be determined.
Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL). Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
""" """
import json import json
@@ -94,6 +118,53 @@ def inconclusive(msg):
sys.exit(2) sys.exit(2)
def _pkg_exists(version):
"""True iff the generic package for `version` is downloadable anonymously."""
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, version, PKG)
status, _ = _get(url)
return status == 200, url
def untagged_probe_set(versions):
"""The versions to probe for invariant (2), as (version, why) pairs.
Bounded on purpose and printed by the caller: the package listing api needs a token (401,
re-measured 2026-08-03), so absence cannot be enumerated. What CAN be done is to probe the
places an orphan would actually land.
FRONTIER a publish that succeeded followed by a tag push that failed leaves the orphan
exactly one version past the newest tag. This is the failure mode the R-188
reordering makes possible, so it is the one that must not be guesswork.
GAPS a patch number skipped between two consecutive tags. Bounded per gap so a typo'd
tag (v0.130.0 after v0.121.1) cannot turn this into a thousand requests.
"""
parsed = sorted(tuple(int(p) for p in v.split(".")) for v in versions)
have = set(parsed)
out = []
if not parsed:
return out
hi = parsed[-1]
for cand, why in (
((hi[0], hi[1], hi[2] + 1), "next patch after the newest tag"),
((hi[0], hi[1], hi[2] + 2), "second patch after the newest tag"),
((hi[0], hi[1] + 1, 0), "next minor after the newest tag"),
((hi[0] + 1, 0, 0), "next major after the newest tag"),
):
if cand not in have:
out.append(("%d.%d.%d" % cand, why))
MAX_GAP_PROBES = 12
for a, b in zip(parsed, parsed[1:]):
if a[0] != b[0] or a[1] != b[1]:
continue # a minor/major step is not a patch gap
for patch in range(a[2] + 1, min(b[2], a[2] + 1 + MAX_GAP_PROBES)):
cand = (a[0], a[1], patch)
if cand not in have:
out.append(("%d.%d.%d" % cand, "patch gap between v%d.%d.%d and v%d.%d.%d" % (a + b)))
return out
def main(): def main():
print("check-published-versions — every released agent version must be INSTALLABLE") print("check-published-versions — every released agent version must be INSTALLABLE")
print(" gitea:", GITEA_BASE) print(" gitea:", GITEA_BASE)
@@ -144,14 +215,42 @@ def main():
else: else:
print(" ok v%s: binary downloadable + tag serves its configs" % v) print(" ok v%s: binary downloadable + tag serves its configs" % v)
# ── invariant (2): no PUBLISHED version may be missing its tag (R-188) ──────────────────────
probes = untagged_probe_set(versions)
orphans = []
print() print()
print(" converse probe — a published version with no tag (bounded; the package listing api")
print(" needs a token, so this cannot enumerate). Probing %d version(s):" % len(probes))
for v, why in probes:
try:
exists, url = _pkg_exists(v)
except Exception as e:
inconclusive("network failure while probing v%s: %s" % (v, e))
mark = "PUBLISHED — NO TAG" if exists else "absent (ok)"
print(" %-10s %-42s %s" % (v, why, mark))
if exists:
orphans.append((v, url))
print()
if bad or orphans:
if orphans:
print("check-published-versions: %d PUBLISHED VERSION(S) WITH NO TAG" % len(orphans))
for v, url in orphans:
print(" v%s is downloadable at %s but has no git tag." % (v, url))
print(" A release publishes and then pushes its tag; a package with no tag means the")
print(" push failed or was skipped. The local tag is probably still in the release")
print(" clone — finish it with:")
for v, _ in orphans:
print(" git push origin v%s" % v)
print(" (and if the tag is gone, re-create it on the released commit before pushing.)")
if bad: if bad:
print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad)) print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —") print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —")
print(" the R-115 defect, three times in five days. Publish it with:") print(" the R-115 defect, three times in five days. Publish it with:")
print(" scripts/release-agent.sh <version>") print(" scripts/release-agent.sh <version>")
if bad or orphans:
return 1 return 1
print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE") print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED")
return 0 return 0
+5 -1
View File
@@ -51,7 +51,11 @@ if [[ -z "$BIN" ]]; then
BIN="$(mktemp -t felhom-agent.XXXXXX)" BIN="$(mktemp -t felhom-agent.XXXXXX)"
CLEANUP_BIN="$BIN" CLEANUP_BIN="$BIN"
log "building felhom-agent $VERSION from $REPO_ROOT" log "building felhom-agent $VERSION from $REPO_ROOT"
( cd "$REPO_ROOT" && CGO_ENABLED=0 go build -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent ) # These flags MUST match release-agent.sh's build exactly — see the long comment there (R-186).
# They used to differ: this line forced CGO_ENABLED=0 and produced a binary 74 KB smaller than
# the one the release path built for the same version. One version name must mean one binary
# whichever entry point produced it.
( cd "$REPO_ROOT" && go build -trimpath -buildvcs=false -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
fi fi
[[ -f "$BIN" ]] || die "binary not found: $BIN" [[ -f "$BIN" ]] || die "binary not found: $BIN"
trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT
+80 -7
View File
@@ -74,7 +74,25 @@ existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
BIN="$(mktemp -t felhom-agent-XXXXXX)" BIN="$(mktemp -t felhom-agent-XXXXXX)"
trap 'rm -f "$BIN"' EXIT trap 'rm -f "$BIN"' EXIT
log "building $VERSION" log "building $VERSION"
go build -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \ # REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and
# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS
# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries.
# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags:
#
# default flags, no tag yet .. 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
# default flags, tagged ...... 4a38f394… 14 085 440 B (mod v0.121.99)
# -trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
#
# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller,
# verified) and the version comes from the explicit ldflag below, which is where it belongs.
# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also
# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens.
#
# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force
# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same
# version — one version name, two binaries, by whichever entry point was used. Both now build the
# same way; if that ever has to change, change it in BOTH or the guarantee is gone.
go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|| die "go build failed" || die "go build failed"
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')" built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
[[ "$built_ver" == "$VERSION" ]] \ [[ "$built_ver" == "$VERSION" ]] \
@@ -82,10 +100,27 @@ built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')" BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
log "built ok: sha256 $BUILT_SHA" log "built ok: sha256 $BUILT_SHA"
# ── 4. Tag (before publishing, so a published version always has a tag) ───────────────────────── # ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
# Order matters in this direction only: a tag with no package is caught by #
# scripts/check-published-versions.py on the next CI run; a package with no tag is invisible to it, # THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
#
# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no
# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it,
# because the Gitea package LISTING api needs a token the gate does not have. # because the Gitea package LISTING api needs a token the gate does not have.
#
# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in
# the seconds between the tag becoming visible and the package existing — and correctly failed. Every
# correct release had roughly a coin-flip chance of emailing the operator a failure for a release
# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18
# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be
# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too.
#
# So the tag is still created HERE, before anything is published — the build and the tag still
# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone.
# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package
# is downloadable. The invariant the old order protected is not traded away: it is asserted directly
# by the gate's new converse probe (a published version with no tag FAILS), so both directions are
# now checked rather than one being arranged for.
log "tagging $TAG at $(git rev-parse --short HEAD)" log "tagging $TAG at $(git rev-parse --short HEAD)"
git tag -a "$TAG" -m "agent $TAG git tag -a "$TAG" -m "agent $TAG
@@ -94,7 +129,6 @@ sha256 of the published binary: $BUILT_SHA
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/, felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
so this tag is part of the released artifact, not a bookmark (R-183)." so this tag is part of the released artifact, not a bookmark (R-183)."
git push origin "$TAG" || die "tag push failed — refusing to publish an untagged version"
# ── 5. Publish (the existing script; deliberately not reimplemented) ──────────────────────────── # ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
log "publishing …" log "publishing …"
@@ -104,9 +138,48 @@ log "publishing …"
# R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in # R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
# the same commit; this line makes the release independent of it, because a file mode is exactly the # the same commit; this line makes the release independent of it, because a file mode is exactly the
# kind of thing that is lost again by a checkout, an archive, or a copy. # kind of thing that is lost again by a checkout, an archive, or a copy.
bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN" || die "publish failed" if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
# The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next
# attempt would die at step 2's "tag $TAG already exists" and read as "this version is already
# released", which would be exactly backwards. Only remove it if nothing was in fact published —
# if a package DOES exist, the tag is wanted and must be pushed, not deleted.
now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
if [[ "$now_published" == "200" ]]; then
log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG"
else
git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried"
fi
die "publish failed"
fi
# ── 6. Verify by an INDEPENDENT download ──────────────────────────────────────────────────────── # ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
#
# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the
# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches.
# The recovery is one line and it is printed rather than described.
log "pushing $TAG"
if ! git push origin "$TAG"; then
cat >&2 <<EOF
RELEASE HALF DONE — the package is PUBLISHED and its tag is NOT pushed.
version : $VERSION
sha256 : $BUILT_SHA
The tag exists in this clone only. Nothing installs from an untagged version (the installer
fetches this version's configs from raw/tag/$TAG/), and scripts/check-published-versions.py will
FAIL on it as a published version with no tag. Finish the release with:
git push origin $TAG
EOF
die "tag push failed after a successful publish — see above"
fi
# ── 7. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
# The publish step's own success is not proof: it reports on its own write. What matters is that a # The publish step's own success is not proof: it reports on its own write. What matters is that a
# box can now GET the bytes and that they are the bytes that were built. This is the same # box can now GET the bytes and that they are the bytes that were built. This is the same
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that # presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that