Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
construction. After it, 'nothing is due' is the NORMAL outcome — and it was
logged at DEBUG, which journald drops. An empty journal would then be equally
consistent with a healthy loop and a dead goroutine: the shape the R-88 watcher
was retired for, re-created by making the quiet path the common one.
A not-due evaluation now logs one INFO line naming every tier's verdict (four
lines a day at the 6h default), and an unlistable tier reads UNKNOWN with its
error in that same line, so a lookup failure can never present as 'nothing due'.
Red-proved through the scheduler's own tick, not the helper.
2026-08-03 15:26:54 +02:00
5 changed files with 373 additions and 93 deletions
So a credential-free gate can ask *"is this version installable"* but not *"which version is
vouched"*. Adding an operator credential to CI to close that is the operator's call, not a gate
author's. The implemented invariant — **every `v<semver>` tag must have a downloadable package and a
tag tree that serves the agent's configs** — needs no credential and **catches all three recorded
instances**, because the release script creates the tag and publishes in one act.
**What it does not catch, stated rather than assumed away:** the hub vouching a version that was
never released at all. Nothing here can see that; it belongs at vouch time in the hub. → **R-184**.
## Proof
| Check | Result |
|---|---|
| `go build ./... && go vet ./...` | OK |
| `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
| `agent_gates.py --fast` | `published` correctly **SKIPPED** — the pre-push hook must not fail because Gitea blinked |
| `agent_gates.py` (full) | `reuse-refs` OK, `published` OK |
| release script: re-release guard | `ERROR: tag v0.120.0 already exists — releasing over it would make one version name two binaries`, rc=1 |
| release script: clean-tree guard | `ERROR: working tree is dirty — commit and push first`, rc=1 |
### Red-proof F — both directions
- **A tagged-but-unpublished version** (`v9.9.9` created for the purpose): gate **rc=1**,
`binary NOT downloadable (HTTP 404 …)`. This is the R-115 shape exactly.
- **The gate deregistered from the entry point**, same bad state: `agent_gates.py` → **rc=0, "all
agent gates OK"**. Restored → **rc=1, CONVICTED: published**. The guard is what catches it, not
something else.
### Scenario F measured on REAL CI, not inferred
Runs **69** and **70** are on the **same commit**`0db7766`:
| run | state of the repo | CI |
| Piece | File | Change |
|---|---|---|
| 69 | no `v9.9.9` | **success** |
| 70 | `v9.9.9` tagged, not published | **failure** |
| the due-check | `internal/backup/restoretest_due.go` (new) | `EvaluateDue` / `evaluateTier` — per-tier verdict + the reason, ordered oldest-proven first |
| the trigger | `internal/backup/schedule.go` | the ticker is now the **evaluation interval**; `pickForThisRun` answers *"is anything due?"*, and "nothing" is a normal answer |
| the state | `internal/backup/restoretest_state.go` | records **which archive** was proven, with migration |
| the picker | `internal/backup/runner.go` | `PickSettledRestoreCandidateOn(ctx, target, notAfter)`; `PickRestoreCandidateOn` is a one-line call into it |
| the knobs | `internal/config/config.go` | `restore_test_eval_interval_seconds` + `restore_test_settle_seconds`; the old key deprecated, not repurposed |
| the wiring | `cmd/felhom-agent/main.go` | settle-aware picker + `Settle`; deprecation WARN; new `--selftest=restore-test-due` |
| the observable (**v0.121.1**) | `internal/backup/schedule.go`, `restoretest_due.go` | a not-due evaluation logs one **INFO** line naming every tier's verdict — see §9 |
Same code, same workflow, one variable — so the gate demonstrably RUNS in CI and fails for exactly
the R-115 condition. This also retrospectively explains runs 67/68, which were red in the window when
`v9.9.9` first existed. **One deliberate CI failure e-mail reached the operator — that was this
proof, not an incident.**
### The state records the archive (§8.2)
I could not read CI's own step log to attribute those runs directly: the Gitea jobs endpoint requires
an API token, and the only credential available on this host (`~/.docker/config.json`) is a registry
password, which the API rejects. The controlled before/after above replaced that log rather than an
assumption standing in for it.
A timestamp cannot answer *"have we proven **this** archive"* — it is the same class as the workspace
rule that a timestamp records an *attempt*, not a *result*: here it records a result, but not **which**
result. `RestoreTestState` now holds `{archive, proven_at}` per tier.
`v9.9.9` was deleted afterwards; `git ls-remote --tags` shows only `v0.120.0`.
**Migration:** a pre-R-86 file (`{"target": "<RFC3339>"}`) keeps its **time** — rotation ordering
survives a deploy, which is why the file exists at all — and yields **no proven archive**, so each
tier is due exactly once after the upgrade. One extra test per tier, once, is the safe direction;
reading a legacy time as proof of whatever archive is current would invent a guarantee.
## Tag convention
### The config (§8.3) — and a correction to the spec
`v<semver>`, at the commit the binary was built from. `v0.120.0` was created retroactively at
`cd6e267` — the commit that produced the published binary (sha `a7763d31b55b5ce7…`). `configs/` is
byte-identical between that commit and `main`, so nothing about the sixteen fetched files depends on
the choice; `cd6e267` is tagged because it is the honest one.
The spec said *"`0` must keep meaning disabled"*. **In the code as it stands, `0` means *use the
default* and NEGATIVE means disabled** (`RestoreTestCadence`, pre-existing). Making `0`disable would
have switched restore-testing off on every box that leaves the key unset — the worst possible reading
— so the actual semantics were preserved and this is flagged rather than silently followed.
-`restore_test_eval_interval_seconds` — how often due-ness is **asked**. Default **6 h**.
-`restore_test_settle_seconds` — how long an archive must sit. Default **24 h**.
-`restore_test_cadence_seconds` — **deprecated**. Negative still **disables**, verbatim. A positive
value seeds the **settle lag** (the quantity a person setting it was expressing: how long may pass
between a backup and confidence that it restores), and the daemon logs one start-up WARN naming
both replacements. It is deliberately **not** carried into the evaluation interval: a box that set
72 h to spare a weak endpoint would otherwise get a 72-hour-latency due-check, whereas what it
wanted — fewer heavy restores — is what per-archive due-ness already gives it.
## 4. Part 1.4 — the measurement, and the interval chosen from it
Measured on demo-felhom, 2026-08-03, via `--selftest=restore-test-due` and by timing the underlying
| both together | one full evaluation | **430 ms** |
**Cost does not set the interval** — even at one evaluation a minute the offsite leg would be ~0.7 %
of the link's time. What sets it is the other bound, and it is not in the brief: **under a per-archive
due-check a FAILING tier stays due, so the evaluation interval is also its RETRY interval — and a
retry is a multi-GB restore.** Every few minutes would be an incident of its own; the old timer
retried a broken tier once a day.
**6 h chosen from both ends:** at most four heavy retries a day in the worst case, and at most 6 h of
latency between an archive settling and its proof — negligible against a 24 h settle lag, so a daily
tier is still proved daily. No second rate limiter was added (§8.4): the pacing remains one test per
archive generation.
## 5. Two hazards the new frequency created, and their fixes
Both are consequences of evaluating often rather than daily, and neither is in the brief:
1.**The due-check now runs BEFORE the heavy-operation gate is taken.** Holding that gate for a read
that answers "nothing to do" would open a window at *every* evaluation in which a starting backup
cannot acquire — and a backup that cannot acquire does not merely wait, it **records a failure and
pages the operator** (F-A1). Nothing heavy starts before the gate; due-ness does not expire while
we check.
2.**The candidate picker skips implausible archives.** Under per-archive due-ness an incomplete
1-byte phantom (F-CRIT-2's artefact, which server-side prune does **not** collect) would be picked
forever, fail forever, never earn proof, and leave the tier due at *every* evaluation — turning the
evaluation interval into the retry rate for a multi-GB restore. `archivePlausiblyComplete` (the
canonical helper, with its warn-once companion) is applied in the shared scan, so both callers
agree. **This is a behaviour change to `PickRestoreCandidateOn`** and is recorded as such.
## 6. Tests and red-proofs
Green gate, both repos: `go build ./... && go vet ./... && go test ./...` — agent **29 packages ok,
rc=0**; hub **rc=0**. The test run and the commit were always separate commands.
| # | Test | Asserts | Mutation | Observed |
|---|---|---|---|---|
| A | `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` | 5 runs over 5 days, each on the settled archive | the naive rule (`now-landed >= settle`, proven-archive check deleted, cutoff removed) | **FAIL** — `a daily tier must be proved once per day; got 0 run(s) over 5 days: []` |
| B | `TestDue_WeeklyTierIsProvedOncePerArchive` | 84 evaluations over 3 weeks → exactly 3 runs, one per archive | — | pass |
| C | `TestDue_RestartRunsNothing` | two restarts + 4 evaluations → **0 runs** | `ProvenArchive` reverted to per-tier time | **FAIL** — `2 restart(s) produced 4 run(s)` |
| D | `TestDue_NewSettledArchiveMakesAProvedTierDueAgain` | a newly settled archive re-arms the tier, and the NEW archive is tested | — | pass |
| E | `TestDue_FailingTierIsRetriedAndNeverProven` | 3 evaluations → 3 retries, no proof recorded | credit on failure (`rt.Pass &&` dropped) | **FAIL** — `got 1 run(s) over 3 evaluations` + `TestRotation_FailureEarnsNoCredit` also failed |
| F | `TestDue_TwoDueTiersRunOneAtATime` | one evaluation → one run; the other is deferred and runs next | — | pass |
| F′ | `TestDue_DeferredBehindABackupStaysDue` | the gate holds; a deferred tier stays DUE | — | pass |
| H | `TestDue_NewbornTierIsNotDueAndNotAnError` | no archive → not due, no error, **and a reason** | — | pass |
| — | `TestDue_UnsettledArchiveIsNotACandidate` | a 2 h-old archive is not a candidate under a 24 h lag | — | pass |
| — | `TestDue_LookupFailureIsUnknownNotNotDue` | a tier that cannot be listed is UNKNOWN, the error travels, the other tier still runs | — | pass |
| — | `TestRestoreTestState_LegacyFileMigratesToNothingProven` | legacy time kept, no archive claimed | — | pass |
| — | `TestPickRestoreCandidate_SkipsImplausibleArchives` | the newest entry is not a candidate if it cannot be complete | guard removed | **FAIL** — `pick = "phantom" want the newest COMPLETE archive 'real'` |
| I | `TestMainWiresTheSettleAwareTierPicker` | **AST** of `main.go`: settle picker wired, old picker gone, `Settle` set, eval-interval accessor called | the wiring line commented out | **FAIL** — `main.go never passes runner.PickSettledRestoreCandidateOn …` (a `strings.Contains` check would have PASSED — the string is still there, in a comment) |
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
```
### 7.4 Teardown — all three layers
| Layer | Before | After |
|---|---|---|
| scratch guest 990000 | `stopped lock=create` during the run | **absent** from `pct list` |
| its volumes | 5 thin LVs (32 G + 200 G + 50 G + 2×1 G) | **0** matches in `lvs` |
| hub-side record | — | the run's **`restore_tests[]` entry is RETAINED deliberately** — it is the proof the hub's staleness check reads, and deleting it would delete the result. No event was created: the run passed, and `restore_test_failed`/`restore_test_stale` fire only on failure or staleness |
`pvesm status` before and after: `local-lvm` 1.95 % used before the run, and the thin volumes are gone
after it — the restore reclaimed to the same shape it started in.
### 7.5 The restart, which is the defect a person would actually notice
## 8. Release and deployment
| Step | Evidence |
|---|---|
| Released via `scripts/release-agent.sh 0.121.0` | tag `v0.121.0` at `4d82591`, package published |
| Verified by **independent download** | sha256 `b2128f3cd4539225a2842f541f56ffaf5390b1d97f3f3a80076ec5f53dbc7d7a`, 14 081 336 B, round-trip GET matched |
| Gate re-run after release | `2 released version(s) to verify: 0.120.0, 0.121.0` → both **installable** |
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
admin