The missing grant is one command; the silence was the defect. On demo-felhom the
agent's token has FelhomAgentStore on local, local-lvm and felhom-pbs — and not
on felhom-backup, the storage the same installer configured as
local_backup_target. That storage answers {"data":[]} through the token while
root sees three archives.
An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return, so
pickForThisRun skipped it as "no settled archive yet" and the tier was never
restore-testable on that box. The permission question, unlike the listing, has a
definite answer, so it is asked directly: Client.Permissions reads
/access/permissions as the agent's OWN token, and one capability.Status per
configured tier reports it — composed around the sudo prober, the way the
pool-read check already is.
Measured first, because the obvious reading is wrong: an ungranted path answers
neither empty nor 403, but with the privileges inherited from the box-wide grant
(Sys.Audit, SDN.Use, Datastore.Audit). Checking for Datastore.Audit would report
a blinded storage healthy — red-proved. The probe tests for
Datastore.AllocateSpace.
The probed set comes from the box's own config, never a fixed list: a hardcoded
probe list is the defect reproduced inside the fix. Critical, because the hub
alerts only on critical — except the "local" fallback target, which is reported
but does not page. It never looks at content, so it cannot alarm on a newborn
tier; it never reports ok when it could not ask. Status wire shape unchanged, so
no hub change.
Scenario A proven on demo-felhom against the exact observation that filed R-189:
a 675 s offsite restore-test passed, the agent was restarted 11 seconds later
(inside the reporting window), and the hub's very next report carried
'1 restore-tests' where the same sequence produced 0 this morning. The hub's own
database holds the archive, the tier, the pass and the ORIGINAL test time, with
the run mechanics deliberately zero.
Also records the property the validation surfaced: the state holds one proof per
tier, so proving an older archive re-arms a newer one — confirmed live after the
defaults were restored.
2026-08-03 16:58:35 +02:00
5 changed files with 587 additions and 273 deletions
| `-trimpath -buildvcs=false`, either way | `7ffcdf1d…` | 14 064 574 B | *(none)* |
**Evidence that a daily tier does become due**, at three levels:
**After, on the real release (v0.122.0) — the three values §15.2 asks for:**
1.**Unit, time-driven** — `TestDue_DailyTierIsProvedDailyOnItsOwnArchive`: five simulated days, one
archive a day, evaluated hourly (120 evaluations) → **exactly 5 runs**, and run *i* tests day
*i−1*'s archive, never the still-settling one.
2.**The red-proof of the naive rule** — implemented and observed failing at **0 runs over 5 days**
(§6), which is the trap made visible rather than argued about.
3.**Live** — the offsite tier on demo-felhom became due on its own archive and ran (§7).
## 3. What changed
| Piece | File | Change |
| artifact | sha256 | size |
|---|---|---|
| the due-check | `internal/backup/restoretest_due.go` (new) | `EvaluateDue` / `evaluateTier` — per-tier verdict + the reason, ordered oldest-proven first |
| the trigger | `internal/backup/schedule.go` | the ticker is now the **evaluation interval**; `pickForThisRun` answers *"is anything due?"*, and "nothing" is a normal answer |
| the state | `internal/backup/restoretest_state.go` | records **which archive** was proven, with migration |
| the picker | `internal/backup/runner.go` | `PickSettledRestoreCandidateOn(ctx, target, notAfter)`; `PickRestoreCandidateOn` is a one-line call into it |
| the knobs | `internal/config/config.go` | `restore_test_eval_interval_seconds` + `restore_test_settle_seconds`; the old key deprecated, not repurposed |
| the wiring | `cmd/felhom-agent/main.go` | settle-aware picker + `Settle`; deprecation WARN; new `--selftest=restore-test-due` |
| the observable (**v0.121.1**) | `internal/backup/schedule.go`, `restoretest_due.go` | a not-due evaluation logs one **INFO** line naming every tier's verdict — see §9 |
| published, downloaded from Gitea | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #1 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #2 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
### The state records the archive (§8.2)
**All three identical.** The property is removed-cause, not sequenced-around: `-buildvcs=false` drops
a stamp nothing reads (no `ReadBuildInfo` caller, verified by grep), and the version still comes from
the explicit `-X main.version` ldflag. `-trimpath` additionally makes a rebuild from a different
checkout directory match.
A timestamp cannot answer *"have we proven **this** archive"* — it is the same class as the workspace
rule that a timestamp records an *attempt*, not a *result*: here it records a result, but not **which**
result. `RestoreTestState` now holds `{archive, proven_at}` per tier.
**A second discrepancy fell out of the measurement and is fixed with it:**`publish-agent.sh`'s
fallback build forced `CGO_ENABLED=0` and produced **13 990 236 B** against the release path's
**14 064 574 B** — a 74 KB difference, i.e. one version name meaning two binaries depending on which
entry point ran. Both paths now use identical flags, each commented with a pointer to the other.
**Migration:** a pre-R-86 file (`{"target": "<RFC3339>"}`) keeps its **time** — rotation ordering
survives a deploy, which is why the file exists at all — and yields **no proven archive**, so each
tier is due exactly once after the upgrade. One extra test per tier, once, is the safe direction;
reading a legacy time as proof of whatever archive is current would invent a guarantee.
## 3. Scenario E — a correct release no longer emails a failure (Part 2)
### The config (§8.3) — and a correction to the spec
**Only the tag PUSH moved.** The order is now build → tag **locally** → publish → push tag. The tag is
still created before anything is published, so the build and the tag describe the same commit; it
becomes *visible* — to CI (`on: [push]`) and to any `raw/tag/…` fetch — only once the package is
downloadable.
The spec said *"`0` must keep meaning disabled"*. **In the code as it stands, `0` means *use the
default* and NEGATIVE means disabled** (`RestoreTestCadence`, pre-existing). Making `0` disable would
have switched restore-testing off on every box that leaves the key unset — the worst possible reading
— so the actual semantics were preserved and this is flagged rather than silently followed.
**The old ordering's invariant is asserted directly rather than arranged for.**
`check-published-versions.py` now carries two invariants: every tag has an installable package (as
before) **and no published version is missing its tag** (new). The second is a **bounded probe** —
the frontier past the newest tag, where a failed tag push leaves an orphan, plus patch gaps — and it
**prints its probe set on every run**, because a check whose coverage is invisible reads as a
guarantee it is not making. The package listing api was **re-measured**, not assumed: `401` without a
token, so absence still cannot be enumerated, and the script says so in its own output.
-`restore_test_eval_interval_seconds` — how often due-ness is **asked**. Default **6 h**.
-`restore_test_settle_seconds` — how long an archive must sit. Default **24 h**.
-`restore_test_cadence_seconds` — **deprecated**. Negative still **disables**, verbatim. A positive
value seeds the **settle lag** (the quantity a person setting it was expressing: how long may pass
between a backup and confidence that it restores), and the daemon logs one start-up WARN naming
both replacements. It is deliberately **not** carried into the evaluation interval: a box that set
72 h to spare a weak endpoint would otherwise get a 72-hour-latency due-check, whereas what it
wanted — fewer heavy restores — is what per-archive due-ness already gives it.
**Live result — this release is the test:**
## 4. Part 1.4 — the measurement, and the interval chosen fromit
Measured on demo-felhom, 2026-08-03, via `--selftest=restore-test-due` and by timing the underlying
API call directly (3 runs each):
| tier | what it is | one due-check |
| release | CI runs on the release commit | outcome |
That report was built **after** the restart, when the in-memory store was empty — so the entry can
only have come from the persisted state. The archive, the tier, and the **original** test time
survived; the run mechanics are zero because they are deliberately not re-invented.
**A 14.5 GB encrypted offsite archive**, restored, booted, verified and destroyed in **675 s** — and
this time the proof outlived the process that produced it.
**Teardown, all three layers:** scratch guest absent from `pct list` (0), its volumes gone from `lvs`
(0), the validation drop-in removed and the daemon back on its defaults
(`eval_interval=6h0m0s settle=24h0m0s`). The hub-side `restore_tests[]` record is **retained
deliberately** — it is the proof the staleness check reads, so deleting it would delete the result.
No `restore_test_*` event was raised, because nothing failed and nothing is stale.
## 7. Tests and red-proofs
Green gate: `go build ./... && go vet ./... && go test ./...` — **29 packages ok, rc=0**, plus
`python3 scripts/agent_gates.py` (reuse-refs + published-versions) all OK. The test run and the commit
were always separate commands.
| # | Test | Asserts | Mutation | Observed |
|---|---|---|---|---|
| A | `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` | 5 runs over 5 days, each on the settled archive | the naive rule (`now-landed >= settle`, proven-archive check deleted, cutoff removed) | **FAIL** — `a daily tier must be proved once per day; got 0 run(s) over 5 days: []` |
| B | `TestDue_WeeklyTierIsProvedOncePerArchive` | 84 evaluations over 3 weeks → exactly 3 runs, one per archive | — | pass |
| C | `TestDue_RestartRunsNothing` | two restarts + 4 evaluations → **0 runs** | `ProvenArchive` reverted to per-tier time | **FAIL** — `2 restart(s) produced 4 run(s)` |
| D | `TestDue_NewSettledArchiveMakesAProvedTierDueAgain` | a newly settled archive re-arms the tier, and the NEW archive is tested | — | pass |
| E | `TestDue_FailingTierIsRetriedAndNeverProven` | 3 evaluations → 3 retries, no proof recorded | credit on failure (`rt.Pass &&` dropped) | **FAIL** — `got 1 run(s) over 3 evaluations` + `TestRotation_FailureEarnsNoCredit` also failed |
| F | `TestDue_TwoDueTiersRunOneAtATime` | one evaluation → one run; the other is deferred and runs next | — | pass |
| F′ | `TestDue_DeferredBehindABackupStaysDue` | the gate holds; a deferred tier stays DUE | — | pass |
| H | `TestDue_NewbornTierIsNotDueAndNotAnError` | no archive → not due, no error, **and a reason** | — | pass |
| — | `TestDue_UnsettledArchiveIsNotACandidate` | a 2 h-old archive is not a candidate under a 24 h lag | — | pass |
| — | `TestDue_LookupFailureIsUnknownNotNotDue` | a tier that cannot be listed is UNKNOWN, the error travels, the other tier still runs | — | pass |
| — | `TestRestoreTestState_LegacyFileMigratesToNothingProven` | legacy time kept, no archive claimed | — | pass |
| — | `TestPickRestoreCandidate_SkipsImplausibleArchives` | the newest entry is not a candidate if it cannot be complete | guard removed | **FAIL** — `pick = "phantom" want the newest COMPLETE archive 'real'` |
| I | `TestMainWiresTheSettleAwareTierPicker` | **AST** of `main.go`: settle picker wired, old picker gone, `Settle` set, eval-interval accessor called | the wiring line commented out | **FAIL** — `main.go never passes runner.PickSettledRestoreCandidateOn …` (a `strings.Contains` check would have PASSED — the string is still there, in a comment) |
| A | `TestMerge_ProofSurvivesARestart` | an empty in-memory store + a persisted proof → the proof is reported, with its archive and its original time | the persisted merge deleted (the pre-R-189 body) | **FAIL** — `after a restart the persisted proof must be reported; got 0 entr(ies): []` — the live observation exactly |
| B | `TestMerge_NeverInventsAPassForAnUnprovenTier` | no proof → no entry; a tier-less record → no entry | — (its state-layer twin below carries the mutation) | pass |
| B′ | `TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe` | v1 + v2 + v3 records side by side → only the describable one is reported | the `reportable()` filter dropped | **FAIL** — `got 3` entries, two with an empty `SourceTier`/`SourceArchive` |
| C | `TestMerge_NewerWinsAndNeverDuplicatesATier` | one entry per tier, newest wins, in both directions | de-duplication removed | **FAIL** — `one entry per tier; got 2 for "pbs" — the hub would read two tests` |
| D | `TestMerge_AFailureIsStillReported` | a fresh failure beats an older stored success | (same mutation) | **FAIL** — 2 entries, i.e. the failure no longer the single answer for that tier |
| G | the gate's converse assertion | a published version with no tag fails | probe set → `[]` | **FAIL** (green over a live orphan) |
| I | `TestMainWiresTheDurableRestoreTestProof` | **AST**: `SetProvenRestoreTests` is called **and fed `rtState`** | the call commented out | **FAIL** — `main.go never calls collector.SetProvenRestoreTests` (a `strings.Contains` check would have passed — the string is still there) |
| H | reproducibility | three identical sha256 | — (measurement, §2) | pass |
Hub-side (Scenario G) is in `felhom.eu/REPORT.md`, including **a hollow test caught by its own
red-proof**: the first weekly fixture had no jitter, sat on exactly 168 h, and PASSED under the
flat-window mutation.
**Jitter, per §10:** every timestamp fixture uses odd minutes and seconds (`13:25:14`, `19:55:41`,
`13:41:07`, `04:41:58`) — several taken from the real box — rather than round hours. Yesterday a test
was hollow because a perfectly regular series landed exactly on a threshold and survived its own
mutation.
### Tests deliberately changed, and why
## 8. Files changed
`TestRotation_BothTiersExercisedAcrossCadences` asserted *4 ticks → 4 runs*. That was a faithful
statement of the defect — every tick produced a heavy restore-test, because the ticker **was** the
trigger. It is now `TestRotation_BothTiersExercisedOncePerArchive`: **2 runs across 4 evaluations**,
one per tier, one per archive. Strictly stronger — it pins both the coverage R-85 won and the pacing
R-86 adds. The old assertion is quoted in the test's comment so the change is legible.
`internal/backup/restoretest_state.go` (v3 record + `ProvenRestoreTests`), `internal/backup/store.go`
(the comment that had become false), `internal/backup/schedule.go` (record tier + verified),
`internal/hub/collect.go` (the seam + the merge), `cmd/felhom-agent/main.go` (wiring),
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
```
### 7.4 Teardown — all three layers
| Layer | Before | After |
|---|---|---|
| scratch guest 990000 | `stopped lock=create` during the run | **absent** from `pct list` |
| its volumes | 5 thin LVs (32 G + 200 G + 50 G + 2×1 G) | **0** matches in `lvs` |
| hub-side record | — | the run's **`restore_tests[]` entry is RETAINED deliberately** — it is the proof the hub's staleness check reads, and deleting it would delete the result. No event was created: the run passed, and `restore_test_failed`/`restore_test_stale` fire only on failure or staleness |
`pvesm status` before and after: `local-lvm` 1.95 % used before the run, and the thin volumes are gone
after it — the restore reclaimed to the same shape it started in.
### 7.5 The restart, which is the defect a person would actually notice
Under the old scheduler every agent deploy restarted the ticker, so a restore-test ran one interval
after each deploy regardless of what had been proven. The proof of the fix cannot be *"nothing
appeared in the log"* — that is the absent-line trap this project has a standing rule about — so
v0.121.1 makes a quiet evaluation say what it decided, and the evaluation interval was shortened to
2 min for the validation so evaluations are **observable**, not assumed:
```
15:32:10 felhom-agent daemon starting version=0.121.1 ← the restart
| Second release, same path | **v0.121.1** — tag `v0.121.1`, sha256 `afaeeb509d1ed70d6e6bebac0393a3cd5be59d51e3db9ff96ef8524bd78546d7`, round-trip verified |
| Config compatibility checked on both boxes | `restore_test_cadence_seconds = 0` on demo-felhom **and** demo-hp, and the installer writes `0` — so no box is on the deprecation path, and a fresh install gets the new defaults with no installer change |
## 9. Findings filed (none fixed blind)
- **R-185 — the agent cannot see demo-felhom's host backup tier at all.** The PVE token has no ACL on
`/storage/felhom-backup`, so the content listing returns `{"data":[]}` where root sees three
archives (6.1–6.3 GB, 08-01/02/03). Verified three ways, including `local` — which *has* a grant —
returning its archives through the same token. **Pre-existing and independent of R-86** (R-85's
rotation had the same blindness). The part worth fixing is the **silence**: a permission-blinded
tier is today indistinguishable from a newborn one, and the agent already records the backups it
wrote to that target, so the contradiction is detectable.
- **R-186 — a released binary's sha cannot be reproduced from its tag.** `release-agent.sh` builds
before tagging, so Go stamps a pseudo-version into the published bytes: published `b2128f3c…`
(14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source and toolchain. The
build order is deliberate, so the fix is not to swap the steps blind. **Mitigated here** by
deploying the published artifact.
- **R-187 — R-115's one-command release had never run its publish leg** (`CLOSED`, fixed in the same
session): `publish-agent.sh` has been mode `0644` since 2026-06-28 because every earlier caller used
`bash …`, and `release-agent.sh` called it directly → `Permission denied` on the first real release.
Fixed both ways: the mode bit restored **and** the call made mode-independent. The tag the failed
run created was withdrawn (nothing had been published under it — verified 404) and recreated on the
fix commit, so one version name still means one binary.
- **R-188 — a correct agent release emails a CI failure about half the time.** `on: [push]` fires the
gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag before publishing
(deliberately). CI can therefore run the published-versions gate inside the window where the tag
exists and the package does not, and correctly report *"every released agent version must be
INSTALLABLE"* for a release that completes seconds later. **Measured across two releases in one
session:** v0.121.0 → runs #12 success / #13 failure on the same sha; v0.121.1 → #17 failure / #18
success on the same sha; and one pair both green — a race, not a rule. It matters because R-168
made CI email on failure so a red gate cannot be missed; a signal that cries wolf on every second
correct release is how that mail becomes something you archive unread.
- **R-189 — a passing restore-test can be invisible to the hub, and R-86 widened that window.**
Observed live: **the 15:25:14 PASS reached no host-report at all**. `restore_tests[]` comes from an
**in-memory** store (*"lost on restart; the cadence re-populates"*) and the report interval is
900 s; the agent was restarted 2 m 43 s after the run for the v0.121.1 deploy. That used to
self-heal within 24 h because the next cadence re-tested the tier — **under per-archive due-ness
the agent will not re-test a proven archive**, so the hub can stay ignorant until the next archive
generation. The persisted proof already exists: `RestoreTestState.Snapshot()` is documented *"for
the host-report gauge"* and has **no production caller** — a seam built and never wired, and an
invariant asserted only in a comment, in one method. Bounded, not over-ranked: the hub scans its
retained window and the offsite tier's window (12 d) is wider than its archive rhythm (7 d), so one
lost report is tolerated. Filed, not fixed — it is a report-contract change.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
admin