14 Commits

Author SHA1 Message Date
admin fe14bc62c0 v0.123.0: a tier the box cannot READ now says so (R-185)
gates / gates (push) Successful in 7s
The missing grant is one command; the silence was the defect. On demo-felhom the
agent's token has FelhomAgentStore on local, local-lvm and felhom-pbs — and not
on felhom-backup, the storage the same installer configured as
local_backup_target. That storage answers {"data":[]} through the token while
root sees three archives.

An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return, so
pickForThisRun skipped it as "no settled archive yet" and the tier was never
restore-testable on that box. The permission question, unlike the listing, has a
definite answer, so it is asked directly: Client.Permissions reads
/access/permissions as the agent's OWN token, and one capability.Status per
configured tier reports it — composed around the sudo prober, the way the
pool-read check already is.

Measured first, because the obvious reading is wrong: an ungranted path answers
neither empty nor 403, but with the privileges inherited from the box-wide grant
(Sys.Audit, SDN.Use, Datastore.Audit). Checking for Datastore.Audit would report
a blinded storage healthy — red-proved. The probe tests for
Datastore.AllocateSpace.

The probed set comes from the box's own config, never a fixed list: a hardcoded
probe list is the defect reproduced inside the fix. Critical, because the hub
alerts only on critical — except the "local" fallback target, which is reported
but does not page. It never looks at content, so it cannot alarm on a newborn
tier; it never reports ok when it could not ask. Status wire shape unchanged, so
no hub change.
2026-08-03 18:53:47 +02:00
admin 0b28eae7bb REPORT: R-189/R-188/R-186 — live evidence, the three sha values, and the observations
gates / gates (push) Successful in 6s
Scenario A proven on demo-felhom against the exact observation that filed R-189:
a 675 s offsite restore-test passed, the agent was restarted 11 seconds later
(inside the reporting window), and the hub's very next report carried
'1 restore-tests' where the same sequence produced 0 this morning. The hub's own
database holds the archive, the tier, the pass and the ORIGINAL test time, with
the run mechanics deliberately zero.

Also records the property the validation surfaced: the state holds one proof per
tier, so proving an older archive re-arms a newer one — confirmed live after the
defaults were restored.
2026-08-03 16:58:35 +02:00
admin 7581f8140a v0.122.0: three ways the signals lied about themselves (R-189, R-188, R-186)
gates / gates (push) Successful in 7s
All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.

R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.

R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.

R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
2026-08-03 16:40:18 +02:00
admin 3d0a1d615d REPORT: restart proof, R-188/R-189, and the restored defaults
gates / gates (push) Successful in 7s
2026-08-03 15:36:46 +02:00
admin 77e2cc4583 CONTEXT: v0.121.1 (a quiet evaluation is audible) + the live proof
gates / gates (push) Successful in 6s
2026-08-03 15:32:23 +02:00
admin cd1b087db7 REPORT: R-86 live results (14.5 GB offsite restore-test, due-triggered, 635 s)
gates / gates (push) Successful in 7s
2026-08-03 15:27:33 +02:00
admin 53d0c6bfc4 v0.121.1: 'nothing is due' must be AUDIBLE (R-86 + standing rule 3)
gates / gates (push) Successful in 6s
Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
construction. After it, 'nothing is due' is the NORMAL outcome — and it was
logged at DEBUG, which journald drops. An empty journal would then be equally
consistent with a healthy loop and a dead goroutine: the shape the R-88 watcher
was retired for, re-created by making the quiet path the common one.

A not-due evaluation now logs one INFO line naming every tier's verdict (four
lines a day at the 6h default), and an unlistable tier reads UNKNOWN with its
error in that same line, so a lookup failure can never present as 'nothing due'.

Red-proved through the scheduler's own tick, not the helper.
2026-08-03 15:26:54 +02:00
admin 4d82591052 release-agent.sh: the publish leg was unrunnable on its first real use
gates / gates (push) Successful in 7s
R-115's whole point is that publishing cannot be forgotten because it rides the
release script. On the first real release through it (v0.121.0, R-86) it died at
exactly that leg:

  scripts/release-agent.sh: line 101: .../scripts/publish-agent.sh: Permission denied

publish-agent.sh has been mode 0644 since it was created on 2026-06-28 — every
earlier caller ran it as `bash scripts/publish-agent.sh`, so nothing ever noticed,
and release-agent.sh (written the same day it was needed) called it directly.

Two fixes, both small and both wanted: restore the executable bit, and invoke it
through `bash` so the release no longer depends on a file mode — the kind of
thing a checkout, an archive or a copy loses again.

The v0.121.0 tag created by the failed run is withdrawn and recreated on this
commit; nothing was published under it (verified 404 on the package endpoint), so
one version name still means one binary.
2026-08-03 15:04:52 +02:00
admin 4618169036 R-86: restore-test follows the backup, not the clock (v0.121.0)
gates / gates (push) Failing after 7s
The ticker survives as the EVALUATION interval only. A tier is DUE when its
newest archive that has settled for `settle` (default 24h) has not been proven:
daily tier -> proved daily on yesterday's archive, weekly tier -> weekly on its
own, newborn -> UNKNOWN.

The trap avoided: the literal reading ("newest archive is >= 24h old") is NEVER
true on a daily tier, so it silently switches restore-testing off where it
matters most. Red-proved at 0 runs over 5 simulated days.

- state records WHICH archive was proven; legacy files keep their time and yield
  no proven archive (each tier due once after the upgrade, deliberately)
- two knobs replace one: restore_test_eval_interval_seconds (6h, measured) and
  restore_test_settle_seconds (24h). The old cadence key keeps its DISABLE
  meaning verbatim and now seeds the settle lag, with a start-up WARN.
- due-check runs BEFORE the heavy-op gate (a frequent poll must not make a
  starting backup record a failure, F-A1)
- candidate picker skips implausible archives (a phantom would be due forever)
- new read-only --selftest=restore-test-due prints the verdict + its cost
2026-08-03 14:54:57 +02:00
admin 1b14cfd0b4 REPORT: Scenario F measured on real CI (runs 69 vs 70, same commit)
gates / gates (push) Successful in 7s
2026-08-03 12:44:37 +02:00
admin 0db77666c6 REPORT: release path + published-versions gate (no version bump)
gates / gates (push) Failing after 7s
2026-08-03 12:37:09 +02:00
admin dd2d1feb6e release path publishes, and an unreleasable version fails CI (R-115, R-183)
gates / gates (push) Failing after 7s
NO VERSION BUMP and nothing built: no Go code changed. The agent stays v0.120.0.

scripts/release-agent.sh — THE way to release. build -> tag -> publish -> verify
by INDEPENDENT download. Publishing was a separate remembered step and was
forgotten three times in five days (R-111's 17 stranded releases, 0.114.0, and
0.120.0 — deployed to both demo hosts and undownloadable, so a documented-path
reinstall would have silently downgraded them WHILE REPORTING SUCCESS). R-111's
own closing line named this leg and closed SHIPPED without it; it recurred the
same afternoon, which is the evidence that a note is not a mechanism.

It tags because felhom-host-install.sh now fetches the sixteen agent config
files from raw/tag/v<version>/ (R-183): a released version with no tag 404s a
box mid-install, as root, on a virgin machine. It verifies by downloading what
it just published and comparing the sha to what it built — the publish step's
own success is a report on its own write; a fetch returning the right bytes is
a different claim. It refuses a dirty/unpushed tree and refuses to re-release an
existing version. It does NOT vouch: that points machines at a version and stays
the operator's act.

scripts/check-published-versions.py — the gate. Every v<semver> tag must have a
downloadable package AND a tag tree serving the agent's configs. Registered as
NOT --fast (needs network; a push must not fail because Gitea blinked), and the
CI workflow now runs the FULL gate set instead of --fast — otherwise the gate
would have been registered and never run, the built-but-never-wired failure this
project has shipped four times.

The invariant is not the one specified, and the reason was measured, not assumed:
the hub artifact manifest is 401 without a per-customer passphrase and Gitea's
package LISTING api is 401 without a token, while the package DOWNLOAD url and
the git TAGS api are anonymous. So CI cannot ask "what is vouched" without an
operator credential — whose addition is the operator's call. The tag-based
invariant needs none and catches all three recorded instances. What it does not
catch (the hub vouching a version never released at all) is filed as R-184.
2026-08-03 12:34:20 +02:00
admin 9dfd89cb94 docs: agent 0.120.0 published + vouched, proven on two reinstalled boxes (R-178)
gates / gates (push) Successful in 6s
v0.120.0 had been built, committed and deployed to both demo hosts but never
published: the Gitea generic package 404'd and the hub manifest vouched 0.119.0.
Installer step 5 skips only on an exact version match, so a documented-path
reinstall would have downgraded both boxes to the pre-merge agent -- and would
have succeeded, since step_grows passes -sysdata-grow 0 and 0.119.0's mp1 resize
never fires. Published from a clean tree (upload 201, round-trip GET verified,
sha a7763d31b55b5ce7...) and vouched; both reinstalls then fetched and verified
it over the real customer path.

Filed as the third instance of R-115. No version bump, nothing built.
Evidence: felhom.eu/REPORT.md
2026-08-03 09:33:50 +02:00
admin 4bb84fc3ca REPORT: v0.120.0 + golden 3.0.0 — built and proven at the bake, NOT proven on a box
gates / gates (push) Successful in 12s
States the scope reduction first: Phases 6-7 (reinstall both demo boxes and
prove one end to end) were NOT done, nothing was wiped, and the golden is
deliberately left unvouched as a result. Filed as R-178.

Also records the two instrument errors this session: a census truncated by
head -10 that gave the wrong answer about --sysdata-grow (the installer does
pass it), and a wait gated on a marker the bake prints before publishing.
2026-08-03 07:15:29 +02:00
25 changed files with 3268 additions and 165 deletions
+13 -1
View File
@@ -43,7 +43,19 @@ jobs:
- name: Run the gate entry point
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
# exit code IS the result: no `|| true`, no pipe that could swallow it.
run: cd ws/felhom-agent && python3 scripts/agent_gates.py --fast
#
# THE FULL SET, NOT `--fast` (R-115, 2026-08-03). `--fast` means "no network and no
# container runtime" and exists for `.githooks/pre-push`, where a push must not fail
# because Gitea blinked or because someone is on a train. CI is the opposite machine: it
# has the network, it is not in anyone's way, and it is the half that emails. The
# published-versions gate — the R-115 mechanism, which asks Gitea whether a released
# version can actually be downloaded — is network-bound and therefore runs ONLY here.
# Leaving `--fast` in place would have registered that gate and never run it, which is the
# built-but-never-wired failure this project has shipped four times.
env:
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
- name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
+282
View File
@@ -1,3 +1,285 @@
## v0.123.0 — a tier the box cannot READ now says so (2026-08-03, R-185)
**The missing permission is one command. The silence was the defect.** On demo-felhom the agent's PVE
token held `FelhomAgentStore` on `local`, `local-lvm` and `felhom-pbs` — and **not** on
`felhom-backup`, the storage the same installer had configured as `local_backup_target`. Asked for
that storage's content the API answers `{"data":[]}` while root sees three archives (6.16.3 GB,
08-01/02/03).
**An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return.** `pickForThisRun` skips
an empty tier — correctly, because a fresh offsite tier legitimately has nothing yet — and reports
*"no settled archive yet"*. So that tier was never restore-testable on that box and nothing ever
mentioned it. This project's own rule, in a new place: an empty answer is not evidence that there is
nothing there.
**The permission question, unlike the listing, has a definite answer — so it is asked directly.**
`Client.Permissions` reads `GET /access/permissions?path=/storage/<target>` **as the agent's own
token** (asking as root answers a different question and always says yes), and one
`capability.Status` per configured tier reports the result. It composes *around* the sudo prober, the
way the pool-read check already does — an API read does not belong inside a sudo-policy probe.
**MEASURED FIRST, and the obvious reading is wrong.** The ungranted path does not answer empty and
does not 403:
```
/storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
/storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
```
It answers with the privileges **inherited** from the box-wide `/` grant. A probe asking *"is the
path present?"* or *"does it have `Datastore.Audit`?"* would report the blinded storage healthy — so
the probe tests for `Datastore.AllocateSpace` specifically, and a red-proof pins that.
**Decisions, each weighed once:**
- **The probed set comes from the box's own config** (`BackupTiers()`), not a fixed list. A hardcoded
probe list is exactly the defect being fixed, reproduced inside the fix.
- **CRITICAL**, because the hub alerts only on critical and a non-critical entry would ride the
report and alert nobody — the same silence with extra steps. **Except** the `local` fallback
target, which host-install's own comment calls the DEGRADED configuration: it is still probed and
still reported, but it does not page, because turning an ordinary documented setup into an alert
is how a signal becomes something an operator archives unread.
- **It never looks at content**, so it cannot alarm on a newborn tier by construction.
- **It never reports ok when it could not ask.** An unreachable PVE is degraded: a self-check that
fails open converts *"I do not know"* into *"fine"*.
The wire shape (`capability.Status`) is unchanged, so the hub's existing critical-degraded alert
applies with no hub change and no hub bump.
## v0.122.0 — three ways the signals lied about themselves (2026-08-03, R-189 · R-188 · R-186)
All three are the reporting and release path misreporting its own work. **No customer machine, no
backup, no restore, no disk layout, no data.** The restore-test itself and when it runs are unchanged
from v0.121.1.
### R-189 — a passing restore-test no longer vanishes on a restart
`restore_tests[]` came only from the in-memory `backup.Store`, whose own comment read *"lost on
restart; the cadence re-populates"*. That was true under a timer. It stopped being true when R-86 made
the agent refuse to re-test an archive it has already proven: a proof lost to a restart is not
repeated for a whole archive generation — **a week on the offsite tier** — and the hub calls the tier
unproven for all of it.
**Observed, not predicted (2026-08-03):** a real 14.5 GB offsite restore-test PASSED at 15:25:14, the
agent was restarted 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two
host-reports.
The durable proof already existed — `RestoreTestState`, on disk, per tier, with the archive since
R-86 — and `Snapshot()` had carried the doc comment *"for the host-report gauge"* since the day it was
written **with no caller at all**: a seam built, documented, and never connected. It now carries the
`tier` and what was `verified` as well (stored at proof time, when they are known for certain, rather
than derived later by a storage lookup that can fail), and `ProvenRestoreTests` renders them as report
entries which the collector merges.
- **Merge rule: one entry per tier, newest by `TestedAt` wins.** A fresh failure beats a stored
success — the failure is the news and lives nowhere else; a stored success beats a stale in-memory
entry after a restart; a tier never appears twice, which the hub would read as two tests. An
unparseable timestamp counts as older, so a malformed entry cannot displace a good one.
- **It refuses to lie.** A record missing the archive or the tier produces NO entry, and run mechanics
(scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one
would be. An unproven tier reading as proven would be worse than the defect being fixed.
- **Only successes are persisted, and that asymmetry is now written down where it will be read:** a
success suppresses future work, so losing it leaves the system quietly less tested than it believes;
a failure causes future work and heals itself at the next evaluation.
- The `Store` comment that stopped being true is corrected in place rather than left to mislead.
### R-188 — a correct release no longer emails a failure
`on: [push]` fires the gates workflow on the **tag** push, and the release pushed its tag *before*
publishing, so CI ran the published-versions gate in the seconds before the package existed and
correctly reported it missing. Measured across two releases in one session: runs 12/13 and 17/18, same
sha each time, opposite results — a race, not a rule. R-168 made that mail the thing that cannot be
missed; one that is wrong half the time is one you stop reading.
**Only the tag PUSH moved** (build → tag locally → publish → push tag). The tag is still created before
anything is published, so the build and the tag still describe the same commit; it simply becomes
*visible* — to CI, and to any `raw/tag/…` fetch — once the package is downloadable.
The invariant the old order protected is **not traded away**: `check-published-versions.py` now asserts
the converse directly — **no published version may be missing its tag** — as a bounded probe of the
frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set every
run because a check whose coverage is invisible reads as a guarantee it is not making. The package
listing api still answers **401** without a token (re-measured), so absence cannot be enumerated, and
the script says so.
A publish that succeeds and a tag push that then fails now **dies loudly**, printing the one-line
recovery; and a publish that *fails* removes the local-only tag so the release can simply be retried
instead of colliding with itself.
### R-186 — a released binary can now be verified by rebuilding it
`go build` stamps a module version derived from VCS state, so a build made before the tag existed and
a rebuild made after it were different binaries. Measured at one commit, same source, same toolchain:
```
default flags, no tag yet ... 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
default flags, tagged ....... 4a38f394… 14 085 440 B (mod v0.121.99)
-trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
```
The stamp is removed rather than sequenced around — nothing in this repo reads it (no `ReadBuildInfo`
caller) and the version comes from the explicit `-X main.version` ldflag. `-trimpath` additionally
makes a rebuild from a different checkout directory match.
**A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and
therefore produced a binary **74 KB smaller** than the release path built for the same version — one
version name, two binaries, decided by which entry point was used. Both now build identically.
`CLAUDE.md` records the exact command an operator can run to verify a published binary independently.
## v0.121.1 — "nothing is due" must be AUDIBLE (2026-08-03, R-86 + standing rule 3)
**Found while live-validating v0.121.0, and it is this project's own rule pointed at the change that
had just shipped.** Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
construction. After it, *"nothing is due"* is the NORMAL outcome — and it was logged at **DEBUG**,
which journald drops. An empty journal would then have been equally consistent with a healthy loop
and with a dead goroutine: the exact shape the R-88 watcher was retired for, re-created in a new
place by making the quiet path the common one.
A not-due evaluation now logs one **INFO** line naming every tier's verdict:
```
backup: restore-test evaluated — nothing due
verdicts="felhom-pbs: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven;
felhom-backup: no settled archive yet — nothing to prove (newborn or still settling)"
```
Four lines a day at the 6 h default, and the answer to *"why did nothing run last night?"* is in the
log instead of being re-derived. A tier whose storage cannot be listed reads `UNKNOWN` with its error
in the same line, so a lookup failure can never present as "nothing due".
Red-proved by reverting to the bare `Debug` line: the test asserts what the SCHEDULER emits on a real
`tick`, not what the helper returns — a helper-level test would have passed against a tick that never
called it.
## v0.121.0 — a restore-test proves each BACKUP, not the clock (2026-08-03, R-86)
**The trigger changed; the restore-test did not.** `Scheduler.Run` still has a ticker, but it is now
the **evaluation interval** — how often "is anything due?" is asked — and no longer the thing that
decides a test happens. What decides is a per-archive due-check
(`internal/backup/restoretest_due.go`):
> Let **A** = the newest archive on this tier that has settled for at least `settle` (default 24 h).
> The tier is **DUE** when **A** exists and **A has not already been proven**.
A daily tier is therefore proved once a day, on yesterday's archive; a weekly tier once a week, on
its own; a newborn tier is UNKNOWN and never a fault. Per-archive due-ness IS the pacing — one test
per archive generation and no more — so there is deliberately no second rate limiter on top of it.
**The trap this avoided, recorded because it is the version a reasonable person writes.** R-86's own
wording ("~24 h after its own newest archive") implemented literally is *"due when the newest archive
is ≥ 24 h old"* — and on a **daily** tier that is never true, because a new archive resets the
newest-archive age to zero long before it reaches 24 h. The literal rule silently switches
restore-testing OFF for the tier that matters most. It has its own red-proof, which was observed
failing with **0 runs over 5 simulated days**.
**What the fix rests on**
- **The state records WHICH archive was proven** (`restoretest_state.go`), not merely when a tier last
passed — a time cannot answer "have we proven *this* archive". A pre-R-86 state file keeps its time
(rotation ordering survives the upgrade) and yields **no** proven archive, so each tier is due
exactly once after the upgrade: one extra test per tier, once, which is the safe direction.
- **Two knobs replace one, and the old one is not silently repurposed.**
`restore_test_eval_interval_seconds` (how often due-ness is asked; default **6 h**) and
`restore_test_settle_seconds` (how long an archive must sit; default **24 h**). The deprecated
`restore_test_cadence_seconds` keeps its DISABLE meaning (negative) verbatim, and a positive value
now seeds the **settle lag** — with a start-up WARN naming both replacements.
- **6 h is bounded from both sides, not picked.** MEASURED cost of one evaluation on demo-felhom
(Part 1.4): local dir storage **18 ms**, the PBS tier over the WAN to ep0 **392 ms**, both together
**430 ms** — cheap enough for minutes, so cost is not the constraint. The **ceiling** is: a tier
whose restore-test keeps failing stays due, so the evaluation interval is also its RETRY interval,
and a retry is a multi-GB restore.
- **The due-check runs BEFORE the heavy-operation gate is taken.** Evaluations are frequent now, and
holding that gate for a read that answers "nothing to do" would open a window at every evaluation
in which a starting backup cannot acquire — and a backup that cannot acquire records a failure and
pages the operator (F-A1). Nothing heavy starts before the gate.
- **The candidate picker skips implausible archives.** Under per-archive due-ness an incomplete
1-byte phantom (F-CRIT-2's artefact, which server-side prune does not collect) would be picked
forever, fail forever, never earn proof, and leave the tier due at EVERY evaluation — turning the
evaluation interval into the retry rate for a multi-GB restore. `PickRestoreCandidateOn` now
delegates to the settle-aware picker, so both callers agree.
**Unchanged, deliberately:** the restore-test itself (restore → boot → verify → destroy the scratch),
its journal, crash recovery, the scratch VMID band, the one-heavy-operation gate, success-only proof
credit, and oldest-proven ordering — which survives as the tie-break **between due tiers**.
**New:** `--selftest=restore-test-due` — read-only; prints the per-tier due verdict the scheduler
would act on, with the measured cost of the lookup.
**Live finding, pre-existing and NOT caused by this change (filed as R-185):** on demo-felhom the
agent's PVE token has no ACL on `/storage/felhom-backup`, so the API returns an EMPTY content listing
for that storage (root sees three archives). The host tier has therefore never been restore-testable
on that box, and both R-85's rotation and R-86's due-check report it indistinguishably from "newborn"
("no settled archive yet"). Verified live against `local` (grant present → 3 archives) and
`felhom-backup` (no grant → `{"data":[]}`).
## Releasing publishes, and an unreleasable version cannot pass CI (2026-08-03, R-115 + R-183) — **NO VERSION BUMP**
**No Go code changed, so nothing is bumped and nothing was built.** This is the release path and a
gate; the agent stays **v0.120.0**.
**`scripts/release-agent.sh` — THE way to release.** Build → **tag** → publish → **verify by an
independent download**. Publishing used to be a separate remembered step and was **forgotten three
times in five days** (R-111's seventeen stranded releases, 0.114.0, and 0.120.0 — which sat deployed
on both demo hosts and undownloadable, so a documented-path reinstall would have silently downgraded
them to the pre-merge agent *while reporting success*). R-111's own closing line named this leg and
closed SHIPPED without it; it recurred the same afternoon, which is the evidence that a note is not a
mechanism.
- It **tags** because `felhom-host-install.sh` now fetches this version's sixteen config files from
`raw/tag/v<version>/` (R-183) — a released version without a tag 404s a box mid-install, as root.
- It **verifies by downloading what it just published** and comparing the sha to what it built. The
publish step's own success is a report on its own write; a fetch returning the right bytes is a
different claim, and it is the one that matters.
- It **refuses** a dirty or unpushed tree, and refuses to re-release an existing version — one
version name must never mean two binaries.
- It **does NOT vouch.** Vouching points machines at a version and stays the operator's act.
**`scripts/check-published-versions.py` — the gate (R-115 mechanism (b)).** Every `v<semver>` tag
must have a downloadable package AND a tag tree that serves the agent's configs. Registered in
`agent_gates.py` as **not `--fast`** (it needs network, and a push must not fail because Gitea
blinked), and **the CI workflow now runs the FULL gate set** rather than `--fast` — otherwise the
gate would have been registered and never run, which is the built-but-never-wired failure this
project has shipped four times.
**The invariant is NOT the one the task specified, and the reason was measured.** The task asked for
*"the version the hub tells machines to install must be downloadable"*. That is the better invariant
and CI cannot see it: the hub's artifact manifest answers **401** without a per-customer passphrase,
and Gitea's package LISTING api answers **401** without a token, while the package DOWNLOAD url and
the git TAGS api are both anonymous. Putting an operator credential into CI to close that gap is the
operator's call, not a gate author's. The tag-based invariant needs no credential and **catches all
three recorded instances**, because the release script creates the tag and publishes in one act.
**What it does not catch — the hub vouching a version that was never released at all — is recorded
as R-184 rather than assumed away.**
## docs — v0.120.0 PUBLISHED + vouched, and proven on two reinstalled boxes (2026-08-03, R-178) — **no version bump, nothing built**
**Nothing shipped in this entry.** It records an operational fact the version history could not
otherwise carry: **v0.120.0 had been built, committed at `cd6e267` and deployed to both demo hosts,
but never published.** `GET https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.120.0/felhom-agent`
returned **HTTP 404** (0.119.0 → 200), and the hub's Day-0 manifest accordingly vouched **0.119.0**.
**Why that mattered more than it looks.** `felhom-host-install.sh` step 5 skips the binary install
only when `installed == vouched` *exactly*, so a documented-path reinstall would have replaced the
merge-aware 0.120.0 with the pre-merge 0.119.0 — **and would have succeeded**, because the current
`step_grows` passes `-sysdata-grow 0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error)
therefore never fires. Both demo boxes would have come back on an agent that predates the merge while
every log line read green.
Published this session on an operator ruling, from a clean tree (`git status --porcelain` empty,
`HEAD == origin/main == 4bb84fc3`): `scripts/publish-agent.sh 0.120.0` → upload **HTTP 201**,
round-trip GET verified, `AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`;
then vouched in the hub manifest, which resolved the sha authoritatively from Gitea rather than
trusting the submitted value.
Both reinstalls then fetched and sha-verified it over the real customer path —
`verified sha256 a7763d31b55b5ce7… matches the hub manifest` — and 4b's single-volume grow was
exercised live on both boxes (`data +46G (->70G, ONE volume)` on demo-hp,
`+226G (->250G)` on demo-felhom), each producing `mp0` at `/var/lib/felhom` with **no `mp1`**.
**Filed as the third instance of R-115** (*"publishing is a remembered step"*), which has been
WAITING-ON-OPERATOR since 2026-07-29. Full evidence: `felhom.eu/REPORT.md`.
## v0.120.0 — one data volume (2026-08-03, R-165 · decision D-a · variant V-c) — `build-golden.sh` 2.1.0 → **3.0.0**
**The dedicated backup partition stops existing.** A golden built by `build-golden.sh` v3.0.0 ships a
+39 -2
View File
@@ -63,13 +63,50 @@ internal/storage/ storage observer + durable ids + role/claim classifiers + S
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).
> **RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115).** There used to be a raw `go build` line
> here and a *separate* "Publish" row, so publishing was a step someone had to remember — and it was
> **forgotten three times in five days**, the last leaving agent v0.120.0 deployed on both demo hosts
> and undownloadable, where a documented-path reinstall would have silently downgraded them while
> reporting success. Do not hand-roll the build: the script also creates the `v<version>` git TAG
> that `felhom-host-install.sh` fetches this version's sixteen config files from (R-183), and it
> verifies by an **independent download** rather than trusting the publish step's own output.
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
> than reimplementing it.
>
> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188,
> R-186).** The tag is created before the publish so the build and the tag describe the same commit;
> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
> package makes the published-versions gate correctly fail a correct release — it did, on roughly
> every second release, and R-168 sends that failure to you by mail. The invariant the old order
> protected is asserted directly instead: the gate now also refuses a **published version with no
> tag**. If the push fails after a successful publish the script says so loudly and prints the
> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean.
>
> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath
> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet —
> before this, a rebuild could not reproduce the sha you were vouching. To check any published
> version yourself:
>
> ```bash
> V=0.122.0
> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
> -o /tmp/felhom-agent-check ./cmd/felhom-agent
> sha256sum /tmp/felhom-agent-check
> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
> ```
>
> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to
> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line
> ever changes, change both or one version name means two binaries again.
| Step | Where | One-liner |
|---|---|---|
| Build | DooPlex (local) | `cd /mnt/5_hdd/felhom.eu/git/felhom-agent && git pull && go build -ldflags '-X main.version=<v>' -o /tmp/felhom-agent-<v> ./cmd/felhom-agent` |
| **Release** (build + tag + publish + verify) | DooPlex (local) | `GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver>` — refuses a dirty/unpushed tree and refuses to re-release an existing version |
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
| Deploy | felhom-pve | backup `.bak-<old>``install -m0755``systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
| Publish | DooPlex (local) | `scripts/publish-agent.sh <ver> <bin>` (REGISTRY_* creds); hub Day-0 manifest vouch = operator follow-up |
| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
## Proxmox model (the load-bearing rules)
+59
View File
@@ -5,6 +5,65 @@
## Current
- **2026-08-03 — v0.122.0 (R-189 · R-188 · R-186): three signals that lied about their own work.**
None touches data; all three cost attention, which every other signal depends on.
- **R-189 — a passing restore-test no longer vanishes on a restart.** `restore_tests[]` came only
from the in-memory `backup.Store` (*"lost on restart; the cadence re-populates"* — true under a
timer, FALSE since R-86, because the agent will not re-test a proven archive). **Observed live:**
a 14.5 GB offsite PASS at 15:25:14, agent restarted 2 m 43 s later, hub logged `0 restore-tests`
twice. `RestoreTestState` now stores `tier` + `verified` beside the archive (v3 shape; v1/v2
still read, and a record missing archive-or-tier is NOT reported), exposes
`ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it — **one entry per tier,
newest by `TestedAt` wins**, so a fresh failure beats a stored success and a tier never appears
twice. Wiring pinned by an AST test: the method this replaces (`Snapshot`) claimed a
"host-report gauge" in its doc comment and had **no caller** for weeks.
- **ONLY SUCCESSES ARE PERSISTED, and the reason is now in the code:** a success *suppresses*
future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less
tested than it believes); a failure *causes* future work and heals itself at the next evaluation.
- **R-188 — the release stopped emailing false failures.** Only the tag PUSH moved (build → tag
locally → publish → push tag): the push is what wakes CI, and a tag visible before its package
made the gate correctly fail a correct release ~half the time. The old order's invariant is now
asserted directly — `check-published-versions.py` refuses a **published version with no tag**, as
a bounded, printed probe (the package listing api is still 401 without a token, re-measured).
- **R-186 — a released binary is verifiable.** `-trimpath -buildvcs=false`: same source → same
bytes whether or not the tag exists. Measured. `publish-agent.sh`'s fallback also forced
`CGO_ENABLED=0` and built a **74 KB different** binary for the same version — both paths now
identical. The verification command is in `CLAUDE.md`.
- **2026-08-03 — v0.121.0 (R-86): the restore-test follows the BACKUP, not the clock.** The ticker is
now only the **evaluation interval**; a tier is **DUE** when its newest archive that has settled for
`settle` (default 24 h) **has not been proven**. Daily tier → proved daily on yesterday's archive;
weekly tier → weekly on its own; newborn → UNKNOWN. **The trap, so it is not reintroduced:** the
literal reading of R-86 — *"due when the newest archive is ≥ 24 h old"* — is NEVER true on a daily
tier (a new archive resets the age before it reaches the lag), so it switches restore-testing off
where it matters most. Red-proved at 0 runs over 5 simulated days.
- **The state now records WHICH archive was proven**, not just when a tier passed. A pre-R-86 file
keeps its time (ordering survives) and yields no proven archive → each tier is due once after the
upgrade, deliberately.
- **The old cadence key:** `restore_test_cadence_seconds` is DEPRECATED. Negative still DISABLES
(verbatim); a positive value now seeds the **settle lag** and the daemon WARNs once at start-up
naming `restore_test_eval_interval_seconds` (default 6 h) and `restore_test_settle_seconds`
(default 24 h). It is NOT carried into the evaluation interval.
- **6 h is bounded from both ends:** measured evaluation cost (local 18 ms, PBS-over-WAN 392 ms,
both 430 ms) says cost is irrelevant; the ceiling is that a FAILING tier stays due, so the
evaluation interval is also its retry interval for a multi-GB restore.
- The due-check now runs **before** the heavy-operation gate is taken (a frequent poll must not be
able to make a starting backup record a failure — F-A1), and the candidate picker skips archives
failing `archivePlausiblyComplete` (a phantom would be due forever and fail forever).
- New read-only `--selftest=restore-test-due` prints the per-tier verdict + its cost.
- **v0.121.1 — a quiet evaluation is AUDIBLE.** "Nothing is due" is now the NORMAL outcome, and at
DEBUG it was silent: an empty journal would have been equally consistent with a healthy loop and
a dead goroutine (standing rule 3 — the shape the R-88 watcher was retired for). A not-due
evaluation logs ONE INFO line naming every tier's verdict; an unlistable tier reads `UNKNOWN`
with its error in that same line.
- **PROVEN LIVE 2026-08-03 on demo-felhom:** due-triggered offsite restore-test of a 14.5 GB
encrypted PBS archive — restored, booted, verified, scratch destroyed, **635 s**; the state then
named that archive, a second evaluation ran nothing, and an agent restart ran nothing.
- **R-185 (filed, NOT fixed here):** on demo-felhom the agent token has no ACL on
`/storage/felhom-backup`, so its content listing comes back EMPTY (root sees 3 archives) — the
host tier has never been restore-testable there, and the due-check cannot distinguish that from
a newborn tier.
- **2026-07-28 — v0.107.0: F-REBOOT fixed — a guest rebooted mid-backup now comes back by itself.**
New `internal/localapi/guestpower.go`: a 60 s watchdog that starts a guest which is `onboot:1`,
stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
+270 -30
View File
@@ -1,41 +1,281 @@
# REPORT — CI runs the gate entry point on every push (R-168, 2026-08-02)
# REPORT — R-189 · R-188 · R-186: three ways the signals lied about themselves
**Overwritten** per the standing rule. The prior contents (session 1's gate entry point, same day) have their durable record in `CHANGELOG.md`.
**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.121.1 → v0.122.0** (`7581f81`) · released,
published, verified by an independent download **and by rebuilding it**, deployed to demo-felhom.
`felhom.eu`: register + docs only, **no hub change and no hub bump** — the hub already reads
`restore_tests[]`; the defect was that the agent stopped sending them.
**No version bump, no build, no deploy.** This adds `.gitea/workflows/gates.yml` and a CHANGELOG
entry. Nothing compiled changed.
---
## What changed
## 1. Baselines, re-read on arrival
`.gitea/workflows/gates.yml` — on every push, a Gitea Actions runner obtains this repo at the
**exact pushed SHA** (shallow `git fetch`, no `uses:` step anywhere) and runs
`python3 scripts/agent_gates.py --fast` and nothing else. The exit code is the job's result: no `|| true`, no
pipe that could swallow it.
| Repo | `main` @ commit | Version | Matched §1? |
|---|---|---|---|
| `felhom-agent` | `3d0a1d615d11` | `v0.121.1` | **yes** |
| `felhom.eu` | `c9a3e48b2106` | hub `v0.91.1` | **yes** |
**It REPORTS, it cannot REFUSE**, and the workflow header says so in the pre-push hook's voice: this
repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand
at. The refusing half is `.githooks/pre-push` (per-clone, `--no-verify`-able); this half notices when
that was skipped. Making CI blocking needs branch protection plus a PR workflow — an operator
decision, tracked as `felhom.eu` `OPEN-ITEMS.md` **R-169**.
Highest register ID in use was **R-189**; no new IDs were needed — all three rows already existed.
(Grep confirmed R-190+ free, in case one had been.)
**A failed run emails the operator.** Probe P5 measured that Gitea itself sends **nothing** on a
failed run — no mail, no notification row, no log line — so the workflow sends its own via Resend and
prints the provider's accepted id, making "a message left the machine" an observable. Demonstrated on
a real red run in `felhom.eu`: `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
## 2. Scenario H — the reproducibility measurement (Part 3, done first on purpose)
**CI reproduces the workspace's SIBLING LAYOUT on purpose.** This repo's entry point invokes the
shared `reuse_refs_check.py` that lives in the `felhom.eu` clone next door and is deliberately never
copied here, and this repo's `REUSE.md` cites `hub/internal/store/dr_recipe.go`, which lives in the hub. The workflow clones
`felhom.eu` as a sibling; without it the gate fails **closed** with `gate is MISSING` — correctly,
but for the wrong reason.
**Before**, one commit, same source, same toolchain, same ldflags — the only difference is whether the
tag existed when the build ran:
## Verification
| build | sha256 | size | embedded module version |
|---|---|---|---|
| default flags, **no tag yet** | `18f4a495…` | 14 085 464 B | `v0.121.2-0.20260803133646-3d0a1d61` |
| default flags, **tagged** | `4a38f394…` | 14 085 440 B | `v0.121.99` |
| `-trimpath -buildvcs=false`, either way | `7ffcdf1d…` | 14 064 574 B | *(none)* |
First run: run #1, id=9, conclusion **success**, sha `eb991445`.
**After, on the real release (v0.122.0) — the three values §15.2 asks for:**
**CI and the local pre-push hook AGREE**, which is the check that mattered: `90 cited paths — exact 88, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: felhom.eu)` — identical to the local run.
| artifact | sha256 | size |
|---|---|---|
| published, downloaded from Gitea | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #1 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #2 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
The runner is unprivileged host-mode, shared with the other three repos on a single owner-scoped
registration (measured: all four repos' tasks claimed by `felhom-gates-runner`). Full probe detail,
the security posture and the teardown: `felhom.eu/documentation/audits/SPIKE-ci-runner-2026-08-02.md`
and `felhom.eu/REPORT.md`.
**All three identical.** The property is removed-cause, not sequenced-around: `-buildvcs=false` drops
a stamp nothing reads (no `ReadBuildInfo` caller, verified by grep), and the version still comes from
the explicit `-X main.version` ldflag. `-trimpath` additionally makes a rebuild from a different
checkout directory match.
**A second discrepancy fell out of the measurement and is fixed with it:** `publish-agent.sh`'s
fallback build forced `CGO_ENABLED=0` and produced **13 990 236 B** against the release path's
**14 064 574 B** — a 74 KB difference, i.e. one version name meaning two binaries depending on which
entry point ran. Both paths now use identical flags, each commented with a pointer to the other.
## 3. Scenario E — a correct release no longer emails a failure (Part 2)
**Only the tag PUSH moved.** The order is now build → tag **locally** → publish → push tag. The tag is
still created before anything is published, so the build and the tag describe the same commit; it
becomes *visible* — to CI (`on: [push]`) and to any `raw/tag/…` fetch — only once the package is
downloadable.
**The old ordering's invariant is asserted directly rather than arranged for.**
`check-published-versions.py` now carries two invariants: every tag has an installable package (as
before) **and no published version is missing its tag** (new). The second is a **bounded probe**
the frontier past the newest tag, where a failed tag push leaves an orphan, plus patch gaps — and it
**prints its probe set on every run**, because a check whose coverage is invisible reads as a
guarantee it is not making. The package listing api was **re-measured**, not assumed: `401` without a
token, so absence still cannot be enumerated, and the script says so in its own output.
**Live result — this release is the test:**
| release | CI runs on the release commit | outcome |
|---|---|---|
| v0.121.0 (yesterday) | #12 / #13 | one **green**, one **red** |
| v0.121.1 (yesterday) | #17 / #18 | one **red**, one **green** |
| **v0.122.0 (this one)** | **#21 (task id 96) / #22 (task id 97)** | **both green** |
**Failure modes made loud rather than tidy:** a publish that succeeds followed by a tag push that
fails now dies printing `git push origin v<ver>` — the local tag is already there, so recovery is one
line — and a publish that *fails* deletes the local-only tag so a retry is clean instead of colliding
with step 2's re-release guard.
## 4. Scenarios F and G — both directions demonstrated, then cleaned up
**G — a published version with no tag must FAIL.** A real fixture: version **0.121.2** (the frontier,
exactly where a failed tag push lands) published to the live registry with no tag.
```
0.121.2 next patch after the newest tag PUBLISHED — NO TAG
check-published-versions: 1 PUBLISHED VERSION(S) WITH NO TAG
v0.121.2 is downloadable at … but has no git tag.
git push origin v0.121.2
EXIT=1
```
**Red-proof (observed):** with the fixture still live, replacing the probe set with `[]` produced
`ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED`, **exit 0** — a green run over a published
orphan. Restored.
**Teardown:** fixture deleted (HTTP 204), absence independently re-verified (`GET → 404`), gate green
again (exit 0). No scratch tag was ever pushed.
**F — a tag with no package must still FAIL.** Demonstrated against a local stand-in serving two tags
where only one has a package:
```
ok v0.120.0: binary downloadable + tag serves its configs
FAIL v0.199.0:
- binary NOT downloadable (HTTP 404 …)
- tag does not serve configs/felhom-agent.service (HTTP 404) — a box would 404 mid-install
EXIT=1
```
**Why not a real pushed tag:** pushing one wakes CI and would have emailed the operator a **true**
alarm about a fixture — the same attention cost R-188 exists to remove. F is also already
demonstrated in the wild: CI runs **#13** and **#17** failed for exactly this reason yesterday.
## 5. R-189 — a proof that survives a restart reaches the hub (Part 1)
`restore_tests[]` came only from the in-memory `backup.Store`, whose comment read *"lost on restart;
the cadence re-populates"*. True under a timer; false since R-86, because the agent refuses to re-test
an archive it has already proven — so a lost proof is not repeated for a whole archive generation.
**What changed**
- `RestoreTestState` stores the **tier** and what was **verified** beside the archive (v3 shape).
Both are recorded **at proof time from the run's own result** — deriving them later would need a
storage-type lookup at report-building time, a network call that can fail on the one path where
failing means mislabelling a proof.
- `ProvenRestoreTests` renders the stored proofs as report entries; `Collector.SetProvenRestoreTests`
merges them with the in-memory result.
- **Merge rule: one entry per tier, newest by `TestedAt` wins.** It falls out of what each source
means rather than from a preference: a fresh failure beats a stored success (the failure is the
news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a
tier never appears twice — the hub would read that as two tests. An unparseable timestamp counts as
**older**, so a malformed entry cannot displace a good one.
- **It refuses to lie.** A record missing the archive **or** the tier produces no entry, and run
mechanics (scratch VMID, duration) are not re-invented: an absent duration is not a claim, a
fabricated one would be.
- **The asymmetry is now in the code** (§8.1): a success *suppresses* future work so it must be
durable; a failure *causes* future work and heals itself, and persisting one would make a healed
tier keep reporting a fault.
- **`Store`'s comment is corrected in place** — leaving it is how the next reader concludes this is
handled.
**Migration, and it is visible on the live box:** a pre-R-189 record has an archive but no tier, so it
is **not** reportable. Upgrading does not retroactively make an old proof visible; the tier's next
real proof fills it in. Confirmed immediately after the deploy — still `0 restore-tests`, with the
v2 record sitting on disk.
## 6. Scenario A, live on demo-felhom — against the observation that filed R-189
**The observation being replaced (2026-08-03, 15:25):** a real offsite restore-test PASSED, the agent
was restarted 2 m 43 s later, and the hub logged `0 restore-tests` on the next two host-reports.
**The same sequence, on v0.122.0:**
```
16:44:06 restore-test tier is DUE target=felhom-pbs
archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z
reason="newest settled archive … has not been proven (last proven archive was a different one)"
16:55:21 restore-test: scratch guest torn down vmid=990000
16:55:21 backup: scheduled restore-test PASSED archive=felhom-pbs:…2026-07-27T19:55:41Z duration_s=675.1
16:55:32 systemctl restart felhom-agent ← INSIDE the 15-minute reporting window
16:55:36 hub: host-report from demo-felhom-8363b5 (… 1 restore-tests …) ← was 0
```
**The proof on disk (v3 — the tier is what the old shape lacked):**
```json
{"felhom-pbs": {"archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z",
"tier": "pbs", "verified": "boot+running", "proven_at": "2026-08-03T14:55:21Z"}}
```
**What the HUB stored** — read from its own database (copied with its `-wal`, freshness confirmed by
the newest row's `received_at` = `2026-08-03 14:55:36` UTC, matching the ingest line):
```json
{ "source_archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z",
"source_tier": "pbs", "pass": true, "verified": "boot+running",
"tested_at": "2026-08-03T14:55:21Z", "scratch_vmid": 0, "duration_seconds": 0 }
```
That report was built **after** the restart, when the in-memory store was empty — so the entry can
only have come from the persisted state. The archive, the tier, and the **original** test time
survived; the run mechanics are zero because they are deliberately not re-invented.
**A 14.5 GB encrypted offsite archive**, restored, booted, verified and destroyed in **675 s** — and
this time the proof outlived the process that produced it.
**Teardown, all three layers:** scratch guest absent from `pct list` (0), its volumes gone from `lvs`
(0), the validation drop-in removed and the daemon back on its defaults
(`eval_interval=6h0m0s settle=24h0m0s`). The hub-side `restore_tests[]` record is **retained
deliberately** — it is the proof the staleness check reads, so deleting it would delete the result.
No `restore_test_*` event was raised, because nothing failed and nothing is stale.
## 7. Tests and red-proofs
Green gate: `go build ./... && go vet ./... && go test ./...`**29 packages ok, rc=0**, plus
`python3 scripts/agent_gates.py` (reuse-refs + published-versions) all OK. The test run and the commit
were always separate commands.
| # | Test | Asserts | Mutation | Observed |
|---|---|---|---|---|
| A | `TestMerge_ProofSurvivesARestart` | an empty in-memory store + a persisted proof → the proof is reported, with its archive and its original time | the persisted merge deleted (the pre-R-189 body) | **FAIL**`after a restart the persisted proof must be reported; got 0 entr(ies): []` — the live observation exactly |
| B | `TestMerge_NeverInventsAPassForAnUnprovenTier` | no proof → no entry; a tier-less record → no entry | — (its state-layer twin below carries the mutation) | pass |
| B | `TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe` | v1 + v2 + v3 records side by side → only the describable one is reported | the `reportable()` filter dropped | **FAIL**`got 3` entries, two with an empty `SourceTier`/`SourceArchive` |
| C | `TestMerge_NewerWinsAndNeverDuplicatesATier` | one entry per tier, newest wins, in both directions | de-duplication removed | **FAIL**`one entry per tier; got 2 for "pbs" — the hub would read two tests` |
| D | `TestMerge_AFailureIsStillReported` | a fresh failure beats an older stored success | (same mutation) | **FAIL** — 2 entries, i.e. the failure no longer the single answer for that tier |
| — | `TestMerge_MalformedTimestampNeverWins` | unparseable ≠ newest | — | pass |
| — | `TestMerge_NilProvenSourceIsANoOp` | pre-R-189 behaviour unchanged when unwired | — | pass |
| — | `TestScheduler_ProofIsRecordedReportably` | a pass **through the scheduler** leaves a reportable proof | — | pass |
| — | `TestScheduler_AFailureLeavesNoPersistedProof` | §8.1's asymmetry, asserted not assumed | — | pass |
| G | the gate's converse assertion | a published version with no tag fails | probe set → `[]` | **FAIL** (green over a live orphan) |
| I | `TestMainWiresTheDurableRestoreTestProof` | **AST**: `SetProvenRestoreTests` is called **and fed `rtState`** | the call commented out | **FAIL**`main.go never calls collector.SetProvenRestoreTests` (a `strings.Contains` check would have passed — the string is still there) |
| H | reproducibility | three identical sha256 | — (measurement, §2) | pass |
**Jitter, per §10:** every timestamp fixture uses odd minutes and seconds (`13:25:14`, `19:55:41`,
`13:41:07`, `04:41:58`) — several taken from the real box — rather than round hours. Yesterday a test
was hollow because a perfectly regular series landed exactly on a threshold and survived its own
mutation.
## 8. Files changed
`internal/backup/restoretest_state.go` (v3 record + `ProvenRestoreTests`), `internal/backup/store.go`
(the comment that had become false), `internal/backup/schedule.go` (record tier + verified),
`internal/hub/collect.go` (the seam + the merge), `cmd/felhom-agent/main.go` (wiring),
`scripts/release-agent.sh` (ordering, reproducible build, loud half-done release),
`scripts/publish-agent.sh` (identical build flags), `scripts/check-published-versions.py` (the
converse invariant), plus `REUSE.md`, `CHANGELOG.md`, `CONTEXT.md`, `CLAUDE.md` and three test files.
**Commits**`felhom-agent`: `7581f81` (v0.122.0). `felhom.eu`: see §10.
## 9. The independent-verification command (Part 3, recorded in `CLAUDE.md`)
```bash
V=0.122.0
git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
-o /tmp/felhom-agent-check ./cmd/felhom-agent
sha256sum /tmp/felhom-agent-check
curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
```
Both print `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` (§2).
## 10. Registers
- **R-189 → CLOSED** (shipped + proven live), **R-188 → CLOSED** (shipped), **R-186 → CLOSED**
(shipped + measured).
- **R-185 remains OPEN and untouched** — it is a missing `/storage/felhom-backup` ACL on demo-felhom,
a permission defect, not a reporting one. Nothing in this session changed it, and the priority list
says so explicitly.
- No new IDs minted. `ROADMAP.md` contains none of these three rows, so there was nothing to collapse.
- `00-capability-map.md`'s restore-proof row now records that the evidence path itself had a gap and
what closed it; `CONTEXT.md` gains **S-19** (the proof/failure asymmetry and the merge rule) and
**S-20** (the release ordering and what each step protects); `STATUS.md` rewritten for the operator
and trimmed to 83 lines.
## 11. Observations — noticed, recorded, NOT acted on
- **The proof state holds ONE record per tier, so proving an OLDER archive re-arms a newer one —
CONFIRMED after the validation, not merely predicted.** With the defaults restored, the due-check
reads: `tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z" proven="…2026-07-27T19:55:41Z" —
newest settled archive has not been proven (last proven archive was a different one)`.
Surfaced by this session's own validation method: to get a fresh proof without waiting a week, the
settle lag was widened so the older, unproven offsite archive became the candidate. That overwrote
the record for the newer archive, so once the default 24 h settle returns, the newest settled
archive is no longer the recorded proof and the tier becomes due once more. **Consequence, stated
rather than left to surprise: demo-felhom will run one further unattended offsite restore-test
within 6 h, after which the newest archive is the recorded proof** — the correct steady state. In
normal operation this cannot arise, because the candidate only ever moves forward.
- **`RestoreTestState.Snapshot()` has no caller again.** The host report is now fed by
`ProvenRestoreTests`, which carries what a bare timestamp cannot. The method's doc comment says in
as many words that it should be deleted if it does not acquire one — deliberately not deleted in
this session, because removing an exported method is a change with no bearing on the three rows.
- **Ten files in this repo are not `gofmt`-clean and were already so on arrival**
(`internal/capability/probe.go`, `internal/escrow/consume.go`, `internal/mgmtplane/mgmtplane.go`,
`internal/reconcile/bringup.go`, `internal/signedjobs/runner.go`, `internal/storage/{candidates,
intent}.go` and three test files). Every file this session touched is clean; the others are
untouched, and no gate checks formatting.
- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for the restore-test
staleness check. Unchanged here and not a defect at this fleet size; it is the cost centre if the
fleet grows, and it is the reason the window read was deliberately left at 14 days yesterday.
- **`felhom.eu/CONTEXT.md` still carries duplicate standing-ruling IDs** (three `S-14`s, two `S-15`s)
from before yesterday. New rulings continue to be numbered above the collision (S-19, S-20) rather
than adding to it; renumbering the existing ones is a separate, purely editorial change.
+3 -2
View File
@@ -148,8 +148,9 @@
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,t)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test rotation state, persisted (atomic tmp+rename) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target) (string,error)` | The per-run restore-test spec + per-tier candidate lookup (R-85) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", nil)`**`""` is NOT an error**, or every fresh box looks broken for its first week. |
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename)**which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)`**`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
+198 -7
View File
@@ -165,7 +165,7 @@ func main() {
showVersion bool
)
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `restore-test-due` = READ-ONLY: print the per-tier due verdict the scheduler would act on, with its cost; `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up")
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)")
@@ -238,6 +238,8 @@ func main() {
os.Exit(runSelftestBackup(context.Background(), cfg, logger, vmid))
case "restore-test":
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
case "restore-test-due":
os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
case "pbs-verify":
os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
case "lanresolver":
@@ -409,6 +411,110 @@ func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
return s
}
// storeGrantStatuses probes whether the agent's OWN TOKEN may read the storages this box depends
// on — one capability.Status per configured backup tier (R-185).
//
// ── WHY THIS EXISTS, AND WHY IT IS NOT A CONTENT LISTING ─────────────────────────────────────
//
// On demo-felhom the token had FelhomAgentStore on local, local-lvm and felhom-pbs — and NOT on
// `felhom-backup`, the storage the same installer had configured as `local_backup_target`. Asked
// for that storage's content the API answers `{"data":[]}` while root sees three archives.
//
// **An empty listing is what a FORBIDDEN tier and a NEWBORN tier both return**, and no care at that
// call site can separate them: `pickForThisRun` skips an empty tier (correctly — a fresh offsite
// tier legitimately has nothing) and says "no settled archive yet". So the host tier on that box was
// never restore-testable and nothing ever mentioned it. That is this project's own rule failing in a
// new place: an empty answer is not evidence that there is nothing there.
//
// The permission question, unlike the listing, has a DEFINITE answer — so it is asked directly.
//
// ── WHAT IS PROBED, AND WHY NOT A FIXED LIST ─────────────────────────────────────────────────
//
// The tiers come from this box's own config (`BackupTiers()`), because a hardcoded probe list is
// precisely the defect being fixed — the installer's hardcoded ACL set is what drifted from the
// target it went on to configure. Probing what the box says it depends on cannot drift from it.
//
// CRITICAL, deliberately: a tier the agent cannot read is a tier whose backups are invisible to it
// and which is never restore-tested. The hub alerts only on Critical, and a non-critical entry here
// would ride the report and alert nobody — the same silence with extra steps.
//
// One exception, so an ordinary configuration is not turned into an alarm: a box with no dedicated
// target (`local_backup_target: "local"`, which host-install's own comment calls the DEGRADED
// fallback) is not treated as critical for that tier — see storeGrantCritical.
func storeGrantStatuses(ctx context.Context, px *proxmox.Client, cfg config.Config) []capability.Status {
tiers, _ := cfg.Backup.BackupTiers() // warnings are logged where the tiers are armed
out := make([]capability.Status, 0, len(tiers))
for _, t := range tiers {
out = append(out, storeGrantStatus(ctx, px, t.TargetID, storeGrantCritical(t.TargetID)))
}
return out
}
// storeGrantRequiredPriv is the privilege whose ABSENCE was measured to blind the content listing.
//
// Measured on demo-felhom 2026-08-03: the two storages that list through the token hold
// Datastore.Allocate + Datastore.AllocateSpace (the FelhomAgentStore role); the one that answers
// empty holds only what the box-wide grant propagates (Sys.Audit, SDN.Use, Datastore.Audit). It is
// NOT Datastore.Audit that is missing — checking for that would report the blinded storage healthy.
const storeGrantRequiredPriv = "Datastore.AllocateSpace"
// storeGrantCritical decides whether a missing grant on this target is Critical (operator-paged).
//
// "local" is host-install's DEGRADED fallback target — a box with no dedicated backup storage is a
// known, ordinary configuration, and turning it into a critical alert is how a signal becomes
// something an operator archives unread. It is still probed and still reported; only the paging
// differs.
func storeGrantCritical(targetID string) bool { return targetID != "local" }
// storeGrantStatus is one tier's grant probe. It NEVER reports ok when it could not ask: a
// self-check that fails open is worse than none, because it converts "I do not know" into "fine".
func storeGrantStatus(ctx context.Context, px *proxmox.Client, targetID string, critical bool) capability.Status {
s := capability.Status{
Name: "pve:store-grant:" + targetID,
Feature: "backup tier " + targetID + " readable by the agent (archive listing, restore-test candidacy)",
Critical: critical,
Status: capability.StatusOK,
}
if px == nil {
s.Status, s.Reason = capability.StatusDegraded, "not configured"
return s
}
if targetID == "" {
s.Status, s.Reason = capability.StatusDegraded, "tier has no target id"
return s
}
pctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
privs, err := px.Permissions(pctx, "/storage/"+targetID)
return storeGrantVerdict(targetID, critical, privs, err)
}
// storeGrantVerdict is the DECISION, split out from the API call so the tests exercise the real
// thing rather than a copy of it. A test that re-implements this branch would pass while production
// diverged — which is the hollow shape this project keeps finding in its own tests.
func storeGrantVerdict(targetID string, critical bool, privs map[string]int, err error) capability.Status {
s := capability.Status{
Name: "pve:store-grant:" + targetID,
Feature: "backup tier " + targetID + " readable by the agent (archive listing, restore-test candidacy)",
Critical: critical,
Status: capability.StatusOK,
}
if err != nil {
// Unreachable PVE is UNKNOWN, and unknown is reported as degraded rather than ok: a
// self-check that fails open converts "I do not know" into "fine".
s.Status, s.Reason = capability.StatusDegraded, "could not read own permissions: "+err.Error()
return s
}
if privs[storeGrantRequiredPriv] != 1 {
// Name the storage AND the missing role: "a storage grant is missing" without saying which
// one costs a diagnosis at 07:00.
s.Status, s.Reason = capability.StatusDegraded,
"the agent token lacks "+storeGrantRequiredPriv+" on /storage/"+targetID+
" (grant FelhomAgentStore there) — this tier's archives are INVISIBLE to the agent and it is never restore-tested"
}
return s
}
// logCapabilities logs the privileged-capability self-check at startup: one INFO summary, plus an
// ERROR per degraded capability naming the gated feature (so a missing grant is loud at cutover,
// not days later). Inactive (config-gated off, plumbing healthy — v0.86.0) is counted in the
@@ -490,8 +596,11 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// A1 (v0.62.0): compose the PVE pool-read check AROUND the sudo prober (an API read does not
// belong inside the sudo-policy probe). Non-critical: a degraded pool read means the stale-lock
// reaper fail-safes (locks stay uncleared) — visible on the hub report, no operator page.
// R-185: the store-grant probes compose around the sudo prober the same way the pool read does
// (an API read does not belong inside the sudo-policy probe — the v0.62.0 A1 precedent).
probeAll := func(ctx context.Context) []capability.Status {
return append(capProber.Probe(ctx), poolReadStatus(ctx, px))
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
return append(out, storeGrantStatuses(ctx, px, cfg)...)
}
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
// already carries the gated view — v0.86.0.)
@@ -660,6 +769,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
heavyOps := &backup.InFlight{}
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
// holds only this process's latest run, and under per-archive due-ness the agent will not
// re-test an archive it has already proven — so without this the hub can report a tier unproven
// for a whole archive generation after a deploy. Observed live on 2026-08-03: a passing 14.5 GB
// offsite restore-test reached no host-report at all.
collector.SetProvenRestoreTests(rtState)
// PBS verify loop (slice 6 Phase B): the fifth daemon goroutine. Cheap, key-free,
// ciphertext-level integrity check on its own cadence (default 6h), reporting per-snapshot
@@ -1273,13 +1388,21 @@ func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
// machinery still works on-demand via --selftest=restore-test.
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler {
cadence := cfg.Backup.RestoreTestCadence()
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
// per-archive due-check in internal/backup/restoretest_due.go.
cadence := cfg.Backup.RestoreTestEvalInterval()
if cadence > 0 {
if err := cfg.Backup.ValidateForRestoreTest(); err != nil {
logger.Warn("daemon: restore-test cadence disabled (config invalid)", "err", err)
logger.Warn("daemon: restore-test disabled (config invalid)", "err", err)
cadence = 0
}
}
if cadence > 0 && cfg.Backup.RestoreTestLegacyCadenceInUse() {
// Said ONCE, at start-up, naming both replacements: a key whose meaning changed under a box
// without a word is the silent repurposing R-86 §8.3 forbids.
logger.Warn("daemon: backup.restore_test_cadence_seconds is DEPRECATED — R-86 replaced the interval trigger with a per-archive due-check; this value now seeds the SETTLE lag only. Set backup.restore_test_settle_seconds and backup.restore_test_eval_interval_seconds explicitly",
"settle", cfg.Backup.RestoreTestSettle(), "eval_interval", cadence)
}
min, max := cfg.Backup.ScratchBand()
target := cfg.Backup.BackupTarget()
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
@@ -1315,13 +1438,18 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
}
},
Cadence: cadence,
Logger: logger,
// R-86: the settle lag — how long an archive must have sat before it is a candidate. With
// the per-archive due-check, this plus the archive rhythm is the whole schedule.
Settle: cfg.Backup.RestoreTestSettle(),
Logger: logger,
// R-85: rotate across EVERY configured tier, oldest-proven first (operator ruling, Option 1).
// Before this the scheduler only ever saw cfg.Backup.BackupTarget(), so the offsite tier's
// archives were never candidates and the DR tier went unproven for its whole existence.
// R-86 demoted that ordering to the tie-break BETWEEN DUE TIERS and widened this picker to
// the settle-aware one, which is what makes due-ness per archive generation.
Tiers: tierIDs,
TierPick: runner.PickRestoreCandidateOn,
TierPick: runner.PickSettledRestoreCandidateOn,
State: rtState,
InFlight: inFlight,
})
@@ -1751,6 +1879,67 @@ func runSelftestBackup(ctx context.Context, cfg config.Config, logger *slog.Logg
// running → teardown) of -archive (or the newest backup on the local target) into a scratch
// guest. Standalone (no hub). Runs engine.Recover first so a leaked scratch from a prior
// crashed test is reaped before this run.
// runSelftestRestoreTestDue prints the per-tier DUE verdict the scheduler would act on, and what
// each evaluation COST — read-only, so it is safe on any box at any time.
//
// It exists for two reasons R-86 needed and could not get from a log line. First, the due-check's
// verdict is the whole schedule now: "why did nothing run last night?" is answerable only by asking
// the same question the scheduler asks, against the same storages, in the same order. Second, the
// evaluation interval had to be chosen from a MEASURED cost rather than a guess — an offsite tier's
// candidate lookup crosses the WAN, and a monitoring loop that costs more than it is worth is how a
// check becomes the load. It reuses the daemon's own construction path (buildRestoreTestScheduler),
// so what it prints is what the daemon would decide, not a re-derivation of it.
func runSelftestRestoreTestDue(ctx context.Context, cfg config.Config, logger *slog.Logger) int {
if err := cfg.Validate(); err != nil {
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
return 1
}
px, err := newProxmoxClient(cfg)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest: proxmox client:", err)
return 1
}
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, logger)
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
start := time.Now()
verdicts := sched.EvaluateDue(ctx)
total := time.Since(start)
if len(verdicts) == 0 {
fmt.Println("no tiers configured for restore-testing (or rotation not wired)")
return 0
}
rc := 0
for _, v := range verdicts {
proven, _ := rtState.ProvenArchive(v.Target)
fmt.Printf("tier=%-16s due=%-5v archive=%q landed=%s proven=%q\n reason: %s\n",
v.Target, v.Due, v.Archive, formatOrDash(v.Landed), proven, v.Reason)
if v.Err != nil {
// A tier we could not list is UNKNOWN, and it is a non-zero exit: an unreadable tier is
// a real condition, not a quiet "nothing to do".
fmt.Printf(" ERROR: %v\n", v.Err)
rc = 3
}
}
// Per-tier timing, measured one tier at a time so the WAN leg is attributable (R-86 Part 1.4).
for _, v := range verdicts {
t0 := time.Now()
_ = sched.EvaluateDueTier(ctx, v.Target)
fmt.Printf("cost tier=%-16s one_lookup=%s\n", v.Target, time.Since(t0).Round(time.Millisecond))
}
fmt.Printf("cost all_tiers=%s\n", total.Round(time.Millisecond))
return rc
}
// formatOrDash renders a time, or "-" when it is zero (no archive).
func formatOrDash(t time.Time) string {
if t.IsZero() {
return "-"
}
return t.UTC().Format(time.RFC3339)
}
func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog.Logger, archive string) int {
if err := cfg.Validate(); err != nil {
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
@@ -2940,6 +3129,8 @@ func (f *selftestFlag) Set(v string) error {
f.mode = "backup"
case "restore-test":
f.mode = "restore-test"
case "restore-test-due":
f.mode = "restore-test-due"
case "pbs-verify":
f.mode = "pbs-verify"
case "lanresolver":
@@ -2957,7 +3148,7 @@ func (f *selftestFlag) Set(v string) error {
case "controller-swap":
f.mode = "controller-swap"
default:
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
}
return nil
}
@@ -0,0 +1,156 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// R-86 Scenario I — the seam-discipline test for the due-check.
//
// A due-check is worth nothing if the daemon still wires the OLD picker: every unit test in
// internal/backup would stay green (they inject the seam directly), the scheduler would ask for the
// newest archive with no settle cutoff, and the per-archive rule would run against a candidate that
// changes every time a backup lands. That is the same shape as the v0.91.0 inert seam — built,
// tested, never called — and this repo has shipped it four times.
//
// It walks main.go's AST rather than grepping: a commented-out call still satisfies a substring
// match, and a comment is not a caller.
func TestMainWiresTheSettleAwareTierPicker(t *testing.T) {
f := parseMainForWiring(t)
var settlePicker, oldPicker, settleWired, evalInterval bool
ast.Inspect(f, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.SelectorExpr:
// runner.PickSettledRestoreCandidateOn passed as a value (not called).
switch node.Sel.Name {
case "PickSettledRestoreCandidateOn":
settlePicker = true
case "PickRestoreCandidateOn":
oldPicker = true
}
case *ast.KeyValueExpr:
key, ok := node.Key.(*ast.Ident)
if !ok {
return true
}
if key.Name == "Settle" {
settleWired = true
}
case *ast.CallExpr:
if sel, ok := node.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "RestoreTestEvalInterval" {
evalInterval = true
}
}
return true
})
if !settlePicker {
t.Error("main.go never passes runner.PickSettledRestoreCandidateOn as the scheduler's TierPick — " +
"the due-check would run without a settle cutoff, i.e. against an archive that may still be being written")
}
if oldPicker {
t.Error("main.go still wires the pre-R-86 PickRestoreCandidateOn as a tier picker — " +
"two pickers means the one under test is not the one running")
}
if !settleWired {
t.Error("main.go never sets SchedulerOptions.Settle — the settle lag would default to 0 in the daemon " +
"and every freshly-landed archive would be an immediate candidate")
}
if !evalInterval {
t.Error("main.go never calls cfg.Backup.RestoreTestEvalInterval() — the scheduler would be driven by " +
"the retired cadence knob")
}
}
// The two R-85 guarantees the due-check must not have quietly dropped: the spec is still built PER
// RUN, and the shared heavy-operation gate is still handed to the scheduler.
func TestMainStillWiresTheHeavyOperationGateAndPerRunSpec(t *testing.T) {
f := parseMainForWiring(t)
var inFlightWired, specIsAFunc bool
ast.Inspect(f, func(n ast.Node) bool {
kv, ok := n.(*ast.KeyValueExpr)
if !ok {
return true
}
key, ok := kv.Key.(*ast.Ident)
if !ok {
return true
}
switch key.Name {
case "InFlight":
inFlightWired = true
case "Spec":
// A FuncLit means it is evaluated per run; anything else is a frozen value.
if _, isFunc := kv.Value.(*ast.FuncLit); isFunc {
specIsAFunc = true
}
}
return true
})
if !inFlightWired {
t.Error("main.go no longer hands the scheduler the shared InFlight gate — a restore-test could pull a " +
"multi-GB archive over the same tunnel an offsite backup is pushing one over (Scenario F)")
}
if !specIsAFunc {
t.Error("SchedulerOptions.Spec is no longer a function literal — a frozen spec is the R-85 defect " +
"(the tier and its timeout evaluated once at daemon start, forever)")
}
}
func parseMainForWiring(t *testing.T) *ast.File {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
return f
}
// R-189 Scenario I — the DURABLE proof source must actually be wired into the collector.
//
// This test exists because the method it feeds is the project's own cautionary tale:
// `RestoreTestState.Snapshot` carried the doc comment "for the host-report gauge" from the day it
// was written and **had no caller at all** — a seam built, documented and never connected, found
// only when a live restore-test's PASS reached no host-report. The fix must not become the next
// instance, so the wiring is asserted rather than trusted.
//
// AST, not grep: a commented-out call still contains the string (proven yesterday, when commenting
// out the tier-picker line failed this test while a `strings.Contains` check would have passed).
func TestMainWiresTheDurableRestoreTestProof(t *testing.T) {
f := parseMainForWiring(t)
var wired, feedsState bool
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "SetProvenRestoreTests" {
return true
}
wired = true
// ...and it must be fed the PERSISTED state, not the in-memory store.
if len(call.Args) == 1 {
if id, ok := call.Args[0].(*ast.Ident); ok && id.Name == "rtState" {
feedsState = true
}
}
return true
})
if !wired {
t.Error("main.go never calls collector.SetProvenRestoreTests — the persisted proof would never " +
"reach the hub, which is the R-189 defect exactly: a passing restore-test that vanishes on restart")
}
if wired && !feedsState {
t.Error("collector.SetProvenRestoreTests is not fed rtState — the in-memory store is the thing " +
"that does NOT survive a restart, so wiring it here would fix nothing")
}
}
+166
View File
@@ -0,0 +1,166 @@
package main
import (
"context"
"go/ast"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
)
// R-185 — a tier the box cannot READ must say so.
//
// THE OBSERVATION (demo-felhom, 2026-08-03, reproduced at the start of this session): root lists
// three archives on `felhom-backup`; the agent's own token gets `{"data":[]}` from the same
// endpoint; and `local`, which has the grant, lists through that same token. The token is the
// variable, not the storage.
//
// The defect is NOT the missing grant — that is one command. It is that an empty content listing is
// what a FORBIDDEN tier and a NEWBORN tier both return, so the box could not tell them apart and
// said nothing. These tests pin the distinction.
// permAnswer is the shape /access/permissions really returns, taken from the live measurement:
// an UNGRANTED path answers with the privileges inherited from the box-wide grant — NOT empty, and
// NOT a 403.
var (
permGranted = map[string]int{"Datastore.Allocate": 1, "Datastore.AllocateSpace": 1}
permUngranted = map[string]int{"Sys.Audit": 1, "SDN.Use": 1, "Datastore.Audit": 1}
)
// probeWith calls the PRODUCTION decision with a permissions answer. **Naming the seam:** everything
// below is true up to `storeGrantVerdict`; that the live call feeds it the real API answer is what
// Part 0's measurement established and what the live run on the box demonstrates. An earlier draft
// of this file re-implemented the branch here — it passed, and would have kept passing while
// production diverged, which is the hollow shape this project keeps catching in its own tests.
func probeWith(privs map[string]int, targetID string, critical bool) capability.Status {
return storeGrantVerdict(targetID, critical, privs, nil)
}
// ── SCENARIO A — a forbidden storage is REPORTED, not passed over ────────────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): delete the store-grant probes from `probeAll` in
// main.go — i.e. restore `append(capProber.Probe(ctx), poolReadStatus(ctx, px))` — and
// TestMainWiresTheStoreGrantProbe fails with "main.go never calls storeGrantStatuses". That is
// today's behaviour on the live box: complete silence about a tier it cannot read.
func TestStoreGrant_ForbiddenStorageIsDegradedAndNamed(t *testing.T) {
s := probeWith(permUngranted, "felhom-backup", true)
if s.Status != capability.StatusDegraded {
t.Fatalf("a storage the agent may not read must be DEGRADED, not %q — silence is the defect", s.Status)
}
if !s.Critical {
t.Fatal("it must be CRITICAL: the hub alerts only on critical, so a non-critical entry is the same silence with extra steps")
}
if !strings.Contains(s.Reason, "felhom-backup") {
t.Fatalf("the reason must NAME the storage — 'a grant is missing' costs a diagnosis at 07:00; got %q", s.Reason)
}
if !strings.Contains(s.Reason, "FelhomAgentStore") {
t.Fatalf("the reason must name the ROLE to grant, so the fix is in the alert; got %q", s.Reason)
}
}
// THE TRAP THE LIVE MEASUREMENT CAUGHT, pinned so it cannot be re-introduced: the ungranted answer
// is not empty and not a 403 — it carries the INHERITED box-wide privileges. A probe that asked
// "did the path come back?" or "does it have Datastore.Audit?" would report the blinded storage
// healthy.
func TestStoreGrant_InheritedPrivilegesAreNotAGrant(t *testing.T) {
if len(permUngranted) == 0 {
t.Fatal("fixture wrong: the ungranted answer is NOT empty — that is the whole trap")
}
if permUngranted["Datastore.Audit"] != 1 {
t.Fatal("fixture wrong: the ungranted path DOES carry Datastore.Audit, inherited box-wide")
}
if s := probeWith(permUngranted, "felhom-backup", true); s.Status != capability.StatusDegraded {
t.Fatalf("checking for the wrong privilege reports a blinded storage healthy; got %q", s.Status)
}
// ...and the privilege actually checked is the one whose absence was measured to blind listing.
if storeGrantRequiredPriv != "Datastore.AllocateSpace" {
t.Fatalf("the probed privilege changed to %q — re-measure before trusting it", storeGrantRequiredPriv)
}
}
// ── SCENARIO B — a newborn tier is still silent ──────────────────────────────────────────────
//
// A storage the agent IS allowed to read but which simply holds no archives yet is HEALTHY. The
// probe must not look at content at all, or every freshly provisioned box alarms and the signal dies.
//
// COMPANION RED-PROOF (observed): make the probe degrade on an empty content listing instead of on
// the permission — a granted-but-empty storage then reports degraded, i.e. every newborn box alarms.
func TestStoreGrant_GrantedButEmptyIsHealthy(t *testing.T) {
s := probeWith(permGranted, "felhom-pbs", true)
if s.Status != capability.StatusOK {
t.Fatalf("a readable tier is healthy whether or not it holds archives yet; got %q (%s)", s.Status, s.Reason)
}
if s.Reason != "" {
t.Fatalf("a healthy probe carries no reason; got %q", s.Reason)
}
}
// ── SCENARIO C — the two states are distinguishable at a glance ──────────────────────────────
func TestStoreGrant_ForbiddenAndNewbornAreDistinguishable(t *testing.T) {
forbidden := probeWith(permUngranted, "felhom-backup", true)
newborn := probeWith(permGranted, "felhom-pbs", true)
if forbidden.Status == newborn.Status {
t.Fatalf("the two states must differ — today both read as 'no settled archive yet'; got %q for both", forbidden.Status)
}
if forbidden.Name == newborn.Name {
t.Fatalf("each tier needs its own capability id, or one tier's fault hides another's; got %q twice", forbidden.Name)
}
}
// §8.3, weighed once and pinned: a box with NO dedicated target ("local" — host-install's own
// DEGRADED fallback) must not turn an ordinary configuration into an operator page. It is still
// probed and still reported; only the paging differs.
func TestStoreGrant_TheFallbackTargetIsNotCritical(t *testing.T) {
if storeGrantCritical("local") {
t.Fatal("a box whose backup target is the 'local' fallback must not page the operator about " +
"an ordinary, documented configuration")
}
for _, dedicated := range []string{"felhom-backup", "felhom-pbs", "some-nvme"} {
if !storeGrantCritical(dedicated) {
t.Fatalf("a DEDICATED target that cannot be read is user-facing and must be critical; %q was not", dedicated)
}
}
// The fallback is still reported — silence for it would be the original defect, scoped smaller.
if s := probeWith(permUngranted, "local", storeGrantCritical("local")); s.Status != capability.StatusDegraded {
t.Fatalf("the fallback target must still report degraded when unreadable; got %q", s.Status)
}
}
// A probe that cannot ask must never answer "ok" — unknown reported as healthy is worse than no
// probe, because it looks like coverage.
func TestStoreGrant_UnreachablePVEIsDegradedNotOK(t *testing.T) {
s := storeGrantStatus(context.Background(), nil, "felhom-backup", true)
if s.Status != capability.StatusDegraded {
t.Fatalf("an unaskable probe must be DEGRADED, never ok; got %q", s.Status)
}
if s.Reason == "" {
t.Fatal("it must say why it could not ask")
}
}
// ── SCENARIO H — the seam ────────────────────────────────────────────────────────────────────
//
// This project's "built but never wired" count reached six last week. The fix for a SILENCE must not
// itself be silent. AST, not grep: a commented-out call still contains the string.
func TestMainWiresTheStoreGrantProbe(t *testing.T) {
f := parseMainForWiring(t)
var wired bool
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "storeGrantStatuses" {
wired = true
}
return true
})
if !wired {
t.Error("main.go never calls storeGrantStatuses — the probe would exist and report to nobody, " +
"which is precisely the silence R-185 is about")
}
}
+24 -3
View File
@@ -135,10 +135,11 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
}
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
const big = 4 << 30 // a plausible whole-guest archive
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "a", Content: "backup", CTime: 10},
{VolID: "b", Content: "backup", CTime: 99},
{VolID: "iso", Content: "iso", CTime: 999}, // not a backup → ignored
{VolID: "a", Content: "backup", CTime: 10, Size: big},
{VolID: "b", Content: "backup", CTime: 99, Size: big},
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
@@ -152,6 +153,26 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
}
}
// R-86: the NEWEST entry is not a candidate if it cannot be a complete archive. An incomplete
// artefact (F-CRIT-2's 1-byte phantom, which server-side prune does not collect) would otherwise be
// picked forever, fail its restore forever, never earn proof, and so leave the tier due at every
// evaluation — turning the evaluation interval into the retry rate for a multi-GB restore.
//
// COMPANION RED-PROOF (observed): drop the `archivePlausiblyComplete` guard from
// PickSettledRestoreCandidateOn and this fails with
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "real" {
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
}
}
// --- scheduler ---
type fakeRTRunner struct {
+168
View File
@@ -0,0 +1,168 @@
package backup
import (
"context"
"fmt"
"time"
)
// R-86 — a restore-test follows the BACKUP, not the clock.
//
// ── WHAT WAS WRONG ───────────────────────────────────────────────────────────────────────────
//
// The trigger was `time.NewTicker(cadence)` started at daemon start, and the tier was chosen by
// oldest-proven rotation. Its phase was therefore the PROCESS'S UPTIME: agent deploys are routine,
// so the test drifted to an arbitrary time of day every week; a fresh archive could sit unproven
// while an older one was re-tested; and a weekly tier was tested on the same rhythm as a daily one,
// sometimes twice on the same archive.
//
// ── THE RULE, AND THE TRAP IN ITS OBVIOUS FORM ───────────────────────────────────────────────
//
// R-86's ask reads "test a tier ~24 h after its own newest archive". Implemented literally —
// *"due when the newest archive is at least `settle` old"* — a DAILY tier is NEVER due: a new
// archive lands every day, so the newest archive's age resets to zero long before it reaches 24 h.
// The naive rule silently switches restore-testing off for the tier that matters most, and it is
// the version a reasonable person would write. It has a red-proof of its own
// (TestDue_NaiveNewestArchiveAgeRuleNeverFiresOnADailyTier).
//
// The rule implemented here:
//
// Let A = the newest archive on this tier that is at least `settle` old.
// The tier is DUE when A exists and A HAS NOT ALREADY BEEN PROVEN.
//
// daily tier → A is yesterday's archive; a new one settles each day → proved once per day
// weekly tier → A is last week's until the next settles → proved once per week
// newborn tier → A does not exist → UNKNOWN, never a fault
//
// Per-archive due-ness IS the pacing: one test per archive generation and no more. There is
// deliberately no second rate limiter on top of it (§8.4) — two independent pacing mechanisms
// produce a cadence nobody can predict from either.
//
// ── WHAT DID NOT CHANGE ──────────────────────────────────────────────────────────────────────
//
// The one-heavy-operation gate, the success-only proof credit, the oldest-proven ordering (now the
// tie-break between two DUE tiers), the restore-test itself, its journal and its scratch band. Only
// the trigger changed.
// DueVerdict is one tier's due-ness, and the evidence for it. Every field is logged: a due-check
// that cannot say WHY is a quiet path, and quiet paths are what this monitor family keeps shipping.
type DueVerdict struct {
Target string // the tier's storage target id
// Due is true only when Archive is set and has not been proven.
Due bool
// Archive is the settled candidate A ("" when the tier holds none).
Archive string
// Landed is when A landed on the tier (zero when Archive is "").
Landed time.Time
// ProvenArchive is what the state says was last proven on this tier ("" = nothing/legacy).
ProvenArchive string
// Err is a candidate-lookup failure. A tier whose archives cannot be listed is UNKNOWN — it is
// NEVER reported as "not due", which would silently retire a tier the moment its storage
// stopped answering. Due stays false (we have no archive to test) and the error travels.
Err error
// Reason is the one-line human account of this verdict.
Reason string
}
// String renders a verdict for the operator log / selftest output.
func (v DueVerdict) String() string {
return fmt.Sprintf("tier=%s due=%v archive=%q reason=%s", v.Target, v.Due, v.Archive, v.Reason)
}
// EvaluateDue returns the due verdict for every configured tier, ordered oldest-proven first.
//
// Ordering is the R-85 rotation, demoted to a TIE-BREAK: it no longer decides whether a test
// happens (due-ness does), only which of several due tiers goes first. Keeping it means a tier can
// still never be starved — a tier that has waited longest is served first — and keeping it as the
// order rather than as the trigger is the whole of this change.
func (s *Scheduler) EvaluateDue(ctx context.Context) []DueVerdict {
if !s.rotating() {
return nil
}
order := s.tiers
if s.rtState != nil {
order = s.rtState.OldestFirst(s.tiers)
}
cutoff := s.settleCutoff()
out := make([]DueVerdict, 0, len(order))
for _, target := range order {
out = append(out, s.evaluateTier(ctx, target, cutoff))
}
return out
}
// settleCutoff is the newest landing time an archive may have and still count as settled.
func (s *Scheduler) settleCutoff() time.Time {
if s.settle <= 0 {
return time.Time{} // no settle requirement configured → any archive is a candidate
}
return s.now().Add(-s.settle)
}
// evaluateTier is the per-tier due-check. PURE given the picker and the state, so the rule is
// unit-tested directly rather than inferred from whether a fake runner happened to be called.
func (s *Scheduler) evaluateTier(ctx context.Context, target string, cutoff time.Time) DueVerdict {
v := DueVerdict{Target: target}
archive, landed, err := s.tierPick(ctx, target, cutoff)
if err != nil {
// UNKNOWN, never "not due", and never silent.
v.Err = err
v.Reason = fmt.Sprintf("candidate lookup FAILED (%v) — tier is unknown this evaluation, not proven and not dismissed", err)
return v
}
v.Archive, v.Landed = archive, landed
if archive == "" {
v.Reason = "no settled archive yet — nothing to prove (newborn or still settling)"
return v
}
proven, ok := "", false
if s.rtState != nil {
proven, ok = s.rtState.ProvenArchive(target)
}
v.ProvenArchive = proven
if ok && proven == archive {
v.Reason = fmt.Sprintf("newest settled archive (landed %s) is already proven", landed.Format(time.RFC3339))
return v
}
v.Due = true
switch {
case !ok && proven == "":
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven; nothing proven on this tier yet", landed.Format(time.RFC3339))
default:
v.Reason = fmt.Sprintf("newest settled archive (landed %s) has not been proven (last proven archive was a different one)", landed.Format(time.RFC3339))
}
return v
}
// EvaluateDueTier is EvaluateDue for ONE named tier — the selftest's per-tier cost probe, so the
// WAN leg of an offsite lookup is attributable rather than buried in an aggregate.
func (s *Scheduler) EvaluateDueTier(ctx context.Context, target string) DueVerdict {
return s.evaluateTier(ctx, target, s.settleCutoff())
}
// verdictSummary renders one compact line of per-tier verdicts for the "nothing due" log.
//
// It re-evaluates rather than threading the verdicts out of pickForThisRun, and that is a
// deliberate trade: this runs only on the path where NOTHING is due, so the cost is one extra
// storage listing per tier on an otherwise idle evaluation (measured 18 ms local / 392 ms offsite,
// R-86 Part 1.4), and in exchange the logging path cannot drift from the deciding path by holding a
// stale copy of it. If that cost ever matters, pass the verdicts in — do not let the two diverge.
func (s *Scheduler) verdictSummary(ctx context.Context) string {
out := ""
for _, v := range s.EvaluateDue(ctx) {
if out != "" {
out += "; "
}
switch {
case v.Err != nil:
out += v.Target + ": UNKNOWN (" + v.Err.Error() + ")"
default:
out += v.Target + ": " + v.Reason
}
}
if out == "" {
return "no tiers configured"
}
return out
}
+593
View File
@@ -0,0 +1,593 @@
package backup
import (
"context"
"errors"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// R-86 — the restore-test follows the BACKUP, not the clock.
//
// Every test here DRIVES time (`s.now` is injected and stepped) rather than waiting for it. A test
// that slept could not say anything about a 24-hour rule in under 24 hours, and one that only
// asserted "no error" would pass against a scheduler that never ran anything at all — which is
// precisely the failure mode §8.1's trap produces. So the assertions are: did a test run, on WHICH
// archive, and did a second evaluation correctly run NOTHING.
// ── the fake tier storage ────────────────────────────────────────────────────────────────────
// archiveStub is one archive on a tier: its volid and when it landed.
type archiveStub struct {
volid string
landed time.Time
}
// tierStorage is a TierPicker over per-tier archive lists. It implements the SAME contract as the
// production picker (*BackupRunner).PickSettledRestoreCandidateOn — newest archive that landed at
// or before the cutoff — which is itself covered against a fake PVE API in backup_test.go, and
// end-to-end by the live run. Naming the seam explicitly: everything below is true up to this
// picker; that the real picker obeys the same rule is asserted there, not here.
type tierStorage struct {
archives map[string][]archiveStub
err map[string]error // target → lookup failure
}
func (ts *tierStorage) pick(_ context.Context, target string, notAfter time.Time) (string, time.Time, error) {
if e, ok := ts.err[target]; ok && e != nil {
return "", time.Time{}, e
}
var best archiveStub
for _, a := range ts.archives[target] {
if !notAfter.IsZero() && a.landed.After(notAfter) {
continue // not settled yet
}
if best.volid == "" || a.landed.After(best.landed) {
best = a
}
}
return best.volid, best.landed, nil
}
// dueHarness is a scheduler with a driven clock over a fake tier storage.
type dueHarness struct {
s *Scheduler
rr *rotRunner
st *RestoreTestState
ts *tierStorage
clock time.Time
path string
}
func newDueHarness(t *testing.T, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
t.Helper()
return newDueHarnessAt(t, filepath.Join(t.TempDir(), "rt.json"), start, settle, pass, tiers, ts)
}
func newDueHarnessAt(t *testing.T, statePath string, start time.Time, settle time.Duration, pass bool, tiers []string, ts *tierStorage) *dueHarness {
t.Helper()
h := &dueHarness{rr: &rotRunner{pass: pass}, ts: ts, clock: start, path: statePath}
h.st = NewRestoreTestState(statePath)
h.s = NewScheduler(SchedulerOptions{
Runner: h.rr,
Store: NewStore(),
Spec: func(_ context.Context, archive string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
},
Cadence: time.Hour,
Settle: settle,
Logger: quiet(),
Tiers: tiers,
TierPick: ts.pick,
State: h.st,
InFlight: &InFlight{},
})
h.s.now = func() time.Time { return h.clock }
return h
}
// advance steps the clock by step, evaluating once at every step — the scheduler's real shape.
func (h *dueHarness) advance(step, total time.Duration) {
for elapsed := time.Duration(0); elapsed < total; elapsed += step {
h.clock = h.clock.Add(step)
h.s.tick(context.Background())
}
}
var day0 = time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)
// dailyArchives lands one archive a day at 02:00 for n days, starting at day0.
func dailyArchives(tier string, n int) []archiveStub {
out := make([]archiveStub, 0, n)
for d := 0; d < n; d++ {
out = append(out, archiveStub{
volid: fmt.Sprintf("%s:backup/vzdump-lxc-9201-day%d.tar.zst", tier, d),
landed: day0.AddDate(0, 0, d),
})
}
return out
}
// ── SCENARIO A — a daily tier is proved daily, on its own archive ────────────────────────────
//
// THE TRAP THIS PINS (§8.1). R-86 reads "trigger a tier ~24 h after its own newest archive", and
// the literal implementation of that — *due when the newest archive is at least `settle` old* — is
// NEVER true on a daily tier: a new archive lands every day, so the newest archive's age resets to
// zero long before it reaches 24 h. The literal reading silently switches restore-testing OFF for
// the tier that matters most.
//
// COMPANION RED-PROOF (observed 2026-08-03). In Scheduler.evaluateTier, the per-archive comparison
// was replaced by the naive age rule:
//
// - if ok && proven == archive { … not due … }
// - if s.now().Sub(landed) < s.settle { … not due … } // and the proven-archive check deleted
//
// and the picker cutoff was removed (`cutoff := time.Time{}`), i.e. exactly "is the newest archive
// old enough". Result:
//
// --- FAIL: TestDue_DailyTierIsProvedDailyOnItsOwnArchive
// restoretest_due_test.go: a daily tier must be proved once per day; got 0 run(s) over 5 days
//
// Zero runs — restore-testing off. Restored immediately afterwards.
func TestDue_DailyTierIsProvedDailyOnItsOwnArchive(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 6)}}
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
// Five days, evaluated hourly.
h.advance(time.Hour, 5*24*time.Hour)
got := h.rr.seen()
if len(got) != 5 {
t.Fatalf("a daily tier must be proved once per day; got %d run(s) over 5 days: %v", len(got), got)
}
// And each run must be on the archive that settled that day — day0's on day 1, and so on.
for i, a := range got {
want := fmt.Sprintf("local:backup/vzdump-lxc-9201-day%d.tar.zst", i)
if a != want {
t.Fatalf("run %d tested %q, want %q — the test is not following the archive", i+1, a, want)
}
}
// The newest archive is NEVER the one tested: it has not settled.
if last := got[len(got)-1]; last == "local:backup/vzdump-lxc-9201-day5.tar.zst" {
t.Fatal("the still-settling archive was tested — the settle cutoff is not being applied")
}
}
// ── SCENARIO B — a weekly tier is proved weekly, not every other day ─────────────────────────
func TestDue_WeeklyTierIsProvedOncePerArchive(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {
{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0},
{volid: "felhom-pbs:backup/ct/9201/w1", landed: day0.AddDate(0, 0, 7)},
{volid: "felhom-pbs:backup/ct/9201/w2", landed: day0.AddDate(0, 0, 14)},
}}}
h := newDueHarness(t, day0.Add(time.Hour), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
// Three weeks, evaluated every 6 hours — 84 evaluations.
h.advance(6*time.Hour, 21*24*time.Hour)
got := h.rr.seen()
want := []string{
"felhom-pbs:backup/ct/9201/w0",
"felhom-pbs:backup/ct/9201/w1",
"felhom-pbs:backup/ct/9201/w2",
}
if len(got) != len(want) {
t.Fatalf("a weekly tier must be proved ONCE PER ARCHIVE (3 archives over 3 weeks); got %d run(s): %v", len(got), got)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("run %d tested %q, want %q", i+1, got[i], want[i])
}
}
}
// ── SCENARIO C — an agent restart does not change the schedule ───────────────────────────────
//
// This is the defect a person actually notices: today every deploy restarts the ticker, so a
// restore-test runs one interval after each deploy regardless of what has already been proven.
//
// COMPANION RED-PROOF (observed 2026-08-03): revert the state to per-tier TIME by making
// ProvenArchive ignore the stored archive —
//
// - if !ok || p.Archive == "" { return "", false }
// - return "", false // per-tier time only, the pre-R-86 state
//
// → --- FAIL: TestDue_RestartRunsNothing
//
// restoretest_due_test.go:226: an agent restart must not trigger a restore-test; 2 restart(s)
// produced 4 run(s)
//
// Four: the same already-proven archive re-tested on EVERY evaluation after EVERY restart, which is
// today's behaviour with the ticker's phase reset by the deploy. Restored.
func TestDue_RestartRunsNothing(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "rt.json")
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
start := day0.AddDate(0, 0, 1).Add(time.Hour) // day 1, 03:00 — day0's archive has settled
h := newDueHarnessAt(t, path, start, 24*time.Hour, true, []string{"local"}, ts)
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("precondition: the settled archive should have been proved once; got %d run(s)", n)
}
// --- two restarts: brand-new scheduler + brand-new state object over the SAME file ---
total := 0
for i := 0; i < 2; i++ {
h2 := newDueHarnessAt(t, path, start.Add(time.Duration(i+1)*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
h2.s.tick(context.Background())
h2.s.tick(context.Background())
total += len(h2.rr.seen())
}
if total != 0 {
t.Fatalf("an agent restart must not trigger a restore-test; 2 restart(s) produced %d run(s)", total)
}
}
// ── SCENARIO D — a new archive makes a tier due even if it was tested yesterday ──────────────
func TestDue_NewSettledArchiveMakesAProvedTierDueAgain(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 2)}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
h.s.tick(context.Background()) // proves day0's archive
h.s.tick(context.Background()) // nothing new has settled → nothing
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("want exactly 1 run before the new archive settles, got %d: %v", n, h.rr.seen())
}
// Day 2, 03:00 — day1's archive has now settled.
h.clock = day0.AddDate(0, 0, 2).Add(time.Hour)
h.s.tick(context.Background())
got := h.rr.seen()
if len(got) != 2 {
t.Fatalf("a newly settled archive must make the tier due again; got %v", got)
}
if got[1] != "local:backup/vzdump-lxc-9201-day1.tar.zst" {
t.Fatalf("the NEW archive must be the one tested; got %q", got[1])
}
}
// ── SCENARIO E — a failing tier keeps being retried, and earns no proof ──────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): give credit on failure in Scheduler.tick —
//
// - if rt.Pass && s.rtState != nil && target != "" {
// - if s.rtState != nil && target != "" {
//
// → --- FAIL: TestDue_FailingTierIsRetriedAndNeverProven
//
// restoretest_due_test.go: a failing tier must keep being retried; got 1 run(s) over 3
// evaluations
//
// A single failure would have retired the archive as proven — a permanently broken DR tier looking
// freshly verified, which is the loudest signal this system produces going silent. Restored.
func TestDue_FailingTierIsRetriedAndNeverProven(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": dailyArchives("local", 1)}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, false, []string{"local"}, ts)
for i := 0; i < 3; i++ {
h.s.tick(context.Background())
}
got := h.rr.seen()
if len(got) != 3 {
t.Fatalf("a failing tier must keep being retried; got %d run(s) over 3 evaluations: %v", len(got), got)
}
if _, ok := h.st.ProvenArchive("local"); ok {
t.Fatal("a FAILED restore-test must not record the archive as proven")
}
if _, ok := h.st.LastSuccess("local"); ok {
t.Fatal("a FAILED restore-test must not stamp the tier as proven")
}
}
// ── SCENARIO F — two tiers due at once do not run at once ────────────────────────────────────
func TestDue_TwoDueTiersRunOneAtATime(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/a", landed: day0}},
}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
// Both tiers are due at this instant.
due := h.s.EvaluateDue(context.Background())
if len(due) != 2 || !due[0].Due || !due[1].Due {
t.Fatalf("precondition: both tiers should be due; got %v", due)
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("ONE evaluation must start ONE restore-test, never two multi-GB restores over one link; got %d: %v", n, h.rr.seen())
}
// The other tier was DEFERRED, not cancelled: it is still due and runs on the next evaluation.
h.s.tick(context.Background())
got := h.rr.seen()
if len(got) != 2 || got[0] == got[1] {
t.Fatalf("the deferred tier must run on the NEXT evaluation, on its own archive; got %v", got)
}
}
// The heavy-operation gate still holds, and a tier deferred behind a backup stays DUE.
func TestDue_DeferredBehindABackupStaysDue(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(time.Hour), 24*time.Hour, true, []string{"local"}, ts)
gate := &InFlight{}
h.s.inFlight = gate
release, _, _ := gate.TryAcquire("backup:felhom-pbs")
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("the restore-test must DEFER while a backup holds the gate; got %d run(s)", n)
}
if due := h.s.EvaluateDue(context.Background()); !due[0].Due {
t.Fatal("a deferred tier must remain DUE — deferral is not dismissal")
}
release()
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 1 {
t.Fatalf("must resume once the gate frees; got %d run(s)", n)
}
}
// ── SCENARIO H — a newborn box is UNKNOWN, not stale and not a fault ─────────────────────────
func TestDue_NewbornTierIsNotDueAndNotAnError(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": nil}}
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"felhom-pbs"}, ts)
due := h.s.EvaluateDue(context.Background())
if len(due) != 1 {
t.Fatalf("want one verdict, got %v", due)
}
v := due[0]
if v.Due || v.Err != nil || v.Archive != "" {
t.Fatalf("a tier with no archive is UNKNOWN — not due, not an error; got %+v", v)
}
if v.Reason == "" {
t.Fatal("every verdict must carry a reason — a due-check that cannot say why is a quiet path")
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("a newborn tier must not be restore-tested; got %d run(s)", n)
}
}
// An archive that exists but has NOT settled yet is not a candidate — and that is not an error.
func TestDue_UnsettledArchiveIsNotACandidate(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"local": {{volid: "local:backup/fresh.tar.zst", landed: day0}}}}
h := newDueHarness(t, day0.Add(2*time.Hour), 24*time.Hour, true, []string{"local"}, ts)
if v := h.s.EvaluateDue(context.Background())[0]; v.Due || v.Archive != "" {
t.Fatalf("an archive 2h old must not be a candidate under a 24h settle lag; got %+v", v)
}
h.s.tick(context.Background())
if n := len(h.rr.seen()); n != 0 {
t.Fatalf("nothing settled → no run; got %d", n)
}
}
// A tier whose archives cannot be LISTED is UNKNOWN — never silently "not due", and never silent.
// Treating a lookup failure as "not due" would retire a tier the moment its storage stopped
// answering, which is the same absence-is-not-evidence error this monitor family keeps making.
func TestDue_LookupFailureIsUnknownNotNotDue(t *testing.T) {
boom := errors.New("storage unreachable")
ts := &tierStorage{
archives: map[string][]archiveStub{"local": {{volid: "local:backup/a.tar.zst", landed: day0}}},
err: map[string]error{"felhom-pbs": boom},
}
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
var pbs DueVerdict
for _, v := range h.s.EvaluateDue(context.Background()) {
if v.Target == "felhom-pbs" {
pbs = v
}
}
if pbs.Err == nil {
t.Fatal("a lookup failure must travel in the verdict, not be swallowed")
}
if pbs.Due {
t.Fatal("a tier we could not list must not be reported DUE — we have no archive to test")
}
if pbs.Reason == "" {
t.Fatal("the failure must be explained, not merely flagged")
}
// And the OTHER tier still runs: one tier's storage being unreadable must not cost the other
// tier its proof.
h.s.tick(context.Background())
if got := h.rr.seen(); len(got) != 1 || got[0] != "local:backup/a.tar.zst" {
t.Fatalf("the readable tier must still be proved; got %v", got)
}
}
// ── the state's migration (§8.2) ─────────────────────────────────────────────────────────────
// A pre-R-86 state file carries a TIME and no archive. It must keep its time (rotation ordering
// survives the upgrade) and yield NO proven archive, so each tier is due exactly once. Reading a
// legacy time as proof of the CURRENT archive would mark an unproven archive proven — a guarantee
// invented by a migration.
func TestRestoreTestState_LegacyFileMigratesToNothingProven(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
legacy := `{"local":"2026-08-01T02:00:00Z","felhom-pbs":"2026-07-30T02:00:00Z"}`
if err := writeFileForTest(path, legacy); err != nil {
t.Fatal(err)
}
st := NewRestoreTestState(path)
if _, ok := st.ProvenArchive("local"); ok {
t.Fatal("a legacy record names no archive — it must NOT be read as proof of the current one")
}
at, ok := st.LastSuccess("local")
if !ok || !at.Equal(time.Date(2026, 8, 1, 2, 0, 0, 0, time.UTC)) {
t.Fatalf("the legacy TIME must survive (rotation ordering depends on it); got %v ok=%v", at, ok)
}
// Ordering still works off the legacy times.
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("oldest-first must still order legacy records; got %v", got)
}
}
// The new shape round-trips, archive and all.
func TestRestoreTestState_ArchiveRoundTrips(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
t.Fatal(err)
}
re := NewRestoreTestState(path)
got, ok := re.ProvenArchive("felhom-pbs")
if !ok || got != "felhom-pbs:backup/ct/9201/x" {
t.Fatalf("the proven ARCHIVE must survive a restart; got %q ok=%v", got, ok)
}
at, ok := re.LastSuccess("felhom-pbs")
if !ok || !at.Equal(now) {
t.Fatalf("the proven TIME must survive too; got %v ok=%v", at, ok)
}
}
// writeFileForTest is a tiny helper so the legacy-migration fixture reads clearly above.
func writeFileForTest(path, content string) error {
return os.WriteFile(path, []byte(content), 0o600)
}
// Standing rule 3: an absent log line is not evidence. "Nothing is due" is now the NORMAL outcome of
// an evaluation, so it must produce a POSITIVE observable naming each tier's verdict — otherwise a
// quiet journal is equally consistent with a healthy loop and a dead goroutine.
//
// COMPANION RED-PROOF (observed 2026-08-03): drop the summary back to a bare
// `s.logger.Debug("backup: restore-test not due this evaluation")` and this fails with
// "a not-due evaluation must name each tier's verdict; got \"\"" — i.e. nothing at INFO at all.
func TestDue_NothingDueStillNamesEveryTiersVerdict(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{
"local": {{volid: "local:backup/a.tar.zst", landed: day0}},
"felhom-pbs": nil, // no archive at all
}}
h := newDueHarness(t, day0.AddDate(0, 0, 1), 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
// Prove the local tier so NOTHING is due.
if err := h.st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", h.clock); err != nil {
t.Fatal(err)
}
// Assert what the SCHEDULER emits on a real evaluation, not what a helper returns — a helper
// test would pass against a tick that never calls it.
var logbuf strings.Builder
h.s.logger = slog.New(slog.NewTextHandler(&logbuf, &slog.HandlerOptions{Level: slog.LevelInfo}))
h.s.tick(context.Background())
got := logbuf.String()
for _, want := range []string{"local", "felhom-pbs", "already proven", "no settled archive"} {
if !strings.Contains(got, want) {
t.Fatalf("a not-due evaluation must name each tier's verdict; got %q (missing %q)", got, want)
}
}
}
// A tier whose storage cannot be listed must say UNKNOWN in that same line — a lookup failure that
// reads as "nothing due" is the silence this rule exists to prevent.
func TestDue_VerdictSummaryNamesAnUnknownTier(t *testing.T) {
ts := &tierStorage{
archives: map[string][]archiveStub{"local": nil},
err: map[string]error{"felhom-pbs": errors.New("storage unreachable")},
}
h := newDueHarness(t, day0, 24*time.Hour, true, []string{"local", "felhom-pbs"}, ts)
got := h.s.verdictSummary(context.Background())
if !strings.Contains(got, "UNKNOWN") || !strings.Contains(got, "storage unreachable") {
t.Fatalf("an unlistable tier must read as UNKNOWN with its error; got %q", got)
}
}
// ── R-189 — the persisted proof must be REPORTABLE, and must refuse to lie ───────────────────
//
// A proof held only in the in-memory store dies with the process, and under per-archive due-ness the
// agent will not repeat the work. So the persisted record has to be able to become a host-report
// entry — without inventing anything it does not know.
//
// COMPANION RED-PROOF (observed 2026-08-03): drop the `reportable()` filter from
// ProvenRestoreTests, so a pre-R-189 record (archive but no tier) is emitted →
//
// --- FAIL: TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe
// restoretest_due_test.go: a record with no TIER must not be reported (the hub keys its
// per-tier proof on it); got [{... SourceTier: ...}]
//
// Restored.
func TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe(t *testing.T) {
path := filepath.Join(t.TempDir(), "rt.json")
// v1 (a bare time), v2 (archive, no tier) and v3 (complete) side by side — every shape this
// file has ever had, which is what a real box carries after two upgrades.
legacy := `{
"old-v1": "2026-07-30T02:11:07Z",
"old-v2": {"archive":"felhom-backup:backup/vzdump-lxc-9201-a.tar.zst","proven_at":"2026-08-01T04:41:58Z"},
"felhom-pbs": {"archive":"felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z","tier":"pbs","verified":"boot+running","proven_at":"2026-08-03T13:25:14Z"}
}`
if err := writeFileForTest(path, legacy); err != nil {
t.Fatal(err)
}
got := NewRestoreTestState(path).ProvenRestoreTests(context.Background())
if len(got) != 1 {
t.Fatalf("only the record that can be described honestly may be reported; got %d: %+v", len(got), got)
}
e := got[0]
if e.SourceTier != "pbs" {
t.Fatalf("a record with no TIER must not be reported (the hub keys its per-tier proof on it); got %+v", got)
}
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" || !e.Pass {
t.Fatalf("the reported entry must be the stored proof, unchanged; got %+v", e)
}
if e.TestedAt != "2026-08-03T13:25:14Z" {
t.Fatalf("the entry must carry the time the run passed, not now(); got %q", e.TestedAt)
}
if e.Verified != "boot+running" {
t.Fatalf("what the run verified must survive the round trip; got %q", e.Verified)
}
// Run mechanics are NOT invented: an absent duration is not a claim, a fabricated one would be.
if e.DurationSeconds != 0 || e.ScratchVMID != 0 {
t.Fatalf("the re-report must not invent run mechanics it never stored; got duration=%v scratch=%d",
e.DurationSeconds, e.ScratchVMID)
}
// The legacy records still serve the DUE-check, which is a separate question from reporting.
if _, ok := NewRestoreTestState(path).ProvenArchive("old-v2"); !ok {
t.Fatal("a v2 record must still answer the due-check even though it cannot be reported")
}
}
// A tier proved through the SCHEDULER (not by hand) lands in the state complete enough to report —
// the production path, not a hand-built fixture.
func TestScheduler_ProofIsRecordedReportably(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, true, []string{"felhom-pbs"}, ts)
// The fake runner echoes the spec's tier; give the spec a tier the way main.go does.
h.s.spec = func(_ context.Context, archive string) reconcile.RestoreTestSpec {
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, SourceTier: "pbs"}
}
h.s.tick(context.Background())
got := h.st.ProvenRestoreTests(context.Background())
if len(got) != 1 {
t.Fatalf("a scheduled pass must leave a REPORTABLE proof; got %d: %+v", len(got), got)
}
if got[0].SourceTier != "pbs" || got[0].SourceArchive != "felhom-pbs:backup/ct/9201/w0" {
t.Fatalf("the proof must name the tier and the archive the run used; got %+v", got[0])
}
}
// A FAILED run leaves nothing to report — the asymmetry of §8.1, asserted rather than assumed.
func TestScheduler_AFailureLeavesNoPersistedProof(t *testing.T) {
ts := &tierStorage{archives: map[string][]archiveStub{"felhom-pbs": {{volid: "felhom-pbs:backup/ct/9201/w0", landed: day0}}}}
h := newDueHarness(t, day0.AddDate(0, 0, 1).Add(97*time.Minute), 24*time.Hour, false, []string{"felhom-pbs"}, ts)
h.s.tick(context.Background())
if got := h.st.ProvenRestoreTests(context.Background()); len(got) != 0 {
t.Fatalf("a FAILED run must persist nothing — a failing tier is retried, and a stored failure "+
"would outlive the fault; got %+v", got)
}
}
+169 -19
View File
@@ -1,12 +1,15 @@
package backup
import (
"context"
"encoding/json"
"os"
"path/filepath"
"sort"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// RestoreTestState persists the last SUCCESSFUL restore-test per backup tier.
@@ -28,42 +31,131 @@ import (
//
// Only SUCCESS is recorded. A failed run must not satisfy rotation, or a tier that fails every time
// would look freshly proven and stop being retried — the same "a failure satisfies the cadence"
// trap the backup due-check avoids.
// trap the backup due-check avoids. R-86 keeps that property unchanged and gives it a second job:
// the due-check reads this state, so a failure that recorded proof would ALSO stop the tier from
// ever becoming due again. The rule earns its keep twice now.
//
// R-86 (1.2) — WHICH ARCHIVE, not just when.
//
// A timestamp alone cannot answer the question the due-check asks. "This tier passed at 04:00" is
// consistent both with "yesterday's archive is proven" and with "an archive from a week ago is
// proven and nothing since has been looked at". Restore-testing is now per ARCHIVE GENERATION —
// a tier is due once it holds a settled archive that has not been proven — so the identity of the
// proven archive is the state, and the time is metadata (rotation ordering, operator reporting).
//
// This is the same class as the workspace rule "a timestamp records an ATTEMPT, not a RESULT":
// here it records a result, but not WHICH result, and that is just as unable to answer the question
// being asked of it.
type RestoreTestState struct {
path string
mu sync.Mutex
last map[string]time.Time // target id → last SUCCESSFUL restore-test (UTC)
last map[string]provenTier // target id → what was last PROVEN on that tier
}
// provenTier is one tier's proof: the archive that passed, which tier it was, what was verified,
// and when.
//
// R-189 added `Tier` and `Verified`. Until then this record could answer the DUE-check but could not
// be REPORTED, and being reportable is what closes R-189: a proof held only in the in-memory result
// store vanishes on restart, and under per-archive due-ness the box will not repeat the work, so the
// hub can stay ignorant of a real success until the next archive generation.
//
// `Tier` is stored rather than derived because it is known for certain at proof time (the run's own
// spec used it to choose the restore timeout) and deriving it later would need a storage-type lookup
// at report-building time — a network call that can fail, on a path where failing means mis-labelling
// a proof. Store what you knew when you knew it.
type provenTier struct {
Archive string // volid of the archive that PASSED; "" = a legacy record with no archive
Tier string // "local" | "pbs" — as the run reported it; "" = pre-R-189 record
Verified string // what the run verified (e.g. "boot+running"); "" = pre-R-189 record
At time.Time // when that run passed (UTC)
}
// reportable reports whether this record can be re-reported to the hub as a restore-test result.
//
// It needs BOTH the archive and the tier: the hub keys its edge-triggered failure state on the
// archive and its per-tier proof lookup on the tier, so an entry missing either is not a usable
// proof — and emitting one anyway would be a report the hub cannot act on, dressed as evidence.
// A pre-R-189 record is therefore silently not reported; the tier's next real proof fills it in.
func (p provenTier) reportable() bool { return p.Archive != "" && p.Tier != "" }
// provenTierJSON is the on-disk shape. Two older shapes are read and neither is written:
//
// v1 (pre-R-86) "<target>": "<RFC3339>" — a time, no archive
// v2 (R-86) "<target>": {archive, proven_at} — due-check usable, not reportable
// v3 (R-189) "<target>": {archive, tier, verified, …} — both
//
// Fields absent in an older file unmarshal to "", which is exactly the "no usable proof" signal the
// readers above test for — the migration needs no version number because the absence IS the answer.
type provenTierJSON struct {
Archive string `json:"archive"`
Tier string `json:"tier,omitempty"`
Verified string `json:"verified,omitempty"`
ProvenAt string `json:"proven_at"`
}
// NewRestoreTestState opens (or creates) the state at path. A missing or unreadable file is NOT an
// error: it degrades to "nothing proven yet", which is the correct starting point and keeps a
// corrupt file from wedging the daemon.
//
// MIGRATION (R-86). The pre-R-86 file is `{"<target>": "<RFC3339>"}` — a time and no archive. A
// legacy record keeps its TIME (rotation ordering survives a deploy, which is why the file exists
// at all) but yields NO proven archive, so every tier is due exactly once on first evaluation after
// the upgrade. One extra restore-test per tier, once, is the safe direction: the alternative is to
// read a legacy time as proof of whatever archive happens to be current, which would mark an
// unproven archive proven — inventing a guarantee out of a migration.
func NewRestoreTestState(path string) *RestoreTestState {
s := &RestoreTestState{path: path, last: map[string]time.Time{}}
s := &RestoreTestState{path: path, last: map[string]provenTier{}}
data, err := os.ReadFile(path)
if err != nil {
return s
}
var raw map[string]string
var raw map[string]json.RawMessage
if json.Unmarshal(data, &raw) != nil {
return s
}
for target, ts := range raw {
if t, perr := time.Parse(time.RFC3339, ts); perr == nil {
s.last[target] = t.UTC()
for target, msg := range raw {
// Legacy shape: a bare RFC3339 string.
var legacy string
if json.Unmarshal(msg, &legacy) == nil {
if t, perr := time.Parse(time.RFC3339, legacy); perr == nil {
s.last[target] = provenTier{At: t.UTC()} // no archive → due once, deliberately
}
continue
}
var cur provenTierJSON
if json.Unmarshal(msg, &cur) != nil {
continue // one unreadable entry must not lose the others
}
t, perr := time.Parse(time.RFC3339, cur.ProvenAt)
if perr != nil {
continue
}
s.last[target] = provenTier{Archive: cur.Archive, Tier: cur.Tier, Verified: cur.Verified, At: t.UTC()}
}
return s
}
// RecordSuccess stamps a tier as proven at t. Only call this for a PASSING restore-test.
func (s *RestoreTestState) RecordSuccess(target string, t time.Time) error {
// RecordSuccess stamps a tier as proven at t, naming the ARCHIVE that passed, the TIER the run
// reported, and what it verified. Only call this for a PASSING restore-test — the archive is what
// makes the tier not-due, so recording one for a failed run would retire the archive unproven.
//
// ONLY SUCCESSES ARE PERSISTED, AND THE ASYMMETRY IS DELIBERATE (R-189 §8.1). Say it here because
// the next reader will notice failures are absent and try to "fix" it:
//
// a SUCCESS suppresses future work — a proven archive is never re-tested, so a lost proof leaves
// the system quietly less tested than it believes. It must survive a restart.
//
// a FAILURE causes future work — a failing tier stays due and is retried at the next evaluation,
// so a lost failure heals itself within one interval. Persisting it would do the opposite of
// helping: a healed tier would keep reporting a failure that is no longer true.
func (s *RestoreTestState) RecordSuccess(target, archive, tier, verified string, t time.Time) error {
if target == "" {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
s.last[target] = t.UTC()
s.last[target] = provenTier{Archive: archive, Tier: tier, Verified: verified, At: t.UTC()}
return s.saveLocked()
}
@@ -71,21 +163,75 @@ func (s *RestoreTestState) RecordSuccess(target string, t time.Time) error {
func (s *RestoreTestState) LastSuccess(target string) (time.Time, bool) {
s.mu.Lock()
defer s.mu.Unlock()
t, ok := s.last[target]
return t, ok
p, ok := s.last[target]
return p.At, ok
}
// Snapshot returns a copy of the whole map — for the host-report gauge.
// ProvenArchive returns the archive last PROVEN on this tier (ok=false = none — either never tested,
// or a legacy record carrying only a time). It is the due-check's whole question: an archive that is
// not this one has not been proven.
func (s *RestoreTestState) ProvenArchive(target string) (string, bool) {
s.mu.Lock()
defer s.mu.Unlock()
p, ok := s.last[target]
if !ok || p.Archive == "" {
return "", false
}
return p.Archive, true
}
// Snapshot returns a copy of the last-proven TIMES.
//
// It carried the comment "for the host-report gauge" from the day it was written and **had no caller
// at all** until R-189 — a seam built and never wired, and an invariant asserted in a comment with
// nothing pinning it, in one method. The host report is now fed by ProvenRestoreTests below, which
// carries the archive and the tier that a bare timestamp cannot. This stays for callers that want
// only the times; if it acquires none, delete it rather than let it claim a purpose again.
func (s *RestoreTestState) Snapshot() map[string]time.Time {
s.mu.Lock()
defer s.mu.Unlock()
out := make(map[string]time.Time, len(s.last))
for k, v := range s.last {
out[k] = v
out[k] = v.At
}
return out
}
// ProvenRestoreTests renders the persisted proofs as host-report entries — the R-189 fix.
//
// It satisfies hub.RestoreTestReporter's shape, so the collector can merge these with the in-memory
// results. What it emits is a RE-REPORT of a run that really happened, not a synthesis:
//
// - `Pass` is true because ONLY successes are stored (RecordSuccess is the sole writer);
// - `SourceArchive`, `SourceTier`, `Verified` and `TestedAt` are the values that run reported;
// - the run mechanics (scratch VMID, duration, warnings) are NOT re-invented. An absent duration
// is not a claim; a fabricated one would be.
//
// A record that cannot be reported honestly is omitted rather than padded — see provenTier.reportable.
// **A tier with no usable proof produces NO entry**: an unproven tier reading as proven would be a
// worse defect than the one this fixes.
func (s *RestoreTestState) ProvenRestoreTests(context.Context) []hub.RestoreTest {
s.mu.Lock()
defer s.mu.Unlock()
out := make([]hub.RestoreTest, 0, len(s.last))
for _, p := range s.last {
if !p.reportable() {
continue
}
out = append(out, hub.RestoreTest{
SourceArchive: p.Archive,
SourceTier: p.Tier,
Pass: true,
Verified: p.Verified,
TestedAt: p.At.UTC().Format(time.RFC3339),
})
}
// Deterministic order: the report is compared byte-wise by the contract test, and Go's map
// iteration is randomised.
sort.Slice(out, func(i, j int) bool { return out[i].SourceTier < out[j].SourceTier })
return out
}
// OldestFirst orders targets by "least recently proven first"; never-proven sorts FIRST.
//
// This is the operator's 2026-07-26 ruling (Option 1): self-balancing, no new config knob, and it
@@ -100,8 +246,9 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
defer s.mu.Unlock()
out := append([]string(nil), targets...)
sort.SliceStable(out, func(i, j int) bool {
ti, oki := s.last[out[i]]
tj, okj := s.last[out[j]]
pi, oki := s.last[out[i]]
pj, okj := s.last[out[j]]
ti, tj := pi.At, pj.At
switch {
case !oki && !okj:
return out[i] < out[j] // both never proven → deterministic
@@ -119,9 +266,12 @@ func (s *RestoreTestState) OldestFirst(targets []string) []string {
}
func (s *RestoreTestState) saveLocked() error {
raw := make(map[string]string, len(s.last))
for target, t := range s.last {
raw[target] = t.UTC().Format(time.RFC3339)
raw := make(map[string]provenTierJSON, len(s.last))
for target, p := range s.last {
raw[target] = provenTierJSON{
Archive: p.Archive, Tier: p.Tier, Verified: p.Verified,
ProvenAt: p.At.UTC().Format(time.RFC3339),
}
}
data, err := json.MarshalIndent(raw, "", " ")
if err != nil {
+36 -17
View File
@@ -39,10 +39,19 @@ func (r *rotRunner) seen() []string {
return append([]string(nil), r.archives...)
}
// testLanded is a landing time old enough to be settled under any cutoff these tests use. R-86
// widened the TierPicker seam with the archive's landing time; the rotation tests below are about
// tier ORDER and the heavy-operation gate, not about settling, so they hold it constant.
var testLanded = time.Date(2026, 7, 1, 0, 0, 0, 0, time.UTC)
// archiveFor is a TierPicker over a fixed map: target → archive ("" = that tier holds none).
func archiveFor(m map[string]string) TierPicker {
return func(_ context.Context, target string) (string, error) {
return m[target], nil
return func(_ context.Context, target string, _ time.Time) (string, time.Time, error) {
a := m[target]
if a == "" {
return "", time.Time{}, nil
}
return a, testLanded, nil
}
}
@@ -63,14 +72,24 @@ func rotScheduler(t *testing.T, rr *rotRunner, st *RestoreTestState, pick TierPi
})
}
// ── SCENARIO A — both tiers get tested across consecutive cadences ───────────────────────────
// ── SCENARIO A — both tiers get tested, each ONCE per archive ────────────────────────────────
//
// R-86 CHANGED THIS TEST'S CONTRACT, deliberately, and the old assertion is worth recording because
// it was a faithful statement of the defect. It read:
//
// 4 ticks → 4 runs, and consecutive runs must hit different tiers
//
// i.e. every tick produced a heavy restore-test, because the ticker WAS the trigger. Under R-86 a
// tick is an EVALUATION: both tiers are still exercised (rotation is intact), but a tier whose
// newest settled archive is already proven is not re-tested just because time passed. So the
// assertion is now 2 runs across 4 evaluations — one per tier, one per archive — which is a
// STRICTLY STRONGER statement: it pins both the coverage R-85 won and the pacing R-86 adds.
//
// COMPANION RED-PROOF (observed): restore the single-target picker — set `Tiers`/`TierPick` to nil
// so `pickForThisRun` falls back to `s.pick` on the primary runner — and this fails with
// "both tiers must be exercised across 4 cadences; got [local:… local:… local:… local:…]",
// i.e. the offsite tier never appears. That is today's behaviour, and it is why demo-hp's DR tier
// went unproven for its entire existence.
func TestRotation_BothTiersExercisedAcrossCadences(t *testing.T) {
// "both tiers must be exercised; got [local:…]", i.e. the offsite tier never appears. That is
// pre-R-85 behaviour, and it is why demo-hp's DR tier went unproven for its entire existence.
func TestRotation_BothTiersExercisedOncePerArchive(t *testing.T) {
rr := &rotRunner{pass: true}
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
s := rotScheduler(t, rr, st, archiveFor(map[string]string{
@@ -94,14 +113,14 @@ func TestRotation_BothTiersExercisedAcrossCadences(t *testing.T) {
}
}
if !sawLocal || !sawPBS {
t.Fatalf("both tiers must be exercised across 4 cadences; got %v", got)
t.Fatalf("both tiers must be exercised; got %v", got)
}
// Oldest-first must ALTERNATE, not clump — otherwise one tier is starved between visits.
if len(got) != 4 {
t.Fatalf("want 4 runs, got %d: %v", len(got), got)
// Exactly one run per tier: the archives never changed, so nothing became due a second time.
if len(got) != 2 {
t.Fatalf("want 2 runs across 4 evaluations (one per archive generation), got %d: %v", len(got), got)
}
if got[0] == got[1] {
t.Fatalf("consecutive runs hit the same tier — oldest-first is not rotating: %v", got)
t.Fatalf("the two runs must be different tiers — oldest-first is not ordering due tiers: %v", got)
}
}
@@ -284,11 +303,11 @@ func TestOldestFirst_Ordering(t *testing.T) {
t.Fatalf("unexpected: %v", got)
}
}
_ = st.RecordSuccess("local", now)
_ = st.RecordSuccess("local", "local:backup/a.tar.zst", "local", "boot+running", now)
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "felhom-pbs" {
t.Fatalf("a never-proven tier must sort before a proven one; got %v", got)
}
_ = st.RecordSuccess("felhom-pbs", now.Add(time.Hour))
_ = st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/b", "pbs", "boot+running", now.Add(time.Hour))
if got := st.OldestFirst([]string{"local", "felhom-pbs"}); got[0] != "local" {
t.Fatalf("the least recently proven must sort first; got %v", got)
}
@@ -299,8 +318,8 @@ func TestOldestFirst_Ordering(t *testing.T) {
func TestOldestFirst_DeterministicOnTies(t *testing.T) {
st := NewRestoreTestState(filepath.Join(t.TempDir(), "rt.json"))
now := time.Now().UTC()
_ = st.RecordSuccess("b-tier", now)
_ = st.RecordSuccess("a-tier", now)
_ = st.RecordSuccess("b-tier", "b:archive", "local", "boot+running", now)
_ = st.RecordSuccess("a-tier", "a:archive", "local", "boot+running", now)
for i := 0; i < 20; i++ {
if got := st.OldestFirst([]string{"b-tier", "a-tier"}); got[0] != "a-tier" {
t.Fatalf("tie-break must be deterministic; iteration %d gave %v", i, got)
@@ -315,7 +334,7 @@ func TestRestoreTestState_PersistenceAndCorruption(t *testing.T) {
now := time.Now().UTC().Truncate(time.Second)
st := NewRestoreTestState(path)
if err := st.RecordSuccess("felhom-pbs", now); err != nil {
if err := st.RecordSuccess("felhom-pbs", "felhom-pbs:backup/ct/9201/x", "pbs", "boot+running", now); err != nil {
t.Fatal(err)
}
reopened := NewRestoreTestState(path)
+42 -4
View File
@@ -260,21 +260,59 @@ func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error)
// restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and turning
// that into a failure would make every fresh box look broken for its first week.
func (r *BackupRunner) PickRestoreCandidateOn(ctx context.Context, target string) (string, error) {
archive, _, err := r.PickSettledRestoreCandidateOn(ctx, target, time.Time{})
return archive, err
}
// PickSettledRestoreCandidateOn is the R-86 due-check's picker: the newest archive on target that
// landed AT OR BEFORE notAfter (the settle cutoff), with the time it landed. A zero notAfter means
// "no cutoff" — that is the pre-R-86 behaviour, which is why PickRestoreCandidateOn is now a
// one-line call into this and its contract is untouched (one scan, one owner).
//
// WHY A CUTOFF AT ALL. An archive that landed minutes ago may still be settling — R-71a's
// settle-gate exists because the offsite tier's day-0 consume raced its own floor update — and
// restore-testing the archive a backup is still writing proves nothing about the backup that
// finished. The due-check therefore asks about the newest SETTLED archive, and §8.1's rule is built
// on that: the tier is due when a settled archive exists that has not been proven.
//
// The plausibility floor is applied here and not in the old path on purpose. Under R-86 the picked
// archive becomes the tier's due-ness: an incomplete 1-byte phantom (F-CRIT-2's artefact — server
// prune does NOT collect it) would be selected forever, fail its restore forever, never earn proof,
// and so make the tier due at EVERY evaluation. Skipping it is what keeps the retry rate bounded by
// the archive generation rather than by the evaluation interval.
//
// Contract preserved: ("", zero, nil) when the storage holds no eligible archive. **A tier with
// nothing to restore is not an error** — a brand-new offsite tier legitimately has nothing yet, and
// turning that into a failure would make every fresh box look broken for its first week.
func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target string, notAfter time.Time) (string, time.Time, error) {
if target == "" {
return "", nil
return "", time.Time{}, nil
}
contents, err := r.api.StorageContent(ctx, target)
if err != nil {
return "", err
return "", time.Time{}, err
}
var best string
var bestCTime int64 = -1
for _, e := range contents {
if e.Content == "backup" && e.CTime > bestCTime {
if e.Content != "backup" {
continue
}
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
continue // not settled yet — a newer archive is not a reason to re-prove an older one
}
if ok, why := archivePlausiblyComplete(e); !ok {
r.warnRejectedArchiveOnce(e, why)
continue
}
if e.CTime > bestCTime {
bestCTime, best = e.CTime, e.VolID
}
}
return best, nil
if best == "" {
return "", time.Time{}, nil
}
return best, time.Unix(bestCTime, 0).UTC(), nil
}
// latestArchive finds the newest backup archive volid + size for vmid on the target.
+119 -59
View File
@@ -32,22 +32,31 @@ type CandidatePicker func(ctx context.Context) (string, error)
// PBS archive was classified "local" and got the 10-minute local wait.
type SpecBuilder func(ctx context.Context, archive string) reconcile.RestoreTestSpec
// TierPicker resolves the newest archive on a NAMED tier, or "" when that tier holds none.
// (*BackupRunner).PickRestoreCandidateOn satisfies it. "" must NOT be an error — a brand-new
// offsite tier legitimately has nothing to restore yet.
type TierPicker func(ctx context.Context, target string) (string, error)
// TierPicker resolves the newest archive on a NAMED tier that landed AT OR BEFORE notAfter (the
// settle cutoff), together with when it landed. (*BackupRunner).PickSettledRestoreCandidateOn
// satisfies it. A zero notAfter means "no settle requirement".
//
// R-86 widened this seam from (target) → archive. The landing time is what makes the due-check's
// verdict explainable — "archive X, which landed at T, has not been proven" — and the cutoff is
// what makes the rule per-ARCHIVE-GENERATION instead of per-interval. "" must NOT be an error: a
// brand-new offsite tier legitimately has nothing to restore yet.
type TierPicker func(ctx context.Context, target string, notAfter time.Time) (archive string, landed time.Time, err error)
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
// AND a valid scratch band is configured (validated by the caller before construction).
type Scheduler struct {
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec SpecBuilder // R-85: evaluated PER RUN, never frozen at construction
// cadence is the EVALUATION interval (R-86) — how often "is anything due?" is asked. It is no
// longer the thing that decides a test happens; see restoretest_due.go.
cadence time.Duration
logger *slog.Logger
now func() time.Time
// settle is how long an archive must have sat before it is a candidate (R-86).
settle time.Duration
logger *slog.Logger
now func() time.Time
// R-85 tier rotation. All optional: without them the scheduler behaves exactly as before
// (single tier via `pick`), which keeps every existing caller and test working untouched.
@@ -64,9 +73,14 @@ type SchedulerOptions struct {
Store *Store
// Spec builds the run's spec (RestoreStorage, ScratchMin/Max, SourceTier, timeouts) from the
// picked archive. Called ONCE PER RUN — see SpecBuilder for why it is not a value.
Spec SpecBuilder
Cadence time.Duration // 0 → disabled
Logger *slog.Logger
Spec SpecBuilder
// Cadence is the EVALUATION interval — how often due-ness is asked, NOT how often a test runs
// (R-86). 0 → disabled.
Cadence time.Duration
// Settle is how long an archive must have sat before it is a restore-test candidate (R-86).
// 0 → no settle requirement (any archive is a candidate).
Settle time.Duration
Logger *slog.Logger
// R-85 (all optional — omit for the pre-R-85 single-tier behaviour):
// Tiers are the configured tier target ids (primary first); TierPick resolves an archive on a
@@ -89,6 +103,7 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
store: opts.Store,
spec: opts.Spec,
cadence: opts.Cadence,
settle: opts.Settle,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
tiers: append([]string(nil), opts.Tiers...),
@@ -98,9 +113,19 @@ func NewScheduler(opts SchedulerOptions) *Scheduler {
}
}
// Run fires a restore-test on the cadence until ctx is cancelled. A 0 cadence disables it
// (the goroutine just waits for shutdown). It does NOT fire immediately on start (a restore
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
// Run EVALUATES due-ness on the interval until ctx is cancelled, and runs a restore-test only when
// a tier is actually due (R-86). A 0 interval disables it (the goroutine just waits for shutdown).
//
// The ticker survives as the evaluation interval and nothing else. It is emphatically NOT the
// trigger any more: its phase is the process's uptime, and agent deploys reset it, which is exactly
// the defect R-86 removes. What decides that a test happens is `EvaluateDue`.
//
// It still does NOT evaluate immediately on start — the first evaluation is one interval in. That
// is an EARNED restraint, kept deliberately: a restore is heavy, agent restarts are routine, and a
// crash-loop that evaluated at start would hammer a permanently-failing tier as fast as it could
// restart. Due-ness does not expire while we wait, so the only cost is up to one interval of
// latency on a tier that just became due. On-demand runs use `--selftest=restore-test`.
//
// Returns nil on ctx cancellation.
func (s *Scheduler) Run(ctx context.Context) error {
if s.cadence <= 0 || s.runner == nil || s.spec == nil || (s.pick == nil && !s.rotating()) {
@@ -108,7 +133,8 @@ func (s *Scheduler) Run(ctx context.Context) error {
<-ctx.Done()
return nil
}
s.logger.Info("backup: restore-test scheduler starting", "cadence", s.cadence)
s.logger.Info("backup: restore-test scheduler starting (per-archive due-check)",
"eval_interval", s.cadence, "settle", s.settle)
t := time.NewTicker(s.cadence)
defer t.Stop()
for {
@@ -122,8 +148,12 @@ func (s *Scheduler) Run(ctx context.Context) error {
}
}
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
// no backup exists yet. Deterministic given s.now — tests call it directly.
// tick is ONE EVALUATION: gate → due-check → run the first due tier → record which archive was
// proven. No-ops cleanly when nothing is due, when no backup exists yet, or when a heavy operation
// is already in flight. Deterministic given s.now — tests call it directly.
//
// One run per evaluation, by construction (Scenario F): a second due tier is left DUE and picked up
// by the next evaluation. Deferred, never cancelled, and never two multi-GB restores over one link.
func (s *Scheduler) tick(ctx context.Context) {
if s.spec == nil {
// Defensive: Run() already refuses to start without a SpecBuilder, but tick is also
@@ -132,27 +162,48 @@ func (s *Scheduler) tick(ctx context.Context) {
s.logger.Error("backup: restore-test has no spec builder — skipping (this is a wiring bug)")
return
}
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
// cancelled backup costs the backup.
release, busy, ok := s.inFlight.TryAcquire("restore-test")
if !ok {
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight", "busy", busy)
return
}
defer release()
// The due-check runs BEFORE the gate is taken, and that ORDER is load-bearing under R-86.
//
// It used to be the other way round, and correctly so: the gate was held for one heavy run a
// day, and the candidate lookup rode along inside it. Evaluations are now frequent, and the
// lookup is a storage listing that for the offsite tier crosses the WAN. Holding the
// one-heavy-operation gate for a read that answers "nothing to do" would open a small window at
// EVERY evaluation in which a starting backup cannot acquire — and a backup that cannot acquire
// does not merely wait, it records a failure and pages the operator (F-A1). A cheap poll must
// not be able to manufacture that.
//
// Nothing is lost by checking first: due-ness does not expire, and the gate is still taken
// before anything heavy begins.
archive, target, err := s.pickForThisRun(ctx)
if err != nil {
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
return
}
if archive == "" {
s.logger.Info("backup: restore-test skipped; no backup available yet")
// A POSITIVE OBSERVABLE, at INFO, and this is not noise — it is standing rule 3.
//
// Before R-86 every tick ran a heavy restore-test, so the scheduler was audible by
// construction. Now "nothing is due" is the NORMAL outcome, and at DEBUG it is silent: an
// empty journal would be equally consistent with a healthy loop and with a dead goroutine,
// which is the exact shape the R-88 watcher was retired for. One line per evaluation is four
// lines a day at the 6h default, and it names each tier's verdict so the answer to "why did
// nothing run last night?" is in the log rather than in a re-derivation.
s.logger.Info("backup: restore-test evaluated — nothing due", "verdicts", s.verdictSummary(ctx))
return
}
// Scenario F: join the one-heavy-operation-at-a-time gate. A restore-test PULLS a multi-GB
// archive over the same tunnel an offsite backup PUSHES one; running both saturates the link and
// drives each toward its timeout, which is how a healthy tier gets recorded as failed. DEFER —
// never cancel what is already running: a deferred restore-test costs hours of coverage, a
// cancelled backup costs the backup. A deferred tier stays DUE, so the next evaluation retries it.
release, busy, ok := s.inFlight.TryAcquire("restore-test")
if !ok {
s.logger.Info("backup: restore-test deferred — a heavy operation is already in flight",
"busy", busy, "target", target, "archive", archive)
return
}
defer release()
// R-85: build the spec for THIS run, from THIS archive. Never a frozen value.
spec := s.spec(ctx, archive)
spec.Archive = archive
@@ -165,8 +216,14 @@ func (s *Scheduler) tick(ctx context.Context) {
// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
// that fails every time would look freshly proven and quietly stop being retried.
if rt.Pass && s.rtState != nil && target != "" {
if err := s.rtState.RecordSuccess(target, s.now()); err != nil {
s.logger.Warn("backup: could not persist the restore-test rotation state", "target", target, "err", err)
// R-86: the ARCHIVE is recorded, not merely the time — that is what makes the tier
// not-due until a NEWER archive settles, and what makes a proof survive a restart.
// R-189: the TIER and what was VERIFIED go with it, so the proof can be RE-REPORTED after a
// restart. Both come from the run's own result, never re-derived — `rt.SourceTier` is what
// this run was actually judged as, and deriving it later would need a storage lookup that
// can fail on the one path where failing means mislabelling a proof.
if err := s.rtState.RecordSuccess(target, archive, rt.SourceTier, rt.Verified, s.now()); err != nil {
s.logger.Warn("backup: could not persist the restore-test proof state", "target", target, "err", err)
}
}
switch {
@@ -189,46 +246,49 @@ func (s *Scheduler) tick(ctx context.Context) {
// rotating reports whether multi-tier rotation is wired.
func (s *Scheduler) rotating() bool { return len(s.tiers) > 0 && s.tierPick != nil }
// pickForThisRun chooses the tier and its newest archive.
// pickForThisRun chooses the tier to test THIS evaluation: the first DUE tier, in oldest-proven
// order.
//
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1): the tier whose last SUCCESSFUL restore-test
// is oldest goes first, never-proven first of all. Self-balancing, no config knob, and it naturally
// prioritises a tier that has never been proven — which on this fleet was the offsite tier, unproven
// for its entire existence while reporting `applied`.
// R-86 changed what this answers. It used to answer "whose turn is it?", and the answer was always
// somebody's — the ticker had fired, so a test was going to happen. It now answers "is anything
// due?", and "nothing" is a normal, frequent and correct answer.
//
// A tier with no archives is SKIPPED, not failed, and the next tier is tried. Skipping to a testable
// tier is strictly better than burning the whole cadence: a brand-new offsite tier has nothing to
// restore yet, and that is normal, not broken. It cannot starve the empty tier either — as soon as
// it has an archive it still sorts first, because it is still the least recently proven.
// OLDEST-FIRST (operator ruling 2026-07-26, Option 1) survives as the ORDER among due tiers: the
// tier whose last successful restore-test is oldest goes first, never-proven first of all. It is
// self-balancing, needs no config knob, and it still cannot starve a tier — but it no longer decides
// that a test happens at all.
//
// Returns ("", "", nil) when nothing anywhere is testable.
// A tier with no settled archive is SKIPPED, not failed — a brand-new offsite tier has nothing to
// restore yet, and that is normal, not broken. A tier whose archives cannot be LISTED is likewise
// skipped, loudly, and its error is returned only when no other tier was testable: one tier's
// storage being unreadable must not cost the other tier its proof, and must not be silent either.
//
// Returns ("", "", nil) when nothing anywhere is due.
func (s *Scheduler) pickForThisRun(ctx context.Context) (archive, target string, err error) {
if !s.rotating() {
// Pre-R-85 single-tier path (tests and any caller that wires only `Pick`): there is no tier
// identity and no persisted proof here, so there is nothing to compare an archive against
// and no due-check is possible. It runs on every evaluation, exactly as it always did.
a, perr := s.pick(ctx)
return a, "", perr // pre-R-85 single-tier path; no rotation credit to record
}
order := s.tiers
if s.rtState != nil {
order = s.rtState.OldestFirst(s.tiers)
return a, "", perr
}
var firstErr error
for _, t := range order {
a, perr := s.tierPick(ctx, t)
if perr != nil {
// One tier's storage being unreadable must not block the others.
for _, v := range s.EvaluateDue(ctx) {
if v.Err != nil {
s.logger.Warn("backup: restore-test candidate lookup failed for a tier; trying the next",
"target", t, "err", perr)
"target", v.Target, "err", v.Err)
if firstErr == nil {
firstErr = perr
firstErr = v.Err
}
continue
}
if a == "" {
s.logger.Debug("backup: restore-test tier has no archive yet; trying the next", "target", t)
if !v.Due {
s.logger.Debug("backup: restore-test tier is not due", "target", v.Target, "reason", v.Reason)
continue
}
s.logger.Info("backup: restore-test tier selected (oldest-proven first)", "target", t, "archive", a)
return a, t, nil
s.logger.Info("backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)",
"target", v.Target, "archive", v.Archive, "landed", v.Landed.Format(time.RFC3339), "reason", v.Reason)
return v.Archive, v.Target, nil
}
if firstErr != nil {
return "", "", firstErr
+16 -2
View File
@@ -10,8 +10,22 @@ import (
// Store holds the agent's LATEST backup result per target and the latest restore-test
// result — the point-in-time state the host-report surfaces. It is updated by the backup
// runner + the restore-test scheduler/selftest and read by the collector via the hub
// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence
// re-populates) and mutex-guarded for the concurrent collector vs scheduler access.
// BackupReporter / RestoreTestReporter seams. In-memory and mutex-guarded for the concurrent
// collector vs scheduler access.
//
// **"lost on restart; the cadence re-populates" — that sentence used to be here and it is now
// FALSE for restore-tests (R-189, 2026-08-03).** It was true while a timer re-tested every tier
// daily. Under R-86's per-archive due-check the agent will NOT re-test an archive it has already
// proven, so a proof lost to a restart is not repeated until the next archive generation — a week on
// the offsite tier — and the hub reports that tier unproven throughout. Observed, not predicted: a
// real 14.5 GB offsite restore passed, the agent was restarted 2 m 43 s later for a deploy, and two
// consecutive host-reports carried `0 restore-tests`.
//
// The durable half is `RestoreTestState` (on disk, per tier, with the archive) and the collector
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
type Store struct {
mu sync.Mutex
byTarget map[string]hub.Backup // latest backup per target id
+95 -13
View File
@@ -333,9 +333,23 @@ type BackupConfig struct {
LocalBackupTarget string `json:"local_backup_target"`
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
RestoreStorage string `json:"restore_storage"`
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
// RestoreTestCadenceSeconds is the LEGACY restore-test knob, retained for one meaning only:
// NEGATIVE still DISABLES the automatic restore-test entirely (on-demand selftest still works),
// and 0 still means "use the default". It no longer sets how often a test runs — R-86 replaced
// the interval trigger with a per-archive due-check — so a positive value now seeds
// RestoreTestSettleSeconds instead (see RestoreTestSettle). Prefer the two explicit keys below.
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
// RestoreTestEvalIntervalSeconds is how often the scheduler ASKS whether any tier is due
// (R-86); 0 → default. It is not how often a test runs: a tier is tested once per archive
// generation no matter how often it is asked. This interval sets two things — the latency
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
// Restore-testing an archive a backup is still writing proves nothing about the backup that
// finished — this is the same settle discipline R-71a's gate applies to the offsite consume.
RestoreTestSettleSeconds int `json:"restore_test_settle_seconds"`
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
// always excluded. Defaults to 990000990009.
@@ -545,26 +559,92 @@ func (b BackupConfig) BackupTarget() string {
return defaultBackupTarget
}
// Default scratch VMID band + restore-test cadence.
// Default scratch VMID band + the two R-86 restore-test knobs.
const (
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
defaultRestoreTestCadence = 24 * time.Hour
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
// defaultRestoreTestEvalInterval is how often due-ness is ASKED. It is bounded from BOTH sides,
// and neither bound alone would have picked it:
//
// FLOOR — what one evaluation costs. MEASURED on demo-felhom, 2026-08-03 (R-86 Part 1.4), via
// --selftest=restore-test-due and by timing the underlying API call directly. One evaluation
// is one storage-content listing per tier:
//
// local dir storage (3 archives) ....... 18 ms (18.7 / 18.3 / 18.5)
// PBS tier, WAN to ep0 (2 snapshots) ... 392 ms (375 / 378 / 424)
// both tiers together .................. 430 ms
//
// So cost does NOT set this: even at one evaluation a minute the offsite leg would be ~0.7 %
// of a WAN link's time and ~9 minutes of ep0's day. Worth writing down anyway, because the
// number that would have forbidden a frequent poll is the one nobody measures.
//
// CEILING — the retry rate of a FAILING tier. Under a per-archive due-check a tier whose
// restore-test keeps failing stays due, so the evaluation interval IS its retry interval, and
// a retry is a multi-GB restore. Every few minutes would be an incident of its own; the old
// timer retried a broken tier once a day.
//
// 6h sits between them: four heavy retries a day at the very worst, latency from settle to
// proof of at most 6h against a 24h settle lag (so a daily tier is still proved daily), and no
// second rate limiter anywhere — the pacing remains one test per archive generation.
defaultRestoreTestEvalInterval = 6 * time.Hour
// defaultRestoreTestSettle is how long an archive must sit before it may be restore-tested.
// 24h is R-86's own figure ("~24 h after its own newest archive") and it is what makes the
// candidate on a daily tier YESTERDAY's archive rather than the one still being written.
defaultRestoreTestSettle = 24 * time.Hour
)
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
// 0 → 24h default, negative → 0 (disabled).
func (b BackupConfig) RestoreTestCadence() time.Duration {
// RestoreTestEvalInterval returns how often the scheduler evaluates due-ness (R-86): a positive
// value as-is, 0 → the measured default, negative → 0 (disabled).
//
// The LEGACY `restore_test_cadence_seconds` keeps exactly one power here, the one a box may be
// relying on: a NEGATIVE value still disables the automatic restore-test outright. It no longer
// sets the interval, because the interval no longer decides that a test happens.
func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
if b.RestoreTestCadenceSeconds < 0 {
return 0 // legacy DISABLE — preserved verbatim
}
switch {
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
case b.RestoreTestCadenceSeconds < 0:
case b.RestoreTestEvalIntervalSeconds > 0:
return time.Duration(b.RestoreTestEvalIntervalSeconds) * time.Second
case b.RestoreTestEvalIntervalSeconds < 0:
return 0 // disabled
default:
return defaultRestoreTestCadence
return defaultRestoreTestEvalInterval
}
}
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
//
// WHAT HAPPENED TO THE OLD KEY. A box that set `restore_test_cadence_seconds` to a positive value
// was expressing "how long may pass between a backup and the confidence that it restores". That
// quantity survives R-86 as the SETTLE LAG, so a positive legacy value seeds this rather than being
// dropped or silently repurposed as the evaluation interval — and the daemon says so at start-up
// (see RestoreTestLegacyCadenceInUse). It is deliberately not carried into the evaluation interval:
// a box that set 72h to spare a weak endpoint would otherwise get a 72h-latency due-check, whereas
// what it actually wanted — fewer heavy restores — is what per-archive due-ness already gives it.
func (b BackupConfig) RestoreTestSettle() time.Duration {
switch {
case b.RestoreTestSettleSeconds > 0:
return time.Duration(b.RestoreTestSettleSeconds) * time.Second
case b.RestoreTestSettleSeconds < 0:
return 0 // explicitly no settle requirement
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second // legacy seeding
default:
return defaultRestoreTestSettle
}
}
// RestoreTestLegacyCadenceInUse reports whether the deprecated key is what is deciding the settle
// lag, so the daemon can name both replacements ONCE at start-up. A config key that changed meaning
// without saying so is exactly the silent repurposing §8.3 forbids.
func (b BackupConfig) RestoreTestLegacyCadenceInUse() bool {
return b.RestoreTestCadenceSeconds > 0 && b.RestoreTestSettleSeconds == 0
}
// PBSVerifyCadence returns the verify-loop interval: positive as-is, 0 → 6h default,
// negative → 0 (disabled).
func (b BackupConfig) PBSVerifyCadence() time.Duration {
@@ -810,6 +890,8 @@ func applyEnv(cfg *Config) {
cfg.Backup.RestoreStorage = v
}
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
cfg.Backup.RestoreTestEvalIntervalSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_EVAL_INTERVAL_SECONDS", cfg.Backup.RestoreTestEvalIntervalSeconds)
cfg.Backup.RestoreTestSettleSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_SETTLE_SECONDS", cfg.Backup.RestoreTestSettleSeconds)
}
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse
+96 -5
View File
@@ -47,6 +47,22 @@ type RestoreTestReporter interface {
RestoreTests(ctx context.Context) []RestoreTest
}
// ProvenRestoreTestReporter is the DURABLE half of the restore-test signal (R-189).
//
// RestoreTestReporter above is backed by an in-memory store whose own comment used to read "lost on
// restart; the cadence re-populates". That was true while a timer re-tested every tier daily. It
// stopped being true on 2026-08-03: under per-archive due-ness the agent will not re-test an archive
// it has already proven, so a proof lost to a restart is not repeated for a whole archive generation
// — a week on the offsite tier — and the hub reports the tier unproven the entire time.
//
// Observed, not predicted: a real 14.5 GB offsite restore passed at 15:25:14, the agent was restarted
// 2 m 43 s later for a deploy, and the hub logged `0 restore-tests` on the next two reports.
//
// (*backup.RestoreTestState).ProvenRestoreTests satisfies this. nil → the merge is a no-op.
type ProvenRestoreTestReporter interface {
ProvenRestoreTests(ctx context.Context) []RestoreTest
}
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
type PBSReporter interface {
@@ -79,6 +95,7 @@ type Collector struct {
storage StorageObserver
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
@@ -427,16 +444,90 @@ func (c *Collector) collectBackups(ctx context.Context) []Backup {
return []Backup{}
}
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
//
// The rule is ONE ENTRY PER TIER, NEWEST WINS, and it falls out of what each source means rather
// than from a preference between them:
//
// - the in-memory store holds this process's latest run, pass OR fail. A failure exists nowhere
// else and must always reach the hub — a failing tier is retried at the next evaluation, so its
// record is short-lived by design;
// - the persisted state holds the last SUCCESS per tier and survives a restart.
//
// Comparing by TestedAt gives the right answer in every case without special-casing: a fresh failure
// beats an older stored success (the failure is the news), a stored success beats a stale in-memory
// entry after a restart, and a tier proved twice never appears twice — two entries for one tier would
// read at the hub as two tests.
//
// A tier with no usable persisted proof contributes NOTHING. Reporting an unproven tier as proven
// would be a worse defect than the one this closes.
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
if c.restoreTests == nil {
return []RestoreTest{}
out := []RestoreTest{}
if c.restoreTests != nil {
if r := c.restoreTests.RestoreTests(ctx); r != nil {
out = append(out, r...)
}
}
if r := c.restoreTests.RestoreTests(ctx); r != nil {
return r
if c.provenTests == nil {
return out
}
return []RestoreTest{}
// Index what we already have by tier, keeping the newest per tier.
best := map[string]int{} // tier → index into out
for i, rt := range out {
if rt.SourceTier == "" {
continue // untiered entry: never deduped, never overwritten — we cannot say what it is
}
if j, seen := best[rt.SourceTier]; !seen || newerRestoreTest(rt, out[j]) {
best[rt.SourceTier] = i
}
}
for _, p := range c.provenTests.ProvenRestoreTests(ctx) {
if p.SourceTier == "" {
continue // not usable as a per-tier proof; the state layer already filters these
}
i, seen := best[p.SourceTier]
if !seen {
out = append(out, p)
best[p.SourceTier] = len(out) - 1
continue
}
if newerRestoreTest(p, out[i]) {
out[i] = p
}
}
return out
}
// newerRestoreTest reports whether a was tested after b. An unparseable or absent timestamp is
// treated as OLDER, so a malformed entry can never displace a good one.
func newerRestoreTest(a, b RestoreTest) bool {
ta, aok := parseRestoreTestedAt(a.TestedAt)
tb, bok := parseRestoreTestedAt(b.TestedAt)
if !aok {
return false
}
if !bok {
return true
}
return ta.After(tb)
}
func parseRestoreTestedAt(s string) (time.Time, bool) {
t, err := time.Parse(time.RFC3339, s)
if err != nil {
return time.Time{}, false
}
return t.UTC(), true
}
// SetProvenRestoreTests wires the durable proof source. It is a setter rather than a constructor
// argument because the persisted state is opened later in main() than the collector is built; the
// same shape as the other late-wired seams here. **The wiring is asserted by an AST test** — the
// method it feeds carried a doc comment naming a "host-report gauge" for weeks with no caller at
// all, and this fix must not become the next instance of that.
func (c *Collector) SetProvenRestoreTests(p ProvenRestoreTestReporter) { c.provenTests = p }
// collectPBSSnapshots reads the latest PBS snapshot inventory via the seam (nil → empty).
func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
if c.pbs == nil {
+213
View File
@@ -0,0 +1,213 @@
package hub
import (
"context"
"testing"
"time"
)
// R-189 — a passing restore-test must survive an agent restart and reach the hub.
//
// THE OBSERVATION THIS EXISTS FOR (2026-08-03, demo-felhom): a real 14.5 GB offsite restore-test
// PASSED at 15:25:14; the agent was restarted 2 m 43 s later for a deploy; the hub logged
// `0 restore-tests` on the next two host-reports. The in-memory store's own comment said "lost on
// restart; the cadence re-populates", which was true under a timer and stopped being true when R-86
// made the agent refuse to re-test an archive it has already proven.
//
// Timestamps here carry JITTER (odd minutes and seconds, not round hours) — yesterday a test was
// hollow because a perfectly regular series landed exactly on a threshold and passed under the
// mutation it was meant to catch.
type fakeLatest struct{ tests []RestoreTest }
func (f *fakeLatest) RestoreTests(context.Context) []RestoreTest { return f.tests }
type fakeProven struct{ tests []RestoreTest }
func (f *fakeProven) ProvenRestoreTests(context.Context) []RestoreTest { return f.tests }
func rt(tier, archive string, pass bool, at time.Time) RestoreTest {
return RestoreTest{
SourceArchive: archive, SourceTier: tier, Pass: pass,
Verified: "boot+running", TestedAt: at.UTC().Format(time.RFC3339),
}
}
// mergeCollector builds a Collector with only the two restore-test seams wired — the merge is what
// is under test, not the rest of the collection.
func mergeCollector(latest, proven []RestoreTest) *Collector {
c := &Collector{}
if latest != nil {
c.restoreTests = &fakeLatest{tests: latest}
}
if proven != nil {
c.provenTests = &fakeProven{tests: proven}
}
return c
}
func findTier(got []RestoreTest, tier string) (RestoreTest, int) {
var hit RestoreTest
n := 0
for _, e := range got {
if e.SourceTier == tier {
hit, n = e, n+1
}
}
return hit, n
}
// ── SCENARIO A — a proof survives a restart and reaches the hub ──────────────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): delete the `c.provenTests` merge from
// collectRestoreTests (return the in-memory slice as it used to) →
//
// --- FAIL: TestMerge_ProofSurvivesARestart
// restoretest_merge_test.go: after a restart the persisted proof must be reported; got 0 entr(ies)
//
// which is exactly the live observation: `0 restore-tests`. Restored.
func TestMerge_ProofSurvivesARestart(t *testing.T) {
provenAt := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC) // the real run's timestamp
// After a restart the in-memory store is EMPTY — this is the whole point.
c := mergeCollector([]RestoreTest{}, []RestoreTest{
rt("pbs", "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", true, provenAt),
})
got := c.collectRestoreTests(context.Background())
if len(got) != 1 {
t.Fatalf("after a restart the persisted proof must be reported; got %d entr(ies): %+v", len(got), got)
}
e := got[0]
if e.SourceArchive != "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" {
t.Fatalf("the entry must name the archive that was proven — the hub keys on it; got %q", e.SourceArchive)
}
if e.SourceTier != "pbs" || !e.Pass {
t.Fatalf("the entry must be a PASS on the tier it was proven on; got tier=%q pass=%v", e.SourceTier, e.Pass)
}
if e.TestedAt != provenAt.Format(time.RFC3339) {
t.Fatalf("the entry must carry the ORIGINAL test time, not now(); got %q", e.TestedAt)
}
}
// ── SCENARIO B — the report does not invent a pass ───────────────────────────────────────────
//
// COMPANION RED-PROOF (observed): make the state layer emit an entry for an unproven tier (drop the
// `reportable()` filter in ProvenRestoreTests, so a legacy record with no archive is emitted) — the
// equivalent at this layer is a proven-source that returns an entry for a tier nothing proved, which
// this test injects directly and the assertion below rejects.
func TestMerge_NeverInventsAPassForAnUnprovenTier(t *testing.T) {
// Nothing proven anywhere: no in-memory result, no persisted proof.
c := mergeCollector([]RestoreTest{}, []RestoreTest{})
if got := c.collectRestoreTests(context.Background()); len(got) != 0 {
t.Fatalf("a tier with no proof must produce NO entry — an unproven tier reading as proven is "+
"worse than the defect being fixed; got %+v", got)
}
// And an entry the state layer could not describe (no tier) is never promoted into a proof.
c2 := mergeCollector([]RestoreTest{}, []RestoreTest{
{SourceArchive: "local:backup/x.tar.zst", SourceTier: "", Pass: true,
TestedAt: time.Date(2026, 8, 1, 4, 41, 58, 0, time.UTC).Format(time.RFC3339)},
})
if got := c2.collectRestoreTests(context.Background()); len(got) != 0 {
t.Fatalf("a persisted record with no tier is not a usable proof and must be dropped; got %+v", got)
}
}
// ── SCENARIO C — a fresh in-memory result wins, and never duplicates ─────────────────────────
//
// COMPANION RED-PROOF (observed 2026-08-03): remove the de-duplication (append every persisted entry
// unconditionally) →
//
// --- FAIL: TestMerge_NewerWinsAndNeverDuplicatesATier
// restoretest_merge_test.go: one entry per tier; got 2 for "pbs" — the hub would read two tests
//
// Restored.
func TestMerge_NewerWinsAndNeverDuplicatesATier(t *testing.T) {
lastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC) // jittered, from the real box
fiveMinAgo := time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC)
c := mergeCollector(
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
)
got := c.collectRestoreTests(context.Background())
e, n := findTier(got, "pbs")
if n != 1 {
t.Fatalf("one entry per tier; got %d for \"pbs\" — the hub would read two tests: %+v", n, got)
}
if e.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
t.Fatalf("the NEWER result must win; got %q tested %q", e.SourceArchive, e.TestedAt)
}
// ...and the older-in-memory / newer-persisted direction, which is the post-restart case.
c2 := mergeCollector(
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, lastWeek)},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", true, fiveMinAgo)},
)
e2, n2 := findTier(c2.collectRestoreTests(context.Background()), "pbs")
if n2 != 1 || e2.SourceArchive != "felhom-pbs:backup/ct/9201/new" {
t.Fatalf("newest must win regardless of which source it came from; got %d entr(ies), archive %q", n2, e2.SourceArchive)
}
}
// ── SCENARIO D — a failure still reaches the hub ─────────────────────────────────────────────
//
// The merge must not mask a failure with an older stored success. A failing tier is retried at the
// next evaluation and its record lives ONLY in memory, so losing it here would silence the loudest
// DR signal this system produces.
func TestMerge_AFailureIsStillReported(t *testing.T) {
provenLastWeek := time.Date(2026, 7, 27, 19, 55, 41, 0, time.UTC)
failedJustNow := time.Date(2026, 8, 3, 13, 41, 7, 0, time.UTC)
c := mergeCollector(
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/new", false, failedJustNow)},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/old", true, provenLastWeek)},
)
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
if n != 1 {
t.Fatalf("one entry per tier; got %d: %+v", n, c.collectRestoreTests(context.Background()))
}
if e.Pass {
t.Fatalf("a FAILURE newer than the stored proof must be what is reported — masking it would "+
"silence the loudest DR signal there is; got pass=%v archive=%q", e.Pass, e.SourceArchive)
}
}
// Two different tiers are both reported — the merge is per tier, not a single slot.
func TestMerge_BothTiersSurvive(t *testing.T) {
c := mergeCollector(
[]RestoreTest{rt("local", "felhom-backup:backup/vzdump-lxc-9201-a.tar.zst", true,
time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))},
[]RestoreTest{rt("pbs", "felhom-pbs:backup/ct/9201/x", true,
time.Date(2026, 8, 2, 5, 12, 33, 0, time.UTC))},
)
got := c.collectRestoreTests(context.Background())
if _, n := findTier(got, "local"); n != 1 {
t.Fatalf("the in-memory tier must survive the merge; got %+v", got)
}
if _, n := findTier(got, "pbs"); n != 1 {
t.Fatalf("the persisted tier must survive the merge; got %+v", got)
}
}
// A malformed timestamp must never displace a good entry — "unparseable" is not "newest".
func TestMerge_MalformedTimestampNeverWins(t *testing.T) {
good := rt("pbs", "felhom-pbs:backup/ct/9201/good", true, time.Date(2026, 8, 3, 13, 25, 14, 0, time.UTC))
bad := RestoreTest{SourceArchive: "felhom-pbs:backup/ct/9201/bad", SourceTier: "pbs", Pass: true, TestedAt: "not-a-time"}
c := mergeCollector([]RestoreTest{good}, []RestoreTest{bad})
e, n := findTier(c.collectRestoreTests(context.Background()), "pbs")
if n != 1 || e.SourceArchive != "felhom-pbs:backup/ct/9201/good" {
t.Fatalf("an unparseable timestamp must not displace a good entry; got %d entr(ies), archive %q", n, e.SourceArchive)
}
}
// A nil proven-source leaves the pre-R-189 behaviour exactly as it was.
func TestMerge_NilProvenSourceIsANoOp(t *testing.T) {
only := rt("local", "felhom-backup:backup/x.tar.zst", true, time.Date(2026, 8, 3, 4, 44, 50, 0, time.UTC))
c := mergeCollector([]RestoreTest{only}, nil)
got := c.collectRestoreTests(context.Background())
if len(got) != 1 || got[0].SourceArchive != only.SourceArchive {
t.Fatalf("a nil durable source must not change anything; got %+v", got)
}
}
+29
View File
@@ -45,6 +45,35 @@ func (c *Client) Pool(ctx context.Context, name string) (PoolInfo, error) {
return p, c.get(ctx, "/pools/"+url.PathEscape(name), &p)
}
// Permissions returns the privileges this API TOKEN holds at an ACL path, as
// GET /access/permissions?path=<path> answers it: privilege name → 1.
//
// R-185. It asks about the CALLER — the agent's own token — which is the only useful form of the
// question. Asking as root answers a different question and always says yes.
//
// MEASURED SHAPE (demo-felhom, 2026-08-03), because the whole value of this call is reading the
// answer correctly and the obvious reading is wrong:
//
// /storage/felhom-pbs → {"Datastore.Allocate":1,"Datastore.AllocateSpace":1}
// /storage/felhom-backup → {"Sys.Audit":1,"SDN.Use":1,"Datastore.Audit":1}
//
// The ungranted path does NOT answer empty, and does NOT 403. It answers with the privileges
// INHERITED from the box-wide `/` grant — so "is this path present in the response" reports OK for a
// storage the agent demonstrably cannot list. The caller must test for the SPECIFIC privilege.
//
// The response is keyed by path; an absent path yields no privileges, which is the same answer as
// "none" and is treated as such by the caller.
func (c *Client) Permissions(ctx context.Context, aclPath string) (map[string]int, error) {
var raw map[string]map[string]int
if err := c.get(ctx, "/access/permissions?path="+url.QueryEscape(aclPath), &raw); err != nil {
return nil, err
}
if p, ok := raw[aclPath]; ok {
return p, nil
}
return map[string]int{}, nil
}
// GuestStatus returns GET /nodes/{node}/lxc/{vmid}/status/current. The API body
// has no vmid field (it is in the path), so it is set from the argument.
func (c *Client) GuestStatus(ctx context.Context, vmid int) (Guest, error) {
+7
View File
@@ -9,6 +9,12 @@
Gates (all must pass; **non-zero exit on any failure**):
1. reuse-refs every path cited by this repo's REUSE.md still resolves
2. published every `v<semver>` tag has a downloadable package AND a tag tree that serves
the agent's configs (R-115). NEEDS NETWORK, so it is **not** in `--fast` and
the pre-push hook does not run it a push must not fail because Gitea blinked
or because someone is offline on a train. CI runs the FULL set for exactly this
reason: it is the machine that can afford a network check, and it is the half
that emails when something is wrong.
WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)).
@@ -39,6 +45,7 @@ SHARED_REUSE = os.path.join(os.path.dirname(ROOT), "felhom.eu", "scripts", "reus
# (label, absolute script path, args, fast)
GATES = [
("reuse-refs", SHARED_REUSE, [ROOT], True),
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
+258
View File
@@ -0,0 +1,258 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""check-published-versions.py — a released agent version must be INSTALLABLE, not merely built.
R-115. A box installs the agent from a Gitea generic package the hub vouches, never from git, and
since R-110/R-183 it also fetches the agent's sixteen config files from `raw/tag/v<version>/`.
Nothing in the build, deploy or session-end path checked that either existed, so "deployed" and
"installable" were independent states that drifted silently. **Three instances in five days:**
* R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published a new customer
would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
* 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
* 0.120.0 (2026-08-03) built, committed, deployed to BOTH demo hosts, never published. A
documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
agent and would have *succeeded* while doing it.
THE INVARIANTS there are TWO now, and the second is R-188's price.
(1) For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
and the tag must serve the agent's configs.
(2) No PUBLISHED version may be missing its tag.
Invariant (2) is new (R-188, 2026-08-03) and it exists because `release-agent.sh` now pushes the tag
AFTER publishing. The old order pushed the tag first, and the old comment said why: a tag with no
package is caught here, a package with no tag is invisible, because the Gitea package LISTING api
needs a token this gate does not have. That reasoning was sound and the ordering was still wrong
the tag push is what wakes CI, so every correct release had a ~50% chance of running this gate in the
seconds before its own package existed and mailing the operator a failure for a release that worked
(measured across two releases: runs 12/13 and 17/18, same shas, opposite results).
Moving the push does not get to trade invariant (2) away, so it is asserted here instead WITHOUT a
token, and therefore as a BOUNDED PROBE rather than an enumeration:
* the FRONTIER the versions immediately above the highest tag. This is the realistic failure the
new ordering makes possible: publish succeeds, tag push fails, so the orphan is exactly one
version beyond the newest tag.
* the GAPS patch versions that fall between two existing tags and have no tag of their own.
Re-measured 2026-08-03, not assumed: `GET /api/v1/packages/admin?type=generic` answers **401** with no
token, so absence still cannot be proven. The probe set is PRINTED on every run, because a check whose
coverage is invisible reads as a guarantee it is not making.
The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
and that was measured rather than assumed:** the hub's artifact manifest
(`GET /api/v1/artifacts/<customer>`) answers **401** without a per-customer retrieval passphrase,
and the Gitea package LISTING api answers **401** without a token, while the package DOWNLOAD url
and the git TAGS api are both anonymous. So a credential-free gate can ask *"is this version
installable"* but not *"which version is vouched"*. Putting an operator credential into CI to close
that gap is the operator's call, not a gate author's it is recorded as a limitation below and as a
backlog row rather than quietly assumed away.
**What this invariant does catch: all three instances above.** `release-agent.sh` creates the tag and
publishes in one act, so a release whose publish was skipped, failed, or was forgotten leaves a tag
with no package which is exactly what this refuses. It needs no version floor: tags begin at
v0.120.0, which is published.
**What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. R-184.
Nor does the converse probe prove that NO untagged package exists only that none exists at the
probed versions, which are printed. Closing that properly needs a read token in CI ( R-184).
FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
INCONCLUSIVE**, naming every URL tried never a pass. "Cannot determine" is not "fine": that is the
standing rule this project earned twice, and a gate that green-lights on its own blindness is worse
than no gate, because it looks like coverage.
Pure python3 + urllib, NO curl and no third-party module: the CI runner is a host-mode container
carrying python3 and git and nothing else, and an earlier workflow step died on
`curl: command not found`.
python3 scripts/check-published-versions.py
Exit: 0 both invariants hold · 1 either is violated · 2 could not be determined.
Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
"""
import json
import os
import re
import sys
import urllib.error
import urllib.request
GITEA_BASE = os.environ.get("GITEA_BASE", "https://gitea.dooplex.hu").rstrip("/")
OWNER = "admin"
REPO = "felhom-agent"
PKG = "felhom-agent"
TIMEOUT = 25
# One config the installer fetches. Its presence proves the TAG's tree carries the configs the
# sixteen `fetch_raw` calls will ask for — a tag that exists but predates them would 404 a box
# mid-install, on a virgin machine, as root.
PROBE_CONFIG = "configs/felhom-agent.service"
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
tried = []
def _get(url, want_body=False):
"""GET a URL. Returns (status, body_or_None). Network failure raises."""
tried.append(url)
req = urllib.request.Request(url, method="GET")
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
body = r.read() if want_body else None
return r.status, body
except urllib.error.HTTPError as e:
return e.code, None
def inconclusive(msg):
print("INCONCLUSIVE:", msg)
print(" URLs tried (a 'no access' claim must name its attempts):")
for u in tried:
print(" ", u)
sys.exit(2)
def _pkg_exists(version):
"""True iff the generic package for `version` is downloadable anonymously."""
url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, version, PKG)
status, _ = _get(url)
return status == 200, url
def untagged_probe_set(versions):
"""The versions to probe for invariant (2), as (version, why) pairs.
Bounded on purpose and printed by the caller: the package listing api needs a token (401,
re-measured 2026-08-03), so absence cannot be enumerated. What CAN be done is to probe the
places an orphan would actually land.
FRONTIER a publish that succeeded followed by a tag push that failed leaves the orphan
exactly one version past the newest tag. This is the failure mode the R-188
reordering makes possible, so it is the one that must not be guesswork.
GAPS a patch number skipped between two consecutive tags. Bounded per gap so a typo'd
tag (v0.130.0 after v0.121.1) cannot turn this into a thousand requests.
"""
parsed = sorted(tuple(int(p) for p in v.split(".")) for v in versions)
have = set(parsed)
out = []
if not parsed:
return out
hi = parsed[-1]
for cand, why in (
((hi[0], hi[1], hi[2] + 1), "next patch after the newest tag"),
((hi[0], hi[1], hi[2] + 2), "second patch after the newest tag"),
((hi[0], hi[1] + 1, 0), "next minor after the newest tag"),
((hi[0] + 1, 0, 0), "next major after the newest tag"),
):
if cand not in have:
out.append(("%d.%d.%d" % cand, why))
MAX_GAP_PROBES = 12
for a, b in zip(parsed, parsed[1:]):
if a[0] != b[0] or a[1] != b[1]:
continue # a minor/major step is not a patch gap
for patch in range(a[2] + 1, min(b[2], a[2] + 1 + MAX_GAP_PROBES)):
cand = (a[0], a[1], patch)
if cand not in have:
out.append(("%d.%d.%d" % cand, "patch gap between v%d.%d.%d and v%d.%d.%d" % (a + b)))
return out
def main():
print("check-published-versions — every released agent version must be INSTALLABLE")
print(" gitea:", GITEA_BASE)
tags_url = "%s/api/v1/repos/%s/%s/tags?limit=200" % (GITEA_BASE, OWNER, REPO)
try:
status, body = _get(tags_url, want_body=True)
except Exception as e:
inconclusive("cannot reach Gitea to list tags: %s" % e)
if status != 200 or not body:
inconclusive("tags api returned HTTP %s — cannot enumerate releases" % status)
try:
tags = [t["name"] for t in json.loads(body.decode("utf-8"))]
except Exception as e:
inconclusive("tags api response is not the expected JSON: %s" % e)
versions = sorted(m.group(1) for m in (TAG_RE.match(t) for t in tags) if m)
if not versions:
# Not a failure: a repo legitimately has no version tags before its first release. Say so
# loudly rather than reporting a vacuous pass — an empty check that prints OK is how an
# unexamined thing becomes a documented-clean one.
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
print("\ncheck-published-versions: NOTHING TO CHECK")
return 0
print(" %d released version(s) to verify: %s" % (len(versions), ", ".join(versions)))
bad = []
for v in versions:
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
try:
pkg_status, _ = _get(pkg_url)
raw_status, _ = _get(raw_url)
except Exception as e:
inconclusive("network failure while checking v%s: %s" % (v, e))
problems = []
if pkg_status != 200:
problems.append("binary NOT downloadable (HTTP %s at %s)" % (pkg_status, pkg_url))
if raw_status != 200:
problems.append("tag does not serve %s (HTTP %s) — a box would 404 mid-install"
% (PROBE_CONFIG, raw_status))
if problems:
bad.append((v, problems))
print(" FAIL v%s:" % v)
for p in problems:
print(" -", p)
else:
print(" ok v%s: binary downloadable + tag serves its configs" % v)
# ── invariant (2): no PUBLISHED version may be missing its tag (R-188) ──────────────────────
probes = untagged_probe_set(versions)
orphans = []
print()
print(" converse probe — a published version with no tag (bounded; the package listing api")
print(" needs a token, so this cannot enumerate). Probing %d version(s):" % len(probes))
for v, why in probes:
try:
exists, url = _pkg_exists(v)
except Exception as e:
inconclusive("network failure while probing v%s: %s" % (v, e))
mark = "PUBLISHED — NO TAG" if exists else "absent (ok)"
print(" %-10s %-42s %s" % (v, why, mark))
if exists:
orphans.append((v, url))
print()
if bad or orphans:
if orphans:
print("check-published-versions: %d PUBLISHED VERSION(S) WITH NO TAG" % len(orphans))
for v, url in orphans:
print(" v%s is downloadable at %s but has no git tag." % (v, url))
print(" A release publishes and then pushes its tag; a package with no tag means the")
print(" push failed or was skipped. The local tag is probably still in the release")
print(" clone — finish it with:")
for v, _ in orphans:
print(" git push origin v%s" % v)
print(" (and if the tag is gone, re-create it on the released commit before pushing.)")
if bad:
print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —")
print(" the R-115 defect, three times in five days. Publish it with:")
print(" scripts/release-agent.sh <version>")
if bad or orphans:
return 1
print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED")
return 0
if __name__ == "__main__":
sys.exit(main())
Regular → Executable
+5 -1
View File
@@ -51,7 +51,11 @@ if [[ -z "$BIN" ]]; then
BIN="$(mktemp -t felhom-agent.XXXXXX)"
CLEANUP_BIN="$BIN"
log "building felhom-agent $VERSION from $REPO_ROOT"
( cd "$REPO_ROOT" && CGO_ENABLED=0 go build -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
# These flags MUST match release-agent.sh's build exactly — see the long comment there (R-186).
# They used to differ: this line forced CGO_ENABLED=0 and produced a binary 74 KB smaller than
# the one the release path built for the same version. One version name must mean one binary
# whichever entry point produced it.
( cd "$REPO_ROOT" && go build -trimpath -buildvcs=false -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
fi
[[ -f "$BIN" ]] || die "binary not found: $BIN"
trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env bash
# release-agent.sh — THE way to release a felhom-agent version. One act: build → tag → publish →
# verify by independent download.
#
# WHY THIS EXISTS (R-115). Publishing used to be a step someone had to remember, and it was
# forgotten THREE TIMES IN FIVE DAYS:
#
# * R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published, so a new customer
# would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
# * 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
# * 0.120.0 (2026-08-03) built, committed and deployed to BOTH demo hosts, never published. A
# documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
# agent — and would have *succeeded* while doing it, because the current `step_grows`
# sets SYSDATA_GROW=0 so the older agent's fatal mp1 resize never fires.
#
# R-111's own closing line said publishing should join the release train rather than stay a
# remembered step. It closed SHIPPED without that leg, and the leg recurred the same afternoon —
# which is the evidence that a note is not a mechanism. This file is the mechanism. The
# documentation now points here instead of at a raw `go build` line, so there is ONE documented way
# to release and it cannot complete without publishing.
#
# WHY IT TAGS (R-183). Since felhom-host-install.sh pins its sixteen agent-config fetches to
# `raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root, on a
# virgin machine. The tag and the package are two halves of one release and are created together.
#
# WHY IT DOES NOT VOUCH. Vouching is what points machines at a version, and it stays the operator's
# deliberate act — the same prove-then-vouch principle that governed the golden two sessions ago.
# This script prints the version and sha to vouch; a human decides when.
#
# Usage:
# GITEA_USER=admin GITEA_TOKEN=<token> ./scripts/release-agent.sh <version>
#
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
# GITEA_BASE / GITEA_OWNER override the defaults.
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
set -euo pipefail
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
GITEA_OWNER="${GITEA_OWNER:-admin}"
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
die() { echo "[release-agent] ERROR: $*" >&2; exit 1; }
log() { echo "[release-agent] $*" >&2; }
VERSION="${1:-}"
[[ -n "$VERSION" ]] || die "version required (usage: GITEA_USER=.. GITEA_TOKEN=.. $0 <version>)"
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "version must be bare semver X.Y.Z (got '$VERSION')"
TAG="v$VERSION"
cd "$REPO_ROOT"
# ── 1. Clean-tree gate ──────────────────────────────────────────────────────────────────────────
# An unpushed change does not exist. Releasing a dirty tree publishes a binary whose source nobody
# else can obtain, and tags a commit that does not contain what was built.
if [[ "${RELEASE_ALLOW_DIRTY:-0}" != "1" ]]; then
[[ -z "$(git status --porcelain)" ]] || die "working tree is dirty — commit and push first"
local_head="$(git rev-parse HEAD)"
git fetch -q origin main
[[ "$local_head" == "$(git rev-parse origin/main)" ]] \
|| die "HEAD != origin/main — push first (an unpushed change does not exist)"
fi
# ── 2. Refuse to re-release a version that already exists ───────────────────────────────────────
# Silently overwriting a published artifact is how "the same version" comes to mean two different
# binaries on two different boxes.
if git rev-parse -q --verify "refs/tags/$TAG" >/dev/null; then
die "tag $TAG already exists — releasing over it would make one version name two binaries"
fi
existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
[[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead"
# ── 3. Build ────────────────────────────────────────────────────────────────────────────────────
BIN="$(mktemp -t felhom-agent-XXXXXX)"
trap 'rm -f "$BIN"' EXIT
log "building $VERSION"
# REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and
# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS
# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries.
# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags:
#
# default flags, no tag yet .. 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
# default flags, tagged ...... 4a38f394… 14 085 440 B (mod v0.121.99)
# -trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
#
# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller,
# verified) and the version comes from the explicit ldflag below, which is where it belongs.
# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also
# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens.
#
# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force
# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same
# version — one version name, two binaries, by whichever entry point was used. Both now build the
# same way; if that ever has to change, change it in BOTH or the guarantee is gone.
go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|| die "go build failed"
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
[[ "$built_ver" == "$VERSION" ]] \
|| die "the built binary reports '$built_ver', not '$VERSION' — the ldflag did not take"
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
log "built ok: sha256 $BUILT_SHA"
# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
#
# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
#
# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no
# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it,
# because the Gitea package LISTING api needs a token the gate does not have.
#
# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in
# the seconds between the tag becoming visible and the package existing — and correctly failed. Every
# correct release had roughly a coin-flip chance of emailing the operator a failure for a release
# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18
# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be
# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too.
#
# So the tag is still created HERE, before anything is published — the build and the tag still
# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone.
# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package
# is downloadable. The invariant the old order protected is not traded away: it is asserted directly
# by the gate's new converse probe (a published version with no tag FAILS), so both directions are
# now checked rather than one being arranged for.
log "tagging $TAG at $(git rev-parse --short HEAD)"
git tag -a "$TAG" -m "agent $TAG
Released by scripts/release-agent.sh.
sha256 of the published binary: $BUILT_SHA
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
so this tag is part of the released artifact, not a bookmark (R-183)."
# ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
log "publishing …"
# Invoked through `bash` DELIBERATELY, not as an executable. On 2026-08-03 the first real release
# through this script died here — `publish-agent.sh` has been mode 0644 since it was created on
# 2026-06-28, because every earlier caller ran it as `bash scripts/publish-agent.sh`. So the one leg
# R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
# the same commit; this line makes the release independent of it, because a file mode is exactly the
# kind of thing that is lost again by a checkout, an archive, or a copy.
if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
# The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next
# attempt would die at step 2's "tag $TAG already exists" and read as "this version is already
# released", which would be exactly backwards. Only remove it if nothing was in fact published —
# if a package DOES exist, the tag is wanted and must be pushed, not deleted.
now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
if [[ "$now_published" == "200" ]]; then
log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG"
else
git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried"
fi
die "publish failed"
fi
# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
#
# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the
# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches.
# The recovery is one line and it is printed rather than described.
log "pushing $TAG"
if ! git push origin "$TAG"; then
cat >&2 <<EOF
RELEASE HALF DONE — the package is PUBLISHED and its tag is NOT pushed.
version : $VERSION
sha256 : $BUILT_SHA
The tag exists in this clone only. Nothing installs from an untagged version (the installer
fetches this version's configs from raw/tag/$TAG/), and scripts/check-published-versions.py will
FAIL on it as a published version with no tag. Finish the release with:
git push origin $TAG
EOF
die "tag push failed after a successful publish — see above"
fi
# ── 7. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
# The publish step's own success is not proof: it reports on its own write. What matters is that a
# box can now GET the bytes and that they are the bytes that were built. This is the same
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that
# returns the right sha are different claims.
log "verifying by independent download …"
DL="$(mktemp -t felhom-agent-dl-XXXXXX)"
trap 'rm -f "$BIN" "$DL"' EXIT
curl -fsS -o "$DL" "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" \
|| die "round-trip GET failed — the version is NOT installable"
DL_SHA="$(sha256sum "$DL" | awk '{print $1}')"
[[ "$DL_SHA" == "$BUILT_SHA" ]] \
|| die "published sha $DL_SHA != built sha $BUILT_SHA — the artifact is not what was built"
# The tag must also serve the configs the installer will fetch from it.
cfg_code="$(curl -fsS -o /dev/null -w '%{http_code}' \
"$GITEA_BASE/$GITEA_OWNER/felhom-agent/raw/tag/$TAG/configs/felhom-agent.service" 2>/dev/null || true)"
[[ "$cfg_code" == "200" ]] \
|| die "tag $TAG does not serve configs/felhom-agent.service (HTTP $cfg_code) — a box would 404 mid-install"
cat <<EOF
RELEASED — and installable, verified by download, not by this script's own say-so.
version : $VERSION
tag : $TAG
sha256 : $BUILT_SHA
NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act:
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
EOF