diff --git a/REPORT.md b/REPORT.md index 5e8e9cb..304c4b2 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,9 +1,9 @@ -# REPORT — R-86: a restore-test proves each BACKUP, not the clock +# REPORT — R-189 · R-188 · R-186: three ways the signals lied about themselves -**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.120.0 → v0.121.0 → v0.121.1** -(`4618169`, `4d82591`, `53d0c6b`) · -released, published, verified by independent download, deployed to demo-felhom and **proven live**. -Sibling half: `felhom.eu` hub **v0.91.0 → v0.91.1** — the two ship together. +**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.121.1 → v0.122.0** (`7581f81`) · released, +published, verified by an independent download **and by rebuilding it**, deployed to demo-felhom. +`felhom.eu`: register + docs only, **no hub change and no hub bump** — the hub already reads +`restore_tests[]`; the defect was that the agent stopped sending them. --- @@ -11,307 +11,271 @@ Sibling half: `felhom.eu` hub **v0.91.0 → v0.91.1** — the two ship together. | Repo | `main` @ commit | Version | Matched §1? | |---|---|---|---| -| `felhom-agent` | `1b14cfd0b48b` | `v0.120.0` | **yes** | -| `felhom.eu` | `e34b614e5b65` | CHANGELOG `v0.90.0`, deployed image `0.90.1` | **yes — the discrepancy was real and is fixed** (entry backfilled) | +| `felhom-agent` | `3d0a1d615d11` | `v0.121.1` | **yes** | +| `felhom.eu` | `c9a3e48b2106` | hub `v0.91.1` | **yes** | -Highest register ID in use was **R-184**; `R-185`–`R-187` established free by grep across all four -repos and `documentation/`. +Highest register ID in use was **R-189**; no new IDs were needed — all three rows already existed. +(Grep confirmed R-190+ free, in case one had been.) -## 2. The rule, in one sentence — and the trap it avoids +## 2. Scenario H — the reproducibility measurement (Part 3, done first on purpose) -> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h). -> The tier is **DUE** when A exists and **A has not already been proven**. +**Before**, one commit, same source, same toolchain, same ldflags — the only difference is whether the +tag existed when the build ran: -The literal reading of R-86's own wording — *"due when the newest archive is ≥ 24 h old"* — is -**never true on a daily tier**, because a new archive lands each day and resets the newest-archive age -to zero long before it reaches the lag. It would have switched restore-testing **off** for the tier -that matters most, silently, while looking like the row was implemented. +| build | sha256 | size | embedded module version | +|---|---|---|---| +| default flags, **no tag yet** | `18f4a495…` | 14 085 464 B | `v0.121.2-0.20260803133646-3d0a1d61` | +| default flags, **tagged** | `4a38f394…` | 14 085 440 B | `v0.121.99` | +| `-trimpath -buildvcs=false`, either way | `7ffcdf1d…` | 14 064 574 B | *(none)* | -**Evidence that a daily tier does become due**, at three levels: +**After, on the real release (v0.122.0) — the three values §15.2 asks for:** -1. **Unit, time-driven** — `TestDue_DailyTierIsProvedDailyOnItsOwnArchive`: five simulated days, one - archive a day, evaluated hourly (120 evaluations) → **exactly 5 runs**, and run *i* tests day - *i−1*'s archive, never the still-settling one. -2. **The red-proof of the naive rule** — implemented and observed failing at **0 runs over 5 days** - (§6), which is the trap made visible rather than argued about. -3. **Live** — the offsite tier on demo-felhom became due on its own archive and ran (§7). - -## 3. What changed - -| Piece | File | Change | +| artifact | sha256 | size | |---|---|---| -| the due-check | `internal/backup/restoretest_due.go` (new) | `EvaluateDue` / `evaluateTier` — per-tier verdict + the reason, ordered oldest-proven first | -| the trigger | `internal/backup/schedule.go` | the ticker is now the **evaluation interval**; `pickForThisRun` answers *"is anything due?"*, and "nothing" is a normal answer | -| the state | `internal/backup/restoretest_state.go` | records **which archive** was proven, with migration | -| the picker | `internal/backup/runner.go` | `PickSettledRestoreCandidateOn(ctx, target, notAfter)`; `PickRestoreCandidateOn` is a one-line call into it | -| the knobs | `internal/config/config.go` | `restore_test_eval_interval_seconds` + `restore_test_settle_seconds`; the old key deprecated, not repurposed | -| the wiring | `cmd/felhom-agent/main.go` | settle-aware picker + `Settle`; deprecation WARN; new `--selftest=restore-test-due` | -| the observable (**v0.121.1**) | `internal/backup/schedule.go`, `restoretest_due.go` | a not-due evaluation logs one **INFO** line naming every tier's verdict — see §9 | +| published, downloaded from Gitea | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B | +| rebuild at the tag, #1 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B | +| rebuild at the tag, #2 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B | -### The state records the archive (§8.2) +**All three identical.** The property is removed-cause, not sequenced-around: `-buildvcs=false` drops +a stamp nothing reads (no `ReadBuildInfo` caller, verified by grep), and the version still comes from +the explicit `-X main.version` ldflag. `-trimpath` additionally makes a rebuild from a different +checkout directory match. -A timestamp cannot answer *"have we proven **this** archive"* — it is the same class as the workspace -rule that a timestamp records an *attempt*, not a *result*: here it records a result, but not **which** -result. `RestoreTestState` now holds `{archive, proven_at}` per tier. +**A second discrepancy fell out of the measurement and is fixed with it:** `publish-agent.sh`'s +fallback build forced `CGO_ENABLED=0` and produced **13 990 236 B** against the release path's +**14 064 574 B** — a 74 KB difference, i.e. one version name meaning two binaries depending on which +entry point ran. Both paths now use identical flags, each commented with a pointer to the other. -**Migration:** a pre-R-86 file (`{"target": ""}`) keeps its **time** — rotation ordering -survives a deploy, which is why the file exists at all — and yields **no proven archive**, so each -tier is due exactly once after the upgrade. One extra test per tier, once, is the safe direction; -reading a legacy time as proof of whatever archive is current would invent a guarantee. +## 3. Scenario E — a correct release no longer emails a failure (Part 2) -### The config (§8.3) — and a correction to the spec +**Only the tag PUSH moved.** The order is now build → tag **locally** → publish → push tag. The tag is +still created before anything is published, so the build and the tag describe the same commit; it +becomes *visible* — to CI (`on: [push]`) and to any `raw/tag/…` fetch — only once the package is +downloadable. -The spec said *"`0` must keep meaning disabled"*. **In the code as it stands, `0` means *use the -default* and NEGATIVE means disabled** (`RestoreTestCadence`, pre-existing). Making `0` disable would -have switched restore-testing off on every box that leaves the key unset — the worst possible reading -— so the actual semantics were preserved and this is flagged rather than silently followed. +**The old ordering's invariant is asserted directly rather than arranged for.** +`check-published-versions.py` now carries two invariants: every tag has an installable package (as +before) **and no published version is missing its tag** (new). The second is a **bounded probe** — +the frontier past the newest tag, where a failed tag push leaves an orphan, plus patch gaps — and it +**prints its probe set on every run**, because a check whose coverage is invisible reads as a +guarantee it is not making. The package listing api was **re-measured**, not assumed: `401` without a +token, so absence still cannot be enumerated, and the script says so in its own output. -- `restore_test_eval_interval_seconds` — how often due-ness is **asked**. Default **6 h**. -- `restore_test_settle_seconds` — how long an archive must sit. Default **24 h**. -- `restore_test_cadence_seconds` — **deprecated**. Negative still **disables**, verbatim. A positive - value seeds the **settle lag** (the quantity a person setting it was expressing: how long may pass - between a backup and confidence that it restores), and the daemon logs one start-up WARN naming - both replacements. It is deliberately **not** carried into the evaluation interval: a box that set - 72 h to spare a weak endpoint would otherwise get a 72-hour-latency due-check, whereas what it - wanted — fewer heavy restores — is what per-archive due-ness already gives it. +**Live result — this release is the test:** -## 4. Part 1.4 — the measurement, and the interval chosen from it - -Measured on demo-felhom, 2026-08-03, via `--selftest=restore-test-due` and by timing the underlying -API call directly (3 runs each): - -| tier | what it is | one due-check | +| release | CI runs on the release commit | outcome | |---|---|---| -| `felhom-backup` (dir) | local, on-box | **18 ms** (18.7 / 18.3 / 18.5) | -| `felhom-pbs` | offsite, **WAN to ep0** | **392 ms** (375 / 378 / 424) | -| both together | one full evaluation | **430 ms** | +| v0.121.0 (yesterday) | #12 / #13 | one **green**, one **red** | +| v0.121.1 (yesterday) | #17 / #18 | one **red**, one **green** | +| **v0.122.0 (this one)** | **#21 (task id 96) / #22 (task id 97)** | **both green** | -**Cost does not set the interval** — even at one evaluation a minute the offsite leg would be ~0.7 % -of the link's time. What sets it is the other bound, and it is not in the brief: **under a per-archive -due-check a FAILING tier stays due, so the evaluation interval is also its RETRY interval — and a -retry is a multi-GB restore.** Every few minutes would be an incident of its own; the old timer -retried a broken tier once a day. +**Failure modes made loud rather than tidy:** a publish that succeeds followed by a tag push that +fails now dies printing `git push origin v` — the local tag is already there, so recovery is one +line — and a publish that *fails* deletes the local-only tag so a retry is clean instead of colliding +with step 2's re-release guard. -**6 h chosen from both ends:** at most four heavy retries a day in the worst case, and at most 6 h of -latency between an archive settling and its proof — negligible against a 24 h settle lag, so a daily -tier is still proved daily. No second rate limiter was added (§8.4): the pacing remains one test per -archive generation. +## 4. Scenarios F and G — both directions demonstrated, then cleaned up -## 5. Two hazards the new frequency created, and their fixes +**G — a published version with no tag must FAIL.** A real fixture: version **0.121.2** (the frontier, +exactly where a failed tag push lands) published to the live registry with no tag. -Both are consequences of evaluating often rather than daily, and neither is in the brief: +``` +0.121.2 next patch after the newest tag PUBLISHED — NO TAG +check-published-versions: 1 PUBLISHED VERSION(S) WITH NO TAG + v0.121.2 is downloadable at … but has no git tag. + git push origin v0.121.2 +EXIT=1 +``` -1. **The due-check now runs BEFORE the heavy-operation gate is taken.** Holding that gate for a read - that answers "nothing to do" would open a window at *every* evaluation in which a starting backup - cannot acquire — and a backup that cannot acquire does not merely wait, it **records a failure and - pages the operator** (F-A1). Nothing heavy starts before the gate; due-ness does not expire while - we check. -2. **The candidate picker skips implausible archives.** Under per-archive due-ness an incomplete - 1-byte phantom (F-CRIT-2's artefact, which server-side prune does **not** collect) would be picked - forever, fail forever, never earn proof, and leave the tier due at *every* evaluation — turning the - evaluation interval into the retry rate for a multi-GB restore. `archivePlausiblyComplete` (the - canonical helper, with its warn-once companion) is applied in the shared scan, so both callers - agree. **This is a behaviour change to `PickRestoreCandidateOn`** and is recorded as such. +**Red-proof (observed):** with the fixture still live, replacing the probe set with `[]` produced +`ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED`, **exit 0** — a green run over a published +orphan. Restored. -## 6. Tests and red-proofs +**Teardown:** fixture deleted (HTTP 204), absence independently re-verified (`GET → 404`), gate green +again (exit 0). No scratch tag was ever pushed. -Green gate, both repos: `go build ./... && go vet ./... && go test ./...` — agent **29 packages ok, -rc=0**; hub **rc=0**. The test run and the commit were always separate commands. +**F — a tag with no package must still FAIL.** Demonstrated against a local stand-in serving two tags +where only one has a package: + +``` + ok v0.120.0: binary downloadable + tag serves its configs + FAIL v0.199.0: + - binary NOT downloadable (HTTP 404 …) + - tag does not serve configs/felhom-agent.service (HTTP 404) — a box would 404 mid-install +EXIT=1 +``` + +**Why not a real pushed tag:** pushing one wakes CI and would have emailed the operator a **true** +alarm about a fixture — the same attention cost R-188 exists to remove. F is also already +demonstrated in the wild: CI runs **#13** and **#17** failed for exactly this reason yesterday. + +## 5. R-189 — a proof that survives a restart reaches the hub (Part 1) + +`restore_tests[]` came only from the in-memory `backup.Store`, whose comment read *"lost on restart; +the cadence re-populates"*. True under a timer; false since R-86, because the agent refuses to re-test +an archive it has already proven — so a lost proof is not repeated for a whole archive generation. + +**What changed** + +- `RestoreTestState` stores the **tier** and what was **verified** beside the archive (v3 shape). + Both are recorded **at proof time from the run's own result** — deriving them later would need a + storage-type lookup at report-building time, a network call that can fail on the one path where + failing means mislabelling a proof. +- `ProvenRestoreTests` renders the stored proofs as report entries; `Collector.SetProvenRestoreTests` + merges them with the in-memory result. +- **Merge rule: one entry per tier, newest by `TestedAt` wins.** It falls out of what each source + means rather than from a preference: a fresh failure beats a stored success (the failure is the + news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a + tier never appears twice — the hub would read that as two tests. An unparseable timestamp counts as + **older**, so a malformed entry cannot displace a good one. +- **It refuses to lie.** A record missing the archive **or** the tier produces no entry, and run + mechanics (scratch VMID, duration) are not re-invented: an absent duration is not a claim, a + fabricated one would be. +- **The asymmetry is now in the code** (§8.1): a success *suppresses* future work so it must be + durable; a failure *causes* future work and heals itself, and persisting one would make a healed + tier keep reporting a fault. +- **`Store`'s comment is corrected in place** — leaving it is how the next reader concludes this is + handled. + +**Migration, and it is visible on the live box:** a pre-R-189 record has an archive but no tier, so it +is **not** reportable. Upgrading does not retroactively make an old proof visible; the tier's next +real proof fills it in. Confirmed immediately after the deploy — still `0 restore-tests`, with the +v2 record sitting on disk. + +## 6. Scenario A, live on demo-felhom — against the observation that filed R-189 + +**The observation being replaced (2026-08-03, 15:25):** a real offsite restore-test PASSED, the agent +was restarted 2 m 43 s later, and the hub logged `0 restore-tests` on the next two host-reports. + +**The same sequence, on v0.122.0:** + +``` +16:44:06 restore-test tier is DUE target=felhom-pbs + archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z + reason="newest settled archive … has not been proven (last proven archive was a different one)" +16:55:21 restore-test: scratch guest torn down vmid=990000 +16:55:21 backup: scheduled restore-test PASSED archive=felhom-pbs:…2026-07-27T19:55:41Z duration_s=675.1 +16:55:32 systemctl restart felhom-agent ← INSIDE the 15-minute reporting window +16:55:36 hub: host-report from demo-felhom-8363b5 (… 1 restore-tests …) ← was 0 +``` + +**The proof on disk (v3 — the tier is what the old shape lacked):** + +```json +{"felhom-pbs": {"archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z", + "tier": "pbs", "verified": "boot+running", "proven_at": "2026-08-03T14:55:21Z"}} +``` + +**What the HUB stored** — read from its own database (copied with its `-wal`, freshness confirmed by +the newest row's `received_at` = `2026-08-03 14:55:36` UTC, matching the ingest line): + +```json +{ "source_archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z", + "source_tier": "pbs", "pass": true, "verified": "boot+running", + "tested_at": "2026-08-03T14:55:21Z", "scratch_vmid": 0, "duration_seconds": 0 } +``` + +That report was built **after** the restart, when the in-memory store was empty — so the entry can +only have come from the persisted state. The archive, the tier, and the **original** test time +survived; the run mechanics are zero because they are deliberately not re-invented. + +**A 14.5 GB encrypted offsite archive**, restored, booted, verified and destroyed in **675 s** — and +this time the proof outlived the process that produced it. + +**Teardown, all three layers:** scratch guest absent from `pct list` (0), its volumes gone from `lvs` +(0), the validation drop-in removed and the daemon back on its defaults +(`eval_interval=6h0m0s settle=24h0m0s`). The hub-side `restore_tests[]` record is **retained +deliberately** — it is the proof the staleness check reads, so deleting it would delete the result. +No `restore_test_*` event was raised, because nothing failed and nothing is stale. + + +## 7. Tests and red-proofs + +Green gate: `go build ./... && go vet ./... && go test ./...` — **29 packages ok, rc=0**, plus +`python3 scripts/agent_gates.py` (reuse-refs + published-versions) all OK. The test run and the commit +were always separate commands. | # | Test | Asserts | Mutation | Observed | |---|---|---|---|---| -| A | `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` | 5 runs over 5 days, each on the settled archive | the naive rule (`now-landed >= settle`, proven-archive check deleted, cutoff removed) | **FAIL** — `a daily tier must be proved once per day; got 0 run(s) over 5 days: []` | -| B | `TestDue_WeeklyTierIsProvedOncePerArchive` | 84 evaluations over 3 weeks → exactly 3 runs, one per archive | — | pass | -| C | `TestDue_RestartRunsNothing` | two restarts + 4 evaluations → **0 runs** | `ProvenArchive` reverted to per-tier time | **FAIL** — `2 restart(s) produced 4 run(s)` | -| D | `TestDue_NewSettledArchiveMakesAProvedTierDueAgain` | a newly settled archive re-arms the tier, and the NEW archive is tested | — | pass | -| E | `TestDue_FailingTierIsRetriedAndNeverProven` | 3 evaluations → 3 retries, no proof recorded | credit on failure (`rt.Pass &&` dropped) | **FAIL** — `got 1 run(s) over 3 evaluations` + `TestRotation_FailureEarnsNoCredit` also failed | -| F | `TestDue_TwoDueTiersRunOneAtATime` | one evaluation → one run; the other is deferred and runs next | — | pass | -| F′ | `TestDue_DeferredBehindABackupStaysDue` | the gate holds; a deferred tier stays DUE | — | pass | -| H | `TestDue_NewbornTierIsNotDueAndNotAnError` | no archive → not due, no error, **and a reason** | — | pass | -| — | `TestDue_UnsettledArchiveIsNotACandidate` | a 2 h-old archive is not a candidate under a 24 h lag | — | pass | -| — | `TestDue_LookupFailureIsUnknownNotNotDue` | a tier that cannot be listed is UNKNOWN, the error travels, the other tier still runs | — | pass | -| — | `TestRestoreTestState_LegacyFileMigratesToNothingProven` | legacy time kept, no archive claimed | — | pass | -| — | `TestPickRestoreCandidate_SkipsImplausibleArchives` | the newest entry is not a candidate if it cannot be complete | guard removed | **FAIL** — `pick = "phantom" want the newest COMPLETE archive 'real'` | -| I | `TestMainWiresTheSettleAwareTierPicker` | **AST** of `main.go`: settle picker wired, old picker gone, `Settle` set, eval-interval accessor called | the wiring line commented out | **FAIL** — `main.go never passes runner.PickSettledRestoreCandidateOn …` (a `strings.Contains` check would have PASSED — the string is still there, in a comment) | -| I′ | `TestMainStillWiresTheHeavyOperationGateAndPerRunSpec` | R-85's gate + per-run spec survive | — | pass | +| A | `TestMerge_ProofSurvivesARestart` | an empty in-memory store + a persisted proof → the proof is reported, with its archive and its original time | the persisted merge deleted (the pre-R-189 body) | **FAIL** — `after a restart the persisted proof must be reported; got 0 entr(ies): []` — the live observation exactly | +| B | `TestMerge_NeverInventsAPassForAnUnprovenTier` | no proof → no entry; a tier-less record → no entry | — (its state-layer twin below carries the mutation) | pass | +| B′ | `TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe` | v1 + v2 + v3 records side by side → only the describable one is reported | the `reportable()` filter dropped | **FAIL** — `got 3` entries, two with an empty `SourceTier`/`SourceArchive` | +| C | `TestMerge_NewerWinsAndNeverDuplicatesATier` | one entry per tier, newest wins, in both directions | de-duplication removed | **FAIL** — `one entry per tier; got 2 for "pbs" — the hub would read two tests` | +| D | `TestMerge_AFailureIsStillReported` | a fresh failure beats an older stored success | (same mutation) | **FAIL** — 2 entries, i.e. the failure no longer the single answer for that tier | +| — | `TestMerge_MalformedTimestampNeverWins` | unparseable ≠ newest | — | pass | +| — | `TestMerge_NilProvenSourceIsANoOp` | pre-R-189 behaviour unchanged when unwired | — | pass | +| — | `TestScheduler_ProofIsRecordedReportably` | a pass **through the scheduler** leaves a reportable proof | — | pass | +| — | `TestScheduler_AFailureLeavesNoPersistedProof` | §8.1's asymmetry, asserted not assumed | — | pass | +| G | the gate's converse assertion | a published version with no tag fails | probe set → `[]` | **FAIL** (green over a live orphan) | +| I | `TestMainWiresTheDurableRestoreTestProof` | **AST**: `SetProvenRestoreTests` is called **and fed `rtState`** | the call commented out | **FAIL** — `main.go never calls collector.SetProvenRestoreTests` (a `strings.Contains` check would have passed — the string is still there) | +| H | reproducibility | three identical sha256 | — (measurement, §2) | pass | -Hub-side (Scenario G) is in `felhom.eu/REPORT.md`, including **a hollow test caught by its own -red-proof**: the first weekly fixture had no jitter, sat on exactly 168 h, and PASSED under the -flat-window mutation. +**Jitter, per §10:** every timestamp fixture uses odd minutes and seconds (`13:25:14`, `19:55:41`, +`13:41:07`, `04:41:58`) — several taken from the real box — rather than round hours. Yesterday a test +was hollow because a perfectly regular series landed exactly on a threshold and survived its own +mutation. -### Tests deliberately changed, and why +## 8. Files changed -`TestRotation_BothTiersExercisedAcrossCadences` asserted *4 ticks → 4 runs*. That was a faithful -statement of the defect — every tick produced a heavy restore-test, because the ticker **was** the -trigger. It is now `TestRotation_BothTiersExercisedOncePerArchive`: **2 runs across 4 evaluations**, -one per tier, one per archive. Strictly stronger — it pins both the coverage R-85 won and the pacing -R-86 adds. The old assertion is quoted in the test's comment so the change is legible. +`internal/backup/restoretest_state.go` (v3 record + `ProvenRestoreTests`), `internal/backup/store.go` +(the comment that had become false), `internal/backup/schedule.go` (record tier + verified), +`internal/hub/collect.go` (the seam + the merge), `cmd/felhom-agent/main.go` (wiring), +`scripts/release-agent.sh` (ordering, reproducible build, loud half-done release), +`scripts/publish-agent.sh` (identical build flags), `scripts/check-published-versions.py` (the +converse invariant), plus `REUSE.md`, `CHANGELOG.md`, `CONTEXT.md`, `CLAUDE.md` and three test files. -## 7. The live run — triggered by due-ness, on real hardware +**Commits** — `felhom-agent`: `7581f81` (v0.122.0). `felhom.eu`: see §10. -Deployed to **demo-felhom** (Tier 0). The deployed binary is the **published artifact downloaded from -Gitea**, not a local rebuild — see R-186. +## 9. The independent-verification command (Part 3, recorded in `CLAUDE.md`) -### 7.1 The due verdict, per tier, before anything ran - -``` -eval_interval=6h0m0s settle=24h0m0s -tier=felhom-backup due=false archive="" landed=- proven="" - reason: no settled archive yet — nothing to prove (newborn or still settling) -tier=felhom-pbs due=true archive="felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" - landed=2026-07-28T04:49:43Z proven="" - reason: newest settled archive (landed 2026-07-28T04:49:43Z) has not been proven; nothing proven yet +```bash +V=0.122.0 +git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \ + -o /tmp/felhom-agent-check ./cmd/felhom-agent +sha256sum /tmp/felhom-agent-check +curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum ``` -`felhom-backup` reads "no settled archive" for a reason that is **not** the one it appears to be — -see **R-185**: the agent cannot list that storage at all. +Both print `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` (§2). -### 7.2 A real run, started by the due-check +## 10. Registers -Only the **evaluation interval** was shortened for the validation (a systemd drop-in, since removed): -the due rule, the settle lag and the restore-test itself were untouched. +- **R-189 → CLOSED** (shipped + proven live), **R-188 → CLOSED** (shipped), **R-186 → CLOSED** + (shipped + measured). +- **R-185 remains OPEN and untouched** — it is a missing `/storage/felhom-backup` ACL on demo-felhom, + a permission defect, not a reporting one. Nothing in this session changed it, and the priority list + says so explicitly. +- No new IDs minted. `ROADMAP.md` contains none of these three rows, so there was nothing to collapse. +- `00-capability-map.md`'s restore-proof row now records that the evidence path itself had a gap and + what closed it; `CONTEXT.md` gains **S-19** (the proof/failure asymmetry and the merge rule) and + **S-20** (the release ordering and what each step protects); `STATUS.md` rewritten for the operator + and trimmed to 83 lines. -``` -15:14:38 backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers) - target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z - landed=2026-07-28T04:49:43Z - reason="newest settled archive … has not been proven; nothing proven on this tier yet" -15:14:39 restore-test: full-fidelity restore params derived from the archive config scratch=990000 - … proxmox-backup-client restore --crypt-mode=encrypt … (felhom-agent@pve!agent) -15:25:08 audit: gate decision class=guest_destroy guest=990000 source=one_shot_job allowed=true -15:25:14 restore-test: scratch guest torn down vmid=990000 -15:25:14 backup: scheduled restore-test PASSED archive=felhom-pbs:… duration_s=635.1 -``` +## 11. Observations — noticed, recorded, NOT acted on -A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored into a scratch guest, -booted, verified and destroyed — **635 s**, unattended, and started by *"this archive has not been -proven"* rather than by a timer. - -### 7.3 The state now names that archive, and a second evaluation runs nothing - -```json -{ "felhom-pbs": { "archive": "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", - "proven_at": "2026-08-03T13:25:14Z" } } -``` - -``` -tier=felhom-pbs due=false proven="felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" - reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven -``` - -### 7.4 Teardown — all three layers - -| Layer | Before | After | -|---|---|---| -| scratch guest 990000 | `stopped lock=create` during the run | **absent** from `pct list` | -| its volumes | 5 thin LVs (32 G + 200 G + 50 G + 2×1 G) | **0** matches in `lvs` | -| hub-side record | — | the run's **`restore_tests[]` entry is RETAINED deliberately** — it is the proof the hub's staleness check reads, and deleting it would delete the result. No event was created: the run passed, and `restore_test_failed`/`restore_test_stale` fire only on failure or staleness | - -`pvesm status` before and after: `local-lvm` 1.95 % used before the run, and the thin volumes are gone -after it — the restore reclaimed to the same shape it started in. - -### 7.5 The restart, which is the defect a person would actually notice - -Under the old scheduler every agent deploy restarted the ticker, so a restore-test ran one interval -after each deploy regardless of what had been proven. The proof of the fix cannot be *"nothing -appeared in the log"* — that is the absent-line trap this project has a standing rule about — so -v0.121.1 makes a quiet evaluation say what it decided, and the evaluation interval was shortened to -2 min for the validation so evaluations are **observable**, not assumed: - -``` -15:32:10 felhom-agent daemon starting version=0.121.1 ← the restart -15:32:11 backup: restore-test scheduler starting (per-archive due-check) eval_interval=2m0s settle=24h0m0s -15:34:12 backup: restore-test evaluated — nothing due - verdicts="felhom-backup: no settled archive yet — nothing to prove (newborn or still settling); - felhom-pbs: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven" -15:36:12 backup: restore-test evaluated — nothing due (same verdicts) - -runs since the restart: 0 -``` - -Evaluations demonstrably **happened** and demonstrably **decided**; nothing ran. The 6 h default was -restored afterwards (§8). - - -## 8. Release and deployment - -| Step | Evidence | -|---|---| -| Released via `scripts/release-agent.sh 0.121.0` | tag `v0.121.0` at `4d82591`, package published | -| Verified by **independent download** | sha256 `b2128f3cd4539225a2842f541f56ffaf5390b1d97f3f3a80076ec5f53dbc7d7a`, 14 081 336 B, round-trip GET matched | -| Gate re-run after release | `2 released version(s) to verify: 0.120.0, 0.121.0` → both **installable** | -| Deployed | `felhom-agent --version` → **0.121.0**, `systemctl is-active` → **active**, prior binary kept as `.bak-0.120.0` | -| Startup | `capabilities self-check ok=68 total=68 degraded=0`, and `backup: restore-test scheduler starting (per-archive due-check) eval_interval=6h0m0s settle=24h0m0s` | -| Second release, same path | **v0.121.1** — tag `v0.121.1`, sha256 `afaeeb509d1ed70d6e6bebac0393a3cd5be59d51e3db9ff96ef8524bd78546d7`, round-trip verified | -| Deployed (published bytes again) | `felhom-agent --version` → **0.121.1**, `active`; prior kept as `.bak-0.121.0` | -| Validation config removed | the 2-min drop-in deleted; the daemon back on **`eval_interval=6h0m0s settle=24h0m0s`** | -| Fleet | demo-hp still runs **0.120.0** — deliberate: pointing machines at a version is what **vouching** does | -| **Vouching** | **NOT done — deliberately the operator's act.** Hub UI → Configs → Day-0 artifacts: agent **`0.121.1`**, sha `afaeeb50…` (0.121.0 also published, sha `b2128f3c…`) | -| Config compatibility checked on both boxes | `restore_test_cadence_seconds = 0` on demo-felhom **and** demo-hp, and the installer writes `0` — so no box is on the deprecation path, and a fresh install gets the new defaults with no installer change | - -## 9. Findings filed (none fixed blind) - -- **R-185 — the agent cannot see demo-felhom's host backup tier at all.** The PVE token has no ACL on - `/storage/felhom-backup`, so the content listing returns `{"data":[]}` where root sees three - archives (6.1–6.3 GB, 08-01/02/03). Verified three ways, including `local` — which *has* a grant — - returning its archives through the same token. **Pre-existing and independent of R-86** (R-85's - rotation had the same blindness). The part worth fixing is the **silence**: a permission-blinded - tier is today indistinguishable from a newborn one, and the agent already records the backups it - wrote to that target, so the contradiction is detectable. -- **R-186 — a released binary's sha cannot be reproduced from its tag.** `release-agent.sh` builds - before tagging, so Go stamps a pseudo-version into the published bytes: published `b2128f3c…` - (14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source and toolchain. The - build order is deliberate, so the fix is not to swap the steps blind. **Mitigated here** by - deploying the published artifact. -- **R-187 — R-115's one-command release had never run its publish leg** (`CLOSED`, fixed in the same - session): `publish-agent.sh` has been mode `0644` since 2026-06-28 because every earlier caller used - `bash …`, and `release-agent.sh` called it directly → `Permission denied` on the first real release. - Fixed both ways: the mode bit restored **and** the call made mode-independent. The tag the failed - run created was withdrawn (nothing had been published under it — verified 404) and recreated on the - fix commit, so one version name still means one binary. - -- **R-188 — a correct agent release emails a CI failure about half the time.** `on: [push]` fires the - gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag before publishing - (deliberately). CI can therefore run the published-versions gate inside the window where the tag - exists and the package does not, and correctly report *"every released agent version must be - INSTALLABLE"* for a release that completes seconds later. **Measured across two releases in one - session:** v0.121.0 → runs #12 success / #13 failure on the same sha; v0.121.1 → #17 failure / #18 - success on the same sha; and one pair both green — a race, not a rule. It matters because R-168 - made CI email on failure so a red gate cannot be missed; a signal that cries wolf on every second - correct release is how that mail becomes something you archive unread. -- **R-189 — a passing restore-test can be invisible to the hub, and R-86 widened that window.** - Observed live: **the 15:25:14 PASS reached no host-report at all**. `restore_tests[]` comes from an - **in-memory** store (*"lost on restart; the cadence re-populates"*) and the report interval is - 900 s; the agent was restarted 2 m 43 s after the run for the v0.121.1 deploy. That used to - self-heal within 24 h because the next cadence re-tested the tier — **under per-archive due-ness - the agent will not re-test a proven archive**, so the hub can stay ignorant until the next archive - generation. The persisted proof already exists: `RestoreTestState.Snapshot()` is documented *"for - the host-report gauge"* and has **no production caller** — a seam built and never wired, and an - invariant asserted only in a comment, in one method. Bounded, not over-ranked: the hub scans its - retained window and the offsite tier's window (12 d) is wider than its archive rhythm (7 d), so one - lost report is tolerated. Filed, not fixed — it is a report-contract change. - -## 10. CI - -| Repo | Run | Commit | Result | -|---|---|---|---| -| `felhom-agent` | **#15** (id 83) | `4d82591` | **success** | -| `felhom-agent` | #13 (id 81) | `4618169` | **failure — explained, and it is CI doing its job** | -| `felhom.eu` | **#48** (id 86) | `ff2655c` | **success** | - -Run #13 fired on the **tag push** from the *failed* first release: `v0.121.0` existed as a tag while -nothing was published, and `check-published-versions.py` correctly refused — *"every released agent -version must be INSTALLABLE"*. That is precisely the state R-115's gate exists to catch, caught within -minutes and self-resolved by the corrected release. Confirmed locally afterwards: both 0.120.0 and -0.121.0 verify. `--no-verify` was **not** used anywhere. - -## 11. Observations — noticed, recorded, not acted on - -- **`felhom.eu/CONTEXT.md` has duplicate standing-ruling IDs** — three `S-14`s and two `S-15`s already - in the file before this session. New rulings were numbered **S-17/S-18** rather than adding to the - collision; the existing duplicates are untouched. -- **`agent_gates.py --fast` skips the published-versions gate**, so the pre-push hook cannot catch an - unpublished release — only CI can. That is the intended split (no network in a hook), and it is why - run #13 mattered. -- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for this check. Not - changed here (the read window is the same as before), but it is the cost centre if the fleet grows. +- **The proof state holds ONE record per tier, so proving an OLDER archive re-arms a newer one — + CONFIRMED after the validation, not merely predicted.** With the defaults restored, the due-check + reads: `tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z" proven="…2026-07-27T19:55:41Z" — + newest settled archive has not been proven (last proven archive was a different one)`. + Surfaced by this session's own validation method: to get a fresh proof without waiting a week, the + settle lag was widened so the older, unproven offsite archive became the candidate. That overwrote + the record for the newer archive, so once the default 24 h settle returns, the newest settled + archive is no longer the recorded proof and the tier becomes due once more. **Consequence, stated + rather than left to surprise: demo-felhom will run one further unattended offsite restore-test + within 6 h, after which the newest archive is the recorded proof** — the correct steady state. In + normal operation this cannot arise, because the candidate only ever moves forward. +- **`RestoreTestState.Snapshot()` has no caller again.** The host report is now fed by + `ProvenRestoreTests`, which carries what a bare timestamp cannot. The method's doc comment says in + as many words that it should be deleted if it does not acquire one — deliberately not deleted in + this session, because removing an exported method is a change with no bearing on the three rows. +- **Ten files in this repo are not `gofmt`-clean and were already so on arrival** + (`internal/capability/probe.go`, `internal/escrow/consume.go`, `internal/mgmtplane/mgmtplane.go`, + `internal/reconcile/bringup.go`, `internal/signedjobs/runner.go`, `internal/storage/{candidates, + intent}.go` and three test files). Every file this session touched is clean; the others are + untouched, and no gate checks formatting. +- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for the restore-test + staleness check. Unchanged here and not a defect at this fleet size; it is the cost centre if the + fleet grows, and it is the reason the window read was deliberately left at 14 days yesterday. +- **`felhom.eu/CONTEXT.md` still carries duplicate standing-ruling IDs** (three `S-14`s, two `S-15`s) + from before yesterday. New rulings continue to be numbered above the collision (S-19, S-20) rather + than adding to it; renumbering the existing ones is a separate, purely editorial change.