From 0b28eae7bb14f49323a461dbed54ac8507a1216c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 3 Aug 2026 16:58:35 +0200 Subject: [PATCH] =?UTF-8?q?REPORT:=20R-189/R-188/R-186=20=E2=80=94=20live?= =?UTF-8?q?=20evidence,=20the=20three=20sha=20values,=20and=20the=20observ?= =?UTF-8?q?ations?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scenario A proven on demo-felhom against the exact observation that filed R-189: a 675 s offsite restore-test passed, the agent was restarted 11 seconds later (inside the reporting window), and the hub's very next report carried '1 restore-tests' where the same sequence produced 0 this morning. The hub's own database holds the archive, the tier, the pass and the ORIGINAL test time, with the run mechanics deliberately zero. Also records the property the validation surfaced: the state holds one proof per tier, so proving an older archive re-arms a newer one — confirmed live after the defaults were restored. --- REPORT.md | 508 +++++++++++++++++++++++++----------------------------- 1 file changed, 236 insertions(+), 272 deletions(-) diff --git a/REPORT.md b/REPORT.md index 5e8e9cb..304c4b2 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,9 +1,9 @@ -# REPORT — R-86: a restore-test proves each BACKUP, not the clock +# REPORT — R-189 · R-188 · R-186: three ways the signals lied about themselves -**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.120.0 → v0.121.0 → v0.121.1** -(`4618169`, `4d82591`, `53d0c6b`) · -released, published, verified by independent download, deployed to demo-felhom and **proven live**. -Sibling half: `felhom.eu` hub **v0.91.0 → v0.91.1** — the two ship together. +**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.121.1 → v0.122.0** (`7581f81`) · released, +published, verified by an independent download **and by rebuilding it**, deployed to demo-felhom. +`felhom.eu`: register + docs only, **no hub change and no hub bump** — the hub already reads +`restore_tests[]`; the defect was that the agent stopped sending them. --- @@ -11,307 +11,271 @@ Sibling half: `felhom.eu` hub **v0.91.0 → v0.91.1** — the two ship together. | Repo | `main` @ commit | Version | Matched §1? | |---|---|---|---| -| `felhom-agent` | `1b14cfd0b48b` | `v0.120.0` | **yes** | -| `felhom.eu` | `e34b614e5b65` | CHANGELOG `v0.90.0`, deployed image `0.90.1` | **yes — the discrepancy was real and is fixed** (entry backfilled) | +| `felhom-agent` | `3d0a1d615d11` | `v0.121.1` | **yes** | +| `felhom.eu` | `c9a3e48b2106` | hub `v0.91.1` | **yes** | -Highest register ID in use was **R-184**; `R-185`–`R-187` established free by grep across all four -repos and `documentation/`. +Highest register ID in use was **R-189**; no new IDs were needed — all three rows already existed. +(Grep confirmed R-190+ free, in case one had been.) -## 2. The rule, in one sentence — and the trap it avoids +## 2. Scenario H — the reproducibility measurement (Part 3, done first on purpose) -> Let **A** be the newest archive on a tier that has settled for at least the settle lag (24 h). -> The tier is **DUE** when A exists and **A has not already been proven**. +**Before**, one commit, same source, same toolchain, same ldflags — the only difference is whether the +tag existed when the build ran: -The literal reading of R-86's own wording — *"due when the newest archive is ≥ 24 h old"* — is -**never true on a daily tier**, because a new archive lands each day and resets the newest-archive age -to zero long before it reaches the lag. It would have switched restore-testing **off** for the tier -that matters most, silently, while looking like the row was implemented. +| build | sha256 | size | embedded module version | +|---|---|---|---| +| default flags, **no tag yet** | `18f4a495…` | 14 085 464 B | `v0.121.2-0.20260803133646-3d0a1d61` | +| default flags, **tagged** | `4a38f394…` | 14 085 440 B | `v0.121.99` | +| `-trimpath -buildvcs=false`, either way | `7ffcdf1d…` | 14 064 574 B | *(none)* | -**Evidence that a daily tier does become due**, at three levels: +**After, on the real release (v0.122.0) — the three values §15.2 asks for:** -1. **Unit, time-driven** — `TestDue_DailyTierIsProvedDailyOnItsOwnArchive`: five simulated days, one - archive a day, evaluated hourly (120 evaluations) → **exactly 5 runs**, and run *i* tests day - *i−1*'s archive, never the still-settling one. -2. **The red-proof of the naive rule** — implemented and observed failing at **0 runs over 5 days** - (§6), which is the trap made visible rather than argued about. -3. **Live** — the offsite tier on demo-felhom became due on its own archive and ran (§7). - -## 3. What changed - -| Piece | File | Change | +| artifact | sha256 | size | |---|---|---| -| the due-check | `internal/backup/restoretest_due.go` (new) | `EvaluateDue` / `evaluateTier` — per-tier verdict + the reason, ordered oldest-proven first | -| the trigger | `internal/backup/schedule.go` | the ticker is now the **evaluation interval**; `pickForThisRun` answers *"is anything due?"*, and "nothing" is a normal answer | -| the state | `internal/backup/restoretest_state.go` | records **which archive** was proven, with migration | -| the picker | `internal/backup/runner.go` | `PickSettledRestoreCandidateOn(ctx, target, notAfter)`; `PickRestoreCandidateOn` is a one-line call into it | -| the knobs | `internal/config/config.go` | `restore_test_eval_interval_seconds` + `restore_test_settle_seconds`; the old key deprecated, not repurposed | -| the wiring | `cmd/felhom-agent/main.go` | settle-aware picker + `Settle`; deprecation WARN; new `--selftest=restore-test-due` | -| the observable (**v0.121.1**) | `internal/backup/schedule.go`, `restoretest_due.go` | a not-due evaluation logs one **INFO** line naming every tier's verdict — see §9 | +| published, downloaded from Gitea | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B | +| rebuild at the tag, #1 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B | +| rebuild at the tag, #2 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B | -### The state records the archive (§8.2) +**All three identical.** The property is removed-cause, not sequenced-around: `-buildvcs=false` drops +a stamp nothing reads (no `ReadBuildInfo` caller, verified by grep), and the version still comes from +the explicit `-X main.version` ldflag. `-trimpath` additionally makes a rebuild from a different +checkout directory match. -A timestamp cannot answer *"have we proven **this** archive"* — it is the same class as the workspace -rule that a timestamp records an *attempt*, not a *result*: here it records a result, but not **which** -result. `RestoreTestState` now holds `{archive, proven_at}` per tier. +**A second discrepancy fell out of the measurement and is fixed with it:** `publish-agent.sh`'s +fallback build forced `CGO_ENABLED=0` and produced **13 990 236 B** against the release path's +**14 064 574 B** — a 74 KB difference, i.e. one version name meaning two binaries depending on which +entry point ran. Both paths now use identical flags, each commented with a pointer to the other. -**Migration:** a pre-R-86 file (`{"target": ""}`) keeps its **time** — rotation ordering -survives a deploy, which is why the file exists at all — and yields **no proven archive**, so each -tier is due exactly once after the upgrade. One extra test per tier, once, is the safe direction; -reading a legacy time as proof of whatever archive is current would invent a guarantee. +## 3. Scenario E — a correct release no longer emails a failure (Part 2) -### The config (§8.3) — and a correction to the spec +**Only the tag PUSH moved.** The order is now build → tag **locally** → publish → push tag. The tag is +still created before anything is published, so the build and the tag describe the same commit; it +becomes *visible* — to CI (`on: [push]`) and to any `raw/tag/…` fetch — only once the package is +downloadable. -The spec said *"`0` must keep meaning disabled"*. **In the code as it stands, `0` means *use the -default* and NEGATIVE means disabled** (`RestoreTestCadence`, pre-existing). Making `0` disable would -have switched restore-testing off on every box that leaves the key unset — the worst possible reading -— so the actual semantics were preserved and this is flagged rather than silently followed. +**The old ordering's invariant is asserted directly rather than arranged for.** +`check-published-versions.py` now carries two invariants: every tag has an installable package (as +before) **and no published version is missing its tag** (new). The second is a **bounded probe** — +the frontier past the newest tag, where a failed tag push leaves an orphan, plus patch gaps — and it +**prints its probe set on every run**, because a check whose coverage is invisible reads as a +guarantee it is not making. The package listing api was **re-measured**, not assumed: `401` without a +token, so absence still cannot be enumerated, and the script says so in its own output. -- `restore_test_eval_interval_seconds` — how often due-ness is **asked**. Default **6 h**. -- `restore_test_settle_seconds` — how long an archive must sit. Default **24 h**. -- `restore_test_cadence_seconds` — **deprecated**. Negative still **disables**, verbatim. A positive - value seeds the **settle lag** (the quantity a person setting it was expressing: how long may pass - between a backup and confidence that it restores), and the daemon logs one start-up WARN naming - both replacements. It is deliberately **not** carried into the evaluation interval: a box that set - 72 h to spare a weak endpoint would otherwise get a 72-hour-latency due-check, whereas what it - wanted — fewer heavy restores — is what per-archive due-ness already gives it. +**Live result — this release is the test:** -## 4. Part 1.4 — the measurement, and the interval chosen from it - -Measured on demo-felhom, 2026-08-03, via `--selftest=restore-test-due` and by timing the underlying -API call directly (3 runs each): - -| tier | what it is | one due-check | +| release | CI runs on the release commit | outcome | |---|---|---| -| `felhom-backup` (dir) | local, on-box | **18 ms** (18.7 / 18.3 / 18.5) | -| `felhom-pbs` | offsite, **WAN to ep0** | **392 ms** (375 / 378 / 424) | -| both together | one full evaluation | **430 ms** | +| v0.121.0 (yesterday) | #12 / #13 | one **green**, one **red** | +| v0.121.1 (yesterday) | #17 / #18 | one **red**, one **green** | +| **v0.122.0 (this one)** | **#21 (task id 96) / #22 (task id 97)** | **both green** | -**Cost does not set the interval** — even at one evaluation a minute the offsite leg would be ~0.7 % -of the link's time. What sets it is the other bound, and it is not in the brief: **under a per-archive -due-check a FAILING tier stays due, so the evaluation interval is also its RETRY interval — and a -retry is a multi-GB restore.** Every few minutes would be an incident of its own; the old timer -retried a broken tier once a day. +**Failure modes made loud rather than tidy:** a publish that succeeds followed by a tag push that +fails now dies printing `git push origin v` — the local tag is already there, so recovery is one +line — and a publish that *fails* deletes the local-only tag so a retry is clean instead of colliding +with step 2's re-release guard. -**6 h chosen from both ends:** at most four heavy retries a day in the worst case, and at most 6 h of -latency between an archive settling and its proof — negligible against a 24 h settle lag, so a daily -tier is still proved daily. No second rate limiter was added (§8.4): the pacing remains one test per -archive generation. +## 4. Scenarios F and G — both directions demonstrated, then cleaned up -## 5. Two hazards the new frequency created, and their fixes +**G — a published version with no tag must FAIL.** A real fixture: version **0.121.2** (the frontier, +exactly where a failed tag push lands) published to the live registry with no tag. -Both are consequences of evaluating often rather than daily, and neither is in the brief: +``` +0.121.2 next patch after the newest tag PUBLISHED — NO TAG +check-published-versions: 1 PUBLISHED VERSION(S) WITH NO TAG + v0.121.2 is downloadable at … but has no git tag. + git push origin v0.121.2 +EXIT=1 +``` -1. **The due-check now runs BEFORE the heavy-operation gate is taken.** Holding that gate for a read - that answers "nothing to do" would open a window at *every* evaluation in which a starting backup - cannot acquire — and a backup that cannot acquire does not merely wait, it **records a failure and - pages the operator** (F-A1). Nothing heavy starts before the gate; due-ness does not expire while - we check. -2. **The candidate picker skips implausible archives.** Under per-archive due-ness an incomplete - 1-byte phantom (F-CRIT-2's artefact, which server-side prune does **not** collect) would be picked - forever, fail forever, never earn proof, and leave the tier due at *every* evaluation — turning the - evaluation interval into the retry rate for a multi-GB restore. `archivePlausiblyComplete` (the - canonical helper, with its warn-once companion) is applied in the shared scan, so both callers - agree. **This is a behaviour change to `PickRestoreCandidateOn`** and is recorded as such. +**Red-proof (observed):** with the fixture still live, replacing the probe set with `[]` produced +`ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED`, **exit 0** — a green run over a published +orphan. Restored. -## 6. Tests and red-proofs +**Teardown:** fixture deleted (HTTP 204), absence independently re-verified (`GET → 404`), gate green +again (exit 0). No scratch tag was ever pushed. -Green gate, both repos: `go build ./... && go vet ./... && go test ./...` — agent **29 packages ok, -rc=0**; hub **rc=0**. The test run and the commit were always separate commands. +**F — a tag with no package must still FAIL.** Demonstrated against a local stand-in serving two tags +where only one has a package: + +``` + ok v0.120.0: binary downloadable + tag serves its configs + FAIL v0.199.0: + - binary NOT downloadable (HTTP 404 …) + - tag does not serve configs/felhom-agent.service (HTTP 404) — a box would 404 mid-install +EXIT=1 +``` + +**Why not a real pushed tag:** pushing one wakes CI and would have emailed the operator a **true** +alarm about a fixture — the same attention cost R-188 exists to remove. F is also already +demonstrated in the wild: CI runs **#13** and **#17** failed for exactly this reason yesterday. + +## 5. R-189 — a proof that survives a restart reaches the hub (Part 1) + +`restore_tests[]` came only from the in-memory `backup.Store`, whose comment read *"lost on restart; +the cadence re-populates"*. True under a timer; false since R-86, because the agent refuses to re-test +an archive it has already proven — so a lost proof is not repeated for a whole archive generation. + +**What changed** + +- `RestoreTestState` stores the **tier** and what was **verified** beside the archive (v3 shape). + Both are recorded **at proof time from the run's own result** — deriving them later would need a + storage-type lookup at report-building time, a network call that can fail on the one path where + failing means mislabelling a proof. +- `ProvenRestoreTests` renders the stored proofs as report entries; `Collector.SetProvenRestoreTests` + merges them with the in-memory result. +- **Merge rule: one entry per tier, newest by `TestedAt` wins.** It falls out of what each source + means rather than from a preference: a fresh failure beats a stored success (the failure is the + news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a + tier never appears twice — the hub would read that as two tests. An unparseable timestamp counts as + **older**, so a malformed entry cannot displace a good one. +- **It refuses to lie.** A record missing the archive **or** the tier produces no entry, and run + mechanics (scratch VMID, duration) are not re-invented: an absent duration is not a claim, a + fabricated one would be. +- **The asymmetry is now in the code** (§8.1): a success *suppresses* future work so it must be + durable; a failure *causes* future work and heals itself, and persisting one would make a healed + tier keep reporting a fault. +- **`Store`'s comment is corrected in place** — leaving it is how the next reader concludes this is + handled. + +**Migration, and it is visible on the live box:** a pre-R-189 record has an archive but no tier, so it +is **not** reportable. Upgrading does not retroactively make an old proof visible; the tier's next +real proof fills it in. Confirmed immediately after the deploy — still `0 restore-tests`, with the +v2 record sitting on disk. + +## 6. Scenario A, live on demo-felhom — against the observation that filed R-189 + +**The observation being replaced (2026-08-03, 15:25):** a real offsite restore-test PASSED, the agent +was restarted 2 m 43 s later, and the hub logged `0 restore-tests` on the next two host-reports. + +**The same sequence, on v0.122.0:** + +``` +16:44:06 restore-test tier is DUE target=felhom-pbs + archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z + reason="newest settled archive … has not been proven (last proven archive was a different one)" +16:55:21 restore-test: scratch guest torn down vmid=990000 +16:55:21 backup: scheduled restore-test PASSED archive=felhom-pbs:…2026-07-27T19:55:41Z duration_s=675.1 +16:55:32 systemctl restart felhom-agent ← INSIDE the 15-minute reporting window +16:55:36 hub: host-report from demo-felhom-8363b5 (… 1 restore-tests …) ← was 0 +``` + +**The proof on disk (v3 — the tier is what the old shape lacked):** + +```json +{"felhom-pbs": {"archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z", + "tier": "pbs", "verified": "boot+running", "proven_at": "2026-08-03T14:55:21Z"}} +``` + +**What the HUB stored** — read from its own database (copied with its `-wal`, freshness confirmed by +the newest row's `received_at` = `2026-08-03 14:55:36` UTC, matching the ingest line): + +```json +{ "source_archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z", + "source_tier": "pbs", "pass": true, "verified": "boot+running", + "tested_at": "2026-08-03T14:55:21Z", "scratch_vmid": 0, "duration_seconds": 0 } +``` + +That report was built **after** the restart, when the in-memory store was empty — so the entry can +only have come from the persisted state. The archive, the tier, and the **original** test time +survived; the run mechanics are zero because they are deliberately not re-invented. + +**A 14.5 GB encrypted offsite archive**, restored, booted, verified and destroyed in **675 s** — and +this time the proof outlived the process that produced it. + +**Teardown, all three layers:** scratch guest absent from `pct list` (0), its volumes gone from `lvs` +(0), the validation drop-in removed and the daemon back on its defaults +(`eval_interval=6h0m0s settle=24h0m0s`). The hub-side `restore_tests[]` record is **retained +deliberately** — it is the proof the staleness check reads, so deleting it would delete the result. +No `restore_test_*` event was raised, because nothing failed and nothing is stale. + + +## 7. Tests and red-proofs + +Green gate: `go build ./... && go vet ./... && go test ./...` — **29 packages ok, rc=0**, plus +`python3 scripts/agent_gates.py` (reuse-refs + published-versions) all OK. The test run and the commit +were always separate commands. | # | Test | Asserts | Mutation | Observed | |---|---|---|---|---| -| A | `TestDue_DailyTierIsProvedDailyOnItsOwnArchive` | 5 runs over 5 days, each on the settled archive | the naive rule (`now-landed >= settle`, proven-archive check deleted, cutoff removed) | **FAIL** — `a daily tier must be proved once per day; got 0 run(s) over 5 days: []` | -| B | `TestDue_WeeklyTierIsProvedOncePerArchive` | 84 evaluations over 3 weeks → exactly 3 runs, one per archive | — | pass | -| C | `TestDue_RestartRunsNothing` | two restarts + 4 evaluations → **0 runs** | `ProvenArchive` reverted to per-tier time | **FAIL** — `2 restart(s) produced 4 run(s)` | -| D | `TestDue_NewSettledArchiveMakesAProvedTierDueAgain` | a newly settled archive re-arms the tier, and the NEW archive is tested | — | pass | -| E | `TestDue_FailingTierIsRetriedAndNeverProven` | 3 evaluations → 3 retries, no proof recorded | credit on failure (`rt.Pass &&` dropped) | **FAIL** — `got 1 run(s) over 3 evaluations` + `TestRotation_FailureEarnsNoCredit` also failed | -| F | `TestDue_TwoDueTiersRunOneAtATime` | one evaluation → one run; the other is deferred and runs next | — | pass | -| F′ | `TestDue_DeferredBehindABackupStaysDue` | the gate holds; a deferred tier stays DUE | — | pass | -| H | `TestDue_NewbornTierIsNotDueAndNotAnError` | no archive → not due, no error, **and a reason** | — | pass | -| — | `TestDue_UnsettledArchiveIsNotACandidate` | a 2 h-old archive is not a candidate under a 24 h lag | — | pass | -| — | `TestDue_LookupFailureIsUnknownNotNotDue` | a tier that cannot be listed is UNKNOWN, the error travels, the other tier still runs | — | pass | -| — | `TestRestoreTestState_LegacyFileMigratesToNothingProven` | legacy time kept, no archive claimed | — | pass | -| — | `TestPickRestoreCandidate_SkipsImplausibleArchives` | the newest entry is not a candidate if it cannot be complete | guard removed | **FAIL** — `pick = "phantom" want the newest COMPLETE archive 'real'` | -| I | `TestMainWiresTheSettleAwareTierPicker` | **AST** of `main.go`: settle picker wired, old picker gone, `Settle` set, eval-interval accessor called | the wiring line commented out | **FAIL** — `main.go never passes runner.PickSettledRestoreCandidateOn …` (a `strings.Contains` check would have PASSED — the string is still there, in a comment) | -| I′ | `TestMainStillWiresTheHeavyOperationGateAndPerRunSpec` | R-85's gate + per-run spec survive | — | pass | +| A | `TestMerge_ProofSurvivesARestart` | an empty in-memory store + a persisted proof → the proof is reported, with its archive and its original time | the persisted merge deleted (the pre-R-189 body) | **FAIL** — `after a restart the persisted proof must be reported; got 0 entr(ies): []` — the live observation exactly | +| B | `TestMerge_NeverInventsAPassForAnUnprovenTier` | no proof → no entry; a tier-less record → no entry | — (its state-layer twin below carries the mutation) | pass | +| B′ | `TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe` | v1 + v2 + v3 records side by side → only the describable one is reported | the `reportable()` filter dropped | **FAIL** — `got 3` entries, two with an empty `SourceTier`/`SourceArchive` | +| C | `TestMerge_NewerWinsAndNeverDuplicatesATier` | one entry per tier, newest wins, in both directions | de-duplication removed | **FAIL** — `one entry per tier; got 2 for "pbs" — the hub would read two tests` | +| D | `TestMerge_AFailureIsStillReported` | a fresh failure beats an older stored success | (same mutation) | **FAIL** — 2 entries, i.e. the failure no longer the single answer for that tier | +| — | `TestMerge_MalformedTimestampNeverWins` | unparseable ≠ newest | — | pass | +| — | `TestMerge_NilProvenSourceIsANoOp` | pre-R-189 behaviour unchanged when unwired | — | pass | +| — | `TestScheduler_ProofIsRecordedReportably` | a pass **through the scheduler** leaves a reportable proof | — | pass | +| — | `TestScheduler_AFailureLeavesNoPersistedProof` | §8.1's asymmetry, asserted not assumed | — | pass | +| G | the gate's converse assertion | a published version with no tag fails | probe set → `[]` | **FAIL** (green over a live orphan) | +| I | `TestMainWiresTheDurableRestoreTestProof` | **AST**: `SetProvenRestoreTests` is called **and fed `rtState`** | the call commented out | **FAIL** — `main.go never calls collector.SetProvenRestoreTests` (a `strings.Contains` check would have passed — the string is still there) | +| H | reproducibility | three identical sha256 | — (measurement, §2) | pass | -Hub-side (Scenario G) is in `felhom.eu/REPORT.md`, including **a hollow test caught by its own -red-proof**: the first weekly fixture had no jitter, sat on exactly 168 h, and PASSED under the -flat-window mutation. +**Jitter, per §10:** every timestamp fixture uses odd minutes and seconds (`13:25:14`, `19:55:41`, +`13:41:07`, `04:41:58`) — several taken from the real box — rather than round hours. Yesterday a test +was hollow because a perfectly regular series landed exactly on a threshold and survived its own +mutation. -### Tests deliberately changed, and why +## 8. Files changed -`TestRotation_BothTiersExercisedAcrossCadences` asserted *4 ticks → 4 runs*. That was a faithful -statement of the defect — every tick produced a heavy restore-test, because the ticker **was** the -trigger. It is now `TestRotation_BothTiersExercisedOncePerArchive`: **2 runs across 4 evaluations**, -one per tier, one per archive. Strictly stronger — it pins both the coverage R-85 won and the pacing -R-86 adds. The old assertion is quoted in the test's comment so the change is legible. +`internal/backup/restoretest_state.go` (v3 record + `ProvenRestoreTests`), `internal/backup/store.go` +(the comment that had become false), `internal/backup/schedule.go` (record tier + verified), +`internal/hub/collect.go` (the seam + the merge), `cmd/felhom-agent/main.go` (wiring), +`scripts/release-agent.sh` (ordering, reproducible build, loud half-done release), +`scripts/publish-agent.sh` (identical build flags), `scripts/check-published-versions.py` (the +converse invariant), plus `REUSE.md`, `CHANGELOG.md`, `CONTEXT.md`, `CLAUDE.md` and three test files. -## 7. The live run — triggered by due-ness, on real hardware +**Commits** — `felhom-agent`: `7581f81` (v0.122.0). `felhom.eu`: see §10. -Deployed to **demo-felhom** (Tier 0). The deployed binary is the **published artifact downloaded from -Gitea**, not a local rebuild — see R-186. +## 9. The independent-verification command (Part 3, recorded in `CLAUDE.md`) -### 7.1 The due verdict, per tier, before anything ran - -``` -eval_interval=6h0m0s settle=24h0m0s -tier=felhom-backup due=false archive="" landed=- proven="" - reason: no settled archive yet — nothing to prove (newborn or still settling) -tier=felhom-pbs due=true archive="felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" - landed=2026-07-28T04:49:43Z proven="" - reason: newest settled archive (landed 2026-07-28T04:49:43Z) has not been proven; nothing proven yet +```bash +V=0.122.0 +git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \ + -o /tmp/felhom-agent-check ./cmd/felhom-agent +sha256sum /tmp/felhom-agent-check +curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum ``` -`felhom-backup` reads "no settled archive" for a reason that is **not** the one it appears to be — -see **R-185**: the agent cannot list that storage at all. +Both print `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` (§2). -### 7.2 A real run, started by the due-check +## 10. Registers -Only the **evaluation interval** was shortened for the validation (a systemd drop-in, since removed): -the due rule, the settle lag and the restore-test itself were untouched. +- **R-189 → CLOSED** (shipped + proven live), **R-188 → CLOSED** (shipped), **R-186 → CLOSED** + (shipped + measured). +- **R-185 remains OPEN and untouched** — it is a missing `/storage/felhom-backup` ACL on demo-felhom, + a permission defect, not a reporting one. Nothing in this session changed it, and the priority list + says so explicitly. +- No new IDs minted. `ROADMAP.md` contains none of these three rows, so there was nothing to collapse. +- `00-capability-map.md`'s restore-proof row now records that the evidence path itself had a gap and + what closed it; `CONTEXT.md` gains **S-19** (the proof/failure asymmetry and the merge rule) and + **S-20** (the release ordering and what each step protects); `STATUS.md` rewritten for the operator + and trimmed to 83 lines. -``` -15:14:38 backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers) - target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z - landed=2026-07-28T04:49:43Z - reason="newest settled archive … has not been proven; nothing proven on this tier yet" -15:14:39 restore-test: full-fidelity restore params derived from the archive config scratch=990000 - … proxmox-backup-client restore --crypt-mode=encrypt … (felhom-agent@pve!agent) -15:25:08 audit: gate decision class=guest_destroy guest=990000 source=one_shot_job allowed=true -15:25:14 restore-test: scratch guest torn down vmid=990000 -15:25:14 backup: scheduled restore-test PASSED archive=felhom-pbs:… duration_s=635.1 -``` +## 11. Observations — noticed, recorded, NOT acted on -A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored into a scratch guest, -booted, verified and destroyed — **635 s**, unattended, and started by *"this archive has not been -proven"* rather than by a timer. - -### 7.3 The state now names that archive, and a second evaluation runs nothing - -```json -{ "felhom-pbs": { "archive": "felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z", - "proven_at": "2026-08-03T13:25:14Z" } } -``` - -``` -tier=felhom-pbs due=false proven="felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z" - reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven -``` - -### 7.4 Teardown — all three layers - -| Layer | Before | After | -|---|---|---| -| scratch guest 990000 | `stopped lock=create` during the run | **absent** from `pct list` | -| its volumes | 5 thin LVs (32 G + 200 G + 50 G + 2×1 G) | **0** matches in `lvs` | -| hub-side record | — | the run's **`restore_tests[]` entry is RETAINED deliberately** — it is the proof the hub's staleness check reads, and deleting it would delete the result. No event was created: the run passed, and `restore_test_failed`/`restore_test_stale` fire only on failure or staleness | - -`pvesm status` before and after: `local-lvm` 1.95 % used before the run, and the thin volumes are gone -after it — the restore reclaimed to the same shape it started in. - -### 7.5 The restart, which is the defect a person would actually notice - -Under the old scheduler every agent deploy restarted the ticker, so a restore-test ran one interval -after each deploy regardless of what had been proven. The proof of the fix cannot be *"nothing -appeared in the log"* — that is the absent-line trap this project has a standing rule about — so -v0.121.1 makes a quiet evaluation say what it decided, and the evaluation interval was shortened to -2 min for the validation so evaluations are **observable**, not assumed: - -``` -15:32:10 felhom-agent daemon starting version=0.121.1 ← the restart -15:32:11 backup: restore-test scheduler starting (per-archive due-check) eval_interval=2m0s settle=24h0m0s -15:34:12 backup: restore-test evaluated — nothing due - verdicts="felhom-backup: no settled archive yet — nothing to prove (newborn or still settling); - felhom-pbs: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven" -15:36:12 backup: restore-test evaluated — nothing due (same verdicts) - -runs since the restart: 0 -``` - -Evaluations demonstrably **happened** and demonstrably **decided**; nothing ran. The 6 h default was -restored afterwards (§8). - - -## 8. Release and deployment - -| Step | Evidence | -|---|---| -| Released via `scripts/release-agent.sh 0.121.0` | tag `v0.121.0` at `4d82591`, package published | -| Verified by **independent download** | sha256 `b2128f3cd4539225a2842f541f56ffaf5390b1d97f3f3a80076ec5f53dbc7d7a`, 14 081 336 B, round-trip GET matched | -| Gate re-run after release | `2 released version(s) to verify: 0.120.0, 0.121.0` → both **installable** | -| Deployed | `felhom-agent --version` → **0.121.0**, `systemctl is-active` → **active**, prior binary kept as `.bak-0.120.0` | -| Startup | `capabilities self-check ok=68 total=68 degraded=0`, and `backup: restore-test scheduler starting (per-archive due-check) eval_interval=6h0m0s settle=24h0m0s` | -| Second release, same path | **v0.121.1** — tag `v0.121.1`, sha256 `afaeeb509d1ed70d6e6bebac0393a3cd5be59d51e3db9ff96ef8524bd78546d7`, round-trip verified | -| Deployed (published bytes again) | `felhom-agent --version` → **0.121.1**, `active`; prior kept as `.bak-0.121.0` | -| Validation config removed | the 2-min drop-in deleted; the daemon back on **`eval_interval=6h0m0s settle=24h0m0s`** | -| Fleet | demo-hp still runs **0.120.0** — deliberate: pointing machines at a version is what **vouching** does | -| **Vouching** | **NOT done — deliberately the operator's act.** Hub UI → Configs → Day-0 artifacts: agent **`0.121.1`**, sha `afaeeb50…` (0.121.0 also published, sha `b2128f3c…`) | -| Config compatibility checked on both boxes | `restore_test_cadence_seconds = 0` on demo-felhom **and** demo-hp, and the installer writes `0` — so no box is on the deprecation path, and a fresh install gets the new defaults with no installer change | - -## 9. Findings filed (none fixed blind) - -- **R-185 — the agent cannot see demo-felhom's host backup tier at all.** The PVE token has no ACL on - `/storage/felhom-backup`, so the content listing returns `{"data":[]}` where root sees three - archives (6.1–6.3 GB, 08-01/02/03). Verified three ways, including `local` — which *has* a grant — - returning its archives through the same token. **Pre-existing and independent of R-86** (R-85's - rotation had the same blindness). The part worth fixing is the **silence**: a permission-blinded - tier is today indistinguishable from a newborn one, and the agent already records the backups it - wrote to that target, so the contradiction is detectable. -- **R-186 — a released binary's sha cannot be reproduced from its tag.** `release-agent.sh` builds - before tagging, so Go stamps a pseudo-version into the published bytes: published `b2128f3c…` - (14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source and toolchain. The - build order is deliberate, so the fix is not to swap the steps blind. **Mitigated here** by - deploying the published artifact. -- **R-187 — R-115's one-command release had never run its publish leg** (`CLOSED`, fixed in the same - session): `publish-agent.sh` has been mode `0644` since 2026-06-28 because every earlier caller used - `bash …`, and `release-agent.sh` called it directly → `Permission denied` on the first real release. - Fixed both ways: the mode bit restored **and** the call made mode-independent. The tag the failed - run created was withdrawn (nothing had been published under it — verified 404) and recreated on the - fix commit, so one version name still means one binary. - -- **R-188 — a correct agent release emails a CI failure about half the time.** `on: [push]` fires the - gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag before publishing - (deliberately). CI can therefore run the published-versions gate inside the window where the tag - exists and the package does not, and correctly report *"every released agent version must be - INSTALLABLE"* for a release that completes seconds later. **Measured across two releases in one - session:** v0.121.0 → runs #12 success / #13 failure on the same sha; v0.121.1 → #17 failure / #18 - success on the same sha; and one pair both green — a race, not a rule. It matters because R-168 - made CI email on failure so a red gate cannot be missed; a signal that cries wolf on every second - correct release is how that mail becomes something you archive unread. -- **R-189 — a passing restore-test can be invisible to the hub, and R-86 widened that window.** - Observed live: **the 15:25:14 PASS reached no host-report at all**. `restore_tests[]` comes from an - **in-memory** store (*"lost on restart; the cadence re-populates"*) and the report interval is - 900 s; the agent was restarted 2 m 43 s after the run for the v0.121.1 deploy. That used to - self-heal within 24 h because the next cadence re-tested the tier — **under per-archive due-ness - the agent will not re-test a proven archive**, so the hub can stay ignorant until the next archive - generation. The persisted proof already exists: `RestoreTestState.Snapshot()` is documented *"for - the host-report gauge"* and has **no production caller** — a seam built and never wired, and an - invariant asserted only in a comment, in one method. Bounded, not over-ranked: the hub scans its - retained window and the offsite tier's window (12 d) is wider than its archive rhythm (7 d), so one - lost report is tolerated. Filed, not fixed — it is a report-contract change. - -## 10. CI - -| Repo | Run | Commit | Result | -|---|---|---|---| -| `felhom-agent` | **#15** (id 83) | `4d82591` | **success** | -| `felhom-agent` | #13 (id 81) | `4618169` | **failure — explained, and it is CI doing its job** | -| `felhom.eu` | **#48** (id 86) | `ff2655c` | **success** | - -Run #13 fired on the **tag push** from the *failed* first release: `v0.121.0` existed as a tag while -nothing was published, and `check-published-versions.py` correctly refused — *"every released agent -version must be INSTALLABLE"*. That is precisely the state R-115's gate exists to catch, caught within -minutes and self-resolved by the corrected release. Confirmed locally afterwards: both 0.120.0 and -0.121.0 verify. `--no-verify` was **not** used anywhere. - -## 11. Observations — noticed, recorded, not acted on - -- **`felhom.eu/CONTEXT.md` has duplicate standing-ruling IDs** — three `S-14`s and two `S-15`s already - in the file before this session. New rulings were numbered **S-17/S-18** rather than adding to the - collision; the existing duplicates are untouched. -- **`agent_gates.py --fast` skips the published-versions gate**, so the pre-push hook cannot catch an - unpublished release — only CI can. That is the intended split (no network in a hook), and it is why - run #13 mattered. -- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for this check. Not - changed here (the read window is the same as before), but it is the cost centre if the fleet grows. +- **The proof state holds ONE record per tier, so proving an OLDER archive re-arms a newer one — + CONFIRMED after the validation, not merely predicted.** With the defaults restored, the due-check + reads: `tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z" proven="…2026-07-27T19:55:41Z" — + newest settled archive has not been proven (last proven archive was a different one)`. + Surfaced by this session's own validation method: to get a fresh proof without waiting a week, the + settle lag was widened so the older, unproven offsite archive became the candidate. That overwrote + the record for the newer archive, so once the default 24 h settle returns, the newest settled + archive is no longer the recorded proof and the tier becomes due once more. **Consequence, stated + rather than left to surprise: demo-felhom will run one further unattended offsite restore-test + within 6 h, after which the newest archive is the recorded proof** — the correct steady state. In + normal operation this cannot arise, because the candidate only ever moves forward. +- **`RestoreTestState.Snapshot()` has no caller again.** The host report is now fed by + `ProvenRestoreTests`, which carries what a bare timestamp cannot. The method's doc comment says in + as many words that it should be deleted if it does not acquire one — deliberately not deleted in + this session, because removing an exported method is a change with no bearing on the three rows. +- **Ten files in this repo are not `gofmt`-clean and were already so on arrival** + (`internal/capability/probe.go`, `internal/escrow/consume.go`, `internal/mgmtplane/mgmtplane.go`, + `internal/reconcile/bringup.go`, `internal/signedjobs/runner.go`, `internal/storage/{candidates, + intent}.go` and three test files). Every file this session touched is clean; the others are + untouched, and no gate checks formatting. +- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for the restore-test + staleness check. Unchanged here and not a defect at this fleet size; it is the cost centre if the + fleet grows, and it is the reason the window read was deliberately left at 14 days yesterday. +- **`felhom.eu/CONTEXT.md` still carries duplicate standing-ruling IDs** (three `S-14`s, two `S-15`s) + from before yesterday. New rulings continue to be numbered above the collision (S-19, S-20) rather + than adding to it; renumbering the existing ones is a separate, purely editorial change.