REPORT: restart proof, R-188/R-189, and the restored defaults
gates / gates (push) Successful in 7s

This commit is contained in:
2026-08-03 15:36:46 +02:00
parent 77e2cc4583
commit 3d0a1d615d
+46 -1
View File
@@ -212,6 +212,25 @@ after it — the restore reclaimed to the same shape it started in.
### 7.5 The restart, which is the defect a person would actually notice
Under the old scheduler every agent deploy restarted the ticker, so a restore-test ran one interval
after each deploy regardless of what had been proven. The proof of the fix cannot be *"nothing
appeared in the log"* — that is the absent-line trap this project has a standing rule about — so
v0.121.1 makes a quiet evaluation say what it decided, and the evaluation interval was shortened to
2 min for the validation so evaluations are **observable**, not assumed:
```
15:32:10 felhom-agent daemon starting version=0.121.1 ← the restart
15:32:11 backup: restore-test scheduler starting (per-archive due-check) eval_interval=2m0s settle=24h0m0s
15:34:12 backup: restore-test evaluated — nothing due
verdicts="felhom-backup: no settled archive yet — nothing to prove (newborn or still settling);
felhom-pbs: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven"
15:36:12 backup: restore-test evaluated — nothing due (same verdicts)
runs since the restart: 0
```
Evaluations demonstrably **happened** and demonstrably **decided**; nothing ran. The 6 h default was
restored afterwards (§8).
## 8. Release and deployment
@@ -223,7 +242,12 @@ after it — the restore reclaimed to the same shape it started in.
| Gate re-run after release | `2 released version(s) to verify: 0.120.0, 0.121.0` → both **installable** |
| Deployed | `felhom-agent --version`**0.121.0**, `systemctl is-active`**active**, prior binary kept as `.bak-0.120.0` |
| Startup | `capabilities self-check ok=68 total=68 degraded=0`, and `backup: restore-test scheduler starting (per-archive due-check) eval_interval=6h0m0s settle=24h0m0s` |
| **Vouching** | **NOT done — deliberately the operator's act.** Hub UI → Configs → Day-0 artifacts: agent `0.121.0`, sha above |
| Second release, same path | **v0.121.1** — tag `v0.121.1`, sha256 `afaeeb509d1ed70d6e6bebac0393a3cd5be59d51e3db9ff96ef8524bd78546d7`, round-trip verified |
| Deployed (published bytes again) | `felhom-agent --version`**0.121.1**, `active`; prior kept as `.bak-0.121.0` |
| Validation config removed | the 2-min drop-in deleted; the daemon back on **`eval_interval=6h0m0s settle=24h0m0s`** |
| Fleet | demo-hp still runs **0.120.0** — deliberate: pointing machines at a version is what **vouching** does |
| **Vouching** | **NOT done — deliberately the operator's act.** Hub UI → Configs → Day-0 artifacts: agent **`0.121.1`**, sha `afaeeb50…` (0.121.0 also published, sha `b2128f3c…`) |
| Config compatibility checked on both boxes | `restore_test_cadence_seconds = 0` on demo-felhom **and** demo-hp, and the installer writes `0` — so no box is on the deprecation path, and a fresh install gets the new defaults with no installer change |
## 9. Findings filed (none fixed blind)
@@ -246,6 +270,27 @@ after it — the restore reclaimed to the same shape it started in.
run created was withdrawn (nothing had been published under it — verified 404) and recreated on the
fix commit, so one version name still means one binary.
- **R-188 — a correct agent release emails a CI failure about half the time.** `on: [push]` fires the
gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag before publishing
(deliberately). CI can therefore run the published-versions gate inside the window where the tag
exists and the package does not, and correctly report *"every released agent version must be
INSTALLABLE"* for a release that completes seconds later. **Measured across two releases in one
session:** v0.121.0 → runs #12 success / #13 failure on the same sha; v0.121.1 → #17 failure / #18
success on the same sha; and one pair both green — a race, not a rule. It matters because R-168
made CI email on failure so a red gate cannot be missed; a signal that cries wolf on every second
correct release is how that mail becomes something you archive unread.
- **R-189 — a passing restore-test can be invisible to the hub, and R-86 widened that window.**
Observed live: **the 15:25:14 PASS reached no host-report at all**. `restore_tests[]` comes from an
**in-memory** store (*"lost on restart; the cadence re-populates"*) and the report interval is
900 s; the agent was restarted 2 m 43 s after the run for the v0.121.1 deploy. That used to
self-heal within 24 h because the next cadence re-tested the tier — **under per-archive due-ness
the agent will not re-test a proven archive**, so the hub can stay ignorant until the next archive
generation. The persisted proof already exists: `RestoreTestState.Snapshot()` is documented *"for
the host-report gauge"* and has **no production caller** — a seam built and never wired, and an
invariant asserted only in a comment, in one method. Bounded, not over-ranked: the hub scans its
retained window and the offsite tier's window (12 d) is wider than its archive rhythm (7 d), so one
lost report is tolerated. Filed, not fixed — it is a report-contract change.
## 10. CI
| Repo | Run | Commit | Result |