felhom-agent/REPORT.md

# REPORT — R-189 · R-188 · R-186: three ways the signals lied about themselves

**Date:** 2026-08-03 · **Repo:** `felhom-agent` **v0.121.1 → v0.122.0** (`7581f81`) · released,
published, verified by an independent download **and by rebuilding it**, deployed to demo-felhom.
`felhom.eu`: register + docs only, **no hub change and no hub bump** — the hub already reads
`restore_tests[]`; the defect was that the agent stopped sending them.

---

## 1. Baselines, re-read on arrival

| Repo | `main` @ commit | Version | Matched §1? |
|---|---|---|---|
| `felhom-agent` | `3d0a1d615d11` | `v0.121.1` | **yes** |
| `felhom.eu` | `c9a3e48b2106` | hub `v0.91.1` | **yes** |

Highest register ID in use was **R-189**; no new IDs were needed — all three rows already existed.
(Grep confirmed R-190+ free, in case one had been.)

## 2. Scenario H — the reproducibility measurement (Part 3, done first on purpose)

**Before**, one commit, same source, same toolchain, same ldflags — the only difference is whether the
tag existed when the build ran:

| build | sha256 | size | embedded module version |
|---|---|---|---|
| default flags, **no tag yet** | `18f4a495…` | 14 085 464 B | `v0.121.2-0.20260803133646-3d0a1d61` |
| default flags, **tagged** | `4a38f394…` | 14 085 440 B | `v0.121.99` |
| `-trimpath -buildvcs=false`, either way | `7ffcdf1d…` | 14 064 574 B | *(none)* |

**After, on the real release (v0.122.0) — the three values §15.2 asks for:**

| artifact | sha256 | size |
|---|---|---|
| published, downloaded from Gitea | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #1 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |
| rebuild at the tag, #2 | `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` | 14 076 649 B |

**All three identical.** The property is removed-cause, not sequenced-around: `-buildvcs=false` drops
a stamp nothing reads (no `ReadBuildInfo` caller, verified by grep), and the version still comes from
the explicit `-X main.version` ldflag. `-trimpath` additionally makes a rebuild from a different
checkout directory match.

**A second discrepancy fell out of the measurement and is fixed with it:** `publish-agent.sh`'s
fallback build forced `CGO_ENABLED=0` and produced **13 990 236 B** against the release path's
**14 064 574 B** — a 74 KB difference, i.e. one version name meaning two binaries depending on which
entry point ran. Both paths now use identical flags, each commented with a pointer to the other.

## 3. Scenario E — a correct release no longer emails a failure (Part 2)

**Only the tag PUSH moved.** The order is now build → tag **locally** → publish → push tag. The tag is
still created before anything is published, so the build and the tag describe the same commit; it
becomes *visible* — to CI (`on: [push]`) and to any `raw/tag/…` fetch — only once the package is
downloadable.

**The old ordering's invariant is asserted directly rather than arranged for.**
`check-published-versions.py` now carries two invariants: every tag has an installable package (as
before) **and no published version is missing its tag** (new). The second is a **bounded probe** —
the frontier past the newest tag, where a failed tag push leaves an orphan, plus patch gaps — and it
**prints its probe set on every run**, because a check whose coverage is invisible reads as a
guarantee it is not making. The package listing api was **re-measured**, not assumed: `401` without a
token, so absence still cannot be enumerated, and the script says so in its own output.

**Live result — this release is the test:**

| release | CI runs on the release commit | outcome |
|---|---|---|
| v0.121.0 (yesterday) | #12 / #13 | one **green**, one **red** |
| v0.121.1 (yesterday) | #17 / #18 | one **red**, one **green** |
| **v0.122.0 (this one)** | **#21 (task id 96) / #22 (task id 97)** | **both green** |

**Failure modes made loud rather than tidy:** a publish that succeeds followed by a tag push that
fails now dies printing `git push origin v<ver>` — the local tag is already there, so recovery is one
line — and a publish that *fails* deletes the local-only tag so a retry is clean instead of colliding
with step 2's re-release guard.

## 4. Scenarios F and G — both directions demonstrated, then cleaned up

**G — a published version with no tag must FAIL.** A real fixture: version **0.121.2** (the frontier,
exactly where a failed tag push lands) published to the live registry with no tag.

```
0.121.2    next patch after the newest tag            PUBLISHED — NO TAG
check-published-versions: 1 PUBLISHED VERSION(S) WITH NO TAG
  v0.121.2 is downloadable at … but has no git tag.
      git push origin v0.121.2
EXIT=1
```

**Red-proof (observed):** with the fixture still live, replacing the probe set with `[]` produced
`ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED`, **exit 0** — a green run over a published
orphan. Restored.

**Teardown:** fixture deleted (HTTP 204), absence independently re-verified (`GET → 404`), gate green
again (exit 0). No scratch tag was ever pushed.

**F — a tag with no package must still FAIL.** Demonstrated against a local stand-in serving two tags
where only one has a package:

```
  ok   v0.120.0: binary downloadable + tag serves its configs
  FAIL v0.199.0:
        - binary NOT downloadable (HTTP 404 …)
        - tag does not serve configs/felhom-agent.service (HTTP 404) — a box would 404 mid-install
EXIT=1
```

**Why not a real pushed tag:** pushing one wakes CI and would have emailed the operator a **true**
alarm about a fixture — the same attention cost R-188 exists to remove. F is also already
demonstrated in the wild: CI runs **#13** and **#17** failed for exactly this reason yesterday.

## 5. R-189 — a proof that survives a restart reaches the hub (Part 1)

`restore_tests[]` came only from the in-memory `backup.Store`, whose comment read *"lost on restart;
the cadence re-populates"*. True under a timer; false since R-86, because the agent refuses to re-test
an archive it has already proven — so a lost proof is not repeated for a whole archive generation.

**What changed**

- `RestoreTestState` stores the **tier** and what was **verified** beside the archive (v3 shape).
  Both are recorded **at proof time from the run's own result** — deriving them later would need a
  storage-type lookup at report-building time, a network call that can fail on the one path where
  failing means mislabelling a proof.
- `ProvenRestoreTests` renders the stored proofs as report entries; `Collector.SetProvenRestoreTests`
  merges them with the in-memory result.
- **Merge rule: one entry per tier, newest by `TestedAt` wins.** It falls out of what each source
  means rather than from a preference: a fresh failure beats a stored success (the failure is the
  news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a
  tier never appears twice — the hub would read that as two tests. An unparseable timestamp counts as
  **older**, so a malformed entry cannot displace a good one.
- **It refuses to lie.** A record missing the archive **or** the tier produces no entry, and run
  mechanics (scratch VMID, duration) are not re-invented: an absent duration is not a claim, a
  fabricated one would be.
- **The asymmetry is now in the code** (§8.1): a success *suppresses* future work so it must be
  durable; a failure *causes* future work and heals itself, and persisting one would make a healed
  tier keep reporting a fault.
- **`Store`'s comment is corrected in place** — leaving it is how the next reader concludes this is
  handled.

**Migration, and it is visible on the live box:** a pre-R-189 record has an archive but no tier, so it
is **not** reportable. Upgrading does not retroactively make an old proof visible; the tier's next
real proof fills it in. Confirmed immediately after the deploy — still `0 restore-tests`, with the
v2 record sitting on disk.

## 6. Scenario A, live on demo-felhom — against the observation that filed R-189

**The observation being replaced (2026-08-03, 15:25):** a real offsite restore-test PASSED, the agent
was restarted 2 m 43 s later, and the hub logged `0 restore-tests` on the next two host-reports.

**The same sequence, on v0.122.0:**

```
16:44:06  restore-test tier is DUE   target=felhom-pbs
          archive=felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z
          reason="newest settled archive … has not been proven (last proven archive was a different one)"
16:55:21  restore-test: scratch guest torn down  vmid=990000
16:55:21  backup: scheduled restore-test PASSED  archive=felhom-pbs:…2026-07-27T19:55:41Z  duration_s=675.1
16:55:32  systemctl restart felhom-agent        ← INSIDE the 15-minute reporting window
16:55:36  hub: host-report from demo-felhom-8363b5 (… 1 restore-tests …)      ← was 0
```

**The proof on disk (v3 — the tier is what the old shape lacked):**

```json
{"felhom-pbs": {"archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z",
                "tier": "pbs", "verified": "boot+running", "proven_at": "2026-08-03T14:55:21Z"}}
```

**What the HUB stored** — read from its own database (copied with its `-wal`, freshness confirmed by
the newest row's `received_at` = `2026-08-03 14:55:36` UTC, matching the ingest line):

```json
{ "source_archive": "felhom-pbs:backup/ct/9201/2026-07-27T19:55:41Z",
  "source_tier": "pbs", "pass": true, "verified": "boot+running",
  "tested_at": "2026-08-03T14:55:21Z", "scratch_vmid": 0, "duration_seconds": 0 }
```

That report was built **after** the restart, when the in-memory store was empty — so the entry can
only have come from the persisted state. The archive, the tier, and the **original** test time
survived; the run mechanics are zero because they are deliberately not re-invented.

**A 14.5 GB encrypted offsite archive**, restored, booted, verified and destroyed in **675 s** — and
this time the proof outlived the process that produced it.

**Teardown, all three layers:** scratch guest absent from `pct list` (0), its volumes gone from `lvs`
(0), the validation drop-in removed and the daemon back on its defaults
(`eval_interval=6h0m0s settle=24h0m0s`). The hub-side `restore_tests[]` record is **retained
deliberately** — it is the proof the staleness check reads, so deleting it would delete the result.
No `restore_test_*` event was raised, because nothing failed and nothing is stale.


## 7. Tests and red-proofs

Green gate: `go build ./... && go vet ./... && go test ./...` — **29 packages ok, rc=0**, plus
`python3 scripts/agent_gates.py` (reuse-refs + published-versions) all OK. The test run and the commit
were always separate commands.

| # | Test | Asserts | Mutation | Observed |
|---|---|---|---|---|
| A | `TestMerge_ProofSurvivesARestart` | an empty in-memory store + a persisted proof → the proof is reported, with its archive and its original time | the persisted merge deleted (the pre-R-189 body) | **FAIL** — `after a restart the persisted proof must be reported; got 0 entr(ies): []` — the live observation exactly |
| B | `TestMerge_NeverInventsAPassForAnUnprovenTier` | no proof → no entry; a tier-less record → no entry | — (its state-layer twin below carries the mutation) | pass |
| B′ | `TestProvenRestoreTests_RefusesToReportWhatItCannotDescribe` | v1 + v2 + v3 records side by side → only the describable one is reported | the `reportable()` filter dropped | **FAIL** — `got 3` entries, two with an empty `SourceTier`/`SourceArchive` |
| C | `TestMerge_NewerWinsAndNeverDuplicatesATier` | one entry per tier, newest wins, in both directions | de-duplication removed | **FAIL** — `one entry per tier; got 2 for "pbs" — the hub would read two tests` |
| D | `TestMerge_AFailureIsStillReported` | a fresh failure beats an older stored success | (same mutation) | **FAIL** — 2 entries, i.e. the failure no longer the single answer for that tier |
| — | `TestMerge_MalformedTimestampNeverWins` | unparseable ≠ newest | — | pass |
| — | `TestMerge_NilProvenSourceIsANoOp` | pre-R-189 behaviour unchanged when unwired | — | pass |
| — | `TestScheduler_ProofIsRecordedReportably` | a pass **through the scheduler** leaves a reportable proof | — | pass |
| — | `TestScheduler_AFailureLeavesNoPersistedProof` | §8.1's asymmetry, asserted not assumed | — | pass |
| G | the gate's converse assertion | a published version with no tag fails | probe set → `[]` | **FAIL** (green over a live orphan) |
| I | `TestMainWiresTheDurableRestoreTestProof` | **AST**: `SetProvenRestoreTests` is called **and fed `rtState`** | the call commented out | **FAIL** — `main.go never calls collector.SetProvenRestoreTests` (a `strings.Contains` check would have passed — the string is still there) |
| H | reproducibility | three identical sha256 | — (measurement, §2) | pass |

**Jitter, per §10:** every timestamp fixture uses odd minutes and seconds (`13:25:14`, `19:55:41`,
`13:41:07`, `04:41:58`) — several taken from the real box — rather than round hours. Yesterday a test
was hollow because a perfectly regular series landed exactly on a threshold and survived its own
mutation.

## 8. Files changed

`internal/backup/restoretest_state.go` (v3 record + `ProvenRestoreTests`), `internal/backup/store.go`
(the comment that had become false), `internal/backup/schedule.go` (record tier + verified),
`internal/hub/collect.go` (the seam + the merge), `cmd/felhom-agent/main.go` (wiring),
`scripts/release-agent.sh` (ordering, reproducible build, loud half-done release),
`scripts/publish-agent.sh` (identical build flags), `scripts/check-published-versions.py` (the
converse invariant), plus `REUSE.md`, `CHANGELOG.md`, `CONTEXT.md`, `CLAUDE.md` and three test files.

**Commits** — `felhom-agent`: `7581f81` (v0.122.0). `felhom.eu`: see §10.

## 9. The independent-verification command (Part 3, recorded in `CLAUDE.md`)

```bash
V=0.122.0
git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
    -o /tmp/felhom-agent-check ./cmd/felhom-agent
sha256sum /tmp/felhom-agent-check
curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
```

Both print `d5f294e56c1ef59055e8e87fb9135aa477632dbbc56a5d4bff46bbd0466c1edf` (§2).

## 10. Registers

- **R-189 → CLOSED** (shipped + proven live), **R-188 → CLOSED** (shipped), **R-186 → CLOSED**
  (shipped + measured).
- **R-185 remains OPEN and untouched** — it is a missing `/storage/felhom-backup` ACL on demo-felhom,
  a permission defect, not a reporting one. Nothing in this session changed it, and the priority list
  says so explicitly.
- No new IDs minted. `ROADMAP.md` contains none of these three rows, so there was nothing to collapse.
- `00-capability-map.md`'s restore-proof row now records that the evidence path itself had a gap and
  what closed it; `CONTEXT.md` gains **S-19** (the proof/failure asymmetry and the merge rule) and
  **S-20** (the release ordering and what each step protects); `STATUS.md` rewritten for the operator
  and trimmed to 83 lines.

## 11. Observations — noticed, recorded, NOT acted on

- **The proof state holds ONE record per tier, so proving an OLDER archive re-arms a newer one —
  CONFIRMED after the validation, not merely predicted.** With the defaults restored, the due-check
  reads: `tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z" proven="…2026-07-27T19:55:41Z" —
  newest settled archive has not been proven (last proven archive was a different one)`.
  Surfaced by this session's own validation method: to get a fresh proof without waiting a week, the
  settle lag was widened so the older, unproven offsite archive became the candidate. That overwrote
  the record for the newer archive, so once the default 24 h settle returns, the newest settled
  archive is no longer the recorded proof and the tier becomes due once more. **Consequence, stated
  rather than left to surprise: demo-felhom will run one further unattended offsite restore-test
  within 6 h, after which the newest archive is the recorded proof** — the correct steady state. In
  normal operation this cannot arise, because the candidate only ever moves forward.
- **`RestoreTestState.Snapshot()` has no caller again.** The host report is now fed by
  `ProvenRestoreTests`, which carries what a bare timestamp cannot. The method's doc comment says in
  as many words that it should be deleted if it does not acquire one — deliberately not deleted in
  this session, because removing an exported method is a change with no bearing on the three rows.
- **Ten files in this repo are not `gofmt`-clean and were already so on arrival**
  (`internal/capability/probe.go`, `internal/escrow/consume.go`, `internal/mgmtplane/mgmtplane.go`,
  `internal/reconcile/bringup.go`, `internal/signedjobs/runner.go`, `internal/storage/{candidates,
  intent}.go` and three test files). Every file this session touched is clean; the others are
  untouched, and no gate checks formatting.
- **The hub sweeps every 60 s and re-reads 14 days of host-reports per customer** for the restore-test
  staleness check. Unchanged here and not a defect at this fleet size; it is the cost centre if the
  fleet grows, and it is the reason the window read was deliberately left at 14 days yesterday.
- **`felhom.eu/CONTEXT.md` still carries duplicate standing-ruling IDs** (three `S-14`s, two `S-15`s)
  from before yesterday. New rulings continue to be numbered above the collision (S-19, S-20) rather
  than adding to it; renumbering the existing ones is a separate, purely editorial change.