v0.122.0: three ways the signals lied about themselves (R-189, R-188, R-186)

All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.

R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.

R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.

R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.

scripts/check-published-versions.py

@@ -14,10 +14,32 @@ Nothing in the build, deploy or session-end path checked that either existed, so
            documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
            agent — and would have *succeeded* while doing it.
 
-THE INVARIANT, AND WHY IT IS THIS ONE.
+THE INVARIANTS — there are TWO now, and the second is R-188's price.
 
-  For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
-  and the tag must serve the agent's configs.
+  (1) For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
+      and the tag must serve the agent's configs.
+
+  (2) No PUBLISHED version may be missing its tag.
+
+Invariant (2) is new (R-188, 2026-08-03) and it exists because `release-agent.sh` now pushes the tag
+AFTER publishing. The old order pushed the tag first, and the old comment said why: a tag with no
+package is caught here, a package with no tag is invisible, because the Gitea package LISTING api
+needs a token this gate does not have. That reasoning was sound and the ordering was still wrong —
+the tag push is what wakes CI, so every correct release had a ~50% chance of running this gate in the
+seconds before its own package existed and mailing the operator a failure for a release that worked
+(measured across two releases: runs 12/13 and 17/18, same shas, opposite results).
+
+Moving the push does not get to trade invariant (2) away, so it is asserted here instead — WITHOUT a
+token, and therefore as a BOUNDED PROBE rather than an enumeration:
+
+  * the FRONTIER — the versions immediately above the highest tag. This is the realistic failure the
+    new ordering makes possible: publish succeeds, tag push fails, so the orphan is exactly one
+    version beyond the newest tag.
+  * the GAPS — patch versions that fall between two existing tags and have no tag of their own.
+
+Re-measured 2026-08-03, not assumed: `GET /api/v1/packages/admin?type=generic` answers **401** with no
+token, so absence still cannot be proven. The probe set is PRINTED on every run, because a check whose
+coverage is invisible reads as a guarantee it is not making.
 
 The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
 downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
@@ -36,6 +58,8 @@ v0.120.0, which is published.
 
 **What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
 all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. → R-184.
+Nor does the converse probe prove that NO untagged package exists — only that none exists at the
+probed versions, which are printed. Closing that properly needs a read token in CI (→ R-184).
 
 FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
 INCONCLUSIVE**, naming every URL tried — never a pass. "Cannot determine" is not "fine": that is the
@@ -48,7 +72,7 @@ carrying python3 and git and nothing else, and an earlier workflow step died on
 
     python3 scripts/check-published-versions.py
 
-Exit: 0 every tag installable · 1 at least one is not · 2 could not be determined.
+Exit: 0 both invariants hold · 1 either is violated · 2 could not be determined.
 Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
 """
 import json
@@ -94,6 +118,53 @@ def inconclusive(msg):
     sys.exit(2)
 
 
+def _pkg_exists(version):
+    """True iff the generic package for `version` is downloadable anonymously."""
+    url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, version, PKG)
+    status, _ = _get(url)
+    return status == 200, url
+
+
+def untagged_probe_set(versions):
+    """The versions to probe for invariant (2), as (version, why) pairs.
+
+    Bounded on purpose and printed by the caller: the package listing api needs a token (401,
+    re-measured 2026-08-03), so absence cannot be enumerated. What CAN be done is to probe the
+    places an orphan would actually land.
+
+      FRONTIER — a publish that succeeded followed by a tag push that failed leaves the orphan
+                 exactly one version past the newest tag. This is the failure mode the R-188
+                 reordering makes possible, so it is the one that must not be guesswork.
+      GAPS     — a patch number skipped between two consecutive tags. Bounded per gap so a typo'd
+                 tag (v0.130.0 after v0.121.1) cannot turn this into a thousand requests.
+    """
+    parsed = sorted(tuple(int(p) for p in v.split(".")) for v in versions)
+    have = set(parsed)
+    out = []
+    if not parsed:
+        return out
+
+    hi = parsed[-1]
+    for cand, why in (
+        ((hi[0], hi[1], hi[2] + 1), "next patch after the newest tag"),
+        ((hi[0], hi[1], hi[2] + 2), "second patch after the newest tag"),
+        ((hi[0], hi[1] + 1, 0), "next minor after the newest tag"),
+        ((hi[0] + 1, 0, 0), "next major after the newest tag"),
+    ):
+        if cand not in have:
+            out.append(("%d.%d.%d" % cand, why))
+
+    MAX_GAP_PROBES = 12
+    for a, b in zip(parsed, parsed[1:]):
+        if a[0] != b[0] or a[1] != b[1]:
+            continue  # a minor/major step is not a patch gap
+        for patch in range(a[2] + 1, min(b[2], a[2] + 1 + MAX_GAP_PROBES)):
+            cand = (a[0], a[1], patch)
+            if cand not in have:
+                out.append(("%d.%d.%d" % cand, "patch gap between v%d.%d.%d and v%d.%d.%d" % (a + b)))
+    return out
+
+
 def main():
     print("check-published-versions — every released agent version must be INSTALLABLE")
     print("  gitea:", GITEA_BASE)
@@ -144,14 +215,42 @@ def main():
         else:
             print("  ok   v%s: binary downloadable + tag serves its configs" % v)
 
+    # ── invariant (2): no PUBLISHED version may be missing its tag (R-188) ──────────────────────
+    probes = untagged_probe_set(versions)
+    orphans = []
     print()
+    print("  converse probe — a published version with no tag (bounded; the package listing api")
+    print("  needs a token, so this cannot enumerate). Probing %d version(s):" % len(probes))
+    for v, why in probes:
+        try:
+            exists, url = _pkg_exists(v)
+        except Exception as e:
+            inconclusive("network failure while probing v%s: %s" % (v, e))
+        mark = "PUBLISHED — NO TAG" if exists else "absent (ok)"
+        print("    %-10s %-42s %s" % (v, why, mark))
+        if exists:
+            orphans.append((v, url))
+
+    print()
+    if bad or orphans:
+        if orphans:
+            print("check-published-versions: %d PUBLISHED VERSION(S) WITH NO TAG" % len(orphans))
+            for v, url in orphans:
+                print("  v%s is downloadable at %s but has no git tag." % (v, url))
+            print("  A release publishes and then pushes its tag; a package with no tag means the")
+            print("  push failed or was skipped. The local tag is probably still in the release")
+            print("  clone — finish it with:")
+            for v, _ in orphans:
+                print("      git push origin v%s" % v)
+            print("  (and if the tag is gone, re-create it on the released commit before pushing.)")
     if bad:
         print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
         print("  A tagged version with no package is a release that was BUILT and never PUBLISHED —")
         print("  the R-115 defect, three times in five days. Publish it with:")
         print("      scripts/release-agent.sh <version>")
+    if bad or orphans:
         return 1
-    print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE")
+    print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE, AND NONE UNTAGGED")
     return 0
 
 

scripts/publish-agent.sh

@@ -51,7 +51,11 @@ if [[ -z "$BIN" ]]; then
     BIN="$(mktemp -t felhom-agent.XXXXXX)"
     CLEANUP_BIN="$BIN"
     log "building felhom-agent $VERSION from $REPO_ROOT …"
-    ( cd "$REPO_ROOT" && CGO_ENABLED=0 go build -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
+    # These flags MUST match release-agent.sh's build exactly — see the long comment there (R-186).
+    # They used to differ: this line forced CGO_ENABLED=0 and produced a binary 74 KB smaller than
+    # the one the release path built for the same version. One version name must mean one binary
+    # whichever entry point produced it.
+    ( cd "$REPO_ROOT" && go build -trimpath -buildvcs=false -ldflags "-X main.version=${VERSION}" -o "$BIN" ./cmd/felhom-agent )
 fi
 [[ -f "$BIN" ]] || die "binary not found: $BIN"
 trap '[[ -n "$CLEANUP_BIN" ]] && rm -f "$CLEANUP_BIN"' EXIT

scripts/release-agent.sh

@@ -74,7 +74,25 @@ existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
 BIN="$(mktemp -t felhom-agent-XXXXXX)"
 trap 'rm -f "$BIN"' EXIT
 log "building $VERSION …"
-go build -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
+# REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and
+# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS
+# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries.
+# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags:
+#
+#   default flags,  no tag yet .. 18f4a495…  14 085 464 B   (mod v0.121.2-0.2026…-3d0a1d61)
+#   default flags,  tagged ...... 4a38f394…  14 085 440 B   (mod v0.121.99)
+#   -trimpath -buildvcs=false ... 7ffcdf1d…  14 064 574 B   IDENTICAL both ways
+#
+# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller,
+# verified) and the version comes from the explicit ldflag below, which is where it belongs.
+# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also
+# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens.
+#
+# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force
+# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same
+# version — one version name, two binaries, by whichever entry point was used. Both now build the
+# same way; if that ever has to change, change it in BOTH or the guarantee is gone.
+go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
     || die "go build failed"
 built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
 [[ "$built_ver" == "$VERSION" ]] \
@@ -82,10 +100,27 @@ built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
 BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
 log "built ok: sha256 $BUILT_SHA"
 
-# ── 4. Tag (before publishing, so a published version always has a tag) ─────────────────────────
-# Order matters in this direction only: a tag with no package is caught by
-# scripts/check-published-versions.py on the next CI run; a package with no tag is invisible to it,
+# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
+#
+# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
+#
+# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no
+# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it,
 # because the Gitea package LISTING api needs a token the gate does not have.
+#
+# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in
+# the seconds between the tag becoming visible and the package existing — and correctly failed. Every
+# correct release had roughly a coin-flip chance of emailing the operator a failure for a release
+# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18
+# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be
+# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too.
+#
+# So the tag is still created HERE, before anything is published — the build and the tag still
+# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone.
+# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package
+# is downloadable. The invariant the old order protected is not traded away: it is asserted directly
+# by the gate's new converse probe (a published version with no tag FAILS), so both directions are
+# now checked rather than one being arranged for.
 log "tagging $TAG at $(git rev-parse --short HEAD) …"
 git tag -a "$TAG" -m "agent $TAG
 
@@ -94,7 +129,6 @@ sha256 of the published binary: $BUILT_SHA
 
 felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
 so this tag is part of the released artifact, not a bookmark (R-183)."
-git push origin "$TAG" || die "tag push failed — refusing to publish an untagged version"
 
 # ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
 log "publishing …"
@@ -104,9 +138,48 @@ log "publishing …"
 # R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
 # the same commit; this line makes the release independent of it, because a file mode is exactly the
 # kind of thing that is lost again by a checkout, an archive, or a copy.
-bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN" || die "publish failed"
+if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
+    # The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next
+    # attempt would die at step 2's "tag $TAG already exists" and read as "this version is already
+    # released", which would be exactly backwards. Only remove it if nothing was in fact published —
+    # if a package DOES exist, the tag is wanted and must be pushed, not deleted.
+    now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \
+        "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
+    if [[ "$now_published" == "200" ]]; then
+        log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG"
+    else
+        git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried"
+    fi
+    die "publish failed"
+fi
 
-# ── 6. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
+# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
+# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
+# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
+#
+# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the
+# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches.
+# The recovery is one line and it is printed rather than described.
+log "pushing $TAG …"
+if ! git push origin "$TAG"; then
+    cat >&2 <<EOF
+
+  RELEASE HALF DONE — the package is PUBLISHED and its tag is NOT pushed.
+
+    version : $VERSION
+    sha256  : $BUILT_SHA
+
+  The tag exists in this clone only. Nothing installs from an untagged version (the installer
+  fetches this version's configs from raw/tag/$TAG/), and scripts/check-published-versions.py will
+  FAIL on it as a published version with no tag. Finish the release with:
+
+      git push origin $TAG
+
+EOF
+    die "tag push failed after a successful publish — see above"
+fi
+
+# ── 7. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
 # The publish step's own success is not proof: it reports on its own write. What matters is that a
 # box can now GET the bytes and that they are the bytes that were built. This is the same
 # presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that