release path publishes, and an unreleasable version fails CI (R-115, R-183)
gates / gates (push) Failing after 7s
gates / gates (push) Failing after 7s
NO VERSION BUMP and nothing built: no Go code changed. The agent stays v0.120.0. scripts/release-agent.sh — THE way to release. build -> tag -> publish -> verify by INDEPENDENT download. Publishing was a separate remembered step and was forgotten three times in five days (R-111's 17 stranded releases, 0.114.0, and 0.120.0 — deployed to both demo hosts and undownloadable, so a documented-path reinstall would have silently downgraded them WHILE REPORTING SUCCESS). R-111's own closing line named this leg and closed SHIPPED without it; it recurred the same afternoon, which is the evidence that a note is not a mechanism. It tags because felhom-host-install.sh now fetches the sixteen agent config files from raw/tag/v<version>/ (R-183): a released version with no tag 404s a box mid-install, as root, on a virgin machine. It verifies by downloading what it just published and comparing the sha to what it built — the publish step's own success is a report on its own write; a fetch returning the right bytes is a different claim. It refuses a dirty/unpushed tree and refuses to re-release an existing version. It does NOT vouch: that points machines at a version and stays the operator's act. scripts/check-published-versions.py — the gate. Every v<semver> tag must have a downloadable package AND a tag tree serving the agent's configs. Registered as NOT --fast (needs network; a push must not fail because Gitea blinked), and the CI workflow now runs the FULL gate set instead of --fast — otherwise the gate would have been registered and never run, the built-but-never-wired failure this project has shipped four times. The invariant is not the one specified, and the reason was measured, not assumed: the hub artifact manifest is 401 without a per-customer passphrase and Gitea's package LISTING api is 401 without a token, while the package DOWNLOAD url and the git TAGS api are anonymous. So CI cannot ask "what is vouched" without an operator credential — whose addition is the operator's call. The tag-based invariant needs none and catches all three recorded instances. What it does not catch (the hub vouching a version never released at all) is filed as R-184.
This commit is contained in:
@@ -9,6 +9,12 @@
|
||||
Gates (all must pass; **non-zero exit on any failure**):
|
||||
|
||||
1. reuse-refs every path cited by this repo's REUSE.md still resolves
|
||||
2. published every `v<semver>` tag has a downloadable package AND a tag tree that serves
|
||||
the agent's configs (R-115). NEEDS NETWORK, so it is **not** in `--fast` and
|
||||
the pre-push hook does not run it — a push must not fail because Gitea blinked
|
||||
or because someone is offline on a train. CI runs the FULL set for exactly this
|
||||
reason: it is the machine that can afford a network check, and it is the half
|
||||
that emails when something is wrong.
|
||||
|
||||
WHY THIS FILE EXISTS, WITH ONE GATE (2026-08-02, R-29 leg (b)).
|
||||
|
||||
@@ -39,6 +45,7 @@ SHARED_REUSE = os.path.join(os.path.dirname(ROOT), "felhom.eu", "scripts", "reus
|
||||
# (label, absolute script path, args, fast)
|
||||
GATES = [
|
||||
("reuse-refs", SHARED_REUSE, [ROOT], True),
|
||||
("published", os.path.join(ROOT, "scripts", "check-published-versions.py"), [], False),
|
||||
]
|
||||
|
||||
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""check-published-versions.py — a released agent version must be INSTALLABLE, not merely built.
|
||||
|
||||
R-115. A box installs the agent from a Gitea generic package the hub vouches, never from git, and
|
||||
since R-110/R-183 it also fetches the agent's sixteen config files from `raw/tag/v<version>/`.
|
||||
Nothing in the build, deploy or session-end path checked that either existed, so "deployed" and
|
||||
"installable" were independent states that drifted silently. **Three instances in five days:**
|
||||
|
||||
* R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published — a new customer
|
||||
would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
|
||||
* 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
|
||||
* 0.120.0 (2026-08-03) built, committed, deployed to BOTH demo hosts, never published. A
|
||||
documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
||||
agent — and would have *succeeded* while doing it.
|
||||
|
||||
THE INVARIANT, AND WHY IT IS THIS ONE.
|
||||
|
||||
For every `v<semver>` git tag in this repo: the matching generic package must be DOWNLOADABLE,
|
||||
and the tag must serve the agent's configs.
|
||||
|
||||
The task's §8.4 asked for a different one — *"the version the hub tells machines to install must be
|
||||
downloadable"* — and that is the better invariant in principle. **It is not implementable from CI,
|
||||
and that was measured rather than assumed:** the hub's artifact manifest
|
||||
(`GET /api/v1/artifacts/<customer>`) answers **401** without a per-customer retrieval passphrase,
|
||||
and the Gitea package LISTING api answers **401** without a token, while the package DOWNLOAD url
|
||||
and the git TAGS api are both anonymous. So a credential-free gate can ask *"is this version
|
||||
installable"* but not *"which version is vouched"*. Putting an operator credential into CI to close
|
||||
that gap is the operator's call, not a gate author's — it is recorded as a limitation below and as a
|
||||
backlog row rather than quietly assumed away.
|
||||
|
||||
**What this invariant does catch: all three instances above.** `release-agent.sh` creates the tag and
|
||||
publishes in one act, so a release whose publish was skipped, failed, or was forgotten leaves a tag
|
||||
with no package — which is exactly what this refuses. It needs no version floor: tags begin at
|
||||
v0.120.0, which is published.
|
||||
|
||||
**What it does NOT catch, stated plainly:** the hub vouching a version that was never released at
|
||||
all (no tag, no package). Nothing here can see that; it belongs at vouch time, in the hub. → R-184.
|
||||
|
||||
FAIL-CLOSED. A network error, an unparseable response or an unreachable Gitea is exit **2
|
||||
INCONCLUSIVE**, naming every URL tried — never a pass. "Cannot determine" is not "fine": that is the
|
||||
standing rule this project earned twice, and a gate that green-lights on its own blindness is worse
|
||||
than no gate, because it looks like coverage.
|
||||
|
||||
Pure python3 + urllib, NO curl and no third-party module: the CI runner is a host-mode container
|
||||
carrying python3 and git and nothing else, and an earlier workflow step died on
|
||||
`curl: command not found`.
|
||||
|
||||
python3 scripts/check-published-versions.py
|
||||
|
||||
Exit: 0 every tag installable · 1 at least one is not · 2 could not be determined.
|
||||
Env: GITEA_BASE overrides the Gitea root (CI sets the in-cluster service URL).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
GITEA_BASE = os.environ.get("GITEA_BASE", "https://gitea.dooplex.hu").rstrip("/")
|
||||
OWNER = "admin"
|
||||
REPO = "felhom-agent"
|
||||
PKG = "felhom-agent"
|
||||
TIMEOUT = 25
|
||||
|
||||
# One config the installer fetches. Its presence proves the TAG's tree carries the configs the
|
||||
# sixteen `fetch_raw` calls will ask for — a tag that exists but predates them would 404 a box
|
||||
# mid-install, on a virgin machine, as root.
|
||||
PROBE_CONFIG = "configs/felhom-agent.service"
|
||||
|
||||
TAG_RE = re.compile(r"^v(\d+\.\d+\.\d+)$")
|
||||
|
||||
tried = []
|
||||
|
||||
|
||||
def _get(url, want_body=False):
|
||||
"""GET a URL. Returns (status, body_or_None). Network failure raises."""
|
||||
tried.append(url)
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
|
||||
body = r.read() if want_body else None
|
||||
return r.status, body
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, None
|
||||
|
||||
|
||||
def inconclusive(msg):
|
||||
print("INCONCLUSIVE:", msg)
|
||||
print(" URLs tried (a 'no access' claim must name its attempts):")
|
||||
for u in tried:
|
||||
print(" ", u)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def main():
|
||||
print("check-published-versions — every released agent version must be INSTALLABLE")
|
||||
print(" gitea:", GITEA_BASE)
|
||||
|
||||
tags_url = "%s/api/v1/repos/%s/%s/tags?limit=200" % (GITEA_BASE, OWNER, REPO)
|
||||
try:
|
||||
status, body = _get(tags_url, want_body=True)
|
||||
except Exception as e:
|
||||
inconclusive("cannot reach Gitea to list tags: %s" % e)
|
||||
if status != 200 or not body:
|
||||
inconclusive("tags api returned HTTP %s — cannot enumerate releases" % status)
|
||||
try:
|
||||
tags = [t["name"] for t in json.loads(body.decode("utf-8"))]
|
||||
except Exception as e:
|
||||
inconclusive("tags api response is not the expected JSON: %s" % e)
|
||||
|
||||
versions = sorted(m.group(1) for m in (TAG_RE.match(t) for t in tags) if m)
|
||||
if not versions:
|
||||
# Not a failure: a repo legitimately has no version tags before its first release. Say so
|
||||
# loudly rather than reporting a vacuous pass — an empty check that prints OK is how an
|
||||
# unexamined thing becomes a documented-clean one.
|
||||
print(" no v<semver> tags in this repo yet — nothing to check, and nothing proven")
|
||||
print("\ncheck-published-versions: NOTHING TO CHECK")
|
||||
return 0
|
||||
print(" %d released version(s) to verify: %s" % (len(versions), ", ".join(versions)))
|
||||
|
||||
bad = []
|
||||
for v in versions:
|
||||
pkg_url = "%s/api/packages/%s/generic/%s/%s/%s" % (GITEA_BASE, OWNER, PKG, v, PKG)
|
||||
raw_url = "%s/%s/%s/raw/tag/v%s/%s" % (GITEA_BASE, OWNER, REPO, v, PROBE_CONFIG)
|
||||
try:
|
||||
pkg_status, _ = _get(pkg_url)
|
||||
raw_status, _ = _get(raw_url)
|
||||
except Exception as e:
|
||||
inconclusive("network failure while checking v%s: %s" % (v, e))
|
||||
|
||||
problems = []
|
||||
if pkg_status != 200:
|
||||
problems.append("binary NOT downloadable (HTTP %s at %s)" % (pkg_status, pkg_url))
|
||||
if raw_status != 200:
|
||||
problems.append("tag does not serve %s (HTTP %s) — a box would 404 mid-install"
|
||||
% (PROBE_CONFIG, raw_status))
|
||||
if problems:
|
||||
bad.append((v, problems))
|
||||
print(" FAIL v%s:" % v)
|
||||
for p in problems:
|
||||
print(" -", p)
|
||||
else:
|
||||
print(" ok v%s: binary downloadable + tag serves its configs" % v)
|
||||
|
||||
print()
|
||||
if bad:
|
||||
print("check-published-versions: %d RELEASED VERSION(S) NOT INSTALLABLE" % len(bad))
|
||||
print(" A tagged version with no package is a release that was BUILT and never PUBLISHED —")
|
||||
print(" the R-115 defect, three times in five days. Publish it with:")
|
||||
print(" scripts/release-agent.sh <version>")
|
||||
return 1
|
||||
print("check-published-versions: ALL RELEASED VERSIONS INSTALLABLE")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+133
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env bash
|
||||
# release-agent.sh — THE way to release a felhom-agent version. One act: build → tag → publish →
|
||||
# verify by independent download.
|
||||
#
|
||||
# WHY THIS EXISTS (R-115). Publishing used to be a step someone had to remember, and it was
|
||||
# forgotten THREE TIMES IN FIVE DAYS:
|
||||
#
|
||||
# * R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published, so a new customer
|
||||
# would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
|
||||
# * 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
|
||||
# * 0.120.0 (2026-08-03) built, committed and deployed to BOTH demo hosts, never published. A
|
||||
# documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
||||
# agent — and would have *succeeded* while doing it, because the current `step_grows`
|
||||
# sets SYSDATA_GROW=0 so the older agent's fatal mp1 resize never fires.
|
||||
#
|
||||
# R-111's own closing line said publishing should join the release train rather than stay a
|
||||
# remembered step. It closed SHIPPED without that leg, and the leg recurred the same afternoon —
|
||||
# which is the evidence that a note is not a mechanism. This file is the mechanism. The
|
||||
# documentation now points here instead of at a raw `go build` line, so there is ONE documented way
|
||||
# to release and it cannot complete without publishing.
|
||||
#
|
||||
# WHY IT TAGS (R-183). Since felhom-host-install.sh pins its sixteen agent-config fetches to
|
||||
# `raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root, on a
|
||||
# virgin machine. The tag and the package are two halves of one release and are created together.
|
||||
#
|
||||
# WHY IT DOES NOT VOUCH. Vouching is what points machines at a version, and it stays the operator's
|
||||
# deliberate act — the same prove-then-vouch principle that governed the golden two sessions ago.
|
||||
# This script prints the version and sha to vouch; a human decides when.
|
||||
#
|
||||
# Usage:
|
||||
# GITEA_USER=admin GITEA_TOKEN=<token> ./scripts/release-agent.sh <version>
|
||||
#
|
||||
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
|
||||
# GITEA_BASE / GITEA_OWNER override the defaults.
|
||||
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
|
||||
set -euo pipefail
|
||||
|
||||
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
|
||||
GITEA_OWNER="${GITEA_OWNER:-admin}"
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
die() { echo "[release-agent] ERROR: $*" >&2; exit 1; }
|
||||
log() { echo "[release-agent] $*" >&2; }
|
||||
|
||||
VERSION="${1:-}"
|
||||
[[ -n "$VERSION" ]] || die "version required (usage: GITEA_USER=.. GITEA_TOKEN=.. $0 <version>)"
|
||||
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "version must be bare semver X.Y.Z (got '$VERSION')"
|
||||
TAG="v$VERSION"
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
# ── 1. Clean-tree gate ──────────────────────────────────────────────────────────────────────────
|
||||
# An unpushed change does not exist. Releasing a dirty tree publishes a binary whose source nobody
|
||||
# else can obtain, and tags a commit that does not contain what was built.
|
||||
if [[ "${RELEASE_ALLOW_DIRTY:-0}" != "1" ]]; then
|
||||
[[ -z "$(git status --porcelain)" ]] || die "working tree is dirty — commit and push first"
|
||||
local_head="$(git rev-parse HEAD)"
|
||||
git fetch -q origin main
|
||||
[[ "$local_head" == "$(git rev-parse origin/main)" ]] \
|
||||
|| die "HEAD != origin/main — push first (an unpushed change does not exist)"
|
||||
fi
|
||||
|
||||
# ── 2. Refuse to re-release a version that already exists ───────────────────────────────────────
|
||||
# Silently overwriting a published artifact is how "the same version" comes to mean two different
|
||||
# binaries on two different boxes.
|
||||
if git rev-parse -q --verify "refs/tags/$TAG" >/dev/null; then
|
||||
die "tag $TAG already exists — releasing over it would make one version name two binaries"
|
||||
fi
|
||||
existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
||||
[[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead"
|
||||
|
||||
# ── 3. Build ────────────────────────────────────────────────────────────────────────────────────
|
||||
BIN="$(mktemp -t felhom-agent-XXXXXX)"
|
||||
trap 'rm -f "$BIN"' EXIT
|
||||
log "building $VERSION …"
|
||||
go build -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|
||||
|| die "go build failed"
|
||||
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
||||
[[ "$built_ver" == "$VERSION" ]] \
|
||||
|| die "the built binary reports '$built_ver', not '$VERSION' — the ldflag did not take"
|
||||
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
|
||||
log "built ok: sha256 $BUILT_SHA"
|
||||
|
||||
# ── 4. Tag (before publishing, so a published version always has a tag) ─────────────────────────
|
||||
# Order matters in this direction only: a tag with no package is caught by
|
||||
# scripts/check-published-versions.py on the next CI run; a package with no tag is invisible to it,
|
||||
# because the Gitea package LISTING api needs a token the gate does not have.
|
||||
log "tagging $TAG at $(git rev-parse --short HEAD) …"
|
||||
git tag -a "$TAG" -m "agent $TAG
|
||||
|
||||
Released by scripts/release-agent.sh.
|
||||
sha256 of the published binary: $BUILT_SHA
|
||||
|
||||
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
|
||||
so this tag is part of the released artifact, not a bookmark (R-183)."
|
||||
git push origin "$TAG" || die "tag push failed — refusing to publish an untagged version"
|
||||
|
||||
# ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
|
||||
log "publishing …"
|
||||
"$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN" || die "publish failed"
|
||||
|
||||
# ── 6. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
|
||||
# The publish step's own success is not proof: it reports on its own write. What matters is that a
|
||||
# box can now GET the bytes and that they are the bytes that were built. This is the same
|
||||
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that
|
||||
# returns the right sha are different claims.
|
||||
log "verifying by independent download …"
|
||||
DL="$(mktemp -t felhom-agent-dl-XXXXXX)"
|
||||
trap 'rm -f "$BIN" "$DL"' EXIT
|
||||
curl -fsS -o "$DL" "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" \
|
||||
|| die "round-trip GET failed — the version is NOT installable"
|
||||
DL_SHA="$(sha256sum "$DL" | awk '{print $1}')"
|
||||
[[ "$DL_SHA" == "$BUILT_SHA" ]] \
|
||||
|| die "published sha $DL_SHA != built sha $BUILT_SHA — the artifact is not what was built"
|
||||
|
||||
# The tag must also serve the configs the installer will fetch from it.
|
||||
cfg_code="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
||||
"$GITEA_BASE/$GITEA_OWNER/felhom-agent/raw/tag/$TAG/configs/felhom-agent.service" 2>/dev/null || true)"
|
||||
[[ "$cfg_code" == "200" ]] \
|
||||
|| die "tag $TAG does not serve configs/felhom-agent.service (HTTP $cfg_code) — a box would 404 mid-install"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||
|
||||
version : $VERSION
|
||||
tag : $TAG
|
||||
sha256 : $BUILT_SHA
|
||||
|
||||
NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act:
|
||||
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
||||
EOF
|
||||
Reference in New Issue
Block a user