admin
7581f8140a
gates / gates (push) Successful in 7s
All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.
R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.
R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.
R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
213 lines
13 KiB
Bash
Executable File
213 lines
13 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# release-agent.sh — THE way to release a felhom-agent version. One act: build → tag → publish →
|
|
# verify by independent download.
|
|
#
|
|
# WHY THIS EXISTS (R-115). Publishing used to be a step someone had to remember, and it was
|
|
# forgotten THREE TIMES IN FIVE DAYS:
|
|
#
|
|
# * R-111 (2026-07-29) 17 releases v0.97.0-v0.113.0 built and never published, so a new customer
|
|
# would have installed without the whole R-82 tiered-backup arc, F-CRIT-2 and F-REBOOT.
|
|
# * 0.114.0 (same afternoon) built, deployed to felhom-pve, never published.
|
|
# * 0.120.0 (2026-08-03) built, committed and deployed to BOTH demo hosts, never published. A
|
|
# documented-path reinstall would have silently DOWNGRADED both boxes to the pre-merge
|
|
# agent — and would have *succeeded* while doing it, because the current `step_grows`
|
|
# sets SYSDATA_GROW=0 so the older agent's fatal mp1 resize never fires.
|
|
#
|
|
# R-111's own closing line said publishing should join the release train rather than stay a
|
|
# remembered step. It closed SHIPPED without that leg, and the leg recurred the same afternoon —
|
|
# which is the evidence that a note is not a mechanism. This file is the mechanism. The
|
|
# documentation now points here instead of at a raw `go build` line, so there is ONE documented way
|
|
# to release and it cannot complete without publishing.
|
|
#
|
|
# WHY IT TAGS (R-183). Since felhom-host-install.sh pins its sixteen agent-config fetches to
|
|
# `raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root, on a
|
|
# virgin machine. The tag and the package are two halves of one release and are created together.
|
|
#
|
|
# WHY IT DOES NOT VOUCH. Vouching is what points machines at a version, and it stays the operator's
|
|
# deliberate act — the same prove-then-vouch principle that governed the golden two sessions ago.
|
|
# This script prints the version and sha to vouch; a human decides when.
|
|
#
|
|
# Usage:
|
|
# GITEA_USER=admin GITEA_TOKEN=<token> ./scripts/release-agent.sh <version>
|
|
#
|
|
# Env: GITEA_USER/GITEA_TOKEN (package write) — same credentials publish-agent.sh already takes.
|
|
# GITEA_BASE / GITEA_OWNER override the defaults.
|
|
# RELEASE_ALLOW_DIRTY=1 skips the clean-tree gate (for a rehearsal; never for a real release).
|
|
set -euo pipefail
|
|
|
|
GITEA_BASE="${GITEA_BASE:-https://gitea.dooplex.hu}"
|
|
GITEA_OWNER="${GITEA_OWNER:-admin}"
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
|
|
die() { echo "[release-agent] ERROR: $*" >&2; exit 1; }
|
|
log() { echo "[release-agent] $*" >&2; }
|
|
|
|
VERSION="${1:-}"
|
|
[[ -n "$VERSION" ]] || die "version required (usage: GITEA_USER=.. GITEA_TOKEN=.. $0 <version>)"
|
|
[[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || die "version must be bare semver X.Y.Z (got '$VERSION')"
|
|
TAG="v$VERSION"
|
|
|
|
cd "$REPO_ROOT"
|
|
|
|
# ── 1. Clean-tree gate ──────────────────────────────────────────────────────────────────────────
|
|
# An unpushed change does not exist. Releasing a dirty tree publishes a binary whose source nobody
|
|
# else can obtain, and tags a commit that does not contain what was built.
|
|
if [[ "${RELEASE_ALLOW_DIRTY:-0}" != "1" ]]; then
|
|
[[ -z "$(git status --porcelain)" ]] || die "working tree is dirty — commit and push first"
|
|
local_head="$(git rev-parse HEAD)"
|
|
git fetch -q origin main
|
|
[[ "$local_head" == "$(git rev-parse origin/main)" ]] \
|
|
|| die "HEAD != origin/main — push first (an unpushed change does not exist)"
|
|
fi
|
|
|
|
# ── 2. Refuse to re-release a version that already exists ───────────────────────────────────────
|
|
# Silently overwriting a published artifact is how "the same version" comes to mean two different
|
|
# binaries on two different boxes.
|
|
if git rev-parse -q --verify "refs/tags/$TAG" >/dev/null; then
|
|
die "tag $TAG already exists — releasing over it would make one version name two binaries"
|
|
fi
|
|
existing="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
|
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
|
[[ "$existing" != "200" ]] || die "version $VERSION is ALREADY PUBLISHED — bump the version instead"
|
|
|
|
# ── 3. Build ────────────────────────────────────────────────────────────────────────────────────
|
|
BIN="$(mktemp -t felhom-agent-XXXXXX)"
|
|
trap 'rm -f "$BIN"' EXIT
|
|
log "building $VERSION …"
|
|
# REPRODUCIBLE BY CONSTRUCTION (R-186). The sha printed below is the one the operator vouches, and
|
|
# until now nobody could rebuild it to check: `go build` stamps a module version derived from VCS
|
|
# state, so a build made BEFORE the tag exists and a rebuild made after it are different binaries.
|
|
# Measured 2026-08-03 at this commit — same source, same toolchain, same ldflags:
|
|
#
|
|
# default flags, no tag yet .. 18f4a495… 14 085 464 B (mod v0.121.2-0.2026…-3d0a1d61)
|
|
# default flags, tagged ...... 4a38f394… 14 085 440 B (mod v0.121.99)
|
|
# -trimpath -buildvcs=false ... 7ffcdf1d… 14 064 574 B IDENTICAL both ways
|
|
#
|
|
# `-buildvcs=false` removes the stamp — nothing in this repo reads it (no `ReadBuildInfo` caller,
|
|
# verified) and the version comes from the explicit ldflag below, which is where it belongs.
|
|
# `-trimpath` removes absolute build paths, so a rebuild from a different checkout directory also
|
|
# matches. Neither is a sequencing trick: the property no longer depends on WHEN the build happens.
|
|
#
|
|
# CGO is deliberately left at its default. publish-agent.sh's fallback build used to force
|
|
# CGO_ENABLED=0 and therefore produced a DIFFERENT binary (13 990 236 B, 74 KB smaller) for the same
|
|
# version — one version name, two binaries, by whichever entry point was used. Both now build the
|
|
# same way; if that ever has to change, change it in BOTH or the guarantee is gone.
|
|
go build -trimpath -buildvcs=false -ldflags "-X main.version=$VERSION" -o "$BIN" ./cmd/felhom-agent \
|
|
|| die "go build failed"
|
|
built_ver="$("$BIN" --version 2>/dev/null | awk '{print $2}')"
|
|
[[ "$built_ver" == "$VERSION" ]] \
|
|
|| die "the built binary reports '$built_ver', not '$VERSION' — the ldflag did not take"
|
|
BUILT_SHA="$(sha256sum "$BIN" | awk '{print $1}')"
|
|
log "built ok: sha256 $BUILT_SHA"
|
|
|
|
# ── 4. Tag LOCALLY (the push comes after the publish — see step 6) ──────────────────────────────
|
|
#
|
|
# THE ORDER CHANGED, AND ONLY THE PUSH MOVED (R-188, 2026-08-03).
|
|
#
|
|
# It used to be tag → push tag → publish, and the reason written here was sound: a tag with no
|
|
# package is caught by scripts/check-published-versions.py, a package with no tag is invisible to it,
|
|
# because the Gitea package LISTING api needs a token the gate does not have.
|
|
#
|
|
# What that reasoning missed is that the tag PUSH is what wakes CI (`on: [push]`), so the gate ran in
|
|
# the seconds between the tag becoming visible and the package existing — and correctly failed. Every
|
|
# correct release had roughly a coin-flip chance of emailing the operator a failure for a release
|
|
# that worked. Measured across two releases in one session: runs 12/13 (v0.121.0) and 17/18
|
|
# (v0.121.1), same sha each time, opposite results. R-168 made that mail the thing that cannot be
|
|
# missed; a mail that is wrong half the time is one you stop reading, and then the real one goes too.
|
|
#
|
|
# So the tag is still created HERE, before anything is published — the build and the tag still
|
|
# describe the same commit, and a failed publish leaves a purely local tag that never misled anyone.
|
|
# It simply becomes VISIBLE (to CI, and to any installer fetching raw/tag/…) only once the package
|
|
# is downloadable. The invariant the old order protected is not traded away: it is asserted directly
|
|
# by the gate's new converse probe (a published version with no tag FAILS), so both directions are
|
|
# now checked rather than one being arranged for.
|
|
log "tagging $TAG at $(git rev-parse --short HEAD) …"
|
|
git tag -a "$TAG" -m "agent $TAG
|
|
|
|
Released by scripts/release-agent.sh.
|
|
sha256 of the published binary: $BUILT_SHA
|
|
|
|
felhom-host-install.sh fetches this version's config files from raw/tag/$TAG/configs/,
|
|
so this tag is part of the released artifact, not a bookmark (R-183)."
|
|
|
|
# ── 5. Publish (the existing script; deliberately not reimplemented) ────────────────────────────
|
|
log "publishing …"
|
|
# Invoked through `bash` DELIBERATELY, not as an executable. On 2026-08-03 the first real release
|
|
# through this script died here — `publish-agent.sh` has been mode 0644 since it was created on
|
|
# 2026-06-28, because every earlier caller ran it as `bash scripts/publish-agent.sh`. So the one leg
|
|
# R-115 exists to make unforgettable was, on its first use, unrunnable. The mode bit is restored in
|
|
# the same commit; this line makes the release independent of it, because a file mode is exactly the
|
|
# kind of thing that is lost again by a checkout, an archive, or a copy.
|
|
if ! bash "$REPO_ROOT/scripts/publish-agent.sh" "$VERSION" "$BIN"; then
|
|
# The tag is LOCAL-ONLY at this point, so a failed publish must not leave one behind: the next
|
|
# attempt would die at step 2's "tag $TAG already exists" and read as "this version is already
|
|
# released", which would be exactly backwards. Only remove it if nothing was in fact published —
|
|
# if a package DOES exist, the tag is wanted and must be pushed, not deleted.
|
|
now_published="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
|
"$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" 2>/dev/null || true)"
|
|
if [[ "$now_published" == "200" ]]; then
|
|
log "publish reported failure but the package IS downloadable — keeping the local tag; push it with: git push origin $TAG"
|
|
else
|
|
git tag -d "$TAG" >/dev/null 2>&1 && log "removed the local-only tag $TAG so the release can be retried"
|
|
fi
|
|
die "publish failed"
|
|
fi
|
|
|
|
# ── 6. Push the tag, now that the package exists ────────────────────────────────────────────────
|
|
# This is the step that makes the release VISIBLE — to CI, and to every `raw/tag/v<version>/` fetch
|
|
# the installer makes. It runs last of the two so CI can never see a tag whose package is not there.
|
|
#
|
|
# If it fails, the release is HALF DONE and must be said so loudly: the package is published and the
|
|
# tag exists only in this clone, which is precisely the orphan the gate's converse probe now catches.
|
|
# The recovery is one line and it is printed rather than described.
|
|
log "pushing $TAG …"
|
|
if ! git push origin "$TAG"; then
|
|
cat >&2 <<EOF
|
|
|
|
RELEASE HALF DONE — the package is PUBLISHED and its tag is NOT pushed.
|
|
|
|
version : $VERSION
|
|
sha256 : $BUILT_SHA
|
|
|
|
The tag exists in this clone only. Nothing installs from an untagged version (the installer
|
|
fetches this version's configs from raw/tag/$TAG/), and scripts/check-published-versions.py will
|
|
FAIL on it as a published version with no tag. Finish the release with:
|
|
|
|
git push origin $TAG
|
|
|
|
EOF
|
|
die "tag push failed after a successful publish — see above"
|
|
fi
|
|
|
|
# ── 7. Verify by an INDEPENDENT download ────────────────────────────────────────────────────────
|
|
# The publish step's own success is not proof: it reports on its own write. What matters is that a
|
|
# box can now GET the bytes and that they are the bytes that were built. This is the same
|
|
# presence-is-not-success rule the project earned twice — a step that says "done" and a fetch that
|
|
# returns the right sha are different claims.
|
|
log "verifying by independent download …"
|
|
DL="$(mktemp -t felhom-agent-dl-XXXXXX)"
|
|
trap 'rm -f "$BIN" "$DL"' EXIT
|
|
curl -fsS -o "$DL" "$GITEA_BASE/api/packages/$GITEA_OWNER/generic/felhom-agent/$VERSION/felhom-agent" \
|
|
|| die "round-trip GET failed — the version is NOT installable"
|
|
DL_SHA="$(sha256sum "$DL" | awk '{print $1}')"
|
|
[[ "$DL_SHA" == "$BUILT_SHA" ]] \
|
|
|| die "published sha $DL_SHA != built sha $BUILT_SHA — the artifact is not what was built"
|
|
|
|
# The tag must also serve the configs the installer will fetch from it.
|
|
cfg_code="$(curl -fsS -o /dev/null -w '%{http_code}' \
|
|
"$GITEA_BASE/$GITEA_OWNER/felhom-agent/raw/tag/$TAG/configs/felhom-agent.service" 2>/dev/null || true)"
|
|
[[ "$cfg_code" == "200" ]] \
|
|
|| die "tag $TAG does not serve configs/felhom-agent.service (HTTP $cfg_code) — a box would 404 mid-install"
|
|
|
|
cat <<EOF
|
|
|
|
RELEASED — and installable, verified by download, not by this script's own say-so.
|
|
|
|
version : $VERSION
|
|
tag : $TAG
|
|
sha256 : $BUILT_SHA
|
|
|
|
NOT VOUCHED. Vouching is what points machines at this version and stays your deliberate act:
|
|
hub operator UI → Configs → Day-0 artifacts. Until then boxes keep installing the previous one.
|
|
EOF
|