admin
dd2d1feb6e
gates / gates (push) Failing after 7s
NO VERSION BUMP and nothing built: no Go code changed. The agent stays v0.120.0. scripts/release-agent.sh — THE way to release. build -> tag -> publish -> verify by INDEPENDENT download. Publishing was a separate remembered step and was forgotten three times in five days (R-111's 17 stranded releases, 0.114.0, and 0.120.0 — deployed to both demo hosts and undownloadable, so a documented-path reinstall would have silently downgraded them WHILE REPORTING SUCCESS). R-111's own closing line named this leg and closed SHIPPED without it; it recurred the same afternoon, which is the evidence that a note is not a mechanism. It tags because felhom-host-install.sh now fetches the sixteen agent config files from raw/tag/v<version>/ (R-183): a released version with no tag 404s a box mid-install, as root, on a virgin machine. It verifies by downloading what it just published and comparing the sha to what it built — the publish step's own success is a report on its own write; a fetch returning the right bytes is a different claim. It refuses a dirty/unpushed tree and refuses to re-release an existing version. It does NOT vouch: that points machines at a version and stays the operator's act. scripts/check-published-versions.py — the gate. Every v<semver> tag must have a downloadable package AND a tag tree serving the agent's configs. Registered as NOT --fast (needs network; a push must not fail because Gitea blinked), and the CI workflow now runs the FULL gate set instead of --fast — otherwise the gate would have been registered and never run, the built-but-never-wired failure this project has shipped four times. The invariant is not the one specified, and the reason was measured, not assumed: the hub artifact manifest is 401 without a per-customer passphrase and Gitea's package LISTING api is 401 without a token, while the package DOWNLOAD url and the git TAGS api are anonymous. So CI cannot ask "what is vouched" without an operator credential — whose addition is the operator's call. The tag-based invariant needs none and catches all three recorded instances. What it does not catch (the hub vouching a version never released at all) is filed as R-184.
117 lines
6.6 KiB
YAML
117 lines
6.6 KiB
YAML
# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
|
|
# pushed or what they typed.
|
|
#
|
|
# *** THIS REPORTS. IT CANNOT REFUSE. ***
|
|
#
|
|
# felhom repos push straight to `main` with no pull request, so there is no merge for a status
|
|
# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
|
|
# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
|
|
# whole thing, and both are named in felhom.eu documentation/backlog/OPEN-ITEMS.md R-168.
|
|
#
|
|
# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
|
|
# the runner is a host-mode container with python3 and git and nothing else (see
|
|
# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
|
|
# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
|
|
#
|
|
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
|
|
# layer up. That is the last step, and it runs ONLY on failure.
|
|
name: gates
|
|
on: [push]
|
|
|
|
jobs:
|
|
gates:
|
|
runs-on: felhom-gates
|
|
steps:
|
|
- name: Fetch the pushed commit and the sibling clone it needs
|
|
# This repo's entry point invokes a SHARED checker that lives in the felhom.eu clone next
|
|
# door and is deliberately never copied here — so CI has to reproduce the workspace's
|
|
# sibling layout or the gate fails closed with "gate is MISSING". The sibling is also
|
|
# needed for CONTENT: this repo's REUSE.md cites a path that lives in the hub.
|
|
run: |
|
|
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip,
|
|
# which can move under us if two pushes race.
|
|
mkdir -p ws/felhom-agent
|
|
cd ws/felhom-agent
|
|
git init -q .
|
|
git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom-agent.git
|
|
git fetch -q --depth 1 origin "$GITHUB_SHA"
|
|
git checkout -q FETCH_HEAD
|
|
echo "checked out $(git rev-parse HEAD)"
|
|
cd .. && git clone -q --depth 1 http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git felhom.eu
|
|
echo "sibling felhom.eu present at $(cd felhom.eu && git rev-parse --short HEAD)"
|
|
|
|
- name: Run the gate entry point
|
|
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
|
|
# exit code IS the result: no `|| true`, no pipe that could swallow it.
|
|
#
|
|
# THE FULL SET, NOT `--fast` (R-115, 2026-08-03). `--fast` means "no network and no
|
|
# container runtime" and exists for `.githooks/pre-push`, where a push must not fail
|
|
# because Gitea blinked or because someone is on a train. CI is the opposite machine: it
|
|
# has the network, it is not in anyone's way, and it is the half that emails. The
|
|
# published-versions gate — the R-115 mechanism, which asks Gitea whether a released
|
|
# version can actually be downloaded — is network-bound and therefore runs ONLY here.
|
|
# Leaving `--fast` in place would have registered that gate and never run it, which is the
|
|
# built-but-never-wired failure this project has shipped four times.
|
|
env:
|
|
# In-cluster, so the check does not depend on public DNS or the ingress TLS chain.
|
|
GITEA_BASE: http://gitea.gitea-system.svc.cluster.local:3000
|
|
run: cd ws/felhom-agent && python3 scripts/agent_gates.py
|
|
|
|
- name: Alarm on failure
|
|
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
|
|
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
|
|
# is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
|
|
# existing transactional path (Resend, the same one the hub uses), and prints the provider's
|
|
# accepted id so "a message left the machine" is an observable, not an assumption.
|
|
#
|
|
# Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
|
|
# else on purpose, and the first version of this step died on `curl: command not found`.
|
|
# Reaching for a bigger image to send one HTTP request would have been the wrong trade.
|
|
if: failure()
|
|
env:
|
|
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
|
run: |
|
|
python3 - <<'PY'
|
|
import json, os, sys, urllib.request, urllib.error
|
|
|
|
key = os.environ.get("RESEND_API_KEY", "")
|
|
if not key:
|
|
sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
|
|
"silent alarm is worse than none. Set the user-level Actions secret.")
|
|
|
|
repo = os.environ.get("GITHUB_REPOSITORY", "?")
|
|
sha = os.environ.get("GITHUB_SHA", "?")
|
|
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
|
|
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
|
|
|
|
body = json.dumps({
|
|
"from": "Felhom CI <monitoring@felhom.eu>",
|
|
"to": ["admin@felhom.eu"],
|
|
"subject": "[felhom CI] gates FAILED in %s" % repo,
|
|
"text": (
|
|
"The gate entry point exited non-zero.\n\n"
|
|
"Repository : %s\n"
|
|
"Commit : %s\n"
|
|
"Run : %s/%s/actions/runs/%s\n\n"
|
|
"The failing gate names itself in the run log.\n\n"
|
|
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
|
|
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
|
|
"outranks whatever the push was for.\n"
|
|
) % (repo, sha, srv, repo, run),
|
|
}).encode()
|
|
|
|
req = urllib.request.Request(
|
|
"https://api.resend.com/emails", data=body, method="POST",
|
|
headers={"Authorization": "Bearer %s" % key,
|
|
"Content-Type": "application/json",
|
|
# Cloudflare fronts api.resend.com and BLOCKS the default
|
|
# "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
|
|
# exactly like an auth failure and is not one. Measured 2026-08-02.
|
|
"User-Agent": "felhom-ci/1.0"})
|
|
try:
|
|
with urllib.request.urlopen(req, timeout=30) as r:
|
|
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
|
|
except urllib.error.HTTPError as e:
|
|
sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
|
|
PY
|