ci(R-1297 A): heavy suites by path on a branch push, all on a tag and nightly; no second full gate run
gates / gates (push) Successful in 3m23s
gates / gates (push) Successful in 3m23s
- script_tests_gate.py: test_gate_decoys / test_due_checks_gate / dooplex-offsite run on a CI branch push only when one of their paths changed (FELHOM_CHANGED_FILES); no list, an empty or unreadable list = every suite - gates.yml: nightly schedule 03:20 UTC; the list is set only for push:refs/heads/*; a trimmed commits array (total_commits > listed) is an unknown list - test_repo_gates.py and test_due_checks_gate.py group G drive the real runner with GATES narrowed to their labels, instead of the whole gate set a second time (130 s each in CI) - red-proved: filter cases, workflow pins (old gates.yml), inert run_gate, unregistered due-checks
This commit is contained in:
1 parent
8143a2c872
commit
ecd82abab8
5 files changed
+194
-10
No files matched your search
@@ -18,8 +18,13 @@
|
|||||||
name: gates
|
name: gates
|
||||||
# R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
|
# R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
|
||||||
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
|
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
|
||||||
|
# R-1297 (A, 2026-10-26): the nightly schedule is the run that still executes the three path-filtered suites when no
|
||||||
|
# push touched their files (scripts/script_tests_gate.py PATH_FILTERED). It sets no file list, so every suite runs —
|
||||||
|
# like a release tag (a tag push) and a manual dispatch. 03:20 UTC: after the 01:40 release copy and the night chain.
|
||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
|
schedule:
|
||||||
|
- cron: "20 3 * * *"
|
||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
inputs:
|
inputs:
|
||||||
force_alarm:
|
force_alarm:
|
||||||
@@ -149,6 +154,12 @@ jobs:
|
|||||||
if not commits:
|
if not commits:
|
||||||
sys.stderr.write("the payload carries no commits array - unknown\n")
|
sys.stderr.write("the payload carries no commits array - unknown\n")
|
||||||
raise SystemExit(0)
|
raise SystemExit(0)
|
||||||
|
# R-1297: Gitea trims the payload's commits array (the newest few only); a push of more commits than it
|
||||||
|
# lists has files nobody named here. That list is UNKNOWN, never short - so nothing is printed.
|
||||||
|
total = ev.get("total_commits")
|
||||||
|
if isinstance(total, int) and total > len(commits):
|
||||||
|
sys.stderr.write("the payload lists %d of %d commits - the file list is unknown\n" % (len(commits), total))
|
||||||
|
raise SystemExit(0)
|
||||||
seen = []
|
seen = []
|
||||||
for c in commits:
|
for c in commits:
|
||||||
for key in ("added", "modified", "removed"):
|
for key in ("added", "modified", "removed"):
|
||||||
@@ -172,6 +183,18 @@ jobs:
|
|||||||
[ "$SCOPE" = "docs" ] || SCOPE=code
|
[ "$SCOPE" = "docs" ] || SCOPE=code
|
||||||
echo "PUSH_SCOPE=$SCOPE" >> "$GITHUB_ENV"
|
echo "PUSH_SCOPE=$SCOPE" >> "$GITHUB_ENV"
|
||||||
echo "scope: $SCOPE"
|
echo "scope: $SCOPE"
|
||||||
|
# R-1297 (A): the heavy suites read this list and skip themselves when none of their paths changed. ONLY a
|
||||||
|
# branch push with a known list sets it; a tag, the nightly schedule and a dispatch run every suite.
|
||||||
|
case "${GITHUB_EVENT_NAME:-}:${GITHUB_REF:-}" in
|
||||||
|
push:refs/heads/*)
|
||||||
|
if [ -s /tmp/pushed-files.txt ]; then
|
||||||
|
echo "FELHOM_CHANGED_FILES=/tmp/pushed-files.txt" >> "$GITHUB_ENV"
|
||||||
|
echo "suite filter: ON (a branch push with a known file list)"
|
||||||
|
else
|
||||||
|
echo "suite filter: OFF (no usable file list) - every suite runs"
|
||||||
|
fi ;;
|
||||||
|
*) echo "suite filter: OFF (${GITHUB_EVENT_NAME:-?} ${GITHUB_REF:-?}) - every suite runs" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
- name: Run the gate entry point
|
- name: Run the gate entry point
|
||||||
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy — those are either
|
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy — those are either
|
||||||
|
|||||||
@@ -45,6 +45,51 @@ def timeout_for(rel):
|
|||||||
return SUITE_TIMEOUT_S.get(rel.replace(os.sep, "/"), TIMEOUT_S)
|
return SUITE_TIMEOUT_S.get(rel.replace(os.sep, "/"), TIMEOUT_S)
|
||||||
SKIP_DIRS = {"__pycache__", ".git", "node_modules"}
|
SKIP_DIRS = {"__pycache__", ".git", "node_modules"}
|
||||||
|
|
||||||
|
# R-1297 (A, 2026-10-26): THREE HEAVY SUITES RUN ON A CI PUSH ONLY WHEN ONE OF THEIR OWN PATHS CHANGED. Measured in CI run
|
||||||
|
# 2366: the decoy suite 232 s, the due-checks suite 133 s, the DooPlex off-site suite 69 s — of a 13-minute run, on every
|
||||||
|
# push, documents included. The decoys test the GATE SCRIPTS (they plant a fixture and expect a conviction), so a push
|
||||||
|
# that touches no script cannot change their verdict; the other two test their own scripts only.
|
||||||
|
#
|
||||||
|
# A RUN WITHOUT A LIST RUNS EVERY SUITE. CI sets FELHOM_CHANGED_FILES only for a branch push whose file list is known
|
||||||
|
# (gates.yml); a release tag, the nightly schedule, a manual dispatch and the pre-push hook set nothing, so they run all
|
||||||
|
# of them. An unreadable or empty list is unknown, and unknown runs everything (fail-closed). What the filter may cost:
|
||||||
|
# a document change that breaks a decoy's FIXTURE (not a gate) is seen by the nightly run, within a day — said here so
|
||||||
|
# nobody reads the filter as free. Pinned by scripts/test_script_tests_gate.py (FILTER_CASES).
|
||||||
|
CHANGED_ENV = "FELHOM_CHANGED_FILES"
|
||||||
|
PATH_FILTERED = {
|
||||||
|
"scripts/test_gate_decoys.py": ("scripts/", ".gitea/", ".githooks/"),
|
||||||
|
"scripts/test_due_checks_gate.py": ("scripts/due_checks_gate.py", "scripts/test_due_checks_gate.py",
|
||||||
|
"scripts/repo_gates.py", "scripts/script_tests_gate.py"),
|
||||||
|
"scripts/dooplex-offsite/test_dooplex_offsite.py": ("scripts/dooplex-offsite/",),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def changed_paths():
|
||||||
|
"""The pushed paths, or None when no usable list exists (then every suite runs)."""
|
||||||
|
p = os.environ.get(CHANGED_ENV, "")
|
||||||
|
if not p:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
with open(p, encoding="utf-8") as fh:
|
||||||
|
paths = [l.strip() for l in fh if l.strip()]
|
||||||
|
except OSError as e:
|
||||||
|
print("script-tests: %s=%s unreadable (%s) — every suite runs" % (CHANGED_ENV, p, e))
|
||||||
|
return None
|
||||||
|
if not paths:
|
||||||
|
print("script-tests: %s=%s is empty — every suite runs" % (CHANGED_ENV, p))
|
||||||
|
return None
|
||||||
|
return paths
|
||||||
|
|
||||||
|
|
||||||
|
def skip_reason(rel, changed):
|
||||||
|
"""Why a suite is skipped on this push, or "" when it runs."""
|
||||||
|
prefixes = PATH_FILTERED.get(rel.replace(os.sep, "/"))
|
||||||
|
if changed is None or not prefixes:
|
||||||
|
return ""
|
||||||
|
if any(c.startswith(pfx) for c in changed for pfx in prefixes):
|
||||||
|
return ""
|
||||||
|
return "none of its paths (%s) changed in this push — it runs on a release tag and nightly" % ", ".join(prefixes)
|
||||||
|
|
||||||
|
|
||||||
def find_suites(scripts_dir):
|
def find_suites(scripts_dir):
|
||||||
out = []
|
out = []
|
||||||
@@ -66,9 +111,16 @@ def main(argv):
|
|||||||
print("script-tests: FAILED — no test_*.py found under %s/scripts (a gate that checked nothing is not green)" % root)
|
print("script-tests: FAILED — no test_*.py found under %s/scripts (a gate that checked nothing is not green)" % root)
|
||||||
return 1
|
return 1
|
||||||
env = dict(os.environ, **{MARK: "1"})
|
env = dict(os.environ, **{MARK: "1"})
|
||||||
|
changed = changed_paths()
|
||||||
failed = []
|
failed = []
|
||||||
|
skipped = 0
|
||||||
for s in suites:
|
for s in suites:
|
||||||
rel = os.path.relpath(s, root)
|
rel = os.path.relpath(s, root)
|
||||||
|
why = skip_reason(rel, changed)
|
||||||
|
if why:
|
||||||
|
print(" SKIP %-55s %s" % (rel, why))
|
||||||
|
skipped += 1
|
||||||
|
continue
|
||||||
t0 = time.time()
|
t0 = time.time()
|
||||||
limit = timeout_for(rel)
|
limit = timeout_for(rel)
|
||||||
try:
|
try:
|
||||||
@@ -86,7 +138,8 @@ def main(argv):
|
|||||||
if failed:
|
if failed:
|
||||||
print("script-tests: FAILED — %d of %d suite(s): %s" % (len(failed), len(suites), ", ".join(failed)))
|
print("script-tests: FAILED — %d of %d suite(s): %s" % (len(failed), len(suites), ", ".join(failed)))
|
||||||
return 1
|
return 1
|
||||||
print("script-tests: OK — %d suite(s), every one exited 0" % len(suites))
|
print("script-tests: OK — %d suite(s) ran, every one exited 0%s" % (
|
||||||
|
len(suites) - skipped, (", %d skipped by path (R-1297)" % skipped) if skipped else ""))
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -199,9 +199,21 @@ def test_g_gate_is_registered_in_the_runner():
|
|||||||
repo_gates.py's source would also match a commented-out line, which is precisely the
|
repo_gates.py's source would also match a commented-out line, which is precisely the
|
||||||
failure mode being guarded against.
|
failure mode being guarded against.
|
||||||
"""
|
"""
|
||||||
p = subprocess.run([sys.executable, RUNNER, "--fast"],
|
# R-1297 (A, 2026-10-26): the SAME runner (`main`, the real run_gate), its GATES table narrowed to the
|
||||||
capture_output=True, text=True, cwd=ROOT)
|
# entries labelled due-checks — taken from the real table, so a commented-out registration still leaves
|
||||||
out = p.stdout + p.stderr
|
# nothing to run and every check below fails. It used to run the whole gate set again (~130 s in CI).
|
||||||
|
import io
|
||||||
|
sys.path.insert(0, HERE)
|
||||||
|
import repo_gates
|
||||||
|
saved, real, buf = repo_gates.GATES, sys.stdout, io.StringIO()
|
||||||
|
try:
|
||||||
|
repo_gates.GATES = [g for g in saved if g[0] == "due-checks"]
|
||||||
|
sys.stdout = buf
|
||||||
|
repo_gates.main(["--fast"])
|
||||||
|
finally:
|
||||||
|
sys.stdout = real
|
||||||
|
repo_gates.GATES = saved
|
||||||
|
out = buf.getvalue()
|
||||||
check("G: runner names the due-checks gate in its output", "due-checks" in out)
|
check("G: runner names the due-checks gate in its output", "due-checks" in out)
|
||||||
check("G: runner reports a verdict line for it",
|
check("G: runner reports a verdict line for it",
|
||||||
any(l.strip().startswith("due-checks") and ("OK" in l or "FAILED" in l
|
any(l.strip().startswith("due-checks") and ("OK" in l or "FAILED" in l
|
||||||
|
|||||||
@@ -11,7 +11,15 @@ runner's actual effect.
|
|||||||
|
|
||||||
Red-proofed 2026-08-02: replacing run_gate's body with `return 0` (the inert runner) turns
|
Red-proofed 2026-08-02: replacing run_gate's body with `return 0` (the inert runner) turns
|
||||||
test_every_member_gate_actually_ran red while the summary still prints "all felhom.eu gates OK".
|
test_every_member_gate_actually_ran red while the summary still prints "all felhom.eu gates OK".
|
||||||
|
|
||||||
|
R-1297 (A, 2026-10-26): IT TESTS THE WIRING, NOT THE WHOLE GATE SET A SECOND TIME. It used to run `repo_gates.py --fast`
|
||||||
|
in full — every gate again, 130 s of every CI run, inside a run that had just executed the same gates. Now it drives the
|
||||||
|
SAME runner (`main`, the real `run_gate`, the real scripts) with the GATES table narrowed to the five fingerprinted
|
||||||
|
gates — taken FROM the real table by label, so a gate commented out of it fails here as before. That every gate in the
|
||||||
|
table runs is the entry point's own run on every push; this test only proves the runner calls what it lists.
|
||||||
|
Red-proved again 2026-10-26 with `return 0, ""` as run_gate's body.
|
||||||
"""
|
"""
|
||||||
|
import io
|
||||||
import os
|
import os
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
@@ -34,10 +42,26 @@ class RepoGatesTest(unittest.TestCase):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def setUpClass(cls):
|
def setUpClass(cls):
|
||||||
p = subprocess.run([sys.executable, ENTRY, "--fast"], cwd=ROOT,
|
sys.path.insert(0, os.path.join(ROOT, "scripts"))
|
||||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
import repo_gates
|
||||||
cls.rc = p.returncode
|
cls.repo_gates = repo_gates
|
||||||
cls.out = p.stdout.decode("utf-8", "replace")
|
labels = [l for l, _f in FINGERPRINTS]
|
||||||
|
cls.picked = [g for g in repo_gates.GATES if g[0] in labels]
|
||||||
|
real, buf = sys.stdout, io.StringIO()
|
||||||
|
saved = repo_gates.GATES
|
||||||
|
try:
|
||||||
|
repo_gates.GATES = cls.picked
|
||||||
|
sys.stdout = buf
|
||||||
|
cls.rc = repo_gates.main(["--fast"])
|
||||||
|
finally:
|
||||||
|
sys.stdout = real
|
||||||
|
repo_gates.GATES = saved
|
||||||
|
cls.out = buf.getvalue()
|
||||||
|
|
||||||
|
def test_every_fingerprinted_gate_is_registered(self):
|
||||||
|
got = sorted(g[0] for g in self.picked)
|
||||||
|
self.assertEqual(got, sorted(l for l, _f in FINGERPRINTS),
|
||||||
|
"a fingerprinted gate is missing from repo_gates.GATES")
|
||||||
|
|
||||||
def test_exit_code_is_zero(self):
|
def test_exit_code_is_zero(self):
|
||||||
self.assertEqual(self.rc, 0, self.out)
|
self.assertEqual(self.rc, 0, self.out)
|
||||||
|
|||||||
@@ -16,7 +16,8 @@ GATE = os.path.join(HERE, "script_tests_gate.py")
|
|||||||
|
|
||||||
COVERS = {
|
COVERS = {
|
||||||
"script-tests": ("a suite that PRINTS OK and exits 1; a failing suite three levels deep; an empty scripts/ "
|
"script-tests": ("a suite that PRINTS OK and exits 1; a failing suite three levels deep; an empty scripts/ "
|
||||||
"tree; plus the genuine article (must pass) and the nested-run mark (must step aside)"),
|
"tree; plus the genuine article (must pass) and the nested-run mark (must step aside); R-1297: a "
|
||||||
|
"failing heavy suite on a push that touched its own path, with no list, an empty or a missing list"),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -54,8 +55,79 @@ def timeouts_ok():
|
|||||||
return (g.timeout_for("scripts/test_gate_decoys.py") == 600 and g.timeout_for("scripts/test_other.py") == g.TIMEOUT_S == 300)
|
return (g.timeout_for("scripts/test_gate_decoys.py") == 600 and g.timeout_for("scripts/test_other.py") == g.TIMEOUT_S == 300)
|
||||||
|
|
||||||
|
|
||||||
|
# R-1297 (A): a heavy suite runs on a CI push only when one of ITS paths changed; with no list (a tag, the nightly run,
|
||||||
|
# the pre-push hook) or an unreadable or empty list, every suite runs. Each case plants a FAILING heavy suite, so
|
||||||
|
# „it ran" is rc 1 and „it was skipped" is rc 0 — the verdict, not a log line, says which happened.
|
||||||
|
CHANGED = "FELHOM_CHANGED_FILES"
|
||||||
|
|
||||||
|
|
||||||
|
def gate_changed(root, changed):
|
||||||
|
"""changed: None (variable unset), "MISSING" (points at no file), or a list of pushed paths."""
|
||||||
|
env = {k: v for k, v in os.environ.items() if k not in ("FELHOM_SCRIPT_TESTS_GATE", CHANGED)}
|
||||||
|
if changed == "MISSING":
|
||||||
|
env[CHANGED] = os.path.join(root, "no-such-list.txt")
|
||||||
|
elif changed is not None:
|
||||||
|
lst = os.path.join(root, "pushed.txt")
|
||||||
|
open(lst, "w").write("".join(f + "\n" for f in changed))
|
||||||
|
env[CHANGED] = lst
|
||||||
|
p = subprocess.run([sys.executable, GATE, root], env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||||
|
return p.returncode, p.stdout.decode("utf-8", "replace")
|
||||||
|
|
||||||
|
|
||||||
|
HEAVY = {"test_ok.py": "pass\n", "test_gate_decoys.py": "raise SystemExit(1)\n",
|
||||||
|
"dooplex-offsite/test_dooplex_offsite.py": "raise SystemExit(1)\n",
|
||||||
|
"test_due_checks_gate.py": "raise SystemExit(1)\n"}
|
||||||
|
FILTER_CASES = [
|
||||||
|
# (name, files, changed, want rc, text the output must hold)
|
||||||
|
("filter: no list -> every suite runs", HEAVY, None, 1, "test_gate_decoys.py"),
|
||||||
|
("filter: docs-only push skips the 3", HEAVY, ["documentation/x.md", "STATUS.md"], 0, "SKIP"),
|
||||||
|
("filter: a gate script runs decoys", {"test_ok.py": "pass\n", "test_gate_decoys.py": "raise SystemExit(1)\n"},
|
||||||
|
["scripts/site_gates.py"], 1, "test_gate_decoys.py"),
|
||||||
|
("filter: the due gate runs its suite", {"test_ok.py": "pass\n", "test_due_checks_gate.py": "raise SystemExit(1)\n"},
|
||||||
|
["scripts/due_checks_gate.py"], 1, "test_due_checks_gate.py"),
|
||||||
|
("filter: offsite path runs offsite", {"test_ok.py": "pass\n", "dooplex-offsite/test_dooplex_offsite.py": "raise SystemExit(1)\n"},
|
||||||
|
["scripts/dooplex-offsite/dooplex-offsite.sh"], 1, "test_dooplex_offsite.py"),
|
||||||
|
("filter: hub push skips offsite", {"test_ok.py": "pass\n", "dooplex-offsite/test_dooplex_offsite.py": "raise SystemExit(1)\n"},
|
||||||
|
["hub/internal/web/hosts.go"], 0, "SKIP"),
|
||||||
|
("filter: empty list -> every suite", HEAVY, [], 1, "test_gate_decoys.py"),
|
||||||
|
("filter: missing list -> every suite", HEAVY, "MISSING", 1, "test_gate_decoys.py"),
|
||||||
|
("filter: other suites never skipped", {"test_other.py": "raise SystemExit(1)\n"}, ["documentation/x.md"], 1, "test_other.py"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def workflow_pins():
|
||||||
|
"""R-1297: the filter is safe only while the nightly run exists and only a branch push sets the list. Read from the
|
||||||
|
real workflow and hook, so removing the schedule, or setting the list on every event, turns this red."""
|
||||||
|
root = os.path.dirname(HERE)
|
||||||
|
wf = open(os.path.join(root, ".gitea", "workflows", "gates.yml"), encoding="utf-8").read()
|
||||||
|
hook = open(os.path.join(root, ".githooks", "pre-push"), encoding="utf-8").read()
|
||||||
|
problems = []
|
||||||
|
if "schedule:" not in wf or "- cron:" not in wf:
|
||||||
|
problems.append("gates.yml has no nightly schedule — the filtered suites would run only when touched")
|
||||||
|
arm = wf.find("push:refs/heads/*)")
|
||||||
|
sets = [i for i in range(len(wf)) if wf.startswith('FELHOM_CHANGED_FILES=', i)]
|
||||||
|
if not sets or arm < 0 or any(i < arm or i > wf.find(";;", arm) for i in sets):
|
||||||
|
problems.append("gates.yml sets FELHOM_CHANGED_FILES outside the branch-push arm")
|
||||||
|
if "total_commits" not in wf:
|
||||||
|
problems.append("gates.yml does not treat a trimmed commits array as an unknown list")
|
||||||
|
if "FELHOM_CHANGED_FILES" in hook:
|
||||||
|
problems.append("the pre-push hook sets the suite filter — the hook must run every suite")
|
||||||
|
for p in problems:
|
||||||
|
print("FAIL workflow pin: " + p)
|
||||||
|
if not problems:
|
||||||
|
print("ok workflow pins (nightly run, branch-push-only list, trimmed payload, hook unfiltered)")
|
||||||
|
return not problems
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
bad = 0
|
bad = 0
|
||||||
|
bad += not workflow_pins()
|
||||||
|
for name, files, changed, want, text in FILTER_CASES:
|
||||||
|
got, out = gate_changed(fake_repo(files), changed)
|
||||||
|
ok = got == want and text in out
|
||||||
|
bad += not ok
|
||||||
|
print("%s %-38s want rc=%d got rc=%d%s" % ("ok " if ok else "FAIL", name, want, got,
|
||||||
|
"" if text in out else " (no %r in output)" % text))
|
||||||
t_ok = timeouts_ok()
|
t_ok = timeouts_ok()
|
||||||
bad += not t_ok
|
bad += not t_ok
|
||||||
print("%s %-32s" % ("ok " if t_ok else "FAIL", "per-suite time limits (R-1209)"))
|
print("%s %-32s" % ("ok " if t_ok else "FAIL", "per-suite time limits (R-1209)"))
|
||||||
@@ -64,7 +136,7 @@ def main():
|
|||||||
ok = got == want
|
ok = got == want
|
||||||
bad += not ok
|
bad += not ok
|
||||||
print("%s %-32s want rc=%d got rc=%d" % ("ok " if ok else "FAIL", name, want, got))
|
print("%s %-32s want rc=%d got rc=%d" % ("ok " if ok else "FAIL", name, want, got))
|
||||||
print("script-tests decoys: %d/%d" % (len(CASES) - bad, len(CASES)))
|
print("script-tests decoys: %d failed of %d" % (bad, len(CASES) + len(FILTER_CASES) + 1))
|
||||||
return 1 if bad else 0
|
return 1 if bad else 0
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user