admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs+gate: instruction files cannot silently regrow (R-229)
gates / gates (push) Successful in 7s
Details

Browse Source 
New shared scripts/instructions_gate.py, registered in controller_gates.py and
agent_gates.py, never copied into a sibling repo (the reuse_refs_check.py
precedent). 20 fixture tests, all asserting the effect: exit code AND that the
message names the file and the reason.

It is a consistency gate, not a budget gate, and the failure message says so. A
/context reading measured the instruction files at 15k tokens against 869k free in
a 1M window -- space is not the constraint, and a future reader must not re-derive
the wrong reason. The 200-line ceiling is adherence guidance; a file nobody can
hold in their head is where contradictions hide, and five were found here.

Checks run against effective text (HTML comments stripped, because they are
stripped before injection): the line ceiling; every .claude/rules/*.md declares
paths: or an explicit unconditional: true; no component version literal; no
TEMPORARY block carrying a past date; and the workspace-root CLAUDE.md is
byte-identical to its versioned copy -- the live file sits outside any git repo,
so that copy is its only version-controlled record.

Two traps recorded so they are not reintroduced: a bare \d+\.\d+\.\d+ matches the
first three octets of every IPv4 (the gate excludes dotted quads, or it fails on
192.168.0.180 in the agent's own file); and unconditional: true is NOT a Claude
Code feature but this project's own marker.

Workspace-root CLAUDE.md 208 -> 182 lines (142 effective), copy kept identical.
The nine-instance invariant table moved into the felhom-testing skill, which
triggers when writing or reviewing a test; all three directive bullets stayed in
the core. felhom.eu/CLAUDE.md got surgical corrections only and is knowingly still
over the ceiling at 227 effective lines -- closing it needs the restructure R-229
defers, said plainly rather than quietly absorbed.

CONTEXT.md gains standing ruling S-35. OPEN-ITEMS.md gains R-229.

Docs only -- no Go, no version bump, nothing built or deployed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2
This commit is contained in:
authentik Default Admin
2026-08-06 09:38:52 +02:00
parent d30c2a51ed
commit c21bcf84f7
10 changed files with 1125 additions and 333 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/CHANGELOG.md
+31
View File
@@ -1,3 +1,34 @@
## instructions_gate.py 1.0.0 — instruction files cannot silently regrow (2026-08-06, R-229)
New shared gate, registered in `controller_gates.py` and `agent_gates.py`, never copied into a
sibling repo (the `reuse_refs_check.py` precedent). `--fast` safe.
**It is a consistency gate, not a budget gate, and the failure message says so.** A `/context`
reading on 2026-08-06 measured the instruction files at 15k tokens against **869k free** in a 1M
window — space is not the constraint, and a future reader must not re-derive the wrong reason. The
200-line ceiling is Anthropic's *adherence* guidance, and a file nobody can hold in their head is
where contradictions hide: four were found in this project on the same day, two of which decided
where a destructive drill runs.
Checks, all against **effective** text (HTML comments stripped, because they are stripped before
injection): the line ceiling; every `.claude/rules/*.md` declares `paths:` or an explicit
`unconditional: true`; no component version literal; no TEMPORARY block carrying a past date; and the
workspace-root `CLAUDE.md` is byte-identical to its versioned copy — the live file sits outside any
git repo, so that copy is its only version-controlled record.
**Two traps found while building it, recorded so they are not reintroduced:**
- A bare `\d+\.\d+\.\d+` matches the first three octets of **every IPv4 address**. The gate
excludes dotted quads; without that it fails on `192.168.0.180` in the agent's own file.
- `unconditional: true` is **not** a Claude Code feature — it is this project's marker, asserting
that always-loading was deliberate. The docstring says so, so nobody looks for it in the product.
`test_instructions_gate.py`: 20 fixture assertions, all on the **effect** (exit code *and* that the
message names the file and the reason). Includes the load-bearing negatives — 400 commented lines
must not trip the ceiling, and a version literal inside a comment must be allowed — because the gate
must not punish the very move it exists to encourage. Companion red-proof recorded in the ledger:
ceiling temporarily set to 100 against the real trimmed files, `felhom-agent` (173) FAILED and was
named, `felhom-controller` (92) still passed; threshold restored, suite re-run green.
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
scripts/instructions_gate.py
+238
View File
@@ -0,0 +1,238 @@
# -*- coding: utf-8 -*-
"""Instruction-file consistency gate — keeps CLAUDE.md files from silently regrowing.
Usage: python3 scripts/instructions_gate.py <repo-root> [<repo-root> ...]
python3 scripts/instructions_gate.py --fast <repo-root> (identical: no network,
no container runtime)
THIS IS A CONSISTENCY GATE, NOT A BUDGET GATE. Measured 2026-08-06 on a 1M-token window: the
instruction files occupied 15k tokens against 869k free. Space is NOT the constraint here and no
failure message may claim it is. The line ceiling exists because longer instruction files reduce
ADHERENCE — Anthropic's guidance is to keep a CLAUDE.md under 200 lines — and because a file nobody
can hold in their head is where contradictions hide. Four were found in this project on 2026-08-06,
two of which decided where a destructive drill runs.
WHAT IS COUNTED. Every check runs against EFFECTIVE text: block-level HTML comments are stripped
first, because they are stripped before injection and never reach the model. Verified empirically on
Claude Code 2.1.222 with a control (two plain markers -> both seen) and a treatment (one marker
inside <!-- -->, twice -> not seen). That is the whole point of the comment convention: earned
rationale stays in the repo for human readers at zero cost to the instructions.
CHECKS (each names the file and the reason; a missing input is a FAILURE, never a skip):
1. <root>/CLAUDE.md is at most MAX_LINES effective lines.
2. Every <root>/.claude/rules/*.md declares `paths:` in frontmatter, or `unconditional: true`.
NOTE: `paths:` is a Claude Code feature — a rule carrying it loads only when a file matching
one of its globs is read. `unconditional: true` is NOT a product feature; it is OUR marker,
asserting that always-loading was deliberate. The product loads a rule with no `paths:` key
unconditionally either way, so this check catches the ACCIDENT, not the product behaviour.
3. No component version literal in effective text. Versions change several times a day and the
fleet is not uniform, so a version in an instruction file is stale within a day (ask the hub's
/hosts + /configs, or the box). Historical citations belong in an HTML comment beside the rule
they justify, where they inform a human and cannot go stale in the model's view.
IPv4 addresses are NOT versions — a bare \\d+\\.\\d+\\.\\d+ matches the first three octets of
every one of them, which is how this check would otherwise fail on its own repo.
4. No TEMPORARY block carrying a date already past. The block this gate was written for said
"TEMPORARY - until ~2026-08-02 ... Delete this block on return" and was still being read as
current fact on 2026-08-06, while a sibling repo's CLAUDE.md asserted the opposite.
5. The workspace-root CLAUDE.md and its versioned copy at
felhom.eu/documentation/runbooks/workspace-CLAUDE.md are byte-identical. The live file sits in
a directory that is not a git repo, so the copy is the only version-controlled record of it;
nothing but this check enforces that they agree.
THE POSITIVE OBSERVABLE. Every root prints a per-check tally with the measured numbers, not just a
verdict. "0 failures" alone cannot tell a working gate from a blind one — if the effective line
count suddenly reads 3, the stripper broke, and the count is where you see it.
The kill condition is pinned by scripts/test_instructions_gate.py: an over-length CLAUDE.md still
FAILS, and a rule file with neither marker still FAILS.
"""
import os
import re
import sys
MAX_LINES = 200
# Block-level HTML comments: stripped before injection, so they cost nothing and are not counted.
COMMENT_RE = re.compile(r"<!--.*?-->", re.S)
# A semver-ish literal that is NOT part of a dotted quad (IPv4) and not part of a longer run.
VERSION_RE = re.compile(r"(?<![\d.])\d+\.\d+\.\d+(?![\d.])")
# "TEMPORARY" anywhere on a line, plus an ISO date somewhere in that block.
TEMPORARY_RE = re.compile(r"TEMPORARY")
ISO_DATE_RE = re.compile(r"(\d{4})-(\d{2})-(\d{2})")
WORKSPACE_COPY = os.path.join(
"felhom.eu", "documentation", "runbooks", "workspace-CLAUDE.md"
)
def effective(text):
"""The text the model actually receives: HTML comments removed."""
return COMMENT_RE.sub("", text)
def today_tuple():
"""Local date as (y, m, d). Injectable via FELHOM_GATE_TODAY for the test suite."""
override = os.environ.get("FELHOM_GATE_TODAY")
if override:
m = ISO_DATE_RE.match(override.strip())
if m:
return tuple(int(g) for g in m.groups())
import datetime
d = datetime.date.today()
return (d.year, d.month, d.day)
def check_length(path, failures, tally):
with io_open(path) as fh:
eff = effective(fh.read())
n = eff.count("\n")
tally.append(" CLAUDE.md effective lines : %d (ceiling %d)" % (n, MAX_LINES))
if n > MAX_LINES:
failures.append(
"%s: %d effective lines, ceiling %d. This is an ADHERENCE limit, not a space "
"limit — long instruction files get followed less reliably and hide "
"contradictions. Move path-bound guidance into .claude/rules/*.md with a `paths:` "
"list, procedures into the skill that already covers them, and earned rationale "
"into an HTML comment (free: stripped before injection)."
% (path, n, MAX_LINES)
)
def check_rules(root, failures, tally):
rules_dir = os.path.join(root, ".claude", "rules")
if not os.path.isdir(rules_dir):
tally.append(" rule files : none (no .claude/rules/)")
return
names = sorted(n for n in os.listdir(rules_dir) if n.endswith(".md"))
scoped = 0
for name in names:
path = os.path.join(rules_dir, name)
with io_open(path) as fh:
head = fh.read(2048)
has_paths = re.search(r"^paths:", head, re.M) is not None
has_uncond = re.search(r"^unconditional:\s*true\s*$", head, re.M) is not None
if has_paths:
scoped += 1
elif not has_uncond:
failures.append(
"%s: rule file has neither a `paths:` frontmatter list nor an explicit "
"`unconditional: true`. Without `paths:` Claude Code loads it in EVERY session, "
"which is rarely what a rule file is for — add the globs it applies to, or "
"declare `unconditional: true` to say the always-loading is deliberate." % path
)
tally.append(
" rule files : %d (%d path-scoped)" % (len(names), scoped)
)
def check_versions(path, failures, tally):
with io_open(path) as fh:
eff = effective(fh.read())
hits = []
for i, line in enumerate(eff.split("\n"), 1):
for m in VERSION_RE.finditer(line):
hits.append((i, m.group(), line.strip()[:90]))
tally.append(" version literals : %d" % len(hits))
for lineno, ver, ctx in hits:
failures.append(
"%s:%d: component version literal %r — versions change several times a day and the "
"fleet is not uniform, so this is stale within a day. Ask the hub (/hosts, /configs) "
"or the box. A historical citation belongs in an HTML comment beside the rule it "
"justifies.\n %s" % (path, lineno, ver, ctx)
)
def check_temporary(path, failures, tally):
with io_open(path) as fh:
eff = effective(fh.read())
lines = eff.split("\n")
today = today_tuple()
found = 0
for i, line in enumerate(lines, 1):
if not TEMPORARY_RE.search(line):
continue
found += 1
window = "\n".join(lines[i - 1 : i + 6])
for m in ISO_DATE_RE.finditer(window):
when = tuple(int(g) for g in m.groups())
if when < today:
failures.append(
"%s:%d: TEMPORARY block carrying the past date %s. A temporary block that "
"outlives its own deadline is read as current fact — this gate exists "
"because one did, for four days, while a sibling repo asserted the "
"opposite. Delete it; the audit trail is the record."
% (path, i, m.group())
)
break
tally.append(" TEMPORARY blocks : %d" % found)
def check_workspace_copy(workspace_root, failures, tally):
live = os.path.join(workspace_root, "CLAUDE.md")
copy = os.path.join(workspace_root, WORKSPACE_COPY)
if not os.path.exists(live) or not os.path.exists(copy):
tally.append(" workspace copy : n/a (not this workspace)")
return
with io_open(live) as a, io_open(copy) as b:
same = a.read() == b.read()
tally.append(" workspace copy identical : %s" % ("yes" if same else "NO"))
if not same:
failures.append(
"%s and %s have diverged. The live workspace file sits in a directory that is not a "
"git repo, so the copy is its only version-controlled record — nothing but this "
"check enforces that they agree. Copy the live file over the versioned one."
% (live, copy)
)
def io_open(path):
return open(path, "r", encoding="utf-8")
def main(argv):
roots = [a for a in argv[1:] if a != "--fast"]
if not roots:
sys.stderr.write("usage: instructions_gate.py [--fast] <repo-root> ...\n")
return 2
failures = []
for root in roots:
root = os.path.abspath(root)
print("instructions_gate: %s" % root)
tally = []
claude_md = os.path.join(root, "CLAUDE.md")
if not os.path.exists(claude_md):
failures.append(
"%s: no CLAUDE.md. A missing input is a FAILURE, never a skip — a gate that "
"silently passes on an absent file is how the check stops running." % root
)
else:
check_length(claude_md, failures, tally)
check_versions(claude_md, failures, tally)
check_temporary(claude_md, failures, tally)
check_rules(root, failures, tally)
check_workspace_copy(os.path.dirname(root), failures, tally)
for line in tally:
print(line)
if failures:
print("")
print("instructions_gate: %d FAILURE(S)" % len(failures))
for f in failures:
print(" - %s" % f)
return 1
print("")
print("instructions_gate: OK")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/test_instructions_gate.py
+200
View File
@@ -0,0 +1,200 @@
# -*- coding: utf-8 -*-
"""Fixture tests for instructions_gate.py.
Run: python3 scripts/test_instructions_gate.py
Every test asserts the EFFECT — the gate's exit code AND that its message names the file and the
reason — not merely that "it ran". A gate that exits non-zero for the wrong reason is not a gate.
The negatives matter as much as the positives here: the whole point of the HTML-comment convention
is that commented content costs nothing, so `test_comments_do_not_count_toward_ceiling` and
`test_version_literal_in_comment_is_allowed` are what stop the gate from punishing the very move it
is meant to encourage.
"""
import os
import shutil
import subprocess
import sys
import tempfile
HERE = os.path.dirname(os.path.abspath(__file__))
GATE = os.path.join(HERE, "instructions_gate.py")
PASSED = []
FAILED = []
def run_gate(root, today="2026-08-06"):
env = dict(os.environ, FELHOM_GATE_TODAY=today)
p = subprocess.run(
[sys.executable, GATE, root],
capture_output=True,
text=True,
env=env,
)
return p.returncode, p.stdout + p.stderr
def make_repo(tmp, claude_md, rules=None):
root = os.path.join(tmp, "repo")
os.makedirs(root, exist_ok=True)
with open(os.path.join(root, "CLAUDE.md"), "w", encoding="utf-8") as fh:
fh.write(claude_md)
if rules:
rd = os.path.join(root, ".claude", "rules")
os.makedirs(rd, exist_ok=True)
for name, body in rules.items():
with open(os.path.join(rd, name), "w", encoding="utf-8") as fh:
fh.write(body)
return root
def check(name, cond, detail=""):
(PASSED if cond else FAILED).append(name + ((" — " + detail) if detail else ""))
print((" PASS " if cond else " FAIL ") + name + ((" " + detail) if detail else ""))
def test_short_file_passes():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(tmp, "# Repo\n" + "a line\n" * 150)
rc, out = run_gate(root)
check("150-line CLAUDE.md exits 0", rc == 0, "rc=%d" % rc)
def test_long_file_fails_and_names_file_and_count():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(tmp, "# Repo\n" + "a line\n" * 204)
rc, out = run_gate(root)
check("205-line CLAUDE.md exits non-zero", rc != 0, "rc=%d" % rc)
check("message names the file", "CLAUDE.md" in out)
check("message names the effective line count", "205 effective lines" in out)
check(
"message says ADHERENCE, not space",
"ADHERENCE limit, not a space limit" in out,
)
def test_comments_do_not_count_toward_ceiling():
"""The convention's load-bearing negative: 400 commented lines must not trip the ceiling."""
with tempfile.TemporaryDirectory() as tmp:
body = "# Repo\n" + "a line\n" * 100 + "<!--\n" + "history\n" * 400 + "-->\n"
root = make_repo(tmp, body)
rc, out = run_gate(root)
check("400 commented lines do not trip the ceiling", rc == 0, "rc=%d" % rc)
def test_rule_with_paths_passes():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(
tmp,
"# Repo\n",
rules={"scoped.md": '---\npaths: ["**/*.go"]\n---\n\n# Scoped\n'},
)
rc, out = run_gate(root)
check("rule with paths: exits 0", rc == 0, "rc=%d" % rc)
def test_rule_without_paths_fails():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(
tmp, "# Repo\n", rules={"bare.md": "---\nname: bare\n---\n\n# Bare\n"}
)
rc, out = run_gate(root)
check("rule with neither marker exits non-zero", rc != 0, "rc=%d" % rc)
check("message names the rule file", "bare.md" in out)
def test_rule_with_unconditional_marker_passes():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(
tmp,
"# Repo\n",
rules={"always.md": "---\nunconditional: true\n---\n\n# Always\n"},
)
rc, out = run_gate(root)
check("explicit unconditional: true exits 0", rc == 0, "rc=%d" % rc)
def test_version_literal_fails():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(tmp, "# Repo\n\nThe box runs agent 0.93.0 today.\n")
rc, out = run_gate(root)
check("version literal exits non-zero", rc != 0, "rc=%d" % rc)
check("message quotes the version", "0.93.0" in out)
def test_ipv4_is_not_a_version():
"""A bare \\d+\\.\\d+\\.\\d+ matches the first three octets of every IPv4."""
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(tmp, "# Repo\n\nDooPlex is 192.168.0.180 and the demo box is 10.0.0.1.\n")
rc, out = run_gate(root)
check("IPv4 addresses are not flagged as versions", rc == 0, "rc=%d" % rc)
def test_version_literal_in_comment_is_allowed():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(tmp, "# Repo\n\n<!-- fixed in hub v0.97.0 -->\n")
rc, out = run_gate(root)
check("version literal inside a comment is allowed", rc == 0, "rc=%d" % rc)
def test_expired_temporary_fails():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(
tmp,
"# Repo\n\n> **TEMPORARY — host is away (until ~2026-08-02).**\n> Delete on return.\n",
)
rc, out = run_gate(root)
check("expired TEMPORARY block exits non-zero", rc != 0, "rc=%d" % rc)
check("message quotes the past date", "2026-08-02" in out)
def test_future_temporary_passes():
with tempfile.TemporaryDirectory() as tmp:
root = make_repo(
tmp, "# Repo\n\n> **TEMPORARY — host is away (until ~2026-12-31).**\n"
)
rc, out = run_gate(root)
check("TEMPORARY with a future date exits 0", rc == 0, "rc=%d" % rc)
def test_missing_claude_md_fails():
with tempfile.TemporaryDirectory() as tmp:
root = os.path.join(tmp, "repo")
os.makedirs(root)
rc, out = run_gate(root)
check("absent CLAUDE.md is a FAILURE, not a skip", rc != 0, "rc=%d" % rc)
def test_diverged_workspace_copy_fails():
with tempfile.TemporaryDirectory() as tmp:
ws = os.path.join(tmp, "ws")
root = os.path.join(ws, "repo")
os.makedirs(root)
with open(os.path.join(root, "CLAUDE.md"), "w", encoding="utf-8") as fh:
fh.write("# Repo\n")
with open(os.path.join(ws, "CLAUDE.md"), "w", encoding="utf-8") as fh:
fh.write("# Workspace live\n")
cp = os.path.join(ws, "felhom.eu", "documentation", "runbooks")
os.makedirs(cp)
with open(os.path.join(cp, "workspace-CLAUDE.md"), "w", encoding="utf-8") as fh:
fh.write("# Workspace copy — DIVERGED\n")
rc, out = run_gate(root)
check("diverged workspace copy exits non-zero", rc != 0, "rc=%d" % rc)
check("message names both files", "workspace-CLAUDE.md" in out)
def main():
print("test_instructions_gate")
for fn in sorted(
(v for k, v in globals().items() if k.startswith("test_")),
key=lambda f: f.__name__,
):
fn()
print("")
print("passed: %d failed: %d" % (len(PASSED), len(FAILED)))
return 1 if FAILED else 0
if __name__ == "__main__":
sys.exit(main())