admin
f27aed87cd
gates / gates (push) Successful in 15s
MEMORY.md is the larger half of what loads before a word is typed (8.4k tokens vs the root CLAUDE.md's 6.6k) and is the one instruction file nobody hand-edits, so nothing was watching it. Three deliberately different outcomes, each pinned by a test: over-ceiling FAILS (auto-memory drops content past the limit with no error), an orphan WARNS (the store is outside git), and an absent store PASSES while PRINTING its reason -- asserted on the reason text, because a pass with no reason is indistinguishable from a gate that stopped running. 39 assertions (was 20). Red-proof run against the real store, not a fixture.
364 lines
16 KiB
Python
364 lines
16 KiB
Python
# -*- coding: utf-8 -*-
|
|
"""Instruction-file consistency gate — keeps CLAUDE.md files from silently regrowing.
|
|
|
|
Usage: python3 scripts/instructions_gate.py <repo-root> [<repo-root> ...]
|
|
python3 scripts/instructions_gate.py --fast <repo-root> (identical: no network,
|
|
no container runtime)
|
|
|
|
THIS IS A CONSISTENCY GATE, NOT A BUDGET GATE. Measured 2026-08-06 on a 1M-token window: the
|
|
instruction files occupied 15k tokens against 869k free. Space is NOT the constraint here and no
|
|
failure message may claim it is. The line ceiling exists because longer instruction files reduce
|
|
ADHERENCE — Anthropic's guidance is to keep a CLAUDE.md under 200 lines — and because a file nobody
|
|
can hold in their head is where contradictions hide. Four were found in this project on 2026-08-06,
|
|
two of which decided where a destructive drill runs.
|
|
|
|
WHAT IS COUNTED. Every check runs against EFFECTIVE text: block-level HTML comments are stripped
|
|
first, because they are stripped before injection and never reach the model. Verified empirically on
|
|
Claude Code 2.1.222 with a control (two plain markers -> both seen) and a treatment (one marker
|
|
inside <!-- -->, twice -> not seen). That is the whole point of the comment convention: earned
|
|
rationale stays in the repo for human readers at zero cost to the instructions.
|
|
|
|
CHECKS (each names the file and the reason; a missing input is a FAILURE, never a skip):
|
|
|
|
1. <root>/CLAUDE.md is at most MAX_LINES effective lines.
|
|
2. Every <root>/.claude/rules/*.md declares `paths:` in frontmatter, or `unconditional: true`.
|
|
NOTE: `paths:` is a Claude Code feature — a rule carrying it loads only when a file matching
|
|
one of its globs is read. `unconditional: true` is NOT a product feature; it is OUR marker,
|
|
asserting that always-loading was deliberate. The product loads a rule with no `paths:` key
|
|
unconditionally either way, so this check catches the ACCIDENT, not the product behaviour.
|
|
3. No component version literal in effective text. Versions change several times a day and the
|
|
fleet is not uniform, so a version in an instruction file is stale within a day (ask the hub's
|
|
/hosts + /configs, or the box). Historical citations belong in an HTML comment beside the rule
|
|
they justify, where they inform a human and cannot go stale in the model's view.
|
|
IPv4 addresses are NOT versions — a bare \\d+\\.\\d+\\.\\d+ matches the first three octets of
|
|
every one of them, which is how this check would otherwise fail on its own repo.
|
|
4. No TEMPORARY block carrying a date already past. The block this gate was written for said
|
|
"TEMPORARY - until ~2026-08-02 ... Delete this block on return" and was still being read as
|
|
current fact on 2026-08-06, while a sibling repo's CLAUDE.md asserted the opposite.
|
|
5. The workspace-root CLAUDE.md and its versioned copy at
|
|
felhom.eu/documentation/runbooks/workspace-CLAUDE.md are byte-identical. The live file sits in
|
|
a directory that is not a git repo, so the copy is the only version-controlled record of it;
|
|
nothing but this check enforces that they agree.
|
|
6. The auto-memory index (<workspace>/.claude-memory/MEMORY.md) is within its line and byte
|
|
limits, and every top-level topic file is referenced by it.
|
|
|
|
WHY 6 EXISTS, AND WHY ITS THREE OUTCOMES DIFFER (2026-08-06).
|
|
|
|
MEMORY.md is the LARGER half of what loads before a word is typed: measured at the workspace root,
|
|
the hand-written root CLAUDE.md was 6.6k tokens and MEMORY.md 8.4k. It is also the one instruction
|
|
file nobody hand-edits — Claude writes it — so nothing was watching it.
|
|
|
|
- OVER THE LIMIT IS A FAILURE. Content past the auto-memory limit is dropped with NO error. A
|
|
silent truncation of the index is the failure mode with no observable at all, which is exactly
|
|
the class this project keeps getting bitten by.
|
|
- AN ORPHAN IS A WARNING, NOT A FAILURE. The store changes between sessions and lives outside git;
|
|
a hard fail would block pushes for something no commit can fix. On 2026-08-06 the index
|
|
referenced 113 files while 157 existed — 44 held knowledge nothing would ever read.
|
|
- AN ABSENT STORE PASSES, AND SAYS SO OUT LOUD. This is a DELIBERATE exception to check 1's
|
|
"a missing input is a FAILURE, never a skip": the store is machine-local by design and a clone
|
|
on any other host legitimately has none. The reason is printed in the tally so an absent store
|
|
can never be mistaken for a silent skip — which is the only thing that made the exception safe
|
|
to grant.
|
|
|
|
THE POSITIVE OBSERVABLE. Every root prints a per-check tally with the measured numbers, not just a
|
|
verdict. "0 failures" alone cannot tell a working gate from a blind one — if the effective line
|
|
count suddenly reads 3, the stripper broke, and the count is where you see it.
|
|
|
|
The kill condition is pinned by scripts/test_instructions_gate.py: an over-length CLAUDE.md still
|
|
FAILS, and a rule file with neither marker still FAILS.
|
|
"""
|
|
import os
|
|
import re
|
|
import sys
|
|
|
|
MAX_LINES = 200
|
|
|
|
# The auto-memory index. Limits are the store's own, not this project's taste: content past them is
|
|
# dropped with no error.
|
|
MEMORY_DIRNAME = ".claude-memory"
|
|
MEMORY_INDEX = "MEMORY.md"
|
|
MEMORY_ARCHIVE = "archive"
|
|
MAX_MEMORY_LINES = 200
|
|
MAX_MEMORY_BYTES = 25 * 1024
|
|
|
|
# Block-level HTML comments: stripped before injection, so they cost nothing and are not counted.
|
|
COMMENT_RE = re.compile(r"<!--.*?-->", re.S)
|
|
|
|
# A semver-ish literal that is NOT part of a dotted quad (IPv4) and not part of a longer run.
|
|
VERSION_RE = re.compile(r"(?<![\d.])\d+\.\d+\.\d+(?![\d.])")
|
|
|
|
# "TEMPORARY" anywhere on a line, plus an ISO date somewhere in that block.
|
|
TEMPORARY_RE = re.compile(r"TEMPORARY")
|
|
ISO_DATE_RE = re.compile(r"(\d{4})-(\d{2})-(\d{2})")
|
|
|
|
WORKSPACE_COPY = os.path.join(
|
|
"felhom.eu", "documentation", "runbooks", "workspace-CLAUDE.md"
|
|
)
|
|
|
|
|
|
def effective(text):
|
|
"""The text the model actually receives: HTML comments removed."""
|
|
return COMMENT_RE.sub("", text)
|
|
|
|
|
|
def today_tuple():
|
|
"""Local date as (y, m, d). Injectable via FELHOM_GATE_TODAY for the test suite."""
|
|
override = os.environ.get("FELHOM_GATE_TODAY")
|
|
if override:
|
|
m = ISO_DATE_RE.match(override.strip())
|
|
if m:
|
|
return tuple(int(g) for g in m.groups())
|
|
import datetime
|
|
|
|
d = datetime.date.today()
|
|
return (d.year, d.month, d.day)
|
|
|
|
|
|
def check_length(path, failures, tally):
|
|
with io_open(path) as fh:
|
|
eff = effective(fh.read())
|
|
n = eff.count("\n")
|
|
tally.append(" CLAUDE.md effective lines : %d (ceiling %d)" % (n, MAX_LINES))
|
|
if n > MAX_LINES:
|
|
failures.append(
|
|
"%s: %d effective lines, ceiling %d. This is an ADHERENCE limit, not a space "
|
|
"limit — long instruction files get followed less reliably and hide "
|
|
"contradictions. Move path-bound guidance into .claude/rules/*.md with a `paths:` "
|
|
"list, procedures into the skill that already covers them, and earned rationale "
|
|
"into an HTML comment (free: stripped before injection)."
|
|
% (path, n, MAX_LINES)
|
|
)
|
|
|
|
|
|
def check_rules(root, failures, tally):
|
|
rules_dir = os.path.join(root, ".claude", "rules")
|
|
if not os.path.isdir(rules_dir):
|
|
tally.append(" rule files : none (no .claude/rules/)")
|
|
return
|
|
names = sorted(n for n in os.listdir(rules_dir) if n.endswith(".md"))
|
|
scoped = 0
|
|
for name in names:
|
|
path = os.path.join(rules_dir, name)
|
|
with io_open(path) as fh:
|
|
head = fh.read(2048)
|
|
has_paths = re.search(r"^paths:", head, re.M) is not None
|
|
has_uncond = re.search(r"^unconditional:\s*true\s*$", head, re.M) is not None
|
|
if has_paths:
|
|
scoped += 1
|
|
elif not has_uncond:
|
|
failures.append(
|
|
"%s: rule file has neither a `paths:` frontmatter list nor an explicit "
|
|
"`unconditional: true`. Without `paths:` Claude Code loads it in EVERY session, "
|
|
"which is rarely what a rule file is for — add the globs it applies to, or "
|
|
"declare `unconditional: true` to say the always-loading is deliberate." % path
|
|
)
|
|
tally.append(
|
|
" rule files : %d (%d path-scoped)" % (len(names), scoped)
|
|
)
|
|
|
|
|
|
def check_versions(path, failures, tally):
|
|
with io_open(path) as fh:
|
|
eff = effective(fh.read())
|
|
hits = []
|
|
for i, line in enumerate(eff.split("\n"), 1):
|
|
for m in VERSION_RE.finditer(line):
|
|
hits.append((i, m.group(), line.strip()[:90]))
|
|
tally.append(" version literals : %d" % len(hits))
|
|
for lineno, ver, ctx in hits:
|
|
failures.append(
|
|
"%s:%d: component version literal %r — versions change several times a day and the "
|
|
"fleet is not uniform, so this is stale within a day. Ask the hub (/hosts, /configs) "
|
|
"or the box. A historical citation belongs in an HTML comment beside the rule it "
|
|
"justifies.\n %s" % (path, lineno, ver, ctx)
|
|
)
|
|
|
|
|
|
def check_temporary(path, failures, tally):
|
|
with io_open(path) as fh:
|
|
eff = effective(fh.read())
|
|
lines = eff.split("\n")
|
|
today = today_tuple()
|
|
found = 0
|
|
for i, line in enumerate(lines, 1):
|
|
if not TEMPORARY_RE.search(line):
|
|
continue
|
|
found += 1
|
|
window = "\n".join(lines[i - 1 : i + 6])
|
|
for m in ISO_DATE_RE.finditer(window):
|
|
when = tuple(int(g) for g in m.groups())
|
|
if when < today:
|
|
failures.append(
|
|
"%s:%d: TEMPORARY block carrying the past date %s. A temporary block that "
|
|
"outlives its own deadline is read as current fact — this gate exists "
|
|
"because one did, for four days, while a sibling repo asserted the "
|
|
"opposite. Delete it; the audit trail is the record."
|
|
% (path, i, m.group())
|
|
)
|
|
break
|
|
tally.append(" TEMPORARY blocks : %d" % found)
|
|
|
|
|
|
def check_workspace_copy(workspace_root, failures, tally):
|
|
live = os.path.join(workspace_root, "CLAUDE.md")
|
|
copy = os.path.join(workspace_root, WORKSPACE_COPY)
|
|
if not os.path.exists(live) or not os.path.exists(copy):
|
|
tally.append(" workspace copy : n/a (not this workspace)")
|
|
return
|
|
with io_open(live) as a, io_open(copy) as b:
|
|
same = a.read() == b.read()
|
|
tally.append(" workspace copy identical : %s" % ("yes" if same else "NO"))
|
|
if not same:
|
|
failures.append(
|
|
"%s and %s have diverged. The live workspace file sits in a directory that is not a "
|
|
"git repo, so the copy is its only version-controlled record — nothing but this "
|
|
"check enforces that they agree. Copy the live file over the versioned one."
|
|
% (live, copy)
|
|
)
|
|
|
|
|
|
def memory_referenced(index_text):
|
|
"""Basenames the index points at, via [](x.md) links and [[wikilink]]s alike.
|
|
|
|
Both forms count as a reference. A file reachable only by a wikilink is indexed — it just is not
|
|
a listed entry — and calling it an orphan would push someone to add a duplicate row for it.
|
|
"""
|
|
md = re.findall(r"\]\(([^)]+\.md)\)", index_text)
|
|
wiki = re.findall(r"\[\[([^\]]+)\]\]", index_text)
|
|
names = set(os.path.basename(x) for x in md)
|
|
for w in wiki:
|
|
names.add(os.path.basename(w if w.endswith(".md") else w + ".md"))
|
|
return names
|
|
|
|
|
|
def check_memory(workspace_root, failures, warnings, tally):
|
|
store = os.path.join(workspace_root, MEMORY_DIRNAME)
|
|
if not os.path.isdir(store):
|
|
# DELIBERATE exception to "a missing input is a FAILURE" — see the module docstring. The
|
|
# reason is printed so this can never read as a silent skip.
|
|
tally.append(
|
|
" memory index : absent (%s/ not on this host — the auto-memory store is "
|
|
"machine-local by design, so a clone elsewhere legitimately has none; PASS with reason)"
|
|
% MEMORY_DIRNAME
|
|
)
|
|
return
|
|
|
|
index = os.path.join(store, MEMORY_INDEX)
|
|
if not os.path.exists(index):
|
|
failures.append(
|
|
"%s: the memory store exists but has no %s. An un-indexed store is unreachable "
|
|
"knowledge — every topic file is read on demand, and the index is the only thing that "
|
|
"names them." % (store, MEMORY_INDEX)
|
|
)
|
|
tally.append(" memory index : MISSING")
|
|
return
|
|
|
|
with io_open(index) as fh:
|
|
text = fh.read()
|
|
nlines = text.count("\n")
|
|
nbytes = len(text.encode("utf-8"))
|
|
tally.append(
|
|
" memory index : %d lines (ceiling %d), %d bytes (ceiling %d)"
|
|
% (nlines, MAX_MEMORY_LINES, nbytes, MAX_MEMORY_BYTES)
|
|
)
|
|
|
|
if nlines > MAX_MEMORY_LINES:
|
|
failures.append(
|
|
"%s: %d lines, ceiling %d. Content past the auto-memory limit is DROPPED WITH NO "
|
|
"ERROR — a truncated index is a silent failure with no observable. Move detail out of "
|
|
"the index into the topic files it points at; the index is what loads, the topic files "
|
|
"are read on demand." % (index, nlines, MAX_MEMORY_LINES)
|
|
)
|
|
if nbytes > MAX_MEMORY_BYTES:
|
|
failures.append(
|
|
"%s: %d bytes, ceiling %d. Same reason as the line ceiling — silent truncation. Move "
|
|
"detail into topic files rather than dropping entries."
|
|
% (index, nbytes, MAX_MEMORY_BYTES)
|
|
)
|
|
|
|
referenced = memory_referenced(text)
|
|
on_disk = set(
|
|
n for n in os.listdir(store)
|
|
if n.endswith(".md") and n != MEMORY_INDEX
|
|
and os.path.isfile(os.path.join(store, n))
|
|
)
|
|
orphans = sorted(on_disk - referenced)
|
|
archived = 0
|
|
arc = os.path.join(store, MEMORY_ARCHIVE)
|
|
if os.path.isdir(arc):
|
|
archived = len([n for n in os.listdir(arc) if n.endswith(".md")])
|
|
tally.append(
|
|
" memory topic files : %d indexed, %d orphaned, %d archived"
|
|
% (len(on_disk) - len(orphans), len(orphans), archived)
|
|
)
|
|
if orphans:
|
|
# WARN, never FAIL: the store is outside git and changes between sessions, so a failure
|
|
# here would block pushes for something no commit can fix.
|
|
warnings.append(
|
|
"%s: %d top-level topic file(s) are not referenced by %s, so nothing will ever read "
|
|
"them: %s%s"
|
|
% (
|
|
store,
|
|
len(orphans),
|
|
MEMORY_INDEX,
|
|
", ".join(orphans[:8]),
|
|
(" (+%d more)" % (len(orphans) - 8)) if len(orphans) > 8 else "",
|
|
)
|
|
)
|
|
|
|
|
|
def io_open(path):
|
|
return open(path, "r", encoding="utf-8")
|
|
|
|
|
|
def main(argv):
|
|
roots = [a for a in argv[1:] if a != "--fast"]
|
|
if not roots:
|
|
sys.stderr.write("usage: instructions_gate.py [--fast] <repo-root> ...\n")
|
|
return 2
|
|
|
|
failures = []
|
|
warnings = []
|
|
for root in roots:
|
|
root = os.path.abspath(root)
|
|
print("instructions_gate: %s" % root)
|
|
tally = []
|
|
|
|
claude_md = os.path.join(root, "CLAUDE.md")
|
|
if not os.path.exists(claude_md):
|
|
failures.append(
|
|
"%s: no CLAUDE.md. A missing input is a FAILURE, never a skip — a gate that "
|
|
"silently passes on an absent file is how the check stops running." % root
|
|
)
|
|
else:
|
|
check_length(claude_md, failures, tally)
|
|
check_versions(claude_md, failures, tally)
|
|
check_temporary(claude_md, failures, tally)
|
|
|
|
check_rules(root, failures, tally)
|
|
check_workspace_copy(os.path.dirname(root), failures, tally)
|
|
check_memory(os.path.dirname(root), failures, warnings, tally)
|
|
|
|
for line in tally:
|
|
print(line)
|
|
|
|
if warnings:
|
|
print("")
|
|
for w in warnings:
|
|
print("WARNING: %s" % w)
|
|
|
|
if failures:
|
|
print("")
|
|
print("instructions_gate: %d FAILURE(S)" % len(failures))
|
|
for f in failures:
|
|
print(" - %s" % f)
|
|
return 1
|
|
|
|
print("")
|
|
print("instructions_gate: OK%s" % (" (%d warning(s))" % len(warnings) if warnings else ""))
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main(sys.argv))
|