admin/app-catalog-felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e81d41e5270ac9205acd10a4f3fbbdf07a6402f1
app-catalog-felhom.eu/scripts/test_gate_decoys.py
T
admin e81d41e527
gates / gates (push) Failing after 2s
Details
English copy — the PILOT: privatebin, paperless-ngx, romm (R-560 slice 5) 
89 of 1 032 strings. No Hungarian byte moved; no image, pin, catalog_since or
compose line changed. EN_MISSING_CEILING 1032 -> 943 in this commit.

Chosen for SHAPE: privatebin exercises the plain case (description, tagline, lists);
paperless-ngx adds select options, a placeholder and a customer-facing folder label;
romm carries the catalog's only optional_config block, whose group has no id of its
own and is matched by `match_group` — the Hungarian group name it translates. All
three run on the demo box, so the English pages can be fetched rather than reasoned
about.

Two gate defects fixed while translating, each found by its own decoy rather than by
reading: coverage was counted only for the apps NAMED on the command line, so
`check-copy-i18n.py privatebin` reported 47 more missing strings than the same tree
unscoped and either number could have been made to "pass"; and the ASCII-Hungarian
stems matched as bare substrings, so „ird be" convicted "the third best" and „angol"
convicted "Angola". Both now have their own case in the decoy suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-20 14:34:22 +02:00

405 lines
22 KiB
Python
Raw Blame History

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""test_gate_decoys.py — can this repo's gates be fooled by a LABEL? (R-421)
The same instrument as `felhom.eu/scripts/test_gate_decoys.py`: a decoy is the LABEL without the
FACT, and a gate that convicts on the label alone — or fails to convict on the fact — is a live hole.
Every case asserts BOTH directions where it can: the genuine article must pass and the decoy must be
judged on what it IS, not on what it says.
Covered here (the `COVERS` literal is AST-read by `felhom.eu/scripts/decoy_coverage_gate.py`):
engine-major — `check-engine-major.py` refuses a database-engine pin that crosses a MAJOR.
Its label is the version string; its fact is the `image:` line of an engine SERVICE. Decoys a
real session would produce:
* the major moves in a COMMENT and in kimai's `serverVersion=11.6.2-MariaDB` env var, while
the image line stays — must PASS (nothing moved);
* the APP's own image crosses a major (kimai 2.57 -> 3.0) — must PASS (not an engine);
* a `mariadb:12.3` string lands in README.md — must PASS (not a template);
* the engine moves WITHIN its major (11.6 -> 11.8) — must PASS (the rule says MAJOR);
and the facts:
* `mariadb:11.6 -> mariadb:12.3` on `kimai-db` — must be REFUSED (exit 1), naming the rule
and its expiry (R-448);
* `postgres:16-alpine -> postgres:17-alpine` on `docmost-postgres` — must be REFUSED (the
eleven PostgreSQL services are covered by NAME MATCH, not by a list);
* `mariadb:11.6 -> mariadb:lts` — INCONCLUSIVE (exit 2), never 0: a major nobody can read is
not a pass.
HOW. The repo is cloned into a scratch directory; the WORKING-TREE gate is run inside the clone
(so the file under test is the one being edited, not HEAD's); each case is one commit on top of the
clone's HEAD and the gate is run with `--range HEAD~1..HEAD`. The real tree is never touched.
Run from the repo root: python3 scripts/test_gate_decoys.py
Exit 0 every decoy judged correctly · 1 a decoy passed or a genuine article was refused.
"""
import io
import os
import re
import shutil
import subprocess
import sys
import tempfile
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# ── WHAT THIS FILE COVERS ────────────────────────────────────────────────────────────────────────
# Read by felhom.eu/scripts/decoy_coverage_gate.py, which AST-parses this literal. A gate named here
# MUST have a decoy below that has been seen to fail.
COVERS = {
"engine-major": "the major moved in a comment/env var/README/app image, not on an engine's image: line",
"catalog-since": "the date bumped in a comment/README while .felhom.yml's field stayed; or only a comment/env moved, no image (R-452)",
"copy-i18n": "Hungarian edited in a COMMENT/README/display_name (label, not copy) vs a real frozen string changed; an English block that is not English, is not matched to a Hungarian twin, or rewrites a credential (R-560)",
}
fails = []
ran = 0
def sh(args, cwd):
return subprocess.run(args, cwd=cwd, capture_output=True, text=True)
def make_clone():
tmp = tempfile.mkdtemp(prefix="catalog-decoys-")
r = sh(["git", "clone", "-q", "file://" + ROOT, tmp], cwd=ROOT)
if r.returncode != 0:
raise SystemExit("clone failed: " + r.stderr)
sh(["git", "config", "user.email", "decoy@gate.invalid"], cwd=tmp)
sh(["git", "config", "user.name", "decoy"], cwd=tmp)
return tmp
def edit(clone, relpath, fn):
p = os.path.join(clone, relpath)
os.makedirs(os.path.dirname(p), exist_ok=True) # a case may ADD a file (a new app directory)
text = io.open(p, encoding="utf-8").read() if os.path.exists(p) else ""
new = fn(text)
if new == text:
raise SystemExit("case did not change %s — the case is broken, not the gate" % relpath)
with io.open(p, "w", encoding="utf-8") as fh:
fh.write(new)
def commit(clone, msg):
sh(["git", "add", "-A"], cwd=clone)
r = sh(["git", "commit", "-q", "-m", msg], cwd=clone)
if r.returncode != 0:
raise SystemExit("commit failed: " + r.stderr)
def reset(clone):
sh(["git", "reset", "-q", "--hard", "HEAD"], cwd=clone)
def case(name, clone, edits, expect_rc, must_contain=(), gate="check-engine-major.py"):
"""edits: list of (relpath, fn). Commits them, runs the gate on HEAD~1..HEAD, restores."""
global ran
ran += 1
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
try:
for relpath, fn in edits:
edit(clone, relpath, fn)
commit(clone, name)
# the WORKING-TREE gate, run inside the clone (it reads git from its cwd)
r = sh([sys.executable, os.path.join(ROOT, "scripts", gate),
"--range", "HEAD~1..HEAD"], cwd=clone)
out = r.stdout + r.stderr
ok = r.returncode == expect_rc and all(m in out for m in must_contain)
if ok:
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
else:
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
name, r.returncode, expect_rc,
[m for m in must_contain if m not in out], out[-900:]))
return out
finally:
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
def case_copy(name, clone, edits, expect_rc, must_contain=(), extra_args=()):
"""The copy-i18n gate reads FILES, not commits, so its cases need neither a commit nor a range —
but they DO need --root, or the gate would read the real repo and judge files nobody edited.
That is the `constant-for-measurement` decoy shape, and it would make every case below pass."""
global ran
ran += 1
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
try:
for relpath, fn in edits:
edit(clone, relpath, fn)
args = list(extra_args)
if "--expect-missing" not in args:
# MEASURE the clone's current coverage and hand it back as the ceiling. The ratchet is
# not what these cases test — they test the freeze, the structure and the language —
# and hard-coding a number here would make every case fail the day a batch lands.
# The ratchet has its own two cases below, in both directions.
probe = sh([sys.executable, os.path.join(ROOT, "scripts", "check-copy-i18n.py"),
"--root", clone, "--expect-missing", "-1"], cwd=clone)
m = re.search(r"(\d+) strings have no English", probe.stdout + probe.stderr)
if not m:
fails.append("%s: could not measure the clone's coverage — the case is broken, "
"not the gate" % name)
return ""
args += ["--expect-missing", m.group(1)]
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-copy-i18n.py"),
"--root", clone] + args, cwd=clone)
out = r.stdout + r.stderr
if r.returncode == expect_rc and all(m in out for m in must_contain):
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
else:
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
name, r.returncode, expect_rc,
[m for m in must_contain if m not in out], out[-900:]))
return out
finally:
sh(["git", "checkout", "-q", "--", "."], cwd=clone)
sh(["git", "clean", "-qfd"], cwd=clone)
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
def swap_image(service, frm, to):
"""Change ONLY the named service's own image: line — the same per-service discipline as the
gate, so the case moves the fact and nothing else."""
def _fn(text):
out, cur, done = [], None, False
for line in text.splitlines():
m = re.match(r"^ ([A-Za-z0-9_-]+):\s*$", line)
if m:
cur = m.group(1)
mi = re.match(r"^(\s+image:\s*)(\S+)\s*$", line)
if mi and cur == service and mi.group(2) == frm:
line = mi.group(1) + to
done = True
out.append(line)
if not done:
raise SystemExit("%s does not carry image %s — fixture drifted" % (service, frm))
return "\n".join(out) + "\n"
return _fn
def main():
gate = os.path.join(ROOT, "scripts", "check-engine-major.py")
if not os.path.isfile(gate):
print("FAIL: scripts/check-engine-major.py is missing — a failure, never a skip")
return 1
clone = make_clone()
try:
KIMAI = "templates/kimai/docker-compose.yml"
DOCMOST = "templates/docmost/docker-compose.yml"
# ── THE FACTS: these must be refused ─────────────────────────────────────────────────
out = case("FACT: kimai-db mariadb:11.6 -> 12.3 (cross-major)", clone,
[(KIMAI, swap_image("kimai-db", "mariadb:11.6", "mariadb:12.3"))],
expect_rc=1,
must_contain=("ENGINE-MAJOR GATE FAILED", "kimai-db", "mariadb 11 -> 12",
"R-448", "EXPIRY"))
if "REFUSAL_TEXT" in os.environ:
print(out)
case("FACT: docmost-postgres postgres:16-alpine -> 17-alpine", clone,
[(DOCMOST, swap_image("docmost-postgres", "postgres:16-alpine", "postgres:17-alpine"))],
expect_rc=1, must_contain=("docmost-postgres", "postgres 16 -> 17"))
case("FACT: kimai-db mariadb:11.6 -> mariadb:lts (major unreadable)", clone,
[(KIMAI, swap_image("kimai-db", "mariadb:11.6", "mariadb:lts"))],
expect_rc=2, must_contain=("INCONCLUSIVE",))
# ── THE GENUINE ARTICLES: these must pass ────────────────────────────────────────────
case("GENUINE: kimai-db mariadb:11.6 -> 11.8 (within major)", clone,
[(KIMAI, swap_image("kimai-db", "mariadb:11.6", "mariadb:11.8"))],
expect_rc=0, must_contain=("engine-major gate OK",))
# ── THE DECOYS: the label moves, the fact does not — these must pass ─────────────────
def comment_and_env(text):
# the version string moves in a COMMENT and in kimai's serverVersion env, image untouched
t = text.replace("serverVersion=11.6.2-MariaDB", "serverVersion=12.3.0-MariaDB")
return t.replace("# Database: mariadb", "# Database: mariadb (image: mariadb:12.3 soon)")
case("DECOY: major moves only in a comment + serverVersion env", clone,
[(KIMAI, comment_and_env)], expect_rc=0, must_contain=("engine-major gate OK",))
case("DECOY: the APP image crosses a major (kimai 2.57 -> 3.0)", clone,
[(KIMAI, swap_image("kimai", "kimai/kimai2:apache-2.57.0", "kimai/kimai2:apache-3.0.0"))],
expect_rc=0, must_contain=("engine-major gate OK",))
case("DECOY: 'mariadb:12.3' lands in README.md, not a template", clone,
[("README.md", lambda t: t + "\nDecoy: mariadb:11.6 -> mariadb:12.3 pending.\n")],
expect_rc=0, must_contain=("0 compose file(s) changed",))
# ── catalog-since (R-452): an image move must bump the app's catalog_since ───────────
import datetime
today = datetime.date.today().isoformat()
KIMAI_FY = "templates/kimai/.felhom.yml"
CS = "check-catalog-since.py"
def set_since(date):
def _fn(text):
new = re.sub(r'^catalog_since:\s*"?\d{4}-\d{2}-\d{2}"?', 'catalog_since: "%s"' % date, text, count=1, flags=re.M)
if new == text:
raise SystemExit("kimai's .felhom.yml carries no catalog_since — fixture drifted")
return new
return _fn
case("FACT: kimai image moves, catalog_since untouched", clone,
[(KIMAI, swap_image("kimai", "kimai/kimai2:apache-2.57.0", "kimai/kimai2:apache-2.58.0"))],
expect_rc=1, must_contain=("CATALOG-SINCE GATE FAILED", "kimai", "catalog_since is still"), gate=CS)
case("FACT: kimai image moves, catalog_since set to a FUTURE year", clone,
[(KIMAI, swap_image("kimai", "kimai/kimai2:apache-2.57.0", "kimai/kimai2:apache-2.58.0")),
(KIMAI_FY, set_since("2036-09-13"))],
expect_rc=1, must_contain=("in the future",), gate=CS)
case("GENUINE: kimai image moves AND catalog_since = today", clone,
[(KIMAI, swap_image("kimai", "kimai/kimai2:apache-2.57.0", "kimai/kimai2:apache-2.58.0")),
(KIMAI_FY, set_since(today))],
expect_rc=0, must_contain=("catalog-since gate OK", "1 image move(s) dated"), gate=CS)
case("DECOY: image moves; today's date lands in a COMMENT and README, the field stays", clone,
[(KIMAI, lambda t: swap_image("kimai", "kimai/kimai2:apache-2.57.0", "kimai/kimai2:apache-2.58.0")(t).replace("services:", "# catalog_since: %s\nservices:" % today, 1)),
("README.md", lambda t: t + "\ncatalog_since: %s (kimai)\n" % today)],
expect_rc=1, must_contain=("CATALOG-SINCE GATE FAILED",), gate=CS)
case("DECOY: only a comment + env line change, images untouched, date untouched", clone,
[(KIMAI, comment_and_env)], expect_rc=0, must_contain=("0 image move(s) dated", "catalog-since gate OK"), gate=CS)
# ── copy-i18n (R-560): Hungarian frozen, English sound ───────────────────────────────
PB = "templates/privatebin/.felhom.yml"
TOTAL = 1032 # every copy string in the catalog, measured on 94bc5febaca2
PB_EN = 14 # what the genuine block below translates
# A CORRECT English block for privatebin — the genuine article every decoy is a twist on.
GENUINE_EN = """
i18n:
en:
description: "Encrypted note and text sharing"
app_info:
tagline: "Encrypted text sharing - the server never sees the content"
use_cases:
- 'Share sensitive text safely'
- 'End-to-end encryption - the server cannot read the content'
- 'Choose how long it lasts (5 minutes to a year, or never)'
- 'Delete after reading, automatically'
- 'Password protection for extra safety'
first_steps:
- 'Open paste.DOMAIN in your browser'
- 'Type your text and select Send'
- 'Share the link you get - the encryption key is inside the URL'
deploy_fields:
- env_var: DOMAIN
label: "Domain"
description: "The server domain name"
- env_var: SUBDOMAIN
label: "Subdomain"
description: "The address this app answers on"
"""
def strip_en(t):
"""Remove an existing English block (and its comment header) — a case must behave the
same before and after that app's batch lands, or the suite rots on a future push."""
t = re.sub(r"\n# --- English copy.*\Z", "\n", t, flags=re.S)
return re.sub(r"\n^i18n:\n.*\Z", "\n", t, flags=re.S | re.M)
def add_en(block=GENUINE_EN):
return lambda t: strip_en(t).rstrip("\n") + "\n" + block
def en_with(old_, new_):
return add_en(GENUINE_EN.replace(old_, new_))
# THE FACTS — each must be refused.
case_copy("FACT: a Hungarian byte changed in a frozen string", clone,
[(PB, lambda t: t.replace("Titkosított jegyzet és szöveg megosztás",
"Titkosított jegyzet- és szövegmegosztás"))],
expect_rc=1, must_contain=("Hungarian CHANGED", "privatebin", "description"))
case_copy("FACT: a Hungarian first_step removed", clone,
[(PB, lambda t: t.replace(" - 'Oszd meg a generált linket - a titkosítási kulcs az URL-ben van'\n", ""))],
expect_rc=1, must_contain=("REMOVED", "first_steps"))
case_copy("FACT: a NEW app is not in the freeze", clone,
[("templates/decoyapp/.felhom.yml",
lambda t: 'display_name: "Decoy"\ndescription: "Uj alkalmazas"\nslug: decoyapp\n')],
expect_rc=1, must_contain=("not in the freeze", "--add-app"))
case_copy("FACT: an unknown key inside the English block", clone,
[(PB, en_with(' description: "Encrypted note and text sharing"',
' description: "Encrypted note and text sharing"\n docs_url: "https://example.invalid"'))],
expect_rc=1, must_contain=("unknown key",), extra_args=("--expect-missing", str(TOTAL - PB_EN)))
case_copy("FACT: an English deploy field with no Hungarian twin", clone,
[(PB, en_with(" - env_var: DOMAIN", " - env_var: NOSUCHFIELD"))],
expect_rc=1, must_contain=("no Hungarian twin", "NOSUCHFIELD"))
case_copy("FACT: an accented Hungarian letter left in the English", clone,
[(PB, en_with("Share sensitive text safely", "Érzékeny text sharing"))],
expect_rc=1, must_contain=("accented Hungarian letter",))
case_copy("FACT: ASCII-only Hungarian left in the English (no accent to find)", clone,
[(PB, en_with(' description: "The address this app answers on"',
' description: "Aldomain for the app"'))],
expect_rc=1, must_contain=("ASCII-only Hungarian", "aldomain"))
case_copy("FACT: the product begs (\"please\")", clone,
[(PB, en_with("Type your text and select Send", "Please type your text and select Send"))],
expect_rc=1, must_contain=("does not beg",))
case_copy("FACT: an English retrieval promise the Hungarian never made", clone,
[(PB, en_with("Password protection for extra safety",
"Deleted notes can still be restored later"))],
expect_rc=1, must_contain=("retrieval promise",))
# A credential is a LOGIN, not prose: gokapi's default_creds carries admin / adminadmin.
GK = "templates/gokapi/.felhom.yml"
case_copy("FACT: a credential token rewritten in translation", clone,
[(GK, lambda t: t.rstrip("\n") + """
i18n:
en:
app_info:
default_creds: "Sign in: administrator / hunter2"
""")],
expect_rc=1, must_contain=("credential token",))
case_copy("FACT: an i18n block for a language the controller does not render", clone,
[(PB, lambda t: t.rstrip("\n") + "\ni18n:\n de:\n description: \"Verschluesselte Notizen\"\n")],
expect_rc=1, must_contain=("renders en only",))
case_copy("FACT: an English list with a different number of steps", clone,
[(PB, en_with(" - 'Share the link you get - the encryption key is inside the URL'\n", ""))],
expect_rc=1, must_contain=("a list is replaced",))
# THE RATCHET — the one thing --expect-missing does not test for the cases above, so it is
# tested here explicitly, in BOTH directions. A ceiling that only convicts upwards can be
# left behind by a push that translated more than it recorded.
case_copy("FACT: ratchet — fewer strings translated than the ceiling records", clone,
[(PB, add_en())], expect_rc=1,
must_contain=("English coverage", "ABOVE"),
extra_args=("--expect-missing", "0"))
case_copy("FACT: ratchet — more translated than the ceiling records", clone,
[(PB, add_en())], expect_rc=1,
must_contain=("English coverage", "BELOW"),
extra_args=("--expect-missing", "999999"))
# THE GENUINE ARTICLE — must pass.
case_copy("GENUINE: a correct English block on privatebin", clone,
[(PB, add_en())], expect_rc=0,
must_contain=("copy-i18n: OK", "privatebin 14/14"))
# The ratchet is a fact about the CATALOG, not about how the gate was invoked. Naming one
# app must not change the count — the first version of this gate counted coverage only for
# the apps in scope, so `check-copy-i18n.py privatebin` reported 47 more missing strings
# than the same tree unscoped, and either number could have been made to "pass".
case_copy("GENUINE: naming an app does not change the coverage count", clone,
[(PB, add_en())], expect_rc=0,
must_contain=("copy-i18n: OK",),
extra_args=("privatebin", "--expect-missing", str(TOTAL - PB_EN)))
# THE DECOYS — the LABEL moves, the FACT does not. Each must pass.
case_copy("DECOY: Hungarian rewritten inside a YAML COMMENT", clone,
[(PB, lambda t: t.replace("# --- App info (info page content) ---",
"# --- Alkalmazas informacio: Titkosított jegyzet MEGVALTOZOTT ---"))],
expect_rc=0, must_contain=("copy-i18n: OK",))
case_copy("DECOY: a frozen Hungarian sentence pasted into README.md", clone,
[("README.md", lambda t: t + "\nTitkositott jegyzet es szoveg megosztas (decoy)\n")],
expect_rc=0, must_contain=("copy-i18n: OK",))
case_copy("DECOY: display_name changed - a NAME, never copy", clone,
[(PB, lambda t: t.replace('display_name: "PrivateBin"', 'display_name: "PrivateBin 2"'))],
expect_rc=0, must_contain=("copy-i18n: OK",))
case_copy("DECOY: docs_url changed - configuration, never copy", clone,
[(PB, lambda t: t.replace("https://github.com/PrivateBin/PrivateBin/wiki",
"https://example.invalid/wiki"))],
expect_rc=0, must_contain=("copy-i18n: OK",))
case_copy("DECOY: Hungarian text added to a docker-compose.yml", clone,
[("templates/privatebin/docker-compose.yml",
lambda t: t.replace("services:", "# Titkosított jegyzet és szöveg megosztás\nservices:", 1))],
expect_rc=0, must_contain=("copy-i18n: OK",))
finally:
shutil.rmtree(clone, ignore_errors=True)
if fails:
print()
for f in fails:
print("FAIL: %s" % f)
return 1
print("\ncatalog gate decoys OK — %d case(s), every label judged on its fact (R-421)" % ran)
return 0
if __name__ == "__main__":
sys.exit(main())
Reference in New Issue View Git Blame Copy Permalink