admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0a960ba58d5b22ca581ca8d7d73ed5c33d5cee58
felhom-controller/controller/scripts/i18n_go_parity.py
T
admin 074ddda352 R-576: go-parity gate also judges the CALL — arity vs printf verbs, and no key glued with + 
Checks 4 (ARITY) and 5 (NO-CONCAT) in i18n_go_parity.py: at every message-helper call
(MsgError/MsgErrorf/Text/Msgf/msg/msgLang/msgHU/note) whose key is a literal hu.json knows,
the argument count must equal the Hungarian value's printf verbs; a known key literal may
never be an operand of +. 485 calls judged, 0 defects today. Decoys: a dropped argument and
a key+suffix concatenation on a real producer convict; a nested-paren argument is accepted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 22:24:23 +02:00

510 lines
22 KiB
Python
Raw Blame History

# -*- coding: utf-8 -*-
"""i18n_go_parity.py -- a Go-side message key may only carry text that ALREADY existed.
Run from controller/: python3 scripts/i18n_go_parity.py
Exit 0 clean * 1 convicted * 2 inconclusive (base capture or bundles missing).
Design: felhom.eu/documentation/architecture/10-localisation.md s1 -- the Hungarian product must
render byte-for-byte the same before and after every slice. Slice 1 proved that for TEMPLATES by
rendering fixtures captured from unconverted templates. Slice 2 moves GO literals into the bundle,
and a rendered fixture cannot cover them all (a flash, an API error, a country name each need their
own request). This gate proves the same property structurally, at push time:
Every key slice 2 introduced maps to a Go string literal that existed at the BASE COMMIT, byte
for byte. A literal that was split, joined, reworded or re-punctuated on the way into hu.json
fails here and names both sides.
Two files carry it:
scripts/i18n_go_base.json -- EVERY Go string literal at the base commit, captured ONCE with
`--capture` from a clean worktree of that commit. FROZEN: it is the
measurement, and regenerating it to make a conversion pass is the
one thing that would make this gate a wish. Slice 2's three
releases all measure against the same capture.
scripts/i18n_go_keys.json -- `key -> from`, written by the conversion. `from` is the base
literal the key replaced, or the ORDERED list of literals a
concatenation joined.
Checks:
1. LISTED. Every hu.json key named by Go code (s.msg / msgN / Msgf / MsgErrorf / b.Msg / b.Plural
with a literal key) appears in i18n_go_keys.json. Forgetting to list a converted key is the
hole this closes.
2. REAL. Every `from` literal appears in the base capture. A key cannot cite text nobody wrote.
3. BYTE-EQUAL. A single-literal key: hu.json[key] == from, exactly. A joined key: hu.json[key]
with its printf verbs removed == the `from` fragments concatenated in order, with their verbs
removed too. Plural keys compare their `.one`/`.other`-less base form.
4. ARITY (R-576). At a message-helper call whose key is a literal hu.json knows, the arguments
after the key are as many as the Hungarian value's printf verbs.
5. NO-CONCAT (R-576). A literal naming a bundle key is never an operand of `+`.
Checks 1-3 ask whether the TEXT is real; 4-5 ask whether the CALL kept all of it. A structural
gate over the text cannot see a defect in the call -- that is how 7 producers lost half their
sentence on 2026-09-18 with this gate green.
The Go literal walker is the inventory's (felhom.eu/scripts/i18n_inventory.py `go_literals`),
copied rather than imported: this gate runs from a controller clone that may not sit beside
felhom.eu, and a gate that can fail to import its own scanner is a gate that can silently skip.
This source is ASCII-only: it holds no Hungarian, so there is nothing for a locale to mangle.
"""
from __future__ import annotations
import argparse
import io
import json
import os
import re
import sys
import urllib.parse
HERE = os.path.dirname(os.path.abspath(__file__))
CTRL = os.path.dirname(HERE)
LOCALES = os.path.join(CTRL, "internal", "i18n", "locales")
BASE_FILE = os.path.join(HERE, "i18n_go_base.json")
KEYS_FILE = os.path.join(HERE, "i18n_go_keys.json")
# Directories whose Go literals are IN SCOPE for the capture. cmd/ and internal/ minus the
# first-boot wizard, which is out of scope and slated for deletion (R-554, 10-localisation.md s6).
SCOPE_ROOTS = ["internal", "cmd"]
SCOPE_SKIP = [os.path.join("internal", "setup")]
# A string literal SHAPED like a bundle key: lower-case dotted segments, no spaces. Every such
# literal that hu.json actually knows is treated as "this key is named in Go".
#
# It deliberately does NOT look for `s.msg(` and its siblings. The first version did, and a decoy
# caught it: a key reaches the bundle through many shapes that are not a call to a message helper --
# `offboxRedirect(w, r, "flash.offbox.app_missing", true)` puts one in a redirect URL, an Alert
# carries one in a struct field, a handler stores one in `data["TitleKey"]`. Every one of those was
# invisible, so a converted key could be dropped from the map and nothing would say so. Matching the
# SHAPE and then requiring the bundle to know it cannot miss a delivery mechanism, because it does
# not model one.
KEY_SHAPE_RE = re.compile(r"^[a-z][a-z0-9_]*(?:\.[a-z0-9_\-]+)+$")
# printf verbs, including explicit argument indexes (%[2]s -- how English reorders while the
# Hungarian format string keeps the plain verbs it always had).
VERB_RE = re.compile(r"%(?:\[\d+\])?[-+# 0-9.*]*[a-zA-Z]")
def go_literals(src: str):
"""Yield (line, literal_body, raw) for every string literal outside comments.
Copied from felhom.eu/scripts/i18n_inventory.py `go_literals` (2026-09-17).
"""
i, n, line = 0, len(src), 1
while i < n:
c = src[i]
if c == "\n":
line += 1
i += 1
elif src.startswith("//", i):
j = src.find("\n", i)
i = n if j < 0 else j
elif src.startswith("/*", i):
j = src.find("*/", i + 2)
j = n if j < 0 else j + 2
line += src.count("\n", i, j)
i = j
elif c == "'":
j = i + 1
while j < n and src[j] != "'":
j += 2 if src[j] == "\\" else 1
i = j + 1
elif c == '"':
j = i + 1
while j < n and src[j] != '"' and src[j] != "\n":
j += 2 if src[j] == "\\" else 1
yield line, src[i + 1:j], False
i = j + 1
elif c == "`":
j = src.find("`", i + 1)
j = n if j < 0 else j
yield line, src[i + 1:j], True
line += src.count("\n", i, j)
i = j + 1
else:
i += 1
def go_files(root: str):
out = []
for d in SCOPE_ROOTS:
base = os.path.join(root, d)
if not os.path.isdir(base):
continue
for dirpath, _dirs, names in os.walk(base):
rel = os.path.relpath(dirpath, root)
if any(rel == s or rel.startswith(s + os.sep) for s in SCOPE_SKIP):
continue
for n in sorted(names):
if n.endswith(".go") and not n.endswith("_test.go"):
out.append(os.path.join(dirpath, n))
return sorted(out)
def read(p: str) -> str:
with io.open(p, encoding="utf-8", errors="replace") as fh:
return fh.read()
def unescape(body: str) -> str:
"""Turn a Go interpreted-literal BODY into the runtime string.
Only the escapes this codebase uses appear here; anything unknown is left alone rather than
guessed, which keeps the comparison honest (an unrecognised escape can only make a key FAIL
to match, never falsely match).
"""
out, i, n = [], 0, len(body)
simple = {"n": "\n", "t": "\t", "r": "\r", "\\": "\\", '"': '"', "'": "'"}
while i < n:
if body[i] == "\\" and i + 1 < n:
c = body[i + 1]
if c in simple:
out.append(simple[c])
i += 2
continue
if c == "u" and i + 6 <= n:
try:
out.append(chr(int(body[i + 2:i + 6], 16)))
i += 6
continue
except ValueError:
pass
out.append(body[i])
i += 1
return "".join(out)
def capture(root: str) -> dict:
"""Index EVERY Go string literal at this commit, not only the ones a word list calls Hungarian.
The first version filtered by `has_hu`, and the filter was wrong in exactly the way R-565 named:
a Hungarian string spelled without accents ("Naponta", "5 percenkent", "Eletjel (Heartbeat)",
"Adatbazis mentes") is invisible to a letter search, and a word list is a list -- it is short by
construction and nobody knows which words are missing. Seven such literals were silently absent
from the first capture and the gate correctly refused the keys that cited them.
Filtering is not needed here at all. This index answers ONE question -- "did the base commit
contain this exact text?" -- and an unfiltered index answers it for every string. It cannot be
fooled by an English literal that happens to equal a Hungarian sentence, because no such literal
exists. So the word list is gone and the blind spot with it.
"""
lits = {}
for p in go_files(root):
rel = os.path.relpath(p, root)
src = read(p)
for line, body, raw in go_literals(src):
text = body if raw else unescape(body)
if text == "":
continue
lits.setdefault(text, []).append("%s:%d" % (rel, line))
return lits
def strip_verbs(s: str) -> str:
"""Remove printf verbs, keeping a literal %% as a single percent sign."""
s = s.replace("%%", "\x00")
s = VERB_RE.sub("", s)
return s.replace("\x00", "%")
def load_json(p: str):
if not os.path.exists(p):
return None
with io.open(p, encoding="utf-8") as fh:
return json.load(fh)
def keys_named_in_go(root: str, hu):
"""Every bundle key a Go literal names -> the sites that name it."""
out = {}
for p in go_files(root):
rel = os.path.relpath(p, root)
src = read(p)
for i, line in enumerate(src.splitlines(), 1):
if line.lstrip().startswith("//"):
continue
for _line, body, raw in go_literals(line):
text = body if raw else unescape(body)
if hu is not None and text in hu and KEY_SHAPE_RE.match(text):
out.setdefault(text, []).append("%s:%d" % (rel, i))
return out
def base_form(hu: dict, key: str):
"""The Hungarian text for key, accepting the plural split (key, key.one, key.other)."""
if key in hu:
return hu[key]
for suffix in (".other", ".one"):
if key + suffix in hu:
return hu[key + suffix]
return None
# ---- R-576: the CALL, not only the TEXT ---------------------------------------------------------
#
# Checks 1-3 ask "is the text a key carries real?". They cannot see a call site that LOST text: on
# 2026-09-18 the bulk converter turned `fmt.Errorf("a: "+ "b: %s", x)` into a call that kept only
# the key and dropped the continuation and its argument -- 7 producers, and this gate stayed GREEN,
# because every surviving fragment WAS a byte-equal base literal. Two more questions convict it:
#
# 4. ARITY. At a message-helper call whose key argument is a literal hu.json knows, the number of
# arguments after the key equals the number of printf verbs in the Hungarian value. A call that
# spreads a slice (`args...`) is not counted -- its arity is not visible in the source.
# 5. NO-CONCAT. A literal that names a bundle key is never an operand of `+`. A key is a whole
# name; a key glued to more text is either a half-converted concatenation or a key nobody can
# find.
#
# helper name -> index of its KEY argument. Matched by NAME (no type checker), so only a call whose
# key argument is a literal hu.json actually knows is ever judged.
MSG_HELPERS = {
"MsgError": 0, # util.MsgError(key, args...)
"MsgErrorf": 1, # util.MsgErrorf(kind, key, args...)
"Text": 1, # util.Text(lang, key, args...)
"Msgf": 1, # (*i18n.Bundle).Msgf(lang, key, args...)
"msg": 1, # (*Router|*Server).msg(req, key, args...)
"msgLang": 1, # (*Router|*Server).msgLang(lang, key, args...)
"msgHU": 0, # stacks.msgHU(key, args...)
"note": 0, # (*backup.Manager).note(key, args...)
}
CALL_RE = re.compile(r"\b(" + "|".join(sorted(MSG_HELPERS, key=len, reverse=True)) + r")\(")
def code_mask(src: str):
"""(mask, lits): mask is 1 where src[i] is CODE (outside comments, strings and runes); lits is
the (start, end) span of every interpreted string literal, quotes included."""
mask = bytearray(len(src))
lits = []
i, n = 0, len(src)
while i < n:
c = src[i]
if src.startswith("//", i):
j = src.find("\n", i)
i = n if j < 0 else j
elif src.startswith("/*", i):
j = src.find("*/", i + 2)
i = n if j < 0 else j + 2
elif c in "'\"":
j = i + 1
while j < n and src[j] != c and src[j] != "\n":
j += 2 if src[j] == "\\" else 1
if c == '"':
lits.append((i, j + 1))
i = j + 1
elif c == "`":
j = src.find("`", i + 1)
i = n if j < 0 else j + 1
else:
mask[i] = 1
i += 1
return mask, lits
def split_args(src: str, mask, open_paren: int):
"""The top-level argument source strings of the call whose '(' is at open_paren, or None."""
depth, start, args, i, n = 0, open_paren + 1, [], open_paren, len(src)
while i < n:
if mask[i]:
c = src[i]
if c in "([{":
depth += 1
elif c in ")]}":
depth -= 1
if depth == 0:
tail = src[start:i].strip()
if tail:
args.append(tail)
return args
elif c == "," and depth == 1:
args.append(src[start:i].strip())
start = i + 1
i += 1
return None
def verb_arity(value: str) -> int:
"""How many arguments the printf verbs of value consume (explicit indexes respected)."""
s = value.replace("%%", "")
need, nxt = 0, 0
for m in VERB_RE.finditer(s):
idx = re.match(r"%\[(\d+)\]", m.group(0))
if idx:
nxt = int(idx.group(1))
else:
nxt += 1
need = max(need, nxt)
return need
STR_LIT_RE = re.compile(r'^"((?:[^"\\\n]|\\.)*)"$')
def call_site_defects(root: str, hu) -> tuple:
"""Checks 4 (ARITY) and 5 (NO-CONCAT) over every in-scope Go file -> (defects, calls judged)."""
bad, checked = [], 0
if hu is None:
return bad, checked
for p in go_files(root):
rel = os.path.relpath(p, root)
src = read(p)
mask, lits = code_mask(src)
# 5. NO-CONCAT: a known key literal with a `+` on either side.
for a, b in lits:
text = unescape(src[a + 1:b - 1])
if text not in hu or not KEY_SHAPE_RE.match(text):
continue
left = src[:a].rstrip(" \t")
right = src[b:].lstrip(" \t")
if left.endswith("+") or right.startswith("+"):
line = src.count("\n", 0, a) + 1
bad.append("CONCAT %s is glued to more text with + at %s:%d -- a key is a whole "
"name; this is a half-converted concatenation" % (text, rel, line))
# 4. ARITY.
for m in CALL_RE.finditer(src):
if not mask[m.start()]:
continue
args = split_args(src, mask, m.end() - 1)
k = MSG_HELPERS[m.group(1)]
if args is None or len(args) <= k:
continue
lit = STR_LIT_RE.match(args[k])
if not lit:
continue
key = unescape(lit.group(1))
if key not in hu or not KEY_SHAPE_RE.match(key):
continue # a plural key (.one/.other only) or not a key at all
rest = args[k + 1:]
if rest and rest[-1].endswith("..."):
continue # a spread slice: arity not visible
checked += 1
want = verb_arity(hu[key])
if len(rest) != want:
line = src.count("\n", 0, m.start()) + 1
bad.append("ARITY %s called with %d argument(s) at %s:%d, its Hungarian value "
"has %d printf verb(s): %r" % (key, len(rest), rel, line, want, hu[key]))
return bad, checked
def main(argv) -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--capture", action="store_true",
help="rewrite i18n_go_base.json (run once, from a CLEAN worktree of the base commit)")
ap.add_argument("--root", default=CTRL,
help="controller root to capture FROM -- point it at a clean worktree of the base "
"commit, never at a tree that has already been converted")
ap.add_argument("--commit", default="", help="the commit being captured, recorded in the file")
args = ap.parse_args(argv[1:])
if args.capture:
lits = capture(args.root)
payload = {
"_comment": (
"FROZEN measurement -- every Hungarian Go string literal at the base commit of "
"localisation slice 2. Regenerating this to make a conversion pass is the one thing "
"that would turn i18n_go_parity.py into a wish. See its docstring."
),
"commit": args.commit,
"count": len(lits),
# One site per literal: enough to name where the text came from in a failure, and it
# keeps a frozen measurement small enough to read in a review.
"literals": {k: sorted(set(v))[0] for k, v in sorted(lits.items())},
}
with io.open(BASE_FILE, "w", encoding="utf-8") as fh:
json.dump(payload, fh, ensure_ascii=False, indent=1, sort_keys=False)
fh.write("\n")
print("captured %d Go string literals from %s -> %s"
% (len(lits), args.root, os.path.relpath(BASE_FILE, CTRL)))
return 0
base = load_json(BASE_FILE)
keys = load_json(KEYS_FILE)
hu = load_json(os.path.join(LOCALES, "hu.json"))
if base is None or keys is None or hu is None:
print("go-parity gate INCONCLUSIVE: base capture, key map or hu.json not found under %s" % CTRL)
return 2
base_lits = base.get("literals", {})
named = keys_named_in_go(CTRL, hu)
# Keys that predate slice 2 (the spike's and slice 1's copy-producing funcs). Each is pinned by
# its own Go test, named in the map; this gate accounts for them so a NEW Go key cannot hide
# among them. They are not measured against the base capture: their Hungarian was moved by
# slice 0/1 and proved by rendered fixtures, which is the stronger measurement.
preexisting = keys.pop("_preexisting", {}) or {}
keys.pop("_comment", None)
bad = []
# 1. LISTED -- a key a Go call site names must be accounted for.
for key in sorted(named):
if key not in keys and key not in preexisting:
bad.append("UNLISTED %s named at %s but absent from %s"
% (key, named[key][0], os.path.basename(KEYS_FILE)))
# 2 + 3. REAL and BYTE-EQUAL.
for key in sorted(keys):
spec = keys[key]
value = base_form(hu, key)
if value is None:
bad.append("NO-KEY %s listed but absent from hu.json" % key)
continue
# A third spec shape, for the handful of sites where the Hungarian was written ALREADY
# URL-ESCAPED inside a redirect target: {"inside": "<the base literal>"}. The literal is
# unescaped and the key's text must occur in it exactly once. Explicit and rare on purpose --
# a substring rule applied everywhere would let a reworded sentence pass.
if isinstance(spec, dict):
lit = spec.get("inside", "")
if lit not in base_lits:
bad.append("INVENTED %s cites a base literal that does not exist: %r" % (key, lit))
continue
decoded = urllib.parse.unquote_plus(lit)
n = decoded.count(value)
if n != 1:
bad.append("CHANGED %s occurs %d times (want 1) in the unescaped literal\n"
" hu.json : %r\n"
" unescaped: %r" % (key, n, value, decoded))
continue
froms = [spec] if isinstance(spec, str) else list(spec)
missing = [f for f in froms if f not in base_lits]
if missing:
bad.append("INVENTED %s cites text that is in no base-commit literal: %r"
% (key, missing[0]))
continue
# A STRING `from` means "this key replaced exactly that literal" -- compared byte for byte, so
# even a verb swapped from %s to %v convicts. A LIST means "this key joined these literals",
# which is how a concatenation or a trailing parameter becomes one message; there the printf
# verbs are what the join replaced, so both sides are compared with the verbs removed.
if isinstance(spec, str):
if value != froms[0]:
bad.append("CHANGED %s\n hu.json: %r\n base : %r (%s)"
% (key, value, froms[0], base_lits[froms[0]]))
else:
got, want = strip_verbs(value), strip_verbs("".join(froms))
if got != want:
bad.append("CHANGED %s (joined from %d literals)\n"
" hu.json (verbs removed): %r\n"
" base (verbs removed): %r"
% (key, len(froms), got, want))
# 4 + 5. ARITY and NO-CONCAT (R-576) -- the call, not the text.
call_bad, judged = call_site_defects(CTRL, hu)
bad.extend(call_bad)
if judged == 0:
bad.append("NO-CALLS no message-helper call with a literal key was found -- the call scanner "
"no longer matches the code, so checks 4-5 would be green by seeing nothing")
print("go-parity: base capture %s (%d literals), %d slice-2 keys listed, %d pre-existing, "
"%d named in Go, %d helper calls judged for arity"
% (base.get("commit", "?")[:12] or "?", len(base_lits), len(keys), len(preexisting),
len(named), judged))
if bad:
print("\ngo-parity gate CONVICTS (%d):" % len(bad))
for b in bad:
print(" " + b)
return 1
print("go-parity gate OK: every Go-side key carries base-commit text, byte for byte, and every "
"judged call passes as many arguments as its message has verbs.")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
Reference in New Issue View Git Blame Copy Permalink