admin
979ababb8f
test_gate_decoys.py runs test_check_volume_persistence.py (the blind and crying-wolf probers refused rc=3, `wrote nothing` never CLEAN, the papra signature convicted) and requires it green, so COVERS is a fact; and runs the working-tree gate in a scratch catalog with PATH = ONLY a stub docker that fails every call: a runtime that answers nothing, no docker, nothing to judge are each HARNESS REFUSED rc=3, never 0. An always-succeeding stub is deliberately not used - it would walk the prober into the host filesystem. COVERS gains "volume-persistence". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
1551 lines
101 KiB
Python
1551 lines
101 KiB
Python
#!/usr/bin/env python3
|
|
# -*- coding: utf-8 -*-
|
|
"""test_gate_decoys.py — can this repo's gates be fooled by a LABEL? (R-421)
|
|
|
|
The same instrument as `felhom.eu/scripts/test_gate_decoys.py`: a decoy is the LABEL without the
|
|
FACT, and a gate that convicts on the label alone — or fails to convict on the fact — is a live hole.
|
|
Every case asserts BOTH directions where it can: the genuine article must pass and the decoy must be
|
|
judged on what it IS, not on what it says.
|
|
|
|
Covered here (the `COVERS` literal is AST-read by `felhom.eu/scripts/decoy_coverage_gate.py`):
|
|
|
|
engine-major — `check-engine-major.py` refuses a database-engine pin that crosses a MAJOR.
|
|
Its label is the version string; its fact is the `image:` line of an engine SERVICE. Decoys a
|
|
real session would produce:
|
|
* the major moves in a COMMENT and in kimai's `serverVersion=11.6.2-MariaDB` env var, while
|
|
the image line stays — must PASS (nothing moved);
|
|
* the APP's own image crosses a major (kimai 2.57 -> 3.0) — must PASS (not an engine);
|
|
* a `mariadb:12.3` string lands in README.md — must PASS (not a template);
|
|
* the engine moves WITHIN its major (11.6 -> 11.8) — must PASS (the rule says MAJOR);
|
|
and the facts:
|
|
* `mariadb:11.6 -> mariadb:12.3` on `kimai-db` — must be REFUSED (exit 1), naming the rule
|
|
and its expiry (R-448);
|
|
* `postgres:16-alpine -> postgres:17-alpine` on `docmost-postgres` — must be REFUSED (the
|
|
eleven PostgreSQL services are covered by NAME MATCH, not by a list);
|
|
* `mariadb:11.6 -> mariadb:lts` — INCONCLUSIVE (exit 2), never 0: a major nobody can read is
|
|
not a pass.
|
|
|
|
HOW. The repo is cloned into a scratch directory; the WORKING-TREE gate is run inside the clone
|
|
(so the file under test is the one being edited, not HEAD's); each case is one commit on top of the
|
|
clone's HEAD and the gate is run with `--range HEAD~1..HEAD`. The real tree is never touched.
|
|
|
|
Run from the repo root: python3 scripts/test_gate_decoys.py
|
|
Exit 0 every decoy judged correctly · 1 a decoy passed or a genuine article was refused.
|
|
"""
|
|
import io
|
|
import json
|
|
import os
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
|
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
|
|
|
# ── WHAT THIS FILE COVERS ────────────────────────────────────────────────────────────────────────
|
|
# Read by felhom.eu/scripts/decoy_coverage_gate.py, which AST-parses this literal. A gate named here
|
|
# MUST have a decoy below that has been seen to fail.
|
|
COVERS = {
|
|
"volume-persistence": "the classifier and the self-test via the injected-prober suite scripts/test_check_volume_persistence.py, RUN FROM HERE and required green (a blind prober and a crying-wolf prober are refused rc=3; `wrote nothing`, a container not running, a path token alone, an unresolved suspect are UNDETERMINED or clean, never a false CLEAN/BROKEN; the papra signature convicts); and END TO END with a PATH-STUB docker that fails every call, no docker at all, nothing to judge - each HARNESS REFUSED rc=3, never 0 (R-426)",
|
|
"image-resolvable": "END TO END through the gate's own `docker manifest inspect` call, with a PATH-STUB docker (PATH holds ONLY the stub, so the real runtime is unreachable): the label of success without the fact - rc=0 carrying a throttle or any error text, a docker that resolves EVERYTHING incl. the .invalid canary, no docker at all, nothing to judge - each INCONCLUSIVE or HARNESS REFUSED, never 0; the cry-wolf direction - a throttle or an unrecognised error on rc=1 is never an accusation; vs a registry that positively says `manifest unknown` (convicted, naming the app) and a clean run (R-426)",
|
|
"image-pins": "the pin's LABEL without the pin: a registry PORT (`:5000`) read as a tag, `@sha256:` with no digest behind it, a QUOTED `\"image\":` key, an interpolated `${APP_IMAGE:-nginx}`, `:LATEST`/`:latest` in quotes - vs the inert shapes that must pass (a commented `# image: nginx`, an `x-image:` extension field, a README line) and the genuine pins (a tag, a real 64-hex digest, `:latest@sha256:` pinned by its digest, a port with a tag) (R-426)",
|
|
"engine-major": "the major moved in a comment/env var/README/app image, not on an engine's image: line",
|
|
"catalog-since": "the date bumped in a comment/README while .felhom.yml's field stayed; or only a comment/env moved, no image (R-452)",
|
|
"probe-matches-compose": "the probe TARGET resolves by exact name, explicit `container`, or a UNIQUE prefix - an ambiguity is refused, not guessed (R-630); the DEGRADED no-PyYAML mode CI actually runs; the port/path moved in a COMMENT, in traefik's loadbalancer label, in "
|
|
"`ports:`/`expose:`, or on a NON-probed service - none of which is where "
|
|
"the app listens; vs a real probe port/path that the app does not answer (R-618)",
|
|
"test-record": "a ladder whose newest step is not the compose's images (a move without a record), a gap, a line that is not one JSON entry, a failed verdict - vs a clean ladder (09 decision 13)",
|
|
"test-record-move": "an image move with NO entry, with the entry only in a COMMENT or in README, with a failed/backfilled entry, with a digest the registry no longer serves, memory_tight without a raised limit - vs a proven entry that matches; a ref moving in a compose COMMENT is not a move (09 decision 13)",
|
|
"probe-measured": "the measurement written in the TAGLINE or another comment block, not directly above setup_done_probe:; a date with no before/after; before/after with no date; 'read upstream' instead of 'measured' - vs a genuine measured comment (R-715)",
|
|
"family-gate": "family_gate written only in a COMMENT (not gated, no min_controller owed); min_controller only in a comment; a golden DIRECTORY named 0.287.0 with no bake log (the mkdir shape, R-410); a sibling with no golden (stated NOT CHECKED, never a pass of rule 3) - vs the facts: an unanchorable exception (regex, '/', '..'), an exception list with no gate, min_controller below 0.287.0, the newest baked golden below 0.287.0; and a genuine family app passes (decisions 63/64, finding F1)",
|
|
"onboarding": "a NEW template with no record; a record missing an id, or carrying it only inside an HTML comment; a `done` whose path does not exist, is an EMPTY directory (the mkdir shape, R-410) or names an absent sibling-repo file; an `n/a` with an empty or two-word reason; an `open` row; `opened:` backdated before the checklist; the template a new app copies lacking a new id - vs a complete record, an id added after `opened:`, and an exempt app's record with open rows (NEW-APP-CHECKLIST.md)",
|
|
"mem-limit-sum": "the right figure only in a COMMENT (the compose header's 'mem_limit: 640M', the .felhom.yml arithmetic) while the field is wrong; a `memory:` under reservations: (not a limit) making the sum come out right; a `mem_limit:` outside resources:; a steps/ file with the old figure (not judged) - vs the facts: a field under the sum, a service with no limit, an unreadable size (R-758)",
|
|
"data-key": "a data_key: true only in a COMMENT or inside the i18n: block (not the field); an 'Encryption key' LABEL on a signing secret (copy, not the fact - must pass unflagged) - vs the facts: an *_ENCRYPTION_KEY field unflagged, a REGISTERED field unflagged, an unexplained flag, a stale registry entry, a non-boolean flag (R-127)",
|
|
"copy-i18n": "Hungarian edited in a COMMENT/README/display_name (label, not copy) vs a real frozen string changed; an English block that is not English, is not matched to a Hungarian twin, or rewrites a credential (R-560). A retrieval promise REGISTERED in ALLOWLIST_EN passes only for its own app+path+sentence with a real reason; an entry for another app, a rewritten sentence, a stale entry or a two-word reason convicts (R-594). Also the DEGRADED mode CI actually runs — PyYAML shadowed out, freeze only (R-595)",
|
|
}
|
|
|
|
fails = []
|
|
ran = 0
|
|
|
|
|
|
def sh(args, cwd):
|
|
return subprocess.run(args, cwd=cwd, capture_output=True, text=True)
|
|
|
|
|
|
def make_clone():
|
|
tmp = tempfile.mkdtemp(prefix="catalog-decoys-")
|
|
r = sh(["git", "clone", "-q", "file://" + ROOT, tmp], cwd=ROOT)
|
|
if r.returncode != 0:
|
|
raise SystemExit("clone failed: " + r.stderr)
|
|
sh(["git", "config", "user.email", "decoy@gate.invalid"], cwd=tmp)
|
|
sh(["git", "config", "user.name", "decoy"], cwd=tmp)
|
|
return tmp
|
|
|
|
|
|
def edit(clone, relpath, fn):
|
|
p = os.path.join(clone, relpath)
|
|
os.makedirs(os.path.dirname(p), exist_ok=True) # a case may ADD a file (a new app directory)
|
|
text = io.open(p, encoding="utf-8").read() if os.path.exists(p) else ""
|
|
new = fn(text)
|
|
if new == text:
|
|
raise SystemExit("case did not change %s — the case is broken, not the gate" % relpath)
|
|
with io.open(p, "w", encoding="utf-8") as fh:
|
|
fh.write(new)
|
|
|
|
|
|
def commit(clone, msg):
|
|
sh(["git", "add", "-A"], cwd=clone)
|
|
r = sh(["git", "commit", "-q", "-m", msg], cwd=clone)
|
|
if r.returncode != 0:
|
|
raise SystemExit("commit failed: " + r.stderr)
|
|
|
|
|
|
def reset(clone):
|
|
sh(["git", "reset", "-q", "--hard", "HEAD"], cwd=clone)
|
|
sh(["git", "clean", "-fdq"], cwd=clone) # a case may ADD a file (a steps/ definition)
|
|
|
|
|
|
def case(name, clone, edits, expect_rc, must_contain=(), gate="check-engine-major.py"):
|
|
"""edits: list of (relpath, fn). Commits them, runs the gate on HEAD~1..HEAD, restores."""
|
|
global ran
|
|
ran += 1
|
|
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
commit(clone, name)
|
|
# the WORKING-TREE gate, run inside the clone (it reads git from its cwd)
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", gate),
|
|
"--range", "HEAD~1..HEAD"], cwd=clone)
|
|
out = r.stdout + r.stderr
|
|
ok = r.returncode == expect_rc and all(m in out for m in must_contain)
|
|
if ok:
|
|
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc,
|
|
[m for m in must_contain if m not in out], out[-900:]))
|
|
return out
|
|
finally:
|
|
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
|
|
|
|
|
|
def case_probe(name, clone, edits, expect_rc, must_contain=(), apps=("tandoor", "zipline", "wger",
|
|
"home-assistant")):
|
|
"""The probe gate reads FILES in a checkout, so its cases need `--root` and no commit.
|
|
|
|
Without `--root` the gate would read the REAL repo while the case edits the clone, every case
|
|
would see identical bytes, and every case would pass — the constant-for-measurement shape. The
|
|
app list is passed explicitly for the same reason: a whole-repo run is dominated by the three
|
|
genuine faults and would mask whether THIS case's edit changed anything.
|
|
"""
|
|
global ran
|
|
ran += 1
|
|
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-probe-matches-compose.py"),
|
|
"--root=" + clone] + list(apps), cwd=clone)
|
|
out = r.stdout + r.stderr
|
|
ok = r.returncode == expect_rc and all(m in out for m in must_contain)
|
|
if ok:
|
|
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc,
|
|
[m for m in must_contain if m not in out], out[-900:]))
|
|
return out
|
|
finally:
|
|
sh(["git", "checkout", "-q", "--", "."], cwd=clone)
|
|
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
|
|
|
|
|
|
def case_probe_noyaml(name, clone, edits, expect_rc, must_contain=(), apps=()):
|
|
"""The same cases with PyYAML SHADOWED OUT — the mode the CI runner actually has.
|
|
|
|
A degraded mode that always passes is worse than no gate, because the summary says OK. So the
|
|
three real faults are re-introduced here too and must still be REFUSED by the line reader.
|
|
"""
|
|
global ran
|
|
ran += 1
|
|
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
|
|
nd = noyaml_dir()
|
|
env = dict(os.environ, PYTHONPATH=nd + os.pathsep + os.environ.get("PYTHONPATH", ""))
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
r = subprocess.run([sys.executable, os.path.join(ROOT, "scripts",
|
|
"check-probe-matches-compose.py"),
|
|
"--root=" + clone] + list(apps),
|
|
cwd=clone, capture_output=True, text=True, env=env)
|
|
out = r.stdout + r.stderr
|
|
ok = r.returncode == expect_rc and all(m in out for m in must_contain)
|
|
if ok:
|
|
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc,
|
|
[m for m in must_contain if m not in out], out[-900:]))
|
|
return out
|
|
finally:
|
|
sh(["git", "checkout", "-q", "--", "."], cwd=clone)
|
|
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
|
|
|
|
|
|
def case_copy(name, clone, edits, expect_rc, must_contain=(), extra_args=()):
|
|
"""The copy-i18n gate reads FILES, not commits, so its cases need neither a commit nor a range —
|
|
but they DO need --root, or the gate would read the real repo and judge files nobody edited.
|
|
That is the `constant-for-measurement` decoy shape, and it would make every case below pass."""
|
|
global ran
|
|
ran += 1
|
|
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
args = list(extra_args)
|
|
if "--expect-missing" not in args:
|
|
# MEASURE the clone's current coverage and hand it back as the ceiling. The ratchet is
|
|
# not what these cases test — they test the freeze, the structure and the language —
|
|
# and hard-coding a number here would make every case fail the day a batch lands.
|
|
# The ratchet has its own two cases below, in both directions.
|
|
probe = sh([sys.executable, os.path.join(ROOT, "scripts", "check-copy-i18n.py"),
|
|
"--root", clone, "--expect-missing", "-1"], cwd=clone)
|
|
m = re.search(r"(\d+) strings have no English", probe.stdout + probe.stderr)
|
|
if not m:
|
|
fails.append("%s: could not measure the clone's coverage — the case is broken, "
|
|
"not the gate" % name)
|
|
return ""
|
|
args += ["--expect-missing", m.group(1)]
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-copy-i18n.py"),
|
|
"--root", clone] + args, cwd=clone)
|
|
out = r.stdout + r.stderr
|
|
if r.returncode == expect_rc and all(m in out for m in must_contain):
|
|
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc,
|
|
[m for m in must_contain if m not in out], out[-900:]))
|
|
return out
|
|
finally:
|
|
sh(["git", "checkout", "-q", "--", "."], cwd=clone)
|
|
sh(["git", "clean", "-qfd"], cwd=clone)
|
|
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
|
|
|
|
|
|
NOYAML = os.path.join(tempfile.gettempdir(), "felhom-decoy-noyaml")
|
|
|
|
|
|
def noyaml_dir():
|
|
"""A directory that shadows PyYAML with a module that refuses to import — the CI runner has
|
|
python3 and git and NOTHING else (.gitea/workflows/gates.yml), and the copy gate's first six
|
|
pushes each turned CI red because it imported yaml. These cases pin the degraded mode."""
|
|
os.makedirs(NOYAML, exist_ok=True)
|
|
with io.open(os.path.join(NOYAML, "yaml.py"), "w", encoding="utf-8") as fh:
|
|
fh.write('raise ImportError("no module named yaml (CI-runner simulation)")\n')
|
|
return NOYAML
|
|
|
|
|
|
def case_copy_noyaml(name, clone, edits, expect_rc, must_contain=()):
|
|
"""case_copy with PyYAML made unimportable — i.e. what CI actually runs."""
|
|
global ran
|
|
ran += 1
|
|
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
|
|
env = dict(os.environ, PYTHONPATH=noyaml_dir())
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
r = subprocess.run([sys.executable, os.path.join(ROOT, "scripts", "check-copy-i18n.py"),
|
|
"--root", clone], cwd=clone, capture_output=True, text=True, env=env)
|
|
out = r.stdout + r.stderr
|
|
if r.returncode == expect_rc and all(m in out for m in must_contain):
|
|
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc,
|
|
[m for m in must_contain if m not in out], out[-900:]))
|
|
return out
|
|
finally:
|
|
sh(["git", "checkout", "-q", "--", "."], cwd=clone)
|
|
sh(["git", "clean", "-qfd"], cwd=clone)
|
|
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
|
|
|
|
|
|
def swap_image(service, frm, to):
|
|
"""Change ONLY the named service's own image: line — the same per-service discipline as the
|
|
gate, so the case moves the fact and nothing else."""
|
|
def _fn(text):
|
|
out, cur, done = [], None, False
|
|
for line in text.splitlines():
|
|
m = re.match(r"^ ([A-Za-z0-9_-]+):\s*$", line)
|
|
if m:
|
|
cur = m.group(1)
|
|
mi = re.match(r"^(\s+image:\s*)(\S+)\s*$", line)
|
|
if mi and cur == service and mi.group(2) == frm:
|
|
line = mi.group(1) + to
|
|
done = True
|
|
out.append(line)
|
|
if not done:
|
|
raise SystemExit("%s does not carry image %s — fixture drifted" % (service, frm))
|
|
return "\n".join(out) + "\n"
|
|
return _fn
|
|
|
|
|
|
|
|
def cur_image(clone, relpath, service):
|
|
"""The service's current image in the clone — read, never typed (R-663)."""
|
|
sys.path.insert(0, os.path.join(ROOT, "scripts"))
|
|
import ladder as _l
|
|
return _l.images_in(io.open(os.path.join(clone, relpath), encoding="utf-8").read())[service]
|
|
|
|
|
|
def strip_ladder(t):
|
|
"""The template WITHOUT its update_ladder block (and the header comment the writer puts above it).
|
|
The writer appends the block at the END of the file (ladder.append_entry), so everything from its
|
|
first line on goes — a case then builds exactly the ladder it describes, whatever the live catalog
|
|
has recorded since (R-663)."""
|
|
lines = t.splitlines()
|
|
for i, l in enumerate(lines):
|
|
if l.startswith("# update_ladder") or l.startswith("update_ladder:"):
|
|
return "\n".join(lines[:i]).rstrip("\n") + "\n"
|
|
return t
|
|
|
|
|
|
# ── test record (09 §3 decision 13) ────────────────────────────────────────────────────────────
|
|
TR_D1 = "sha256:" + "a" * 64
|
|
TR_D2 = "sha256:" + "b" * 64
|
|
|
|
|
|
def tr_entry(frm, to, digest, verdict="proven", peak=41.0, tight=False, **extra):
|
|
import json as _j
|
|
e = {"from": frm, "to": to, "digest": digest, "verdict": verdict,
|
|
"tested_at": "2026-09-23T22:00:00Z", "harness_version": 2,
|
|
"evidence": "felhom.eu/documentation/audits/night-2026-09-23/apps/x/", "memory_peak_pct": peak,
|
|
"marks": {"files_may_change": False, "needs_person": None, "memory_tight": tight}}
|
|
e.update(extra)
|
|
return " - " + _j.dumps(e)
|
|
|
|
|
|
def tr_append(line, header=True):
|
|
"""Append one entry line. A template that already HAS a ladder (the writer puts it at the end of
|
|
the file) gets the line appended to it; one without gets the block (R-663: navidrome gained a
|
|
ladder on 2026-09-23 night, and a second `update_ladder:` key is its own conviction)."""
|
|
def _fn(t):
|
|
has = any(l.startswith("update_ladder:") for l in t.splitlines())
|
|
block = ("\nupdate_ladder:\n" if header and not has else "") + line + "\n"
|
|
return t.rstrip("\n") + "\n" + block
|
|
return _fn
|
|
|
|
|
|
def case_tr_move(name, clone, edits, expect_rc, must_contain=(), table=None):
|
|
import json as _j
|
|
tf = os.path.join(clone, "..", os.path.basename(clone) + "-digests.json")
|
|
_j.dump(table or {}, open(tf, "w"))
|
|
global ran
|
|
ran += 1
|
|
base = sh(["git", "rev-parse", "HEAD"], cwd=clone).stdout.strip()
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
commit(clone, name)
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-test-record-move.py"),
|
|
"--range", "HEAD~1..HEAD", "--digests-from", tf], cwd=clone)
|
|
out = r.stdout + r.stderr
|
|
ok = r.returncode == expect_rc and all(m in out for m in must_contain)
|
|
print(" %s %-52s rc=%d (expected %d)" % ("ok" if ok else "XX", name, r.returncode, expect_rc))
|
|
if not ok:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, [m for m in must_contain if m not in out], out[-900:]))
|
|
finally:
|
|
sh(["git", "reset", "-q", "--hard", base], cwd=clone)
|
|
os.remove(tf)
|
|
|
|
|
|
def case_tr_static(name, clone, edits, expect_rc, must_contain=(), apps=("navidrome",)):
|
|
global ran
|
|
ran += 1
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-test-record.py"),
|
|
"--root", clone] + list(apps), cwd=clone)
|
|
out = r.stdout + r.stderr
|
|
ok = r.returncode == expect_rc and all(m in out for m in must_contain)
|
|
print(" %s %-52s rc=%d (expected %d)" % ("ok" if ok else "XX", name, r.returncode, expect_rc))
|
|
if not ok:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, [m for m in must_contain if m not in out], out[-900:]))
|
|
finally:
|
|
reset(clone)
|
|
|
|
|
|
def test_record_cases(clone):
|
|
NC, NF = "templates/navidrome/docker-compose.yml", "templates/navidrome/.felhom.yml"
|
|
# READ, never typed (R-663): the live pin moves with every proven step.
|
|
old = cur_image(clone, NC, "navidrome")
|
|
new = "deluan/navidrome:0.99.1"
|
|
prev = "deluan/navidrome:0.1.0" # an invented older step, for the static cases
|
|
frm, to = {"navidrome": old}, {"navidrome": new}
|
|
move = (NC, swap_image("navidrome", old, new))
|
|
good = tr_entry(frm, to, {"navidrome": TR_D1})
|
|
table = {new: TR_D1}
|
|
print("-- test-record-move: an image move needs its own proven record")
|
|
case_tr_move("FACT: a bare image move, no entry", clone, [move], 1,
|
|
("adds NO update_ladder entry",), table)
|
|
case_tr_move("GENUINE: a proven entry whose digest the registry serves", clone,
|
|
[move, (NF, tr_append(good))], 0, ("0.64.1" if False else "test-record-move gate",), table)
|
|
case_tr_move("FACT: the entry's verdict is failed", clone,
|
|
[move, (NF, tr_append(tr_entry(frm, to, {"navidrome": TR_D1}, verdict="failed")))], 1,
|
|
("not allowed in a ladder",), table)
|
|
case_tr_move("FACT: the registry now serves another digest", clone,
|
|
[move, (NF, tr_append(good))], 1, ("the registry serves",), {new: TR_D2})
|
|
case_tr_move("INCONCLUSIVE: the registry cannot be asked", clone,
|
|
[move, (NF, tr_append(good))], 2, ("could not be asked",), {})
|
|
case_tr_move("DECOY: the entry only in a COMMENT under update_ladder", clone,
|
|
[move, (NF, lambda t: strip_ladder(t).rstrip("\n") + "\nupdate_ladder:\n # " + good.strip() + "\n")], 1,
|
|
("holds no entry",), table)
|
|
case_tr_move("DECOY: the entry only in README.md", clone,
|
|
[move, ("README.md", lambda t: t + "\n" + good + "\n")], 1,
|
|
("adds NO update_ladder entry",), table)
|
|
case_tr_move("FACT: a new move carrying a BACKFILLED entry", clone,
|
|
[move, (NF, tr_append(tr_entry(frm, to, {"navidrome": TR_D1}, backfilled="2026-09-23")))], 1,
|
|
("marked backfilled",), table)
|
|
case_tr_move("FACT: memory_tight and the limit did not move", clone,
|
|
[move, (NF, tr_append(tr_entry(frm, to, {"navidrome": TR_D1}, peak=86.0, tight=True)))], 1,
|
|
("memory_tight",), table)
|
|
case_tr_move("GENUINE: memory_tight WITH the limit raised", clone,
|
|
[move, (NC, lambda t: t.replace("memory: 256M", "memory: 384M")),
|
|
(NF, tr_append(tr_entry(frm, to, {"navidrome": TR_D1}, peak=86.0, tight=True)))], 0,
|
|
(), table)
|
|
case_tr_move("DECOY: a ref moves in a compose COMMENT only", clone,
|
|
[(NC, lambda t: t + "\n# was: deluan/navidrome:0.63.2\n")], 0, (), table)
|
|
print("-- test-record (static): the newest step IS the compose")
|
|
def ladder_of(*lines):
|
|
"""Replace the template's ladder with exactly these entry lines."""
|
|
return lambda t: tr_append("\n".join(lines))(strip_ladder(t))
|
|
head = tr_entry({"navidrome": prev}, frm, {"navidrome": TR_D1})
|
|
case_tr_static("GENUINE: a ladder whose head is the compose", clone, [(NF, ladder_of(head))], 0)
|
|
case_tr_static("FACT: the compose moved past the ladder's head", clone,
|
|
[(NF, ladder_of(head)), move], 1, ("not the ladder's newest step",))
|
|
case_tr_static("FACT: a line that is not one JSON entry", clone,
|
|
[(NF, lambda t: strip_ladder(t) + "\nupdate_ladder:\n - from: x\n")], 1, ("not a one-line JSON entry",))
|
|
case_tr_static("FACT: a gap between steps", clone,
|
|
[(NF, ladder_of(tr_entry({"navidrome": "deluan/navidrome:0.0.1"}, {"navidrome": "deluan/navidrome:0.0.2"}, {"navidrome": TR_D1}), head))],
|
|
1, ("the ladder has a gap",))
|
|
case_tr_static("FACT: a failed verdict sits in the ladder", clone,
|
|
[(NF, ladder_of(tr_entry({"navidrome": prev}, frm, {"navidrome": TR_D1}, verdict="failed")))],
|
|
1, ("not allowed in a ladder",))
|
|
|
|
# ── `09` §6.4 part 5 (v0.268.0 on the box): every step but the newest carries its OWN definition
|
|
# at steps/<StepKey(to)>.yml — the box climbs one step at a time and pins that file. ──────────
|
|
print("-- test-record (static): every intermediate step carries its own definition")
|
|
sys.path.insert(0, os.path.join(ROOT, "scripts"))
|
|
import ladder as _l
|
|
mid = {"navidrome": "deluan/navidrome:0.2.0"}
|
|
two = ladder_of(tr_entry({"navidrome": prev}, mid, {"navidrome": TR_D1}),
|
|
tr_entry(mid, frm, {"navidrome": TR_D1}))
|
|
step_rel = "templates/navidrome/" + _l.step_file(mid)
|
|
meta_rel = "templates/navidrome/" + _l.step_meta_file(mid)
|
|
meta_body = lambda t: _l.strip_ladder_block(io.open(os.path.join(clone, NF), encoding="utf-8").read())
|
|
step_body = lambda t: swap_image("navidrome", old, mid["navidrome"])(io.open(os.path.join(clone, NC), encoding="utf-8").read())
|
|
case_tr_static("FACT: a two-step ladder with NO steps/ file for the first step", clone,
|
|
[(NF, two)], 1, ("has no definition",))
|
|
case_tr_static("GENUINE: a two-step ladder whose first step carries its definition", clone,
|
|
[(NF, two), (step_rel, step_body), (meta_rel, meta_body)], 0)
|
|
case_tr_static("DECOY: the steps/ file has the right NAME and names the head's image", clone,
|
|
[(NF, two), (step_rel, lambda t: io.open(os.path.join(clone, NC), encoding="utf-8").read()), (meta_rel, meta_body)],
|
|
1, ("names",))
|
|
case_tr_static("DECOY: the step's definition sits beside the template under another name", clone,
|
|
[(NF, two), ("templates/navidrome/steps/0.2.0.yml", step_body)], 1, ("has no definition",))
|
|
# R-664: the step's own .felhom.yml
|
|
case_tr_static("FACT: the step has its compose but no .felhom.yml", clone,
|
|
[(NF, two), (step_rel, step_body)], 1, ("R-664",))
|
|
case_tr_static("DECOY: the step's .felhom.yml exists by NAME and holds no healthcheck", clone,
|
|
[(NF, two), (step_rel, step_body), (meta_rel, lambda t: "display_name: Navidrome\n")], 1, ("not a real step file",))
|
|
|
|
# ── `09` §3 decision 52: a RE-TEST — the same tag proved at a NEW digest (from == to). ─────────────
|
|
print("-- test-record: a same-tag re-test (decision 52)")
|
|
TR_D3 = "sha256:" + "c" * 64
|
|
frm_step_rel = "templates/navidrome/" + _l.step_file(frm)
|
|
frm_meta_rel = "templates/navidrome/" + _l.step_meta_file(frm)
|
|
same_step = lambda t: io.open(os.path.join(clone, NC), encoding="utf-8").read()
|
|
rt = lambda dig, dfrom, **kw: tr_entry(frm, frm, {"navidrome": dig}, digest_from={"navidrome": dfrom}, **kw)
|
|
good_rt = rt(TR_D2, TR_D1, box_evidence="felhom.eu/documentation/audits/x/box/")
|
|
with_steps = [(frm_step_rel, same_step), (frm_meta_rel, meta_body)]
|
|
case_tr_static("GENUINE: head + a re-test from its digest, both venues", clone,
|
|
[(NF, ladder_of(head, good_rt))] + with_steps, 0)
|
|
case_tr_static("FACT: a re-test with NO new digest", clone,
|
|
[(NF, ladder_of(head, rt(TR_D1, TR_D1, box_evidence="x")))] + with_steps, 1, ("no new digest",))
|
|
case_tr_static("FACT: a re-test without the box venue", clone,
|
|
[(NF, ladder_of(head, rt(TR_D2, TR_D1)))] + with_steps, 1, ("BOTH venues",))
|
|
case_tr_static("FACT: a re-test FROM a digest the previous entry never tested", clone,
|
|
[(NF, ladder_of(head, rt(TR_D2, TR_D3, box_evidence="x")))] + with_steps, 1, ("starts from the previous entry",))
|
|
case_tr_static("DECOY: a re-test that names digest_from only in its evidence text", clone,
|
|
[(NF, ladder_of(head, tr_entry(frm, frm, {"navidrome": TR_D2}, box_evidence="x", evidence="digest_from sha256:aaaa")))] + with_steps,
|
|
1, ("needs digest_from",))
|
|
rt_live = lambda t: tr_append(rt(TR_D2, TR_D1, box_evidence="felhom.eu/documentation/audits/x/box/"))(t)
|
|
case_tr_move("GENUINE: a re-test whose digest the registry serves now", clone,
|
|
[(NF, rt_live)] + with_steps, 0, ("test-record-move gate",), {old: TR_D2})
|
|
case_tr_move("FACT: a re-test whose digest the registry no longer serves", clone,
|
|
[(NF, rt_live)] + with_steps, 1, ("re-test navidrome",), {old: TR_D3})
|
|
case_tr_move("FACT: a re-test with no new digest reaches the move gate too", clone,
|
|
[(NF, tr_append(rt(TR_D1, TR_D1, box_evidence="x")))] + with_steps, 1, ("no new digest",), {old: TR_D1})
|
|
|
|
|
|
def isolate_onboarding_clone(cat):
|
|
"""R-781: the cases judge ONLY the records they build. The clone carries the REAL published records (the new apps',
|
|
and the exempt apps' shape-checked ones), and those cite evidence in the real felhom.eu — which the stand-in
|
|
sibling does not hold — so a genuine case read incomplete on an untouched tree (4 FAILs). Remove every real record
|
|
from the scratch clone, and every non-exempt template directory with it (without its record a real new app would
|
|
itself convict). The real tree is never touched; the real records stay judged by the real gate run."""
|
|
src = io.open(os.path.join(cat, "scripts", "check-onboarding.py"), encoding="utf-8").read()
|
|
exempt = set(re.search(r'EXEMPT = frozenset\("""(.*?)"""', src, re.S).group(1).split())
|
|
if len(exempt) < 40:
|
|
raise SystemExit("check-onboarding.py's EXEMPT list read as %d apps — the fixture drifted" % len(exempt))
|
|
onb = os.path.join(cat, "onboarding")
|
|
for name in os.listdir(onb):
|
|
if name.endswith(".md") and name != "_TEMPLATE.md":
|
|
os.remove(os.path.join(onb, name))
|
|
tdir = os.path.join(cat, "templates")
|
|
for name in os.listdir(tdir):
|
|
if os.path.isdir(os.path.join(tdir, name)) and name not in exempt:
|
|
shutil.rmtree(os.path.join(tdir, name))
|
|
|
|
|
|
def onboarding_cases():
|
|
"""The onboarding gate reads FILES — the checklist, the template, the records, and evidence paths that may live
|
|
in a SIBLING repository. So each case runs in its own scratch WORKSPACE: <ws>/app-catalog-felhom.eu (a clone,
|
|
with this working tree's checklist + template copied in — the files under test are the ones being edited) and
|
|
<ws>/felhom.eu (a stand-in sibling holding one evidence file). The real tree is never touched."""
|
|
global ran
|
|
ws = tempfile.mkdtemp(prefix="catalog-onboarding-")
|
|
cat = os.path.join(ws, "app-catalog-felhom.eu")
|
|
sh(["git", "clone", "-q", "file://" + ROOT, cat], cwd=ROOT)
|
|
for rel in ("NEW-APP-CHECKLIST.md", os.path.join("onboarding", "_TEMPLATE.md")):
|
|
os.makedirs(os.path.dirname(os.path.join(cat, rel)) or cat, exist_ok=True)
|
|
shutil.copy(os.path.join(ROOT, rel), os.path.join(cat, rel))
|
|
isolate_onboarding_clone(cat)
|
|
sib = os.path.join(ws, "felhom.eu", "documentation", "audits", "onb")
|
|
os.makedirs(sib)
|
|
with io.open(os.path.join(sib, "proof.txt"), "w", encoding="utf-8") as fh:
|
|
fh.write("measured\n")
|
|
shutil.copytree(os.path.join(cat, "templates", "vaultwarden"), os.path.join(cat, "templates", "newapp"))
|
|
ev = os.path.join(cat, "onboarding", "evidence", "newapp")
|
|
os.makedirs(ev)
|
|
with io.open(os.path.join(ev, "e.txt"), "w", encoding="utf-8") as fh:
|
|
fh.write("bench output\n")
|
|
ids = [m.group(1) for m in (re.match(r"^(\d+\.\d+) \|", l) for l in
|
|
io.open(os.path.join(cat, "onboarding", "_TEMPLATE.md"), encoding="utf-8")) if m]
|
|
if len(ids) < 50:
|
|
raise SystemExit("the template carries %d ids — the fixture drifted, not the gate" % len(ids))
|
|
|
|
def record(rows=None, opened="2026-10-01", app="newapp"):
|
|
rows = rows if rows is not None else ["%s | done | app-catalog-felhom.eu/onboarding/evidence/newapp/e.txt" % i
|
|
for i in ids]
|
|
return "# Onboarding record\n\napp: %s\nopened: %s\n\n%s\n" % (app, opened, "\n".join(rows))
|
|
|
|
def full(**over):
|
|
out = []
|
|
for i in ids:
|
|
out.append(over.get(i, "%s | done | app-catalog-felhom.eu/onboarding/evidence/newapp/e.txt" % i))
|
|
return [r for r in out if r is not None]
|
|
|
|
REC = os.path.join(cat, "onboarding", "newapp.md")
|
|
|
|
def case_onb(name, setup, expect_rc, must=()):
|
|
global ran
|
|
ran += 1
|
|
try:
|
|
setup()
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-onboarding.py"), "--root=" + cat,
|
|
"--today=2026-10-02"], cwd=cat)
|
|
out = r.stdout + r.stderr
|
|
if r.returncode == expect_rc and all(m in out for m in must):
|
|
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, [m for m in must if m not in out], out[-900:]))
|
|
finally:
|
|
for f in (REC, os.path.join(cat, "onboarding", "wger.md")):
|
|
if os.path.exists(f):
|
|
os.remove(f)
|
|
shutil.copy(os.path.join(ROOT, "NEW-APP-CHECKLIST.md"), os.path.join(cat, "NEW-APP-CHECKLIST.md"))
|
|
shutil.copy(os.path.join(ROOT, "onboarding", "_TEMPLATE.md"), os.path.join(cat, "onboarding", "_TEMPLATE.md"))
|
|
empty = os.path.join(cat, "onboarding", "evidence", "hollow")
|
|
if os.path.isdir(empty):
|
|
shutil.rmtree(empty)
|
|
|
|
def put(text, path=REC):
|
|
def f():
|
|
with io.open(path, "w", encoding="utf-8") as fh:
|
|
fh.write(text)
|
|
return f
|
|
|
|
try:
|
|
print("\n-- onboarding: the facts (each MUST be refused)")
|
|
case_onb("FACT: a new template with NO record", lambda: None, 1, ("newapp: NEW app with no onboarding record",))
|
|
case_onb("FACT: a record missing id 1.4", put(record(full(**{"1.4": None}))), 1, ("missing id(s): 1.4",))
|
|
case_onb("FACT: 1.4 answered only inside an HTML comment",
|
|
put(record(full(**{"1.4": "<!--\n1.4 | done | app-catalog-felhom.eu/onboarding/evidence/newapp/e.txt\n-->"}))),
|
|
1, ("missing id(s): 1.4",))
|
|
case_onb("FACT: done with a path that does not exist",
|
|
put(record(full(**{"2.5": "2.5 | done | app-catalog-felhom.eu/onboarding/evidence/newapp/restore.txt"}))),
|
|
1, ("id 2.5 is done but its evidence", "restore.txt"))
|
|
def hollow():
|
|
os.makedirs(os.path.join(cat, "onboarding", "evidence", "hollow"))
|
|
put(record(full(**{"5.1": "5.1 | done | app-catalog-felhom.eu/onboarding/evidence/hollow"})))()
|
|
case_onb("FACT: done with an EMPTY directory (the mkdir shape)", hollow, 1, ("id 5.1 is done but its evidence",))
|
|
case_onb("FACT: done naming an absent file in the sibling repo",
|
|
put(record(full(**{"3.6": "3.6 | done | felhom.eu/documentation/audits/onb/lockout.txt"}))),
|
|
1, ("id 3.6 is done but its evidence",))
|
|
case_onb("FACT: n/a with an EMPTY reason", put(record(full(**{"7.1": "7.1 | n/a | "}))), 1, ("id 7.1 is n/a",))
|
|
case_onb("FACT: n/a with a two-word reason", put(record(full(**{"7.1": "7.1 | n/a | not needed"}))), 1, ("id 7.1 is n/a",))
|
|
case_onb("FACT: an OPEN row", put(record(full(**{"6.3": "6.3 | open | the forced-fail case is not run yet"}))),
|
|
1, ("id 6.3 is OPEN",))
|
|
case_onb("FACT: opened: backdated before the checklist", put(record(full(), opened="2026-09-01")),
|
|
1, ("before the checklist existed",))
|
|
def new_id_template_lacks():
|
|
t = io.open(os.path.join(cat, "NEW-APP-CHECKLIST.md"), encoding="utf-8").read()
|
|
t = t.replace("\n## 7. Mail", "\n| 6.9 | 2026-10-01 | a new check | how | why |\n\n## 7. Mail", 1)
|
|
io.open(os.path.join(cat, "NEW-APP-CHECKLIST.md"), "w", encoding="utf-8").write(t)
|
|
put(record(full() + ["6.9 | done | app-catalog-felhom.eu/onboarding/evidence/newapp/e.txt"]))()
|
|
case_onb("FACT: a checklist id the template a new app copies lacks", new_id_template_lacks, 1,
|
|
("_TEMPLATE.md lacks checklist id(s): 6.9",))
|
|
case_onb("FACT: an exempt app's record with a done that points nowhere",
|
|
lambda: (put(record(full()))(), put(record(["1.5 | done | app-catalog-felhom.eu/nowhere.txt"],
|
|
app="wger"), os.path.join(cat, "onboarding", "wger.md"))()),
|
|
1, ("wger: id 1.5 is done but its evidence",))
|
|
|
|
print("-- onboarding: the genuine articles (each MUST pass)")
|
|
case_onb("GENUINE: a complete record (catalog + sibling evidence)",
|
|
put(record(full(**{"3.6": "3.6 | done | felhom.eu/documentation/audits/onb/proof.txt — measured on 9202",
|
|
"7.1": "7.1 | n/a | the app sends no mail at all"}))), 0, ("onboarding gate OK",))
|
|
def later_id():
|
|
t = io.open(os.path.join(cat, "NEW-APP-CHECKLIST.md"), encoding="utf-8").read()
|
|
t = t.replace("\n## 7. Mail", "\n| 6.9 | 2026-11-01 | a later check | how | why |\n\n## 7. Mail", 1)
|
|
io.open(os.path.join(cat, "NEW-APP-CHECKLIST.md"), "w", encoding="utf-8").write(t)
|
|
tp = os.path.join(cat, "onboarding", "_TEMPLATE.md")
|
|
io.open(tp, "a", encoding="utf-8").write("6.9 | open | not started: a later check\n")
|
|
put(record(full()))()
|
|
case_onb("GENUINE: an id added AFTER opened: does not bind", later_id, 0, ("onboarding gate OK",))
|
|
case_onb("GENUINE: an exempt app's record may say open",
|
|
lambda: (put(record(full()))(), put(record(["1.5 | open | the dev server runs (R-755)"], app="wger"),
|
|
os.path.join(cat, "onboarding", "wger.md"))()),
|
|
0, ("exempt app(s) with a record (shape-checked): wger",))
|
|
def no_sibling():
|
|
shutil.move(os.path.join(ws, "felhom.eu"), os.path.join(ws, "felhom.eu.away"))
|
|
put(record(full(**{"3.6": "3.6 | done | felhom.eu/documentation/audits/onb/lockout.txt"})))()
|
|
try:
|
|
case_onb("STATED SKIP: sibling repo absent (the CI shape) - printed, not checked", no_sibling, 0,
|
|
("NOT CHECKED here", "felhom.eu/documentation/audits/onb/lockout.txt"))
|
|
finally:
|
|
if os.path.isdir(os.path.join(ws, "felhom.eu.away")):
|
|
shutil.move(os.path.join(ws, "felhom.eu.away"), os.path.join(ws, "felhom.eu"))
|
|
finally:
|
|
shutil.rmtree(ws, ignore_errors=True)
|
|
|
|
def family_gate_cases():
|
|
"""check-family-gate.py reads FILES: templates/*/.felhom.yml and the sibling felhom.eu's golden bake records."""
|
|
global ran
|
|
import tempfile
|
|
ws = tempfile.mkdtemp(prefix="catalog-familygate-")
|
|
try:
|
|
cat, sib = os.path.join(ws, "cat"), os.path.join(ws, "felhom.eu")
|
|
def golden(ver, baked=True):
|
|
d = os.path.join(sib, "documentation", "tests", "golden-%s-2026-10-02" % ver)
|
|
os.makedirs(d, exist_ok=True)
|
|
if baked:
|
|
io.open(os.path.join(d, "bake.log"), "w").write("GOLDEN_SHA256=" + "a" * 64 + "\n")
|
|
def app(body, name="famapp"):
|
|
d = os.path.join(cat, "templates", name)
|
|
os.makedirs(d, exist_ok=True)
|
|
io.open(os.path.join(d, ".felhom.yml"), "w").write("display_name: X\n" + body)
|
|
GOOD = 'family_gate: true\nfamily_gate_except:\n - "/api/v1/opds" # e-readers\n - "/api/kobo/"\nmin_controller: "0.287.0"\n'
|
|
def run(name, setup, expect_rc, must=(), sibling=True):
|
|
global ran
|
|
shutil.rmtree(cat, ignore_errors=True); shutil.rmtree(sib, ignore_errors=True)
|
|
os.makedirs(os.path.join(cat, "templates"))
|
|
setup()
|
|
args = [sys.executable, os.path.join(ROOT, "scripts", "check-family-gate.py"), "--root=" + cat,
|
|
"--felhom-eu=" + (sib if sibling else os.path.join(ws, "absent"))]
|
|
r = sh(args, ROOT); out = r.stdout + r.stderr; ran += 1
|
|
ok = r.returncode == expect_rc and all(m in out for m in must)
|
|
print(" %s %-70s rc=%d (expected %d)" % ("ok" if ok else "XX", name, r.returncode, expect_rc))
|
|
if not ok:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (name, r.returncode, expect_rc, [m for m in must if m not in out], out[-400:]))
|
|
print("\n-- family-gate: genuine and decoys")
|
|
run("GENUINE: a family app, literal exceptions, min 0.287.0, golden 0.287.0", lambda: (app(GOOD), golden("0.287.0")), 0, ("family-gate gate OK",))
|
|
run("DECOY: family_gate only in a COMMENT -> not gated, nothing owed", lambda: (app("# family_gate: true\n"), golden("0.286.1")), 0, ("0 family-gated",))
|
|
run("DECOY: no sibling -> rule 3 stated NOT CHECKED", lambda: app(GOOD), 0, ("NOT CHECKED",), sibling=False)
|
|
print("-- family-gate: the facts (each MUST be refused)")
|
|
run("FACT: an exception that is a regex", lambda: (app(GOOD.replace('"/api/kobo/"', '"/api/(.*)"')), golden("0.287.0")), 1, ("not a literal path prefix",))
|
|
run("FACT: the exception '/' (the whole app)", lambda: (app(GOOD.replace('"/api/kobo/"', '"/"')), golden("0.287.0")), 1, ("not a literal",))
|
|
run("FACT: an exception with '..'", lambda: (app(GOOD.replace('"/api/kobo/"', '"/api/../admin"')), golden("0.287.0")), 1, ("not a literal",))
|
|
run("FACT: an exception list with no gate", lambda: (app('family_gate_except:\n - "/x"\n'), golden("0.287.0")), 1, ("with no gate",))
|
|
run("FACT: min_controller only in a comment", lambda: (app(GOOD.replace('min_controller: "0.287.0"', '# min_controller: "0.287.0"')), golden("0.287.0")), 1, ("needs min_controller",))
|
|
run("FACT: min_controller below the release", lambda: (app(GOOD.replace('0.287.0', '0.286.1')), golden("0.287.0")), 1, ("needs min_controller",))
|
|
run("FACT: newest baked golden below the release", lambda: (app(GOOD), golden("0.286.1")), 1, ("publish the app OPEN",))
|
|
run("FACT: a golden DIRECTORY 0.287.0 with no bake log (a name is not a bake)", lambda: (app(GOOD), golden("0.286.1"), golden("0.287.0", baked=False)), 1, ("0.286.1",))
|
|
finally:
|
|
shutil.rmtree(ws, ignore_errors=True)
|
|
|
|
|
|
def image_pins_cases():
|
|
"""check-image-pins.py reads templates/*/docker-compose.yml under --root (R-426).
|
|
|
|
Each case is ONE planted template in a scratch catalog, so a case's verdict is that line's verdict
|
|
and nothing else. The real tree is also judged (it must pass), so a gate that convicts everything
|
|
fails here too.
|
|
"""
|
|
global ran
|
|
ws = tempfile.mkdtemp(prefix="catalog-pins-")
|
|
DIG = "@sha256:" + "0123456789abcdef" * 4
|
|
try:
|
|
def run(name, image_line, expect_rc, must=(), extra_file=None):
|
|
global ran
|
|
cat = os.path.join(ws, "cat")
|
|
shutil.rmtree(cat, ignore_errors=True)
|
|
d = os.path.join(cat, "templates", "demo")
|
|
os.makedirs(d)
|
|
io.open(os.path.join(d, "docker-compose.yml"), "w", encoding="utf-8").write(
|
|
"services:\n demo:\n image: demo/demo:1.0\n demo-web:\n" + image_line + "\n"
|
|
" restart: unless-stopped\n")
|
|
if extra_file:
|
|
io.open(os.path.join(d, extra_file[0]), "w", encoding="utf-8").write(extra_file[1])
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-image-pins.py"), "--root=" + cat], ROOT)
|
|
out = r.stdout + r.stderr
|
|
ran += 1
|
|
if r.returncode == expect_rc and all(m in out for m in must):
|
|
print(" ok %-52s rc=%d (expected %d)" % ("image-pins: " + name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("image-pins: %s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, [m for m in must if m not in out], out[-600:]))
|
|
|
|
# ── THE FACTS: an image the box would pull floating. Each must be REFUSED. ───────────────
|
|
run("FACT: untagged", " image: nginx", 1, ("NO TAG",))
|
|
run("FACT: a registry PORT is not a tag", " image: registry.local:5000/nginx", 1, ("NO TAG",))
|
|
run("FACT: quoted :latest", ' image: "nginx:latest"', 1, ("floating tag :latest",))
|
|
run("FACT: :LATEST in capitals", " image: nginx:LATEST", 1, ("floating tag",))
|
|
run("FACT: a floating alias (:edge)", " image: alpine:edge", 1, ("floating tag :edge",))
|
|
run("FACT: untagged with a comment saying pinned", " image: nginx # pinned 1.27", 1, ("NO TAG",))
|
|
run("FACT: a QUOTED key is the same field", ' "image": nginx', 1, ("NO TAG",))
|
|
run("FACT: an interpolated ref cannot be read", " image: ${APP_IMAGE:-nginx}", 1, ("INTERPOLATED",))
|
|
run("FACT: @sha256: with no digest behind it", " image: nginx@sha256:pinned", 1, ("NOT A DIGEST",))
|
|
# ── INERT / GENUINE: must PASS ───────────────────────────────────────────────────────────
|
|
run("INERT: a commented-out untagged image", " # image: nginx", 0, ("image-pin gate OK",))
|
|
run("INERT: an x- extension field (Compose ignores it)", " x-image: nginx", 0, ("image-pin gate OK",))
|
|
run("INERT: an untagged image in README.md", " image: nginx:1.27.3", 0, ("image-pin gate OK",),
|
|
extra_file=("README.md", "image: nginx\n"))
|
|
run("GENUINE: a concrete tag", " image: nginx:1.27.3-alpine", 0, ("image-pin gate OK",))
|
|
run("GENUINE: a registry port WITH a tag", " image: registry.local:5000/nginx:1.27", 0, ("image-pin gate OK",))
|
|
run("GENUINE: a real digest", " image: nginx" + DIG, 0, ("image-pin gate OK",))
|
|
run("GENUINE: :latest pinned by its digest", " image: nginx:latest" + DIG, 0, ("image-pin gate OK",))
|
|
# the real catalog must pass too — a gate that convicts everything is not a gate
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-image-pins.py")], ROOT)
|
|
ran += 1
|
|
if r.returncode == 0:
|
|
print(" ok %-52s rc=0 (expected 0)" % "image-pins: GENUINE: the real catalog")
|
|
else:
|
|
fails.append("image-pins: the real catalog was refused rc=%d\n%s" % (r.returncode, (r.stdout + r.stderr)[-600:]))
|
|
finally:
|
|
shutil.rmtree(ws, ignore_errors=True)
|
|
|
|
|
|
STUB_DOCKER = """#!%s
|
|
# PATH-stub docker for the image-resolvable decoys. Answers ONLY `docker manifest inspect <ref>`, from a JSON table;
|
|
# logs every argv. It never runs anything.
|
|
import json, os, sys
|
|
table = json.load(open(os.environ["DECOY_DOCKER_TABLE"]))
|
|
with open(os.environ["DECOY_DOCKER_LOG"], "a") as fh:
|
|
fh.write(json.dumps(sys.argv[1:]) + "\\n")
|
|
if sys.argv[1:3] != ["manifest", "inspect"] or len(sys.argv) != 4:
|
|
sys.stderr.write("stub docker: unexpected call %%r\\n" %% (sys.argv[1:],))
|
|
sys.exit(97)
|
|
rc, err = table.get(sys.argv[3], table["*"])
|
|
sys.stderr.write(err)
|
|
sys.exit(rc)
|
|
"""
|
|
|
|
|
|
def image_resolvable_cases():
|
|
"""check-image-resolvable.py END TO END, its docker replaced by a PATH stub (R-426).
|
|
|
|
The gate's unit tests inject `resolver` and so never run `docker_resolver` — the function that turns a real
|
|
`docker manifest inspect` exit code and stderr into a verdict, which is where both of its live traps sit. Here the
|
|
working-tree script is copied into a scratch catalog and run with PATH = a directory holding ONLY the stub, so the
|
|
real docker (which acts on DooPlex) cannot be reached even by accident, and nothing touches the network.
|
|
"""
|
|
global ran
|
|
ws = tempfile.mkdtemp(prefix="catalog-resolvable-")
|
|
GOOD, DEAD = "example.org/alive/app:1.0", "example.org/rotten/app:2.0"
|
|
try:
|
|
stub = os.path.join(ws, "stubbin")
|
|
os.makedirs(stub)
|
|
io.open(os.path.join(stub, "docker"), "w", encoding="utf-8").write(STUB_DOCKER % sys.executable)
|
|
os.chmod(os.path.join(stub, "docker"), 0o755)
|
|
empty = os.path.join(ws, "nobin")
|
|
os.makedirs(empty)
|
|
|
|
def run(name, table, expect_rc, must=(), path=stub, templates=True, asked=()):
|
|
global ran
|
|
cat = os.path.join(ws, "cat")
|
|
shutil.rmtree(cat, ignore_errors=True)
|
|
os.makedirs(os.path.join(cat, "scripts"))
|
|
shutil.copy(os.path.join(ROOT, "scripts", "check-image-resolvable.py"), os.path.join(cat, "scripts"))
|
|
if templates:
|
|
for app, img in (("alive", GOOD), ("rotten", DEAD)):
|
|
d = os.path.join(cat, "templates", app)
|
|
os.makedirs(d)
|
|
io.open(os.path.join(d, "docker-compose.yml"), "w", encoding="utf-8").write(
|
|
"services:\n %s:\n image: %s\n" % (app, img))
|
|
tab, log = os.path.join(ws, "table.json"), os.path.join(ws, "calls.log")
|
|
json.dump(table, io.open(tab, "w", encoding="utf-8"))
|
|
io.open(log, "w").close()
|
|
env = {"PATH": path, "DECOY_DOCKER_TABLE": tab, "DECOY_DOCKER_LOG": log}
|
|
r = subprocess.run([sys.executable, os.path.join(cat, "scripts", "check-image-resolvable.py")],
|
|
cwd=cat, env=env, capture_output=True, text=True, input="")
|
|
out = r.stdout + r.stderr
|
|
calls = io.open(log).read()
|
|
ran += 1
|
|
miss = [m for m in must if m not in out] + ["(never asked docker about %s)" % a for a in asked if a not in calls]
|
|
if r.returncode == expect_rc and not miss:
|
|
print(" ok %-52s rc=%d (expected %d)" % ("image-resolvable: " + name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("image-resolvable: %s: rc=%d expected %d%s; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, " - LIVE HOLE" if expect_rc != 0 and r.returncode == 0 else "",
|
|
miss, out[-600:]))
|
|
|
|
GONE = [1, "manifest unknown: manifest unknown\n"]
|
|
okall = {"*": [0, ""], "felhom-nonexistent.invalid/no/such:image": GONE}
|
|
run("GENUINE: every pin resolves", okall, 0, ("all resolve",), asked=(GOOD, DEAD))
|
|
run("FACT: the registry says one pin is gone", dict(okall, **{DEAD: GONE}), 1,
|
|
("GONE", "pinned at templates/rotten:3"), asked=(DEAD,))
|
|
run("LABEL: rc=0 carrying a throttle message", dict(okall, **{DEAD: [0, "toomanyrequests: rate limit\n"]}), 2,
|
|
("INCONCLUSIVE",))
|
|
run("LABEL: rc=0 carrying any error text", dict(okall, **{DEAD: [0, "error: half an answer\n"]}), 2,
|
|
("INCONCLUSIVE",))
|
|
run("CRY-WOLF: a throttle on rc=1 is not an accusation", dict(okall, **{DEAD: [1, "toomanyrequests: slow down\n"]}),
|
|
2, ("INCONCLUSIVE",))
|
|
run("CRY-WOLF: an unrecognised failure is not an accusation", dict(okall, **{DEAD: [1, "error: something odd\n"]}),
|
|
2, ("INCONCLUSIVE",))
|
|
run("LABEL: a docker that resolves EVERYTHING, the canary too", {"*": [0, ""]}, 3,
|
|
("HARNESS REFUSED",))
|
|
run("LABEL: no docker at all", okall, 2, ("could not run docker",), path=empty)
|
|
run("LABEL: nothing to judge", okall, 3, ("HARNESS REFUSED",), templates=False)
|
|
finally:
|
|
shutil.rmtree(ws, ignore_errors=True)
|
|
|
|
|
|
FAIL_DOCKER = """#!%s
|
|
# PATH-stub docker for the volume-persistence decoys: a host whose runtime answers nothing. Every call fails.
|
|
import sys
|
|
sys.stderr.write("Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\\n")
|
|
sys.exit(1)
|
|
"""
|
|
|
|
|
|
def volume_persistence_cases():
|
|
"""check-volume-persistence.py (R-426). Two halves.
|
|
|
|
1. Its classifier and self-test are pure and already fixture-tested with an INJECTED prober — the blind prober,
|
|
the crying-wolf prober, `wrote nothing`, the papra signature. Those ARE decoys; this suite runs that file and
|
|
requires it green, so the COVERS line above is a fact and not a label (felhom.eu's guide-quote precedent).
|
|
2. End to end, the working-tree script copied into a scratch catalog and run with PATH = ONLY a stub docker. The
|
|
stub FAILS every call, so the gate stops at its canary build and nothing — no compose, no container, no
|
|
volume — is ever created; an always-succeeding stub is deliberately NOT used, because it would walk the
|
|
prober into the host filesystem. The real docker acts on DooPlex and cannot be reached from here.
|
|
"""
|
|
global ran
|
|
ran += 1
|
|
r = subprocess.run([sys.executable, os.path.join(ROOT, "scripts", "test_check_volume_persistence.py")],
|
|
cwd=ROOT, capture_output=True, text=True, input="")
|
|
tail = (r.stdout + r.stderr).strip().splitlines()
|
|
if r.returncode != 0:
|
|
fails.append("volume-persistence: its injected-prober decoy suite FAILED rc=%d\n%s"
|
|
% (r.returncode, "\n".join(tail[-25:])))
|
|
else:
|
|
print(" ok %-52s %s" % ("volume-persistence: injected-prober suite", tail[-1] if tail else "?"))
|
|
|
|
ws = tempfile.mkdtemp(prefix="catalog-volpersist-")
|
|
try:
|
|
stub = os.path.join(ws, "stubbin")
|
|
os.makedirs(stub)
|
|
io.open(os.path.join(stub, "docker"), "w", encoding="utf-8").write(FAIL_DOCKER % sys.executable)
|
|
os.chmod(os.path.join(stub, "docker"), 0o755)
|
|
empty = os.path.join(ws, "nobin")
|
|
os.makedirs(empty)
|
|
|
|
def run(name, expect_rc, must=(), path=stub, templates=True):
|
|
global ran
|
|
cat = os.path.join(ws, "cat")
|
|
shutil.rmtree(cat, ignore_errors=True)
|
|
os.makedirs(os.path.join(cat, "scripts"))
|
|
shutil.copy(os.path.join(ROOT, "scripts", "check-volume-persistence.py"), os.path.join(cat, "scripts"))
|
|
if templates:
|
|
d = os.path.join(cat, "templates", "demo")
|
|
os.makedirs(d)
|
|
io.open(os.path.join(d, "docker-compose.yml"), "w", encoding="utf-8").write(
|
|
"services:\n demo:\n image: example.org/demo:1.0\n volumes:\n - demo_data:/data\n"
|
|
"volumes:\n demo_data:\n")
|
|
r = subprocess.run([sys.executable, os.path.join(cat, "scripts", "check-volume-persistence.py")],
|
|
cwd=cat, env={"PATH": path}, capture_output=True, text=True, input="", timeout=300)
|
|
out = r.stdout + r.stderr
|
|
ran += 1
|
|
miss = [m for m in must if m not in out]
|
|
if r.returncode == expect_rc and not miss:
|
|
print(" ok %-52s rc=%d (expected %d)" % ("volume-persistence: " + name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("volume-persistence: %s: rc=%d expected %d%s; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, " - LIVE HOLE" if r.returncode == 0 else "", miss, out[-600:]))
|
|
|
|
run("LABEL: a runtime that answers nothing", 3, ("HARNESS REFUSED", "failed its canary"))
|
|
run("LABEL: no docker at all", 3, ("HARNESS REFUSED",), path=empty)
|
|
run("LABEL: nothing to judge", 3, ("HARNESS REFUSED",), templates=False)
|
|
finally:
|
|
shutil.rmtree(ws, ignore_errors=True)
|
|
|
|
|
|
def mem_sum_cases():
|
|
"""check-mem-limit-sum.py reads FILES: templates/*/docker-compose.yml + .felhom.yml, via --root (R-758)."""
|
|
global ran
|
|
import tempfile
|
|
ws = tempfile.mkdtemp(prefix="catalog-memsum-")
|
|
try:
|
|
COMPOSE = (
|
|
"# demo - header\n# RAM: ~100M (mem_limit: 640M)\n"
|
|
"services:\n"
|
|
" demo:\n image: demo/demo:1.0\n container_name: demo\n environment:\n - X=1\n"
|
|
" deploy:\n resources:\n limits:\n memory: 384M\n"
|
|
" demo-db:\n image: postgres:16-alpine\n command: |\n memory: 9999M\n"
|
|
" deploy:\n resources:\n limits:\n memory: 256M # the DB\n"
|
|
"volumes:\n demo_data:\n")
|
|
META = ('display_name: "Demo"\n# mem_limit: "999M" is not this line\n'
|
|
'resources:\n mem_request: "100M"\n mem_limit: "640M" # 384+256\n pi_compatible: true\n')
|
|
|
|
def run(name, compose, meta, expect_rc, must=(), extra=None):
|
|
global ran
|
|
cat = os.path.join(ws, "cat")
|
|
shutil.rmtree(cat, ignore_errors=True)
|
|
d = os.path.join(cat, "templates", "demo")
|
|
os.makedirs(d)
|
|
io.open(os.path.join(d, "docker-compose.yml"), "w", encoding="utf-8").write(compose)
|
|
io.open(os.path.join(d, ".felhom.yml"), "w", encoding="utf-8").write(meta)
|
|
if extra:
|
|
extra(d)
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-mem-limit-sum.py"), "--root=" + cat], ROOT)
|
|
out = r.stdout + r.stderr
|
|
ran += 1
|
|
ok = r.returncode == expect_rc and all(m in out for m in must)
|
|
print(" %s %-70s rc=%d (expected %d)" % ("ok" if ok else "XX", name, r.returncode, expect_rc))
|
|
if not ok:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, [m for m in must if m not in out], out[-400:]))
|
|
|
|
def steps(d):
|
|
os.makedirs(os.path.join(d, "steps"))
|
|
io.open(os.path.join(d, "steps", "abc.felhom.yml"), "w").write('resources:\n mem_limit: "384M"\n')
|
|
|
|
print("\n-- mem-limit-sum: genuine and decoys")
|
|
run("GENUINE: 384+256 = 640M, the field says 640M", COMPOSE, META, 0, ("mem-limit-sum gate OK",))
|
|
run("GENUINE: 1G is 1024M (1G + 256M = 1280M)", COMPOSE.replace("memory: 384M", "memory: 1G"),
|
|
META.replace('"640M"', '"1280M"'), 0, ("gate OK",))
|
|
run("DECOY: a steps/ file with an old figure is not judged", COMPOSE, META, 0, ("gate OK",), extra=steps)
|
|
print("-- mem-limit-sum: the facts (each MUST be refused)")
|
|
run("FACT: the field under the sum; the right figure only in comments", COMPOSE,
|
|
META.replace('mem_limit: "640M" # 384+256', 'mem_limit: "384M" # 384+256=640M'), 1, ("not the sum", "384+256=640M"))
|
|
run("FACT: reservations: memory makes nothing a limit (service w/o limit)",
|
|
COMPOSE.replace(" limits:\n memory: 256M", " reservations:\n memory: 256M"),
|
|
META.replace('"640M"', '"384M"'), 1, ("NO deploy.resources.limits.memory", "demo-db"))
|
|
run("FACT: mem_limit only OUTSIDE resources: (top level) is no declaration", COMPOSE,
|
|
META.replace(' mem_limit: "640M" # 384+256\n', '').replace('display_name: "Demo"\n', 'display_name: "Demo"\nmem_limit: "640M"\n'),
|
|
1, ("declares no resources.mem_limit",))
|
|
run("FACT: an unreadable size is INCONCLUSIVE, never a pass", COMPOSE.replace("memory: 384M", "memory: lots"), META, 2,
|
|
("INCONCLUSIVE",))
|
|
finally:
|
|
shutil.rmtree(ws, ignore_errors=True)
|
|
|
|
|
|
def data_key_cases():
|
|
"""check-data-key.py reads templates/*/.felhom.yml via --root (R-127 leg a)."""
|
|
global ran
|
|
import tempfile
|
|
ws = tempfile.mkdtemp(prefix="catalog-datakey-")
|
|
try:
|
|
GOOD = ('display_name: "X"\n'
|
|
'deploy_fields:\n'
|
|
' - env_var: DOMAIN\n type: domain\n'
|
|
' - env_var: SECRET_KEY\n label: "Titkositasi kulcs"\n type: secret\n generate: "hex:32"\n'
|
|
' - env_var: APP_ENCRYPTION_KEY\n type: secret\n generate: "hex:16"\n'
|
|
' # why: encrypts stored data\n data_key: true\n'
|
|
'\ni18n:\n en:\n deploy_fields:\n - env_var: SECRET_KEY\n label: "Encryption key"\n')
|
|
BOOK = ('deploy_fields:\n - env_var: APP_KEY\n type: secret\n generate: "base64key:32"\n data_key: true\n')
|
|
|
|
def run(name, files, expect_rc, must=()):
|
|
global ran
|
|
cat = os.path.join(ws, "cat")
|
|
shutil.rmtree(cat, ignore_errors=True)
|
|
for app, text in files.items():
|
|
d = os.path.join(cat, "templates", app)
|
|
os.makedirs(d)
|
|
io.open(os.path.join(d, ".felhom.yml"), "w", encoding="utf-8").write(text)
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-data-key.py"), "--root=" + cat], ROOT)
|
|
out = r.stdout + r.stderr
|
|
ran += 1
|
|
ok = r.returncode == expect_rc and all(m in out for m in must)
|
|
print(" %s %-70s rc=%d (expected %d)" % ("ok" if ok else "XX", name, r.returncode, expect_rc))
|
|
if not ok:
|
|
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
|
name, r.returncode, expect_rc, [m for m in must if m not in out], out[-400:]))
|
|
|
|
print("\n-- data-key: genuine and decoys")
|
|
run("GENUINE: a flagged *_ENCRYPTION_KEY; a signing key labelled 'Encryption key' unflagged; bookstack registered",
|
|
{"demo": GOOD, "bookstack": BOOK}, 0, ("data-key gate OK", "2 data key(s)"))
|
|
print("-- data-key: the facts (each MUST be refused)")
|
|
run("FACT: the flag only in a COMMENT", {"demo": GOOD.replace(" data_key: true\n", " # data_key: true\n"),
|
|
"bookstack": BOOK}, 1, ("demo/APP_ENCRYPTION_KEY is a data key",))
|
|
run("FACT: the flag only inside the i18n: block", {"demo": GOOD.replace(" data_key: true\n", "") +
|
|
" data_key: true\n", "bookstack": BOOK}, 1,
|
|
("demo/APP_ENCRYPTION_KEY is a data key",))
|
|
run("FACT: a REGISTERED data key unflagged (bookstack APP_KEY)",
|
|
{"demo": GOOD, "bookstack": BOOK.replace(" data_key: true\n", "")}, 1, ("bookstack/APP_KEY is a data key", "registered"))
|
|
run("FACT: an unexplained flag (a signing key flagged)", {"demo": GOOD.replace(
|
|
' generate: "hex:32"\n', ' generate: "hex:32"\n data_key: true\n', 1), "bookstack": BOOK}, 1,
|
|
("demo/SECRET_KEY carries `data_key: true` but neither",))
|
|
run("FACT: a stale REGISTRY entry (bookstack has no APP_KEY any more)",
|
|
{"demo": GOOD, "bookstack": BOOK.replace("APP_KEY", "OTHER_KEY").replace(" data_key: true\n", "")}, 1, ("STALE",))
|
|
run("FACT: a non-boolean flag is INCONCLUSIVE", {"demo": GOOD.replace("data_key: true", "data_key: yes"),
|
|
"bookstack": BOOK}, 2, ("INCONCLUSIVE",))
|
|
finally:
|
|
shutil.rmtree(ws, ignore_errors=True)
|
|
|
|
|
|
def main():
|
|
gate = os.path.join(ROOT, "scripts", "check-engine-major.py")
|
|
if not os.path.isfile(gate):
|
|
print("FAIL: scripts/check-engine-major.py is missing — a failure, never a skip")
|
|
return 1
|
|
clone = make_clone()
|
|
try:
|
|
KIMAI = "templates/kimai/docker-compose.yml"
|
|
DOCMOST = "templates/docmost/docker-compose.yml"
|
|
# The engine ref is READ from the clone, not typed: the night of 2026-09-23 moved kimai-db
|
|
# 11.6 -> 11.8 through its own test record, and a literal here broke every case below
|
|
# ("fixture drifted") without a single gate changing. R-663.
|
|
KDB = cur_image(clone, KIMAI, "kimai-db")
|
|
# docmost-postgres too (2026-09-29): it moved 16 -> 18 through its own ladder, and the typed "16-alpine" here
|
|
# made every run fail "fixture drifted" before a single case ran — the same shape as R-663. Read it.
|
|
DMDB = cur_image(clone, DOCMOST, "docmost-postgres")
|
|
DM_MAJ = int(DMDB.split(":")[1].split("-")[0].split(".")[0])
|
|
DM_NEXT = DMDB.replace(":%d" % DM_MAJ, ":%d" % (DM_MAJ + 1), 1)
|
|
# kimai's own image too (2026-09-30): it moved apache-2.57.0 -> 2.67.0 through its ladder, and the typed tag here
|
|
# failed every run "fixture drifted" before a single case ran (R-663's shape a third time). Read it.
|
|
KAPP = cur_image(clone, KIMAI, "kimai")
|
|
_krepo, _ktag = KAPP.rsplit(":", 1)
|
|
_kpre = _ktag[:len(_ktag) - len(_ktag.lstrip("abcdefghijklmnopqrstuvwxyz-"))]
|
|
_kmaj, _kmin = (int(x) for x in _ktag[len(_kpre):].split(".")[:2])
|
|
KAPP_MINOR = "%s:%s%d.%d.0" % (_krepo, _kpre, _kmaj, _kmin + 1)
|
|
KAPP_MAJOR = "%s:%s%d.0.0" % (_krepo, _kpre, _kmaj + 1)
|
|
if not KDB.startswith("mariadb:11."):
|
|
raise SystemExit("kimai-db is %s — the cases below assume a MariaDB 11 line; fixture drifted" % KDB)
|
|
|
|
# ── THE FACTS: these must be refused ─────────────────────────────────────────────────
|
|
out = case("FACT: docmost-postgres major + 1 (read from the clone)", clone,
|
|
[(DOCMOST, lambda t: swap_image("docmost-postgres", DMDB, DM_NEXT)(t))],
|
|
expect_rc=1,
|
|
must_contain=("ENGINE-MAJOR GATE FAILED", "docmost-postgres", "postgres %d -> %d" % (DM_MAJ, DM_MAJ + 1)))
|
|
if "REFUSAL_TEXT" in os.environ:
|
|
print(out)
|
|
case("FACT: docmost-postgres major + 1, alone", clone,
|
|
[(DOCMOST, swap_image("docmost-postgres", DMDB, DM_NEXT))],
|
|
expect_rc=1, must_contain=("docmost-postgres", "postgres %d -> %d" % (DM_MAJ, DM_MAJ + 1), "R-463"))
|
|
# R-469 + R-450 (2026-09-21): a MariaDB major bundled with the app's own bump is the
|
|
# bookstack 0b73e5e shape — two migrations behind one edge — and stays refused.
|
|
case("FACT: kimai-db 11.6 -> 12.3 BUNDLED with the kimai app bump", clone,
|
|
[(KIMAI, lambda t: swap_image("kimai", KAPP, KAPP_MINOR)(
|
|
swap_image("kimai-db", KDB, "mariadb:12.3")(t)))],
|
|
expect_rc=1, must_contain=("IN THE SAME COMMIT as kimai", "OWN EDGE", "R-450"))
|
|
case("FACT: kimai-db mariadb:11.6 -> mariadb:lts (major unreadable)", clone,
|
|
[(KIMAI, swap_image("kimai-db", KDB, "mariadb:lts"))],
|
|
expect_rc=2, must_contain=("INCONCLUSIVE",))
|
|
|
|
# ── THE GENUINE ARTICLES: these must pass ────────────────────────────────────────────
|
|
case("GENUINE: kimai-db mariadb:11.x -> 11.99 (within major)", clone,
|
|
[(KIMAI, swap_image("kimai-db", KDB, "mariadb:11.99"))],
|
|
expect_rc=0, must_contain=("engine-major gate OK",))
|
|
# R-469: the LIFT itself. A MariaDB major ALONE in its template is now permitted, and the
|
|
# gate says so by name rather than passing in silence.
|
|
case("GENUINE: kimai-db mariadb:11.6 -> 12.3 ALONE (the R-469 lift)", clone,
|
|
[(KIMAI, swap_image("kimai-db", KDB, "mariadb:12.3"))],
|
|
expect_rc=0, must_contain=("ALLOWED", "kimai-db", "mariadb 11 -> 12", "R-469"))
|
|
|
|
# ── `09` §3 decision 35 (2026-09-25): a PostgreSQL major passes ONLY with its proven,
|
|
# two-venue, MARKED ladder entry, alone in its commit. The fact is the entry's CONTENT for
|
|
# THIS step — never the word "engine_conversion" somewhere, never one venue.
|
|
DOCMOST_FY = "templates/docmost/.felhom.yml"
|
|
DM, DR = cur_image(clone, DOCMOST, "docmost"), cur_image(clone, DOCMOST, "docmost-redis")
|
|
DPG = cur_image(clone, DOCMOST, "docmost-postgres")
|
|
# Read, never typed (2026-09-29: docmost moved 16 -> 18 and the typed 16 stopped every run). The conversion
|
|
# case moves the CURRENT major one up.
|
|
PG_FROM = int(DPG.split(":")[1].split("-")[0].split(".")[0]); PG_TO = PG_FROM + 1
|
|
PG_TO_REF = DPG.replace(":%d" % PG_FROM, ":%d" % PG_TO, 1)
|
|
def pg_entry(mark=True, box=True, to_pg=None, extra=""):
|
|
e = {"from": {"docmost": DM, "docmost-postgres": DPG, "docmost-redis": DR},
|
|
"to": {"docmost": DM, "docmost-postgres": to_pg or PG_TO_REF, "docmost-redis": DR},
|
|
"digest": {"docmost": "sha256:" + "a" * 64, "docmost-postgres": "sha256:" + "b" * 64, "docmost-redis": "sha256:" + "c" * 64},
|
|
"verdict": "proven", "tested_at": "2026-09-25T20:00:00Z", "harness_version": 4,
|
|
"evidence": "x/bench.json", "box_evidence": "x/box.json" if box else None, "memory_peak_pct": 20.0,
|
|
"marks": {"files_may_change": False, "needs_person": None, "memory_tight": False}}
|
|
if mark:
|
|
e["engine_conversion"] = {"service": "docmost-postgres", "engine": "postgres", "from": PG_FROM, "to": PG_TO}
|
|
return lambda t: t.rstrip("\n") + "\n - " + json.dumps(e) + "\n" + extra
|
|
pg18 = swap_image("docmost-postgres", DPG, PG_TO_REF)
|
|
case("GENUINE: docmost-postgres major+1 ALONE with its proven two-venue MARKED entry", clone,
|
|
[(DOCMOST, pg18), (DOCMOST_FY, pg_entry())],
|
|
expect_rc=0, must_contain=("ALLOWED", "docmost-postgres", "postgres %d -> %d" % (PG_FROM, PG_TO), "decision 35"))
|
|
case("DECOY: the entry is proven on both venues but carries NO conversion mark", clone,
|
|
[(DOCMOST, pg18), (DOCMOST_FY, pg_entry(mark=False))],
|
|
expect_rc=1, must_contain=("NOT PROVEN FOR THIS APP", "engine_conversion None"))
|
|
case("DECOY: the marked entry cites ONE venue only (no box_evidence)", clone,
|
|
[(DOCMOST, pg18), (DOCMOST_FY, pg_entry(box=False))],
|
|
expect_rc=1, must_contain=("NOT PROVEN FOR THIS APP", "BOTH venues"))
|
|
case("DECOY: the mark sits in a COMMENT, the entry has none", clone,
|
|
[(DOCMOST, pg18), (DOCMOST_FY, pg_entry(mark=False, extra='# engine_conversion: {"service": "docmost-postgres", "engine": "postgres", "from": PG_FROM, "to": PG_TO}\n'))],
|
|
expect_rc=1, must_contain=("NOT PROVEN FOR THIS APP",))
|
|
case("DECOY: the marked entry, but the move is BUNDLED with the app's own bump", clone,
|
|
[(DOCMOST, lambda t: swap_image("docmost", DM, DM + "-next")(pg18(t))), (DOCMOST_FY, pg_entry())],
|
|
expect_rc=1, must_contain=("ENGINE-MAJOR GATE FAILED", "docmost-postgres"))
|
|
case("FACT: adventurelog's postgis 16 -> 17 (the postgis family was never judged before)", clone,
|
|
[("templates/adventurelog/docker-compose.yml",
|
|
swap_image("adventurelog-postgres", cur_image(clone, "templates/adventurelog/docker-compose.yml", "adventurelog-postgres"), "postgis/postgis:17-3.5-alpine"))],
|
|
expect_rc=1, must_contain=("adventurelog-postgres", "postgis 16 -> 17"))
|
|
|
|
# ── THE DECOYS: the label moves, the fact does not — these must pass ─────────────────
|
|
def comment_and_env(text):
|
|
# the version string moves in a COMMENT and in kimai's serverVersion env, image untouched
|
|
t = text.replace("serverVersion=11.6.2-MariaDB", "serverVersion=12.3.0-MariaDB")
|
|
return t.replace("# Database: mariadb", "# Database: mariadb (image: mariadb:12.3 soon)")
|
|
case("DECOY: major moves only in a comment + serverVersion env", clone,
|
|
[(KIMAI, comment_and_env)], expect_rc=0, must_contain=("engine-major gate OK",))
|
|
case("DECOY: the APP image crosses a major (kimai 2.57 -> 3.0)", clone,
|
|
[(KIMAI, swap_image("kimai", KAPP, KAPP_MAJOR))],
|
|
expect_rc=0, must_contain=("engine-major gate OK",))
|
|
case("DECOY: 'mariadb:12.3' lands in README.md, not a template", clone,
|
|
[("README.md", lambda t: t + "\nDecoy: mariadb:11.6 -> mariadb:12.3 pending.\n")],
|
|
expect_rc=0, must_contain=("0 compose file(s) changed",))
|
|
|
|
# ── catalog-since (R-452): an image move must bump the app's catalog_since ───────────
|
|
import datetime
|
|
today = datetime.date.today().isoformat()
|
|
KIMAI_FY = "templates/kimai/.felhom.yml"
|
|
CS = "check-catalog-since.py"
|
|
def set_since(date):
|
|
def _fn(text):
|
|
new, n = re.subn(r'^catalog_since:\s*"?\d{4}-\d{2}-\d{2}"?', 'catalog_since: "%s"' % date, text, count=1, flags=re.M)
|
|
if n == 0: # the FIELD is missing (2026-09-30: "unchanged" is not drift — kimai's date was today)
|
|
raise SystemExit("kimai's .felhom.yml carries no catalog_since — fixture drifted")
|
|
return new
|
|
return _fn
|
|
# The cases below need kimai's catalog_since to be a PAST date at HEAD (2026-09-30: kimai moved that day, so
|
|
# "untouched" and "set to today" were the same text). The throwaway clone gets one fixture commit.
|
|
edit(clone, KIMAI_FY, lambda t: re.sub(r'^catalog_since:.*$', 'catalog_since: "2026-01-01"', t, count=1, flags=re.M))
|
|
commit(clone, "fixture: kimai catalog_since in the past")
|
|
case("FACT: kimai image moves, catalog_since untouched", clone,
|
|
[(KIMAI, swap_image("kimai", KAPP, KAPP_MINOR))],
|
|
expect_rc=1, must_contain=("CATALOG-SINCE GATE FAILED", "kimai", "catalog_since is still"), gate=CS)
|
|
case("FACT: kimai image moves, catalog_since set to a FUTURE year", clone,
|
|
[(KIMAI, swap_image("kimai", KAPP, KAPP_MINOR)),
|
|
(KIMAI_FY, set_since("2036-09-13"))],
|
|
expect_rc=1, must_contain=("in the future",), gate=CS)
|
|
case("GENUINE: kimai image moves AND catalog_since = today", clone,
|
|
[(KIMAI, swap_image("kimai", KAPP, KAPP_MINOR)),
|
|
(KIMAI_FY, set_since(today))],
|
|
expect_rc=0, must_contain=("catalog-since gate OK", "1 image move(s) dated"), gate=CS)
|
|
case("DECOY: image moves; today's date lands in a COMMENT and README, the field stays", clone,
|
|
[(KIMAI, lambda t: swap_image("kimai", KAPP, KAPP_MINOR)(t).replace("services:", "# catalog_since: %s\nservices:" % today, 1)),
|
|
("README.md", lambda t: t + "\ncatalog_since: %s (kimai)\n" % today)],
|
|
expect_rc=1, must_contain=("CATALOG-SINCE GATE FAILED",), gate=CS)
|
|
case("DECOY: only a comment + env line change, images untouched, date untouched", clone,
|
|
[(KIMAI, comment_and_env)], expect_rc=0, must_contain=("0 image move(s) dated", "catalog-since gate OK"), gate=CS)
|
|
|
|
# ── copy-i18n (R-560): Hungarian frozen, English sound ───────────────────────────────
|
|
PB = "templates/privatebin/.felhom.yml"
|
|
TOTAL = 1032 # every copy string in the catalog, measured on 94bc5febaca2
|
|
PB_EN = 14 # what the genuine block below translates
|
|
|
|
# A CORRECT English block for privatebin — the genuine article every decoy is a twist on.
|
|
GENUINE_EN = """
|
|
i18n:
|
|
en:
|
|
description: "Encrypted note and text sharing"
|
|
app_info:
|
|
tagline: "Encrypted text sharing - the server never sees the content"
|
|
use_cases:
|
|
- 'Share sensitive text safely'
|
|
- 'End-to-end encryption - the server cannot read the content'
|
|
- 'Choose how long it lasts (5 minutes to a year, or never)'
|
|
- 'Delete after reading, automatically'
|
|
- 'Password protection for extra safety'
|
|
first_steps:
|
|
- 'Open paste.DOMAIN in your browser'
|
|
- 'Type your text and select Send'
|
|
- 'Share the link you get - the encryption key is inside the URL'
|
|
deploy_fields:
|
|
- env_var: DOMAIN
|
|
label: "Domain"
|
|
description: "The server domain name"
|
|
- env_var: SUBDOMAIN
|
|
label: "Subdomain"
|
|
description: "The address this app answers on"
|
|
"""
|
|
|
|
def strip_en(t):
|
|
"""Remove an existing English block (and its comment header) — a case must behave the
|
|
same before and after that app's batch lands, or the suite rots on a future push."""
|
|
t = re.sub(r"\n# --- English copy.*\Z", "\n", t, flags=re.S)
|
|
return re.sub(r"\n^i18n:\n.*\Z", "\n", t, flags=re.S | re.M)
|
|
|
|
def add_en(block=GENUINE_EN):
|
|
return lambda t: strip_en(t).rstrip("\n") + "\n" + block
|
|
|
|
def en_with(old_, new_):
|
|
return add_en(GENUINE_EN.replace(old_, new_))
|
|
|
|
# THE FACTS — each must be refused.
|
|
case_copy("FACT: a Hungarian byte changed in a frozen string", clone,
|
|
[(PB, lambda t: t.replace("Titkosított jegyzet és szöveg megosztás",
|
|
"Titkosított jegyzet- és szövegmegosztás"))],
|
|
expect_rc=1, must_contain=("Hungarian CHANGED", "privatebin", "description"))
|
|
case_copy("FACT: a Hungarian first_step removed", clone,
|
|
[(PB, lambda t: t.replace(" - 'Oszd meg a generált linket - a titkosítási kulcs az URL-ben van'\n", ""))],
|
|
expect_rc=1, must_contain=("REMOVED", "first_steps"))
|
|
case_copy("FACT: a NEW app is not in the freeze", clone,
|
|
[("templates/decoyapp/.felhom.yml",
|
|
lambda t: 'display_name: "Decoy"\ndescription: "Uj alkalmazas"\nslug: decoyapp\n')],
|
|
expect_rc=1, must_contain=("not in the freeze", "--add-app"))
|
|
case_copy("FACT: an unknown key inside the English block", clone,
|
|
[(PB, en_with(' description: "Encrypted note and text sharing"',
|
|
' description: "Encrypted note and text sharing"\n docs_url: "https://example.invalid"'))],
|
|
expect_rc=1, must_contain=("unknown key",), extra_args=("--expect-missing", str(TOTAL - PB_EN)))
|
|
case_copy("FACT: an English deploy field with no Hungarian twin", clone,
|
|
[(PB, en_with(" - env_var: DOMAIN", " - env_var: NOSUCHFIELD"))],
|
|
expect_rc=1, must_contain=("no Hungarian twin", "NOSUCHFIELD"))
|
|
case_copy("FACT: an accented Hungarian letter left in the English", clone,
|
|
[(PB, en_with("Share sensitive text safely", "Érzékeny text sharing"))],
|
|
expect_rc=1, must_contain=("accented Hungarian letter",))
|
|
case_copy("FACT: ASCII-only Hungarian left in the English (no accent to find)", clone,
|
|
[(PB, en_with(' description: "The address this app answers on"',
|
|
' description: "Aldomain for the app"'))],
|
|
expect_rc=1, must_contain=("ASCII-only Hungarian", "aldomain"))
|
|
case_copy("FACT: the product begs (\"please\")", clone,
|
|
[(PB, en_with("Type your text and select Send", "Please type your text and select Send"))],
|
|
expect_rc=1, must_contain=("does not beg",))
|
|
case_copy("FACT: an English retrieval promise the Hungarian never made", clone,
|
|
[(PB, en_with("Password protection for extra safety",
|
|
"Deleted notes can still be restored later"))],
|
|
expect_rc=1, must_contain=("retrieval promise",))
|
|
|
|
# ALLOWLIST_EN (R-594): a TRUE retrieval promise can be REGISTERED with a reason; the label without the fact
|
|
# must not pass. The register lives beside the freeze so the clone's own copy is the one judged.
|
|
AL = "scripts/copy_freeze/allowlist_en.json"
|
|
PROMISE = ("Password protection for extra safety", "Deleted notes can still be restored later")
|
|
WHY = "privatebin keeps a burned paste for its expiry window, so it is true here (decoy)"
|
|
|
|
def allow(*entries):
|
|
return lambda t: json.dumps({"_what": "decoy", "entries": [
|
|
dict(zip(("app", "path", "match", "reason"), e)) for e in entries]}, indent=1) + "\n"
|
|
|
|
case_copy("GENUINE: a REGISTERED true retrieval promise passes", clone,
|
|
[(PB, en_with(*PROMISE)),
|
|
(AL, allow(("privatebin", "app_info.use_cases[4]", "can still be restored", WHY)))],
|
|
expect_rc=0, must_contain=("copy-i18n: OK", "1 registered retrieval promise(s) in ALLOWLIST_EN, 1 used"))
|
|
case_copy("FACT: registered for ANOTHER app - the promise still convicts, the entry is stale", clone,
|
|
[(PB, en_with(*PROMISE)),
|
|
(AL, allow(("vaultwarden", "app_info.use_cases[4]", "can still be restored", WHY)))],
|
|
expect_rc=1, must_contain=("retrieval promise the Hungarian does not make", "STALE entry vaultwarden"))
|
|
case_copy("FACT: registered on the path, but the sentence was rewritten (match gone)", clone,
|
|
[(PB, en_with(PROMISE[0], "Deleted notes can be recovered at any time")),
|
|
(AL, allow(("privatebin", "app_info.use_cases[4]", "can still be restored", WHY)))],
|
|
expect_rc=1, must_contain=("retrieval promise the Hungarian does not make", "STALE entry privatebin"))
|
|
case_copy("FACT: a STALE entry - nothing in the English promises it any more", clone,
|
|
[(AL, allow(("privatebin", "app_info.use_cases[4]", "can still be restored", WHY)))],
|
|
expect_rc=1, must_contain=("STALE entry privatebin",))
|
|
case_copy("FACT: a registered promise with a two-word reason", clone,
|
|
[(PB, en_with(*PROMISE)),
|
|
(AL, allow(("privatebin", "app_info.use_cases[4]", "can still be restored", "it's fine")))],
|
|
expect_rc=1, must_contain=("no reason worth the name", "retrieval promise the Hungarian does not make"))
|
|
# A credential is a LOGIN, not prose: gokapi's default_creds carries admin / adminadmin.
|
|
GK = "templates/gokapi/.felhom.yml"
|
|
case_copy("FACT: a credential token rewritten in translation", clone,
|
|
[(GK, lambda t: t.rstrip("\n") + """
|
|
i18n:
|
|
en:
|
|
app_info:
|
|
default_creds: "Sign in: administrator / hunter2"
|
|
""")],
|
|
expect_rc=1, must_contain=("credential token",))
|
|
case_copy("FACT: an i18n block for a language the controller does not render", clone,
|
|
[(PB, lambda t: t.rstrip("\n") + "\ni18n:\n de:\n description: \"Verschluesselte Notizen\"\n")],
|
|
expect_rc=1, must_contain=("renders en only",))
|
|
case_copy("FACT: an English list with a different number of steps", clone,
|
|
[(PB, en_with(" - 'Share the link you get - the encryption key is inside the URL'\n", ""))],
|
|
expect_rc=1, must_contain=("a list is replaced",))
|
|
|
|
# THE RATCHET — the one thing --expect-missing does not test for the cases above, so it is
|
|
# tested here explicitly, in BOTH directions. A ceiling that only convicts upwards can be
|
|
# left behind by a push that translated more than it recorded.
|
|
# The ABOVE case REMOVES privatebin's English (2026-10-05): it used to ADD it and rely on the rest
|
|
# of the catalog being untranslated, so the day the catalog reached full coverage (ceiling 0) the
|
|
# tree had nothing missing, the gate rightly said OK, and the case failed on its own premise.
|
|
case_copy("FACT: ratchet — fewer strings translated than the ceiling records", clone,
|
|
[(PB, strip_en)], expect_rc=1,
|
|
must_contain=("English coverage", "ABOVE"),
|
|
extra_args=("--expect-missing", "0"))
|
|
case_copy("FACT: ratchet — more translated than the ceiling records", clone,
|
|
[(PB, add_en())], expect_rc=1,
|
|
must_contain=("English coverage", "BELOW"),
|
|
extra_args=("--expect-missing", "999999"))
|
|
|
|
# THE GENUINE ARTICLE — must pass.
|
|
case_copy("GENUINE: a correct English block on privatebin", clone,
|
|
[(PB, add_en())], expect_rc=0,
|
|
must_contain=("copy-i18n: OK", "privatebin 14/14"))
|
|
|
|
# The ratchet is a fact about the CATALOG, not about how the gate was invoked. Naming one
|
|
# app must not change the count — the first version of this gate counted coverage only for
|
|
# the apps in scope, so `check-copy-i18n.py privatebin` reported 47 more missing strings
|
|
# than the same tree unscoped, and either number could have been made to "pass".
|
|
case_copy("GENUINE: naming an app does not change the coverage count", clone,
|
|
[(PB, add_en())], expect_rc=0,
|
|
must_contain=("copy-i18n: OK",),
|
|
extra_args=("privatebin",))
|
|
|
|
# DEGRADED MODE — what CI actually runs, because its runner has no PyYAML.
|
|
case_copy_noyaml("FACT(no-yaml): a Hungarian byte changed in a frozen string", clone,
|
|
[(PB, lambda t: t.replace("Titkosított jegyzet és szöveg megosztás",
|
|
"Titkosított jegyzet- és szövegmegosztás"))],
|
|
expect_rc=1, must_contain=("DEGRADED", "no longer in the file", "privatebin"))
|
|
case_copy_noyaml("FACT(no-yaml): a frozen Hungarian line deleted", clone,
|
|
[(PB, lambda t: t.replace(" - 'Oszd meg a generált linket - a titkosítási kulcs az URL-ben van'\n", ""))],
|
|
expect_rc=1, must_contain=("no longer in the file",))
|
|
case_copy_noyaml("FACT(no-yaml): a NEW app is not in the freeze", clone,
|
|
[("templates/decoyapp2/.felhom.yml",
|
|
lambda t: 'display_name: "Decoy"\ndescription: "Uj"\nslug: decoyapp2\n')],
|
|
expect_rc=1, must_contain=("not in the freeze",))
|
|
case_copy_noyaml("GENUINE(no-yaml): the untouched tree passes, and SAYS what it did not check",
|
|
clone, [("README.md", lambda t: t + "\n<!-- decoy: a harmless line -->\n")],
|
|
expect_rc=0,
|
|
must_contain=("DEGRADED", "OK (degraded", "NOT checked here"))
|
|
# The escaped-quote pair: romm's two help_texts whose YAML escapes an inner double quote.
|
|
# The degraded check must find them anyway — if it cannot, it convicts an honest tree.
|
|
case_copy_noyaml("GENUINE(no-yaml): romm's escaped-quote help_texts are still found", clone,
|
|
[("templates/romm/docker-compose.yml", lambda t: t + "\n# decoy comment\n")],
|
|
expect_rc=0, must_contain=("OK (degraded",))
|
|
|
|
# THE DECOYS — the LABEL moves, the FACT does not. Each must pass.
|
|
case_copy("DECOY: Hungarian rewritten inside a YAML COMMENT", clone,
|
|
[(PB, lambda t: t.replace("# --- App info (info page content) ---",
|
|
"# --- Alkalmazas informacio: Titkosított jegyzet MEGVALTOZOTT ---"))],
|
|
expect_rc=0, must_contain=("copy-i18n: OK",))
|
|
case_copy("DECOY: a frozen Hungarian sentence pasted into README.md", clone,
|
|
[("README.md", lambda t: t + "\nTitkositott jegyzet es szoveg megosztas (decoy)\n")],
|
|
expect_rc=0, must_contain=("copy-i18n: OK",))
|
|
case_copy("DECOY: display_name changed - a NAME, never copy", clone,
|
|
[(PB, lambda t: t.replace('display_name: "PrivateBin"', 'display_name: "PrivateBin 2"'))],
|
|
expect_rc=0, must_contain=("copy-i18n: OK",))
|
|
case_copy("DECOY: docs_url changed - configuration, never copy", clone,
|
|
[(PB, lambda t: t.replace("https://github.com/PrivateBin/PrivateBin/wiki",
|
|
"https://example.invalid/wiki"))],
|
|
expect_rc=0, must_contain=("copy-i18n: OK",))
|
|
case_copy("DECOY: Hungarian text added to a docker-compose.yml", clone,
|
|
[("templates/privatebin/docker-compose.yml",
|
|
lambda t: t.replace("services:", "# Titkosított jegyzet és szöveg megosztás\nservices:", 1))],
|
|
expect_rc=0, must_contain=("copy-i18n: OK",))
|
|
|
|
# ── probe-matches-compose (R-618) ────────────────────────────────────────────────────────
|
|
# The FACT is where the app LISTENS inside its container. The LABEL is every other number
|
|
# in the file that looks like a port: the traefik label, `ports:`, `expose:`, a comment,
|
|
# and a sidecar's own healthcheck. A gate that reads any of those would have passed the
|
|
# three templates that stopped a working app on 2026-09-21.
|
|
BS = "templates/bookstack/.felhom.yml"
|
|
BSC = "templates/bookstack/docker-compose.yml"
|
|
|
|
print("\n-- probe-matches-compose: the facts (each MUST be refused)")
|
|
# The three real faults, RE-INTRODUCED rather than read off the tree. Reading them off the
|
|
# tree passed only while the tree was broken; the case would have gone green for the wrong
|
|
# reason the moment R-618 was fixed, which is the `constant-for-measurement` shape again.
|
|
case_probe("FACT: tandoor's real R-618 port fault, re-introduced", clone,
|
|
[("templates/tandoor/.felhom.yml", lambda t: t.replace(" port: 80\n",
|
|
" port: 8080\n"))],
|
|
expect_rc=1, must_contain=("FAIL tandoor", "Nothing listens on 8080"),
|
|
apps=("tandoor",))
|
|
case_probe("FACT: wger's real R-618 port fault, re-introduced", clone,
|
|
[("templates/wger/.felhom.yml", lambda t: t.replace(" port: 8000\n",
|
|
" port: 80\n"))],
|
|
expect_rc=1, must_contain=("FAIL wger", "dials 8000 on loopback"),
|
|
apps=("wger",))
|
|
case_probe("FACT: zipline's real R-618 path fault, re-introduced", clone,
|
|
[("templates/zipline/.felhom.yml",
|
|
lambda t: t.replace('path: "/api/healthcheck"', 'path: "/api/health"'))],
|
|
expect_rc=1, must_contain=("FAIL zipline", "This probe CAN fail on it"),
|
|
apps=("zipline",))
|
|
case_probe("GENUINE: the fixed tree passes", clone, [], expect_rc=0,
|
|
must_contain=("probe-matches-compose: OK",),
|
|
apps=("tandoor", "zipline", "wger"))
|
|
case_probe("FACT: a clean app given a wrong probe port", clone,
|
|
[(BS, lambda t: t.replace("port: 80", "port: 8080"))],
|
|
expect_rc=1, must_contain=("FAIL bookstack", "dials 80 on loopback"),
|
|
apps=("bookstack",))
|
|
case_probe("FACT: api+expect probe given a path the app does not answer", clone,
|
|
[(BS, lambda t: t.replace(" - type: http\n port: 80",
|
|
" - type: api\n port: 80\n"
|
|
" path: /status\n expect:\n"
|
|
" status: 200"))],
|
|
expect_rc=1, must_contain=("FAIL bookstack", "This probe CAN fail on it"),
|
|
apps=("bookstack",))
|
|
|
|
print("-- probe-matches-compose: the decoys (the label moves, the fact does not)")
|
|
case_probe("DECOY: the port moves in a COMMENT in .felhom.yml", clone,
|
|
[(BS, lambda t: t.replace("healthcheck:",
|
|
"# the app listens on 8080 (decoy comment)\nhealthcheck:"))],
|
|
expect_rc=0, must_contain=("probe-matches-compose: OK",), apps=("bookstack",))
|
|
case_probe("DECOY: traefik loadbalancer.server.port changed", clone,
|
|
[(BSC, lambda t: t.replace("loadbalancer.server.port=80",
|
|
"loadbalancer.server.port=9999"))],
|
|
expect_rc=0, must_contain=("probe-matches-compose: OK",), apps=("bookstack",))
|
|
case_probe("DECOY: a published `ports:` mapping changed", clone,
|
|
[(BSC, lambda t: t.replace(" container_name: bookstack\n",
|
|
" container_name: bookstack\n"
|
|
" ports:\n - \"9999:80\"\n", 1))],
|
|
expect_rc=0, must_contain=("probe-matches-compose: OK",), apps=("bookstack",))
|
|
case_probe("DECOY: a NON-probed sidecar's healthcheck port changed", clone,
|
|
[(BSC, lambda t: t.replace(
|
|
'["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]',
|
|
'["CMD", "curl", "-f", "http://127.0.0.1:7777/"]'))],
|
|
expect_rc=0, must_contain=("probe-matches-compose: OK",), apps=("bookstack",))
|
|
case_probe("DECOY: path mismatch on a probe that CANNOT fail on it -> WARN, not FAIL", clone,
|
|
[], expect_rc=0,
|
|
must_contain=("WARN home-assistant", "Harmless TODAY"),
|
|
apps=("home-assistant",))
|
|
|
|
|
|
print("-- probe-matches-compose: the DEGRADED mode CI actually runs (no PyYAML)")
|
|
case_probe_noyaml("DEGRADED: the fixed tree passes, and SAYS it is degraded", clone, [],
|
|
expect_rc=0, must_contain=("mode: DEGRADED", "OK (degraded)"),
|
|
apps=("tandoor", "zipline", "wger"))
|
|
case_probe_noyaml("DEGRADED FACT: tandoor's port fault still refused", clone,
|
|
[("templates/tandoor/.felhom.yml",
|
|
lambda t: t.replace(" port: 80\n", " port: 8080\n"))],
|
|
expect_rc=1, must_contain=("FAIL tandoor", "Nothing listens on 8080"),
|
|
apps=("tandoor",))
|
|
case_probe_noyaml("DEGRADED FACT: wger's port fault still refused", clone,
|
|
[("templates/wger/.felhom.yml",
|
|
lambda t: t.replace(" port: 8000\n", " port: 80\n"))],
|
|
expect_rc=1, must_contain=("FAIL wger",), apps=("wger",))
|
|
case_probe_noyaml("DEGRADED FACT: zipline's path fault still refused", clone,
|
|
[("templates/zipline/.felhom.yml",
|
|
lambda t: t.replace('path: "/api/healthcheck"', 'path: "/api/health"'))],
|
|
expect_rc=1, must_contain=("FAIL zipline", "This probe CAN fail on it"),
|
|
apps=("zipline",))
|
|
case_probe_noyaml("DEGRADED DECOY: traefik label moves, the fact does not", clone,
|
|
[("templates/bookstack/docker-compose.yml",
|
|
lambda t: t.replace("loadbalancer.server.port=80",
|
|
"loadbalancer.server.port=9999"))],
|
|
expect_rc=0, must_contain=("OK (degraded)",), apps=("bookstack",))
|
|
|
|
|
|
print("-- probe-matches-compose: the probe TARGET (R-630)")
|
|
case_probe("FACT: paperless-ngx without its explicit container is refused", clone,
|
|
[("templates/paperless-ngx/.felhom.yml",
|
|
lambda t: t.replace(" container: paperless-webserver\n", ""))],
|
|
expect_rc=1, must_contain=("FAIL paperless-ngx", "resolves to no container"),
|
|
apps=("paperless-ngx",))
|
|
case_probe("FACT: immich's FOUR prefix candidates are an ambiguity, not a pick", clone,
|
|
[("templates/immich/.felhom.yml",
|
|
lambda t: t.replace(" container: immich-server\n", ""))],
|
|
expect_rc=1, must_contain=("FAIL immich", "ambiguous"),
|
|
apps=("immich",))
|
|
case_probe("FACT: an explicit container no service declares is refused", clone,
|
|
[("templates/paperless-ngx/.felhom.yml",
|
|
lambda t: t.replace("container: paperless-webserver",
|
|
"container: paperless-nope"))],
|
|
expect_rc=1, must_contain=("FAIL paperless-ngx", "no service declares"),
|
|
apps=("paperless-ngx",))
|
|
case_probe("GENUINE: both explicit containers resolve", clone, [], expect_rc=0,
|
|
must_contain=("probe-matches-compose: OK",), apps=("paperless-ngx", "immich"))
|
|
case_probe("DECOY: the container name moves in a COMMENT, not the field", clone,
|
|
[("templates/paperless-ngx/.felhom.yml",
|
|
lambda t: t.replace("healthcheck:", "# container: paperless-nope\nhealthcheck:"))],
|
|
expect_rc=0, must_contain=("probe-matches-compose: OK",), apps=("paperless-ngx",))
|
|
# A unique prefix must still resolve WITHOUT the explicit field — the third rule is a
|
|
# narrowing, not a removal. immich with three of its four sidecars renamed leaves exactly
|
|
# one `immich-*` container, which is no longer an ambiguity.
|
|
case_probe("DECOY: a UNIQUE prefix still resolves without the field", clone,
|
|
[("templates/immich/docker-compose.yml",
|
|
lambda t: t.replace("container_name: immich-postgres", "container_name: db-immich")
|
|
.replace("container_name: immich-redis", "container_name: cache-immich")
|
|
.replace("container_name: immich-machine-learning", "container_name: ml-immich")),
|
|
("templates/immich/.felhom.yml",
|
|
lambda t: t.replace(" container: immich-server\n", ""))],
|
|
expect_rc=0, must_contain=("probe-matches-compose: OK",), apps=("immich",))
|
|
case_probe_noyaml("DEGRADED: the explicit container is read without PyYAML too", clone, [],
|
|
expect_rc=0, must_contain=("mode: DEGRADED", "OK (degraded)"),
|
|
apps=("paperless-ngx", "immich"))
|
|
case_probe_noyaml("DEGRADED FACT: removing it still refuses", clone,
|
|
[("templates/paperless-ngx/.felhom.yml",
|
|
lambda t: t.replace(" container: paperless-webserver\n", ""))],
|
|
expect_rc=1, must_contain=("FAIL paperless-ngx",), apps=("paperless-ngx",))
|
|
|
|
test_record_cases(clone)
|
|
|
|
# probe-measured (R-715): the measurement must sit DIRECTLY above setup_done_probe:, with a date and both answers.
|
|
KOMGA = "templates/komga/.felhom.yml"
|
|
def drop_note(t):
|
|
return "\n".join(l for l in t.split("\n") if not (l.startswith("# measured on 9202") and "isClaimed" in l))
|
|
def case_pm(name, edits, expect_rc, must=()):
|
|
global ran
|
|
ran += 1
|
|
try:
|
|
for relpath, fn in edits:
|
|
edit(clone, relpath, fn)
|
|
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-probe-measured.py"), "--root=" + clone, "komga"], cwd=clone)
|
|
out = r.stdout + r.stderr
|
|
if r.returncode == expect_rc and all(m in out for m in must):
|
|
print(" ok %-52s rc=%d (expected %d)" % (name, r.returncode, expect_rc))
|
|
else:
|
|
fails.append("%s: rc=%d expected %d\n%s" % (name, r.returncode, expect_rc, out[-600:]))
|
|
finally:
|
|
sh(["git", "checkout", "-q", "--", "."], cwd=clone)
|
|
case_pm("probe-measured: genuine komga note passes", [], 0, ("probe-measured: OK",))
|
|
case_pm("probe-measured: note moved into the TAGLINE", [(KOMGA, lambda t: drop_note(t).replace(
|
|
"\ntagline:", "\ntagline:", 1).replace("app_info:\n", "app_info:\n # measured on 9202 2026-09-29: isClaimed false -> true\n", 1))], 1, ("FAIL",))
|
|
case_pm("probe-measured: date but no before/after", [(KOMGA, lambda t: drop_note(t).replace(
|
|
"\nsetup_done_probe:", "\n# measured on 9202 2026-09-29\nsetup_done_probe:"))], 1, ("before/after",))
|
|
case_pm("probe-measured: before/after but no date", [(KOMGA, lambda t: drop_note(t).replace(
|
|
"\nsetup_done_probe:", "\n# measured: isClaimed false -> true\nsetup_done_probe:"))], 1, ("a date",))
|
|
case_pm("probe-measured: 'read upstream' is not a measurement", [(KOMGA, lambda t: drop_note(t).replace(
|
|
"\nsetup_done_probe:", "\n# read upstream 2026-09-29: isClaimed false -> true\nsetup_done_probe:"))], 1, ("measured",))
|
|
|
|
finally:
|
|
shutil.rmtree(clone, ignore_errors=True)
|
|
|
|
image_pins_cases()
|
|
image_resolvable_cases()
|
|
volume_persistence_cases()
|
|
onboarding_cases()
|
|
family_gate_cases()
|
|
mem_sum_cases()
|
|
data_key_cases()
|
|
|
|
if fails:
|
|
print()
|
|
for f in fails:
|
|
print("FAIL: %s" % f)
|
|
return 1
|
|
print("\ncatalog gate decoys OK — %d case(s), every label judged on its fact (R-421)" % ran)
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|