R-127 (a): four data-encrypting keys flagged data_key (n8n, calcom, wanderer, bookstack APP_KEY) + gate data-key with decoys
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -27,6 +27,8 @@ Gates, in order (all must pass; **non-zero exit on any failure**):
|
||||
record (NEW-APP-CHECKLIST.md; the 53 apps published before 2026-10-01 are exempt by name)
|
||||
11. mem-limit-sum static, instant, whole repo — `.felhom.yml` resources.mem_limit equals the sum of every
|
||||
service's deploy.resources.limits.memory, and every service has one (R-758)
|
||||
12. data-key static, instant, whole repo — every data-encrypting key carries data_key: true (name rule
|
||||
or REGISTRY), and no flag is unexplained (R-127)
|
||||
4. engine-major static, needs GIT HISTORY — no database engine pin crosses a MAJOR version
|
||||
(operator ruling 2026-09-13; expires when Slice 4 / R-448 ships). Runs in the
|
||||
pre-push hook, which has the full clone; on a SHALLOW clone (CI fetches at
|
||||
@@ -123,6 +125,9 @@ GATES = [
|
||||
# templates were under it — the deploy screen and the box's overcommit warning read less than the app may take.
|
||||
# Static, stdlib only (no PyYAML on CI), so --fast: the hook and CI.
|
||||
("mem-limit-sum", "check-mem-limit-sum.py", True, True, False),
|
||||
# R-127 leg (a) (2026-10-05): every data-encrypting key carries `data_key: true` (by its NAME, or by a registry
|
||||
# entry with its reason) and every flag is accounted for — the restore's fail-closed gate reads only that flag.
|
||||
("data-key", "check-data-key.py", True, True, False),
|
||||
]
|
||||
|
||||
# 3 (R-605): the gate's HARNESS refused to run — its canary failed or it had nothing to judge — so NO app was
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""check-data-key.py — every data-encrypting key carries `data_key: true`, and every flag is accounted for (R-127 a).
|
||||
|
||||
WHAT WENT WRONG. `data_key: true` on a deploy field tells the controller the app ENCRYPTS STORED DATA with it: the
|
||||
restore RECOVERS the value and refuses (fail-closed) when it cannot, and the box never generates a new one
|
||||
(felhom-controller internal/backup/restore_unit.go missingDataKeys; internal/stacks/deploy.go GenerateSecretForField).
|
||||
In 2026-08 only five fields carried it, while n8n's N8N_ENCRYPTION_KEY, calcom's CALENDSO_ENCRYPTION_KEY, wanderer's
|
||||
POCKETBASE_ENCRYPTION_KEY and bookstack's APP_KEY (two-factor secrets) did not — so a restore missing one of them
|
||||
would have proceeded onto data it cannot decrypt instead of refusing.
|
||||
|
||||
WHY NOT THE LABEL. The Hungarian label „Titkosítási kulcs" ("Encryption key") sits on 24 secrets, most of which only
|
||||
SIGN sessions (Django SECRET_KEY, Phoenix SECRET_KEY_BASE, JWT secrets): regenerating those signs everyone out, it
|
||||
loses no data. A label is copy, frozen byte for byte, and it is not the fact. The facts this gate reads:
|
||||
|
||||
1. NAME RULE — a field whose env var names an encryption key or a pepper (`ENCRYPTION_KEY`, `PEPPER`) is a data key
|
||||
by what the app calls it. It must carry `data_key: true`.
|
||||
2. REGISTRY — data keys whose name does not say so (bookstack APP_KEY encrypts two-factor secrets; …) are listed
|
||||
below BY APP AND FIELD with the reason. Each must carry the flag (unflagging one is a regression), and an entry
|
||||
whose field no longer exists is STALE (refused — a registry nobody prunes stops meaning anything).
|
||||
3. AGREEMENT — a `data_key: true` that neither rule names is refused: add it to REGISTRY with its reason. Over-
|
||||
flagging is not harmless either: the restore then REFUSES for a key that could have been regenerated.
|
||||
|
||||
stdlib only (the CI runner has no PyYAML). The flag is read only from a field's own block inside the top-level
|
||||
`deploy_fields:` — never from a comment, never from the `i18n:` block, never from `steps/` files.
|
||||
|
||||
USAGE
|
||||
python3 scripts/check-data-key.py [app …] [--root=<dir>] (`--all` accepted, a no-op: every directory is judged)
|
||||
Exit: 0 agree · 1 convicted · 2 inconclusive.
|
||||
Decoys: scripts/test_gate_decoys.py `data_key_cases` (COVERS "data-key").
|
||||
"""
|
||||
import io
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
NAME_RULE = re.compile(r"ENCRYPTION_KEY|PEPPER")
|
||||
|
||||
# (app, env_var) -> why it is a data key although its name does not say so.
|
||||
REGISTRY = {
|
||||
("adventurelog", "SECRET_KEY"): "the template's own comment: encrypts stored data; restore must recover it",
|
||||
("bookstack", "APP_KEY"): "Laravel encrypt() on every member's two-factor secret (app/Access/Mfa/MfaValue.php)",
|
||||
("dawarich", "SECRET_KEY_BASE"): "the template's own comment: stored data unreadable if it changes",
|
||||
("papra", "AUTH_SECRET"): "the template's own comment: stored tokens invalid if it changes; keep the old key",
|
||||
("sparkyfitness", "BETTER_AUTH_SECRET"): "signs sessions AND encrypts 2FA/TOTP secrets (template comment)",
|
||||
}
|
||||
|
||||
FIELD_RE = re.compile(r"^ - env_var:\s*['\"]?([A-Za-z0-9_]+)['\"]?\s*(?:#.*)?$")
|
||||
DATA_KEY_RE = re.compile(r"^ data_key:\s*(\S+?)\s*(?:#.*)?$")
|
||||
TOP_KEY_RE = re.compile(r"^[A-Za-z0-9_]+:")
|
||||
|
||||
|
||||
def fields(meta_text):
|
||||
"""[(env_var, data_key_raw_or_None)] from the top-level deploy_fields: block only."""
|
||||
out, cur, inside = [], None, False
|
||||
for line in meta_text.split("\n"):
|
||||
if TOP_KEY_RE.match(line):
|
||||
inside = line.startswith("deploy_fields:")
|
||||
cur = None
|
||||
continue
|
||||
if not inside:
|
||||
continue
|
||||
m = FIELD_RE.match(line)
|
||||
if m:
|
||||
cur = [m.group(1), None]
|
||||
out.append(cur)
|
||||
continue
|
||||
m = DATA_KEY_RE.match(line)
|
||||
if m and cur is not None:
|
||||
cur[1] = m.group(1).strip("'\"")
|
||||
return [(a, b) for a, b in out]
|
||||
|
||||
|
||||
def main(argv):
|
||||
root, apps = ROOT, []
|
||||
for a in argv:
|
||||
if a.startswith("--root="):
|
||||
root = a.split("=", 1)[1]
|
||||
elif a == "--all":
|
||||
continue
|
||||
elif a.startswith("-"):
|
||||
print("unknown option: %s" % a)
|
||||
return 2
|
||||
else:
|
||||
apps.append(a)
|
||||
tdir = os.path.join(root, "templates")
|
||||
if not os.path.isdir(tdir):
|
||||
print("data-key: no templates/ under %s" % root)
|
||||
return 2
|
||||
every = sorted(n for n in os.listdir(tdir) if os.path.isfile(os.path.join(tdir, n, ".felhom.yml")))
|
||||
judged = apps or every
|
||||
unknown = [a for a in judged if a not in every]
|
||||
if unknown:
|
||||
print("data-key: no such template: %s" % ", ".join(unknown))
|
||||
return 2
|
||||
bad, undecided, flagged = [], [], 0
|
||||
seen = set()
|
||||
for app in judged:
|
||||
text = io.open(os.path.join(tdir, app, ".felhom.yml"), encoding="utf-8").read()
|
||||
for env, raw in fields(text):
|
||||
seen.add((app, env))
|
||||
if raw is not None and raw.lower() not in ("true", "false"):
|
||||
undecided.append("%s/%s: data_key %r is not true/false" % (app, env, raw))
|
||||
continue
|
||||
on = raw is not None and raw.lower() == "true"
|
||||
flagged += on
|
||||
named = bool(NAME_RULE.search(env))
|
||||
reg = (app, env) in REGISTRY
|
||||
if (named or reg) and not on:
|
||||
why = "its name says it encrypts" if named else "registered: " + REGISTRY[(app, env)]
|
||||
bad.append("%s/%s is a data key (%s) but carries no `data_key: true` — a restore missing it would "
|
||||
"proceed onto data it cannot decrypt" % (app, env, why))
|
||||
elif on and not (named or reg):
|
||||
bad.append("%s/%s carries `data_key: true` but neither its name nor REGISTRY says why — add it to "
|
||||
"REGISTRY in scripts/check-data-key.py with the reason (or drop the flag)" % (app, env))
|
||||
for (app, env), why in sorted(REGISTRY.items()):
|
||||
if app in judged and (app, env) not in seen:
|
||||
bad.append("REGISTRY entry %s/%s is STALE — no such deploy field (%s)" % (app, env, why))
|
||||
for b in bad:
|
||||
print("REFUSED: " + b)
|
||||
for u in undecided:
|
||||
print("INCONCLUSIVE: " + u)
|
||||
if bad:
|
||||
print("data-key: %d problem(s) in %d template(s)" % (len(bad), len(judged)))
|
||||
return 1
|
||||
if undecided:
|
||||
return 2
|
||||
print("data-key gate OK: %d template(s), %d data key(s), every flag agrees with the name rule and REGISTRY"
|
||||
% (len(judged), flagged))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -65,7 +65,9 @@ class CatalogGatesFastTest(unittest.TestCase):
|
||||
# 11 since 2026-10-01: onboarding (NEW-APP-CHECKLIST.md), fast and history-free, so it runs in CI too.
|
||||
# 12 since family-gate joined — STALE AGAIN until 2026-10-05 (found by the burn-down, round 2).
|
||||
# 13 since 2026-10-05: mem-limit-sum (R-758), fast and history-free, so it runs in CI too.
|
||||
self.assertEqual(len(mod.GATES), 13)
|
||||
# 14 since the same night: data-key (R-127 leg a).
|
||||
self.assertEqual(len(mod.GATES), 14)
|
||||
self.assertIn("data-key", [g[0] for g in mod.GATES if g[3] and not g[4]])
|
||||
self.assertIn("mem-limit-sum", [g[0] for g in mod.GATES if g[3] and not g[4]])
|
||||
self.assertIn("onboarding", [g[0] for g in mod.GATES if g[3] and not g[4]])
|
||||
self.assertEqual([g[0] for g in mod.GATES if not g[3]], ["image-resolvable", "volume-persistence"])
|
||||
|
||||
@@ -58,6 +58,7 @@ COVERS = {
|
||||
"family-gate": "family_gate written only in a COMMENT (not gated, no min_controller owed); min_controller only in a comment; a golden DIRECTORY named 0.287.0 with no bake log (the mkdir shape, R-410); a sibling with no golden (stated NOT CHECKED, never a pass of rule 3) - vs the facts: an unanchorable exception (regex, '/', '..'), an exception list with no gate, min_controller below 0.287.0, the newest baked golden below 0.287.0; and a genuine family app passes (decisions 63/64, finding F1)",
|
||||
"onboarding": "a NEW template with no record; a record missing an id, or carrying it only inside an HTML comment; a `done` whose path does not exist, is an EMPTY directory (the mkdir shape, R-410) or names an absent sibling-repo file; an `n/a` with an empty or two-word reason; an `open` row; `opened:` backdated before the checklist; the template a new app copies lacking a new id - vs a complete record, an id added after `opened:`, and an exempt app's record with open rows (NEW-APP-CHECKLIST.md)",
|
||||
"mem-limit-sum": "the right figure only in a COMMENT (the compose header's 'mem_limit: 640M', the .felhom.yml arithmetic) while the field is wrong; a `memory:` under reservations: (not a limit) making the sum come out right; a `mem_limit:` outside resources:; a steps/ file with the old figure (not judged) - vs the facts: a field under the sum, a service with no limit, an unreadable size (R-758)",
|
||||
"data-key": "a data_key: true only in a COMMENT or inside the i18n: block (not the field); an 'Encryption key' LABEL on a signing secret (copy, not the fact - must pass unflagged) - vs the facts: an *_ENCRYPTION_KEY field unflagged, a REGISTERED field unflagged, an unexplained flag, a stale registry entry, a non-boolean flag (R-127)",
|
||||
"copy-i18n": "Hungarian edited in a COMMENT/README/display_name (label, not copy) vs a real frozen string changed; an English block that is not English, is not matched to a Hungarian twin, or rewrites a credential (R-560). A retrieval promise REGISTERED in ALLOWLIST_EN passes only for its own app+path+sentence with a real reason; an entry for another app, a rewritten sentence, a stale entry or a two-word reason convicts (R-594). Also the DEGRADED mode CI actually runs — PyYAML shadowed out, freeze only (R-595)",
|
||||
}
|
||||
|
||||
@@ -747,6 +748,60 @@ def mem_sum_cases():
|
||||
shutil.rmtree(ws, ignore_errors=True)
|
||||
|
||||
|
||||
def data_key_cases():
|
||||
"""check-data-key.py reads templates/*/.felhom.yml via --root (R-127 leg a)."""
|
||||
global ran
|
||||
import tempfile
|
||||
ws = tempfile.mkdtemp(prefix="catalog-datakey-")
|
||||
try:
|
||||
GOOD = ('display_name: "X"\n'
|
||||
'deploy_fields:\n'
|
||||
' - env_var: DOMAIN\n type: domain\n'
|
||||
' - env_var: SECRET_KEY\n label: "Titkositasi kulcs"\n type: secret\n generate: "hex:32"\n'
|
||||
' - env_var: APP_ENCRYPTION_KEY\n type: secret\n generate: "hex:16"\n'
|
||||
' # why: encrypts stored data\n data_key: true\n'
|
||||
'\ni18n:\n en:\n deploy_fields:\n - env_var: SECRET_KEY\n label: "Encryption key"\n')
|
||||
BOOK = ('deploy_fields:\n - env_var: APP_KEY\n type: secret\n generate: "base64key:32"\n data_key: true\n')
|
||||
|
||||
def run(name, files, expect_rc, must=()):
|
||||
global ran
|
||||
cat = os.path.join(ws, "cat")
|
||||
shutil.rmtree(cat, ignore_errors=True)
|
||||
for app, text in files.items():
|
||||
d = os.path.join(cat, "templates", app)
|
||||
os.makedirs(d)
|
||||
io.open(os.path.join(d, ".felhom.yml"), "w", encoding="utf-8").write(text)
|
||||
r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-data-key.py"), "--root=" + cat], ROOT)
|
||||
out = r.stdout + r.stderr
|
||||
ran += 1
|
||||
ok = r.returncode == expect_rc and all(m in out for m in must)
|
||||
print(" %s %-70s rc=%d (expected %d)" % ("ok" if ok else "XX", name, r.returncode, expect_rc))
|
||||
if not ok:
|
||||
fails.append("%s: rc=%d expected %d; missing %s\n%s" % (
|
||||
name, r.returncode, expect_rc, [m for m in must if m not in out], out[-400:]))
|
||||
|
||||
print("\n-- data-key: genuine and decoys")
|
||||
run("GENUINE: a flagged *_ENCRYPTION_KEY; a signing key labelled 'Encryption key' unflagged; bookstack registered",
|
||||
{"demo": GOOD, "bookstack": BOOK}, 0, ("data-key gate OK", "2 data key(s)"))
|
||||
print("-- data-key: the facts (each MUST be refused)")
|
||||
run("FACT: the flag only in a COMMENT", {"demo": GOOD.replace(" data_key: true\n", " # data_key: true\n"),
|
||||
"bookstack": BOOK}, 1, ("demo/APP_ENCRYPTION_KEY is a data key",))
|
||||
run("FACT: the flag only inside the i18n: block", {"demo": GOOD.replace(" data_key: true\n", "") +
|
||||
" data_key: true\n", "bookstack": BOOK}, 1,
|
||||
("demo/APP_ENCRYPTION_KEY is a data key",))
|
||||
run("FACT: a REGISTERED data key unflagged (bookstack APP_KEY)",
|
||||
{"demo": GOOD, "bookstack": BOOK.replace(" data_key: true\n", "")}, 1, ("bookstack/APP_KEY is a data key", "registered"))
|
||||
run("FACT: an unexplained flag (a signing key flagged)", {"demo": GOOD.replace(
|
||||
' generate: "hex:32"\n', ' generate: "hex:32"\n data_key: true\n', 1), "bookstack": BOOK}, 1,
|
||||
("demo/SECRET_KEY carries `data_key: true` but neither",))
|
||||
run("FACT: a stale REGISTRY entry (bookstack has no APP_KEY any more)",
|
||||
{"demo": GOOD, "bookstack": BOOK.replace("APP_KEY", "OTHER_KEY").replace(" data_key: true\n", "")}, 1, ("STALE",))
|
||||
run("FACT: a non-boolean flag is INCONCLUSIVE", {"demo": GOOD.replace("data_key: true", "data_key: yes"),
|
||||
"bookstack": BOOK}, 2, ("INCONCLUSIVE",))
|
||||
finally:
|
||||
shutil.rmtree(ws, ignore_errors=True)
|
||||
|
||||
|
||||
def main():
|
||||
gate = os.path.join(ROOT, "scripts", "check-engine-major.py")
|
||||
if not os.path.isfile(gate):
|
||||
@@ -1259,6 +1314,7 @@ i18n:
|
||||
onboarding_cases()
|
||||
family_gate_cases()
|
||||
mem_sum_cases()
|
||||
data_key_cases()
|
||||
|
||||
if fails:
|
||||
print()
|
||||
|
||||
Reference in New Issue
Block a user