diff --git a/REUSE.md b/REUSE.md index c8dcdc2..e2ffe76 100644 --- a/REUSE.md +++ b/REUSE.md @@ -20,7 +20,7 @@ Templates are config; the few script helpers other scripts must REUSE, never re- |---|---|---| | **The one canonical example app** | `templates/paperless-ngx/` (both files) | Multi-container (app + postgres + redis), HDD + userdata mounts, full deploy_fields spectrum (domain/subdomain/secret/password/text/path/select). Copy this structure for any new app. | | `.felhom.yml` required fields | `templates/paperless-ngx/.felhom.yml` | All 53 apps: `display_name`, `description` (Hungarian), `category`, `subdomain`, `slug`, `resources{mem_request, mem_limit, pi_compatible, needs_hdd}`, `deploy_fields`, `app_info{tagline, use_cases, first_steps, ...}`, `healthcheck`. Optional: `smtp_mapping` (email-capable apps), `open_path` (non-root landing page, e.g. ghost). | -| deploy_fields conventions | `templates/paperless-ngx/.felhom.yml` (`deploy_fields:` block) | Every app starts with `DOMAIN` (type `domain`) + `SUBDOMAIN` (type `subdomain`, `locked_after_deploy: true`). Secrets: `type: secret` + `generate:` — dominant generators `password:24` (DB passwords) and `hex:32` (app secret keys); `password:16` for shown admin passwords (`type: password`). HDD apps add `HDD_PATH` (`type: path`, placeholder `/mnt/felhom-drives/hdd_1`, locked). Labels/descriptions in Hungarian. | +| deploy_fields conventions | `templates/paperless-ngx/.felhom.yml` (`deploy_fields:` block) | Every app starts with `DOMAIN` (type `domain`) + `SUBDOMAIN` (type `subdomain`, `locked_after_deploy: true`). Secrets: `type: secret` + `generate:` — dominant generators `password:24` (DB passwords) and `hex:32` (app secret keys); `password:16` for shown admin passwords (`type: password`). HDD apps add `HDD_PATH` (`type: path`, placeholder `/mnt/felhom-drives/hdd_1`, locked). Labels/descriptions in Hungarian. **A key that ENCRYPTS STORED DATA gets `data_key: true`** + a comment saying why (the restore recovers it and refuses without it; the box never regenerates it) — gated by `scripts/check-data-key.py` (R-127): an env var naming `ENCRYPTION_KEY`/`PEPPER` must be flagged, any other data key goes in its `REGISTRY` with the reason, and an unexplained flag is refused. A key that only SIGNS sessions is not one, whatever its label says. | | **Known default login → `after_install:`** (decision 45, controller ≥ 0.279.0) | `templates/bookstack/.felhom.yml` (`ADMIN_PASSWORD` field + `after_install:` block); `FIRST-ADMIN.md` for every app | An app that starts with a known admin login gets a generated `type: password` field (`generate: "password:24"`, `locked_after_deploy: true`) and ONE `after_install: {service, env: [ADMIN_PASSWORD], command: [...], success: ""}` through the app's OWN CLI, run once after a FRESH install. Keep `app_info.default_creds` — the page hides it once the command succeeded and warns while it is in effect. **Prove on 9202 (drill catalog) before live: the default fails, the generated password works, a wrong one fails.** TRAPS: `success:` is required because a CLI can exit 0 on an error (claper's `rpc`); **pass the password as its own argument, never inside program code** (`sys.argv[1]` — mealie, wger; security review 2026-09-29); a special-character policy uses `generate: "password:24:special"` (controller ≥ 0.280.0, calibre-web); a Hungarian first-steps change needs `check-copy-i18n.py --capture-freeze`. | | **Open first-run screen → `setup_gate:`** (decision 46, controller ≥ 0.280.0) | `templates/n8n/.felhom.yml` (probe), `templates/uptime-kuma/.felhom.yml` (no probe → the household's button); `FIRST-ADMIN.md` | `setup_gate: true` + optional `setup_done_probe: {url: http://:/, field: , done: ""}` | TRAPS: the probe must FLIP on the setup — measure it before and after on 9202; an app with open sign-up after setup (R-711) is not closed by the gate; `url` is read on the docker network, so it names the container, not the subdomain. | | **Open sign-up after the setup → `signup_block:`** (decision 47, controller ≥ 0.281.0) | `templates/opengist/.felhom.yml`, `templates/calcom/.felhom.yml` | `signup_block: ""` + `app_info.add_people` (hu) / `i18n.en.app_info.add_people` | TRAPS: block the app's API sign-up call, not only the page; an app's own invite link often uses the same address (the household's 15-minute window covers it); `add_people` is copy — `--capture-freeze`. | diff --git a/scripts/catalog_gates.py b/scripts/catalog_gates.py index 9d3f753..362e64f 100644 --- a/scripts/catalog_gates.py +++ b/scripts/catalog_gates.py @@ -27,6 +27,8 @@ Gates, in order (all must pass; **non-zero exit on any failure**): record (NEW-APP-CHECKLIST.md; the 53 apps published before 2026-10-01 are exempt by name) 11. mem-limit-sum static, instant, whole repo — `.felhom.yml` resources.mem_limit equals the sum of every service's deploy.resources.limits.memory, and every service has one (R-758) + 12. data-key static, instant, whole repo — every data-encrypting key carries data_key: true (name rule + or REGISTRY), and no flag is unexplained (R-127) 4. engine-major static, needs GIT HISTORY — no database engine pin crosses a MAJOR version (operator ruling 2026-09-13; expires when Slice 4 / R-448 ships). Runs in the pre-push hook, which has the full clone; on a SHALLOW clone (CI fetches at @@ -123,6 +125,9 @@ GATES = [ # templates were under it — the deploy screen and the box's overcommit warning read less than the app may take. # Static, stdlib only (no PyYAML on CI), so --fast: the hook and CI. ("mem-limit-sum", "check-mem-limit-sum.py", True, True, False), + # R-127 leg (a) (2026-10-05): every data-encrypting key carries `data_key: true` (by its NAME, or by a registry + # entry with its reason) and every flag is accounted for — the restore's fail-closed gate reads only that flag. + ("data-key", "check-data-key.py", True, True, False), ] # 3 (R-605): the gate's HARNESS refused to run — its canary failed or it had nothing to judge — so NO app was diff --git a/scripts/check-data-key.py b/scripts/check-data-key.py new file mode 100644 index 0000000..b127cc1 --- /dev/null +++ b/scripts/check-data-key.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""check-data-key.py — every data-encrypting key carries `data_key: true`, and every flag is accounted for (R-127 a). + +WHAT WENT WRONG. `data_key: true` on a deploy field tells the controller the app ENCRYPTS STORED DATA with it: the +restore RECOVERS the value and refuses (fail-closed) when it cannot, and the box never generates a new one +(felhom-controller internal/backup/restore_unit.go missingDataKeys; internal/stacks/deploy.go GenerateSecretForField). +In 2026-08 only five fields carried it, while n8n's N8N_ENCRYPTION_KEY, calcom's CALENDSO_ENCRYPTION_KEY, wanderer's +POCKETBASE_ENCRYPTION_KEY and bookstack's APP_KEY (two-factor secrets) did not — so a restore missing one of them +would have proceeded onto data it cannot decrypt instead of refusing. + +WHY NOT THE LABEL. The Hungarian label „Titkosítási kulcs" ("Encryption key") sits on 24 secrets, most of which only +SIGN sessions (Django SECRET_KEY, Phoenix SECRET_KEY_BASE, JWT secrets): regenerating those signs everyone out, it +loses no data. A label is copy, frozen byte for byte, and it is not the fact. The facts this gate reads: + + 1. NAME RULE — a field whose env var names an encryption key or a pepper (`ENCRYPTION_KEY`, `PEPPER`) is a data key + by what the app calls it. It must carry `data_key: true`. + 2. REGISTRY — data keys whose name does not say so (bookstack APP_KEY encrypts two-factor secrets; …) are listed + below BY APP AND FIELD with the reason. Each must carry the flag (unflagging one is a regression), and an entry + whose field no longer exists is STALE (refused — a registry nobody prunes stops meaning anything). + 3. AGREEMENT — a `data_key: true` that neither rule names is refused: add it to REGISTRY with its reason. Over- + flagging is not harmless either: the restore then REFUSES for a key that could have been regenerated. + +stdlib only (the CI runner has no PyYAML). The flag is read only from a field's own block inside the top-level +`deploy_fields:` — never from a comment, never from the `i18n:` block, never from `steps/` files. + +USAGE + python3 scripts/check-data-key.py [app …] [--root=] (`--all` accepted, a no-op: every directory is judged) +Exit: 0 agree · 1 convicted · 2 inconclusive. +Decoys: scripts/test_gate_decoys.py `data_key_cases` (COVERS "data-key"). +""" +import io +import os +import re +import sys + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + +NAME_RULE = re.compile(r"ENCRYPTION_KEY|PEPPER") + +# (app, env_var) -> why it is a data key although its name does not say so. +REGISTRY = { + ("adventurelog", "SECRET_KEY"): "the template's own comment: encrypts stored data; restore must recover it", + ("bookstack", "APP_KEY"): "Laravel encrypt() on every member's two-factor secret (app/Access/Mfa/MfaValue.php)", + ("dawarich", "SECRET_KEY_BASE"): "the template's own comment: stored data unreadable if it changes", + ("papra", "AUTH_SECRET"): "the template's own comment: stored tokens invalid if it changes; keep the old key", + ("sparkyfitness", "BETTER_AUTH_SECRET"): "signs sessions AND encrypts 2FA/TOTP secrets (template comment)", +} + +FIELD_RE = re.compile(r"^ - env_var:\s*['\"]?([A-Za-z0-9_]+)['\"]?\s*(?:#.*)?$") +DATA_KEY_RE = re.compile(r"^ data_key:\s*(\S+?)\s*(?:#.*)?$") +TOP_KEY_RE = re.compile(r"^[A-Za-z0-9_]+:") + + +def fields(meta_text): + """[(env_var, data_key_raw_or_None)] from the top-level deploy_fields: block only.""" + out, cur, inside = [], None, False + for line in meta_text.split("\n"): + if TOP_KEY_RE.match(line): + inside = line.startswith("deploy_fields:") + cur = None + continue + if not inside: + continue + m = FIELD_RE.match(line) + if m: + cur = [m.group(1), None] + out.append(cur) + continue + m = DATA_KEY_RE.match(line) + if m and cur is not None: + cur[1] = m.group(1).strip("'\"") + return [(a, b) for a, b in out] + + +def main(argv): + root, apps = ROOT, [] + for a in argv: + if a.startswith("--root="): + root = a.split("=", 1)[1] + elif a == "--all": + continue + elif a.startswith("-"): + print("unknown option: %s" % a) + return 2 + else: + apps.append(a) + tdir = os.path.join(root, "templates") + if not os.path.isdir(tdir): + print("data-key: no templates/ under %s" % root) + return 2 + every = sorted(n for n in os.listdir(tdir) if os.path.isfile(os.path.join(tdir, n, ".felhom.yml"))) + judged = apps or every + unknown = [a for a in judged if a not in every] + if unknown: + print("data-key: no such template: %s" % ", ".join(unknown)) + return 2 + bad, undecided, flagged = [], [], 0 + seen = set() + for app in judged: + text = io.open(os.path.join(tdir, app, ".felhom.yml"), encoding="utf-8").read() + for env, raw in fields(text): + seen.add((app, env)) + if raw is not None and raw.lower() not in ("true", "false"): + undecided.append("%s/%s: data_key %r is not true/false" % (app, env, raw)) + continue + on = raw is not None and raw.lower() == "true" + flagged += on + named = bool(NAME_RULE.search(env)) + reg = (app, env) in REGISTRY + if (named or reg) and not on: + why = "its name says it encrypts" if named else "registered: " + REGISTRY[(app, env)] + bad.append("%s/%s is a data key (%s) but carries no `data_key: true` — a restore missing it would " + "proceed onto data it cannot decrypt" % (app, env, why)) + elif on and not (named or reg): + bad.append("%s/%s carries `data_key: true` but neither its name nor REGISTRY says why — add it to " + "REGISTRY in scripts/check-data-key.py with the reason (or drop the flag)" % (app, env)) + for (app, env), why in sorted(REGISTRY.items()): + if app in judged and (app, env) not in seen: + bad.append("REGISTRY entry %s/%s is STALE — no such deploy field (%s)" % (app, env, why)) + for b in bad: + print("REFUSED: " + b) + for u in undecided: + print("INCONCLUSIVE: " + u) + if bad: + print("data-key: %d problem(s) in %d template(s)" % (len(bad), len(judged))) + return 1 + if undecided: + return 2 + print("data-key gate OK: %d template(s), %d data key(s), every flag agrees with the name rule and REGISTRY" + % (len(judged), flagged)) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/test_catalog_gates.py b/scripts/test_catalog_gates.py index d438b6b..1b9ddcb 100644 --- a/scripts/test_catalog_gates.py +++ b/scripts/test_catalog_gates.py @@ -65,7 +65,9 @@ class CatalogGatesFastTest(unittest.TestCase): # 11 since 2026-10-01: onboarding (NEW-APP-CHECKLIST.md), fast and history-free, so it runs in CI too. # 12 since family-gate joined — STALE AGAIN until 2026-10-05 (found by the burn-down, round 2). # 13 since 2026-10-05: mem-limit-sum (R-758), fast and history-free, so it runs in CI too. - self.assertEqual(len(mod.GATES), 13) + # 14 since the same night: data-key (R-127 leg a). + self.assertEqual(len(mod.GATES), 14) + self.assertIn("data-key", [g[0] for g in mod.GATES if g[3] and not g[4]]) self.assertIn("mem-limit-sum", [g[0] for g in mod.GATES if g[3] and not g[4]]) self.assertIn("onboarding", [g[0] for g in mod.GATES if g[3] and not g[4]]) self.assertEqual([g[0] for g in mod.GATES if not g[3]], ["image-resolvable", "volume-persistence"]) diff --git a/scripts/test_gate_decoys.py b/scripts/test_gate_decoys.py index 61ddf50..8de99fe 100644 --- a/scripts/test_gate_decoys.py +++ b/scripts/test_gate_decoys.py @@ -58,6 +58,7 @@ COVERS = { "family-gate": "family_gate written only in a COMMENT (not gated, no min_controller owed); min_controller only in a comment; a golden DIRECTORY named 0.287.0 with no bake log (the mkdir shape, R-410); a sibling with no golden (stated NOT CHECKED, never a pass of rule 3) - vs the facts: an unanchorable exception (regex, '/', '..'), an exception list with no gate, min_controller below 0.287.0, the newest baked golden below 0.287.0; and a genuine family app passes (decisions 63/64, finding F1)", "onboarding": "a NEW template with no record; a record missing an id, or carrying it only inside an HTML comment; a `done` whose path does not exist, is an EMPTY directory (the mkdir shape, R-410) or names an absent sibling-repo file; an `n/a` with an empty or two-word reason; an `open` row; `opened:` backdated before the checklist; the template a new app copies lacking a new id - vs a complete record, an id added after `opened:`, and an exempt app's record with open rows (NEW-APP-CHECKLIST.md)", "mem-limit-sum": "the right figure only in a COMMENT (the compose header's 'mem_limit: 640M', the .felhom.yml arithmetic) while the field is wrong; a `memory:` under reservations: (not a limit) making the sum come out right; a `mem_limit:` outside resources:; a steps/ file with the old figure (not judged) - vs the facts: a field under the sum, a service with no limit, an unreadable size (R-758)", + "data-key": "a data_key: true only in a COMMENT or inside the i18n: block (not the field); an 'Encryption key' LABEL on a signing secret (copy, not the fact - must pass unflagged) - vs the facts: an *_ENCRYPTION_KEY field unflagged, a REGISTERED field unflagged, an unexplained flag, a stale registry entry, a non-boolean flag (R-127)", "copy-i18n": "Hungarian edited in a COMMENT/README/display_name (label, not copy) vs a real frozen string changed; an English block that is not English, is not matched to a Hungarian twin, or rewrites a credential (R-560). A retrieval promise REGISTERED in ALLOWLIST_EN passes only for its own app+path+sentence with a real reason; an entry for another app, a rewritten sentence, a stale entry or a two-word reason convicts (R-594). Also the DEGRADED mode CI actually runs — PyYAML shadowed out, freeze only (R-595)", } @@ -747,6 +748,60 @@ def mem_sum_cases(): shutil.rmtree(ws, ignore_errors=True) +def data_key_cases(): + """check-data-key.py reads templates/*/.felhom.yml via --root (R-127 leg a).""" + global ran + import tempfile + ws = tempfile.mkdtemp(prefix="catalog-datakey-") + try: + GOOD = ('display_name: "X"\n' + 'deploy_fields:\n' + ' - env_var: DOMAIN\n type: domain\n' + ' - env_var: SECRET_KEY\n label: "Titkositasi kulcs"\n type: secret\n generate: "hex:32"\n' + ' - env_var: APP_ENCRYPTION_KEY\n type: secret\n generate: "hex:16"\n' + ' # why: encrypts stored data\n data_key: true\n' + '\ni18n:\n en:\n deploy_fields:\n - env_var: SECRET_KEY\n label: "Encryption key"\n') + BOOK = ('deploy_fields:\n - env_var: APP_KEY\n type: secret\n generate: "base64key:32"\n data_key: true\n') + + def run(name, files, expect_rc, must=()): + global ran + cat = os.path.join(ws, "cat") + shutil.rmtree(cat, ignore_errors=True) + for app, text in files.items(): + d = os.path.join(cat, "templates", app) + os.makedirs(d) + io.open(os.path.join(d, ".felhom.yml"), "w", encoding="utf-8").write(text) + r = sh([sys.executable, os.path.join(ROOT, "scripts", "check-data-key.py"), "--root=" + cat], ROOT) + out = r.stdout + r.stderr + ran += 1 + ok = r.returncode == expect_rc and all(m in out for m in must) + print(" %s %-70s rc=%d (expected %d)" % ("ok" if ok else "XX", name, r.returncode, expect_rc)) + if not ok: + fails.append("%s: rc=%d expected %d; missing %s\n%s" % ( + name, r.returncode, expect_rc, [m for m in must if m not in out], out[-400:])) + + print("\n-- data-key: genuine and decoys") + run("GENUINE: a flagged *_ENCRYPTION_KEY; a signing key labelled 'Encryption key' unflagged; bookstack registered", + {"demo": GOOD, "bookstack": BOOK}, 0, ("data-key gate OK", "2 data key(s)")) + print("-- data-key: the facts (each MUST be refused)") + run("FACT: the flag only in a COMMENT", {"demo": GOOD.replace(" data_key: true\n", " # data_key: true\n"), + "bookstack": BOOK}, 1, ("demo/APP_ENCRYPTION_KEY is a data key",)) + run("FACT: the flag only inside the i18n: block", {"demo": GOOD.replace(" data_key: true\n", "") + + " data_key: true\n", "bookstack": BOOK}, 1, + ("demo/APP_ENCRYPTION_KEY is a data key",)) + run("FACT: a REGISTERED data key unflagged (bookstack APP_KEY)", + {"demo": GOOD, "bookstack": BOOK.replace(" data_key: true\n", "")}, 1, ("bookstack/APP_KEY is a data key", "registered")) + run("FACT: an unexplained flag (a signing key flagged)", {"demo": GOOD.replace( + ' generate: "hex:32"\n', ' generate: "hex:32"\n data_key: true\n', 1), "bookstack": BOOK}, 1, + ("demo/SECRET_KEY carries `data_key: true` but neither",)) + run("FACT: a stale REGISTRY entry (bookstack has no APP_KEY any more)", + {"demo": GOOD, "bookstack": BOOK.replace("APP_KEY", "OTHER_KEY").replace(" data_key: true\n", "")}, 1, ("STALE",)) + run("FACT: a non-boolean flag is INCONCLUSIVE", {"demo": GOOD.replace("data_key: true", "data_key: yes"), + "bookstack": BOOK}, 2, ("INCONCLUSIVE",)) + finally: + shutil.rmtree(ws, ignore_errors=True) + + def main(): gate = os.path.join(ROOT, "scripts", "check-engine-major.py") if not os.path.isfile(gate): @@ -1259,6 +1314,7 @@ i18n: onboarding_cases() family_gate_cases() mem_sum_cases() + data_key_cases() if fails: print() diff --git a/templates/bookstack/.felhom.yml b/templates/bookstack/.felhom.yml index 1ae5d1a..3bbf17f 100644 --- a/templates/bookstack/.felhom.yml +++ b/templates/bookstack/.felhom.yml @@ -40,6 +40,10 @@ deploy_fields: type: secret generate: "base64key:32" locked_after_deploy: true + # R-127: Laravel's APP_KEY encrypts each user's two-factor secret (app/Access/Mfa/MfaValue.php encrypt()). + # Regenerating it locks out every member who turned two-factor on, so a restore RECOVERS it and refuses rather + # than make a new one. + data_key: true - env_var: DB_PASSWORD label: "Adatbázis jelszó" diff --git a/templates/calcom/.felhom.yml b/templates/calcom/.felhom.yml index 83bcf99..798c6a8 100644 --- a/templates/calcom/.felhom.yml +++ b/templates/calcom/.felhom.yml @@ -46,6 +46,9 @@ deploy_fields: type: secret generate: "hex:16" locked_after_deploy: true + # R-127: encrypts the connected-app credentials (calendar and video integrations) stored in the database. + # Regenerating it breaks every connected calendar, so a restore RECOVERS it and refuses rather than make a new one. + data_key: true - env_var: DB_PASSWORD label: "Adatbázis jelszó" diff --git a/templates/n8n/.felhom.yml b/templates/n8n/.felhom.yml index 8e58269..0f3ec03 100644 --- a/templates/n8n/.felhom.yml +++ b/templates/n8n/.felhom.yml @@ -40,6 +40,9 @@ deploy_fields: type: secret generate: "hex:16" locked_after_deploy: true + # R-127: encrypts every saved credential in n8n's database (packages/core cipher). Regenerating it makes every + # stored credential unreadable, so a restore RECOVERS it and refuses rather than make a new one. + data_key: true # --- The setup gate (controller >= 0.280.0, `09` §3 decision 46) --- diff --git a/templates/wanderer/.felhom.yml b/templates/wanderer/.felhom.yml index b5d0bfb..2052020 100644 --- a/templates/wanderer/.felhom.yml +++ b/templates/wanderer/.felhom.yml @@ -54,6 +54,9 @@ deploy_fields: type: secret generate: "hex:16" locked_after_deploy: true + # R-127: PocketBase encrypts its stored settings (mail and sign-in provider secrets) with it. Regenerating it + # leaves them unreadable, so a restore RECOVERS it and refuses rather than make a new one. + data_key: true # --- Sign-up (controller >= 0.282.0, `09` §3 decisions 47-48) --- # NOT gated: its web server calls its own database through the public name, which a gate would refuse (measured on