catalog gates: one entry point, mandated in CLAUDE.md (R-161 ruling)

scripts/catalog_gates.py runs all three gates - image-pins, image-resolvable,
volume-persistence - and exits non-zero if any fails. Mandated in CLAUDE.md the way
felhom.eu/scripts/site_gates.py is: run it after any template change, naming the
app(s) you touched.

Operator ruling, recorded because both alternatives were rejected for measured
reasons. Controller-side enforcement at template load was rejected because such a
check can only read the file, and a static audit of all 53 templates reports the
catalog clean INCLUDING papra - it would pass on the exact defect it exists to
catch; the property is decidable only at runtime. CI was rejected for now: neither
repo has any, and there are no users yet. What was chosen copies the shape that
demonstrably works here - of this project's gates, the only ones that ever get run
are the ones with a single entry point named in a CLAUDE.md; site_gates.py is run,
and R-29's three orphans are named nowhere and have stopped nothing.

Behaviour: 0 all clean / 1 convicted / 2 UNDETERMINED, never a pass; a conviction
outranks an undetermined result so the reader knows which they have. Gate output is
streamed, not captured. App names scope the two gates that accept scoping; with no
names the runtime gate deploys every template and belongs on a scratch host.
Adding a fourth gate means one line in GATES.

R-161 stays OPEN at reduced scope: this is convention, run by a person. Real
automatic enforcement is owed when a second person touches templates.

Verified: image-pins passes standalone (53 templates, 0 unpinned), the
unknown-option path exits 2, and the aggregation was unit-checked over five
gate-code combinations. The runtime leg was deliberately NOT executed - it deploys
templates via docker compose and DooPlex is the recovery chain - so the runner's
end-to-end invocation of that third gate is inferred, not measured, and is flagged
in REPORT.md to be closed on a scratch host at the next campaign.

REPORT.md overwritten per convention; the persistence sweep's report is preserved
at audits/persistence-sweep-2026-08-02/ and pointed to from the new one.
This commit is contained in:
2026-08-02 14:03:55 +02:00
parent 6d45b60f94
commit fd7747d129
5 changed files with 213 additions and 35 deletions
+28
View File
@@ -1,5 +1,33 @@
# Changelog
## 2026-08-02 — one entry point for the catalog's gates (R-161 ruling)
`scripts/catalog_gates.py` runs all three gates — image-pins, image-resolvable, volume-persistence —
and exits non-zero if any fails. Mandated in `CLAUDE.md` the way `felhom.eu/scripts/site_gates.py` is:
**run it after any template change**, naming the app(s) you touched.
**Operator ruling, recorded because the alternatives were rejected for measured reasons.**
Controller-side enforcement at template load was rejected: such a check can only read the file, and a
static audit of all 53 templates reports the catalog clean **including papra** — it would pass on the
exact defect it exists to catch. CI was rejected for now: neither repo has any, and there are no users
yet. What was chosen copies the shape that demonstrably works here — of this project's gates, the only
ones that ever get run are the ones with a single entry point named in a CLAUDE.md; `site_gates.py` is
run, and R-29's three orphaned gates are named nowhere and have stopped nothing.
Behaviour: `0` all clean · `1` convicted · `2` UNDETERMINED, **never a pass**; a conviction outranks an
undetermined result in the summary so the reader knows which they have. Gate output is streamed, not
captured — a runner that swallows diagnostics makes a conviction unreadable. Scoping passes app names
through to the two gates that accept them; with no names the runtime gate deploys every template and
belongs on a scratch host.
**R-161 stays OPEN at reduced scope:** this is convention, run by a person. Real automatic enforcement
is owed when a second person touches templates.
Verified: `image-pins` passes standalone (53 templates, 0 unpinned); the unknown-option path exits 2;
the aggregation was unit-checked over five gate-code combinations. **The runtime leg was deliberately
NOT executed on DooPlex** — it deploys templates via `docker compose`, and DooPlex is the recovery
chain; it belongs on a scratch host.
## 2026-08-02 — persistence sweep: does every app's data land in a folder the template preserves?
Campaign 10's R-156 found papra writing its database into the container's writable layer while the
+14
View File
@@ -27,6 +27,20 @@ deployed `app.yaml` (customer secrets) is never overwritten. Full deploy details
- Update `CHANGELOG.md` (newest on top) and overwrite `REPORT.md` with every pushed change.
- No secrets in any committed file; secrets are generated at deploy time via `deploy_fields`
`generate:` specs.
- **Run `python3 scripts/catalog_gates.py <app>` after ANY template change** — it is the ONE entry
point and runs all three gates below, exiting non-zero if any fails. Name the app(s) you touched
and it scopes the two gates that accept scoping, which is fast; with no names the runtime gate
deploys **every** template, so that form belongs **on a scratch host, never a customer box**.
Exit: 0 all clean · 1 convicted · 2 UNDETERMINED, which is never a pass.
**Why a runner and not four separate invocations** (operator ruling 2026-08-02, R-161): of this
project's gates, the only ones that ever get run are the ones with a single entry point named in a
CLAUDE.md — `felhom.eu/scripts/site_gates.py` is run, and R-29's three orphans are named nowhere and
have stopped nothing. Controller-side enforcement was rejected because a check at template load can
only read the file, and a static audit of all 53 templates reports the catalog clean **including
papra** — it would pass on the exact defect it exists to catch. CI was rejected for now: neither
repo has any, and there are no users yet. **R-161 stays open at reduced scope** — this is
convention, run by a person; real automatic enforcement is owed when a second person touches
templates.
- **Never `:latest` or untagged images in templates** — pin a concrete version tag; an app deployed
anywhere in the fleet is pinned to the digest it is currently running (a pin must never cause a
version jump). Digest pins (`@sha256:`) also count. Gate: `python scripts/check-image-pins.py`
+51 -35
View File
@@ -1,44 +1,60 @@
# REPORT — catalog persistence sweep (2026-08-02)
# REPORT — one entry point for the catalog's gates (2026-08-02)
**Question:** does every app's data actually land in a folder the template preserves?
**Answer, over all 53 templates:** **43 CLEAN · 3 BROKEN · 7 UNDETERMINED.**
**Change:** `scripts/catalog_gates.py` — runs all three catalog gates, non-zero exit on any failure —
plus its mandate in `CLAUDE.md` and a `REUSE.md` row. **No gate logic changed; no template touched.**
Full report, per-app evidence, proofs and proposed register entries:
**`audits/persistence-sweep-2026-08-02/README.md`**.
## What was found
| app | defect | disposition |
|---|---|---|
| **gramps-web** | Mounted `/app/data` (a path the app never writes), `/app/media`, `/tmp`. Its accounts database AND **its family tree** landed in the container's writable layer — destroyed by any redeploy, absent from every backup, while `gramps_data` was tarred nightly as an empty directory. | **FIXED** — persists the 8 paths the image's own environment names. Survives-a-redeploy proven: `users.sqlite` and the family-tree files byte-identical, same inode. |
| **wishlist** | Mounted `wishlist_data:/data`, another path the app never writes. `prod.db` landed in the **anonymous** volume from the image's `VOLUME` directive — never in `ResolveDockerVolumeNames`, so never backed up, and orphaned by a redeploy. | **FIXED** — mounts `/usr/src/app/data` + `/usr/src/app/uploads`. `prod.db` byte-identical with the same inode across a redeploy. |
| **papra** (R-156) | Mounts `papra_data:/app/data`, which is root-owned, unwritable by uid 999, and **does not exist in the papra image at all**; the database is written to `/app/app-data/db/`. | **NOT FIXED — referred to the operator.** Fix prepared and proven in the scratch guest (current: `db.sqlite` differs after a redeploy, so a real account created via the API is lost; fixed: byte-identical, it survives). It was deployed on one box; that box (`c10-soak`) was torn down by the other session mid-run and papra now appears nowhere in hub telemetry — but that is *absence* evidence, and `demo-hp` is fenced. One command settles it: §6.1a. |
7 UNDETERMINED are listed with reasons and **never folded into CLEAN** (§4.2): 4 apps write nothing
until a user completes setup, 3 could not be started.
> The previous REPORT.md described the **catalog persistence sweep** (43 CLEAN · 3 BROKEN ·
> 7 UNDETERMINED over 53 templates). It is not lost: the full report, per-app evidence and proofs are
> at `audits/persistence-sweep-2026-08-02/README.md`, and its CHANGELOG entry sits directly below this
> one. This file is overwritten per the repo convention.
## What was built
**`scripts/check-volume-persistence.py`** — the third catalog gate and the only RUNTIME one. This
defect class is **invisible to static analysis**, measured rather than assumed: a static audit of
all 53 composes reports the catalog clean *and reports papra clean*. So the gate deploys each
template, exercises it into writing data, and compares where the data landed with what is mounted.
Exit 0 clean / **1 REFUSED** / 2 undecided; UNDETERMINED is never a pass.
python3 scripts/catalog_gates.py # every AVAILABLE app, all three gates
python3 scripts/catalog_gates.py papra wishlist # only these app dirs — the normal case
python3 scripts/catalog_gates.py --all # include hidden/abandoned apps too
It **refuses to report at all** unless it has just re-proven itself in both directions against two
canary templates that differ only in which path the volume mounts at — so every run carries a live
demonstration of R-156 and of its fix. 41 fixture tests driving `check()` (the function `__main__`
calls); every rule red-proofed.
| Gate | Kind | Scoped by app name |
|---|---|---|
| `check-image-pins.py` | static, instant, whole repo | no |
| `check-image-resolvable.py` | network | yes |
| `check-volume-persistence.py` | **runtime** | yes |
Registered in `CLAUDE.md` and `REUSE.md`. **Enforcement is convention, not CI** — the catalog repo
has no CI of any kind. Raising that is proposed as R-161.
Exit: **0** all clean · **1** convicted · **2** UNDETERMINED. A conviction outranks an undetermined
result in the summary, and **2 is never folded into a pass** — an app that wrote nothing has not been
shown correct, and a throttled registry has not shown an image alive. Gate output is streamed rather
than captured: a runner that swallows diagnostics makes a conviction unreadable.
## Method notes worth carrying forward
## Why a runner, and not the two alternatives (operator ruling, R-161)
- **Seven detector defects were found by adjudicating flagged apps rather than trusting the label**,
and six of them made a *correct* app look broken. Every one came from a rule identifying data by
its **name** rather than by evidence of what it is. Detail: §3.5.
- The sweep was **restarted from the first app twice** rather than merge a matrix built by more than
one version of the rules.
- No `docker exec` anywhere — Campaign 7 §1.1's OCI-error-to-stdout trap.
- No hub record was created; the hub was read-only throughout. Teardown, all three layers: §9.
- **Controller-side at template load — rejected, and this is the substantive reason.** Such a check
can only read the file. A static audit of all 53 templates reports the catalog clean **including
papra**, whose compose is well-formed while its database goes to the container's writable layer.
**It would pass on the exact defect it exists to catch.** The property is decidable only at runtime.
- **CI — rejected for now.** Neither repo has any CI to build on, and there are no users yet.
- **A named single entry point — chosen, because it is the shape that works here.** Of this project's
gates, the only ones that ever get run are those with one entry point named in a CLAUDE.md:
`felhom.eu/scripts/site_gates.py` is run; R-29's three orphaned gates are named nowhere and have
stopped nothing. This copies that shape rather than adding a fourth gate nobody invokes.
**R-161 stays OPEN at reduced scope** — this is convention, run by a person. Real automatic
enforcement is owed when a second person touches templates.
## Verification — and what was deliberately not run
| Check | Result |
|---|---|
| syntax | OK |
| `check-image-pins.py` standalone | **OK — 53 templates, 0 unpinned images** (exit 0) |
| unknown-option path | exit **2** |
| aggregation, unit-checked over 5 gate-code combinations | `(0,0,0)→0` · `(0,0,1)→1` · `(0,2,0)→2` · `(0,2,1)→1` · `(1,2,0)→1` |
**The runtime leg was NOT executed here, deliberately.** `check-volume-persistence.py` deploys each
template with `docker compose` on the invoking host; DooPlex is Tier 2 — the recovery chain — and the
gate's own documentation says scratch host, never a customer box. Its correctness was already
established by the sweep that wrote it (53 templates, canary self-test in both directions).
**What is therefore unproven here:** the runner's end-to-end invocation of that third gate. The
plumbing is identical to the two it did invoke and the argument passing is unit-checked, but that is an
inference, not a measurement. **Run it once on a scratch host at the start of the next catalog
campaign** — that is the cheapest moment to close it.
+1
View File
@@ -17,6 +17,7 @@ None — this repo is templates/config, not code. See §2/§5.
| deploy_fields conventions | `templates/paperless-ngx/.felhom.yml` (`deploy_fields:` block) | Every app starts with `DOMAIN` (type `domain`) + `SUBDOMAIN` (type `subdomain`, `locked_after_deploy: true`). Secrets: `type: secret` + `generate:` — dominant generators `password:24` (DB passwords) and `hex:32` (app secret keys); `password:16` for shown admin passwords (`type: password`). HDD apps add `HDD_PATH` (`type: path`, placeholder `/mnt/felhom-drives/hdd_1`, locked). Labels/descriptions in Hungarian. |
| Controller-side health probe | `templates/vaultwarden/.felhom.yml` (`healthcheck:` block) | `healthcheck.checks[]` with `type: http` (port only), `type: api` (port + `path` + `expect.status: 200`), or `type: tcp` (port only — mealie, crafty-controller). Prefer `api` with a real health path when the app has one. |
| App lifecycle (`available`/`hidden`/`abandoned`) | `templates/plant-it/.felhom.yml` (`lifecycle:` block) | Optional top-level `lifecycle:` in `.felhom.yml`. Absent/empty ≡ `available`. `hidden` = not offered for new installs; `abandoned` = same, PLUS a permanent "Nem karbantartott" badge + notice on every box already running it. **Deployed instances keep full function in both states** — lifecycle governs what is OFFERED, never what runs; the controller refuses a deploy of a non-available template server-side (fail-closed, so a stale link or direct POST cannot install one). Unknown value → treated as `available` + one WARN, never a broken template. **Do NOT take an app out of circulation by deleting or moving its directory** — that orphans every customer already running it, which is what the 2026-07-21 `retired/` experiment got wrong. The resolvability gate skips non-available apps, so an abandoned app's dead image is not a standing red. |
| **Catalog gates — THE entry point** | `scripts/catalog_gates.py` | **Run `python3 scripts/catalog_gates.py <app>` after ANY template change** (mandated in `CLAUDE.md`). Runs all three gates below in order — image-pins, image-resolvable, volume-persistence — and exits **non-zero if any fails**; **2 (UNDETERMINED) is reported distinctly and is never a pass**, 1 (convicted) outranks 2 in the summary. Naming app(s) scopes the two gates that accept scoping, which is the normal after-a-change run; with no names the RUNTIME gate deploys every template, so that form is **scratch host only**. **Why a runner** (operator ruling 2026-08-02, R-161): the only gates in this project that ever get run are the ones with a single entry point named in a CLAUDE.md — `felhom.eu/scripts/site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. Controller-side enforcement was rejected because a load-time check reads only the file and a static audit reports the catalog clean **including papra** — it would pass on the very defect it exists to catch; CI was rejected for now (neither repo has any, no users yet). Adding a fourth gate here means adding it to `GATES` in this file — nothing else. |
| Image pinning | ALL `templates/*/docker-compose.yml` (`image:` line) | **Never `:latest` or untagged** (recovery-unit `ImagePins` pins the tag — `:latest` breaks restore fidelity). Pin a concrete version tag; an app deployed anywhere in the fleet pins to the digest it is RUNNING (pin ≠ upgrade); `@sha256:` digest pins also count. Gate: `python scripts/check-image-pins.py` after any compose change (swept 2026-07-12: 5 pins). TRAP: ghcr `tags/list` can be stale/partial — verify tag existence via `docker manifest inspect`, never the tag list. |
| Image RESOLVABILITY (does the pin still exist?) | `scripts/check-image-resolvable.py` + `scripts/test_check_image_resolvable.py` | The complement to the pin gate, which is purely syntactic and cannot see rot. Run it at the START of every catalog campaign and before any publish train that vouches the catalog: `python3 scripts/check-image-resolvable.py [app …]`. Exit **0** all resolve, **1** the registry says an image is GONE, **2** INCONCLUSIVE/harness error. **Two traps it encodes, both live-observed:** (a) `docker manifest inspect` prints `toomanyrequests` and **still exits 0** — never trust the exit code alone (same shape as the ISO `validate-answer` trap); (b) the inverse — the first sweep called 24 of 65 pins dead, `postgres:16-alpine` among them, because Docker Hub throttled it partway. Ambiguity therefore resolves to INCONCLUSIVE, never to an accusation; a gate that cries wolf gets ignored. Unauthenticated Hub lookups WILL throttle on a full 65-pin sweep — `docker login` first, or expect exit 2. |
| Volume PERSISTENCE (does the app write where the template preserves?) | `scripts/check-volume-persistence.py` + `scripts/test_check_volume_persistence.py` | The third gate and the only RUNTIME one — **the two image gates are static and this class is invisible to static analysis**, which was measured, not assumed: a static audit of all 53 composes (every declared volume attached, no anonymous mounts, no stray host binds) reports the catalog clean AND reports papra clean. papra's compose is well-formed; it mounts `papra_data:/app/data` while the app writes `/app/app-data/db/db.sqlite` into the container's **writable layer** and cannot write `/app/data` at all — so `DumpAppVolumes` (`felhom-controller internal/backup/backup.go:543`) tars an empty directory and the backup verifies (R-156, Campaign 10). Run: `python3 scripts/check-volume-persistence.py [app …]` **on a scratch host, never a customer box**. Exit **0** all CLEAN, **1** REFUSED, **2** UNDETERMINED/prober untrustworthy. **Traps it encodes:** (a) `A` vs `C` in `docker diff` — a linuxserver.io entrypoint chowning its app tree produced 1305 `C` entries and called calibre-web BROKEN on the first pass, so DATA is decided from `A` only and a `C` on a DB file is adjudicated by comparing bytes against a pristine container of the same image; (b) no `docker exec` anywhere — Campaign 7 §1.1's OCI-error-to-stdout trap, so uid comes from `/proc/<pid>/status` and writability from a host-side `stat`; (c) `base64key` secrets need the controller's `base64:` prefix (`deploy.go:904`) or bookstack serves 500s and the harness looks like an app defect; (d) it self-tests in BOTH directions against two canary templates before reporting anything — a detector that flags nothing turns an unexamined catalog into a documented-clean one. UNDETERMINED is **never** folded into CLEAN. |
+119
View File
@@ -0,0 +1,119 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""catalog_gates.py — THE entry point for this repo's gates. Run from the repo root:
python3 scripts/catalog_gates.py # every AVAILABLE app, all three gates
python3 scripts/catalog_gates.py papra wishlist # only these app dirs (the normal case)
python3 scripts/catalog_gates.py --all # include hidden/abandoned apps too
Gates, in order (all must pass; **non-zero exit on any failure**):
1. image-pins static, instant, whole repo — no :latest / untagged / floating alias
2. image-resolvable network — every pinned tag still EXISTS upstream
3. volume-persistence RUNTIME — the folder a template preserves is the folder the app writes to
WHY THIS FILE EXISTS (operator ruling, 2026-08-02 — R-161).
The volume-persistence gate was built because papra's backup completed, verified, and contained an
empty directory. The obvious enforcement points were both rejected, each for a measured reason:
- **Controller-side, at template load: rejected because it would PASS on the defect it exists to
catch.** A check at load time can only read the file, and papra's compose is well-formed — a
static audit of all 53 templates reports the catalog clean, papra included. The property is only
decidable at runtime (see `check-volume-persistence.py`'s header).
- **CI: rejected for now** — neither repo has any CI to build on, and there are no users yet.
What was chosen instead is the shape that demonstrably works in this project. Of every gate written
here, **the only ones that ever get run are the ones with a single entry point named in a CLAUDE.md**:
`felhom.eu/scripts/site_gates.py` is run; R-29's three orphaned gates are named nowhere and have
stopped nothing. So this copies that shape rather than adding a fourth gate nobody invokes. It is
mandated in `CLAUDE.md` the way `site_gates.py` is.
**R-161 stays OPEN at reduced scope:** this is convention, run by a person. Real automatic
enforcement is owed when a second person touches templates.
EXIT CODES. Each gate returns 0 clean / 1 convicted / 2 inconclusive. This runner exits **non-zero if
any gate is non-zero**, and reports 2 distinctly as INCONCLUSIVE — an undetermined result is never a
pass (an app that wrote nothing has not been shown correct; a throttled registry has not shown an
image alive), but it is also not a conviction, and the operator reading the summary needs to know
which they have.
SCOPE. With app names, every gate that accepts scoping is scoped to them — that is the normal
after-a-template-change run and it is fast. With no names the runtime gate deploys **every** template,
which takes minutes per app and **belongs on a scratch host, never a customer box** (see CLAUDE.md).
"""
import os
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SCRIPTS = os.path.join(ROOT, "scripts")
# (label, filename, accepts_app_scope)
GATES = [
("image-pins", "check-image-pins.py", False),
("image-resolvable", "check-image-resolvable.py", True),
("volume-persistence", "check-volume-persistence.py", True),
]
VERDICT = {0: "OK", 1: "FAILED", 2: "INCONCLUSIVE"}
def run_gate(label, script, args):
path = os.path.join(SCRIPTS, script)
if not os.path.exists(path):
print("FAIL: %s%s is missing from scripts/" % (label, script))
return 1
print("\n" + "=" * 78)
print("== gate: %s (%s%s)" % (label, script, (" " + " ".join(args)) if args else ""))
print("=" * 78, flush=True)
# stream the gate's own output rather than capturing it — its diagnostics are the point,
# and a runner that swallows them makes a conviction unreadable.
return subprocess.call([sys.executable, path] + args, cwd=ROOT)
def main(argv):
include_hidden = "--all" in argv
apps = [a for a in argv if not a.startswith("-")]
unknown = [a for a in argv if a.startswith("-") and a != "--all"]
if unknown:
print("unknown option(s): %s" % " ".join(unknown))
print(__doc__.strip().splitlines()[0])
return 2
scope_note = ("apps: " + ", ".join(apps)) if apps else (
"ALL apps (runtime gate deploys every template — scratch host only)")
print("catalog_gates — %s%s" % (scope_note, " [--all: incl. hidden/abandoned]" if include_hidden else ""))
results = []
for label, script, scoped in GATES:
args = []
if include_hidden:
args.append("--all")
if scoped and apps:
args += apps
results.append((label, run_gate(label, script, args)))
print("\n" + "=" * 78)
print("== summary")
print("=" * 78)
worst = 0
for label, rc in results:
print(" %-20s %-13s (exit %d)" % (label, VERDICT.get(rc, "ERROR"), rc))
# 1 (a conviction) outranks 2 (undetermined) in what it tells the operator to do
if rc != 0:
worst = 1 if rc == 1 or worst == 1 else 2
if worst == 0:
print("\nall catalog gates OK")
return 0
convicted = [l for l, rc in results if rc == 1]
undecided = [l for l, rc in results if rc not in (0, 1)]
if convicted:
print("\nCONVICTED: %s" % ", ".join(convicted))
if undecided:
print("UNDETERMINED (never a pass): %s" % ", ".join(undecided))
return worst
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))