security watch reads zstd layers in CI by itself (R-1327): pinned zstandard 0.25.0 wheel, sha256-checked; RUNTIME log line; HA 2026.9.4 and RomM 5.4.1 dropped from the cache for the CI fill proof; Immich v3.3.5 power-cut repair evidence

scripts/CHANGELOG.md

@@ -1,3 +1,16 @@
+## 2026-10-28 — the security watch reads zstd layers in CI by itself (R-1327)
+
+- **`image_runtime.py`:** zstd layers are decoded by `zstandard` 0.25.0 — an importable one, else the wheel pinned by
+  PyPI's sha256 for this Python and libc (`ZSTANDARD_WHEELS`: CPython 3.12/3.13, musl/glibc), from
+  `~/.cache/felhom-wheels`, fetched only with `FELHOM_WHEEL_FETCH=1` (both workflows set it) — else the `zstd` program,
+  else a NOTE (NOT CHECKED). The wheel is hash-checked every time it is opened and loaded from the checked bytes. Any
+  error while a layer decodes is a NOTE, never a pass.
+- **`security_watch.py`:** a `RUNTIME read=N (S s) from-cache=M not-checked=K zstd-decoder=…` line and one
+  `RUNTIME-READ <image> <s> ok|NOT CHECKED` line per image read tonight.
+- Tests: `test_image_runtime.py` 22 checks (zstd above gzip with no `zstd` program, a damaged and a cut-off zstd layer,
+  no decoder, the wheel loader's hash refusals) — the zstd read seen failing on the 2026-10-27 module first;
+  `test_security_watch.py` W10 counts (59 checks).
+
 ## 2026-10-27 — the security watch reads two notice shapes right (R-1310, part)
 
 - **`security_watch.py` `ghsa_affects`:** a GitHub Actions advisory (ecosystem `actions`, the project's own CI) is no

scripts/image_runtime.py

@@ -12,18 +12,30 @@ boxes ran the bad code. This module reads what the image itself carries:
      `brotli-*` / `brotlicffi-*` `.dist-info` folders of Python. The topmost layer that holds a file wins (a later layer
      replaces an earlier one's copy).
 
-What it cannot read is said, never guessed: a registry it has no token route for, a zstd layer where no `zstd` program exists (Python 3.13 has no zstd),
-a layer over the byte budget — each makes the result `unchecked` with its reason, and the watch prints NOT CHECKED.
+ZSTD LAYERS (R-1327, 2026-10-28). Home Assistant and other images ship zstd-compressed layers, and Python before 3.14
+has no zstd. The decoder is the `zstandard` package (0.25.0), pinned by the sha256 PyPI publishes for each wheel: an
+importable `zstandard` first, else the pinned wheel from the cache (`~/.cache/felhom-wheels`; fetched only when
+FELHOM_WHEEL_FETCH=1 — the CI workflows set it), else the `zstd` program. Not the CI runner image: that image is built
+in the homelab repo and a change there is a DooPlex change. The wheel's hash is checked every time it is opened.
+
+What it cannot read is said, never guessed: a registry it has no token route for, a zstd layer where no decoder exists,
+a layer that does not decode, a layer over the byte budget — each makes the result `unchecked` with its reason, and the
+watch prints NOT CHECKED.
 
 USAGE (also a library: `read_image(ref)` -> dict)
   python3 scripts/image_runtime.py IMAGE[:TAG] ... [--budget-mb=N] [--json]
 """
+import atexit
+import hashlib
 import json
+import os
+import platform
 import re
 import shutil
 import subprocess
 import sys
-import zlib
+import tempfile
+import zipfile
 import tarfile
 import urllib.error
 import urllib.request
@@ -43,6 +55,101 @@ class Unreadable(Exception):
     pass
 
 
+ZSTANDARD_VERSION = "0.25.0"
+# (Python, libc) -> (wheel URL, sha256) — the digests PyPI publishes for the 0.25.0 files (released 2025-09-14), read
+# 2026-10-28. The CI runner is Alpine (musl) with Python 3.12; DooPlex is Debian (glibc) with Python 3.13.
+_PYPI = "https://files.pythonhosted.org/packages/"
+ZSTANDARD_WHEELS = {
+    ("3.12", "musl"): (_PYPI + "fc/5f/75aafd4b9d11b5407b641b8e41a57864097663699f23e9ad4dbb91dc6bfe/"
+                       "zstandard-0.25.0-cp312-cp312-musllinux_1_2_x86_64.whl",
+                       "474d2596a2dbc241a556e965fb76002c1ce655445e4e3bf38e5477d413165ffa"),
+    ("3.13", "musl"): (_PYPI + "b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/"
+                       "zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl",
+                       "4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf"),
+    ("3.12", "glibc"): (_PYPI + "53/60/7be26e610767316c028a2cbedb9a3beabdbe33e2182c373f71a1c0b88f36/"
+                        "zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
+                        "5a56ba0db2d244117ed744dfa8f6f5b366e14148e00de44723413b2f3938a902"),
+    ("3.13", "glibc"): (_PYPI + "63/4b/e3678b4e776db00f9f7b2fe58e547e8928ef32727d7a1ff01dea010f3f13/"
+                        "zstandard-0.25.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
+                        "8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a"),
+}
+_ZSTANDARD = {}   # the loaded module, or the reason there is none — decided once per process
+
+
+def _libc():
+    if platform.libc_ver()[0] == "glibc":
+        return "glibc"
+    import glob
+    return "musl" if glob.glob("/lib/ld-musl-*") else "unknown"
+
+
+def _wheel_cache():
+    return os.path.join(os.environ.get("XDG_CACHE_HOME") or os.path.join(os.path.expanduser("~"), ".cache"),
+                        "felhom-wheels")
+
+
+def zstandard_module(fetch=None, wheels=None, cache=None, try_import=True):
+    """(module, how) or (None, why). An importable `zstandard` first; else the pinned wheel for this Python and libc,
+    from the cache or — only when `fetch` (default: FELHOM_WHEEL_FETCH=1) — from PyPI; its sha256 is checked before
+    every use, and a wheel with any other hash is never opened. Pinned by test_image_runtime.py (wheel_*)."""
+    if try_import:
+        try:
+            import zstandard
+            return zstandard, "zstandard %s (installed)" % getattr(zstandard, "__version__", "?")
+        except ImportError:
+            pass
+    fetch = os.environ.get("FELHOM_WHEEL_FETCH") == "1" if fetch is None else fetch
+    key = ("%d.%d" % sys.version_info[:2], _libc())
+    pin = (wheels if wheels is not None else ZSTANDARD_WHEELS).get(key)
+    if not pin:
+        return None, "no zstd decoder: no pinned zstandard wheel for Python %s / %s" % key
+    url, want = pin
+    cache = cache or _wheel_cache()
+    path = os.path.join(cache, url.rsplit("/", 1)[1])
+    data = open(path, "rb").read() if os.path.isfile(path) else None
+    if data is None or hashlib.sha256(data).hexdigest() != want:
+        if not fetch:
+            return None, "no zstd decoder: the pinned zstandard wheel is not in %s (FELHOM_WHEEL_FETCH=1 fetches it)" % cache
+        try:
+            data = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": UA}), timeout=60).read()
+        except Exception as e:  # noqa: BLE001 — network, DNS, TLS: the zstd layers are NOT CHECKED, said out loud
+            return None, "no zstd decoder: the pinned zstandard wheel could not be fetched (%s)" % str(e)[:100]
+        if hashlib.sha256(data).hexdigest() != want:
+            return None, "no zstd decoder: the fetched zstandard wheel's sha256 is not the pinned one — not used"
+        os.makedirs(cache, exist_ok=True)
+        with open(path + ".new", "wb") as f:
+            f.write(data)
+        os.replace(path + ".new", path)
+    into = tempfile.mkdtemp(prefix="felhom-zstandard-")
+    atexit.register(shutil.rmtree, into, True)
+    import io
+    with zipfile.ZipFile(io.BytesIO(data)) as z:   # from the bytes just hashed, never from the file again
+        z.extractall(into)
+    sys.path.insert(0, into)
+    try:
+        import importlib
+        sys.modules.pop("zstandard", None)
+        mod = importlib.import_module("zstandard")
+    except Exception as e:  # noqa: BLE001
+        return None, "no zstd decoder: the pinned zstandard wheel did not load (%s)" % str(e)[:100]
+    finally:
+        sys.path.remove(into)
+    return mod, "zstandard %s (pinned wheel, sha256 checked)" % getattr(mod, "__version__", "?")
+
+
+def zstd_decoder():
+    """How this process decodes a zstd layer: ('module', mod) | ('program', 'zstd') | (None, why). Decided once."""
+    if "d" not in _ZSTANDARD:
+        mod, how = zstandard_module()
+        if mod:
+            _ZSTANDARD["d"] = ("module", mod)
+        elif shutil.which("zstd"):
+            _ZSTANDARD["d"] = ("program", "zstd")
+        else:
+            _ZSTANDARD["d"] = (None, how)
+    return _ZSTANDARD["d"]
+
+
 def split_ref(ref):
     """'rommapp/romm:5.4.1' -> (registry, path, tag). lscr.io is Docker Hub's mirror (the watch's rule)."""
     tag = "latest"
@@ -152,15 +259,18 @@ def parse_node_h(text):
     return ".".join(p.group(1) for p in parts) if all(parts) else None
 
 
-def scan_layers(reg, layers, budget=DEFAULT_BUDGET):
-    """Stream layers top-down. Returns (found, notes): found holds apk/dpkg/node_h/pydist; the first (topmost) wins."""
+def scan_layers(reg, layers, budget=DEFAULT_BUDGET, decoder=None):
+    """Stream layers top-down. Returns (found, notes): found holds apk/dpkg/node_h/pydist; the first (topmost) wins.
+    `decoder` is zstd_decoder()'s answer (a seam for the tests); a zstd layer with none is a NOTE, never skipped quietly."""
     found, notes, spent = {}, [], 0
     for layer in reversed(layers):
         mt, size = layer.get("mediaType", ""), int(layer.get("size") or 0)
         zstd = "zstd" in mt
-        if zstd and not shutil.which("zstd"):
-            notes.append("zstd layer %s: no zstd program here" % layer["digest"][:19])
-            continue
+        if zstd:
+            kind, dec = decoder or zstd_decoder()
+            if not kind:
+                notes.append("zstd layer %s: %s" % (layer["digest"][:19], dec))
+                continue
         if spent + size > budget:
             notes.append("byte budget reached before layer %s (%d MB)" % (layer["digest"][:19], size >> 20))
             break
@@ -168,7 +278,9 @@ def scan_layers(reg, layers, budget=DEFAULT_BUDGET):
         resp = reg.blob(layer["digest"], stream=True)
         proc = None
         try:
-            if zstd:   # Python 3.13 has no zstd: the zstd program decompresses the stream (a pipe, nothing on disk)
+            if zstd and kind == "module":   # the pinned zstandard: a streaming reader, nothing on disk
+                fileobj, mode = dec.ZstdDecompressor().stream_reader(resp, read_across_frames=True), "r|"
+            elif zstd:   # the zstd program decompresses the stream (a pipe, nothing on disk)
                 proc = subprocess.Popen(["zstd", "-dc"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
                 import threading
 
@@ -202,7 +314,7 @@ def scan_layers(reg, layers, budget=DEFAULT_BUDGET):
                         if pm:
                             key = pm.group(1).lower()
                             found.setdefault("pydist", {}).setdefault(key, pm.group(2))
-        except (tarfile.TarError, OSError, EOFError, zlib.error) as e:
+        except Exception as e:  # noqa: BLE001 — tar, zlib, zstd (ZstdError), I/O: the layer is NOT CHECKED, never a pass
             notes.append("layer %s unreadable: %s" % (layer["digest"][:19], type(e).__name__))
         finally:
             if proc:

scripts/security_watch.py

@@ -275,6 +275,14 @@ def cached_runtime_reader(cache, resolve=None, read=None):
     caches a COMPLETE result only (an unchecked one is tried again the next night). `cache` lives in the state file."""
     resolve = resolve or image_runtime.resolve_digest
     read = read or image_runtime.read_image
+    import threading
+    import time
+    lock = threading.Lock()
+    stats = {"cached": 0, "read": 0, "read_s": 0.0, "not_checked": 0}
+
+    def count(key, n=1):
+        with lock:
+            stats[key] += n
 
     def reader(ref):
         try:
@@ -287,12 +295,21 @@ def cached_runtime_reader(cache, resolve=None, read=None):
             hit = dict(cache[digest], ref=ref)
             hit["_used"] = True
             cache[digest]["_seen"] = True
+            count("cached")
             return hit
+        t0 = time.monotonic()
         res = read(ref)
+        count("read_s", time.monotonic() - t0)
+        count("read")
+        print("RUNTIME-READ %s %.0f s %s" % (ref, time.monotonic() - t0, "NOT CHECKED: " + "; ".join(res["unchecked"])[:160]
+                                             if res.get("unchecked") else "ok"), flush=True)
         if digest and not res.get("unchecked"):
             cache[digest] = {k: v for k, v in res.items() if k != "ref"}
             cache[digest]["_seen"] = True
+        else:
+            count("not_checked")
         return res
+    reader.stats = stats   # the RUNTIME line of the log: how much was read tonight and how long it took (R-1327)
     return reader
 
 
@@ -855,14 +872,19 @@ def main(argv):
     if state_path and os.path.isfile(state_path):
         prior = json.load(open(state_path, encoding="utf-8"))
     rcache = {k: dict(v, _seen=False) for k, v in (prior.get("runtime_cache") or {}).items()}
-    run(items, net, today, rotate=not token,
-        runtime=None if "no-runtime" in opts else cached_runtime_reader(rcache))
+    rreader = None if "no-runtime" in opts else cached_runtime_reader(rcache)
+    run(items, net, today, rotate=not token, runtime=rreader)
     for line in table(items):
         print(line)
     counts = {}
     for it in items:
         counts[it["class"]] = counts.get(it["class"], 0) + 1
     print("COUNTS " + " ".join("%s=%d" % (k.replace(" ", "_"), v) for k, v in sorted(counts.items())))
+    if rreader:
+        st = rreader.stats
+        print("RUNTIME read=%d (%.0f s) from-cache=%d not-checked=%d zstd-decoder=%s" % (
+            st["read"], st["read_s"], st["cached"], st["not_checked"],
+            image_runtime.zstd_decoder()[1] if st["read"] else "not needed"))
     print("REQUESTS " + " ".join("%s=%d" % kv for kv in sorted(net.requests.items())))
     state_path = opts.get("state")
     state, missing = {}, True

scripts/test_image_runtime.py

@@ -6,12 +6,22 @@ version header gives the version; an image reference splits the way the watch's
 Hub; a bare name is library/); a tar layer streamed top-down returns the TOPMOST copy of a file and the Python
 `.dist-info` versions; the DooPlex list keeps one line per image:tag with every file that names it, and skips templated
 and variable images. Red-proof: a scan that keeps the FIRST (bottom) copy instead of the topmost fails `topmost`.
+
+ZSTD (R-1327, 2026-10-28): a zstd layer above a gzip one is read with NO `zstd` program on the PATH (the CI runner has
+none) — through the pinned `zstandard` wheel; a zstd layer that does not decode, and one with no decoder at all, are
+NOTES (the watch's NOT CHECKED), never a pass; the wheel loader opens a wheel only when its sha256 is the pinned one.
+Where no decoder can be had (no wheel in the cache, FELHOM_WHEEL_FETCH unset) the read check says NOT CHECKED out loud
+and is not counted; with FELHOM_WHEEL_FETCH=1 (CI sets it) a missing decoder FAILS. Red-proof: the 2026-10-27 module
+(the `zstd` program or nothing) fails `zstd: a zstd layer is read with no zstd program`.
 """
+import base64
+import hashlib
 import io
 import os
 import sys
 import tarfile
 import tempfile
+import zipfile
 
 HERE = os.path.dirname(os.path.abspath(__file__))
 sys.path.insert(0, HERE)
@@ -47,6 +57,78 @@ class FakeReg:
         return io.BytesIO(self.blobs[digest])
 
 
+# A tar holding an Alpine package database (nodejs 24.22.1, brotli-libs 1.2.1), compressed by `zstd -19` (2026-10-28).
+ZSTD_LAYER = base64.b64decode(
+    "KLUv/QRoLQMAsgUSEJB9hC/xJSHf5yhLWm0eYqY+iotURKotdmoB8FiUYcw4RjGQJXy2quX/jpvO2z94QaiU9/dP/+/npPcHKW0610jcI/M24iI"
+    "NCiAwMT3M5Vx0jvfWN08dUPKASwQMGmAZDXf7YaaA")
+ZSTD_MT = "application/vnd.oci.image.layer.v1.tar+zstd"
+GZ_MT = "application/vnd.oci.image.layer.v1.tar+gzip"
+
+
+def zstd_checks():
+    gz = tar_bytes({"usr/local/lib/python3.12/site-packages/brotli-1.2.1.dist-info/RECORD": b"x",
+                    "lib/apk/db/installed": b"P:nodejs\nV:20.0.0-r0\n"})
+    layers = [{"digest": "sha256:gz", "mediaType": GZ_MT, "size": len(gz)},
+              {"digest": "sha256:zst", "mediaType": ZSTD_MT, "size": len(ZSTD_LAYER)}]
+    reg = FakeReg({"sha256:gz": gz, "sha256:zst": ZSTD_LAYER})
+    path = os.environ.get("PATH", "")
+    os.environ["PATH"] = tempfile.mkdtemp()          # no `zstd` program: the CI runner's case
+    ir._ZSTANDARD.clear()                            # the decoder is decided once per process: decide it HERE
+    try:
+        found, notes = ir.scan_layers(reg, layers)
+    finally:
+        os.environ["PATH"] = path
+        ir._ZSTANDARD.clear()
+    no_decoder = [n for n in notes if "no zstd decoder" in n]
+    if no_decoder and os.environ.get("FELHOM_WHEEL_FETCH") != "1":
+        print("  NOT CHECKED  zstd: a zstd layer is read with no zstd program — %s" % no_decoder[0])
+    else:
+        check("zstd: a zstd layer is read with no zstd program (the top layer's apk db wins)",
+              (found.get("apk") or {}).get("nodejs") == "24.22.1-r0" and not notes, (found.get("apk"), notes))
+        check("zstd: the gzip layer under it is still read", (found.get("pydist") or {}).get("brotli") == "1.2.1",
+              found.get("pydist"))
+    found, notes = ir.scan_layers(reg, [{"digest": "sha256:zst", "mediaType": ZSTD_MT, "size": 9}],
+                                  decoder=(None, "no zstd decoder: test"))
+    check("zstd: no decoder is a NOTE (NOT CHECKED), never a silent pass", notes and "no zstd decoder" in notes[0]
+          and not found, (found, notes))
+    mod, _how = ir.zstandard_module()
+    if mod:
+        bad = ZSTD_LAYER[:40] + b"\x00" * 30 + ZSTD_LAYER[70:]
+        found, notes = ir.scan_layers(FakeReg({"sha256:bad": bad}), [{"digest": "sha256:bad", "mediaType": ZSTD_MT,
+                                                                      "size": len(bad)}], decoder=("module", mod))
+        check("zstd: a layer that does not decode is a NOTE (NOT CHECKED), never a pass",
+              any("unreadable" in n for n in notes), (found, notes))
+        found, notes = ir.scan_layers(FakeReg({"sha256:t": ZSTD_LAYER[:60]}), [{"digest": "sha256:t", "mediaType": ZSTD_MT,
+                                                                                "size": 60}], decoder=("module", mod))
+        check("zstd: a cut-off layer is a NOTE (NOT CHECKED), never a pass", any("unreadable" in n for n in notes),
+              (found, notes))
+
+
+def wheel_checks():
+    """The loader opens a wheel only when its sha256 is the pinned one. A fake `zstandard` wheel, served by file://."""
+    d = tempfile.mkdtemp()
+    whl = os.path.join(d, "zstandard-0.0-fake.whl")
+    with zipfile.ZipFile(whl, "w") as z:
+        z.writestr("zstandard/__init__.py", "__version__ = 'fake-for-test'\n")
+    good = hashlib.sha256(open(whl, "rb").read()).hexdigest()
+    key = ("%d.%d" % sys.version_info[:2], ir._libc())
+    url = "file://" + whl
+    mod, how = ir.zstandard_module(fetch=True, wheels={key: (url, good)}, cache=os.path.join(d, "c1"), try_import=False)
+    check("wheel: the pinned wheel is fetched, hash-checked and loaded", mod is not None and
+          getattr(mod, "__version__", "") == "fake-for-test" and "sha256 checked" in how, how)
+    mod, how = ir.zstandard_module(fetch=True, wheels={key: (url, "0" * 64)}, cache=os.path.join(d, "c2"), try_import=False)
+    check("wheel: a wheel whose sha256 is not the pinned one is never loaded", mod is None and "not the pinned" in how, how)
+    c3 = os.path.join(d, "c3")
+    os.makedirs(c3)
+    open(os.path.join(c3, os.path.basename(whl)), "wb").write(b"changed")
+    mod, how = ir.zstandard_module(fetch=False, wheels={key: (url, good)}, cache=c3, try_import=False)
+    check("wheel: a changed wheel in the cache is not loaded, and without FETCH nothing is fetched", mod is None and
+          "not in" in how, how)
+    mod, how = ir.zstandard_module(fetch=True, wheels={}, cache=c3, try_import=False)
+    check("wheel: no pin for this Python is a reason, never a guess", mod is None and "no pinned" in how, how)
+    sys.modules.pop("zstandard", None)
+
+
 def main():
     apk = ir.parse_apk("P:musl\nV:1.2.5-r10\n\nP:nodejs\nV:24.18.1-r0\n\nP:brotli-libs\nV:1.2.0-r1\n")
     check("apk database -> nodejs and brotli-libs", apk.get("nodejs") == "24.18.1-r0" and apk.get("brotli-libs") == "1.2.0-r1", apk)
@@ -75,6 +157,8 @@ def main():
     found, notes = ir.scan_layers(FakeReg({}), [{"digest": "sha256:z", "mediaType": "application/vnd.oci.image.layer.v1.tar+zstd",
                                                 "size": 10}], budget=0)
     check("a layer over the byte budget is a NOTE, never a silent pass", notes and ("budget" in notes[0] or "zstd" in notes[0]), notes)
+    zstd_checks()
+    wheel_checks()
     d = tempfile.mkdtemp()
     open(os.path.join(d, "a.yaml"), "w").write("spec:\n  containers:\n  - image: rommapp/romm:5.4.0\n    name: x\n"
                                                 "  - image: '{{ .Values.image }}'\n  - image: $IMG\n")

scripts/test_security_watch.py

@@ -389,7 +389,11 @@ def t_runtime():
     check("W10 an UNCHECKED result is not cached (tomorrow tries again)", "sha256:f" not in cache, list(cache))
     src = open(os.path.join(HERE, "security_watch.py"), encoding="utf-8").read()
     check("W10 main() hands the reader to run() (off only with --no-runtime)",
-          "runtime=None if \"no-runtime\" in opts else cached_runtime_reader(rcache))" in src)
+          "rreader = None if \"no-runtime\" in opts else cached_runtime_reader(rcache)" in src
+          and "runtime=rreader)" in src)
+    check("W10 the reader counts what it read and from the cache (the log's RUNTIME line)",
+          r.stats["read"] == 2 and r.stats["cached"] == 1 and r.stats["not_checked"] == 1 and "RUNTIME read=" in src,
+          r.stats)
 
 
 def t_ghcr_pages():