release-copy: the released images (skopeo dir:, digest-true) and agent packages copied to ep0 nightly, verified by sha256 and a checksum compare, capped at 3 GB (D33, R-1227); tests red-proved

scripts/CHANGELOG.md

@@ -1,3 +1,8 @@
+## 2026-10-23 — `release-copy/`: the released images and agent packages leave DooPlex nightly (D33, R-1227); `hub-db-backup/`: every 6 hours (D34)
+
+- **`release-copy/felhom-release-copy`** (new, DooPlex root, timer 01:40): the newest 3 release tags of felhom-controller, felhom-hub and felhom-samba (`skopeo copy --all --preserve-digests` into `dir:` — the registry's own bytes, so a push back gives the SAME digest; a `docker save` tar would not) and the newest 5 agent versions (binary + config bundle). Before anything leaves: the top manifest hashes to the registry's digest, every blob and child manifest to its name, every named blob is present, each agent file matches the registry's sha256; the staged tree ≤ the cap (3072 MB). Then `rsync --delete` to ep0's limited user `felhom-relcopy` (rrsync, one directory), and a `--checksum --dry-run` must show no difference before the success signal. Spoiled tags (0.315.0, 0.316.3) are never copied. 13 tests (`test_release_copy.py`), four red-proved.
+- **`hub-db-backup/`** — the push runs at 02:30, 08:30, 14:30 and 20:30 (the hub snapshots every 6 hours since hub 0.167.0) and refuses a newest snapshot older than 2 h (was 26 h). Tests: the timer read against the hub's slots; a missed slot is refused.
+
 ## 2026-10-23 — felhom-host-install.sh 1.36.0 (pushed; the tag `installer-v1.36.0` waits for the operator's yes): the uninstall removes the root-SSH drop-in (R-982, D4)
 
 The agent's config bundle (next agent release) gives the host's own sshd `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf`

scripts/release-copy/felhom-release-copy

@@ -0,0 +1,300 @@
+#!/usr/bin/env python3
+"""felhom-release-copy — copy the released images and agent packages off DooPlex to ep0, nightly (D33, R-1227).
+
+Runs on DooPlex as root from felhom-release-copy.timer. Runbook: documentation/runbooks/total-loss-of-dooplex.md
+(„Restore the release copy"). Pinned by test_release_copy.py.
+
+WHAT. The newest KEEP release tags (X.Y.Z only) of each image in REPOS, and the newest AGENT_KEEP agent versions (the
+binary and its config bundle). After a DooPlex loss these are what every box update, reinstall and new install pulls.
+
+HOW, and why skopeo, not `docker save`. Each image tag is copied with `skopeo copy --all --preserve-digests` into
+skopeo's `dir:` layout: the registry's own manifest bytes and compressed layer blobs, unchanged. Pushed back into any
+registry, the image has THE SAME DIGEST as the release (a `docker save` tar is re-compressed on the next push, so the
+digest — what the „a version is pushed once" rule and the hub's spoiled-tag list are about — would differ). It also
+needs no Docker daemon on DooPlex.
+
+CHECKS, before anything leaves: the sha256 of the copied top manifest equals the registry's digest; every blob and
+child manifest hashes to its own name; every blob a manifest names is present; each agent file's sha256 equals the
+registry's record. The staged tree must be ≤ CAP_MB (the cap — the sender enforces it; ep0 has no quota). Then
+`rsync --delete` to ep0 (the limited user `felhom-relcopy`, rrsync-confined to one directory), and a second
+`rsync --checksum --dry-run` must report NO difference. Only then is the success signal written (CLAUDE.md „presence
+is not success"). Versions beyond KEEP / AGENT_KEEP are deleted locally, and so on ep0 by `--delete`.
+"""
+import base64
+import hashlib
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import urllib.request
+
+CONF = os.environ.get("FELHOM_RC_CONF", "/etc/felhom-release-copy")
+STATE = os.environ.get("FELHOM_RC_STATE", "/var/lib/felhom-release-copy")
+TEXTFILE_DIR = os.environ.get("FELHOM_RC_TEXTFILE_DIR", "/var/lib/node_exporter/textfile_collector")
+REGISTRY = "gitea.dooplex.hu"
+OWNER = "admin"
+SPOILED = {"felhom-controller": {"0.315.0", "0.316.3"}}  # withdrawn builds (R-1085) — never offered, never copied
+AGENT_FILES = ("felhom-agent", "felhom-config-bundle.json")
+HEX64 = re.compile(r"^[0-9a-f]{64}$")
+RELEASE = re.compile(r"^\d+\.\d+\.\d+$")
+
+
+class Refused(Exception):
+    pass
+
+
+def log(msg):
+    print("felhom-release-copy: " + msg, flush=True)
+
+
+def load_env(path):
+    env = {}
+    for line in open(path, encoding="utf-8"):
+        line = line.strip()
+        if line and not line.startswith("#") and "=" in line:
+            k, v = line.split("=", 1)
+            env[k.strip()] = v.strip().strip("'\"")
+    return env
+
+
+def newest(versions, n, spoiled=()):
+    rel = [v for v in versions if RELEASE.match(v) and v not in spoiled]
+    rel.sort(key=lambda v: tuple(int(x) for x in v.split(".")))
+    return rel[-n:] if n > 0 else []
+
+
+def sha256_file(path):
+    h = hashlib.sha256()
+    with open(path, "rb") as f:
+        for chunk in iter(lambda: f.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def verify_image_dir(d, want_digest):
+    """Refuses unless the dir: copy in d is exactly the image want_digest names (see the module docstring)."""
+    top = os.path.join(d, "manifest.json")
+    if not os.path.isfile(top):
+        raise Refused(f"{d}: no manifest.json")
+    got = "sha256:" + sha256_file(top)
+    if got != want_digest:
+        raise Refused(f"{d}: manifest digest {got} != registry {want_digest}")
+    names = set(os.listdir(d))
+    manifests = [top]
+    for n in sorted(names):
+        p = os.path.join(d, n)
+        if HEX64.match(n):
+            if sha256_file(p) != n:
+                raise Refused(f"{d}: blob {n} does not hash to its name")
+        elif n.endswith(".manifest.json") and HEX64.match(n[: -len(".manifest.json")]):
+            if sha256_file(p) != n[: -len(".manifest.json")]:
+                raise Refused(f"{d}: child manifest {n} does not hash to its name")
+            manifests.append(p)
+    for m in manifests:
+        doc = json.load(open(m, encoding="utf-8"))
+        for ref in doc.get("manifests", []):
+            hx = ref["digest"].split(":", 1)[1]
+            if hx + ".manifest.json" not in names:
+                raise Refused(f"{d}: child manifest {hx[:12]} named by the index is missing")
+        for ref in ([doc["config"]] if "config" in doc else []) + doc.get("layers", []):
+            hx = ref["digest"].split(":", 1)[1]
+            if hx not in names:
+                raise Refused(f"{d}: blob {hx[:12]} named by a manifest is missing")
+
+
+def tree_bytes(root):
+    total = 0
+    for dp, _, fs in os.walk(root):
+        for f in fs:
+            total += os.lstat(os.path.join(dp, f)).st_size
+    return total
+
+
+def prune(root, keep):
+    """Removes every directory under root whose name is not in keep; returns the removed names."""
+    gone = []
+    if os.path.isdir(root):
+        for n in sorted(os.listdir(root)):
+            if n not in keep:
+                shutil.rmtree(os.path.join(root, n))
+                gone.append(n)
+    return gone
+
+
+class Registry:
+    def __init__(self, user, token):
+        self.auth = "Basic " + base64.b64encode(f"{user}:{token}".encode()).decode()
+
+    def get(self, url, raw=False):
+        r = urllib.request.urlopen(urllib.request.Request(url, headers={"Authorization": self.auth}), timeout=120)
+        return r.read() if raw else json.load(r)
+
+    def agent_versions(self):
+        out, page = [], 1
+        while True:
+            batch = self.get(f"https://{REGISTRY}/api/v1/packages/{OWNER}?type=generic&q=felhom-agent&limit=50&page={page}")
+            if not batch:
+                return out
+            out += [p["version"] for p in batch if p.get("name") == "felhom-agent"]
+            page += 1
+
+    def agent_files(self, ver):
+        return {f["name"]: f["sha256"] for f in self.get(
+            f"https://{REGISTRY}/api/v1/packages/{OWNER}/generic/felhom-agent/{ver}/files")}
+
+    def agent_download(self, ver, name, dest):
+        with open(dest, "wb") as f:
+            f.write(self.get(f"https://{REGISTRY}/api/packages/{OWNER}/generic/felhom-agent/{ver}/{name}", raw=True))
+
+
+def run(cmd, **kw):
+    return subprocess.run(cmd, check=True, text=True, capture_output=True, **kw)
+
+
+def skopeo(authfile, *args):
+    return run(["skopeo", *args[:1], "--authfile", authfile, *args[1:]]).stdout
+
+
+def copy_images(reg_authfile, stage, repos, keep):
+    images = os.path.join(stage, "images")
+    os.makedirs(images, exist_ok=True)
+    copied = []
+    for repo in repos:
+        tags = json.loads(skopeo(reg_authfile, "list-tags", f"docker://{REGISTRY}/{OWNER}/{repo}"))["Tags"]
+        want = newest(tags, keep, SPOILED.get(repo, ()))
+        if not want:
+            raise Refused(f"{repo}: no release tag")
+        rdir = os.path.join(images, repo)
+        os.makedirs(rdir, exist_ok=True)
+        for tag in want:
+            ref = f"docker://{REGISTRY}/{OWNER}/{repo}:{tag}"
+            digest = "sha256:" + hashlib.sha256(
+                skopeo(reg_authfile, "inspect", "--raw", ref).encode()).hexdigest()
+            d = os.path.join(rdir, tag)
+            if os.path.isfile(os.path.join(d, "DIGEST")) and open(os.path.join(d, "DIGEST")).read().strip() == digest:
+                verify_image_dir(d, digest)  # already here — re-checked every night, not trusted
+                continue
+            tmp = d + ".tmp"
+            shutil.rmtree(tmp, ignore_errors=True)
+            shutil.rmtree(d, ignore_errors=True)
+            skopeo(reg_authfile, "copy", "--all", "--preserve-digests", "--quiet", ref, f"dir:{tmp}")
+            verify_image_dir(tmp, digest)
+            with open(os.path.join(tmp, "DIGEST"), "w") as f:
+                f.write(digest + "\n")
+            os.rename(tmp, d)
+            copied.append(f"{repo}:{tag}")
+        gone = prune(rdir, set(want))
+        log(f"{repo}: keeping {', '.join(want)}" + (f"; removed {', '.join(gone)}" if gone else ""))
+    prune(images, set(repos))
+    return copied
+
+
+def copy_agents(reg, stage, keep):
+    adir = os.path.join(stage, "agent")
+    os.makedirs(adir, exist_ok=True)
+    want = newest(reg.agent_versions(), keep)
+    if not want:
+        raise Refused("felhom-agent: no release version")
+    for ver in want:
+        d = os.path.join(adir, ver)
+        os.makedirs(d, exist_ok=True)
+        recorded = reg.agent_files(ver)
+        for name in AGENT_FILES:
+            if name not in recorded:
+                raise Refused(f"felhom-agent {ver}: the registry has no {name}")
+            p = os.path.join(d, name)
+            if not (os.path.isfile(p) and sha256_file(p) == recorded[name]):
+                reg.agent_download(ver, name, p + ".tmp")
+                if sha256_file(p + ".tmp") != recorded[name]:
+                    os.remove(p + ".tmp")
+                    raise Refused(f"felhom-agent {ver}/{name}: the download does not match the registry's sha256")
+                os.rename(p + ".tmp", p)
+            with open(os.path.join(d, name + ".sha256"), "w") as f:
+                f.write(f"{recorded[name]}  {name}\n")
+    gone = prune(adir, set(want))
+    log(f"felhom-agent: keeping {', '.join(want)}" + (f"; removed {', '.join(gone)}" if gone else ""))
+
+
+def write_sums(stage):
+    lines = []
+    for dp, _, fs in os.walk(stage):
+        for f in fs:
+            if f == "SHA256SUMS":
+                continue
+            p = os.path.join(dp, f)
+            lines.append(f"{sha256_file(p)}  {os.path.relpath(p, stage)}")
+    with open(os.path.join(stage, "SHA256SUMS"), "w") as f:
+        f.write("\n".join(sorted(lines, key=lambda l: l[66:])) + "\n")
+
+
+def rsync_to(env, stage, dry_check=False):
+    ssh = (f"ssh -i {CONF}/id_ed25519 -o UserKnownHostsFile={CONF}/known_hosts -o StrictHostKeyChecking=yes "
+           f"-o BatchMode=yes -o ConnectTimeout=30")
+    cmd = ["rsync", "-a", "--delete", "-e", ssh]
+    if dry_check:
+        cmd += ["--checksum", "--dry-run", "--itemize-changes"]
+    return run(cmd + [stage.rstrip("/") + "/", env["REMOTE"] + ":./"]).stdout
+
+
+def signal(nbytes):
+    tmp = os.path.join(TEXTFILE_DIR, f"felhom_release_copy.prom.{os.getpid()}")
+    with open(tmp, "w") as f:
+        f.write("# HELP felhom_release_copy_last_success_timestamp_seconds Last verified copy of the releases to ep0 (D33).\n"
+                "# TYPE felhom_release_copy_last_success_timestamp_seconds gauge\n"
+                f"felhom_release_copy_last_success_timestamp_seconds {int(time.time())}\n"
+                f"felhom_release_copy_last_success_bytes {nbytes}\n")
+    os.chmod(tmp, 0o644)
+    os.rename(tmp, os.path.join(TEXTFILE_DIR, "felhom_release_copy.prom"))
+
+
+def main():
+    env = load_env(os.path.join(CONF, "env"))
+    keep, akeep, cap = int(env.get("KEEP", "3")), int(env.get("AGENT_KEEP", "5")), int(env["CAP_MB"])
+    repos = env["REPOS"].split()
+    token = open(os.path.join(CONF, "registry-token"), encoding="utf-8").read().strip()
+    user = env.get("REGISTRY_USER", OWNER)
+    stage = os.path.join(STATE, "stage")
+    os.makedirs(stage, exist_ok=True)
+    os.chmod(STATE, 0o700)
+    with tempfile.TemporaryDirectory(dir=STATE) as td:
+        authfile = os.path.join(td, "auth.json")
+        fd = os.open(authfile, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
+        with os.fdopen(fd, "w") as f:
+            json.dump({"auths": {REGISTRY: {"auth": base64.b64encode(f"{user}:{token}".encode()).decode()}}}, f)
+        start = time.time()
+        copied = copy_images(authfile, stage, repos, keep)
+    copy_agents(Registry(user, token), stage, akeep)
+    for n in os.listdir(stage):
+        if n not in ("images", "agent", "SHA256SUMS", "COPIED_AT"):
+            p = os.path.join(stage, n)
+            shutil.rmtree(p) if os.path.isdir(p) else os.remove(p)
+    with open(os.path.join(stage, "COPIED_AT"), "w") as f:
+        f.write(time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + "\n")
+    write_sums(stage)
+    nbytes = tree_bytes(stage)
+    if nbytes > cap * 1024 * 1024:
+        raise Refused(f"the copy is {nbytes // (1024 * 1024)} MB, above the cap {cap} MB — nothing sent")
+    log(f"staged and verified: {nbytes // (1024 * 1024)} MB, new this run: {', '.join(copied) or 'none'}")
+    rsync_to(env, stage)
+    diff = rsync_to(env, stage, dry_check=True).strip()
+    if diff:
+        raise Refused("ep0's copy differs from the staged one after the push: " + diff.splitlines()[0])
+    log(f"pushed to ep0 and compared by checksum: no difference ({int(time.time() - start)} s)")
+    signal(nbytes)
+    log("success signal written")
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except Refused as e:
+        print(f"felhom-release-copy: FAILED: {e}", file=sys.stderr)
+        sys.exit(1)
+    except subprocess.CalledProcessError as e:
+        print(f"felhom-release-copy: FAILED: {e.cmd[0]} {e.cmd[1] if len(e.cmd) > 1 else ''} rc={e.returncode}: "
+              f"{(e.stderr or '').strip()[:300]}", file=sys.stderr)
+        sys.exit(1)

scripts/release-copy/felhom-release-copy.service

@@ -0,0 +1,19 @@
+# Versioned in felhom.eu/scripts/release-copy/ (D33, R-1227); installed by install.sh. Runbook: total-loss-of-dooplex.md.
+[Unit]
+Description=Felhom: copy the released images and agent packages to ep0 (D33)
+Wants=network-online.target
+After=network-online.target
+OnFailure=felhom-backup-failmail@%n.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/felhom-release-copy
+UMask=0077
+TimeoutStartSec=2h
+Nice=10
+IOSchedulingClass=idle
+PrivateTmp=yes
+ProtectSystem=strict
+ProtectHome=yes
+ReadWritePaths=/var/lib/felhom-release-copy /var/lib/node_exporter/textfile_collector
+NoNewPrivileges=yes

scripts/release-copy/felhom-release-copy.timer

@@ -0,0 +1,12 @@
+# Versioned in felhom.eu/scripts/release-copy/ (D33, R-1227); installed by install.sh.
+[Unit]
+Description=Felhom: copy the released images and agent packages to ep0 (D33) — nightly
+
+[Timer]
+# 01:40: after the 00:20 off-site push, before the 02:30 hub DB push and the 03:00 local backup.
+OnCalendar=*-*-* 01:40:00
+Persistent=true
+RandomizedDelaySec=5min
+
+[Install]
+WantedBy=timers.target

scripts/release-copy/install.sh

@@ -0,0 +1,30 @@
+#!/bin/sh
+# install.sh — install the release copy job on DooPlex (D33, R-1227). Root. Idempotent.
+# Installs the script and two units, writes /etc/felhom-release-copy/env (no secrets) when absent, and does NOT enable
+# the timer — enable it by hand after the first manual run:  systemctl enable --now felhom-release-copy.timer
+# The registry token (registry-token), the SSH key (id_ed25519) and ep0's host key (known_hosts) are created
+# separately, file to file, never by this script (runbook total-loss-of-dooplex.md, „The release copy").
+set -eu
+HERE=$(cd "$(dirname "$0")" && pwd)
+[ "$(id -u)" = 0 ] || { echo "install.sh: run as root" >&2; exit 1; }
+install -m 0755 "$HERE/felhom-release-copy" /usr/local/sbin/felhom-release-copy
+for u in felhom-release-copy.service felhom-release-copy.timer; do
+  install -m 0644 "$HERE/$u" "/etc/systemd/system/$u"
+done
+install -d -m 0700 /etc/felhom-release-copy /var/lib/felhom-release-copy
+if [ ! -f /etc/felhom-release-copy/env ]; then
+  umask 077
+  cat > /etc/felhom-release-copy/env <<'ENV'
+# Not secret. The token is in registry-token (0600), the SSH key in id_ed25519 (0600).
+REMOTE='felhom-relcopy@167.233.158.164'
+REGISTRY_USER='admin'
+# the newest KEEP release tags of each image; the newest AGENT_KEEP agent versions (the vouched one can lag the newest)
+REPOS='felhom-controller felhom-hub felhom-samba'
+KEEP=3
+AGENT_KEEP=5
+# the cap: the sender refuses to send a bigger copy (ep0's / has ~14 GB free; ep0 has no quota)
+CAP_MB=3072
+ENV
+fi
+systemctl daemon-reload
+echo "install.sh: installed; timer NOT enabled (see the header)"

scripts/release-copy/test_release_copy.py

@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+"""Tests for felhom-release-copy (D33). No network, no registry, no ep0: a synthetic skopeo `dir:` copy, and the
+end-to-end refusals through main() with the copy steps replaced. Run: python3 test_release_copy.py"""
+import hashlib
+import importlib.machinery
+import importlib.util
+import json
+import os
+import tempfile
+import unittest
+from unittest import mock
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+_loader = importlib.machinery.SourceFileLoader("rc", os.path.join(HERE, "felhom-release-copy"))
+_spec = importlib.util.spec_from_loader("rc", _loader)
+rc = importlib.util.module_from_spec(_spec)
+_loader.exec_module(rc)
+
+
+def h(b):
+    return hashlib.sha256(b).hexdigest()
+
+
+def make_image(d):
+    """A two-level image like gitea's: an OCI index → one platform manifest → config + one layer. Returns the digest."""
+    os.makedirs(d)
+    layer, config = b"layer-bytes", b'{"config":1}'
+    child = json.dumps({"schemaVersion": 2, "config": {"digest": "sha256:" + h(config)},
+                        "layers": [{"digest": "sha256:" + h(layer)}]}).encode()
+    index = json.dumps({"schemaVersion": 2, "manifests": [{"digest": "sha256:" + h(child)}]}).encode()
+    for name, data in ((h(layer), layer), (h(config), config), (h(child) + ".manifest.json", child),
+                       ("manifest.json", index), ("version", b"Directory Transport Version: 1.1\n")):
+        open(os.path.join(d, name), "wb").write(data)
+    return "sha256:" + h(index)
+
+
+class Verify(unittest.TestCase):
+    def setUp(self):
+        self.t = tempfile.mkdtemp()
+        self.d = os.path.join(self.t, "img")
+        self.digest = make_image(self.d)
+
+    def test_a_true_copy_passes(self):
+        rc.verify_image_dir(self.d, self.digest)
+
+    def test_a_changed_layer_is_refused(self):
+        # RED-PROOF: drop the per-blob hash check in verify_image_dir → this passes silently
+        blob = [n for n in os.listdir(self.d) if rc.HEX64.match(n)][0]
+        with open(os.path.join(self.d, blob), "ab") as f:
+            f.write(b"x")
+        with self.assertRaisesRegex(rc.Refused, "does not hash to its name"):
+            rc.verify_image_dir(self.d, self.digest)
+
+    def test_another_image_is_refused(self):
+        with self.assertRaisesRegex(rc.Refused, "!= registry"):
+            rc.verify_image_dir(self.d, "sha256:" + "0" * 64)
+
+    def test_a_missing_layer_is_refused(self):
+        # RED-PROOF: drop the „named by a manifest is missing" check → a copy without its layer passes
+        layer = [n for n in os.listdir(self.d) if rc.HEX64.match(n)
+                 and open(os.path.join(self.d, n), "rb").read() == b"layer-bytes"][0]
+        os.remove(os.path.join(self.d, layer))
+        with self.assertRaisesRegex(rc.Refused, "missing"):
+            rc.verify_image_dir(self.d, self.digest)
+
+    def test_a_missing_child_manifest_is_refused(self):
+        child = [n for n in os.listdir(self.d) if n.endswith(".manifest.json")][0]
+        os.remove(os.path.join(self.d, child))
+        with self.assertRaisesRegex(rc.Refused, "child manifest .* missing"):
+            rc.verify_image_dir(self.d, self.digest)
+
+
+class Select(unittest.TestCase):
+    def test_newest_releases_by_number_not_text(self):
+        tags = ["0.9.0", "0.10.0", "0.340.0", "0.339.0", "0.340.0-rc.8", "latest", "0.99.1"]
+        self.assertEqual(rc.newest(tags, 3), ["0.99.1", "0.339.0", "0.340.0"])
+
+    def test_spoiled_tags_are_never_copied(self):
+        self.assertEqual(rc.newest(["0.314.0", "0.315.0", "0.316.3"], 3, rc.SPOILED["felhom-controller"]), ["0.314.0"])
+
+    def test_prune_deletes_beyond_keep(self):
+        t = tempfile.mkdtemp()
+        for v in ("0.1.0", "0.2.0", "0.3.0"):
+            os.makedirs(os.path.join(t, v))
+        self.assertEqual(rc.prune(t, {"0.3.0"}), ["0.1.0", "0.2.0"])
+        self.assertEqual(os.listdir(t), ["0.3.0"])
+
+
+class Main(unittest.TestCase):
+    """main() with the registry copies replaced: the cap and the post-push comparison gate the success signal."""
+
+    def setUp(self):
+        self.t = tempfile.mkdtemp()
+        self.conf, self.state, self.tf = (os.path.join(self.t, n) for n in ("conf", "state", "textfile"))
+        for d in (self.conf, self.state, self.tf):
+            os.makedirs(d)
+        open(os.path.join(self.conf, "env"), "w").write(
+            "REMOTE=felhom-relcopy@ep0\nKEEP=3\nAGENT_KEEP=5\nCAP_MB=1\nREPOS='felhom-hub'\n")
+        open(os.path.join(self.conf, "registry-token"), "w").write("not-a-token\n")
+        self.patches = [mock.patch.object(rc, "CONF", self.conf), mock.patch.object(rc, "STATE", self.state),
+                        mock.patch.object(rc, "TEXTFILE_DIR", self.tf)]
+        for p in self.patches:
+            p.start()
+        self.addCleanup(lambda: [p.stop() for p in self.patches])
+        self.pushed = []
+
+    def fake_images(self, size):
+        def f(authfile, stage, repos, keep):
+            d = os.path.join(stage, "images", "felhom-hub")
+            os.makedirs(d, exist_ok=True)
+            open(os.path.join(d, "blob"), "wb").write(b"x" * size)
+            return ["felhom-hub:0.167.0"]
+        return f
+
+    def fake_rsync(self, diff=""):
+        def f(env, stage, dry_check=False):
+            self.pushed.append(dry_check)
+            return diff if dry_check else ""
+        return f
+
+    def go(self, size, diff=""):
+        with mock.patch.object(rc, "copy_images", self.fake_images(size)), \
+                mock.patch.object(rc, "copy_agents", lambda reg, stage, keep: None), \
+                mock.patch.object(rc, "rsync_to", self.fake_rsync(diff)):
+            rc.main()
+
+    def signal(self):
+        return os.path.exists(os.path.join(self.tf, "felhom_release_copy.prom"))
+
+    def test_success_pushes_compares_then_signals(self):
+        self.go(1000)
+        self.assertEqual(self.pushed, [False, True])
+        self.assertTrue(self.signal())
+        sums = open(os.path.join(self.state, "stage", "SHA256SUMS")).read()
+        self.assertIn("images/felhom-hub/blob", sums)
+
+    def test_above_the_cap_nothing_is_sent(self):
+        # RED-PROOF: drop the cap check in main() → the push happens and the signal is written
+        with self.assertRaisesRegex(rc.Refused, "above the cap"):
+            self.go(2 * 1024 * 1024)
+        self.assertEqual(self.pushed, [])
+        self.assertFalse(self.signal())
+
+    def test_a_difference_after_the_push_writes_no_signal(self):
+        # RED-PROOF: ignore the dry-run's output → a copy ep0 does not hold reads as success
+        with self.assertRaisesRegex(rc.Refused, "differs"):
+            self.go(1000, diff=">fcsT...... images/felhom-hub/blob\n")
+        self.assertFalse(self.signal())
+
+
+class Units(unittest.TestCase):
+    def read(self, n):
+        return open(os.path.join(HERE, n)).read()
+
+    def test_unit_runs_the_installed_script_and_mails_on_failure(self):
+        s = self.read("felhom-release-copy.service")
+        self.assertIn("ExecStart=/usr/local/sbin/felhom-release-copy\n", s)
+        self.assertIn("OnFailure=felhom-backup-failmail@%n.service", s)
+
+    def test_nightly(self):
+        self.assertIn("OnCalendar=*-*-* 01:40:00", self.read("felhom-release-copy.timer"))
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=2)