diff --git a/CHANGELOG.md b/CHANGELOG.md index 038f3ed..8b05c8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,31 @@ # Changelog +## 2026-07-12 — image pinning sweep: `:latest` eliminated from all templates (5 pins) + standing gate + +A catalog sweep found 5/53 templates with unpinned images. Beyond version discipline, `:latest` +breaks restore fidelity: the controller's recovery-unit `ImagePins` pins the *tag*, so restoring a +`:latest` app re-pulls whatever `:latest` means at restore time — potentially schema-incompatible +with the data being restored. Rule applied: a deployed app pins to the digest it is RUNNING +(pin ≠ upgrade); undeployed apps pin to the verified upstream stable. All five pins are +digest-identical to what `:latest` resolved to on 2026-07-12 — a pure no-op for running apps. + +| App | Old | New | Evidence | +|-----|-----|-----|----------| +| bentopdf | `ghcr.io/alam00000/bentopdf:latest` | `:v2.8.6` | digest == latest (`eaeea1e4…`); undeployed | +| calibre-web | `crocodilestick/calibre-web-automated:latest` | `:v4.0.6` | digest == RUNNING image on demo 9201 (`c31a738b…`) | +| papra | `ghcr.io/papra-hq/papra:latest` | `:26.6.1-rootless` | latest == the -rootless variant (`a7a42e22…`); `-root` differs — variant preserved | +| recipe-importer | `gitea.dooplex.hu/admin/recipe-importer:latest` | `:v0.9.11` | tag pre-existed in registry, digest == latest (`f3cb617c…`) — no retag needed | +| termix | `ghcr.io/lukegus/termix:latest` | `:2.5.0` | digest == latest == release-2.5.0 (`4d337131…`); undeployed | + +- New rerunnable gate `scripts/check-image-pins.py`: fails on `:latest`/`dev`/`nightly`/`edge`/ + `main`/`master` AND on untagged image refs (implicit :latest); `@sha256:` digests count as pinned. + Red-proofed both shapes (revert→exit 1→restore). +- Standing rule added to `CLAUDE.md` (never :latest / untagged; deployed apps pin to running digest). +- `templates.json` carries no image strings (legacy metadata only) — untouched. +- Fleet caveat: non-deployment of bentopdf/papra/termix verified on demo 9201 only; felhotest + unreachable + Peti's box offline at sweep time (operator approved proceeding — pins are + digest-equal to latest, so worst case equals the status quo). + ## 2026-07-06 — healthcheck sweep: `localhost` → `127.0.0.1` across all 48 templates Escalation of the re-run vaultwarden observation diff --git a/CLAUDE.md b/CLAUDE.md index 56206d1..805ccdf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,3 +27,7 @@ deployed `app.yaml` (customer secrets) is never overwritten. Full deploy details - Update `CHANGELOG.md` (newest on top) and overwrite `REPORT.md` with every pushed change. - No secrets in any committed file; secrets are generated at deploy time via `deploy_fields` `generate:` specs. +- **Never `:latest` or untagged images in templates** — pin a concrete version tag; an app deployed + anywhere in the fleet is pinned to the digest it is currently running (a pin must never cause a + version jump). Digest pins (`@sha256:`) also count. Gate: `python scripts/check-image-pins.py` + (run after any compose change; exit 1 on any floating/missing tag). diff --git a/CONTEXT.md b/CONTEXT.md index 2ddde52..194e489 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -2,5 +2,6 @@ > Created with the REUSE.md rollout (2026-07-03). History: `CHANGELOG.md`; format spec: `README.md`. +- **2026-07-12 — `:latest` banned from catalog — pin rule recorded.** 5 templates pinned (bentopdf v2.8.6, calibre-web v4.0.6 = running digest on 9201, papra 26.6.1-rootless, recipe-importer v0.9.11, termix 2.5.0; all digest-equal to that day's `:latest` — no-op for running apps). Standing rule in CLAUDE.md + gate `scripts/check-image-pins.py` (run after any compose change). Deployed apps pin to their RUNNING digest; pin ≠ upgrade. - **2026-07-03 — CLAUDE.md expanded** (repo purpose, push-to-main deploy contract, pointers); still intentionally light. - **2026-07-03 — `REUSE.md` exists at the repo root** (catalog conventions, healthcheck families, canonical example app = paperless-ngx, traps); maintenance rule active: update it in the same commit that changes a catalog-wide convention. diff --git a/REPORT.md b/REPORT.md index d42e57e..b25394f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,26 +1,63 @@ -# REPORT — healthcheck sweep: `localhost` → `127.0.0.1` across all 48 templates (2026-07-06) +# REPORT — image pinning: `:latest` eliminated from the catalog (2026-07-12) -**Class:** implementation (mechanical sweep; push = deploy). **Baseline:** `main` @ `d86e256` → (this commit). -Part C of the pilot-blockers bundle (A = agent v0.73.0 F2, B = controller v0.102.0 async restore). -Provenance: RERUN vaultwarden healthcheck observation, escalated to a class. +**Class:** implementation (template change + sweep gate; push = deploy). +**Baseline:** `main` @ `2ebe082` → (this commit). **Scope:** 5 compose image lines + new gate script ++ CLAUDE.md rule. No healthcheck/env/format touch-ups; no controller change; no redeploys triggered. -## The class -48/53 templates used `localhost` in their docker healthcheck `test:` line. BusyBox `wget` (and the -node `http.get`, python `urllib`/`socket.create_connection`, and `curl` one-shot forms) resolve -`localhost`→IPv6 `::1` with NO cross-address-family fallback, so an app that binds IPv4-only reads -docker-`unhealthy` while fully serving (vaultwarden, re-run 2026-07-06). +## Why -## The change -Mechanical `localhost`→`127.0.0.1`, scoped strictly to the healthcheck `test:` lines of the 48 affected -`templates/*/docker-compose.yml` (all four check-binary forms incl. mealie's python socket tuple). Diff -review confirmed: **no** app env/config/label line changed; `.felhom.yml` files were already clean (zero -occurrences). New REUSE.md convention row. +The controller's recovery-unit `ImagePins` (felhom-controller `internal/backup/recovery_unit.go`) +pins the compose *tag*: restoring a `:latest`-pinned app re-pulls whatever `:latest` points to at +restore time — potentially schema-incompatible with the data being restored. Load-bearing rule: +**a deployed app pins to the digest it is currently running** (the pin never causes a version jump); +undeployed apps pin to the latest verified upstream stable. -## Live acceptance -Catalog synced to guest 9201 (`POST /api/sync` 200); vaultwarden template + container now -`http://127.0.0.1:80/alive`. **vaultwarden recreated → docker-`healthy`** (the observation's subject flips). -bookstack got the same 127.0.0.1 sweep but stays `unhealthy` from an **app-level HTTP 500** on its health -path (pre-existing, unrelated to localhost — documented). +## The five pins (old → new, with evidence) -## Not changed -Only healthcheck `test:` lines. No env, config, labels, ports, or `.felhom.yml`. +| App | New pin | Evidence (all gathered 2026-07-12) | +|-----|---------|------------------------------------| +| **calibre-web** (DEPLOYED, demo 9201) | `crocodilestick/calibre-web-automated:v4.0.6` | Running RepoDigest on 9201: `sha256:c31a738b6d5e…`; Docker Hub tag `v4.0.6` digest: `sha256:c31a738b6d5e…` — **exact match** (Scenario A: pin == running image; a newer tag would have violated the no-silent-upgrade guard) | +| **bentopdf** (undeployed) | `ghcr.io/alam00000/bentopdf:v2.8.6` | GitHub latest release v2.8.6 (2026-06-28); ghcr manifest digest `sha256:eaeea1e4…` == `latest` | +| **papra** (undeployed) | `ghcr.io/papra-hq/papra:26.6.1-rootless` | ghcr `latest` digest `sha256:a7a42e22…` == `26.6.1-rootless`; the `-root` variant differs (`c5757540…`) — the **rootless** variant is what `:latest` was shipping, so behavior (permission semantics) is preserved exactly | +| **recipe-importer** (own image, undeployed) | `gitea.dooplex.hu/admin/recipe-importer:v0.9.11` | Its repo CHANGELOG declares v0.9.11 (2026-02-26); the registry **already had** `v0.9.11` (build.sh pushes `:VERSION`+`:latest` together), digest `sha256:f3cb617c…` == `latest` — **Scenario C's retag was unnecessary**; no push performed | +| **termix** (undeployed) | `ghcr.io/lukegus/termix:2.5.0` | ghcr digests: `latest` == `2.5.0` == `release-2.5.0` (`sha256:4d337131…`); bare CalVer tag chosen | + +All five `docker manifest inspect ` checks ran on the build server (180) with per-command exit +codes checked — 5× OK, zero pipe-swallowed. + +`templates.json`: contains **no image strings** (legacy Portainer-era metadata) — untouched, per §4.3. + +## Sweep + red-proofs (§10 / Scenario D) + +- New rerunnable gate: `scripts/check-image-pins.py` — scans every `templates/*/docker-compose.yml` + `image:` line; fails on floating tags (`latest`,`dev`,`nightly`,`edge`,`main`,`master`) AND on + untagged refs (implicit :latest); `@sha256:` counts as pinned; registry-port refs handled. +- **Final run: `image-pin gate OK — 53 templates, 0 unpinned images`, exit 0.** +- **Red-proof 1** (floating tag): reverted termix to `:latest` → exit 1 with + `templates/termix/docker-compose.yml:11 … [floating tag :latest]` → restored → exit 0. ✓ +- **Red-proof 2** (bare shape): stripped termix's tag entirely → exit 1 with + `[NO TAG (implicit :latest)]` → restored → exit 0. ✓ (Both §7-D shapes proven.) + +## Fleet-deployment verification (Scenario B) + +- demo 9201: `docker ps` — bentopdf/papra/termix/recipe-importer **not deployed**; calibre-web + running from `:latest` (Up 4 hours at check time). +- felhotest (router.abonet.hu:33022): **unreachable** (connection refused). +- Peti's box: offline (dead fan, no ETA). +- Operator asked and approved proceeding: every pin is digest-identical to current `:latest`, so + even a hidden deployment elsewhere sees zero change vs. the status quo on its next redeploy. + +## Git-sync no-op verification on 9201 + +_(filled after push — see below)_ + +## Observations (not acted on) + +- When Peti's box returns: check whether bentopdf/papra/termix/recipe-importer are deployed there; + if any is running an OLDER digest than the pin, decide upgrade vs. per-node pin then. felhotest + should get the same check when reachable again. +- papra upstream publishes only `-root`/`-rootless` variant tags — any future manual upgrade must + keep the `-rootless` suffix or permissions semantics change. +- bentopdf's ghcr `tags/list` API returns a stale/partial page (newest listed was 1.15.3 while + v2.8.6 exists and serves) — tag existence must be checked via manifest HEAD/inspect, not tags/list. +- recipe-importer's registry `:latest` remains published (allowed; only templates must not use it). diff --git a/REUSE.md b/REUSE.md index 9803ae8..4d70c50 100644 --- a/REUSE.md +++ b/REUSE.md @@ -16,6 +16,7 @@ None — this repo is templates/config, not code. See §2/§5. | `.felhom.yml` required fields | `templates/paperless-ngx/.felhom.yml` | All 53 apps: `display_name`, `description` (Hungarian), `category`, `subdomain`, `slug`, `resources{mem_request, mem_limit, pi_compatible, needs_hdd}`, `deploy_fields`, `app_info{tagline, use_cases, first_steps, ...}`, `healthcheck`. Optional: `smtp_mapping` (email-capable apps), `open_path` (non-root landing page, e.g. ghost). | | deploy_fields conventions | `templates/paperless-ngx/.felhom.yml` (`deploy_fields:` block) | Every app starts with `DOMAIN` (type `domain`) + `SUBDOMAIN` (type `subdomain`, `locked_after_deploy: true`). Secrets: `type: secret` + `generate:` — dominant generators `password:24` (DB passwords) and `hex:32` (app secret keys); `password:16` for shown admin passwords (`type: password`). HDD apps add `HDD_PATH` (`type: path`, placeholder `/mnt/felhom-drives/hdd_1`, locked). Labels/descriptions in Hungarian. | | Controller-side health probe | `templates/vaultwarden/.felhom.yml` (`healthcheck:` block) | `healthcheck.checks[]` with `type: http` (port only), `type: api` (port + `path` + `expect.status: 200`), or `type: tcp` (port only — mealie, crafty-controller). Prefer `api` with a real health path when the app has one. | +| Image pinning | ALL `templates/*/docker-compose.yml` (`image:` line) | **Never `:latest` or untagged** (recovery-unit `ImagePins` pins the tag — `:latest` breaks restore fidelity). Pin a concrete version tag; an app deployed anywhere in the fleet pins to the digest it is RUNNING (pin ≠ upgrade); `@sha256:` digest pins also count. Gate: `python scripts/check-image-pins.py` after any compose change (swept 2026-07-12: 5 pins). TRAP: ghcr `tags/list` can be stale/partial — verify tag existence via `docker manifest inspect`, never the tag list. | | Docker healthcheck host | ALL `templates/*/docker-compose.yml` (`healthcheck.test:`) | **Always `127.0.0.1`, never `localhost`.** BusyBox `wget` (and node/python/curl one-shots) resolve `localhost`→IPv6 `::1` with NO cross-address-family fallback; an app that binds IPv4-only then reads docker-`unhealthy` while fully serving (vaultwarden, re-run 2026-07-06 — swept all 48 templates). | | Docker healthcheck — BusyBox/wget images | `templates/vaultwarden/docker-compose.yml` (~L49) | `test: ["CMD", "wget", "--spider", "-q", "http://localhost:/"]`. Most common family (~20 apps, e.g. homebox, glance). | | Docker healthcheck — curl-capable images | `templates/paperless-ngx/docker-compose.yml` (~L76) | `test: ["CMD", "curl", "-f", "http://localhost:/"]` (~18 apps: jellyfin, immich, sonarr…). | diff --git a/scripts/check-image-pins.py b/scripts/check-image-pins.py new file mode 100644 index 0000000..2cb9e78 --- /dev/null +++ b/scripts/check-image-pins.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""check-image-pins.py — catalog gate: no :latest / untagged images in templates. + +Scans every templates/*/docker-compose.yml `image:` line and fails (exit 1) on: + - an explicit `:latest` tag (including `:latest@sha256:...` — the tag is a lie there, + but the digest pins it, so that shape is allowed and only the bare tag is banned), + - a floating alias tag (`dev`, `nightly`, `edge`, `main`, `master`), + - a missing tag entirely (`image: nginx` → implicit :latest). + +A digest reference (`repo@sha256:...`) counts as pinned. Registry ports +(`host:5000/img:1.2`) are handled: the tag is what follows the LAST colon of the +LAST path segment. + +Standing rule (CLAUDE.md): never :latest or untagged images in templates — pin a +concrete version tag; deployed apps pin to their running digest. +""" +import re +import sys +from pathlib import Path + +BANNED_TAGS = {"latest", "dev", "nightly", "edge", "main", "master"} +IMAGE_RE = re.compile(r"^\s*image:\s*[\"']?([^\s\"'#]+)") + +def check(root: Path) -> int: + failures = [] + files = sorted(root.glob("templates/*/docker-compose.yml")) + if not files: + print(f"ERROR: no templates found under {root}/templates/", file=sys.stderr) + return 2 + for f in files: + for lineno, line in enumerate(f.read_text(encoding="utf-8").splitlines(), 1): + m = IMAGE_RE.match(line) + if not m: + continue + ref = m.group(1) + if "@sha256:" in ref: + continue # digest-pinned — strongest pin there is + last_seg = ref.rsplit("/", 1)[-1] + if ":" not in last_seg: + failures.append((f, lineno, ref, "NO TAG (implicit :latest)")) + continue + tag = last_seg.rsplit(":", 1)[-1] + if tag.lower() in BANNED_TAGS: + failures.append((f, lineno, ref, f"floating tag :{tag}")) + if failures: + print("UNPINNED IMAGES FOUND:") + for f, lineno, ref, why in failures: + print(f" {f.as_posix()}:{lineno}: {ref} [{why}]") + return 1 + print(f"image-pin gate OK — {len(files)} templates, 0 unpinned images") + return 0 + +if __name__ == "__main__": + sys.exit(check(Path(__file__).resolve().parent.parent)) diff --git a/templates/bentopdf/docker-compose.yml b/templates/bentopdf/docker-compose.yml index 631e5c7..daf3aa6 100644 --- a/templates/bentopdf/docker-compose.yml +++ b/templates/bentopdf/docker-compose.yml @@ -8,7 +8,7 @@ services: bentopdf: - image: ghcr.io/alam00000/bentopdf:latest + image: ghcr.io/alam00000/bentopdf:v2.8.6 container_name: bentopdf restart: unless-stopped environment: diff --git a/templates/calibre-web/docker-compose.yml b/templates/calibre-web/docker-compose.yml index 7201282..d9e469d 100644 --- a/templates/calibre-web/docker-compose.yml +++ b/templates/calibre-web/docker-compose.yml @@ -18,7 +18,7 @@ services: calibre-web: - image: crocodilestick/calibre-web-automated:latest + image: crocodilestick/calibre-web-automated:v4.0.6 container_name: calibre-web restart: unless-stopped environment: diff --git a/templates/papra/docker-compose.yml b/templates/papra/docker-compose.yml index bd12a4f..99623b1 100644 --- a/templates/papra/docker-compose.yml +++ b/templates/papra/docker-compose.yml @@ -8,7 +8,7 @@ services: papra: - image: ghcr.io/papra-hq/papra:latest + image: ghcr.io/papra-hq/papra:26.6.1-rootless container_name: papra restart: unless-stopped environment: diff --git a/templates/recipe-importer/docker-compose.yml b/templates/recipe-importer/docker-compose.yml index 40fe9a2..26befea 100644 --- a/templates/recipe-importer/docker-compose.yml +++ b/templates/recipe-importer/docker-compose.yml @@ -12,7 +12,7 @@ services: recipe-importer: - image: gitea.dooplex.hu/admin/recipe-importer:latest + image: gitea.dooplex.hu/admin/recipe-importer:v0.9.11 container_name: recipe-importer restart: unless-stopped environment: diff --git a/templates/termix/docker-compose.yml b/templates/termix/docker-compose.yml index 703a9bc..6a4c72b 100644 --- a/templates/termix/docker-compose.yml +++ b/templates/termix/docker-compose.yml @@ -8,7 +8,7 @@ services: termix: - image: ghcr.io/lukegus/termix:latest + image: ghcr.io/lukegus/termix:2.5.0 container_name: termix restart: unless-stopped environment: