CHANGELOG + REPORT: campaign 7 full catalog sweep (53/53, 45 pass, 13 fixes)

Campaign 7 catalog sweep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE

CHANGELOG.md

@@ -1,5 +1,45 @@
 # Changelog
 
+## 2026-07-19 — CAMPAIGN 7: full catalog sweep (53/53 apps deployed + validated on the demo box)
+
+Every app in the catalog was bumped to its newest stable upstream tag where one existed, then
+**actually deployed** through the controller's real endpoints on the demo box (controller 0.146.0),
+validated (all containers healthy, HTTP through the real Traefik ingress, log scan), and removed
+again through the real delete flow. Full evidence + result matrix:
+`felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md`.
+
+**Result: 45 apps pass end-to-end, 4 do not, 1 is not automatable (plex needs a real PLEX_CLAIM).**
+
+**Version bumps** — ~40 templates moved to current upstream, 15 of them across a major
+(bookstack 25.02→26.05, immich v2→v3, calcom v4→v6, nextcloud 31→34, grafana 11→13, n8n 1→2,
+outline 0.82→1.9, vikunja 0.24→2.3, tandoor 1→2, romm 4→5, radarr 5→6, privatebin 1→2,
+onlyoffice 8→9, claper 1→2, gramps-web v24→v25). `uptime-kuma` moved off the floating `:2` tag
+to `2.4.0`. **DB/cache sidecar majors were deliberately NOT bumped** — rationale in the campaign
+doc §4 (a DB major is the application's decision, and `postgres:16-alpine` already tracks 16.x).
+
+**13 template fixes, every one live-re-validated:**
+
+- **7 broken healthchecks.** This is not cosmetic: Traefik will not route to an `unhealthy`
+  container, so a probe that cannot run makes the app return **404 to the customer while it serves
+  200 on its own port**. adventurelog (wget in a distroless image → Node-exec at an absolute path),
+  emby (curl absent, BusyBox only), papra + wishlist (node-only images), homebox (`--spider` sends
+  HEAD, endpoint answers 405 to HEAD / 200 to GET), zipline (v4 renamed `/api/health` →
+  `/api/healthcheck`), tandoor (`start_period` too short for gunicorn).
+- **5 apps that had NEVER been deployable** and were fixed: papra (missing required `AUTH_SECRET`,
+  now a generated `data_key` secret), zipline (v4 `CORE_DATABASE_URL` → `DATABASE_URL`), wishlist
+  (dead Docker Hub image → followed upstream to `ghcr.io/cmintey/wishlist:v0.66.0`), homebox
+  (upstream dropped the `v` tag prefix + new required `HBOX_AUTH_API_KEY_PEPPER`), wger (2.6 needs
+  the full `DJANGO_DB_*` set and listens on :8000, not :80 — the Traefik port was wrong too).
+- **4 memory/OOM corrections proven by a live OOM:** gramps-web 384M→1024M, n8n 512M→1536M
+  (V8 heap), rallly 256M→768M, tandoor 512M→1024M (+ its `mem_limit` sum was already wrong).
+- **gokapi reverted v2.2.4 → v1.9.6**: v2 refuses to run against the seeded ConfigVersion-21
+  config and demands an intermediate v2.0.0 pass, even on a fresh deploy. Shipping it would have
+  broken every new gokapi deploy. Needs a dedicated v2 config-migration task.
+
+**Still failing (recorded, not fixed):** `glance` (needs a seeded `glance.yml`; PROVEN pre-existing —
+the pre-campaign v0.7.4 pin fails identically), `gokapi` (above), `plant-it` and `wanderer`
+(their images do not resolve at all — neither the new tag nor the one the catalog already shipped).
+
 ## 2026-07-14 — backup classification `backup:` blocks for the 13 bind-bearing apps (controller v0.132.0)
 
 Adds the referential-coupling `backup:` classification block to every catalog app that binds

REPORT.md

@@ -1,71 +1,68 @@
-# REPORT — image pinning: `:latest` eliminated from the catalog (2026-07-12)
+# REPORT — CAMPAIGN 7: full app-catalog sweep (2026-07-18 → 2026-07-19)
 
-**Class:** implementation (template change + sweep gate; push = deploy).
-**Baseline:** `main` @ `2ebe082` → (this commit). **Scope:** 5 compose image lines + new gate script
-+ CLAUDE.md rule. No healthcheck/env/format touch-ups; no controller change; no redeploys triggered.
+**Scope:** every app under `templates/` — bump the pin, deploy it through the real pipeline on the
+demo box, prove it works, audit its healthcheck, record, remove.
+**Box:** demo guest 9201 (`demo-felhom` on `felhom-pve`), controller **0.146.0**.
+**Full matrix + ranked findings:**
+`felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md`
 
-## Why
+## Outcome
 
-The controller's recovery-unit `ImagePins` (felhom-controller `internal/backup/recovery_unit.go`)
-pins the compose *tag*: restoring a `:latest`-pinned app re-pulls whatever `:latest` points to at
-restore time — potentially schema-incompatible with the data being restored. Load-bearing rule:
-**a deployed app pins to the digest it is currently running** (the pin never causes a version jump);
-undeployed apps pin to the latest verified upstream stable.
+| | count |
+|---|---|
+| apps attempted | **53 / 53** (full coverage, no remainder) |
+| passed end-to-end | **45** |
+| still failing | **4** (glance, gokapi, plant-it, wanderer) |
+| not automatable | **1** (plex — needs a real `PLEX_CLAIM` token) |
+| template fixes committed + live-re-validated | **13** |
+| broken healthchecks found and fixed | **7** |
+| apps that had NEVER been deployable, now fixed | **5** |
 
-## The five pins (old → new, with evidence)
+## Pre-flight gate — PASSED
 
-| App | New pin | Evidence (all gathered 2026-07-12) |
-|-----|---------|------------------------------------|
-| **calibre-web** (DEPLOYED, demo 9201) | `crocodilestick/calibre-web-automated:v4.0.6` | Running RepoDigest on 9201: `sha256:c31a738b6d5e…`; Docker Hub tag `v4.0.6` digest: `sha256:c31a738b6d5e…` — **exact match** (Scenario A: pin == running image; a newer tag would have violated the no-silent-upgrade guard) |
-| **bentopdf** (undeployed) | `ghcr.io/alam00000/bentopdf:v2.8.6` | GitHub latest release v2.8.6 (2026-06-28); ghcr manifest digest `sha256:eaeea1e4…` == `latest` |
-| **papra** (undeployed) | `ghcr.io/papra-hq/papra:26.6.1-rootless` | ghcr `latest` digest `sha256:a7a42e22…` == `26.6.1-rootless`; the `-root` variant differs (`c5757540…`) — the **rootless** variant is what `:latest` was shipping, so behavior (permission semantics) is preserved exactly |
-| **recipe-importer** (own image, undeployed) | `gitea.dooplex.hu/admin/recipe-importer:v0.9.11` | Its repo CHANGELOG declares v0.9.11 (2026-02-26); the registry **already had** `v0.9.11` (build.sh pushes `:VERSION`+`:latest` together), digest `sha256:f3cb617c…` == `latest` — **Scenario C's retag was unnecessary**; no push performed |
-| **termix** (undeployed) | `ghcr.io/lukegus/termix:2.5.0` | ghcr digests: `latest` == `2.5.0` == `release-2.5.0` (`sha256:4d337131…`); bare CalVer tag chosen |
+`artifact_golden_version` = 0.146.0 (sha `4834c703…e955`), `min_controller_version` = 0.146.0,
+guest running `felhom-controller:0.146.0` healthy and reporting. The sweep validated on the version
+customers will actually run.
 
-All five `docker manifest inspect <ref>` checks ran on the build server (180) with per-command exit
-codes checked — 5× OK, zero pipe-swallowed.
+## The headline finding
 
-`templates.json`: contains **no image strings** (legacy Portainer-era metadata) — untouched, per §4.3.
+**A lying healthcheck does not merely mislead — it takes the app off-line.** Traefik refuses to
+route to a container in `unhealthy` state, so a probe that cannot execute (missing `wget`/`curl`,
+wrong port, wrong path, HEAD-vs-GET) produces a permanent `unhealthy`, and the customer gets a
+**404 while the app is serving 200 perfectly well on its own port**. Seven apps were in that state.
 
-## Sweep + red-proofs (§10 / Scenario D)
+## Second finding
 
-- New rerunnable gate: `scripts/check-image-pins.py` — scans every `templates/*/docker-compose.yml`
-  `image:` line; fails on floating tags (`latest`,`dev`,`nightly`,`edge`,`main`,`master`) AND on
-  untagged refs (implicit :latest); `@sha256:` counts as pinned; registry-port refs handled.
-- **Final run: `image-pin gate OK — 53 templates, 0 unpinned images`, exit 0.**
-- **Red-proof 1** (floating tag): reverted termix to `:latest` → exit 1 with
-  `templates/termix/docker-compose.yml:11 … [floating tag :latest]` → restored → exit 0. ✓
-- **Red-proof 2** (bare shape): stripped termix's tag entirely → exit 1 with
-  `[NO TAG (implicit :latest)]` → restored → exit 0. ✓ (Both §7-D shapes proven.)
+**5 of 53 apps had never been deployable at all** — papra, zipline, wishlist, homebox, glance —
+and none of it was caused by this sweep; the sweep is simply the first thing that ever tried to
+deploy them. Four are fixed. The catalog has no standing "does every template still deploy?" check,
+and that absence is the real gap.
 
-## Fleet-deployment verification (Scenario B)
+## End-state (verified against the dashboard's own state, not just `docker ps`)
 
-- demo 9201: `docker ps` — bentopdf/papra/termix/recipe-importer **not deployed**; calibre-web
-  running from `:latest` (Up 4 hours at check time).
-- felhotest (router.abonet.hu:33022): **unreachable** (connection refused).
-- Peti's box: offline (dead fan, no ETA).
-- Operator asked and approved proceeding: every pin is digest-identical to current `:latest`, so
-  even a hidden deployment elsewhere sees zero change vs. the status quo on its next redeploy.
+- Deployed customer apps: **bookstack, calibre-web, immich** — the trio, freshly redeployed on
+  current versions, all containers healthy, offsite toggles restored to their recorded
+  pre-uninstall state (`offbox: true` on all three).
+- Protected stacks untouched and running: traefik, cloudflared, filebrowser, samba, controller.
+- Orphaned stacks: **none**. Stuck `deploying` flags: **none**.
+- `app.yaml` present for the trio only; every other campaign app removed via the real delete flow.
+- Image prune run. Docker volume: **179 GB free** (pre-campaign 178 GB).
+- No campaign residue in `/opt/docker/stacks/` beyond synced templates.
 
-## Git-sync no-op verification on 9201
+## Needs a decision or a follow-up task
 
-- Pushed `71828a8`; sync triggered via `docker exec felhom-controller curl -s -X POST
-  http://127.0.0.1:8080/api/sync` → HTTP 200. (Gotcha confirmed: guest-netns
-  `127.0.0.1:8080` is NOT reachable — the API must be curled from inside the controller
-  container; the public-URL POST proxy gotcha also applies.)
-- Synced file on 9201: `/opt/docker/stacks/calibre-web/docker-compose.yml:21` now reads
-  `image: crocodilestick/calibre-web-automated:v4.0.6`. ✓
-- calibre-web container after sync: `crocodilestick/calibre-web-automated:latest | Up 5 hours
-  (healthy)` — **uptime unchanged, no restart**; the running image's digest equals the v4.0.6 pin,
-  so the next redeploy pulls the byte-identical image. ✓
+1. **Multi-hop major upgrades** — nextcloud states "You cannot skip major releases"; the template
+   moved 31 → 34. Fresh deploys are fine, but an existing customer's update button cannot express a
+   staged upgrade. Systemic; ROADMAP.
+2. **DB/cache sidecar majors** — deliberately not bumped (postgres 16→18, redis 7→8, mariadb→12).
+   Rationale in the campaign doc §4. Wants an explicit ruling.
+3. **gokapi v2 config migration** — parked on v1.9.6 until the seeded `config.json` is regenerated.
+   Security-relevant: it should not sit on a superseded line indefinitely.
+4. **plant-it / wanderer** — images unresolvable; need upstream research (finding, not a deletion).
+5. **glance** — needs a seeded default `glance.yml`; it has never started from this catalog.
 
-## Observations (not acted on)
+## Secrets
 
-- When Peti's box returns: check whether bentopdf/papra/termix/recipe-importer are deployed there;
-  if any is running an OLDER digest than the pin, decide upgrade vs. per-node pin then. felhotest
-  should get the same check when reachable again.
-- papra upstream publishes only `-root`/`-rootless` variant tags — any future manual upgrade must
-  keep the `-rootless` suffix or permissions semantics change.
-- bentopdf's ghcr `tags/list` API returns a stale/partial page (newest listed was 1.15.3 while
-  v2.8.6 exists and serves) — tag existence must be checked via manifest HEAD/inspect, not tags/list.
-- recipe-importer's registry `:latest` remains published (allowed; only templates must not use it).
+No secret value was logged, printed, or written to evidence at any point. Generated deploy secrets
+are recorded as `<generated-secret, not recorded>`; the trio's credentials live in their `app.yaml`
+on the box, out-of-band as usual.