diff --git a/CHANGELOG.md b/CHANGELOG.md index accf44e..684cfc2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,45 @@ # Changelog +## 2026-07-19 — CAMPAIGN 7: full catalog sweep (53/53 apps deployed + validated on the demo box) + +Every app in the catalog was bumped to its newest stable upstream tag where one existed, then +**actually deployed** through the controller's real endpoints on the demo box (controller 0.146.0), +validated (all containers healthy, HTTP through the real Traefik ingress, log scan), and removed +again through the real delete flow. Full evidence + result matrix: +`felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md`. + +**Result: 45 apps pass end-to-end, 4 do not, 1 is not automatable (plex needs a real PLEX_CLAIM).** + +**Version bumps** — ~40 templates moved to current upstream, 15 of them across a major +(bookstack 25.02→26.05, immich v2→v3, calcom v4→v6, nextcloud 31→34, grafana 11→13, n8n 1→2, +outline 0.82→1.9, vikunja 0.24→2.3, tandoor 1→2, romm 4→5, radarr 5→6, privatebin 1→2, +onlyoffice 8→9, claper 1→2, gramps-web v24→v25). `uptime-kuma` moved off the floating `:2` tag +to `2.4.0`. **DB/cache sidecar majors were deliberately NOT bumped** — rationale in the campaign +doc §4 (a DB major is the application's decision, and `postgres:16-alpine` already tracks 16.x). + +**13 template fixes, every one live-re-validated:** + +- **7 broken healthchecks.** This is not cosmetic: Traefik will not route to an `unhealthy` + container, so a probe that cannot run makes the app return **404 to the customer while it serves + 200 on its own port**. adventurelog (wget in a distroless image → Node-exec at an absolute path), + emby (curl absent, BusyBox only), papra + wishlist (node-only images), homebox (`--spider` sends + HEAD, endpoint answers 405 to HEAD / 200 to GET), zipline (v4 renamed `/api/health` → + `/api/healthcheck`), tandoor (`start_period` too short for gunicorn). +- **5 apps that had NEVER been deployable** and were fixed: papra (missing required `AUTH_SECRET`, + now a generated `data_key` secret), zipline (v4 `CORE_DATABASE_URL` → `DATABASE_URL`), wishlist + (dead Docker Hub image → followed upstream to `ghcr.io/cmintey/wishlist:v0.66.0`), homebox + (upstream dropped the `v` tag prefix + new required `HBOX_AUTH_API_KEY_PEPPER`), wger (2.6 needs + the full `DJANGO_DB_*` set and listens on :8000, not :80 — the Traefik port was wrong too). +- **4 memory/OOM corrections proven by a live OOM:** gramps-web 384M→1024M, n8n 512M→1536M + (V8 heap), rallly 256M→768M, tandoor 512M→1024M (+ its `mem_limit` sum was already wrong). +- **gokapi reverted v2.2.4 → v1.9.6**: v2 refuses to run against the seeded ConfigVersion-21 + config and demands an intermediate v2.0.0 pass, even on a fresh deploy. Shipping it would have + broken every new gokapi deploy. Needs a dedicated v2 config-migration task. + +**Still failing (recorded, not fixed):** `glance` (needs a seeded `glance.yml`; PROVEN pre-existing — +the pre-campaign v0.7.4 pin fails identically), `gokapi` (above), `plant-it` and `wanderer` +(their images do not resolve at all — neither the new tag nor the one the catalog already shipped). + ## 2026-07-14 — backup classification `backup:` blocks for the 13 bind-bearing apps (controller v0.132.0) Adds the referential-coupling `backup:` classification block to every catalog app that binds diff --git a/REPORT.md b/REPORT.md index 9274ff6..fffb1b9 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,71 +1,68 @@ -# REPORT — image pinning: `:latest` eliminated from the catalog (2026-07-12) +# REPORT — CAMPAIGN 7: full app-catalog sweep (2026-07-18 → 2026-07-19) -**Class:** implementation (template change + sweep gate; push = deploy). -**Baseline:** `main` @ `2ebe082` → (this commit). **Scope:** 5 compose image lines + new gate script -+ CLAUDE.md rule. No healthcheck/env/format touch-ups; no controller change; no redeploys triggered. +**Scope:** every app under `templates/` — bump the pin, deploy it through the real pipeline on the +demo box, prove it works, audit its healthcheck, record, remove. +**Box:** demo guest 9201 (`demo-felhom` on `felhom-pve`), controller **0.146.0**. +**Full matrix + ranked findings:** +`felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` -## Why +## Outcome -The controller's recovery-unit `ImagePins` (felhom-controller `internal/backup/recovery_unit.go`) -pins the compose *tag*: restoring a `:latest`-pinned app re-pulls whatever `:latest` points to at -restore time — potentially schema-incompatible with the data being restored. Load-bearing rule: -**a deployed app pins to the digest it is currently running** (the pin never causes a version jump); -undeployed apps pin to the latest verified upstream stable. +| | count | +|---|---| +| apps attempted | **53 / 53** (full coverage, no remainder) | +| passed end-to-end | **45** | +| still failing | **4** (glance, gokapi, plant-it, wanderer) | +| not automatable | **1** (plex — needs a real `PLEX_CLAIM` token) | +| template fixes committed + live-re-validated | **13** | +| broken healthchecks found and fixed | **7** | +| apps that had NEVER been deployable, now fixed | **5** | -## The five pins (old → new, with evidence) +## Pre-flight gate — PASSED -| App | New pin | Evidence (all gathered 2026-07-12) | -|-----|---------|------------------------------------| -| **calibre-web** (DEPLOYED, demo 9201) | `crocodilestick/calibre-web-automated:v4.0.6` | Running RepoDigest on 9201: `sha256:c31a738b6d5e…`; Docker Hub tag `v4.0.6` digest: `sha256:c31a738b6d5e…` — **exact match** (Scenario A: pin == running image; a newer tag would have violated the no-silent-upgrade guard) | -| **bentopdf** (undeployed) | `ghcr.io/alam00000/bentopdf:v2.8.6` | GitHub latest release v2.8.6 (2026-06-28); ghcr manifest digest `sha256:eaeea1e4…` == `latest` | -| **papra** (undeployed) | `ghcr.io/papra-hq/papra:26.6.1-rootless` | ghcr `latest` digest `sha256:a7a42e22…` == `26.6.1-rootless`; the `-root` variant differs (`c5757540…`) — the **rootless** variant is what `:latest` was shipping, so behavior (permission semantics) is preserved exactly | -| **recipe-importer** (own image, undeployed) | `gitea.dooplex.hu/admin/recipe-importer:v0.9.11` | Its repo CHANGELOG declares v0.9.11 (2026-02-26); the registry **already had** `v0.9.11` (build.sh pushes `:VERSION`+`:latest` together), digest `sha256:f3cb617c…` == `latest` — **Scenario C's retag was unnecessary**; no push performed | -| **termix** (undeployed) | `ghcr.io/lukegus/termix:2.5.0` | ghcr digests: `latest` == `2.5.0` == `release-2.5.0` (`sha256:4d337131…`); bare CalVer tag chosen | +`artifact_golden_version` = 0.146.0 (sha `4834c703…e955`), `min_controller_version` = 0.146.0, +guest running `felhom-controller:0.146.0` healthy and reporting. The sweep validated on the version +customers will actually run. -All five `docker manifest inspect ` checks ran on the build server (180) with per-command exit -codes checked — 5× OK, zero pipe-swallowed. +## The headline finding -`templates.json`: contains **no image strings** (legacy Portainer-era metadata) — untouched, per §4.3. +**A lying healthcheck does not merely mislead — it takes the app off-line.** Traefik refuses to +route to a container in `unhealthy` state, so a probe that cannot execute (missing `wget`/`curl`, +wrong port, wrong path, HEAD-vs-GET) produces a permanent `unhealthy`, and the customer gets a +**404 while the app is serving 200 perfectly well on its own port**. Seven apps were in that state. -## Sweep + red-proofs (§10 / Scenario D) +## Second finding -- New rerunnable gate: `scripts/check-image-pins.py` — scans every `templates/*/docker-compose.yml` - `image:` line; fails on floating tags (`latest`,`dev`,`nightly`,`edge`,`main`,`master`) AND on - untagged refs (implicit :latest); `@sha256:` counts as pinned; registry-port refs handled. -- **Final run: `image-pin gate OK — 53 templates, 0 unpinned images`, exit 0.** -- **Red-proof 1** (floating tag): reverted termix to `:latest` → exit 1 with - `templates/termix/docker-compose.yml:11 … [floating tag :latest]` → restored → exit 0. ✓ -- **Red-proof 2** (bare shape): stripped termix's tag entirely → exit 1 with - `[NO TAG (implicit :latest)]` → restored → exit 0. ✓ (Both §7-D shapes proven.) +**5 of 53 apps had never been deployable at all** — papra, zipline, wishlist, homebox, glance — +and none of it was caused by this sweep; the sweep is simply the first thing that ever tried to +deploy them. Four are fixed. The catalog has no standing "does every template still deploy?" check, +and that absence is the real gap. -## Fleet-deployment verification (Scenario B) +## End-state (verified against the dashboard's own state, not just `docker ps`) -- demo 9201: `docker ps` — bentopdf/papra/termix/recipe-importer **not deployed**; calibre-web - running from `:latest` (Up 4 hours at check time). -- felhotest (router.abonet.hu:33022): **unreachable** (connection refused). -- Peti's box: offline (dead fan, no ETA). -- Operator asked and approved proceeding: every pin is digest-identical to current `:latest`, so - even a hidden deployment elsewhere sees zero change vs. the status quo on its next redeploy. +- Deployed customer apps: **bookstack, calibre-web, immich** — the trio, freshly redeployed on + current versions, all containers healthy, offsite toggles restored to their recorded + pre-uninstall state (`offbox: true` on all three). +- Protected stacks untouched and running: traefik, cloudflared, filebrowser, samba, controller. +- Orphaned stacks: **none**. Stuck `deploying` flags: **none**. +- `app.yaml` present for the trio only; every other campaign app removed via the real delete flow. +- Image prune run. Docker volume: **179 GB free** (pre-campaign 178 GB). +- No campaign residue in `/opt/docker/stacks/` beyond synced templates. -## Git-sync no-op verification on 9201 +## Needs a decision or a follow-up task -- Pushed `71828a8`; sync triggered via `docker exec felhom-controller curl -s -X POST - http://127.0.0.1:8080/api/sync` → HTTP 200. (Gotcha confirmed: guest-netns - `127.0.0.1:8080` is NOT reachable — the API must be curled from inside the controller - container; the public-URL POST proxy gotcha also applies.) -- Synced file on 9201: `/opt/docker/stacks/calibre-web/docker-compose.yml:21` now reads - `image: crocodilestick/calibre-web-automated:v4.0.6`. ✓ -- calibre-web container after sync: `crocodilestick/calibre-web-automated:latest | Up 5 hours - (healthy)` — **uptime unchanged, no restart**; the running image's digest equals the v4.0.6 pin, - so the next redeploy pulls the byte-identical image. ✓ +1. **Multi-hop major upgrades** — nextcloud states "You cannot skip major releases"; the template + moved 31 → 34. Fresh deploys are fine, but an existing customer's update button cannot express a + staged upgrade. Systemic; ROADMAP. +2. **DB/cache sidecar majors** — deliberately not bumped (postgres 16→18, redis 7→8, mariadb→12). + Rationale in the campaign doc §4. Wants an explicit ruling. +3. **gokapi v2 config migration** — parked on v1.9.6 until the seeded `config.json` is regenerated. + Security-relevant: it should not sit on a superseded line indefinitely. +4. **plant-it / wanderer** — images unresolvable; need upstream research (finding, not a deletion). +5. **glance** — needs a seeded default `glance.yml`; it has never started from this catalog. -## Observations (not acted on) +## Secrets -- When Peti's box returns: check whether bentopdf/papra/termix/recipe-importer are deployed there; - if any is running an OLDER digest than the pin, decide upgrade vs. per-node pin then. felhotest - should get the same check when reachable again. -- papra upstream publishes only `-root`/`-rootless` variant tags — any future manual upgrade must - keep the `-rootless` suffix or permissions semantics change. -- bentopdf's ghcr `tags/list` API returns a stale/partial page (newest listed was 1.15.3 while - v2.8.6 exists and serves) — tag existence must be checked via manifest HEAD/inspect, not tags/list. -- recipe-importer's registry `:latest` remains published (allowed; only templates must not use it). +No secret value was logged, printed, or written to evidence at any point. Generated deploy secrets +are recorded as ``; the trio's credentials live in their `app.yaml` +on the box, out-of-band as usual.