CHANGELOG + REPORT: campaign 7 full catalog sweep (53/53, 45 pass, 13 fixes)
Campaign 7 catalog sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
@@ -1,71 +1,68 @@
|
||||
# REPORT — image pinning: `:latest` eliminated from the catalog (2026-07-12)
|
||||
# REPORT — CAMPAIGN 7: full app-catalog sweep (2026-07-18 → 2026-07-19)
|
||||
|
||||
**Class:** implementation (template change + sweep gate; push = deploy).
|
||||
**Baseline:** `main` @ `2ebe082` → (this commit). **Scope:** 5 compose image lines + new gate script
|
||||
+ CLAUDE.md rule. No healthcheck/env/format touch-ups; no controller change; no redeploys triggered.
|
||||
**Scope:** every app under `templates/` — bump the pin, deploy it through the real pipeline on the
|
||||
demo box, prove it works, audit its healthcheck, record, remove.
|
||||
**Box:** demo guest 9201 (`demo-felhom` on `felhom-pve`), controller **0.146.0**.
|
||||
**Full matrix + ranked findings:**
|
||||
`felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md`
|
||||
|
||||
## Why
|
||||
## Outcome
|
||||
|
||||
The controller's recovery-unit `ImagePins` (felhom-controller `internal/backup/recovery_unit.go`)
|
||||
pins the compose *tag*: restoring a `:latest`-pinned app re-pulls whatever `:latest` points to at
|
||||
restore time — potentially schema-incompatible with the data being restored. Load-bearing rule:
|
||||
**a deployed app pins to the digest it is currently running** (the pin never causes a version jump);
|
||||
undeployed apps pin to the latest verified upstream stable.
|
||||
| | count |
|
||||
|---|---|
|
||||
| apps attempted | **53 / 53** (full coverage, no remainder) |
|
||||
| passed end-to-end | **45** |
|
||||
| still failing | **4** (glance, gokapi, plant-it, wanderer) |
|
||||
| not automatable | **1** (plex — needs a real `PLEX_CLAIM` token) |
|
||||
| template fixes committed + live-re-validated | **13** |
|
||||
| broken healthchecks found and fixed | **7** |
|
||||
| apps that had NEVER been deployable, now fixed | **5** |
|
||||
|
||||
## The five pins (old → new, with evidence)
|
||||
## Pre-flight gate — PASSED
|
||||
|
||||
| App | New pin | Evidence (all gathered 2026-07-12) |
|
||||
|-----|---------|------------------------------------|
|
||||
| **calibre-web** (DEPLOYED, demo 9201) | `crocodilestick/calibre-web-automated:v4.0.6` | Running RepoDigest on 9201: `sha256:c31a738b6d5e…`; Docker Hub tag `v4.0.6` digest: `sha256:c31a738b6d5e…` — **exact match** (Scenario A: pin == running image; a newer tag would have violated the no-silent-upgrade guard) |
|
||||
| **bentopdf** (undeployed) | `ghcr.io/alam00000/bentopdf:v2.8.6` | GitHub latest release v2.8.6 (2026-06-28); ghcr manifest digest `sha256:eaeea1e4…` == `latest` |
|
||||
| **papra** (undeployed) | `ghcr.io/papra-hq/papra:26.6.1-rootless` | ghcr `latest` digest `sha256:a7a42e22…` == `26.6.1-rootless`; the `-root` variant differs (`c5757540…`) — the **rootless** variant is what `:latest` was shipping, so behavior (permission semantics) is preserved exactly |
|
||||
| **recipe-importer** (own image, undeployed) | `gitea.dooplex.hu/admin/recipe-importer:v0.9.11` | Its repo CHANGELOG declares v0.9.11 (2026-02-26); the registry **already had** `v0.9.11` (build.sh pushes `:VERSION`+`:latest` together), digest `sha256:f3cb617c…` == `latest` — **Scenario C's retag was unnecessary**; no push performed |
|
||||
| **termix** (undeployed) | `ghcr.io/lukegus/termix:2.5.0` | ghcr digests: `latest` == `2.5.0` == `release-2.5.0` (`sha256:4d337131…`); bare CalVer tag chosen |
|
||||
`artifact_golden_version` = 0.146.0 (sha `4834c703…e955`), `min_controller_version` = 0.146.0,
|
||||
guest running `felhom-controller:0.146.0` healthy and reporting. The sweep validated on the version
|
||||
customers will actually run.
|
||||
|
||||
All five `docker manifest inspect <ref>` checks ran on the build server (180) with per-command exit
|
||||
codes checked — 5× OK, zero pipe-swallowed.
|
||||
## The headline finding
|
||||
|
||||
`templates.json`: contains **no image strings** (legacy Portainer-era metadata) — untouched, per §4.3.
|
||||
**A lying healthcheck does not merely mislead — it takes the app off-line.** Traefik refuses to
|
||||
route to a container in `unhealthy` state, so a probe that cannot execute (missing `wget`/`curl`,
|
||||
wrong port, wrong path, HEAD-vs-GET) produces a permanent `unhealthy`, and the customer gets a
|
||||
**404 while the app is serving 200 perfectly well on its own port**. Seven apps were in that state.
|
||||
|
||||
## Sweep + red-proofs (§10 / Scenario D)
|
||||
## Second finding
|
||||
|
||||
- New rerunnable gate: `scripts/check-image-pins.py` — scans every `templates/*/docker-compose.yml`
|
||||
`image:` line; fails on floating tags (`latest`,`dev`,`nightly`,`edge`,`main`,`master`) AND on
|
||||
untagged refs (implicit :latest); `@sha256:` counts as pinned; registry-port refs handled.
|
||||
- **Final run: `image-pin gate OK — 53 templates, 0 unpinned images`, exit 0.**
|
||||
- **Red-proof 1** (floating tag): reverted termix to `:latest` → exit 1 with
|
||||
`templates/termix/docker-compose.yml:11 … [floating tag :latest]` → restored → exit 0. ✓
|
||||
- **Red-proof 2** (bare shape): stripped termix's tag entirely → exit 1 with
|
||||
`[NO TAG (implicit :latest)]` → restored → exit 0. ✓ (Both §7-D shapes proven.)
|
||||
**5 of 53 apps had never been deployable at all** — papra, zipline, wishlist, homebox, glance —
|
||||
and none of it was caused by this sweep; the sweep is simply the first thing that ever tried to
|
||||
deploy them. Four are fixed. The catalog has no standing "does every template still deploy?" check,
|
||||
and that absence is the real gap.
|
||||
|
||||
## Fleet-deployment verification (Scenario B)
|
||||
## End-state (verified against the dashboard's own state, not just `docker ps`)
|
||||
|
||||
- demo 9201: `docker ps` — bentopdf/papra/termix/recipe-importer **not deployed**; calibre-web
|
||||
running from `:latest` (Up 4 hours at check time).
|
||||
- felhotest (router.abonet.hu:33022): **unreachable** (connection refused).
|
||||
- Peti's box: offline (dead fan, no ETA).
|
||||
- Operator asked and approved proceeding: every pin is digest-identical to current `:latest`, so
|
||||
even a hidden deployment elsewhere sees zero change vs. the status quo on its next redeploy.
|
||||
- Deployed customer apps: **bookstack, calibre-web, immich** — the trio, freshly redeployed on
|
||||
current versions, all containers healthy, offsite toggles restored to their recorded
|
||||
pre-uninstall state (`offbox: true` on all three).
|
||||
- Protected stacks untouched and running: traefik, cloudflared, filebrowser, samba, controller.
|
||||
- Orphaned stacks: **none**. Stuck `deploying` flags: **none**.
|
||||
- `app.yaml` present for the trio only; every other campaign app removed via the real delete flow.
|
||||
- Image prune run. Docker volume: **179 GB free** (pre-campaign 178 GB).
|
||||
- No campaign residue in `/opt/docker/stacks/` beyond synced templates.
|
||||
|
||||
## Git-sync no-op verification on 9201
|
||||
## Needs a decision or a follow-up task
|
||||
|
||||
- Pushed `71828a8`; sync triggered via `docker exec felhom-controller curl -s -X POST
|
||||
http://127.0.0.1:8080/api/sync` → HTTP 200. (Gotcha confirmed: guest-netns
|
||||
`127.0.0.1:8080` is NOT reachable — the API must be curled from inside the controller
|
||||
container; the public-URL POST proxy gotcha also applies.)
|
||||
- Synced file on 9201: `/opt/docker/stacks/calibre-web/docker-compose.yml:21` now reads
|
||||
`image: crocodilestick/calibre-web-automated:v4.0.6`. ✓
|
||||
- calibre-web container after sync: `crocodilestick/calibre-web-automated:latest | Up 5 hours
|
||||
(healthy)` — **uptime unchanged, no restart**; the running image's digest equals the v4.0.6 pin,
|
||||
so the next redeploy pulls the byte-identical image. ✓
|
||||
1. **Multi-hop major upgrades** — nextcloud states "You cannot skip major releases"; the template
|
||||
moved 31 → 34. Fresh deploys are fine, but an existing customer's update button cannot express a
|
||||
staged upgrade. Systemic; ROADMAP.
|
||||
2. **DB/cache sidecar majors** — deliberately not bumped (postgres 16→18, redis 7→8, mariadb→12).
|
||||
Rationale in the campaign doc §4. Wants an explicit ruling.
|
||||
3. **gokapi v2 config migration** — parked on v1.9.6 until the seeded `config.json` is regenerated.
|
||||
Security-relevant: it should not sit on a superseded line indefinitely.
|
||||
4. **plant-it / wanderer** — images unresolvable; need upstream research (finding, not a deletion).
|
||||
5. **glance** — needs a seeded default `glance.yml`; it has never started from this catalog.
|
||||
|
||||
## Observations (not acted on)
|
||||
## Secrets
|
||||
|
||||
- When Peti's box returns: check whether bentopdf/papra/termix/recipe-importer are deployed there;
|
||||
if any is running an OLDER digest than the pin, decide upgrade vs. per-node pin then. felhotest
|
||||
should get the same check when reachable again.
|
||||
- papra upstream publishes only `-root`/`-rootless` variant tags — any future manual upgrade must
|
||||
keep the `-rootless` suffix or permissions semantics change.
|
||||
- bentopdf's ghcr `tags/list` API returns a stale/partial page (newest listed was 1.15.3 while
|
||||
v2.8.6 exists and serves) — tag existence must be checked via manifest HEAD/inspect, not tags/list.
|
||||
- recipe-importer's registry `:latest` remains published (allowed; only templates must not use it).
|
||||
No secret value was logged, printed, or written to evidence at any point. Generated deploy secrets
|
||||
are recorded as `<generated-secret, not recorded>`; the trio's credentials live in their `app.yaml`
|
||||
on the box, out-of-band as usual.
|
||||
|
||||
Reference in New Issue
Block a user