felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md

# CAMPAIGN 7 — full app-catalog sweep (bump · deploy · validate · clean)

**Run:** 2026-07-18 evening → 2026-07-19 morning (overnight, hard wall-clock 06:30 CEST)
**Repos:** `app-catalog-felhom.eu` (the work), `felhom.eu/documentation` (this doc)
**Box:** demo (guest 9201 `demo-felhom` on `felhom-pve`), controller **0.146.0**
**Class:** Campaign — record-and-rank, with a defined allowed-fix set (the fixes are the deliverable)

---

## 0. Pre-flight gate — PASSED

Both of Viktor's saves had landed before the sweep began:

| check | value |
|---|---|
| `artifact_golden_version` | **0.146.0**, sha `4834c703…e955` ✓ |
| `min_controller_version` (floor) | **0.146.0** ✓ (saved last, as required) |
| guest 9201 running | `felhom-controller:0.146.0`, healthy ✓ |
| box reporting | `controller_started (0.146.0)` @ 18:52:07 ✓ |

So the sweep validated on the version customers actually run.

> **Finding C7-F1 (observability, MEDIUM) — the floor-lift update emitted no
> `controller_updated` event.** The gate asked for `controller_updated 0.145.0→0.146.0`
> in the events. It is **absent**, although 0.145.0→0.146.0 went through the *same*
> agent-driven bootstrap path (`/etc/felhom-bootstrap/bootstrap.json` →
> `felhom-controller-bootstrap.service`) that DID emit the event for 0.143.0→0.145.0 two
> hours earlier. `controller_started (0.146.0)` was emitted normally.
> Consequence: the operator-visible event trail under-reports version transitions, so
> "did the box converge?" cannot be answered from the events alone. Not a blocker — the
> runtime version, the golden record and the floor all agreed — but it is the one signal
> the gate was written around.

---

## 1. Method

- Every deploy/undeploy went through the controller's **real endpoints** — the ones the UI
  calls: `POST /api/stacks/<app>/deploy`, `POST /api/stacks/<app>/stop`,
  `POST /api/stacks/<app>/remove`, `POST /api/sync`. No raw `docker compose` against a
  managed stack, no hand-edited `app.yaml`.
- Deploy fields auto-filled from `deploy-fields` metadata (DOMAIN / SUBDOMAIN / HDD_PATH);
  secrets minted per the `.felhom.yml` `generate:` spec. **No secret value was ever logged
  or written to evidence** — evidence records the field NAME and `<generated-secret, not
  recorded>`.
- One campaign app deployed at a time. Evidence per app under
  `180:~/campaign7/evidence/<app>/` (containers, healthcheck audit, per-container logs,
  traefik view, log scan).
- Validation asserted **effect**, not absence-of-error: terminal health verdict per
  container, healthcheck-binary audit, HTTP probe through the real Traefik ingress, login
  where feasible, log scan with judgment for benign startup noise.

### 1.1 Three engine bugs found and fixed mid-run (they would have corrupted the matrix)

Recorded because they are exactly the "hollow validation" the testing doctrine warns about
— each one made a broken thing look fine, or a fine thing look broken:

1. **Traefik registration race → false 404s.** The first probe fired the instant a
   container reported healthy, but Traefik registers the router a few seconds later.
   actualbudget and calcom were both recorded FAIL and were actually fine (200 / 307).
   Fixed with a retry window; both re-validated green.
2. **`docker exec` writes its OCI error to STDOUT, not stderr.** The healthcheck audit
   checked "did `command -v <tool>` print anything" — so a *missing* binary printed
   `executable file not found` and read as **present**. This made the entire healthcheck
   audit — the campaign's core deliverable — report every app "honest". Fixed to require
   rc==0 **and** an absolute path, with a direct-exec fallback for shell-less images.
3. **`created` sampled as a terminal state.** Containers still starting were recorded as
   settled, so docmost and ghost were flagged FAIL while healthy moments later. Fixed to
   require `running`; both re-validated green.

---

## 2. Uninstall semantics — the real delete flow (map row: PARTIAL → **PROVEN**)

Live evidence from the customer trio (`POST /api/stacks/<app>/remove`, both checkboxes
default-off, which is what the UI sends):

| what | behaviour | evidence |
|---|---|---|
| stack still running | **refused** — `409 "still running — stop it first"`; the flow is stop→remove | bookstack |
| named docker volumes | **ALWAYS DESTROYED** (`compose down --volumes`), incl. the app's database | bookstack ×2, calibre ×1, immich ×3 |
| HDD bind-mount data (`userdata/`, `appdata/`) | **PRESERVED** unless `remove_hdd_data=true` | calibre books 440K, immich 54M — byte-identical after |
| HDD backup dirs (`backups/primary/<app>`) | **PRESERVED** unless `remove_backups=true` | calibre 24K, immich 28K |
| docker images | **KEPT** (for redeploy) | all three |
| `app.yaml` (deploy config) | **REMOVED** | all three |
| `docker-compose.yml` (template) | **KEPT** | all three |
| per-app **offsite** toggle (`app_backup.<app>.offbox`) | **SURVIVES** the uninstall | all three still `offbox:true` after removal |
| per-app **tier-2 / cross-drive** config | **CLEARED** (`SetCrossDriveConfig(name, nil)`) | `removeStack` |

The confirmation modal is honest about the destructive part — it states *"Mindig törlődik:
Docker kötetek (adatbázis, alkalmazás konfiguráció)"* before the customer confirms — so
there is **no consent gap**. Two narrower issues:

> **Finding C7-F2 (evidence trail, LOW) — `volumes_removed` is always `null`.**
> The remove response reported `volumes_removed: null` while actually destroying
> `bookstack_bookstack_config` + `bookstack_bookstack_db_data` (and 3 immich volumes).
> `delete.go` step 3 scrapes `compose down` stdout for `"Removing volume"` / `"Volume"`,
> which current Docker Compose no longer prints in that shape. The modal pre-warns, so this
> is not a safety issue — but the removal receipt is useless as evidence, and any future
> "what did we delete?" audit built on it would silently return nothing.

> **Finding C7-F3 (asymmetry, LOW) — offsite toggle outlives the app.** Removing an app
> clears its tier-2 schedule but leaves `app_backup.<app>.offbox = true` set forever. Benign
> today (a re-deploy inherits the customer's prior intent, which is arguably right, and it
> is why the trio's toggles needed no restoring). Worth an explicit ruling: is a removed
> app's offsite intent meant to persist, or should removal clear it like it clears tier-2?

---

## 3. The customer trio — uninstall → bump → fresh redeploy (Viktor's ruling), done FIRST

All three travelled the full path and are **left RUNNING** as the end state.

| app | pin(s) | MAJOR | deploy | health | http | login | logs | settle |
|---|---|---|---|---|---|---|---|---|
| **bookstack** | `25.02.2 → 26.05.2`; mariadb `11.6 → 12.3` | ✅ ×2 | ok | ok (2/2 healthy) | 302 | **ok** | clean | 20s |
| **calibre-web** | `v4.0.6` (already newest stable) | — | ok | ok | 302 | **ok** | noisy-benign | 20s |
| **immich** | `v2.5.5 → v3.0.3`; postgres `16-vectorchord0.3.0 → 16-vectorchord0.4.3-pgvectors0.2.0` | ✅ | ok | ok (4/4 healthy) | 200 | page-only | clean | 20s |

**Recorded pre-uninstall offsite state — and it contradicted the campaign note.** The note
said "bookstack ON, calibre ON, immich OFF". The box said:

```
bookstack   enabled=false  offbox=true
calibre-web enabled=false  offbox=true
immich      enabled=false  offbox=true      <-- note said OFF
```

Per the rail ("trust the recorded pre-uninstall state over this note") all three were
restored to `offbox:true` — which they already were, since the toggle survives uninstall
(§2). Verified post-redeploy.

**Detail per app**

- **bookstack** — MAJOR ×2. Upstream v26.05 needs `storage/fonts` writable for PDF export and
  makes revision-viewing a separate permission; neither bites a fresh deploy. Logged in with
  the documented `admin@admin.com / password` → 302 + 6 authenticated markers on the
  dashboard. mariadb 12.3 healthy via `healthcheck.sh`.
- **calibre-web** — already at the newest STABLE tag; everything newer upstream is `dev-*`
  (correctly excluded). Logged in with `admin / admin123` (Flask CSRF token required) →
  title renders `Calibre-Web Automated | Books (0)`. The empty library is expected: the DB
  volume was destroyed while the HDD book files were preserved, so the library needs
  re-importing — the accepted consequence. The traceback in its log is **benign**: calibre's
  installer fails headless `xdg-desktop-menu` setup, catches it, reports "There were 1
  warnings", and continues.
- **immich** — MAJOR. v3.0.0 drops pgvecto.rs and requires VectorChord; our pin was already
  VectorChord so the fresh deploy was unaffected. Sidecar moved to the extension versions
  immich v3.0.3 ships in its own compose (vectorchord 0.4.3 / pgvectors 0.2.0) while
  **keeping our PG major 16** rather than following upstream down to 14 — a `16-` build of
  that exact extension pair is published, so no needless major change. Proven functional,
  not merely healthy: `/api/server/ping` → `pong`, `/api/server/version` → `{3,0,3}`,
  122 migration/init log lines, "Immich Microservices is running [v3.0.3]".
  `login: page-only` is deliberate — the fresh install sits at first-run admin signup
  (`isInitialized:false`, signup page 200) and creating an admin would mint a credential
  that would then have to be transmitted to Viktor out-of-band; the owner creates his own.
- **redis vs valkey (recorded, not fixed):** immich v3 upstream migrated `redis` →
  `valkey:9`. We kept `redis:7-alpine` and **it works fine on v3** (all 4 containers
  healthy). Swapping the image is a structural change, not a pin bump, so it is recorded
  for a considered follow-up rather than done here.

### 3.1 Offsite continuity across uninstall → reinstall — **PROVEN** (free evidence for R-34)

The rail asked whether a re-deployed app's offsite run RESUMES its pre-uninstall snapshot tags.
An offsite run was triggered after the trio was back up (`POST /backup/offbox/run` — a run, not a
config change). It completed `ok` in 1m13s, and the tags line up exactly:

| app | pre-uninstall snapshot | post-reinstall snapshot | tags |
|---|---|---|---|
| calibre-web | `243ad08c` 07-18 17:13:42 | `9698ba74` 07-19 01:26:58 | `felhom-offbox,calibre-web` — **identical** |
| immich | `6df12205` 07-18 17:13:46 | `ea511336` 07-19 01:27:02 | `felhom-offbox,immich` — **identical** |
| bookstack | *(none — it was the „kimaradtak" app)* | `63850757` 07-19 01:26:52 | `felhom-offbox,bookstack` — **new** |
| shares | — | `bbbc2fae` 07-19 01:27:55 | `felhom-offbox,_shares` (R-7b path still working) |

**Continuity holds.** The tag lineage is stable across a full uninstall → volume-destroy → fresh
redeploy, so a `restic --tag <app>` query still spans the reinstall boundary and the pre-uninstall
restore points remain addressable — reinstalling an app does not orphan its offsite history.

**A side effect worth recording:** the standing `kimaradtak: bookstack` warning is now **cleared**
(`last_warning: None`, 6 snapshots, 211.6 MB). bookstack had no offsite snapshot before this
campaign because it had no tier-1 backup to push; the fresh redeploy produced one, so its first-ever
offsite snapshot landed here. The rail predicted the warning might FIRE during the sweep — instead
the sweep resolved a pre-existing one.

---

## 4. Version-bump policy actually applied (deviation, stated deliberately)

App images: newest STABLE upstream tag, rc/beta/nightly/dev excluded, digest-free pinned
tags per house style.

**DB/cache sidecar majors were deliberately NOT bumped** (postgres 16→18, redis 7→8,
mariadb→12.3 on apps other than bookstack, postgis 16→17). This is a conscious deviation
from a literal reading of "newest stable for every pin":

- a DB major is a **data-plane decision the application owns**, not a currency decision —
  immich proves it, upstream pins one specific tested postgres build;
- tags like `postgres:16-alpine` / `redis:7-alpine` already track the newest patch inside
  their major, so they are not stale;
- blind-bumping ~8 apps onto PG18 would have manufactured failures that are artifacts of
  the sweep's own choice rather than real findings, and would have burned the wall-clock the
  rail explicitly told us to protect.

→ carried to ROADMAP as a decision item, not silently skipped. See §7.

---

## 5. Healthcheck audit — the wget lesson, systematically

Every deployed app's compose healthcheck was parsed and the probe binary checked **inside
the image**.

> **Finding C7-F4 (HIGH) — a lying healthcheck does not merely mislead; it takes the app
> OFF-LINE.** Traefik will not route to a container in an `unhealthy` state. So a probe that
> ENOENTs → permanent `unhealthy` → **Traefik returns 404 to the customer while the app is
> serving 200 perfectly well on its own port**. This was not theoretical: both apps below
> were completely unreachable for that reason, and both looked "deployed" in docker ps.

| app | lie | reality | fix | verified |
|---|---|---|---|---|
| **adventurelog** (frontend) | `wget --spider` | distroless image: no shell, no wget, no curl; node only at `/nodejs/bin/node`, off PATH | Node-exec family via the absolute interpreter path | 404 → **200**, 3/3 healthy |
| **emby** | `curl -f` | no curl, no standalone wget — image ships only BusyBox v1.38 | BusyBox-wget family via `/bin/busybox wget` | unhealthy/404 → **healthy/302** |

Both fixed in-template and live-re-validated on the demo box.

---

## 6. Result matrix

**All 53 catalog apps were attempted.** `http` is through the real Traefik ingress; a 3xx to a login
page is a PASS. `login` was attempted where credentials are documented and the flow is scriptable.

### 6.1 Passed (45)

| app | old → new pin | MAJOR | deploy | health | http | logs | settle |
|---|---|---|---|---|---|---|---|
| actualbudget | 26.1.0 → 26.7.0 | | ok | ok | 200 | clean | 10s |
| adventurelog | v0.11.0 → v0.12.1 | | ok | **fixed** | 200 | clean | 100s |
| audiobookshelf | 2.19.5 → 2.35.1 | | ok | ok | 200 | clean | 30s |
| bentopdf | v2.8.6 (newest) | | ok | ok | 200 | clean | 20s |
| **bookstack** ¹ | 25.02.2 → 26.05.2 · mariadb 11.6 → 12.3 | ✅ | ok | ok | 302 | clean | 20s |
| calcom | v4.6.9 → v6.2.0 | ✅ | ok | ok | 307 | clean | 20s |
| **calibre-web** ¹ | v4.0.6 (newest) | | ok | ok | 302 | noisy-benign | 20s |
| claper | 1.8 → 2.5 | ✅ | ok | ok | 200 | clean | 70s |
| code-server | 4.96.4 → 4.129.0 | | ok | ok | 302 | clean | 40s |
| crafty-controller | 4.10.7 (GitLab registry, manual) | | ok | ok | 302 | clean | 50s |
| docmost | 0.25.3 → 0.95.0 | | ok | ok | 200 | clean | 20s |
| emby | 4.9.0.42 → 4.10.0.20 | | ok | **fixed** | 302 | clean | 10s |
| ghost | 6.19.2-alpine → 6.53.0-alpine | | ok | ok | 200 | clean | 10s |
| gitea | 1.23.4 → 1.27.0 | | ok | ok | 200 | clean | 10s |
| grafana | 11.5.1 → 13.1.0 | ✅ | ok | ok | 302 | clean | 20s |
| gramps-web | v24.12.1 → v25.6.0 | ✅ | ok | **fixed** (mem) | 200 | clean | 110s |
| home-assistant | 2026.2.2 → 2026.7.2 | | ok | ok | 302 | clean | 70s |
| homebox | v0.16.3 → 0.26.2 | | ok | **fixed** ×3 | 200 | clean | — |
| homepage | v1.2.0 → v1.13.2 | | ok | ok | 200 | clean | 20s |
| **immich** ¹ | v2.5.5 → v3.0.3 · pg → 16-vectorchord0.4.3 | ✅ | ok | ok | 200 | clean | 20s |
| jellyfin | 10.11.6 → 10.11.11 | | ok | ok | 302 | clean | 30s |
| kimai | apache-2.25.0 → apache-2.57.0 | | ok | ok | 302 | clean | 50s |
| komga | 1.20.0 → 1.25.0 | | ok | ok | 200 | clean | 20s |
| mealie | v3.10.2 → v3.20.1 | | ok | ok | 200 | clean | 40s |
| n8n | 1.79.3 → 2.31.3 | ✅ | ok | **fixed** (mem) | 200 | clean | 50s |
| navidrome | 0.54.5 → 0.63.2 | | ok | ok | 302 | clean | 10s |
| nextcloud | 31.0.14-apache → 34.0.1-apache | ✅ | ok | ok | 302 | clean | 50s |
| onlyoffice | 8.3.0 → 9.4.0 | ✅ | ok | ok | 302 | clean | 40s |
| opengist | 1.10 → 1.13 | | ok | ok | 302 | clean | 10s |
| outline | 0.82.0 → 1.9.1 | ✅ | ok | **fixed** (env) | 200 | clean | 40s |
| paperless-ngx | 2.15.3 → 2.20.15 | | ok | ok | 302 | clean | 70s |
| papra | 26.6.1-rootless (newest) | | ok | **fixed** ×2 | 200 | clean | — |
| privatebin | 1.7.5 → 2.0.5 | ✅ | ok | ok | 200 | clean | 10s |
| radarr | 5.17.2 → 6.3.0 | ✅ | ok | ok | 200 | clean | 10s |
| rallly | 3.11.2 → 4.11.1 | ✅ | ok | **fixed** (mem) | 200 | clean | 20s |
| recipe-importer | v0.9.11 (ours, internal registry) | | ok | ok | 302 | clean | 10s |
| romm | 4.5.0 → 5.0.0 | ✅ | ok | ok | 200 | clean | 50s |
| seerr | 2.3.0 → 2.7.3 | | ok | ok | 307 | clean | 30s |
| sonarr | 4.0.13 → 4.0.19 | | ok | ok | 200 | clean | 10s |
| sparkyfitness | v0.17.2 → v0.17.3 (server + web) | | ok | ok | 200 | clean | 40s |
| tandoor | 1.5.26 → 2.6.13 | ✅ | ok | **fixed** ×4 | 302 | clean | 40s |
| termix | 2.5.0 (newest) | | ok | ok | 200 | clean | 20s |
| uptime-kuma | `:2` → 2.4.0 (**floating tag pinned**) | | ok | ok | 302 | clean | 30s |
| vaultwarden | 1.33.2-alpine → 1.36.0-alpine | | ok | ok | 200 | clean | 10s |
| wger | 2.3 → **2.6** (2.3 is gone upstream) | | ok | **fixed** ×3 | 302 | clean | 60s |
| vikunja | 0.24.6 → 2.3.0 | ✅ | ok | ok ⚠ no healthcheck | 200 | clean | 0s |
| wishlist | Hub 1.9.0 → **ghcr v0.66.0** | | ok | **fixed** ×2 | 200 | clean | — |
| zipline | 4.0.0 → 4.6.1 | | ok | **fixed** ×2 | 200 | clean | — |

¹ the customer trio — left RUNNING as the end state (see §3). Login: bookstack **ok**,
calibre-web **ok**, immich page-only (deliberate, §3). For the other 40 apps login was
`not-attempted` — a rendered login page through the real ingress was taken as PASS per the rail.

### 6.2 Not passing (4) + 1 not attempted

| app | outcome | cause | disposition |
|---|---|---|---|
| **glance** | FAIL — crash-loop | needs `/app/config/glance.yml`; the template mounts an EMPTY config volume and never seeds one | **pre-existing** — proven: v0.7.4 (the pre-campaign pin) fails identically. Bump KEPT, finding raised |
| **gokapi** | FAIL — crash-loop | v2.2.4 refuses to run against the seeded ConfigVersion-21 config: *"Please update to version 2.0.0 before running this version"* | **pin REVERTED to v1.9.6** (last known-good); v2 migration needs the seeded config regenerated |
| **plant-it** | FAIL — image | `msdeluise/plant-it` does not resolve on Docker Hub — **neither 1.0.1 nor the shipped 0.10.0** | **pre-existing**; no replacement registry found. Needs upstream research |
| **wanderer** | FAIL — image | `ghcr.io/flomp/wanderer` does not resolve — **neither 0.20.0 nor the shipped 0.16.0** | **pre-existing**; meilisearch sidecar bumped v1.12 → v1.49. Needs upstream research |
| **plex** | not attempted | `PLEX_CLAIM` is a required field with no default — a real claim token from plex.tv is needed | not automatable; genuinely owner-supplied. Not a defect |

### 6.3 MAJOR breaking notes (for the future update-path story — see R-40)

Fresh deploys do not migrate, so none of these bit during the sweep. They matter for the moment an
EXISTING customer presses update.

| app | major | one-line breaking note | source |
|---|---|---|---|
| bookstack | 25.02 → 26.05 | `storage/fonts` must be writable for PDF export; revision-viewing became a separate permission | upstream releases |
| immich | v2 → v3 | drops pgvecto.rs, requires VectorChord (our pin was already VectorChord); most other breaks are API-only | v3.0.0 notes |
| nextcloud | 31 → 34 | **"You cannot skip major releases"** — sequential upgrade required. The sharpest case for R-40 | admin manual |
| n8n | 1 → 2 | MySQL/MariaDB support removed; several nodes removed; `ExecuteCommand`/`LocalFileTrigger` disabled by default; env access blocked in code expressions | 2.0.0 notes |
| grafana | 11 → 13 | Angular plugin support **removed** in 12 (deprecated in 11); `editors_can_admin` gone; v2 dashboard schema is one-way | whatsnew v12 |
| vikunja | 0.24 → 2.3 | CORS on by default and `service.publicurl` becomes **required to start**; OpenID provider config array→map; SQLite paths now relative to `service.rootpath`; API `/tasks/all`→`/tasks` | 1.0.0 changelog |
| privatebin | 1 → 2 | min PHP 7.3→7.4; `page` template removed; legacy model class names must become `Filesystem`/`Database`; v1/ZeroBin paste format support dropped (DB column drops needed) | 2.0.0 notes |
| radarr | 5 → 6 | no breaking config/DB changes documented; docker users must update the image (no in-container update) | 6.3.0 notes |
| romm | 4 → 5 | no breaking config/DB/folder changes documented (UI overhaul release) | 5.0.0 notes |
| outline | 0.82 → 1.9 | **found empirically:** 1.x defaults to SSL for Postgres — needs `PGSSLMODE=disable` against a sidecar DB | live deploy |
| tandoor | 1 → 2 | **found empirically:** app port moved 8080 → 80 and `ALLOWED_HOSTS` became mandatory (Django 400s everything without it) | live deploy |
| zipline | (4.0 → 4.6) | **found empirically:** v4 renamed `CORE_DATABASE_URL` → `DATABASE_URL` and `/api/health` → `/api/healthcheck` | live deploy |
| homebox | (v0.16 → 0.26) | **found empirically:** upstream dropped the `v` tag prefix; `HBOX_AUTH_API_KEY_PEPPER` now required | live deploy |
| wger | (2.3 → 2.6) | **found empirically:** full `DJANGO_DB_*` set required even for sqlite; app listens on :8000 | live deploy |
| calcom · claper · gramps-web · onlyoffice | v4→v6 · 1→2 · v24→v25 · 8→9 | deployed clean; upstream one-liner **not retrieved** within the wall-clock — flagged MAJOR without a fabricated note | — |

The empirically-found rows are the more valuable half: they are what actually happens on a real
deploy, not what a changelog remembered to mention.

---

## 7. Findings, ranked

**F4 (HIGH) — a lying healthcheck takes the app OFF-LINE, it does not merely mislead.**
Traefik refuses to route to a container in `unhealthy` state, so an ENOENT'ing probe →
permanent `unhealthy` → **404 for the customer while the app serves 200 on its own port**.
Seven apps were affected. Detail in §5 and F5 below.

**F5 (HIGH) — 7 of 53 apps shipped a broken or wrong healthcheck.** All fixed and live-re-validated:

| app | the lie | reality | fix |
|---|---|---|---|
| adventurelog (frontend) | `wget --spider` | distroless: no shell/wget/curl; node only at an absolute path | Node-exec via `/nodejs/bin/node` |
| emby | `curl -f` | no curl, no standalone wget — BusyBox only | `/bin/busybox wget` |
| papra | `wget --spider` | image ships only `node` | Node-exec |
| wishlist | `wget --spider` | image ships only `node` | Node-exec |
| homebox | `wget --spider` (**HEAD**) | endpoint answers **405 to HEAD, 200 to GET** | `wget -q -O /dev/null` (GET) |
| zipline | `GET /api/health` | v4 renamed it — `/api/health` 404, `/api/healthcheck` 200 | corrected path |
| tandoor | `start_period: 30s` | gunicorn still booting; probes exhausted at ~2 min | `start_period: 240s` |

**F6 (HIGH) — 5 apps were ALREADY undeployable before this campaign.** None was caused by the
sweep; the sweep is simply the first thing that ever tried to deploy them:
`glance` (no seeded config), `papra` (no `AUTH_SECRET`), `zipline` (v4 `DATABASE_URL` rename),
`wishlist` (dead Docker Hub image), `plant-it` + `wanderer` (images do not resolve at all).
**Four are now fixed; plant-it and wanderer need upstream research.**
→ The catalog had no standing "does every template still deploy?" check. That is the real gap.

**F7 (HIGH, systemic) — the update path cannot express a multi-hop major upgrade.**
Nextcloud states plainly: *"You cannot skip major releases."* This campaign moved its template
31 → 34. A fresh deploy is fine (validated, 302), but an EXISTING customer's update button would
attempt 31 → 34 in one step, which Nextcloud forbids. Same shape for any app with sequential-major
rules. The template pin is a single value with no notion of an upgrade path.
→ ROADMAP item; nextcloud is the sharpest case but not the only one.

**F8 (MEDIUM, security-relevant) — gokapi is parked on a superseded version line.**
The revert to v1.9.6 restores the pre-campaign state rather than introducing a new regression, but
gokapi cannot reach v2 until the seeded `config.json` is regenerated in the v2 format. It should not
sit on v1 indefinitely — this wants a dedicated task, not a backlog line.

**F1 (MEDIUM) — the floor-lift controller update emitted no `controller_updated` event.** §0.

**F9 (MEDIUM) — an "obvious" env fix nearly orphaned customer data.** While chasing wger 2.6 the
interim fix pinned `DJANGO_DB_DATABASE=/home/wger/db/database.sqlite`. On 2.3 that path came from
wger's own default; hard-coding a different explicit path would have pointed an existing customer's
wger at an **empty** database while looking perfectly healthy. Reverted deliberately along with the
pin. Worth remembering as a class: *adding an explicit path for a value that previously defaulted is
a data-location change, not a config tidy-up.*

**F10 (MEDIUM) — 24 templates probe with `wget --spider`, which issues HEAD.** homebox proved the
trap (405 to HEAD, 200 to GET). The other 23 validated green, so their endpoints do answer HEAD —
but the default choice is fragile and silently costs availability when an app tightens its methods.
→ convention note for REUSE.md: prefer a real GET (`wget -q -O /dev/null`) unless HEAD is verified.

**F11 (LOW) — vikunja ships with NO healthcheck at all.** It serves 200 and Traefik routes it
(no health state to filter on), so it works — but it has no liveness signal, and the controller-side
probe is the only thing watching it. Not fixed: adding one was not "trivial-and-certain" within the
allowed set.

**F2 (LOW) — the remove receipt never lists destroyed volumes** (`volumes_removed: null`). §2.

**F3 (LOW) — a removed app keeps its offsite toggle forever.** §2. Needs a ruling.

**F12 (LOW) — `.felhom.yml` `mem_limit` can drift from the compose sum.** tandoor claimed 512M
while its services summed to 768M. Only found because tandoor OOM'd. A mechanical gate would catch
the whole catalog at once (the repo already has `check-image-pins.py` as the pattern to copy).

### 7.1 Sweep-engine bugs (recorded because they are the "hollow validation" failure mode)

Three bugs in the campaign's own harness would have written a **false matrix** — two made broken
things look fine, one made fine things look broken. All fixed mid-run and the affected apps
re-validated (§1.1): the Traefik registration race (false 404s), `docker exec` writing its OCI error
to **stdout** (which turned the entire healthcheck audit green), and sampling `created` as a
terminal state.

---

## 8. Coverage

**53 of 53 catalog apps attempted — full coverage. No resumable remainder.**

- **45 passed** end-to-end through the real pipeline (deploy → healthy → HTTP through Traefik → log
  scan → removed via the real delete flow).
- **3 left RUNNING** as the end state (the customer trio), on current versions, offsite toggles
  restored, both scripted logins green.
- **4 failed**, each with a diagnosed root cause and a disposition (§6.2); 2 of those are pinned
  back to their last known-good version rather than shipping a broken pin.
- **1 not attempted** (plex — needs a real `PLEX_CLAIM` token; not a defect).
- **13 template fixes** committed, every one live-re-validated on the demo box.
- **Alert e-mails observed:** none fired during the sweep window. The pre-existing
  `kimaradtak: bookstack` offsite warning (from the 17:14 run, before the campaign) was still on
  record at the start and is expected per the rail — not chased.

### 8.1 Not done / explicitly out of scope

- **DB/cache sidecar majors** (postgres 16→18, redis 7→8, mariadb→12, postgis 16→17) —
  deliberately not bumped, rationale in §4. **ROADMAP decision item.**
- **MAJOR breaking notes** are in §6.3. Retrieved from upstream for bookstack, immich, nextcloud,
  n8n, grafana, vikunja, privatebin, radarr and romm; found EMPIRICALLY (better) for outline,
  tandoor, zipline, homebox and wger. Four (calcom, claper, gramps-web, onlyoffice) deployed clean
  but have **no upstream one-liner** — recorded as not-retrieved rather than fabricated.
- **immich redis → valkey** (upstream migrated in v3; redis:7-alpine works fine on v3) — recorded,
  not done: an image swap is structural, not a pin bump.
- **glance config seeding**, **gokapi v2 config migration**, **plant-it / wanderer upstream
  research** — all outside the allowed-fix set, each needs its own task.