Campaign 10's R-156 found papra writing its database into the container's writable layer while the
volume the template preserves stayed empty — a backup that completes, verifies, and contains
nothing. papra was never the point: nothing anywhere checked that the folder a template preserves
is the folder the app writes to. All 53 templates have now been measured live.
43 CLEAN / 3 BROKEN / 7 UNDETERMINED. UNDETERMINED is counted separately, each with its reason,
and never folded into CLEAN.
FIXED (neither app is deployed anywhere, so nothing was stranded):
- gramps-web mounted /app/data, /app/media, /tmp — and /app/data is a path the application never
writes. Its accounts database and ITS FAMILY TREE both landed in the writable layer while
gramps_data was tarred nightly as an empty directory. Now persists the eight paths the image's
own environment names, matching upstream's reference compose. Proven: users.sqlite and the
family-tree files survive a redeploy byte-identical, same inode.
- wishlist mounted wishlist_data:/data, another path the app never writes; prod.db landed in the
ANONYMOUS volume from the image's VOLUME directive — absent from ResolveDockerVolumeNames, so
never backed up, and orphaned by a redeploy. Now mounts /usr/src/app/data + /usr/src/app/uploads.
Proven: prod.db byte-identical, same inode, across a redeploy.
Every corrected path confirmed by two independent sources — the shipped image's own
environment/Config.Volumes and upstream's reference compose — never inferred from a directory name.
papra is NOT fixed. It is live on one box, and changing the mount target makes the next compose up
recreate the container and destroy the writable layer its documents live in. The fix is prepared
and proven in the scratch guest (current: db.sqlite differs after a redeploy, so a real account
created via the API is lost; fixed: byte-identical, it survives). Referred to the operator with the
two options; no migration written.
NEW GATE scripts/check-volume-persistence.py — the third catalog gate and the only RUNTIME one.
This class is invisible to static analysis, measured not assumed: a static audit of all 53 composes
reports the catalog clean AND reports papra clean. Exit 0 clean / 1 REFUSED / 2 undecided. It
refuses to report at all unless it has just re-proven itself in both directions against two canary
templates that differ only in which path the volume mounts at, so every run carries a live
demonstration of R-156 and of its fix. No docker exec anywhere (Campaign 7 §1.1). 44 fixture tests
driving check(), the function __main__ calls; every rule red-proofed.
Enforcement is convention, not CI — this repo has no CI. Stated plainly in the report; raising it
is proposed as R-160.
Report, per-app evidence, proofs and proposed register entries (R-158..R-161, NOT filed — felhom.eu
is fenced this session): audits/persistence-sweep-2026-08-02/
paperless-ngx and calibre-web are the only two catalog apps with a drop-zone,
and each has exactly ONE ingest bind. Both move from ${USERDATA_PATH}/import/<app>
to ${IMPORT_PATH}/<app> — the canonical root on the system drive — so a
multi-drive box has one drop-zone instead of one real folder plus a dead
lookalike on every other drive (and import/* is class: excluded, so files
stranded in a dead one would never be backed up either).
The matching backup: entries move to the new `import:` list IN THE SAME COMMIT.
This is not cosmetic: ValidateBackupSpec rejects an entry matching no compose
bind, and the rejection is WHOLE-BLOCK, so a stale `userdata: import/paperless`
would have discarded paperless's `hdd: appdata/paperless/media class: mandatory`
too and silently degraded the customer's document originals to legacy handling.
Both classes stay `excluded` — the move must not change data handling.
New data_paths: blocks on paperless-ngx, calibre-web and romm — role + Hungarian
label over paths that already exist as compose binds. Covers all three roles and
the multi-entry case. Requires controller v0.172.0 (deployed to both demo boxes
before this push, since ${IMPORT_PATH} is unset on older controllers).
Storage-layout header comments updated in both composes — they are the only
in-repo documentation of the layout.
Moving a template out of templates/ un-offers it but also makes the
controller's orphan detector see it as GONE for anyone already running the
app - flagging their working install Elavult with a Torles button. Withdrawing
an app must never take a working app away from a customer.
Optional lifecycle: available|hidden|abandoned in .felhom.yml instead.
plant-it returns to templates/ as the first abandoned app; retired/ removed.
Resolvability gate skips (and reports) non-available apps.
wanderer: ghcr.io/flomp/wanderer:0.16.0 is a ghost - upstream split the app
into web+db images, moved registry and renamed the org. Restructured to
upstream's own v0.20.0 compose (3 services, new /data/plugins volume, second
public hostname for PocketBase, meilisearch pinned DOWN to upstream's v1.36.0
per the R-42 ruling).
plant-it: retired. The repo name was wrong (plant-it-server) but upstream has
DELETED self-hosting; last server image is 2024-12-10 and it needs MySQL+Redis
the template never had. Moved to retired/ rather than deleted - reversible.
R-41 slice 1: check-image-resolvable.py. Encodes two traps - manifest inspect
exits 0 while printing toomanyrequests, and the inverse, where the first sweep
called 24 of 65 pins dead because Hub throttled it. Ambiguity is INCONCLUSIVE,
never an accusation.
With 2.6 pulling and booting, wger still sat unhealthy: the healthcheck got
wget rc=4 (network failure, not an HTTP error) because nothing listens on :80.
wger's own log says it plainly -- 'Using django's development server on port
8000...' -- and the image declares ExposedPorts {"8000/tcp"}.
Corrected both the healthcheck target and the Traefik loadbalancer port, which
was pointing at :80 as well (so routing would have failed even once healthy).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Correcting my own earlier revert. Reverting to 2.3 was WRONG: wger/server:2.3 is
no longer published on Docker Hub (only 2.4, 2.5, 2.6 resolve), so that pin could
not be pulled at all -- the deploy was accepted and no container was ever created.
An unpullable pin is strictly worse than the problem it was meant to avoid.
Every available version (2.4/2.5/2.6, verified) reads the whole DJANGO_DB_* set
unconditionally, even with the sqlite engine. So supplying it is not a hack
around one version -- it is now the only way to run wger at all. Verified live:
2.6 with the full set boots and applies its migrations ('Applying auth.0001_initial
... OK').
USER/PASSWORD/HOST/PORT are ignored by the sqlite backend but must be present.
DJANGO_DB_DATABASE points into the existing wger_data volume (/home/wger/db),
which is where wger's own default sqlite file lived -- so an existing install is
not pointed at an empty database somewhere else.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
With the DATABASE_URL fix zipline started cleanly ('server started
hostname=0.0.0.0 port=3000') but stayed unhealthy. Probing the running container
from inside:
/api/health -> 404
/api/healthcheck -> 200
v4 renamed the endpoint; the template still probed the v3 path.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
After the memory fix tandoor still never went healthy. The probe output was
'wget: can't connect to remote host (127.0.0.1): Connection refused' while the
app log was still printing 'Booting worker' / 'Running django-vite' -- i.e. the
app had simply not finished starting. With start_period 30s and retries 3 the
healthcheck gave up at roughly two minutes.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
With the image and the pepper fixed, homebox still sat unhealthy. The probe's own
exit code was 8 (server error response) and homebox's log shows exactly why:
method=HEAD path=/api/v1/status status=405
method=GET path=/api/v1/status status=200
'wget --spider' issues a HEAD request; homebox's status endpoint only implements
GET. Switched to 'wget -q -O /dev/null' (a real GET).
NOTE: 24 templates use --spider. The rest validated green, so their endpoints do
answer HEAD -- but this is a latent trap worth a convention note (see the
campaign doc).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
wger 2.6 crash-loops on a fresh deploy. Its settings/main.py reads the whole
DJANGO_DB_* set unconditionally -- even when the engine is sqlite:
ImproperlyConfigured: Set the DJANGO_DB_DATABASE environment variable
...then, once that was supplied:
ImproperlyConfigured: Set the DJANGO_DB_USER environment variable
Satisfying it means either stuffing in dummy USER/PASSWORD/HOST/PORT values that
the sqlite backend ignores, or giving wger a real Postgres sidecar. The first is
a hack, the second is compose restructuring -- both outside this campaign's
allowed-fix set.
Reverted to the previously shipped 2.3. NOTE: the interim DJANGO_DB_DATABASE
line added earlier in this campaign is reverted TOO, deliberately -- on 2.3 the
sqlite path came from wger's own default, and pinning a different explicit path
would have pointed an existing customer's wger at an empty database.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Both crashes above were masking a second defect: once each app actually stayed
up, it sat permanently unhealthy because its healthcheck called wget, and
neither image has wget or curl -- only node. Traefik does not route to an
unhealthy container, so both would have served 404 to the customer regardless.
Switched both to the Node-exec family (http.get, exit non-zero on 5xx/error).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
zipline crash-looped 15 times on a fresh deploy:
ERROR config::readDbVars] No database environment variables found
(DATABASE_URL or all of [DATABASE_USERNAME, DATABASE_PASSWORD,
DATABASE_HOST, DATABASE_PORT, DATABASE_NAME]), exiting...
In Zipline v4 the database variable is DATABASE_URL with NO prefix, while
CORE_SECRET kept its prefix (confirmed against the upstream config docs) -- so
only the DB variable was renamed here.
The template was already pinned to the v4 line (4.0.0) before this campaign, and
v4 has always required DATABASE_URL, so zipline has been undeployable from the
catalog for the whole v4 series. PRE-EXISTING defect, surfaced by the sweep.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
wishlist deployed but never created a container: the pull fails because
cmintey/wishlist:1.9.0 no longer exists on Docker Hub. Upstream publishes to
ghcr.io/cmintey/wishlist, where the current release is v0.66.0 (2026-07-08,
confirmed against the GitHub releases API).
Note the version string goes '1.9.0' -> 'v0.66.0'. That is NOT a downgrade: the
old Hub pin used a scheme upstream does not publish under, so the catalog was
pinned to an image reference that has no counterpart in the real release stream.
This was a PRE-EXISTING defect -- wishlist was undeployable before this campaign
too (it was not version-bumped by the sweep).
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
wger 2.6 crash-looped 15 times on a fresh deploy:
django.core.exceptions.ImproperlyConfigured:
Set the DJANGO_DB_DATABASE environment variable
The template already selected the sqlite3 engine, but 2.6 no longer supplies a
default database path -- it must be given explicitly even for sqlite. Pointed at
the existing wger_data volume (/home/wger/db) so the database survives redeploys.
Env var proven wrong by the deploy.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
tandoor 2.6.13 never reached healthy; gunicorn workers were SIGKILLed in a loop:
[ERROR] Worker (pid:366) was sent SIGKILL! Perhaps out of memory?
[ERROR] Worker (pid:367) was sent SIGKILL! Perhaps out of memory?
Separately, .felhom.yml mem_limit was already inconsistent with the compose file:
it claimed 512M while the services summed to 512+256 = 768M. Per the REUSE.md
rule mem_limit is the SUM of the compose limits, so it is now 1024+256 = 1280M.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
rallly 4.11.1 (Next.js 16) booted and was immediately killed by the cgroup OOM
killer, 14 times in 420s:
▲ Next.js 16.2.6
- Local: http://localhost:3000
✓ Ready in 0ms
Killed
Migrations applied fine; it simply cannot live in 256M any more. App limit
256M -> 768M; sidecar Postgres left at 256M; .felhom.yml mem_limit updated to
the new sum (768+256 = 1024M) per the REUSE.md rule.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
With the tag fixed so the image actually pulls, homebox 0.26.2 then panicked on
every start (16 restarts):
panic: auth.api_key_pepper must be set to at least 32 bytes; generate with
`openssl rand -base64 48` and provide via HBOX_AUTH_API_KEY_PEPPER.
Rotating it invalidates all issued API keys
Added as a generated base64key:48 secret (matching upstream's own suggestion)
and marked data_key: true -- rotating it invalidates every issued API key, so
restore must recover the original rather than mint a new one.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
papra was NOT version-bumped by this campaign -- it crash-looped at its existing
26.6.1-rootless pin, so this is a PRE-EXISTING catalog defect: papra has never
been deployable from this template.
Invalid configuration: In production, the auth secret must not be the default
one. Please set a secure auth secret using the AUTH_SECRET environment
variable.
Added as a generated hex:32 secret and marked data_key: true -- it signs
sessions, so regenerating it on restore would invalidate every login.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
n8n 2.31.3 died on a fresh deploy under the 512M limit:
FATAL ERROR: Ineffective mark-compacts near heap limit
Allocation failed - JavaScript heap out of memory
Note the container still reported *healthy* while the Node process was crashing,
so docker health alone did not catch this -- the log scan did.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
outline 1.9.1 crash-looped (15 restarts in 420s) on a fresh deploy:
SequelizeConnectionError: The server does not support SSL connections
Set the PGSSLMODE environment variable to 'disable' or enable SSL on your
database server.
The stack's own Postgres sidecar is on the app-internal network and does not
serve SSL, so the correct answer is to disable it client-side. Env var proven
wrong by the deploy.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
v25.6.0 could not stay up under the 384M limit. Live evidence from the demo box:
[ERROR] Worker (pid:559) was sent SIGKILL! Perhaps out of memory?
[ERROR] Worker (pid:556) was sent SIGKILL! Perhaps out of memory?
gunicorn kept booting replacement workers that were killed in turn, so the
container never reached healthy and Traefik returned 404. mem_limit in
.felhom.yml raised to match the compose limit; mem_request 100M -> 384M.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Upstream stopped publishing v-prefixed tags on ghcr.io/sysadminsmedia/homebox.
Neither the new v0.26.2 NOR the previously shipped v0.16.3 resolves any more --
so homebox was already undeployable from the catalog before this campaign; the
deploy is accepted and then no container is ever created because the pull fails.
Pinned to the un-prefixed 0.26.2, which resolves.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Campaign 7 bumped gokapi to v2.2.4. It crash-loops on a FRESH deploy:
Gokapi v2.2.4 starting
Warning: Salt for admin password invalid, generating new salt.
Please update to version 2.0.0 before running this version.
The catalog seeds a v1-era config.json (ConfigVersion 21). v2.2.4 sees that
config version and demands an intermediate v2.0.0 migration pass -- even with an
empty data volume -- so it never comes up.
Making v2 work needs the seeded config regenerated in the v2 format, which is
template/config restructuring and therefore OUTSIDE this campaign's allowed-fix
set (pins, healthchecks, start_period, proven-wrong env, OOM limits). Shipping
the broken pin would break gokapi for every NEW customer deploy, so the pin is
reverted to the last known-good version and the upgrade is recorded as a
finding for a dedicated task.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
House style is a specific upstream tag. ':2' is a floating major that silently
moves under customers on every pull -- the same class of problem as ':latest',
just narrower. Pinned to the current 2.x release.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The healthcheck ran 'curl -f', but the Emby image has neither curl nor a
standalone wget -- only BusyBox. The probe ENOENT'd every run, the container
stayed permanently unhealthy, and Traefik will not route to an unhealthy
container, so the customer got a 404 while Emby itself was serving fine.
Switched to the BusyBox-wget family via the /bin/busybox multi-call binary.
Campaign 7 catalog sweep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
admin