app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02

Catalog persistence sweep — does every app's data land in a persisted volume?

Run: 2026-08-02 · Repo: app-catalog-felhom.eu (only) · Class: implementation — detection first, fixes second.

Campaign 10 found that papra stores its database somewhere the backup never looks (R-156). papra is not the interesting part. Nothing anywhere checked that the folder a template preserves is the folder the app writes to, across 53 templates. This is what that check found.

Parallel-session fence, observed. Another session was running the M-band RTO measurement on demo-hp, writing to felhom.eu. This session ran on demo-felhom and committed only to app-catalog-felhom.eu. No commit to felhom.eu, no register edit, nothing on demo-hp. All register entries below are proposed text for later filing by one session, in one place.

---

1. Venue and baselines

Host	demo-felhom (N100), Tier 0 — ssh felhom-pve (Tailscale 100.70.170.35)
Guest	LXC 9301 catalog-persist, created for this run and destroyed at teardown (§9)
Guest spec	Debian 13, 4 cores, 6 GB + 2 GB swap, local-lvm:120, nesting=1,keyctl=1, DHCP 192.168.0.22
Reach	ssh felhom-pve "pct exec 9301 -- …". No direct SSH: the N100 sits on a remote LAN whose 192.168.0.0/24 collides with DooPlex's, so 192.168.0.22 is unroutable from here (No route to host)
Docker	29.7.1, compose v5.3.1
Untouched	guest 9201 on demo-felhom (read-only docker ps/volume ls only), drill-r50, Peti's cluster, both boxes' backup configuration, everything on demo-hp

Baselines, read fresh

What	Value	How
app-catalog-felhom.eu HEAD	4252121 = origin/main, clean at start	git rev-parse
Templates	53	ls templates | wc -l
felhom.eu HEAD (read-only, for citation)	7ba7c2a	git rev-parse — not committed to
felhom-controller main	v0.188.0, HEAD 4115e88	CHANGELOG.md top
Hub	0.86.0	live /apps footer

Why the scratch guest runs no controller — stated, because the brief asks for its version

There is no felhom-controller on 9301, deliberately. The property under test is a property of the template, and the controller's role in it is exactly one thing: it resolves the ${…} variables and runs docker compose up -d (felhom-controller/internal/stacks/deploy.go:374, :551-582). It does not rewrite the compose file, so a plain Docker host reproducing that env substitution exercises the identical code path in the app.

Standing a controller up would have meant enrolling a box with the hub — creating hub-side state this session is told not to create, and which R-131 records four orphaned instances of. The substitution the controller performs is instead mirrored exactly, including the trap that generateValue prefixes base64key with base64: (deploy.go:904); getting that wrong cost one false verdict before it was caught (§3.4).

---

2. Part 0 — had this ground been covered?

No. Neither Campaign 7 nor any gate, .felhom.yml field, or CI check tests this property. Each negative below names what was searched.

2.1 Campaign 7 tested deployability and health, not persistence

felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md:38-51 states its method: per-container terminal health verdict, healthcheck-binary audit, HTTP probe through the real Traefik ingress, login where scriptable, and a log scan. Its result matrix (:114) carries the columns pin · MAJOR · deploy · health · http · logs · settle. There is no persistence column, and no step in the method reads where data landed.

The proof that this is a gap rather than a technicality is in that same matrix: papra passed it — deploy ok, health **fixed** ×2, http 200, logs clean (:151) — while its database was being written to a directory no backup would ever contain. Campaign 7 also demonstrated the narrower fact that a healthcheck can be a lie (C7-F4, :210-224); this sweep demonstrates that a truthful healthcheck is still not evidence about data.

What changed since: nothing that would have closed it. Between Campaign 7 and today the catalog gained the resolvability gate (2026-07-21), lifecycle: (2026-07-21) and R-75's data_paths: (2026-07-26) — image currency, circulation status, and folder labelling respectively.

2.2 No gate asserts a volume ⟷ data-path relationship. Searched:

Where	What was searched	Result
app-catalog-felhom.eu/scripts/	full listing	exactly two gates, check-image-pins.py (syntactic tag check) and check-image-resolvable.py (does the tag still exist) — both about images, neither about volumes
catalog repo CI	.gitea/workflows, .github, drone/woodpecker/ci config; every non-template YAML	none exists. Catalog gates are convention-run, invoked by whoever edits the catalog — this matters for §7
all four repos	grep -rniE "volume.*(persist|data.?path|writable.layer)|writable.layer|docker diff|persistence.(gate|check)" --include=*.py --include=*.go --include=*.sh	no check of this kind; the only hits are comments and a test fixture name
all gate scripts, four repos	ls felhom-controller/scripts/*.py felhom.eu/scripts/*.py app-catalog-felhom.eu/scripts/*.py felhom-agent/scripts/*	11 gates. The closest by name, felhom-controller/scripts/docker_run_volume_path_gate.py, is unrelated: it allowlists docker … -v arguments in controller Go code so a controller-local host path cannot be mounted from inside a container (v0.125.0 scenario D)
.felhom.yml schema	backup:, data_paths:, lifecycle:, deploy_fields:	none expresses "the app writes here"

2.3 The two validators that look closest cannot see this class

ValidateBackupSpec (felhom-controller/internal/appbackup/classify.go) and ValidateDataPaths (internal/stacks/datapaths.go:58) both reject an entry matching no compose bind — real checks, but of declaration consistency, not of runtime behaviour. And both operate on ParseComposeClassifiableBinds (internal/stacks/classify_binds.go), which keeps only mounts whose host side passes classifyRoot — i.e. paths under ${USERDATA_PATH} / ${HDD_PATH} / ${IMPORT_PATH}. Named volumes are skipped entirely.

That is decisive for papra, which has zero binds: every one of these validators is vacuously satisfied by it. It is also decisive for scale — 39 of 53 templates declare no felhom-root bind at all, so for three-quarters of the catalog the entire data-protection story is DumpAppVolumes (internal/backup/backup.go:543) tarring a named volume, and nothing checks that the volume has anything in it.

2.4 R-156's evidence, read before building

felhom.eu/documentation/tests/campaign10-evidence-2026-07-31/r156-papra-volume.txt — five numbered observations: the volume mounted at /app/data; uid=999(nonroot) with /app/data root root; touch: cannot touch '/app/data/.w': Permission denied; the volume empty but for a canary the campaign wrote as root; and db.sqlite 475136 B in the container's writable layer at /app/app-data/. The detector was built against those observations, and it reproduces all of them.

---

3. The detector

scripts/check-volume-persistence.py. The detector and the gate are the same program — the sweep below was produced by running the committed gate, so the durable artifact is the thing that generated the evidence rather than a paper copy of it.

Per app: deploy from the template → exercise it into writing data → compare where the data landed against what the compose mounts.

3.1 What is observed, and why nothing uses docker exec

	signal	how
M	mount inventory, by persistence class	docker inspect .Mounts
D	the writable layer	docker diff — Docker excludes every mounted path from it, so an entry here is definitionally in no volume and no bind
U	the app's real uid/gid	/proc/<State.Pid>/status, not Config.User — which is empty for papra even though it drops to 999
W	writability of each mount by that uid	host-side stat of the mount source
E	mount occupancy	host-side walk of the mount source

No docker exec anywhere. Campaign 7 §1.1 recorded that docker exec writes its OCI error to STDOUT, so a missing binary printed executable file not found and read as present — which made that campaign's entire healthcheck audit report every app honest. Everything here is read from docker inspect, docker diff, /proc and the host filesystem, so a distroless or shell-less image is observed exactly like any other.

3.2 Mount classes — the second defect class this exposes

class	persists a redeploy?	backed up?
named-declared — in the compose volumes: block	yes	yes
anonymous — an image VOLUME or unnamed mount (64-hex docker name)	survives a restart	NO
bind	yes	per the backup spec
tmpfs	no	no

The anonymous case is deceptive precisely because it survives a restart. ResolveDockerVolumeNames (felhom-controller/internal/appbackup/appdata.go) returns <project>_<name> only for volumes declared in the compose file, so an anonymous volume is invisible to DumpAppVolumes — and a compose down + up orphans it and creates a fresh one. Data there is unbacked-up and lost on redeploy, with no symptom until someone looks.

3.3 What counts as "exercised", and what counts as data

Exercised. Every container reaches running, and healthy where a healthcheck exists; then an HTTP request the app's own router answers. Any status code — including 3xx/4xx/5xx — proves the request reached application code; a connection refusal (000) does not. If that produced no writes anywhere, a second-chance pass walks a wider path list following redirects, so a first-run wizard is actually reached, and waits again. An app that still writes nothing is UNDETERMINED, never CLEAN — health-check-passing is not data-writing, and conflating the two is what let papra look fine.

Data versus noise. Three rules, deliberately explicit rather than buried in a heuristic:

1. Noise — /tmp, /run, /var/log, /var/cache, package-manager state, __pycache__, node_modules, .cache, *.pid, *.sock, *.log, *.pyc, and the per-container files Docker itself writes (/etc/hosts, /etc/resolv.conf, …).
2. Database signature — a filename that proves a database is in that directory (*.sqlite*, PG_VERSION, ib_logfile*, data.mdb, dump.rdb, MANIFEST-*, data.ms, …). The strongest signal, and it needs no path heuristics.
3. Data token — a path saying "app state" (data, storage, uploads, library, documents, var/lib/postgresql, …), used only where rule 2 is silent.

A versus C is the whole difficulty, and rule 3 is where judgement actually lives:

• A (added) — the app created this file; it exists only in the writable layer. papra's db.sqlite is an A. DATA is decided from A entries only.
• C (changed) — a file that ships in the image was touched. Usually a chown sweep and entirely benign. A C on a database-signature file is genuinely ambiguous, because an app writing into a DB shipped in its image looks identical. Those are held as SUSPECT and settled by bytes: the file is copied out of the running container and out of a pristine container created from the same image, and compared. Identical → a chown, recorded and dismissed. Different → a real write into an image-layer file, which is papra's defect with a different verb. Unreadable → UNDETERMINED, never folded into CLEAN.

3.4 Red-proofs — both directions, before the sweep, and again on every run

	claim	result
positive	it flags papra	BROKEN, reproducing R-156's evidence line for line: /app/data EMPTY and NOT writable by the app's own uid=999 gid=999, db.sqlite in the writable layer at /app/app-data/db
negative	it stays quiet on a known-good app	vaultwarden CLEAN — and demonstrably good, not merely unflagged (below)

The negative is a proven negative. state/survive2.py deployed vaultwarden, let it write, ran a real docker compose down + up -d (no -v — that is a redeploy; down --volumes is the uninstall path, Campaign 7 §2), and re-fingerprinted every file by sha256 and inode:

vaultwarden   SURVIVES   before=4 after=4 survived=2 changed=2 lost=0
  survived: /data::db.sqlite3     <- the database itself, byte-identical, SAME INODE
            /data::rsa_key.pem
  changed : /data::db.sqlite3-shm , /data::db.sqlite3-wal   <- SQLite sidecars, expected on restart

The self-test runs on every invocation, in both directions. Before the gate reports anything it puts two canary templates through the same prober and refuses to issue a verdict unless it calls one BROKEN and the other CLEAN. Both are built from a purpose-made image reproducing papra's exact ownership shape — /app/data root-owned, /app/app-data owned by the app's uid — and differ only in which path the volume is mounted at, so the canary pair is a live demonstration of R-156 and of its fix, every run. A detector that flags nothing is worse than none: it converts an unexamined catalog into a documented-clean one.

3.5 Seven defects in the detector, every one found by adjudicating a flag rather than trusting it

Recorded because each would have put a wrong number in the matrix, and six of the seven made a correct app look broken — the failure mode that gets a gate switched off and then protects nothing.

#	defect	what it did
1	base64key lost the controller's base64: prefix (deploy.go:904)	bookstack's Laravel APP_KEY was invalid, it served 500s, read UNDETERMINED. With the prefix: CLEAN. A harness bug wearing an app defect's clothes — the class Campaign 7 §1.1 and Campaign 10 §4d both record
2	C entries scored as data	a linuxserver.io entrypoint re-owning its application tree produced 1305 C entries for calibre-web, 92 of them PNGs under cps/static/css/images/**. Called BROKEN. Led to the A/C split and the byte-level adjudicator (§3.3)
3	a cache directory escaped the noise filter	docker diff lists directories, so the bytecode cache appeared as a bare …/crafty/config/__pycache__ entry while its .pyc children were filtered by suffix — the directory then being the only surviving entry under a path containing config. crafty-controller called BROKEN four times over with its data correctly placed. Fixed by normalising with a trailing slash, covering node_modules, .cache, .git at once
4	postgresql.conf listed as a database signature	it is a config file the postgres entrypoint writes at init. immich called BROKEN with its PGDATA correctly in its volume, 1831 files
5	path tokens convicted on their own	onlyoffice unpacks its OWN static assets into the writable layer at first boot — plugin icons, slide-theme media/, web-apps/apps/api/documents/api.js, 2560 added entries — while its real data mount received data normally. Rule 3 was demoted to report-only
6	the structural check asked per container, not per app	fired on docmost, immich and claper — three correct apps, identical shape: the app container's only volume is for user uploads and legitimately empty on a fresh install, while all real state sits in the sibling database container's volume (1540 / 1833 / 1470 files)
7	mount rules never consulted the noise list	the prefixes are written /run/, so a bare mount target /run matched none of them. privatebin's image declares VOLUME /run; the anonymous volume docker made there holds nginx.pid, php-fpm.sock and s6 fifos, and read as 14 files of unbacked-up customer data

Plus one that was not a false verdict but a reporting defect, and arguably the worst of the set:

8. A finding was being swallowed by a louder one. gramps-web's structural finding — its family tree landing outside every mount, the entire point of the app — went into the undetermined bucket, and that bucket is discarded whenever a BROKEN leg exists. Its lesser defect (the accounts database) convicted first and hid it. A finding that disappears because a different finding won belongs to the same family as an absent log line read as health, and the fix is that the structural finding is now reported whatever the verdict.

The pattern, which is the real lesson. Every false positive came from a rule that identified data by its name rather than by evidence of what it is: C assumed to be a write, postgresql.conf assumed to be a database, media/ and config/ assumed to be customer content, /run assumed to be storage because it was a volume. The rules that never needed correcting are the ones grounded in something checkable — a database file the app created, a mount its own uid cannot write, bytes that differ from the image's copy, and did anything land in a preserved folder at all.

Restarts, stated plainly. The sweep was restarted from the first app twice rather than merge a matrix built by more than one version of the rules — the call Campaign 10 made restarting run 2b from cycle 1. Defects 6, 7 and 8 were found after the final pass had begun; all three are in classify(), which reads only what the probe already captured, so those verdicts were re-derived rather than re-measured (state/reclassify.py), and then every non-CLEAN and every flipped app was re-run live end-to-end through the final gate so no reported verdict rests on post-processing (§5.3). Six verdicts changed on re-derivation: claper, docmost, immich, sparkyfitness and zipline UNDETERMINED→CLEAN, privatebin BROKEN→UNDETERMINED.

And one rule was written, red-proofed, and deleted. A second "are all this directory's children noise?" filter looked like a sensible belt to defect 3's brace. Its red-proof came back green, which is the tell: e["added"] can only ever contain entries that already passed is_noise, so the check is always False — dead code wearing the costume of a safeguard, exactly the trap CLAUDE.md catalogues. It is gone, the code says why it is not there, and test_cache_directories_are_filtered_at_ENTRY_level pins the mechanism that actually does the work.

4. The sweep — all 53 templates

43 CLEAN · 3 BROKEN · 7 UNDETERMINED. UNDETERMINED is counted separately and is never folded into CLEAN; each one's reason is in §4.2.

Evidence per app: evidence/<app>/probe.json — the full probe (mount inventory with occupancy and writability, the app's real uid, the writable-layer rollup, the adjudicated suspects) plus the verdict and its reasons. Runner log: state/gate.log.

How to read the mount column: [vol] a named volume declared in the compose (persisted AND backed up) · [ANON] an anonymous volume from the image's own VOLUME directive (persisted across a restart, never backed up, orphaned by a redeploy) · [bind] a host bind · ,EMPTY the mount received nothing.

4.1 The matrix

app	verdict	what the template mounts	where the app actually wrote	mount not writable by app uid
gramps-web	BROKEN	/app/data[vol,EMPTY], /app/media[vol,EMPTY], /tmp[vol,EMPTY]	/app/users (writable layer)	—
papra	BROKEN	/app/data[vol,EMPTY]	/app/app-data/db (writable layer)	/app/data
wishlist	BROKEN	/data[vol,EMPTY], /usr/src/app/uploads[ANON,EMPTY], /usr/src/app/data[ANON]	/usr/src/app/data(1)	—
bentopdf	UNDETERMINED	—	nothing	—
glance	UNDETERMINED	/app/config[vol,EMPTY]	nothing	—
plant-it	UNDETERMINED	—	nothing	—
privatebin	UNDETERMINED	/tmp[ANON,EMPTY], /var/lib/nginx/tmp[ANON,EMPTY], /srv/data[vol,EMPTY], /run[ANON]	/run(14)	—
recipe-importer	UNDETERMINED	/data[vol,EMPTY]	nothing	—
uptime-kuma	UNDETERMINED	/app/data[vol,EMPTY]	nothing	—
wanderer	UNDETERMINED	/app/uploads[vol,EMPTY], /pb_data[vol], /data/plugins[vol,EMPTY], /meili_data[vol]	/pb_data(6), /meili_data(11)	—
actualbudget	CLEAN	/data[vol]	/data(2)	—
adventurelog	CLEAN	/code/media[vol], /var/lib/postgresql/data[vol]	/code/media(251), /var/lib/postgresql/data(2287)	—
audiobookshelf	CLEAN	/config[vol], /metadata[vol], /audiobooks[bind,EMPTY], /podcasts[bind,EMPTY]	/config(16), /metadata(1)	—
bookstack	CLEAN	/config[vol], /var/lib/mysql[vol]	/config(20), /var/lib/mysql(291)	—
calcom	CLEAN	/var/lib/postgresql/data[vol]	/var/lib/postgresql/data(2261)	—
calibre-web	CLEAN	/config[vol], /cwa-book-ingest[bind,EMPTY], /calibre-library[bind]	/config(11), /calibre-library(3)	—
claper	CLEAN	/app/priv/static/uploads[vol,EMPTY], /var/lib/postgresql/data[vol]	/var/lib/postgresql/data(1470)	—
code-server	CLEAN	/config[vol]	/config(6)	—
crafty-controller	CLEAN	/crafty/servers[vol,EMPTY], /crafty/app/config[vol], /crafty/import[vol,EMPTY], /crafty/backups[vol,EMPTY], /crafty/logs[vol]	/crafty/app/config(16), /crafty/logs(8)	—
docmost	CLEAN	/app/data/storage[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol]	/var/lib/postgresql/data(1540), /data(3)	—
emby	CLEAN	/config[vol], /media[bind]	/config(46), /media(3)	—
ghost	CLEAN	/var/lib/ghost/content[vol]	/var/lib/ghost/content(7)	—
gitea	CLEAN	/data[vol]	/data(8)	—
gokapi	CLEAN	/app/data[vol], /app/config[vol]	/app/data(4), /app/config(1)	—
grafana	CLEAN	/var/lib/grafana[vol]	/var/lib/grafana(543)	—
home-assistant	CLEAN	/config[vol]	/config(21)	—
homebox	CLEAN	/data[vol]	/data(3)	—
homepage	CLEAN	/app/config[vol], /var/run/docker.sock[bind,EMPTY]	/app/config(3)	—
immich	CLEAN	/cache[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol], /usr/src/app/upload[bind], /external/photos[bind,EMPTY], /data[ANON,EMPTY]	/var/lib/postgresql/data(1833), /data(3), /usr/src/app/upload(6)	—
jellyfin	CLEAN	/config[vol], /cache[vol], /media[bind]	/config(20), /cache(2), /media(3)	—
kimai	CLEAN	/opt/kimai/var[vol], /var/lib/mysql[vol]	/opt/kimai/var(2547), /var/lib/mysql(279)	—
komga	CLEAN	/config[vol], /data[bind,EMPTY]	/config(11)	—
mealie	CLEAN	/app/data[vol]	/app/data(5)	—
n8n	CLEAN	/home/node/.n8n[vol]	/home/node/.n8n(7)	—
navidrome	CLEAN	/data[vol], /music[bind,EMPTY]	/data(6)	—
nextcloud	CLEAN	/var/www/html/data[bind], /var/www/html[vol], /var/lib/mysql[vol], /data[vol]	/var/www/html/data(69), /var/www/html(4001), /var/lib/mysql(311), /data(3)	—
onlyoffice	CLEAN	/var/www/onlyoffice/Data[vol], /var/log/onlyoffice[vol], /usr/share/fonts/truetype/custom[ANON,EMPTY], /var/lib/onlyoffice[ANON,EMPTY]	/var/www/onlyoffice/Data(3), /var/log/onlyoffice(11)	—
opengist	CLEAN	/opengist[vol]	/opengist(9)	—
outline	CLEAN	/var/lib/outline/data[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol,EMPTY]	/var/lib/postgresql/data(1655)	—
paperless-ngx	CLEAN	/var/lib/postgresql/data[vol], /data[vol], /usr/src/paperless/export[bind,EMPTY], /usr/src/paperless/data[vol], /usr/src/paperless/media[bind], /usr/src/paperless/consume[bind,EMPTY]	/var/lib/postgresql/data(1927), /data(3), /usr/src/paperless/data(9), /usr/src/paperless/media(1)	—
plex	CLEAN	/media[bind], /config[vol], /transcode[vol,EMPTY]	/media(3), /config(82)	—
radarr	CLEAN	/config[vol], /media/movies[bind,EMPTY], /downloads[bind,EMPTY]	/config(11)	—
rallly	CLEAN	/var/lib/postgresql/data[vol]	/var/lib/postgresql/data(1580)	—
romm	CLEAN	/redis-data[ANON,EMPTY], /romm[ANON,EMPTY], /romm/library[bind,EMPTY], /romm/resources[bind,EMPTY], /romm/config[vol], /var/lib/mysql[vol], /data[vol]	/romm/config(1), /var/lib/mysql(271), /data(4)	—
seerr	CLEAN	/app/config[vol]	/app/config(11)	—
sonarr	CLEAN	/media/tv[bind,EMPTY], /downloads[bind,EMPTY], /config[vol]	/config(11)	—
sparkyfitness	CLEAN	/var/lib/postgresql/data[vol], /app/SparkyFitnessServer/uploads[vol,EMPTY], /app/SparkyFitnessServer/backup[vol,EMPTY]	/var/lib/postgresql/data(1815)	—
tandoor	CLEAN	/opt/recipes/staticfiles[vol], /opt/recipes/mediafiles[vol,EMPTY], /var/lib/postgresql/data[vol]	/opt/recipes/staticfiles(3047), /var/lib/postgresql/data(1984)	—
termix	CLEAN	/app/data[vol]	/app/data(4)	—
vaultwarden	CLEAN	/data[vol]	/data(4)	—
vikunja	CLEAN	/app/vikunja/files[vol,EMPTY], /db[vol]	/db(3)	—
wger	CLEAN	/home/wger/db[vol], /home/wger/media[vol,EMPTY]	/home/wger/db(1)	—
zipline	CLEAN	/zipline/uploads[vol,EMPTY], /zipline/public[vol,EMPTY], /var/lib/postgresql/data[vol]	/var/lib/postgresql/data(1357)	—

4.2 UNDETERMINED — seven, each with its reason

None of these is a pass. Four are apps that genuinely wrote nothing, which is a limit of the exercise rather than a property of the template; three could not be started at all.

app	why it is undetermined	is this a template defect?
bentopdf	Declares zero volumes and wrote nothing. Stateless by design — a browser-side PDF tool.	No — but the gate cannot distinguish "stateless" from "not exercised hard enough", and saying CLEAN would claim a check that was never made.
uptime-kuma	Healthy, 302 to /setup, 0 entries in docker diff, volume empty. v2 creates its database only when the setup wizard is submitted.	Unknown. Needs a completed wizard to decide.
privatebin	Wrote nothing to /srv/data; no paste was ever posted. (Its /run, /tmp, /var/lib/nginx/tmp anonymous volumes are runtime state — see §3.5.6.)	Unknown. Needs a posted paste to decide.
recipe-importer	Its one mount /data stayed empty and the only writes were at /.	Unknown. Needs a real import to decide.
glance	Crash-loop (restarting, exit 1). It needs /app/config/glance.yml and the template mounts an empty config volume without seeding one.	Pre-existing and already recorded — Campaign 7 §6.2 proved the same failure on the pre-campaign pin. Not a persistence finding; the app never ran.
plant-it	pull access denied for msdeluise/plant-it, repository does not exist — no container was created.	No. The app is lifecycle: abandoned for exactly this reason (CHANGELOG 2026-07-21); its dead image is the expected end state.
wanderer	Container unhealthy. Its /pb_data (6) and /meili_data (11) did receive data, so the app is not obviously broken — but a verdict on an unhealthy app would be a verdict on a half-started one.	Unknown. Needs a healthy deploy to decide.

4.3 What the CLEAN column actually establishes

For the 43 CLEAN apps the claim is bounded, and worth stating precisely: the data each app wrote during a first-boot-plus-one-HTTP-request exercise landed inside a mount the template preserves, none of it landed in the writable layer, and every mount was writable by the app's own uid. It is not a claim that every future write will land there — an app that stores uploads correctly but writes, say, a search index elsewhere on first use would still read CLEAN here. That is the residual gap, and it is what the periodic re-run in §7.1 exists for.

5. Fixes — two made, each with a survives-a-redeploy proof

Every corrected path is confirmed by two independent sources, never inferred from a directory name: the shipped image's own environment/Config.Volumes, and upstream's reference compose. Guessing at a data path is how this defect class was created.

Neither app is deployed anywhere in the fleet (hub /apps lists 10 deployed apps; neither is among them), so neither fix strands anything. papra is deployed and is therefore NOT fixed — §6.1.

5.1 gramps-web

Mounted /app/data, /app/media, /tmp. /app/data appears nowhere in the image's environment — the application never writes it. Now mounts the eight paths the image itself names, matching upstream's reference compose:

volume	path	what lives there
gramps_users	/app/users	GRAMPSWEB_USER_DB_URI — the accounts database
gramps_db	/root/.gramps/grampsdb	GRAMPS_DATABASE_PATH — the family tree
gramps_index	/app/indexdir	GRAMPSWEB_SEARCH_INDEX_DB_URI
gramps_media	/app/media	GRAMPSWEB_MEDIA_BASE_DIR (was already correct)
gramps_thumb_cache · gramps_cache · gramps_secret · gramps_tmp		caches, exports, secret

Proof — proofs/gramps-web/survive-gramps-web.json. Deploy → exercise → docker compose down (no -v; that is a redeploy, down --volumes is the uninstall path) → up -d → re-fingerprint by sha256 and inode:

gramps-web   SURVIVES   before=5 after=5 survived=4 changed=1 lost=0
  survived (byte-identical, SAME INODE):
    /app/users::users.sqlite                                      <- the accounts database
    /root/.gramps/grampsdb::<uuid>/database.txt                   <- THE FAMILY TREE
    /root/.gramps/grampsdb::<uuid>/name.txt
    /app/thumbnail_cache::<hash>
  changed in place: /app/cache::request_cache/<hash>              <- a request cache, expected

5.2 wishlist

Mounted wishlist_data:/data — again a path the application never writes. The image declares VOLUME /usr/src/app/data and VOLUME /usr/src/app/uploads (Config.Volumes), and upstream's own compose mounts exactly those two, so prod.db was landing in the anonymous volume docker creates for an image VOLUME directive: invisible to ResolveDockerVolumeNames, therefore never backed up, and orphaned by down + up. Now mounts both real paths.

Proof — proofs/wishlist/survive-wishlist-content.json.

before redeploy:  prod.db  inode=47151  size=307200  sha256=f35df23b…
after  redeploy:  prod.db  inode=47151  size=307200  sha256=f35df23b…
same_inode=true   verdict: SURVIVES

Same file object, byte-identical content.

A correction, and the limit of this proof. The generic prover (state/survive2.py) first reported DATA-LOST for wishlist. That was the prover, not the app: its rule counts any change in bytes as a loss, and cannot separate "the app modified its own database on boot" (fine — vaultwarden's SQLite WAL does exactly that) from "the app recreated it" (data gone). The focused check settles it on inode. And the limit: the seeded content step 404'd — the signup endpoint was a guess and wrong — so this is a file-level proof, not a user-row-level one. Byte-identical content means nothing in the database was lost, but no row was independently confirmed present beforehand.

5.3 Both fixes re-checked with the gate that convicted them

The strongest available confirmation is the same instrument reaching the opposite verdict, with its own two-way self-test green on both runs. proofs/fixed-check/:

self-testing the prober (both directions)…
  prober flags the R-156 signature and clears a correct template — trustworthy
gramps-web           CLEAN
wishlist             CLEAN
2 clean · 0 broken · 0 undetermined (of 2 in scope)
volume-persistence gate OK          rc=0

Before the fixes the same gate reported both BROKEN (state/verify.log).

5.4 papra — the fix is proven, and deliberately not applied

Run in the scratch guest only, on a temporary copy of the template (proofs/_papra-fix-proof/papra-fix-proof.json). A real account was created through papra's own API in both legs (sign-up -> HTTP/1.1 200 OK), so real user data was written to the database before each redeploy:

leg	db.sqlite before	after
current papra_data:/app/data	1e9b695a…	5d179bf1…	different — the account is gone
fixed papra_data:/app/app-data	9dd5cf37…	9dd5cf37…	byte-identical — it survives

An incidental confirmation from the same run: under the fixed template /app/data reports present: false. The directory does not exist in the papra image at all — it existed only because the template mounted a volume there. So the volume was not merely unwritable; it was mounted at a path the application has no concept of.

Limit of this proof: the document-upload step did not complete (the follow-up /api/organizations call returned auth.unauthorized — a session-cookie handling problem in the harness, not in papra), so the user data verified is the account, not an uploaded document.

This fix is not committed. §6.1.

---

6. Referred to the operator — not decided here

6.1 papra is deployed. Its fix would strand the data it is meant to protect.

The fix is prepared and deliberately NOT committed.

       volumes:
-      - papra_data:/app/data
+      - papra_data:/app/app-data

Why it is a strand risk. The volume name is unchanged, so no volume is orphaned — but the mount target changes, which makes docker compose up -d recreate the container, and the container's writable layer is where papra's documents and database currently live. Applying this to a running papra destroys exactly the data the change exists to protect.

The honest framing, both halves. That data is already living on borrowed time: an image bump, a RedeployFromEnv, a restore, or any other container recreation destroys it just the same, and no backup has ever contained it. But the fix is what schedules the loss, on a date the customer does not choose. Scaling that decision down is not this session's call.

Note that pushing the template alone destroys nothing: the controller's git-sync updates the two template files and does not restart the stack (CLAUDE.md, deploy contract). The change would take effect at the next deploy. It is still an armed change, which is why it is held.

Which box — determined as far as the fences allow, and no further.

checked	result
hub /apps (read-only)	Papra: 1 deployment, 218 MB avg / 250 MB P95 — it is live somewhere
demo-felhom guest 9201	not there — docker ps -a shows no papra container, docker volume ls no papra volume
hub /configs	boxes currently reporting: c10-soak, demo-felhom, demo-hp (david —, drill-r50 BLOCKED, peti-felhom DOWN, sess-f DOWN)
hub /hosts/<id> and /apps/papra	neither page attributes a deployment to a host

So it is on c10-soak or demo-hp, both of which live on demo-hp hardware that this session is fenced from. c10-soak is the likelier of the two and would make this moot: it is Campaign 10's own rig, its hub customer record reads "Campaign 10 two-storage soak (SCRATCH - delete at teardown)", and papra was deployed on it to find R-156 in the first place — so the "customer data" would be the campaign's own canary. That is a guess, and it is labelled as one. One docker ps on either box settles it, and the other session owns both.

What is asked: confirm which box, and then either (a) it is c10-soak → push the fix, nothing of value is lost, or (b) it is a demo box someone cares about → copy /app/app-data out of the running container first (docker cp papra:/app/app-data <dest>), then push, then copy back into the new volume. No migration is shipped here — §5 of the brief, and inventing one is how this class of defect gets worse.

6.1a — UPDATE at teardown: the deployment disappeared mid-session

Between the start of this run and its teardown, the hub's /configs and /hosts both stopped listing c10-soak — the other session tore Campaign 10's rig down while this work was in flight (its teardown was owed). In the same interval the hub's /apps went from 10 deployed apps to 6, and the four that disappeared are papra, grafana, homebox and rallly — precisely the four apps Campaign 10's audit §A4 names as its discriminator set. papra's single deployment was therefore on c10-soak, as §6.1 suspected, and it is gone.

The fix is still not applied, deliberately. The evidence is strong but it is absence evidence — papra no longer appearing in aggregated telemetry is not the same as a positive observation that no box runs it, and this project's standing rule 3 exists for exactly that distinction. The one box I cannot check is demo-hp, which is fenced this session; the hub says it is reporting normally and shows no papra, but that is the same absence argument one level down.

The asymmetry decides it: applying the fix while a papra instance does exist destroys its documents at the next redeploy and is irreversible; leaving it is a one-line change the operator can push in seconds. So it is left.

What settles it: ssh demo-hp "pct exec 9201 -- docker ps -a --filter name=papra" — one command, on the box this session may not touch. If it returns nothing, apply the diff in §6.1; there is then no strand risk and papra joins gramps-web and wishlist as an ordinary fix.

6.2 Nothing else needed a ruling

Both other BROKEN apps have an unambiguous correct path — each confirmed by two independent sources, never inferred — and neither is deployed anywhere in the fleet, so both were fixed outright (§5). No app was left BROKEN for want of a decision, and no UNDETERMINED app was touched.

---

7. Part 4 — the gate

7.1 Where catalog gates live, and what runs them

They live in app-catalog-felhom.eu/scripts/ as dependency-free Python that exits non-zero on violation, and nothing automated runs them. There is no CI in this repo — no .gitea/workflows, no .github, no drone/woodpecker config; the only YAML outside templates/ is generated customer output. Each gate is invoked by whoever edits the catalog, and CLAUDE.md says when.

That is the honest description, and it is stated rather than glossed because this project's record on the point is unambiguous: R-29 has three orphaned gates and one enforced one, and the enforced one is the only one that has ever stopped anything. This gate is at the same enforcement level as its two siblings — convention, not CI. What would raise it is named in §8 as a proposed register entry; building it would mean touching the controller's sync path, which is out of scope here (§5 fences) and would need a decision about failing a customer's template sync.

The gate could be built here, so it was. It did not have to wait for felhom.eu.

7.2 Why the gate has to be a runtime check — measured, not assumed

A static audit of all 53 composes — every declared top-level volume attached to a service, no anonymous mounts, no host binds outside the felhom roots — reports the catalog completely clean. The single hit is homepage's deliberate read-only /var/run/docker.sock.

It reports papra clean too. papra's compose is well-formed: it declares papra_data and mounts it at /app/data. Nothing in the file is wrong. What is wrong is only observable at runtime.

So a static gate would have been a gate that could never have caught the defect it was written for — coverage that isn't. scripts/check-volume-persistence.py deploys, exercises, and measures.

7.3 What it refuses

Exit 1 — REFUSED, not a warning:

• DATA in a container's writable layer (never persisted, never backed up);
• an app-data mount the app's own uid cannot write (R-156's second leg);
• data in an anonymous volume (never backed up, orphaned by a redeploy);
• a database-signature file in the writable layer whose bytes differ from the image's copy.

Exit 2 — UNDETERMINED, and explicitly not a clean bill of health: a container that never reached running/healthy; nothing written anywhere; a suspect that could not be adjudicated; or — load-bearing — the prober failing its own two-way canary, in which case the gate refuses to report any verdict at all rather than issue a false all-clear.

Exit 0 only when every app in scope is CLEAN.

7.4 The test through the path that actually invokes it

scripts/test_check_volume_persistence.py — 31 tests, no Docker (the prober is injected). They drive check(), the function __main__ calls, not just the verdict logic: a gate whose decision logic is tested but whose entry point is not has shipped inert in this project before (the seam-wiring rule, four recorded instances).

Covered at the entry point: one broken app ⇒ rc 1 and the app is named; all clean ⇒ rc 0; undetermined ⇒ rc 2, never rc 0; broken outranks undetermined; out-of-circulation apps skipped and reported; a prober that flags nothing ⇒ rc 2; a prober that flags everything ⇒ rc 2.

Every rule was red-proofed by breaking it and confirming the suite goes red:

break	suite
score C entries as data again (the calibre-web false positive)	RED
revert the trailing-slash noise normalisation (the crafty false positive)	RED
fold "wrote nothing" into CLEAN	RED
warn instead of refuse (rc 0 on broken)	RED
drop the writable-layer leg (the papra signature)	RED
let an unresolved suspect fall through to CLEAN	RED
drop the base64: prefix	RED
(the deleted all-children-noise rule)	GREEN — which is why it was deleted

Registered in CLAUDE.md (conventions) and REUSE.md (the map), in the same commit.

---

8. Findings for later filing — PROPOSED TEXT, NOT FILED

The register is felhom.eu/documentation/backlog/{ROADMAP,OPEN-ITEMS}.md, and felhom.eu is fenced this session. These are drafts for one session to file, in one place.

Register grep before proposing (read-only, ROADMAP.md + OPEN-ITEMS.md): wishlist -> 1 hit, R-41's deployability gate, unrelated · gramps -> 0 · anonymous -> 2 hits, both about the golden bake, unrelated · ResolveDockerVolumeNames -> 0 · DumpAppVolumes -> 0 · writable layer -> 0 · check-volume-persistence -> 0. Highest filed: R-155. So these start at R-158.

8.0 First, a process finding — R-156 and R-157 are themselves unfiled

grep -c "R-156\|R-157" over both register files returns 0 / 0. They exist only inside audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md. This is the identical failure the brief cites for R-154/R-155 — "minted in a spike doc and never carried across" — recurring immediately, on the very finding this session was commissioned to act on. R-156 is the reason this work happened and it is not in the register. Whoever files the entries below should carry R-156 and R-157 across at the same time.

And it happened again, to this document, while it was being written. The first draft of this section numbered its proposals R-158…R-161 on the strength of a register grep showing R-155 as the highest filed. Mid-session the parallel session pushed audits/SPIKE-recovery-unit-space-2026-08-02.md and audits/CAMPAIGN-10-closeout-2026-08-02.md, both of which use R-158 for an unrelated finding (recovery-unit space ceiling on mp1) — and neither adds it to ROADMAP.md or OPEN-ITEMS.md, which still stop at R-155. So a grep of the register was true when it was run and stale within hours, and two sessions minted the same number for different findings on the same day.

These proposals are therefore renumbered R-159…R-162, and the number to claim should be taken at filing time, not at drafting time. The register being the only source of truth is precisely what is not happening: R-156, R-157 and R-158 are all live in audit documents and none of them is filed.

R-159 — wishlist's data lands in an ANONYMOUS volume: never backed up, orphaned by a redeploy

FIXED HERE (templates/wishlist/docker-compose.yml); filed to record the class.

The image declares VOLUME /usr/src/app/data and VOLUME /usr/src/app/uploads (docker image inspect ghcr.io/cmintey/wishlist:v0.66.0 -> Config.Volumes). The template mounted wishlist_data:/data — a path the application never writes. So prod.db went into the anonymous volume docker creates for an image VOLUME directive.

Why this is worse than an ordinary miss: it survives a restart. ResolveDockerVolumeNames (felhom-controller/internal/appbackup/appdata.go) returns <project>_<name> only for volumes declared in the compose file, so DumpAppVolumes never sees an anonymous one — and compose down + up orphans it and mounts a fresh empty one. Meanwhile wishlist_data was tarred nightly as an empty directory. A store that survives a restart, loses on redeploy, and is never in a backup is harder to notice than papra's, because casual testing finds nothing wrong.

The class, not just the app: any image VOLUME directive at a path the template does not explicitly mount creates unbacked-up storage silently. immich-server has one today at /data (Config.Volumes = {"/data":{}}) — empty when measured, so nothing is at risk now, but anything immich ever writes there is invisible to backup. Suggested standing rule for REUSE.md: a template must mount every path in its image's Config.Volumes, or state why not. Checkable statically only if the image is pulled, which is why it lives in the runtime gate.

R-160 — gramps-web persisted three paths and wrote to none of them

FIXED HERE (templates/gramps-web/docker-compose.yml).

The template mounted /app/data, /app/media and /tmp; /app/data appears nowhere in the image's environment — the application never writes it. Its accounts database (GRAMPSWEB_USER_DB_URI=sqlite:////app/users/users.sqlite) and its family tree (GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb) both landed in the container's writable layer. Upstream's reference compose persists eight paths; this template persisted three, one a phantom.

Severity above papra's: papra loses documents a customer uploaded, which they may still hold elsewhere. gramps-web loses the family tree — the artefact the customer built inside the app, of which no other copy exists by construction.

R-161 — the gate exists, and nothing automated runs it

The check R-156 asks for now exists (scripts/check-volume-persistence.py, §7). Its enforcement level is convention: the catalog repo has no CI of any kind (.gitea/workflows, .github, drone/woodpecker — none), so like its two sibling gates it runs when a person runs it.

This project's own record says that is not enough: R-29 has three orphaned gates and one enforced one, and the enforced one is the only one that has ever stopped anything. Two directions, neither buildable from this repo:

1. CI in the catalog repo — a Gitea Actions runner. Nothing exists to build on today.
2. Controller-side at sync — the controller already validates .felhom.yml at template load (internal/stacks/metadata.go:336-358) and a persistence assertion could ride the same choke point. But that needs a runtime probe on a customer box, and a decision about what a failing sync should do to a customer's working app. Cross-repo, and a design question, not a task.

Interim position, recorded in CLAUDE.md: run it when a template's volumes: block or image tag changes, and at the start of every catalog campaign.

R-162 — docker diff is the only witness, and its failure mode is quiet

A limitation, not a defect. The gate's power comes from docker diff excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay storage driver. On a driver where docker diff is unsupported or lies, the gate degrades to the mount-occupancy and writability legs and would not say so. The canary self-test would catch it (the broken canary would stop reporting BROKEN, and the gate would then refuse to report at all), so it fails closed — but the message it prints would blame the prober rather than the driver.

---

9. Teardown — all three layers

Per felhom.eu/documentation/PROMPT-TEMPLATE.md §13. Layer 3 is the one that has been missed every time (drill-r50, sess-c, sess-d), so it is stated explicitly rather than implied.

Layer 1 — the machine

DONE. pct stop 9301 && pct destroy 9301 --purge → Logical volume "vm-9301-disk-0" successfully removed / purging CT 9301 from related configurations / rc=0. Verified after: pct list shows only 9201, and lvs | grep 9301 returns nothing. Evidence: state/teardown.txt.

Layer 2 — the host

pvesm status before and after, and the space actually returned.

	local-lvm available (KiB)	used
before the run began	258 702 410	29.27 %
end of run, before teardown	169 968 909	53.53 %
after teardown	258 702 410	29.27 %

The space returned exactly to the pre-run baseline — 258 702 410 KiB, identical to the figure recorded before the guest was created. local is unchanged at 26.00 %. Full before/after: state/teardown.txt.

The guest's 120 G volume was thin-provisioned on an over-subscribed pool (pct create warned: sum of thin volume sizes 402 G against a 348 G pool), which is why a disk watchdog ran throughout (state/diskwatch.log) and why the space actually returning is measured rather than assumed.

Layer 3 — the hub

Nothing to remove: this run created no hub record of any kind, by design.

That is an absence of creation, not an absence of checking. The scratch guest ran no felhom-controller and was never enrolled (§1), precisely so no customer or appliance record could exist to orphan. The hub was touched read-only throughout — /apps, /apps/papra, /configs, /hosts, /hosts/<id>, over the ClusterIP with Basic auth — to answer "is papra deployed anywhere?" (§6.1).

Confirmed after teardown against the live hub: /configs contains no entry for this run (catalog-persist / 9301: absent), and /hosts lists only demo-felhom-8363b5, demo-hp-bb76ea, drill-r50-0a4f9a, sess-f-2670b5. The list is two shorter than at the start of the run — c10-soak was removed by the other session mid-run, not by this one (§6.1a).

The scratch host paths the gate creates for ${HDD_PATH} / ${IMPORT_PATH} (/srv/felhom-gate/…) and the felhom-volgate-canary:1 image existed only inside the guest, so both go with layer 1.

---

10. What did not happen, and why

• papra was not fixed. It is deployed, and the fix would destroy the data it protects (§6.1). Prepared, proven, held. The brief says stop rather than ship a migration; no migration was written.
• No data migration of any kind was shipped, for papra or anything else.
• No UNDETERMINED app was fixed. Seven remain undecided (§4.2) and none was nudged toward CLEAN to tidy the table. Four need a deeper exercise than a first-boot HTTP request (a completed setup wizard, a posted paste, a real import); three could not be started at all.
• glance was not fixed even though its defect is understood (it crash-loops for want of a seeded /app/config/glance.yml). It is a deployability defect, not a persistence one, already recorded as pre-existing in Campaign 7 §6.2 — outside this brief's scope.
• plant-it was not touched. Its image does not resolve; it is lifecycle: abandoned for exactly that reason.
• Nothing outside the catalog templates was changed. The controller-side observations (ResolveDockerVolumeNames cannot see anonymous volumes; ParseComposeClassifiableBinds cannot see named volumes at all) are findings for §8, not edits.
• No commit to felhom.eu, no register edit, nothing on demo-hp. The other session owns all three. Every R-number in §8 is proposed text.
• No hub-side state was created. The hub was read only — /apps, /configs, /hosts, /apps/papra — via the ClusterIP with Basic auth. Nothing to clean up, so nothing recorded as a command (R-131's four orphans).
• Guest 9201 on demo-felhom was read, not touched — docker ps -a and docker volume ls, to establish that papra is not deployed there.
• claude-in-chrome was not used — unavailable on DooPlex. Every observation is endpoint-level or filesystem-level, and each is named where it is used.
• No secret value was written anywhere. The probe records env var KEYS only; generated credentials are minted per-run and discarded with the container.

---