app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/README.md

# Catalog persistence sweep — does every app's data land in a persisted volume?

**Run:** 2026-08-02 · **Repo:** `app-catalog-felhom.eu` (only) · **Class:** implementation —
detection first, fixes second.

Campaign 10 found that papra stores its database somewhere the backup never looks (R-156). papra is
not the interesting part. **Nothing anywhere checked that the folder a template preserves is the
folder the app writes to**, across 53 templates. This is what that check found.

> **Parallel-session fence, observed.** Another session was running the M-band RTO measurement on
> `demo-hp`, writing to `felhom.eu`. This session ran on `demo-felhom` and committed only to
> `app-catalog-felhom.eu`. **No commit to `felhom.eu`, no register edit, nothing on `demo-hp`.**
> All register entries below are proposed text for later filing by one session, in one place.

---

## 1. Venue and baselines

| | |
|---|---|
| Host | **`demo-felhom`** (N100), Tier 0 — `ssh felhom-pve` (Tailscale `100.70.170.35`) |
| Guest | **LXC 9301 `catalog-persist`**, created for this run and destroyed at teardown (§9) |
| Guest spec | Debian 13, 4 cores, 6 GB + 2 GB swap, `local-lvm:120`, `nesting=1,keyctl=1`, DHCP `192.168.0.22` |
| Reach | `ssh felhom-pve "pct exec 9301 -- …"`. **No direct SSH**: the N100 sits on a remote LAN whose `192.168.0.0/24` collides with DooPlex's, so `192.168.0.22` is unroutable from here (`No route to host`) |
| Docker | 29.7.1, compose v5.3.1 |
| Untouched | guest **9201** on `demo-felhom` (read-only `docker ps`/`volume ls` only), `drill-r50`, Peti's cluster, both boxes' backup configuration, **everything on `demo-hp`** |

### Baselines, read fresh

| What | Value | How |
|---|---|---|
| `app-catalog-felhom.eu` HEAD | **`4252121`** = `origin/main`, clean at start | `git rev-parse` |
| Templates | **53** | `ls templates \| wc -l` |
| `felhom.eu` HEAD (read-only, for citation) | **`7ba7c2a`** | `git rev-parse` — **not committed to** |
| felhom-controller `main` | **v0.188.0**, HEAD `4115e88` | `CHANGELOG.md` top |
| Hub | 0.86.0 | live `/apps` footer |

### Why the scratch guest runs no controller — stated, because the brief asks for its version

There is **no felhom-controller on 9301, deliberately.** The property under test is a property of
the *template*, and the controller's role in it is exactly one thing: it resolves the `${…}`
variables and runs `docker compose up -d` (`felhom-controller/internal/stacks/deploy.go:374`,
`:551-582`). It does not rewrite the compose file, so a plain Docker host reproducing that env
substitution exercises the identical code path in the app.

Standing a controller up would have meant enrolling a box with the hub — creating hub-side state
this session is told not to create, and which R-131 records four orphaned instances of. The
substitution the controller performs is instead **mirrored exactly**, including the trap that
`generateValue` prefixes `base64key` with `base64:` (`deploy.go:904`); getting that wrong cost one
false verdict before it was caught (§3.4).

---

## 2. Part 0 — had this ground been covered?

**No. Neither Campaign 7 nor any gate, `.felhom.yml` field, or CI check tests this property.**
Each negative below names what was searched.

### 2.1 Campaign 7 tested deployability and health, not persistence

`felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md:38-51` states its method:
per-container terminal health verdict, healthcheck-binary audit, HTTP probe through the real
Traefik ingress, login where scriptable, and a log scan. Its result matrix (`:114`) carries the
columns *pin · MAJOR · deploy · health · http · logs · settle*. **There is no persistence column,
and no step in the method reads where data landed.**

The proof that this is a gap rather than a technicality is in that same matrix: **papra passed it**
— `deploy ok`, `health **fixed** ×2`, `http 200`, `logs clean` (`:151`) — while its database was
being written to a directory no backup would ever contain. Campaign 7 also demonstrated the
narrower fact that a healthcheck can be a lie (C7-F4, `:210-224`); this sweep demonstrates that a
*truthful* healthcheck is still not evidence about data.

**What changed since:** nothing that would have closed it. Between Campaign 7 and today the catalog
gained the resolvability gate (2026-07-21), `lifecycle:` (2026-07-21) and R-75's `data_paths:`
(2026-07-26) — image currency, circulation status, and folder *labelling* respectively.

### 2.2 No gate asserts a volume ⟷ data-path relationship. Searched:

| Where | What was searched | Result |
|---|---|---|
| `app-catalog-felhom.eu/scripts/` | full listing | exactly two gates, `check-image-pins.py` (syntactic tag check) and `check-image-resolvable.py` (does the tag still exist) — **both about images, neither about volumes** |
| catalog repo CI | `.gitea/workflows`, `.github`, drone/woodpecker/ci config; every non-template YAML | **none exists.** Catalog gates are convention-run, invoked by whoever edits the catalog — this matters for §7 |
| all four repos | `grep -rniE "volume.*(persist\|data.?path\|writable.layer)\|writable.layer\|docker diff\|persistence.(gate\|check)" --include=*.py --include=*.go --include=*.sh` | no check of this kind; the only hits are comments and a test fixture name |
| all gate scripts, four repos | `ls felhom-controller/scripts/*.py felhom.eu/scripts/*.py app-catalog-felhom.eu/scripts/*.py felhom-agent/scripts/*` | 11 gates. The closest by name, `felhom-controller/scripts/docker_run_volume_path_gate.py`, is unrelated: it allowlists `docker … -v` arguments in **controller Go code** so a controller-local host path cannot be mounted from inside a container (v0.125.0 scenario D) |
| `.felhom.yml` schema | `backup:`, `data_paths:`, `lifecycle:`, `deploy_fields:` | none expresses "the app writes here" |

### 2.3 The two validators that look closest cannot see this class

`ValidateBackupSpec` (`felhom-controller/internal/appbackup/classify.go`) and `ValidateDataPaths`
(`internal/stacks/datapaths.go:58`) both reject an entry matching no compose bind — real checks, but
of **declaration consistency**, not of runtime behaviour. And both operate on
`ParseComposeClassifiableBinds` (`internal/stacks/classify_binds.go`), which keeps only mounts whose
host side passes `classifyRoot` — i.e. paths under `${USERDATA_PATH}` / `${HDD_PATH}` /
`${IMPORT_PATH}`. **Named volumes are skipped entirely.**

That is decisive for papra, which has **zero binds**: every one of these validators is vacuously
satisfied by it. It is also decisive for scale — **39 of 53 templates declare no felhom-root bind at
all**, so for three-quarters of the catalog the entire data-protection story is `DumpAppVolumes`
(`internal/backup/backup.go:543`) tarring a named volume, and nothing checks that the volume has
anything in it.

### 2.4 R-156's evidence, read before building

`felhom.eu/documentation/tests/campaign10-evidence-2026-07-31/r156-papra-volume.txt` — five numbered
observations: the volume mounted at `/app/data`; `uid=999(nonroot)` with `/app/data` `root root`;
`touch: cannot touch '/app/data/.w': Permission denied`; the volume empty but for a canary the
campaign wrote **as root**; and `db.sqlite` 475136 B in the container's writable layer at
`/app/app-data/`. The detector was built against those observations, and it reproduces all of them.

---

## 3. The detector

`scripts/check-volume-persistence.py`. **The detector and the gate are the same program** — the
sweep below was produced by running the committed gate, so the durable artifact is the thing that
generated the evidence rather than a paper copy of it.

Per app: deploy from the template → exercise it into writing data → compare where the data landed
against what the compose mounts.

### 3.1 What is observed, and why nothing uses `docker exec`

| | signal | how |
|---|---|---|
| M | mount inventory, by persistence class | `docker inspect .Mounts` |
| D | the writable layer | `docker diff` — Docker **excludes every mounted path** from it, so an entry here is definitionally in no volume and no bind |
| U | the app's real uid/gid | `/proc/<State.Pid>/status`, **not** `Config.User` — which is empty for papra even though it drops to 999 |
| W | writability of each mount by that uid | host-side `stat` of the mount source |
| E | mount occupancy | host-side walk of the mount source |

**No `docker exec` anywhere.** Campaign 7 §1.1 recorded that `docker exec` writes its OCI error to
STDOUT, so a *missing* binary printed `executable file not found` and read as **present** — which
made that campaign's entire healthcheck audit report every app honest. Everything here is read from
`docker inspect`, `docker diff`, `/proc` and the host filesystem, so a distroless or shell-less
image is observed exactly like any other.

### 3.2 Mount classes — the second defect class this exposes

| class | persists a redeploy? | backed up? |
|---|---|---|
| `named-declared` — in the compose `volumes:` block | yes | **yes** |
| `anonymous` — an image `VOLUME` or unnamed mount (64-hex docker name) | survives a *restart* | **NO** |
| `bind` | yes | per the backup spec |
| `tmpfs` | no | no |

The anonymous case is deceptive precisely because it survives a restart. `ResolveDockerVolumeNames`
(`felhom-controller/internal/appbackup/appdata.go`) returns `<project>_<name>` only for volumes
**declared in the compose file**, so an anonymous volume is invisible to `DumpAppVolumes` — and a
`compose down` + `up` orphans it and creates a fresh one. Data there is unbacked-up *and* lost on
redeploy, with no symptom until someone looks.

### 3.3 What counts as "exercised", and what counts as data

**Exercised.** Every container reaches `running`, and `healthy` where a healthcheck exists; then an
HTTP request the app's *own router* answers. Any status code — including 3xx/4xx/5xx — proves the
request reached application code; a connection refusal (`000`) does not. If that produced no writes
anywhere, a **second-chance pass** walks a wider path list following redirects, so a first-run
wizard is actually reached, and waits again. **An app that still writes nothing is UNDETERMINED,
never CLEAN** — health-check-passing is not data-writing, and conflating the two is what let papra
look fine.

**Data versus noise.** Three rules, deliberately explicit rather than buried in a heuristic:

1. **Noise** — `/tmp`, `/run`, `/var/log`, `/var/cache`, package-manager state, `__pycache__`,
   `node_modules`, `.cache`, `*.pid`, `*.sock`, `*.log`, `*.pyc`, and the per-container files
   Docker itself writes (`/etc/hosts`, `/etc/resolv.conf`, …).
2. **Database signature** — a filename that *proves* a database is in that directory
   (`*.sqlite*`, `PG_VERSION`, `ib_logfile*`, `data.mdb`, `dump.rdb`, `MANIFEST-*`, `data.ms`, …).
   The strongest signal, and it needs no path heuristics.
3. **Data token** — a path saying "app state" (`data`, `storage`, `uploads`, `library`, `documents`,
   `var/lib/postgresql`, …), used only where rule 2 is silent.

**`A` versus `C` is the whole difficulty**, and rule 3 is where judgement actually lives:

- `A` (added) — the app **created** this file; it exists only in the writable layer. papra's
  `db.sqlite` is an `A`. **DATA is decided from `A` entries only.**
- `C` (changed) — a file that **ships in the image** was touched. Usually a chown sweep and
  entirely benign. A `C` on a *database-signature* file is genuinely ambiguous, because an app
  writing into a DB shipped in its image looks identical. Those are held as **SUSPECT** and settled
  **by bytes**: the file is copied out of the running container and out of a pristine container
  created from the same image, and compared. Identical → a chown, recorded and dismissed.
  Different → a real write into an image-layer file, which is papra's defect with a different verb.
  Unreadable → **UNDETERMINED**, never folded into CLEAN.

### 3.4 Red-proofs — both directions, before the sweep, and again on every run

| | claim | result |
|---|---|---|
| **positive** | it flags **papra** | **BROKEN**, reproducing R-156's evidence line for line: `/app/data` EMPTY and `NOT writable by the app's own uid=999 gid=999`, `db.sqlite` in the writable layer at `/app/app-data/db` |
| **negative** | it stays quiet on a known-good app | **vaultwarden CLEAN** — and *demonstrably* good, not merely unflagged (below) |

The negative is a **proven** negative. `state/survive2.py` deployed vaultwarden, let it write, ran a
real `docker compose down` + `up -d` (no `-v` — that is a redeploy; `down --volumes` is the
uninstall path, Campaign 7 §2), and re-fingerprinted every file by sha256 **and inode**:

```
vaultwarden   SURVIVES   before=4 after=4 survived=2 changed=2 lost=0
  survived: /data::db.sqlite3     <- the database itself, byte-identical, SAME INODE
            /data::rsa_key.pem
  changed : /data::db.sqlite3-shm , /data::db.sqlite3-wal   <- SQLite sidecars, expected on restart
```

**The self-test runs on every invocation, in both directions.** Before the gate reports anything it
puts two canary templates through the same prober and refuses to issue a verdict unless it calls
one BROKEN and the other CLEAN. Both are built from a purpose-made image reproducing papra's exact
ownership shape — `/app/data` root-owned, `/app/app-data` owned by the app's uid — and **differ only
in which path the volume is mounted at**, so the canary pair is a live demonstration of R-156 *and*
of its fix, every run. A detector that flags nothing is worse than none: it converts an unexamined
catalog into a documented-clean one.

### 3.5 Seven defects in the detector, every one found by adjudicating a flag rather than trusting it

Recorded because each would have put a wrong number in the matrix, and **six of the seven made a
correct app look broken** — the failure mode that gets a gate switched off and then protects nothing.

| # | defect | what it did |
|---|---|---|
| 1 | **`base64key` lost the controller's `base64:` prefix** (`deploy.go:904`) | bookstack's Laravel `APP_KEY` was invalid, it served 500s, read UNDETERMINED. With the prefix: CLEAN. A harness bug wearing an app defect's clothes — the class Campaign 7 §1.1 and Campaign 10 §4d both record |
| 2 | **`C` entries scored as data** | a linuxserver.io entrypoint re-owning its application tree produced **1305 `C` entries** for calibre-web, 92 of them PNGs under `cps/static/css/images/**`. Called BROKEN. Led to the `A`/`C` split and the byte-level adjudicator (§3.3) |
| 3 | **a cache *directory* escaped the noise filter** | `docker diff` lists directories, so the bytecode cache appeared as a bare `…/crafty/config/__pycache__` entry while its `.pyc` children were filtered by suffix — the directory then being the only surviving entry under a path containing `config`. crafty-controller called BROKEN four times over with its data correctly placed. Fixed by normalising with a trailing slash, covering `node_modules`, `.cache`, `.git` at once |
| 4 | **`postgresql.conf` listed as a database signature** | it is a *config* file the postgres entrypoint writes at init. immich called BROKEN with its PGDATA correctly in its volume, **1831 files** |
| 5 | **path tokens convicted on their own** | onlyoffice unpacks its OWN static assets into the writable layer at first boot — plugin icons, slide-theme `media/`, `web-apps/apps/api/documents/api.js`, **2560 added entries** — while its real data mount received data normally. Rule 3 was **demoted to report-only** |
| 6 | **the structural check asked per container, not per app** | fired on **docmost, immich and claper** — three correct apps, identical shape: the app container's only volume is for user uploads and legitimately empty on a fresh install, while all real state sits in the sibling database container's volume (1540 / 1833 / 1470 files) |
| 7 | **mount rules never consulted the noise list** | the prefixes are written `/run/`, so a bare mount target `/run` matched none of them. privatebin's image declares `VOLUME /run`; the anonymous volume docker made there holds `nginx.pid`, `php-fpm.sock` and s6 fifos, and read as 14 files of unbacked-up customer data |

Plus one that was not a false verdict but a **reporting** defect, and arguably the worst of the set:

8. **A finding was being swallowed by a louder one.** gramps-web's structural finding — its *family
   tree* landing outside every mount, the entire point of the app — went into the undetermined
   bucket, and that bucket is discarded whenever a BROKEN leg exists. Its lesser defect (the
   accounts database) convicted first and hid it. A finding that disappears because a different
   finding won belongs to the same family as an absent log line read as health, and the fix is that
   the structural finding is now reported whatever the verdict.

**The pattern, which is the real lesson.** Every false positive came from a rule that identified data
by its **name** rather than by evidence of what it is: `C` assumed to be a write, `postgresql.conf`
assumed to be a database, `media/` and `config/` assumed to be customer content, `/run` assumed to be
storage because it was a volume. The rules that never needed correcting are the ones grounded in
something checkable — a database file the app **created**, a mount its own uid **cannot write**,
bytes that **differ** from the image's copy, and *did anything land in a preserved folder at all*.

**Restarts, stated plainly.** The sweep was restarted from the first app **twice** rather than merge
a matrix built by more than one version of the rules — the call Campaign 10 made restarting run 2b
from cycle 1. Defects 6, 7 and 8 were found after the final pass had begun; all three are in
`classify()`, which reads only what the probe already captured, so those verdicts were **re-derived**
rather than re-measured (`state/reclassify.py`), and then **every non-CLEAN and every flipped app was
re-run live end-to-end** through the final gate so no reported verdict rests on post-processing
(§5.3). Six verdicts changed on re-derivation: claper, docmost, immich, sparkyfitness and zipline
UNDETERMINED→CLEAN, privatebin BROKEN→UNDETERMINED.

**And one rule was written, red-proofed, and deleted.** A second "are all this directory's children
noise?" filter looked like a sensible belt to defect 3's brace. Its red-proof came back **green**,
which is the tell: `e["added"]` can only ever contain entries that already passed `is_noise`, so the
check is always False — dead code wearing the costume of a safeguard, exactly the trap `CLAUDE.md`
catalogues. It is gone, the code says why it is not there, and
`test_cache_directories_are_filtered_at_ENTRY_level` pins the mechanism that actually does the work.

## 4. The sweep — all 53 templates

**43 CLEAN · 3 BROKEN · 7 UNDETERMINED.** UNDETERMINED is counted separately and is never folded
into CLEAN; each one's reason is in §4.2.

Evidence per app: `evidence/<app>/probe.json` — the full probe (mount inventory with occupancy and
writability, the app's real uid, the writable-layer rollup, the adjudicated suspects) plus the
verdict and its reasons. Runner log: `state/gate.log`.

**How to read the mount column:** `[vol]` a named volume declared in the compose (persisted AND
backed up) · `[ANON]` an anonymous volume from the image's own `VOLUME` directive (persisted across
a restart, **never backed up**, orphaned by a redeploy) · `[bind]` a host bind · `,EMPTY` the mount
received nothing.

### 4.1 The matrix

| app | verdict | what the template mounts | where the app actually wrote | mount not writable by app uid |
|---|---|---|---|---|
| `gramps-web` | **BROKEN** | /app/data[vol,EMPTY], /app/media[vol,EMPTY], /tmp[vol,EMPTY] | **/app/users (writable layer)** | — |
| `papra` | **BROKEN** | /app/data[vol,EMPTY] | **/app/app-data/db (writable layer)** | /app/data |
| `wishlist` | **BROKEN** | /data[vol,EMPTY], /usr/src/app/uploads[ANON,EMPTY], /usr/src/app/data[ANON] | /usr/src/app/data(1) | — |
| `bentopdf` | **UNDETERMINED** | — | nothing | — |
| `glance` | **UNDETERMINED** | /app/config[vol,EMPTY] | nothing | — |
| `plant-it` | **UNDETERMINED** | — | nothing | — |
| `privatebin` | **UNDETERMINED** | /tmp[ANON,EMPTY], /var/lib/nginx/tmp[ANON,EMPTY], /srv/data[vol,EMPTY], /run[ANON] | /run(14) | — |
| `recipe-importer` | **UNDETERMINED** | /data[vol,EMPTY] | nothing | — |
| `uptime-kuma` | **UNDETERMINED** | /app/data[vol,EMPTY] | nothing | — |
| `wanderer` | **UNDETERMINED** | /app/uploads[vol,EMPTY], /pb_data[vol], /data/plugins[vol,EMPTY], /meili_data[vol] | /pb_data(6), /meili_data(11) | — |
| `actualbudget` | **CLEAN** | /data[vol] | /data(2) | — |
| `adventurelog` | **CLEAN** | /code/media[vol], /var/lib/postgresql/data[vol] | /code/media(251), /var/lib/postgresql/data(2287) | — |
| `audiobookshelf` | **CLEAN** | /config[vol], /metadata[vol], /audiobooks[bind,EMPTY], /podcasts[bind,EMPTY] | /config(16), /metadata(1) | — |
| `bookstack` | **CLEAN** | /config[vol], /var/lib/mysql[vol] | /config(20), /var/lib/mysql(291) | — |
| `calcom` | **CLEAN** | /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(2261) | — |
| `calibre-web` | **CLEAN** | /config[vol], /cwa-book-ingest[bind,EMPTY], /calibre-library[bind] | /config(11), /calibre-library(3) | — |
| `claper` | **CLEAN** | /app/priv/static/uploads[vol,EMPTY], /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(1470) | — |
| `code-server` | **CLEAN** | /config[vol] | /config(6) | — |
| `crafty-controller` | **CLEAN** | /crafty/servers[vol,EMPTY], /crafty/app/config[vol], /crafty/import[vol,EMPTY], /crafty/backups[vol,EMPTY], /crafty/logs[vol] | /crafty/app/config(16), /crafty/logs(8) | — |
| `docmost` | **CLEAN** | /app/data/storage[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol] | /var/lib/postgresql/data(1540), /data(3) | — |
| `emby` | **CLEAN** | /config[vol], /media[bind] | /config(46), /media(3) | — |
| `ghost` | **CLEAN** | /var/lib/ghost/content[vol] | /var/lib/ghost/content(7) | — |
| `gitea` | **CLEAN** | /data[vol] | /data(8) | — |
| `gokapi` | **CLEAN** | /app/data[vol], /app/config[vol] | /app/data(4), /app/config(1) | — |
| `grafana` | **CLEAN** | /var/lib/grafana[vol] | /var/lib/grafana(543) | — |
| `home-assistant` | **CLEAN** | /config[vol] | /config(21) | — |
| `homebox` | **CLEAN** | /data[vol] | /data(3) | — |
| `homepage` | **CLEAN** | /app/config[vol], /var/run/docker.sock[bind,EMPTY] | /app/config(3) | — |
| `immich` | **CLEAN** | /cache[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol], /usr/src/app/upload[bind], /external/photos[bind,EMPTY], /data[ANON,EMPTY] | /var/lib/postgresql/data(1833), /data(3), /usr/src/app/upload(6) | — |
| `jellyfin` | **CLEAN** | /config[vol], /cache[vol], /media[bind] | /config(20), /cache(2), /media(3) | — |
| `kimai` | **CLEAN** | /opt/kimai/var[vol], /var/lib/mysql[vol] | /opt/kimai/var(2547), /var/lib/mysql(279) | — |
| `komga` | **CLEAN** | /config[vol], /data[bind,EMPTY] | /config(11) | — |
| `mealie` | **CLEAN** | /app/data[vol] | /app/data(5) | — |
| `n8n` | **CLEAN** | /home/node/.n8n[vol] | /home/node/.n8n(7) | — |
| `navidrome` | **CLEAN** | /data[vol], /music[bind,EMPTY] | /data(6) | — |
| `nextcloud` | **CLEAN** | /var/www/html/data[bind], /var/www/html[vol], /var/lib/mysql[vol], /data[vol] | /var/www/html/data(69), /var/www/html(4001), /var/lib/mysql(311), /data(3) | — |
| `onlyoffice` | **CLEAN** | /var/www/onlyoffice/Data[vol], /var/log/onlyoffice[vol], /usr/share/fonts/truetype/custom[ANON,EMPTY], /var/lib/onlyoffice[ANON,EMPTY] | /var/www/onlyoffice/Data(3), /var/log/onlyoffice(11) | — |
| `opengist` | **CLEAN** | /opengist[vol] | /opengist(9) | — |
| `outline` | **CLEAN** | /var/lib/outline/data[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol,EMPTY] | /var/lib/postgresql/data(1655) | — |
| `paperless-ngx` | **CLEAN** | /var/lib/postgresql/data[vol], /data[vol], /usr/src/paperless/export[bind,EMPTY], /usr/src/paperless/data[vol], /usr/src/paperless/media[bind], /usr/src/paperless/consume[bind,EMPTY] | /var/lib/postgresql/data(1927), /data(3), /usr/src/paperless/data(9), /usr/src/paperless/media(1) | — |
| `plex` | **CLEAN** | /media[bind], /config[vol], /transcode[vol,EMPTY] | /media(3), /config(82) | — |
| `radarr` | **CLEAN** | /config[vol], /media/movies[bind,EMPTY], /downloads[bind,EMPTY] | /config(11) | — |
| `rallly` | **CLEAN** | /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(1580) | — |
| `romm` | **CLEAN** | /redis-data[ANON,EMPTY], /romm[ANON,EMPTY], /romm/library[bind,EMPTY], /romm/resources[bind,EMPTY], /romm/config[vol], /var/lib/mysql[vol], /data[vol] | /romm/config(1), /var/lib/mysql(271), /data(4) | — |
| `seerr` | **CLEAN** | /app/config[vol] | /app/config(11) | — |
| `sonarr` | **CLEAN** | /media/tv[bind,EMPTY], /downloads[bind,EMPTY], /config[vol] | /config(11) | — |
| `sparkyfitness` | **CLEAN** | /var/lib/postgresql/data[vol], /app/SparkyFitnessServer/uploads[vol,EMPTY], /app/SparkyFitnessServer/backup[vol,EMPTY] | /var/lib/postgresql/data(1815) | — |
| `tandoor` | **CLEAN** | /opt/recipes/staticfiles[vol], /opt/recipes/mediafiles[vol,EMPTY], /var/lib/postgresql/data[vol] | /opt/recipes/staticfiles(3047), /var/lib/postgresql/data(1984) | — |
| `termix` | **CLEAN** | /app/data[vol] | /app/data(4) | — |
| `vaultwarden` | **CLEAN** | /data[vol] | /data(4) | — |
| `vikunja` | **CLEAN** | /app/vikunja/files[vol,EMPTY], /db[vol] | /db(3) | — |
| `wger` | **CLEAN** | /home/wger/db[vol], /home/wger/media[vol,EMPTY] | /home/wger/db(1) | — |
| `zipline` | **CLEAN** | /zipline/uploads[vol,EMPTY], /zipline/public[vol,EMPTY], /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(1357) | — |

### 4.2 UNDETERMINED — seven, each with its reason

**None of these is a pass.** Four are apps that genuinely wrote nothing, which is a limit of the
exercise rather than a property of the template; three could not be started at all.

| app | why it is undetermined | is this a template defect? |
|---|---|---|
| `bentopdf` | Declares **zero volumes** and wrote nothing. Stateless by design — a browser-side PDF tool. | No — but the gate cannot distinguish "stateless" from "not exercised hard enough", and saying CLEAN would claim a check that was never made. |
| `uptime-kuma` | Healthy, 302 to `/setup`, **0 entries in `docker diff`**, volume empty. v2 creates its database only when the setup wizard is submitted. | Unknown. Needs a completed wizard to decide. |
| `privatebin` | Wrote nothing to `/srv/data`; no paste was ever posted. (Its `/run`, `/tmp`, `/var/lib/nginx/tmp` anonymous volumes are runtime state — see §3.5.6.) | Unknown. Needs a posted paste to decide. |
| `recipe-importer` | Its one mount `/data` stayed empty and the only writes were at `/`. | Unknown. Needs a real import to decide. |
| `glance` | **Crash-loop** (`restarting`, exit 1). It needs `/app/config/glance.yml` and the template mounts an empty config volume without seeding one. | Pre-existing and already recorded — Campaign 7 §6.2 proved the same failure on the pre-campaign pin. Not a persistence finding; the app never ran. |
| `plant-it` | `pull access denied for msdeluise/plant-it, repository does not exist` — no container was created. | No. The app is `lifecycle: abandoned` for exactly this reason (CHANGELOG 2026-07-21); its dead image is the expected end state. |
| `wanderer` | Container `unhealthy`. Its `/pb_data` (6) and `/meili_data` (11) did receive data, so the app is not obviously broken — but a verdict on an unhealthy app would be a verdict on a half-started one. | Unknown. Needs a healthy deploy to decide. |

### 4.3 What the CLEAN column actually establishes

For the 43 CLEAN apps the claim is bounded, and worth stating precisely: **the data each app wrote
during a first-boot-plus-one-HTTP-request exercise landed inside a mount the template preserves,
none of it landed in the writable layer, and every mount was writable by the app's own uid.** It is
not a claim that every future write will land there — an app that stores uploads correctly but
writes, say, a search index elsewhere on first use would still read CLEAN here. That is the
residual gap, and it is what the periodic re-run in §7.1 exists for.



## 5. Fixes — two made, each with a survives-a-redeploy proof

Every corrected path is confirmed by **two independent sources**, never inferred from a directory
name: the shipped image's own environment/`Config.Volumes`, and upstream's reference compose.
Guessing at a data path is how this defect class was created.

Neither app is deployed anywhere in the fleet (hub `/apps` lists 10 deployed apps; neither is among
them), so neither fix strands anything. **papra is deployed and is therefore NOT fixed** — §6.1.

### 5.1 `gramps-web`

Mounted `/app/data`, `/app/media`, `/tmp`. **`/app/data` appears nowhere in the image's environment**
— the application never writes it. Now mounts the eight paths the image itself names, matching
upstream's reference compose:

| volume | path | what lives there |
|---|---|---|
| `gramps_users` | `/app/users` | `GRAMPSWEB_USER_DB_URI` — the accounts database |
| `gramps_db` | `/root/.gramps/grampsdb` | `GRAMPS_DATABASE_PATH` — **the family tree** |
| `gramps_index` | `/app/indexdir` | `GRAMPSWEB_SEARCH_INDEX_DB_URI` |
| `gramps_media` | `/app/media` | `GRAMPSWEB_MEDIA_BASE_DIR` (was already correct) |
| `gramps_thumb_cache` · `gramps_cache` · `gramps_secret` · `gramps_tmp` | | caches, exports, secret |

**Proof — `proofs/gramps-web/survive-gramps-web.json`.** Deploy → exercise → `docker compose down`
(no `-v`; that is a redeploy, `down --volumes` is the uninstall path) → `up -d` → re-fingerprint by
sha256 **and inode**:

```
gramps-web   SURVIVES   before=5 after=5 survived=4 changed=1 lost=0
  survived (byte-identical, SAME INODE):
    /app/users::users.sqlite                                      <- the accounts database
    /root/.gramps/grampsdb::<uuid>/database.txt                   <- THE FAMILY TREE
    /root/.gramps/grampsdb::<uuid>/name.txt
    /app/thumbnail_cache::<hash>
  changed in place: /app/cache::request_cache/<hash>              <- a request cache, expected
```

### 5.2 `wishlist`

Mounted `wishlist_data:/data` — again a path the application never writes. The image declares
`VOLUME /usr/src/app/data` and `VOLUME /usr/src/app/uploads` (`Config.Volumes`), and upstream's own
compose mounts exactly those two, so `prod.db` was landing in the **anonymous** volume docker
creates for an image `VOLUME` directive: invisible to `ResolveDockerVolumeNames`, therefore never
backed up, and orphaned by `down` + `up`. Now mounts both real paths.

**Proof — `proofs/wishlist/survive-wishlist-content.json`.**

```
before redeploy:  prod.db  inode=47151  size=307200  sha256=f35df23b…
after  redeploy:  prod.db  inode=47151  size=307200  sha256=f35df23b…
same_inode=true   verdict: SURVIVES
```

Same file object, byte-identical content.

> **A correction, and the limit of this proof.** The generic prover (`state/survive2.py`) first
> reported `DATA-LOST` for wishlist. That was the prover, not the app: its rule counts any change
> in bytes as a loss, and cannot separate "the app modified its own database on boot" (fine —
> vaultwarden's SQLite WAL does exactly that) from "the app recreated it" (data gone). The focused
> check settles it on inode. **And the limit:** the seeded content step 404'd — the signup endpoint
> was a guess and wrong — so this is a **file-level** proof, not a user-row-level one. Byte-identical
> content means nothing in the database was lost, but no row was independently confirmed present
> beforehand.

### 5.3 Both fixes re-checked with the gate that convicted them

The strongest available confirmation is the same instrument reaching the opposite verdict, with its
own two-way self-test green on both runs. `proofs/fixed-check/`:

```
self-testing the prober (both directions)…
  prober flags the R-156 signature and clears a correct template — trustworthy
gramps-web           CLEAN
wishlist             CLEAN
2 clean · 0 broken · 0 undetermined (of 2 in scope)
volume-persistence gate OK          rc=0
```

Before the fixes the same gate reported both BROKEN (`state/verify.log`).

### 5.4 papra — the fix is proven, and deliberately not applied

Run in the scratch guest only, on a temporary copy of the template
(`proofs/_papra-fix-proof/papra-fix-proof.json`). **A real account was created through papra's own
API in both legs** (`sign-up -> HTTP/1.1 200 OK`), so real user data was written to the database
before each redeploy:

| leg | `db.sqlite` before | after | |
|---|---|---|---|
| **current** `papra_data:/app/data` | `1e9b695a…` | `5d179bf1…` | **different — the account is gone** |
| **fixed** `papra_data:/app/app-data` | `9dd5cf37…` | `9dd5cf37…` | **byte-identical — it survives** |

An incidental confirmation from the same run: under the fixed template `/app/data` reports
`present: false`. **The directory does not exist in the papra image at all** — it existed only
because the template mounted a volume there. So the volume was not merely unwritable; it was
mounted at a path the application has no concept of.

**Limit of this proof:** the document-upload step did not complete (the follow-up
`/api/organizations` call returned `auth.unauthorized` — a session-cookie handling problem in the
harness, not in papra), so the user data verified is the account, not an uploaded document.

**This fix is not committed.** §6.1.

---

## 6. Referred to the operator — not decided here

### 6.1 papra is deployed. Its fix would strand the data it is meant to protect.

**The fix is prepared and deliberately NOT committed.**

```diff
       volumes:
-      - papra_data:/app/data
+      - papra_data:/app/app-data
```

**Why it is a strand risk.** The volume *name* is unchanged, so no volume is orphaned — but the
mount target changes, which makes `docker compose up -d` **recreate the container**, and the
container's writable layer is where papra's documents and database currently live. Applying this to
a running papra destroys exactly the data the change exists to protect.

**The honest framing, both halves.** That data is already living on borrowed time: an image bump, a
`RedeployFromEnv`, a restore, or any other container recreation destroys it just the same, and no
backup has ever contained it. But the fix is what *schedules* the loss, on a date the customer does
not choose. Scaling that decision down is not this session's call.

Note that pushing the template alone destroys nothing: the controller's git-sync updates the two
template files and does **not** restart the stack (`CLAUDE.md`, deploy contract). The change would
take effect at the next deploy. It is still an armed change, which is why it is held.

**Which box — determined as far as the fences allow, and no further.**

| checked | result |
|---|---|
| hub `/apps` (read-only) | **Papra: 1 deployment**, 218 MB avg / 250 MB P95 — it is live somewhere |
| `demo-felhom` guest 9201 | **not there** — `docker ps -a` shows no papra container, `docker volume ls` no papra volume |
| hub `/configs` | boxes currently reporting: `c10-soak`, `demo-felhom`, `demo-hp` (`david` —, `drill-r50` BLOCKED, `peti-felhom` DOWN, `sess-f` DOWN) |
| hub `/hosts/<id>` and `/apps/papra` | neither page attributes a deployment to a host |

So it is on **`c10-soak`** or **`demo-hp`**, both of which live on demo-hp hardware that this
session is fenced from. `c10-soak` is the likelier of the two and would make this moot: it is
Campaign 10's own rig, its hub customer record reads *"Campaign 10 two-storage soak (SCRATCH -
delete at teardown)"*, and papra was deployed on it **to find R-156 in the first place** — so the
"customer data" would be the campaign's own canary. **That is a guess, and it is labelled as one.**
One `docker ps` on either box settles it, and the other session owns both.

**What is asked:** confirm which box, and then either (a) it is `c10-soak` → push the fix, nothing
of value is lost, or (b) it is a demo box someone cares about → copy `/app/app-data` out of the
running container first (`docker cp papra:/app/app-data <dest>`), then push, then copy back into
the new volume. **No migration is shipped here** — §5 of the brief, and inventing one is how this
class of defect gets worse.


> ### 6.1a — UPDATE at teardown: the deployment disappeared mid-session
>
> Between the start of this run and its teardown, the hub's `/configs` and `/hosts` both **stopped
> listing `c10-soak`** — the other session tore Campaign 10's rig down while this work was in
> flight (its teardown was owed). In the same interval the hub's `/apps` went from 10 deployed apps
> to 6, and the four that disappeared are **papra, grafana, homebox and rallly** — precisely the
> four apps Campaign 10's audit §A4 names as its discriminator set. papra's single deployment was
> therefore on `c10-soak`, as §6.1 suspected, and it is gone.
>
> **The fix is still not applied, deliberately.** The evidence is strong but it is *absence*
> evidence — papra no longer appearing in aggregated telemetry is not the same as a positive
> observation that no box runs it, and this project's standing rule 3 exists for exactly that
> distinction. The one box I cannot check is `demo-hp`, which is fenced this session; the hub says
> it is reporting normally and shows no papra, but that is the same absence argument one level down.
>
> The asymmetry decides it: applying the fix while a papra instance does exist destroys its
> documents at the next redeploy and is irreversible; leaving it is a one-line change the operator
> can push in seconds. **So it is left.**
>
> **What settles it:** `ssh demo-hp "pct exec 9201 -- docker ps -a --filter name=papra"` — one
> command, on the box this session may not touch. If it returns nothing, apply the diff in §6.1;
> there is then no strand risk and papra joins gramps-web and wishlist as an ordinary fix.

### 6.2 Nothing else needed a ruling

Both other BROKEN apps have an unambiguous correct path — each confirmed by **two independent
sources**, never inferred — and neither is deployed anywhere in the fleet, so both were fixed
outright (§5). No app was left BROKEN for want of a decision, and no UNDETERMINED app was touched.

---

## 7. Part 4 — the gate

### 7.1 Where catalog gates live, and what runs them

They live in `app-catalog-felhom.eu/scripts/` as dependency-free Python that exits non-zero on
violation, and **nothing automated runs them.** There is no CI in this repo — no `.gitea/workflows`,
no `.github`, no drone/woodpecker config; the only YAML outside `templates/` is generated customer
output. Each gate is invoked by whoever edits the catalog, and `CLAUDE.md` says when.

That is the honest description, and it is stated rather than glossed because this project's record
on the point is unambiguous: R-29 has three orphaned gates and one enforced one, and the enforced
one is the only one that has ever stopped anything. **This gate is at the same enforcement level as
its two siblings — convention, not CI.** What would raise it is named in §8 as a proposed register
entry; building it would mean touching the controller's sync path, which is out of scope here
(§5 fences) and would need a decision about failing a customer's template sync.

The gate could be built here, so it was. It did not have to wait for `felhom.eu`.

### 7.2 Why the gate has to be a runtime check — measured, not assumed

A static audit of all 53 composes — every declared top-level volume attached to a service, no
anonymous mounts, no host binds outside the felhom roots — reports the catalog **completely clean**.
The single hit is `homepage`'s deliberate read-only `/var/run/docker.sock`.

**It reports papra clean too.** papra's compose is well-formed: it declares `papra_data` and mounts
it at `/app/data`. Nothing in the file is wrong. What is wrong is only observable at runtime.

So a static gate would have been a gate that could never have caught the defect it was written for —
coverage that isn't. `scripts/check-volume-persistence.py` deploys, exercises, and measures.

### 7.3 What it refuses

**Exit 1 — REFUSED, not a warning:**

- DATA in a container's writable layer (never persisted, never backed up);
- an app-data mount the app's own uid cannot write (R-156's second leg);
- data in an **anonymous** volume (never backed up, orphaned by a redeploy);
- a database-signature file in the writable layer whose bytes **differ** from the image's copy.

**Exit 2 — UNDETERMINED, and explicitly *not* a clean bill of health:** a container that never
reached running/healthy; nothing written anywhere; a suspect that could not be adjudicated; or —
load-bearing — **the prober failing its own two-way canary**, in which case the gate refuses to
report any verdict at all rather than issue a false all-clear.

**Exit 0** only when every app in scope is CLEAN.

### 7.4 The test through the path that actually invokes it

`scripts/test_check_volume_persistence.py` — **31 tests, no Docker** (the prober is injected).
They drive **`check()`, the function `__main__` calls**, not just the verdict logic: a gate whose
decision logic is tested but whose entry point is not has shipped inert in this project before
(the seam-wiring rule, four recorded instances).

Covered at the entry point: one broken app ⇒ **rc 1** and the app is named; all clean ⇒ rc 0;
undetermined ⇒ **rc 2, never rc 0**; broken outranks undetermined; out-of-circulation apps skipped
and reported; **a prober that flags nothing ⇒ rc 2**; **a prober that flags everything ⇒ rc 2**.

Every rule was red-proofed by breaking it and confirming the suite goes red:

| break | suite |
|---|---|
| score `C` entries as data again (the calibre-web false positive) | **RED** |
| revert the trailing-slash noise normalisation (the crafty false positive) | **RED** |
| fold "wrote nothing" into CLEAN | **RED** |
| warn instead of refuse (rc 0 on broken) | **RED** |
| drop the writable-layer leg (the papra signature) | **RED** |
| let an unresolved suspect fall through to CLEAN | **RED** |
| drop the `base64:` prefix | **RED** |
| *(the deleted all-children-noise rule)* | **GREEN — which is why it was deleted** |

Registered in `CLAUDE.md` (conventions) and `REUSE.md` (the map), in the same commit.

---

## 8. Findings for later filing — PROPOSED TEXT, NOT FILED

The register is `felhom.eu/documentation/backlog/{ROADMAP,OPEN-ITEMS}.md`, and `felhom.eu` is
fenced this session. These are drafts for one session to file, in one place.

**Register grep before proposing** (read-only, `ROADMAP.md` + `OPEN-ITEMS.md`):
`wishlist` -> 1 hit, R-41's deployability gate, unrelated · `gramps` -> 0 · `anonymous` -> 2 hits,
both about the golden bake, unrelated · `ResolveDockerVolumeNames` -> 0 · `DumpAppVolumes` -> 0 ·
`writable layer` -> 0 · `check-volume-persistence` -> 0. **Highest filed: R-155.** So these start
at **R-158**.

### 8.0 First, a process finding — R-156 and R-157 are themselves unfiled

`grep -c "R-156\|R-157"` over both register files returns **0 / 0**. They exist only inside
`audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`. This is the identical failure the brief cites
for R-154/R-155 — *"minted in a spike doc and never carried across"* — recurring immediately, on
the very finding this session was commissioned to act on. **R-156 is the reason this work happened
and it is not in the register.** Whoever files the entries below should carry R-156 and R-157
across at the same time.

**And it happened again, to this document, while it was being written.** The first draft of this
section numbered its proposals R-158…R-161 on the strength of a register grep showing R-155 as the
highest filed. Mid-session the parallel session pushed
`audits/SPIKE-recovery-unit-space-2026-08-02.md` and `audits/CAMPAIGN-10-closeout-2026-08-02.md`,
both of which **use R-158** for an unrelated finding (recovery-unit space ceiling on mp1) — and
neither adds it to `ROADMAP.md` or `OPEN-ITEMS.md`, which still stop at R-155. So a grep of the
register was true when it was run and stale within hours, and two sessions minted the same number
for different findings on the same day.

These proposals are therefore renumbered **R-159…R-162**, and the number to claim should be taken
at filing time, not at drafting time. The register being the only source of truth is precisely what
is not happening: **R-156, R-157 and R-158 are all live in audit documents and none of them is
filed.**

### R-159 — wishlist's data lands in an ANONYMOUS volume: never backed up, orphaned by a redeploy

**FIXED HERE** (`templates/wishlist/docker-compose.yml`); filed to record the class.

The image declares `VOLUME /usr/src/app/data` and `VOLUME /usr/src/app/uploads`
(`docker image inspect ghcr.io/cmintey/wishlist:v0.66.0` -> `Config.Volumes`). The template mounted
`wishlist_data:/data` — **a path the application never writes**. So `prod.db` went into the
anonymous volume docker creates for an image `VOLUME` directive.

**Why this is worse than an ordinary miss: it survives a restart.** `ResolveDockerVolumeNames`
(`felhom-controller/internal/appbackup/appdata.go`) returns `<project>_<name>` only for volumes
**declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one — and
`compose down` + `up` orphans it and mounts a fresh empty one. Meanwhile `wishlist_data` was tarred
nightly as an empty directory. A store that survives a restart, loses on redeploy, and is never in
a backup is harder to notice than papra's, because casual testing finds nothing wrong.

**The class, not just the app:** any image `VOLUME` directive at a path the template does not
explicitly mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data`
(`Config.Volumes = {"/data":{}}`) — empty when measured, so nothing is at risk now, but anything
immich ever writes there is invisible to backup. Suggested standing rule for `REUSE.md`: *a template
must mount every path in its image's `Config.Volumes`, or state why not.* Checkable statically only
if the image is pulled, which is why it lives in the runtime gate.

### R-160 — gramps-web persisted three paths and wrote to none of them

**FIXED HERE** (`templates/gramps-web/docker-compose.yml`).

The template mounted `/app/data`, `/app/media` and `/tmp`; **`/app/data` appears nowhere in the
image's environment** — the application never writes it. Its accounts database
(`GRAMPSWEB_USER_DB_URI=sqlite:////app/users/users.sqlite`) and **its family tree**
(`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the container's writable layer.
Upstream's reference compose persists **eight** paths; this template persisted three, one a phantom.

Severity above papra's: papra loses documents a customer uploaded, which they may still hold
elsewhere. gramps-web loses the family tree — the artefact the customer *built inside the app*, of
which no other copy exists by construction.

### R-161 — the gate exists, and nothing automated runs it

The check R-156 asks for now exists (`scripts/check-volume-persistence.py`, §7). Its enforcement
level is **convention**: the catalog repo has no CI of any kind (`.gitea/workflows`, `.github`,
drone/woodpecker — none), so like its two sibling gates it runs when a person runs it.

This project's own record says that is not enough: R-29 has three orphaned gates and one enforced
one, and the enforced one is the only one that has ever stopped anything. Two directions, neither
buildable from this repo:

1. **CI in the catalog repo** — a Gitea Actions runner. Nothing exists to build on today.
2. **Controller-side at sync** — the controller already validates `.felhom.yml` at template load
   (`internal/stacks/metadata.go:336-358`) and a persistence assertion could ride the same choke
   point. But that needs a runtime probe on a customer box, and a decision about what a failing
   sync should do to a customer's working app. **Cross-repo, and a design question, not a task.**

Interim position, recorded in `CLAUDE.md`: run it when a template's `volumes:` block or image tag
changes, and at the start of every catalog campaign.

### R-162 — `docker diff` is the only witness, and its failure mode is quiet

A limitation, not a defect. The gate's power comes from `docker diff` excluding mounted paths, which
makes "in the writable layer" mechanically decidable — an implementation detail of the overlay
storage driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the
mount-occupancy and writability legs and would not say so. The canary self-test *would* catch it
(the broken canary would stop reporting BROKEN, and the gate would then refuse to report at all),
so it fails closed — but the message it prints would blame the prober rather than the driver.

---

## 9. Teardown — all three layers

Per `felhom.eu/documentation/PROMPT-TEMPLATE.md` §13. Layer 3 is the one that has been missed every
time (`drill-r50`, `sess-c`, `sess-d`), so it is stated explicitly rather than implied.

### Layer 1 — the machine

**DONE.** `pct stop 9301 && pct destroy 9301 --purge` → `Logical volume "vm-9301-disk-0"
successfully removed` / `purging CT 9301 from related configurations` / rc=0. Verified after:
`pct list` shows only 9201, and `lvs | grep 9301` returns nothing. Evidence: `state/teardown.txt`.

### Layer 2 — the host

`pvesm status` before and after, and the space actually returned.

| | `local-lvm` available (KiB) | used |
|---|---|---|
| before the run began | 258 702 410 | 29.27 % |
| end of run, before teardown | 169 968 909 | 53.53 % |
| **after teardown** | **258 702 410** | **29.27 %** |

**The space returned exactly to the pre-run baseline** — 258 702 410 KiB, identical to the figure
recorded before the guest was created. `local` is unchanged at 26.00 %. Full before/after:
`state/teardown.txt`.

The guest's 120 G volume was thin-provisioned on an **over-subscribed** pool (`pct create` warned:
sum of thin volume sizes 402 G against a 348 G pool), which is why a disk watchdog ran throughout
(`state/diskwatch.log`) and why the space actually returning is measured rather than assumed.

### Layer 3 — the hub

**Nothing to remove: this run created no hub record of any kind, by design.**

That is an absence of *creation*, not an absence of checking. The scratch guest ran **no
felhom-controller** and was never enrolled (§1), precisely so no customer or appliance record could
exist to orphan. The hub was touched **read-only** throughout — `/apps`, `/apps/papra`, `/configs`,
`/hosts`, `/hosts/<id>`, over the ClusterIP with Basic auth — to answer "is papra deployed
anywhere?" (§6.1).

Confirmed after teardown against the live hub: `/configs` contains no entry for this run
(`catalog-persist` / `9301`: absent), and `/hosts` lists only `demo-felhom-8363b5`,
`demo-hp-bb76ea`, `drill-r50-0a4f9a`, `sess-f-2670b5`. The list is **two shorter** than at the
start of the run — `c10-soak` was removed by the other session mid-run, not by this one (§6.1a).

The scratch host paths the gate creates for `${HDD_PATH}` / `${IMPORT_PATH}` (`/srv/felhom-gate/…`)
and the `felhom-volgate-canary:1` image existed only inside the guest, so both go with layer 1.

---

## 10. What did not happen, and why

- **papra was not fixed.** It is deployed, and the fix would destroy the data it protects (§6.1).
  Prepared, proven, held. The brief says stop rather than ship a migration; no migration was written.
- **No data migration of any kind was shipped**, for papra or anything else.
- **No UNDETERMINED app was fixed.** Seven remain undecided (§4.2) and none was nudged toward CLEAN
  to tidy the table. Four need a deeper exercise than a first-boot HTTP request (a completed setup
  wizard, a posted paste, a real import); three could not be started at all.
- **`glance` was not fixed** even though its defect is understood (it crash-loops for want of a
  seeded `/app/config/glance.yml`). It is a deployability defect, not a persistence one, already
  recorded as pre-existing in Campaign 7 §6.2 — outside this brief's scope.
- **`plant-it` was not touched.** Its image does not resolve; it is `lifecycle: abandoned` for
  exactly that reason.
- **Nothing outside the catalog templates was changed.** The controller-side observations
  (`ResolveDockerVolumeNames` cannot see anonymous volumes; `ParseComposeClassifiableBinds` cannot
  see named volumes at all) are findings for §8, not edits.
- **No commit to `felhom.eu`, no register edit, nothing on `demo-hp`.** The other session owns all
  three. Every R-number in §8 is proposed text.
- **No hub-side state was created.** The hub was read only — `/apps`, `/configs`, `/hosts`,
  `/apps/papra` — via the ClusterIP with Basic auth. Nothing to clean up, so nothing recorded as a
  command (R-131's four orphans).
- **Guest 9201 on `demo-felhom` was read, not touched** — `docker ps -a` and `docker volume ls`,
  to establish that papra is not deployed there.
- **`claude-in-chrome` was not used** — unavailable on DooPlex. Every observation is endpoint-level
  or filesystem-level, and each is named where it is used.
- **No secret value was written anywhere.** The probe records env var KEYS only; generated
  credentials are minted per-run and discarded with the container.

---