app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/README.md

# Catalog persistence sweep — does every app's data land in a persisted volume?

**Run:** 2026-08-02 · **Repo:** `app-catalog-felhom.eu` (only) · **Class:** implementation —
detection first, fixes second.

Campaign 10 found that papra stores its database somewhere the backup never looks (R-156). papra is
not the interesting part. **Nothing anywhere checked that the folder a template preserves is the
folder the app writes to**, across 53 templates. This is what that check found.

> **Parallel-session fence, observed.** Another session was running the M-band RTO measurement on
> `demo-hp`, writing to `felhom.eu`. This session ran on `demo-felhom` and committed only to
> `app-catalog-felhom.eu`. **No commit to `felhom.eu`, no register edit, nothing on `demo-hp`.**
> All register entries below are proposed text for later filing by one session, in one place.

---

## 1. Venue and baselines

| | |
|---|---|
| Host | **`demo-felhom`** (N100), Tier 0 — `ssh felhom-pve` (Tailscale `100.70.170.35`) |
| Guest | **LXC 9301 `catalog-persist`**, created for this run and destroyed at teardown (§9) |
| Guest spec | Debian 13, 4 cores, 6 GB + 2 GB swap, `local-lvm:120`, `nesting=1,keyctl=1`, DHCP `192.168.0.22` |
| Reach | `ssh felhom-pve "pct exec 9301 -- …"`. **No direct SSH**: the N100 sits on a remote LAN whose `192.168.0.0/24` collides with DooPlex's, so `192.168.0.22` is unroutable from here (`No route to host`) |
| Docker | 29.7.1, compose v5.3.1 |
| Untouched | guest **9201** on `demo-felhom` (read-only `docker ps`/`volume ls` only), `drill-r50`, Peti's cluster, both boxes' backup configuration, **everything on `demo-hp`** |

### Baselines, read fresh

| What | Value | How |
|---|---|---|
| `app-catalog-felhom.eu` HEAD | **`4252121`** = `origin/main`, clean at start | `git rev-parse` |
| Templates | **53** | `ls templates \| wc -l` |
| `felhom.eu` HEAD (read-only, for citation) | **`7ba7c2a`** | `git rev-parse` — **not committed to** |
| felhom-controller `main` | **v0.188.0**, HEAD `4115e88` | `CHANGELOG.md` top |
| Hub | 0.86.0 | live `/apps` footer |

### Why the scratch guest runs no controller — stated, because the brief asks for its version

There is **no felhom-controller on 9301, deliberately.** The property under test is a property of
the *template*, and the controller's role in it is exactly one thing: it resolves the `${…}`
variables and runs `docker compose up -d` (`felhom-controller/internal/stacks/deploy.go:374`,
`:551-582`). It does not rewrite the compose file, so a plain Docker host reproducing that env
substitution exercises the identical code path in the app.

Standing a controller up would have meant enrolling a box with the hub — creating hub-side state
this session is told not to create, and which R-131 records four orphaned instances of. The
substitution the controller performs is instead **mirrored exactly**, including the trap that
`generateValue` prefixes `base64key` with `base64:` (`deploy.go:904`); getting that wrong cost one
false verdict before it was caught (§3.4).

---

## 2. Part 0 — had this ground been covered?

**No. Neither Campaign 7 nor any gate, `.felhom.yml` field, or CI check tests this property.**
Each negative below names what was searched.

### 2.1 Campaign 7 tested deployability and health, not persistence

`felhom.eu/documentation/audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md:38-51` states its method:
per-container terminal health verdict, healthcheck-binary audit, HTTP probe through the real
Traefik ingress, login where scriptable, and a log scan. Its result matrix (`:114`) carries the
columns *pin · MAJOR · deploy · health · http · logs · settle*. **There is no persistence column,
and no step in the method reads where data landed.**

The proof that this is a gap rather than a technicality is in that same matrix: **papra passed it**
— `deploy ok`, `health **fixed** ×2`, `http 200`, `logs clean` (`:151`) — while its database was
being written to a directory no backup would ever contain. Campaign 7 also demonstrated the
narrower fact that a healthcheck can be a lie (C7-F4, `:210-224`); this sweep demonstrates that a
*truthful* healthcheck is still not evidence about data.

**What changed since:** nothing that would have closed it. Between Campaign 7 and today the catalog
gained the resolvability gate (2026-07-21), `lifecycle:` (2026-07-21) and R-75's `data_paths:`
(2026-07-26) — image currency, circulation status, and folder *labelling* respectively.

### 2.2 No gate asserts a volume ⟷ data-path relationship. Searched:

| Where | What was searched | Result |
|---|---|---|
| `app-catalog-felhom.eu/scripts/` | full listing | exactly two gates, `check-image-pins.py` (syntactic tag check) and `check-image-resolvable.py` (does the tag still exist) — **both about images, neither about volumes** |
| catalog repo CI | `.gitea/workflows`, `.github`, drone/woodpecker/ci config; every non-template YAML | **none exists.** Catalog gates are convention-run, invoked by whoever edits the catalog — this matters for §7 |
| all four repos | `grep -rniE "volume.*(persist\|data.?path\|writable.layer)\|writable.layer\|docker diff\|persistence.(gate\|check)" --include=*.py --include=*.go --include=*.sh` | no check of this kind; the only hits are comments and a test fixture name |
| all gate scripts, four repos | `ls felhom-controller/scripts/*.py felhom.eu/scripts/*.py app-catalog-felhom.eu/scripts/*.py felhom-agent/scripts/*` | 11 gates. The closest by name, `felhom-controller/scripts/docker_run_volume_path_gate.py`, is unrelated: it allowlists `docker … -v` arguments in **controller Go code** so a controller-local host path cannot be mounted from inside a container (v0.125.0 scenario D) |
| `.felhom.yml` schema | `backup:`, `data_paths:`, `lifecycle:`, `deploy_fields:` | none expresses "the app writes here" |

### 2.3 The two validators that look closest cannot see this class

`ValidateBackupSpec` (`felhom-controller/internal/appbackup/classify.go`) and `ValidateDataPaths`
(`internal/stacks/datapaths.go:58`) both reject an entry matching no compose bind — real checks, but
of **declaration consistency**, not of runtime behaviour. And both operate on
`ParseComposeClassifiableBinds` (`internal/stacks/classify_binds.go`), which keeps only mounts whose
host side passes `classifyRoot` — i.e. paths under `${USERDATA_PATH}` / `${HDD_PATH}` /
`${IMPORT_PATH}`. **Named volumes are skipped entirely.**

That is decisive for papra, which has **zero binds**: every one of these validators is vacuously
satisfied by it. It is also decisive for scale — **39 of 53 templates declare no felhom-root bind at
all**, so for three-quarters of the catalog the entire data-protection story is `DumpAppVolumes`
(`internal/backup/backup.go:543`) tarring a named volume, and nothing checks that the volume has
anything in it.

### 2.4 R-156's evidence, read before building

`felhom.eu/documentation/tests/campaign10-evidence-2026-07-31/r156-papra-volume.txt` — five numbered
observations: the volume mounted at `/app/data`; `uid=999(nonroot)` with `/app/data` `root root`;
`touch: cannot touch '/app/data/.w': Permission denied`; the volume empty but for a canary the
campaign wrote **as root**; and `db.sqlite` 475136 B in the container's writable layer at
`/app/app-data/`. The detector was built against those observations, and it reproduces all of them.

---

## 3. The detector

`scripts/check-volume-persistence.py`. **The detector and the gate are the same program** — the
sweep below was produced by running the committed gate, so the durable artifact is the thing that
generated the evidence rather than a paper copy of it.

Per app: deploy from the template → exercise it into writing data → compare where the data landed
against what the compose mounts.

### 3.1 What is observed, and why nothing uses `docker exec`

| | signal | how |
|---|---|---|
| M | mount inventory, by persistence class | `docker inspect .Mounts` |
| D | the writable layer | `docker diff` — Docker **excludes every mounted path** from it, so an entry here is definitionally in no volume and no bind |
| U | the app's real uid/gid | `/proc/<State.Pid>/status`, **not** `Config.User` — which is empty for papra even though it drops to 999 |
| W | writability of each mount by that uid | host-side `stat` of the mount source |
| E | mount occupancy | host-side walk of the mount source |

**No `docker exec` anywhere.** Campaign 7 §1.1 recorded that `docker exec` writes its OCI error to
STDOUT, so a *missing* binary printed `executable file not found` and read as **present** — which
made that campaign's entire healthcheck audit report every app honest. Everything here is read from
`docker inspect`, `docker diff`, `/proc` and the host filesystem, so a distroless or shell-less
image is observed exactly like any other.

### 3.2 Mount classes — the second defect class this exposes

| class | persists a redeploy? | backed up? |
|---|---|---|
| `named-declared` — in the compose `volumes:` block | yes | **yes** |
| `anonymous` — an image `VOLUME` or unnamed mount (64-hex docker name) | survives a *restart* | **NO** |
| `bind` | yes | per the backup spec |
| `tmpfs` | no | no |

The anonymous case is deceptive precisely because it survives a restart. `ResolveDockerVolumeNames`
(`felhom-controller/internal/appbackup/appdata.go`) returns `<project>_<name>` only for volumes
**declared in the compose file**, so an anonymous volume is invisible to `DumpAppVolumes` — and a
`compose down` + `up` orphans it and creates a fresh one. Data there is unbacked-up *and* lost on
redeploy, with no symptom until someone looks.

### 3.3 What counts as "exercised", and what counts as data

**Exercised.** Every container reaches `running`, and `healthy` where a healthcheck exists; then an
HTTP request the app's *own router* answers. Any status code — including 3xx/4xx/5xx — proves the
request reached application code; a connection refusal (`000`) does not. If that produced no writes
anywhere, a **second-chance pass** walks a wider path list following redirects, so a first-run
wizard is actually reached, and waits again. **An app that still writes nothing is UNDETERMINED,
never CLEAN** — health-check-passing is not data-writing, and conflating the two is what let papra
look fine.

**Data versus noise.** Three rules, deliberately explicit rather than buried in a heuristic:

1. **Noise** — `/tmp`, `/run`, `/var/log`, `/var/cache`, package-manager state, `__pycache__`,
   `node_modules`, `.cache`, `*.pid`, `*.sock`, `*.log`, `*.pyc`, and the per-container files
   Docker itself writes (`/etc/hosts`, `/etc/resolv.conf`, …).
2. **Database signature** — a filename that *proves* a database is in that directory
   (`*.sqlite*`, `PG_VERSION`, `ib_logfile*`, `data.mdb`, `dump.rdb`, `MANIFEST-*`, `data.ms`, …).
   The strongest signal, and it needs no path heuristics.
3. **Data token** — a path saying "app state" (`data`, `storage`, `uploads`, `library`, `documents`,
   `var/lib/postgresql`, …), used only where rule 2 is silent.

**`A` versus `C` is the whole difficulty**, and rule 3 is where judgement actually lives:

- `A` (added) — the app **created** this file; it exists only in the writable layer. papra's
  `db.sqlite` is an `A`. **DATA is decided from `A` entries only.**
- `C` (changed) — a file that **ships in the image** was touched. Usually a chown sweep and
  entirely benign. A `C` on a *database-signature* file is genuinely ambiguous, because an app
  writing into a DB shipped in its image looks identical. Those are held as **SUSPECT** and settled
  **by bytes**: the file is copied out of the running container and out of a pristine container
  created from the same image, and compared. Identical → a chown, recorded and dismissed.
  Different → a real write into an image-layer file, which is papra's defect with a different verb.
  Unreadable → **UNDETERMINED**, never folded into CLEAN.

### 3.4 Red-proofs — both directions, before the sweep, and again on every run

| | claim | result |
|---|---|---|
| **positive** | it flags **papra** | **BROKEN**, reproducing R-156's evidence line for line: `/app/data` EMPTY and `NOT writable by the app's own uid=999 gid=999`, `db.sqlite` in the writable layer at `/app/app-data/db` |
| **negative** | it stays quiet on a known-good app | **vaultwarden CLEAN** — and *demonstrably* good, not merely unflagged (below) |

The negative is a **proven** negative. `state/survive2.py` deployed vaultwarden, let it write, ran a
real `docker compose down` + `up -d` (no `-v` — that is a redeploy; `down --volumes` is the
uninstall path, Campaign 7 §2), and re-fingerprinted every file by sha256 **and inode**:

```
vaultwarden   SURVIVES   before=4 after=4 survived=2 changed=2 lost=0
  survived: /data::db.sqlite3     <- the database itself, byte-identical, SAME INODE
            /data::rsa_key.pem
  changed : /data::db.sqlite3-shm , /data::db.sqlite3-wal   <- SQLite sidecars, expected on restart
```

**The self-test runs on every invocation, in both directions.** Before the gate reports anything it
puts two canary templates through the same prober and refuses to issue a verdict unless it calls
one BROKEN and the other CLEAN. Both are built from a purpose-made image reproducing papra's exact
ownership shape — `/app/data` root-owned, `/app/app-data` owned by the app's uid — and **differ only
in which path the volume is mounted at**, so the canary pair is a live demonstration of R-156 *and*
of its fix, every run. A detector that flags nothing is worse than none: it converts an unexamined
catalog into a documented-clean one.

### 3.5 Seven defects in the detector, every one found by adjudicating a flag rather than trusting it

Recorded because each would have put a wrong number in the matrix, and **six of the seven made a
correct app look broken** — the failure mode that gets a gate switched off and then protects nothing.

| # | defect | what it did |
|---|---|---|
| 1 | **`base64key` lost the controller's `base64:` prefix** (`deploy.go:904`) | bookstack's Laravel `APP_KEY` was invalid, it served 500s, read UNDETERMINED. With the prefix: CLEAN. A harness bug wearing an app defect's clothes — the class Campaign 7 §1.1 and Campaign 10 §4d both record |
| 2 | **`C` entries scored as data** | a linuxserver.io entrypoint re-owning its application tree produced **1305 `C` entries** for calibre-web, 92 of them PNGs under `cps/static/css/images/**`. Called BROKEN. Led to the `A`/`C` split and the byte-level adjudicator (§3.3) |
| 3 | **a cache *directory* escaped the noise filter** | `docker diff` lists directories, so the bytecode cache appeared as a bare `…/crafty/config/__pycache__` entry while its `.pyc` children were filtered by suffix — the directory then being the only surviving entry under a path containing `config`. crafty-controller called BROKEN four times over with its data correctly placed. Fixed by normalising with a trailing slash, covering `node_modules`, `.cache`, `.git` at once |
| 4 | **`postgresql.conf` listed as a database signature** | it is a *config* file the postgres entrypoint writes at init. immich called BROKEN with its PGDATA correctly in its volume, **1831 files** |
| 5 | **path tokens convicted on their own** | onlyoffice unpacks its OWN static assets into the writable layer at first boot — plugin icons, slide-theme `media/`, `web-apps/apps/api/documents/api.js`, **2560 added entries** — while its real data mount received data normally. Rule 3 was **demoted to report-only** |
| 6 | **the structural check asked per container, not per app** | fired on **docmost, immich and claper** — three correct apps, identical shape: the app container's only volume is for user uploads and legitimately empty on a fresh install, while all real state sits in the sibling database container's volume (1540 / 1833 / 1470 files) |
| 7 | **mount rules never consulted the noise list** | the prefixes are written `/run/`, so a bare mount target `/run` matched none of them. privatebin's image declares `VOLUME /run`; the anonymous volume docker made there holds `nginx.pid`, `php-fpm.sock` and s6 fifos, and read as 14 files of unbacked-up customer data |

Plus one that was not a false verdict but a **reporting** defect, and arguably the worst of the set:

8. **A finding was being swallowed by a louder one.** gramps-web's structural finding — its *family
   tree* landing outside every mount, the entire point of the app — went into the undetermined
   bucket, and that bucket is discarded whenever a BROKEN leg exists. Its lesser defect (the
   accounts database) convicted first and hid it. A finding that disappears because a different
   finding won belongs to the same family as an absent log line read as health, and the fix is that
   the structural finding is now reported whatever the verdict.

**The pattern, which is the real lesson.** Every false positive came from a rule that identified data
by its **name** rather than by evidence of what it is: `C` assumed to be a write, `postgresql.conf`
assumed to be a database, `media/` and `config/` assumed to be customer content, `/run` assumed to be
storage because it was a volume. The rules that never needed correcting are the ones grounded in
something checkable — a database file the app **created**, a mount its own uid **cannot write**,
bytes that **differ** from the image's copy, and *did anything land in a preserved folder at all*.

**Restarts, stated plainly.** The sweep was restarted from the first app **twice** rather than merge
a matrix built by more than one version of the rules — the call Campaign 10 made restarting run 2b
from cycle 1. Defects 6, 7 and 8 were found after the final pass had begun; all three are in
`classify()`, which reads only what the probe already captured, so those verdicts were **re-derived**
rather than re-measured (`state/reclassify.py`), and then **every non-CLEAN and every flipped app was
re-run live end-to-end** through the final gate so no reported verdict rests on post-processing
(§5.3). Six verdicts changed on re-derivation: claper, docmost, immich, sparkyfitness and zipline
UNDETERMINED→CLEAN, privatebin BROKEN→UNDETERMINED.

**And one rule was written, red-proofed, and deleted.** A second "are all this directory's children
noise?" filter looked like a sensible belt to defect 3's brace. Its red-proof came back **green**,
which is the tell: `e["added"]` can only ever contain entries that already passed `is_noise`, so the
check is always False — dead code wearing the costume of a safeguard, exactly the trap `CLAUDE.md`
catalogues. It is gone, the code says why it is not there, and
`test_cache_directories_are_filtered_at_ENTRY_level` pins the mechanism that actually does the work.

## 4. The sweep — all 53 templates

**43 CLEAN · 3 BROKEN · 7 UNDETERMINED.** UNDETERMINED is counted separately and is never folded
into CLEAN; each one's reason is in §4.2.

Evidence per app: `evidence/<app>/probe.json` — the full probe (mount inventory with occupancy and
writability, the app's real uid, the writable-layer rollup, the adjudicated suspects) plus the
verdict and its reasons. Runner log: `state/gate.log`.

**How to read the mount column:** `[vol]` a named volume declared in the compose (persisted AND
backed up) · `[ANON]` an anonymous volume from the image's own `VOLUME` directive (persisted across
a restart, **never backed up**, orphaned by a redeploy) · `[bind]` a host bind · `,EMPTY` the mount
received nothing.

### 4.1 The matrix

| app | verdict | what the template mounts | where the app actually wrote | mount not writable by app uid |
|---|---|---|---|---|
| `gramps-web` | **BROKEN** | /app/data[vol,EMPTY], /app/media[vol,EMPTY], /tmp[vol,EMPTY] | **/app/users (writable layer)** | — |
| `papra` | **BROKEN** | /app/data[vol,EMPTY] | **/app/app-data/db (writable layer)** | /app/data |
| `wishlist` | **BROKEN** | /data[vol,EMPTY], /usr/src/app/uploads[ANON,EMPTY], /usr/src/app/data[ANON] | /usr/src/app/data(1) | — |
| `bentopdf` | **UNDETERMINED** | — | nothing | — |
| `glance` | **UNDETERMINED** | /app/config[vol,EMPTY] | nothing | — |
| `plant-it` | **UNDETERMINED** | — | nothing | — |
| `privatebin` | **UNDETERMINED** | /tmp[ANON,EMPTY], /var/lib/nginx/tmp[ANON,EMPTY], /srv/data[vol,EMPTY], /run[ANON] | /run(14) | — |
| `recipe-importer` | **UNDETERMINED** | /data[vol,EMPTY] | nothing | — |
| `uptime-kuma` | **UNDETERMINED** | /app/data[vol,EMPTY] | nothing | — |
| `wanderer` | **UNDETERMINED** | /app/uploads[vol,EMPTY], /pb_data[vol], /data/plugins[vol,EMPTY], /meili_data[vol] | /pb_data(6), /meili_data(11) | — |
| `actualbudget` | **CLEAN** | /data[vol] | /data(2) | — |
| `adventurelog` | **CLEAN** | /code/media[vol], /var/lib/postgresql/data[vol] | /code/media(251), /var/lib/postgresql/data(2287) | — |
| `audiobookshelf` | **CLEAN** | /config[vol], /metadata[vol], /audiobooks[bind,EMPTY], /podcasts[bind,EMPTY] | /config(16), /metadata(1) | — |
| `bookstack` | **CLEAN** | /config[vol], /var/lib/mysql[vol] | /config(20), /var/lib/mysql(291) | — |
| `calcom` | **CLEAN** | /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(2261) | — |
| `calibre-web` | **CLEAN** | /config[vol], /cwa-book-ingest[bind,EMPTY], /calibre-library[bind] | /config(11), /calibre-library(3) | — |
| `claper` | **CLEAN** | /app/priv/static/uploads[vol,EMPTY], /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(1470) | — |
| `code-server` | **CLEAN** | /config[vol] | /config(6) | — |
| `crafty-controller` | **CLEAN** | /crafty/servers[vol,EMPTY], /crafty/app/config[vol], /crafty/import[vol,EMPTY], /crafty/backups[vol,EMPTY], /crafty/logs[vol] | /crafty/app/config(16), /crafty/logs(8) | — |
| `docmost` | **CLEAN** | /app/data/storage[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol] | /var/lib/postgresql/data(1540), /data(3) | — |
| `emby` | **CLEAN** | /config[vol], /media[bind] | /config(46), /media(3) | — |
| `ghost` | **CLEAN** | /var/lib/ghost/content[vol] | /var/lib/ghost/content(7) | — |
| `gitea` | **CLEAN** | /data[vol] | /data(8) | — |
| `gokapi` | **CLEAN** | /app/data[vol], /app/config[vol] | /app/data(4), /app/config(1) | — |
| `grafana` | **CLEAN** | /var/lib/grafana[vol] | /var/lib/grafana(543) | — |
| `home-assistant` | **CLEAN** | /config[vol] | /config(21) | — |
| `homebox` | **CLEAN** | /data[vol] | /data(3) | — |
| `homepage` | **CLEAN** | /app/config[vol], /var/run/docker.sock[bind,EMPTY] | /app/config(3) | — |
| `immich` | **CLEAN** | /cache[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol], /usr/src/app/upload[bind], /external/photos[bind,EMPTY], /data[ANON,EMPTY] | /var/lib/postgresql/data(1833), /data(3), /usr/src/app/upload(6) | — |
| `jellyfin` | **CLEAN** | /config[vol], /cache[vol], /media[bind] | /config(20), /cache(2), /media(3) | — |
| `kimai` | **CLEAN** | /opt/kimai/var[vol], /var/lib/mysql[vol] | /opt/kimai/var(2547), /var/lib/mysql(279) | — |
| `komga` | **CLEAN** | /config[vol], /data[bind,EMPTY] | /config(11) | — |
| `mealie` | **CLEAN** | /app/data[vol] | /app/data(5) | — |
| `n8n` | **CLEAN** | /home/node/.n8n[vol] | /home/node/.n8n(7) | — |
| `navidrome` | **CLEAN** | /data[vol], /music[bind,EMPTY] | /data(6) | — |
| `nextcloud` | **CLEAN** | /var/www/html/data[bind], /var/www/html[vol], /var/lib/mysql[vol], /data[vol] | /var/www/html/data(69), /var/www/html(4001), /var/lib/mysql(311), /data(3) | — |
| `onlyoffice` | **CLEAN** | /var/www/onlyoffice/Data[vol], /var/log/onlyoffice[vol], /usr/share/fonts/truetype/custom[ANON,EMPTY], /var/lib/onlyoffice[ANON,EMPTY] | /var/www/onlyoffice/Data(3), /var/log/onlyoffice(11) | — |
| `opengist` | **CLEAN** | /opengist[vol] | /opengist(9) | — |
| `outline` | **CLEAN** | /var/lib/outline/data[vol,EMPTY], /var/lib/postgresql/data[vol], /data[vol,EMPTY] | /var/lib/postgresql/data(1655) | — |
| `paperless-ngx` | **CLEAN** | /var/lib/postgresql/data[vol], /data[vol], /usr/src/paperless/export[bind,EMPTY], /usr/src/paperless/data[vol], /usr/src/paperless/media[bind], /usr/src/paperless/consume[bind,EMPTY] | /var/lib/postgresql/data(1927), /data(3), /usr/src/paperless/data(9), /usr/src/paperless/media(1) | — |
| `plex` | **CLEAN** | /media[bind], /config[vol], /transcode[vol,EMPTY] | /media(3), /config(82) | — |
| `radarr` | **CLEAN** | /config[vol], /media/movies[bind,EMPTY], /downloads[bind,EMPTY] | /config(11) | — |
| `rallly` | **CLEAN** | /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(1580) | — |
| `romm` | **CLEAN** | /redis-data[ANON,EMPTY], /romm[ANON,EMPTY], /romm/library[bind,EMPTY], /romm/resources[bind,EMPTY], /romm/config[vol], /var/lib/mysql[vol], /data[vol] | /romm/config(1), /var/lib/mysql(271), /data(4) | — |
| `seerr` | **CLEAN** | /app/config[vol] | /app/config(11) | — |
| `sonarr` | **CLEAN** | /media/tv[bind,EMPTY], /downloads[bind,EMPTY], /config[vol] | /config(11) | — |
| `sparkyfitness` | **CLEAN** | /var/lib/postgresql/data[vol], /app/SparkyFitnessServer/uploads[vol,EMPTY], /app/SparkyFitnessServer/backup[vol,EMPTY] | /var/lib/postgresql/data(1815) | — |
| `tandoor` | **CLEAN** | /opt/recipes/staticfiles[vol], /opt/recipes/mediafiles[vol,EMPTY], /var/lib/postgresql/data[vol] | /opt/recipes/staticfiles(3047), /var/lib/postgresql/data(1984) | — |
| `termix` | **CLEAN** | /app/data[vol] | /app/data(4) | — |
| `vaultwarden` | **CLEAN** | /data[vol] | /data(4) | — |
| `vikunja` | **CLEAN** | /app/vikunja/files[vol,EMPTY], /db[vol] | /db(3) | — |
| `wger` | **CLEAN** | /home/wger/db[vol], /home/wger/media[vol,EMPTY] | /home/wger/db(1) | — |
| `zipline` | **CLEAN** | /zipline/uploads[vol,EMPTY], /zipline/public[vol,EMPTY], /var/lib/postgresql/data[vol] | /var/lib/postgresql/data(1357) | — |

### 4.2 UNDETERMINED — seven, each with its reason

**None of these is a pass.** Four are apps that genuinely wrote nothing, which is a limit of the
exercise rather than a property of the template; three could not be started at all.

| app | why it is undetermined | is this a template defect? |
|---|---|---|
| `bentopdf` | Declares **zero volumes** and wrote nothing. Stateless by design — a browser-side PDF tool. | No — but the gate cannot distinguish "stateless" from "not exercised hard enough", and saying CLEAN would claim a check that was never made. |
| `uptime-kuma` | Healthy, 302 to `/setup`, **0 entries in `docker diff`**, volume empty. v2 creates its database only when the setup wizard is submitted. | Unknown. Needs a completed wizard to decide. |
| `privatebin` | Wrote nothing to `/srv/data`; no paste was ever posted. (Its `/run`, `/tmp`, `/var/lib/nginx/tmp` anonymous volumes are runtime state — see §3.5.6.) | Unknown. Needs a posted paste to decide. |
| `recipe-importer` | Its one mount `/data` stayed empty and the only writes were at `/`. | Unknown. Needs a real import to decide. |
| `glance` | **Crash-loop** (`restarting`, exit 1). It needs `/app/config/glance.yml` and the template mounts an empty config volume without seeding one. | Pre-existing and already recorded — Campaign 7 §6.2 proved the same failure on the pre-campaign pin. Not a persistence finding; the app never ran. |
| `plant-it` | `pull access denied for msdeluise/plant-it, repository does not exist` — no container was created. | No. The app is `lifecycle: abandoned` for exactly this reason (CHANGELOG 2026-07-21); its dead image is the expected end state. |
| `wanderer` | Container `unhealthy`. Its `/pb_data` (6) and `/meili_data` (11) did receive data, so the app is not obviously broken — but a verdict on an unhealthy app would be a verdict on a half-started one. | Unknown. Needs a healthy deploy to decide. |

### 4.3 What the CLEAN column actually establishes

For the 43 CLEAN apps the claim is bounded, and worth stating precisely: **the data each app wrote
during a first-boot-plus-one-HTTP-request exercise landed inside a mount the template preserves,
none of it landed in the writable layer, and every mount was writable by the app's own uid.** It is
not a claim that every future write will land there — an app that stores uploads correctly but
writes, say, a search index elsewhere on first use would still read CLEAN here. That is the
residual gap, and it is what the periodic re-run in §7.1 exists for.



## 5. Fixes — two made, each with a survives-a-redeploy proof

Every corrected path is confirmed by **two independent sources**, never inferred from a directory
name: the shipped image's own environment/`Config.Volumes`, and upstream's reference compose.
Guessing at a data path is how this defect class was created.

Neither app is deployed anywhere in the fleet (hub `/apps` lists 10 deployed apps; neither is among
them), so neither fix strands anything. **papra is deployed and is therefore NOT fixed** — §6.1.

### 5.1 `gramps-web`

Mounted `/app/data`, `/app/media`, `/tmp`. **`/app/data` appears nowhere in the image's environment**
— the application never writes it. Now mounts the eight paths the image itself names, matching
upstream's reference compose:

| volume | path | what lives there |
|---|---|---|
| `gramps_users` | `/app/users` | `GRAMPSWEB_USER_DB_URI` — the accounts database |
| `gramps_db` | `/root/.gramps/grampsdb` | `GRAMPS_DATABASE_PATH` — **the family tree** |
| `gramps_index` | `/app/indexdir` | `GRAMPSWEB_SEARCH_INDEX_DB_URI` |
| `gramps_media` | `/app/media` | `GRAMPSWEB_MEDIA_BASE_DIR` (was already correct) |
| `gramps_thumb_cache` · `gramps_cache` · `gramps_secret` · `gramps_tmp` | | caches, exports, secret |

**Proof — `proofs/gramps-web/survive-gramps-web.json`.** Deploy → exercise → `docker compose down`
(no `-v`; that is a redeploy, `down --volumes` is the uninstall path) → `up -d` → re-fingerprint by
sha256 **and inode**:

```
gramps-web   SURVIVES   before=5 after=5 survived=4 changed=1 lost=0
  survived (byte-identical, SAME INODE):
    /app/users::users.sqlite                                      <- the accounts database
    /root/.gramps/grampsdb::<uuid>/database.txt                   <- THE FAMILY TREE
    /root/.gramps/grampsdb::<uuid>/name.txt
    /app/thumbnail_cache::<hash>
  changed in place: /app/cache::request_cache/<hash>              <- a request cache, expected
```

### 5.2 `wishlist`

Mounted `wishlist_data:/data` — again a path the application never writes. The image declares
`VOLUME /usr/src/app/data` and `VOLUME /usr/src/app/uploads` (`Config.Volumes`), and upstream's own
compose mounts exactly those two, so `prod.db` was landing in the **anonymous** volume docker
creates for an image `VOLUME` directive: invisible to `ResolveDockerVolumeNames`, therefore never
backed up, and orphaned by `down` + `up`. Now mounts both real paths.

**Proof — `proofs/wishlist/survive-wishlist-content.json`.**

```
before redeploy:  prod.db  inode=47151  size=307200  sha256=f35df23b…
after  redeploy:  prod.db  inode=47151  size=307200  sha256=f35df23b…
same_inode=true   verdict: SURVIVES
```

Same file object, byte-identical content.

> **A correction, and the limit of this proof.** The generic prover (`state/survive2.py`) first
> reported `DATA-LOST` for wishlist. That was the prover, not the app: its rule counts any change
> in bytes as a loss, and cannot separate "the app modified its own database on boot" (fine —
> vaultwarden's SQLite WAL does exactly that) from "the app recreated it" (data gone). The focused
> check settles it on inode. **And the limit:** the seeded content step 404'd — the signup endpoint
> was a guess and wrong — so this is a **file-level** proof, not a user-row-level one. Byte-identical
> content means nothing in the database was lost, but no row was independently confirmed present
> beforehand.

### 5.3 Both fixes re-checked with the gate that convicted them

The strongest available confirmation is the same instrument reaching the opposite verdict, with its
own two-way self-test green on both runs. `proofs/fixed-check/`:

```
self-testing the prober (both directions)…
  prober flags the R-156 signature and clears a correct template — trustworthy
gramps-web           CLEAN
wishlist             CLEAN
2 clean · 0 broken · 0 undetermined (of 2 in scope)
volume-persistence gate OK          rc=0
```

Before the fixes the same gate reported both BROKEN (`state/verify.log`).

### 5.4 papra — the fix is proven, and deliberately not applied

Run in the scratch guest only, on a temporary copy of the template
(`proofs/_papra-fix-proof/papra-fix-proof.json`). **A real account was created through papra's own
API in both legs** (`sign-up -> HTTP/1.1 200 OK`), so real user data was written to the database
before each redeploy:

| leg | `db.sqlite` before | after | |
|---|---|---|---|
| **current** `papra_data:/app/data` | `1e9b695a…` | `5d179bf1…` | **different — the account is gone** |
| **fixed** `papra_data:/app/app-data` | `9dd5cf37…` | `9dd5cf37…` | **byte-identical — it survives** |

An incidental confirmation from the same run: under the fixed template `/app/data` reports
`present: false`. **The directory does not exist in the papra image at all** — it existed only
because the template mounted a volume there. So the volume was not merely unwritable; it was
mounted at a path the application has no concept of.

**Limit of this proof:** the document-upload step did not complete (the follow-up
`/api/organizations` call returned `auth.unauthorized` — a session-cookie handling problem in the
harness, not in papra), so the user data verified is the account, not an uploaded document.

**This fix is not committed.** §6.1.

---

## 6. Referred to the operator — not decided here

### 6.1 papra is deployed. Its fix would strand the data it is meant to protect.

**The fix is prepared and deliberately NOT committed.**

```diff
       volumes:
-      - papra_data:/app/data
+      - papra_data:/app/app-data
```

**Why it is a strand risk.** The volume *name* is unchanged, so no volume is orphaned — but the
mount target changes, which makes `docker compose up -d` **recreate the container**, and the
container's writable layer is where papra's documents and database currently live. Applying this to
a running papra destroys exactly the data the change exists to protect.

**The honest framing, both halves.** That data is already living on borrowed time: an image bump, a
`RedeployFromEnv`, a restore, or any other container recreation destroys it just the same, and no
backup has ever contained it. But the fix is what *schedules* the loss, on a date the customer does
not choose. Scaling that decision down is not this session's call.

Note that pushing the template alone destroys nothing: the controller's git-sync updates the two
template files and does **not** restart the stack (`CLAUDE.md`, deploy contract). The change would
take effect at the next deploy. It is still an armed change, which is why it is held.

**Which box — determined as far as the fences allow, and no further.**

| checked | result |
|---|---|
| hub `/apps` (read-only) | **Papra: 1 deployment**, 218 MB avg / 250 MB P95 — it is live somewhere |
| `demo-felhom` guest 9201 | **not there** — `docker ps -a` shows no papra container, `docker volume ls` no papra volume |
| hub `/configs` | boxes currently reporting: `c10-soak`, `demo-felhom`, `demo-hp` (`david` —, `drill-r50` BLOCKED, `peti-felhom` DOWN, `sess-f` DOWN) |
| hub `/hosts/<id>` and `/apps/papra` | neither page attributes a deployment to a host |

So it is on **`c10-soak`** or **`demo-hp`**, both of which live on demo-hp hardware that this
session is fenced from. `c10-soak` is the likelier of the two and would make this moot: it is
Campaign 10's own rig, its hub customer record reads *"Campaign 10 two-storage soak (SCRATCH -
delete at teardown)"*, and papra was deployed on it **to find R-156 in the first place** — so the
"customer data" would be the campaign's own canary. **That is a guess, and it is labelled as one.**
One `docker ps` on either box settles it, and the other session owns both.

**What is asked:** confirm which box, and then either (a) it is `c10-soak` → push the fix, nothing
of value is lost, or (b) it is a demo box someone cares about → copy `/app/app-data` out of the
running container first (`docker cp papra:/app/app-data <dest>`), then push, then copy back into
the new volume. **No migration is shipped here** — §5 of the brief, and inventing one is how this
class of defect gets worse.

### 6.2 Nothing else needed a ruling

Both other BROKEN apps have an unambiguous correct path — each confirmed by **two independent
sources**, never inferred — and neither is deployed anywhere in the fleet, so both were fixed
outright (§5). No app was left BROKEN for want of a decision, and no UNDETERMINED app was touched.

---

## 7. Part 4 — the gate

### 7.1 Where catalog gates live, and what runs them

They live in `app-catalog-felhom.eu/scripts/` as dependency-free Python that exits non-zero on
violation, and **nothing automated runs them.** There is no CI in this repo — no `.gitea/workflows`,
no `.github`, no drone/woodpecker config; the only YAML outside `templates/` is generated customer
output. Each gate is invoked by whoever edits the catalog, and `CLAUDE.md` says when.

That is the honest description, and it is stated rather than glossed because this project's record
on the point is unambiguous: R-29 has three orphaned gates and one enforced one, and the enforced
one is the only one that has ever stopped anything. **This gate is at the same enforcement level as
its two siblings — convention, not CI.** What would raise it is named in §8 as a proposed register
entry; building it would mean touching the controller's sync path, which is out of scope here
(§5 fences) and would need a decision about failing a customer's template sync.

The gate could be built here, so it was. It did not have to wait for `felhom.eu`.

### 7.2 Why the gate has to be a runtime check — measured, not assumed

A static audit of all 53 composes — every declared top-level volume attached to a service, no
anonymous mounts, no host binds outside the felhom roots — reports the catalog **completely clean**.
The single hit is `homepage`'s deliberate read-only `/var/run/docker.sock`.

**It reports papra clean too.** papra's compose is well-formed: it declares `papra_data` and mounts
it at `/app/data`. Nothing in the file is wrong. What is wrong is only observable at runtime.

So a static gate would have been a gate that could never have caught the defect it was written for —
coverage that isn't. `scripts/check-volume-persistence.py` deploys, exercises, and measures.

### 7.3 What it refuses

**Exit 1 — REFUSED, not a warning:**

- DATA in a container's writable layer (never persisted, never backed up);
- an app-data mount the app's own uid cannot write (R-156's second leg);
- data in an **anonymous** volume (never backed up, orphaned by a redeploy);
- a database-signature file in the writable layer whose bytes **differ** from the image's copy.

**Exit 2 — UNDETERMINED, and explicitly *not* a clean bill of health:** a container that never
reached running/healthy; nothing written anywhere; a suspect that could not be adjudicated; or —
load-bearing — **the prober failing its own two-way canary**, in which case the gate refuses to
report any verdict at all rather than issue a false all-clear.

**Exit 0** only when every app in scope is CLEAN.

### 7.4 The test through the path that actually invokes it

`scripts/test_check_volume_persistence.py` — **31 tests, no Docker** (the prober is injected).
They drive **`check()`, the function `__main__` calls**, not just the verdict logic: a gate whose
decision logic is tested but whose entry point is not has shipped inert in this project before
(the seam-wiring rule, four recorded instances).

Covered at the entry point: one broken app ⇒ **rc 1** and the app is named; all clean ⇒ rc 0;
undetermined ⇒ **rc 2, never rc 0**; broken outranks undetermined; out-of-circulation apps skipped
and reported; **a prober that flags nothing ⇒ rc 2**; **a prober that flags everything ⇒ rc 2**.

Every rule was red-proofed by breaking it and confirming the suite goes red:

| break | suite |
|---|---|
| score `C` entries as data again (the calibre-web false positive) | **RED** |
| revert the trailing-slash noise normalisation (the crafty false positive) | **RED** |
| fold "wrote nothing" into CLEAN | **RED** |
| warn instead of refuse (rc 0 on broken) | **RED** |
| drop the writable-layer leg (the papra signature) | **RED** |
| let an unresolved suspect fall through to CLEAN | **RED** |
| drop the `base64:` prefix | **RED** |
| *(the deleted all-children-noise rule)* | **GREEN — which is why it was deleted** |

Registered in `CLAUDE.md` (conventions) and `REUSE.md` (the map), in the same commit.

---

## 8. Findings for later filing — PROPOSED TEXT, NOT FILED

The register is `felhom.eu/documentation/backlog/{ROADMAP,OPEN-ITEMS}.md`, and `felhom.eu` is
fenced this session. These are drafts for one session to file, in one place.

**Register grep before proposing** (read-only, `ROADMAP.md` + `OPEN-ITEMS.md`):
`wishlist` -> 1 hit, R-41's deployability gate, unrelated · `gramps` -> 0 · `anonymous` -> 2 hits,
both about the golden bake, unrelated · `ResolveDockerVolumeNames` -> 0 · `DumpAppVolumes` -> 0 ·
`writable layer` -> 0 · `check-volume-persistence` -> 0. **Highest filed: R-155.** So these start
at **R-158**.

### 8.0 First, a process finding — R-156 and R-157 are themselves unfiled

`grep -c "R-156\|R-157"` over both register files returns **0 / 0**. They exist only inside
`audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`. This is the identical failure the brief cites
for R-154/R-155 — *"minted in a spike doc and never carried across"* — recurring immediately, on
the very finding this session was commissioned to act on. **R-156 is the reason this work happened
and it is not in the register.** Whoever files the entries below should carry R-156 and R-157
across at the same time.

### R-158 — wishlist's data lands in an ANONYMOUS volume: never backed up, orphaned by a redeploy

**FIXED HERE** (`templates/wishlist/docker-compose.yml`); filed to record the class.

The image declares `VOLUME /usr/src/app/data` and `VOLUME /usr/src/app/uploads`
(`docker image inspect ghcr.io/cmintey/wishlist:v0.66.0` -> `Config.Volumes`). The template mounted
`wishlist_data:/data` — **a path the application never writes**. So `prod.db` went into the
anonymous volume docker creates for an image `VOLUME` directive.

**Why this is worse than an ordinary miss: it survives a restart.** `ResolveDockerVolumeNames`
(`felhom-controller/internal/appbackup/appdata.go`) returns `<project>_<name>` only for volumes
**declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one — and
`compose down` + `up` orphans it and mounts a fresh empty one. Meanwhile `wishlist_data` was tarred
nightly as an empty directory. A store that survives a restart, loses on redeploy, and is never in
a backup is harder to notice than papra's, because casual testing finds nothing wrong.

**The class, not just the app:** any image `VOLUME` directive at a path the template does not
explicitly mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data`
(`Config.Volumes = {"/data":{}}`) — empty when measured, so nothing is at risk now, but anything
immich ever writes there is invisible to backup. Suggested standing rule for `REUSE.md`: *a template
must mount every path in its image's `Config.Volumes`, or state why not.* Checkable statically only
if the image is pulled, which is why it lives in the runtime gate.

### R-159 — gramps-web persisted three paths and wrote to none of them

**FIXED HERE** (`templates/gramps-web/docker-compose.yml`).

The template mounted `/app/data`, `/app/media` and `/tmp`; **`/app/data` appears nowhere in the
image's environment** — the application never writes it. Its accounts database
(`GRAMPSWEB_USER_DB_URI=sqlite:////app/users/users.sqlite`) and **its family tree**
(`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the container's writable layer.
Upstream's reference compose persists **eight** paths; this template persisted three, one a phantom.

Severity above papra's: papra loses documents a customer uploaded, which they may still hold
elsewhere. gramps-web loses the family tree — the artefact the customer *built inside the app*, of
which no other copy exists by construction.

### R-160 — the gate exists, and nothing automated runs it

The check R-156 asks for now exists (`scripts/check-volume-persistence.py`, §7). Its enforcement
level is **convention**: the catalog repo has no CI of any kind (`.gitea/workflows`, `.github`,
drone/woodpecker — none), so like its two sibling gates it runs when a person runs it.

This project's own record says that is not enough: R-29 has three orphaned gates and one enforced
one, and the enforced one is the only one that has ever stopped anything. Two directions, neither
buildable from this repo:

1. **CI in the catalog repo** — a Gitea Actions runner. Nothing exists to build on today.
2. **Controller-side at sync** — the controller already validates `.felhom.yml` at template load
   (`internal/stacks/metadata.go:336-358`) and a persistence assertion could ride the same choke
   point. But that needs a runtime probe on a customer box, and a decision about what a failing
   sync should do to a customer's working app. **Cross-repo, and a design question, not a task.**

Interim position, recorded in `CLAUDE.md`: run it when a template's `volumes:` block or image tag
changes, and at the start of every catalog campaign.

### R-161 — `docker diff` is the only witness, and its failure mode is quiet

A limitation, not a defect. The gate's power comes from `docker diff` excluding mounted paths, which
makes "in the writable layer" mechanically decidable — an implementation detail of the overlay
storage driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the
mount-occupancy and writability legs and would not say so. The canary self-test *would* catch it
(the broken canary would stop reporting BROKEN, and the gate would then refuse to report at all),
so it fails closed — but the message it prints would blame the prober rather than the driver.

---

## 9. Teardown — all three layers

Per `felhom.eu/documentation/PROMPT-TEMPLATE.md` §13. Layer 3 is the one that has been missed every
time (`drill-r50`, `sess-c`, `sess-d`), so it is stated explicitly rather than implied.

### Layer 1 — the machine

`pct stop 9301 && pct destroy 9301 --purge`, and its `local-lvm` volume with it. Verified by
`pct list` (no 9301) and `lvs` (no `vm-9301-disk-0`). **Evidence: `state/teardown.txt`.**

### Layer 2 — the host

`pvesm status` before and after, and the space actually returned.

| | `local-lvm` available (KiB) | used |
|---|---|---|
| before the run began | 258 702 410 | 29.27 % |
| end of run, before teardown | 169 968 909 | 53.53 % |
| after teardown | *see `state/teardown.txt`* | |

The guest's 120 G volume was thin-provisioned on an **over-subscribed** pool (`pct create` warned:
sum of thin volume sizes 402 G against a 348 G pool), which is why a disk watchdog ran throughout
(`state/diskwatch.log`) and why the space actually returning is measured rather than assumed.

### Layer 3 — the hub

**Nothing to remove: this run created no hub record of any kind, by design.**

That is an absence of *creation*, not an absence of checking. The scratch guest ran **no
felhom-controller** and was never enrolled (§1), precisely so no customer or appliance record could
exist to orphan. The hub was touched **read-only** throughout — `/apps`, `/apps/papra`, `/configs`,
`/hosts`, `/hosts/<id>`, over the ClusterIP with Basic auth — to answer "is papra deployed
anywhere?" (§6.1).

Confirmed against the live hub `/configs` list, unchanged from the pre-run state and containing no
entry for this run: `c10-soak`, `david`, `demo-felhom`, `demo-hp`, `drill-r50`, `peti-felhom`,
`sess-f`.

The scratch host paths the gate creates for `${HDD_PATH}` / `${IMPORT_PATH}` (`/srv/felhom-gate/…`)
and the `felhom-volgate-canary:1` image existed only inside the guest, so both go with layer 1.

---

## 10. What did not happen, and why

- **papra was not fixed.** It is deployed, and the fix would destroy the data it protects (§6.1).
  Prepared, proven, held. The brief says stop rather than ship a migration; no migration was written.
- **No data migration of any kind was shipped**, for papra or anything else.
- **No UNDETERMINED app was fixed.** Seven remain undecided (§4.2) and none was nudged toward CLEAN
  to tidy the table. Four need a deeper exercise than a first-boot HTTP request (a completed setup
  wizard, a posted paste, a real import); three could not be started at all.
- **`glance` was not fixed** even though its defect is understood (it crash-loops for want of a
  seeded `/app/config/glance.yml`). It is a deployability defect, not a persistence one, already
  recorded as pre-existing in Campaign 7 §6.2 — outside this brief's scope.
- **`plant-it` was not touched.** Its image does not resolve; it is `lifecycle: abandoned` for
  exactly that reason.
- **Nothing outside the catalog templates was changed.** The controller-side observations
  (`ResolveDockerVolumeNames` cannot see anonymous volumes; `ParseComposeClassifiableBinds` cannot
  see named volumes at all) are findings for §8, not edits.
- **No commit to `felhom.eu`, no register edit, nothing on `demo-hp`.** The other session owns all
  three. Every R-number in §8 is proposed text.
- **No hub-side state was created.** The hub was read only — `/apps`, `/configs`, `/hosts`,
  `/apps/papra` — via the ClusterIP with Basic auth. Nothing to clean up, so nothing recorded as a
  command (R-131's four orphans).
- **Guest 9201 on `demo-felhom` was read, not touched** — `docker ps -a` and `docker volume ls`,
  to establish that papra is not deployed there.
- **`claude-in-chrome` was not used** — unavailable on DooPlex. Every observation is endpoint-level
  or filesystem-level, and each is named where it is used.
- **No secret value was written anywhere.** The probe records env var KEYS only; generated
  credentials are minted per-run and discarded with the container.

---