docs: R-172 CLOSED (hub v0.88.0), R-173 filed, session report

R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been
applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite,
which ignores unknown parameters without an error. Recorded that way so nobody
re-reads it as "SQLite was slow".

R-173 NEW: while establishing who copies hub.db for the WAL change, found
pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with
backup-daily and backup-weekly the only recurring jobs and both on the default
group — so the hub database has no volume-level backup, and it holds every box's
break-glass root password plus the escrow custody records. Filed, not fixed:
whether the exclusion is deliberate is an operator question.

The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the
parallel-session rule — REPORT.md belongs to the controller session that ran
immediately before this one.

It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub
pod three times against a 256Mi limit. Not the WAL change, and not a test I
should have run against a Tier-2 box; the unit tests already proved the property.

REPORT-r172-hub-wal.md

@@ -0,0 +1,160 @@
+# REPORT — hub v0.88.0: the WAL that never was (R-172), plus R-173 found
+
+**Session artefact naming:** written as `REPORT-r172-hub-wal.md`, not `REPORT.md`, per this repo's
+parallel-session rule — the shared `REPORT.md` belongs to the controller boot-recovery session that
+ran immediately before this one and must not be clobbered.
+
+**Repo:** `felhom.eu` (hub `v0.87.0` → **`v0.88.0`**) · **Trigger:** a `HOST STALE` banner the
+operator spotted on `hub.felhom.eu` after the previous session finished.
+
+---
+
+## 1. What the alarm actually was
+
+**Not the agent, not the guest.** The agent was up **2 days**, never restarted, and actively
+reconciling; the controller was reporting normally (the header read "Last report just now", 0.190.0,
+10/10 containers). The failure was the hub **writing** the host report:
+
+```
+20:26:34 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
+20:41:32 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY)
+20:42:32 [INFO]  Host staleness: demo-felhom-8363b5 ok → stale (host_stale)
+20:42:33 [INFO]  Operator email sent for demo-felhom/host_stale
+```
+
+The chain, and the margin is exactly one retry wide: reports are every **15 min**, staleness fires at
+**30 min**, the hub returns **500** on `SQLITE_BUSY` without retrying, and the agent logs
+`keeping current interval` and waits a full interval without retrying either. **Two consecutive
+collisions = a false alarm.** It had already fired once that day (19:12:32, recovered 19:20:32).
+
+**Was it caused by the preceding session?** Partly amplified, not caused. 13 collisions in one pod
+lifetime; **the first at 15:56 CEST, ~3 h before that session's first deploy**. 7 of 13 fell inside
+its window of ~13 controller restarts, which raises write concurrency — so the burst made a
+pre-existing fault more likely, and the fault was not new.
+
+## 2. Root cause — the pragmas were never applied
+
+The DSN was `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. The driver
+is **modernc.org/sqlite v1.45.0**, whose `applyQueryParams` (confirmed at source in the module cache)
+reads only `_pragma`, `_time_format`, `_time_integer_format`, `_txlock` and `_inttotime` — and
+**silently ignores everything else**. No error, no warning.
+
+So the hub ran in the default **rollback-journal** mode with **`busy_timeout=0`** for its entire life
+while its own source said WAL. In rollback-journal mode a reader excludes a writer, so rendering an
+operator page can block a host report — which is precisely the observed 500.
+
+**The observable that proved it before any code changed:** a 128 MB `/data/hub.db` with **no
+`-wal`/`-shm` file beside it while the database was open**. In WAL mode those must exist.
+
+This is the project's recurring class — a configuration asserting an invariant the code does not
+provide — and it is the second one this week.
+
+## 3. The fix
+
+```
+?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
+```
+
+| Parameter | Why it is not optional |
+|---|---|
+| `journal_mode(WAL)` | readers and one writer proceed concurrently, so a page render can no longer block a report; it is a property of the database FILE and persists once set |
+| `busy_timeout(5000)` | writers still serialise; without a timeout SQLite returns `SQLITE_BUSY` *immediately* rather than waiting |
+| `_txlock=immediate` | **the one that is easy to miss.** `database/sql`'s `Begin()` is DEFERRED, so a read-then-write transaction must upgrade its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT` — which **`busy_timeout` does not retry**. This store has **10+ `db.Begin()` sites and they are all write paths** (customer delete/reset, wg, appliance, pbsdr, telemetry, log bundles). WAL + busy_timeout alone would have shipped half a fix with a known un-retryable path left open |
+
+**Retry options (b) and (c) from R-172 were deliberately NOT taken.** With readers no longer blocking
+writers and the upgrade path covered, a `SQLITE_BUSY` reaching a handler should now be rare enough to
+be a real signal; a retry would hide it. Revisit only on evidence.
+
+## 4. Tests and the red-proof
+
+**Every assertion reads the value back from the DATABASE, never the DSN string** — a string assertion
+would have passed happily for the entire life of the bug. Six tests in `internal/store/pragma_test.go`:
+
+| Test | Asserts |
+|---|---|
+| `TestStorePragmasAreActuallyApplied` | runtime `journal_mode` = wal, `busy_timeout` ≥ 5000 |
+| `TestStoreWALFilesExistWhileOpen` | `-wal`/`-shm` exist beside an open DB — **the production signature, pinned** |
+| `TestStoreReaderDoesNotBlockWriter` | the CONSEQUENCE: a write during a held read succeeds |
+| `TestStoreConcurrentWritersDoNotReturnBusy` | 8 concurrent writers all wait rather than error |
+| `TestStoreTransactionUpgradeDoesNotReturnBusySnapshot` | 6 racing read-then-write transactions all commit |
+| `TestSQLiteDriverIgnoresMattnStyleParams` | guards the ROOT CAUSE: fails if the pragmas are "tidied" back to mattn form; skips itself with instructions if a future driver starts honouring them |
+
+**Red-proof — restore the DSN that shipped.** Observed FAIL, then reverted with a passing control:
+
+```
+journal_mode = "delete", want "wal"
+hub.db-wal is missing beside an OPEN database
+a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY)   ← the live error, exactly
+```
+
+`go build ./... && go vet ./... && go test ./...` in `hub/` → **rc=0**. `scripts/repo_gates.py --fast`
+→ all 5 gates OK.
+
+## 5. Operational consequence — handled, not discovered later
+
+**A WAL database cannot be copied by taking `hub.db` alone.** A committed transaction may still live
+in `hub.db-wal`, so a bare `cat` yields a copy that **opens cleanly and silently omits the newest
+writes** — the worst possible shape for a credential lookup. The break-glass root-password retrieval
+in `documentation/operations/nodes.md` used exactly that command, and `_recovery-inventory-2026-07-28.md`
+records it as a past action that reads like a recipe. Both are now WAL-aware: copy the `-wal`
+alongside, `|| true` because an absent `-wal` is legitimate, and **shred both** (the WAL holds the
+same secrets).
+
+Not hypothetical: the live `-wal` measured **729,272 bytes** during verification, all of which a bare
+`cat` would have dropped.
+
+## 6. Live verification
+
+- Image built and pushed; **`manifests/hub.yaml` 0.87.0 → 0.88.0** (the only thing ArgoCD deploys
+  from), hard-refresh + deliberate sync (auto-sync is OFF). ArgoCD **Synced / Healthy**.
+- `/data/` now shows **`hub.db-wal` and `hub.db-shm`** beside the open DB — the exact observable whose
+  absence proved the bug.
+- **Zero `SQLITE_BUSY` since the rollout.**
+- Host report landed at 21:11:33; staleness checker reports `2 ok, 0 stale` — `demo-felhom-8363b5`
+  is back to `ok`.
+- `PRAGMA integrity_check` → **`ok`**, `journal_mode` → **`wal`**, all tables intact
+  (hosts 4, customer_configs 6, host_recovery 4, host_reports 2756, events 2293). Read via the new
+  WAL-aware copy recipe and shredded afterwards.
+
+## 7. A mistake I made, and it caused a real outage
+
+**I OOM-killed the hub pod three times** with a 60-concurrent page-render load test intended to prove
+the fix under contention. The pod's limit is **256 Mi**; 60 simultaneous renders of a heavy customer
+page exceeded it (`OOMKilled`, exit 137, readiness probe timeouts). The hub was unavailable for parts
+of ~6 minutes and recovered on its own.
+
+Three things worth stating plainly:
+
+1. **It was not the WAL change.** WAL's extra footprint is the 32 KB `-shm` mapping; the OOM was
+   template rendering under concurrency I created.
+2. **I should not have run it.** DooPlex is **Tier 2 — precious**, and the hub is part of the
+   recovery chain. A synthetic load test at that concurrency against a memory-limited pod on that box
+   was the wrong call; the unit tests already proved the property, and the live proof needed was the
+   `-wal` file plus a clean report — both of which I already had.
+3. **It did produce one piece of genuine evidence**, which does not excuse it: after three hard kills
+   the WAL replayed cleanly and `integrity_check` returned `ok`, which is a real (if unplanned)
+   demonstration of WAL crash-safety on this volume.
+
+No data was lost. The pod is `Ready`, restart count 3, serving normally.
+
+## 8. Backlog
+
+- **R-172 → CLOSED**, with the root cause recorded as *the pragmas were never applied*, not as tuning.
+- **R-173 → NEW.** While checking who copies `hub.db` for the WAL change, I found `pvc/hub-data`
+  carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly`
+  are the only recurring jobs and both target `default`. **The hub database has no volume-level
+  backup** — and it holds `host_recovery` (every box's break-glass root password), `host_escrow` +
+  `host_escrow_superseded`, `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg peers.
+  Filed rather than fixed: whether the exclusion is deliberate is a question for the operator, and the
+  manual hot copy recorded in `_recovery-inventory` is not a backup. `grep` established the ID free.
+
+## 9. Observations — noticed, NOT acted on
+
+1. **The hub returns HTTP 500 for a transient lock**, which is what turned a retryable condition into
+   an alarm. Left as-is deliberately (§3) so a surviving `SQLITE_BUSY` stays visible.
+2. **The agent does not retry a failed report**, so one collision costs a full 15-minute interval —
+   half the staleness budget. Same reasoning; if collisions recur, this is the cheaper of the two
+   retry fixes.
+3. **`store.New` sets no `SetMaxOpenConns`**, so `database/sql` may open unbounded connections against
+   a single-writer database. Not changed here — WAL plus the immediate-lock covers the observed
+   failure, and bounding the pool changes latency characteristics that nothing currently measures.

STATUS.md

@@ -66,6 +66,15 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)*
 
 ## Changed since last update
 
+- **2026-08-02** — The false "host offline" warning is fixed, and the cause was not what it looked
+  like. The hub's database was supposed to be in a mode where reading a page cannot block a machine's
+  status update — the code said so, but a one-word syntax difference meant the setting had **never
+  taken effect**, for the hub's whole life. So opening an operator page could make a machine's report
+  fail; two failures in a row crossed the half-hour threshold and sent you an alert about a machine
+  that was up and healthy. It had already done that twice that day. Now genuinely fixed and verified
+  live. **Also found while checking it: the hub's own database is not in any automatic backup** — it
+  holds every machine's emergency password and the escrow records. Filed, not yet fixed.
+
 - **2026-08-02** — Boot recovery finished. Both halves of the power-cut problem are closed: the
   machine records what the customer asked for, and it now waits for the system to finish starting
   before deciding what is missing. Six hard resets in a row, everything back every time. A hole the

documentation/backlog/OPEN-ITEMS.md

@@ -88,7 +88,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
 | **R-157** | ~~**`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (B: controller v0.189.0; A: v0.190.0, 2026-08-02) | — | **Both mechanisms closed. (B)** the container-count signal → recorded intent (R-166). **(A)** the sweep looked ONCE at T+5 s, deriving candidates from a fleet docker was still restoring — 3 of 6 hard resets. Now a **settle-then-sweep window**: sample the fleet every 5 s, settled after 3 identical samples, sweep ONCE at the end; ends on settled OR a 50 s budget, and the log says which. **The budget is 50 s because a test rejected 60 s**: settle+budget+one 30 s retry must stay under the 90 s `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s. A window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps — the grace was NOT widened to hide it (§8.3). **A defect in the fix, found by live validation not review:** `GetStacks()` is the Manager's cache, refreshed by the scheduler every 10 s, so sampling it every 5 s without refreshing let "settled" mean "the cache did not update" — observed missing a container removed 5 s before the window closed. `sampleBootFleet` now refreshes first. **Live: 6/6 hard resets on the shipped build, every app back every time** (settle times 10/40/10/10/15/15 s — i.e. the window routinely waited 2–8× longer than the old fixed 5 s), plus a before/after on ONE app on ONE box: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35, the fixed one found and recovered it at 18:18:50 | — |
 | **R-170** | ~~**The drive-backed boot gate infers a customer's Stop from a container count.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | `shouldRecreateOnBoot` now reads `desired_state` with the SAME three-way table as `isBootOrphan`: `stopped` → never; `running` → recreate whatever the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. `presentStable` untouched and still load-bearing (an absent drive is never recreated here — the very term the boot sweep was missing, R-171). Its comment argued at length FOR the container count and was rewritten; a correct implementation under a comment arguing the opposite is worse than either alone. **The agreement is pinned from BOTH sides** against one fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`) because the two gates cannot be called from one package without an import cycle. **Live on 9201, both halves in one reboot:** calibre-web (drive-backed, `running`, ZERO containers) → `recreating drive-backed app calibre-web`; immich (`stopped`) → `1 drive-backed app(s) left stopped on purpose` | — |
 | **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — |
-| **R-172** | **A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports, and the margin is exactly ONE RETRY wide.** The hub logs `Failed to save host-report from <host>: database is locked (5) (SQLITE_BUSY)` and returns **HTTP 500**; the agent logs `hub: report failed; keeping current interval` and **does not retry**, waiting its full **15-minute** interval. Staleness fires at **30 minutes**, so **two consecutive collisions = a false alarm + an operator e-mail** for a host that is perfectly healthy. Observed 2026-08-02: reports at 18:56:33 + 19:11:33 failed → `host_stale` 19:12:32; reports at 20:26:34 + 20:41:32 failed → `host_stale` 20:42:32 + operator e-mail, while the agent was up 2 days and actively reconciling throughout | **READY (S) — NEW 2026-08-02** | — | **Root cause: `/data/hub.db` is 128 MB in ROLLBACK-JOURNAL mode, not WAL** — no `-wal`/`-shm` file exists beside it while the DB is open. In that mode a writer excludes readers and vice versa, so a UI page render can block a report write. **13 collisions in one pod lifetime, first at 15:56 CEST — ~3 h before that day's controller work — so this is pre-existing**, though a burst of controller restarts amplifies it (7 of the 13 fell in a ~2 h window of heavy restarts). **Three candidate fixes, cheapest first, and they are not exclusive:** (a) `PRAGMA journal_mode=WAL` + a `busy_timeout` on the hub's connection — readers stop blocking writers, which removes most collisions; (b) the hub retries a `SQLITE_BUSY` write once before returning 500 — a transient lock is not an internal error; (c) the agent retries a failed report once instead of waiting a full interval, so ONE collision cannot consume half the staleness budget. **Note the class:** this is a false alarm about a healthy machine, the same class as R-171, and it has already trained one operator e-mail to be noise. Evidence: hub pod `hub-67774ccf4f-74wwx` logs + `felhom-agent` journal, 2026-08-02 | CC |
+| **R-172** | ~~**A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (hub v0.88.0, 2026-08-02) | — | **ROOT CAUSE WAS NOT TUNING — THE PRAGMAS WERE NEVER APPLIED.** `store.New` used `?_journal_mode=WAL&_busy_timeout=5000`, which is **mattn/go-sqlite3** syntax; the driver is **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`/`_time_format`/`_time_integer_format`/`_txlock`/`_inttotime` and **ignores the rest without an error**. The hub ran in rollback-journal mode with `busy_timeout=0` for its entire life while its own source said WAL — a configuration asserting an invariant the code did not provide. Proof: a 128 MB open `/data/hub.db` with **no `-wal`/`-shm` beside it**. **Fix:** `?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate`. **`_txlock=immediate` is not optional** — `database/sql`'s `Begin()` is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**; this store has 10+ `db.Begin()` sites, all write paths. **Retry options (b) and (c) were deliberately NOT taken** — with readers no longer blocking writers a surviving `SQLITE_BUSY` would be a real signal, and a retry would hide it; revisit only on evidence. **Live:** `-wal`+`-shm` now present, **zero `SQLITE_BUSY` since rollout**, host back to `ok`, and `PRAGMA integrity_check` = `ok` with `journal_mode=wal` after three unrelated OOM restarts. **Operational consequence handled:** a WAL DB cannot be copied by taking `hub.db` alone — the break-glass retrieval in `operations/nodes.md` did exactly that and is now WAL-aware (the live `-wal` was 729 KB, i.e. a bare `cat` would have silently omitted it) | — |
+| **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC |
 | **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC |
 | **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `<project>_<name>` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC |
 | **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |