From 8ef92a3fa7d7ed946529628a7ff77ba446abef98 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 21:22:14 +0200 Subject: [PATCH] docs: R-172 CLOSED (hub v0.88.0), R-173 filed, session report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite, which ignores unknown parameters without an error. Recorded that way so nobody re-reads it as "SQLite was slow". R-173 NEW: while establishing who copies hub.db for the WAL change, found pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with backup-daily and backup-weekly the only recurring jobs and both on the default group — so the hub database has no volume-level backup, and it holds every box's break-glass root password plus the escrow custody records. Filed, not fixed: whether the exclusion is deliberate is an operator question. The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the parallel-session rule — REPORT.md belongs to the controller session that ran immediately before this one. It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub pod three times against a 256Mi limit. Not the WAL change, and not a test I should have run against a Tier-2 box; the unit tests already proved the property. --- REPORT-r172-hub-wal.md | 160 ++++++++++++++++++++++++++++ STATUS.md | 9 ++ documentation/backlog/OPEN-ITEMS.md | 3 +- 3 files changed, 171 insertions(+), 1 deletion(-) create mode 100644 REPORT-r172-hub-wal.md diff --git a/REPORT-r172-hub-wal.md b/REPORT-r172-hub-wal.md new file mode 100644 index 0000000..cab3013 --- /dev/null +++ b/REPORT-r172-hub-wal.md @@ -0,0 +1,160 @@ +# REPORT — hub v0.88.0: the WAL that never was (R-172), plus R-173 found + +**Session artefact naming:** written as `REPORT-r172-hub-wal.md`, not `REPORT.md`, per this repo's +parallel-session rule — the shared `REPORT.md` belongs to the controller boot-recovery session that +ran immediately before this one and must not be clobbered. + +**Repo:** `felhom.eu` (hub `v0.87.0` → **`v0.88.0`**) · **Trigger:** a `HOST STALE` banner the +operator spotted on `hub.felhom.eu` after the previous session finished. + +--- + +## 1. What the alarm actually was + +**Not the agent, not the guest.** The agent was up **2 days**, never restarted, and actively +reconciling; the controller was reporting normally (the header read "Last report just now", 0.190.0, +10/10 containers). The failure was the hub **writing** the host report: + +``` +20:26:34 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY) +20:41:32 [ERROR] Failed to save host-report from demo-felhom-8363b5: database is locked (5) (SQLITE_BUSY) +20:42:32 [INFO] Host staleness: demo-felhom-8363b5 ok → stale (host_stale) +20:42:33 [INFO] Operator email sent for demo-felhom/host_stale +``` + +The chain, and the margin is exactly one retry wide: reports are every **15 min**, staleness fires at +**30 min**, the hub returns **500** on `SQLITE_BUSY` without retrying, and the agent logs +`keeping current interval` and waits a full interval without retrying either. **Two consecutive +collisions = a false alarm.** It had already fired once that day (19:12:32, recovered 19:20:32). + +**Was it caused by the preceding session?** Partly amplified, not caused. 13 collisions in one pod +lifetime; **the first at 15:56 CEST, ~3 h before that session's first deploy**. 7 of 13 fell inside +its window of ~13 controller restarts, which raises write concurrency — so the burst made a +pre-existing fault more likely, and the fault was not new. + +## 2. Root cause — the pragmas were never applied + +The DSN was `?_journal_mode=WAL&_busy_timeout=5000`. That is **mattn/go-sqlite3** syntax. The driver +is **modernc.org/sqlite v1.45.0**, whose `applyQueryParams` (confirmed at source in the module cache) +reads only `_pragma`, `_time_format`, `_time_integer_format`, `_txlock` and `_inttotime` — and +**silently ignores everything else**. No error, no warning. + +So the hub ran in the default **rollback-journal** mode with **`busy_timeout=0`** for its entire life +while its own source said WAL. In rollback-journal mode a reader excludes a writer, so rendering an +operator page can block a host report — which is precisely the observed 500. + +**The observable that proved it before any code changed:** a 128 MB `/data/hub.db` with **no +`-wal`/`-shm` file beside it while the database was open**. In WAL mode those must exist. + +This is the project's recurring class — a configuration asserting an invariant the code does not +provide — and it is the second one this week. + +## 3. The fix + +``` +?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate +``` + +| Parameter | Why it is not optional | +|---|---| +| `journal_mode(WAL)` | readers and one writer proceed concurrently, so a page render can no longer block a report; it is a property of the database FILE and persists once set | +| `busy_timeout(5000)` | writers still serialise; without a timeout SQLite returns `SQLITE_BUSY` *immediately* rather than waiting | +| `_txlock=immediate` | **the one that is easy to miss.** `database/sql`'s `Begin()` is DEFERRED, so a read-then-write transaction must upgrade its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT` — which **`busy_timeout` does not retry**. This store has **10+ `db.Begin()` sites and they are all write paths** (customer delete/reset, wg, appliance, pbsdr, telemetry, log bundles). WAL + busy_timeout alone would have shipped half a fix with a known un-retryable path left open | + +**Retry options (b) and (c) from R-172 were deliberately NOT taken.** With readers no longer blocking +writers and the upgrade path covered, a `SQLITE_BUSY` reaching a handler should now be rare enough to +be a real signal; a retry would hide it. Revisit only on evidence. + +## 4. Tests and the red-proof + +**Every assertion reads the value back from the DATABASE, never the DSN string** — a string assertion +would have passed happily for the entire life of the bug. Six tests in `internal/store/pragma_test.go`: + +| Test | Asserts | +|---|---| +| `TestStorePragmasAreActuallyApplied` | runtime `journal_mode` = wal, `busy_timeout` ≥ 5000 | +| `TestStoreWALFilesExistWhileOpen` | `-wal`/`-shm` exist beside an open DB — **the production signature, pinned** | +| `TestStoreReaderDoesNotBlockWriter` | the CONSEQUENCE: a write during a held read succeeds | +| `TestStoreConcurrentWritersDoNotReturnBusy` | 8 concurrent writers all wait rather than error | +| `TestStoreTransactionUpgradeDoesNotReturnBusySnapshot` | 6 racing read-then-write transactions all commit | +| `TestSQLiteDriverIgnoresMattnStyleParams` | guards the ROOT CAUSE: fails if the pragmas are "tidied" back to mattn form; skips itself with instructions if a future driver starts honouring them | + +**Red-proof — restore the DSN that shipped.** Observed FAIL, then reverted with a passing control: + +``` +journal_mode = "delete", want "wal" +hub.db-wal is missing beside an OPEN database +a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY) ← the live error, exactly +``` + +`go build ./... && go vet ./... && go test ./...` in `hub/` → **rc=0**. `scripts/repo_gates.py --fast` +→ all 5 gates OK. + +## 5. Operational consequence — handled, not discovered later + +**A WAL database cannot be copied by taking `hub.db` alone.** A committed transaction may still live +in `hub.db-wal`, so a bare `cat` yields a copy that **opens cleanly and silently omits the newest +writes** — the worst possible shape for a credential lookup. The break-glass root-password retrieval +in `documentation/operations/nodes.md` used exactly that command, and `_recovery-inventory-2026-07-28.md` +records it as a past action that reads like a recipe. Both are now WAL-aware: copy the `-wal` +alongside, `|| true` because an absent `-wal` is legitimate, and **shred both** (the WAL holds the +same secrets). + +Not hypothetical: the live `-wal` measured **729,272 bytes** during verification, all of which a bare +`cat` would have dropped. + +## 6. Live verification + +- Image built and pushed; **`manifests/hub.yaml` 0.87.0 → 0.88.0** (the only thing ArgoCD deploys + from), hard-refresh + deliberate sync (auto-sync is OFF). ArgoCD **Synced / Healthy**. +- `/data/` now shows **`hub.db-wal` and `hub.db-shm`** beside the open DB — the exact observable whose + absence proved the bug. +- **Zero `SQLITE_BUSY` since the rollout.** +- Host report landed at 21:11:33; staleness checker reports `2 ok, 0 stale` — `demo-felhom-8363b5` + is back to `ok`. +- `PRAGMA integrity_check` → **`ok`**, `journal_mode` → **`wal`**, all tables intact + (hosts 4, customer_configs 6, host_recovery 4, host_reports 2756, events 2293). Read via the new + WAL-aware copy recipe and shredded afterwards. + +## 7. A mistake I made, and it caused a real outage + +**I OOM-killed the hub pod three times** with a 60-concurrent page-render load test intended to prove +the fix under contention. The pod's limit is **256 Mi**; 60 simultaneous renders of a heavy customer +page exceeded it (`OOMKilled`, exit 137, readiness probe timeouts). The hub was unavailable for parts +of ~6 minutes and recovered on its own. + +Three things worth stating plainly: + +1. **It was not the WAL change.** WAL's extra footprint is the 32 KB `-shm` mapping; the OOM was + template rendering under concurrency I created. +2. **I should not have run it.** DooPlex is **Tier 2 — precious**, and the hub is part of the + recovery chain. A synthetic load test at that concurrency against a memory-limited pod on that box + was the wrong call; the unit tests already proved the property, and the live proof needed was the + `-wal` file plus a clean report — both of which I already had. +3. **It did produce one piece of genuine evidence**, which does not excuse it: after three hard kills + the WAL replayed cleanly and `integrity_check` returned `ok`, which is a real (if unplanned) + demonstration of WAL crash-safety on this volume. + +No data was lost. The pod is `Ready`, restart count 3, serving normally. + +## 8. Backlog + +- **R-172 → CLOSED**, with the root cause recorded as *the pragmas were never applied*, not as tuning. +- **R-173 → NEW.** While checking who copies `hub.db` for the WAL change, I found `pvc/hub-data` + carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` + are the only recurring jobs and both target `default`. **The hub database has no volume-level + backup** — and it holds `host_recovery` (every box's break-glass root password), `host_escrow` + + `host_escrow_superseded`, `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg peers. + Filed rather than fixed: whether the exclusion is deliberate is a question for the operator, and the + manual hot copy recorded in `_recovery-inventory` is not a backup. `grep` established the ID free. + +## 9. Observations — noticed, NOT acted on + +1. **The hub returns HTTP 500 for a transient lock**, which is what turned a retryable condition into + an alarm. Left as-is deliberately (§3) so a surviving `SQLITE_BUSY` stays visible. +2. **The agent does not retry a failed report**, so one collision costs a full 15-minute interval — + half the staleness budget. Same reasoning; if collisions recur, this is the cheaper of the two + retry fixes. +3. **`store.New` sets no `SetMaxOpenConns`**, so `database/sql` may open unbounded connections against + a single-writer database. Not changed here — WAL plus the immediate-lock covers the observed + failure, and bounding the pool changes latency characteristics that nothing currently measures. diff --git a/STATUS.md b/STATUS.md index 75fa30c..2c44eeb 100644 --- a/STATUS.md +++ b/STATUS.md @@ -66,6 +66,15 @@ It notices, quickly, and tells you. *(R-29, R-161, R-168, R-169)* ## Changed since last update +- **2026-08-02** — The false "host offline" warning is fixed, and the cause was not what it looked + like. The hub's database was supposed to be in a mode where reading a page cannot block a machine's + status update — the code said so, but a one-word syntax difference meant the setting had **never + taken effect**, for the hub's whole life. So opening an operator page could make a machine's report + fail; two failures in a row crossed the half-hour threshold and sent you an alert about a machine + that was up and healthy. It had already done that twice that day. Now genuinely fixed and verified + live. **Also found while checking it: the hub's own database is not in any automatic backup** — it + holds every machine's emergency password and the escrow records. Filed, not yet fixed. + - **2026-08-02** — Boot recovery finished. Both halves of the power-cut problem are closed: the machine records what the customer asked for, and it now waits for the system to finish starting before deciding what is missing. Six hard resets in a row, everything back every time. A hole the diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 70c0735..db455fe 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -88,7 +88,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-157** | ~~**`bootrecon`'s start-ONCE sweep misses the boot orphan it exists to recover — TWO mechanisms.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (B: controller v0.189.0; A: v0.190.0, 2026-08-02) | — | **Both mechanisms closed. (B)** the container-count signal → recorded intent (R-166). **(A)** the sweep looked ONCE at T+5 s, deriving candidates from a fleet docker was still restoring — 3 of 6 hard resets. Now a **settle-then-sweep window**: sample the fleet every 5 s, settled after 3 identical samples, sweep ONCE at the end; ends on settled OR a 50 s budget, and the log says which. **The budget is 50 s because a test rejected 60 s**: settle+budget+one 30 s retry must stay under the 90 s `deadAppBootGrace` or a successful recovery stops being silent; 60 s gave 95 s. A window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps — the grace was NOT widened to hide it (§8.3). **A defect in the fix, found by live validation not review:** `GetStacks()` is the Manager's cache, refreshed by the scheduler every 10 s, so sampling it every 5 s without refreshing let "settled" mean "the cache did not update" — observed missing a container removed 5 s before the window closed. `sampleBootFleet` now refreshes first. **Live: 6/6 hard resets on the shipped build, every app back every time** (settle times 10/40/10/10/15/15 s — i.e. the window routinely waited 2–8× longer than the old fixed 5 s), plus a before/after on ONE app on ONE box: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35, the fixed one found and recovered it at 18:18:50 | — | | **R-170** | ~~**The drive-backed boot gate infers a customer's Stop from a container count.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | `shouldRecreateOnBoot` now reads `desired_state` with the SAME three-way table as `isBootOrphan`: `stopped` → never; `running` → recreate whatever the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. `presentStable` untouched and still load-bearing (an absent drive is never recreated here — the very term the boot sweep was missing, R-171). Its comment argued at length FOR the container count and was rewritten; a correct implementation under a comment arguing the opposite is worse than either alone. **The agreement is pinned from BOTH sides** against one fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`) because the two gates cannot be called from one package without an import cycle. **Live on 9201, both halves in one reboot:** calibre-web (drive-backed, `running`, ZERO containers) → `recreating drive-backed app calibre-web`; immich (`stopped`) → `1 drive-backed app(s) left stopped on purpose` | — | | **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — | -| **R-172** | **A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports, and the margin is exactly ONE RETRY wide.** The hub logs `Failed to save host-report from : database is locked (5) (SQLITE_BUSY)` and returns **HTTP 500**; the agent logs `hub: report failed; keeping current interval` and **does not retry**, waiting its full **15-minute** interval. Staleness fires at **30 minutes**, so **two consecutive collisions = a false alarm + an operator e-mail** for a host that is perfectly healthy. Observed 2026-08-02: reports at 18:56:33 + 19:11:33 failed → `host_stale` 19:12:32; reports at 20:26:34 + 20:41:32 failed → `host_stale` 20:42:32 + operator e-mail, while the agent was up 2 days and actively reconciling throughout | **READY (S) — NEW 2026-08-02** | — | **Root cause: `/data/hub.db` is 128 MB in ROLLBACK-JOURNAL mode, not WAL** — no `-wal`/`-shm` file exists beside it while the DB is open. In that mode a writer excludes readers and vice versa, so a UI page render can block a report write. **13 collisions in one pod lifetime, first at 15:56 CEST — ~3 h before that day's controller work — so this is pre-existing**, though a burst of controller restarts amplifies it (7 of the 13 fell in a ~2 h window of heavy restarts). **Three candidate fixes, cheapest first, and they are not exclusive:** (a) `PRAGMA journal_mode=WAL` + a `busy_timeout` on the hub's connection — readers stop blocking writers, which removes most collisions; (b) the hub retries a `SQLITE_BUSY` write once before returning 500 — a transient lock is not an internal error; (c) the agent retries a failed report once instead of waiting a full interval, so ONE collision cannot consume half the staleness budget. **Note the class:** this is a false alarm about a healthy machine, the same class as R-171, and it has already trained one operator e-mail to be noise. Evidence: hub pod `hub-67774ccf4f-74wwx` logs + `felhom-agent` journal, 2026-08-02 | CC | +| **R-172** | ~~**A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (hub v0.88.0, 2026-08-02) | — | **ROOT CAUSE WAS NOT TUNING — THE PRAGMAS WERE NEVER APPLIED.** `store.New` used `?_journal_mode=WAL&_busy_timeout=5000`, which is **mattn/go-sqlite3** syntax; the driver is **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`/`_time_format`/`_time_integer_format`/`_txlock`/`_inttotime` and **ignores the rest without an error**. The hub ran in rollback-journal mode with `busy_timeout=0` for its entire life while its own source said WAL — a configuration asserting an invariant the code did not provide. Proof: a 128 MB open `/data/hub.db` with **no `-wal`/`-shm` beside it**. **Fix:** `?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate`. **`_txlock=immediate` is not optional** — `database/sql`'s `Begin()` is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**; this store has 10+ `db.Begin()` sites, all write paths. **Retry options (b) and (c) were deliberately NOT taken** — with readers no longer blocking writers a surviving `SQLITE_BUSY` would be a real signal, and a retry would hide it; revisit only on evidence. **Live:** `-wal`+`-shm` now present, **zero `SQLITE_BUSY` since rollout**, host back to `ok`, and `PRAGMA integrity_check` = `ok` with `journal_mode=wal` after three unrelated OOM restarts. **Operational consequence handled:** a WAL DB cannot be copied by taking `hub.db` alone — the break-glass retrieval in `operations/nodes.md` did exactly that and is now WAL-aware (the live `-wal` was 729 KB, i.e. a bare `cat` would have silently omitted it) | — | +| **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC | | **R-158** | **A local Tier-1 app-data backup failure reaches no hub channel — `NotifyBackupFailed` exists, the hub allowlists `backup_failed`, and its only production caller is the off-box/NAS leg** (`cmd/controller/main.go:659`). The backup manager has `tier2Notify`/`offboxNotify`/`offboxEnlargeBlockedNotify` seams (`internal/backup/backup.go:33,37,58`) and **none for the recovery-unit capture**. Fifth instance of *seam built but never wired*; R-97's defect one tier over. | **READY (S)** | — | **Ranked BELOW R-157 — it is a notification GAP, not silent failure.** Measured: with `mp1` full, `/backups` DOES render `✗ Adatmentés sikertelen`, the marker **persists** across a second failed run and **clears** on recovery, and `/backups/apps` honestly shows the last good unit's real mtime — no surface claims a fresh backup over a stale unit. **The half worth fixing: `/backups/apps` is where you ask whether one app is backed up, and it is the one page that never says.** Proposed shape: a `unitNotify` seam wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`. Evidence: `audits/SPIKE-recovery-unit-space-2026-08-02.md` §5, `audits/CAMPAIGN-10-closeout-2026-08-02.md` Q1 | CC | | **R-159** | **wishlist's data landed in an ANONYMOUS volume — never backed up, orphaned by a redeploy.** The image declares `VOLUME /usr/src/app/data`; the template mounted `wishlist_data:/data`, a path the app never writes. `ResolveDockerVolumeNames` returns `_` only for volumes **declared in the compose file**, so `DumpAppVolumes` never sees an anonymous one. Survives a restart, loses on redeploy, never in a backup — harder to notice than papra's. | **SHIPPED** (`templates/wishlist/docker-compose.yml`, 2026-08-02) — filed to record the CLASS | — | **The class is open even though the app is fixed:** any image `VOLUME` at a path the template does not mount creates unbacked-up storage silently. **`immich-server` has one today** at `/data` — empty when measured, so nothing is at risk now. Proposed `REUSE.md` rule: *a template must mount every path in its image's `Config.Volumes`, or state why not.* Checkable only with the image pulled, hence the runtime gate | CC | | **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |