# Why immich's database was killed on its first start (R-732) — the cause, measured

2026-09-30, by day. Bench: LXC 9401 on demo-hp (6 cores, 10 GiB, **swap 0**), rebuilt for this. Box: scratch guest 9202 on
demo-hp (7 cores, 25 GiB, **swap 512 MiB**). Both disks are raw images on the same `nvme-scratch` store; both run cgroup v2.
Image: `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` at the ladder's digest `1a078b23…` (PostgreSQL 16.10).
Sampler: `tools/pgsample.sh` — the database container's `memory.current`, `memory.stat` (anon, file, shmem, file_dirty,
file_writeback) and `memory.events` (oom, oom_kill) every 2 s from the container's birth. Every run is a fresh install.

## The mechanism, in one paragraph

On its very first start (and again whenever an update ships a new geodata file), immich imports ~228 000 places into
`geodata_places` in 5 000-row INSERTs, **up to 9 at once** (`server/src/repositories/map.repository.ts` v3.2.2 L284–321:
`if (futures.length >= 9) await Promise.all(futures)`). Each INSERT is its own database process. Together they need
**~400 MB of their own memory (anon)**, on top of **~170 MB of shared_buffers** the import touches (the image's own
`/etc/postgresql/postgresql.conf` fixes `shared_buffers = 512MB`, `work_mem = 16MB` — `S1-image-conf.txt`, `S1-settings.txt`).
About 575 MB does not fit a 512 MB limit. **With swap** the kernel pushes ~70–110 MB of it out and the import survives at the
limit (that is why 9202 passed this morning and at 15:27). **Without swap** the kernel kills a database process — always
the one running `insert into "geodata_places"` (`S1-postgres.log`) — PostgreSQL restarts every backend, the server's worker
dies, and the cycle repeats (61–104 kills in 7–15 min).

## The claims the brief listed, each tested

| candidate | test | result |
|---|---|---|
| 1. the image sizes memory from the HOST's RAM | `pg_settings.source` + the image's config source | **false** — `shared_buffers` 512MB and `work_mem` 16MB come from the image's fixed `postgresql.conf` (`configuration file /etc/postgresql/postgresql.conf`); the rest are PostgreSQL defaults. The container sees the host's RAM (30.7 GB on the bench, 25.9 GB on the box) but nothing reads it |
| 2. the import needs more than 512M at the image's defaults, on any host | limit changed alone | **true without swap**: 1024M → 0 kills (peak anon 403 MB); 768M → 0 kills twice (412/394 MB) |
| 3a. `shared_buffers` | 512MB → 128MB alone (`command:` flag), limit 512M, no swap | **false** — still 104 kills; the import finished only after 14 min of retries (`S3`) |
| 3b. dirty page cache (slow writeback) | `file_dirty` sampled | **false** — 2–13 MB dirty at the kills (`S4`) |
| 3c. `/dev/shm` | read | 64M, 1.1M used — not involved |
| bench vs box = host RAM | configs | **false** — same host; the difference is **swap** (box 512 MiB, bench 0) |
| swap | the bench given 512 MiB swap alone, limit 512M | **true** — 0 kills, swap.peak 74.7 MB (`S6`); on the box swap.peak 119 MB, pswpout 27 669 pages (`box/S2-box-positive.txt`) |

## The two curves (peaks; the full 2-second series are in the files)

| run | limit | swap | peak current | peak anon | oom_kill | outcome |
|---|---|---|---|---|---|---|
| bench S1 — **positive control**, R-732 repeated | 512M | 0 | 512M | 420M | 84 (in 631 s) | never healthy |
| bench S4 — positive again, with dirty pages | 512M | 0 | 511M | 421M | 24 (in 186 s) | never healthy |
| box S2 — same template, fresh | 512M | 512 MiB | 511M | 280M (+~108 MB swapped) | 0 | import done in 20 s, at 99 % of the limit |
| bench S3 — shared_buffers 128MB only | 512M | 0 | 512M | 434M | 104 | import after ~14 min |
| bench S6 — **swap only** | 512M | 512 MiB | 511M | 320M | 0 | healthy, import 8.2 s |
| bench S5 — **limit only** | 1024M | 0 | 956M | 403M | 0 | healthy, import 8.3 s |
| bench S7, S8 — the fix | 768M | 0 | 765M / 759M | 412M / 394M (54 % / 51 %) | 0 / 0 | healthy, import 7.7 / 7.9 s |

**Control pair:** S1/S4 (fail) vs S6 (only swap changed → pass) and vs S5/S7/S8 (only the limit changed → pass); S3 (only
`shared_buffers` changed → still fails). The cause is named on the variables that flip the result.

## Why settings cannot fix it

The 400 MB is nine concurrent INSERT backends; their memory is the statement's own (5 000 rows × 11 columns), not
`work_mem` or `shared_buffers`. immich exposes no knob for the import's concurrency. Lowering `shared_buffers` saves at most
the ~40–60 MB actually touched and was measured not to be enough (S3). So the limit rises: **768M** is the smallest step with
room (anon 51–54 %, under the 80 % `memory_tight` line; shmem ~170 MB and file cache fill the rest, reclaimable).

## Not measured, stated

- Whether a real customer guest has swap: both demo guests have 512 MiB; the golden's guest config is not recorded here.
  The fix does not depend on it (proven with swap off on both venues).
- The PostgreSQL 14 line upstream runs uses the SAME `postgresql.ssd.conf` (base-images copies one file for every major) — the
  same 512MB `shared_buffers`; its import behaviour would be the same. Note only.
- The chaos-night restarts of 2026-09-17 (R-676) fit this mechanism (a DB connection dropped during the first-start import on a
  6 GB guest) — consistent, not re-measured.