felhom.eu/documentation/audits/immich-repair-2026-10-14

Immich repairs its broken photos after a power cut (R-1007) — 2026-10-14

Times UTC. Scratch guest 9202 on demo-hp (Tier 0). Immich v3.2.4. No key, token, password, photo or file name of a household in this folder (bench content only).

Part A — how can the box ask Immich without the household's password?

Predictions (written 2026-10-14 before any test):

#	Candidate	Prediction
A1	immich-admin CLI	it exists in v3.2.4 but has only account/maintenance commands (reset admin password, password/OAuth login on/off, list users, media location) — no job command; fails
A2	a scoped API key owned by the box	v3.2.4 has per-permission keys (asset.update for the per-asset job); but an API key acts as its user: the per-asset job checks access per asset, so the admin's key cannot repair another family member's photos; and the box has no clean moment to mint it without the password or the household's session → works only partly, and needs a capture we do not want
A3	Immich's nightly „missing thumbnails"	„missing" reads the database (the asset's file rows), not the disk; nothing short of a database write makes an empty file count as missing → only with A5
A4	queue the same job in Immich's Redis	works, with no key: the job is AssetGenerateThumbnails {id} on queue thumbnailGeneration; done with Immich's OWN queue library inside immich-server (docker exec … node) it is the same job the API queues; breaks if Immich renames the job or the queue (an update can)
A5	a database write	works (the box has DB_PASSWORD) but writes into the household's data; refused unless chosen
A6	the household presses a button	works always; costs the household a step and a full rebuild

Lost originals: the integrity job can be queued the same way as A4, and its report READ from the database (a read, not a write) — or the box compares the database's originalPaths with the disk itself.

Set-up (A/A0-*.txt): Immich deployed through the product on 9202. The admin made as a household: a stranger's admin-sign-up → 401 (the setup gate); the household passed the gate with its dashboard session (felhom_gate cookie) → 201, isInitialized true. A second family member made by the admin. 20 photos as the admin, 10 as the member. From here the household password was used only to look at results as the household; no candidate used it.

Measured:

#	Works?	What it writes into the household's Immich	What the household sees	What an Immich update breaks	What a compromised controller gains
A1 immich-admin	no — v3.2.4's commands: reset-admin-password, password/OAuth/maintenance on-off, list-users, grant/revoke-admin, change-media-location, schema-check, version. No job command (A/A0-photos.txt)	—	—	—	—
A2 scoped API key	partly — needs asset.update + job.create (with asset.update alone: 403 „Missing required permission: job.create"); the admin's own photo 204, the family member's photo 400 „Not found or no asset.update access"; the lost-file report needs maintenance + an admin. The box cannot make the key itself: only the household's session can (A/A2-A3-key-and-missing.txt)	one API key row in the admin's account	„Felhom box repair" under Account settings → API keys; deleting it stops the repair	little: the public API is versioned	the key: update the admin's photos' metadata, run any job — less than it has already
A3 nightly „missing thumbnails"	no (on by default, 00:00) — it reads only the database: no thumbnail/preview ROW, an edited asset without a full-size row, or thumbhash null (asset-job.repository.js streamForThumbnailJob). An empty file never counts; only a database write (A5) would make it	—	—	—	—
A4 Immich's own queue	yes — AssetGenerateThumbnails {id} on thumbnailGeneration (prefix immich_bull), added with Immich's own bullmq inside immich-server (docker exec … node) — the same job POST /api/assets/jobs queues: 5 broken assets of BOTH users repaired in 1.08 s; their 10 files rewritten; 0 of the other 50 files touched. Lost originals: IntegrityMissingFilesQueueAll on integrityCheck → one missing_file row in 0.44 s, read from the database with a SELECT (name, date taken, owner) (A/A4-*.txt)	only what Immich itself writes: the rebuilt thumbnails and its own integrity-report rows	nothing (the admin's Jobs page shows the queue counts while it runs)	the job name, the queue name, the prefix, the report table — internal; an update can rename them	nothing new: the controller already holds the docker socket and DB_PASSWORD
A5 database write	not tried (not chosen)	rows in the household's database	depends	the schema	—
A6 the household presses	always works	—	a note; „thumbnails: all" rebuilds the whole library	—	—

Scan cost (A/scan-20000.txt): 20,000 files in the same layout (10,525 folders), 10 empty → found in 0.29 s (find -xdev -type f -size 0; warm cache — a cold scan cannot be forced inside the guest). Path: <upload>/thumbs/<user id>/<aa>/<bb>/<asset id>_thumbnail.webp / _preview.jpeg; the asset id is the first 36 characters, followed by _ (the prompt's draft regex had -). Teardown: Immich removed through the product; no folder, container, volume or image left (A/teardown.txt).

Part B — the decision (operator)

Question: how should the box ask Immich to repair the broken photos after a power cut?

	Option 1 — Immich's own queue (A4) — my pick	Option 2 — an API key the household makes (A2)
How	the controller runs Immich's own queue library inside immich-server and adds the same job Immich's API adds; reads the lost-file report from Immich's database (read only)	the household makes a key in Immich (admin, 3 permissions) and pastes it into the dashboard; the box calls the public API
Writes into Immich	only what Immich itself writes when it repairs (thumbnails, its report rows)	one API key in the admin's account
The household sees	nothing (a note only if a photo is lost)	the key in its account; one extra setup step
Covers	every user's photos	only the admin's — a family member's photos are refused (measured)
An Immich update can break	the job/queue names and the report table → the catalog pins the tested Immich version; any other version: skip and tell the operator, never guess	little (public API)
Threat	none new: the controller already has the docker socket and the database password	a stored key the box must guard

If you do nothing: the fallback A6 — no automatic repair; a note tells the household to run „thumbnails: all".

Operator answer (2026-10-14, in chat): Option 1 — Immich's own queue. Recorded as 09 §3 decision 236 and in R-1007.

Part C — the build

• Catalog 6a1bb04: Immich's after_unclean_boot block — the two scripts tested on 9202 (C/scripts-on-9202.txt: 4 broken assets of two users repaired in 2.4 s, 0 of 53 controls rewritten; the deleted original reported with its name and date), byte-equal in the template. No min_controller (an older controller ignores the block).
• Controller v0.311.0 (8d862e4, 3b85a0c): internal/aftercrash, internal/stacks/after_unclean_boot.go, the app page's lost-files note, app_repaired_after_crash. 15 guards red-proofed. A security review of 8d862e4 found that a failing command's output could reach the log and the event's problem; fixed before the build (3b85a0c).
• Hub v0.150.0 (822d0393): allow-list + operator-only + per-app cooldown + the counts line; 3 guards red-proofed. Deployed with the operator present (ArgoCD, the hub Deployment only; Synced/Healthy; image 0.150.0 ready).

Part D — the live proof on demo-hp guest 9201 (D/)

Controller 0.311.0 on 9201 by demo-hp's own floor (operator, MinAgent 0.155.0; hub „managed floor SERVED … from declared"). Immich deployed through the product onto hdd_1; the admin through the setup gate as a household (a stranger's sign-up 401, the household's 201). Uploads 1–60 settled (121 thumbnail files, 0 empty) and were copied off as the CONTROLS before the cut; uploads 61–200 ran into the cut.

Step	Result
fences	clear 15:25:00 (demo-hp and Tester 1: no vzdump, PBS client, OS step, restic, lock)
the cut	15:25:04.680 off → 15:25:14.753 on (plug countdown); about 135 uploads done; SSH back 15:25:57
crash verdict	controller 15:28:16 „boot grace 15m0s: the host's last boot (15:25:38Z) followed an UNCLEAN stop"
damage (read only, 15:38)	273 thumbnail files, 82 empty: 41 photos with both thumbnail and preview empty
the run	15:42:46 „1 app(s) with an after_unclean_boot block" → 15:42:56 „immich: found 41 empty file(s), repaired 41, still broken 0, lost 0 (checked=true)" — 10.2 s (app.yaml 10,172 ms)
after	0 empty files, 273 files; the 120 control photo files untouched (size + mtime; the 121st, .immich, is Immich's own marker, rewritten at its start)
the operator	hub 17:42:56 local: app_repaired_after_crash (info) — recorded, not mailed (all repaired)
the household	the app page (200) shows no lost-files card — nothing was lost. The lost-photo note was not exercised live (pinned by unit tests and the 9202 run)
repeat cut	not needed (the first cut left 41 broken)
teardown	Immich removed through the product; no container, volume, image or data folder; /root/sizing and the empty /mnt/sizing-bench skeleton (0 files, this morning's bench) removed

Findings: the event's and the log's duration read 0 (R-1020); the hub build script builds a stale checkout when its pull fails (R-1021). Not this session: Plex was removed on Tester 1 at 15:41 and its controller restarted at 15:42 — another session's work; this session's cut also restarted Tester 1 (VM 341 on the HP), and that session was told.

E — Rollout to every box, the image gate, the shared checkout (2026-10-14 evening)

• Gate: catalog scripts/check-unclean-boot-images.py (unclean-boot-images, --fast: hook and CI). 11 decoy cases (E/catalog-decoys.txt); five mutations of the gate each turned a case red (E/redproof-unclean-boot.txt). Catalog 8f5ad8a.
• Golden 0.311.0: baked in the drill VM, sha efe5f02e…, round trip equal (documentation/tests/golden-0.311.0-2026-10-14/). Vouched by the operator with agent 0.156.1 and min_agent 0.155.0; global floor 0.311.0 / 0.155.0.
• Read-back (E/rollout-readback.txt; the hub pages AND each box's own docker ps): N100, demo-hp and Tester 1 run controller 0.311.0 and agent 0.156.1; app containers kept running (not restarted by the update). demo-hp's own floor 0.311.0 is still set. Tester 2 is off: it gets the floor when it next reports.
• Shared checkout (R-1021): leftovers moved to /mnt/5_hdd/felhom.eu/leftovers/2026-10-14/ (README there); clean at origin/main. build-hub.sh v1.24.0 refuses a stale or dirty tree (E/bh-new.txt; the old script E/bh-old.txt).