107 lines
11 KiB
Markdown
107 lines
11 KiB
Markdown
# Immich repairs its broken photos after a power cut (R-1007) — 2026-10-14
|
||
|
||
Times UTC. Scratch guest 9202 on demo-hp (Tier 0). Immich v3.2.4. No key, token, password, photo or file name of a
|
||
household in this folder (bench content only).
|
||
|
||
## Part A — how can the box ask Immich without the household's password?
|
||
|
||
**Predictions (written 2026-10-14 before any test):**
|
||
|
||
| # | Candidate | Prediction |
|
||
|---|---|---|
|
||
| A1 | `immich-admin` CLI | it exists in v3.2.4 but has only account/maintenance commands (reset admin password, password/OAuth login on/off, list users, media location) — **no job command**; fails |
|
||
| A2 | a scoped API key owned by the box | v3.2.4 has per-permission keys (`asset.update` for the per-asset job); but **an API key acts as its user**: the per-asset job checks access per asset, so the admin's key cannot repair another family member's photos; and the box has no clean moment to mint it without the password or the household's session → works only partly, and needs a capture we do not want |
|
||
| A3 | Immich's nightly „missing thumbnails" | „missing" reads the database (the asset's file rows), not the disk; nothing short of a database write makes an empty file count as missing → only with A5 |
|
||
| A4 | queue the same job in Immich's Redis | works, with no key: the job is `AssetGenerateThumbnails {id}` on queue `thumbnailGeneration`; done with Immich's OWN queue library inside `immich-server` (`docker exec … node`) it is the same job the API queues; breaks if Immich renames the job or the queue (an update can) |
|
||
| A5 | a database write | works (the box has `DB_PASSWORD`) but writes into the household's data; refused unless chosen |
|
||
| A6 | the household presses a button | works always; costs the household a step and a full rebuild |
|
||
|
||
Lost originals: the integrity job can be queued the same way as A4, and its report READ from the database (a read, not a
|
||
write) — or the box compares the database's `originalPath`s with the disk itself.
|
||
|
||
**Set-up (`A/A0-*.txt`):** Immich deployed through the product on 9202. The admin made **as a household**: a stranger's
|
||
`admin-sign-up` → 401 (the setup gate); the household passed the gate with its dashboard session (`felhom_gate` cookie)
|
||
→ 201, `isInitialized` true. A second family member made by the admin. 20 photos as the admin, 10 as the member. From
|
||
here the household password was used only to look at results as the household; no candidate used it.
|
||
|
||
**Measured:**
|
||
|
||
| # | Works? | What it writes into the household's Immich | What the household sees | What an Immich update breaks | What a compromised controller gains |
|
||
|---|---|---|---|---|---|
|
||
| A1 `immich-admin` | **no** — v3.2.4's commands: reset-admin-password, password/OAuth/maintenance on-off, list-users, grant/revoke-admin, change-media-location, schema-check, version. No job command (`A/A0-photos.txt`) | — | — | — | — |
|
||
| A2 scoped API key | **partly** — needs `asset.update` + `job.create` (with `asset.update` alone: 403 „Missing required permission: job.create"); the admin's own photo 204, **the family member's photo 400 „Not found or no asset.update access"**; the lost-file report needs `maintenance` + an admin. The box cannot make the key itself: only the household's session can (`A/A2-A3-key-and-missing.txt`) | one API key row in the admin's account | „Felhom box repair" under Account settings → API keys; deleting it stops the repair | little: the public API is versioned | the key: update the admin's photos' metadata, run any job — less than it has already |
|
||
| A3 nightly „missing thumbnails" | **no** (on by default, 00:00) — it reads only the database: no thumbnail/preview ROW, an edited asset without a full-size row, or `thumbhash` null (`asset-job.repository.js` `streamForThumbnailJob`). An empty file never counts; only a database write (A5) would make it | — | — | — | — |
|
||
| **A4 Immich's own queue** | **yes** — `AssetGenerateThumbnails {id}` on `thumbnailGeneration` (prefix `immich_bull`), added with Immich's own `bullmq` inside `immich-server` (`docker exec … node`) — the same job `POST /api/assets/jobs` queues: **5 broken assets of BOTH users repaired in 1.08 s; their 10 files rewritten; 0 of the other 50 files touched**. Lost originals: `IntegrityMissingFilesQueueAll` on `integrityCheck` → one `missing_file` row in 0.44 s, read from the database with a SELECT (name, date taken, owner) (`A/A4-*.txt`) | only what Immich itself writes: the rebuilt thumbnails and its own integrity-report rows | nothing (the admin's Jobs page shows the queue counts while it runs) | the job name, the queue name, the prefix, the report table — internal; an update can rename them | nothing new: the controller already holds the docker socket and `DB_PASSWORD` |
|
||
| A5 database write | not tried (not chosen) | rows in the household's database | depends | the schema | — |
|
||
| A6 the household presses | always works | — | a note; „thumbnails: all" rebuilds the whole library | — | — |
|
||
|
||
**Scan cost (`A/scan-20000.txt`):** 20,000 files in the same layout (10,525 folders), 10 empty → found in **0.29 s**
|
||
(`find -xdev -type f -size 0`; warm cache — a cold scan cannot be forced inside the guest).
|
||
**Path:** `<upload>/thumbs/<user id>/<aa>/<bb>/<asset id>_thumbnail.webp` / `_preview.jpeg`; the asset id is the first
|
||
36 characters, followed by **`_`** (the prompt's draft regex had `-`).
|
||
**Teardown:** Immich removed through the product; no folder, container, volume or image left (`A/teardown.txt`).
|
||
|
||
## Part B — the decision (operator)
|
||
|
||
**Question: how should the box ask Immich to repair the broken photos after a power cut?**
|
||
|
||
| | Option 1 — Immich's own queue (A4) — **my pick** | Option 2 — an API key the household makes (A2) |
|
||
|---|---|---|
|
||
| How | the controller runs Immich's own queue library inside `immich-server` and adds the same job Immich's API adds; reads the lost-file report from Immich's database (read only) | the household makes a key in Immich (admin, 3 permissions) and pastes it into the dashboard; the box calls the public API |
|
||
| Writes into Immich | only what Immich itself writes when it repairs (thumbnails, its report rows) | one API key in the admin's account |
|
||
| The household sees | nothing (a note only if a photo is lost) | the key in its account; one extra setup step |
|
||
| Covers | every user's photos | **only the admin's** — a family member's photos are refused (measured) |
|
||
| An Immich update can break | the job/queue names and the report table → the catalog pins the tested Immich version; any other version: skip and tell the operator, never guess | little (public API) |
|
||
| Threat | none new: the controller already has the docker socket and the database password | a stored key the box must guard |
|
||
|
||
**If you do nothing:** the fallback A6 — no automatic repair; a note tells the household to run „thumbnails: all".
|
||
|
||
**Operator answer (2026-10-14, in chat): Option 1 — Immich's own queue.** Recorded as `09` §3 decision 236 and in R-1007.
|
||
|
||
## Part C — the build
|
||
|
||
- **Catalog** `6a1bb04`: Immich's `after_unclean_boot` block — the two scripts tested on 9202 (`C/scripts-on-9202.txt`:
|
||
4 broken assets of two users repaired in 2.4 s, 0 of 53 controls rewritten; the deleted original reported with its name
|
||
and date), byte-equal in the template. No `min_controller` (an older controller ignores the block).
|
||
- **Controller** v0.311.0 (`8d862e4`, `3b85a0c`): `internal/aftercrash`, `internal/stacks/after_unclean_boot.go`, the
|
||
app page's lost-files note, `app_repaired_after_crash`. 15 guards red-proofed. A security review of `8d862e4` found that
|
||
a failing command's output could reach the log and the event's `problem`; fixed before the build (`3b85a0c`).
|
||
- **Hub** v0.150.0 (`822d0393`): allow-list + operator-only + per-app cooldown + the counts line; 3 guards red-proofed.
|
||
Deployed with the operator present (ArgoCD, the hub Deployment only; Synced/Healthy; image 0.150.0 ready).
|
||
|
||
## Part D — the live proof on demo-hp guest 9201 (`D/`)
|
||
|
||
Controller 0.311.0 on 9201 by demo-hp's own floor (operator, MinAgent 0.155.0; hub „managed floor SERVED … from
|
||
declared"). Immich deployed through the product onto `hdd_1`; the admin through the setup gate as a household (a
|
||
stranger's sign-up 401, the household's 201). Uploads 1–60 settled (121 thumbnail files, 0 empty) and were copied off as
|
||
the CONTROLS before the cut; uploads 61–200 ran into the cut.
|
||
|
||
| Step | Result |
|
||
|---|---|
|
||
| fences | clear 15:25:00 (demo-hp and Tester 1: no vzdump, PBS client, OS step, restic, lock) |
|
||
| the cut | 15:25:04.680 off → 15:25:14.753 on (plug countdown); about 135 uploads done; SSH back 15:25:57 |
|
||
| crash verdict | controller 15:28:16 „boot grace 15m0s: the host's last boot (15:25:38Z) followed an UNCLEAN stop" |
|
||
| damage (read only, 15:38) | 273 thumbnail files, **82 empty: 41 photos with both thumbnail and preview empty** |
|
||
| the run | 15:42:46 „1 app(s) with an after_unclean_boot block" → 15:42:56 „immich: found 41 empty file(s), repaired 41, still broken 0, lost 0 (checked=true)" — 10.2 s (`app.yaml` 10,172 ms) |
|
||
| after | **0 empty files**, 273 files; **the 120 control photo files untouched** (size + mtime; the 121st, `.immich`, is Immich's own marker, rewritten at its start) |
|
||
| the operator | hub 17:42:56 local: `app_repaired_after_crash` (info) — recorded, not mailed (all repaired) |
|
||
| the household | the app page (200) shows no lost-files card — nothing was lost. **The lost-photo note was not exercised live** (pinned by unit tests and the 9202 run) |
|
||
| repeat cut | not needed (the first cut left 41 broken) |
|
||
| teardown | Immich removed through the product; no container, volume, image or data folder; `/root/sizing` and the empty `/mnt/sizing-bench` skeleton (0 files, this morning's bench) removed |
|
||
|
||
Findings: the event's and the log's duration read 0 (R-1020); the hub build script builds a stale checkout when its pull
|
||
fails (R-1021). Not this session: Plex was removed on Tester 1 at 15:41 and its controller restarted at 15:42 — another
|
||
session's work; this session's cut also restarted Tester 1 (VM 341 on the HP), and that session was told.
|
||
|
||
## E — Rollout to every box, the image gate, the shared checkout (2026-10-14 evening)
|
||
|
||
- **Gate:** catalog `scripts/check-unclean-boot-images.py` (`unclean-boot-images`, `--fast`: hook and CI). 11 decoy cases
|
||
(`E/catalog-decoys.txt`); five mutations of the gate each turned a case red (`E/redproof-unclean-boot.txt`). Catalog `8f5ad8a`.
|
||
- **Golden 0.311.0:** baked in the drill VM, sha `efe5f02e…`, round trip equal (`documentation/tests/golden-0.311.0-2026-10-14/`).
|
||
Vouched by the operator with agent 0.156.1 and min_agent 0.155.0; global floor 0.311.0 / 0.155.0.
|
||
- **Read-back** (`E/rollout-readback.txt`; the hub pages AND each box's own `docker ps`): N100, demo-hp and Tester 1 run
|
||
controller 0.311.0 and agent 0.156.1; app containers kept running (not restarted by the update). demo-hp's own floor
|
||
0.311.0 is still set. Tester 2 is off: it gets the floor when it next reports.
|
||
- **Shared checkout (R-1021):** leftovers moved to `/mnt/5_hdd/felhom.eu/leftovers/2026-10-14/` (README there); clean at
|
||
`origin/main`. `build-hub.sh` v1.24.0 refuses a stale or dirty tree (`E/bh-new.txt`; the old script `E/bh-old.txt`).
|