admin
97249099d6
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
documentation/backlog/
Verified-LIVE findings with implementable fix plans that are not yet implemented. Preserved here
(instead of on git branches) per the trunk-based, no-branches rule — the fix itself is implemented later
directly on main, during a normal/supervised session.
-
FIX-M18-NOTES.md — dump re-validation runs every 5 min (perf). FIXED in controller v0.62.0 @
f8afe5c(2026-06-14). (was on the deletedfelhom-controllerbranchfix/m18-dump-validation-cache.) -
FIX-M19-NOTES.md —
deriveStackNamemisattribution edge (low-incidence correctness). FIXED in controller v0.62.0 @6bab68b(2026-06-14). (was on the deleted branchfix/m19-stackname-crossref.) -
FOLLOWUP-golden-default-controller-tag.md — the golden bakes a stale controller (
:0.43.0when queued; had rotted again to:0.85.1by resolution). FIXED in felhom-agent @ceca355(2026-07-03):build-golden.shv2.0.0 makes the controller tag a MANDATORY argument (a required arg cannot rot) and golden 0.98.3 was baked + clean-room-validated (bake → first-boot-current → self-manage → app deploy, on the drill VM — no supervised touch of live guests needed) + published + vouched. Evidence:../audits/DRILL-golden-098-2026-07-03.md.
Related: the live-drive fixspec (../audits/live-drive-fixspec-2026-06-14.md) carries the deferred
supervised items F9 (HDD provisioning/guest-attach), F20-BUG2 (durable_id scheme), F20-BUG3 (async
mkfs) — to be implemented in the agent/golden supervised session.