# REPORT — R-47: the DB replay must not race the app (both restore paths) · felhom-controller v0.153.0

**Date:** 2026-07-20 · **Repo:** `felhom-controller` (v0.152.0 → **v0.153.0**) · Trunk, pushed to
`main`. · **Baseline:** `main` @ `fd40b29` (clean, equal to `origin/main` at session start)

---

## 1. What was wrong

`felhom.eu/documentation/audits/DIAG-immich-restore-round2-2026-07-19.md`, finding **H4**. The
offsite reconstitution ran its designed sequence — safety dump → stop → start → replay — and the
replay aborted:

```
10:58:25  controller: replaying DB dump into immich-postgres
10:58:33  immich-server: "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35  controller: ERROR relation "clip_index" already exists - exit status 3
```

`ImportDump` needs a running database container, so the code started the WHOLE stack first. That gave
the application an eight-second window to rebuild the very schema objects the dump was about to
create; under `ON_ERROR_STOP=1` the collision aborted the script. The data survived only because
`pg_dump` emits COPY before CREATE INDEX — a collision earlier in the script would have left a
genuinely half-restored database and reported it identically.

**Class defect.** The local `RestoreFromRecoveryUnit` had the same start-then-replay shape, hidden
inside `RecreateStackFromUnit` (which ended in a full `compose up -d`). Both are fixed here.

## 2. What was built

**Part 1 — the seams**

| Change | File |
|---|---|
| `dbTypeForImage` extracted from `DiscoverDatabases` (behaviour byte-equivalent) and shared | `internal/appbackup/dbdump.go`, `internal/appbackup/dbservices.go` (new) |
| `DBServiceNames(composePath)` — sorted compose SERVICE names holding a DB; yaml.v3 `services:` map parse | `internal/appbackup/dbservices.go` (new) |
| `Manager.StartStackServices(name, services)` — scoped `up -d`, **refuses an empty list** | `internal/stacks/manager.go` |
| `RedeployFromEnv` split; persist half is `PersistUnitRedeployConfig` (starts nothing) | `internal/stacks/deploy.go` |
| `StackDataProvider`: `RecreateStackFromUnit` → `RecreateStackDefinitionFromUnit` (+ `StartStackServices`) | `internal/appbackup/appdata.go` |
| Adapter: definition-only recreate + delegation | `cmd/controller/main.go` |
| `DBServiceNames` forwarder | `internal/backup/appbackup_bridge.go` |

**Part 2 — offsite** (`internal/backup/offbox_reconstitute.go`): DB services resolved from the LIVE
compose before any mutation; fail-closed refusal when a DB exists but no service is identifiable;
sequence is now **stop → files → `StartStackServices(dbServices)` → replay → `StartStack` (full) →
health wait**; both failure exits from the window do a best-effort full start.

**Part 3 — local** (`internal/backup/restore_unit.go`): DB services resolved from the UNIT's compose
(it is about to become the live one) plus `hasReplayableDump` (excludes `pre-restore-` safety dumps);
same fail-closed gate before the first mutation; sequence is now **stop → volumes →
`RecreateStackDefinitionFromUnit` → `StartStackServices` → replay → `StartStack` (full) → health
wait**, with the pre-existing `dataErr` / "completed with data errors" semantics preserved.

Untouched, as specified: `restore_db.go`, `ImportDump`, `waitDBReady`, the dump flags
(`--clean --if-exists`, `ON_ERROR_STOP=1`), `mapOffsiteRestorePaths`, the copiers, the honesty
surfaces, `IsDownState`/alerting (R-51), and the agent/hub.

## 3. Tests — 19 new, Groups A–G

| Group | Test | Result |
|---|---|---|
| A | `TestReconstituteReplaysWithOnlyTheDBServiceUp` — order **plus state-at-replay-time** | PASS |
| A | `TestReconstituteReplaysDBAndOrdersOperations` (existing, sequence assertion updated) | PASS |
| B | `TestReconstituteNoDBAppNeverStartsServicesOnly` — negative, zero scoped starts | PASS |
| C | `TestReconstituteRefusesWhenNoDBServiceIdentifiable` — zero-mutation effect | PASS |
| C | `TestRestoreFromUnitRefusesWhenNoDBServiceIdentifiable` — zero-mutation effect | PASS |
| D | `TestRestoreFromUnitReplaysWithOnlyTheDBServiceUp` | PASS |
| D | `TestRestoreFromUnitNoDumpsTakesOneFullStart` | PASS |
| D | `TestRestoreFromUnitIgnoresSafetyDumpsWhenDecidingToReplay` | PASS |
| E | `TestReconstituteReplayFailureStillBringsTheStackUp` | PASS |
| E | `TestReconstituteDBOnlyStartFailureStillBringsTheStackUp` | PASS |
| E | `TestRestoreFromUnitReplayFailureStillBringsTheStackUp` | PASS |
| F | `TestDBTypeForImage`, `TestDBServiceNames` (8 sub-cases), `TestDBServiceNames_TopLevelKeysAreNotServices`, `TestDBServiceNames_UnreadableAndUnparseableError`, `TestDiscoverAndComposeAgreeOnTheSameImages` | PASS |
| G | `TestStartStackServicesRefusesEmptyList`, `TestPersistUnitRedeployConfigPersistsWithoutStarting`, `TestPersistUnitRedeployConfigRejectsUnknownStack` | PASS |

The core assertion is deliberately not "no error": a recording provider captures whether the FULL
stack had been started at the moment the import fired. Asserting only `err == nil` passes on the
pre-fix shape — which is exactly how this shipped.

The compose-parser decoys use the catalog's REAL immich template shape (`immich_ml_cache:`,
`immich_postgres_data:` as top-level `volumes:` keys, `ghcr.io/immich-app/postgres:16-vectorchord…`
as the pin) — the exact input a line scan would misread.

### Companion red-proofs — three run, all reverted, tree clean

| # | Pre-fix shape restored | Failure observed |
|---|---|---|
| 1 | offsite: `StartStackServices` → full `StartStack` before the replay | `TestReconstituteReplaysWithOnlyTheDBServiceUp`: *"the database service was NOT started before the replay"*; `TestReconstituteReplaysDBAndOrdersOperations`: sequence `"stop,start,start"` |
| 2 | local: full `StartStack` inserted before the replay | `TestRestoreFromUnitReplaysWithOnlyTheDBServiceUp`: *"the FULL stack was already up when the replay fired — the H4 race, on the local path"* |
| 3 | both fail-closed gates deleted | both `RefusesWhenNoDBServiceIdentifiable` tests: *"expected a refusal…"* |

### Green gate

`go build ./... && go vet ./... && go test ./...` — **23/23 packages green**, exit 0
(`internal/backup` 174 s). New tests by package: backup +11, appbackup +5, stacks +3.

## 4. Deployment

Recorded on completion of Phase B — see the CHANGELOG entry for the shipped version.

## 5. NOT yet live-validated — remaining human/supervised work

- **STOP-1 (supervised, Viktor present):** prepare a full offsite restore scratch for immich through
  the real endpoints, then fire `/backup/offbox/reconstitute` and verify through the system's own
  surfaces — controller log showing stop → db-only up → replay rc-0 → full up, no `already exists`,
  no drift, immich healthy with content visible. Timestamps to be recorded here afterwards.
- **Phase C (golden 0.153.0):** probes P1–P3 first. **P3 is load-bearing** — the drill environment is
  at the vacation site and must be proven able to pull
  `gitea.dooplex.hu/admin/felhom-controller:0.153.0` BEFORE any bake. If unreachable: stop and
  report; change no routing/DNS/nft.
- **STOP-2 (Viktor, password-gated):** Day-0 manifest Golden → 0.153.0 (Agent 0.90.1 / MinAgent
  0.90.0 unchanged — the CHANGELOG's no-coupling declaration is the authority), then floor →
  v0.153.0 saved **LAST**. Watching the demo box wake during the manifest save banks the **R-23(a)**
  operator-UI save→apply evidence — log the timestamps if observed.
- **Viktor's C6 customer-restore UI run** (note the empty-the-trash method).

## 6. Observations (out of scope, recorded not acted on)

- **The `pre-restore-` prefix is load-bearing in three separate places** (the replay's exact-name
  match, `OffsiteScratchPair`'s dump sniff, and now `hasReplayableDump`) with no shared predicate
  deciding "is this file a replay source". A fourth consumer that forgets the exclusion would arm the
  DB-only window for an app that has nothing to replay. Worth a single helper at some point.
- **`reimportDBDumpsFrom`'s own `hasDump` scan does NOT exclude the safety prefix** (unchanged here —
  `restore_db.go` was explicitly out of scope). Harmless today because the per-DB lookup is an exact
  `<stack>-<dbtype>.sql` match, so a directory holding only safety dumps merely produces the
  "no matching running DB container" WARN instead of a clean zero.
- **`dbTypeForImage` maps both `mysql` and `mariadb` to `DBTypeMariaDB`.** Pre-existing and correct
  for the current catalog (the `mariadb` client speaks to both), but it is an assumption, not an
  invariant, and it is now written down in one place instead of two.
- **`RedeployFromEnv` has no end-to-end test** (it shells out to compose), so the split's equivalence
  is asserted on the persist half only. That is the half the split could break; the tail is
  byte-identical code that was moved, not rewritten.
- **R-29a (`estimate.go` gate finding)** remains open and was not touched.