v0.153.0 — R-47: the DB replay no longer races the app, on BOTH restore paths

Closes R-47. No new agent coupling — MinAgent stays 0.90.0.

The replay needs a running DB container, so both restore paths started the
WHOLE stack first, giving the application a window to rebuild the very schema
objects the dump was about to create. Measured live on 2026-07-19 (H4,
DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds
before the dump's CREATE INDEX, the replay aborted "already exists" under
ON_ERROR_STOP=1, and immich reported schema drift. The data survived only
because pg_dump emits COPY before CREATE INDEX.

Both paths now open a DB-ONLY window: only the stack's database service(s)
come up, the dump is replayed with the app still down, and the full start
runs only after the replay exits 0. Fail-closed: a dump with no identifiable
DB service refuses BEFORE the first mutation. Every exit from the window
still does a best-effort full start, so a failed restore never leaves a box
with a database and no application.

New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line
scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage
with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty
list — argument-less `up -d` is a full start); RedeployFromEnv split into
PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's
RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden
`up -d` inside the old name is what carried the defect on the local path.

19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed
effects, replay-failure bring-up, parser decoys, empty-list refusal); three
companion red-proofs run and reverted. 23/23 packages green.

Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
This commit is contained in:
2026-07-20 17:01:52 +02:00
parent fd40b29119
commit 78ff991f1c
25 changed files with 1323 additions and 201 deletions
+21 -6
View File
@@ -307,14 +307,24 @@ Each app can define rich metadata in `.felhom.yml`:
- **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing.
`ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok +
adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files
overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → start → the
snapshot's dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit) → health wait**.
overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → the DATABASE
SERVICE ONLY started (`StartStackServices`, v0.153.0) → the snapshot's dump replayed
(`reimportDBDumpsFrom`, reading the SCRATCH unit) → the full stack started → health wait**.
Two invariants: nothing is ever deleted (post-snapshot files survive as extras), and the
`pre-restore-` safety dump is verified on disk BEFORE anything is stopped or overwritten — if it
cannot be taken the operation refuses with zero changes. Safety dumps appear in `ListDumpFiles`
(they are the undo). The live recovery unit is still never overwritten, which is why the replay
source is the scratch. Honesty surfaces (`OffsiteScratchPair`): dump age, an unstamped-pair
warning, and the R-44 empty-dump sniff — all warn-level, none of them gates.
- **The DB-only replay window (v0.153.0, R-47).** Until v0.153.0 the whole stack was started before
the replay, so the application's own schema management raced the dump: measured live on
2026-07-19 (H4), immich-server rebuilt `clip_index` two seconds before the dump's `CREATE INDEX`
and the replay aborted `already exists` under `ON_ERROR_STOP=1`. The DB service is now brought up
alone (`appbackup.DBServiceNames` reads the LIVE compose's `services:` map to name it), the dump
is replayed with the app still down, and only then does the full start run. **Fail-closed:** a
dump with no identifiable DB service refuses before the first mutation. Every exit from the window
— replay failure, DB-only start failure — still does a best-effort full start, so a failed restore
never leaves the box with a database and no application.
The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form.
@@ -595,7 +605,7 @@ backups/primary/<app>/
**Resettable secrets (O4, v0.99.0):** an unrecoverable resettable secret (DB password etc.) gets a
**generated replacement** from its catalog `generate` spec (`stacks.GenerateSecretForField` via the
`backup.SetSecretGenerator` seam) instead of redeploying blank (which failed compose-up); the new
value persists encrypted through the normal `RecreateStackFromUnit``SaveAppConfig` path. Fields
value persists encrypted through the normal `RecreateStackDefinitionFromUnit``SaveAppConfig` path. Fields
with no `generate` spec still proceed with a loud "may fail to start" WARN. Residual case: a restored
volume tar carrying the OLD internal credential hash may still need a manual in-DB reset.
- Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath`
@@ -604,9 +614,14 @@ backups/primary/<app>/
env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so
the capture never touches a secret). `data_key` fields are marked in `.felhom.yml`
(`DeployField.DataKey`).
- **Restore replays the DB dump (F17, v0.61.0).** `RestoreFromRecoveryUnit` (and the `RestoreApp`
fallback) stops the app → restores named-volume tars → recreates the compose definition + redeploys
with the recovered env → **replays each `db-dumps/*.sql` into the now-running DB** via
- **Restore replays the DB dump (F17, v0.61.0; re-sequenced v0.153.0, R-47).** `RestoreFromRecoveryUnit`
(and the `RestoreApp` fallback) stops the app → restores named-volume tars → recreates the compose
definition and persists the recovered env (`RecreateStackDefinitionFromUnit`**starts nothing**)
→ starts the DATABASE SERVICE ONLY (`StartStackServices`, named from the unit's compose) →
**replays each `db-dumps/*.sql` into that DB** → starts the full stack → health wait. Before
v0.153.0 `RecreateStackFromUnit` ended in a full `compose up -d`, so this path carried the same
H4 race as the offsite one (see the reconstitution section above), with the same fail-closed rule
and the same guarantee that every exit still brings the app back up. The replay itself uses
`backup.reimportDBDumps``appbackup.ImportDump` (psql / mariadb client, using the live container's own
discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins**
over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`,