v0.153.0 — R-47: the DB replay no longer races the app, on BOTH restore paths
Closes R-47. No new agent coupling — MinAgent stays 0.90.0. The replay needs a running DB container, so both restore paths started the WHOLE stack first, giving the application a window to rebuild the very schema objects the dump was about to create. Measured live on 2026-07-19 (H4, DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds before the dump's CREATE INDEX, the replay aborted "already exists" under ON_ERROR_STOP=1, and immich reported schema drift. The data survived only because pg_dump emits COPY before CREATE INDEX. Both paths now open a DB-ONLY window: only the stack's database service(s) come up, the dump is replayed with the app still down, and the full start runs only after the replay exits 0. Fail-closed: a dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window still does a best-effort full start, so a failed restore never leaves a box with a database and no application. New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty list — argument-less `up -d` is a full start); RedeployFromEnv split into PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden `up -d` inside the old name is what carried the defect on the local path. 19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed effects, replay-failure bring-up, parser decoys, empty-list refusal); three companion red-proofs run and reverted. 23/23 packages green. Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
This commit is contained in:
+21
-6
@@ -307,14 +307,24 @@ Each app can define rich metadata in `.felhom.yml`:
|
||||
- **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing.
|
||||
`ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok +
|
||||
adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files
|
||||
overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → start → the
|
||||
snapshot's dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit) → health wait**.
|
||||
overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → the DATABASE
|
||||
SERVICE ONLY started (`StartStackServices`, v0.153.0) → the snapshot's dump replayed
|
||||
(`reimportDBDumpsFrom`, reading the SCRATCH unit) → the full stack started → health wait**.
|
||||
Two invariants: nothing is ever deleted (post-snapshot files survive as extras), and the
|
||||
`pre-restore-` safety dump is verified on disk BEFORE anything is stopped or overwritten — if it
|
||||
cannot be taken the operation refuses with zero changes. Safety dumps appear in `ListDumpFiles`
|
||||
(they are the undo). The live recovery unit is still never overwritten, which is why the replay
|
||||
source is the scratch. Honesty surfaces (`OffsiteScratchPair`): dump age, an unstamped-pair
|
||||
warning, and the R-44 empty-dump sniff — all warn-level, none of them gates.
|
||||
- **The DB-only replay window (v0.153.0, R-47).** Until v0.153.0 the whole stack was started before
|
||||
the replay, so the application's own schema management raced the dump: measured live on
|
||||
2026-07-19 (H4), immich-server rebuilt `clip_index` two seconds before the dump's `CREATE INDEX`
|
||||
and the replay aborted `already exists` under `ON_ERROR_STOP=1`. The DB service is now brought up
|
||||
alone (`appbackup.DBServiceNames` reads the LIVE compose's `services:` map to name it), the dump
|
||||
is replayed with the app still down, and only then does the full start run. **Fail-closed:** a
|
||||
dump with no identifiable DB service refuses before the first mutation. Every exit from the window
|
||||
— replay failure, DB-only start failure — still does a best-effort full start, so a failed restore
|
||||
never leaves the box with a database and no application.
|
||||
|
||||
The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form.
|
||||
|
||||
@@ -595,7 +605,7 @@ backups/primary/<app>/
|
||||
**Resettable secrets (O4, v0.99.0):** an unrecoverable resettable secret (DB password etc.) gets a
|
||||
**generated replacement** from its catalog `generate` spec (`stacks.GenerateSecretForField` via the
|
||||
`backup.SetSecretGenerator` seam) instead of redeploying blank (which failed compose-up); the new
|
||||
value persists encrypted through the normal `RecreateStackFromUnit` → `SaveAppConfig` path. Fields
|
||||
value persists encrypted through the normal `RecreateStackDefinitionFromUnit` → `SaveAppConfig` path. Fields
|
||||
with no `generate` spec still proceed with a loud "may fail to start" WARN. Residual case: a restored
|
||||
volume tar carrying the OLD internal credential hash may still need a manual in-DB reset.
|
||||
- Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath`
|
||||
@@ -604,9 +614,14 @@ backups/primary/<app>/
|
||||
env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so
|
||||
the capture never touches a secret). `data_key` fields are marked in `.felhom.yml`
|
||||
(`DeployField.DataKey`).
|
||||
- **Restore replays the DB dump (F17, v0.61.0).** `RestoreFromRecoveryUnit` (and the `RestoreApp`
|
||||
fallback) stops the app → restores named-volume tars → recreates the compose definition + redeploys
|
||||
with the recovered env → **replays each `db-dumps/*.sql` into the now-running DB** via
|
||||
- **Restore replays the DB dump (F17, v0.61.0; re-sequenced v0.153.0, R-47).** `RestoreFromRecoveryUnit`
|
||||
(and the `RestoreApp` fallback) stops the app → restores named-volume tars → recreates the compose
|
||||
definition and persists the recovered env (`RecreateStackDefinitionFromUnit` — **starts nothing**)
|
||||
→ starts the DATABASE SERVICE ONLY (`StartStackServices`, named from the unit's compose) →
|
||||
**replays each `db-dumps/*.sql` into that DB** → starts the full stack → health wait. Before
|
||||
v0.153.0 `RecreateStackFromUnit` ended in a full `compose up -d`, so this path carried the same
|
||||
H4 race as the offsite one (see the reconstitution section above), with the same fail-closed rule
|
||||
and the same guarantee that every exit still brings the app back up. The replay itself uses
|
||||
`backup.reimportDBDumps` → `appbackup.ImportDump` (psql / mariadb client, using the live container's own
|
||||
discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins**
|
||||
over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`,
|
||||
|
||||
Reference in New Issue
Block a user