4.6 KiB
R-893 — after a failed off-site replay: put the app back exactly as it was? (design note, 2026-10-19)
Status: design only, for the operator. Measured first (README.md in this folder, 9202, controller 0.335.0).
Architecture: 07 §6.3 (the failure ladder „replay → rollback → hold"), 09 §3 decision 192 (D8: first hold —
built in 0.304.0 — then „put back exactly as it was"). No restore code was changed.
What the measurement showed (plain words)
An off-site restore of one app replaces the app's database volume with the backup's, then loads the backup's database copy. When that load fails, the box puts the household's own database copy (taken a minute before) back on top of the backup's volume. Measured on Rallly: all the newest rows came back, but a table that only the backup had stayed behind (37 tables instead of 36). The app is then held stopped, and a start is refused — that part works. But nothing in the product can make the data right again; the household waits for support. And the restore page tells them the database „is back as it was before the restore" — measured not true.
The question
When an off-site restore of one app fails half-way, should the box put the app back exactly as it was and start it, or keep holding it stopped for support?
The options
| A — keep a copy of the app's volumes, and put it back | B — keep holding; only make the database part exact | |
|---|---|---|
| What happens | After the undo dump and the stop, the box saves the app's live volumes (the same volume-dump helper the nightly copy uses) and keeps the live definition (already done since 0.304.0). On a failed replay it puts those volumes and the definition back — the state from one minute before — and starts the app. The undo dump stays as a second way back. | On a failed replay the box empties the database volume before it loads the undo dump (a rebuilding load, on the rollback path only). Then the database is exactly as it was. The app stays held while files or other volumes moved; an app whose only change was its database volume can start. |
| Disk | One extra copy of the app's volumes for the length of the restore (Rallly: 69 MB, made in about 1 s). A free-space check refuses BEFORE the stop when the copy does not fit — then nothing is touched. | None. |
| Code | ~1 session: the save step, the put-back on every failed branch, the fit check, tests; then one 9202 drill (tonight's setup can be reused). | ~½–1 session: the emptying step and its tests; the R-638 option-B loader, but only on this one path. |
| Risk | Low: both helpers exist and already run every night. A put-back that fails leaves the hold, as today. | Higher: the volume is emptied before the undo is loaded, so a bad undo dump leaves an empty database (the backup's volume is gone too). Files and other volumes are still the backup's. |
| What the household sees | „The restore did not finish. Nothing changed: the app runs as before." No support call. | Usually still „the app is stopped; contact us" — fewer cases than today, but the same support call. |
Pick: A
It is the direction the operator already ruled (D8, second half), it makes the restore page's promise true, and the measured cost is small: the extra disk is one copy of the app's own volumes, for minutes, and a fit check refuses cleanly when there is no room. B removes only one of the mixed parts and adds a new way to lose the database.
One small fix either way, now: the sentence err.backup.db_restore_failed_held_mixed says the database is back as
it was and that files and other data came from the backup. Measured: neither is reliably true (Rallly had no files or
other volumes from the backup, and its database was not as before). It should say only what is certain: the restore
stopped half-way, the app is held stopped, and nothing is lost yet.
If the operator does nothing: a failed off-site restore keeps ending in a held app that only support can bring back, and the page keeps telling the household their database is back as it was.
First slice and its red test (for A)
- Red test beside
r893_mixed_restore_test.go: a failed replay after a volume replay must call the volume put-back with the live copy, write the live definition back, and START the app (no hold); with too little free space for the copy, the restore must refuse withStopStackcalled 0 times. - Then the 9202 drill (this folder's
README.md, same steps): after the forced failure, the table count and the markers must equal the live state before the restore (36 tables, A–D, nor893_only_old), and the app must run.