# P3 — R-893: a failed off-site replay, measured on scratch guest 9202 (2026-10-19 evening)

Night brief 2026-10-19, Part 3 — **measure and design only; no restore code changed.** Tier 0 only (demo-hp, scratch
guest 9202). Code read live: `felhom-controller/controller/internal/backup/offbox_reconstitute.go` at `15ea002`
(controller 0.335.0). Architecture: `07-backup-architecture.md` §6.3 (the R-893 known limit, the failure ladder
„replay → rollback → hold"); `09` §3 decision 192 (D8: first hold, then „put back exactly as it was").

## How it was reproduced (the product path, one forced error)

- 9202 put on controller **0.335.0** for the run (was 0.331.0; put back after). Hub is OFF on 9202.
- A **scratch off-site target**: an SFTP server container (`atmoz/sftp:alpine`, `r893-sftp`, port 2222) on 9202 itself,
  repo on 9202's own disk (`/var/lib/felhom/r893-sftp`, nvme-scratch — not `pve/data`). Configured through the
  household's own endpoint `POST /backup/offbox/config` (a throwaway ed25519 key, the server's host key pinned),
  escrow confirmed with the legacy manual endpoint (9202 has no agent), only `rallly` switched on.
- Throwaway database app: **Rallly 4.15.3 + postgres:18-alpine**, the DB in the named volume
  `rallly_rallly_postgres_data` (deployed through the product, `box_walk.deploy`).
- Markers in the real database (positive observables): table `r893_marker` with one row per phase, plus tables that
  exist only at certain times.
- The **forced error**: one line `SELECT r893_forced_replay_failure();` put into the PREPARED scratch copy of the
  snapshot's dump, just before its completion marker (so the completeness gate still passes). The real
  `ReconstituteFromOffsite` then replaced the volume, started the DB alone and failed the replay — i.e. the replay
  failed **after the volume was replaced**, as the brief asked.

## Timeline (UTC)

| Time | Act | File |
|---|---|---|
| 18:45 | baseline: 9202 on 0.331.0, no off-site target, settings + off-box folder saved | `00-baseline.txt` |
| 18:48 | Rallly deployed; marker **A** | `01-marker-A.txt` |
| 18:51:19 | off-site run → snapshot 1; then marker **B** + table `r893_after` | `02-snapshot-and-marker-B.txt` |
| 18:53 | table `r893_only_old` + marker **C**; off-site run → **snapshot 2** (18:54:29, `93dbb37d`) | `04-before-snapshot-2.txt` |
| 18:54:53 | after snapshot 2: `r893_only_old` DROPPED (like a table a newer version removes), marker **D**, table `r893_new2` — the live state the household has | `05-live-before-restore.txt` |
| 18:54:59 | full restore prepared into scratch (66.1 MB); the failing line injected | `03-forced-failure.txt` |
| 18:55:42 | `POST /backup/offbox/reconstitute` (confirm) | `06-reconstitute-log.txt`, `09-controller-log-window.txt` |
| 18:55:43 | undo dump of the LIVE db written (`pre-restore-…-rallly-postgres.sql`, 36 tables) | |
| 18:55:55 | the snapshot's volume put in place (`Restored 1 Docker volume(s)`) | |
| 18:55:57 | replay FAILED (`exit status 3`, single transaction — nothing of it applied) | |
| 18:55:58 | rollback: the undo dump loaded into the database **that now sits on snapshot 2's volume** — „rollback complete" | |
| 18:55:58–59 | hold SET (`reason: restore_mixed`), app stopped | |
| 18:56:42 | the held volume read through a throwaway COPY (the held state untouched) | `07-after-failed-replay.txt` |
| 18:57 | household start → **409 held** | `08-app-state-and-start.txt` |
| 18:58–19:01 | teardown | `10-teardown.txt` |

## What the failed replay left

| | Live before the restore | After the failed replay + rollback |
|---|---|---|
| marker rows | A, B, C, D | A, B, C, D (the newest rows — the undo dump won) |
| `r893_new2` (written after snapshot 2) | present, 1 row | present, 1 row |
| `r893_only_old` (in snapshot 2 only, dropped after) | **absent** | **present, 1 row** — left from the snapshot's volume |
| public tables | 36 | **37** |
| definition on disk | rallly 4.15.3, postgres 18 | rallly 4.15.3, postgres 18 (unchanged — same version; the R-1106 password overlay was empty, so the definition did not move) |
| app | running | **stopped, held** (`restore_mixed`); a start answers 409 „held" |
| other volumes / files | — | none for Rallly (its only volume is the DB's) |

So the rollback gives the **newest rows on top of the older volume**: everything the undo dump knows is exact, and
whatever only the older volume had stays behind (the dump's `--clean --if-exists` drops only the objects IT names).
That is the R-638 / R-893 mix, now measured, not read. The 0.304.0 hold works as built: the app does not start on it.

**The sentences the household sees:**
- the restore page (`err.backup.db_restore_failed_held_mixed`): „Az adatbázis a visszaállítás előtti állapotban van,
  de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze." — **measured false on
  both halves for this app**: the database is NOT as it was (a table from the backup stayed), and Rallly had no files
  or other data from the backup (only its database volume moved).
- the start refusal: „…félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás
  biztonsági okból leállítva marad." — true.

## Not measured tonight (said plainly)

- The **version-change** branch (the snapshot's older definition written, then the live one written back by
  `holdAppAfterMixedRestore`). It needs an app installed at an older catalog step and updated before the restore; the
  drill catalog that could do it is a git repo this helper was told not to commit to. The branch is pinned by
  `r893_mixed_restore_test.go`; it ends in the same hold.
- What support does next: no product action releases a `restore_mixed` hold with the data made right — that is the
  gap option A closes.

## Side observations

- The manual off-site run's pre-push dump leg dumped and restarted **every** app on the box (paperless, immich), not
  only the one switched on (log: `App-data backup completed: 3 databases … 4 volume dump(s)`). By design (the
  coherent pair), outside R-893's scope.
- 9202's address moved (DHCP: .114 → .121); `box_walk.py`'s 9202 row still names .114 — run with `BASE=`.

## Teardown (three layers)

- **Machine (9202):** Rallly removed through the product (`volumes_removed: rallly_rallly_postgres_data`, its backups
  and the off-site scratch gone); `r893-sftp` container, `atmoz/sftp:alpine` and `alpine:3` images (pulled tonight),
  the repo folder, the key, all `/root/r893-*` and `/tmp/r893-*` removed; `settings.json` and the `offbox/` folder put
  back from the copies taken at 18:45 (off-site target unset, no per-app off-site switch, restore holds as before:
  chaoscrash, chaosoom, chaosoomb, gokapi); controller back on **0.331.0** (healthy, 0 `[ERROR]`), the 0.335.0 image
  removed from 9202. The template folder `/opt/docker/stacks/rallly` (catalog files + an old `hold-logs`) was there
  before and stays.
- **Host (demo-hp):** only `/tmp/r893-*` copies, removed. Nothing on `pve/data`.
- **Hub:** nothing. 9202 has no hub; the hold's events were dropped on the box („notifier disabled (no hub
  configured)").