96 lines
7.1 KiB
Markdown
96 lines
7.1 KiB
Markdown
# P3 — R-893: a failed off-site replay, measured on scratch guest 9202 (2026-10-19 evening)
|
||
|
||
Night brief 2026-10-19, Part 3 — **measure and design only; no restore code changed.** Tier 0 only (demo-hp, scratch
|
||
guest 9202). Code read live: `felhom-controller/controller/internal/backup/offbox_reconstitute.go` at `15ea002`
|
||
(controller 0.335.0). Architecture: `07-backup-architecture.md` §6.3 (the R-893 known limit, the failure ladder
|
||
„replay → rollback → hold"); `09` §3 decision 192 (D8: first hold, then „put back exactly as it was").
|
||
|
||
## How it was reproduced (the product path, one forced error)
|
||
|
||
- 9202 put on controller **0.335.0** for the run (was 0.331.0; put back after). Hub is OFF on 9202.
|
||
- A **scratch off-site target**: an SFTP server container (`atmoz/sftp:alpine`, `r893-sftp`, port 2222) on 9202 itself,
|
||
repo on 9202's own disk (`/var/lib/felhom/r893-sftp`, nvme-scratch — not `pve/data`). Configured through the
|
||
household's own endpoint `POST /backup/offbox/config` (a throwaway ed25519 key, the server's host key pinned),
|
||
escrow confirmed with the legacy manual endpoint (9202 has no agent), only `rallly` switched on.
|
||
- Throwaway database app: **Rallly 4.15.3 + postgres:18-alpine**, the DB in the named volume
|
||
`rallly_rallly_postgres_data` (deployed through the product, `box_walk.deploy`).
|
||
- Markers in the real database (positive observables): table `r893_marker` with one row per phase, plus tables that
|
||
exist only at certain times.
|
||
- The **forced error**: one line `SELECT r893_forced_replay_failure();` put into the PREPARED scratch copy of the
|
||
snapshot's dump, just before its completion marker (so the completeness gate still passes). The real
|
||
`ReconstituteFromOffsite` then replaced the volume, started the DB alone and failed the replay — i.e. the replay
|
||
failed **after the volume was replaced**, as the brief asked.
|
||
|
||
## Timeline (UTC)
|
||
|
||
| Time | Act | File |
|
||
|---|---|---|
|
||
| 18:45 | baseline: 9202 on 0.331.0, no off-site target, settings + off-box folder saved | `00-baseline.txt` |
|
||
| 18:48 | Rallly deployed; marker **A** | `01-marker-A.txt` |
|
||
| 18:51:19 | off-site run → snapshot 1; then marker **B** + table `r893_after` | `02-snapshot-and-marker-B.txt` |
|
||
| 18:53 | table `r893_only_old` + marker **C**; off-site run → **snapshot 2** (18:54:29, `93dbb37d`) | `04-before-snapshot-2.txt` |
|
||
| 18:54:53 | after snapshot 2: `r893_only_old` DROPPED (like a table a newer version removes), marker **D**, table `r893_new2` — the live state the household has | `05-live-before-restore.txt` |
|
||
| 18:54:59 | full restore prepared into scratch (66.1 MB); the failing line injected | `03-forced-failure.txt` |
|
||
| 18:55:42 | `POST /backup/offbox/reconstitute` (confirm) | `06-reconstitute-log.txt`, `09-controller-log-window.txt` |
|
||
| 18:55:43 | undo dump of the LIVE db written (`pre-restore-…-rallly-postgres.sql`, 36 tables) | |
|
||
| 18:55:55 | the snapshot's volume put in place (`Restored 1 Docker volume(s)`) | |
|
||
| 18:55:57 | replay FAILED (`exit status 3`, single transaction — nothing of it applied) | |
|
||
| 18:55:58 | rollback: the undo dump loaded into the database **that now sits on snapshot 2's volume** — „rollback complete" | |
|
||
| 18:55:58–59 | hold SET (`reason: restore_mixed`), app stopped | |
|
||
| 18:56:42 | the held volume read through a throwaway COPY (the held state untouched) | `07-after-failed-replay.txt` |
|
||
| 18:57 | household start → **409 held** | `08-app-state-and-start.txt` |
|
||
| 18:58–19:01 | teardown | `10-teardown.txt` |
|
||
|
||
## What the failed replay left
|
||
|
||
| | Live before the restore | After the failed replay + rollback |
|
||
|---|---|---|
|
||
| marker rows | A, B, C, D | A, B, C, D (the newest rows — the undo dump won) |
|
||
| `r893_new2` (written after snapshot 2) | present, 1 row | present, 1 row |
|
||
| `r893_only_old` (in snapshot 2 only, dropped after) | **absent** | **present, 1 row** — left from the snapshot's volume |
|
||
| public tables | 36 | **37** |
|
||
| definition on disk | rallly 4.15.3, postgres 18 | rallly 4.15.3, postgres 18 (unchanged — same version; the R-1106 password overlay was empty, so the definition did not move) |
|
||
| app | running | **stopped, held** (`restore_mixed`); a start answers 409 „held" |
|
||
| other volumes / files | — | none for Rallly (its only volume is the DB's) |
|
||
|
||
So the rollback gives the **newest rows on top of the older volume**: everything the undo dump knows is exact, and
|
||
whatever only the older volume had stays behind (the dump's `--clean --if-exists` drops only the objects IT names).
|
||
That is the R-638 / R-893 mix, now measured, not read. The 0.304.0 hold works as built: the app does not start on it.
|
||
|
||
**The sentences the household sees:**
|
||
- the restore page (`err.backup.db_restore_failed_held_mixed`): „Az adatbázis a visszaállítás előtti állapotban van,
|
||
de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze." — **measured false on
|
||
both halves for this app**: the database is NOT as it was (a table from the backup stayed), and Rallly had no files
|
||
or other data from the backup (only its database volume moved).
|
||
- the start refusal: „…félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás
|
||
biztonsági okból leállítva marad." — true.
|
||
|
||
## Not measured tonight (said plainly)
|
||
|
||
- The **version-change** branch (the snapshot's older definition written, then the live one written back by
|
||
`holdAppAfterMixedRestore`). It needs an app installed at an older catalog step and updated before the restore; the
|
||
drill catalog that could do it is a git repo this helper was told not to commit to. The branch is pinned by
|
||
`r893_mixed_restore_test.go`; it ends in the same hold.
|
||
- What support does next: no product action releases a `restore_mixed` hold with the data made right — that is the
|
||
gap option A closes.
|
||
|
||
## Side observations
|
||
|
||
- The manual off-site run's pre-push dump leg dumped and restarted **every** app on the box (paperless, immich), not
|
||
only the one switched on (log: `App-data backup completed: 3 databases … 4 volume dump(s)`). By design (the
|
||
coherent pair), outside R-893's scope.
|
||
- 9202's address moved (DHCP: .114 → .121); `box_walk.py`'s 9202 row still names .114 — run with `BASE=`.
|
||
|
||
## Teardown (three layers)
|
||
|
||
- **Machine (9202):** Rallly removed through the product (`volumes_removed: rallly_rallly_postgres_data`, its backups
|
||
and the off-site scratch gone); `r893-sftp` container, `atmoz/sftp:alpine` and `alpine:3` images (pulled tonight),
|
||
the repo folder, the key, all `/root/r893-*` and `/tmp/r893-*` removed; `settings.json` and the `offbox/` folder put
|
||
back from the copies taken at 18:45 (off-site target unset, no per-app off-site switch, restore holds as before:
|
||
chaoscrash, chaosoom, chaosoomb, gokapi); controller back on **0.331.0** (healthy, 0 `[ERROR]`), the 0.335.0 image
|
||
removed from 9202. The template folder `/opt/docker/stacks/rallly` (catalog files + an old `hold-logs`) was there
|
||
before and stays.
|
||
- **Host (demo-hp):** only `/tmp/r893-*` copies, removed. Nothing on `pve/data`.
|
||
- **Hub:** nothing. 9202 has no hub; the hold's events were dropped on the box („notifier disabled (no hub
|
||
configured)").
|