felhom.eu/documentation/audits/night-2026-10-19/P3

P3 — R-893: a failed off-site replay, measured on scratch guest 9202 (2026-10-19 evening)

Night brief 2026-10-19, Part 3 — measure and design only; no restore code changed. Tier 0 only (demo-hp, scratch guest 9202). Code read live: felhom-controller/controller/internal/backup/offbox_reconstitute.go at 15ea002 (controller 0.335.0). Architecture: 07-backup-architecture.md §6.3 (the R-893 known limit, the failure ladder „replay → rollback → hold"); 09 §3 decision 192 (D8: first hold, then „put back exactly as it was").

How it was reproduced (the product path, one forced error)

• 9202 put on controller 0.335.0 for the run (was 0.331.0; put back after). Hub is OFF on 9202.
• A scratch off-site target: an SFTP server container (atmoz/sftp:alpine, r893-sftp, port 2222) on 9202 itself, repo on 9202's own disk (/var/lib/felhom/r893-sftp, nvme-scratch — not pve/data). Configured through the household's own endpoint POST /backup/offbox/config (a throwaway ed25519 key, the server's host key pinned), escrow confirmed with the legacy manual endpoint (9202 has no agent), only rallly switched on.
• Throwaway database app: Rallly 4.15.3 + postgres:18-alpine, the DB in the named volume rallly_rallly_postgres_data (deployed through the product, box_walk.deploy).
• Markers in the real database (positive observables): table r893_marker with one row per phase, plus tables that exist only at certain times.
• The forced error: one line SELECT r893_forced_replay_failure(); put into the PREPARED scratch copy of the snapshot's dump, just before its completion marker (so the completeness gate still passes). The real ReconstituteFromOffsite then replaced the volume, started the DB alone and failed the replay — i.e. the replay failed after the volume was replaced, as the brief asked.

Timeline (UTC)

Time	Act	File
18:45	baseline: 9202 on 0.331.0, no off-site target, settings + off-box folder saved	00-baseline.txt
18:48	Rallly deployed; marker A	01-marker-A.txt
18:51:19	off-site run → snapshot 1; then marker B + table r893_after	02-snapshot-and-marker-B.txt
18:53	table r893_only_old + marker C; off-site run → snapshot 2 (18:54:29, 93dbb37d)	04-before-snapshot-2.txt
18:54:53	after snapshot 2: r893_only_old DROPPED (like a table a newer version removes), marker D, table r893_new2 — the live state the household has	05-live-before-restore.txt
18:54:59	full restore prepared into scratch (66.1 MB); the failing line injected	03-forced-failure.txt
18:55:42	POST /backup/offbox/reconstitute (confirm)	06-reconstitute-log.txt, 09-controller-log-window.txt
18:55:43	undo dump of the LIVE db written (pre-restore-…-rallly-postgres.sql, 36 tables)
18:55:55	the snapshot's volume put in place (Restored 1 Docker volume(s))
18:55:57	replay FAILED (exit status 3, single transaction — nothing of it applied)
18:55:58	rollback: the undo dump loaded into the database that now sits on snapshot 2's volume — „rollback complete"
18:55:58–59	hold SET (reason: restore_mixed), app stopped
18:56:42	the held volume read through a throwaway COPY (the held state untouched)	07-after-failed-replay.txt
18:57	household start → 409 held	08-app-state-and-start.txt
18:58–19:01	teardown	10-teardown.txt

What the failed replay left

	Live before the restore	After the failed replay + rollback
marker rows	A, B, C, D	A, B, C, D (the newest rows — the undo dump won)
r893_new2 (written after snapshot 2)	present, 1 row	present, 1 row
r893_only_old (in snapshot 2 only, dropped after)	absent	present, 1 row — left from the snapshot's volume
public tables	36	37
definition on disk	rallly 4.15.3, postgres 18	rallly 4.15.3, postgres 18 (unchanged — same version; the R-1106 password overlay was empty, so the definition did not move)
app	running	stopped, held (restore_mixed); a start answers 409 „held"
other volumes / files	—	none for Rallly (its only volume is the DB's)

So the rollback gives the newest rows on top of the older volume: everything the undo dump knows is exact, and whatever only the older volume had stays behind (the dump's --clean --if-exists drops only the objects IT names). That is the R-638 / R-893 mix, now measured, not read. The 0.304.0 hold works as built: the app does not start on it.

The sentences the household sees:

• the restore page (err.backup.db_restore_failed_held_mixed): „Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze." — measured false on both halves for this app: the database is NOT as it was (a table from the backup stayed), and Rallly had no files or other data from the backup (only its database volume moved).
• the start refusal: „…félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad." — true.

Not measured tonight (said plainly)

• The version-change branch (the snapshot's older definition written, then the live one written back by holdAppAfterMixedRestore). It needs an app installed at an older catalog step and updated before the restore; the drill catalog that could do it is a git repo this helper was told not to commit to. The branch is pinned by r893_mixed_restore_test.go; it ends in the same hold.
• What support does next: no product action releases a restore_mixed hold with the data made right — that is the gap option A closes.

Side observations

• The manual off-site run's pre-push dump leg dumped and restarted every app on the box (paperless, immich), not only the one switched on (log: App-data backup completed: 3 databases … 4 volume dump(s)). By design (the coherent pair), outside R-893's scope.
• 9202's address moved (DHCP: .114 → .121); box_walk.py's 9202 row still names .114 — run with BASE=.

Teardown (three layers)

• Machine (9202): Rallly removed through the product (volumes_removed: rallly_rallly_postgres_data, its backups and the off-site scratch gone); r893-sftp container, atmoz/sftp:alpine and alpine:3 images (pulled tonight), the repo folder, the key, all /root/r893-* and /tmp/r893-* removed; settings.json and the offbox/ folder put back from the copies taken at 18:45 (off-site target unset, no per-app off-site switch, restore holds as before: chaoscrash, chaosoom, chaosoomb, gokapi); controller back on 0.331.0 (healthy, 0 [ERROR]), the 0.335.0 image removed from 9202. The template folder /opt/docker/stacks/rallly (catalog files + an old hold-logs) was there before and stays.
• Host (demo-hp): only /tmp/r893-* copies, removed. Nothing on pve/data.
• Hub: nothing. 9202 has no hub; the hold's events were dropped on the box („notifier disabled (no hub configured)").