felhom.eu/documentation/audits/night2-2026-10-20/P2/live

P2 live: R-893 option A, proved on scratch guest 9202 (2026-10-20, by day)

Night brief 2026-10-20 part 2, Part 2 (operator answer „D20 A"). Controller 0.336.0-rc.1 (commit 93fb251, image id sha256:0756d4d2…) on Tier 0 only: demo-hp, scratch guest 9202. Code read live: felhom-controller/controller/internal/backup/offbox_restore_undo.go and offbox_reconstitute.go at 93fb251. Architecture: 07-backup-architecture.md §6.3 (R-893); 09 §3 decision 192 and D20 = A. The method repeats night-2026-10-19/P3/ (the measurement of the old behaviour) step for step, so the two sets of numbers can be compared.

Method (endpoint level, the household's buttons; no browser)

• 9202 was on 0.331.0. For the run it went to 0.336.0-rc.1: pull, write /etc/felhom-controller-image, restart felhom-controller-bootstrap.service. Healthy, 0 [ERROR] (01-upgrade.txt). It was put back after the run.
• Scratch off-site target: atmoz/sftp:alpine container r893b-sftp on 9202 itself (port 2222). The repo is on 9202's own disk at /var/lib/felhom/r893b-sftp (nvme-scratch, not pve/data). The controller reached it at 172.17.0.1:2222, which does not depend on DHCP. Set up through POST /backup/offbox/config with a throwaway ed25519 key and the server's own host key pinned. 9202 has no agent, so the escrow was confirmed with the legacy manual POST /backup/offbox/confirm-escrow. Only rallly was switched on (POST /backup/offbox/toggle).
• Throwaway app: Rallly 4.15.3 + postgres:18-alpine. Its DB is in the named volume rallly_rallly_postgres_data. Deployed through the product (box_walk.deploy).
• Observables, all positive (tool-obs.sh.txt):
  • the r893_marker rows;
  • tables that exist only at certain times;
  • the public base-table count;
  • an md5 of pg_dump --schema-only, with pg 18's random \restrict lines removed and a second dump as its control (equal every time);
  • an md5 of the per-table row counts;
  • an md5 of pg_dump --data-only;
  • the definition's image lines;
  • the backups/restore-undo/rallly folder.
• Forced error: the same as last night. One line, SELECT r893_forced_replay_failure();, goes into the PREPARED scratch copy of the snapshot's dump, just before its completion marker. The completeness gate still passes, and the replay fails after the volume was replaced.
• Restore: POST /backup/offbox/reconstitute with app=rallly confirm=1. The outcome was read from /api/backup/restore-status, the source the restore page polls.
• Watch on the undo folder: while each restore ran, a watcher on 9202 listed /mnt/sys_drive/felhom-data/backups/restore-undo every 0.3 s. This shows the copy from a different channel than the controller's log.

Timeline (UTC)

Time	Act	File
07:25:33	baseline: 0.331.0, no off-site target, settings.json and the offbox/ folder copied	00-baseline.txt
07:25:47	9202 on 0.336.0-rc.1, healthy	01-upgrade.txt
07:27:37–07:27:58	SFTP target up; off-site configured, escrow confirmed, rallly switched on	02-sftp-target.txt
07:28:03–07:28:53	Rallly deployed	03-deploy.txt
07:29:13	marker A (34 tables)	04-marker-A.txt
07:29:29–07:30:38	off-site run → snapshot 1	05-snapshot-1.txt
07:30:50	marker B + r893_after; r893_only_old + marker C (36 tables)	05-snapshot-1.txt
07:30:56–07:32:04	off-site run → snapshot 2 (c3d94d03)	06-snapshot-2.txt
07:32:38	r893_only_old DROPPED, marker D, r893_new2: the LIVE record	07-live-before-restore.txt
07:32:57–07:33:13	full restore of snapshot 2 prepared (66.1 MB; content checked: C present, D absent); the failing line injected	08-prepare-and-inject.txt
07:33:28	reconstitute (forced failure)	10-forced-failure-http.txt, 11-controller-log-window*.txt
07:33:29	undo dump of the live DB; app stopped 07:33:29–07:33:40
07:33:40.9–41.6	live volume copied to restore-undo/rallly/20261020T073340Z/rallly_rallly_postgres_data.tar (69 198 848 B), tmp then rename	09-undo-folder-watch.txt
07:33:42	snapshot's volume in place (Restored 1 Docker volume(s))
07:33:44	replay FAILED (function r893_forced_replay_failure() does not exist, exit 3); rows rolled back 07:33:45
07:33:46–47	app stopped; the copy poured back (Restored 1 Docker volume(s), second time); „the live volumes and definition are put back; starting the app (R-893)"
07:33:53	Stack rallly started successfully
07:33:54	restore op ends ok=false; the copy folder gone (watcher: absent at 07:33:54.45)
07:34:25	after state read: identical to 07:32:38	12-after-failed-restore.txt
07:36:09–07:36:15	full restore prepared again, with no injection (injected-line count 0)	13-normal-prepare.txt
07:36:28–07:36:57	normal reconstitute: copy made 07:36:41–42, volume + dump replayed, app restarted, copy removed 07:36:57.9	14-…, 15-…, 16-controller-log-normal*.txt
07:38:31	after state read = snapshot 2	17-after-normal-restore.txt
(stop)	the session was paused by an API rate limit. When it resumed, 9202 was re-read first: nothing was in progress, no restore-undo copy, step 4 already recorded
08:43–08:45	teardown, three layers	18-teardown.txt

Step 3: the forced failure, before and after

	LIVE before the restore (07:32:38)	After the failed restore + put-back (07:34:25)	2026-10-19 on 0.335.0 (old behaviour)
marker rows	A, B, C, D	A, B, C, D	A, B, C, D
r893_only_old (in snapshot 2 only)	absent	absent	present, 1 row
r893_new2 (written after snapshot 2)	present, 1 row	present, 1 row	present, 1 row
public base tables	36	36	37
schema-only md5	05828c77…	05828c77… (control equal)	—
per-table row-count md5	d8bcbc4b…	d8bcbc4b…	—
data-only md5	e5c1a72f…	e5c1a72f…	—
definition	rallly 4.15.3, postgres 18 (digest pins)	same	same
app	running	running, healthy; controller state=running; front door 200	stopped, held (restore_mixed)
restore hold for rallly	none	none (holds: chaoscrash, chaosoom, chaosoomb, gokapi, as at baseline)	restore_mixed
backups/restore-undo/rallly	absent	absent (it existed 07:33:40–07:33:54)	(did not exist)

The DB volume's CreatedAt moved from 07:28:21 to 07:33:46. That is expected: the put-back recreates the volume from the copy. Every content observable is equal, the data-only dump included.

Step 4: the normal restore of snapshot 2

	Recorded just before snapshot 2 (07:30:50)	After the normal restore (07:38:31)
marker rows	A, B, C	A, B, C
r893_only_old	present, 1 row	present, 1 row
r893_new2	absent	absent
public base tables	36	36
schema-only md5	f872bd54…	f872bd54…
per-table row-count md5	c032377e…	c032377e…
data-only md5	db5903ab…	1e688d1b… (differs; see below)
app	running	running, healthy; front door 200; no hold
restore-undo folder	—	absent (existed 07:36:41–07:36:57)

• Table count. Snapshot 2 has 36 base tables, not 37. It held r893_only_old but not r893_new2. The „37" in the brief is last night's MIXED state (36 live tables + the stray r893_only_old), not the snapshot's.
• The data-only md5. It differs from the read taken 6 s before the snapshot's own dump started, while every row count and the schema are equal. Not explained here. It may be a row value that Rallly touched between that read and the dump (the off-site run restarted the app), or row order after the reload. It was not part of the acceptance list, and no row count moved. In step 3 the same observable was byte-equal, because there the volume itself was put back.

The sentences the household sees

• Failed restore, put back (err.backup.restore_failed_put_back, from /api/backup/restore-status): „A teljes visszaállítás sikertelen: a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut". Measured true: the data is identical and the app runs. It does not contain „vedd fel velünk a kapcsolatot".
• Normal restore: „A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult." True. 09:30 is local time (CEST) for snapshot 2's dump at 07:30 UTC.
• Flash keys seen on the way: flash.offbox.target_saved_escrow_agent_down (no agent on 9202, expected), escrow_confirmed, run_started, restore_started, full_restore_started.

Not measured (said plainly)

• The version-change branch (the snapshot's older definition written, then the live one written back). It needs an app at an older catalog step, the same limit as last night. Pinned by r893_putback_test.go.
• The _files sentence (restore_failed_put_back_files). Rallly has no drive files, so 0 files were placed.
• The refusals before the stop: the fit check refusing (restore_undo_no_space), an unmeasurable volume (restore_undo_unmeasured) and a failed copy (restore_undo_copy_failed). Only the positive path of the fit check ran: it passed and the app was stopped after it.
• A put-back that itself fails (hold + kept copy).
• The operator notification for a put-back. 9202 has no hub, so it was not observable.

Side observations

• Each manual off-site run restarted Rallly (its dump leg), as last night.
• The controller logged one expected [ERROR] pair per failure (the import and the rollback line) and one [ERROR] [web] off-box reconstitute … (async) line that carries the household sentence. The sentence is good news, so ERROR is a loud level for a put-back that worked. This is noted for the parent, not changed.
• The empty folder backups/restore-undo/ is left behind after the per-app folder is removed (the code removes the per-app folder only). This drill removed it by name.

Teardown (three layers)

• Machine (9202):
  • Rallly was removed through the product (POST /api/stacks/rallly/stop, then /remove with remove_hdd_data + remove_backups). Result: volumes_removed: rallly_rallly_postgres_data; 0 containers, 0 volumes; no backups/primary/rallly, no offsite-restore/rallly. The template folder /opt/docker/stacks/rallly (.felhom.yml, docker-compose.yml, hold-logs) was there before and stays.
  • r893b-sftp was removed. So were the images atmoz/sftp:alpine and lukevella/rallly:4.15.3 (both absent at baseline), the repo folder, the key and the empty backups/restore-undo/.
  • settings.json was put back from the 07:25 copy (sha256 cb6b18f8… = baseline) and offbox/ from its tar, with the controller stopped.
  • The controller is back on 0.331.0: healthy, 0 [ERROR]. Off-site target unset, no per-app off-site switch, restore holds as at baseline.
  • The 0.336.0-rc.1 image was removed by name. The image list is identical to the baseline. All /root/r893b-* and /tmp/r893b-* files are gone.
  • Not put back: restore-status.json, the controller's record of the last op. It now names this drill's last restore, as it named last night's at baseline.
• Host (demo-hp): only the per-call /tmp/r893b-g-* script copies, each removed after its call; 0 left. Nothing on pve/data. No whole-guest backup taken.
• Hub: nothing. 9202 has no hub.
• DooPlex: the throwaway key was shredded from the session scratchpad.