162 lines
12 KiB
Markdown
162 lines
12 KiB
Markdown
# P2 live: R-893 option A, proved on scratch guest 9202 (2026-10-20, by day)
|
||
|
||
Night brief 2026-10-20 part 2, Part 2 (operator answer „D20 A"). Controller **0.336.0-rc.1** (commit `93fb251`,
|
||
image id `sha256:0756d4d2…`) on Tier 0 only: demo-hp, scratch guest 9202. Code read live:
|
||
`felhom-controller/controller/internal/backup/offbox_restore_undo.go` and `offbox_reconstitute.go` at `93fb251`.
|
||
Architecture: `07-backup-architecture.md` §6.3 (R-893); `09` §3 decision 192 and D20 = A.
|
||
The method repeats `night-2026-10-19/P3/` (the measurement of the old behaviour) step for step, so the two sets of
|
||
numbers can be compared.
|
||
|
||
## Method (endpoint level, the household's buttons; no browser)
|
||
|
||
- 9202 was on **0.331.0**. For the run it went to 0.336.0-rc.1: pull, write `/etc/felhom-controller-image`,
|
||
restart `felhom-controller-bootstrap.service`. Healthy, 0 `[ERROR]` (`01-upgrade.txt`). It was put back after the run.
|
||
- **Scratch off-site target:** `atmoz/sftp:alpine` container `r893b-sftp` on 9202 itself (port 2222). The repo is on
|
||
9202's own disk at `/var/lib/felhom/r893b-sftp` (nvme-scratch, not `pve/data`). The controller reached it at
|
||
`172.17.0.1:2222`, which does not depend on DHCP. Set up through `POST /backup/offbox/config` with a throwaway
|
||
ed25519 key and the server's own host key pinned. 9202 has no agent, so the escrow was confirmed with the legacy
|
||
manual `POST /backup/offbox/confirm-escrow`. Only `rallly` was switched on (`POST /backup/offbox/toggle`).
|
||
- **Throwaway app:** Rallly 4.15.3 + postgres:18-alpine. Its DB is in the named volume `rallly_rallly_postgres_data`.
|
||
Deployed through the product (`box_walk.deploy`).
|
||
- **Observables, all positive** (`tool-obs.sh.txt`):
|
||
- the `r893_marker` rows;
|
||
- tables that exist only at certain times;
|
||
- the public base-table count;
|
||
- an md5 of `pg_dump --schema-only`, with pg 18's random `\restrict` lines removed and **a second dump as its
|
||
control** (equal every time);
|
||
- an md5 of the per-table row counts;
|
||
- an md5 of `pg_dump --data-only`;
|
||
- the definition's image lines;
|
||
- the `backups/restore-undo/rallly` folder.
|
||
- **Forced error:** the same as last night. One line, `SELECT r893_forced_replay_failure();`, goes into the PREPARED
|
||
scratch copy of the snapshot's dump, just before its completion marker. The completeness gate still passes, and the
|
||
replay fails **after the volume was replaced**.
|
||
- **Restore:** `POST /backup/offbox/reconstitute` with `app=rallly confirm=1`. The outcome was read from
|
||
`/api/backup/restore-status`, the source the restore page polls.
|
||
- **Watch on the undo folder:** while each restore ran, a watcher on 9202 listed
|
||
`/mnt/sys_drive/felhom-data/backups/restore-undo` every 0.3 s. This shows the copy from a different channel than
|
||
the controller's log.
|
||
|
||
## Timeline (UTC)
|
||
|
||
| Time | Act | File |
|
||
|---|---|---|
|
||
| 07:25:33 | baseline: 0.331.0, no off-site target, `settings.json` and the `offbox/` folder copied | `00-baseline.txt` |
|
||
| 07:25:47 | 9202 on 0.336.0-rc.1, healthy | `01-upgrade.txt` |
|
||
| 07:27:37–07:27:58 | SFTP target up; off-site configured, escrow confirmed, `rallly` switched on | `02-sftp-target.txt` |
|
||
| 07:28:03–07:28:53 | Rallly deployed | `03-deploy.txt` |
|
||
| 07:29:13 | marker **A** (34 tables) | `04-marker-A.txt` |
|
||
| 07:29:29–07:30:38 | off-site run → snapshot 1 | `05-snapshot-1.txt` |
|
||
| 07:30:50 | marker **B** + `r893_after`; `r893_only_old` + marker **C** (36 tables) | `05-snapshot-1.txt` |
|
||
| 07:30:56–07:32:04 | off-site run → **snapshot 2** (`c3d94d03`) | `06-snapshot-2.txt` |
|
||
| 07:32:38 | `r893_only_old` DROPPED, marker **D**, `r893_new2`: the LIVE record | `07-live-before-restore.txt` |
|
||
| 07:32:57–07:33:13 | full restore of snapshot 2 prepared (66.1 MB; content checked: C present, D absent); the failing line injected | `08-prepare-and-inject.txt` |
|
||
| 07:33:28 | reconstitute (forced failure) | `10-forced-failure-http.txt`, `11-controller-log-window*.txt` |
|
||
| 07:33:29 | undo dump of the live DB; app stopped 07:33:29–07:33:40 | |
|
||
| 07:33:40.9–41.6 | **live volume copied** to `restore-undo/rallly/20261020T073340Z/rallly_rallly_postgres_data.tar` (69 198 848 B), tmp then rename | `09-undo-folder-watch.txt` |
|
||
| 07:33:42 | snapshot's volume in place (`Restored 1 Docker volume(s)`) | |
|
||
| 07:33:44 | replay FAILED (`function r893_forced_replay_failure() does not exist`, exit 3); rows rolled back 07:33:45 | |
|
||
| 07:33:46–47 | app stopped; **the copy poured back** (`Restored 1 Docker volume(s)`, second time); „the live volumes and definition are put back; starting the app (R-893)" | |
|
||
| 07:33:53 | `Stack rallly started successfully` | |
|
||
| 07:33:54 | restore op ends ok=false; the copy folder **gone** (watcher: absent at 07:33:54.45) | |
|
||
| 07:34:25 | after state read: identical to 07:32:38 | `12-after-failed-restore.txt` |
|
||
| 07:36:09–07:36:15 | full restore prepared again, with no injection (injected-line count 0) | `13-normal-prepare.txt` |
|
||
| 07:36:28–07:36:57 | normal reconstitute: copy made 07:36:41–42, volume + dump replayed, app restarted, copy removed 07:36:57.9 | `14-…`, `15-…`, `16-controller-log-normal*.txt` |
|
||
| 07:38:31 | after state read = snapshot 2 | `17-after-normal-restore.txt` |
|
||
| (stop) | the session was paused by an API rate limit. When it resumed, 9202 was re-read first: nothing was in progress, no restore-undo copy, step 4 already recorded | |
|
||
| 08:43–08:45 | teardown, three layers | `18-teardown.txt` |
|
||
|
||
## Step 3: the forced failure, before and after
|
||
|
||
| | LIVE before the restore (07:32:38) | After the failed restore + put-back (07:34:25) | 2026-10-19 on 0.335.0 (old behaviour) |
|
||
|---|---|---|---|
|
||
| marker rows | A, B, C, D | **A, B, C, D** | A, B, C, D |
|
||
| `r893_only_old` (in snapshot 2 only) | absent | **absent** | present, 1 row |
|
||
| `r893_new2` (written after snapshot 2) | present, 1 row | **present, 1 row** | present, 1 row |
|
||
| public base tables | 36 | **36** | 37 |
|
||
| schema-only md5 | `05828c77…` | **`05828c77…`** (control equal) | — |
|
||
| per-table row-count md5 | `d8bcbc4b…` | **`d8bcbc4b…`** | — |
|
||
| data-only md5 | `e5c1a72f…` | **`e5c1a72f…`** | — |
|
||
| definition | rallly 4.15.3, postgres 18 (digest pins) | same | same |
|
||
| app | running | **running, healthy**; controller `state=running`; front door 200 | stopped, held (`restore_mixed`) |
|
||
| restore hold for rallly | none | **none** (holds: chaoscrash, chaosoom, chaosoomb, gokapi, as at baseline) | `restore_mixed` |
|
||
| `backups/restore-undo/rallly` | absent | **absent** (it existed 07:33:40–07:33:54) | (did not exist) |
|
||
|
||
The DB volume's `CreatedAt` moved from 07:28:21 to 07:33:46. That is expected: the put-back recreates the volume
|
||
from the copy. Every content observable is equal, the data-only dump included.
|
||
|
||
## Step 4: the normal restore of snapshot 2
|
||
|
||
| | Recorded just before snapshot 2 (07:30:50) | After the normal restore (07:38:31) |
|
||
|---|---|---|
|
||
| marker rows | A, B, C | **A, B, C** |
|
||
| `r893_only_old` | present, 1 row | **present, 1 row** |
|
||
| `r893_new2` | absent | **absent** |
|
||
| public base tables | 36 | **36** |
|
||
| schema-only md5 | `f872bd54…` | **`f872bd54…`** |
|
||
| per-table row-count md5 | `c032377e…` | **`c032377e…`** |
|
||
| data-only md5 | `db5903ab…` | `1e688d1b…` (differs; see below) |
|
||
| app | running | **running, healthy**; front door 200; no hold |
|
||
| restore-undo folder | — | **absent** (existed 07:36:41–07:36:57) |
|
||
|
||
- **Table count.** Snapshot 2 has **36** base tables, not 37. It held `r893_only_old` but not `r893_new2`. The
|
||
„37" in the brief is last night's MIXED state (36 live tables + the stray `r893_only_old`), not the snapshot's.
|
||
- **The data-only md5.** It differs from the read taken 6 s before the snapshot's own dump started, while every row
|
||
count and the schema are equal. Not explained here. It may be a row value that Rallly touched between that read
|
||
and the dump (the off-site run restarted the app), or row order after the reload. It was not part of the
|
||
acceptance list, and no row count moved. In step 3 the same observable was byte-equal, because there the volume
|
||
itself was put back.
|
||
|
||
## The sentences the household sees
|
||
|
||
- **Failed restore, put back** (`err.backup.restore_failed_put_back`, from `/api/backup/restore-status`): „A teljes
|
||
visszaállítás sikertelen: a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai
|
||
visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut". Measured **true**: the data is
|
||
identical and the app runs. It does **not** contain „vedd fel velünk a kapcsolatot".
|
||
- **Normal restore:** „A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20.
|
||
09:30) — az alkalmazás újraindult." True. 09:30 is local time (CEST) for snapshot 2's dump at 07:30 UTC.
|
||
- Flash keys seen on the way: `flash.offbox.target_saved_escrow_agent_down` (no agent on 9202, expected),
|
||
`escrow_confirmed`, `run_started`, `restore_started`, `full_restore_started`.
|
||
|
||
## Not measured (said plainly)
|
||
|
||
- **The version-change branch** (the snapshot's older definition written, then the live one written back). It needs
|
||
an app at an older catalog step, the same limit as last night. Pinned by `r893_putback_test.go`.
|
||
- **The `_files` sentence** (`restore_failed_put_back_files`). Rallly has no drive files, so 0 files were placed.
|
||
- **The refusals before the stop:** the fit check refusing (`restore_undo_no_space`), an unmeasurable volume
|
||
(`restore_undo_unmeasured`) and a failed copy (`restore_undo_copy_failed`). Only the positive path of the fit
|
||
check ran: it passed and the app was stopped after it.
|
||
- **A put-back that itself fails** (hold + kept copy).
|
||
- **The operator notification** for a put-back. 9202 has no hub, so it was not observable.
|
||
|
||
## Side observations
|
||
|
||
- Each manual off-site run restarted Rallly (its dump leg), as last night.
|
||
- The controller logged one expected `[ERROR]` pair per failure (the import and the rollback line) and one
|
||
`[ERROR] [web] off-box reconstitute … (async)` line that carries the household sentence. The sentence is good
|
||
news, so ERROR is a loud level for a put-back that worked. This is noted for the parent, not changed.
|
||
- The empty folder `backups/restore-undo/` is left behind after the per-app folder is removed (the code removes the
|
||
per-app folder only). This drill removed it by name.
|
||
|
||
## Teardown (three layers)
|
||
|
||
- **Machine (9202):**
|
||
- Rallly was removed through the product (`POST /api/stacks/rallly/stop`, then `/remove` with `remove_hdd_data` +
|
||
`remove_backups`). Result: `volumes_removed: rallly_rallly_postgres_data`; 0 containers, 0 volumes; no
|
||
`backups/primary/rallly`, no `offsite-restore/rallly`. The template folder `/opt/docker/stacks/rallly`
|
||
(`.felhom.yml`, `docker-compose.yml`, `hold-logs`) was there before and stays.
|
||
- `r893b-sftp` was removed. So were the images `atmoz/sftp:alpine` and `lukevella/rallly:4.15.3` (both absent at
|
||
baseline), the repo folder, the key and the empty `backups/restore-undo/`.
|
||
- `settings.json` was put back from the 07:25 copy (sha256 `cb6b18f8…` = baseline) and `offbox/` from its tar,
|
||
with the controller stopped.
|
||
- The controller is back on **0.331.0**: healthy, 0 `[ERROR]`. Off-site target unset, no per-app off-site switch,
|
||
restore holds as at baseline.
|
||
- The 0.336.0-rc.1 image was removed by name. The image list is **identical to the baseline**. All
|
||
`/root/r893b-*` and `/tmp/r893b-*` files are gone.
|
||
- Not put back: `restore-status.json`, the controller's record of the last op. It now names this drill's last
|
||
restore, as it named last night's at baseline.
|
||
- **Host (demo-hp):** only the per-call `/tmp/r893b-g-*` script copies, each removed after its call; 0 left. Nothing
|
||
on `pve/data`. No whole-guest backup taken.
|
||
- **Hub:** nothing. 9202 has no hub.
|
||
- **DooPlex:** the throwaway key was shredded from the session scratchpad.
|