Night 2026-10-19: register 161 -> 161 (closed R-921, R-953, R-986; opened R-1168..R-1170; R-1001/R-1011/R-1010/R-1070/R-1167 READY, R-893 READY with the measured design note -> D20, R-1083 gap); morning list in STATUS; 03/05/07/11 updated; R-893 evidence
gates / gates (push) Successful in 13m5s
gates / gates (push) Successful in 13m5s
This commit is contained in:
@@ -0,0 +1,25 @@
|
||||
Mon Oct 19 18:45:25 UTC 2026
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 16 hours (healthy)
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 18 hours (healthy)
|
||||
paperless-postgres postgres:18-alpine Up 18 hours (healthy)
|
||||
paperless-redis redis:7-alpine Up 18 hours (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 18 hours (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 18 hours (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 18 hours (healthy)
|
||||
immich-redis redis:7-alpine Up 18 hours (healthy)
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 22 hours (healthy)
|
||||
traefik traefik:v3.7.13 Up 3 days
|
||||
/opt/docker/stacks/rallly
|
||||
docker-compose.yml
|
||||
hold-logs
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/loop1 69G 11G 54G 17% /var/lib/felhom
|
||||
total 20
|
||||
drwx------ 2 root root 4096 Sep 30 08:40 .
|
||||
drwxr-xr-x 4 root root 4096 Oct 19 18:46 ..
|
||||
-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts
|
||||
-rw------- 1 root root 64 Sep 30 08:40 repo_password
|
||||
-rw------- 1 root root 411 Sep 30 08:40 ssh_key
|
||||
83723cedb2aa77ace51345e585937720fde114d1f0b24da21eb37fba4bd39374 settings.json
|
||||
-rw-r--r-- 1 root root 27003 Oct 19 03:14 /root/r893-settings.json.before
|
||||
@@ -0,0 +1,13 @@
|
||||
INSERT 0 1
|
||||
=== A-written 18:49:07 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up About a minute (healthy)
|
||||
rallly-postgres postgres:18-alpine Up About a minute (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1
|
||||
r893_after table exists: 0
|
||||
r893_after rows: ERROR: relation "r893_after" does not exist
|
||||
LINE 1: select count(*) from r893_after;
|
||||
^
|
||||
public tables: 34
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,13 @@
|
||||
INSERT 0 1
|
||||
CREATE TABLE
|
||||
INSERT 0 1
|
||||
=== B-written 18:51:46 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 35 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 41 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1
|
||||
r893_after table exists: 1
|
||||
r893_after rows: 1
|
||||
public tables: 35
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,11 @@
|
||||
126
|
||||
34
|
||||
2660:-- PostgreSQL database dump complete
|
||||
2660-SELECT r893_forced_replay_failure(); -- R-893 measurement 2026-10-19: forced error AFTER the volumes are replaced
|
||||
2661:-- PostgreSQL database dump complete
|
||||
2720-SELECT r893_forced_replay_failure(); -- R-893 measurement 2026-10-19: forced error AFTER the volumes are replaced
|
||||
2721:-- PostgreSQL database dump complete
|
||||
total 67608
|
||||
drwxr-xr-x 2 root root 4096 Oct 19 18:54 .
|
||||
drwxr-xr-x 5 root root 4096 Oct 19 18:54 ..
|
||||
-rw-r--r-- 1 root root 69221888 Oct 19 18:54 rallly_rallly_postgres_data.tar
|
||||
@@ -0,0 +1,15 @@
|
||||
CREATE TABLE
|
||||
INSERT 0 1
|
||||
INSERT 0 1
|
||||
=== before-snapshot-2 18:53:21 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 2 minutes (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 2 minutes (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1
|
||||
r893_after table exists: 1
|
||||
r893_after rows: 1
|
||||
r893_only_old exists: 1
|
||||
r893_new2 exists: 0
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,16 @@
|
||||
DROP TABLE
|
||||
INSERT 0 1
|
||||
CREATE TABLE
|
||||
INSERT 0 1
|
||||
=== LIVE-before-the-restore 18:54:53 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 30 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 35 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1, D-after-snapshot-2=1
|
||||
r893_after table exists: 1
|
||||
r893_after rows: 1
|
||||
r893_only_old exists: 0
|
||||
r893_new2 exists: 1
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,40 @@
|
||||
2026/10/19 18:55:43 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:43 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec 8ad4e7b50692 pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges
|
||||
2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/19 18:55:43 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (size=110.0 KB, valid=true, tables=36, duration=325ms)
|
||||
2026/10/19 18:55:43 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB, 325ms, 36 tables)
|
||||
2026/10/19 18:55:43 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB)
|
||||
2026/10/19 18:55:43 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2
|
||||
2026/10/19 18:55:43 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:43 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s)
|
||||
2026/10/19 18:55:54 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/19 18:55:55 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/19 18:55:55 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/19 18:55:55 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/19 18:55:55 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:56 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/19 18:55:56 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:56 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:56 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:57 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows:
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:1062: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:57 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (8ad4e7b50692 → b55136567813) — rolling back into the live container
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:57 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 dbdump.go:820: [INFO] [backup] Imported DB dump pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state
|
||||
2026/10/19 18:55:58 settings.go:1951: [WARN] [settings] restore hold SET for rallly — the app stays stopped until it is cleared
|
||||
2026/10/19 18:55:58 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=starting deployed=true containers=1
|
||||
2026/10/19 18:55:58 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:58 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:59 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s)
|
||||
2026/10/19 18:55:59 offbox_reconstitute.go:531: [ERROR] [offbox] rallly: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:59 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot
|
||||
@@ -0,0 +1,14 @@
|
||||
=== after the failed replay 18:56:42 UTC
|
||||
(no rallly container listed = compose down; app held)
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
Unable to find image 'alpine:3' locally
|
||||
3: Pulling from library/alpine
|
||||
Digest: sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
|
||||
Status: Downloaded newer image for alpine:3
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1, D-after-snapshot-2=1
|
||||
r893_after exists: 1 rows: 1
|
||||
r893_only_old exists: 1 rows: 1
|
||||
r893_new2 exists: 1 rows: 1
|
||||
public tables: 37
|
||||
prisma migrations: 153
|
||||
read copy removed
|
||||
@@ -0,0 +1,4 @@
|
||||
GET stack: 200 {'status': None, 'state': 'stopped', 'held': None, 'hold': None, 'restore_hold': None, 'deployed': True}
|
||||
POST start: 409 {"ok": false, "data": {"reason": "held"}, "error": "a(z) rallly visszaállítása 2026-10-19 18:55-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot"}
|
||||
restore-status: 200 {"ok": true, "data": {"running": false, "op": "offbox-reconstitute", "stack": "rallly", "started_at": "2026-10-19T18:55:42.483513793Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": false, "message": "A teljes visszaállítás sikertelen: a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot", "finished_at": "2026-10-19T18:55:59.246941172
|
||||
settings restore hold: {'stack': 'rallly', 'at': '2026-10-19T18:55:58Z', 'replay_error': 'importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3', 'reason': 'restore_mixed'}
|
||||
@@ -0,0 +1,176 @@
|
||||
2026/10/19 18:55:42 auth.go:160: [DEBUG] [web] login attempt: visitor 192.168.0.180 (peer 172.18.0.3:41876, X-Forwarded-For "192.168.0.180", CF-Connecting-IP "")
|
||||
2026/10/19 18:55:42 auth.go:192: [DEBUG] [web] login successful from 192.168.0.180, creating session
|
||||
2026/10/19 18:55:42 session_store.go:140: [DEBUG] [web] sessions: saved 142 session(s) to dashboard-sessions.json
|
||||
2026/10/19 18:55:42 auth.go:316: [DEBUG] [web] session created, expires=2026-10-26T18:55:42Z, active_sessions=146
|
||||
2026/10/19 18:55:42 auth.go:212: [INFO] [web] Login from 192.168.0.180
|
||||
2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for GET /
|
||||
2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.3:41876
|
||||
2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for GET /launcher
|
||||
2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.3:41876
|
||||
2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
|
||||
2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.3:41876
|
||||
2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/19 18:55:43 offbox_reconstitute.go:1206: [WARN] [offbox] rallly: the snapshot dump rallly-postgres.sql has no account rows — it may predate the customer's data
|
||||
2026/10/19 18:55:43 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/19 18:55:43 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/19 18:55:43 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 5e59e7bdadc1 rallly rallly lukevella/rallly:4.15.3
|
||||
8ad4e7b50692 rallly-postgres rallly postgres:18-alpine
|
||||
197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine
|
||||
8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4
|
||||
c030b51054cb...
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container rallly (image=lukevella/rallly:4.15.3, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=8ad4e7b50692)
|
||||
2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7)
|
||||
2026/10/19 18:55:43 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb)
|
||||
2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 10 non-DB container(s)
|
||||
2026/10/19 18:55:43 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/19 18:55:43 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:43 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec 8ad4e7b50692 pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges
|
||||
2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/19 18:55:43 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (size=110.0 KB, valid=true, tables=36, duration=325ms)
|
||||
2026/10/19 18:55:43 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB, 325ms, 36 tables)
|
||||
2026/10/19 18:55:43 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB)
|
||||
2026/10/19 18:55:43 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2
|
||||
2026/10/19 18:55:43 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:43 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/19 18:55:43 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714348KB) → used≈12771MB avail≈13127MB
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714348 availKB=15567204 → total=25898MB avail=13127MB used=12771MB (49.3%)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=12.0GB avail=52.8GB (18.5%)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=175.4GB avail=714.7GB (19.7%)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.25 2.62 2.11 8/1611 4393" → 1m=2.25 5m=2.62 15m=2.11
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 57.6°C (hwmon2)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] GetInfo done in 59ms — mem=12771MB/25898MB (49.3%), rootDisk=12.0GB/68.4GB (18.5%), load=2.25/2.62/2.11, temp=57.6°C (hwmon2), cpu=13.9%
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 1m40s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container)
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/19 18:55:44 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/19 18:55:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/19 18:55:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/19 18:55:54 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 1m50s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container)
|
||||
2026/10/19 18:55:54 manager.go:1717: [DEBUG] Command completed: docker compose down (took 10.7s)
|
||||
2026/10/19 18:55:54 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s)
|
||||
2026/10/19 18:55:54 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
2026/10/19 18:55:54 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/19 18:55:55 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/19 18:55:55 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/19 18:55:55 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/19 18:55:55 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, DB_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/19 18:55:55 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:56 manager.go:1717: [DEBUG] Command completed: docker compose up -d rallly-postgres (took 0.4s)
|
||||
2026/10/19 18:55:56 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/19 18:55:56 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks
|
||||
2026/10/19 18:55:56 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/19 18:55:56 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine
|
||||
8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4
|
||||
c030b51054cb immich-postgres immich ghcr.io/immich-app/postgres...
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7)
|
||||
2026/10/19 18:55:56 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb)
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s)
|
||||
2026/10/19 18:55:56 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/19 18:55:56 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:56 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:57 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows:
|
||||
ERROR: function r893_forced_replay_failure() does not exist
|
||||
LINE 1: SELECT r893_forced_replay_failure();
|
||||
^
|
||||
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:1062: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:57 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/19 18:55:57 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine
|
||||
8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4
|
||||
c030b51054cb immich-postgres immich ghcr.io/immich-app/postgres...
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7)
|
||||
2026/10/19 18:55:57 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb)
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s)
|
||||
2026/10/19 18:55:57 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (8ad4e7b50692 → b55136567813) — rolling back into the live container
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:57 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 dbdump.go:820: [INFO] [backup] Imported DB dump pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state
|
||||
2026/10/19 18:55:58 settings.go:1951: [WARN] [settings] restore hold SET for rallly — the app stays stopped until it is cleared
|
||||
2026/10/19 18:55:58 settings.go:922: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (27892 bytes)
|
||||
2026/10/19 18:55:58 settings.go:925: [INFO] [settings] Settings saved
|
||||
2026/10/19 18:55:58 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=starting deployed=true containers=1
|
||||
2026/10/19 18:55:58 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:58 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/19 18:55:58 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:59 manager.go:1717: [DEBUG] Command completed: docker compose down (took 0.9s)
|
||||
2026/10/19 18:55:59 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s)
|
||||
2026/10/19 18:55:59 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
2026/10/19 18:55:59 offbox_reconstitute.go:531: [ERROR] [offbox] rallly: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:59 notifier.go:1373: [WARN] notifier disabled (no hub configured): DROPPED event backup_run_failures (severity error) — further backup_run_failures events are logged at DEBUG only
|
||||
2026/10/19 18:55:59 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot
|
||||
2026/10/19 18:55:59 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/19 18:56:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/19 18:56:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/19 18:56:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 2m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:56:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:56:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (2 skipped not due, 0 skipped no container)
|
||||
2026/10/19 18:56:04 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
@@ -0,0 +1,37 @@
|
||||
20:57:56 [X] stop -> 200 {'ok': True, 'message': 'Stack rallly stop completed'}
|
||||
20:58:28 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'rallly', 'volumes_removed': ['rallly_rallly_postgres_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': '
|
||||
20:58:36 [X] after remove: deployed=False leftovers='/opt/docker/stacks/rallly'
|
||||
remove: 200
|
||||
r893-sftp
|
||||
images-removed
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly': No such file or directory
|
||||
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
|
||||
vol-check-done
|
||||
Stopping 'felhom-controller-bootstrap.service', but its triggering units are still active:
|
||||
felhom-controller-bootstrap.path
|
||||
felhom-controller Up 2 seconds (health: starting)
|
||||
83723cedb2aa77ace51345e585937720fde114d1f0b24da21eb37fba4bd39374 /var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json
|
||||
total 20
|
||||
drwx------ 2 root root 4096 Sep 30 08:40 .
|
||||
drwxr-xr-x 4 root root 4096 Oct 19 18:59 ..
|
||||
-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts
|
||||
-rw------- 1 root root 64 Sep 30 08:40 repo_password
|
||||
-rw------- 1 root root 411 Sep 30 08:40 ssh_key
|
||||
0.335.0-image-removed
|
||||
0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 50 seconds (healthy)
|
||||
3bc7298260da6566d4c09245124acd92f20f62bbebe46b0181c11d99a8314f2b /var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json
|
||||
0
|
||||
filebrowser Up 2 minutes (healthy)
|
||||
paperless-webserver Up 7 minutes (healthy)
|
||||
paperless-redis Up 7 minutes (healthy)
|
||||
paperless-postgres Up 7 minutes (healthy)
|
||||
immich-server Up 7 minutes (healthy)
|
||||
immich-machine-learning Up 8 minutes (healthy)
|
||||
immich-postgres Up 8 minutes (healthy)
|
||||
immich-redis Up 8 minutes (healthy)
|
||||
traefik Up 3 days
|
||||
/var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json {'immich': None, 'paperless-ngx': None, 'privatebin': None}
|
||||
offbox_target set: False
|
||||
restore holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi']
|
||||
0
|
||||
@@ -0,0 +1,95 @@
|
||||
# P3 — R-893: a failed off-site replay, measured on scratch guest 9202 (2026-10-19 evening)
|
||||
|
||||
Night brief 2026-10-19, Part 3 — **measure and design only; no restore code changed.** Tier 0 only (demo-hp, scratch
|
||||
guest 9202). Code read live: `felhom-controller/controller/internal/backup/offbox_reconstitute.go` at `15ea002`
|
||||
(controller 0.335.0). Architecture: `07-backup-architecture.md` §6.3 (the R-893 known limit, the failure ladder
|
||||
„replay → rollback → hold"); `09` §3 decision 192 (D8: first hold, then „put back exactly as it was").
|
||||
|
||||
## How it was reproduced (the product path, one forced error)
|
||||
|
||||
- 9202 put on controller **0.335.0** for the run (was 0.331.0; put back after). Hub is OFF on 9202.
|
||||
- A **scratch off-site target**: an SFTP server container (`atmoz/sftp:alpine`, `r893-sftp`, port 2222) on 9202 itself,
|
||||
repo on 9202's own disk (`/var/lib/felhom/r893-sftp`, nvme-scratch — not `pve/data`). Configured through the
|
||||
household's own endpoint `POST /backup/offbox/config` (a throwaway ed25519 key, the server's host key pinned),
|
||||
escrow confirmed with the legacy manual endpoint (9202 has no agent), only `rallly` switched on.
|
||||
- Throwaway database app: **Rallly 4.15.3 + postgres:18-alpine**, the DB in the named volume
|
||||
`rallly_rallly_postgres_data` (deployed through the product, `box_walk.deploy`).
|
||||
- Markers in the real database (positive observables): table `r893_marker` with one row per phase, plus tables that
|
||||
exist only at certain times.
|
||||
- The **forced error**: one line `SELECT r893_forced_replay_failure();` put into the PREPARED scratch copy of the
|
||||
snapshot's dump, just before its completion marker (so the completeness gate still passes). The real
|
||||
`ReconstituteFromOffsite` then replaced the volume, started the DB alone and failed the replay — i.e. the replay
|
||||
failed **after the volume was replaced**, as the brief asked.
|
||||
|
||||
## Timeline (UTC)
|
||||
|
||||
| Time | Act | File |
|
||||
|---|---|---|
|
||||
| 18:45 | baseline: 9202 on 0.331.0, no off-site target, settings + off-box folder saved | `00-baseline.txt` |
|
||||
| 18:48 | Rallly deployed; marker **A** | `01-marker-A.txt` |
|
||||
| 18:51:19 | off-site run → snapshot 1; then marker **B** + table `r893_after` | `02-snapshot-and-marker-B.txt` |
|
||||
| 18:53 | table `r893_only_old` + marker **C**; off-site run → **snapshot 2** (18:54:29, `93dbb37d`) | `04-before-snapshot-2.txt` |
|
||||
| 18:54:53 | after snapshot 2: `r893_only_old` DROPPED (like a table a newer version removes), marker **D**, table `r893_new2` — the live state the household has | `05-live-before-restore.txt` |
|
||||
| 18:54:59 | full restore prepared into scratch (66.1 MB); the failing line injected | `03-forced-failure.txt` |
|
||||
| 18:55:42 | `POST /backup/offbox/reconstitute` (confirm) | `06-reconstitute-log.txt`, `09-controller-log-window.txt` |
|
||||
| 18:55:43 | undo dump of the LIVE db written (`pre-restore-…-rallly-postgres.sql`, 36 tables) | |
|
||||
| 18:55:55 | the snapshot's volume put in place (`Restored 1 Docker volume(s)`) | |
|
||||
| 18:55:57 | replay FAILED (`exit status 3`, single transaction — nothing of it applied) | |
|
||||
| 18:55:58 | rollback: the undo dump loaded into the database **that now sits on snapshot 2's volume** — „rollback complete" | |
|
||||
| 18:55:58–59 | hold SET (`reason: restore_mixed`), app stopped | |
|
||||
| 18:56:42 | the held volume read through a throwaway COPY (the held state untouched) | `07-after-failed-replay.txt` |
|
||||
| 18:57 | household start → **409 held** | `08-app-state-and-start.txt` |
|
||||
| 18:58–19:01 | teardown | `10-teardown.txt` |
|
||||
|
||||
## What the failed replay left
|
||||
|
||||
| | Live before the restore | After the failed replay + rollback |
|
||||
|---|---|---|
|
||||
| marker rows | A, B, C, D | A, B, C, D (the newest rows — the undo dump won) |
|
||||
| `r893_new2` (written after snapshot 2) | present, 1 row | present, 1 row |
|
||||
| `r893_only_old` (in snapshot 2 only, dropped after) | **absent** | **present, 1 row** — left from the snapshot's volume |
|
||||
| public tables | 36 | **37** |
|
||||
| definition on disk | rallly 4.15.3, postgres 18 | rallly 4.15.3, postgres 18 (unchanged — same version; the R-1106 password overlay was empty, so the definition did not move) |
|
||||
| app | running | **stopped, held** (`restore_mixed`); a start answers 409 „held" |
|
||||
| other volumes / files | — | none for Rallly (its only volume is the DB's) |
|
||||
|
||||
So the rollback gives the **newest rows on top of the older volume**: everything the undo dump knows is exact, and
|
||||
whatever only the older volume had stays behind (the dump's `--clean --if-exists` drops only the objects IT names).
|
||||
That is the R-638 / R-893 mix, now measured, not read. The 0.304.0 hold works as built: the app does not start on it.
|
||||
|
||||
**The sentences the household sees:**
|
||||
- the restore page (`err.backup.db_restore_failed_held_mixed`): „Az adatbázis a visszaállítás előtti állapotban van,
|
||||
de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze." — **measured false on
|
||||
both halves for this app**: the database is NOT as it was (a table from the backup stayed), and Rallly had no files
|
||||
or other data from the backup (only its database volume moved).
|
||||
- the start refusal: „…félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás
|
||||
biztonsági okból leállítva marad." — true.
|
||||
|
||||
## Not measured tonight (said plainly)
|
||||
|
||||
- The **version-change** branch (the snapshot's older definition written, then the live one written back by
|
||||
`holdAppAfterMixedRestore`). It needs an app installed at an older catalog step and updated before the restore; the
|
||||
drill catalog that could do it is a git repo this helper was told not to commit to. The branch is pinned by
|
||||
`r893_mixed_restore_test.go`; it ends in the same hold.
|
||||
- What support does next: no product action releases a `restore_mixed` hold with the data made right — that is the
|
||||
gap option A closes.
|
||||
|
||||
## Side observations
|
||||
|
||||
- The manual off-site run's pre-push dump leg dumped and restarted **every** app on the box (paperless, immich), not
|
||||
only the one switched on (log: `App-data backup completed: 3 databases … 4 volume dump(s)`). By design (the
|
||||
coherent pair), outside R-893's scope.
|
||||
- 9202's address moved (DHCP: .114 → .121); `box_walk.py`'s 9202 row still names .114 — run with `BASE=`.
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine (9202):** Rallly removed through the product (`volumes_removed: rallly_rallly_postgres_data`, its backups
|
||||
and the off-site scratch gone); `r893-sftp` container, `atmoz/sftp:alpine` and `alpine:3` images (pulled tonight),
|
||||
the repo folder, the key, all `/root/r893-*` and `/tmp/r893-*` removed; `settings.json` and the `offbox/` folder put
|
||||
back from the copies taken at 18:45 (off-site target unset, no per-app off-site switch, restore holds as before:
|
||||
chaoscrash, chaosoom, chaosoomb, gokapi); controller back on **0.331.0** (healthy, 0 `[ERROR]`), the 0.335.0 image
|
||||
removed from 9202. The template folder `/opt/docker/stacks/rallly` (catalog files + an old `hold-logs`) was there
|
||||
before and stays.
|
||||
- **Host (demo-hp):** only `/tmp/r893-*` copies, removed. Nothing on `pve/data`.
|
||||
- **Hub:** nothing. 9202 has no hub; the hold's events were dropped on the box („notifier disabled (no hub
|
||||
configured)").
|
||||
@@ -0,0 +1,51 @@
|
||||
# R-893 — after a failed off-site replay: put the app back exactly as it was? (design note, 2026-10-19)
|
||||
|
||||
Status: **design only, for the operator.** Measured first (`README.md` in this folder, 9202, controller 0.335.0).
|
||||
Architecture: `07` §6.3 (the failure ladder „replay → rollback → hold"), `09` §3 decision 192 (D8: first hold —
|
||||
built in 0.304.0 — then „put back exactly as it was"). No restore code was changed.
|
||||
|
||||
## What the measurement showed (plain words)
|
||||
|
||||
An off-site restore of one app replaces the app's database volume with the backup's, then loads the backup's
|
||||
database copy. When that load fails, the box puts the household's own database copy (taken a minute before) back
|
||||
**on top of the backup's volume**. Measured on Rallly: all the newest rows came back, but a table that only the backup
|
||||
had **stayed behind** (37 tables instead of 36). The app is then held stopped, and a start is refused — that part
|
||||
works. But nothing in the product can make the data right again; the household waits for support. And the restore
|
||||
page tells them the database „is back as it was before the restore" — measured not true.
|
||||
|
||||
## The question
|
||||
|
||||
**When an off-site restore of one app fails half-way, should the box put the app back exactly as it was and start
|
||||
it, or keep holding it stopped for support?**
|
||||
|
||||
## The options
|
||||
|
||||
| | A — keep a copy of the app's volumes, and put it back | B — keep holding; only make the database part exact |
|
||||
|---|---|---|
|
||||
| What happens | After the undo dump and the stop, the box saves the app's live volumes (the same volume-dump helper the nightly copy uses) and keeps the live definition (already done since 0.304.0). On a failed replay it puts those volumes and the definition back — the state from one minute before — and starts the app. The undo dump stays as a second way back. | On a failed replay the box empties the database volume before it loads the undo dump (a rebuilding load, on the rollback path only). Then the database is exactly as it was. The app stays held while files or other volumes moved; an app whose only change was its database volume can start. |
|
||||
| Disk | One extra copy of the app's volumes for the length of the restore (Rallly: 69 MB, made in about 1 s). A free-space check refuses BEFORE the stop when the copy does not fit — then nothing is touched. | None. |
|
||||
| Code | ~1 session: the save step, the put-back on every failed branch, the fit check, tests; then one 9202 drill (tonight's setup can be reused). | ~½–1 session: the emptying step and its tests; the R-638 option-B loader, but only on this one path. |
|
||||
| Risk | Low: both helpers exist and already run every night. A put-back that fails leaves the hold, as today. | Higher: the volume is emptied before the undo is loaded, so a bad undo dump leaves an empty database (the backup's volume is gone too). Files and other volumes are still the backup's. |
|
||||
| What the household sees | „The restore did not finish. Nothing changed: the app runs as before." No support call. | Usually still „the app is stopped; contact us" — fewer cases than today, but the same support call. |
|
||||
|
||||
## Pick: A
|
||||
|
||||
It is the direction the operator already ruled (D8, second half), it makes the restore page's promise true, and the
|
||||
measured cost is small: the extra disk is one copy of the app's own volumes, for minutes, and a fit check refuses
|
||||
cleanly when there is no room. B removes only one of the mixed parts and adds a new way to lose the database.
|
||||
|
||||
**One small fix either way, now:** the sentence `err.backup.db_restore_failed_held_mixed` says the database is back as
|
||||
it was and that files and other data came from the backup. Measured: neither is reliably true (Rallly had no files or
|
||||
other volumes from the backup, and its database was not as before). It should say only what is certain: the restore
|
||||
stopped half-way, the app is held stopped, and nothing is lost yet.
|
||||
|
||||
**If the operator does nothing:** a failed off-site restore keeps ending in a held app that only support can bring
|
||||
back, and the page keeps telling the household their database is back as it was.
|
||||
|
||||
## First slice and its red test (for A)
|
||||
|
||||
- Red test beside `r893_mixed_restore_test.go`: a failed replay after a volume replay must call the volume put-back
|
||||
with the live copy, write the live definition back, and START the app (no hold); with too little free space for the
|
||||
copy, the restore must refuse with `StopStack` called 0 times.
|
||||
- Then the 9202 drill (this folder's `README.md`, same steps): after the forced failure, the table count and the
|
||||
markers must equal the live state before the restore (36 tables, A–D, no `r893_only_old`), and the app must run.
|
||||
Reference in New Issue
Block a user