10:10:26Z == 1 deploy
10:10:27Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
10:11:12Z state: running
10:11:42Z == 2 marker A + backup
10:11:46Z CREATE TABLE
INSERT 0 1
10:11:49Z [2] DB markers: 'A'
10:11:49Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
10:12:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m9.055678483s", "last_run": "2026-10-20T10:12:58.690358712Z", "success": true}, "enabled": true, "running": false}
10:13:04Z unit on disk:
UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly
2026-10-20 10:12 487 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml
2026-10-20 10:12 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml
2026-10-20 10:12 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml
2026-10-20 10:12 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
2026-10-20 10:11 111371 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
2026-10-20 10:12 3195 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json
2026-10-20 10:12 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json
10:13:04Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore
10:13:09Z INSERT 0 1
10:13:12Z [3-before] DB markers: 'A,B'
10:13:12Z [3] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
10:13:12Z [3] POST /backup/restore -> HTTP/2 302  ['location: /backups/restore?flash=flash.restore.started']
10:13:12Z [3]   restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}}
10:13:40Z [3]   restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
10:14:00Z [3] after: state=running hold=None
10:14:18Z [3-after] DB markers: 'A'
10:14:18Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half
10:14:22Z INSERT 0 1
10:14:25Z [4-live] DB markers: 'A,C'
10:14:29Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304
10:14:29Z == 5 the unit restore over the damaged copy
10:14:29Z [5] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
10:14:29Z [5] POST /backup/restore -> HTTP/2 302  ['location: /backups/restore?flash=flash.restore.started']
10:14:29Z [5]   restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
10:14:44Z [5]   restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-p
10:15:04Z [5] after: state=degraded hold=None
10:15:38Z [5-after] DB markers: 'Error response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running'
10:15:41Z containers:
rallly Created
rallly-postgres Restarting (1) 5 seconds ago
10:15:44Z db container log tail:
Data page checksums are enabled.

initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
10:15:47Z any undo / safety copy of the live state on disk?
/mnt/sizing-bench/backups/secondary/paperless-ngx/recovery-unit/db-dumps/pre-restore-20260927T094411Z-paperless-ngx-postgres.sql
/mnt/sys_drive/felhom-data/backups/primary/rallly
/mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
10:15:51Z controller log (restore lines):
Container rallly Created 
 Container rallly-postgres Starting 
 Container rallly-postgres Started 
 Container rallly-postgres Waiting 
 Container rallly-postgres Error dependency rallly-postgres failed to start
dependency failed to start: container rallly-postgres is unhealthy
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
stderr: ...twork rallly_rallly-internal Creating 
 Network rallly_rallly-internal Created 
 Network rallly_rallly-internal Created 
 Container rallly-postgres Creating 
 Container rallly-postgres Created 
 Container rallly Creating 
 Container rallly Created 
 Container rallly-postgres Starting 
 Container rallly-postgres Started 
 Container rallly-postgres Waiting 
 Container rallly-postgres Error dependency rallly-postgres failed to start
dependency failed to start: container rallly-postgres is unhealthy
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
stderr: ...twork rallly_rallly-internal Creating 
 Network rallly_rallly-internal Created 
 Network rallly_rallly-internal Created 
 Container rallly-postgres Creating 
 Container rallly-postgres Created 
 Container rallly Creating 
 Container rallly Created 
 Container rallly-postgres Starting 
 Container rallly-postgres Started 
 Container rallly-postgres Waiting 
 Container rallly-postgres Error dependency rallly-postgres failed to start
dependency failed to start: container rallly-postgres is unhealthy
2026/10/20 10:14:49 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
2026/10/20 10:14:59 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
2026/10/20 10:15:09 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
2026/10/20 10:15:09 main.go:4054: [INFO] [deadapp] rallly crossed the crash_loop threshold (8 in 10m0s) while a backup/quiesce/update holds it — NOT stopped
2026/10/20 10:15:19 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
2026/10/20 10:15:29 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
2026/10/20 10:15:39 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
2026/10/20 10:15:49 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
exit 0