106 lines
12 KiB
Plaintext
106 lines
12 KiB
Plaintext
19:23:48Z == 1 deploy
|
||
19:23:49Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||
19:25:00Z state: running
|
||
19:25:30Z == 2 marker A + backup
|
||
19:25:34Z CREATE TABLE
|
||
INSERT 0 1
|
||
19:25:38Z [2] DB markers: 'A'
|
||
19:25:38Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||
19:27:09Z backup finished: {"db_dump": {"count": 3, "duration": "1m28.092103803s", "last_run": "2026-10-21T19:27:06.731987264Z", "success": true}, "enabled": true, "running": false}
|
||
19:27:14Z unit on disk:
|
||
UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly
|
||
2026-10-21 19:27 488 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml
|
||
2026-10-21 19:27 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml
|
||
2026-10-21 19:27 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml
|
||
2026-10-21 19:26 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||
2026-10-21 19:25 111383 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||
2026-10-21 19:27 3200 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json
|
||
2026-10-21 19:26 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json
|
||
19:27:14Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore
|
||
19:27:20Z INSERT 0 1
|
||
19:27:26Z [3-before] DB markers: 'A,B'
|
||
19:27:26Z [3] restore points offered: [{"time": "2026-10-21T19:25:39Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||
19:27:26Z [3] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||
19:27:26Z [3] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-21T19:27:26.239218667Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-po
|
||
19:27:59Z [3] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-21T19:27:26.239218667Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-21T19:27:56.717480993Z"}, "last_recent": true}
|
||
19:28:19Z [3] after: state=running hold=None
|
||
19:28:38Z [3-after] DB markers: 'A'
|
||
19:28:38Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half
|
||
19:28:42Z INSERT 0 1
|
||
19:28:47Z [4-live] DB markers: 'A,C'
|
||
19:28:51Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304
|
||
19:28:51Z == 5 the unit restore over the damaged copy (D22: the live data must come back, the app must run)
|
||
19:28:51Z [5] restore points offered: [{"time": "2026-10-21T19:25:39Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||
19:28:51Z [5] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||
19:28:51Z [5] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-21T19:28:51.801475739Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-21T19:27:56.717480993Z"}, "last_recent": true}
|
||
19:29:19Z [5] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-21T19:28:51.801475739Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut", "finished_at": "2026-10-21T19:29:18.735655321Z"}, "last_recent": true}
|
||
19:29:39Z [5] after: state=running hold=None
|
||
19:30:14Z [5-after (want A,C)] DB markers: 'A,C'
|
||
19:30:18Z containers:
|
||
rallly Up 59 seconds (healthy)
|
||
rallly-postgres Up About a minute (healthy)
|
||
19:30:30Z undo copies left on disk:
|
||
/mnt/sys_drive/felhom-data/backups/restore-undo
|
||
/var/lib/felhom/sys_drive/felhom-data/backups/restore-undo
|
||
19:30:34Z controller log (restore lines):
|
||
2026/10/21 19:29:09 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||
2026/10/21 19:29:11 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||
2026/10/21 19:29:11 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||
2026/10/21 19:29:11 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/rallly — 4 env vars, 2 encrypted, 2 sensitive fields
|
||
2026/10/21 19:29:11 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||
2026/10/21 19:29:11 deploy.go:883: [INFO] [stacks] Redeploying rallly from recovery unit with 4 env vars
|
||
2026/10/21 19:29:11 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/rallly — 4 env vars, 0 encrypted, 2 sensitive fields
|
||
2026/10/21 19:29:11 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||
2026/10/21 19:29:11 pin.go:93: [INFO] [stacks] pin rallly: rallly=lukevella/rallly:4.15.3, rallly-postgres=postgres:18-alpine
|
||
2026/10/21 19:29:11 offbox_restore_undo.go:241: [WARN] [offbox] rallly: the restore failed (failed to restore 1 volume(s): [rallly_rallly_postgres_data]) — the live volumes and definition are put back; starting the app (R-893)
|
||
2026/10/21 19:29:11 manager.go:1348: [DEBUG] [stacks] StartStack rallly: current state=stopped deployed=true
|
||
2026/10/21 19:29:11 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||
2026/10/21 19:29:11 manager.go:1358: [DEBUG] [stacks] StartStack rallly: prepared 10 env vars for compose
|
||
2026/10/21 19:29:11 manager.go:1695: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/rallly)
|
||
2026/10/21 19:29:15 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
|
||
2026/10/21 19:29:18 manager.go:1366: [INFO] [stacks] Stack rallly started successfully (took 6.5s)
|
||
2026/10/21 19:29:18 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/rallly — 4 env vars, 0 encrypted, 2 sensitive fields
|
||
2026/10/21 19:29:18 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||
2026/10/21 19:29:18 installed.go:420: [INFO] [stacks] installed-images rallly: recorded 2 service(s) (rallly=lukevella/rallly:4.15.3 (sha256:8cd979aefe8d…), rallly-postgres=postgres:18-alpine (sha256:77f585114c32…))
|
||
2026/10/21 19:29:18 handlers.go:1883: [ERROR] [web] Restore failed (async): stack=rallly: a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut
|
||
2026/10/21 19:29:21 manager.go:1695: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/rallly)
|
||
2026/10/21 19:29:21 manager.go:1811: [INFO] [stacks] Stack rallly post-start status:
|
||
2026/10/21 19:29:21 manager.go:1814: [INFO] [stacks] rallly lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 running Up 3 seconds (health: starting)
|
||
2026/10/21 19:29:21 manager.go:1814: [INFO] [stacks] rallly-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 9 seconds (healthy)
|
||
2026/10/21 19:29:45 healthprobe.go:187: [DEBUG] Health probe rallly: HTTP GET :3000/ → 200 (188ms)
|
||
2026/10/21 19:29:55 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 10s ago, effective interval 5m0s, healthy=true
|
||
2026/10/21 19:30:05 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 20s ago, effective interval 5m0s, healthy=true
|
||
2026/10/21 19:30:15 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 30s ago, effective interval 5m0s, healthy=true
|
||
2026/10/21 19:30:25 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 40s ago, effective interval 5m0s, healthy=true
|
||
2026/10/21 19:30:25 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=true composePath=/opt/docker/stacks/rallly/docker-compose.yml
|
||
19:30:34Z == 6 a fresh backup (the damaged unit is replaced) and marker D (the newest live row)
|
||
19:30:34Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||
19:31:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m23.279234156s", "last_run": "2026-10-21T19:31:57.486157078Z", "success": true}, "enabled": true, "running": false}
|
||
19:32:04Z INSERT 0 1
|
||
19:32:10Z [6-live (want A,C,D)] DB markers: 'A,C,D'
|
||
19:32:10Z == 7 the restore, the controller KILLED (kill -9) right after the volume step, before the replay
|
||
19:32:10Z [7] POST /backup/restore -> HTTP/2 302
|
||
19:32:28Z killer:
|
||
KILLED pid 850421 at 19:32:27.386610327
|
||
19:32:28Z waiting for the controller to come back
|
||
19:33:19Z [7] restore-status after the restart: {"running": false, "started_at": "0001-01-01T00:00:00Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak.", "finished_at": "2026-10-21T19:32:37.86861273Z", "interrupted": true, "put_back": true}, "last_recent": true}
|
||
19:33:23Z [7-after (want A,C,D)] DB markers: 'A,C,D'
|
||
19:33:27Z containers:
|
||
rallly Up 49 seconds (healthy)
|
||
rallly-postgres Up 54 seconds (healthy)
|
||
19:33:33Z undo copies left on disk:
|
||
/mnt/sys_drive/felhom-data/backups/restore-undo
|
||
/var/lib/felhom/sys_drive/felhom-data/backups/restore-undo
|
||
19:33:37Z controller log after the restart (appstop / restore):
|
||
2026/10/21 19:29:05 restore_crash_putback.go:80: [INFO] [backup] rallly: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)
|
||
2026/10/21 19:29:11 offbox_restore_undo.go:241: [WARN] [offbox] rallly: the restore failed (failed to restore 1 volume(s): [rallly_rallly_postgres_data]) — the live volumes and definition are put back; starting the app (R-893)
|
||
2026/10/21 19:32:24 restore_crash_putback.go:80: [INFO] [backup] rallly: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)
|
||
2026/10/21 19:32:28 appstop_marker.go:350: [WARN] [appstop] crash recovery: a local restore (op "unit-restore:rallly") was interrupted and left 1 app(s) stopped — restarting them: [rallly]
|
||
2026/10/21 19:32:31 restore_crash_putback.go:169: [WARN] [appstop] rallly: the interrupted restore is put back from /mnt/sys_drive/felhom-data/backups/restore-undo/rallly/20261021T193223Z — 1 volume(s) poured back, 0 removed, definition=true (R-1195)
|
||
2026/10/21 19:32:31 appstop_marker.go:373: [WARN] [appstop] crash recovery: rallly's interrupted restore is undone — its data is as before the restore (R-1195)
|
||
2026/10/21 19:32:37 appstop_marker.go:391: [INFO] [appstop] crash recovery: restarted rallly after the interrupted a local restore
|
||
2026/10/21 19:32:37 restore_record.go:105: [WARN] [backup] restore of rallly (restore, started 2026-10-21T19:32:10Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again
|
||
2026/10/21 19:32:37 restore_record_wiring.go:36: [INFO] [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: true)
|
||
2026/10/21 19:32:37 notifier.go:1373: [WARN] notifier disabled (no hub configured): DROPPED event restore_interrupted (severity warning) — further restore_interrupted events are logged at DEBUG only
|
||
exit 0
|