245 lines
17 KiB
Plaintext
245 lines
17 KiB
Plaintext
# S4-restore-window-step2 — restore pressed at 2026-09-26T07:41:33Z (own unit, snapshot helyi)
|
|
{"ok": true, "snapshot_id": "helyi", "snapshots": [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}], "http": "HTTP/2 302", "location": ["location: /backups/restore?flash=flash.restore.started"], "seconds": 36.5, "state_after": "restarting", "hold_after": null}
|
|
controller lines:
|
|
2026/09/26 07:41:34 handlers.go:1707: [WARN] [web] Restore requested (async): stack=docmost, snapshot=helyi from 172.18.0.6:42900
|
|
2026/09/26 07:41:34 restore_unit.go:383: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
|
|
2026/09/26 07:41:34 manager.go:1305: [INFO] [stacks] Stopping stack: docmost
|
|
2026/09/26 07:41:35 manager.go:1314: [INFO] [stacks] Stack docmost stopped successfully (took 0.8s)
|
|
2026/09/26 07:41:35 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
|
|
2026/09/26 07:41:36 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
|
|
2026/09/26 07:41:36 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
|
|
2026/09/26 07:41:36 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for docmost
|
|
2026/09/26 07:41:36 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
|
2026/09/26 07:41:36 deploy.go:708: [INFO] [stacks] Redeploying docmost from recovery unit with 4 env vars
|
|
2026/09/26 07:41:36 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
|
2026/09/26 07:41:36 pin.go:93: [INFO] [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:18-alpine, docmost-redis=redis:7-alpine
|
|
2026/09/26 07:41:36 manager.go:1269: [INFO] [stacks] Starting stack docmost services only: [docmost-postgres]
|
|
2026/09/26 07:41:37 manager.go:1280: [INFO] [stacks] Stack docmost services [docmost-postgres] started (took 0.4s)
|
|
2026/09/26 07:41:37 restore_db.go:87: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
|
2026/09/26 07:41:53 main.go:3724: [INFO] [deadapp] docmost crossed the crash_loop threshold but a backup or restore is running (single-flight held) — NOT stopped
|
|
2026/09/26 07:42:08 restore_unit.go:450: [ERROR] [backup] DB re-import for docmost: importing postgres dump for docmost: waiting for docmost-postgres (postgres) readiness: timeout after 30s
|
|
2026/09/26 07:42:08 manager.go:1216: [INFO] [stacks] Starting stack: docmost
|
|
2026/09/26 07:42:09 manager.go:1491: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/docmost) — exit code 1 (took 0.9s)
|
|
2026/09/26 07:42:09 manager.go:1497: [ERROR] [stacks] stderr: Container docmost-redis Creating
|
|
Container docmost-redis Created
|
|
Container docmost Creating
|
|
Container docmost Created
|
|
Container docmost-postgres Starting
|
|
Container docmost-redis Starting
|
|
Container docmost-postgres Started
|
|
Container docmost-redis Started
|
|
Container docmost-postgres Waiting
|
|
Container docmost-redis Waiting
|
|
Container docmost-postgres Error dependency docmost-postgres failed to start
|
|
dependency failed to start: container docmost-postgres is unhealthy
|
|
2026/09/26 07:42:09 manager.go:1227: [ERROR] [stacks] Stack docmost start failed after 0.9s: exit code 1
|
|
stderr: Container docmost-redis Creating
|
|
Container docmost-redis Created
|
|
Container docmost Creating
|
|
Container docmost Created
|
|
Container docmost-postgres Starting
|
|
Container docmost-redis Starting
|
|
Container docmost-postgres Started
|
|
Container docmost-redis Started
|
|
Container docmost-postgres Waiting
|
|
Container docmost-redis Waiting
|
|
Container docmost-postgres Error dependency docmost-postgres failed to start
|
|
dependency failed to start: container docmost-postgres is unhealthy
|
|
2026/09/26 07:42:09 handlers.go:1715: [ERROR] [web] Restore failed (async): stack=docmost: starting docmost after restore from unit: starting stack docmost: exit code 1
|
|
stderr: Container docmost-redis Creating
|
|
Container docmost-redis Created
|
|
Container docmost Creating
|
|
Container docmost Created
|
|
Container docmost-postgres Starting
|
|
Container docmost-redis Starting
|
|
Container docmost-postgres Started
|
|
Container docmost-redis Started
|
|
Container docmost-postgres Waiting
|
|
Container docmost-redis Waiting
|
|
Container docmost-postgres Error dependency docmost-postgres failed to start
|
|
dependency failed to start: container docmost-postgres is unhealthy
|
|
|
|
docmost-postgres log (tail):
|
|
major-version-specific directory names). This better reflects how
|
|
PostgreSQL itself works, and how upgrades are to be performed.
|
|
|
|
See also https://github.com/docker-library/postgres/pull/1259
|
|
|
|
Counter to that, there appears to be PostgreSQL data in:
|
|
/var/lib/postgresql
|
|
|
|
This is usually the result of upgrading the Docker image without
|
|
upgrading the underlying database using "pg_upgrade" (which requires both
|
|
versions).
|
|
|
|
The suggested container configuration for 18+ is to place a single mount
|
|
at /var/lib/postgresql which will then place PostgreSQL data in a
|
|
subdirectory, allowing usage of "pg_upgrade --link" without mount point
|
|
boundary issues.
|
|
|
|
See https://github.com/docker-library/postgres/issues/37 for a (long)
|
|
discussion around this process, and suggestions for how to do so.
|
|
Error: in 18+, these Docker images are configured to store database data in a
|
|
format which is compatible with "pg_ctlcluster" (specifically, using
|
|
major-version-specific directory names). This better reflects how
|
|
PostgreSQL itself works, and how upgrades are to be performed.
|
|
|
|
See also https://github.com/docker-library/postgres/pull/1259
|
|
|
|
Counter to that, there appears to be PostgreSQL data in:
|
|
/var/lib/postgresql
|
|
|
|
This is usually the result of upgrading the Docker image without
|
|
upgrading the underlying database using "pg_upgrade" (which requires both
|
|
versions).
|
|
|
|
The suggested container configuration for 18+ is to place a single mount
|
|
at /var/lib/postgresql which will then place PostgreSQL data in a
|
|
subdirectory, allowing usage of "pg_upgrade --link" without mount point
|
|
boundary issues.
|
|
|
|
See https://github.com/docker-library/postgres/issues/37 for a (long)
|
|
discussion around this process, and suggestions for how to do so.
|
|
|
|
docmost log (tail):
|
|
|
|
seed reads back through the front door: False
|
|
# S4-restore-window-step2 — 2026-09-26T09:42:34+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
|
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
|
installed=null
|
|
conversion_copy=None
|
|
state=degraded phase=done
|
|
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
|
manifest.controller_version = "0.274.0"
|
|
manifest.created_at = "2026-09-26T07:32:53Z"
|
|
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
|
manifest.db_dumps = ["docmost-postgres.sql"]
|
|
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
|
compose/ images:
|
|
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
|
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
|
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
|
compose/ postgres mount:
|
|
- docmost_postgres_data:/var/lib/postgresql
|
|
docmost_postgres_data:
|
|
mtimes (UTC):
|
|
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
|
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
|
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
|
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
|
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
|
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
|
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
|
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
|
docmost-postgres.sql: Dumped from database version 16.15
|
|
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
|
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
|
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
|
manifest.controller_version = "0.274.0"
|
|
manifest.created_at = "2026-09-26T07:27:53Z"
|
|
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
|
manifest.db_dumps = ["docmost-postgres.sql"]
|
|
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
|
compose/ images:
|
|
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
|
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
|
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
|
compose/ postgres mount:
|
|
- docmost_postgres_data:/var/lib/postgresql/data
|
|
docmost_postgres_data:
|
|
mtimes (UTC):
|
|
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
|
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
|
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
|
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
|
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
|
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
|
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
|
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
|
docmost-postgres.sql: Dumped from database version 16.15
|
|
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
|
== live
|
|
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
|
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
|
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
|
Error response from daemon: Container d38697606a1e60ebffa27adc7e262b04d6690fd593c2fc02035d6bcb1fcb83e5 is restarting, wait until the container is running
|
|
docmost docmost/docmost:0.96.0 Created
|
|
docmost-redis redis:7-alpine Up 30 seconds (healthy)
|
|
docmost-postgres postgres:18-alpine Restarting (1) 8 seconds ago
|
|
volume docmost_docmost_postgres_data
|
|
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
|
volume docmost_docmost_redis_data
|
|
volume docmost_docmost_storage
|
|
pg volume root:
|
|
18 PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts
|
|
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
|
# S4-restore-window-step2 — 2026-09-26T09:42:50+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
|
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
|
installed=null
|
|
conversion_copy=None
|
|
state=degraded phase=done
|
|
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
|
manifest.controller_version = "0.274.0"
|
|
manifest.created_at = "2026-09-26T07:32:53Z"
|
|
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
|
manifest.db_dumps = ["docmost-postgres.sql"]
|
|
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
|
compose/ images:
|
|
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
|
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
|
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
|
compose/ postgres mount:
|
|
- docmost_postgres_data:/var/lib/postgresql
|
|
docmost_postgres_data:
|
|
mtimes (UTC):
|
|
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
|
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
|
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
|
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
|
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
|
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
|
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
|
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
|
docmost-postgres.sql: Dumped from database version 16.15
|
|
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
|
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
|
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
|
manifest.controller_version = "0.274.0"
|
|
manifest.created_at = "2026-09-26T07:27:53Z"
|
|
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
|
manifest.db_dumps = ["docmost-postgres.sql"]
|
|
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
|
compose/ images:
|
|
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
|
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
|
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
|
compose/ postgres mount:
|
|
- docmost_postgres_data:/var/lib/postgresql/data
|
|
docmost_postgres_data:
|
|
mtimes (UTC):
|
|
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
|
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
|
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
|
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
|
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
|
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
|
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
|
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
|
docmost-postgres.sql: Dumped from database version 16.15
|
|
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
|
== live
|
|
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
|
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
|
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
|
Error response from daemon: Container d38697606a1e60ebffa27adc7e262b04d6690fd593c2fc02035d6bcb1fcb83e5 is restarting, wait until the container is running
|
|
docmost docmost/docmost:0.96.0 Created
|
|
docmost-redis redis:7-alpine Up 46 seconds (healthy)
|
|
docmost-postgres postgres:18-alpine Restarting (1) 24 seconds ago
|
|
volume docmost_docmost_postgres_data
|
|
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
|
volume docmost_docmost_redis_data
|
|
volume docmost_docmost_storage
|
|
pg volume root:
|
|
18 PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts
|
|
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|