09 decisions 40-41 (operator rulings 2026-09-26); A1 spike: the unit's definition and data drift apart (R-696 widened), R-697 filed
gates / gates (push) Successful in 25s
gates / gates (push) Successful in 25s
This commit is contained in:
@@ -0,0 +1,91 @@
|
||||
# A1 — the window between an update and the next backup, measured (9202, controller 0.274.0, drill catalog)
|
||||
|
||||
2026-09-26 07:24–07:44Z. Guest 9202 on demo-hp, pointed at the drill catalog (`tools/repoint.py drill`, saved
|
||||
config `controller.yaml.pre-vt0926`). docmost deployed at `docmost/docmost:0.95.0` + `postgres:16-alpine` from a
|
||||
one-commit drill template (`85af330`, reverted by `8c3d099` right after the deploy, so the drill equals live
|
||||
`main` again), seeded through docmost's own front door (a user + login, `fixtures.py`), then climbed the live
|
||||
ladder with two presses of the guarded Update. Every file here is the raw record; this page is the reading.
|
||||
|
||||
Method: endpoint-level (the buttons' own endpoints: `POST /api/stacks/docmost/update`, the backups page's
|
||||
`POST /backup/restore` form with `snapshot_id=helyi`, the second-drive „Teljes visszaállítás" form
|
||||
`POST /backup/tier2/unit-restore`), and the unit read off the disk (`tools/a1.py unit`).
|
||||
|
||||
## Shape 1 — an app-image step (0.95.0 → 0.96.0, PostgreSQL 16 both sides)
|
||||
|
||||
| when (UTC) | unit `compose/` + `image_pins` | unit `db-dumps/docmost-postgres.sql` | restore point offered |
|
||||
|---|---|---|---|
|
||||
| 07:25:57 (the update's own "back up first") | 0.95.0 / 16 | 07:25:43, 0.95.0 schema | — |
|
||||
| 07:26:33 update `done` | 0.95.0 / 16 (unchanged) | 07:25:43 | 07:25:57 |
|
||||
| **07:27:53 the 5-minute status refresh** | **0.96.0 / 16** (`created_at` 07:27:53) | **07:25:43, 0.95.0 schema** | **07:27:53** |
|
||||
|
||||
`S1a-right-after-step1.txt`, `S1b-after-refresh.txt`. **The claim holds:** `CaptureRecoveryUnit`, run by the
|
||||
refresh (`RefreshCache` → `captureAllRecoveryUnits`), rewrote `compose/` and `image_pins` from the stack dir
|
||||
the first time it ran after the pin moved, and the dumps stayed as the backup legs left them. Tier 1's time
|
||||
(`ListRestorePoints` = newest of manifest + dumps) moved to the refresh: **07:27:53 for data from 07:25:43**.
|
||||
|
||||
**Restore in the window (own unit, `helyi`) — `S2-restore-window-step1.txt`:** volumes first (3 tars,
|
||||
including the database's own volume), then the definition from the unit's `compose/` (0.96.0), then the SQL
|
||||
with only the database up. It completed ("3 volume(s) of 3 listed, 1 database(s) of 1 listed"), the seed read
|
||||
back — **and docmost 0.96.0 ran four schema migrations on 0.95.0's data at its first start**
|
||||
(`20260824T211732-page-title-trgm-index` … `20260904T171920-public-spaces`). So the restore performed the
|
||||
update step itself, with no precondition, no safety copy of its own and no undo. Whole here; unguarded always.
|
||||
|
||||
## Shape 2 — an engine step (PostgreSQL 16 → 18, docmost-shaped, with the box's conversion)
|
||||
|
||||
| when (UTC) | unit `compose/` | unit `.sql` | pg volume tar |
|
||||
|---|---|---|---|
|
||||
| 07:29:45 back up first | 0.96.0 / **16**, mount `/var/lib/postgresql/data` | 07:29:45, `Dumped from database version 16.15` | 07:29:47, a 16 datadir at the volume root |
|
||||
| 07:30:39 update `done` (converting 7 s) | unchanged | unchanged | unchanged |
|
||||
| **07:32:53 refresh** | **0.96.0 / 18, mount `/var/lib/postgresql`** | **07:29:45, 16.15** | **07:29:47, 16 datadir** |
|
||||
|
||||
`S3a-right-after-step2.txt`, `S3b-after-refresh-step2.txt`. `conversion_copy` recorded (at 07:30:39).
|
||||
|
||||
**Restore in the window (own unit) — `S4-restore-window-step2.txt`. The claim is TRUE and the outcome is the
|
||||
worst of the three the brief listed: REFUSED by the engine, and the app left DOWN.** The 16 datadir tar was
|
||||
poured into the volume first (the volume is removed and recreated, so the live 18 datadir is GONE at that
|
||||
moment); the unit's 18 definition was then written; the DB-only start of `postgres:18-alpine` refused the
|
||||
old layout (`Error: in 18+, these Docker images are configured to store database data in a format which is
|
||||
compatible with "pg_ctlcluster" … Counter to that, there appears to be PostgreSQL data in:
|
||||
/var/lib/postgresql`) and restarted in a loop; the replay gave up (`waiting for docmost-postgres (postgres)
|
||||
readiness: timeout after 30s`); the full start failed (`docker compose up -d … exit code 1`). The page said
|
||||
the restore failed; docmost stayed `restarting`; the seed did NOT read back (the app's router answered 404).
|
||||
Neither "a fresh empty cluster with the SQL loaded into it" nor "the app comes back whole".
|
||||
|
||||
## A1.3 — the other two copies
|
||||
|
||||
**Second drive (Tier 2) — `S5-restore-tier2-window.txt`.** The mirror had been written by the update's own
|
||||
"back up first" (07:29) and was NOT touched by the refresh: its `compose/` still said 0.96.0 / **16** with
|
||||
the 16 dump and tar. „Teljes visszaállítás" from it brought docmost back **whole at PostgreSQL 16**: 3 volumes,
|
||||
1 database, the seed read back, `PG_VERSION` 16, and the pin moved back to `postgres:16-alpine` (the restore
|
||||
pins to the unit's definition, `09` §5.3). This is option A's shape — by timing, not by design: the mirror
|
||||
is rewritten on the next Tier-2 run from whatever the primary unit then holds, so after the nightly Tier 2
|
||||
it would carry the same mix as the primary.
|
||||
|
||||
**Off-site (Tier 3) — NOT measured live**: 9202 has no off-site target, and the only boxes that have one
|
||||
(9201, demo-felhom) are fenced read-only for this session. **Read from source instead
|
||||
(`offbox_reconstitute.go` `ReconstituteFromOffsite`):** the off-site restore **never writes the definition**
|
||||
— it resolves the database service from the LIVE compose ("reconstitution never overwrites the stack dir"),
|
||||
replays the snapshot's volume tars (the database's raw volume included), then the SQL. The snapshot is taken
|
||||
nightly after its own dump leg, so it is internally coherent — but an update during the day moves the live
|
||||
definition, and until the next night's off-site run **every** off-site restore puts the old data under the
|
||||
new definition. For an engine step that is S4's mechanism exactly, with one difference: this path takes a
|
||||
safety dump and rolls back on a replay failure — but the DB-only start fails before any replay, so the path
|
||||
goes to `restartStack` with the live datadir already replaced. Inferred from source + S4, not observed.
|
||||
|
||||
## Also found
|
||||
|
||||
1. **A restore drops the `conversion_copy` record but not the volume** (`S4`, `S5`: `conversion_copy=None`
|
||||
while `docmost_docmost_postgres_data.pre-update-20260926T073003Z` still exists). `PersistUnitRedeployConfig`
|
||||
builds a fresh `AppConfig`, so the record the hourly release reads is gone and the copy is never released —
|
||||
disk only, never data. Filed.
|
||||
2. **The refresh also re-captures `.felhom.yml`** (6426 B → 3998 B, the applied record of the pinned version,
|
||||
R-669) — expected; it belongs with the definition and is kept with it by the fix.
|
||||
3. **The automatic leg already refuses an app older than the ladder** (`unattended.go` `legCandidate`,
|
||||
`LegSkipOlderThanLadder`); a PERSON's press jumps to the template (`ladder.go` `nextLadderStep` Index −1,
|
||||
measured live by part 5 on vikunja 2.5.0). So the brief's claim "a restore of a version older than the
|
||||
ladder's first entry would then jump" is **true for a person's press and false for the automatic leg**.
|
||||
|
||||
## State left (for Part A's live proof)
|
||||
|
||||
docmost on 9202 at 0.96.0 / PostgreSQL 16, pinned there, seed readable; the orphaned pre-conversion copy
|
||||
volume stays until teardown. The drill repo equals live `main` (`8c3d099` is a revert of `85af330`).
|
||||
@@ -0,0 +1,25 @@
|
||||
seed: True
|
||||
seed reads back through the front door: True
|
||||
# S0-deployed — 2026-09-26T09:25:26+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.95.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.95.0", "digest": "sha256:41c8d777cf23c74e78f94e676aec328b7d7856f48df5e573543dac68d371e37c", "at": "2026-09-26T07:24:50Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:24:50Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:24:50Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=None
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
(absent)
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
(absent)
|
||||
== live
|
||||
compose image: docmost/docmost:0.95.0
|
||||
compose image: postgres:16-alpine
|
||||
compose image: redis:7-alpine
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.95.0 Up 41 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 52 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 52 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 []
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"accepted": true,
|
||||
"http": "202",
|
||||
"phases": [
|
||||
{
|
||||
"t": 0.0,
|
||||
"phase": "backing-up",
|
||||
"label": "Biztons\u00e1gi ment\u00e9s k\u00e9sz\u00fcl a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 14.4,
|
||||
"phase": "safety-dump",
|
||||
"label": "Adatb\u00e1zis pillanatk\u00e9p\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 16.4,
|
||||
"phase": "copying",
|
||||
"label": "Az adatok m\u00e1sol\u00e1sa a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 18.5,
|
||||
"phase": "starting",
|
||||
"label": "Ind\u00edt\u00e1s az \u00faj verzi\u00f3val\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 29.8,
|
||||
"phase": "verifying",
|
||||
"label": "M\u0171k\u00f6d\u00e9s ellen\u0151rz\u00e9se\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 50.3,
|
||||
"phase": "done",
|
||||
"label": "Friss\u00edtve",
|
||||
"updating": false,
|
||||
"error": null,
|
||||
"hold": null
|
||||
}
|
||||
],
|
||||
"duration_s": 50.3,
|
||||
"final_phase": "done",
|
||||
"update_error": null,
|
||||
"hold_reason": null,
|
||||
"state": "running"
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
# S1a-right-after-step1 — 2026-09-26T09:26:37+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:26:33Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:24:50Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:24:50Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up 30 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 40 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 40 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:25:57Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,66 @@
|
||||
# S1b-after-refresh — 2026-09-26T09:28:05+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:26:33Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:24:50Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:24:50Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:27:53.7715190260Z 1514 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up About a minute (healthy)
|
||||
docmost-redis redis:7-alpine Up 2 minutes (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 2 minutes (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:27:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,124 @@
|
||||
# S2-restore-window-step1 — restore pressed at 2026-09-26T07:28:25Z (own unit, snapshot helyi)
|
||||
{"ok": true, "snapshot_id": "helyi", "snapshots": [{"time": "2026-09-26T07:27:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}], "http": "HTTP/2 302", "location": ["location: /backups/restore?flash=flash.restore.started"], "seconds": 30.4, "state_after": "running", "hold_after": null}
|
||||
controller lines:
|
||||
2026/09/26 07:28:26 handlers.go:1707: [WARN] [web] Restore requested (async): stack=docmost, snapshot=helyi from 172.18.0.6:40436
|
||||
2026/09/26 07:28:26 restore_unit.go:383: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
|
||||
2026/09/26 07:28:26 manager.go:1305: [INFO] [stacks] Stopping stack: docmost
|
||||
2026/09/26 07:28:27 manager.go:1314: [INFO] [stacks] Stack docmost stopped successfully (took 1.3s)
|
||||
2026/09/26 07:28:27 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
|
||||
2026/09/26 07:28:28 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
|
||||
2026/09/26 07:28:28 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
|
||||
2026/09/26 07:28:29 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for docmost
|
||||
2026/09/26 07:28:29 deploy.go:708: [INFO] [stacks] Redeploying docmost from recovery unit with 4 env vars
|
||||
2026/09/26 07:28:29 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:28:29 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:28:29 pin.go:93: [INFO] [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:16-alpine, docmost-redis=redis:7-alpine
|
||||
2026/09/26 07:28:29 manager.go:1269: [INFO] [stacks] Starting stack docmost services only: [docmost-postgres]
|
||||
2026/09/26 07:28:29 manager.go:1280: [INFO] [stacks] Stack docmost services [docmost-postgres] started (took 0.4s)
|
||||
2026/09/26 07:28:29 restore_db.go:87: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
||||
2026/09/26 07:28:31 dbdump.go:801: [INFO] [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)
|
||||
2026/09/26 07:28:31 restore_db.go:97: [INFO] [backup] Restore docmost: replayed 1 DB dump(s)
|
||||
2026/09/26 07:28:31 manager.go:1216: [INFO] [stacks] Starting stack: docmost
|
||||
2026/09/26 07:28:42 manager.go:1231: [INFO] [stacks] Stack docmost started successfully (took 11.1s)
|
||||
2026/09/26 07:28:42 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:28:42 installed.go:420: [INFO] [stacks] installed-images docmost: recorded 3 service(s) (docmost=docmost/docmost:0.96.0 (sha256:b56947fcfd08…), docmost-postgres=postgres:16-alpine (sha256:721873c34ceb…), docmost-redis=redis:7-alpine (sha256:858f009f9709…))
|
||||
2026/09/26 07:28:46 manager.go:1562: [INFO] [stacks] Stack docmost post-start status:
|
||||
2026/09/26 07:28:46 manager.go:1565: [INFO] [stacks] docmost docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a running Up 3 seconds (health: starting)
|
||||
2026/09/26 07:28:46 manager.go:1565: [INFO] [stacks] docmost-postgres postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea running Up 16 seconds (healthy)
|
||||
2026/09/26 07:28:46 manager.go:1565: [INFO] [stacks] docmost-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy)
|
||||
2026/09/26 07:28:56 restore_unit.go:468: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
|
||||
2026/09/26 07:28:56 handlers.go:1719: [INFO] [web] Restore completed (async): stack=docmost in 29.878718951s (volumes 3/3, dbs 1/1)
|
||||
|
||||
docmost-postgres log (tail):
|
||||
|
||||
PostgreSQL Database directory appears to contain a database; Skipping initialization
|
||||
|
||||
2026-09-26 09:28:29.836 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
|
||||
2026-09-26 09:28:29.837 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
|
||||
2026-09-26 09:28:29.837 CEST [1] LOG: listening on IPv6 address "::", port 5432
|
||||
2026-09-26 09:28:29.846 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
|
||||
2026-09-26 09:28:29.855 CEST [30] LOG: database system was shut down at 2026-09-26 09:25:44 CEST
|
||||
2026-09-26 09:28:29.865 CEST [1] LOG: database system is ready to accept connections
|
||||
|
||||
docmost log (tail):
|
||||
$ pnpm --filter ./apps/server run start:prod
|
||||
$ cross-env NODE_ENV=production node dist/main
|
||||
(node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided.
|
||||
(Use `node --trace-warnings ...` to show where the warning was created)
|
||||
{"level":"info","time":"2026-09-26T07:28:51.324Z","pid":45,"hostname":"66cf0179fc93","context":"RedisModule","msg":"default: the connection was successfully established"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.574Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseModule","msg":"Establishing database connection"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.597Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseModule","msg":"Database connection successful"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260824T211732-page-title-trgm-index\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260825T022612-oauth\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260902T121326-siem-destinations\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260904T171920-public-spaces\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:52.025Z","pid":45,"hostname":"66cf0179fc93","context":"NestApplication","msg":"Nest application successfully started"}
|
||||
{"level":"info","time":"2026-09-26T07:28:52.040Z","pid":45,"hostname":"66cf0179fc93","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://c-docs.enkisfelhom.hu"}
|
||||
|
||||
seed reads back through the front door: True
|
||||
# S2-restore-window-step1 — 2026-09-26T09:29:20+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:28:42Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:27:53.7715190260Z 1514 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up 43 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 53 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 56 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:27:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"accepted": true,
|
||||
"http": "202",
|
||||
"phases": [
|
||||
{
|
||||
"t": 0.0,
|
||||
"phase": "backing-up",
|
||||
"label": "Biztons\u00e1gi ment\u00e9s k\u00e9sz\u00fcl a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 15.4,
|
||||
"phase": "safety-dump",
|
||||
"label": "Adatb\u00e1zis pillanatk\u00e9p\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 17.5,
|
||||
"phase": "pulling",
|
||||
"label": "\u00daj verzi\u00f3 let\u00f6lt\u00e9se\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 18.5,
|
||||
"phase": "copying",
|
||||
"label": "Az adatok m\u00e1sol\u00e1sa a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 20.5,
|
||||
"phase": "converting",
|
||||
"label": "Adatb\u00e1zis \u00e1talak\u00edt\u00e1sa",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 27.7,
|
||||
"phase": "starting",
|
||||
"label": "Ind\u00edt\u00e1s az \u00faj verzi\u00f3val\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 39.0,
|
||||
"phase": "verifying",
|
||||
"label": "M\u0171k\u00f6d\u00e9s ellen\u0151rz\u00e9se\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 54.3,
|
||||
"phase": "done",
|
||||
"label": "Friss\u00edtve",
|
||||
"updating": false,
|
||||
"error": null,
|
||||
"hold": null
|
||||
}
|
||||
],
|
||||
"duration_s": 54.4,
|
||||
"final_phase": "done",
|
||||
"update_error": null,
|
||||
"hold_reason": null,
|
||||
"state": "running"
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
# S3a-right-after-step2 — 2026-09-26T09:30:39+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:18-alpine", "digest": "sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "at": "2026-09-26T07:30:39Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy={'volume': 'docmost_docmost_postgres_data', 'copy': 'docmost_docmost_postgres_data.pre-update-20260926T073003Z', 'at': '2026-09-26T07:30:39Z', 'from': 16, 'to': 18}
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/18/docker PG_VERSION=18
|
||||
docmost docmost/docmost:0.96.0 Up 21 seconds (healthy)
|
||||
docmost-postgres postgres:18-alpine Up 37 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 32 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:30:00Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,145 @@
|
||||
# S3b-after-refresh-step2 — 2026-09-26T09:41:18+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:18-alpine", "digest": "sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "at": "2026-09-26T07:30:39Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy={'volume': 'docmost_docmost_postgres_data', 'copy': 'docmost_docmost_postgres_data.pre-update-20260926T073003Z', 'at': '2026-09-26T07:30:39Z', 'from': 16, 'to': 18}
|
||||
state=running phase=done
|
||||
# S3b-after-refresh-step2 — 2026-09-26T09:41:18+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:18-alpine", "digest": "sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "at": "2026-09-26T07:30:39Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy={'volume': 'docmost_docmost_postgres_data', 'copy': 'docmost_docmost_postgres_data.pre-update-20260926T073003Z', 'at': '2026-09-26T07:30:39Z', 'from': 16, 'to': 18}
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/18/docker PG_VERSION=18
|
||||
docmost docmost/docmost:0.96.0 Up 10 minutes (healthy)
|
||||
docmost-postgres postgres:18-alpine Up 11 minutes (healthy)
|
||||
docmost-redis redis:7-alpine Up 11 minutes (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/18/docker PG_VERSION=18
|
||||
docmost docmost/docmost:0.96.0 Up 10 minutes (healthy)
|
||||
docmost-postgres postgres:18-alpine Up 11 minutes (healthy)
|
||||
docmost-redis redis:7-alpine Up 11 minutes (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
2026/09/26 07:25:43 update_guard.go:248: [DEBUG] [backup] update precondition for docmost: no Tier-2 copy (nincs másodlagos fájlmásolat ehhez az alkalmazáshoz)
|
||||
2026/09/26 07:25:57 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
|
||||
2026/09/26 07:25:57 update.go:831: [INFO] [stacks] update docmost: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-26T07:25:57Z
|
||||
2026/09/26 07:27:53 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
|
||||
2026/09/26 07:29:59 update.go:831: [INFO] [stacks] update docmost: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-26T07:29:59Z
|
||||
2026/09/26 07:30:39 update.go:1015: [INFO] [stacks] update docmost: KEEPING the pre-conversion datadir copy docmost_docmost_postgres_data.pre-update-20260926T073003Z (PostgreSQL 16) until a backup of the converted app is proven
|
||||
2026/09/26 07:32:53 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
|
||||
@@ -0,0 +1,244 @@
|
||||
# S4-restore-window-step2 — restore pressed at 2026-09-26T07:41:33Z (own unit, snapshot helyi)
|
||||
{"ok": true, "snapshot_id": "helyi", "snapshots": [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}], "http": "HTTP/2 302", "location": ["location: /backups/restore?flash=flash.restore.started"], "seconds": 36.5, "state_after": "restarting", "hold_after": null}
|
||||
controller lines:
|
||||
2026/09/26 07:41:34 handlers.go:1707: [WARN] [web] Restore requested (async): stack=docmost, snapshot=helyi from 172.18.0.6:42900
|
||||
2026/09/26 07:41:34 restore_unit.go:383: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
|
||||
2026/09/26 07:41:34 manager.go:1305: [INFO] [stacks] Stopping stack: docmost
|
||||
2026/09/26 07:41:35 manager.go:1314: [INFO] [stacks] Stack docmost stopped successfully (took 0.8s)
|
||||
2026/09/26 07:41:35 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
|
||||
2026/09/26 07:41:36 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
|
||||
2026/09/26 07:41:36 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
|
||||
2026/09/26 07:41:36 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for docmost
|
||||
2026/09/26 07:41:36 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:41:36 deploy.go:708: [INFO] [stacks] Redeploying docmost from recovery unit with 4 env vars
|
||||
2026/09/26 07:41:36 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:41:36 pin.go:93: [INFO] [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:18-alpine, docmost-redis=redis:7-alpine
|
||||
2026/09/26 07:41:36 manager.go:1269: [INFO] [stacks] Starting stack docmost services only: [docmost-postgres]
|
||||
2026/09/26 07:41:37 manager.go:1280: [INFO] [stacks] Stack docmost services [docmost-postgres] started (took 0.4s)
|
||||
2026/09/26 07:41:37 restore_db.go:87: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
||||
2026/09/26 07:41:53 main.go:3724: [INFO] [deadapp] docmost crossed the crash_loop threshold but a backup or restore is running (single-flight held) — NOT stopped
|
||||
2026/09/26 07:42:08 restore_unit.go:450: [ERROR] [backup] DB re-import for docmost: importing postgres dump for docmost: waiting for docmost-postgres (postgres) readiness: timeout after 30s
|
||||
2026/09/26 07:42:08 manager.go:1216: [INFO] [stacks] Starting stack: docmost
|
||||
2026/09/26 07:42:09 manager.go:1491: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/docmost) — exit code 1 (took 0.9s)
|
||||
2026/09/26 07:42:09 manager.go:1497: [ERROR] [stacks] stderr: Container docmost-redis Creating
|
||||
Container docmost-redis Created
|
||||
Container docmost Creating
|
||||
Container docmost Created
|
||||
Container docmost-postgres Starting
|
||||
Container docmost-redis Starting
|
||||
Container docmost-postgres Started
|
||||
Container docmost-redis Started
|
||||
Container docmost-postgres Waiting
|
||||
Container docmost-redis Waiting
|
||||
Container docmost-postgres Error dependency docmost-postgres failed to start
|
||||
dependency failed to start: container docmost-postgres is unhealthy
|
||||
2026/09/26 07:42:09 manager.go:1227: [ERROR] [stacks] Stack docmost start failed after 0.9s: exit code 1
|
||||
stderr: Container docmost-redis Creating
|
||||
Container docmost-redis Created
|
||||
Container docmost Creating
|
||||
Container docmost Created
|
||||
Container docmost-postgres Starting
|
||||
Container docmost-redis Starting
|
||||
Container docmost-postgres Started
|
||||
Container docmost-redis Started
|
||||
Container docmost-postgres Waiting
|
||||
Container docmost-redis Waiting
|
||||
Container docmost-postgres Error dependency docmost-postgres failed to start
|
||||
dependency failed to start: container docmost-postgres is unhealthy
|
||||
2026/09/26 07:42:09 handlers.go:1715: [ERROR] [web] Restore failed (async): stack=docmost: starting docmost after restore from unit: starting stack docmost: exit code 1
|
||||
stderr: Container docmost-redis Creating
|
||||
Container docmost-redis Created
|
||||
Container docmost Creating
|
||||
Container docmost Created
|
||||
Container docmost-postgres Starting
|
||||
Container docmost-redis Starting
|
||||
Container docmost-postgres Started
|
||||
Container docmost-redis Started
|
||||
Container docmost-postgres Waiting
|
||||
Container docmost-redis Waiting
|
||||
Container docmost-postgres Error dependency docmost-postgres failed to start
|
||||
dependency failed to start: container docmost-postgres is unhealthy
|
||||
|
||||
docmost-postgres log (tail):
|
||||
major-version-specific directory names). This better reflects how
|
||||
PostgreSQL itself works, and how upgrades are to be performed.
|
||||
|
||||
See also https://github.com/docker-library/postgres/pull/1259
|
||||
|
||||
Counter to that, there appears to be PostgreSQL data in:
|
||||
/var/lib/postgresql
|
||||
|
||||
This is usually the result of upgrading the Docker image without
|
||||
upgrading the underlying database using "pg_upgrade" (which requires both
|
||||
versions).
|
||||
|
||||
The suggested container configuration for 18+ is to place a single mount
|
||||
at /var/lib/postgresql which will then place PostgreSQL data in a
|
||||
subdirectory, allowing usage of "pg_upgrade --link" without mount point
|
||||
boundary issues.
|
||||
|
||||
See https://github.com/docker-library/postgres/issues/37 for a (long)
|
||||
discussion around this process, and suggestions for how to do so.
|
||||
Error: in 18+, these Docker images are configured to store database data in a
|
||||
format which is compatible with "pg_ctlcluster" (specifically, using
|
||||
major-version-specific directory names). This better reflects how
|
||||
PostgreSQL itself works, and how upgrades are to be performed.
|
||||
|
||||
See also https://github.com/docker-library/postgres/pull/1259
|
||||
|
||||
Counter to that, there appears to be PostgreSQL data in:
|
||||
/var/lib/postgresql
|
||||
|
||||
This is usually the result of upgrading the Docker image without
|
||||
upgrading the underlying database using "pg_upgrade" (which requires both
|
||||
versions).
|
||||
|
||||
The suggested container configuration for 18+ is to place a single mount
|
||||
at /var/lib/postgresql which will then place PostgreSQL data in a
|
||||
subdirectory, allowing usage of "pg_upgrade --link" without mount point
|
||||
boundary issues.
|
||||
|
||||
See https://github.com/docker-library/postgres/issues/37 for a (long)
|
||||
discussion around this process, and suggestions for how to do so.
|
||||
|
||||
docmost log (tail):
|
||||
|
||||
seed reads back through the front door: False
|
||||
# S4-restore-window-step2 — 2026-09-26T09:42:34+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed=null
|
||||
conversion_copy=None
|
||||
state=degraded phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
Error response from daemon: Container d38697606a1e60ebffa27adc7e262b04d6690fd593c2fc02035d6bcb1fcb83e5 is restarting, wait until the container is running
|
||||
docmost docmost/docmost:0.96.0 Created
|
||||
docmost-redis redis:7-alpine Up 30 seconds (healthy)
|
||||
docmost-postgres postgres:18-alpine Restarting (1) 8 seconds ago
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18 PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
# S4-restore-window-step2 — 2026-09-26T09:42:50+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed=null
|
||||
conversion_copy=None
|
||||
state=degraded phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
Error response from daemon: Container d38697606a1e60ebffa27adc7e262b04d6690fd593c2fc02035d6bcb1fcb83e5 is restarting, wait until the container is running
|
||||
docmost docmost/docmost:0.96.0 Created
|
||||
docmost-redis redis:7-alpine Up 46 seconds (healthy)
|
||||
docmost-postgres postgres:18-alpine Restarting (1) 24 seconds ago
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18 PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,131 @@
|
||||
# S5-restore-tier2-window — restore pressed at 2026-09-26T07:42:58Z (second-drive unit restore)
|
||||
POST /backup/tier2/unit-restore -> HTTP/2 302 ['location: /backups/apps?flash=flash.restore.full_started']
|
||||
restore-status at end: {"running": false, "op": "tier2-unit-restore", "stack": "docmost", "started_at": "2026-09-26T07:42:59.597470183Z", "last": {"op": "tier2-unit-restore", "stack": "docmost", "ok": true, "message": "A(z) docmost: 3 adatk\u00f6tet \u00e9s az adatb\u00e1zis vissza\u00e1ll\u00edtva \u2014 az alkalmaz\u00e1s \u00fajraindult. A vissza\u00e1ll\u00edt\u00e1s forr\u00e1sa a m\u00e1sodik meghajt\u00f3n l\u00e9v\u0151 m\u00e1solat volt (2026-09-26 09:29).", "finished_at": "2026-09-26T07:43:28.98815976Z"}, "last_recent": true}
|
||||
controller lines:
|
||||
2026/09/26 07:42:59 handlers.go:2106: [WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=docmost from 172.18.0.6:42900
|
||||
2026/09/26 07:42:59 tier2_restore.go:206: [WARN] [backup] Tier-2 UNIT restore for docmost from the secondary mirror /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit — this OVERWRITES live app data
|
||||
2026/09/26 07:42:59 restore_unit.go:383: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
|
||||
2026/09/26 07:42:59 manager.go:1305: [INFO] [stacks] Stopping stack: docmost
|
||||
2026/09/26 07:43:00 manager.go:1314: [INFO] [stacks] Stack docmost stopped successfully (took 0.4s)
|
||||
2026/09/26 07:43:00 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
|
||||
2026/09/26 07:43:01 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
|
||||
2026/09/26 07:43:01 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
|
||||
2026/09/26 07:43:01 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for docmost
|
||||
2026/09/26 07:43:01 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:43:01 deploy.go:708: [INFO] [stacks] Redeploying docmost from recovery unit with 4 env vars
|
||||
2026/09/26 07:43:01 pin.go:93: [INFO] [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:16-alpine, docmost-redis=redis:7-alpine
|
||||
2026/09/26 07:43:01 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:43:01 manager.go:1269: [INFO] [stacks] Starting stack docmost services only: [docmost-postgres]
|
||||
2026/09/26 07:43:02 manager.go:1280: [INFO] [stacks] Stack docmost services [docmost-postgres] started (took 0.4s)
|
||||
2026/09/26 07:43:02 restore_db.go:87: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
||||
2026/09/26 07:43:04 dbdump.go:801: [INFO] [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)
|
||||
2026/09/26 07:43:04 restore_db.go:97: [INFO] [backup] Restore docmost: replayed 1 DB dump(s)
|
||||
2026/09/26 07:43:04 manager.go:1216: [INFO] [stacks] Starting stack: docmost
|
||||
2026/09/26 07:43:15 manager.go:1231: [INFO] [stacks] Stack docmost started successfully (took 11.1s)
|
||||
2026/09/26 07:43:15 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:43:15 installed.go:420: [INFO] [stacks] installed-images docmost: recorded 3 service(s) (docmost=docmost/docmost:0.96.0 (sha256:b56947fcfd08…), docmost-postgres=postgres:16-alpine (sha256:721873c34ceb…), docmost-redis=redis:7-alpine (sha256:858f009f9709…))
|
||||
2026/09/26 07:43:19 manager.go:1562: [INFO] [stacks] Stack docmost post-start status:
|
||||
2026/09/26 07:43:19 manager.go:1565: [INFO] [stacks] docmost docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a running Up 3 seconds (health: starting)
|
||||
2026/09/26 07:43:19 manager.go:1565: [INFO] [stacks] docmost-postgres postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea running Up 16 seconds (healthy)
|
||||
2026/09/26 07:43:19 manager.go:1565: [INFO] [stacks] docmost-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy)
|
||||
2026/09/26 07:43:28 restore_unit.go:468: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
|
||||
2026/09/26 07:43:28 tier2_restore.go:251: [INFO] [backup] docmost: primary unit already carries data — left untouched (R-403 never overwrites a richer package with a poorer one)
|
||||
2026/09/26 07:43:28 handlers.go:2116: [INFO] [web] Tier-2 unit restore completed (async): stack=docmost in 29.390287688s (volumes 3/3, dbs 1/1)
|
||||
|
||||
docmost-postgres log (tail):
|
||||
|
||||
PostgreSQL Database directory appears to contain a database; Skipping initialization
|
||||
|
||||
2026-09-26 09:43:02.469 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
|
||||
2026-09-26 09:43:02.469 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
|
||||
2026-09-26 09:43:02.469 CEST [1] LOG: listening on IPv6 address "::", port 5432
|
||||
2026-09-26 09:43:02.475 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
|
||||
2026-09-26 09:43:02.484 CEST [29] LOG: database system was shut down at 2026-09-26 09:29:46 CEST
|
||||
2026-09-26 09:43:02.495 CEST [1] LOG: database system is ready to accept connections
|
||||
|
||||
docmost log (tail):
|
||||
$ pnpm --filter ./apps/server run start:prod
|
||||
$ cross-env NODE_ENV=production node dist/main
|
||||
(node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided.
|
||||
(Use `node --trace-warnings ...` to show where the warning was created)
|
||||
{"level":"info","time":"2026-09-26T07:43:22.964Z","pid":45,"hostname":"905c49590089","context":"RedisModule","msg":"default: the connection was successfully established"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.210Z","pid":45,"hostname":"905c49590089","context":"DatabaseModule","msg":"Establishing database connection"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.238Z","pid":45,"hostname":"905c49590089","context":"DatabaseModule","msg":"Database connection successful"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.399Z","pid":45,"hostname":"905c49590089","context":"DatabaseMigrationService","msg":"No pending database migrations"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.437Z","pid":45,"hostname":"905c49590089","context":"NestApplication","msg":"Nest application successfully started"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.448Z","pid":45,"hostname":"905c49590089","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://c-docs.enkisfelhom.hu"}
|
||||
|
||||
seed reads back through the front door: True
|
||||
# S5-restore-tier2-window — 2026-09-26T09:43:50+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:43:15Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:43:15Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:43:15Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up 40 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 50 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 53 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
docmost_docmost_postgres_data
|
||||
docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
docmost_docmost_redis_data
|
||||
docmost_docmost_storage
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env python3
|
||||
"""a1.py <cmd> — Part A1's spike on guest 9202 (drill catalog): the window between an update and the next
|
||||
backup, measured, then a restore pressed inside that window. EVIDENCE, NOT PRODUCT: it presses only what a
|
||||
person presses (deploy, Update, the backups page's restore) and reads the unit off the disk.
|
||||
|
||||
unit <label> the unit's manifest + compose images + dump headers + mtimes, and the live facts
|
||||
restore <label> [tier2] press the restore (own unit „helyi", or the second-drive unit restore) and read back
|
||||
seed / read <label> docmost's seed through its own front door (fixtures.py)
|
||||
"""
|
||||
import json, os, sys, time
|
||||
import walk as w, fixtures
|
||||
|
||||
APP, SUB = "docmost", "c-docs"
|
||||
OUT_DIR = f"{w.EV}/A1"
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
TOK = w.SC + "/seed-tokens.json"
|
||||
|
||||
|
||||
def out(label):
|
||||
return open(f"{OUT_DIR}/{label}.txt", "a", buffering=1)
|
||||
|
||||
|
||||
UNIT_SH = r"""
|
||||
for U in /mnt/sys_drive/felhom-data/backups/primary/docmost /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit; do
|
||||
echo "== unit $U"
|
||||
[ -d "$U" ] || { echo "(absent)"; continue; }
|
||||
python3 - "$U" <<'PY'
|
||||
import json, sys, os
|
||||
u = sys.argv[1]
|
||||
try:
|
||||
m = json.load(open(u + "/manifest.json"))
|
||||
for k in ("controller_version", "created_at", "image_pins", "db_dumps", "volume_dumps", "dumps_at", "offsite_run_id", "data"):
|
||||
if k in m:
|
||||
print(f" manifest.{k} = {json.dumps(m[k])}")
|
||||
except Exception as e:
|
||||
print(" manifest unreadable:", e)
|
||||
PY
|
||||
echo " compose/ images:"; grep -E '^\s+image:' "$U/compose/docker-compose.yml" 2>/dev/null | sed 's/^ */ /'
|
||||
echo " compose/ postgres mount:"; grep -E 'docmost_postgres_data:' "$U/compose/docker-compose.yml" 2>/dev/null | sed 's/^ */ /'
|
||||
echo " mtimes (UTC):"; find "$U" -maxdepth 2 -type f -printf ' %TY-%Tm-%TdT%TH:%TM:%TSZ %10s %P\n' | sort -k3 | grep -v pre-restore
|
||||
for f in "$U"/db-dumps/*.sql; do [ -f "$f" ] && echo " $(basename $f): $(grep -m1 -o 'Dumped from database version [0-9.]*' "$f")"; done
|
||||
done
|
||||
echo "== live"
|
||||
grep -E '^\s+image:' /opt/docker/stacks/docmost/docker-compose.yml 2>/dev/null | sed 's/^ */ compose /'
|
||||
docker exec docmost-postgres sh -c 'echo " PGDATA=$PGDATA PG_VERSION=$(cat $PGDATA/PG_VERSION 2>&1)"' 2>&1
|
||||
docker ps -a --filter label=com.docker.compose.project=docmost --format ' {{.Names}} {{.Image}} {{.Status}}'
|
||||
docker volume ls -q | grep -i docmost | sed 's/^/ volume /'
|
||||
V=$(docker volume inspect docmost_docmost_postgres_data --format '{{.Mountpoint}}' 2>/dev/null); [ -n "$V" ] && { echo " pg volume root:"; ls "$V" | head -30 | tr '\n' ' '; echo; }
|
||||
"""
|
||||
|
||||
|
||||
def unit(label):
|
||||
o = out(label)
|
||||
st = w.stack(APP)
|
||||
ac = st.get("app_config") or {}
|
||||
o.write(f"# {label} — {time.strftime('%Y-%m-%dT%H:%M:%S%z')}; controller {w.guest('cat /etc/felhom-controller-image').strip()}\n")
|
||||
o.write(f"pinned={ac.get('pinned_images')}\ninstalled={json.dumps(ac.get('installed_images'))}\n"
|
||||
f"conversion_copy={ac.get('conversion_copy')}\nstate={st.get('state')} phase={st.get('update_phase')}\n")
|
||||
o.write(w.guest(UNIT_SH, timeout=120))
|
||||
code, d = w.ctl("GET", f"/api/backup/snapshots?stack={APP}")
|
||||
o.write(f"restore points offered (GET /api/backup/snapshots): {code} {json.dumps(d.get('data') if isinstance(d, dict) else d)[:600]}\n")
|
||||
o.close()
|
||||
print(open(f"{OUT_DIR}/{label}.txt").read()[-4000:])
|
||||
|
||||
|
||||
def seed(label):
|
||||
toks = json.load(open(TOK)) if os.path.exists(TOK) else {}
|
||||
toks[APP] = fixtures.FIXTURES[APP].seed(w, SUB, w.say)
|
||||
json.dump(toks, open(TOK, "w"), default=str); os.chmod(TOK, 0o600)
|
||||
out(label).write(f"seed: {toks[APP] is not None}\n")
|
||||
|
||||
|
||||
def read(label):
|
||||
toks = json.load(open(TOK))
|
||||
ok = fixtures.FIXTURES[APP].verify(w, SUB, toks[APP], w.say)
|
||||
out(label).write(f"seed reads back through the front door: {ok}\n")
|
||||
return ok
|
||||
|
||||
|
||||
def restore(label, tier2=False):
|
||||
o = out(label)
|
||||
since = w.guest("date -u +%Y-%m-%dT%H:%M:%SZ").strip()
|
||||
o.write(f"# {label} — restore pressed at {since} ({'second-drive unit restore' if tier2 else 'own unit, snapshot helyi'})\n")
|
||||
if tier2:
|
||||
# The „Teljes visszaállítás" button: a FORM post, like the own-unit restore (web/server.go).
|
||||
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
|
||||
csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip()
|
||||
r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}",
|
||||
"-X", "POST", "--data-urlencode", f"_csrf={csrf}", "--data-urlencode", f"stack_name={APP}",
|
||||
f"{w.BASE}/backup/tier2/unit-restore"], timeout=180)
|
||||
loc = [l for l in (r.stdout or "").split("\n") if l.lower().startswith("location:")]
|
||||
o.write(f"POST /backup/tier2/unit-restore -> {(r.stdout or '').split(chr(10))[0].strip()} {loc[:1]}\n")
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 900:
|
||||
c2, s = w.ctl("GET", "/api/backup/restore-status")
|
||||
dd = (s.get("data") or {}) if isinstance(s, dict) else {}
|
||||
if not dd.get("running") and time.time() - t0 > 5:
|
||||
break
|
||||
time.sleep(3)
|
||||
o.write(f"restore-status at end: {json.dumps(dd)[:600]}\n")
|
||||
else:
|
||||
r = w.restore(APP, "helyi")
|
||||
o.write(json.dumps({k: v for k, v in r.items() if k != 'observables_after'}, default=str)[:1500] + "\n")
|
||||
time.sleep(10)
|
||||
o.write("controller lines:\n" + w.guest(
|
||||
f"docker logs --since {since} felhom-controller 2>&1 | grep -iE 'docmost|restor|replay|recreat|REFUSED' | grep -v DEBUG | cut -c1-420") + "\n")
|
||||
o.write("docmost-postgres log (tail):\n" + w.guest("docker logs --tail 40 docmost-postgres 2>&1 | cut -c1-300") + "\n")
|
||||
o.write("docmost log (tail):\n" + w.guest("docker logs --tail 25 docmost 2>&1 | cut -c1-300") + "\n")
|
||||
o.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
cmd, label = sys.argv[1], sys.argv[2]
|
||||
w.login()
|
||||
{"unit": unit, "seed": seed, "read": read,
|
||||
"restore": lambda l: restore(l, tier2=len(sys.argv) > 3 and sys.argv[3] == "tier2")}[cmd](label)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Point guest 9202 at the drill catalog (and a 90 s health timeout), or restore the saved config.
|
||||
|
||||
`09` §6.5: `git.repo_url` alone is INERT (R-615) — the cache dir must go too. The saved copy is
|
||||
`controller.yaml.pre-vt0926` (NOT the older `.pre-28`, which a restore must never pick up).
|
||||
"""
|
||||
import re, sys, io
|
||||
sys.path.insert(0, '.')
|
||||
import walk as w
|
||||
|
||||
VOL = "/var/lib/docker/volumes/felhom-controller-data/_data"
|
||||
DRILL_REPO = "https://gitea.dooplex.hu/admin/app-catalog-drill.git"
|
||||
|
||||
|
||||
def creds():
|
||||
for l in io.open("/home/kisfenyo/.git-credentials").read().strip().split("\n"):
|
||||
m = re.match(r'https://(admin):([^@]+)@gitea\.dooplex\.hu', l)
|
||||
if m:
|
||||
return m.group(1), m.group(2)
|
||||
raise SystemExit("no admin credential")
|
||||
|
||||
|
||||
def to_drill():
|
||||
u, t = creds()
|
||||
print(w.guest(f"""
|
||||
set -e
|
||||
test -f {VOL}/controller.yaml.pre-vt0926 || cp -p {VOL}/controller.yaml {VOL}/controller.yaml.pre-vt0926
|
||||
python3 - <<'PY'
|
||||
import re
|
||||
p = "{VOL}/controller.yaml"
|
||||
s = open(p).read()
|
||||
s = re.sub(r'(^\\s+repo_url: ).*$', r'\\g<1>{DRILL_REPO}', s, count=1, flags=re.M)
|
||||
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+token: ).*$', r'\\g<1>"{t}"', s, count=1, flags=re.M)
|
||||
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+username: ).*$', r'\\g<1>"{u}"', s, count=1, flags=re.M)
|
||||
if not re.search(r'^update:', s, re.M):
|
||||
s += "update:\\n health_timeout: 90s\\n"
|
||||
open(p, "w").write(s)
|
||||
PY
|
||||
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
|
||||
docker restart felhom-controller >/dev/null
|
||||
sleep 15
|
||||
grep -A6 '^git:' {VOL}/controller.yaml | sed 's/token:.*/token: <redacted>/'
|
||||
grep -A2 '^update:' {VOL}/controller.yaml
|
||||
"""))
|
||||
|
||||
|
||||
def restore():
|
||||
print(w.guest(f"""
|
||||
set -e
|
||||
cp -p {VOL}/controller.yaml.pre-vt0926 {VOL}/controller.yaml
|
||||
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
|
||||
docker restart felhom-controller >/dev/null
|
||||
sleep 15
|
||||
grep -A6 '^git:' {VOL}/controller.yaml | sed 's/token:.*/token: <redacted>/'
|
||||
grep -c '^update:' {VOL}/controller.yaml || true
|
||||
"""))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
to_drill() if sys.argv[1] == "drill" else restore()
|
||||
@@ -0,0 +1,24 @@
|
||||
"""seed | read — the four drill apps' data through their OWN front doors (fixtures.py, R-156). Tokens are kept in the
|
||||
scratchpad (they can carry generated credentials); the evidence gets only True/False per app."""
|
||||
import json, sys, os
|
||||
import walk as w, fixtures
|
||||
SUBS = {"wishlist": "c-wish", "navidrome": "c-navi", "romm": "c-romm", "vikunja": "c-vik"}
|
||||
TOK = w.SC + "/seed-tokens.json"
|
||||
mode, label = sys.argv[1], sys.argv[2]
|
||||
w.login()
|
||||
res = {}
|
||||
if mode == "seed":
|
||||
toks = {}
|
||||
for a, sub in SUBS.items():
|
||||
toks[a] = fixtures.FIXTURES[a].seed(w, sub, w.say)
|
||||
res[a] = toks[a] is not None
|
||||
json.dump(toks, open(TOK, "w"), default=str); os.chmod(TOK, 0o600)
|
||||
else:
|
||||
toks = json.load(open(TOK))
|
||||
for a, sub in SUBS.items():
|
||||
if len(sys.argv) > 3 and a not in sys.argv[3].split(","):
|
||||
continue
|
||||
res[a] = bool(toks.get(a)) and fixtures.FIXTURES[a].verify(w, sub, toks[a], w.say)
|
||||
line = f"{label}: {mode} " + " ".join(f"{a}={v}" for a, v in res.items())
|
||||
w.say(line)
|
||||
open(w.EV + "/C/data-readback.txt", "a").write(line + "\n")
|
||||
@@ -0,0 +1,19 @@
|
||||
"""sr.py seed|read <label> app=sub[,app=sub] — seed or read back through each app's OWN front door (fixtures.py).
|
||||
Tokens stay in the scratchpad; the evidence gets True/False only."""
|
||||
import json, sys, os
|
||||
import walk as w, fixtures
|
||||
mode, label = sys.argv[1], sys.argv[2]
|
||||
SUBS = dict(x.split("=") for x in sys.argv[3].split(","))
|
||||
TOK = w.SC + "/seed-tokens.json"
|
||||
w.login()
|
||||
toks = json.load(open(TOK)) if os.path.exists(TOK) else {}
|
||||
res = {}
|
||||
for a, sub in SUBS.items():
|
||||
if mode == "seed":
|
||||
toks[a] = fixtures.FIXTURES[a].seed(w, sub, w.say); res[a] = toks[a] is not None
|
||||
else:
|
||||
res[a] = bool(toks.get(a)) and fixtures.FIXTURES[a].verify(w, sub, toks[a], w.say)
|
||||
if mode == "seed":
|
||||
json.dump(toks, open(TOK, "w"), default=str); os.chmod(TOK, 0o600)
|
||||
line = f"{label}: {mode} " + " ".join(f"{a}={v}" for a, v in res.items())
|
||||
w.say(line); open(w.EV + "/data-readback.txt", "a").write(line + "\n")
|
||||
@@ -0,0 +1,507 @@
|
||||
#!/usr/bin/env python3
|
||||
"""walk.py — ONE app's full update walk on guest 9202, through the product's own endpoints.
|
||||
|
||||
EVIDENCE, NOT PRODUCT. It presses exactly the buttons a person presses:
|
||||
POST /api/stacks/<n>/deploy · POST /api/sync · POST /api/stacks/rescan
|
||||
POST /api/stacks/<n>/update · POST /api/stacks/<n>/remove
|
||||
and reads GET /api/stacks/<n>. No controller code exists for it.
|
||||
|
||||
The walk, per `09` §6.4 and the update-night brief §4:
|
||||
1 deploy from the DRILL catalog at the LIVE pin
|
||||
2 seed through the app's OWN front door (R-156: never a volume, never SQL)
|
||||
3 read the seed back <- control C1; a fixture that cannot prove itself proves nothing
|
||||
4 „Mentés most"
|
||||
5 commit the real one-step bump to the DRILL repo, sync, rescan, read the badge in BOTH languages
|
||||
6 press the guarded Update, record every phase with timestamps
|
||||
7 read the seed back through the front door
|
||||
8 the four version observables side by side
|
||||
9 write the verdict record in `09`'s JSON shape
|
||||
|
||||
`inconclusive` is a first-class verdict and is NEVER collapsed into `failed`.
|
||||
"""
|
||||
import argparse, json, os, re, subprocess, sys, time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
SC = os.environ.get('SC', '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/a061fcea-2c08-478c-9fd6-991dc8e748f0/scratchpad')
|
||||
EV = '/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/version-travel-2026-09-26'
|
||||
DRILL = "/mnt/5_hdd/felhom.eu/drill/app-catalog-drill"
|
||||
# GUEST=9201 selects demo-hp's hub-enabled guest (the mail proof); default 9202, the scratch guest.
|
||||
GUEST = os.environ.get("GUEST", "9202")
|
||||
BASE = os.environ.get("BASE") or {"9202": "https://192.168.0.114", "9201": "https://192.168.0.155"}[GUEST]
|
||||
DOMAIN = os.environ.get("DOMAIN", "enkisfelhom.hu")
|
||||
HOSTHDR = f"Host: felhom.{DOMAIN}"
|
||||
HP = "demo-hp"
|
||||
|
||||
LOG = []
|
||||
|
||||
|
||||
def say(*a):
|
||||
line = " ".join(str(x) for x in a)
|
||||
ts = datetime.now().strftime("%H:%M:%S")
|
||||
print(f"{ts} {line}", flush=True)
|
||||
LOG.append(f"{ts} {line}")
|
||||
|
||||
|
||||
def sh(args, timeout=300, inp=None):
|
||||
try:
|
||||
return subprocess.run(args, capture_output=True, text=True, timeout=timeout, input=inp)
|
||||
except (subprocess.TimeoutExpired, OSError) as e:
|
||||
return subprocess.CompletedProcess(args, 124, "", f"{e}")
|
||||
|
||||
|
||||
def guest(script, timeout=600):
|
||||
"""Run a bash script inside guest 9202. Piped as a file — never as an argument (quoting)."""
|
||||
# ONE TEMP FILE PER CALL (night 2026-09-23): the shared /tmp/w<guest>.sh swapped scripts under
|
||||
# two concurrent walks (memory: guest-helper-shares-one-tmp-file).
|
||||
import secrets as _s
|
||||
t = f"/tmp/w{GUEST}-{os.getpid()}-{_s.token_hex(4)}.sh"
|
||||
r = sh(["ssh", "-o", "ConnectTimeout=20", "-o", "StrictHostKeyChecking=accept-new", HP,
|
||||
f"export LC_ALL=C; cat > {t}; pct push {GUEST} {t} {t} >/dev/null 2>&1; "
|
||||
f"pct exec {GUEST} -- bash {t}; pct exec {GUEST} -- rm -f {t}; rm -f {t}"],
|
||||
timeout=timeout, inp=script)
|
||||
return r.stdout or ""
|
||||
|
||||
|
||||
def login():
|
||||
pw = open(f"{SC}/.ctlpw").read().strip()
|
||||
sh(["curl", "-sk", "-D", f"{SC}/hdr{os.getpid()}.txt", "-o", "/dev/null", "-H", HOSTHDR,
|
||||
"-X", "POST", "--data-urlencode", f"password={pw}", f"{BASE}/login"])
|
||||
h = open(f"{SC}/hdr{os.getpid()}.txt").read()
|
||||
m = re.search(r"felhom_session=[A-Za-z0-9._-]+", h, re.I)
|
||||
if not m:
|
||||
sys.exit("login failed: no session cookie")
|
||||
open(f"{SC}/sess{os.getpid()}.txt", "w").write(m.group(0))
|
||||
r = sh(["curl", "-sk", "-L", "-H", HOSTHDR, "-H", f"Cookie: {m.group(0)}", f"{BASE}/"])
|
||||
c = re.search(r'<meta name="csrf-token" content="([^"]+)"', r.stdout or "")
|
||||
if not c:
|
||||
sys.exit("login failed: no csrf token")
|
||||
open(f"{SC}/csrf{os.getpid()}.txt", "w").write(c.group(1))
|
||||
|
||||
|
||||
def ctl(method, path, data=None, raw=False, tries=2):
|
||||
"""One controller API call. Re-logs in once on a 302/401 — the controller's session store is
|
||||
in memory, so any controller restart during the night invalidates it silently."""
|
||||
for attempt in range(tries):
|
||||
sess = open(f"{SC}/sess{os.getpid()}.txt").read().strip()
|
||||
csrf = open(f"{SC}/csrf{os.getpid()}.txt").read().strip()
|
||||
args = ["curl", "-sk", "-H", HOSTHDR, "-H", f"Cookie: {sess}", "-w", "\n%{http_code}"]
|
||||
if method != "GET":
|
||||
args += ["-H", f"X-CSRF-Token: {csrf}", "-H", "Content-Type: application/json",
|
||||
"-X", method]
|
||||
if data is not None:
|
||||
args += ["--data", json.dumps(data)]
|
||||
args.append(f"{BASE}{path}")
|
||||
r = sh(args)
|
||||
body, _, code = (r.stdout or "").rpartition("\n")
|
||||
if code.strip() in ("302", "401") and attempt + 1 < tries:
|
||||
login()
|
||||
continue
|
||||
if raw:
|
||||
return code.strip(), body
|
||||
try:
|
||||
return code.strip(), json.loads(body)
|
||||
except Exception:
|
||||
return code.strip(), {"_raw": body[:600]}
|
||||
return code.strip(), {"_raw": body[:600]}
|
||||
|
||||
|
||||
def page(path):
|
||||
sess = open(f"{SC}/sess{os.getpid()}.txt").read().strip()
|
||||
r = sh(["curl", "-sk", "-H", HOSTHDR, "-H", f"Cookie: {sess}", f"{BASE}{path}"])
|
||||
return r.stdout or ""
|
||||
|
||||
|
||||
def app_curl(sub, path, *extra, method=None, data=None, timeout=45):
|
||||
"""A call to the APP's own front door on 9202 — the household's route, not ours."""
|
||||
args = ["curl", "-sSk", "--max-time", str(timeout), "-H", f"Host: {sub}.{DOMAIN}",
|
||||
"-w", "\n%{http_code}"]
|
||||
if method:
|
||||
args += ["-X", method]
|
||||
if data is not None:
|
||||
args += ["--data-binary", "@-"]
|
||||
args += list(extra) + [f"{BASE}{path}"]
|
||||
r = sh(args, timeout=timeout + 30, inp=data)
|
||||
body, _, code = (r.stdout or "").rpartition("\n")
|
||||
return r.returncode, code.strip(), body
|
||||
|
||||
|
||||
def stack(name):
|
||||
_, d = ctl("GET", f"/api/stacks/{name}")
|
||||
return (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
|
||||
|
||||
def wait_app(sub, path="/", want=("200", "302", "303", "401", "403"), tries=60, delay=5):
|
||||
"""Settling says the container runs; this says the APP answers. Not the same thing."""
|
||||
last = None
|
||||
for _ in range(tries):
|
||||
rc, code, _ = app_curl(sub, path, timeout=15)
|
||||
last = (rc, code)
|
||||
if rc == 0 and code in want:
|
||||
return True
|
||||
time.sleep(delay)
|
||||
say(f" app never answered on {sub}{path} (last rc={last[0]} code={last[1]})")
|
||||
return False
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ the walk
|
||||
|
||||
|
||||
DRIVE = "/mnt/felhom-drives/scratch_hdd/userdata"
|
||||
|
||||
# What THIS run generated for a deploy, per app. Deploy secrets are ENCRYPTED AT REST in
|
||||
# `app.yaml` (`ENC:…`), which is right and which means a fixture cannot read an app's admin
|
||||
# password back off the box — the household sees it once. So the value the harness itself
|
||||
# generated is kept here for the life of the run, and nowhere else.
|
||||
GENERATED = {}
|
||||
|
||||
|
||||
def deploy_values(name, sub):
|
||||
"""Fill EVERY required deploy field the way the wizard would, by asking the box what this app
|
||||
asks for — `GET /api/stacks/<n>/deploy-fields` — instead of assuming DOMAIN+SUBDOMAIN.
|
||||
|
||||
Measured 2026-09-21: three apps in one batch refused at the deploy with a correct 400 because
|
||||
a required field was absent — `HDD_PATH` (navidrome, audiobookshelf) and an admin password
|
||||
(grafana). The refusals happen BEFORE anything is created (`deploy.go:324`), which is the only
|
||||
reason this was safe to discover by running it (live-probes rule).
|
||||
|
||||
A `path` field must name a directory that ALREADY EXISTS (`deploy.go:330`), so one is made on
|
||||
the scratch drive first — the same act the drive browser performs for a household.
|
||||
"""
|
||||
code, d = ctl("GET", f"/api/stacks/{name}/deploy-fields")
|
||||
fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or []
|
||||
values = {"DOMAIN": DOMAIN, "SUBDOMAIN": sub}
|
||||
made = []
|
||||
for f in fields:
|
||||
ev, ty = f.get("env_var"), f.get("type")
|
||||
if ev in values:
|
||||
continue
|
||||
# `type: password` is MANDATORY whatever `required` says — `deploy.go:305-312` refuses
|
||||
# when the caller sends none, deliberately ("the user needs to know their password"),
|
||||
# while `.felhom.yml` declares `required: false` and the API serves that verbatim. A
|
||||
# caller that trusts the contract gets a 400. Measured tonight on grafana; filed.
|
||||
if not f.get("required") and ty != "password":
|
||||
continue # the controller generates the optional secrets itself
|
||||
if ty == "path":
|
||||
p = f"{DRIVE}/{name}"
|
||||
values[ev] = p
|
||||
made.append(p)
|
||||
elif ty in ("secret", "password"):
|
||||
import secrets as _s
|
||||
values[ev] = "Drill-" + _s.token_hex(12)
|
||||
GENERATED.setdefault(name, {})[ev] = values[ev]
|
||||
elif f.get("default"):
|
||||
values[ev] = f["default"]
|
||||
else:
|
||||
values[ev] = f"drill-{name}"
|
||||
if made:
|
||||
guest("mkdir -p " + " ".join(made) + "; ls -ld " + " ".join(made))
|
||||
say(f" [1] made the drive paths this app requires: {made}")
|
||||
extra = [k for k in values if k not in ("DOMAIN", "SUBDOMAIN")]
|
||||
if extra:
|
||||
say(f" [1] required fields filled beyond DOMAIN/SUBDOMAIN: {extra}")
|
||||
return values
|
||||
|
||||
|
||||
def deploy(name, sub, extra_values=None):
|
||||
st = stack(name)
|
||||
if st.get("deployed"):
|
||||
say(f" [1] {name} already deployed — reusing")
|
||||
return True
|
||||
values = deploy_values(name, sub)
|
||||
if extra_values:
|
||||
values.update(extra_values)
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/deploy", {"values": values})
|
||||
say(f" [1] deploy -> {code} {str(d)[:120]}")
|
||||
if code != "202":
|
||||
return False
|
||||
# WAIT FOR `deployed`, NOT FOR `running`. Measured 2026-09-21 on tandoor: docker reported the
|
||||
# container `healthy` while the controller's own state read `unhealthy` — a gate on `running`
|
||||
# alone therefore times out on an app that is up. The state is RECORDED rather than required;
|
||||
# the real gate is the fixture's own `wait_app`, which asks whether the APP answers.
|
||||
seen = None
|
||||
for _ in range(90):
|
||||
time.sleep(5)
|
||||
st = stack(name)
|
||||
seen = st.get("state")
|
||||
# `deployed` alone is NOT enough and `state` alone is NOT right. Measured 2026-09-21:
|
||||
# tandoor reads `unhealthy` while serving (R-618), so gating on "running" hangs; and romm
|
||||
# read `deployed=True, state=degraded, pinned_images=None` twenty seconds in, i.e. the
|
||||
# deploy had not finished writing app.yaml. The PIN is the deploy's own completion mark
|
||||
# (`runComposeDeploy` writes it), so that is what to wait for.
|
||||
pins = (st.get("app_config") or {}).get("pinned_images")
|
||||
if st.get("deployed") and pins and seen in ("running", "unhealthy", "degraded"):
|
||||
say(f" [1] deployed, controller state={seen}, "
|
||||
f"pinned={(st.get('app_config') or {}).get('pinned_images')}")
|
||||
if seen != "running":
|
||||
say(f" [1] NOTE: the controller's own state is {seen!r}, not 'running' — recorded, "
|
||||
f"not treated as a failure; the fixture's front-door wait is the real gate")
|
||||
return True
|
||||
say(f" [1] never became deployed (last controller state={seen!r})")
|
||||
return False
|
||||
|
||||
|
||||
def backup_now(name):
|
||||
"""R-648 (2026-09-23): NO whole-box „Mentés most" from a drill, ever.
|
||||
|
||||
`POST /api/backup/run` is the only backup endpoint and it is WHOLE-BOX: on 9201 it stopped and
|
||||
restarted 9 of 10 standing apps twice, and on 9202 it broke a deploy in flight (R-634). The product
|
||||
has NO per-app backup endpoint (router.go: /backup/run, /backup/tier2 only); the per-app backup
|
||||
exists only inside the guarded update, whose `backing-up` phase calls RunAppBackupNow for the one
|
||||
app. So this presses nothing: the update takes the throwaway app's own backup, and says so in its
|
||||
phase list. A seed written "after the backup" is therefore written before the update's own backup
|
||||
— the undo's last-second copy is still the one that must bring it back."""
|
||||
say(f" [4] backup press SKIPPED for {name} (R-648: whole-box only; the update's backing-up phase backs up {name} alone)")
|
||||
return None
|
||||
|
||||
def drill_bump(app, frm, to, service_hint=None):
|
||||
"""Serialised across concurrent walks: one git working tree, one lock."""
|
||||
import fcntl
|
||||
with open(f"{SC}/drill.lock", "w") as lk:
|
||||
fcntl.flock(lk, fcntl.LOCK_EX)
|
||||
sh(["git", "-C", DRILL, "pull", "-q", "--rebase", "origin", "main"], timeout=120)
|
||||
return _drill_bump(app, frm, to, service_hint)
|
||||
|
||||
|
||||
def _drill_bump(app, frm, to, service_hint=None):
|
||||
"""Commit the edge to the DRILL repo. catalog_since set by hand (the drill repo has no gates).
|
||||
|
||||
`frm`/`to` may be comma-separated lists of the SAME length: an app whose own version lives in
|
||||
two images (adventurelog's backend and frontend) moves both in one edge, while its engine
|
||||
sidecar stays where it is — `09` §3b Q3's rule is per SERVICE, and an app-half edge must move
|
||||
every service that carries the app's own version and no others.
|
||||
"""
|
||||
comp = f"{DRILL}/templates/{app}/docker-compose.yml"
|
||||
fy = f"{DRILL}/templates/{app}/.felhom.yml"
|
||||
s = open(comp).read()
|
||||
froms = [x.strip() for x in frm.split(",") if x.strip()]
|
||||
tos = [x.strip() for x in to.split(",") if x.strip()]
|
||||
if len(froms) != len(tos):
|
||||
say(f" [5] from/to lists differ in length: {froms} vs {tos}")
|
||||
return None
|
||||
for f1, t1 in zip(froms, tos):
|
||||
if f"image: {f1}" not in s:
|
||||
say(f" [5] FROM ref not found in compose: {f1}")
|
||||
return None
|
||||
s = s.replace(f"image: {f1}", f"image: {t1}")
|
||||
open(comp, "w").write(s)
|
||||
f = open(fy).read()
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
f = re.sub(r'^catalog_since:.*$', f'catalog_since: "{today}"', f, count=1, flags=re.M)
|
||||
open(fy, "w").write(f)
|
||||
sh(["git", "-C", DRILL, "add", "-A"])
|
||||
sh(["git", "-C", DRILL, "commit", "-q", "-m", f"DRILL {app}: {frm} -> {to}"])
|
||||
r = sh(["git", "-C", DRILL, "push", "-q", "origin", "main"], timeout=120)
|
||||
h = sh(["git", "-C", DRILL, "rev-parse", "--short=12", "HEAD"]).stdout.strip()
|
||||
say(f" [5] drill commit {h}: {app} {frm} -> {to} (push rc={r.returncode})")
|
||||
return h
|
||||
|
||||
|
||||
def sync_rescan(expect_app=None, expect_ref=None, tries=12, delay=5):
|
||||
"""Sync, rescan, and — when told what to expect — WAIT FOR THE BADGE TO CATCH UP.
|
||||
|
||||
R-607: `POST /api/sync` answers "nincs valtozas" while the catalog HAS moved, and
|
||||
`catalog_images` stays stale until a separate rescan. Tonight showed the rescan alone is not
|
||||
enough either: mealie's badge read "Naprakesz" seconds after its bump was pushed, and the
|
||||
Update that followed moved nothing and still reported "Frissitve". So when the caller knows
|
||||
which reference should appear, this polls for it and SAYS HOW LONG IT TOOK — which is the
|
||||
NUMBER R-607 asks for and has never had.
|
||||
"""
|
||||
t0 = time.time()
|
||||
ctl("POST", "/api/sync")
|
||||
time.sleep(2)
|
||||
ctl("POST", "/api/stacks/rescan")
|
||||
time.sleep(2)
|
||||
if not expect_app or not expect_ref:
|
||||
return None
|
||||
for i in range(tries):
|
||||
cat = stack(expect_app).get("catalog_images") or {}
|
||||
if expect_ref in cat.values():
|
||||
waited = round(time.time() - t0, 1)
|
||||
if i:
|
||||
say(f" [sync] the badge needed {waited}s and {i+1} sync+rescan rounds to catch up "
|
||||
f"to {expect_ref} — R-607's window, measured")
|
||||
return waited
|
||||
time.sleep(delay)
|
||||
ctl("POST", "/api/sync")
|
||||
time.sleep(1)
|
||||
ctl("POST", "/api/stacks/rescan")
|
||||
say(f" [sync] the badge NEVER caught up to {expect_ref} in {round(time.time()-t0,1)}s — "
|
||||
f"catalog_images = {stack(expect_app).get('catalog_images')}")
|
||||
return None
|
||||
|
||||
|
||||
def badges(name):
|
||||
out = {}
|
||||
for lang, suffix in (("hu", ""), ("en", "?lang=en")):
|
||||
h = page(f"/apps/{name}{suffix}")
|
||||
m = re.findall(r'<span class="tag tag-[^"]*"[^>]*title="([^"]*)"[^>]*>([^<]*)<', h)
|
||||
out[lang] = [{"title": a.strip(), "text": b.strip()} for a, b in m][:3]
|
||||
return out
|
||||
|
||||
|
||||
def press_update(name, poll=1.0, cap_s=1800):
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/update")
|
||||
say(f" [6] Update -> {code} {str(d)[:220]}")
|
||||
if code not in ("202", "200"):
|
||||
return {"accepted": False, "http": code, "refusal": d, "phases": [], "duration_s": 0}
|
||||
phases, seen, t0 = [], None, time.time()
|
||||
while time.time() - t0 < cap_s:
|
||||
st = stack(name)
|
||||
ph = st.get("update_phase")
|
||||
if ph != seen:
|
||||
seen = ph
|
||||
rec = {"t": round(time.time() - t0, 1), "phase": ph,
|
||||
"label": st.get("update_phase_label"), "updating": st.get("updating"),
|
||||
"error": st.get("update_error"), "hold": st.get("hold_reason")}
|
||||
phases.append(rec)
|
||||
say(f" +{rec['t']:>6.1f}s phase={ph} label={rec['label']} "
|
||||
f"err={rec['error']} hold={rec['hold']}")
|
||||
if not st.get("updating") and ph in ("done", "failed", "undone", None) and time.time() - t0 > 3:
|
||||
break
|
||||
time.sleep(poll)
|
||||
st = stack(name)
|
||||
return {"accepted": True, "http": code, "phases": phases,
|
||||
"duration_s": round(time.time() - t0, 1),
|
||||
"final_phase": st.get("update_phase"), "update_error": st.get("update_error"),
|
||||
"hold_reason": st.get("hold_reason"), "state": st.get("state")}
|
||||
|
||||
|
||||
def observables(name):
|
||||
st = stack(name)
|
||||
ac = st.get("app_config") or {}
|
||||
live = guest(f"""
|
||||
grep -E '^\\s+image:' /opt/docker/stacks/{name}/docker-compose.yml 2>/dev/null | sed 's/^ *//'
|
||||
echo '---inspect---'
|
||||
for c in $(docker ps -a --filter label=com.docker.compose.project={name} --format '{{{{.Names}}}}'); do
|
||||
echo -n "$c "; docker inspect "$c" --format '{{{{.Config.Image}}}} running={{{{.State.Running}}}} restarts={{{{.RestartCount}}}}'
|
||||
done
|
||||
""")
|
||||
a, _, b = live.partition("---inspect---")
|
||||
return {
|
||||
"pinned_images": ac.get("pinned_images"),
|
||||
"installed_images": {k: (v.get("ref") if isinstance(v, dict) else v)
|
||||
for k, v in (ac.get("installed_images") or {}).items()},
|
||||
"catalog_images": st.get("catalog_images"),
|
||||
"live_compose_image_lines": [x for x in a.strip().splitlines() if x.strip()],
|
||||
"docker_inspect": [x for x in b.strip().splitlines() if x.strip()],
|
||||
}
|
||||
|
||||
|
||||
def app_logs(name, lines=400):
|
||||
"""The app's own container log, DECODED. The endpoint answers a JSON envelope whose `logs` is
|
||||
one string with escaped newlines — a scan over the envelope sees a single enormous line and
|
||||
finds nothing, which reads exactly like "the app printed no migration line" and is not. R-96
|
||||
rule 3 in a new place: an absent line is not evidence when the instrument cannot see lines."""
|
||||
code, d = ctl("GET", f"/api/stacks/{name}/logs?lines={lines}")
|
||||
if isinstance(d, dict):
|
||||
data = d.get("data")
|
||||
if isinstance(data, dict) and isinstance(data.get("logs"), str):
|
||||
return data["logs"]
|
||||
if isinstance(d.get("_raw"), str):
|
||||
return d["_raw"]
|
||||
return str(d)
|
||||
|
||||
|
||||
def write_verdict(rec, appdir):
|
||||
os.makedirs(appdir, exist_ok=True)
|
||||
p = os.path.join(appdir, "verdict.json")
|
||||
json.dump(rec, open(p, "w"), indent=2, ensure_ascii=False)
|
||||
say(f" [9] verdict {rec['verdict']} -> {p}")
|
||||
|
||||
|
||||
def remove(name):
|
||||
"""Remove through the PRODUCT, never `docker rm` (live-probes rule). The remove endpoint
|
||||
refuses a running stack — `409 still running` — so the stop is part of the act, not a tidy-up."""
|
||||
c1, d1 = ctl("POST", f"/api/stacks/{name}/stop")
|
||||
say(f" [X] stop -> {c1} {str(d1)[:100]}")
|
||||
for _ in range(24):
|
||||
time.sleep(5)
|
||||
if stack(name).get("state") != "running":
|
||||
break
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/remove",
|
||||
{"remove_hdd_data": True, "remove_backups": True})
|
||||
say(f" [X] remove (with drive data) -> {code} {str(d)[:160]}")
|
||||
if code == "409":
|
||||
# R-442's fail-closed guard: when the storage subsystem cannot RESOLVE the app's drive
|
||||
# path, the removal is REFUSED and the app is kept rather than half-deleted. On guest 9202
|
||||
# `/api/disks` answers `agent not configured`, so every app deployed with an HDD_PATH hits
|
||||
# this. The household's other choice — remove the app, KEEP the data — is accepted, and the
|
||||
# harness takes it, then tidies its own directory by name at teardown.
|
||||
say(" [X] refused because the drive path cannot be resolved (R-442, fail-closed and right)"
|
||||
" — removing the app and KEEPING the drive data instead")
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/remove",
|
||||
{"remove_hdd_data": False, "remove_backups": True})
|
||||
say(f" [X] remove (keeping drive data) -> {code} {str(d)[:160]}")
|
||||
time.sleep(5)
|
||||
st = stack(name)
|
||||
left = guest(f"ls -d /opt/docker/stacks/{name} 2>/dev/null; "
|
||||
f"docker ps -a --filter label=com.docker.compose.project={name} --format '{{{{.Names}}}}'")
|
||||
say(f" [X] after remove: deployed={st.get('deployed')} leftovers={left.strip()!r}")
|
||||
return code
|
||||
|
||||
|
||||
def app_env(name, key):
|
||||
"""Read one deploy value the CUSTOMER was given (e.g. the generated admin password) from the
|
||||
app's own `app.yaml`. This is not seeding — it is how the household logs in; the controller
|
||||
shows them the same value. Data still goes in through the app's own front door."""
|
||||
out = guest(f"grep -E '^\\s*{key}:' /opt/docker/stacks/{name}/app.yaml 2>/dev/null | head -1")
|
||||
if ":" in out:
|
||||
return out.split(":", 1)[1].strip().strip('"').strip("'")
|
||||
return ""
|
||||
|
||||
|
||||
def snapshots(name):
|
||||
"""The restorable copies the backups page offers for this app."""
|
||||
code, d = ctl("GET", f"/api/backup/snapshots?stack={name}")
|
||||
data = d.get("data") if isinstance(d, dict) else None
|
||||
if isinstance(data, dict):
|
||||
for k in ("snapshots", "items", "restore_points"):
|
||||
if isinstance(data.get(k), list):
|
||||
return data[k]
|
||||
return data if isinstance(data, list) else []
|
||||
|
||||
|
||||
def restore(name, snapshot_id=None, wait_s=1200):
|
||||
"""The household's own way out: the „Visszaállítás a mentésből" button on the backups page.
|
||||
|
||||
A FORM post, not an API call — `POST /backup/restore` with `_csrf`, `stack_name`,
|
||||
`snapshot_id` — because that is the button the sentence tells them to press.
|
||||
"""
|
||||
snaps = snapshots(name)
|
||||
if snapshot_id is None:
|
||||
if not snaps:
|
||||
say(f" [R] no restorable copy offered for {name}")
|
||||
return {"ok": False, "why": "no snapshot offered", "snapshots": snaps}
|
||||
first = snaps[0]
|
||||
snapshot_id = first.get("id") or first.get("snapshot_id") or first.get("short_id")
|
||||
say(f" [R] restoring {name} from snapshot {snapshot_id!r} (of {len(snaps)} offered)")
|
||||
sess = open(f"{SC}/sess{os.getpid()}.txt").read().strip()
|
||||
csrf = open(f"{SC}/csrf{os.getpid()}.txt").read().strip()
|
||||
r = sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", HOSTHDR, "-H", f"Cookie: {sess}",
|
||||
"-X", "POST",
|
||||
"--data-urlencode", f"_csrf={csrf}",
|
||||
"--data-urlencode", f"stack_name={name}",
|
||||
"--data-urlencode", f"snapshot_id={snapshot_id}",
|
||||
f"{BASE}/backup/restore"], timeout=180)
|
||||
head = (r.stdout or "").split("\n")[0].strip()
|
||||
loc = [l for l in (r.stdout or "").split("\n") if l.lower().startswith("location:")]
|
||||
say(f" [R] POST /backup/restore -> {head} {loc[:1]}")
|
||||
t0 = time.time()
|
||||
last = None
|
||||
while time.time() - t0 < wait_s:
|
||||
code, d = ctl("GET", "/api/backup/restore-status")
|
||||
dd = d.get("data") or {}
|
||||
cur = (dd.get("running"), dd.get("phase") or dd.get("state"), dd.get("message"))
|
||||
if cur != last:
|
||||
say(f" +{round(time.time()-t0,1):>6.1f}s restore {cur}")
|
||||
last = cur
|
||||
if not dd.get("running", False) and time.time() - t0 > 5:
|
||||
break
|
||||
time.sleep(2)
|
||||
st = stack(name)
|
||||
say(f" [R] after restore: state={st.get('state')} hold={st.get('hold_reason')!r} "
|
||||
f"phase={st.get('update_phase')}")
|
||||
return {"ok": True, "snapshot_id": snapshot_id, "snapshots": snaps,
|
||||
"http": head, "location": loc[:1], "seconds": round(time.time() - t0, 1),
|
||||
"state_after": st.get("state"), "hold_after": st.get("hold_reason"),
|
||||
"observables_after": observables(name)}
|
||||
Reference in New Issue
Block a user