09 decisions 40-41 (operator rulings 2026-09-26); A1 spike: the unit's definition and data drift apart (R-696 widened), R-697 filed
gates / gates (push) Successful in 25s
gates / gates (push) Successful in 25s
This commit is contained in:
@@ -575,9 +575,8 @@ so („Erről nem készül mentés." / "This is not backed up."). What brings it
|
||||
(or the app's own unit on the drive), listed per row as „Visszatölthető innen" / "Can be loaded from".
|
||||
|
||||
**Who deletes it.** Only the household, by Delete on that page with the app's name typed (a wrong name, or a path that
|
||||
is not a listed item, is refused — proven live 2026-09-25). **The box never deletes kept data by itself** — whether it
|
||||
ever should (e.g. after 90 days with warnings) is operator decision **D3, open** (`STATUS.md`). Until then kept data
|
||||
can fill a drive; the drive-full warning names the kept folders and their sizes as space the household can free.
|
||||
is not a listed item, is refused — proven live 2026-09-25). **The box never deletes kept data by itself** — operator ruling
|
||||
2026-09-26 (`09` §3 decision 40, D3 option A): no age limit, no automatic clean-up. So kept data can fill a drive; the drive-full warning names the kept folders and their sizes as space the household can free.
|
||||
|
||||
**Read-only view.** The file browser shows each kept item under „Megőrzött adatok", one `:ro` bind per item, and
|
||||
follows the list at the next sync (a write is refused: `Read-only file system`, proven live). Not yet readable there:
|
||||
|
||||
@@ -451,6 +451,21 @@ R-636's louder repeated alarm.
|
||||
watches (~12 000 requests each). **Why (b):** 91 % of the old limit is too close for a household box, the gate's
|
||||
rule is followed rather than bypassed, and the mark's weakness for such apps is filed (R-693). Reversible.
|
||||
|
||||
### 2026-09-26 — two operator rulings
|
||||
|
||||
40. **The box never deletes kept data by itself** (operator ruling 2026-09-26, D3 option A). Only the household
|
||||
deletes it, by Delete on the „Megőrzött adatok" page with the app's name typed. No age limit, no automatic
|
||||
clean-up with warnings. Kept data can fill a drive; the drive-full warning names the kept folders and their
|
||||
sizes as space the household can free (as built in v0.274.0). **Why:** the household decided to keep it; a box
|
||||
that deletes it later reverses their decision without them. Closes D3 in `STATUS.md`.
|
||||
41. **adventurelog keeps its world-data download** (operator ruling 2026-09-26, R-655 option B). The template does
|
||||
NOT set `SKIP_WORLD_DATA=1`, so a fresh install still gets the countries/regions data. The move to v0.13.0 is
|
||||
proven with the file already in the `adventurelog_media` volume (an update re-uses what v0.12.1 left), plus a
|
||||
check that a truncated file is replaced (`download-countries --force` is the command's own repair). The
|
||||
healthcheck override fix (`/usr/bin/node`) rides the image move in the same commit. **Why:** a fresh install
|
||||
without world data is a smaller product; the internet dependence is at first start, which the update's own
|
||||
health wait and undo already cover.
|
||||
|
||||
---
|
||||
|
||||
## 3b. ANSWERED 2026-09-23 — the seven questions Slices 6 and 7 needed
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# A1 — the window between an update and the next backup, measured (9202, controller 0.274.0, drill catalog)
|
||||
|
||||
2026-09-26 07:24–07:44Z. Guest 9202 on demo-hp, pointed at the drill catalog (`tools/repoint.py drill`, saved
|
||||
config `controller.yaml.pre-vt0926`). docmost deployed at `docmost/docmost:0.95.0` + `postgres:16-alpine` from a
|
||||
one-commit drill template (`85af330`, reverted by `8c3d099` right after the deploy, so the drill equals live
|
||||
`main` again), seeded through docmost's own front door (a user + login, `fixtures.py`), then climbed the live
|
||||
ladder with two presses of the guarded Update. Every file here is the raw record; this page is the reading.
|
||||
|
||||
Method: endpoint-level (the buttons' own endpoints: `POST /api/stacks/docmost/update`, the backups page's
|
||||
`POST /backup/restore` form with `snapshot_id=helyi`, the second-drive „Teljes visszaállítás" form
|
||||
`POST /backup/tier2/unit-restore`), and the unit read off the disk (`tools/a1.py unit`).
|
||||
|
||||
## Shape 1 — an app-image step (0.95.0 → 0.96.0, PostgreSQL 16 both sides)
|
||||
|
||||
| when (UTC) | unit `compose/` + `image_pins` | unit `db-dumps/docmost-postgres.sql` | restore point offered |
|
||||
|---|---|---|---|
|
||||
| 07:25:57 (the update's own "back up first") | 0.95.0 / 16 | 07:25:43, 0.95.0 schema | — |
|
||||
| 07:26:33 update `done` | 0.95.0 / 16 (unchanged) | 07:25:43 | 07:25:57 |
|
||||
| **07:27:53 the 5-minute status refresh** | **0.96.0 / 16** (`created_at` 07:27:53) | **07:25:43, 0.95.0 schema** | **07:27:53** |
|
||||
|
||||
`S1a-right-after-step1.txt`, `S1b-after-refresh.txt`. **The claim holds:** `CaptureRecoveryUnit`, run by the
|
||||
refresh (`RefreshCache` → `captureAllRecoveryUnits`), rewrote `compose/` and `image_pins` from the stack dir
|
||||
the first time it ran after the pin moved, and the dumps stayed as the backup legs left them. Tier 1's time
|
||||
(`ListRestorePoints` = newest of manifest + dumps) moved to the refresh: **07:27:53 for data from 07:25:43**.
|
||||
|
||||
**Restore in the window (own unit, `helyi`) — `S2-restore-window-step1.txt`:** volumes first (3 tars,
|
||||
including the database's own volume), then the definition from the unit's `compose/` (0.96.0), then the SQL
|
||||
with only the database up. It completed ("3 volume(s) of 3 listed, 1 database(s) of 1 listed"), the seed read
|
||||
back — **and docmost 0.96.0 ran four schema migrations on 0.95.0's data at its first start**
|
||||
(`20260824T211732-page-title-trgm-index` … `20260904T171920-public-spaces`). So the restore performed the
|
||||
update step itself, with no precondition, no safety copy of its own and no undo. Whole here; unguarded always.
|
||||
|
||||
## Shape 2 — an engine step (PostgreSQL 16 → 18, docmost-shaped, with the box's conversion)
|
||||
|
||||
| when (UTC) | unit `compose/` | unit `.sql` | pg volume tar |
|
||||
|---|---|---|---|
|
||||
| 07:29:45 back up first | 0.96.0 / **16**, mount `/var/lib/postgresql/data` | 07:29:45, `Dumped from database version 16.15` | 07:29:47, a 16 datadir at the volume root |
|
||||
| 07:30:39 update `done` (converting 7 s) | unchanged | unchanged | unchanged |
|
||||
| **07:32:53 refresh** | **0.96.0 / 18, mount `/var/lib/postgresql`** | **07:29:45, 16.15** | **07:29:47, 16 datadir** |
|
||||
|
||||
`S3a-right-after-step2.txt`, `S3b-after-refresh-step2.txt`. `conversion_copy` recorded (at 07:30:39).
|
||||
|
||||
**Restore in the window (own unit) — `S4-restore-window-step2.txt`. The claim is TRUE and the outcome is the
|
||||
worst of the three the brief listed: REFUSED by the engine, and the app left DOWN.** The 16 datadir tar was
|
||||
poured into the volume first (the volume is removed and recreated, so the live 18 datadir is GONE at that
|
||||
moment); the unit's 18 definition was then written; the DB-only start of `postgres:18-alpine` refused the
|
||||
old layout (`Error: in 18+, these Docker images are configured to store database data in a format which is
|
||||
compatible with "pg_ctlcluster" … Counter to that, there appears to be PostgreSQL data in:
|
||||
/var/lib/postgresql`) and restarted in a loop; the replay gave up (`waiting for docmost-postgres (postgres)
|
||||
readiness: timeout after 30s`); the full start failed (`docker compose up -d … exit code 1`). The page said
|
||||
the restore failed; docmost stayed `restarting`; the seed did NOT read back (the app's router answered 404).
|
||||
Neither "a fresh empty cluster with the SQL loaded into it" nor "the app comes back whole".
|
||||
|
||||
## A1.3 — the other two copies
|
||||
|
||||
**Second drive (Tier 2) — `S5-restore-tier2-window.txt`.** The mirror had been written by the update's own
|
||||
"back up first" (07:29) and was NOT touched by the refresh: its `compose/` still said 0.96.0 / **16** with
|
||||
the 16 dump and tar. „Teljes visszaállítás" from it brought docmost back **whole at PostgreSQL 16**: 3 volumes,
|
||||
1 database, the seed read back, `PG_VERSION` 16, and the pin moved back to `postgres:16-alpine` (the restore
|
||||
pins to the unit's definition, `09` §5.3). This is option A's shape — by timing, not by design: the mirror
|
||||
is rewritten on the next Tier-2 run from whatever the primary unit then holds, so after the nightly Tier 2
|
||||
it would carry the same mix as the primary.
|
||||
|
||||
**Off-site (Tier 3) — NOT measured live**: 9202 has no off-site target, and the only boxes that have one
|
||||
(9201, demo-felhom) are fenced read-only for this session. **Read from source instead
|
||||
(`offbox_reconstitute.go` `ReconstituteFromOffsite`):** the off-site restore **never writes the definition**
|
||||
— it resolves the database service from the LIVE compose ("reconstitution never overwrites the stack dir"),
|
||||
replays the snapshot's volume tars (the database's raw volume included), then the SQL. The snapshot is taken
|
||||
nightly after its own dump leg, so it is internally coherent — but an update during the day moves the live
|
||||
definition, and until the next night's off-site run **every** off-site restore puts the old data under the
|
||||
new definition. For an engine step that is S4's mechanism exactly, with one difference: this path takes a
|
||||
safety dump and rolls back on a replay failure — but the DB-only start fails before any replay, so the path
|
||||
goes to `restartStack` with the live datadir already replaced. Inferred from source + S4, not observed.
|
||||
|
||||
## Also found
|
||||
|
||||
1. **A restore drops the `conversion_copy` record but not the volume** (`S4`, `S5`: `conversion_copy=None`
|
||||
while `docmost_docmost_postgres_data.pre-update-20260926T073003Z` still exists). `PersistUnitRedeployConfig`
|
||||
builds a fresh `AppConfig`, so the record the hourly release reads is gone and the copy is never released —
|
||||
disk only, never data. Filed.
|
||||
2. **The refresh also re-captures `.felhom.yml`** (6426 B → 3998 B, the applied record of the pinned version,
|
||||
R-669) — expected; it belongs with the definition and is kept with it by the fix.
|
||||
3. **The automatic leg already refuses an app older than the ladder** (`unattended.go` `legCandidate`,
|
||||
`LegSkipOlderThanLadder`); a PERSON's press jumps to the template (`ladder.go` `nextLadderStep` Index −1,
|
||||
measured live by part 5 on vikunja 2.5.0). So the brief's claim "a restore of a version older than the
|
||||
ladder's first entry would then jump" is **true for a person's press and false for the automatic leg**.
|
||||
|
||||
## State left (for Part A's live proof)
|
||||
|
||||
docmost on 9202 at 0.96.0 / PostgreSQL 16, pinned there, seed readable; the orphaned pre-conversion copy
|
||||
volume stays until teardown. The drill repo equals live `main` (`8c3d099` is a revert of `85af330`).
|
||||
@@ -0,0 +1,25 @@
|
||||
seed: True
|
||||
seed reads back through the front door: True
|
||||
# S0-deployed — 2026-09-26T09:25:26+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.95.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.95.0", "digest": "sha256:41c8d777cf23c74e78f94e676aec328b7d7856f48df5e573543dac68d371e37c", "at": "2026-09-26T07:24:50Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:24:50Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:24:50Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=None
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
(absent)
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
(absent)
|
||||
== live
|
||||
compose image: docmost/docmost:0.95.0
|
||||
compose image: postgres:16-alpine
|
||||
compose image: redis:7-alpine
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.95.0 Up 41 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 52 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 52 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 []
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"accepted": true,
|
||||
"http": "202",
|
||||
"phases": [
|
||||
{
|
||||
"t": 0.0,
|
||||
"phase": "backing-up",
|
||||
"label": "Biztons\u00e1gi ment\u00e9s k\u00e9sz\u00fcl a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 14.4,
|
||||
"phase": "safety-dump",
|
||||
"label": "Adatb\u00e1zis pillanatk\u00e9p\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 16.4,
|
||||
"phase": "copying",
|
||||
"label": "Az adatok m\u00e1sol\u00e1sa a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 18.5,
|
||||
"phase": "starting",
|
||||
"label": "Ind\u00edt\u00e1s az \u00faj verzi\u00f3val\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 29.8,
|
||||
"phase": "verifying",
|
||||
"label": "M\u0171k\u00f6d\u00e9s ellen\u0151rz\u00e9se\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 50.3,
|
||||
"phase": "done",
|
||||
"label": "Friss\u00edtve",
|
||||
"updating": false,
|
||||
"error": null,
|
||||
"hold": null
|
||||
}
|
||||
],
|
||||
"duration_s": 50.3,
|
||||
"final_phase": "done",
|
||||
"update_error": null,
|
||||
"hold_reason": null,
|
||||
"state": "running"
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
# S1a-right-after-step1 — 2026-09-26T09:26:37+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:26:33Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:24:50Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:24:50Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up 30 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 40 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 40 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:25:57Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,66 @@
|
||||
# S1b-after-refresh — 2026-09-26T09:28:05+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:26:33Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:24:50Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:24:50Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:27:53.7715190260Z 1514 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up About a minute (healthy)
|
||||
docmost-redis redis:7-alpine Up 2 minutes (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 2 minutes (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:27:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,124 @@
|
||||
# S2-restore-window-step1 — restore pressed at 2026-09-26T07:28:25Z (own unit, snapshot helyi)
|
||||
{"ok": true, "snapshot_id": "helyi", "snapshots": [{"time": "2026-09-26T07:27:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}], "http": "HTTP/2 302", "location": ["location: /backups/restore?flash=flash.restore.started"], "seconds": 30.4, "state_after": "running", "hold_after": null}
|
||||
controller lines:
|
||||
2026/09/26 07:28:26 handlers.go:1707: [WARN] [web] Restore requested (async): stack=docmost, snapshot=helyi from 172.18.0.6:40436
|
||||
2026/09/26 07:28:26 restore_unit.go:383: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
|
||||
2026/09/26 07:28:26 manager.go:1305: [INFO] [stacks] Stopping stack: docmost
|
||||
2026/09/26 07:28:27 manager.go:1314: [INFO] [stacks] Stack docmost stopped successfully (took 1.3s)
|
||||
2026/09/26 07:28:27 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
|
||||
2026/09/26 07:28:28 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
|
||||
2026/09/26 07:28:28 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
|
||||
2026/09/26 07:28:29 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for docmost
|
||||
2026/09/26 07:28:29 deploy.go:708: [INFO] [stacks] Redeploying docmost from recovery unit with 4 env vars
|
||||
2026/09/26 07:28:29 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:28:29 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:28:29 pin.go:93: [INFO] [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:16-alpine, docmost-redis=redis:7-alpine
|
||||
2026/09/26 07:28:29 manager.go:1269: [INFO] [stacks] Starting stack docmost services only: [docmost-postgres]
|
||||
2026/09/26 07:28:29 manager.go:1280: [INFO] [stacks] Stack docmost services [docmost-postgres] started (took 0.4s)
|
||||
2026/09/26 07:28:29 restore_db.go:87: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
||||
2026/09/26 07:28:31 dbdump.go:801: [INFO] [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)
|
||||
2026/09/26 07:28:31 restore_db.go:97: [INFO] [backup] Restore docmost: replayed 1 DB dump(s)
|
||||
2026/09/26 07:28:31 manager.go:1216: [INFO] [stacks] Starting stack: docmost
|
||||
2026/09/26 07:28:42 manager.go:1231: [INFO] [stacks] Stack docmost started successfully (took 11.1s)
|
||||
2026/09/26 07:28:42 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:28:42 installed.go:420: [INFO] [stacks] installed-images docmost: recorded 3 service(s) (docmost=docmost/docmost:0.96.0 (sha256:b56947fcfd08…), docmost-postgres=postgres:16-alpine (sha256:721873c34ceb…), docmost-redis=redis:7-alpine (sha256:858f009f9709…))
|
||||
2026/09/26 07:28:46 manager.go:1562: [INFO] [stacks] Stack docmost post-start status:
|
||||
2026/09/26 07:28:46 manager.go:1565: [INFO] [stacks] docmost docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a running Up 3 seconds (health: starting)
|
||||
2026/09/26 07:28:46 manager.go:1565: [INFO] [stacks] docmost-postgres postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea running Up 16 seconds (healthy)
|
||||
2026/09/26 07:28:46 manager.go:1565: [INFO] [stacks] docmost-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy)
|
||||
2026/09/26 07:28:56 restore_unit.go:468: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
|
||||
2026/09/26 07:28:56 handlers.go:1719: [INFO] [web] Restore completed (async): stack=docmost in 29.878718951s (volumes 3/3, dbs 1/1)
|
||||
|
||||
docmost-postgres log (tail):
|
||||
|
||||
PostgreSQL Database directory appears to contain a database; Skipping initialization
|
||||
|
||||
2026-09-26 09:28:29.836 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
|
||||
2026-09-26 09:28:29.837 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
|
||||
2026-09-26 09:28:29.837 CEST [1] LOG: listening on IPv6 address "::", port 5432
|
||||
2026-09-26 09:28:29.846 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
|
||||
2026-09-26 09:28:29.855 CEST [30] LOG: database system was shut down at 2026-09-26 09:25:44 CEST
|
||||
2026-09-26 09:28:29.865 CEST [1] LOG: database system is ready to accept connections
|
||||
|
||||
docmost log (tail):
|
||||
$ pnpm --filter ./apps/server run start:prod
|
||||
$ cross-env NODE_ENV=production node dist/main
|
||||
(node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided.
|
||||
(Use `node --trace-warnings ...` to show where the warning was created)
|
||||
{"level":"info","time":"2026-09-26T07:28:51.324Z","pid":45,"hostname":"66cf0179fc93","context":"RedisModule","msg":"default: the connection was successfully established"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.574Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseModule","msg":"Establishing database connection"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.597Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseModule","msg":"Database connection successful"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260824T211732-page-title-trgm-index\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260825T022612-oauth\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260902T121326-siem-destinations\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:51.974Z","pid":45,"hostname":"66cf0179fc93","context":"DatabaseMigrationService","msg":"Migration \"20260904T171920-public-spaces\" executed successfully"}
|
||||
{"level":"info","time":"2026-09-26T07:28:52.025Z","pid":45,"hostname":"66cf0179fc93","context":"NestApplication","msg":"Nest application successfully started"}
|
||||
{"level":"info","time":"2026-09-26T07:28:52.040Z","pid":45,"hostname":"66cf0179fc93","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://c-docs.enkisfelhom.hu"}
|
||||
|
||||
seed reads back through the front door: True
|
||||
# S2-restore-window-step1 — 2026-09-26T09:29:20+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:28:42Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:27:53.7715190260Z 1514 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:25:57Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.95.0", "postgres:16-alpine", "redis:7-alpine"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.95.0
|
||||
image: postgres:16-alpine
|
||||
image: redis:7-alpine
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:25:57.3060902190Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:25:57.3060902190Z 447 compose/app.yaml
|
||||
2026-09-26T07:25:57.3060902190Z 3105 compose/docker-compose.yml
|
||||
2026-09-26T07:25:43.7249236060Z 134715 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:25:57.3078222630Z 1298 manifest.json
|
||||
2026-09-26T07:25:44.9049380830Z 51837440 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:25:45.3785498360Z 25088 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:25:45.7447035310Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up 43 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 53 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 56 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:27:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"accepted": true,
|
||||
"http": "202",
|
||||
"phases": [
|
||||
{
|
||||
"t": 0.0,
|
||||
"phase": "backing-up",
|
||||
"label": "Biztons\u00e1gi ment\u00e9s k\u00e9sz\u00fcl a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 15.4,
|
||||
"phase": "safety-dump",
|
||||
"label": "Adatb\u00e1zis pillanatk\u00e9p\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 17.5,
|
||||
"phase": "pulling",
|
||||
"label": "\u00daj verzi\u00f3 let\u00f6lt\u00e9se\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 18.5,
|
||||
"phase": "copying",
|
||||
"label": "Az adatok m\u00e1sol\u00e1sa a friss\u00edt\u00e9s el\u0151tt\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 20.5,
|
||||
"phase": "converting",
|
||||
"label": "Adatb\u00e1zis \u00e1talak\u00edt\u00e1sa",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 27.7,
|
||||
"phase": "starting",
|
||||
"label": "Ind\u00edt\u00e1s az \u00faj verzi\u00f3val\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 39.0,
|
||||
"phase": "verifying",
|
||||
"label": "M\u0171k\u00f6d\u00e9s ellen\u0151rz\u00e9se\u2026",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 54.3,
|
||||
"phase": "done",
|
||||
"label": "Friss\u00edtve",
|
||||
"updating": false,
|
||||
"error": null,
|
||||
"hold": null
|
||||
}
|
||||
],
|
||||
"duration_s": 54.4,
|
||||
"final_phase": "done",
|
||||
"update_error": null,
|
||||
"hold_reason": null,
|
||||
"state": "running"
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
# S3a-right-after-step2 — 2026-09-26T09:30:39+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:18-alpine", "digest": "sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "at": "2026-09-26T07:30:39Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy={'volume': 'docmost_docmost_postgres_data', 'copy': 'docmost_docmost_postgres_data.pre-update-20260926T073003Z', 'at': '2026-09-26T07:30:39Z', 'from': 16, 'to': 18}
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/18/docker PG_VERSION=18
|
||||
docmost docmost/docmost:0.96.0 Up 21 seconds (healthy)
|
||||
docmost-postgres postgres:18-alpine Up 37 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 32 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:30:00Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,145 @@
|
||||
# S3b-after-refresh-step2 — 2026-09-26T09:41:18+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:18-alpine", "digest": "sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "at": "2026-09-26T07:30:39Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy={'volume': 'docmost_docmost_postgres_data', 'copy': 'docmost_docmost_postgres_data.pre-update-20260926T073003Z', 'at': '2026-09-26T07:30:39Z', 'from': 16, 'to': 18}
|
||||
state=running phase=done
|
||||
# S3b-after-refresh-step2 — 2026-09-26T09:41:18+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:28:42Z"}, "docmost-postgres": {"ref": "postgres:18-alpine", "digest": "sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "at": "2026-09-26T07:30:39Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:28:42Z"}}
|
||||
conversion_copy={'volume': 'docmost_docmost_postgres_data', 'copy': 'docmost_docmost_postgres_data.pre-update-20260926T073003Z', 'at': '2026-09-26T07:30:39Z', 'from': 16, 'to': 18}
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/18/docker PG_VERSION=18
|
||||
docmost docmost/docmost:0.96.0 Up 10 minutes (healthy)
|
||||
docmost-postgres postgres:18-alpine Up 11 minutes (healthy)
|
||||
docmost-redis redis:7-alpine Up 11 minutes (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/18/docker PG_VERSION=18
|
||||
docmost docmost/docmost:0.96.0 Up 10 minutes (healthy)
|
||||
docmost-postgres postgres:18-alpine Up 11 minutes (healthy)
|
||||
docmost-redis redis:7-alpine Up 11 minutes (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
2026/09/26 07:25:43 update_guard.go:248: [DEBUG] [backup] update precondition for docmost: no Tier-2 copy (nincs másodlagos fájlmásolat ehhez az alkalmazáshoz)
|
||||
2026/09/26 07:25:57 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
|
||||
2026/09/26 07:25:57 update.go:831: [INFO] [stacks] update docmost: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-26T07:25:57Z
|
||||
2026/09/26 07:27:53 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
|
||||
2026/09/26 07:29:59 update.go:831: [INFO] [stacks] update docmost: precondition met after the backup — Tier 2 (second drive) copy from 2026-09-26T07:29:59Z
|
||||
2026/09/26 07:30:39 update.go:1015: [INFO] [stacks] update docmost: KEEPING the pre-conversion datadir copy docmost_docmost_postgres_data.pre-update-20260926T073003Z (PostgreSQL 16) until a backup of the converted app is proven
|
||||
2026/09/26 07:32:53 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
|
||||
@@ -0,0 +1,244 @@
|
||||
# S4-restore-window-step2 — restore pressed at 2026-09-26T07:41:33Z (own unit, snapshot helyi)
|
||||
{"ok": true, "snapshot_id": "helyi", "snapshots": [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}], "http": "HTTP/2 302", "location": ["location: /backups/restore?flash=flash.restore.started"], "seconds": 36.5, "state_after": "restarting", "hold_after": null}
|
||||
controller lines:
|
||||
2026/09/26 07:41:34 handlers.go:1707: [WARN] [web] Restore requested (async): stack=docmost, snapshot=helyi from 172.18.0.6:42900
|
||||
2026/09/26 07:41:34 restore_unit.go:383: [INFO] [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0
|
||||
2026/09/26 07:41:34 manager.go:1305: [INFO] [stacks] Stopping stack: docmost
|
||||
2026/09/26 07:41:35 manager.go:1314: [INFO] [stacks] Stack docmost stopped successfully (took 0.8s)
|
||||
2026/09/26 07:41:35 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
|
||||
2026/09/26 07:41:36 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
|
||||
2026/09/26 07:41:36 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
|
||||
2026/09/26 07:41:36 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for docmost
|
||||
2026/09/26 07:41:36 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:41:36 deploy.go:708: [INFO] [stacks] Redeploying docmost from recovery unit with 4 env vars
|
||||
2026/09/26 07:41:36 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:41:36 pin.go:93: [INFO] [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:18-alpine, docmost-redis=redis:7-alpine
|
||||
2026/09/26 07:41:36 manager.go:1269: [INFO] [stacks] Starting stack docmost services only: [docmost-postgres]
|
||||
2026/09/26 07:41:37 manager.go:1280: [INFO] [stacks] Stack docmost services [docmost-postgres] started (took 0.4s)
|
||||
2026/09/26 07:41:37 restore_db.go:87: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
||||
2026/09/26 07:41:53 main.go:3724: [INFO] [deadapp] docmost crossed the crash_loop threshold but a backup or restore is running (single-flight held) — NOT stopped
|
||||
2026/09/26 07:42:08 restore_unit.go:450: [ERROR] [backup] DB re-import for docmost: importing postgres dump for docmost: waiting for docmost-postgres (postgres) readiness: timeout after 30s
|
||||
2026/09/26 07:42:08 manager.go:1216: [INFO] [stacks] Starting stack: docmost
|
||||
2026/09/26 07:42:09 manager.go:1491: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/docmost) — exit code 1 (took 0.9s)
|
||||
2026/09/26 07:42:09 manager.go:1497: [ERROR] [stacks] stderr: Container docmost-redis Creating
|
||||
Container docmost-redis Created
|
||||
Container docmost Creating
|
||||
Container docmost Created
|
||||
Container docmost-postgres Starting
|
||||
Container docmost-redis Starting
|
||||
Container docmost-postgres Started
|
||||
Container docmost-redis Started
|
||||
Container docmost-postgres Waiting
|
||||
Container docmost-redis Waiting
|
||||
Container docmost-postgres Error dependency docmost-postgres failed to start
|
||||
dependency failed to start: container docmost-postgres is unhealthy
|
||||
2026/09/26 07:42:09 manager.go:1227: [ERROR] [stacks] Stack docmost start failed after 0.9s: exit code 1
|
||||
stderr: Container docmost-redis Creating
|
||||
Container docmost-redis Created
|
||||
Container docmost Creating
|
||||
Container docmost Created
|
||||
Container docmost-postgres Starting
|
||||
Container docmost-redis Starting
|
||||
Container docmost-postgres Started
|
||||
Container docmost-redis Started
|
||||
Container docmost-postgres Waiting
|
||||
Container docmost-redis Waiting
|
||||
Container docmost-postgres Error dependency docmost-postgres failed to start
|
||||
dependency failed to start: container docmost-postgres is unhealthy
|
||||
2026/09/26 07:42:09 handlers.go:1715: [ERROR] [web] Restore failed (async): stack=docmost: starting docmost after restore from unit: starting stack docmost: exit code 1
|
||||
stderr: Container docmost-redis Creating
|
||||
Container docmost-redis Created
|
||||
Container docmost Creating
|
||||
Container docmost Created
|
||||
Container docmost-postgres Starting
|
||||
Container docmost-redis Starting
|
||||
Container docmost-postgres Started
|
||||
Container docmost-redis Started
|
||||
Container docmost-postgres Waiting
|
||||
Container docmost-redis Waiting
|
||||
Container docmost-postgres Error dependency docmost-postgres failed to start
|
||||
dependency failed to start: container docmost-postgres is unhealthy
|
||||
|
||||
docmost-postgres log (tail):
|
||||
major-version-specific directory names). This better reflects how
|
||||
PostgreSQL itself works, and how upgrades are to be performed.
|
||||
|
||||
See also https://github.com/docker-library/postgres/pull/1259
|
||||
|
||||
Counter to that, there appears to be PostgreSQL data in:
|
||||
/var/lib/postgresql
|
||||
|
||||
This is usually the result of upgrading the Docker image without
|
||||
upgrading the underlying database using "pg_upgrade" (which requires both
|
||||
versions).
|
||||
|
||||
The suggested container configuration for 18+ is to place a single mount
|
||||
at /var/lib/postgresql which will then place PostgreSQL data in a
|
||||
subdirectory, allowing usage of "pg_upgrade --link" without mount point
|
||||
boundary issues.
|
||||
|
||||
See https://github.com/docker-library/postgres/issues/37 for a (long)
|
||||
discussion around this process, and suggestions for how to do so.
|
||||
Error: in 18+, these Docker images are configured to store database data in a
|
||||
format which is compatible with "pg_ctlcluster" (specifically, using
|
||||
major-version-specific directory names). This better reflects how
|
||||
PostgreSQL itself works, and how upgrades are to be performed.
|
||||
|
||||
See also https://github.com/docker-library/postgres/pull/1259
|
||||
|
||||
Counter to that, there appears to be PostgreSQL data in:
|
||||
/var/lib/postgresql
|
||||
|
||||
This is usually the result of upgrading the Docker image without
|
||||
upgrading the underlying database using "pg_upgrade" (which requires both
|
||||
versions).
|
||||
|
||||
The suggested container configuration for 18+ is to place a single mount
|
||||
at /var/lib/postgresql which will then place PostgreSQL data in a
|
||||
subdirectory, allowing usage of "pg_upgrade --link" without mount point
|
||||
boundary issues.
|
||||
|
||||
See https://github.com/docker-library/postgres/issues/37 for a (long)
|
||||
discussion around this process, and suggestions for how to do so.
|
||||
|
||||
docmost log (tail):
|
||||
|
||||
seed reads back through the front door: False
|
||||
# S4-restore-window-step2 — 2026-09-26T09:42:34+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed=null
|
||||
conversion_copy=None
|
||||
state=degraded phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
Error response from daemon: Container d38697606a1e60ebffa27adc7e262b04d6690fd593c2fc02035d6bcb1fcb83e5 is restarting, wait until the container is running
|
||||
docmost docmost/docmost:0.96.0 Created
|
||||
docmost-redis redis:7-alpine Up 30 seconds (healthy)
|
||||
docmost-postgres postgres:18-alpine Restarting (1) 8 seconds ago
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18 PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
# S4-restore-window-step2 — 2026-09-26T09:42:50+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed=null
|
||||
conversion_copy=None
|
||||
state=degraded phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
Error response from daemon: Container d38697606a1e60ebffa27adc7e262b04d6690fd593c2fc02035d6bcb1fcb83e5 is restarting, wait until the container is running
|
||||
docmost docmost/docmost:0.96.0 Created
|
||||
docmost-redis redis:7-alpine Up 46 seconds (healthy)
|
||||
docmost-postgres postgres:18-alpine Restarting (1) 24 seconds ago
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
18 PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
@@ -0,0 +1,131 @@
|
||||
# S5-restore-tier2-window — restore pressed at 2026-09-26T07:42:58Z (second-drive unit restore)
|
||||
POST /backup/tier2/unit-restore -> HTTP/2 302 ['location: /backups/apps?flash=flash.restore.full_started']
|
||||
restore-status at end: {"running": false, "op": "tier2-unit-restore", "stack": "docmost", "started_at": "2026-09-26T07:42:59.597470183Z", "last": {"op": "tier2-unit-restore", "stack": "docmost", "ok": true, "message": "A(z) docmost: 3 adatk\u00f6tet \u00e9s az adatb\u00e1zis vissza\u00e1ll\u00edtva \u2014 az alkalmaz\u00e1s \u00fajraindult. A vissza\u00e1ll\u00edt\u00e1s forr\u00e1sa a m\u00e1sodik meghajt\u00f3n l\u00e9v\u0151 m\u00e1solat volt (2026-09-26 09:29).", "finished_at": "2026-09-26T07:43:28.98815976Z"}, "last_recent": true}
|
||||
controller lines:
|
||||
2026/09/26 07:42:59 handlers.go:2106: [WARN] [web] Tier-2 UNIT restore requested (async, OVERWRITES live data): stack=docmost from 172.18.0.6:42900
|
||||
2026/09/26 07:42:59 tier2_restore.go:206: [WARN] [backup] Tier-2 UNIT restore for docmost from the secondary mirror /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit — this OVERWRITES live app data
|
||||
2026/09/26 07:42:59 restore_unit.go:383: [INFO] [backup] Restoring docmost from recovery unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit: images=3, secrets recovered=2/2, data_keys=0
|
||||
2026/09/26 07:42:59 manager.go:1305: [INFO] [stacks] Stopping stack: docmost
|
||||
2026/09/26 07:43:00 manager.go:1314: [INFO] [stacks] Stack docmost stopped successfully (took 0.4s)
|
||||
2026/09/26 07:43:00 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost
|
||||
2026/09/26 07:43:01 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_redis_data for docmost
|
||||
2026/09/26 07:43:01 restore.go:152: [INFO] [backup] Restoring Docker volume docmost_docmost_storage for docmost
|
||||
2026/09/26 07:43:01 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for docmost
|
||||
2026/09/26 07:43:01 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:43:01 deploy.go:708: [INFO] [stacks] Redeploying docmost from recovery unit with 4 env vars
|
||||
2026/09/26 07:43:01 pin.go:93: [INFO] [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:16-alpine, docmost-redis=redis:7-alpine
|
||||
2026/09/26 07:43:01 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:43:01 manager.go:1269: [INFO] [stacks] Starting stack docmost services only: [docmost-postgres]
|
||||
2026/09/26 07:43:02 manager.go:1280: [INFO] [stacks] Stack docmost services [docmost-postgres] started (took 0.4s)
|
||||
2026/09/26 07:43:02 restore_db.go:87: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
||||
2026/09/26 07:43:04 dbdump.go:801: [INFO] [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)
|
||||
2026/09/26 07:43:04 restore_db.go:97: [INFO] [backup] Restore docmost: replayed 1 DB dump(s)
|
||||
2026/09/26 07:43:04 manager.go:1216: [INFO] [stacks] Starting stack: docmost
|
||||
2026/09/26 07:43:15 manager.go:1231: [INFO] [stacks] Stack docmost started successfully (took 11.1s)
|
||||
2026/09/26 07:43:15 [INFO] [stacks] SaveAppConfig: saved config for docmost
|
||||
2026/09/26 07:43:15 installed.go:420: [INFO] [stacks] installed-images docmost: recorded 3 service(s) (docmost=docmost/docmost:0.96.0 (sha256:b56947fcfd08…), docmost-postgres=postgres:16-alpine (sha256:721873c34ceb…), docmost-redis=redis:7-alpine (sha256:858f009f9709…))
|
||||
2026/09/26 07:43:19 manager.go:1562: [INFO] [stacks] Stack docmost post-start status:
|
||||
2026/09/26 07:43:19 manager.go:1565: [INFO] [stacks] docmost docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a running Up 3 seconds (health: starting)
|
||||
2026/09/26 07:43:19 manager.go:1565: [INFO] [stacks] docmost-postgres postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea running Up 16 seconds (healthy)
|
||||
2026/09/26 07:43:19 manager.go:1565: [INFO] [stacks] docmost-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy)
|
||||
2026/09/26 07:43:28 restore_unit.go:468: [INFO] [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed
|
||||
2026/09/26 07:43:28 tier2_restore.go:251: [INFO] [backup] docmost: primary unit already carries data — left untouched (R-403 never overwrites a richer package with a poorer one)
|
||||
2026/09/26 07:43:28 handlers.go:2116: [INFO] [web] Tier-2 unit restore completed (async): stack=docmost in 29.390287688s (volumes 3/3, dbs 1/1)
|
||||
|
||||
docmost-postgres log (tail):
|
||||
|
||||
PostgreSQL Database directory appears to contain a database; Skipping initialization
|
||||
|
||||
2026-09-26 09:43:02.469 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
|
||||
2026-09-26 09:43:02.469 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
|
||||
2026-09-26 09:43:02.469 CEST [1] LOG: listening on IPv6 address "::", port 5432
|
||||
2026-09-26 09:43:02.475 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
|
||||
2026-09-26 09:43:02.484 CEST [29] LOG: database system was shut down at 2026-09-26 09:29:46 CEST
|
||||
2026-09-26 09:43:02.495 CEST [1] LOG: database system is ready to accept connections
|
||||
|
||||
docmost log (tail):
|
||||
$ pnpm --filter ./apps/server run start:prod
|
||||
$ cross-env NODE_ENV=production node dist/main
|
||||
(node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided.
|
||||
(Use `node --trace-warnings ...` to show where the warning was created)
|
||||
{"level":"info","time":"2026-09-26T07:43:22.964Z","pid":45,"hostname":"905c49590089","context":"RedisModule","msg":"default: the connection was successfully established"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.210Z","pid":45,"hostname":"905c49590089","context":"DatabaseModule","msg":"Establishing database connection"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.238Z","pid":45,"hostname":"905c49590089","context":"DatabaseModule","msg":"Database connection successful"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.399Z","pid":45,"hostname":"905c49590089","context":"DatabaseMigrationService","msg":"No pending database migrations"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.437Z","pid":45,"hostname":"905c49590089","context":"NestApplication","msg":"Nest application successfully started"}
|
||||
{"level":"info","time":"2026-09-26T07:43:23.448Z","pid":45,"hostname":"905c49590089","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://c-docs.enkisfelhom.hu"}
|
||||
|
||||
seed reads back through the front door: True
|
||||
# S5-restore-tier2-window — 2026-09-26T09:43:50+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0
|
||||
pinned={'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'}
|
||||
installed={"docmost": {"ref": "docmost/docmost:0.96.0", "digest": "sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "at": "2026-09-26T07:43:15Z"}, "docmost-postgres": {"ref": "postgres:16-alpine", "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "at": "2026-09-26T07:43:15Z"}, "docmost-redis": {"ref": "redis:7-alpine", "digest": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499", "at": "2026-09-26T07:43:15Z"}}
|
||||
conversion_copy=None
|
||||
state=running phase=done
|
||||
== unit /mnt/sys_drive/felhom-data/backups/primary/docmost
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:32:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:32:53.8191999810Z 6426 compose/.felhom.yml
|
||||
2026-09-26T07:32:53.8201999930Z 447 compose/app.yaml
|
||||
2026-09-26T07:32:53.8191999810Z 3545 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:32:53.8201999930Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T073000Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== unit /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit
|
||||
manifest.controller_version = "0.274.0"
|
||||
manifest.created_at = "2026-09-26T07:27:53Z"
|
||||
manifest.image_pins = ["docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a", "postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", "redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499"]
|
||||
manifest.db_dumps = ["docmost-postgres.sql"]
|
||||
manifest.volume_dumps = ["docmost_docmost_postgres_data.tar", "docmost_docmost_redis_data.tar", "docmost_docmost_storage.tar"]
|
||||
compose/ images:
|
||||
image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
compose/ postgres mount:
|
||||
- docmost_postgres_data:/var/lib/postgresql/data
|
||||
docmost_postgres_data:
|
||||
mtimes (UTC):
|
||||
2026-09-26T07:27:53.7705190140Z 3998 compose/.felhom.yml
|
||||
2026-09-26T07:27:53.7715190260Z 447 compose/app.yaml
|
||||
2026-09-26T07:27:53.7705190140Z 3321 compose/docker-compose.yml
|
||||
2026-09-26T07:29:45.4578891810Z 154012 db-dumps/docmost-postgres.sql
|
||||
2026-09-26T07:29:59.6671961880Z 1514 manifest.json
|
||||
2026-09-26T07:29:47.5169144410Z 70258176 volume-dumps/docmost_docmost_postgres_data.tar
|
||||
2026-09-26T07:29:48.0792382720Z 58880 volume-dumps/docmost_docmost_redis_data.tar
|
||||
2026-09-26T07:29:47.0539087610Z 1536 volume-dumps/docmost_docmost_storage.tar
|
||||
docmost-postgres.sql: Dumped from database version 16.15
|
||||
pre-restore-20260926T072557Z-docmost-postgres.sql: Dumped from database version 16.15
|
||||
== live
|
||||
compose image: docmost/docmost:0.96.0@sha256:b56947fcfd08aab8fae12a377e1792784786adbf8b96e4281f14ef4fc072685a
|
||||
compose image: postgres:16-alpine@sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea
|
||||
compose image: redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499
|
||||
PGDATA=/var/lib/postgresql/data PG_VERSION=16
|
||||
docmost docmost/docmost:0.96.0 Up 40 seconds (healthy)
|
||||
docmost-redis redis:7-alpine Up 50 seconds (healthy)
|
||||
docmost-postgres postgres:16-alpine Up 53 seconds (healthy)
|
||||
volume docmost_docmost_postgres_data
|
||||
volume docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
volume docmost_docmost_redis_data
|
||||
volume docmost_docmost_storage
|
||||
pg volume root:
|
||||
PG_VERSION base global pg_commit_ts pg_dynshmem pg_hba.conf pg_ident.conf pg_logical pg_multixact pg_notify pg_replslot pg_serial pg_snapshots pg_stat pg_stat_tmp pg_subtrans pg_tblspc pg_twophase pg_wal pg_xact postgresql.auto.conf postgresql.conf postmaster.opts postmaster.pid
|
||||
restore points offered (GET /api/backup/snapshots): 200 [{"time": "2026-09-26T07:32:53Z", "short_id": "helyi", "tier": 1, "drive_label": "Bels\u0151 SSD (rendszer)"}]
|
||||
docmost_docmost_postgres_data
|
||||
docmost_docmost_postgres_data.pre-update-20260926T073003Z
|
||||
docmost_docmost_redis_data
|
||||
docmost_docmost_storage
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env python3
|
||||
"""a1.py <cmd> — Part A1's spike on guest 9202 (drill catalog): the window between an update and the next
|
||||
backup, measured, then a restore pressed inside that window. EVIDENCE, NOT PRODUCT: it presses only what a
|
||||
person presses (deploy, Update, the backups page's restore) and reads the unit off the disk.
|
||||
|
||||
unit <label> the unit's manifest + compose images + dump headers + mtimes, and the live facts
|
||||
restore <label> [tier2] press the restore (own unit „helyi", or the second-drive unit restore) and read back
|
||||
seed / read <label> docmost's seed through its own front door (fixtures.py)
|
||||
"""
|
||||
import json, os, sys, time
|
||||
import walk as w, fixtures
|
||||
|
||||
APP, SUB = "docmost", "c-docs"
|
||||
OUT_DIR = f"{w.EV}/A1"
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
TOK = w.SC + "/seed-tokens.json"
|
||||
|
||||
|
||||
def out(label):
|
||||
return open(f"{OUT_DIR}/{label}.txt", "a", buffering=1)
|
||||
|
||||
|
||||
UNIT_SH = r"""
|
||||
for U in /mnt/sys_drive/felhom-data/backups/primary/docmost /mnt/felhom-drives/scratch_hdd/backups/secondary/docmost/recovery-unit; do
|
||||
echo "== unit $U"
|
||||
[ -d "$U" ] || { echo "(absent)"; continue; }
|
||||
python3 - "$U" <<'PY'
|
||||
import json, sys, os
|
||||
u = sys.argv[1]
|
||||
try:
|
||||
m = json.load(open(u + "/manifest.json"))
|
||||
for k in ("controller_version", "created_at", "image_pins", "db_dumps", "volume_dumps", "dumps_at", "offsite_run_id", "data"):
|
||||
if k in m:
|
||||
print(f" manifest.{k} = {json.dumps(m[k])}")
|
||||
except Exception as e:
|
||||
print(" manifest unreadable:", e)
|
||||
PY
|
||||
echo " compose/ images:"; grep -E '^\s+image:' "$U/compose/docker-compose.yml" 2>/dev/null | sed 's/^ */ /'
|
||||
echo " compose/ postgres mount:"; grep -E 'docmost_postgres_data:' "$U/compose/docker-compose.yml" 2>/dev/null | sed 's/^ */ /'
|
||||
echo " mtimes (UTC):"; find "$U" -maxdepth 2 -type f -printf ' %TY-%Tm-%TdT%TH:%TM:%TSZ %10s %P\n' | sort -k3 | grep -v pre-restore
|
||||
for f in "$U"/db-dumps/*.sql; do [ -f "$f" ] && echo " $(basename $f): $(grep -m1 -o 'Dumped from database version [0-9.]*' "$f")"; done
|
||||
done
|
||||
echo "== live"
|
||||
grep -E '^\s+image:' /opt/docker/stacks/docmost/docker-compose.yml 2>/dev/null | sed 's/^ */ compose /'
|
||||
docker exec docmost-postgres sh -c 'echo " PGDATA=$PGDATA PG_VERSION=$(cat $PGDATA/PG_VERSION 2>&1)"' 2>&1
|
||||
docker ps -a --filter label=com.docker.compose.project=docmost --format ' {{.Names}} {{.Image}} {{.Status}}'
|
||||
docker volume ls -q | grep -i docmost | sed 's/^/ volume /'
|
||||
V=$(docker volume inspect docmost_docmost_postgres_data --format '{{.Mountpoint}}' 2>/dev/null); [ -n "$V" ] && { echo " pg volume root:"; ls "$V" | head -30 | tr '\n' ' '; echo; }
|
||||
"""
|
||||
|
||||
|
||||
def unit(label):
|
||||
o = out(label)
|
||||
st = w.stack(APP)
|
||||
ac = st.get("app_config") or {}
|
||||
o.write(f"# {label} — {time.strftime('%Y-%m-%dT%H:%M:%S%z')}; controller {w.guest('cat /etc/felhom-controller-image').strip()}\n")
|
||||
o.write(f"pinned={ac.get('pinned_images')}\ninstalled={json.dumps(ac.get('installed_images'))}\n"
|
||||
f"conversion_copy={ac.get('conversion_copy')}\nstate={st.get('state')} phase={st.get('update_phase')}\n")
|
||||
o.write(w.guest(UNIT_SH, timeout=120))
|
||||
code, d = w.ctl("GET", f"/api/backup/snapshots?stack={APP}")
|
||||
o.write(f"restore points offered (GET /api/backup/snapshots): {code} {json.dumps(d.get('data') if isinstance(d, dict) else d)[:600]}\n")
|
||||
o.close()
|
||||
print(open(f"{OUT_DIR}/{label}.txt").read()[-4000:])
|
||||
|
||||
|
||||
def seed(label):
|
||||
toks = json.load(open(TOK)) if os.path.exists(TOK) else {}
|
||||
toks[APP] = fixtures.FIXTURES[APP].seed(w, SUB, w.say)
|
||||
json.dump(toks, open(TOK, "w"), default=str); os.chmod(TOK, 0o600)
|
||||
out(label).write(f"seed: {toks[APP] is not None}\n")
|
||||
|
||||
|
||||
def read(label):
|
||||
toks = json.load(open(TOK))
|
||||
ok = fixtures.FIXTURES[APP].verify(w, SUB, toks[APP], w.say)
|
||||
out(label).write(f"seed reads back through the front door: {ok}\n")
|
||||
return ok
|
||||
|
||||
|
||||
def restore(label, tier2=False):
|
||||
o = out(label)
|
||||
since = w.guest("date -u +%Y-%m-%dT%H:%M:%SZ").strip()
|
||||
o.write(f"# {label} — restore pressed at {since} ({'second-drive unit restore' if tier2 else 'own unit, snapshot helyi'})\n")
|
||||
if tier2:
|
||||
# The „Teljes visszaállítás" button: a FORM post, like the own-unit restore (web/server.go).
|
||||
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
|
||||
csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip()
|
||||
r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}",
|
||||
"-X", "POST", "--data-urlencode", f"_csrf={csrf}", "--data-urlencode", f"stack_name={APP}",
|
||||
f"{w.BASE}/backup/tier2/unit-restore"], timeout=180)
|
||||
loc = [l for l in (r.stdout or "").split("\n") if l.lower().startswith("location:")]
|
||||
o.write(f"POST /backup/tier2/unit-restore -> {(r.stdout or '').split(chr(10))[0].strip()} {loc[:1]}\n")
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 900:
|
||||
c2, s = w.ctl("GET", "/api/backup/restore-status")
|
||||
dd = (s.get("data") or {}) if isinstance(s, dict) else {}
|
||||
if not dd.get("running") and time.time() - t0 > 5:
|
||||
break
|
||||
time.sleep(3)
|
||||
o.write(f"restore-status at end: {json.dumps(dd)[:600]}\n")
|
||||
else:
|
||||
r = w.restore(APP, "helyi")
|
||||
o.write(json.dumps({k: v for k, v in r.items() if k != 'observables_after'}, default=str)[:1500] + "\n")
|
||||
time.sleep(10)
|
||||
o.write("controller lines:\n" + w.guest(
|
||||
f"docker logs --since {since} felhom-controller 2>&1 | grep -iE 'docmost|restor|replay|recreat|REFUSED' | grep -v DEBUG | cut -c1-420") + "\n")
|
||||
o.write("docmost-postgres log (tail):\n" + w.guest("docker logs --tail 40 docmost-postgres 2>&1 | cut -c1-300") + "\n")
|
||||
o.write("docmost log (tail):\n" + w.guest("docker logs --tail 25 docmost 2>&1 | cut -c1-300") + "\n")
|
||||
o.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
cmd, label = sys.argv[1], sys.argv[2]
|
||||
w.login()
|
||||
{"unit": unit, "seed": seed, "read": read,
|
||||
"restore": lambda l: restore(l, tier2=len(sys.argv) > 3 and sys.argv[3] == "tier2")}[cmd](label)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Point guest 9202 at the drill catalog (and a 90 s health timeout), or restore the saved config.
|
||||
|
||||
`09` §6.5: `git.repo_url` alone is INERT (R-615) — the cache dir must go too. The saved copy is
|
||||
`controller.yaml.pre-vt0926` (NOT the older `.pre-28`, which a restore must never pick up).
|
||||
"""
|
||||
import re, sys, io
|
||||
sys.path.insert(0, '.')
|
||||
import walk as w
|
||||
|
||||
VOL = "/var/lib/docker/volumes/felhom-controller-data/_data"
|
||||
DRILL_REPO = "https://gitea.dooplex.hu/admin/app-catalog-drill.git"
|
||||
|
||||
|
||||
def creds():
|
||||
for l in io.open("/home/kisfenyo/.git-credentials").read().strip().split("\n"):
|
||||
m = re.match(r'https://(admin):([^@]+)@gitea\.dooplex\.hu', l)
|
||||
if m:
|
||||
return m.group(1), m.group(2)
|
||||
raise SystemExit("no admin credential")
|
||||
|
||||
|
||||
def to_drill():
|
||||
u, t = creds()
|
||||
print(w.guest(f"""
|
||||
set -e
|
||||
test -f {VOL}/controller.yaml.pre-vt0926 || cp -p {VOL}/controller.yaml {VOL}/controller.yaml.pre-vt0926
|
||||
python3 - <<'PY'
|
||||
import re
|
||||
p = "{VOL}/controller.yaml"
|
||||
s = open(p).read()
|
||||
s = re.sub(r'(^\\s+repo_url: ).*$', r'\\g<1>{DRILL_REPO}', s, count=1, flags=re.M)
|
||||
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+token: ).*$', r'\\g<1>"{t}"', s, count=1, flags=re.M)
|
||||
s = re.sub(r'(^git:(?:\\n\\s+.*)*?\\n\\s+username: ).*$', r'\\g<1>"{u}"', s, count=1, flags=re.M)
|
||||
if not re.search(r'^update:', s, re.M):
|
||||
s += "update:\\n health_timeout: 90s\\n"
|
||||
open(p, "w").write(s)
|
||||
PY
|
||||
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
|
||||
docker restart felhom-controller >/dev/null
|
||||
sleep 15
|
||||
grep -A6 '^git:' {VOL}/controller.yaml | sed 's/token:.*/token: <redacted>/'
|
||||
grep -A2 '^update:' {VOL}/controller.yaml
|
||||
"""))
|
||||
|
||||
|
||||
def restore():
|
||||
print(w.guest(f"""
|
||||
set -e
|
||||
cp -p {VOL}/controller.yaml.pre-vt0926 {VOL}/controller.yaml
|
||||
rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache
|
||||
docker restart felhom-controller >/dev/null
|
||||
sleep 15
|
||||
grep -A6 '^git:' {VOL}/controller.yaml | sed 's/token:.*/token: <redacted>/'
|
||||
grep -c '^update:' {VOL}/controller.yaml || true
|
||||
"""))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
to_drill() if sys.argv[1] == "drill" else restore()
|
||||
@@ -0,0 +1,24 @@
|
||||
"""seed | read — the four drill apps' data through their OWN front doors (fixtures.py, R-156). Tokens are kept in the
|
||||
scratchpad (they can carry generated credentials); the evidence gets only True/False per app."""
|
||||
import json, sys, os
|
||||
import walk as w, fixtures
|
||||
SUBS = {"wishlist": "c-wish", "navidrome": "c-navi", "romm": "c-romm", "vikunja": "c-vik"}
|
||||
TOK = w.SC + "/seed-tokens.json"
|
||||
mode, label = sys.argv[1], sys.argv[2]
|
||||
w.login()
|
||||
res = {}
|
||||
if mode == "seed":
|
||||
toks = {}
|
||||
for a, sub in SUBS.items():
|
||||
toks[a] = fixtures.FIXTURES[a].seed(w, sub, w.say)
|
||||
res[a] = toks[a] is not None
|
||||
json.dump(toks, open(TOK, "w"), default=str); os.chmod(TOK, 0o600)
|
||||
else:
|
||||
toks = json.load(open(TOK))
|
||||
for a, sub in SUBS.items():
|
||||
if len(sys.argv) > 3 and a not in sys.argv[3].split(","):
|
||||
continue
|
||||
res[a] = bool(toks.get(a)) and fixtures.FIXTURES[a].verify(w, sub, toks[a], w.say)
|
||||
line = f"{label}: {mode} " + " ".join(f"{a}={v}" for a, v in res.items())
|
||||
w.say(line)
|
||||
open(w.EV + "/C/data-readback.txt", "a").write(line + "\n")
|
||||
@@ -0,0 +1,19 @@
|
||||
"""sr.py seed|read <label> app=sub[,app=sub] — seed or read back through each app's OWN front door (fixtures.py).
|
||||
Tokens stay in the scratchpad; the evidence gets True/False only."""
|
||||
import json, sys, os
|
||||
import walk as w, fixtures
|
||||
mode, label = sys.argv[1], sys.argv[2]
|
||||
SUBS = dict(x.split("=") for x in sys.argv[3].split(","))
|
||||
TOK = w.SC + "/seed-tokens.json"
|
||||
w.login()
|
||||
toks = json.load(open(TOK)) if os.path.exists(TOK) else {}
|
||||
res = {}
|
||||
for a, sub in SUBS.items():
|
||||
if mode == "seed":
|
||||
toks[a] = fixtures.FIXTURES[a].seed(w, sub, w.say); res[a] = toks[a] is not None
|
||||
else:
|
||||
res[a] = bool(toks.get(a)) and fixtures.FIXTURES[a].verify(w, sub, toks[a], w.say)
|
||||
if mode == "seed":
|
||||
json.dump(toks, open(TOK, "w"), default=str); os.chmod(TOK, 0o600)
|
||||
line = f"{label}: {mode} " + " ".join(f"{a}={v}" for a, v in res.items())
|
||||
w.say(line); open(w.EV + "/data-readback.txt", "a").write(line + "\n")
|
||||
@@ -0,0 +1,507 @@
|
||||
#!/usr/bin/env python3
|
||||
"""walk.py — ONE app's full update walk on guest 9202, through the product's own endpoints.
|
||||
|
||||
EVIDENCE, NOT PRODUCT. It presses exactly the buttons a person presses:
|
||||
POST /api/stacks/<n>/deploy · POST /api/sync · POST /api/stacks/rescan
|
||||
POST /api/stacks/<n>/update · POST /api/stacks/<n>/remove
|
||||
and reads GET /api/stacks/<n>. No controller code exists for it.
|
||||
|
||||
The walk, per `09` §6.4 and the update-night brief §4:
|
||||
1 deploy from the DRILL catalog at the LIVE pin
|
||||
2 seed through the app's OWN front door (R-156: never a volume, never SQL)
|
||||
3 read the seed back <- control C1; a fixture that cannot prove itself proves nothing
|
||||
4 „Mentés most"
|
||||
5 commit the real one-step bump to the DRILL repo, sync, rescan, read the badge in BOTH languages
|
||||
6 press the guarded Update, record every phase with timestamps
|
||||
7 read the seed back through the front door
|
||||
8 the four version observables side by side
|
||||
9 write the verdict record in `09`'s JSON shape
|
||||
|
||||
`inconclusive` is a first-class verdict and is NEVER collapsed into `failed`.
|
||||
"""
|
||||
import argparse, json, os, re, subprocess, sys, time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
SC = os.environ.get('SC', '/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/a061fcea-2c08-478c-9fd6-991dc8e748f0/scratchpad')
|
||||
EV = '/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/version-travel-2026-09-26'
|
||||
DRILL = "/mnt/5_hdd/felhom.eu/drill/app-catalog-drill"
|
||||
# GUEST=9201 selects demo-hp's hub-enabled guest (the mail proof); default 9202, the scratch guest.
|
||||
GUEST = os.environ.get("GUEST", "9202")
|
||||
BASE = os.environ.get("BASE") or {"9202": "https://192.168.0.114", "9201": "https://192.168.0.155"}[GUEST]
|
||||
DOMAIN = os.environ.get("DOMAIN", "enkisfelhom.hu")
|
||||
HOSTHDR = f"Host: felhom.{DOMAIN}"
|
||||
HP = "demo-hp"
|
||||
|
||||
LOG = []
|
||||
|
||||
|
||||
def say(*a):
|
||||
line = " ".join(str(x) for x in a)
|
||||
ts = datetime.now().strftime("%H:%M:%S")
|
||||
print(f"{ts} {line}", flush=True)
|
||||
LOG.append(f"{ts} {line}")
|
||||
|
||||
|
||||
def sh(args, timeout=300, inp=None):
|
||||
try:
|
||||
return subprocess.run(args, capture_output=True, text=True, timeout=timeout, input=inp)
|
||||
except (subprocess.TimeoutExpired, OSError) as e:
|
||||
return subprocess.CompletedProcess(args, 124, "", f"{e}")
|
||||
|
||||
|
||||
def guest(script, timeout=600):
|
||||
"""Run a bash script inside guest 9202. Piped as a file — never as an argument (quoting)."""
|
||||
# ONE TEMP FILE PER CALL (night 2026-09-23): the shared /tmp/w<guest>.sh swapped scripts under
|
||||
# two concurrent walks (memory: guest-helper-shares-one-tmp-file).
|
||||
import secrets as _s
|
||||
t = f"/tmp/w{GUEST}-{os.getpid()}-{_s.token_hex(4)}.sh"
|
||||
r = sh(["ssh", "-o", "ConnectTimeout=20", "-o", "StrictHostKeyChecking=accept-new", HP,
|
||||
f"export LC_ALL=C; cat > {t}; pct push {GUEST} {t} {t} >/dev/null 2>&1; "
|
||||
f"pct exec {GUEST} -- bash {t}; pct exec {GUEST} -- rm -f {t}; rm -f {t}"],
|
||||
timeout=timeout, inp=script)
|
||||
return r.stdout or ""
|
||||
|
||||
|
||||
def login():
|
||||
pw = open(f"{SC}/.ctlpw").read().strip()
|
||||
sh(["curl", "-sk", "-D", f"{SC}/hdr{os.getpid()}.txt", "-o", "/dev/null", "-H", HOSTHDR,
|
||||
"-X", "POST", "--data-urlencode", f"password={pw}", f"{BASE}/login"])
|
||||
h = open(f"{SC}/hdr{os.getpid()}.txt").read()
|
||||
m = re.search(r"felhom_session=[A-Za-z0-9._-]+", h, re.I)
|
||||
if not m:
|
||||
sys.exit("login failed: no session cookie")
|
||||
open(f"{SC}/sess{os.getpid()}.txt", "w").write(m.group(0))
|
||||
r = sh(["curl", "-sk", "-L", "-H", HOSTHDR, "-H", f"Cookie: {m.group(0)}", f"{BASE}/"])
|
||||
c = re.search(r'<meta name="csrf-token" content="([^"]+)"', r.stdout or "")
|
||||
if not c:
|
||||
sys.exit("login failed: no csrf token")
|
||||
open(f"{SC}/csrf{os.getpid()}.txt", "w").write(c.group(1))
|
||||
|
||||
|
||||
def ctl(method, path, data=None, raw=False, tries=2):
|
||||
"""One controller API call. Re-logs in once on a 302/401 — the controller's session store is
|
||||
in memory, so any controller restart during the night invalidates it silently."""
|
||||
for attempt in range(tries):
|
||||
sess = open(f"{SC}/sess{os.getpid()}.txt").read().strip()
|
||||
csrf = open(f"{SC}/csrf{os.getpid()}.txt").read().strip()
|
||||
args = ["curl", "-sk", "-H", HOSTHDR, "-H", f"Cookie: {sess}", "-w", "\n%{http_code}"]
|
||||
if method != "GET":
|
||||
args += ["-H", f"X-CSRF-Token: {csrf}", "-H", "Content-Type: application/json",
|
||||
"-X", method]
|
||||
if data is not None:
|
||||
args += ["--data", json.dumps(data)]
|
||||
args.append(f"{BASE}{path}")
|
||||
r = sh(args)
|
||||
body, _, code = (r.stdout or "").rpartition("\n")
|
||||
if code.strip() in ("302", "401") and attempt + 1 < tries:
|
||||
login()
|
||||
continue
|
||||
if raw:
|
||||
return code.strip(), body
|
||||
try:
|
||||
return code.strip(), json.loads(body)
|
||||
except Exception:
|
||||
return code.strip(), {"_raw": body[:600]}
|
||||
return code.strip(), {"_raw": body[:600]}
|
||||
|
||||
|
||||
def page(path):
|
||||
sess = open(f"{SC}/sess{os.getpid()}.txt").read().strip()
|
||||
r = sh(["curl", "-sk", "-H", HOSTHDR, "-H", f"Cookie: {sess}", f"{BASE}{path}"])
|
||||
return r.stdout or ""
|
||||
|
||||
|
||||
def app_curl(sub, path, *extra, method=None, data=None, timeout=45):
|
||||
"""A call to the APP's own front door on 9202 — the household's route, not ours."""
|
||||
args = ["curl", "-sSk", "--max-time", str(timeout), "-H", f"Host: {sub}.{DOMAIN}",
|
||||
"-w", "\n%{http_code}"]
|
||||
if method:
|
||||
args += ["-X", method]
|
||||
if data is not None:
|
||||
args += ["--data-binary", "@-"]
|
||||
args += list(extra) + [f"{BASE}{path}"]
|
||||
r = sh(args, timeout=timeout + 30, inp=data)
|
||||
body, _, code = (r.stdout or "").rpartition("\n")
|
||||
return r.returncode, code.strip(), body
|
||||
|
||||
|
||||
def stack(name):
|
||||
_, d = ctl("GET", f"/api/stacks/{name}")
|
||||
return (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
|
||||
|
||||
def wait_app(sub, path="/", want=("200", "302", "303", "401", "403"), tries=60, delay=5):
|
||||
"""Settling says the container runs; this says the APP answers. Not the same thing."""
|
||||
last = None
|
||||
for _ in range(tries):
|
||||
rc, code, _ = app_curl(sub, path, timeout=15)
|
||||
last = (rc, code)
|
||||
if rc == 0 and code in want:
|
||||
return True
|
||||
time.sleep(delay)
|
||||
say(f" app never answered on {sub}{path} (last rc={last[0]} code={last[1]})")
|
||||
return False
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ the walk
|
||||
|
||||
|
||||
DRIVE = "/mnt/felhom-drives/scratch_hdd/userdata"
|
||||
|
||||
# What THIS run generated for a deploy, per app. Deploy secrets are ENCRYPTED AT REST in
|
||||
# `app.yaml` (`ENC:…`), which is right and which means a fixture cannot read an app's admin
|
||||
# password back off the box — the household sees it once. So the value the harness itself
|
||||
# generated is kept here for the life of the run, and nowhere else.
|
||||
GENERATED = {}
|
||||
|
||||
|
||||
def deploy_values(name, sub):
|
||||
"""Fill EVERY required deploy field the way the wizard would, by asking the box what this app
|
||||
asks for — `GET /api/stacks/<n>/deploy-fields` — instead of assuming DOMAIN+SUBDOMAIN.
|
||||
|
||||
Measured 2026-09-21: three apps in one batch refused at the deploy with a correct 400 because
|
||||
a required field was absent — `HDD_PATH` (navidrome, audiobookshelf) and an admin password
|
||||
(grafana). The refusals happen BEFORE anything is created (`deploy.go:324`), which is the only
|
||||
reason this was safe to discover by running it (live-probes rule).
|
||||
|
||||
A `path` field must name a directory that ALREADY EXISTS (`deploy.go:330`), so one is made on
|
||||
the scratch drive first — the same act the drive browser performs for a household.
|
||||
"""
|
||||
code, d = ctl("GET", f"/api/stacks/{name}/deploy-fields")
|
||||
fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or []
|
||||
values = {"DOMAIN": DOMAIN, "SUBDOMAIN": sub}
|
||||
made = []
|
||||
for f in fields:
|
||||
ev, ty = f.get("env_var"), f.get("type")
|
||||
if ev in values:
|
||||
continue
|
||||
# `type: password` is MANDATORY whatever `required` says — `deploy.go:305-312` refuses
|
||||
# when the caller sends none, deliberately ("the user needs to know their password"),
|
||||
# while `.felhom.yml` declares `required: false` and the API serves that verbatim. A
|
||||
# caller that trusts the contract gets a 400. Measured tonight on grafana; filed.
|
||||
if not f.get("required") and ty != "password":
|
||||
continue # the controller generates the optional secrets itself
|
||||
if ty == "path":
|
||||
p = f"{DRIVE}/{name}"
|
||||
values[ev] = p
|
||||
made.append(p)
|
||||
elif ty in ("secret", "password"):
|
||||
import secrets as _s
|
||||
values[ev] = "Drill-" + _s.token_hex(12)
|
||||
GENERATED.setdefault(name, {})[ev] = values[ev]
|
||||
elif f.get("default"):
|
||||
values[ev] = f["default"]
|
||||
else:
|
||||
values[ev] = f"drill-{name}"
|
||||
if made:
|
||||
guest("mkdir -p " + " ".join(made) + "; ls -ld " + " ".join(made))
|
||||
say(f" [1] made the drive paths this app requires: {made}")
|
||||
extra = [k for k in values if k not in ("DOMAIN", "SUBDOMAIN")]
|
||||
if extra:
|
||||
say(f" [1] required fields filled beyond DOMAIN/SUBDOMAIN: {extra}")
|
||||
return values
|
||||
|
||||
|
||||
def deploy(name, sub, extra_values=None):
|
||||
st = stack(name)
|
||||
if st.get("deployed"):
|
||||
say(f" [1] {name} already deployed — reusing")
|
||||
return True
|
||||
values = deploy_values(name, sub)
|
||||
if extra_values:
|
||||
values.update(extra_values)
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/deploy", {"values": values})
|
||||
say(f" [1] deploy -> {code} {str(d)[:120]}")
|
||||
if code != "202":
|
||||
return False
|
||||
# WAIT FOR `deployed`, NOT FOR `running`. Measured 2026-09-21 on tandoor: docker reported the
|
||||
# container `healthy` while the controller's own state read `unhealthy` — a gate on `running`
|
||||
# alone therefore times out on an app that is up. The state is RECORDED rather than required;
|
||||
# the real gate is the fixture's own `wait_app`, which asks whether the APP answers.
|
||||
seen = None
|
||||
for _ in range(90):
|
||||
time.sleep(5)
|
||||
st = stack(name)
|
||||
seen = st.get("state")
|
||||
# `deployed` alone is NOT enough and `state` alone is NOT right. Measured 2026-09-21:
|
||||
# tandoor reads `unhealthy` while serving (R-618), so gating on "running" hangs; and romm
|
||||
# read `deployed=True, state=degraded, pinned_images=None` twenty seconds in, i.e. the
|
||||
# deploy had not finished writing app.yaml. The PIN is the deploy's own completion mark
|
||||
# (`runComposeDeploy` writes it), so that is what to wait for.
|
||||
pins = (st.get("app_config") or {}).get("pinned_images")
|
||||
if st.get("deployed") and pins and seen in ("running", "unhealthy", "degraded"):
|
||||
say(f" [1] deployed, controller state={seen}, "
|
||||
f"pinned={(st.get('app_config') or {}).get('pinned_images')}")
|
||||
if seen != "running":
|
||||
say(f" [1] NOTE: the controller's own state is {seen!r}, not 'running' — recorded, "
|
||||
f"not treated as a failure; the fixture's front-door wait is the real gate")
|
||||
return True
|
||||
say(f" [1] never became deployed (last controller state={seen!r})")
|
||||
return False
|
||||
|
||||
|
||||
def backup_now(name):
|
||||
"""R-648 (2026-09-23): NO whole-box „Mentés most" from a drill, ever.
|
||||
|
||||
`POST /api/backup/run` is the only backup endpoint and it is WHOLE-BOX: on 9201 it stopped and
|
||||
restarted 9 of 10 standing apps twice, and on 9202 it broke a deploy in flight (R-634). The product
|
||||
has NO per-app backup endpoint (router.go: /backup/run, /backup/tier2 only); the per-app backup
|
||||
exists only inside the guarded update, whose `backing-up` phase calls RunAppBackupNow for the one
|
||||
app. So this presses nothing: the update takes the throwaway app's own backup, and says so in its
|
||||
phase list. A seed written "after the backup" is therefore written before the update's own backup
|
||||
— the undo's last-second copy is still the one that must bring it back."""
|
||||
say(f" [4] backup press SKIPPED for {name} (R-648: whole-box only; the update's backing-up phase backs up {name} alone)")
|
||||
return None
|
||||
|
||||
def drill_bump(app, frm, to, service_hint=None):
|
||||
"""Serialised across concurrent walks: one git working tree, one lock."""
|
||||
import fcntl
|
||||
with open(f"{SC}/drill.lock", "w") as lk:
|
||||
fcntl.flock(lk, fcntl.LOCK_EX)
|
||||
sh(["git", "-C", DRILL, "pull", "-q", "--rebase", "origin", "main"], timeout=120)
|
||||
return _drill_bump(app, frm, to, service_hint)
|
||||
|
||||
|
||||
def _drill_bump(app, frm, to, service_hint=None):
|
||||
"""Commit the edge to the DRILL repo. catalog_since set by hand (the drill repo has no gates).
|
||||
|
||||
`frm`/`to` may be comma-separated lists of the SAME length: an app whose own version lives in
|
||||
two images (adventurelog's backend and frontend) moves both in one edge, while its engine
|
||||
sidecar stays where it is — `09` §3b Q3's rule is per SERVICE, and an app-half edge must move
|
||||
every service that carries the app's own version and no others.
|
||||
"""
|
||||
comp = f"{DRILL}/templates/{app}/docker-compose.yml"
|
||||
fy = f"{DRILL}/templates/{app}/.felhom.yml"
|
||||
s = open(comp).read()
|
||||
froms = [x.strip() for x in frm.split(",") if x.strip()]
|
||||
tos = [x.strip() for x in to.split(",") if x.strip()]
|
||||
if len(froms) != len(tos):
|
||||
say(f" [5] from/to lists differ in length: {froms} vs {tos}")
|
||||
return None
|
||||
for f1, t1 in zip(froms, tos):
|
||||
if f"image: {f1}" not in s:
|
||||
say(f" [5] FROM ref not found in compose: {f1}")
|
||||
return None
|
||||
s = s.replace(f"image: {f1}", f"image: {t1}")
|
||||
open(comp, "w").write(s)
|
||||
f = open(fy).read()
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
f = re.sub(r'^catalog_since:.*$', f'catalog_since: "{today}"', f, count=1, flags=re.M)
|
||||
open(fy, "w").write(f)
|
||||
sh(["git", "-C", DRILL, "add", "-A"])
|
||||
sh(["git", "-C", DRILL, "commit", "-q", "-m", f"DRILL {app}: {frm} -> {to}"])
|
||||
r = sh(["git", "-C", DRILL, "push", "-q", "origin", "main"], timeout=120)
|
||||
h = sh(["git", "-C", DRILL, "rev-parse", "--short=12", "HEAD"]).stdout.strip()
|
||||
say(f" [5] drill commit {h}: {app} {frm} -> {to} (push rc={r.returncode})")
|
||||
return h
|
||||
|
||||
|
||||
def sync_rescan(expect_app=None, expect_ref=None, tries=12, delay=5):
|
||||
"""Sync, rescan, and — when told what to expect — WAIT FOR THE BADGE TO CATCH UP.
|
||||
|
||||
R-607: `POST /api/sync` answers "nincs valtozas" while the catalog HAS moved, and
|
||||
`catalog_images` stays stale until a separate rescan. Tonight showed the rescan alone is not
|
||||
enough either: mealie's badge read "Naprakesz" seconds after its bump was pushed, and the
|
||||
Update that followed moved nothing and still reported "Frissitve". So when the caller knows
|
||||
which reference should appear, this polls for it and SAYS HOW LONG IT TOOK — which is the
|
||||
NUMBER R-607 asks for and has never had.
|
||||
"""
|
||||
t0 = time.time()
|
||||
ctl("POST", "/api/sync")
|
||||
time.sleep(2)
|
||||
ctl("POST", "/api/stacks/rescan")
|
||||
time.sleep(2)
|
||||
if not expect_app or not expect_ref:
|
||||
return None
|
||||
for i in range(tries):
|
||||
cat = stack(expect_app).get("catalog_images") or {}
|
||||
if expect_ref in cat.values():
|
||||
waited = round(time.time() - t0, 1)
|
||||
if i:
|
||||
say(f" [sync] the badge needed {waited}s and {i+1} sync+rescan rounds to catch up "
|
||||
f"to {expect_ref} — R-607's window, measured")
|
||||
return waited
|
||||
time.sleep(delay)
|
||||
ctl("POST", "/api/sync")
|
||||
time.sleep(1)
|
||||
ctl("POST", "/api/stacks/rescan")
|
||||
say(f" [sync] the badge NEVER caught up to {expect_ref} in {round(time.time()-t0,1)}s — "
|
||||
f"catalog_images = {stack(expect_app).get('catalog_images')}")
|
||||
return None
|
||||
|
||||
|
||||
def badges(name):
|
||||
out = {}
|
||||
for lang, suffix in (("hu", ""), ("en", "?lang=en")):
|
||||
h = page(f"/apps/{name}{suffix}")
|
||||
m = re.findall(r'<span class="tag tag-[^"]*"[^>]*title="([^"]*)"[^>]*>([^<]*)<', h)
|
||||
out[lang] = [{"title": a.strip(), "text": b.strip()} for a, b in m][:3]
|
||||
return out
|
||||
|
||||
|
||||
def press_update(name, poll=1.0, cap_s=1800):
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/update")
|
||||
say(f" [6] Update -> {code} {str(d)[:220]}")
|
||||
if code not in ("202", "200"):
|
||||
return {"accepted": False, "http": code, "refusal": d, "phases": [], "duration_s": 0}
|
||||
phases, seen, t0 = [], None, time.time()
|
||||
while time.time() - t0 < cap_s:
|
||||
st = stack(name)
|
||||
ph = st.get("update_phase")
|
||||
if ph != seen:
|
||||
seen = ph
|
||||
rec = {"t": round(time.time() - t0, 1), "phase": ph,
|
||||
"label": st.get("update_phase_label"), "updating": st.get("updating"),
|
||||
"error": st.get("update_error"), "hold": st.get("hold_reason")}
|
||||
phases.append(rec)
|
||||
say(f" +{rec['t']:>6.1f}s phase={ph} label={rec['label']} "
|
||||
f"err={rec['error']} hold={rec['hold']}")
|
||||
if not st.get("updating") and ph in ("done", "failed", "undone", None) and time.time() - t0 > 3:
|
||||
break
|
||||
time.sleep(poll)
|
||||
st = stack(name)
|
||||
return {"accepted": True, "http": code, "phases": phases,
|
||||
"duration_s": round(time.time() - t0, 1),
|
||||
"final_phase": st.get("update_phase"), "update_error": st.get("update_error"),
|
||||
"hold_reason": st.get("hold_reason"), "state": st.get("state")}
|
||||
|
||||
|
||||
def observables(name):
|
||||
st = stack(name)
|
||||
ac = st.get("app_config") or {}
|
||||
live = guest(f"""
|
||||
grep -E '^\\s+image:' /opt/docker/stacks/{name}/docker-compose.yml 2>/dev/null | sed 's/^ *//'
|
||||
echo '---inspect---'
|
||||
for c in $(docker ps -a --filter label=com.docker.compose.project={name} --format '{{{{.Names}}}}'); do
|
||||
echo -n "$c "; docker inspect "$c" --format '{{{{.Config.Image}}}} running={{{{.State.Running}}}} restarts={{{{.RestartCount}}}}'
|
||||
done
|
||||
""")
|
||||
a, _, b = live.partition("---inspect---")
|
||||
return {
|
||||
"pinned_images": ac.get("pinned_images"),
|
||||
"installed_images": {k: (v.get("ref") if isinstance(v, dict) else v)
|
||||
for k, v in (ac.get("installed_images") or {}).items()},
|
||||
"catalog_images": st.get("catalog_images"),
|
||||
"live_compose_image_lines": [x for x in a.strip().splitlines() if x.strip()],
|
||||
"docker_inspect": [x for x in b.strip().splitlines() if x.strip()],
|
||||
}
|
||||
|
||||
|
||||
def app_logs(name, lines=400):
|
||||
"""The app's own container log, DECODED. The endpoint answers a JSON envelope whose `logs` is
|
||||
one string with escaped newlines — a scan over the envelope sees a single enormous line and
|
||||
finds nothing, which reads exactly like "the app printed no migration line" and is not. R-96
|
||||
rule 3 in a new place: an absent line is not evidence when the instrument cannot see lines."""
|
||||
code, d = ctl("GET", f"/api/stacks/{name}/logs?lines={lines}")
|
||||
if isinstance(d, dict):
|
||||
data = d.get("data")
|
||||
if isinstance(data, dict) and isinstance(data.get("logs"), str):
|
||||
return data["logs"]
|
||||
if isinstance(d.get("_raw"), str):
|
||||
return d["_raw"]
|
||||
return str(d)
|
||||
|
||||
|
||||
def write_verdict(rec, appdir):
|
||||
os.makedirs(appdir, exist_ok=True)
|
||||
p = os.path.join(appdir, "verdict.json")
|
||||
json.dump(rec, open(p, "w"), indent=2, ensure_ascii=False)
|
||||
say(f" [9] verdict {rec['verdict']} -> {p}")
|
||||
|
||||
|
||||
def remove(name):
|
||||
"""Remove through the PRODUCT, never `docker rm` (live-probes rule). The remove endpoint
|
||||
refuses a running stack — `409 still running` — so the stop is part of the act, not a tidy-up."""
|
||||
c1, d1 = ctl("POST", f"/api/stacks/{name}/stop")
|
||||
say(f" [X] stop -> {c1} {str(d1)[:100]}")
|
||||
for _ in range(24):
|
||||
time.sleep(5)
|
||||
if stack(name).get("state") != "running":
|
||||
break
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/remove",
|
||||
{"remove_hdd_data": True, "remove_backups": True})
|
||||
say(f" [X] remove (with drive data) -> {code} {str(d)[:160]}")
|
||||
if code == "409":
|
||||
# R-442's fail-closed guard: when the storage subsystem cannot RESOLVE the app's drive
|
||||
# path, the removal is REFUSED and the app is kept rather than half-deleted. On guest 9202
|
||||
# `/api/disks` answers `agent not configured`, so every app deployed with an HDD_PATH hits
|
||||
# this. The household's other choice — remove the app, KEEP the data — is accepted, and the
|
||||
# harness takes it, then tidies its own directory by name at teardown.
|
||||
say(" [X] refused because the drive path cannot be resolved (R-442, fail-closed and right)"
|
||||
" — removing the app and KEEPING the drive data instead")
|
||||
code, d = ctl("POST", f"/api/stacks/{name}/remove",
|
||||
{"remove_hdd_data": False, "remove_backups": True})
|
||||
say(f" [X] remove (keeping drive data) -> {code} {str(d)[:160]}")
|
||||
time.sleep(5)
|
||||
st = stack(name)
|
||||
left = guest(f"ls -d /opt/docker/stacks/{name} 2>/dev/null; "
|
||||
f"docker ps -a --filter label=com.docker.compose.project={name} --format '{{{{.Names}}}}'")
|
||||
say(f" [X] after remove: deployed={st.get('deployed')} leftovers={left.strip()!r}")
|
||||
return code
|
||||
|
||||
|
||||
def app_env(name, key):
|
||||
"""Read one deploy value the CUSTOMER was given (e.g. the generated admin password) from the
|
||||
app's own `app.yaml`. This is not seeding — it is how the household logs in; the controller
|
||||
shows them the same value. Data still goes in through the app's own front door."""
|
||||
out = guest(f"grep -E '^\\s*{key}:' /opt/docker/stacks/{name}/app.yaml 2>/dev/null | head -1")
|
||||
if ":" in out:
|
||||
return out.split(":", 1)[1].strip().strip('"').strip("'")
|
||||
return ""
|
||||
|
||||
|
||||
def snapshots(name):
|
||||
"""The restorable copies the backups page offers for this app."""
|
||||
code, d = ctl("GET", f"/api/backup/snapshots?stack={name}")
|
||||
data = d.get("data") if isinstance(d, dict) else None
|
||||
if isinstance(data, dict):
|
||||
for k in ("snapshots", "items", "restore_points"):
|
||||
if isinstance(data.get(k), list):
|
||||
return data[k]
|
||||
return data if isinstance(data, list) else []
|
||||
|
||||
|
||||
def restore(name, snapshot_id=None, wait_s=1200):
|
||||
"""The household's own way out: the „Visszaállítás a mentésből" button on the backups page.
|
||||
|
||||
A FORM post, not an API call — `POST /backup/restore` with `_csrf`, `stack_name`,
|
||||
`snapshot_id` — because that is the button the sentence tells them to press.
|
||||
"""
|
||||
snaps = snapshots(name)
|
||||
if snapshot_id is None:
|
||||
if not snaps:
|
||||
say(f" [R] no restorable copy offered for {name}")
|
||||
return {"ok": False, "why": "no snapshot offered", "snapshots": snaps}
|
||||
first = snaps[0]
|
||||
snapshot_id = first.get("id") or first.get("snapshot_id") or first.get("short_id")
|
||||
say(f" [R] restoring {name} from snapshot {snapshot_id!r} (of {len(snaps)} offered)")
|
||||
sess = open(f"{SC}/sess{os.getpid()}.txt").read().strip()
|
||||
csrf = open(f"{SC}/csrf{os.getpid()}.txt").read().strip()
|
||||
r = sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", HOSTHDR, "-H", f"Cookie: {sess}",
|
||||
"-X", "POST",
|
||||
"--data-urlencode", f"_csrf={csrf}",
|
||||
"--data-urlencode", f"stack_name={name}",
|
||||
"--data-urlencode", f"snapshot_id={snapshot_id}",
|
||||
f"{BASE}/backup/restore"], timeout=180)
|
||||
head = (r.stdout or "").split("\n")[0].strip()
|
||||
loc = [l for l in (r.stdout or "").split("\n") if l.lower().startswith("location:")]
|
||||
say(f" [R] POST /backup/restore -> {head} {loc[:1]}")
|
||||
t0 = time.time()
|
||||
last = None
|
||||
while time.time() - t0 < wait_s:
|
||||
code, d = ctl("GET", "/api/backup/restore-status")
|
||||
dd = d.get("data") or {}
|
||||
cur = (dd.get("running"), dd.get("phase") or dd.get("state"), dd.get("message"))
|
||||
if cur != last:
|
||||
say(f" +{round(time.time()-t0,1):>6.1f}s restore {cur}")
|
||||
last = cur
|
||||
if not dd.get("running", False) and time.time() - t0 > 5:
|
||||
break
|
||||
time.sleep(2)
|
||||
st = stack(name)
|
||||
say(f" [R] after restore: state={st.get('state')} hold={st.get('hold_reason')!r} "
|
||||
f"phase={st.get('update_phase')}")
|
||||
return {"ok": True, "snapshot_id": snapshot_id, "snapshots": snaps,
|
||||
"http": head, "location": loc[:1], "seconds": round(time.time() - t0, 1),
|
||||
"state_after": st.get("state"), "hold_after": st.get("hold_reason"),
|
||||
"observables_after": observables(name)}
|
||||
@@ -814,7 +814,9 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
| **R-693** | **[P3-LOW] The memory watch marks a Node app `memory_tight` at any limit — its heap sizes itself from the limit.** Measured 2026-09-25 on the bench (docmost 0.96.0, harness v4): the app's own memory (`anon`) peaked at **349 MB of 384 MB (90.9 %)**, then, with the limit raised to 512 MB, at **431 MB of 512 MB (80.4 %)** — 0 OOM kills and 0 restarts in both 10-minute watches (~12 000 requests each). So the mark (decision 22's "does not fit the memory") fires for an app that fits, and the gate's remedy (raise the limit) cannot clear it. docmost moved with the limit raised to 512 MB (decision 39). **Needs:** a basis that tells growth-to-fill from pressure (e.g. kills/restarts plus a GC-pressure signal, or a second watch at a higher limit showing the peak scales), or a per-app `memory_scales_with_limit` fact. `audits/night-2026-09-26/C/bench-run1/`, `…/bench-run2/` | **OPEN — P3; owner: CC** |
|
||||
| **R-694** | **[P3-LOW] Loading kept data (and every unit restore) regenerates a withheld login secret — does the household's shown password still work?** Seen 2026-09-25 on 9202 (E5, nextcloud): `generated replacement for [NEXTCLOUD_ADMIN_PASSWORD] — the credential was reset (old value unrecoverable)`. The unit deliberately carries no internet-reachable admin login (D5). For nextcloud the real admin password lives in the loaded DATABASE, so the new env value is likely inert — but if the app page shows the regenerated value as "your password", it is a false one. **Not measured:** what the page shows after a load, per app. `audits/night-2026-09-26/E/E5-5-use.txt` | **OPEN — P3; owner: CC (measure first)** |
|
||||
| **R-695** | **[P3-LOW] Two kept-data Deletes in the same second can leave a self-perpetuating empty kept folder.** Seen 2026-09-25 on 9202 (v0.274.0, teardown): two Deletes at 13:35:16 each started `SyncFileBrowserMounts` in a goroutine; one restarted the file browser with the OLD bind list, Docker recreated the deleted folder `kept/nextcloud/2026-09-25_141014` EMPTY (root, 13:35:17), and the next sync LISTED that empty folder as a kept item and kept binding it — so the bind recreated the folder and the folder kept the bind. Broken by removing the folder and one controller restart. Harm: an empty 0-byte row and a stale empty source in the view; no data. **Fix direction:** single-flight the file-browser sync (the second waits and re-reads), and never list or bind an EMPTY dated kept folder. `audits/night-2026-09-26/G/T1-teardown-9202.txt` | **OPEN — P3; owner: CC** |
|
||||
| **R-696** | **[P2-MEDIUM] Tier 1's "proven at" is the unit MANIFEST's refresh time, not its data's — so the kept pre-conversion copy was released on a backup taken BEFORE the conversion.** Seen 2026-09-26 on demo-hp (v0.274.0, the automatic leg): docmost converted 16 → 18 at 02:18:52Z; at 02:20:16Z the recovery unit was re-captured (its pins changed) — `manifest.json` `created_at` 02:20:16Z — while its `db-dumps/docmost-postgres.sql` is from **02:15:01Z, on PostgreSQL 16**. `UpdateRestorePoints` reads Tier 1's time from `ListRestorePoints` (the unit's time), so at 02:30:16Z `ReleaseConversionCopies` logged "a backup proven on 18: Tier 1 at 02:20:16Z" and removed the 16 datadir copy. **Harm tonight: low** — the 02:15 logical dump holds the rows the conversion's check proved equal, and the next night's dump is on 18. **The class is wider:** the update's own precondition reads the same time, and a unit is re-captured on every controller release or pin change (`CaptureRecoveryUnit`), so a stale dump can read as minutes old (9202 2026-09-25 11:06: "Tier 1 copy 2m0s old" from the 11:04 restart, dumps older). On 9202 the release was right by luck (dump 11:57 > conversion 11:13). **Fix direction:** Tier 1's time = the newest DATA in the unit (its db/volume dumps' own times), never the manifest's; the release additionally requires a dump of the converted service written after the conversion. Presence-is-not-success, `CLAUDE.md`. `audits/night-2026-09-26/G/G4-demo-hp-docmost-after.txt` | **OPEN — P2; owner: CC** |
|
||||
| **R-696** | **[P2-MEDIUM] Tier 1's "proven at" is the unit MANIFEST's refresh time, not its data's — so the kept pre-conversion copy was released on a backup taken BEFORE the conversion.** Seen 2026-09-26 on demo-hp (v0.274.0, the automatic leg): docmost converted 16 → 18 at 02:18:52Z; at 02:20:16Z the recovery unit was re-captured (its pins changed) — `manifest.json` `created_at` 02:20:16Z — while its `db-dumps/docmost-postgres.sql` is from **02:15:01Z, on PostgreSQL 16**. `UpdateRestorePoints` reads Tier 1's time from `ListRestorePoints` (the unit's time), so at 02:30:16Z `ReleaseConversionCopies` logged "a backup proven on 18: Tier 1 at 02:20:16Z" and removed the 16 datadir copy. **Harm tonight: low** — the 02:15 logical dump holds the rows the conversion's check proved equal, and the next night's dump is on 18. **The class is wider:** the update's own precondition reads the same time, and a unit is re-captured on every controller release or pin change (`CaptureRecoveryUnit`), so a stale dump can read as minutes old (9202 2026-09-25 11:06: "Tier 1 copy 2m0s old" from the 11:04 restart, dumps older). On 9202 the release was right by luck (dump 11:57 > conversion 11:13). **Fix direction:** Tier 1's time = the newest DATA in the unit (its db/volume dumps' own times), never the manifest's; the release additionally requires a dump of the converted service written after the conversion. Presence-is-not-success, `CLAUDE.md`. `audits/night-2026-09-26/G/G4-demo-hp-docmost-after.txt` **-- A1 SPIKE 2026-09-26 (9202, controller 0.274.0, `audits/version-travel-2026-09-26/A1/README.md`) — the class is WIDER than the time: the refresh re-captures the unit's DEFINITION too, so the unit pairs new pins with old data.** Measured both shapes: an app step (docmost 0.95.0 → 0.96.0) — the refresh 2 min after the update wrote 0.96.0 into `compose/` over 0.95.0's dump; a restore in that window came back whole only because docmost migrated the old data forward at its first start (four migrations, no guard, no undo). An engine step (PostgreSQL 16 → 18) — the refresh wrote the 18 definition over the 16 dump and 16 datadir tar; a restore in that window poured the 16 datadir back, `postgres:18` REFUSED it, the replay timed out and **the app was left DOWN**. The second-drive mirror, written before the update, brought it back whole at 16. From source: the OFF-SITE restore never writes the definition at all (it uses the live compose), so after any update it mixes versions until the next night's off-site run. Being fixed as Part A of the version-travel brief (controller v0.275.0). | **OPEN — P2; owner: CC** |
|
||||
| **R-697** | **[P3-LOW] A restore drops the `conversion_copy` record but not the kept pre-conversion volume — the copy is then never released.** Seen 2026-09-26 on 9202 (A1, controller 0.274.0): after docmost's 16 → 18 conversion `app.yaml` held `conversion_copy` (`docmost_docmost_postgres_data.pre-update-20260926T073003Z`); a restore from the own unit, then one from the second drive, both left `conversion_copy=None` while the volume stayed. `PersistUnitRedeployConfig` builds a fresh `AppConfig` (Deployed, DeployedAt, Env, LockedFields), so every record kept beside the env is dropped by a restore — the hourly release reads the record, so the volume is orphaned. **Harm: disk only** (the size of the old datadir), never data. **Fix direction:** carry `conversion_copy` across the restore's app.yaml rewrite (the release then removes it when a backup on the new major is proven), and say in the log when a restore supersedes one. `audits/version-travel-2026-09-26/A1/S4-restore-window-step2.txt`, `S5-restore-tier2-window.txt` | **OPEN — P3; owner: CC** |
|
||||
|
||||
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
|
||||
One row per dated check. The R-number must have a row above. Dates are UTC.
|
||||
Clearing a row means the check was DONE and its result recorded in that R-row —
|
||||
|
||||
Reference in New Issue
Block a user