From 302f24e47c411a3acc8075c531c21c9e7dadb93e Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 25 Sep 2026 14:35:15 +0200 Subject: [PATCH] docs: 07 kept data; register (R-657, R-690, R-692 closed; R-450/463/687/688/691 narrowed; R-693, R-694 opened); night-2026-09-26 evidence (part0, C, D, E, F) Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- .../architecture/07-backup-architecture.md | 28 + .../night-2026-09-26/C/bench-run1/C3.log | 42 + .../C/bench-run1/MV-docmost.log | 166 ++ .../bench-run1/evidence/C3/abort-states.json | 8 + .../bench-run1/evidence/C3/compose-final.log | 5 + .../bench-run1/evidence/C3/engine-state.json | 1 + .../C/bench-run1/evidence/C3/files-after.json | 1 + .../bench-run1/evidence/C3/files-before.json | 1 + .../evidence/C3/migration-lines.txt | 0 .../C/bench-run1/evidence/C3/run.log | 14 + .../C/bench-run1/evidence/C3/to-full.log | 0 .../C/bench-run1/evidence/C3/to-states.json | 8 + .../C/bench-run1/evidence/C3/verdict.json | 28 + .../evidence/MV-docmost/abort-refusal.txt | 80 + .../evidence/MV-docmost/abort-states.json | 20 + .../evidence/MV-docmost/compose-final.log | 176 ++ .../MV-docmost/convert-check-after.txt | 54 + .../MV-docmost/convert-check-before.txt | 54 + .../evidence/MV-docmost/convert-load.err | 0 .../evidence/MV-docmost/engine-state.json | 8 + .../evidence/MV-docmost/files-after.json | 1 + .../evidence/MV-docmost/files-before.json | 1 + .../evidence/MV-docmost/memory-samples.json | 1562 +++++++++++++++++ .../evidence/MV-docmost/migration-lines.txt | 1 + .../C/bench-run1/evidence/MV-docmost/run.log | 59 + .../evidence/MV-docmost/to-full.log | 166 ++ .../evidence/MV-docmost/to-states.json | 20 + .../evidence/MV-docmost/verdict.json | 107 ++ .../C/bench-run2/MV-docmost.log | 166 ++ .../evidence/MV-docmost/abort-refusal.txt | 80 + .../evidence/MV-docmost/abort-states.json | 20 + .../evidence/MV-docmost/compose-final.log | 176 ++ .../MV-docmost/convert-check-after.txt | 54 + .../MV-docmost/convert-check-before.txt | 54 + .../evidence/MV-docmost/convert-load.err | 0 .../evidence/MV-docmost/engine-state.json | 8 + .../evidence/MV-docmost/files-after.json | 1 + .../evidence/MV-docmost/files-before.json | 1 + .../evidence/MV-docmost/memory-samples.json | 1562 +++++++++++++++++ .../evidence/MV-docmost/migration-lines.txt | 1 + .../C/bench-run2/evidence/MV-docmost/run.log | 59 + .../evidence/MV-docmost/to-full.log | 166 ++ .../evidence/MV-docmost/to-states.json | 20 + .../evidence/MV-docmost/verdict.json | 107 ++ .../audits/night-2026-09-26/D/D2-arrival.txt | 1 + .../audits/night-2026-09-26/D/D2-floor.txt | 4 + .../night-2026-09-26/D/D3-release-9202.txt | 13 + .../night-2026-09-26/E/CHANGELOG-fragment.md | 17 + .../night-2026-09-26/E/E5-00-deploy-rc.txt | 3 + .../audits/night-2026-09-26/E/E5-1-list.txt | 18 + .../audits/night-2026-09-26/E/E5-2-ask.txt | 38 + .../audits/night-2026-09-26/E/E5-3-fresh.txt | 31 + .../night-2026-09-26/E/E5-4-remove-keep.txt | 14 + .../E/E5-4a-window-to-13:57.txt | 12 + .../night-2026-09-26/E/E5-4b-dbdump.txt | 19 + .../night-2026-09-26/E/E5-4c-window-back.txt | 12 + .../audits/night-2026-09-26/E/E5-5-use.txt | 24 + .../night-2026-09-26/E/E5-6-fresh-again.txt | 19 + .../night-2026-09-26/E/E5-7-load-delete.txt | 24 + .../E/E5-8-filebrowser-ro.txt | 17 + .../night-2026-09-26/E/E5-redproof-owner.txt | 4 + .../E/E6-demo-hp-kept-page.txt | 2 + .../night-2026-09-26/E/E6-floor-0274.txt | 5 + .../night-2026-09-26/E/redproofs/kept.txt | 25 + .../night-2026-09-26/F/F2-live-preview.txt | 1 + .../audits/night-2026-09-26/part0/README.md | 33 + .../audits/night-2026-09-26/tools/e5.py | 88 + .../audits/night-2026-09-26/tools/e5b.py | 47 + documentation/backlog/CLOSED-ITEMS.md | 8 + documentation/backlog/OPEN-ITEMS.md | 13 +- 70 files changed, 5572 insertions(+), 6 deletions(-) create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/C3.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/MV-docmost.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/abort-states.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/compose-final.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/engine-state.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-after.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-before.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/migration-lines.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/run.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/to-full.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/to-states.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/verdict.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-refusal.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-states.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/compose-final.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-after.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-before.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-load.err create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/engine-state.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-after.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-before.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/memory-samples.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/migration-lines.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/run.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-full.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-states.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/verdict.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/MV-docmost.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-refusal.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-states.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/compose-final.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-after.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-before.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-load.err create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/engine-state.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-after.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-before.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/memory-samples.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/migration-lines.txt create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/run.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-full.log create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-states.json create mode 100644 documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/verdict.json create mode 100644 documentation/audits/night-2026-09-26/D/D2-arrival.txt create mode 100644 documentation/audits/night-2026-09-26/D/D2-floor.txt create mode 100644 documentation/audits/night-2026-09-26/D/D3-release-9202.txt create mode 100644 documentation/audits/night-2026-09-26/E/CHANGELOG-fragment.md create mode 100644 documentation/audits/night-2026-09-26/E/E5-00-deploy-rc.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-1-list.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-2-ask.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-3-fresh.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-4-remove-keep.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-4a-window-to-13:57.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-4b-dbdump.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-4c-window-back.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-5-use.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-6-fresh-again.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-7-load-delete.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-8-filebrowser-ro.txt create mode 100644 documentation/audits/night-2026-09-26/E/E5-redproof-owner.txt create mode 100644 documentation/audits/night-2026-09-26/E/E6-demo-hp-kept-page.txt create mode 100644 documentation/audits/night-2026-09-26/E/E6-floor-0274.txt create mode 100644 documentation/audits/night-2026-09-26/E/redproofs/kept.txt create mode 100644 documentation/audits/night-2026-09-26/F/F2-live-preview.txt create mode 100644 documentation/audits/night-2026-09-26/part0/README.md create mode 100644 documentation/audits/night-2026-09-26/tools/e5.py create mode 100644 documentation/audits/night-2026-09-26/tools/e5b.py diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 63434385..70de25d8 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -557,6 +557,34 @@ successes only. After an agent restart the success is read back from the tier's anything is stopped, logged, and reported once as `backup_tier_skipped`; `unknown` is never skipped. **Still open:** quiescing per tier, so a slow second tier does not keep every app down. +### 6.5 Kept data — what a removed app leaves on the drive (controller v0.274.0, `09` §3 decision 36) + +**What it is.** "Remove the app, keep my data" leaves the app's drive folder (`/appdata/`, or the folder +its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files +(R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of +THIS drive's install — the app's own unit or the second-drive mirror — loaded under the kept files, then the +template's `after_load:`, e.g. nextcloud's `occ files:scan --all`) or „Tiszta lappal kezdem" / "Start fresh". + +**Where it lives.** "Start fresh" RENAMES the folder — same drive, never a copy, never across drives — to +`/kept///`, together with the removed app's unit when one sits on that drive (so a later +Load has its database). `/kept` is in `ProtectedHDDPaths`, never under `userdata/`, never bound by a live app. +The page „Megőrzött adatok" / "Kept data" lists every dated folder and every `appdata/` no installed app binds. + +**It is NOT backed up.** No tier captures `/kept` or a leftover `appdata/` of a removed app; the page says +so („Erről nem készül mentés." / "This is not backed up."). What brings it back into an app is the unit it carries +(or the app's own unit on the drive), listed per row as „Visszatölthető innen" / "Can be loaded from". + +**Who deletes it.** Only the household, by Delete on that page with the app's name typed (a wrong name, or a path that +is not a listed item, is refused — proven live 2026-09-25). **The box never deletes kept data by itself** — whether it +ever should (e.g. after 90 days with warnings) is operator decision **D3, open** (`STATUS.md`). Until then kept data +can fill a drive; the drive-full warning names the kept folders and their sizes as space the household can free. + +**Read-only view.** The file browser shows each kept item under „Megőrzött adatok", one `:ro` bind per item, and +follows the list at the next sync (a write is refused: `Read-only file system`, proven live). Not yet readable there: +a folder its app owns with mode 0770 (nextcloud, `www-data`) — R-691. + +Evidence: `audits/night-2026-09-26/E/` (E1 spike, E5 live proof). + ## 7. The recovery chain (D3) — the reason this document exists **[DESIGN] 3-2-1 describes copies. It does not describe recovery.** diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/C3.log b/documentation/audits/night-2026-09-26/C/bench-run1/C3.log new file mode 100644 index 00000000..78b3b75e --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/C3.log @@ -0,0 +1,42 @@ +[11:21:19] scratch drive folders cleared before FROM (R-656): none existed +[11:21:19] C3: deploying privatebin at FROM {'privatebin': 'privatebin/pdo:2.0.5'} +[11:21:56] FROM settled=True in 30.8s :: {"privatebin": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:21:56] privatebin: seeded paste id=6d06e345497ece79 +[11:21:56] privatebin: readback http=200 marker_present=True +[11:21:56] C1 (seed reads back BEFORE): True +[11:21:56] C3: swapping to TO {'privatebin': 'alpine:3.20'} +[11:22:00] TO up -d rc=0 +[11:29:04] TO settled=False in 424.4s :: {"privatebin": {"status": "restarting", "health": "unhealthy", "restarts": 0, "exit": 0}} +[11:29:04] migration lines observed: 0 +[11:29:04] RESULT (seed reads back AFTER): False +[11:29:05] C3: ABORT — putting the FROM images back +[11:29:36] privatebin: readback http=200 marker_present=True +[11:29:36] ABORT: app came back in 30.8s; data present=True +{ + "harness_version": 4, + "edge": "C3", + "app": "privatebin", + "note": "NEGATIVE control: the TO image starts and exits immediately", + "from": { + "privatebin": "privatebin/pdo:2.0.5" + }, + "to": { + "privatebin": "alpine:3.20" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "starts-and-serves", + "abort_detail": null, + "engine_state_after": null, + "memory": null, + "marks": [], + "duration_s": 424.4, + "measured_at": "2026-09-25T11:29:36Z", + "evidence": "evidence/C3", + "scratch_cleared": [], + "files_changed": [], + "total_s": 496.4 +} diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/MV-docmost.log b/documentation/audits/night-2026-09-26/C/bench-run1/MV-docmost.log new file mode 100644 index 00000000..cb5fc380 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/MV-docmost.log @@ -0,0 +1,166 @@ +[11:29:36] scratch drive folders cleared before FROM (R-656): none existed +[11:29:36] MV-docmost: deploying docmost at FROM {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:31:07] FROM settled=True in 31.1s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:31:07] docmost: /api/auth/setup http=200 rc=0 +[11:31:07] docmost: login as the seeded user http=200 ok=True +[11:31:07] C1 (seed reads back BEFORE): True +[11:31:08] MV-docmost: PostgreSQL major move {'service': 'docmost-postgres', 'engine': 'postgres', 'from': 16, 'to': 18} — converting on the bench before the swap +[11:31:33] CONVERTED docmost-postgres PostgreSQL 16 -> 18 in 25.0s (dump 0.54s / 132184 B, load 2.06s, check equal over 48 tables) +[11:31:33] MV-docmost: swapping to TO {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:31:44] TO up -d rc=0 +[11:32:15] TO settled=True in 31.1s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:32:15] engine state docmost-postgres: 18 +[11:32:15] migration lines observed: 1 +[11:32:15] docmost: login as the seeded user http=200 ok=True +[11:32:15] RESULT (seed reads back AFTER): True +[11:32:16] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:3000 +[11:32:31] + 15s docmost=342M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=9M kills=0 rs=0 reqs=300 +[11:32:46] + 30s docmost=346M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=600 +[11:33:01] + 46s docmost=352M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=896 +[11:33:17] + 61s docmost=353M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=1196 +[11:33:32] + 76s docmost=375M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=1496 +[11:33:47] + 91s docmost=373M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=1796 +[11:34:02] + 106s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=2092 +[11:34:17] + 121s docmost=372M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=2392 +[11:34:32] + 136s docmost=373M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=2692 +[11:34:47] + 152s docmost=372M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=2988 +[11:35:03] + 167s docmost=371M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=3288 +[11:35:18] + 182s docmost=370M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=3588 +[11:35:33] + 197s docmost=370M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=3888 +[11:35:48] + 212s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=4186 +[11:36:03] + 227s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=4484 +[11:36:18] + 242s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=4784 +[11:36:34] + 258s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=183M/256M peak=187M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=5084 +[11:36:49] + 273s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=185M/256M peak=188M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=5380 +[11:37:04] + 288s docmost=370M/384M peak=384M kills=0 rs=0 docmost-postgres=186M/256M peak=189M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=5680 +[11:37:19] + 303s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=187M/256M peak=189M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=5980 +[11:37:34] + 318s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=188M/256M peak=191M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=6280 +[11:37:49] + 334s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=189M/256M peak=191M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=6576 +[11:38:05] + 349s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=191M/256M peak=193M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=6876 +[11:38:20] + 364s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=194M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=7176 +[11:38:35] + 379s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=7476 +[11:38:50] + 394s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=194M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=7772 +[11:39:05] + 409s docmost=364M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=8072 +[11:39:20] + 424s docmost=364M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=8372 +[11:39:36] + 440s docmost=366M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=8672 +[11:39:51] + 455s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=8969 +[11:40:06] + 470s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=9269 +[11:40:21] + 485s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=9569 +[11:40:36] + 500s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=9869 +[11:40:51] + 515s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=10165 +[11:41:06] + 530s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=10465 +[11:41:22] + 546s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=10765 +[11:41:37] + 561s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=11065 +[11:41:52] + 576s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=11361 +[11:42:07] + 591s docmost=365M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=11661 +[11:42:22] + 606s docmost=366M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=11961 +[11:42:22] memory watch: killed=False tight=['docmost'] requests=11961 codes={'200': 11961} +[11:42:22] MV-docmost: ABORT — putting the FROM images back +[11:45:26] ABORT: the app did NOT come back (rc=1, 180.7s) +{ + "harness_version": 4, + "edge": "MV-docmost", + "app": "docmost", + "note": "night 2026-09-23 within-a-major move: docmost-postgres=postgres:18-alpine", + "from": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "to": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:18-alpine", + "docmost-redis": "redis:7-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "\u001b[2Kdocmost | {\"level\":\"info\",\"time\":\"2026-09-25T11:31:51.496Z\",\"pid\":45,\"hostname\":\"2ef0af245c57\",\"context\":\"DatabaseMigrationService\",\"msg\":\"No pending database migrations\"}", + "abort": "refuses", + "abort_detail": "26 13:36:34.064 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:36:34.064 * Background saving started by pid 191\n\u001b[2Kdocmost-redis | 191:C 25 Sep 2026 13:36:34.076 * DB saved on disk\n\u001b[2Kdocmost-redis | 191:C 25 Sep 2026 13:36:34.077 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:36:34.165 * Background saving terminated with success\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:41:35.035 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:41:35.036 * Background saving started by pid 374\n\u001b[2Kdocmost-redis | 374:C 25 Sep 2026 13:41:35.048 * DB saved on disk\n\u001b[2Kdocmost-redis | 374:C 25 Sep 2026 13:41:35.049 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:41:35.136 * Background saving terminated with success", + "engine_state_after": { + "docmost-postgres": { + "image": "postgres:18-alpine", + "probe": "datadir major version", + "answer": "18", + "probe_rc": 0 + } + }, + "memory": { + "soak_s": 606.6, + "requested_s": 600, + "requests": 11961, + "codes": { + "200": 11961 + }, + "first_kill": null, + "containers": { + "docmost": { + "limit": 402653184, + "peak": 402665472, + "peak_pct": 1.0, + "anon_peak_sampled": 366161920, + "anon_peak_pct": 0.909, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-postgres": { + "limit": 268435456, + "peak": 206811136, + "peak_pct": 0.77, + "anon_peak_sampled": 16371712, + "anon_peak_pct": 0.061, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-redis": { + "limit": 134217728, + "peak": 11501568, + "peak_pct": 0.086, + "anon_peak_sampled": 4087808, + "anon_peak_pct": 0.03, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + } + }, + "unmeasured": [], + "load": "reached" + }, + "marks": [ + "memory_tight" + ], + "duration_s": 31.1, + "measured_at": "2026-09-25T11:45:26Z", + "evidence": "evidence/MV-docmost", + "scratch_cleared": [], + "engine_conversion": { + "service": "docmost-postgres", + "engine": "postgres", + "from": 16, + "to": 18, + "dump_s": 0.54, + "dump_bytes": 132184, + "new_engine_ready_s": 2.2, + "load_s": 2.06, + "dropped": [ + "docmost" + ], + "skipped_create_role": [ + "CREATE ROLE docmost;" + ], + "result": "converted", + "check_equal": true, + "pg_version": "18", + "tables": 48, + "convert_s": 25.0 + }, + "files_changed": [], + "total_s": 950.2 +} diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/abort-states.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/abort-states.json new file mode 100644 index 00000000..f7a71c94 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/abort-states.json @@ -0,0 +1,8 @@ +{ + "privatebin": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/compose-final.log b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/compose-final.log new file mode 100644 index 00000000..bc19041f --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/compose-final.log @@ -0,0 +1,5 @@ +privatebin | [25-Sep-2026 13:29:05] NOTICE: fpm is running, pid 10 +privatebin | [25-Sep-2026 13:29:05] NOTICE: ready to handle connections +privatebin | 127.0.0.1 - - [25/Sep/2026:13:29:35 +0200] "GET / HTTP/1.1" 200 19664 "-" "Wget" "-" +privatebin | 172.18.0.1 - - [25/Sep/2026:13:29:36 +0200] "GET / HTTP/1.1" 200 22914 "-" "curl/8.14.1" "-" +privatebin | 172.18.0.1 - - [25/Sep/2026:13:29:36 +0200] "GET /?pasteid=6d06e345497ece79 HTTP/1.1" 200 310 "-" "curl/8.14.1" "-" diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/engine-state.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/engine-state.json new file mode 100644 index 00000000..ec747fa4 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/engine-state.json @@ -0,0 +1 @@ +null \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-after.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-after.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-after.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-before.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-before.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/files-before.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/migration-lines.txt b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/migration-lines.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/run.log b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/run.log new file mode 100644 index 00000000..3a511548 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/run.log @@ -0,0 +1,14 @@ +[11:21:19] scratch drive folders cleared before FROM (R-656): none existed +[11:21:19] C3: deploying privatebin at FROM {'privatebin': 'privatebin/pdo:2.0.5'} +[11:21:56] FROM settled=True in 30.8s :: {"privatebin": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:21:56] privatebin: seeded paste id=6d06e345497ece79 +[11:21:56] privatebin: readback http=200 marker_present=True +[11:21:56] C1 (seed reads back BEFORE): True +[11:21:56] C3: swapping to TO {'privatebin': 'alpine:3.20'} +[11:22:00] TO up -d rc=0 +[11:29:04] TO settled=False in 424.4s :: {"privatebin": {"status": "restarting", "health": "unhealthy", "restarts": 0, "exit": 0}} +[11:29:04] migration lines observed: 0 +[11:29:04] RESULT (seed reads back AFTER): False +[11:29:05] C3: ABORT — putting the FROM images back +[11:29:36] privatebin: readback http=200 marker_present=True +[11:29:36] ABORT: app came back in 30.8s; data present=True \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/to-full.log b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/to-full.log new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/to-states.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/to-states.json new file mode 100644 index 00000000..04f4a178 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/to-states.json @@ -0,0 +1,8 @@ +{ + "privatebin": { + "status": "restarting", + "health": "unhealthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/verdict.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/verdict.json new file mode 100644 index 00000000..e1d29e6c --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/C3/verdict.json @@ -0,0 +1,28 @@ +{ + "harness_version": 4, + "edge": "C3", + "app": "privatebin", + "note": "NEGATIVE control: the TO image starts and exits immediately", + "from": { + "privatebin": "privatebin/pdo:2.0.5" + }, + "to": { + "privatebin": "alpine:3.20" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "starts-and-serves", + "abort_detail": null, + "engine_state_after": null, + "memory": null, + "marks": [], + "duration_s": 424.4, + "measured_at": "2026-09-25T11:29:36Z", + "evidence": "evidence/C3", + "scratch_cleared": [], + "files_changed": [], + "total_s": 496.4 +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-refusal.txt b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-refusal.txt new file mode 100644 index 00000000..d9a8695f --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-refusal.txt @@ -0,0 +1,80 @@ +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-redis | 1:M 25 Sep 2026 13:30:25.238 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:30:25.239 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:30:25.240 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:30:25.244 * Creating AOF base file appendonly.aof.1.base.rdb on server start +docmost-redis | 1:M 25 Sep 2026 13:30:25.251 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:30:25.251 * Ready to accept connections tcp +docmost-redis | 1:signal-handler (1790335869) Received SIGTERM scheduling shutdown... +docmost-redis | 1:M 25 Sep 2026 13:31:09.303 * User requested shutdown... +docmost-redis | 1:M 25 Sep 2026 13:31:09.303 * Calling fsync() on the AOF file. +docmost-redis | 1:M 25 Sep 2026 13:31:09.322 * Saving the final RDB snapshot before exiting. +docmost-redis | 1:M 25 Sep 2026 13:31:09.334 * DB saved on disk +docmost-redis | 1:M 25 Sep 2026 13:31:09.335 # Redis is now ready to exit, bye bye... +docmost-redis | 1:C 25 Sep 2026 13:31:33.400 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:31:33.401 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:31:33.402 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:31:33.403 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:31:33.403 * Reading RDB base file on AOF loading... +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * Loading RDB produced by version 7.4.11 +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB age 68 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB memory usage when created 0.90 Mb +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB is base AOF +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * Done loading RDB, keys loaded: 0, keys expired: 0. +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * DB loaded from base file appendonly.aof.1.base.rdb: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * DB loaded from append only file: 0.002 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * Opening AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * Ready to accept connections tcp +docmost-redis | 1:M 25 Sep 2026 13:36:34.064 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:36:34.064 * Background saving started by pid 191 +docmost-redis | 191:C 25 Sep 2026 13:36:34.076 * DB saved on disk +docmost-redis | 191:C 25 Sep 2026 13:36:34.077 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:36:34.165 * Background saving terminated with success +docmost-redis | 1:M 25 Sep 2026 13:41:35.035 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:41:35.036 * Background saving started by pid 374 +docmost-redis | 374:C 25 Sep 2026 13:41:35.048 * DB saved on disk +docmost-redis | 374:C 25 Sep 2026 13:41:35.049 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:41:35.136 * Background saving terminated with success \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-states.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-states.json new file mode 100644 index 00000000..25094113 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/abort-states.json @@ -0,0 +1,20 @@ +{ + "docmost": { + "status": "created", + "health": null, + "restarts": 0, + "exit": 0 + }, + "docmost-postgres": { + "status": "restarting", + "health": "unhealthy", + "restarts": 0, + "exit": 1 + }, + "docmost-redis": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/compose-final.log b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/compose-final.log new file mode 100644 index 00000000..9f892479 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/compose-final.log @@ -0,0 +1,176 @@ +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-redis | 1:C 25 Sep 2026 13:30:25.237 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:30:25.238 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:30:25.238 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:30:25.238 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:30:25.238 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:30:25.239 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:30:25.240 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:30:25.244 * Creating AOF base file appendonly.aof.1.base.rdb on server start +docmost-postgres | +docmost-redis | 1:M 25 Sep 2026 13:30:25.251 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:30:25.251 * Ready to accept connections tcp +docmost-redis | 1:signal-handler (1790335869) Received SIGTERM scheduling shutdown... +docmost-redis | 1:M 25 Sep 2026 13:31:09.303 * User requested shutdown... +docmost-redis | 1:M 25 Sep 2026 13:31:09.303 * Calling fsync() on the AOF file. +docmost-redis | 1:M 25 Sep 2026 13:31:09.322 * Saving the final RDB snapshot before exiting. +docmost-redis | 1:M 25 Sep 2026 13:31:09.334 * DB saved on disk +docmost-redis | 1:M 25 Sep 2026 13:31:09.335 # Redis is now ready to exit, bye bye... +docmost-redis | 1:C 25 Sep 2026 13:31:33.400 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:31:33.401 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:31:33.402 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:31:33.403 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:31:33.403 * Reading RDB base file on AOF loading... +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * Loading RDB produced by version 7.4.11 +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB age 68 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB memory usage when created 0.90 Mb +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB is base AOF +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * Done loading RDB, keys loaded: 0, keys expired: 0. +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * DB loaded from base file appendonly.aof.1.base.rdb: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * DB loaded from append only file: 0.002 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * Opening AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * Ready to accept connections tcp +docmost-redis | 1:M 25 Sep 2026 13:36:34.064 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:36:34.064 * Background saving started by pid 191 +docmost-redis | 191:C 25 Sep 2026 13:36:34.076 * DB saved on disk +docmost-redis | 191:C 25 Sep 2026 13:36:34.077 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:36:34.165 * Background saving terminated with success +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-redis | 1:M 25 Sep 2026 13:41:35.035 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:41:35.036 * Background saving started by pid 374 +docmost-redis | 374:C 25 Sep 2026 13:41:35.048 * DB saved on disk +docmost-redis | 374:C 25 Sep 2026 13:41:35.049 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:41:35.136 * Background saving terminated with success +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-after.txt b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-after.txt new file mode 100644 index 00000000..ab236c2a --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-after.txt @@ -0,0 +1,54 @@ +db:docmost owner=docmost enc=UTF8 coll=en_US.utf8 +db:postgres owner=docmost enc=UTF8 coll=en_US.utf8 +ext:docmost:pg_trgm +ext:docmost:plpgsql +ext:docmost:unaccent +role:docmost super=true login=true pw=true +rows:docmost:public.ai_chat_messages=0 +rows:docmost:public.ai_chats=0 +rows:docmost:public.api_keys=0 +rows:docmost:public.attachments=0 +rows:docmost:public.audit=0 +rows:docmost:public.auth_accounts=0 +rows:docmost:public.auth_providers=0 +rows:docmost:public.backlinks=0 +rows:docmost:public.base_properties=0 +rows:docmost:public.base_rows=0 +rows:docmost:public.base_views=0 +rows:docmost:public.billing=0 +rows:docmost:public.comments=0 +rows:docmost:public.favorites=0 +rows:docmost:public.file_tasks=0 +rows:docmost:public.group_users=1 +rows:docmost:public.groups=1 +rows:docmost:public.kysely_migration=52 +rows:docmost:public.kysely_migration_lock=1 +rows:docmost:public.labels=0 +rows:docmost:public.notifications=0 +rows:docmost:public.oauth_authorization_codes=0 +rows:docmost:public.oauth_clients=0 +rows:docmost:public.oauth_grants=0 +rows:docmost:public.oauth_tokens=0 +rows:docmost:public.page_access=0 +rows:docmost:public.page_history=0 +rows:docmost:public.page_labels=0 +rows:docmost:public.page_permissions=0 +rows:docmost:public.page_transclusion_references=0 +rows:docmost:public.page_transclusions=0 +rows:docmost:public.page_verifications=0 +rows:docmost:public.page_verifiers=0 +rows:docmost:public.pages=0 +rows:docmost:public.public_spaces=0 +rows:docmost:public.scim_tokens=0 +rows:docmost:public.shares=0 +rows:docmost:public.siem_destinations=0 +rows:docmost:public.space_members=2 +rows:docmost:public.spaces=1 +rows:docmost:public.templates=0 +rows:docmost:public.user_mfa=0 +rows:docmost:public.user_sessions=2 +rows:docmost:public.user_tokens=0 +rows:docmost:public.users=1 +rows:docmost:public.watchers=0 +rows:docmost:public.workspace_invitations=0 +rows:docmost:public.workspaces=1 diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-before.txt b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-before.txt new file mode 100644 index 00000000..ab236c2a --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-check-before.txt @@ -0,0 +1,54 @@ +db:docmost owner=docmost enc=UTF8 coll=en_US.utf8 +db:postgres owner=docmost enc=UTF8 coll=en_US.utf8 +ext:docmost:pg_trgm +ext:docmost:plpgsql +ext:docmost:unaccent +role:docmost super=true login=true pw=true +rows:docmost:public.ai_chat_messages=0 +rows:docmost:public.ai_chats=0 +rows:docmost:public.api_keys=0 +rows:docmost:public.attachments=0 +rows:docmost:public.audit=0 +rows:docmost:public.auth_accounts=0 +rows:docmost:public.auth_providers=0 +rows:docmost:public.backlinks=0 +rows:docmost:public.base_properties=0 +rows:docmost:public.base_rows=0 +rows:docmost:public.base_views=0 +rows:docmost:public.billing=0 +rows:docmost:public.comments=0 +rows:docmost:public.favorites=0 +rows:docmost:public.file_tasks=0 +rows:docmost:public.group_users=1 +rows:docmost:public.groups=1 +rows:docmost:public.kysely_migration=52 +rows:docmost:public.kysely_migration_lock=1 +rows:docmost:public.labels=0 +rows:docmost:public.notifications=0 +rows:docmost:public.oauth_authorization_codes=0 +rows:docmost:public.oauth_clients=0 +rows:docmost:public.oauth_grants=0 +rows:docmost:public.oauth_tokens=0 +rows:docmost:public.page_access=0 +rows:docmost:public.page_history=0 +rows:docmost:public.page_labels=0 +rows:docmost:public.page_permissions=0 +rows:docmost:public.page_transclusion_references=0 +rows:docmost:public.page_transclusions=0 +rows:docmost:public.page_verifications=0 +rows:docmost:public.page_verifiers=0 +rows:docmost:public.pages=0 +rows:docmost:public.public_spaces=0 +rows:docmost:public.scim_tokens=0 +rows:docmost:public.shares=0 +rows:docmost:public.siem_destinations=0 +rows:docmost:public.space_members=2 +rows:docmost:public.spaces=1 +rows:docmost:public.templates=0 +rows:docmost:public.user_mfa=0 +rows:docmost:public.user_sessions=2 +rows:docmost:public.user_tokens=0 +rows:docmost:public.users=1 +rows:docmost:public.watchers=0 +rows:docmost:public.workspace_invitations=0 +rows:docmost:public.workspaces=1 diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-load.err b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/convert-load.err new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/engine-state.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/engine-state.json new file mode 100644 index 00000000..6c38c07a --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/engine-state.json @@ -0,0 +1,8 @@ +{ + "docmost-postgres": { + "image": "postgres:18-alpine", + "probe": "datadir major version", + "answer": "18", + "probe_rc": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-after.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-after.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-after.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-before.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-before.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/files-before.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/memory-samples.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/memory-samples.json new file mode 100644 index 00000000..bd58b70f --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/memory-samples.json @@ -0,0 +1,1562 @@ +[ + { + "t": 15.2, + "containers": { + "docmost": { + "limit": 402653184, + "current": 358961152, + "peak": 402665472, + "anon": 331591680, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191037440, + "peak": 195977216, + "anon": 15458304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7983104, + "peak": 10334208, + "anon": 4083712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 300 + }, + { + "t": 30.3, + "containers": { + "docmost": { + "limit": 402653184, + "current": 363266048, + "peak": 402665472, + "anon": 335933440, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191074304, + "peak": 195977216, + "anon": 15462400, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8146944, + "peak": 11079680, + "anon": 3997696, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 600 + }, + { + "t": 45.5, + "containers": { + "docmost": { + "limit": 402653184, + "current": 369631232, + "peak": 402665472, + "anon": 342032384, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191221760, + "peak": 195977216, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7700480, + "peak": 11079680, + "anon": 4030464, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 896 + }, + { + "t": 60.6, + "containers": { + "docmost": { + "limit": 402653184, + "current": 370556928, + "peak": 402665472, + "anon": 343056384, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191156224, + "peak": 195977216, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8196096, + "peak": 11079680, + "anon": 4030464, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1196 + }, + { + "t": 75.8, + "containers": { + "docmost": { + "limit": 402653184, + "current": 394047488, + "peak": 402665472, + "anon": 366161920, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191406080, + "peak": 195977216, + "anon": 15482880, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8187904, + "peak": 11079680, + "anon": 4034560, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1496 + }, + { + "t": 91.0, + "containers": { + "docmost": { + "limit": 402653184, + "current": 391593984, + "peak": 402665472, + "anon": 363929600, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191488000, + "peak": 195977216, + "anon": 15474688, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7950336, + "peak": 11079680, + "anon": 4038656, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1796 + }, + { + "t": 106.1, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386625536, + "peak": 402665472, + "anon": 359018496, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191488000, + "peak": 196571136, + "anon": 15474688, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7712768, + "peak": 11079680, + "anon": 4030464, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2092 + }, + { + "t": 121.3, + "containers": { + "docmost": { + "limit": 402653184, + "current": 390475776, + "peak": 402665472, + "anon": 362840064, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191516672, + "peak": 196571136, + "anon": 15474688, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8187904, + "peak": 11079680, + "anon": 4050944, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2392 + }, + { + "t": 136.4, + "containers": { + "docmost": { + "limit": 402653184, + "current": 391278592, + "peak": 402665472, + "anon": 362872832, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191721472, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8167424, + "peak": 11079680, + "anon": 4026368, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2692 + }, + { + "t": 151.6, + "containers": { + "docmost": { + "limit": 402653184, + "current": 390774784, + "peak": 402665472, + "anon": 362897408, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191721472, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8482816, + "peak": 11079680, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2988 + }, + { + "t": 166.7, + "containers": { + "docmost": { + "limit": 402653184, + "current": 389210112, + "peak": 402665472, + "anon": 361361408, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191823872, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7741440, + "peak": 11079680, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3288 + }, + { + "t": 181.9, + "containers": { + "docmost": { + "limit": 402653184, + "current": 388227072, + "peak": 402665472, + "anon": 361099264, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191721472, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8458240, + "peak": 11079680, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3588 + }, + { + "t": 197.1, + "containers": { + "docmost": { + "limit": 402653184, + "current": 388800512, + "peak": 402665472, + "anon": 360652800, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191721472, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7974912, + "peak": 11079680, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3888 + }, + { + "t": 212.2, + "containers": { + "docmost": { + "limit": 402653184, + "current": 385859584, + "peak": 402665472, + "anon": 358563840, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191725568, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8245248, + "peak": 11124736, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4186 + }, + { + "t": 227.4, + "containers": { + "docmost": { + "limit": 402653184, + "current": 387112960, + "peak": 402665472, + "anon": 359235584, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191725568, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8249344, + "peak": 11124736, + "anon": 4059136, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4484 + }, + { + "t": 242.5, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386764800, + "peak": 402665472, + "anon": 358932480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 191721472, + "peak": 196571136, + "anon": 15478784, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7970816, + "peak": 11124736, + "anon": 4067328, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4784 + }, + { + "t": 257.7, + "containers": { + "docmost": { + "limit": 402653184, + "current": 385994752, + "peak": 402665472, + "anon": 358510592, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 192765952, + "peak": 196571136, + "anon": 15548416, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8830976, + "peak": 11124736, + "anon": 4067328, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5084 + }, + { + "t": 272.8, + "containers": { + "docmost": { + "limit": 402653184, + "current": 387620864, + "peak": 402665472, + "anon": 360202240, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 194117632, + "peak": 197398528, + "anon": 15548416, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7983104, + "peak": 11124736, + "anon": 4067328, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5380 + }, + { + "t": 288.0, + "containers": { + "docmost": { + "limit": 402653184, + "current": 388128768, + "peak": 402665472, + "anon": 360255488, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 195604480, + "peak": 198250496, + "anon": 15552512, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8232960, + "peak": 11124736, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5680 + }, + { + "t": 303.1, + "containers": { + "docmost": { + "limit": 402653184, + "current": 387842048, + "peak": 402665472, + "anon": 360275968, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 196911104, + "peak": 199053312, + "anon": 15568896, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8073216, + "peak": 11124736, + "anon": 4071424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5980 + }, + { + "t": 318.3, + "containers": { + "docmost": { + "limit": 402653184, + "current": 387100672, + "peak": 402665472, + "anon": 359510016, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 197926912, + "peak": 200437760, + "anon": 15568896, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8028160, + "peak": 11124736, + "anon": 4059136, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6280 + }, + { + "t": 333.5, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386203648, + "peak": 402665472, + "anon": 358793216, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 199180288, + "peak": 200937472, + "anon": 15572992, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8314880, + "peak": 11124736, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6576 + }, + { + "t": 348.6, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386240512, + "peak": 402665472, + "anon": 358723584, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 200937472, + "peak": 203296768, + "anon": 15589376, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8318976, + "peak": 11124736, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6876 + }, + { + "t": 363.8, + "containers": { + "docmost": { + "limit": 402653184, + "current": 385441792, + "peak": 402665472, + "anon": 357896192, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202399744, + "peak": 204136448, + "anon": 15605760, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7782400, + "peak": 11124736, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7176 + }, + { + "t": 378.9, + "containers": { + "docmost": { + "limit": 402653184, + "current": 385880064, + "peak": 402665472, + "anon": 358830080, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202227712, + "peak": 205582336, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8343552, + "peak": 11124736, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7476 + }, + { + "t": 394.1, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386682880, + "peak": 402665472, + "anon": 358858752, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 203714560, + "peak": 205582336, + "anon": 16371712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8339456, + "peak": 11124736, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7772 + }, + { + "t": 409.3, + "containers": { + "docmost": { + "limit": 402653184, + "current": 382152704, + "peak": 402665472, + "anon": 355229696, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202215424, + "peak": 205582336, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8085504, + "peak": 11124736, + "anon": 4067328, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8072 + }, + { + "t": 424.4, + "containers": { + "docmost": { + "limit": 402653184, + "current": 382668800, + "peak": 402665472, + "anon": 355229696, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202579968, + "peak": 205582336, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 7819264, + "peak": 11124736, + "anon": 4050944, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8372 + }, + { + "t": 439.6, + "containers": { + "docmost": { + "limit": 402653184, + "current": 384790528, + "peak": 402665472, + "anon": 356646912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202207232, + "peak": 205582336, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8364032, + "peak": 11501568, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8672 + }, + { + "t": 454.8, + "containers": { + "docmost": { + "limit": 402653184, + "current": 384864256, + "peak": 402665472, + "anon": 357343232, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202305536, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8605696, + "peak": 11501568, + "anon": 4059136, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8969 + }, + { + "t": 469.9, + "containers": { + "docmost": { + "limit": 402653184, + "current": 387006464, + "peak": 402665472, + "anon": 359481344, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202461184, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8376320, + "peak": 11501568, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9269 + }, + { + "t": 485.1, + "containers": { + "docmost": { + "limit": 402653184, + "current": 387362816, + "peak": 402665472, + "anon": 359534592, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202211328, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8118272, + "peak": 11501568, + "anon": 4071424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9569 + }, + { + "t": 500.2, + "containers": { + "docmost": { + "limit": 402653184, + "current": 385798144, + "peak": 402665472, + "anon": 358772736, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202207232, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8069120, + "peak": 11501568, + "anon": 4083712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9869 + }, + { + "t": 515.4, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386527232, + "peak": 402665472, + "anon": 358744064, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202186752, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8130560, + "peak": 11501568, + "anon": 4071424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10165 + }, + { + "t": 530.5, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386060288, + "peak": 402665472, + "anon": 358768640, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202170368, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8228864, + "peak": 11501568, + "anon": 4087808, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10465 + }, + { + "t": 545.7, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386818048, + "peak": 402665472, + "anon": 359260160, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202489856, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8155136, + "peak": 11501568, + "anon": 4083712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10765 + }, + { + "t": 560.9, + "containers": { + "docmost": { + "limit": 402653184, + "current": 385470464, + "peak": 402665472, + "anon": 357699584, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202170368, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8183808, + "peak": 11501568, + "anon": 4087808, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11065 + }, + { + "t": 576.0, + "containers": { + "docmost": { + "limit": 402653184, + "current": 386740224, + "peak": 402665472, + "anon": 359219200, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202170368, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8433664, + "peak": 11501568, + "anon": 4083712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11361 + }, + { + "t": 591.2, + "containers": { + "docmost": { + "limit": 402653184, + "current": 382836736, + "peak": 402665472, + "anon": 355418112, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202170368, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8187904, + "peak": 11501568, + "anon": 4087808, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11661 + }, + { + "t": 606.3, + "containers": { + "docmost": { + "limit": 402653184, + "current": 384700416, + "peak": 402665472, + "anon": 357015552, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 202223616, + "peak": 206811136, + "anon": 15671296, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 8437760, + "peak": 11501568, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11961 + } +] \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/migration-lines.txt b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/migration-lines.txt new file mode 100644 index 00000000..b67bbc2e --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/migration-lines.txt @@ -0,0 +1 @@ +docmost | {"level":"info","time":"2026-09-25T11:31:51.496Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"No pending database migrations"} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/run.log b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/run.log new file mode 100644 index 00000000..c1f34436 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/run.log @@ -0,0 +1,59 @@ +[11:29:36] scratch drive folders cleared before FROM (R-656): none existed +[11:29:36] MV-docmost: deploying docmost at FROM {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:31:07] FROM settled=True in 31.1s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:31:07] docmost: /api/auth/setup http=200 rc=0 +[11:31:07] docmost: login as the seeded user http=200 ok=True +[11:31:07] C1 (seed reads back BEFORE): True +[11:31:08] MV-docmost: PostgreSQL major move {'service': 'docmost-postgres', 'engine': 'postgres', 'from': 16, 'to': 18} — converting on the bench before the swap +[11:31:33] CONVERTED docmost-postgres PostgreSQL 16 -> 18 in 25.0s (dump 0.54s / 132184 B, load 2.06s, check equal over 48 tables) +[11:31:33] MV-docmost: swapping to TO {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:31:44] TO up -d rc=0 +[11:32:15] TO settled=True in 31.1s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:32:15] engine state docmost-postgres: 18 +[11:32:15] migration lines observed: 1 +[11:32:15] docmost: login as the seeded user http=200 ok=True +[11:32:15] RESULT (seed reads back AFTER): True +[11:32:16] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:3000 +[11:32:31] + 15s docmost=342M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=9M kills=0 rs=0 reqs=300 +[11:32:46] + 30s docmost=346M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=600 +[11:33:01] + 46s docmost=352M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=896 +[11:33:17] + 61s docmost=353M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=1196 +[11:33:32] + 76s docmost=375M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=1496 +[11:33:47] + 91s docmost=373M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=186M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=1796 +[11:34:02] + 106s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=2092 +[11:34:17] + 121s docmost=372M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=2392 +[11:34:32] + 136s docmost=373M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=2692 +[11:34:47] + 152s docmost=372M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=2988 +[11:35:03] + 167s docmost=371M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=3288 +[11:35:18] + 182s docmost=370M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=3588 +[11:35:33] + 197s docmost=370M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=3888 +[11:35:48] + 212s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=4186 +[11:36:03] + 227s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=4484 +[11:36:18] + 242s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=182M/256M peak=187M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=4784 +[11:36:34] + 258s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=183M/256M peak=187M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=5084 +[11:36:49] + 273s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=185M/256M peak=188M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=5380 +[11:37:04] + 288s docmost=370M/384M peak=384M kills=0 rs=0 docmost-postgres=186M/256M peak=189M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=5680 +[11:37:19] + 303s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=187M/256M peak=189M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=5980 +[11:37:34] + 318s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=188M/256M peak=191M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=6280 +[11:37:49] + 334s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=189M/256M peak=191M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=6576 +[11:38:05] + 349s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=191M/256M peak=193M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=6876 +[11:38:20] + 364s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=194M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=7176 +[11:38:35] + 379s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=7476 +[11:38:50] + 394s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=194M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=7772 +[11:39:05] + 409s docmost=364M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=8072 +[11:39:20] + 424s docmost=364M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=8372 +[11:39:36] + 440s docmost=366M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=196M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=8672 +[11:39:51] + 455s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=8969 +[11:40:06] + 470s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=9269 +[11:40:21] + 485s docmost=369M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=9569 +[11:40:36] + 500s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=9869 +[11:40:51] + 515s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=10165 +[11:41:06] + 530s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=10465 +[11:41:22] + 546s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=193M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=10765 +[11:41:37] + 561s docmost=367M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=11065 +[11:41:52] + 576s docmost=368M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=11361 +[11:42:07] + 591s docmost=365M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=7M/128M peak=10M kills=0 rs=0 reqs=11661 +[11:42:22] + 606s docmost=366M/384M peak=384M kills=0 rs=0 docmost-postgres=192M/256M peak=197M kills=0 rs=0 docmost-redis=8M/128M peak=10M kills=0 rs=0 reqs=11961 +[11:42:22] memory watch: killed=False tight=['docmost'] requests=11961 codes={'200': 11961} +[11:42:22] MV-docmost: ABORT — putting the FROM images back +[11:45:26] ABORT: the app did NOT come back (rc=1, 180.7s) \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-full.log b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-full.log new file mode 100644 index 00000000..ea11ba65 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-full.log @@ -0,0 +1,166 @@ +docmost-redis | 1:C 25 Sep 2026 13:30:25.237 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:30:25.238 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:30:25.238 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:30:25.238 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:30:25.238 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:30:25.239 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:30:25.240 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:30:25.244 * Creating AOF base file appendonly.aof.1.base.rdb on server start +docmost-redis | 1:M 25 Sep 2026 13:30:25.251 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:30:25.251 * Ready to accept connections tcp +docmost-redis | 1:signal-handler (1790335869) Received SIGTERM scheduling shutdown... +docmost-redis | 1:M 25 Sep 2026 13:31:09.303 * User requested shutdown... +docmost-redis | 1:M 25 Sep 2026 13:31:09.303 * Calling fsync() on the AOF file. +docmost-redis | 1:M 25 Sep 2026 13:31:09.322 * Saving the final RDB snapshot before exiting. +docmost-redis | 1:M 25 Sep 2026 13:31:09.334 * DB saved on disk +docmost-redis | 1:M 25 Sep 2026 13:31:09.335 # Redis is now ready to exit, bye bye... +docmost-redis | 1:C 25 Sep 2026 13:31:33.400 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:31:33.401 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:31:33.401 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:31:33.402 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:31:33.403 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:31:33.403 * Reading RDB base file on AOF loading... +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * Loading RDB produced by version 7.4.11 +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB age 68 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB memory usage when created 0.90 Mb +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * RDB is base AOF +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * Done loading RDB, keys loaded: 0, keys expired: 0. +docmost-redis | 1:M 25 Sep 2026 13:31:33.404 * DB loaded from base file appendonly.aof.1.base.rdb: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * DB loaded from append only file: 0.002 seconds +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * Opening AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:31:33.406 * Ready to accept connections tcp +docmost | $ pnpm --filter ./apps/server run start:prod +docmost | $ cross-env NODE_ENV=production node dist/main +docmost | (node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided. +docmost | (Use `node --trace-warnings ...` to show where the warning was created) +docmost | {"level":"info","time":"2026-09-25T11:30:42.747Z","pid":45,"hostname":"2ef0af245c57","context":"RedisModule","msg":"default: the connection was successfully established"} +docmost | {"level":"info","time":"2026-09-25T11:30:43.084Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseModule","msg":"Establishing database connection"} +docmost | {"level":"info","time":"2026-09-25T11:30:43.111Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseModule","msg":"Database connection successful"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.842Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T085400-uuid_v7_fn\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T085500-workspaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T085600-users\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T085700-groups\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T085900-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T086100-add-workspace-fk\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T086200-workspace_invitations\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T086300-pages\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T086400-page_history\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T086600-comments\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T086700-attachments\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240324T086800-pages-tsvector-trigger\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240707T145623-drop-redundant-pages-slug_id-index\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20240903T124647-user-tokens\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20241218T223249-backlinks\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250106T195516-billing\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250118T194658-sso-auth\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250222T114520-add_license_key_to_workspace\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250327T145832-add-contributorIds-to-pages\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250408T191830-shares\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250521T154949-file_tasks\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250623T215045-more-billing-columns\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250715T070817-mfa\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250725T052004-add-new-comments-columns\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250729T213756-add-unaccent-pg_trm-update-tsvector.\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250831T191600-add-group-sync-to-auth-providers\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250831T202306-ldap-auth\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250901T184612-attachments-search\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20250912T101500-api-keys\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260205T214213-add-settings-to-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260209T120000-add-contributor_ids-to-page-history\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260213T085259-notifications\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260213T085320-watchers\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260213T085337-backfill-watchers\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260224T233803-page-permissions\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260228T223532-audit\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260326T121350-user-sessions\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.843Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260329T163516-add-new-indexes\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260409T132415-ai-chat\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260412T135891-templates\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260412T162318-favorites\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260413T121647-page-verifications\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260414T124451-update-file_tasks\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260501T092214-scim\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260501T202258-page-transclusions\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260509T121236-labels\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260529T125146-bases\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260620T010047-personal-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260824T211732-page-title-trgm-index\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260825T022612-oauth\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260902T121326-siem-destinations\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.844Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"Migration \"20260904T171920-public-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.918Z","pid":45,"hostname":"2ef0af245c57","context":"NestApplication","msg":"Nest application successfully started"} +docmost | {"level":"info","time":"2026-09-25T11:30:44.932Z","pid":45,"hostname":"2ef0af245c57","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://docs.gate.invalid"} +docmost | [ELIFECYCLE] Command failed. +docmost | $ pnpm --filter ./apps/server run start:prod +docmost | $ cross-env NODE_ENV=production node dist/main +docmost | (node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided. +docmost | (Use `node --trace-warnings ...` to show where the warning was created) +docmost | {"level":"info","time":"2026-09-25T11:31:51.005Z","pid":45,"hostname":"2ef0af245c57","context":"RedisModule","msg":"default: the connection was successfully established"} +docmost | {"level":"info","time":"2026-09-25T11:31:51.300Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseModule","msg":"Establishing database connection"} +docmost | {"level":"info","time":"2026-09-25T11:31:51.355Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseModule","msg":"Database connection successful"} +docmost | {"level":"info","time":"2026-09-25T11:31:51.496Z","pid":45,"hostname":"2ef0af245c57","context":"DatabaseMigrationService","msg":"No pending database migrations"} +docmost | {"level":"info","time":"2026-09-25T11:31:51.537Z","pid":45,"hostname":"2ef0af245c57","context":"NestApplication","msg":"Nest application successfully started"} +docmost | {"level":"info","time":"2026-09-25T11:31:51.548Z","pid":45,"hostname":"2ef0af245c57","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://docs.gate.invalid"} +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are enabled. +docmost-postgres | +docmost-postgres | fixing permissions on existing directory /var/lib/postgresql/18/docker ... ok +docmost-postgres | creating subdirectories ... ok +docmost-postgres | selecting dynamic shared memory implementation ... posix +docmost-postgres | selecting default "max_connections" ... 100 +docmost-postgres | selecting default "shared_buffers" ... 128MB +docmost-postgres | selecting default time zone ... Europe/Budapest +docmost-postgres | creating configuration files ... ok +docmost-postgres | running bootstrap script ... ok +docmost-postgres | sh: locale: not found +docmost-postgres | 2026-09-25 13:31:28.303 CEST [47] WARNING: no usable system locales were found +docmost-postgres | performing post-bootstrap initialization ... ok +docmost-postgres | syncing data to disk ... ok +docmost-postgres | +docmost-postgres | +docmost-postgres | Success. You can now start the database server using: +docmost-postgres | +docmost-postgres | pg_ctl -D /var/lib/postgresql/18/docker -l logfile start +docmost-postgres | +docmost-postgres | initdb: warning: enabling "trust" authentication for local connections +docmost-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. +docmost-postgres | waiting for server to start....2026-09-25 13:31:29.431 CEST [53] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +docmost-postgres | 2026-09-25 13:31:29.434 CEST [53] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +docmost-postgres | 2026-09-25 13:31:29.448 CEST [59] LOG: database system was shut down at 2026-09-25 13:31:28 CEST +docmost-postgres | 2026-09-25 13:31:29.458 CEST [53] LOG: database system is ready to accept connections +docmost-postgres | done +docmost-postgres | server started +docmost-postgres | CREATE DATABASE +docmost-postgres | +docmost-postgres | +docmost-postgres | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* +docmost-postgres | +docmost-postgres | waiting for server to shut down....2026-09-25 13:31:29.656 CEST [53] LOG: received fast shutdown request +docmost-postgres | 2026-09-25 13:31:29.660 CEST [53] LOG: aborting any active transactions +docmost-postgres | 2026-09-25 13:31:29.666 CEST [53] LOG: background worker "logical replication launcher" (PID 62) exited with exit code 1 +docmost-postgres | 2026-09-25 13:31:29.666 CEST [57] LOG: shutting down +docmost-postgres | 2026-09-25 13:31:29.669 CEST [57] LOG: checkpoint starting: shutdown immediate +docmost-postgres | 2026-09-25 13:31:29.813 CEST [57] LOG: checkpoint complete: wrote 943 buffers (5.8%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.024 s, sync=0.107 s, total=0.147 s; sync files=303, longest=0.075 s, average=0.001 s; distance=4362 kB, estimate=4362 kB; lsn=0/1BA8858, redo lsn=0/1BA8858 +docmost-postgres | 2026-09-25 13:31:29.847 CEST [53] LOG: database system is shut down +docmost-postgres | done +docmost-postgres | server stopped +docmost-postgres | +docmost-postgres | PostgreSQL init process complete; ready for start up. +docmost-postgres | +docmost-postgres | 2026-09-25 13:31:29.900 CEST [1] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +docmost-postgres | 2026-09-25 13:31:29.900 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +docmost-postgres | 2026-09-25 13:31:29.900 CEST [1] LOG: listening on IPv6 address "::", port 5432 +docmost-postgres | 2026-09-25 13:31:29.907 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +docmost-postgres | 2026-09-25 13:31:29.918 CEST [81] LOG: database system was shut down at 2026-09-25 13:31:29 CEST +docmost-postgres | 2026-09-25 13:31:29.927 CEST [1] LOG: database system is ready to accept connections +docmost-postgres | 2026-09-25 13:31:30.254 CEST [79] LOG: checkpoint starting: immediate force wait +docmost-postgres | 2026-09-25 13:31:30.286 CEST [79] LOG: checkpoint complete: wrote 1 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.008 s, sync=0.006 s, total=0.032 s; sync files=3, longest=0.003 s, average=0.002 s; distance=1 kB, estimate=1 kB; lsn=0/1BA8D60, redo lsn=0/1BA8D08 diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-states.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-states.json new file mode 100644 index 00000000..c6bae4a5 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/to-states.json @@ -0,0 +1,20 @@ +{ + "docmost": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "docmost-postgres": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "docmost-redis": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/verdict.json b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/verdict.json new file mode 100644 index 00000000..056f83ed --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run1/evidence/MV-docmost/verdict.json @@ -0,0 +1,107 @@ +{ + "harness_version": 4, + "edge": "MV-docmost", + "app": "docmost", + "note": "night 2026-09-23 within-a-major move: docmost-postgres=postgres:18-alpine", + "from": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "to": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:18-alpine", + "docmost-redis": "redis:7-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "\u001b[2Kdocmost | {\"level\":\"info\",\"time\":\"2026-09-25T11:31:51.496Z\",\"pid\":45,\"hostname\":\"2ef0af245c57\",\"context\":\"DatabaseMigrationService\",\"msg\":\"No pending database migrations\"}", + "abort": "refuses", + "abort_detail": "26 13:36:34.064 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:36:34.064 * Background saving started by pid 191\n\u001b[2Kdocmost-redis | 191:C 25 Sep 2026 13:36:34.076 * DB saved on disk\n\u001b[2Kdocmost-redis | 191:C 25 Sep 2026 13:36:34.077 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:36:34.165 * Background saving terminated with success\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:41:35.035 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:41:35.036 * Background saving started by pid 374\n\u001b[2Kdocmost-redis | 374:C 25 Sep 2026 13:41:35.048 * DB saved on disk\n\u001b[2Kdocmost-redis | 374:C 25 Sep 2026 13:41:35.049 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:41:35.136 * Background saving terminated with success", + "engine_state_after": { + "docmost-postgres": { + "image": "postgres:18-alpine", + "probe": "datadir major version", + "answer": "18", + "probe_rc": 0 + } + }, + "memory": { + "soak_s": 606.6, + "requested_s": 600, + "requests": 11961, + "codes": { + "200": 11961 + }, + "first_kill": null, + "containers": { + "docmost": { + "limit": 402653184, + "peak": 402665472, + "peak_pct": 1.0, + "anon_peak_sampled": 366161920, + "anon_peak_pct": 0.909, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-postgres": { + "limit": 268435456, + "peak": 206811136, + "peak_pct": 0.77, + "anon_peak_sampled": 16371712, + "anon_peak_pct": 0.061, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-redis": { + "limit": 134217728, + "peak": 11501568, + "peak_pct": 0.086, + "anon_peak_sampled": 4087808, + "anon_peak_pct": 0.03, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + } + }, + "unmeasured": [], + "load": "reached" + }, + "marks": [ + "memory_tight" + ], + "duration_s": 31.1, + "measured_at": "2026-09-25T11:45:26Z", + "evidence": "evidence/MV-docmost", + "scratch_cleared": [], + "engine_conversion": { + "service": "docmost-postgres", + "engine": "postgres", + "from": 16, + "to": 18, + "dump_s": 0.54, + "dump_bytes": 132184, + "new_engine_ready_s": 2.2, + "load_s": 2.06, + "dropped": [ + "docmost" + ], + "skipped_create_role": [ + "CREATE ROLE docmost;" + ], + "result": "converted", + "check_equal": true, + "pg_version": "18", + "tables": 48, + "convert_s": 25.0 + }, + "files_changed": [], + "total_s": 950.2 +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/MV-docmost.log b/documentation/audits/night-2026-09-26/C/bench-run2/MV-docmost.log new file mode 100644 index 00000000..24838d21 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/MV-docmost.log @@ -0,0 +1,166 @@ +[11:46:17] scratch drive folders cleared before FROM (R-656): none existed +[11:46:17] MV-docmost: deploying docmost at FROM {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:46:59] FROM settled=True in 31.1s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:47:00] docmost: /api/auth/setup http=200 rc=0 +[11:47:00] docmost: login as the seeded user http=200 ok=True +[11:47:00] C1 (seed reads back BEFORE): True +[11:47:00] MV-docmost: PostgreSQL major move {'service': 'docmost-postgres', 'engine': 'postgres', 'from': 16, 'to': 18} — converting on the bench before the swap +[11:47:11] CONVERTED docmost-postgres PostgreSQL 16 -> 18 in 11.0s (dump 0.54s / 132196 B, load 2.02s, check equal over 48 tables) +[11:47:11] MV-docmost: swapping to TO {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:47:23] TO up -d rc=0 +[11:47:54] TO settled=True in 31.2s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:47:54] engine state docmost-postgres: 18 +[11:47:54] migration lines observed: 1 +[11:47:54] docmost: login as the seeded user http=200 ok=True +[11:47:54] RESULT (seed reads back AFTER): True +[11:47:55] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:3000 +[11:48:10] + 15s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=300 +[11:48:25] + 30s docmost=432M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=600 +[11:48:40] + 45s docmost=437M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=898 +[11:48:55] + 61s docmost=434M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=1196 +[11:49:10] + 76s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=1496 +[11:49:26] + 91s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=1796 +[11:49:41] + 106s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=2096 +[11:49:56] + 121s docmost=421M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=2396 +[11:50:11] + 136s docmost=423M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=2696 +[11:50:26] + 152s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=2992 +[11:50:41] + 167s docmost=420M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=3292 +[11:50:57] + 182s docmost=415M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=3592 +[11:51:12] + 197s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=3892 +[11:51:27] + 212s docmost=414M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=4188 +[11:51:42] + 227s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=4488 +[11:51:57] + 243s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=4788 +[11:52:12] + 258s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=5088 +[11:52:28] + 273s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=5384 +[11:52:43] + 288s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=125M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=5684 +[11:52:58] + 303s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=123M/256M peak=125M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=5984 +[11:53:13] + 318s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=124M/256M peak=127M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=6284 +[11:53:28] + 334s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=126M/256M peak=128M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=6584 +[11:53:43] + 349s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=127M/256M peak=130M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=6884 +[11:53:59] + 364s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=130M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=7184 +[11:54:14] + 379s docmost=414M/512M peak=460M kills=0 rs=0 docmost-postgres=126M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=7484 +[11:54:29] + 394s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=7780 +[11:54:44] + 409s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=8080 +[11:54:59] + 425s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=8380 +[11:55:14] + 440s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=8680 +[11:55:30] + 455s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=8980 +[11:55:45] + 470s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=9276 +[11:56:00] + 485s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=9576 +[11:56:15] + 500s docmost=415M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=9876 +[11:56:30] + 516s docmost=415M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=10176 +[11:56:45] + 531s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=10472 +[11:57:01] + 546s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=10772 +[11:57:16] + 561s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=11072 +[11:57:31] + 576s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=11372 +[11:57:46] + 591s docmost=419M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=11672 +[11:58:01] + 606s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=11972 +[11:58:02] memory watch: killed=False tight=['docmost'] requests=11972 codes={'200': 11972} +[11:58:02] MV-docmost: ABORT — putting the FROM images back +[12:01:05] ABORT: the app did NOT come back (rc=1, 181.1s) +{ + "harness_version": 4, + "edge": "MV-docmost", + "app": "docmost", + "note": "night 2026-09-23 within-a-major move: docmost-postgres=postgres:18-alpine", + "from": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "to": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:18-alpine", + "docmost-redis": "redis:7-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "\u001b[2Kdocmost | {\"level\":\"info\",\"time\":\"2026-09-25T11:47:30.079Z\",\"pid\":45,\"hostname\":\"807844441a94\",\"context\":\"DatabaseMigrationService\",\"msg\":\"No pending database migrations\"}", + "abort": "refuses", + "abort_detail": "26 13:52:13.041 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:52:13.042 * Background saving started by pid 190\n\u001b[2Kdocmost-redis | 190:C 25 Sep 2026 13:52:13.056 * DB saved on disk\n\u001b[2Kdocmost-redis | 190:C 25 Sep 2026 13:52:13.056 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:52:13.143 * Background saving terminated with success\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:57:14.072 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:57:14.072 * Background saving started by pid 373\n\u001b[2Kdocmost-redis | 373:C 25 Sep 2026 13:57:14.085 * DB saved on disk\n\u001b[2Kdocmost-redis | 373:C 25 Sep 2026 13:57:14.086 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:57:14.173 * Background saving terminated with success", + "engine_state_after": { + "docmost-postgres": { + "image": "postgres:18-alpine", + "probe": "datadir major version", + "answer": "18", + "probe_rc": 0 + } + }, + "memory": { + "soak_s": 606.8, + "requested_s": 600, + "requests": 11972, + "codes": { + "200": 11972 + }, + "first_kill": null, + "containers": { + "docmost": { + "limit": 536870912, + "peak": 482848768, + "peak_pct": 0.899, + "anon_peak_sampled": 431427584, + "anon_peak_pct": 0.804, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-postgres": { + "limit": 268435456, + "peak": 138342400, + "peak_pct": 0.515, + "anon_peak_sampled": 13156352, + "anon_peak_pct": 0.049, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-redis": { + "limit": 134217728, + "peak": 9109504, + "peak_pct": 0.068, + "anon_peak_sampled": 4136960, + "anon_peak_pct": 0.031, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + } + }, + "unmeasured": [], + "load": "reached" + }, + "marks": [ + "memory_tight" + ], + "duration_s": 31.2, + "measured_at": "2026-09-25T12:01:05Z", + "evidence": "evidence/MV-docmost", + "scratch_cleared": [], + "engine_conversion": { + "service": "docmost-postgres", + "engine": "postgres", + "from": 16, + "to": 18, + "dump_s": 0.54, + "dump_bytes": 132196, + "new_engine_ready_s": 2.2, + "load_s": 2.02, + "dropped": [ + "docmost" + ], + "skipped_create_role": [ + "CREATE ROLE docmost;" + ], + "result": "converted", + "check_equal": true, + "pg_version": "18", + "tables": 48, + "convert_s": 11.0 + }, + "files_changed": [], + "total_s": 887.8 +} diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-refusal.txt b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-refusal.txt new file mode 100644 index 00000000..a028238f --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-refusal.txt @@ -0,0 +1,80 @@ +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-redis | 1:M 25 Sep 2026 13:46:18.044 * monotonic clock: POSIX clock_gettime +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-redis | 1:M 25 Sep 2026 13:46:18.045 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:46:18.045 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:46:18.052 * Creating AOF base file appendonly.aof.1.base.rdb on server start +docmost-redis | 1:M 25 Sep 2026 13:46:18.059 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:46:18.059 * Ready to accept connections tcp +docmost-redis | 1:signal-handler (1790336821) Received SIGTERM scheduling shutdown... +docmost-redis | 1:M 25 Sep 2026 13:47:01.307 * User requested shutdown... +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-redis | 1:M 25 Sep 2026 13:47:01.307 * Calling fsync() on the AOF file. +docmost-redis | 1:M 25 Sep 2026 13:47:01.308 * Saving the final RDB snapshot before exiting. +docmost-redis | 1:M 25 Sep 2026 13:47:01.321 * DB saved on disk +docmost-redis | 1:M 25 Sep 2026 13:47:01.321 # Redis is now ready to exit, bye bye... +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-redis | 1:C 25 Sep 2026 13:47:12.207 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:47:12.208 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:47:12.209 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Reading RDB base file on AOF loading... +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Loading RDB produced by version 7.4.11 +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB age 54 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB memory usage when created 0.90 Mb +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB is base AOF +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Done loading RDB, keys loaded: 0, keys expired: 0. +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * DB loaded from append only file: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * Opening AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * Ready to accept connections tcp +docmost-redis | 1:M 25 Sep 2026 13:52:13.041 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:52:13.042 * Background saving started by pid 190 +docmost-redis | 190:C 25 Sep 2026 13:52:13.056 * DB saved on disk +docmost-redis | 190:C 25 Sep 2026 13:52:13.056 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:52:13.143 * Background saving terminated with success +docmost-redis | 1:M 25 Sep 2026 13:57:14.072 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:57:14.072 * Background saving started by pid 373 +docmost-redis | 373:C 25 Sep 2026 13:57:14.085 * DB saved on disk +docmost-redis | 373:C 25 Sep 2026 13:57:14.086 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:57:14.173 * Background saving terminated with success \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-states.json b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-states.json new file mode 100644 index 00000000..25094113 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/abort-states.json @@ -0,0 +1,20 @@ +{ + "docmost": { + "status": "created", + "health": null, + "restarts": 0, + "exit": 0 + }, + "docmost-postgres": { + "status": "restarting", + "health": "unhealthy", + "restarts": 0, + "exit": 1 + }, + "docmost-redis": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/compose-final.log b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/compose-final.log new file mode 100644 index 00000000..8bbd5836 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/compose-final.log @@ -0,0 +1,176 @@ +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:46:18.044 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:46:18.045 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:46:18.045 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:46:18.052 * Creating AOF base file appendonly.aof.1.base.rdb on server start +docmost-redis | 1:M 25 Sep 2026 13:46:18.059 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:46:18.059 * Ready to accept connections tcp +docmost-redis | 1:signal-handler (1790336821) Received SIGTERM scheduling shutdown... +docmost-redis | 1:M 25 Sep 2026 13:47:01.307 * User requested shutdown... +docmost-redis | 1:M 25 Sep 2026 13:47:01.307 * Calling fsync() on the AOF file. +docmost-redis | 1:M 25 Sep 2026 13:47:01.308 * Saving the final RDB snapshot before exiting. +docmost-redis | 1:M 25 Sep 2026 13:47:01.321 * DB saved on disk +docmost-redis | 1:M 25 Sep 2026 13:47:01.321 # Redis is now ready to exit, bye bye... +docmost-redis | 1:C 25 Sep 2026 13:47:12.207 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:47:12.208 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:47:12.209 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Reading RDB base file on AOF loading... +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Loading RDB produced by version 7.4.11 +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB age 54 seconds +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB memory usage when created 0.90 Mb +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB is base AOF +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Done loading RDB, keys loaded: 0, keys expired: 0. +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * DB loaded from append only file: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * Opening AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * Ready to accept connections tcp +docmost-redis | 1:M 25 Sep 2026 13:52:13.041 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:52:13.042 * Background saving started by pid 190 +docmost-redis | 190:C 25 Sep 2026 13:52:13.056 * DB saved on disk +docmost-redis | 190:C 25 Sep 2026 13:52:13.056 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:52:13.143 * Background saving terminated with success +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-redis | 1:M 25 Sep 2026 13:57:14.072 * 100 changes in 300 seconds. Saving... +docmost-redis | 1:M 25 Sep 2026 13:57:14.072 * Background saving started by pid 373 +docmost-redis | 373:C 25 Sep 2026 13:57:14.085 * DB saved on disk +docmost-redis | 373:C 25 Sep 2026 13:57:14.086 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +docmost-redis | 1:M 25 Sep 2026 13:57:14.173 * Background saving terminated with success +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are disabled. +docmost-postgres | +docmost-postgres | initdb: error: directory "/var/lib/postgresql/data" exists but is not empty +docmost-postgres | initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/data" or run initdb with an argument other than "/var/lib/postgresql/data". diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-after.txt b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-after.txt new file mode 100644 index 00000000..ab236c2a --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-after.txt @@ -0,0 +1,54 @@ +db:docmost owner=docmost enc=UTF8 coll=en_US.utf8 +db:postgres owner=docmost enc=UTF8 coll=en_US.utf8 +ext:docmost:pg_trgm +ext:docmost:plpgsql +ext:docmost:unaccent +role:docmost super=true login=true pw=true +rows:docmost:public.ai_chat_messages=0 +rows:docmost:public.ai_chats=0 +rows:docmost:public.api_keys=0 +rows:docmost:public.attachments=0 +rows:docmost:public.audit=0 +rows:docmost:public.auth_accounts=0 +rows:docmost:public.auth_providers=0 +rows:docmost:public.backlinks=0 +rows:docmost:public.base_properties=0 +rows:docmost:public.base_rows=0 +rows:docmost:public.base_views=0 +rows:docmost:public.billing=0 +rows:docmost:public.comments=0 +rows:docmost:public.favorites=0 +rows:docmost:public.file_tasks=0 +rows:docmost:public.group_users=1 +rows:docmost:public.groups=1 +rows:docmost:public.kysely_migration=52 +rows:docmost:public.kysely_migration_lock=1 +rows:docmost:public.labels=0 +rows:docmost:public.notifications=0 +rows:docmost:public.oauth_authorization_codes=0 +rows:docmost:public.oauth_clients=0 +rows:docmost:public.oauth_grants=0 +rows:docmost:public.oauth_tokens=0 +rows:docmost:public.page_access=0 +rows:docmost:public.page_history=0 +rows:docmost:public.page_labels=0 +rows:docmost:public.page_permissions=0 +rows:docmost:public.page_transclusion_references=0 +rows:docmost:public.page_transclusions=0 +rows:docmost:public.page_verifications=0 +rows:docmost:public.page_verifiers=0 +rows:docmost:public.pages=0 +rows:docmost:public.public_spaces=0 +rows:docmost:public.scim_tokens=0 +rows:docmost:public.shares=0 +rows:docmost:public.siem_destinations=0 +rows:docmost:public.space_members=2 +rows:docmost:public.spaces=1 +rows:docmost:public.templates=0 +rows:docmost:public.user_mfa=0 +rows:docmost:public.user_sessions=2 +rows:docmost:public.user_tokens=0 +rows:docmost:public.users=1 +rows:docmost:public.watchers=0 +rows:docmost:public.workspace_invitations=0 +rows:docmost:public.workspaces=1 diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-before.txt b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-before.txt new file mode 100644 index 00000000..ab236c2a --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-check-before.txt @@ -0,0 +1,54 @@ +db:docmost owner=docmost enc=UTF8 coll=en_US.utf8 +db:postgres owner=docmost enc=UTF8 coll=en_US.utf8 +ext:docmost:pg_trgm +ext:docmost:plpgsql +ext:docmost:unaccent +role:docmost super=true login=true pw=true +rows:docmost:public.ai_chat_messages=0 +rows:docmost:public.ai_chats=0 +rows:docmost:public.api_keys=0 +rows:docmost:public.attachments=0 +rows:docmost:public.audit=0 +rows:docmost:public.auth_accounts=0 +rows:docmost:public.auth_providers=0 +rows:docmost:public.backlinks=0 +rows:docmost:public.base_properties=0 +rows:docmost:public.base_rows=0 +rows:docmost:public.base_views=0 +rows:docmost:public.billing=0 +rows:docmost:public.comments=0 +rows:docmost:public.favorites=0 +rows:docmost:public.file_tasks=0 +rows:docmost:public.group_users=1 +rows:docmost:public.groups=1 +rows:docmost:public.kysely_migration=52 +rows:docmost:public.kysely_migration_lock=1 +rows:docmost:public.labels=0 +rows:docmost:public.notifications=0 +rows:docmost:public.oauth_authorization_codes=0 +rows:docmost:public.oauth_clients=0 +rows:docmost:public.oauth_grants=0 +rows:docmost:public.oauth_tokens=0 +rows:docmost:public.page_access=0 +rows:docmost:public.page_history=0 +rows:docmost:public.page_labels=0 +rows:docmost:public.page_permissions=0 +rows:docmost:public.page_transclusion_references=0 +rows:docmost:public.page_transclusions=0 +rows:docmost:public.page_verifications=0 +rows:docmost:public.page_verifiers=0 +rows:docmost:public.pages=0 +rows:docmost:public.public_spaces=0 +rows:docmost:public.scim_tokens=0 +rows:docmost:public.shares=0 +rows:docmost:public.siem_destinations=0 +rows:docmost:public.space_members=2 +rows:docmost:public.spaces=1 +rows:docmost:public.templates=0 +rows:docmost:public.user_mfa=0 +rows:docmost:public.user_sessions=2 +rows:docmost:public.user_tokens=0 +rows:docmost:public.users=1 +rows:docmost:public.watchers=0 +rows:docmost:public.workspace_invitations=0 +rows:docmost:public.workspaces=1 diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-load.err b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/convert-load.err new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/engine-state.json b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/engine-state.json new file mode 100644 index 00000000..6c38c07a --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/engine-state.json @@ -0,0 +1,8 @@ +{ + "docmost-postgres": { + "image": "postgres:18-alpine", + "probe": "datadir major version", + "answer": "18", + "probe_rc": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-after.json b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-after.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-after.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-before.json b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-before.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/files-before.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/memory-samples.json b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/memory-samples.json new file mode 100644 index 00000000..eb733baa --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/memory-samples.json @@ -0,0 +1,1562 @@ +[ + { + "t": 15.2, + "containers": { + "docmost": { + "limit": 536870912, + "current": 443064320, + "peak": 482848768, + "anon": 416133120, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124661760, + "peak": 129396736, + "anon": 11550720, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 4915200, + "peak": 7737344, + "anon": 4136960, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 300 + }, + { + "t": 30.3, + "containers": { + "docmost": { + "limit": 536870912, + "current": 453984256, + "peak": 482848768, + "anon": 427085824, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124624896, + "peak": 129396736, + "anon": 11550720, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5152768, + "peak": 8044544, + "anon": 4096000, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 600 + }, + { + "t": 45.4, + "containers": { + "docmost": { + "limit": 536870912, + "current": 458665984, + "peak": 482848768, + "anon": 431427584, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124809216, + "peak": 129396736, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5103616, + "peak": 8265728, + "anon": 4083712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 898 + }, + { + "t": 60.6, + "containers": { + "docmost": { + "limit": 536870912, + "current": 455692288, + "peak": 482848768, + "anon": 428613632, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124735488, + "peak": 129396736, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5541888, + "peak": 8265728, + "anon": 4026368, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1196 + }, + { + "t": 75.8, + "containers": { + "docmost": { + "limit": 536870912, + "current": 442867712, + "peak": 482848768, + "anon": 416555008, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124977152, + "peak": 129396736, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5087232, + "peak": 8265728, + "anon": 4050944, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1496 + }, + { + "t": 91.0, + "containers": { + "docmost": { + "limit": 536870912, + "current": 443006976, + "peak": 482848768, + "anon": 416870400, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124989440, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5099520, + "peak": 8265728, + "anon": 4059136, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1796 + }, + { + "t": 106.1, + "containers": { + "docmost": { + "limit": 536870912, + "current": 443199488, + "peak": 482848768, + "anon": 416755712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124682240, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5435392, + "peak": 8265728, + "anon": 4059136, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2096 + }, + { + "t": 121.3, + "containers": { + "docmost": { + "limit": 536870912, + "current": 441503744, + "peak": 482848768, + "anon": 414699520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124973056, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5353472, + "peak": 8265728, + "anon": 4038656, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2396 + }, + { + "t": 136.4, + "containers": { + "docmost": { + "limit": 536870912, + "current": 443674624, + "peak": 482848768, + "anon": 416755712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 125071360, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5124096, + "peak": 8265728, + "anon": 4067328, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2696 + }, + { + "t": 151.6, + "containers": { + "docmost": { + "limit": 536870912, + "current": 443142144, + "peak": 482848768, + "anon": 416772096, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124882944, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5169152, + "peak": 8265728, + "anon": 4071424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2992 + }, + { + "t": 166.8, + "containers": { + "docmost": { + "limit": 536870912, + "current": 441040896, + "peak": 482848768, + "anon": 414793728, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124833792, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5390336, + "peak": 8265728, + "anon": 4059136, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3292 + }, + { + "t": 181.9, + "containers": { + "docmost": { + "limit": 536870912, + "current": 435974144, + "peak": 482848768, + "anon": 409571328, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124833792, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5124096, + "peak": 8265728, + "anon": 4042752, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3592 + }, + { + "t": 197.1, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437919744, + "peak": 482848768, + "anon": 412205056, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124833792, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5140480, + "peak": 8265728, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3892 + }, + { + "t": 212.2, + "containers": { + "docmost": { + "limit": 536870912, + "current": 434978816, + "peak": 482848768, + "anon": 409104384, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124833792, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5144576, + "peak": 8265728, + "anon": 4055040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4188 + }, + { + "t": 227.4, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437485568, + "peak": 482848768, + "anon": 411549696, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124833792, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5419008, + "peak": 8265728, + "anon": 4063232, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4488 + }, + { + "t": 242.6, + "containers": { + "docmost": { + "limit": 536870912, + "current": 436523008, + "peak": 482848768, + "anon": 410132480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 124833792, + "peak": 129658880, + "anon": 11558912, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5439488, + "peak": 8302592, + "anon": 4075520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4788 + }, + { + "t": 257.7, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437624832, + "peak": 482848768, + "anon": 411680768, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 125153280, + "peak": 129658880, + "anon": 11628544, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5406720, + "peak": 8302592, + "anon": 4079616, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5088 + }, + { + "t": 272.9, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437141504, + "peak": 482848768, + "anon": 411459584, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127205376, + "peak": 129658880, + "anon": 11628544, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5464064, + "peak": 8302592, + "anon": 4083712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5384 + }, + { + "t": 288.1, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438034432, + "peak": 482848768, + "anon": 412061696, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 128917504, + "peak": 131280896, + "anon": 11632640, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5218304, + "peak": 8302592, + "anon": 4091904, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5684 + }, + { + "t": 303.2, + "containers": { + "docmost": { + "limit": 536870912, + "current": 436764672, + "peak": 482848768, + "anon": 410587136, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 129974272, + "peak": 132034560, + "anon": 11649024, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5472256, + "peak": 8302592, + "anon": 4079616, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5984 + }, + { + "t": 318.4, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438128640, + "peak": 482848768, + "anon": 412135424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 130990080, + "peak": 133681152, + "anon": 11649024, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5222400, + "peak": 8302592, + "anon": 4083712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6284 + }, + { + "t": 333.6, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437854208, + "peak": 482848768, + "anon": 412135424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 132472832, + "peak": 134348800, + "anon": 11653120, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5234688, + "peak": 8302592, + "anon": 4087808, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6584 + }, + { + "t": 348.8, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437792768, + "peak": 482848768, + "anon": 411840512, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 134098944, + "peak": 136671232, + "anon": 11669504, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 4980736, + "peak": 8302592, + "anon": 4091904, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6884 + }, + { + "t": 363.9, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437538816, + "peak": 482848768, + "anon": 411852800, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 136978432, + "peak": 137990144, + "anon": 13156352, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5246976, + "peak": 8302592, + "anon": 4091904, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7184 + }, + { + "t": 379.1, + "containers": { + "docmost": { + "limit": 536870912, + "current": 434196480, + "peak": 482848768, + "anon": 408616960, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 132284416, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 4993024, + "peak": 8302592, + "anon": 4091904, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7484 + }, + { + "t": 394.3, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437071872, + "peak": 482848768, + "anon": 411148288, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 131928064, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5267456, + "peak": 8396800, + "anon": 4096000, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7780 + }, + { + "t": 409.4, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438939648, + "peak": 482848768, + "anon": 412958720, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 131477504, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5275648, + "peak": 8396800, + "anon": 4096000, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8080 + }, + { + "t": 424.6, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437637120, + "peak": 482848768, + "anon": 410951680, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 131600384, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5279744, + "peak": 8396800, + "anon": 4104192, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8380 + }, + { + "t": 439.7, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438435840, + "peak": 482848768, + "anon": 412504064, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 131465216, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5562368, + "peak": 8396800, + "anon": 4108288, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8680 + }, + { + "t": 454.9, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438661120, + "peak": 482848768, + "anon": 412438528, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 128016384, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 4947968, + "peak": 9109504, + "anon": 4104192, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8980 + }, + { + "t": 470.1, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438247424, + "peak": 482848768, + "anon": 412540928, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127684608, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 4976640, + "peak": 9109504, + "anon": 4104192, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9276 + }, + { + "t": 485.2, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438628352, + "peak": 482848768, + "anon": 412651520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127901696, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5251072, + "peak": 9109504, + "anon": 4108288, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9576 + }, + { + "t": 500.4, + "containers": { + "docmost": { + "limit": 536870912, + "current": 435908608, + "peak": 482848768, + "anon": 410267648, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127684608, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 4988928, + "peak": 9109504, + "anon": 4108288, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9876 + }, + { + "t": 515.5, + "containers": { + "docmost": { + "limit": 536870912, + "current": 435449856, + "peak": 482848768, + "anon": 409341952, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127971328, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5505024, + "peak": 9109504, + "anon": 4108288, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10176 + }, + { + "t": 530.7, + "containers": { + "docmost": { + "limit": 536870912, + "current": 436940800, + "peak": 482848768, + "anon": 411013120, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127684608, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5332992, + "peak": 9109504, + "anon": 4116480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10472 + }, + { + "t": 545.9, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437010432, + "peak": 482848768, + "anon": 411086848, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127692800, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5255168, + "peak": 9109504, + "anon": 4116480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10772 + }, + { + "t": 561.0, + "containers": { + "docmost": { + "limit": 536870912, + "current": 438517760, + "peak": 482848768, + "anon": 412319744, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 127684608, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5492736, + "peak": 9109504, + "anon": 4116480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11072 + }, + { + "t": 576.2, + "containers": { + "docmost": { + "limit": 536870912, + "current": 439300096, + "peak": 482848768, + "anon": 412577792, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 128585728, + "peak": 138342400, + "anon": 12001280, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5419008, + "peak": 9109504, + "anon": 4120576, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11372 + }, + { + "t": 591.4, + "containers": { + "docmost": { + "limit": 536870912, + "current": 439468032, + "peak": 482848768, + "anon": 413024256, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 128569344, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5160960, + "peak": 9109504, + "anon": 4120576, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11672 + }, + { + "t": 606.5, + "containers": { + "docmost": { + "limit": 536870912, + "current": 437956608, + "peak": 482848768, + "anon": 411541504, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-postgres": { + "limit": 268435456, + "current": 128552960, + "peak": 138342400, + "anon": 11751424, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "docmost-redis": { + "limit": 134217728, + "current": 5603328, + "peak": 9109504, + "anon": 4116480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 11972 + } +] \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/migration-lines.txt b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/migration-lines.txt new file mode 100644 index 00000000..4bb3d47b --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/migration-lines.txt @@ -0,0 +1 @@ +docmost | {"level":"info","time":"2026-09-25T11:47:30.079Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"No pending database migrations"} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/run.log b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/run.log new file mode 100644 index 00000000..fdfc2f10 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/run.log @@ -0,0 +1,59 @@ +[11:46:17] scratch drive folders cleared before FROM (R-656): none existed +[11:46:17] MV-docmost: deploying docmost at FROM {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:46:59] FROM settled=True in 31.1s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:47:00] docmost: /api/auth/setup http=200 rc=0 +[11:47:00] docmost: login as the seeded user http=200 ok=True +[11:47:00] C1 (seed reads back BEFORE): True +[11:47:00] MV-docmost: PostgreSQL major move {'service': 'docmost-postgres', 'engine': 'postgres', 'from': 16, 'to': 18} — converting on the bench before the swap +[11:47:11] CONVERTED docmost-postgres PostgreSQL 16 -> 18 in 11.0s (dump 0.54s / 132196 B, load 2.02s, check equal over 48 tables) +[11:47:11] MV-docmost: swapping to TO {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:18-alpine', 'docmost-redis': 'redis:7-alpine'} +[11:47:23] TO up -d rc=0 +[11:47:54] TO settled=True in 31.2s :: {"docmost": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "docmost-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[11:47:54] engine state docmost-postgres: 18 +[11:47:54] migration lines observed: 1 +[11:47:54] docmost: login as the seeded user http=200 ok=True +[11:47:54] RESULT (seed reads back AFTER): True +[11:47:55] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:3000 +[11:48:10] + 15s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=300 +[11:48:25] + 30s docmost=432M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=600 +[11:48:40] + 45s docmost=437M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=898 +[11:48:55] + 61s docmost=434M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=1196 +[11:49:10] + 76s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=1496 +[11:49:26] + 91s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=1796 +[11:49:41] + 106s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=118M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=2096 +[11:49:56] + 121s docmost=421M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=2396 +[11:50:11] + 136s docmost=423M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=2696 +[11:50:26] + 152s docmost=422M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=2992 +[11:50:41] + 167s docmost=420M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=3292 +[11:50:57] + 182s docmost=415M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=3592 +[11:51:12] + 197s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=3892 +[11:51:27] + 212s docmost=414M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=4188 +[11:51:42] + 227s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=4488 +[11:51:57] + 243s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=4788 +[11:52:12] + 258s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=119M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=5088 +[11:52:28] + 273s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=123M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=5384 +[11:52:43] + 288s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=125M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=5684 +[11:52:58] + 303s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=123M/256M peak=125M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=5984 +[11:53:13] + 318s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=124M/256M peak=127M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=6284 +[11:53:28] + 334s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=126M/256M peak=128M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=6584 +[11:53:43] + 349s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=127M/256M peak=130M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=6884 +[11:53:59] + 364s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=130M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=7M kills=0 rs=0 reqs=7184 +[11:54:14] + 379s docmost=414M/512M peak=460M kills=0 rs=0 docmost-postgres=126M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=7M kills=0 rs=0 reqs=7484 +[11:54:29] + 394s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=7780 +[11:54:44] + 409s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=8080 +[11:54:59] + 425s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=8380 +[11:55:14] + 440s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=125M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=8680 +[11:55:30] + 455s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=8980 +[11:55:45] + 470s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=9276 +[11:56:00] + 485s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=9576 +[11:56:15] + 500s docmost=415M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=9876 +[11:56:30] + 516s docmost=415M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=10176 +[11:56:45] + 531s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=10472 +[11:57:01] + 546s docmost=416M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=10772 +[11:57:16] + 561s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=121M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=11072 +[11:57:31] + 576s docmost=418M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=11372 +[11:57:46] + 591s docmost=419M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=4M/128M peak=8M kills=0 rs=0 reqs=11672 +[11:58:01] + 606s docmost=417M/512M peak=460M kills=0 rs=0 docmost-postgres=122M/256M peak=131M kills=0 rs=0 docmost-redis=5M/128M peak=8M kills=0 rs=0 reqs=11972 +[11:58:02] memory watch: killed=False tight=['docmost'] requests=11972 codes={'200': 11972} +[11:58:02] MV-docmost: ABORT — putting the FROM images back +[12:01:05] ABORT: the app did NOT come back (rc=1, 181.1s) \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-full.log b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-full.log new file mode 100644 index 00000000..40b03ac8 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-full.log @@ -0,0 +1,166 @@ +docmost-postgres | The files belonging to this database system will be owned by user "postgres". +docmost-postgres | This user must also own the server process. +docmost-postgres | +docmost-postgres | The database cluster will be initialized with locale "en_US.utf8". +docmost-postgres | The default database encoding has accordingly been set to "UTF8". +docmost-postgres | The default text search configuration will be set to "english". +docmost-postgres | +docmost-postgres | Data page checksums are enabled. +docmost-postgres | +docmost-postgres | fixing permissions on existing directory /var/lib/postgresql/18/docker ... ok +docmost-postgres | creating subdirectories ... ok +docmost-postgres | selecting dynamic shared memory implementation ... posix +docmost-postgres | selecting default "max_connections" ... 100 +docmost-postgres | selecting default "shared_buffers" ... 128MB +docmost-postgres | selecting default time zone ... Europe/Budapest +docmost-postgres | creating configuration files ... ok +docmost-postgres | running bootstrap script ... ok +docmost-postgres | sh: locale: not found +docmost-postgres | 2026-09-25 13:47:07.004 CEST [47] WARNING: no usable system locales were found +docmost-postgres | performing post-bootstrap initialization ... ok +docmost-postgres | syncing data to disk ... ok +docmost-postgres | +docmost-postgres | +docmost-postgres | Success. You can now start the database server using: +docmost-postgres | +docmost-postgres | pg_ctl -D /var/lib/postgresql/18/docker -l logfile start +docmost-postgres | +docmost-postgres | initdb: warning: enabling "trust" authentication for local connections +docmost-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. +docmost-postgres | waiting for server to start....2026-09-25 13:47:08.179 CEST [53] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +docmost-postgres | 2026-09-25 13:47:08.183 CEST [53] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +docmost-postgres | 2026-09-25 13:47:08.195 CEST [59] LOG: database system was shut down at 2026-09-25 13:47:07 CEST +docmost-postgres | 2026-09-25 13:47:08.203 CEST [53] LOG: database system is ready to accept connections +docmost-postgres | done +docmost-postgres | server started +docmost-postgres | CREATE DATABASE +docmost-postgres | +docmost-postgres | +docmost-postgres | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* +docmost-postgres | +docmost-postgres | waiting for server to shut down....2026-09-25 13:47:08.418 CEST [53] LOG: received fast shutdown request +docmost-postgres | 2026-09-25 13:47:08.422 CEST [53] LOG: aborting any active transactions +docmost-postgres | 2026-09-25 13:47:08.425 CEST [53] LOG: background worker "logical replication launcher" (PID 62) exited with exit code 1 +docmost-postgres | 2026-09-25 13:47:08.427 CEST [57] LOG: shutting down +docmost-postgres | 2026-09-25 13:47:08.430 CEST [57] LOG: checkpoint starting: shutdown immediate +docmost-postgres | 2026-09-25 13:47:08.581 CEST [57] LOG: checkpoint complete: wrote 943 buffers (5.8%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.025 s, sync=0.114 s, total=0.155 s; sync files=303, longest=0.084 s, average=0.001 s; distance=4362 kB, estimate=4362 kB; lsn=0/1BA8858, redo lsn=0/1BA8858 +docmost-postgres | 2026-09-25 13:47:08.608 CEST [53] LOG: database system is shut down +docmost-postgres | done +docmost-postgres | server stopped +docmost-postgres | +docmost-postgres | PostgreSQL init process complete; ready for start up. +docmost-postgres | +docmost-postgres | 2026-09-25 13:47:08.660 CEST [1] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +docmost-postgres | 2026-09-25 13:47:08.660 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +docmost-postgres | 2026-09-25 13:47:08.660 CEST [1] LOG: listening on IPv6 address "::", port 5432 +docmost-postgres | 2026-09-25 13:47:08.666 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +docmost-postgres | 2026-09-25 13:47:08.676 CEST [76] LOG: database system was shut down at 2026-09-25 13:47:08 CEST +docmost-postgres | 2026-09-25 13:47:08.685 CEST [1] LOG: database system is ready to accept connections +docmost-postgres | 2026-09-25 13:47:09.008 CEST [74] LOG: checkpoint starting: immediate force wait +docmost-postgres | 2026-09-25 13:47:09.038 CEST [74] LOG: checkpoint complete: wrote 1 buffers (0.0%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.008 s, sync=0.006 s, total=0.031 s; sync files=3, longest=0.003 s, average=0.002 s; distance=1 kB, estimate=1 kB; lsn=0/1BA8D60, redo lsn=0/1BA8D08 +docmost | $ pnpm --filter ./apps/server run start:prod +docmost | $ cross-env NODE_ENV=production node dist/main +docmost | (node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided. +docmost | (Use `node --trace-warnings ...` to show where the warning was created) +docmost | {"level":"info","time":"2026-09-25T11:46:35.273Z","pid":45,"hostname":"807844441a94","context":"RedisModule","msg":"default: the connection was successfully established"} +docmost | {"level":"info","time":"2026-09-25T11:46:35.492Z","pid":45,"hostname":"807844441a94","context":"DatabaseModule","msg":"Establishing database connection"} +docmost | {"level":"info","time":"2026-09-25T11:46:35.541Z","pid":45,"hostname":"807844441a94","context":"DatabaseModule","msg":"Database connection successful"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.246Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T085400-uuid_v7_fn\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T085500-workspaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T085600-users\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T085700-groups\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T085900-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T086100-add-workspace-fk\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T086200-workspace_invitations\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T086300-pages\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T086400-page_history\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T086600-comments\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T086700-attachments\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240324T086800-pages-tsvector-trigger\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240707T145623-drop-redundant-pages-slug_id-index\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20240903T124647-user-tokens\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20241218T223249-backlinks\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250106T195516-billing\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250118T194658-sso-auth\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250222T114520-add_license_key_to_workspace\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250327T145832-add-contributorIds-to-pages\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250408T191830-shares\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250521T154949-file_tasks\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250623T215045-more-billing-columns\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250715T070817-mfa\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250725T052004-add-new-comments-columns\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250729T213756-add-unaccent-pg_trm-update-tsvector.\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250831T191600-add-group-sync-to-auth-providers\" executed successfully"} +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250831T202306-ldap-auth\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250901T184612-attachments-search\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20250912T101500-api-keys\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260205T214213-add-settings-to-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260209T120000-add-contributor_ids-to-page-history\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260213T085259-notifications\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260213T085320-watchers\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260213T085337-backfill-watchers\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260224T233803-page-permissions\" executed successfully"} +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:46:18.043 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:46:18.044 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:46:18.045 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:46:18.045 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:46:18.052 * Creating AOF base file appendonly.aof.1.base.rdb on server start +docmost-redis | 1:M 25 Sep 2026 13:46:18.059 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:46:18.059 * Ready to accept connections tcp +docmost-redis | 1:signal-handler (1790336821) Received SIGTERM scheduling shutdown... +docmost-redis | 1:M 25 Sep 2026 13:47:01.307 * User requested shutdown... +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260228T223532-audit\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260326T121350-user-sessions\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260329T163516-add-new-indexes\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260409T132415-ai-chat\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260412T135891-templates\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260412T162318-favorites\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260413T121647-page-verifications\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260414T124451-update-file_tasks\" executed successfully"} +docmost-redis | 1:M 25 Sep 2026 13:47:01.307 * Calling fsync() on the AOF file. +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260501T092214-scim\" executed successfully"} +docmost-redis | 1:M 25 Sep 2026 13:47:01.308 * Saving the final RDB snapshot before exiting. +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260501T202258-page-transclusions\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.247Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260509T121236-labels\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.248Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260529T125146-bases\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.248Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260620T010047-personal-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.248Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260824T211732-page-title-trgm-index\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.248Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260825T022612-oauth\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.248Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260902T121326-siem-destinations\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.248Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"Migration \"20260904T171920-public-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-25T11:46:37.301Z","pid":45,"hostname":"807844441a94","context":"NestApplication","msg":"Nest application successfully started"} +docmost-redis | 1:M 25 Sep 2026 13:47:01.321 * DB saved on disk +docmost-redis | 1:M 25 Sep 2026 13:47:01.321 # Redis is now ready to exit, bye bye... +docmost-redis | 1:C 25 Sep 2026 13:47:12.207 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 25 Sep 2026 13:47:12.208 * Configuration loaded +docmost-redis | 1:M 25 Sep 2026 13:47:12.208 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 25 Sep 2026 13:47:12.209 * Running mode=standalone, port=6379. +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Server initialized +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Reading RDB base file on AOF loading... +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Loading RDB produced by version 7.4.11 +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB age 54 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB memory usage when created 0.90 Mb +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * RDB is base AOF +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * Done loading RDB, keys loaded: 0, keys expired: 0. +docmost-redis | 1:M 25 Sep 2026 13:47:12.210 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * DB loaded from append only file: 0.001 seconds +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * Opening AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 25 Sep 2026 13:47:12.211 * Ready to accept connections tcp +docmost | {"level":"info","time":"2026-09-25T11:46:37.313Z","pid":45,"hostname":"807844441a94","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://docs.gate.invalid"} +docmost | [ELIFECYCLE] Command failed. +docmost | $ pnpm --filter ./apps/server run start:prod +docmost | $ cross-env NODE_ENV=production node dist/main +docmost | (node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided. +docmost | (Use `node --trace-warnings ...` to show where the warning was created) +docmost | {"level":"info","time":"2026-09-25T11:47:29.662Z","pid":45,"hostname":"807844441a94","context":"RedisModule","msg":"default: the connection was successfully established"} +docmost | {"level":"info","time":"2026-09-25T11:47:29.878Z","pid":45,"hostname":"807844441a94","context":"DatabaseModule","msg":"Establishing database connection"} +docmost | {"level":"info","time":"2026-09-25T11:47:29.921Z","pid":45,"hostname":"807844441a94","context":"DatabaseModule","msg":"Database connection successful"} +docmost | {"level":"info","time":"2026-09-25T11:47:30.079Z","pid":45,"hostname":"807844441a94","context":"DatabaseMigrationService","msg":"No pending database migrations"} +docmost | {"level":"info","time":"2026-09-25T11:47:30.114Z","pid":45,"hostname":"807844441a94","context":"NestApplication","msg":"Nest application successfully started"} +docmost | {"level":"info","time":"2026-09-25T11:47:30.126Z","pid":45,"hostname":"807844441a94","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://docs.gate.invalid"} diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-states.json b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-states.json new file mode 100644 index 00000000..c6bae4a5 --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/to-states.json @@ -0,0 +1,20 @@ +{ + "docmost": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "docmost-postgres": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "docmost-redis": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/verdict.json b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/verdict.json new file mode 100644 index 00000000..18d0bf9b --- /dev/null +++ b/documentation/audits/night-2026-09-26/C/bench-run2/evidence/MV-docmost/verdict.json @@ -0,0 +1,107 @@ +{ + "harness_version": 4, + "edge": "MV-docmost", + "app": "docmost", + "note": "night 2026-09-23 within-a-major move: docmost-postgres=postgres:18-alpine", + "from": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "to": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:18-alpine", + "docmost-redis": "redis:7-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "\u001b[2Kdocmost | {\"level\":\"info\",\"time\":\"2026-09-25T11:47:30.079Z\",\"pid\":45,\"hostname\":\"807844441a94\",\"context\":\"DatabaseMigrationService\",\"msg\":\"No pending database migrations\"}", + "abort": "refuses", + "abort_detail": "26 13:52:13.041 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:52:13.042 * Background saving started by pid 190\n\u001b[2Kdocmost-redis | 190:C 25 Sep 2026 13:52:13.056 * DB saved on disk\n\u001b[2Kdocmost-redis | 190:C 25 Sep 2026 13:52:13.056 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:52:13.143 * Background saving terminated with success\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:57:14.072 * 100 changes in 300 seconds. Saving...\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:57:14.072 * Background saving started by pid 373\n\u001b[2Kdocmost-redis | 373:C 25 Sep 2026 13:57:14.085 * DB saved on disk\n\u001b[2Kdocmost-redis | 373:C 25 Sep 2026 13:57:14.086 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB\n\u001b[2Kdocmost-redis | 1:M 25 Sep 2026 13:57:14.173 * Background saving terminated with success", + "engine_state_after": { + "docmost-postgres": { + "image": "postgres:18-alpine", + "probe": "datadir major version", + "answer": "18", + "probe_rc": 0 + } + }, + "memory": { + "soak_s": 606.8, + "requested_s": 600, + "requests": 11972, + "codes": { + "200": 11972 + }, + "first_kill": null, + "containers": { + "docmost": { + "limit": 536870912, + "peak": 482848768, + "peak_pct": 0.899, + "anon_peak_sampled": 431427584, + "anon_peak_pct": 0.804, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-postgres": { + "limit": 268435456, + "peak": 138342400, + "peak_pct": 0.515, + "anon_peak_sampled": 13156352, + "anon_peak_pct": 0.049, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "docmost-redis": { + "limit": 134217728, + "peak": 9109504, + "peak_pct": 0.068, + "anon_peak_sampled": 4136960, + "anon_peak_pct": 0.031, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + } + }, + "unmeasured": [], + "load": "reached" + }, + "marks": [ + "memory_tight" + ], + "duration_s": 31.2, + "measured_at": "2026-09-25T12:01:05Z", + "evidence": "evidence/MV-docmost", + "scratch_cleared": [], + "engine_conversion": { + "service": "docmost-postgres", + "engine": "postgres", + "from": 16, + "to": 18, + "dump_s": 0.54, + "dump_bytes": 132196, + "new_engine_ready_s": 2.2, + "load_s": 2.02, + "dropped": [ + "docmost" + ], + "skipped_create_role": [ + "CREATE ROLE docmost;" + ], + "result": "converted", + "check_equal": true, + "pg_version": "18", + "tables": 48, + "convert_s": 11.0 + }, + "files_changed": [], + "total_s": 887.8 +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-26/D/D2-arrival.txt b/documentation/audits/night-2026-09-26/D/D2-arrival.txt new file mode 100644 index 00000000..59e0f7ee --- /dev/null +++ b/documentation/audits/night-2026-09-26/D/D2-arrival.txt @@ -0,0 +1 @@ +# D2 arrival, 15s after polling began (11:32:26Z): demo-felhom=[gitea.dooplex.hu/admin/felhom-controller:0.273.0 healthy] demo-hp=[gitea.dooplex.hu/admin/felhom-controller:0.273.0 healthy] diff --git a/documentation/audits/night-2026-09-26/D/D2-floor.txt b/documentation/audits/night-2026-09-26/D/D2-floor.txt new file mode 100644 index 00000000..31afbc26 --- /dev/null +++ b/documentation/audits/night-2026-09-26/D/D2-floor.txt @@ -0,0 +1,4 @@ +# D2 global floor 0.273.0 (declared MinAgent 0.131.0) — 2026-09-25T11:32:00Z +impact before: {"below":4,"valid":true,"version":"0.273.0"} +POST /configuration/global-floor -> HTTP/1.1 303 See Other Location: /configuration?flash=floor_set +read back: Effective floor: v0.273.0 — source: DB (hub_settings) ; env fallback would be v0.120.0 Save global floor… DB override: v0.273.0 … Type the version again to conf diff --git a/documentation/audits/night-2026-09-26/D/D3-release-9202.txt b/documentation/audits/night-2026-09-26/D/D3-release-9202.txt new file mode 100644 index 00000000..eaeea22b --- /dev/null +++ b/documentation/audits/night-2026-09-26/D/D3-release-9202.txt @@ -0,0 +1,13 @@ +# 9202, 2026-09-25T12:33:08Z: the kept pre-conversion copy of docmost, and the release job +docmost_docmost_postgres_data.pre-update-20260925T111256Z +conversion_copy: + volume: docmost_docmost_postgres_data + copy: docmost_docmost_postgres_data.pre-update-20260925T111256Z + at: "2026-09-25T11:13:34Z" + from: 16 + to: 18 +2026/09/25 12:30:39 scheduler.go:102: [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s) +2026/09/25 12:30:39 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="conversion-copy-release" interval=1h0m0s totalJobs=10 +-rw-r--r-- 1 root root 154903 2026-09-25T11:08:02 pre-restore-20260925T110802Z-docmost-postgres.sql +-rw-r--r-- 1 root root 155357 2026-09-25T11:11:16 pre-restore-20260925T111115Z-docmost-postgres.sql +-rw-r--r-- 1 root root 155810 2026-09-25T11:12:53 pre-restore-20260925T111253Z-docmost-postgres.sql diff --git a/documentation/audits/night-2026-09-26/E/CHANGELOG-fragment.md b/documentation/audits/night-2026-09-26/E/CHANGELOG-fragment.md new file mode 100644 index 00000000..370834d5 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/CHANGELOG-fragment.md @@ -0,0 +1,17 @@ +- **Kept data — `09` §3 decision 36 (Part E).** A reinstall over an app's kept drive folder asks the household: + „A megőrzött adataimat használom" / "Use my kept data" (a load from the newest copy of THIS drive's install — own + unit or second-drive mirror — then the template's `after_load:`) or „Tiszta lappal kezdem" / "Start fresh" (the + folder is renamed into `/kept///`, with the removed app's unit; nothing is deleted). The install + API answers 409 `kept_data_choice` until one is chosen; `DeployStack` refuses too (`ErrKeptDataChoice`). New page + „Megőrzött adatok" / "Kept data" (`/kept-data`, linked from Tárhely): app, date, size, which copy can bring it back; + Load / Look / Delete (typed confirmation — the only deletion of kept data; the box deletes none by itself, D3 open). + FileBrowser gains a read-only „Megőrzött adatok" source (one `:ro` bind per item). The drive-full warning names the + kept folders and sizes. `/kept` is in `ProtectedHDDPaths` and outside every backup leg. `.felhom.yml` + `after_load:` (nextcloud: `occ files:scan --all`, catalog `9cc829c`). +- **R-690 fixed** — the removed-app restore (R-487) never found a unit on a DATA drive: `primaryUnitDirFor` and + `ListRestorePoints` asked `GetStackComposePath` (true for every catalog app), so nextcloud came back with no env, + no database and its files bound on the guest's root disk (measured on 0.272.0, `E/E1-README.md`). Now + `isStackDeployed`; pinned by a production-shaped provider (the R-487 fake answered it for deployed apps only). +- Red-proofs (each seen failing, tree restored): R-690 ×2, kept rules 1–4, the install guard, the install API, + the copy's drive check, the `:ro` bind — `E/redproofs/`. Parity: 7 Hungarian fixtures regenerated, the only + removed lines are the two edited deploy-page lines; 2 new (`kept_data_full`, `kept_data_empty`). diff --git a/documentation/audits/night-2026-09-26/E/E5-00-deploy-rc.txt b/documentation/audits/night-2026-09-26/E/E5-00-deploy-rc.txt new file mode 100644 index 00000000..c3fbd3f6 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-00-deploy-rc.txt @@ -0,0 +1,3 @@ +# 9202 before: gitea.dooplex.hu/admin/felhom-controller:0.273.0-rc1 +# 9202 after: gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc1 healthy +# 9202 now: gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc2 healthy (rc2 = rc1 + R-692's fix) diff --git a/documentation/audits/night-2026-09-26/E/E5-1-list.txt b/documentation/audits/night-2026-09-26/E/E5-1-list.txt new file mode 100644 index 00000000..8e18b31d --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-1-list.txt @@ -0,0 +1,18 @@ +# E5 1-list — 2026-09-25T13:52:57+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc2 +drive: total 20 +drwxr-xr-x 5 root root 4096 Sep 15 09:13 . +drwxr-xr-x 3 root root 4096 Sep 13 20:23 .. +drwxr-xr-x 4 root root 4096 Sep 25 10:39 appdata +drwxr-xr-x 4 root root 4096 Sep 15 09:16 backups +drwxrwsr-x 18 root 1000 4096 Sep 25 10:39 userdata +ls: cannot access '/mnt/felhom-drives/scratch_hdd/kept': No such file or directory +/mnt/felhom-drives/scratch_hdd/appdata: +total 16 +drwxr-xr-x 4 root root 4096 Sep 25 10:39 . +drwxr-xr-x 5 root root 4096 Sep 15 09:13 .. +drwxrwx--- 5 www-data www-data 4096 Sep 25 10:39 nextcloud +drwxr-xr-x 4 root root 4096 Sep 15 09:13 paperless +126M /mnt/felhom-drives/scratch_hdd/appdata/nextcloud + +GET /kept-data: ...s ↗ Hub kapcsolat kikapcsolva — a központi monitoring nem aktív Rendszermonitor → Vissza Megőrzött adatok Az eltávolított alkalmazások lemezen maradt adatai. Megnézheted őket, betöltheted egy új telepítésbe, vagy törölheted. Magától a szerver soha nem töröl közülük semmit. Erről nem készül mentés. Nextcloud 2026-09-25 12:39 · 125.4 MB Visszatölthető innen: nincs mentés — csak a fájlok Megnézem Törlés A(z) Nextcloud megőrzött adatai (125.4 MB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: Nextcloud Végleges törlés RomM 2026-09-24 22:11 · 12.0 KB Visszatölthető innen: nincs mentés — csak a fájlok Megnézem Törlés A(z) RomM megőrzött adatai (12.0 KB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: RomM Végleges törlés Paperless-ngx 2026-09-15 1... +GET /kept-data?lang=en: ...-rc2 Magyar English Sign out ↗ The hub connection is off — central monitoring is not running System monitor → Back Kept data Data that removed apps left on the drives. You can look at it, load it into a new install, or delete it. The server never deletes any of it by itself. This is not backed up. Nextcloud 2026-09-25 12:39 · 125.4 MB Can be loaded from: no backup — the files only Look Delete The kept data of Nextcloud (125.4 MB) is deleted for good. This cannot be undone. To delete it, type: Nextcloud Delete for good RomM 2026-09-24 22:11 · 12.0 KB Can be loaded from: no backup — the files only Look Delete The kept data of RomM (12.0 KB) is deleted for good. This cannot be undone. To delete it, type: RomM Delete for good Paperless-ngx 2026-09-15 11:13 · 1.2 MB Can be loaded from: no backu... diff --git a/documentation/audits/night-2026-09-26/E/E5-2-ask.txt b/documentation/audits/night-2026-09-26/E/E5-2-ask.txt new file mode 100644 index 00000000..d8e04c22 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-2-ask.txt @@ -0,0 +1,38 @@ +# E5 2-ask — 2026-09-25T13:53:08+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc2 +deploy, no choice, lang=hu -> 409 +{ + "ok": false, + "data": { + "code": "kept_data_choice", + "title": "Ennek az alkalmazásnak megvannak a régi adatai", + "body": "A(z) Nextcloud korábbi adatai (125.4 MB, 2026-09-25) még a lemezen vannak. Mit csináljunk velük?", + "use_label": "A megőrzött adataimat használom", + "use_desc": "", + "use_off": "Az adatbázisról nincs mentés, ezért az alkalmazás nem tudja betölteni a régi fájlokat. A fájlokat megnézheted a Megőrzött adatok között.", + "use_offered": false, + "fresh_label": "Tiszta lappal kezdem", + "fresh_desc": "A régi adatok egy mai dátummal jelölt mappába kerülnek. Nem törlünk semmit.", + "not_backed_up": "Erről nem készül mentés." + }, + "error": "Ennek az alkalmazásnak megvannak a régi adatai" +} +deploy, no choice, lang=en -> 409 +{ + "ok": false, + "data": { + "code": "kept_data_choice", + "title": "This app's old data is still here", + "body": "The old data of Nextcloud (125.4 MB, from 2026-09-25) is still on the drive. What should happen to it?", + "use_label": "Use my kept data", + "use_desc": "", + "use_off": "There is no backup of the database, so the app cannot load the old files. You can look at the files under Kept data.", + "use_offered": false, + "fresh_label": "Start fresh", + "fresh_desc": "The old data moves to a folder marked with today's date. Nothing is deleted.", + "not_backed_up": "This is not backed up." + }, + "error": "This app's old data is still here" +} +still not installed: False nextcloud +paperless + diff --git a/documentation/audits/night-2026-09-26/E/E5-3-fresh.txt b/documentation/audits/night-2026-09-26/E/E5-3-fresh.txt new file mode 100644 index 00000000..f612617f --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-3-fresh.txt @@ -0,0 +1,31 @@ +# E5 3-fresh — 2026-09-25T13:53:25+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc2 +deploy kept_data=fresh -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +deployed after (80.4, 'unhealthy') +drive now: /mnt/felhom-drives/scratch_hdd/appdata: +total 16 +drwxr-xr-x 4 root root 4096 Sep 25 11:53 . +drwxr-xr-x 6 root root 4096 Sep 25 11:53 .. +drwxrwx--- 4 www-data www-data 4096 Sep 25 11:54 nextcloud +drwxr-xr-x 4 root root 4096 Sep 15 09:13 paperless + +/mnt/felhom-drives/scratch_hdd/appdata/nextcloud: +total 24 +drwxrwx--- 4 www-data www-data 4096 Sep 25 11:54 . +drwxr-xr-x 4 root root 4096 Sep 25 11:53 .. +-rw-rw-r-- 1 www-data www-data 542 Sep 25 11:54 .htaccess +-rw-rw-r-- 1 www-data www-data 52 Sep 25 11:54 .ncdata +drwxr-xr-x 3 www-data www-data 4096 Sep 25 11:54 admin +drwxr-xr-x 3 www-data www-data 4096 Sep 25 11:54 appdata_occ3u0b61xxr +-rw-rw-r-- 1 www-data www-data 0 Sep 25 11:54 index.html +-rw-rw-r-- 1 www-data www-data 0 Sep 25 11:53 nextcloud.log +/mnt/felhom-drives/scratch_hdd/kept +/mnt/felhom-drives/scratch_hdd/kept/nextcloud +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332 +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332/appdata +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332/.felhom-kept.json +126M /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332 + + nextcloud: occ user:add :: The account "drill369ffe" was created successfully Display name set to "drill369ffe" + nextcloud: seeded user drill369ffe +seeded: True +kept page: datok Az eltávolított alkalmazások lemezen maradt adatai. Megnézheted őket, betöltheted egy új telepítésbe, vagy törölheted. Magától a szerver soha nem töröl közülük semmit. Erről nem készül mentés. Nextcloud 2026-09-25 13:53 · 125.4 MB Visszatölthető innen: nincs mentés — csak a fájlok Az alkalmazás újra telepítve van. Megnézem Törlés A(z) Nextcloud megőrzött adatai (125.4 MB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: Nextcloud Végleges törlés RomM 2026-09-24 22:11 · 12.0 KB Visszatölthető innen: nincs mentés — csak a fájlok Megnézem Törlés A(z) RomM megőrzött adatai (12.0 KB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: RomM Végleges törlés Paperless-ngx 2026-09-15 11:13 · 1.2 MB Visszatölthető innen: nincs mentés — csak a fájlok A diff --git a/documentation/audits/night-2026-09-26/E/E5-4-remove-keep.txt b/documentation/audits/night-2026-09-26/E/E5-4-remove-keep.txt new file mode 100644 index 00000000..3ce8481a --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-4-remove-keep.txt @@ -0,0 +1,14 @@ +# E5 4-remove-keep — 2026-09-25T13:58:37+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc2 +stop 200 +remove keep data + backups -> 200 {'ok': True, 'data': {'removed': 'nextcloud', 'volumes_removed': ['nextcloud_nextcloud_db_data', 'nextcloud_nextcloud_html', 'nextcloud_nextcloud_redis_data'], 'hdd_paths_removed': [], 'hdd_paths_pres +after: False nextcloud +paperless +-rw-r--r-- 1 root root 1591 2026-09-25T11:57:50 manifest.json +drwxr-xr-x 2 root root 4096 2026-09-25T11:57:41 volume-dumps + +/mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud/db-dumps: +total 864 +drwxr-xr-x 2 root root 4096 2026-09-25T11:57:00 . +drwxr-xr-x 5 root root 4096 2026-09-25T11:57:50 .. +-rw-r--r-- 1 root root 873058 2026-09-25T11:57:00 nextcloud-mariadb.sql + diff --git a/documentation/audits/night-2026-09-26/E/E5-4a-window-to-13:57.txt b/documentation/audits/night-2026-09-26/E/E5-4a-window-to-13:57.txt new file mode 100644 index 00000000..d3678406 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-4a-window-to-13:57.txt @@ -0,0 +1,12 @@ +13:55:26 before: +2026/09/25 11:52:50 scheduler.go:132: [INFO] [scheduler] Daily job offsite-proof scheduled for 2026-09-26 05:30 CEST +2026/09/25 11:52:50 scheduler.go:132: [INFO] [scheduler] Daily job metrics-prune scheduled for 2026-09-26 04:00 CEST +2026/09/25 11:52:50 scheduler.go:132: [INFO] [scheduler] Daily job fill-watch scheduled for 2026-09-26 03:30 CEST +13:55:26 POST /backups/window window_start=13:57 -> 303 +13:55:32 after: +2026/09/25 11:55:26 auth.go:142: [DEBUG] [web] auth: valid session for POST /backups/window +2026/09/25 11:55:26 server.go:545: [DEBUG] [web] ServeHTTP: POST /backups/window from 172.18.0.6:51602 +2026/09/25 11:55:26 scheduler.go:172: [INFO] [scheduler] Daily job db-dump rescheduled 02:30 → 13:57 (next run 2026-09-25 13:57 CEST) +2026/09/25 11:55:26 scheduler.go:172: [INFO] [scheduler] Daily job tier2-backup rescheduled 03:30 → 14:57 (next run 2026-09-25 14:57 CEST) +2026/09/25 11:55:26 scheduler.go:172: [INFO] [scheduler] Daily job offbox-backup rescheduled 04:15 → 15:42 (next run 2026-09-25 15:42 CEST) +2026/09/25 11:55:26 backup_handlers.go:65: [INFO] [web] backup window set to 13:57 (legs 13:57/14:57/15:42) diff --git a/documentation/audits/night-2026-09-26/E/E5-4b-dbdump.txt b/documentation/audits/night-2026-09-26/E/E5-4b-dbdump.txt new file mode 100644 index 00000000..38ff21cc --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-4b-dbdump.txt @@ -0,0 +1,19 @@ +2026/09/25 11:55:26 scheduler.go:172: [INFO] [scheduler] Daily job db-dump rescheduled 02:30 → 13:57 (next run 2026-09-25 13:57 CEST) +2026/09/25 11:55:26 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/09/25 11:57:00 scheduler.go:346: [INFO] [scheduler] Running job: db-dump +2026/09/25 11:57:00 dbdump.go:410: [INFO] [backup] DB dump: nextcloud-db → nextcloud-mariadb.sql (852.6 KB, 500ms, 131 tables) +2026/09/25 11:57:17 backup.go:923: [INFO] [backup] Stopping nextcloud for safe volume dump +2026/09/25 11:57:25 backup.go:824: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_db_data → 164.5 MB +2026/09/25 11:57:40 backup.go:824: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_html → 761.6 MB +2026/09/25 11:57:41 backup.go:824: [INFO] [backup] Volume dump: nextcloud/nextcloud_nextcloud_redis_data → 182.0 KB +2026/09/25 11:57:41 backup.go:933: [INFO] [backup] Restarting nextcloud after volume dump +2026/09/25 11:57:50 recovery_unit.go:235: [INFO] [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026/09/25 11:58:09 scheduler.go:363: [INFO] [scheduler] Job db-dump completed (took 1m9.379s) +-rw-r--r-- 1 root root 1591 2026-09-25T11:57:50 manifest.json +drwxr-xr-x 2 root root 4096 2026-09-25T11:57:41 volume-dumps + +/mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud/db-dumps: +total 864 +drwxr-xr-x 2 root root 4096 2026-09-25T11:57:00 . +drwxr-xr-x 5 root root 4096 2026-09-25T11:57:50 .. +-rw-r--r-- 1 root root 873058 2026-09-25T11:57:00 nextcloud-mariadb.sql diff --git a/documentation/audits/night-2026-09-26/E/E5-4c-window-back.txt b/documentation/audits/night-2026-09-26/E/E5-4c-window-back.txt new file mode 100644 index 00000000..91b54278 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-4c-window-back.txt @@ -0,0 +1,12 @@ +13:58:19 before: +2026/09/25 11:55:26 scheduler.go:172: [INFO] [scheduler] Daily job db-dump rescheduled 02:30 → 13:57 (next run 2026-09-25 13:57 CEST) +2026/09/25 11:55:26 scheduler.go:172: [INFO] [scheduler] Daily job tier2-backup rescheduled 03:30 → 14:57 (next run 2026-09-25 14:57 CEST) +2026/09/25 11:55:26 scheduler.go:172: [INFO] [scheduler] Daily job offbox-backup rescheduled 04:15 → 15:42 (next run 2026-09-25 15:42 CEST) +13:58:20 POST /backups/window window_start=02:30 -> 303 +13:58:26 after: +2026/09/25 11:58:19 auth.go:142: [DEBUG] [web] auth: valid session for POST /backups/window +2026/09/25 11:58:19 server.go:545: [DEBUG] [web] ServeHTTP: POST /backups/window from 172.18.0.6:36118 +2026/09/25 11:58:19 scheduler.go:172: [INFO] [scheduler] Daily job db-dump rescheduled 13:57 → 02:30 (next run 2026-09-26 02:30 CEST) +2026/09/25 11:58:19 scheduler.go:172: [INFO] [scheduler] Daily job tier2-backup rescheduled 14:57 → 03:30 (next run 2026-09-26 03:30 CEST) +2026/09/25 11:58:19 scheduler.go:172: [INFO] [scheduler] Daily job offbox-backup rescheduled 15:42 → 04:15 (next run 2026-09-26 04:15 CEST) +2026/09/25 11:58:20 backup_handlers.go:65: [INFO] [web] backup window set to 02:30 (legs 02:30/03:30/04:15) diff --git a/documentation/audits/night-2026-09-26/E/E5-5-use.txt b/documentation/audits/night-2026-09-26/E/E5-5-use.txt new file mode 100644 index 00000000..b7ab9bdc --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-5-use.txt @@ -0,0 +1,24 @@ +# E5 5-use — 2026-09-25T13:59:46+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc2 +ask first -> 409 use_offered=True use_desc=Az adatbázist a 2026-09-25-i mentésből töltjük vissza, és a régi fájlokkal indítjuk az alkalmazást. Ami ezután változott, hiányozhat. +deploy kept_data=use -> 202 {'ok': True, 'message': 'A(z) Nextcloud megőrzött adatainak betöltése elindult.'} +deployed after (35.2, 'running') +controller lines: +2026/09/25 11:59:58 kept_install.go:94: [INFO] [api] Deploy nextcloud: USE MY KEPT DATA — loading from /mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud (2026-09-25T11:57:50Z) under the kept files [/mnt/felhom-drives/scratch_hdd/appdata/nextcloud] +2026/09/25 11:59:58 restore_unit.go:372: [WARN] [backup] Restore nextcloud: generated replacement for [NEXTCLOUD_ADMIN_PASSWORD] — the credential was reset (old value unrecoverable); no data-encrypting key was involved, but a regenerated DATABASE password will not match the restored data directory +2026/09/25 11:59:58 restore_unit.go:383: [INFO] [backup] Restoring nextcloud from recovery unit /mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud: images=3, secrets recovered=2/3, data_keys=0 +2026/09/25 11:59:58 restore.go:152: [INFO] [backup] Restoring Docker volume nextcloud_nextcloud_db_data for nextcloud +2026/09/25 11:59:59 restore.go:152: [INFO] [backup] Restoring Docker volume nextcloud_nextcloud_html for nextcloud +2026/09/25 12:00:05 restore.go:152: [INFO] [backup] Restoring Docker volume nextcloud_nextcloud_redis_data for nextcloud +2026/09/25 12:00:06 restore.go:195: [INFO] [backup] Restored 3 Docker volume(s) for nextcloud +2026/09/25 12:00:06 deploy.go:708: [INFO] [stacks] Redeploying nextcloud from recovery unit with 7 env vars +2026/09/25 12:00:07 restore_db.go:87: [INFO] [backup] Restore nextcloud: replaying DB dump into nextcloud-db (mariadb) +2026/09/25 12:00:16 restore_db.go:97: [INFO] [backup] Restore nextcloud: replayed 1 DB dump(s) +2026/09/25 12:00:30 restore_unit.go:468: [INFO] [backup] Restore-from-unit completed: nextcloud — 3 volume(s) of 3 listed, 1 database(s) of 1 listed +2026/09/25 12:00:30 kept_load.go:136: [INFO] [backup] kept load nextcloud from /mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud done in 32s (volumes 3/3, dbs 1/1) +2026/09/25 12:00:31 kept.go:515: [INFO] [stacks] after_load nextcloud: nextcloud [php occ files:scan --all] in 688ms (err=): Starting scan for user 1 out of 2 (admin) + + nextcloud: readback of the seeded user found=True +seed (the account) reads back: True +PROPFIND files of drill369ffe -> 207; before-backup.txt visible=True +PROPFIND files of drill369ffe -> 207; after-backup.txt visible=True +PROPFIND files of drill369ffe -> 207; never-written.txt visible=False diff --git a/documentation/audits/night-2026-09-26/E/E5-6-fresh-again.txt b/documentation/audits/night-2026-09-26/E/E5-6-fresh-again.txt new file mode 100644 index 00000000..547535bd --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-6-fresh-again.txt @@ -0,0 +1,19 @@ +# E5 6-fresh-again — 2026-09-25T14:09:14+0200; controller gitea.dooplex.hu/admin/felhom-controller:0.274.0-rc2 +remove keep data + backups -> 200 +deploy fresh -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +deployed after (50.3, 'running') +kept now: /mnt/felhom-drives/scratch_hdd/kept +/mnt/felhom-drives/scratch_hdd/kept/nextcloud +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332 +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332/appdata +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332/.felhom-kept.json +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014 +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/appdata +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/.felhom-kept.json +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/unit +126M /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332 +1.1G /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014 + +remove the fresh install (keep data, delete backups) -> 200 {'ok': True, 'data': {'removed': 'nextcloud', 'volumes_removed': ['nextcloud_nextcloud_db_data', 'nextcloud_nextcloud_html', 'nextcloud_nextcloud_redis_data'], +kept page hu: datok Az eltávolított alkalmazások lemezen maradt adatai. Megnézheted őket, betöltheted egy új telepítésbe, vagy törölheted. Magától a szerver soha nem töröl közülük semmit. Erről nem készül mentés. Nextcloud 2026-09-25 14:10 · 62.8 MB Visszatölthető innen: nincs mentés — csak a fájlok Megnézem Törlés A(z) Nextcloud megőrzött adatai (62.8 MB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: Nextcloud Végleges törlés Nextcloud 2026-09-25 14:10 · 1.0 GB Visszatölthető innen: a vele megőrzött mentés, 2026-09-25 13:57 Betöltés Megnézem Törlés A(z) Nextcloud megőrzött adatai (1.0 GB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: Nextcloud Végleges törlés Nextcloud 2026-09-25 13:53 · 125.4 MB Visszatölthető innen: nincs mentés — csak a fájlok Megnézem Törlés A(z) Nextcloud megőrzött adatai (125.4 MB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: Nextcloud Végleges törlés RomM 2026-09-24 22:11 · 12.0 KB Visszatölthető innen: nincs mentés — csak a fájlok Megnézem Törlés A(z) RomM megőrzött adatai (12.0 KB) véglegesen törlődnek. Ezt nem lehet visszacsinálni. A törléshez írd be: RomM Végleges törlés Paperless-ngx 2026-09-15 11:13 · 1.2 MB Visszatölthető innen: nincs mentés — csak a fájlok Az alkalmazás újra telepítve van. Megnézem Törlés A(z) Paperless-ngx megőrzött adatai (1.2 MB) véglegesen törlődnek. Ezt nem le +kept page en: monitor → Back Kept data Data that removed apps left on the drives. You can look at it, load it into a new install, or delete it. The server never deletes any of it by itself. This is not backed up. Nextcloud 2026-09-25 14:10 · 62.8 MB Can be loaded from: no backup — the files only Look Delete The kept data of Nextcloud (62.8 MB) is deleted for good. This cannot be undone. To delete it, type: Nextcloud Delete for good Nextcloud 2026-09-25 14:10 · 1.0 GB Can be loaded from: the backup kept with it, 2026-09-25 13:57 Load Look Delete The kept data of Nextcloud (1.0 GB) is deleted for good. This cannot be undone. To delete it, type: Nextcloud Delete for good Nextcloud 2026-09-25 13:53 · 125.4 MB Can be loaded from: no backup — the files only Look Delete The kept data of Nextcloud (125.4 MB) is deleted for good. This cannot be undone. To delete it, type: Nextcloud Delete for good RomM 2026-09-24 22:11 · 12.0 KB Can be loaded from: no backup — the files only Look Delete The kept data of RomM (12.0 KB) is deleted for good. This cannot be undone. To delete it, type: RomM Delete for good Paperless-ngx 2026-09-15 11:13 · 1.2 MB Can be loaded from: no backup — the files only The app is installed again. Look Delete The kept data of Paperless-ngx (1.2 MB) is deleted for good. This cannot be undone. To delete it, type: Paperless-ngx Delete for good (function(){ var burger=document.querySelec diff --git a/documentation/audits/night-2026-09-26/E/E5-7-load-delete.txt b/documentation/audits/night-2026-09-26/E/E5-7-load-delete.txt new file mode 100644 index 00000000..5f61357b --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-7-load-delete.txt @@ -0,0 +1,24 @@ +listed paths: ['/mnt/felhom-drives/scratch_hdd/appdata/nextcloud', '/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014', '/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014', '/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332', '/mnt/felhom-drives/scratch_hdd/userdata/romm/appdata/romm', '/mnt/felhom-drives/scratch_hdd/appdata/paperless'] +the loadable item: /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014 +LOAD while the leftover occupies the folder -> 302 {'flash_error': 'Az alkalmazás mappájában már vannak adatok, ezért a megőrzött fájlokat nem tettük vissza. Nem változott semmi.'} +DELETE leftover with a WRONG name -> 302 {'flash_error': 'A beírt név nem egyezik (Nextcloud), ezért nem töröltünk semmit.'} +leftover still there: yes +DELETE a path that is not listed (userdata) -> 302 {'flash_error': 'Ez nem megőrzött adat, ezért nem nyúltunk hozzá.'} +DELETE leftover with the right name -> 302 {'flash': 'A(z) Nextcloud megőrzött adatai törölve.'} +leftover now: gone +LOAD the dated item with its backup -> 302 {'flash': 'flash.restore.started'} +controller lines: +2026/09/25 12:12:38 kept.go:424: [INFO] [stacks] kept nextcloud: moved /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/appdata/nextcloud back → /mnt/felhom-drives/scratch_hdd/appdata/nextcloud (load) +2026/09/25 12:12:38 kept_handlers.go:176: [INFO] [web] kept LOAD nextcloud: /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014 from /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/unit (from 172.18.0.6:43270) +2026/09/25 12:12:38 restore_unit.go:383: [INFO] [backup] Restoring nextcloud from recovery unit /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/unit: images=3, secrets recovered=2/3, data_keys=0 +2026/09/25 12:13:12 restore_unit.go:468: [INFO] [backup] Restore-from-unit completed: nextcloud — 3 volume(s) of 3 listed, 1 database(s) of 1 listed +2026/09/25 12:13:12 kept_load.go:136: [INFO] [backup] kept load nextcloud from /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/unit done in 34s (volumes 3/3, dbs 1/1) +2026/09/25 12:13:13 kept.go:515: [INFO] [stacks] after_load nextcloud: nextcloud [php occ files:scan --all] in 765ms (err=): Starting scan for user 1 out of 2 (admin) +2026/09/25 12:13:13 kept.go:444: [INFO] [stacks] kept nextcloud: /mnt/felhom-drives/scratch_hdd/backups/primary/nextcloud already holds a unit — the loaded one stays in /mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014/unit (listed; the househo + + nextcloud: readback of the seeded user found=True +account reads back: True +the loaded item left the list: False | list now: ['/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014', '/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332', '/mnt/felhom-drives/scratch_hdd/userdata/romm/appdata/romm', '/mnt/felhom-drives/scratch_hdd/appdata/paperless'] +PROPFIND files of drill369ffe -> 207; before-backup.txt visible=True +PROPFIND files of drill369ffe -> 207; after-backup.txt visible=True +PROPFIND files of drill369ffe -> 207; never-written.txt visible=False diff --git a/documentation/audits/night-2026-09-26/E/E5-8-filebrowser-ro.txt b/documentation/audits/night-2026-09-26/E/E5-8-filebrowser-ro.txt new file mode 100644 index 00000000..eb6d9382 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-8-filebrowser-ro.txt @@ -0,0 +1,17 @@ +# the file browser's kept-data mounts (docker inspect) — 2026-09-25T12:28:38Z +/mnt/felhom-drives/scratch_hdd/userdata/romm/appdata/romm -> /srv/megorzott/romm-romm rw=false +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_135332 -> /srv/megorzott/scratch_hdd-nextcloud-2026-09-25_135332 rw=false +/mnt/felhom-drives/scratch_hdd/kept/nextcloud/2026-09-25_141014 -> /srv/megorzott/scratch_hdd-nextcloud-2026-09-25_141014 rw=false +/mnt/felhom-drives/scratch_hdd/appdata/paperless -> /srv/megorzott/scratch_hdd-paperless rw=false +# the kept source in the file browser's own config + defaultEnabled: true + - path: "/srv/megorzott" + name: "Megőrzött adatok" + config: + defaultEnabled: true +userDefaults: +# a write into /srv/megorzott/romm-romm from inside the file browser's container: +touch: /srv/megorzott/romm-romm/cc-write-test: Read-only file system +rc=1 +# control: a write into its userdata source works: +rc=0 diff --git a/documentation/audits/night-2026-09-26/E/E5-redproof-owner.txt b/documentation/audits/night-2026-09-26/E/E5-redproof-owner.txt new file mode 100644 index 00000000..f600dedc --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E5-redproof-owner.txt @@ -0,0 +1,4 @@ +### RED-PROOF R-692: drop the ${HDD_PATH} owner filter +--- FAIL: TestKept_OwnerIsNeverTheFileBrowser (0.01s) + kept_test.go:295: the leftovers must be named by the app that binds them through HDD_PATH, never the file browser: map[cloudapp:Cloud App paperless:Filebrowser] +FAIL diff --git a/documentation/audits/night-2026-09-26/E/E6-demo-hp-kept-page.txt b/documentation/audits/night-2026-09-26/E/E6-demo-hp-kept-page.txt new file mode 100644 index 00000000..d8e2e322 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E6-demo-hp-kept-page.txt @@ -0,0 +1,2 @@ +demo-hp /kept-data items: [] +Megőrzött adatok Az eltávolított alkalmazások lemezen maradt adatai. Megnézheted őket, betöltheted egy új telepítésbe, vagy törölheted. Magától a szerver soha nem töröl közülük semmit. Erről nem készül mentés. Nincs megőrzött adat. (function(){ var burger=document.querySelector('.nav-burger'); var sidebar=document.getElementById('sidebar'); var backdrop=document.querySelector('.nav-backdrop'); if( diff --git a/documentation/audits/night-2026-09-26/E/E6-floor-0274.txt b/documentation/audits/night-2026-09-26/E/E6-floor-0274.txt new file mode 100644 index 00000000..fa5ed768 --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/E6-floor-0274.txt @@ -0,0 +1,5 @@ +# global floor 0.274.0 (declared MinAgent 0.131.0) — 2026-09-25T12:30:04Z +impact before: {"below":4,"valid":true,"version":"0.274.0"} +POST -> HTTP/1.1 303 See Other Location: /configuration?flash=floor_set +read back: Effective floor: v0.274.0 — source: DB (hub_settings) ; env fallback would be v0 +arrival after 16s of polling: demo-felhom=[gitea.dooplex.hu/admin/felhom-controller:0.274.0 healthy] demo-hp=[gitea.dooplex.hu/admin/felhom-controller:0.274.0 healthy] diff --git a/documentation/audits/night-2026-09-26/E/redproofs/kept.txt b/documentation/audits/night-2026-09-26/E/redproofs/kept.txt new file mode 100644 index 00000000..f9b8274c --- /dev/null +++ b/documentation/audits/night-2026-09-26/E/redproofs/kept.txt @@ -0,0 +1,25 @@ +## rule1: first attempt removed the block — build failed (unused variable), not a result +## rule2: no undo on a failed rename +--- FAIL: TestKept_KeepAsideIsARenameAndRollsBack (0.00s) + kept_test.go:101: the first folder was not put back after the failed move: stat /tmp/TestKept_KeepAsideIsARenameAndRollsBack288451927/001/drive/appdata/cloudapp/user1/file.txt: no such file or directory +## rule3: kept not protected +--- FAIL: TestKept_KeptDirIsProtected (0.00s) + kept_test.go:132: /kept is not in ProtectedHDDPaths +## rule4: delete without the listed check +--- FAIL: TestKept_DeleteRefusesAnythingNotListed (0.00s) + kept_test.go:185: delete of /tmp/TestKept_DeleteRefusesAnythingNotListed3626460414/001/drive/appdata/cloudapp: want ErrKeptNotListed, got +## install guard removed +--- FAIL: TestKept_DeployRefusesOverOldDataWithoutAChoice (0.01s) + kept_test.go:214: choice "": no choice was accepted — want ErrKeptDataChoice, got +## rule1 (redone — the first mutation did not compile): appdata prefix check disabled +--- FAIL: TestKept_OnlyAppdataBindsAreOldData (0.00s) + kept_test.go:64: old data = [/tmp/TestKept_OnlyAppdataBindsAreOldData2495434777/001/drive/appdata/cloudapp /tmp/TestKept_OnlyAppdataBindsAreOldData2495434777/001/drive/media /tmp/TestKept_OnlyAppdataBindsAreOldData2495434777/001/drive/userdata/Photos], want only [/tmp/TestKept_OnlyAppdataBindsAreOldData2495434777/001/drive/appdata/cloudapp] (a household folder must never be moved) +## backup: the unit's HDD_PATH check disabled +--- FAIL: TestKept_DBCopyIsTheNewestUsableForThisDrive (0.00s) + kept_load_test.go:43: a unit taken of another drive was offered: {UnitDir:/tmp/TestKept_DBCopyIsTheNewestUsableForThisDrive3608708086/004/backups/primary/nextcloud Tier:1 Time:2026-09-25 12:29:20.057488373 +0200 CEST DriveLabel:HDD} +## web: the :ro suffix dropped +--- FAIL: TestKept_FileBrowserBindsAreReadOnly (0.00s) + kept_fb_test.go:23: bind " - /mnt/felhom-drives/hdd_1/kept/nextcloud/2026-09-25_130000:/srv/megorzott/hdd_1-nextcloud-2026-09-25_130000" is not a read-only bind of /mnt/felhom-drives/hdd_1/kept/nextcloud/2026-09-25_130000 under /srv/megorzott +## api: keptDataAtInstall returns false at once +--- FAIL: TestKept_InstallAPIAsksAndInstallsNothing (0.00s) + kept_install_test.go:57: no choice: handled=false code=200 body=map[] diff --git a/documentation/audits/night-2026-09-26/F/F2-live-preview.txt b/documentation/audits/night-2026-09-26/F/F2-live-preview.txt new file mode 100644 index 00000000..1e5bec4b --- /dev/null +++ b/documentation/audits/night-2026-09-26/F/F2-live-preview.txt @@ -0,0 +1 @@ +cloudflare_manual = ["the Cloudflare tunnel that serves enkisfelhom.hu (this customer's config carried its own tunnel token)", 'the DNS records of enkisfelhom.hu (the apex and *.enkisfelhom.hu)'] diff --git a/documentation/audits/night-2026-09-26/part0/README.md b/documentation/audits/night-2026-09-26/part0/README.md new file mode 100644 index 00000000..3504795e --- /dev/null +++ b/documentation/audits/night-2026-09-26/part0/README.md @@ -0,0 +1,33 @@ +# Part 0 — the rulings recorded, last night read (read-only), 2026-09-25 midday + +**Recorded:** `09` §3 decisions 35 and 36; D3 in `STATUS.md`'s decision section; `CONTEXT.md`. + +## Last night (24 → 25 September) on both demo boxes + +The controllers were restarted by the 0.272.0 delivery at ~11:30 CEST, so their own logs and debug rings no +longer reach the night (5 000-line ring). Read instead from the hub's stored reports (`P0-hub-update-leg.txt`, +a copy of the hub DB with its `-wal`, deleted after) and the agents' journals (`P0-wholebox-backup.txt`): + +| box | update leg | steps | whole-box backup | +|---|---|---|---| +| demo-felhom | 04:15:46–04:16:06 CEST, after the off-site copy | opengist 1.13 → 1.15 **done** in 20.1 s | ran **07:29:19–07:30:31 CEST** to `felhom-backup` — after the leg had ended at 04:16; it had nothing to wait for | +| demo-hp | 04:18:41, 0.07 s | none (every app current) — **reported as `"steps": null`** (R-687's cosmetic item, seen live; fixed in v0.273.0) | **none** — its last was the evening's 21:59 hand-triggered one (Part A4 of the previous night), not due | + +**R-687 item 4 (the full-system gate waiting for the leg) did not occur on either box again**: on demo-felhom the +backup came three hours after the leg; demo-hp had none due. + +**Found while reading, filed as R-689:** demo-hp's scheduled restore test picks the golden template +`local:backup/felhom-golden-0.236.0.tar.zst` as "the newest settled archive" and fails with +`HTTP 403 … missing privilege VM.Backup` every 6 h (2026-09-24 10:36, 2026-09-25 04:57 and 10:57) +— `P0-demo-hp-restoretest-golden.txt`. Agent-side; the agent is untouched this session. + +## Which boxes run docmost (the brief said: demo-hp only) + +**TRUE.** demo-felhom runs cloudflared, controller, filebrowser, opengist, traefik — no docmost. demo-hp runs +docmost 0.96.0 on `postgres:16-alpine` (`P0-docmost-demo-hp.txt`): pinned and installed refs equal, digests +recorded 2026-09-22; deployed 2026-08-31. + +Its last copies, per tier: **own unit (Tier 1)** `db-dumps/` 2026-09-25 02:15 UTC, manifest 09:31 UTC; +**second drive (Tier 2)** `hdd_1/backups/secondary/docmost/recovery-unit` mirrored 2026-09-25 01:30 UTC (the layout +marker) — directory mtime 2026-09-24 20:59; **off-site** — the night's off-site run covered 9 apps / 90 snapshots +at 04:15–04:18 CEST (the previous night's record); per-app snapshot times were not read. diff --git a/documentation/audits/night-2026-09-26/tools/e5.py b/documentation/audits/night-2026-09-26/tools/e5.py new file mode 100644 index 00000000..724e1288 --- /dev/null +++ b/documentation/audits/night-2026-09-26/tools/e5.py @@ -0,0 +1,88 @@ +"""e5.py — Part E's live proof on 9202 (controller 0.274.0-rc1, drill catalog), one step per call. +Every act through the product's own endpoints (the ones the pages call). Evidence: E/E5-.txt.""" +import json, sys, time, re +import walk as w, fixtures +step = sys.argv[1] +OUT = open(f"{w.EV}/E/E5-{step}.txt", "w", buffering=1) +def say(*a): + w.say(*a); OUT.write(" ".join(map(str, a)) + "\n") +HDD = "/mnt/felhom-drives/scratch_hdd" +TOK = w.SC + "/seed-tokens.json" +w.login() +def kept_page(lang=""): + h = w.page("/kept-data" + ("?lang=en" if lang else "")) + t = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", h)) + return t +def deploy(choice=None, lang=""): + vals = w.deploy_values("nextcloud", "c-nc"); vals["HDD_PATH"] = HDD + body = {"values": vals} + if choice: + body["kept_data"] = choice + return w.ctl("POST", "/api/stacks/nextcloud/deploy" + ("?lang=en" if lang else ""), body) +def wait_deployed(limit=600): + t0 = time.time() + while time.time() - t0 < limit: + st = w.stack("nextcloud") + if st.get("deployed") and (st.get("app_config") or {}).get("pinned_images") and st.get("state") in ("running", "unhealthy", "degraded"): + return round(time.time() - t0, 1), st.get("state") + time.sleep(5) + return None, w.stack("nextcloud").get("state") +def g(cmd): + return w.guest(cmd) +say(f"# E5 {step} — {time.strftime('%FT%T%z')}; controller {g('cat /etc/felhom-controller-image').strip()}") +if step == "1-list": + say("drive:", g(f"ls -la {HDD}; ls -la {HDD}/appdata {HDD}/kept 2>&1; du -sh {HDD}/appdata/nextcloud 2>&1")) + for lang in ("", "en"): + t = kept_page(lang) + i = t.find("extcloud") + say(f"GET /kept-data{'?lang=en' if lang else ''}: ...{t[max(0,i-300):i+500]}...") +elif step == "2-ask": + for lang in ("", "en"): + code, d = deploy(None, lang) + say(f"deploy, no choice, lang={lang or 'hu'} -> {code}") + say(json.dumps(d, ensure_ascii=False, indent=1)[:1600]) + say("still not installed:", w.stack("nextcloud").get("deployed"), g(f"ls {HDD}/appdata")) +elif step == "3-fresh": + code, d = deploy("fresh") + say(f"deploy kept_data=fresh -> {code} {str(d)[:200]}") + say("deployed after", wait_deployed()) + say("drive now:", g(f"ls -la {HDD}/appdata {HDD}/appdata/nextcloud 2>&1 | head -20; find {HDD}/kept -maxdepth 3 | head; du -sh {HDD}/kept/nextcloud/* 2>&1")) + toks = json.load(open(TOK)); toks["nextcloud"] = fixtures.FIXTURES["nextcloud"].seed(w, "c-nc", say) + json.dump(toks, open(TOK, "w"), default=str) + say("seeded:", toks["nextcloud"] is not None) + t = kept_page(); i = t.find("extcloud"); say("kept page:", t[max(0, i-200):i+600]) +elif step == "4-remove-keep": + code, d = w.ctl("POST", "/api/stacks/nextcloud/stop"); say("stop", code) + time.sleep(15) + code, d = w.ctl("POST", "/api/stacks/nextcloud/remove", {"remove_hdd_data": False, "remove_backups": False}) + say(f"remove keep data + backups -> {code} {str(d)[:200]}") + time.sleep(8) + say("after:", w.stack("nextcloud").get("deployed"), g(f"ls {HDD}/appdata; ls -la --time-style=+%FT%T {HDD}/backups/primary/nextcloud {HDD}/backups/primary/nextcloud/db-dumps 2>&1 | tail -8")) +elif step == "5-use": + code, d = deploy(None) + say(f"ask first -> {code} use_offered={(d.get('data') or {}).get('use_offered')} use_desc={(d.get('data') or {}).get('use_desc')}") + since = g("date -u +%Y-%m-%dT%H:%M:%SZ").strip() + code, d = deploy("use") + say(f"deploy kept_data=use -> {code} {str(d)[:200]}") + say("deployed after", wait_deployed(1200)) + for i in range(60): + if "after_load" in g(f"docker logs --since {since} felhom-controller 2>&1 | grep -c after_load") or i > 58: + break + time.sleep(5) + time.sleep(20) + say("controller lines:\n" + g(f"docker logs --since {since} felhom-controller 2>&1 | grep -iE 'kept|after_load|restore|unit|Restor' | grep -v DEBUG | cut -c1-300 | head -40")) + toks = json.load(open(TOK)) + say("seed (the account) reads back:", fixtures.FIXTURES["nextcloud"].verify(w, "c-nc", toks["nextcloud"], say)) +elif step == "6-fresh-again": + code, d = w.ctl("POST", "/api/stacks/nextcloud/stop"); time.sleep(15) + code, d = w.ctl("POST", "/api/stacks/nextcloud/remove", {"remove_hdd_data": False, "remove_backups": False}) + say(f"remove keep data + backups -> {code}") + time.sleep(8) + code, d = deploy("fresh"); say(f"deploy fresh -> {code} {str(d)[:160]}"); say("deployed after", wait_deployed()) + say("kept now:", g(f"find {HDD}/kept -maxdepth 3 | head -20; du -sh {HDD}/kept/nextcloud/* 2>&1")) + code, d = w.ctl("POST", "/api/stacks/nextcloud/stop"); time.sleep(15) + code, d = w.ctl("POST", "/api/stacks/nextcloud/remove", {"remove_hdd_data": False, "remove_backups": True}) + say(f"remove the fresh install (keep data, delete backups) -> {code} {str(d)[:160]}") + time.sleep(8) + for lang in ("", "en"): + t = kept_page(lang); i = t.find("extcloud"); say(f"kept page {lang or 'hu'}:", t[max(0, i-200):i+1200]) diff --git a/documentation/audits/night-2026-09-26/tools/e5b.py b/documentation/audits/night-2026-09-26/tools/e5b.py new file mode 100644 index 00000000..dfc20b1a --- /dev/null +++ b/documentation/audits/night-2026-09-26/tools/e5b.py @@ -0,0 +1,47 @@ +import re, time, json, sys +import walk as w, fixtures +OUT = open(f"{w.EV}/E/E5-7-load-delete.txt", "w", buffering=1) +def say(*a): + w.say(*a); OUT.write(" ".join(map(str, a)) + "\n") +HDD = "/mnt/felhom-drives/scratch_hdd" +w.login() +import os +sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip(); csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip() +def form(path, data): + args = ["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-X", "POST", "--data-urlencode", f"_csrf={csrf}"] + for k, v in data.items(): + args += ["--data-urlencode", f"{k}={v}"] + r = w.sh(args + [f"{w.BASE}{path}"]) + loc = re.search(r"(?im)^location:\s*(\S+)", r.stdout or "") + code = (r.stdout or "").split(" ", 2)[1] if r.stdout else "?" + return code, (loc.group(1) if loc else "") +import urllib.parse +def flash(loc): + q = urllib.parse.parse_qs(urllib.parse.urlparse(loc).query) + return {k: v[0] for k, v in q.items()} +items = re.findall(r'name="path" value="([^"]+)"', w.page("/kept-data")) +say("listed paths:", items) +leftover = f"{HDD}/appdata/nextcloud" +dated = [p for p in items if "/kept/nextcloud/" in p] +unitdated = [p for p in dated if w.guest(f"test -d {p}/unit && echo yes").strip() == "yes"][0] +say("the loadable item:", unitdated) +c, loc = form("/kept-data/load", {"path": unitdated}); say(f"LOAD while the leftover occupies the folder -> {c} {flash(loc)}") +c, loc = form("/kept-data/delete", {"path": leftover, "confirm": "Nextclod"}); say(f"DELETE leftover with a WRONG name -> {c} {flash(loc)}") +say("leftover still there:", w.guest(f"test -d {leftover} && echo yes || echo GONE").strip()) +c, loc = form("/kept-data/delete", {"path": "/mnt/felhom-drives/scratch_hdd/userdata", "confirm": "userdata"}); say(f"DELETE a path that is not listed (userdata) -> {c} {flash(loc)}") +c, loc = form("/kept-data/delete", {"path": leftover, "confirm": "Nextcloud"}); say(f"DELETE leftover with the right name -> {c} {flash(loc)}") +say("leftover now:", w.guest(f"test -d {leftover} && echo STILL || echo gone").strip()) +since = w.guest("date -u +%Y-%m-%dT%H:%M:%SZ").strip() +c, loc = form("/kept-data/load", {"path": unitdated}); say(f"LOAD the dated item with its backup -> {c} {flash(loc)}") +t0 = time.time() +while time.time() - t0 < 900: + st = w.stack("nextcloud") + if st.get("deployed") and st.get("state") == "running" and "after_load" in w.guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -c after_load"): + break + time.sleep(10) +time.sleep(20) +say("controller lines:\n" + w.guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -iE 'kept|after_load|Restore-from-unit' | grep -v DEBUG | cut -c1-260 | head -12")) +toks = json.load(open(w.SC + "/seed-tokens.json")) +say("account reads back:", fixtures.FIXTURES["nextcloud"].verify(w, "c-nc", toks["nextcloud"], say)) +items2 = re.findall(r'name="path" value="([^"]+)"', w.page("/kept-data")) +say("the loaded item left the list:", unitdated not in items2, "| list now:", items2) diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index f197b67d..8f473eee 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -398,3 +398,11 @@ Compressed here to title, shipping version, evidence, and the sentences that sta | **R-671** | **The undo copies kept by a hold survived the hold's clearing by a restore (P3).** v0.272.0: the restore that lifts an update hold removes that hold's undo copies (never a restore hold, never mid-update). Live on 9202: navidrome held with 1 copy kept → the backup page's restore → hold CLEARED, "removed 1 undo cop(y/ies)", 0 copies left, data read back. | v0.272.0, 2026-09-25 | `git show eb1c56a:documentation/backlog/OPEN-ITEMS.md`; `audits/retire-peti-2026-09-25/B/live/` | | **R-670** | **Every undo logged a false `backup block rejected … docker-compose.yml unreadable` (P3).** v0.272.0: probe-only copies load with `stacks.LoadProbeMetadata`. Live on 9202: an undo of navidrome (a backup block in its `.felhom.yml`) logged 0 such lines. | v0.272.0, 2026-09-25 | `git show eb1c56a:documentation/backlog/OPEN-ITEMS.md`; `audits/retire-peti-2026-09-25/B/live/r670-verdict.txt` | | **R-677** | **A re-tested floating tag's badge age read the tag's date (P3).** v0.272.0: `stacks.BehindSinceAge` — a digest-only move counts from `tested_at`; both producers. Live on 9202: „Frissítés elérhető — ma” / "Update available — today" with `catalog_since` two days old. | v0.272.0, 2026-09-25 | `git show eb1c56a:documentation/backlog/OPEN-ITEMS.md`; `audits/retire-peti-2026-09-25/B/live/live272.json` | + +## 2026-09-25 (evening) — the box converts a PostgreSQL major; kept data (controller v0.273.0 + v0.274.0, hub v0.125.0) + +| id | what closed | closed | where the full text is | +|---|---|---|---| +| **R-657** | **A reinstall over kept data ran silently into the old files (nextcloud never installed) (P2).** Operator ruling 2026-09-25 (`09` §3 decision 36), built in v0.274.0: the install asks „use my kept data" / „start fresh" (409 `kept_data_choice` until chosen); start fresh renames into `/kept///`; the „Megőrzött adatok" page lists, loads, deletes (typed). Proven live on 9202 in both languages (`audits/night-2026-09-26/E/E5-*`). | CLOSED 2026-09-25 — controller v0.274.0 | `git show 3386041e6215:documentation/backlog/OPEN-ITEMS.md`; `audits/night-2026-09-26/E/` | +| **R-690** | **The removed-app restore (R-487) never found a unit kept on a DATA drive (P1).** v0.274.0: `isStackDeployed` instead of `GetStackComposePath` (true for every catalog app); the R-487 test's fake had answered it for deployed apps only. Proven live: „use my kept data" loaded from `scratch_hdd/backups/primary/nextcloud`. | CLOSED 2026-09-25 — controller v0.274.0 | filed and closed this session; `audits/night-2026-09-26/E/E1-README.md` | +| **R-692** | **The Kept-data list named two leftovers "Filebrowser" (P2).** Found live on 9202 (0.274.0-rc1, 2026-09-25): the read-only view makes the file browser's compose bind every kept folder by absolute path, and the owner lookup took it. No live folder was at risk. Fixed before release in v0.274.0: an owner binds the folder through `${HDD_PATH}` (the folder or one inside it), never a protected stack. `TestKept_OwnerIsNeverTheFileBrowser`, red-proofed. `audits/night-2026-09-26/E/` | CLOSED 2026-09-25 — fixed in v0.274.0 before it shipped | filed and closed this session; `audits/night-2026-09-26/E/E5-redproof-owner.txt` | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 418fa969..55f08887 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -665,14 +665,14 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-444** | **[P3-LOW] Nothing runs `pct fstrim` on the fleet, and demo-hp's thin pool was carrying ~23.8 GB of blocks the guest had already freed.** MEASURED 2026-09-01 during this spike's teardown: the run itself added ~1.05 GiB that `local-lvm` did not reclaim on delete (68.97% -> 70.91%); `fstrim` INSIDE the unprivileged container is refused (`FITRIM ioctl failed: Operation not permitted`, all three mounts); `pct fstrim 9201` from the PVE host then trimmed **30.2 GiB + 57 GiB** and took `local-lvm` to **26.78%** — **23.8 GB BELOW this run's own starting point**, i.e. the surplus was long-standing, not ours. **Why it is not merely housekeeping:** a thin pool that only ever grows can reach 100% from DELETED data alone, and a full thin pool takes every guest on the host read-only. demo-hp had 16.4 GB free before the trim. **Not urgent, and the row says so** — but the appliance has no periodic trim and no operator surface reports the gap between guest-free and pool-used. Owner: **CC.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: CC** | | **R-445** | **[P3-LOW] Hub app telemetry survives the app's removal, so a 15-minute throwaway now sets a FLEET-WIDE memory recommendation.** MEASURED 2026-09-01: this spike's Phase 6 Nextcloud existed for ~15 minutes on demo-hp, spent part of it crash-looping, and was then removed with all volumes. The hub's `/apps/nextcloud` page still reports `Deployments`, `Avg Memory 208 MB`, `P95 Memory 280 MB` and **`Suggested Limit (P95x1.2) = 352 MB`**, plus three MariaDB `io_uring` rows under Known Issues attributed to demo-hp. **The suggested limit is an operator-facing recommendation derived from a sample that no longer exists anywhere** — and Nextcloud is a real catalog app whose limit someone may act on. **RETAINED DELIBERATELY BY THIS RUN, NOT CLEARED, and the reason is part of the row:** the hub offers `POST /apps/nextcloud/reset-telemetry` whose own confirm reads *"Delete all telemetry data for nextcloud? This cannot be undone."* — an irreversible write on the operator's surface, and the operator authorised Phase 6, not this. **The one-line command is recorded in the audit doc so it is a decision, not a task.** The general question is the row: should telemetry for an app with zero live deployments age out, or be excluded from the suggestion? Owner: **VIKTOR rules, CC implements.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: VIKTOR rules, CC implements** | | **R-446** | **[P2-MEDIUM] „Naprakész" can be FALSE, and the badge that says it cannot tell.** Slice 2 (controller v0.233.0, 2026-09-02) compares the RECORDED image reference per compose service against the reference the current template pins, and **queries no registry** — deliberately: a customer's box must not depend on reaching eight upstream registries to render a page (`felhom-controller/controller/internal/web/updatebadge.go`, `compareInstalledToTemplate`). **For the 23 floating pins that comparison is blind by construction:** `postgres:16-alpine`, `mariadb:11.6` and 21 others can carry an identical reference over an image that has moved. **MEASURED, not theorised — spike §5 found `mariadb:11.4` and `mariadb:12.3` had BOTH already moved upstream while two fully-pinned CONTROLS held.** So `romm` and `bookstack` on demo-hp would read „Naprakész" over a database engine build that is not the one the catalog now resolves to. **This is a KNOWN LIMITATION OF A SHIPPED FEATURE, filed the same session rather than left implicit**, and it is stated in the same words in `architecture/09-update-architecture.md` §8.1 and in the controller's `README.md`. The close is a digest comparison against the registry, which needs a network call, a cache and a failure posture — it is not a one-liner and it is not slice 2's job. **Depends on R-440**, whose fix (stop floating) would remove the problem instead of measuring it — take that route first if it is available. `architecture/09-update-architecture.md` **MEASURED 2026-09-21, and the blind spot is not one or two pins.** `audits/UPDATE-ARC-STATE-2026-09-21.md` §3.3: the catalog carries **10 floating pins of 66** (recounted — the old "23" was stale), and **6 of the 7 measurable engine pins have been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps), `postgres:15-alpine`, `redis:7-alpine` (6 apps), `mariadb:11.4`, `mariadb:12.3`, `postgis:16-3.5-alpine`; only `mariadb:11.6` has not. The 8th (immich's own ghcr build) is UNMEASURED — ghcr exposes no anonymous last-modified timestamp. **So on demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.** The fix does NOT need the box to query a registry: the catalog can record each pin's digest at push time (`check-image-resolvable.py` already resolves it) and the box compares digests. Put to the operator as `09` §3b **Q6**, recommended YES — the cheapest real improvement on the arc's list. **— UPDATE NIGHT 2026-09-21:** **MEASURED ON A BOX 2026-09-21 (update night, leg B8), and it REFINES the row in two ways rather than merely confirming it.** §8.1's numbers came from a registry sweep on DooPlex; this is the same question asked of a customer-shaped box, where the badge actually renders. On guest 9202, `docmost`'s two floating pins were read as `installed_images` records them and compared against the upstream digests measured the same night: `postgres:16-alpine` → **`sha256:721873c34ceb9…` on the box and `sha256:721873c34ceb9…` upstream**, and `redis:7-alpine` → **`sha256:858f009f9709c…` both sides**. **Identical. So the badge „Naprakész" is TRUE for this box**, and the app reads correctly. **(1) The defect's size is set by INSTALL AGE, not by the catalog.** A floating pin is wrong only for a box that pulled BEFORE the tag moved; a box deployed after the repush holds the current image and its badge is right. R-446's "six repushed pins" measured the tag against the date the CATALOG set it, which is the right measure for *the catalog* and not for *a box*. **(2) The producer Q6 needs ALREADY EXISTS on the box.** `installed_images` records a real `digest` per service (`installed.go` §7.1) — the box knows exactly what it is running. What it cannot do is COMPARE, because the catalog carries no digest to compare against. That is Q6's proposal, and this is a concrete confirmation that only the catalog half is missing. Evidence: `audits/update-night-2026-09-21/23-B8-floating-pin.txt`. **-- RULED 2026-09-23 (`09` §3 decision 17):** YES — the catalog records the image digest of every pin at push time; the box compares against it and, where the catalog carries one, pulls **that exact image**, which makes a floating tag reproducible, not only the badge honest. *Pull-by-digest while the definition names a tag is a claim to verify in the build, not a ruling on mechanism.* **-- 2026-09-23:** pull-by-digest MEASURED on 9202 — Docker and Compose both pull and run `redis:7-alpine@sha256:858f…` and refuse a digest that does not exist. **Build trap, read from source:** `splitImageRef` returns "unorderable" for any ref containing `@` (`stacks/updateorder.go:134`), so a digest-carrying pin must have its digest split off before ordering or every such app reads Unknown. `09` §6.4 part 6. **— NIGHT 2026-09-23:** the CATALOG half of the close shipped: every ladder entry records the digest the registry served for each `to` ref (`scripts/image_digest.py`), and the move gate refuses a digest the registry no longer serves. The box does not compare it yet (`09` §6.4 part 6, box half). | **READY TO BUILD — owner: CC; `09` §6.4** | -| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 **HALF SHIPPED 2026-09-21 (catalog `5ff36d098cbc`): the second half — an engine change gets its OWN edge — is now ENFORCED** by `check-engine-major.py`, which refuses a commit moving a MariaDB major together with any other image move in that template, naming what it was bundled with. The FIRST half (automatic within a major) is Slice 6 and needs four operator answers — `09` §3b **Q1–Q4**, with the shape it would take in `09` §6.2. **The urgency is now measured:** 46 of the catalog's 58 exact pins are behind upstream and **39 of those are within a major** — the population the 2026-09-02 ruling already says may move without a human. **-- RULED 2026-09-23 (operator, `09` §3 decisions 11–15):** Q1–Q4 answered. The update is a leg of the backup chain after off-site and before the full-system backup (11); automatic with a per-box switch ON by default (12); **the TEST decides, not the tag** — the box applies every step the catalog holds because the catalog holds only tested steps, and `CompareImageRefs` moves to the catalog gate (13, REPLACES "never across a major"); a box behind climbs **one tested step at a time** (14); **the box UNDOES a failed update itself** — old definition + the pre-pin safety dump + health check again, HOLD only if the undo fails (15, REPLACES §6.1's no-auto-undo). The undo and the ladder were SPIKED the same day before any build (`audits/update-rulings-2026-09-23/`); build order and costs in `09` §6.4. **-- SPIKED 2026-09-23 (`audits/update-rulings-2026-09-23/`):** the undo works by hand on three real migrating edges and needs eight product additions (R-637..R-642); **the ladder is measured absent** — one press on a box two steps behind jumped vikunja 2.3.0 → 2.5.0 in 9.5 s and 2.4.0 never ran, and the box cannot see intermediate steps at all because its catalog clone is `--depth 1` (`sync.go:283`/`:300`, one commit visible on both demo guests). The ladder's recommended format is an `update_ladder:` list in `.felhom.yml` with each intermediate step's own definition, NOT the git history (romm's image-moving commit is the definition that OOM-looped). The chain's update leg has ≤15 min as ruled (R-643). Build order `09` §6.4. **— NIGHT 2026-09-23: `09` §6.4 part 4 SHIPPED** (catalog `6db08a5`): the test record `update_ladder:` + two gates + the only writer + the 21-move backfill; 12 more steps published with records. Parts 5 (the box climbs), 6-box-half and 7 remain; the romm press on demo-hp showed today's jump live — 5.3.0 → 5.3.1 AND mariadb 11.4 → 11.8 in one press (both tested steps; `done`). **— 2026-09-24: `09` §6.4 PART 5 SHIPPED** (controller v0.268.0 `206b035`, catalog `5ed599c`): one press = one tested step, each step's own definition at `templates//steps/.yml`, proven live on 9202 (romm 5.3.0/11.4 → 5.3.1/11.4 → 5.3.1/11.8 in two presses, `audits/ladder-2026-09-24/partD/`). Left in this row: part 6's box half, part 7 (the automatic leg), part 10 (PostgreSQL majors). **— 2026-09-25 night: part 6's box half shipped in v0.269.0/v0.269.1; PART 7 SHIPPED as controller v0.271.0** (the automatic leg, proven over six simulated nights on 9202 and watched through the demo boxes' first real night, `audits/DRILL-night-2026-09-25.md`). **Left: part 10 only (PostgreSQL majors, R-463).** | **NARROWED to `09` §6.4 part 10 — owner: CC; returns to the operator for go/no-go** | +| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 **HALF SHIPPED 2026-09-21 (catalog `5ff36d098cbc`): the second half — an engine change gets its OWN edge — is now ENFORCED** by `check-engine-major.py`, which refuses a commit moving a MariaDB major together with any other image move in that template, naming what it was bundled with. The FIRST half (automatic within a major) is Slice 6 and needs four operator answers — `09` §3b **Q1–Q4**, with the shape it would take in `09` §6.2. **The urgency is now measured:** 46 of the catalog's 58 exact pins are behind upstream and **39 of those are within a major** — the population the 2026-09-02 ruling already says may move without a human. **-- RULED 2026-09-23 (operator, `09` §3 decisions 11–15):** Q1–Q4 answered. The update is a leg of the backup chain after off-site and before the full-system backup (11); automatic with a per-box switch ON by default (12); **the TEST decides, not the tag** — the box applies every step the catalog holds because the catalog holds only tested steps, and `CompareImageRefs` moves to the catalog gate (13, REPLACES "never across a major"); a box behind climbs **one tested step at a time** (14); **the box UNDOES a failed update itself** — old definition + the pre-pin safety dump + health check again, HOLD only if the undo fails (15, REPLACES §6.1's no-auto-undo). The undo and the ladder were SPIKED the same day before any build (`audits/update-rulings-2026-09-23/`); build order and costs in `09` §6.4. **-- SPIKED 2026-09-23 (`audits/update-rulings-2026-09-23/`):** the undo works by hand on three real migrating edges and needs eight product additions (R-637..R-642); **the ladder is measured absent** — one press on a box two steps behind jumped vikunja 2.3.0 → 2.5.0 in 9.5 s and 2.4.0 never ran, and the box cannot see intermediate steps at all because its catalog clone is `--depth 1` (`sync.go:283`/`:300`, one commit visible on both demo guests). The ladder's recommended format is an `update_ladder:` list in `.felhom.yml` with each intermediate step's own definition, NOT the git history (romm's image-moving commit is the definition that OOM-looped). The chain's update leg has ≤15 min as ruled (R-643). Build order `09` §6.4. **— NIGHT 2026-09-23: `09` §6.4 part 4 SHIPPED** (catalog `6db08a5`): the test record `update_ladder:` + two gates + the only writer + the 21-move backfill; 12 more steps published with records. Parts 5 (the box climbs), 6-box-half and 7 remain; the romm press on demo-hp showed today's jump live — 5.3.0 → 5.3.1 AND mariadb 11.4 → 11.8 in one press (both tested steps; `done`). **— 2026-09-24: `09` §6.4 PART 5 SHIPPED** (controller v0.268.0 `206b035`, catalog `5ed599c`): one press = one tested step, each step's own definition at `templates//steps/.yml`, proven live on 9202 (romm 5.3.0/11.4 → 5.3.1/11.4 → 5.3.1/11.8 in two presses, `audits/ladder-2026-09-24/partD/`). Left in this row: part 6's box half, part 7 (the automatic leg), part 10 (PostgreSQL majors). **— 2026-09-25 night: part 6's box half shipped in v0.269.0/v0.269.1; PART 7 SHIPPED as controller v0.271.0** (the automatic leg, proven over six simulated nights on 9202 and watched through the demo boxes' first real night, `audits/DRILL-night-2026-09-25.md`). **Left: part 10 only (PostgreSQL majors, R-463).** **-- 2026-09-25 (evening): part 10 SHIPPED for docmost** (controller v0.273.0, catalog `afd3a60`, decisions 35–39; `audits/night-2026-09-26/`). What stays open: each of the other ten PostgreSQL apps needs its own two-venue proof before the gate lets its major move. | **NARROWED — the other ten PostgreSQL apps; owner: CC** | | **R-451** | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **RULED — build deferred until the fleet grows; owner: CC** | | **R-454** | **[P3-LOW] Five `internal/web` test files have been `gofmt`-unclean for an unknown length of time, and nothing notices.** MEASURED 2026-09-02: `gofmt -l controller/internal/web/` reports `backups_split_test.go`, `claim_code_naming_test.go`, `disk_health_test.go`, `r400_debug_routes_test.go`, `recovery_test.go` — at the **baseline** commit `960d29b0612c`, i.e. not introduced by v0.233.0 (both files added that day are clean). **`go vet` does not check formatting and `controller_gates.py` has no formatting gate**, so the only thing that would ever surface this is someone running `gofmt -l` by hand, which is how it was found. **Not reformatted in the same session, deliberately** — the minimal-changes rule, and a five-file whitespace commit inside a feature release makes that release's diff unreadable. **Small, and the cost of NOT having the instrument is the row:** the count can only grow, and every future `gofmt -l` run produces noise that hides a real one. Fix is two lines: a `gofmt -l` gate in `controller_gates.py` plus one formatting commit, in that order (the gate first, so the commit is provably complete). Owner: **CC.** | **READY — rank P3-LOW; owner: CC** | | **R-457** | **[P3-LOW] A test that hardcodes a date AND asserts an age derived from it is green on the day it is written and red the next morning — one instance PROVEN, six candidate files named.** MEASURED 2026-09-03: `TestGroupD_BadgeRendersOnBothSurfaces` (shipped the previous day in v0.233.0) pinned a fixture `catalog_since: "2026-07-18"` and asserted the rendered string `"Frissítés elérhető — 46 napja"`. **The pure badge tests inject a clock; the RENDER test does not and cannot** — it goes through the production templates, which call the funcmap entry `updateBadge`, which reads `time.Now()`. The suite was green on 2026-09-02 and **FAILED on 2026-09-03** with *"the behind badge is missing"* on both surfaces, because the true answer had become 47. **Fixed by DERIVING the fixture** — `catalog_since` is computed as *today minus 46 days*, so the test asserts the real number through the real clock and cannot rot. **THE CLASS, which is why this is a row and not just a fix:** a clock-reading test that also carries a date LITERAL is a bomb with a fuse of unknown length, and the suite being green is not evidence it is defused — it is evidence the fuse has not burned down yet. **NAMED AS UNCHECKED CANDIDATES, NOT ACCUSED** — six other test files contain both a `20xx-xx-xx` literal and `time.Now()`: `internal/backup/offbox_test.go`, `internal/web/handler_export_upload_test.go`, `internal/web/r103_tier2_action_test.go`, `internal/web/dashboard_backup_card_test.go`, `internal/web/async_restore_test.go`, `internal/stacks/installed_test.go`. Mixing the two is not itself a defect — it is one only where a literal feeds an assertion evaluated against the real clock — so each needs reading, which is a sweep and not this session. **The instrument that would end the class:** run the suite once under a faked future date in CI and see what turns red. Owner: **CC.** `felhom-controller` v0.234.0 CHANGELOG | **READY — rank P3-LOW; owner: CC** | | **R-458** | **[P3-LOW] `.felhom.yml` keeps flowing to an app whose compose file is FROZEN, so a frozen app can receive a health check written for a version it is not running.** The v0.235.0 render freezes `docker-compose.yml` for a pinned app once the catalog moves past its version, but copies `.felhom.yml` **verbatim in every case** (`Syncer.copyTemplates`). **The asymmetry is deliberate and both directions were considered:** `.felhom.yml` carries no image, and it carries `catalog_since` — the single input the update badge uses to say *„Frissítés elérhető — N napja"* — so freezing it would silently withhold the one number that tells a customer they are behind, i.e. it would break slice 2 to protect slice 3. **What it costs:** the file also carries the controller-side `healthcheck:` block and resource hints, so a template updated for a newer version can hand a frozen app a probe written for software it is not running. **THE FAILURE DIRECTION IS A FALSE ALARM, NEVER DATA LOSS** — the app keeps running; at worst it renders as degraded and, if it persisted, could reach the dead-app alarm path. That is the same class as R-330's false e-mails, which is why this is a row and not a footnote. **Not fixed now, and the reason is that the cheap fix is wrong:** freezing the whole file breaks the badge, and freezing only the `healthcheck:` key means the syncer would have to parse and re-assemble a customer-facing metadata file — new surface on the one path that touches every app on every box every 15 minutes. **What would settle it:** whether any catalog `healthcheck:` has ever been changed in the same commit as an `image:` line (measurable from the catalog's own history, no box needed). If the answer is "never", the exposure is theoretical and the row can be closed by measurement instead of by code. Owner: **CC.** `architecture/09-update-architecture.md` §5.4, §8.5 **— UPDATE NIGHT 2026-09-21:** **MEASURED 2026-09-21 (update night), leg B9, and the row's risk is NARROWER than it states.** A `.felhom.yml`-only change (a health check for a path only a newer version would serve) was pushed to a FROZEN `bentopdf` — installed `v2.8.6`, catalog ahead. §5.4's asymmetry is confirmed live: the new `.felhom.yml` reached the box while the compose `image:` line stayed `v2.8.6`. **But no false alarm was produced**: ten samples over two minutes all read `state=running` with the front door at `200`. The reason is the probe's own semantics, not luck — `healthprobe.go:258-261` treats **any response** as healthy for `type: http`, and the bogus path answers 404, which is a response. **So this row's false-alarm risk exists only for `type: api` probes carrying an `expect` block**, where the status is compared; for every `type: http` template and every `type: api` without `expect`, a newer version's path is invisible to the probe. The row's actual claim — the failure direction is a false alarm, never data loss — stands and is now measured. Evidence: `audits/update-night-2026-09-21/21-B9-frozen-app-newer-felhomyml.md`. | **READY — rank P3-LOW; owner: CC** | | **R-460** | **[P3-LOW] BookStack's FILE half cannot be seeded or verified without a browser, so its upgrades can only ever be auto-proven for the DATABASE.** MEASURED 2026-09-06 while building the R-449 harness. BookStack's API needs a token that is only mintable through its web UI, and its HTTP login is unusable headlessly for a second, independent reason: `APP_URL` comes from the template as `https://${SUBDOMAIN}.${DOMAIN}`, so the app marks its session and XSRF cookies **`secure`**; curl over plain http stores neither and **every login POST returns 419 Page Expired**, which looks exactly like a wrong password. The container serves no TLS. **The database half IS provable** — the harness seeds with `php artisan bookstack:create-admin` and reads back with a DIFFERENT artisan command that must find the record, carrying its own negative control on every call. **What is unprovable is an uploaded image or attachment**, i.e. exactly the half a customer would notice. **THIS IS A FACT ABOUT THE APP, NOT A DEFECT IN THE HARNESS**, and it is recorded because Slice 6 needs to know which apps can be auto-verified and which can only be partly verified — nobody had that list before. **Deliberately NOT worked around:** planting a file in the volume would make the test pass while proving nothing, which is R-156's exact failure. **What would remove it:** a headless token route (upstream), or accepting a browser-driven step for this app alone, which DooPlex cannot run. Owner: **CC.** `audits/SPIKE-upgrade-test-2026-09-06.md` §6 **-- UPDATE NIGHT 2026-09-21:** bookstack's edge was walked again on 2026-09-21 and is again **half-proven**: the database half read back through `php artisan` with its own negative control, the file half untouched. The limitation is unchanged and is now measured on the box as well as on the harness. Two more apps joined the same class tonight for a different reason (R-624). | **READY — rank P3-LOW; owner: CC** | | **R-462** | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). | **READY — rank P2-MEDIUM; owner: CC (scope already ruled, `09` §3 decision 6)** | -| **R-463** | **[P2-MEDIUM] The day the catalog moves `postgres:16` to `17`, ELEVEN apps are affected and the container image will NOT perform the conversion — and nothing anywhere records that.** MEASURED 2026-09-06: 11 of the 53 templates carry PostgreSQL — **8 on `postgres:16-alpine`**, 1 on `postgres:15-alpine`, plus `postgis/postgis:16-3.5-alpine` and Immich's own `postgres:16-vectorchord…` build. **A grep of the whole register for `pg_upgrade`, "postgres major" or "postgresql major" returns ZERO** (confirmed this session, and confirmed again before filing). **WHY IT IS NOT THE SAME PROBLEM AS R-459, and this is the point of the row: the two engines fail in OPPOSITE directions.** MariaDB starts anyway and skips the conversion quietly, which is why R-459 went unnoticed until a harness looked. **PostgreSQL REFUSES TO START on a datadir from an older major** — the official image performs no `pg_upgrade` and exits with a message naming both versions. So the Postgres case cannot hide; it will present as eight apps down at once, on the sync after the catalog moves. **DELIBERATELY NOT MEASURED HERE, and saying so is the scope discipline:** R-459's task was scoped to MariaDB, and measuring the Postgres analogue is its own piece of work with its own venue. **This row exists so the gap is a record rather than a sentence in an audit nobody greps.** What would settle it: one edge on the existing harness (`postgres:16-alpine` → `17-alpine`) on a scratch host, which would also exercise the `engine_state_after` field's Postgres probe end to end — it is written but has never run against a real Postgres major. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §7 **-- UPDATE NIGHT 2026-09-21:** **Measured 2026-09-21, both halves.** (a) What a household sees TODAY: the guarded Update of `postgres:16-alpine` to `17-alpine` on docmost ended **`failed` in 5.1 s**, the app stopped and held, **the pin naming 17 while `installed_images` still said 16 and nothing ran**, the data intact, and the restore the hold sentence names back in **29.1 s**. The engine's refusal line had to be REPRODUCED independently because `failAndHold` destroyed it (R-621): *FATAL: database files are incompatible with server / DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.11.* The datadir was still `16` afterwards, and the same copy started under 16 holding 48 tables as the positive control. (b) The conversion **COSTED** on a real seeded 49 MB / 48-table datadir: `pg_dumpall` **2.6 s / 132 201 B**, fresh 17 plus replay **6.5 s / 48 tables restored**, the app up on 17 saying *Database connection successful*, **the seeded account read back**, total **155.9 s of which ~9 s is engine work**. `pg_upgrade` was NOT run: it needs both majors' binaries in one image and no such image exists here. Full paragraph: `audits/update-night-2026-09-21/24-Q5-postgres-conversion-costed.md`. **-- RULED 2026-09-23 (`09` §3 decision 16):** PostgreSQL majors are converted BY THE BOX as a guarded-update step — save everything from the old engine, start the new one empty, load it back, check. Each of the eleven apps is proven on the test bench before the catalog may move it; the engine-major gate stays until then. | **READY TO BUILD — owner: CC; `09` §6.4; the gate stays until all eleven are proven** | +| **R-463** | **[P2-MEDIUM] The day the catalog moves `postgres:16` to `17`, ELEVEN apps are affected and the container image will NOT perform the conversion — and nothing anywhere records that.** MEASURED 2026-09-06: 11 of the 53 templates carry PostgreSQL — **8 on `postgres:16-alpine`**, 1 on `postgres:15-alpine`, plus `postgis/postgis:16-3.5-alpine` and Immich's own `postgres:16-vectorchord…` build. **A grep of the whole register for `pg_upgrade`, "postgres major" or "postgresql major" returns ZERO** (confirmed this session, and confirmed again before filing). **WHY IT IS NOT THE SAME PROBLEM AS R-459, and this is the point of the row: the two engines fail in OPPOSITE directions.** MariaDB starts anyway and skips the conversion quietly, which is why R-459 went unnoticed until a harness looked. **PostgreSQL REFUSES TO START on a datadir from an older major** — the official image performs no `pg_upgrade` and exits with a message naming both versions. So the Postgres case cannot hide; it will present as eight apps down at once, on the sync after the catalog moves. **DELIBERATELY NOT MEASURED HERE, and saying so is the scope discipline:** R-459's task was scoped to MariaDB, and measuring the Postgres analogue is its own piece of work with its own venue. **This row exists so the gap is a record rather than a sentence in an audit nobody greps.** What would settle it: one edge on the existing harness (`postgres:16-alpine` → `17-alpine`) on a scratch host, which would also exercise the `engine_state_after` field's Postgres probe end to end — it is written but has never run against a real Postgres major. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §7 **-- UPDATE NIGHT 2026-09-21:** **Measured 2026-09-21, both halves.** (a) What a household sees TODAY: the guarded Update of `postgres:16-alpine` to `17-alpine` on docmost ended **`failed` in 5.1 s**, the app stopped and held, **the pin naming 17 while `installed_images` still said 16 and nothing ran**, the data intact, and the restore the hold sentence names back in **29.1 s**. The engine's refusal line had to be REPRODUCED independently because `failAndHold` destroyed it (R-621): *FATAL: database files are incompatible with server / DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.11.* The datadir was still `16` afterwards, and the same copy started under 16 holding 48 tables as the positive control. (b) The conversion **COSTED** on a real seeded 49 MB / 48-table datadir: `pg_dumpall` **2.6 s / 132 201 B**, fresh 17 plus replay **6.5 s / 48 tables restored**, the app up on 17 saying *Database connection successful*, **the seeded account read back**, total **155.9 s of which ~9 s is engine work**. `pg_upgrade` was NOT run: it needs both majors' binaries in one image and no such image exists here. Full paragraph: `audits/update-night-2026-09-21/24-Q5-postgres-conversion-costed.md`. **-- RULED 2026-09-23 (`09` §3 decision 16):** PostgreSQL majors are converted BY THE BOX as a guarded-update step — save everything from the old engine, start the new one empty, load it back, check. Each of the eleven apps is proven on the test bench before the catalog may move it; the engine-major gate stays until then. **-- 2026-09-25 (evening): docmost CONVERTED by the box** (16 → 18, controller v0.273.0; catalog `afd3a60` moved it with the conversion mark; proven on the bench and on 9202 incl. three undo cases). Measured: `postgres:18` refuses even an EMPTY volume at `/var/lib/postgresql/data` — an 18 step moves the mount. Open: the other ten, one proof each (calcom, adventurelog/postgis, claper, rallly, immich's own image, zipline, outline, paperless-ngx, tandoor, sparkyfitness on 15). | **NARROWED — ten apps left; owner: CC; the gate stays per app** | | **R-464** | **[P3-LOW] MariaDB's entrypoint prints `MariaDB upgrade not required` on an UNSUPPORTED DOWNGRADE, so that line cannot be used as a soundness signal.** MEASURED 2026-09-06. After converting a datadir to `12.3.3-MariaDB` and then starting **11.6** on it, the entrypoint logs, on every start: **`[Note] [Entrypoint]: MariaDB upgrade not required`**. Asked properly, the same engine answers **`FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!`** **The entrypoint compares the datadir's recorded version against its own and concludes there is nothing to DO. That is true, and it is not a statement that the state is sound.** **THIS IS THIS PROJECT'S MOST-REPEATED CLASS, in a new costume** — the same shape as `CLAUDE.md`'s "presence is not success" and as R-443's HTTP 200 over a crash-looping app: a reassuring sentence that answers a narrower question than the one a reader will take it for. **Why it is worth a row rather than a footnote: the obvious cheap instrument for R-459 is to grep container logs for that exact line**, and such an instrument would report "fine" for an unsupported downgrade. **The correct probe is `mariadb-upgrade --check-if-upgrade-is-needed`**, which is what `upgrade-test.py`'s `engine_state_after` now uses. **Also recorded, because it nearly produced a wrong answer here: run without credentials that command returns `ERROR 1045 … FATAL ERROR: Upgrade failed` with exit 1** — an authentication failure wearing the shape of a verdict. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §5.4 | **READY — rank P3-LOW; owner: CC** | | **R-468** | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** | | **R-469** | **[P3-LOW] REMOVE THE ENGINE-MAJOR RULE when Slice 4 (R-448) ships — a tracked act, not a lapse.** Since 2026-09-13 `app-catalog-felhom.eu` `CLAUDE.md` rules that *until the Update button takes a verified backup as its precondition, no template may move a database-engine image across a major version* (four MariaDB, eleven PostgreSQL services), and `scripts/check-engine-major.py` (fourth row of `catalog_gates.py`, run by `.githooks/pre-push` with the push range) refuses one, naming the rule and this expiry. **Why the rule:** every `mariadb:` sidecar now carries `MARIADB_AUTO_UPGRADE=1` (R-459), so a MariaDB major move CONVERTS the customer's datadir on the next Update; PostgreSQL converts nothing and refuses to start (R-463). Either way a customer-data event with no backup in front of it. **Honest limit, not re-filed:** the gate needs a parent commit and CI fetches at `--depth 1` — the R-452 gap — so on a shallow clone the runner skips it out loud and only the hook bites. **When R-448 ships:** delete the CLAUDE.md rule, the gate's row and the gate, in one commit that cites this row; then close this. **2026-09-13 — UNBLOCKED, NOT LIFTED.** R-448 shipped in controller v0.237.0/v0.238.0 (slice 4): an update now refuses without a restorable, proven Tier-2 copy, backs up first when it is stale, takes a safety dump, and holds an app that does not come up — the precondition this rule was waiting for. **The rule stays in force until someone deliberately removes it**, which is a separate act (and is worth weighing against R-475: an app with no Tier-2 copy cannot be updated at all, so the guard does not yet cover every app a major engine move would touch). **HALF-LIFTED 2026-09-21, catalog `5ff36d098cbc`.** Slice 4 shipped 2026-09-13, so the rule's own expiry condition is met — **for MariaDB**: the four `mariadb:` sidecars have both halves they need, a verified backup in front of the Update (any tier since v0.239.0) and `MARIADB_AUTO_UPGRADE=1` whose conversion the harness WATCHED run on E3/E3b with the seeded data read back after. **PostgreSQL and MySQL stay refused** — postgres performs no `pg_upgrade` and REFUSES to start on an older major's datadir across eleven templates (R-463); a backup is a route BACK, not a conversion. The refusal text now cites R-463 instead of the shipped R-448. **R-450's second half is enforced in its place:** a MariaDB major must be the ONLY image move in its template in that commit (the bookstack `0b73e5e` shape — two migrations behind one edge). The gate now PRINTS what it allowed, by name — a lifted rule that goes quiet is a lifted rule nobody can audit. Two new decoy cases; two red-proofs, each seen to fail; 40 cases green. **What remains of this row:** the PostgreSQL half, which is R-463's to clear — see `09` §3b **Q5**. **-- NARROWED 2026-09-25 (evening), catalog `6a4a5f0`, `09` §3 decision 35:** the PostgreSQL half now passes ONE app at a time — only a template whose ladder entry for the step is proven on BOTH venues and carries `engine_conversion` (the box converts it, controller v0.273.0), as the only image move in its commit. Every other PostgreSQL app stays refused; the postgis family is judged now (it was not). CLAUDE.md rule text updated the same commit. Decoys + red-proof: `audits/night-2026-09-26/C/`. | **PARTLY CLOSED 2026-09-21 — MariaDB lifted; the PostgreSQL half stands until R-463** | @@ -803,15 +803,16 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-652** | **[P3-LOW] The memory watch counted the kernel's file cache as the app's memory.** MEASURED 2026-09-23 night on the bench: nextcloud 34.0.4 read **100 %** of its 1 GiB and immich's PostgreSQL **100 %**, each with **0** kernel `oom_kill`s — `memory.peak` includes page cache, which the kernel drops before it kills anything. Under the watch as built (R-635 follow-up) both would be marked `memory_tight`, and the gate would demand a raised `mem_limit` — a customer-box capacity figure — for cache. **Done the same night (09 §3 decision 22, CC — operator may reverse):** the watch samples the app's own memory (`anon` of `memory.stat`) every 15 s; the mark and the ladder's `memory_peak_pct` read it where measured; the cgroup peak stays beside it (`memory_cgroup_peak_pct`). **Open:** romm's backfilled entry carries M1's 80.9 % cgroup peak (measured before the anon sample existed) — re-measure it on its next step; and decide whether an app whose anon is low but whose cgroup stays pinned at its limit (cache thrash) should be marked at all. Evidence: `audits/night-2026-09-23/apps/nextcloud/bench-1024M/`, `apps/immich/bench-noanon/`. | **READY — P3; owner: CC (catalog harness)** | | **R-654** | **[P3-LOW] opengist 1.15 moved every page under `/-/` — a household's `/login` bookmark answers 404 after the update.** MEASURED 2026-09-23 night: 1.13 serves `/login`, `/register`, `/all`; 1.15.2 answers **404** on all three and serves `/-/login`, `/-/register`, `/-/all`; `/` redirects to `/-/all`. The app, its data and its probe (`/healthcheck`) are fine, and a household arriving at the root lands correctly — only a deep link breaks. 1.15 also marks its session cookie `Secure`. **Needs:** a line in opengist's `app_info` if the operator wants households told; nothing in the product. Evidence: `apps/opengist-oldfixture/`, `apps/opengist/`. | **READY — P3; owner: operator (copy decision) / CC (writes it)** | | **R-655** | **[P2-MEDIUM] adventurelog v0.13.0 cannot become healthy in the catalog's template — its update is undone on every box.** MEASURED 2026-09-23 night, both venues. v0.13.0's FRONTEND image adds its own `HEALTHCHECK` (`node -e fetch('http://127.0.0.1:3000/health')`), and `/health` answers **503 `{"ok":false,"backend":"unreachable"}`** unless the backend's `/health/` answers OK to the frontend's own request (read from the image's `django-proxy` chunk: `fetch(${getServerEndpoint()}/health/)`). On the bench the backend served `/api/` 200, the seed read back, the migration ran (6 lines) — and the frontend stayed `unhealthy` for 420 s; on 9202 the guarded Update went `verifying` → **`undoing` → `undone`** (under the drill's 90 s timeout). **MEASURED LATER THE SAME NIGHT — two causes, and the first hypothesis was wrong.** (1) The backend's `/health/` answers `200 {"ok": true}` to the frontend (read on a fresh v0.13.0 install); the unhealthy frontend is **our template's own healthcheck override**, `["CMD", "/nodejs/bin/node", …]` — v0.12.1's distroless image keeps node there, v0.13.0 moved it to `/usr/bin/node`, and Docker's health log reads `exec: "/nodejs/bin/node": no such file or directory` 19 times in a row. (2) With the override removed, a second bench run hit a harder wall: v0.13.0's backend runs `download-countries` at EVERY start, fetching world data from the internet, and a cut-short download (`ijson.common.IncompleteJSONError: Incomplete JSON content`) crash-loops the entrypoint — the backend never became healthy in 15 min. The first bench run's download had succeeded. **So v0.13.0's boot depends on an outside download, and whether an update succeeds depends on it too.** **The catalog did NOT move adventurelog.** **Needs:** the override dropped (or pointed at `/usr/bin/node`) in the SAME commit as the image move; and a measured answer on whether `download-countries` can be skipped or pre-seeded (an env switch, or the data in the volume) before the edge is re-proven on both venues. Evidence: `audits/night-2026-09-23/apps/adventurelog/`. | **READY — P2; owner: CC (catalog)** | -| **R-657** | **[P2-MEDIUM] "Remove the app, keep my data", then install it again: nextcloud never installs, and the box only says „unhealthy".** MEASURED 2026-09-23 night on 9202 (v0.267.0): nextcloud was removed through the product keeping its drive folder (the remove with data was refused — R-442's fail-closed guard, as on every drill on 9202 — and the product's keep-data remove taken). An hour later a fresh install of nextcloud on the same box: the template binds `${HDD_PATH}/appdata/nextcloud` to `/var/www/html/data`, the kept folder still holds `admin/`, `appdata_*`, `.ncdata` and a 145 MB `nextcloud.log`, and the image's installer loops **„Login is invalid because files already exist for this user — Retrying install..."**; `occ status` reads `installed: false`. The controller records the deploy as done and the app as `unhealthy`; nothing tells the household that their kept files are what blocks the new install, or what to do. **Why it matters:** keep-data is the choice the product OFFERS a household at remove time — and for nextcloud the kept data makes the app uninstallable. **Needs:** decide the product's promise for a reinstall over kept data, per app class (adopt the data? refuse with a sentence? offer to move it aside?); at minimum a deploy-time refusal or warning when the app's drive folder is not empty. Evidence: `audits/night-2026-09-23/chaos/00-nextcloud-reinstall-over-kept-data.txt`. | **READY — P2; owner: operator (the promise) / CC (the build)** | | **R-675** | **[P3-LOW] The unit-only restore's refusal for a file app still points to „Fájlok visszaállítása" instead of the second drive's whole restore.** `missingFileLegsRefusal` predates decision 26 (v0.269.0); when a whole copy exists on the second drive the sentence should name it. | **READY — P3; owner: CC (controller)** | | **R-676** | **[P3-LOW] Watch: immich's first start restarted 12 times — decision 28's crash-loop stop (6 in 10 min) would stop it.** From the 2026-09-17 chaos night (DB connection dropped during the first-start geocoding import on a 6 GB guest; it did not recover that night). No healthy app in any drill evidence restarts on a first start (1831 samples, 40 live containers), so the threshold stands; this row exists so the first immich install under v0.269.x is watched. `audits/night-2026-09-24/A3/40-first-start-restarts.txt` **2026-09-25 night (read from source, v0.271.0): a DEPLOY's first start is NOT covered by decision 28's suppression** — `Deploying` clears when `compose up -d` returns (`deploy.go` "Clear deploying flag"), and `ObserveUnhealthy` then samples the app; an automatic update's step, verify and undo ARE covered (`Updating`, pinned by `TestD28_NoCrashLoopStopDuringAnAutomaticStep`). So a first start that restarts ≥ 6 times in 10 min is stopped — which R-676 already accepts for a broken first start; a healthy slow first start would be stopped too. | **OPEN — P3; owner: CC (watch)** | | **R-682** | **[P3-LOW] A Remove interrupted by a controller kill leaves the app half-removed: containers gone, the app still listed as installed (and held).** MEASURED 2026-09-24 on 9202 (chaos round 9): the kill 2 s after the Remove press answered the household `502 Bad Gateway`; after the restart `chaoscrash` read deployed, stopped, `unhealthy_stop`, with NO container left. Pressing Remove again completed it cleanly (200, only the catalog template left). Recoverable by the household's own second press; nothing tells them to press it. **Fix direction:** the remove journals its intent and finishes (or says it was interrupted) at boot, as the update does. `audits/night-2026-09-24/E/round-09*.json`, `E/round-09b-remove-again.txt` | **READY — P3; owner: CC (controller)** | | **R-683** | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | -| **R-687** | **[P3-LOW] Part 7's live proof has four gaps a scratch box cannot close, and one observability gap.** (1) **W+5h reached with steps left** is proven by unit test only (`TestLeg_NoStepAtOrAfterW5h`) — the leg starts at W+105m and would need a 3-hour leg live; (2) **the off-site leg FAILING** before the update leg: 9202 has no off-site tier, so only the "no target" path ran live — failure and panic are `TestChainUpdateLeg_EveryPath`; (3) **a `files_may_change` step WITHOUT a whole copy**: both drill apps given the mark (wishlist, navidrome, romm) turned out whole on 9202 by the backup side's truth table (why, per app, is not logged — see the observability gap), so only "with a copy" ran live; (4) **the full-system gate waiting** cannot run on 9202 (no agent), and **did not occur on the demo boxes' real night either** (2026-09-25: both legs ended by 04:19, before the gate opened at 04:30, and no whole-box backup was due on either box) — unit + red-proof only (`TestD20_GateWaitsForTheLeg`). **Also cosmetic:** a leg with no steps reports `"steps": null` to the hub, not `[]`. **Observability:** when the leg TAKES a `files_may_change` step it does not log which whole copy allowed it (only the skip says why). `audits/night-2026-09-25/C/` | **OPEN — P3; owner: CC** | -| **R-688** | **[P3-LOW] The customer delete says it removes the tunnel and zone, but no leg of it calls Cloudflare.** The dialog's acknowledgement reads "the customer will be RESET — offsite repo DESTROYED, PBS revoked, tunnel/zone removed" (`hub/internal/web/customer_delete.go` `deleteCascadeAcks`), while `commitCustomerReset` has legs for Hetzner, PBS, claim, descriptor and DB only. Seen 2026-09-25 retiring `peti-felhom`, whose config carried a Cloudflare tunnel token and API token (`sajatfelhom.hu`): the tokens went with the record; any tunnel or DNS record on Cloudflare's side was neither listed nor removed. **Fix direction:** either a Cloudflare leg (tunnel + DNS by the customer's ids), or the dialog stops promising it and lists what to remove by hand. `audits/RETIRE-peti-2026-09-25.md` | **READY — P3; owner: CC (hub) / operator (Peti's Cloudflare leftovers, if any)** | +| **R-687** | **[P3-LOW] Part 7's live proof has four gaps a scratch box cannot close, and one observability gap.** (1) **W+5h reached with steps left** is proven by unit test only (`TestLeg_NoStepAtOrAfterW5h`) — the leg starts at W+105m and would need a 3-hour leg live; (2) **the off-site leg FAILING** before the update leg: 9202 has no off-site tier, so only the "no target" path ran live — failure and panic are `TestChainUpdateLeg_EveryPath`; (3) **a `files_may_change` step WITHOUT a whole copy**: both drill apps given the mark (wishlist, navidrome, romm) turned out whole on 9202 by the backup side's truth table (why, per app, is not logged — see the observability gap), so only "with a copy" ran live; (4) **the full-system gate waiting** cannot run on 9202 (no agent), and **did not occur on the demo boxes' real night either** (2026-09-25: both legs ended by 04:19, before the gate opened at 04:30, and no whole-box backup was due on either box) — unit + red-proof only (`TestD20_GateWaitsForTheLeg`). **Also cosmetic:** a leg with no steps reports `"steps": null` to the hub, not `[]`. **Observability:** when the leg TAKES a `files_may_change` step it does not log which whole copy allowed it (only the skip says why). `audits/night-2026-09-25/C/` **-- NARROWED 2026-09-25 (controller v0.273.0):** the cosmetic `"steps": null` → `[]` and the taken `files_may_change` step's missing log line are FIXED (red-proofed, `audits/night-2026-09-26/F/`). Items (1)–(4) stay; (4) did not occur on 2026-09-25 either (demo-felhom's whole-box backup ran at 07:29, three hours after its leg; demo-hp had none due). | **OPEN — P3, gaps (1)–(4) only; owner: CC** | +| **R-688** | **[P3-LOW] The customer delete says it removes the tunnel and zone, but no leg of it calls Cloudflare.** The dialog's acknowledgement reads "the customer will be RESET — offsite repo DESTROYED, PBS revoked, tunnel/zone removed" (`hub/internal/web/customer_delete.go` `deleteCascadeAcks`), while `commitCustomerReset` has legs for Hetzner, PBS, claim, descriptor and DB only. Seen 2026-09-25 retiring `peti-felhom`, whose config carried a Cloudflare tunnel token and API token (`sajatfelhom.hu`): the tokens went with the record; any tunnel or DNS record on Cloudflare's side was neither listed nor removed. **Fix direction:** either a Cloudflare leg (tunnel + DNS by the customer's ids), or the dialog stops promising it and lists what to remove by hand. `audits/RETIRE-peti-2026-09-25.md` **-- HALF DONE 2026-09-25 (hub v0.125.0):** the dialog no longer promises a Cloudflare removal; the preview lists what the operator removes by hand, by domain (the tunnel, the DNS records), never the token — proven live on the hub (`audits/night-2026-09-26/F/`). The Cloudflare leg itself is NOT built. | **NARROWED — the Cloudflare leg only; owner: operator (decide if it is wanted) / CC (build)** | | **R-689** | **[P3-LOW] demo-hp's scheduled restore test picks the golden template in `local:backup/` as a backup and fails every 6 h.** Read 2026-09-25 (agent 0.134.0): `restore-test tier is DUE … archive=local:backup/felhom-golden-0.236.0.tar.zst … newest settled archive (landed 2026-09-13T20:17:01Z)` then `scheduled restore-test FAILED … extractconfig … HTTP 403: permission denied at /vms/ (missing privilege VM.Backup)` — 2026-09-24 10:36, 2026-09-25 04:57 and 10:57. The golden is not a backup of any guest; the real archive (`vzdump-lxc-9201-2026_09_24-21_59_25`) was younger than the 24 h settle. So the restore test proves nothing on demo-hp and logs an ERROR each cycle. **Fix direction:** the restore test selects only `vzdump-*` archives of a known guest (or the golden moves out of `local:backup/`). Agent-side; not built (the agent is untouched this session). `audits/night-2026-09-26/part0/P0-demo-hp-restoretest-golden.txt` | **READY — P3; owner: CC (agent)** | -| **R-690** | **[P1-HIGH] The removed-app restore (R-487) never finds a unit kept on a DATA drive: nextcloud came back with no env, no database and its files mounted on the guest's root disk.** MEASURED 2026-09-25 on 9202 (controller 0.272.0): nextcloud removed with „keep my data" and backups kept; the `/backups/restore` picker listed it but offered 0 copies; the restore logged `No readable recovery unit for nextcloud at /mnt/sys_drive/... — falling back to volume-only restore` while the unit sat readable on the data drive; `compose up` without env (`HDD_PATH` blank → bind `/appdata/nextcloud` on the root disk), `nextcloud-db` crash-looping, `volumes 0/0, dbs 0/0`. **Cause:** `backup.primaryUnitDirFor` and `ListRestorePoints` ask `GetStackComposePath` whether the app is removed; production answers true for EVERY catalog app (the stack exists), while the R-487 test's fake answers it for deployed apps only — the test passes, the box fails. **Fix:** ask `isStackDeployed` (the list's own predicate), pinned by a test on a provider that answers like production. `audits/night-2026-09-26/E/E1-README.md` Q1 | **FIX BUILT — controller (Part E build, 2026-09-25); live proof pending the release** — owner CC | +| **R-691** | **[P3-LOW] Kept data (09 §3 decision 36): two gaps of the first build.** (1) **The read-only file-browser view cannot open a folder another user owns with mode 0770** — nextcloud's `appdata/nextcloud` is `www-data` `drwxrwx---` (measured on 9202 2026-09-25), FileBrowser runs as uid 1000, so „Megőrzött adatok" shows the folder and not its files; the files are still listed, sized, loadable and deletable. Fix direction needs a decision (a read-only ACL, or a helper that lists as root) — not a chmod of the household's data. (2) **„Use my kept data" / Load looks only at the own unit (Tier 1) and the second-drive mirror (Tier 2)**; an app whose only database copy is off-site gets "no backup". Controller `43e99d1`. `audits/night-2026-09-26/E/` **-- 2026-09-25 live proof:** (1) confirmed on 9202 — the view mounts nextcloud's kept folders `:ro` but its files are `www-data` 0770. Also seen: the source's name „Megőrzött adatok" is Hungarian on an English box (the file browser's config holds one name). | **OPEN — P3; owner: CC (needs a small decision for (1))** | +| **R-693** | **[P3-LOW] The memory watch marks a Node app `memory_tight` at any limit — its heap sizes itself from the limit.** Measured 2026-09-25 on the bench (docmost 0.96.0, harness v4): the app's own memory (`anon`) peaked at **349 MB of 384 MB (90.9 %)**, then, with the limit raised to 512 MB, at **431 MB of 512 MB (80.4 %)** — 0 OOM kills and 0 restarts in both 10-minute watches (~12 000 requests each). So the mark (decision 22's "does not fit the memory") fires for an app that fits, and the gate's remedy (raise the limit) cannot clear it. docmost moved with the limit raised to 512 MB (decision 39). **Needs:** a basis that tells growth-to-fill from pressure (e.g. kills/restarts plus a GC-pressure signal, or a second watch at a higher limit showing the peak scales), or a per-app `memory_scales_with_limit` fact. `audits/night-2026-09-26/C/bench-run1/`, `…/bench-run2/` | **OPEN — P3; owner: CC** | +| **R-694** | **[P3-LOW] Loading kept data (and every unit restore) regenerates a withheld login secret — does the household's shown password still work?** Seen 2026-09-25 on 9202 (E5, nextcloud): `generated replacement for [NEXTCLOUD_ADMIN_PASSWORD] — the credential was reset (old value unrecoverable)`. The unit deliberately carries no internet-reachable admin login (D5). For nextcloud the real admin password lives in the loaded DATABASE, so the new env value is likely inert — but if the app page shows the regenerated value as "your password", it is a false one. **Not measured:** what the page shows after a load, per app. `audits/night-2026-09-26/E/E5-5-use.txt` | **OPEN — P3; owner: CC (measure first)** |