DRILL R-356b: the off-site restore for a driveless app that HAS a database
gates / gates (push) Successful in 16s
gates / gates (push) Successful in 16s
A drill, not an implementation. No code, no version bump, no CHANGELOG entry. Ten of the forty driveless apps carry a database; I re-measured that count and got 10. For those ten the restore is a five-leg operation that never ran at all until this week, because R-356 refused before any of it started. Walked end to end on demo-hp for both engines - docmost (Postgres 16) and bookstack (MariaDB 12.3) - each deployed for the drill, planted through the app's own interface, destroyed for real, restored through the endpoint the UI posts to. Q1 does it complete: YES. All five legs ran and succeeded, 32s / 25s. Accented names byte-identical both directions. Q2 which leg won: the SQL DUMP. Three-way discriminator returned the altered dump's value. This confirms R-164's F17 ordering on the OFF-SITE path; R-164 only ever cited the local one. Scratch-only mutation; store proved unmutated. Q3 does a failure tell the truth: partly, and two defects. Filed R-379 (HIGH, the undo copy is valid, named, and unappliable by any product action - proven by applying it by hand on both engines), R-380 (HIGH, a failed MariaDB replay leaves a partial database behind an app reporting healthy, where Postgres crash-loops visibly), R-381 (MEDIUM, the failure message pastes engine stderr including customer table rows into the Hungarian surface), R-382 (LOW, the summary log omits the volume count it already has). H1, H2 and H4 did NOT fire and that is recorded. H3 fired in a shape nobody predicted: not a quiet success, but a loud error over a silent inconsistency. R-361 reproduced independently on a second app. restic check: no errors, 29 snapshots. A flaw in the drill's own planting - a double-escaped accented title - was caught by reading stored bytes as hex, recorded, and re-measured in Phase 1b. Register 325236 -> 330683 bytes. Nothing dropped. Teardown: two apps retained with reason, no pvesm before-snapshot taken (said plainly), no hub-side record created.
This commit is contained in:
@@ -0,0 +1,190 @@
|
||||
# DRILL — R-356b: the off-site restore for a driveless app that HAS a database (2026-08-22)
|
||||
|
||||
**This was a drill, not an implementation. No production code was written, no version bumped, no
|
||||
CHANGELOG entry made.** The output is this document and four register rows.
|
||||
|
||||
**Subject:** `demo-hp` (Tier 0, disposable), guest 9201, controller **v0.219.0**, floor **0.219.0**.
|
||||
**Method:** endpoint-level. `claude-in-chrome` is not available on DooPlex, so every product action was
|
||||
the exact HTTP endpoint the UI's own form posts to — `/api/stacks/<app>/deploy`,
|
||||
`/backup/offbox/{toggle,run,restore,reconstitute}` — driven with the session cookie and the session
|
||||
CSRF token scraped from the page. App content was planted through each app's **own** interface
|
||||
(docmost's JSON API; BookStack's login form + `POST /books`).
|
||||
|
||||
## 1. Baselines, confirmed at drill start
|
||||
|
||||
| repo | HEAD | note |
|
||||
|---|---|---|
|
||||
| felhom-controller | `0f3cf0dbb2c0` | v0.219.0 |
|
||||
| felhom-agent | `40d857b52711` | v0.130.0 |
|
||||
| felhom.eu | `8c9f1b798b12` | ahead of the runbook's hash — this session's own golden-bake commit |
|
||||
| app-catalog | `459766cb1639` | |
|
||||
|
||||
Read from the **hub** (`/configuration`, Basic auth, ClusterIP): `golden_version` **0.219.0**,
|
||||
`agent_version` **0.130.0**, `min_agent` **0.129.0**, global controller floor **0.219.0**. All four as
|
||||
expected — the operator's three-field vouch and the floor raise both landed.
|
||||
|
||||
**The 10-app count, measured in this session and not taken from the runbook: 10.**
|
||||
Method: `needs_hdd: false` in `.felhom.yml` **and** a `postgres|mariadb|mysql` image in the compose.
|
||||
Same ten names as the runbook: bookstack, calcom, claper, docmost, kimai, outline, rallly,
|
||||
sparkyfitness, tandoor, zipline. Evidence `01`.
|
||||
|
||||
**Architecture document read:** `documentation/architecture/07-backup-architecture.md` §6.1, §6.2 and
|
||||
§6.3 **as corrected on 2026-08-22** — including the corrected Tier-3 row, the note that R-102 is *not*
|
||||
closed, and the `[DESIGN]` paragraph on destination resolution.
|
||||
|
||||
Register ceiling before this drill: **R-378**.
|
||||
|
||||
## 2. The three questions
|
||||
|
||||
### Q1 — Does it complete at all? **YES.**
|
||||
|
||||
A driveless app with a database restores end to end, comes back healthy, and returns the planted data.
|
||||
|
||||
Measured on `docmost` (Postgres 16). All five legs that had **never run in this combination** executed
|
||||
in order and every one succeeded, in **32 seconds**:
|
||||
|
||||
| leg | evidence |
|
||||
|---|---|
|
||||
| undo copy of the live DB, taken while up | `pre-restore-20260822T135827Z-docmost-postgres.sql`, 135 624 B |
|
||||
| database-service identification | `[docmost-postgres]` |
|
||||
| stop → place files → **replay volumes** | "Restored **3** Docker volume(s)" — incl. the 52 MB Postgres data directory |
|
||||
| start **only** the DB service | `docker compose up -d docmost-postgres`, 0.4 s |
|
||||
| replay the SQL dump on top | "Imported DB dump docmost-postgres.sql", 2 s |
|
||||
|
||||
3 pages planted through docmost's own API → destroyed (`count(*) FROM pages` = **0**, no soft-deleted
|
||||
rows either) → **3 pages back**, same ids. The discriminator went `ORIGINAL-VALUE-A` →
|
||||
`DESTROYED-VALUE-C` → **`ORIGINAL-VALUE-A`**.
|
||||
|
||||
Repeated on `bookstack` (MariaDB 12.3): all five legs, **25 seconds**, 2 volumes replayed including a
|
||||
**161 MB** MariaDB data directory, the planted book back with its accented name byte-identical.
|
||||
|
||||
**Accented round-trip, byte-for-byte:**
|
||||
`c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662`
|
||||
("Árvíztűrő tükörfúrógép 2 — R356b", docmost) and
|
||||
`C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662`
|
||||
("Árvíztűrő könyvespolc — R356b", bookstack). Both identical before and after. Non-ASCII never crossed
|
||||
a shell chain: strings were embedded in python files inside the guest, or passed with
|
||||
`--data-urlencode "name@file"`.
|
||||
|
||||
> **A flaw in this drill's own method, recorded rather than hidden.** The FIRST accented title was
|
||||
> planted through a shell argument chain that double-escaped it, so it was stored as the literal ASCII
|
||||
> text `Árv…` rather than as accented bytes. It was caught by reading the stored value back **as
|
||||
> hex** — the same discipline that makes the rest of these numbers worth anything. The pages
|
||||
> round-tripped exactly either way, so Q1's answer stands, but the accented coverage did not, and was
|
||||
> re-measured in Phase 1b. The faulty page is deliberately left in place as the record.
|
||||
|
||||
### Q2 — Which leg returned the data? **The SQL dump.**
|
||||
|
||||
A three-way discriminator was built so the answer could not be ambiguous:
|
||||
|
||||
| holder | value |
|
||||
|---|---|
|
||||
| live database before the restore | `LIVE-VALUE-C3` — if it survived, neither leg wrote |
|
||||
| the volume tar (52 MB Postgres data dir) | `ORIGINAL-VALUE-A` |
|
||||
| the SQL dump, **altered in the scratch only** | `ALTERED-VALUE-B` |
|
||||
|
||||
**Result: `ALTERED-VALUE-B`.** The dump won. The ordering the code comment asserts —
|
||||
*"volumes FIRST, database after, so a logical .sql dump still wins over whatever copy of the same
|
||||
database a volume tar happens to contain"* (`offbox_reconstitute.go`, the R-354 block) — **holds in
|
||||
practice on the off-site path.** It was an intention; it is now an observation.
|
||||
|
||||
Mutation asserted: dump sha256 `c5414f24…` → `834c32b1…`, zero occurrences of the original left, one of
|
||||
the altered. Evidence `13`.
|
||||
|
||||
**The fence held.** Only the prepared scratch was mutated. Proved afterwards by re-preparing a fresh
|
||||
scratch from the same snapshot: sha256 back to `c5414f24…`, reading `ORIGINAL-VALUE-A`. Evidence `14`.
|
||||
|
||||
> **This confirms an existing claim on a new path rather than establishing a new one.** **R-164**
|
||||
> already records *"the dump is authoritative and replayed after the tar so it WINS (F17)"* — but cites
|
||||
> `internal/backup/restore_unit.go:262-266`, the **local** restore. Q2 extends that to
|
||||
> `offbox_reconstitute.go`, the **off-site** path, which had never been exercised for this app class.
|
||||
|
||||
### Q3 — Does a failure in the database leg tell the truth? **Partly. Two defects.**
|
||||
|
||||
Measured by truncating the dump in the scratch mid-statement, on both engines.
|
||||
|
||||
| | Postgres (docmost) | MariaDB (bookstack) |
|
||||
|---|---|---|
|
||||
| customer sees a failure | **yes** — `alert alert-error`, "sikertelen" | **yes** |
|
||||
| message names the undo copy | **yes**, by filename | **yes** |
|
||||
| app afterwards | **crash-loops** — visibly broken | **`health=healthy, running=true, restarts=0`** |
|
||||
| database afterwards | **emptied** — 43 tables, 0 rows in pages/users/spaces | **partially applied** — book and users intact, `migrations` **wiped to 0 rows** |
|
||||
| undo copy valid? | **yes** — proven by applying it | **yes** — proven by applying it |
|
||||
| product can apply the undo copy | **NO** | **NO** |
|
||||
|
||||
## 3. Hypotheses — what fired and what did not
|
||||
|
||||
- **H1 (schema collision).** **DID NOT FIRE.** The dump replayed cleanly onto a freshly-replaced
|
||||
Postgres data directory in 2 s under `ON_ERROR_STOP=1`. This is the notable negative: the two legs do
|
||||
not collide in practice for this app class.
|
||||
- **H2 (replayed data directory will not start).** **DID NOT FIRE.** Both engines started healthy on a
|
||||
data directory that had just been replaced wholesale from a tar.
|
||||
- **H3 (MariaDB and Postgres do not fail the same way).** **FIRED — but not in the predicted shape.**
|
||||
The prediction was a quiet partial success with no error. What actually happens is a **visible
|
||||
error** *and* a **silently inconsistent database behind a healthy-looking app**. See R-380.
|
||||
- **H4 (credentials).** **DID NOT FIRE.** `ImportDump` discovered `dbUser=docmost/dbName=docmost` from
|
||||
the **live** container and they matched the restored directory, exactly as predicted — secrets are
|
||||
not regenerated on reconstitution.
|
||||
|
||||
## 4. Findings filed
|
||||
|
||||
| id | severity | what |
|
||||
|---|---|---|
|
||||
| **R-379** | HIGH | The undo copy is valid, is named to the customer, and **no product action can apply it** |
|
||||
| **R-380** | HIGH | A failed **MariaDB** replay leaves a partially-applied database behind a **healthy** app |
|
||||
| **R-381** | MEDIUM | The failure message pastes raw engine stderr — **including customer database rows** — into the Hungarian customer surface |
|
||||
| **R-382** | LOW | The reconstitution's summary log line omits the volume count it already has |
|
||||
|
||||
**An independent reproduction of an existing row.** After the first reconstitution, docmost's
|
||||
`db-dumps/` held **only** `pre-restore-*` files — the app's own `docmost-postgres.sql` was gone. That is
|
||||
**R-361** reproducing on a second app, unprompted. Not re-filed; noted here as corroboration.
|
||||
|
||||
## 5. Observations — noticed, recorded, NOT acted on
|
||||
|
||||
- **A newly deployed app is not in the off-site set.** `docmost` and `bookstack` were both absent from
|
||||
`app_backup` entirely until switched on by hand. This is plausibly deliberate (the switch is the
|
||||
customer's), and it is **not filed as a defect** — but the consequence is that an app can be deployed,
|
||||
run, and never leave the box, while the nightly run reports "backup OK". Stated so the next reader can
|
||||
decide whether the default is right.
|
||||
- **`restic check` was run by hand and passed** — `no errors were found`, 29 snapshots. Nothing in the
|
||||
product runs it (**R-359**, already filed). Reproducing the invocation took three attempts because the
|
||||
key file is `ssh_key`, not `id_offbox` as the arg-builder's variable naming suggests.
|
||||
- **The `-db` container-suffix attribution is correct here.** `bookstack-db`'s dump landed in
|
||||
`bookstack`'s own unit as `bookstack-mariadb.sql` — the R-355 failure shape does **not** reproduce,
|
||||
because v0.218.0's compose-project attribution handles it.
|
||||
|
||||
## 6. Phases run
|
||||
|
||||
| phase | status |
|
||||
|---|---|
|
||||
| 1 — Postgres, does it complete | **run** |
|
||||
| 1b — accented round-trip, re-measured after a method flaw | **run** |
|
||||
| 2 — which leg won | **run** |
|
||||
| 3 — failure path, Postgres | **run** |
|
||||
| 4 — MariaDB: Phase 1 and Phase 3 equivalents | **run** |
|
||||
|
||||
Nothing was dropped.
|
||||
|
||||
## 7. Evidence
|
||||
|
||||
`evidence/`, files `00`–`31`, copied off `demo-hp` at the end of each phase. Nothing was reverted or
|
||||
torn down mid-run, so no intermediate teardown could have taken any of it.
|
||||
|
||||
## 8. Teardown — three layers
|
||||
|
||||
1. **Apps.** `docmost` and `bookstack` were **deployed by this drill** and are **RETAINED**, with their
|
||||
planted data. Reason: the planted data *is* the evidence for every claim above, and they are the only
|
||||
deployed members of the 10-app class on the box, so the next drill in this area starts from a real
|
||||
subject instead of rebuilding one. Both are healthy and sane at the end (docmost restored via its
|
||||
undo copy; bookstack's `migrations` table restored 0 → 102 rows the same way).
|
||||
2. **Storage.** No `pvesm` "before" snapshot was taken — **stated plainly rather than reconstructed.**
|
||||
Measured directly instead: the two apps' volumes total ~233 MB
|
||||
(docmost 67 MB + 228 KB + 4 KB, bookstack 166 MB + 224 KB) inside guest 9201, which sits at
|
||||
16 % of 69 GB. Host `local-lvm` 34.81 %, `local` 14.27 %.
|
||||
3. **Hub-side record.** **None was created.** This drill created no customer and no appliance record;
|
||||
it deployed two apps inside an existing guest of an existing customer. Nothing to dispose of.
|
||||
|
||||
## 9. Store integrity at the end
|
||||
|
||||
`restic check` against the live repository: **`no errors were found`**, 29 snapshots, exclusive lock
|
||||
taken and released. Evidence `30`.
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
Baselines re-verified at drill start, 2026-08-22.
|
||||
|
||||
repo HEAD clean in-sync-with-origin/main
|
||||
felhom-controller 0f3cf0dbb2c0 yes yes (v0.219.0)
|
||||
felhom-agent 40d857b52711 yes yes (v0.130.0)
|
||||
felhom.eu 8c9f1b798b12 yes yes (ahead of the runbook's f2edf7e54595:
|
||||
this session's own golden-bake commit)
|
||||
app-catalog-felhom.eu 459766cb1639 yes yes
|
||||
|
||||
Read from the HUB (operator UI, Basic auth, ClusterIP 10.43.52.34:8080/configuration):
|
||||
golden_version 0.219.0 <- vouched
|
||||
agent_version 0.130.0
|
||||
min_agent 0.129.0
|
||||
global controller floor 0.219.0 <- raised
|
||||
|
||||
Register ceiling before this drill: R-378 (highest id across OPEN-ITEMS.md + CLOSED-ITEMS.md).
|
||||
Next free id: R-379.
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
bookstack | image: mariadb:12.3
|
||||
calcom | image: postgres:16-alpine
|
||||
claper | image: postgres:16-alpine
|
||||
docmost | image: postgres:16-alpine
|
||||
kimai | image: mariadb:11.6
|
||||
outline | image: postgres:16-alpine
|
||||
rallly | image: postgres:16-alpine
|
||||
sparkyfitness | image: postgres:15-alpine
|
||||
tandoor | image: postgres:16-alpine
|
||||
zipline | image: postgres:16-alpine
|
||||
|
||||
Method: for each templates/<app>/, an app qualifies when BOTH
|
||||
(a) .felhom.yml declares needs_hdd: false
|
||||
(b) docker-compose.yml has an image: line matching postgres|mariadb|mysql
|
||||
Catalog @ 459766cb1639.
|
||||
COUNT MEASURED THIS SESSION: 10 -- agrees with the runbook's table.
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
== 1. plant the control row
|
||||
INSERT 0 1
|
||||
== 2. FIND it
|
||||
found: R356B-FINDABILITY-CONTROL
|
||||
== 3. remove it
|
||||
DELETE 1
|
||||
== 4. fail to find it
|
||||
absent, as required
|
||||
== the planted state that must survive:
|
||||
1|ORIGINAL-VALUE-A
|
||||
== pages visible through the app's own API:
|
||||
page: "\\u00c1rv\\u00edzt\\u0171r\\u0151 t\\u00fck\\u00f6rf\\u00far\\u00f3g\\u00e9p \\u2014 R356b drill"
|
||||
page: "R356B-PAGE-2-sentinel"
|
||||
page: "R356B-PAGE-1-sentinel"
|
||||
count: 3
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
PLANTED STATE — docmost on demo-hp, 2026-08-22, before the off-site run.
|
||||
|
||||
Through the app's OWN API (POST /api/pages/create, format=markdown), 3 pages:
|
||||
id=01a029ad-81fe-792b-87c2-b940ffc80c0c slugId=F4G3YyHtFj title="R356B-PAGE-1-sentinel"
|
||||
body: FELHOM-R356B-SENTINEL-ALPHA-2026-08-22 plain ascii body
|
||||
id=01a029ad-8298-712b-bf80-e41d1186a746 slugId=87aKZR4rlm title="R356B-PAGE-2-sentinel"
|
||||
body: FELHOM-R356B-SENTINEL-BETA-2026-08-22 second body
|
||||
id=01a029ad-8314-76e9-bef0-2108aed8b7a6 slugId=fJOFUooa0N title=<accented, see below>
|
||||
body: FELHOM-R356B-SENTINEL-GAMMA-2026-08-22 akcentusos oldal torzse
|
||||
|
||||
ACCENTED TITLE, as explicit UTF-8 hex bytes:
|
||||
c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a97020e28094205233353662206472696c6c
|
||||
= "Árvíztűrő tükörfúrógép — R356b drill"
|
||||
Non-ASCII never crossed a shell chain: the planting script was base64-encoded on DooPlex and
|
||||
decoded inside the guest.
|
||||
|
||||
DIRECTLY IN THE DATABASE — the Q2 discriminator:
|
||||
table felhom_r356b_discriminator
|
||||
row id=1, marker='ORIGINAL-VALUE-A'
|
||||
|
||||
FINDABILITY CONTROL (evidence 02): a control row was planted, FOUND, removed, and then NOT found.
|
||||
Every absence claim later in this drill rests on that control.
|
||||
+1
@@ -0,0 +1 @@
|
||||
{"last_duration":"2m38s","last_error":"","last_run":"2026-08-22T13:37:41Z","orphaned":false,"progress":{"active":false,"current_app":"","percent":0,"bytes_done":0,"total_bytes":0,"done_human":"","total_human":"","files_done":0,"total_files":0,"current_file":"","elapsed_sec":0,"phase":""},"repo_size_human":"54.1 MB","snapshots":28,"status":"ok"}
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
2332 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/.felhom.yml
|
||||
445 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/app.yaml
|
||||
3105 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/compose/docker-compose.yml
|
||||
139770 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/docmost-postgres.sql
|
||||
1376 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/manifest.json
|
||||
52344320 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_postgres_data.tar
|
||||
56320 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_redis_data.tar
|
||||
1536 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/volume-dumps/docmost_docmost_storage.tar
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
== BEFORE — pages in the database:
|
||||
01a029ad-81fe-792b-87c2-b940ffc80c0c | R356B-PAGE-1-sentinel
|
||||
01a029ad-8298-712b-bf80-e41d1186a746 | R356B-PAGE-2-sentinel
|
||||
01a029ad-8314-76e9-bef0-2108aed8b7a6 | \u00c1rv\u00edzt\u0171r\u0151 t\u00fck\u00f6rf\u00far\u00f3g\u00e9p \u2014 R356b drill
|
||||
== deleting all three pages through the app's own API (force delete, not trash):
|
||||
01a029ad-81fe-792b-87c2-b940ffc80c0c -> {"success":true,"status":200}
|
||||
01a029ad-8298-712b-bf80-e41d1186a746 -> {"success":true,"status":200}
|
||||
01a029ad-8314-76e9-bef0-2108aed8b7a6 -> {"success":true,"status":200}
|
||||
== move the discriminator to a POST-SNAPSHOT value, so a revert is visible:
|
||||
UPDATE 1
|
||||
== AFTER — pages in the database (must be empty):
|
||||
(none — genuinely gone)
|
||||
== AFTER — any row at all in pages, incl. soft-deleted (a trash that still holds them is not a deletion):
|
||||
0
|
||||
== AFTER — discriminator:
|
||||
1|DESTROYED-VALUE-C
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
HTTP/2 302
|
||||
location: /backups/restore/app?name=docmost&flash=A+teljes+visszaall%C3%ADtas+elindult+%E2%80%94+az+allapot+itt+friss%C3%BCl.
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:140: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5be5e049dab2)
|
||||
2026/08/22 13:58:30 dbdump.go:805: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/08/22 13:58:30 dbdump.go:160: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container opengist (image=ghcr.io/thomiceli/opengist:1.13, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container kimai (image=kimai/kimai2:apache-2.57.0, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:140: [DEBUG] DiscoverDatabases: found mariadb container: kimai-db (id=53e87f2a117d)
|
||||
2026/08/22 13:58:30 dbdump.go:160: [DEBUG] DiscoverDatabases: kimai-db → stack=kimai, dbUser=root, dbName=kimai
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container calibre-web (image=crocodilestick/calibre-web-automated:v4.0.6, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.219.0, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container cloudflared (image=cloudflare/cloudflared:2026.6.0, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:133: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database)
|
||||
2026/08/22 13:58:30 dbdump.go:167: [DEBUG] DiscoverDatabases: found 4 database(s), skipped 12 non-DB container(s)
|
||||
2026/08/22 13:58:30 dbdump.go:170: [INFO] [backup] Discovered 4 databases
|
||||
2026/08/22 13:58:30 restore_db.go:77: [INFO] [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)
|
||||
2026/08/22 13:58:30 dbdump.go:700: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/docmost-postgres.sql into docmost-postgres (postgres)
|
||||
2026/08/22 13:58:32 dbdump.go:710: [INFO] [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)
|
||||
2026/08/22 13:58:32 restore_db.go:87: [INFO] [backup] Restore docmost: replayed 1 DB dump(s)
|
||||
2026/08/22 13:58:32 manager.go:986: [DEBUG] [stacks] StartStack docmost: current state=starting deployed=true
|
||||
2026/08/22 13:58:32 manager.go:989: [INFO] [stacks] Starting stack: docmost
|
||||
2026/08/22 13:58:32 manager.go:996: [DEBUG] [stacks] StartStack docmost: prepared 10 env vars for compose
|
||||
2026/08/22 13:58:32 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, APP_SECRET, DB_PASSWORD, DOMAIN, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/08/22 13:58:32 manager.go:1270: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/docmost)
|
||||
2026/08/22 13:58:43 manager.go:1004: [INFO] [stacks] Stack docmost started successfully (took 11.1s)
|
||||
2026/08/22 13:58:46 manager.go:1261: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, APP_SECRET, DB_PASSWORD, DOMAIN, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/08/22 13:58:46 manager.go:1270: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/docmost)
|
||||
2026/08/22 13:58:46 restore.go:204: [DEBUG] [backup] Post-restore health check: docmost not yet running, waiting...
|
||||
2026/08/22 13:58:46 manager.go:1350: [INFO] [stacks] Stack docmost post-start status:
|
||||
2026/08/22 13:58:46 manager.go:1353: [INFO] [stacks] docmost docmost/docmost:0.95.0 running Up 3 seconds (health: starting)
|
||||
2026/08/22 13:58:46 manager.go:1353: [INFO] [stacks] docmost-postgres postgres:16-alpine running Up 16 seconds (healthy)
|
||||
2026/08/22 13:58:46 manager.go:1353: [INFO] [stacks] docmost-redis redis:7-alpine running Up 14 seconds (healthy)
|
||||
2026/08/22 13:58:46 auth.go:134: [DEBUG] [web] auth: valid session for GET /backup/offbox/status
|
||||
2026/08/22 13:58:46 server.go:393: [DEBUG] [web] ServeHTTP: GET /backup/offbox/status from 172.18.0.4:52436
|
||||
2026/08/22 13:58:51 restore.go:204: [DEBUG] [backup] Post-restore health check: docmost not yet running, waiting...
|
||||
2026/08/22 13:58:56 restore.go:199: [DEBUG] [backup] Post-restore health check: docmost is running
|
||||
2026/08/22 13:58:56 offbox_reconstitute.go:452: [INFO] [offbox] reconstituted docmost from snapshot 239dd86c: 0 file(s) placed, 1 DB dump(s) replayed, safety dump=pre-restore-20260822T135827Z-docmost-postgres.sql, skewed=false
|
||||
2026/08/22 13:58:56 offbox_handlers.go:455: [INFO] [web] off-box reconstitute docmost completed (async): files=0 dbs=1 snapshot=239dd86c
|
||||
2026/08/22 13:59:02 healthprobe.go:153: [DEBUG] Health probe docmost: HTTP GET :3000/ → 200 (6ms)
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
== pages in the database after the restore:
|
||||
01a029ad-81fe-792b-87c2-b940ffc80c0c | R356B-PAGE-1-sentinel
|
||||
01a029ad-8298-712b-bf80-e41d1186a746 | R356B-PAGE-2-sentinel
|
||||
01a029ad-8314-76e9-bef0-2108aed8b7a6 | \u00c1rv\u00edzt\u0171r\u0151 t\u00fck\u00f6rf\u00far\u00f3g\u00e9p \u2014 R356b drill
|
||||
== page count:
|
||||
3
|
||||
== THE DISCRIMINATOR (Q2):
|
||||
1|ORIGINAL-VALUE-A
|
||||
== through the app's OWN interface — log in fresh and list:
|
||||
HTTP/2 400
|
||||
count: 0
|
||||
== body of the accented page, read back through the app:
|
||||
title hex:
|
||||
content : null
|
||||
login: HTTP/2 200
|
||||
== pages listed through the app's OWN interface:
|
||||
title(hex)=5c753030633172765c75303065647a745c7530313731725c753031353120745c75303066636b5c753030663672665c7530306661725c7530306633675c753030653970205c7532303134205233353662206472696c6c id=01a029ad-8314-76e9-bef0-2108aed8b7a6
|
||||
title(hex)=52333536422d504147452d322d73656e74696e656c id=01a029ad-8298-712b-bf80-e41d1186a746
|
||||
title(hex)=52333536422d504147452d312d73656e74696e656c id=01a029ad-81fe-792b-87c2-b940ffc80c0c
|
||||
count: 3
|
||||
== the accented page, body read back through the app:
|
||||
title hex : 5c753030633172765c75303065647a745c7530313731725c753031353120745c75303066636b5c753030663672665c7530306661725c7530306633675c753030653970205c7532303134205233353662206472696c6c
|
||||
title : "\\u00c1rv\\u00edzt\\u0171r\\u0151 t\\u00fck\\u00f6rf\\u00far\\u00f3g\\u00e9p \\u2014 R356b drill"
|
||||
body has GAMMA sentinel: True
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
PHASE 1 RESULT — docmost (driveless, Postgres 16), demo-hp, controller v0.219.0
|
||||
|
||||
THE FIVE LEGS, in order, from the controller log (evidence 08). Every one ran; every one succeeded.
|
||||
|
||||
13:58:24 POST /backup/offbox/reconstitute (the endpoint the UI's button posts to)
|
||||
13:58:27 leg 1 safety dump written -> pre-restore-20260822T135827Z-docmost-postgres.sql (135 624 B)
|
||||
13:58:2x leg 2 DB service identified -> [docmost-postgres]
|
||||
13:58:28 leg 3a stack stopped
|
||||
13:58:28 leg 3b volumes replayed -> docmost_docmost_postgres_data
|
||||
13:58:29 docmost_docmost_redis_data
|
||||
13:58:30 docmost_docmost_storage = "Restored 3 Docker volume(s)"
|
||||
13:58:30 leg 4 DB service ONLY started -> docker compose up -d docmost-postgres (0.4s)
|
||||
13:58:30 leg 5 ImportDump docmost-postgres.sql -> docmost-postgres (postgres)
|
||||
13:58:32 "Imported DB dump ... " / "replayed 1 DB dump(s)" (2 s)
|
||||
13:58:32 full StartStack
|
||||
13:58:56 reconstituted docmost from snapshot 239dd86c:
|
||||
0 file(s) placed, 1 DB dump(s) replayed,
|
||||
safety dump=pre-restore-20260822T135827Z-docmost-postgres.sql, skewed=false
|
||||
|
||||
Total 32 s. All three containers healthy afterwards.
|
||||
|
||||
WHAT CAME BACK
|
||||
pages before destruction : 3
|
||||
pages after destruction : 0 (count(*) FROM pages = 0 — not even soft-deleted rows)
|
||||
pages after restore : 3 same ids, same titles
|
||||
discriminator before : ORIGINAL-VALUE-A
|
||||
discriminator moved to : DESTROYED-VALUE-C (deliberately, so a revert would be visible)
|
||||
discriminator after : ORIGINAL-VALUE-A -> the snapshot's value was restored
|
||||
|
||||
Read back through the APP'S OWN interface after the restore: login HTTP 200, /api/pages/recent
|
||||
returned all 3 pages, and the body of the third still contained its GAMMA sentinel.
|
||||
|
||||
HYPOTHESES — what fired and what did not
|
||||
H1 (schema collision, dump replays over a restored data directory) — DID NOT FIRE.
|
||||
The import completed in 2 s with no error under ON_ERROR_STOP=1. This is the notable result:
|
||||
the two legs do not collide in practice for this app.
|
||||
H2 (replayed data directory will not start) — DID NOT FIRE. docmost-postgres came up healthy
|
||||
on a data directory that had just been replaced wholesale from a tar.
|
||||
H4 (credentials) — DID NOT FIRE. ImportDump discovered dbUser=docmost/dbName=docmost from the
|
||||
LIVE container and they matched the restored directory, as predicted: secrets are not
|
||||
regenerated on reconstitution.
|
||||
H3 (MariaDB vs Postgres error handling) — not applicable to this phase; Phase 4.
|
||||
|
||||
A FLAW IN THIS DRILL'S OWN PLANTING, recorded rather than hidden
|
||||
The first accented title was planted through a shell argument chain that double-escaped it, so
|
||||
it was stored as the LITERAL ASCII text Árvízt... rather than as accented bytes.
|
||||
Caught by reading the stored value back as hex. The pages round-tripped exactly, so Phase 1's
|
||||
Q1 answer stands — but the ACCENTED-BYTES coverage did not, and was re-measured in Phase 1b
|
||||
with the title embedded in a python source file inside the guest, never passed as a shell
|
||||
argument. Wire bytes and stored bytes were compared and are identical.
|
||||
|
||||
PHASE 1b — the accented round-trip, re-measured properly
|
||||
planted (wire bytes) : c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662
|
||||
stored at plant time : identical
|
||||
destroyed : all 4 pages permanently; count(*) FROM pages = 0
|
||||
restored (snapshot a49de51d, 14:05:33): 4 pages back
|
||||
accented title after : c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662
|
||||
MATCH : TRUE ("Árvíztűrő tükörfúrógép 2 — R356b")
|
||||
discriminator : DESTROYED-VALUE-C2 -> ORIGINAL-VALUE-A
|
||||
|
||||
The first three pages also came back; page 01a029ad-8314-... still carries the double-escaped
|
||||
ASCII title, which is what this drill's own faulty plant stored. It is left in place as the
|
||||
record of that flaw rather than quietly re-planted.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
== BEFORE: page count and the accented title as hex
|
||||
4
|
||||
accented title hex: c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662
|
||||
== destroy ALL pages through the app (permanent, not trash):
|
||||
01a029ad-81fe-792b-87c2-b940ffc80c0c -> {"success":true,"status":200}
|
||||
01a029ad-8298-712b-bf80-e41d1186a746 -> {"success":true,"status":200}
|
||||
01a029ad-8314-76e9-bef0-2108aed8b7a6 -> {"success":true,"status":200}
|
||||
01a029c6-47b8-7252-8ea2-4c2036f9e1d0 -> {"success":true,"status":200}
|
||||
UPDATE 1
|
||||
== AFTER destruction — page count (must be 0) and discriminator:
|
||||
0
|
||||
1|DESTROYED-VALUE-C2
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
== page count after restore (was 0 before):
|
||||
4
|
||||
== all titles, as UTF-8 hex:
|
||||
01a029ad-81fe-792b-87c2-b940ffc80c0c hex=52333536422d504147452d312d73656e74696e656c
|
||||
01a029ad-8298-712b-bf80-e41d1186a746 hex=52333536422d504147452d322d73656e74696e656c
|
||||
01a029ad-8314-76e9-bef0-2108aed8b7a6 hex=5c753030633172765c75303065647a745c7530313731725c753031353120745c75303066636b5c753030663672665c7530306661725c7530306633675c753030653970205c7532303134205233353662206472696c6c
|
||||
01a029c6-47b8-7252-8ea2-4c2036f9e1d0 hex=c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662
|
||||
== THE ACCENTED PAGE — expected hex:
|
||||
c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662
|
||||
actual hex: c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662
|
||||
MATCH: True
|
||||
== discriminator (was DESTROYED-VALUE-C2 before the restore):
|
||||
1|ORIGINAL-VALUE-A
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
BEFORE sha256: c5414f24426a4a0050becadeb564990465bf48772a7cb029daaebd22ef788441
|
||||
BEFORE line 1536: 1 ORIGINAL-VALUE-A 2026-08-22 15:34:22.455005+02
|
||||
AFTER line 1536: 1 ALTERED-VALUE-B 2026-08-22 15:34:22.455005+02
|
||||
AFTER sha256: 834c32b19b9ec053b2909f031751cbbcf8d14941b5af2fdf1f3779b113b82d36
|
||||
occurrences of ORIGINAL-VALUE-A left in the scratch dump: 0
|
||||
occurrences of ALTERED-VALUE-B: 1
|
||||
--- the volume tar still holds the ORIGINAL (it is the other candidate):
|
||||
-rw-r--r-- 1 root root 53301760 Aug 22 14:01 /mnt/felhom-drives/hdd_1/backups/offsite-restore/docmost/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/../volume-dumps/docmost_docmost_postgres_data.tar
|
||||
|
||||
THE THREE-WAY DISCRIMINATOR, set up before the reconstitution:
|
||||
live database now holds : LIVE-VALUE-C3 <- if this survives, NEITHER leg wrote
|
||||
volume tar holds : ORIGINAL-VALUE-A <- if this wins, the volume tar supplied the state
|
||||
altered SQL dump holds : ALTERED-VALUE-B <- if this wins, the dump supplied it (as the comment intends)
|
||||
|
||||
FENCE: the mutation was applied to the PREPARED SCRATCH ONLY. The off-site store and the live
|
||||
recovery unit were not touched. Proved after the run by re-preparing a fresh scratch from the same
|
||||
snapshot and confirming it still reads ORIGINAL-VALUE-A.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
fresh scratch sha256: c5414f24426a4a0050becadeb564990465bf48772a7cb029daaebd22ef788441
|
||||
line 1536 : 1 ORIGINAL-VALUE-A 2026-08-22 15:34:22.455005+02
|
||||
ORIGINAL-VALUE-A present: 1
|
||||
ALTERED-VALUE-B present: 0
|
||||
0
|
||||
--- the LIVE recovery unit db-dumps dir (never touched by a restore):
|
||||
total 420
|
||||
drwxr-xr-x 2 root root 4096 Aug 22 14:07 .
|
||||
drwxr-xr-x 5 root root 4096 Aug 22 14:05 ..
|
||||
-rw-r--r-- 1 root root 135624 Aug 22 13:58 pre-restore-20260822T135827Z-docmost-postgres.sql
|
||||
-rw-r--r-- 1 root root 135847 Aug 22 14:05 pre-restore-20260822T140501Z-docmost-postgres.sql
|
||||
-rw-r--r-- 1 root root 141361 Aug 22 14:07 pre-restore-20260822T140713Z-docmost-postgres.sql
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
live discriminator before Phase 3: ALTERED-VALUE-B
|
||||
live page count before Phase 3 : 4
|
||||
--- CORRUPTION METHOD: truncate the dump mid-COPY-block (a realistic partial/corrupt dump)
|
||||
size before: 141364
|
||||
size after : 62000
|
||||
last line now: COPY public.felhom_r356b_discriminator
|
||||
sha256 after: 30e84d2c0d0bcf3e7043459a0d135e96cdb65947f4138f181837b1ee35d2e087
|
||||
+1
@@ -0,0 +1 @@
|
||||
2026/08/22 14:09:40 offbox_handlers.go:451: [ERROR] [web] off-box reconstitute docmost (async): az adatbázis visszaállítása sikertelen: importing postgres dump for docmost: postgres import into docmost-postgres failed: ERROR: syntax error at end of input
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
=== CUSTOMER-FACING MESSAGE, VERBATIM ===
|
||||
A teljes visszaállítás sikertelen: az adatbázis visszaállítása sikertelen: importing postgres dump for docmost: postgres import into docmost-postgres failed: ERROR: syntax error at end of input
|
||||
LINE 1: COPY public.felhom_r356b_discriminator
|
||||
^ — exit status 3 — a korábbi állapot mentése megvan: pre-restore-20260822T140924Z-docmost-postgres.sql
|
||||
|
||||
=== UTF-8 hex ===
|
||||
412074656c6a657320766973737a61c3a16c6cc3ad74c3a1732073696b657274656c656e3a20617a206164617462c3a17a697320766973737a61c3a16c6cc3ad74c3a173612073696b657274656c656e3a20696d706f7274696e6720706f7374677265732064756d7020666f7220646f636d6f73743a20706f73746772657320696d706f727420696e746f20646f636d6f73742d706f737467726573206661696c65643a204552524f523a202073796e746178206572726f7220617420656e64206f6620696e7075740a4c494e4520313a20434f5059207075626c69632e66656c686f6d5f72333536625f6469736372696d696e61746f72200a20202020202020202020202020202020202020202020202020202020202020202020202020202020202020202020205e20e28094206578697420737461747573203320e280942061206b6f72c3a162626920c3a16c6c61706f74206d656e74c3a97365206d656776616e3a207072652d726573746f72652d3230323630383232543134303932345a2d646f636d6f73742d706f7374677265732e73716c
|
||||
|
||||
=== byte length: 407
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
== tables left in the public schema (a healthy docmost has ~30):
|
||||
43
|
||||
== does pages still exist?
|
||||
pages
|
||||
== rows in pages (was 4 before the failed restore):
|
||||
0
|
||||
== rows in users (the workspace owner):
|
||||
0
|
||||
== rows in spaces :
|
||||
0
|
||||
== discriminator rows (was 1):
|
||||
0
|
||||
== the undo copy that the message named:
|
||||
-rw-r--r-- 1 root root 141363 Aug 22 14:09 /mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260822T140924Z-docmost-postgres.sql
|
||||
== is that undo copy a VALID dump? (header + row count for pages)
|
||||
43
|
||||
--
|
||||
-- PostgreSQL database dump
|
||||
--
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
== does the undo copy CONTAIN the lost data?
|
||||
rows in its pages COPY block: 4
|
||||
rows in its users COPY block: 1
|
||||
contains the accented title : 1
|
||||
== APPLY it by hand — proving the undo copy WORKS (nothing in the product will do this):
|
||||
|
||||
(1 row)
|
||||
|
||||
== state after applying the undo by hand:
|
||||
pages: 4
|
||||
users: 1
|
||||
discriminator: ALTERED-VALUE-B
|
||||
+53
@@ -0,0 +1,53 @@
|
||||
PHASE 3 RESULT — the failure path (docmost, Postgres)
|
||||
|
||||
CORRUPTION METHOD: the prepared scratch's db-dumps/docmost-postgres.sql was TRUNCATED from
|
||||
141 364 B to 62 000 B, landing mid-statement at "COPY public.felhom_r356b_discriminator ".
|
||||
sha256 30e84d2c0d0bcf3e7043459a0d135e96cdb65947f4138f181837b1ee35d2e087.
|
||||
Scratch only. Store and live unit untouched.
|
||||
|
||||
DOES THE CUSTOMER SEE A FAILURE? YES.
|
||||
Rendered in an `alert alert-error` block under the heading "Eredmény", beginning
|
||||
"A teljes visszaállítás sikertelen:". Not a warning beside a success.
|
||||
|
||||
DOES THE MESSAGE NAME THE UNDO COPY? YES.
|
||||
"... — a korábbi állapot mentése megvan: pre-restore-20260822T140924Z-docmost-postgres.sql"
|
||||
|
||||
IS THE APP RUNNING AFTERWARDS? NO — it crash-loops.
|
||||
`docmost | Restarting (1)`; docmost-postgres and docmost-redis stay healthy. Login through
|
||||
traefik returns 404 because there is no healthy backend.
|
||||
This is recorded as HONEST rather than as a defect: the app is visibly broken, not falsely
|
||||
green. An earlier reading in this session said "reports healthy" — that was wrong, taken from
|
||||
a "health: starting" line during the restart, and is corrected here.
|
||||
|
||||
WHAT STATE IS THE DATA IN? EMPTY, and that is inherent to the mechanism.
|
||||
The truncated dump ran its DROP/CREATE sequence and died partway through the data:
|
||||
tables in public schema : 43 (schema intact)
|
||||
pages : 0 (was 4)
|
||||
users : 0 (was 1 — the workspace owner)
|
||||
spaces : 0
|
||||
felhom_r356b_discriminator: 0 (was 1)
|
||||
|
||||
IS THE UNDO COPY VALID AND SUFFICIENT? YES — proven by using it.
|
||||
141 363 B, proper pg_dump header, 43 COPY blocks.
|
||||
Its pages block holds 4 rows, its users block 1 row, and it contains the accented title.
|
||||
Applied BY HAND (docker cp + psql -v ON_ERROR_STOP=1): pages 4, users 1,
|
||||
discriminator ALTERED-VALUE-B — exactly the pre-restore state.
|
||||
|
||||
THE GAP — and it is the finding of this phase:
|
||||
NOTHING IN THE PRODUCT CAN APPLY THAT UNDO COPY.
|
||||
- The filename appears ONLY inside the error text. There is no button, no list entry, no route.
|
||||
- `preRestoreDumpPrefix` ("pre-restore-") is explicitly SKIPPED at three places so these files
|
||||
are never offered as a restore source:
|
||||
internal/backup/restore_unit.go:125
|
||||
internal/backup/offbox_reconstitute.go:489
|
||||
internal/backup/offbox_reconstitute.go:539
|
||||
- So a customer whose off-site dump is corrupt is left with a crash-looping app, an emptied
|
||||
database, and a filename they cannot act on. The data is recoverable — by us, by hand.
|
||||
|
||||
SECOND FINDING — the message leaks raw engine internals at a Hungarian customer.
|
||||
407 bytes, of which the middle ~250 are untranslated English psql output including a caret
|
||||
diagram and "exit status 3":
|
||||
"importing postgres dump for docmost: postgres import into docmost-postgres failed:
|
||||
ERROR: syntax error at end of input
|
||||
LINE 1: COPY public.felhom_r356b_discriminator
|
||||
^ — exit status 3"
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
PHASE 4 PLANTED STATE — bookstack (driveless, MariaDB 12.3, container bookstack-db)
|
||||
|
||||
Deployed by this drill (it was not on the box). Off-site switch turned ON and verified.
|
||||
|
||||
Through the app's OWN web interface (login form -> POST /books, session cookie + Laravel _token):
|
||||
entities: id=1 type=book
|
||||
name (UTF-8 hex): C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662
|
||||
= "Árvíztűrő könyvespolc — R356b"
|
||||
description : FELHOM-R356B-BOOKSTACK-SENTINEL-2026-08-22
|
||||
The name was written from a python file inside the guest and passed to curl with
|
||||
--data-urlencode "name@/tmp/bs.name", so it never crossed a shell argument.
|
||||
|
||||
NOTE: this BookStack version has no `books` table — the 2025_09_15 migration
|
||||
`drop_old_entity_tables` unified books/chapters/pages into `entities`. 42 tables total.
|
||||
|
||||
Directly in the database — the discriminator:
|
||||
felhom_r356b_disc: id=1, marker='ORIGINAL-VALUE-A'
|
||||
|
||||
FINDABILITY CONTROL: a control row was planted (id=99), FOUND, removed, and then NOT found.
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
== BEFORE:
|
||||
1 book C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662
|
||||
1 ORIGINAL-VALUE-A
|
||||
== DESTROY (delete the book through the app is a soft-delete; this removes it outright):
|
||||
== AFTER destruction — entities count (must be 0) and discriminator:
|
||||
0
|
||||
1 DESTROYED-VALUE-C
|
||||
== also check the deletions/recycle-bin table is not holding it:
|
||||
0
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
2075 ./mnt/sys_drive/felhom-data/backups/primary/bookstack/compose/.felhom.yml
|
||||
426 ./mnt/sys_drive/felhom-data/backups/primary/bookstack/compose/app.yaml
|
||||
2330 ./mnt/sys_drive/felhom-data/backups/primary/bookstack/compose/docker-compose.yml
|
||||
58775 ./mnt/sys_drive/felhom-data/backups/primary/bookstack/db-dumps/bookstack-mariadb.sql
|
||||
1325 ./mnt/sys_drive/felhom-data/backups/primary/bookstack/manifest.json
|
||||
136704 ./mnt/sys_drive/felhom-data/backups/primary/bookstack/volume-dumps/bookstack_bookstack_config.tar
|
||||
160945152 ./mnt/sys_drive/felhom-data/backups/primary/bookstack/volume-dumps/bookstack_bookstack_db_data.tar
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
== entities after restore (was 0):
|
||||
1 book C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662
|
||||
== expected accented hex:
|
||||
C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662
|
||||
== discriminator (was DESTROYED-VALUE-C):
|
||||
1 ORIGINAL-VALUE-A
|
||||
== containers:
|
||||
bookstack | Up 41 seconds (healthy)
|
||||
bookstack-db | Up 47 seconds (healthy)
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
size before : 30000
|
||||
sha before : e4575b65d84ef24a605c56883f54384c3cd0d91e75862bad9ae69ec7bff52da4
|
||||
size after : 30000
|
||||
sha after : e4575b65d84ef24a605c56883f54384c3cd0d91e75862bad9ae69ec7bff52da4
|
||||
tail (truncation point):
|
||||
51_update_entity_relation_columns',1),
|
||||
(96,'2025_09_15_134813_drop_old_entity_tables',1),
|
||||
(97,'2025_
|
||||
+1
@@ -0,0 +1 @@
|
||||
2026/08/22 14:24:30 offbox_handlers.go:451: [ERROR] [web] off-box reconstitute bookstack (async): az adatbázis visszaállítása sikertelen: importing mariadb dump for bookstack: mariadb import into bookstack-db failed: --------------
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
=== CUSTOMER-FACING MESSAGE, VERBATIM ===
|
||||
A teljes visszaállítás sikertelen: az adatbázis visszaállítása sikertelen: importing mariadb dump for bookstack: mariadb import into bookstack-db failed: --------------
|
||||
INSERT INTO `migrations` VALUES
|
||||
(1,'2014_10_12_000000_create_users_table',1),
|
||||
(2,'2014_10_12_100000_create_password_resets_table',1),
|
||||
(3,'2015_07_12_114933_create_books_table',1),
|
||||
(4,'2015_07_12_190027_create_pages_table',1),
|
||||
(5,'2015_07_13_172121_create_images_table',1),
|
||||
(6,'2015_07_ — exit status 1 — a korábbi állapot mentése megvan: pre-restore-20260822T142418Z-bookstack-mariadb.sql
|
||||
|
||||
=== UTF-8 hex ===
|
||||
412074656c6a657320766973737a61c3a16c6cc3ad74c3a1732073696b657274656c656e3a20617a206164617462c3a17a697320766973737a61c3a16c6cc3ad74c3a173612073696b657274656c656e3a20696d706f7274696e67206d6172696164622064756d7020666f7220626f6f6b737461636b3a206d61726961646220696d706f727420696e746f20626f6f6b737461636b2d6462206661696c65643a202d2d2d2d2d2d2d2d2d2d2d2d2d2d0a494e5345525420494e544f20606d6967726174696f6e73602056414c5545530a28312c262333393b323031345f31305f31325f3030303030305f6372656174655f75736572735f7461626c65262333393b2c31292c0a28322c262333393b323031345f31305f31325f3130303030305f6372656174655f70617373776f72645f7265736574735f7461626c65262333393b2c31292c0a28332c262333393b323031355f30375f31325f3131343933335f6372656174655f626f6f6b735f7461626c65262333393b2c31292c0a28342c262333393b323031355f30375f31325f3139303032375f6372656174655f70616765735f7461626c65262333393b2c31292c0a28352c262333393b323031355f30375f31335f3137323132315f6372656174655f696d616765735f7461626c65262333393b2c31292c0a28362c262333393b323031355f30375f20e28094206578697420737461747573203120e280942061206b6f72c3a162626920c3a16c6c61706f74206d656e74c3a97365206d656776616e3a207072652d726573746f72652d3230323630383232543134323431385a2d626f6f6b737461636b2d6d6172696164622e73716c
|
||||
|
||||
=== byte length: 615
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
== tables left:
|
||||
42
|
||||
== entities (was 1):
|
||||
1
|
||||
== users:
|
||||
2
|
||||
== discriminator table still there?
|
||||
1
|
||||
== migrations rows (the table the dump died inside):
|
||||
0
|
||||
== containers:
|
||||
bookstack | Up 2 minutes (healthy)
|
||||
bookstack-db | Up 3 minutes (healthy)
|
||||
== undo copy on disk:
|
||||
total 128
|
||||
drwxr-xr-x 2 root root 4096 Aug 22 14:24 .
|
||||
drwxr-xr-x 5 root root 4096 Aug 22 14:25 ..
|
||||
-rw-r--r-- 1 root root 58595 Aug 22 14:22 pre-restore-20260822T142223Z-bookstack-mariadb.sql
|
||||
-rw-r--r-- 1 root root 58775 Aug 22 14:24 pre-restore-20260822T142418Z-bookstack-mariadb.sql
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
== applying the undo copy by hand (nothing in the product will do this):
|
||||
== state after:
|
||||
entities : 1
|
||||
users : 2
|
||||
migrations: 102
|
||||
book name : C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662
|
||||
disc : ORIGINAL-VALUE-A
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
repo: sftp:u629488-sub3@u629488-sub3.your-storagebox.de:/home/felhom-repo (port 23)
|
||||
using temporary cache in /tmp/restic-check-cache-2896564091
|
||||
create exclusive lock for repository
|
||||
load indexes
|
||||
check all packs
|
||||
check snapshots, trees and blobs
|
||||
[0:01] 100.00% 29 / 29 snapshots
|
||||
|
||||
no errors were found
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
felhom-pbs pbs active 0 0 0 0.00%
|
||||
local dir active 40453376 5772488 32593772 14.27%
|
||||
local-lvm lvmthin active 56487936 19663450 36824485 34.81%
|
||||
--- guest 9201 disk:
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 10G 56G 16% /var/lib/felhom
|
||||
/dev/mapper/pve-vm--9201--disk--1 69G 10G 56G 16% /mnt/sys_drive
|
||||
Reference in New Issue
Block a user