admin
4e488321bf
gates / gates (push) Successful in 16s
A drill, not an implementation. No code, no version bump, no CHANGELOG entry. Ten of the forty driveless apps carry a database; I re-measured that count and got 10. For those ten the restore is a five-leg operation that never ran at all until this week, because R-356 refused before any of it started. Walked end to end on demo-hp for both engines - docmost (Postgres 16) and bookstack (MariaDB 12.3) - each deployed for the drill, planted through the app's own interface, destroyed for real, restored through the endpoint the UI posts to. Q1 does it complete: YES. All five legs ran and succeeded, 32s / 25s. Accented names byte-identical both directions. Q2 which leg won: the SQL DUMP. Three-way discriminator returned the altered dump's value. This confirms R-164's F17 ordering on the OFF-SITE path; R-164 only ever cited the local one. Scratch-only mutation; store proved unmutated. Q3 does a failure tell the truth: partly, and two defects. Filed R-379 (HIGH, the undo copy is valid, named, and unappliable by any product action - proven by applying it by hand on both engines), R-380 (HIGH, a failed MariaDB replay leaves a partial database behind an app reporting healthy, where Postgres crash-loops visibly), R-381 (MEDIUM, the failure message pastes engine stderr including customer table rows into the Hungarian surface), R-382 (LOW, the summary log omits the volume count it already has). H1, H2 and H4 did NOT fire and that is recorded. H3 fired in a shape nobody predicted: not a quiet success, but a loud error over a silent inconsistency. R-361 reproduced independently on a second app. restic check: no errors, 29 snapshots. A flaw in the drill's own planting - a double-escaped accented title - was caught by reading stored bytes as hex, recorded, and re-measured in Phase 1b. Register 325236 -> 330683 bytes. Nothing dropped. Teardown: two apps retained with reason, no pvesm before-snapshot taken (said plainly), no hub-side record created.
191 lines
11 KiB
Markdown
191 lines
11 KiB
Markdown
# DRILL — R-356b: the off-site restore for a driveless app that HAS a database (2026-08-22)
|
||
|
||
**This was a drill, not an implementation. No production code was written, no version bumped, no
|
||
CHANGELOG entry made.** The output is this document and four register rows.
|
||
|
||
**Subject:** `demo-hp` (Tier 0, disposable), guest 9201, controller **v0.219.0**, floor **0.219.0**.
|
||
**Method:** endpoint-level. `claude-in-chrome` is not available on DooPlex, so every product action was
|
||
the exact HTTP endpoint the UI's own form posts to — `/api/stacks/<app>/deploy`,
|
||
`/backup/offbox/{toggle,run,restore,reconstitute}` — driven with the session cookie and the session
|
||
CSRF token scraped from the page. App content was planted through each app's **own** interface
|
||
(docmost's JSON API; BookStack's login form + `POST /books`).
|
||
|
||
## 1. Baselines, confirmed at drill start
|
||
|
||
| repo | HEAD | note |
|
||
|---|---|---|
|
||
| felhom-controller | `0f3cf0dbb2c0` | v0.219.0 |
|
||
| felhom-agent | `40d857b52711` | v0.130.0 |
|
||
| felhom.eu | `8c9f1b798b12` | ahead of the runbook's hash — this session's own golden-bake commit |
|
||
| app-catalog | `459766cb1639` | |
|
||
|
||
Read from the **hub** (`/configuration`, Basic auth, ClusterIP): `golden_version` **0.219.0**,
|
||
`agent_version` **0.130.0**, `min_agent` **0.129.0**, global controller floor **0.219.0**. All four as
|
||
expected — the operator's three-field vouch and the floor raise both landed.
|
||
|
||
**The 10-app count, measured in this session and not taken from the runbook: 10.**
|
||
Method: `needs_hdd: false` in `.felhom.yml` **and** a `postgres|mariadb|mysql` image in the compose.
|
||
Same ten names as the runbook: bookstack, calcom, claper, docmost, kimai, outline, rallly,
|
||
sparkyfitness, tandoor, zipline. Evidence `01`.
|
||
|
||
**Architecture document read:** `documentation/architecture/07-backup-architecture.md` §6.1, §6.2 and
|
||
§6.3 **as corrected on 2026-08-22** — including the corrected Tier-3 row, the note that R-102 is *not*
|
||
closed, and the `[DESIGN]` paragraph on destination resolution.
|
||
|
||
Register ceiling before this drill: **R-378**.
|
||
|
||
## 2. The three questions
|
||
|
||
### Q1 — Does it complete at all? **YES.**
|
||
|
||
A driveless app with a database restores end to end, comes back healthy, and returns the planted data.
|
||
|
||
Measured on `docmost` (Postgres 16). All five legs that had **never run in this combination** executed
|
||
in order and every one succeeded, in **32 seconds**:
|
||
|
||
| leg | evidence |
|
||
|---|---|
|
||
| undo copy of the live DB, taken while up | `pre-restore-20260822T135827Z-docmost-postgres.sql`, 135 624 B |
|
||
| database-service identification | `[docmost-postgres]` |
|
||
| stop → place files → **replay volumes** | "Restored **3** Docker volume(s)" — incl. the 52 MB Postgres data directory |
|
||
| start **only** the DB service | `docker compose up -d docmost-postgres`, 0.4 s |
|
||
| replay the SQL dump on top | "Imported DB dump docmost-postgres.sql", 2 s |
|
||
|
||
3 pages planted through docmost's own API → destroyed (`count(*) FROM pages` = **0**, no soft-deleted
|
||
rows either) → **3 pages back**, same ids. The discriminator went `ORIGINAL-VALUE-A` →
|
||
`DESTROYED-VALUE-C` → **`ORIGINAL-VALUE-A`**.
|
||
|
||
Repeated on `bookstack` (MariaDB 12.3): all five legs, **25 seconds**, 2 volumes replayed including a
|
||
**161 MB** MariaDB data directory, the planted book back with its accented name byte-identical.
|
||
|
||
**Accented round-trip, byte-for-byte:**
|
||
`c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662`
|
||
("Árvíztűrő tükörfúrógép 2 — R356b", docmost) and
|
||
`C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662`
|
||
("Árvíztűrő könyvespolc — R356b", bookstack). Both identical before and after. Non-ASCII never crossed
|
||
a shell chain: strings were embedded in python files inside the guest, or passed with
|
||
`--data-urlencode "name@file"`.
|
||
|
||
> **A flaw in this drill's own method, recorded rather than hidden.** The FIRST accented title was
|
||
> planted through a shell argument chain that double-escaped it, so it was stored as the literal ASCII
|
||
> text `Árv…` rather than as accented bytes. It was caught by reading the stored value back **as
|
||
> hex** — the same discipline that makes the rest of these numbers worth anything. The pages
|
||
> round-tripped exactly either way, so Q1's answer stands, but the accented coverage did not, and was
|
||
> re-measured in Phase 1b. The faulty page is deliberately left in place as the record.
|
||
|
||
### Q2 — Which leg returned the data? **The SQL dump.**
|
||
|
||
A three-way discriminator was built so the answer could not be ambiguous:
|
||
|
||
| holder | value |
|
||
|---|---|
|
||
| live database before the restore | `LIVE-VALUE-C3` — if it survived, neither leg wrote |
|
||
| the volume tar (52 MB Postgres data dir) | `ORIGINAL-VALUE-A` |
|
||
| the SQL dump, **altered in the scratch only** | `ALTERED-VALUE-B` |
|
||
|
||
**Result: `ALTERED-VALUE-B`.** The dump won. The ordering the code comment asserts —
|
||
*"volumes FIRST, database after, so a logical .sql dump still wins over whatever copy of the same
|
||
database a volume tar happens to contain"* (`offbox_reconstitute.go`, the R-354 block) — **holds in
|
||
practice on the off-site path.** It was an intention; it is now an observation.
|
||
|
||
Mutation asserted: dump sha256 `c5414f24…` → `834c32b1…`, zero occurrences of the original left, one of
|
||
the altered. Evidence `13`.
|
||
|
||
**The fence held.** Only the prepared scratch was mutated. Proved afterwards by re-preparing a fresh
|
||
scratch from the same snapshot: sha256 back to `c5414f24…`, reading `ORIGINAL-VALUE-A`. Evidence `14`.
|
||
|
||
> **This confirms an existing claim on a new path rather than establishing a new one.** **R-164**
|
||
> already records *"the dump is authoritative and replayed after the tar so it WINS (F17)"* — but cites
|
||
> `internal/backup/restore_unit.go:262-266`, the **local** restore. Q2 extends that to
|
||
> `offbox_reconstitute.go`, the **off-site** path, which had never been exercised for this app class.
|
||
|
||
### Q3 — Does a failure in the database leg tell the truth? **Partly. Two defects.**
|
||
|
||
Measured by truncating the dump in the scratch mid-statement, on both engines.
|
||
|
||
| | Postgres (docmost) | MariaDB (bookstack) |
|
||
|---|---|---|
|
||
| customer sees a failure | **yes** — `alert alert-error`, "sikertelen" | **yes** |
|
||
| message names the undo copy | **yes**, by filename | **yes** |
|
||
| app afterwards | **crash-loops** — visibly broken | **`health=healthy, running=true, restarts=0`** |
|
||
| database afterwards | **emptied** — 43 tables, 0 rows in pages/users/spaces | **partially applied** — book and users intact, `migrations` **wiped to 0 rows** |
|
||
| undo copy valid? | **yes** — proven by applying it | **yes** — proven by applying it |
|
||
| product can apply the undo copy | **NO** | **NO** |
|
||
|
||
## 3. Hypotheses — what fired and what did not
|
||
|
||
- **H1 (schema collision).** **DID NOT FIRE.** The dump replayed cleanly onto a freshly-replaced
|
||
Postgres data directory in 2 s under `ON_ERROR_STOP=1`. This is the notable negative: the two legs do
|
||
not collide in practice for this app class.
|
||
- **H2 (replayed data directory will not start).** **DID NOT FIRE.** Both engines started healthy on a
|
||
data directory that had just been replaced wholesale from a tar.
|
||
- **H3 (MariaDB and Postgres do not fail the same way).** **FIRED — but not in the predicted shape.**
|
||
The prediction was a quiet partial success with no error. What actually happens is a **visible
|
||
error** *and* a **silently inconsistent database behind a healthy-looking app**. See R-380.
|
||
- **H4 (credentials).** **DID NOT FIRE.** `ImportDump` discovered `dbUser=docmost/dbName=docmost` from
|
||
the **live** container and they matched the restored directory, exactly as predicted — secrets are
|
||
not regenerated on reconstitution.
|
||
|
||
## 4. Findings filed
|
||
|
||
| id | severity | what |
|
||
|---|---|---|
|
||
| **R-379** | HIGH | The undo copy is valid, is named to the customer, and **no product action can apply it** |
|
||
| **R-380** | HIGH | A failed **MariaDB** replay leaves a partially-applied database behind a **healthy** app |
|
||
| **R-381** | MEDIUM | The failure message pastes raw engine stderr — **including customer database rows** — into the Hungarian customer surface |
|
||
| **R-382** | LOW | The reconstitution's summary log line omits the volume count it already has |
|
||
|
||
**An independent reproduction of an existing row.** After the first reconstitution, docmost's
|
||
`db-dumps/` held **only** `pre-restore-*` files — the app's own `docmost-postgres.sql` was gone. That is
|
||
**R-361** reproducing on a second app, unprompted. Not re-filed; noted here as corroboration.
|
||
|
||
## 5. Observations — noticed, recorded, NOT acted on
|
||
|
||
- **A newly deployed app is not in the off-site set.** `docmost` and `bookstack` were both absent from
|
||
`app_backup` entirely until switched on by hand. This is plausibly deliberate (the switch is the
|
||
customer's), and it is **not filed as a defect** — but the consequence is that an app can be deployed,
|
||
run, and never leave the box, while the nightly run reports "backup OK". Stated so the next reader can
|
||
decide whether the default is right.
|
||
- **`restic check` was run by hand and passed** — `no errors were found`, 29 snapshots. Nothing in the
|
||
product runs it (**R-359**, already filed). Reproducing the invocation took three attempts because the
|
||
key file is `ssh_key`, not `id_offbox` as the arg-builder's variable naming suggests.
|
||
- **The `-db` container-suffix attribution is correct here.** `bookstack-db`'s dump landed in
|
||
`bookstack`'s own unit as `bookstack-mariadb.sql` — the R-355 failure shape does **not** reproduce,
|
||
because v0.218.0's compose-project attribution handles it.
|
||
|
||
## 6. Phases run
|
||
|
||
| phase | status |
|
||
|---|---|
|
||
| 1 — Postgres, does it complete | **run** |
|
||
| 1b — accented round-trip, re-measured after a method flaw | **run** |
|
||
| 2 — which leg won | **run** |
|
||
| 3 — failure path, Postgres | **run** |
|
||
| 4 — MariaDB: Phase 1 and Phase 3 equivalents | **run** |
|
||
|
||
Nothing was dropped.
|
||
|
||
## 7. Evidence
|
||
|
||
`evidence/`, files `00`–`31`, copied off `demo-hp` at the end of each phase. Nothing was reverted or
|
||
torn down mid-run, so no intermediate teardown could have taken any of it.
|
||
|
||
## 8. Teardown — three layers
|
||
|
||
1. **Apps.** `docmost` and `bookstack` were **deployed by this drill** and are **RETAINED**, with their
|
||
planted data. Reason: the planted data *is* the evidence for every claim above, and they are the only
|
||
deployed members of the 10-app class on the box, so the next drill in this area starts from a real
|
||
subject instead of rebuilding one. Both are healthy and sane at the end (docmost restored via its
|
||
undo copy; bookstack's `migrations` table restored 0 → 102 rows the same way).
|
||
2. **Storage.** No `pvesm` "before" snapshot was taken — **stated plainly rather than reconstructed.**
|
||
Measured directly instead: the two apps' volumes total ~233 MB
|
||
(docmost 67 MB + 228 KB + 4 KB, bookstack 166 MB + 224 KB) inside guest 9201, which sits at
|
||
16 % of 69 GB. Host `local-lvm` 34.81 %, `local` 14.27 %.
|
||
3. **Hub-side record.** **None was created.** This drill created no customer and no appliance record;
|
||
it deployed two apps inside an existing guest of an existing customer. Nothing to dispose of.
|
||
|
||
## 9. Store integrity at the end
|
||
|
||
`restic check` against the live repository: **`no errors were found`**, 29 snapshots, exclusive lock
|
||
taken and released. Evidence `30`.
|