felhom.eu/documentation/audits/DRILL-r356b-driveless-db-restore-2026-08-22

DRILL — R-356b: the off-site restore for a driveless app that HAS a database (2026-08-22)

This was a drill, not an implementation. No production code was written, no version bumped, no CHANGELOG entry made. The output is this document and four register rows.

Subject: demo-hp (Tier 0, disposable), guest 9201, controller v0.219.0, floor 0.219.0. Method: endpoint-level. claude-in-chrome is not available on DooPlex, so every product action was the exact HTTP endpoint the UI's own form posts to — /api/stacks/<app>/deploy, /backup/offbox/{toggle,run,restore,reconstitute} — driven with the session cookie and the session CSRF token scraped from the page. App content was planted through each app's own interface (docmost's JSON API; BookStack's login form + POST /books).

1. Baselines, confirmed at drill start

repo	HEAD	note
felhom-controller	0f3cf0dbb2c0	v0.219.0
felhom-agent	40d857b52711	v0.130.0
felhom.eu	8c9f1b798b12	ahead of the runbook's hash — this session's own golden-bake commit
app-catalog	459766cb1639

Read from the hub (/configuration, Basic auth, ClusterIP): golden_version 0.219.0, agent_version 0.130.0, min_agent 0.129.0, global controller floor 0.219.0. All four as expected — the operator's three-field vouch and the floor raise both landed.

The 10-app count, measured in this session and not taken from the runbook: 10. Method: needs_hdd: false in .felhom.yml and a postgres|mariadb|mysql image in the compose. Same ten names as the runbook: bookstack, calcom, claper, docmost, kimai, outline, rallly, sparkyfitness, tandoor, zipline. Evidence 01.

Architecture document read: documentation/architecture/07-backup-architecture.md §6.1, §6.2 and §6.3 as corrected on 2026-08-22 — including the corrected Tier-3 row, the note that R-102 is not closed, and the [DESIGN] paragraph on destination resolution.

Register ceiling before this drill: R-378.

2. The three questions

Q1 — Does it complete at all? YES.

A driveless app with a database restores end to end, comes back healthy, and returns the planted data.

Measured on docmost (Postgres 16). All five legs that had never run in this combination executed in order and every one succeeded, in 32 seconds:

leg	evidence
undo copy of the live DB, taken while up	pre-restore-20260822T135827Z-docmost-postgres.sql, 135 624 B
database-service identification	[docmost-postgres]
stop → place files → replay volumes	"Restored 3 Docker volume(s)" — incl. the 52 MB Postgres data directory
start only the DB service	docker compose up -d docmost-postgres, 0.4 s
replay the SQL dump on top	"Imported DB dump docmost-postgres.sql", 2 s

3 pages planted through docmost's own API → destroyed (count(*) FROM pages = 0, no soft-deleted rows either) → 3 pages back, same ids. The discriminator went ORIGINAL-VALUE-A → DESTROYED-VALUE-C → ORIGINAL-VALUE-A.

Repeated on bookstack (MariaDB 12.3): all five legs, 25 seconds, 2 volumes replayed including a 161 MB MariaDB data directory, the planted book back with its accented name byte-identical.

Accented round-trip, byte-for-byte: c3817276c3ad7a74c5b172c5912074c3bc6bc3b67266c3ba72c3b367c3a970203220e28094205233353662 ("Árvíztűrő tükörfúrógép 2 — R356b", docmost) and C3817276C3AD7A74C5B172C591206BC3B66E79766573706F6C6320E28094205233353662 ("Árvíztűrő könyvespolc — R356b", bookstack). Both identical before and after. Non-ASCII never crossed a shell chain: strings were embedded in python files inside the guest, or passed with --data-urlencode "name@file".

A flaw in this drill's own method, recorded rather than hidden. The FIRST accented title was planted through a shell argument chain that double-escaped it, so it was stored as the literal ASCII text Árv… rather than as accented bytes. It was caught by reading the stored value back as hex — the same discipline that makes the rest of these numbers worth anything. The pages round-tripped exactly either way, so Q1's answer stands, but the accented coverage did not, and was re-measured in Phase 1b. The faulty page is deliberately left in place as the record.

Q2 — Which leg returned the data? The SQL dump.

A three-way discriminator was built so the answer could not be ambiguous:

holder	value
live database before the restore	LIVE-VALUE-C3 — if it survived, neither leg wrote
the volume tar (52 MB Postgres data dir)	ORIGINAL-VALUE-A
the SQL dump, altered in the scratch only	ALTERED-VALUE-B

Result: ALTERED-VALUE-B. The dump won. The ordering the code comment asserts — "volumes FIRST, database after, so a logical .sql dump still wins over whatever copy of the same database a volume tar happens to contain" (offbox_reconstitute.go, the R-354 block) — holds in practice on the off-site path. It was an intention; it is now an observation.

Mutation asserted: dump sha256 c5414f24… → 834c32b1…, zero occurrences of the original left, one of the altered. Evidence 13.

The fence held. Only the prepared scratch was mutated. Proved afterwards by re-preparing a fresh scratch from the same snapshot: sha256 back to c5414f24…, reading ORIGINAL-VALUE-A. Evidence 14.

This confirms an existing claim on a new path rather than establishing a new one. R-164 already records "the dump is authoritative and replayed after the tar so it WINS (F17)" — but cites internal/backup/restore_unit.go:262-266, the local restore. Q2 extends that to offbox_reconstitute.go, the off-site path, which had never been exercised for this app class.

Q3 — Does a failure in the database leg tell the truth? Partly. Two defects.

Measured by truncating the dump in the scratch mid-statement, on both engines.

	Postgres (docmost)	MariaDB (bookstack)
customer sees a failure	yes — alert alert-error, "sikertelen"	yes
message names the undo copy	yes, by filename	yes
app afterwards	crash-loops — visibly broken	health=healthy, running=true, restarts=0
database afterwards	emptied — 43 tables, 0 rows in pages/users/spaces	partially applied — book and users intact, migrations wiped to 0 rows
undo copy valid?	yes — proven by applying it	yes — proven by applying it
product can apply the undo copy	NO	NO

3. Hypotheses — what fired and what did not

• H1 (schema collision). DID NOT FIRE. The dump replayed cleanly onto a freshly-replaced Postgres data directory in 2 s under ON_ERROR_STOP=1. This is the notable negative: the two legs do not collide in practice for this app class.
• H2 (replayed data directory will not start). DID NOT FIRE. Both engines started healthy on a data directory that had just been replaced wholesale from a tar.
• H3 (MariaDB and Postgres do not fail the same way). FIRED — but not in the predicted shape. The prediction was a quiet partial success with no error. What actually happens is a visible error and a silently inconsistent database behind a healthy-looking app. See R-380.
• H4 (credentials). DID NOT FIRE. ImportDump discovered dbUser=docmost/dbName=docmost from the live container and they matched the restored directory, exactly as predicted — secrets are not regenerated on reconstitution.

4. Findings filed

id	severity	what
R-379	HIGH	The undo copy is valid, is named to the customer, and no product action can apply it
R-380	HIGH	A failed MariaDB replay leaves a partially-applied database behind a healthy app
R-381	MEDIUM	The failure message pastes raw engine stderr — including customer database rows — into the Hungarian customer surface
R-382	LOW	The reconstitution's summary log line omits the volume count it already has

An independent reproduction of an existing row. After the first reconstitution, docmost's db-dumps/ held only pre-restore-* files — the app's own docmost-postgres.sql was gone. That is R-361 reproducing on a second app, unprompted. Not re-filed; noted here as corroboration.

5. Observations — noticed, recorded, NOT acted on

• A newly deployed app is not in the off-site set. docmost and bookstack were both absent from app_backup entirely until switched on by hand. This is plausibly deliberate (the switch is the customer's), and it is not filed as a defect — but the consequence is that an app can be deployed, run, and never leave the box, while the nightly run reports "backup OK". Stated so the next reader can decide whether the default is right.
• restic check was run by hand and passed — no errors were found, 29 snapshots. Nothing in the product runs it (R-359, already filed). Reproducing the invocation took three attempts because the key file is ssh_key, not id_offbox as the arg-builder's variable naming suggests.
• The -db container-suffix attribution is correct here. bookstack-db's dump landed in bookstack's own unit as bookstack-mariadb.sql — the R-355 failure shape does not reproduce, because v0.218.0's compose-project attribution handles it.

6. Phases run

phase	status
1 — Postgres, does it complete	run
1b — accented round-trip, re-measured after a method flaw	run
2 — which leg won	run
3 — failure path, Postgres	run
4 — MariaDB: Phase 1 and Phase 3 equivalents	run

Nothing was dropped.

7. Evidence

evidence/, files 00–31, copied off demo-hp at the end of each phase. Nothing was reverted or torn down mid-run, so no intermediate teardown could have taken any of it.

8. Teardown — three layers

1. Apps. docmost and bookstack were deployed by this drill and are RETAINED, with their planted data. Reason: the planted data is the evidence for every claim above, and they are the only deployed members of the 10-app class on the box, so the next drill in this area starts from a real subject instead of rebuilding one. Both are healthy and sane at the end (docmost restored via its undo copy; bookstack's migrations table restored 0 → 102 rows the same way).
2. Storage. No pvesm "before" snapshot was taken — stated plainly rather than reconstructed. Measured directly instead: the two apps' volumes total ~233 MB (docmost 67 MB + 228 KB + 4 KB, bookstack 166 MB + 224 KB) inside guest 9201, which sits at 16 % of 69 GB. Host local-lvm 34.81 %, local 14.27 %.
3. Hub-side record. None was created. This drill created no customer and no appliance record; it deployed two apps inside an existing guest of an existing customer. Nothing to dispose of.

9. Store integrity at the end

restic check against the live repository: no errors were found, 29 snapshots, exclusive lock taken and released. Evidence 30.