R-379/R-380 docs: the failure ladder, the drill record, register housekeeping
gates / gates (push) Successful in 17s

07-backup-architecture.md 6.3 gains a dated [DESIGN] paragraph on replay ->
rollback -> hold, including why no engine flag closes it: --single-transaction
makes Postgres atomic, MariaDB DDL is not transactional, so the rollback is the
fix and the flag is a belt.

Drill record for the live walk, including the TWO defects the walk found in the
fix itself (a rollback into a re-created container; an operator route that
cleared the file while the running controller kept refusing) and the ONE
red-proof that PASSED, which is reported rather than omitted.

R-379..R-382 compressed into CLOSED-ITEMS.md. OPEN-ITEMS 330683 -> 325236 bytes.

STATUS.md restates the outcome and names the next operator step.
This commit is contained in:
2026-08-22 18:31:23 +02:00
parent 4e488321bf
commit a8caa0fdde
23 changed files with 744 additions and 75 deletions
+58 -61
View File
@@ -1,80 +1,77 @@
# REPORT — DRILL R-356b: the off-site restore for a driveless app that HAS a database (2026-08-22)
# REPORT — R-379/R-380/R-381/R-382: the undo copy goes back (2026-08-22)
**A drill, not an implementation.** No production code was written, no version bumped, no CHANGELOG
entry made. The deliverables are a findings document, four register rows and a capability-map update.
Companion to `felhom-controller` **v0.220.0 → v0.220.1 → v0.220.2**. Full record:
`documentation/audits/DRILL-r379-rollback-2026-08-22/`.
Full record: `documentation/audits/DRILL-r356b-driveless-db-restore-2026-08-22/`
## What shipped
## What was measured
**R-379 and R-380 were one failure with one fix.** Both ended with a half-restored database; the only
difference was whether it looked broken. When the replay fails, the product now re-applies the
customer's own pre-restore copy — the same `ImportDump` call a person ran by hand yesterday to recover
both apps — and the app comes back with a message saying **both** that the restore failed and that the
data is as it was.
Ten of the forty driveless apps carry a database. **I re-measured that count myself and got 10** — the
same ten the runbook names. For those ten, restoring is a five-leg operation that, until this week,
never ran at all: R-356 refused before any of it started.
**The whole undo set, matched on the run's own stamp**, never on the `pre-restore-` prefix and never
just the first file. **When the rollback also fails the app is held stopped** — the operator's ruling —
with every start path refusing it, the app-stop marker ended so nothing auto-restarts it, and the row
red rather than green.
Both engines were walked end to end on `demo-hp`: `docmost` (Postgres 16) and `bookstack`
(MariaDB 12.3), each deployed for this drill, planted through the app's **own** interface, destroyed
for real, and restored through the exact endpoint the UI's button posts to.
**R-381:** the failure message stopped pasting engine output (407→257 bytes on Postgres; the MariaDB
one had been 615 bytes with rows out of the customer's own database). The full text now reaches the
operator log, which never had it.
**R-382:** the summary log prints the volume count it already held.
**Also:** undo copies resolve to their own app and are capped at 3.
## The three answers
## Documents updated here
**Q1 — does it complete? YES.** All five legs ran in order and all succeeded — 32 s for Postgres,
25 s for MariaDB. Data back, apps healthy, accented names byte-identical in both directions.
- `documentation/architecture/07-backup-architecture.md` §6.3 — a dated **[DESIGN]** paragraph on the
failure ladder **replay → rollback → hold**, and why an engine flag does not close it.
- `STATUS.md` — the outcome in plain words; the deciding section says what happens if nothing is done.
- `documentation/backlog/` — R-379…R-382 compressed into `CLOSED-ITEMS.md`, each keeping its title,
shipping version, evidence path and every sentence that states a rule. Full text:
`git show 4e488321bfd1:documentation/backlog/OPEN-ITEMS.md`.
**Q2 — which leg returned the data? The SQL dump.** A three-way discriminator (volume tar
`ORIGINAL-VALUE-A`, altered dump `ALTERED-VALUE-B`, live `LIVE-VALUE-C3`) returned **`ALTERED-VALUE-B`**.
The ordering the code comment asserts holds in practice. This **confirms R-164's F17 claim on a second
path** — R-164 cites `restore_unit.go`, the local restore; this measures `offbox_reconstitute.go`.
The mutation was applied to the prepared scratch only, and the store was proved unmutated afterwards
by re-preparing a fresh scratch (sha256 back to `c5414f24…`).
**Register size:** `OPEN-ITEMS.md` **330 683 → 325 236 bytes**; `CLOSED-ITEMS.md` **63 507 → 66 777**.
**Q3 — does a failure tell the truth? Partly.** The customer does see a failure and the undo copy is
named. But two things are wrong, and they are the drill's findings.
## The live walk found two defects in the fix itself
## Findings filed — R-379 … R-382
**Both are recorded because the walk, not the tests, caught them.**
- **R-379 (HIGH)** — the undo copy is valid, is named, and **nothing in the product can apply it**.
Proven by applying it by hand on both engines and getting the exact prior state back.
`pre-restore-` files are deliberately skipped at three code sites; the filename appears only inside
an error string.
- **R-380 (HIGH)** — a failed **MariaDB** replay leaves a partially-applied database behind an app
reporting `health=healthy, running=true, restarts=0`. `bookstack`'s schema-version ledger was wiped
to 0 rows while its user data stayed intact and the dashboard said fine. Postgres, by contrast,
fails visibly (crash-loop). H3 fired — but not in its predicted shape: the prediction was a *quiet
success*; what happens is a loud error and a silent inconsistency.
- **R-381 (MEDIUM)** — the failure message pastes raw engine stderr into the Hungarian customer
surface: 407 bytes for Postgres, **615 for MariaDB, whose middle is an `INSERT INTO migrations
VALUES (…)` listing — actual table rows shown to the customer.**
- **R-382 (LOW)** — the reconstitution's summary log omits the volume count it already has. The
customer-facing flash names the volumes; the operator log does not.
1. **The rollback used a dead container (fixed v0.220.1).** The DB-only start re-creates the DB
container, so the id captured at dump time is dead by rollback time. Measured: captured
`9adbc14f9af6`, re-created `309795897b82`, rollback timed out after 30 s — **the app was held for an
infrastructure reason while its data was recoverable.** No unit test could see it: they all inject
the import seam and never look at container identity.
2. **The operator route did not take effect (fixed v0.220.2).** `--clear-restore-hold` runs as a second
process; it cleared the file and the running controller went on refusing. Found by using it.
**Register: 325 236 bytes before, 330 683 after.** Ceiling was R-378; next free id is now R-383.
## A red-proof that PASSED
## Also recorded
Of nine mutations, **one did not fail its test** and is reported rather than omitted: the R-381
behavioural test injected below `ImportDump`, so a leak reintroduced inside `ImportDump` was invisible
to it. A guard now sits at that layer and the mutation convicts.
- **R-361 reproduced independently** on a second app: after the first reconstitution docmost's
`db-dumps/` held only `pre-restore-*` files. Not re-filed — noted as corroboration.
- **`restic check` passed** at the end: `no errors were found`, 29 snapshots.
- **The `-db` suffix attribution is correct** for `bookstack-db` — the R-355 shape does not reproduce.
- **Observed, not filed:** a newly deployed app is absent from the off-site set until switched on by
hand. Plausibly deliberate; the consequence is stated so the default can be judged.
- **A flaw in the drill's own method, recorded rather than hidden:** the first accented title was
double-escaped by a shell chain and stored as literal ASCII. Caught by reading the stored bytes back
as hex, and re-measured properly in Phase 1b.
## What did not reproduce
## Capability map
The undo copies rendering as app rows on the customer's backup page. The live page was read **before**
any change: zero `pre-restore` strings while four such files sat on disk, with a positive control
showing 8 real rows. The phantom name was real as a map **key**, never a row. Fixed as a naming defect;
their visibility is unchanged and deliberate.
The 2026-08-21 narrowing of *"A customer's file survives a machine rebuild and comes back"* is now
history: both defects it named (R-354, R-356) are closed and proven. The row records what is now
walked — including this drill — and states plainly what is still **not** claimed: the success path is
proven, the recovery-from-a-bad-restore path is not.
## The golden was baked in this session, and why
## Teardown, three layers
The `golden-currency` gate refused this docs push: 0.220.2 was released with no golden. **That block
is not circular** — a golden needs the controller image, which was already pushed, not this commit —
so the gate was satisfied by doing the work it asked for rather than bypassed with `--no-verify`.
**No push in this session used `--no-verify`.**
1. `docmost` and `bookstack` were deployed by this drill and are **RETAINED** with their planted data —
it is the evidence, and they are the only deployed members of this app class on the box. Both left
healthy and sane.
2. No `pvesm` "before" snapshot was taken — **said plainly rather than reconstructed.** Measured
directly: ~233 MB of volumes inside guest 9201 (16 % of 69 GB used).
3. **No hub-side record was created.** No customer, no appliance. Nothing to dispose of.
Golden **0.220.2**, sha256 `cb439418c7005ce01bcb8126bb6385688c2f408c3c4649f6740001bc2c864ed5`,
657 271 965 B, round-trip verified. All five markers hit, both negative controls at zero, both
token-leak greps proved able to convict before their zeros were accepted. Record:
`documentation/tests/golden-0.220.2-2026-08-22/`.
All phases were run. Nothing was dropped.
## Operator follow-up
**Vouch** the golden — Hub → Configuration → Day-0 artifacts, a **three-field** save:
`golden_version` **0.220.2**, `agent_version` **0.130.0**, `min_agent` **0.129.0**. **Then** raise the
floor to **0.220.2**, last, in its own save.