R-379/R-380 docs: the failure ladder, the drill record, register housekeeping
gates / gates (push) Successful in 17s
gates / gates (push) Successful in 17s
07-backup-architecture.md 6.3 gains a dated [DESIGN] paragraph on replay -> rollback -> hold, including why no engine flag closes it: --single-transaction makes Postgres atomic, MariaDB DDL is not transactional, so the rollback is the fix and the flag is a belt. Drill record for the live walk, including the TWO defects the walk found in the fix itself (a rollback into a re-created container; an operator route that cleared the file while the running controller kept refusing) and the ONE red-proof that PASSED, which is reported rather than omitted. R-379..R-382 compressed into CLOSED-ITEMS.md. OPEN-ITEMS 330683 -> 325236 bytes. STATUS.md restates the outcome and names the next operator step.
This commit is contained in:
@@ -1,80 +1,77 @@
|
||||
# REPORT — DRILL R-356b: the off-site restore for a driveless app that HAS a database (2026-08-22)
|
||||
# REPORT — R-379/R-380/R-381/R-382: the undo copy goes back (2026-08-22)
|
||||
|
||||
**A drill, not an implementation.** No production code was written, no version bumped, no CHANGELOG
|
||||
entry made. The deliverables are a findings document, four register rows and a capability-map update.
|
||||
Companion to `felhom-controller` **v0.220.0 → v0.220.1 → v0.220.2**. Full record:
|
||||
`documentation/audits/DRILL-r379-rollback-2026-08-22/`.
|
||||
|
||||
Full record: `documentation/audits/DRILL-r356b-driveless-db-restore-2026-08-22/`
|
||||
## What shipped
|
||||
|
||||
## What was measured
|
||||
**R-379 and R-380 were one failure with one fix.** Both ended with a half-restored database; the only
|
||||
difference was whether it looked broken. When the replay fails, the product now re-applies the
|
||||
customer's own pre-restore copy — the same `ImportDump` call a person ran by hand yesterday to recover
|
||||
both apps — and the app comes back with a message saying **both** that the restore failed and that the
|
||||
data is as it was.
|
||||
|
||||
Ten of the forty driveless apps carry a database. **I re-measured that count myself and got 10** — the
|
||||
same ten the runbook names. For those ten, restoring is a five-leg operation that, until this week,
|
||||
never ran at all: R-356 refused before any of it started.
|
||||
**The whole undo set, matched on the run's own stamp**, never on the `pre-restore-` prefix and never
|
||||
just the first file. **When the rollback also fails the app is held stopped** — the operator's ruling —
|
||||
with every start path refusing it, the app-stop marker ended so nothing auto-restarts it, and the row
|
||||
red rather than green.
|
||||
|
||||
Both engines were walked end to end on `demo-hp`: `docmost` (Postgres 16) and `bookstack`
|
||||
(MariaDB 12.3), each deployed for this drill, planted through the app's **own** interface, destroyed
|
||||
for real, and restored through the exact endpoint the UI's button posts to.
|
||||
**R-381:** the failure message stopped pasting engine output (407→257 bytes on Postgres; the MariaDB
|
||||
one had been 615 bytes with rows out of the customer's own database). The full text now reaches the
|
||||
operator log, which never had it.
|
||||
**R-382:** the summary log prints the volume count it already held.
|
||||
**Also:** undo copies resolve to their own app and are capped at 3.
|
||||
|
||||
## The three answers
|
||||
## Documents updated here
|
||||
|
||||
**Q1 — does it complete? YES.** All five legs ran in order and all succeeded — 32 s for Postgres,
|
||||
25 s for MariaDB. Data back, apps healthy, accented names byte-identical in both directions.
|
||||
- `documentation/architecture/07-backup-architecture.md` §6.3 — a dated **[DESIGN]** paragraph on the
|
||||
failure ladder **replay → rollback → hold**, and why an engine flag does not close it.
|
||||
- `STATUS.md` — the outcome in plain words; the deciding section says what happens if nothing is done.
|
||||
- `documentation/backlog/` — R-379…R-382 compressed into `CLOSED-ITEMS.md`, each keeping its title,
|
||||
shipping version, evidence path and every sentence that states a rule. Full text:
|
||||
`git show 4e488321bfd1:documentation/backlog/OPEN-ITEMS.md`.
|
||||
|
||||
**Q2 — which leg returned the data? The SQL dump.** A three-way discriminator (volume tar
|
||||
`ORIGINAL-VALUE-A`, altered dump `ALTERED-VALUE-B`, live `LIVE-VALUE-C3`) returned **`ALTERED-VALUE-B`**.
|
||||
The ordering the code comment asserts holds in practice. This **confirms R-164's F17 claim on a second
|
||||
path** — R-164 cites `restore_unit.go`, the local restore; this measures `offbox_reconstitute.go`.
|
||||
The mutation was applied to the prepared scratch only, and the store was proved unmutated afterwards
|
||||
by re-preparing a fresh scratch (sha256 back to `c5414f24…`).
|
||||
**Register size:** `OPEN-ITEMS.md` **330 683 → 325 236 bytes**; `CLOSED-ITEMS.md` **63 507 → 66 777**.
|
||||
|
||||
**Q3 — does a failure tell the truth? Partly.** The customer does see a failure and the undo copy is
|
||||
named. But two things are wrong, and they are the drill's findings.
|
||||
## The live walk found two defects in the fix itself
|
||||
|
||||
## Findings filed — R-379 … R-382
|
||||
**Both are recorded because the walk, not the tests, caught them.**
|
||||
|
||||
- **R-379 (HIGH)** — the undo copy is valid, is named, and **nothing in the product can apply it**.
|
||||
Proven by applying it by hand on both engines and getting the exact prior state back.
|
||||
`pre-restore-` files are deliberately skipped at three code sites; the filename appears only inside
|
||||
an error string.
|
||||
- **R-380 (HIGH)** — a failed **MariaDB** replay leaves a partially-applied database behind an app
|
||||
reporting `health=healthy, running=true, restarts=0`. `bookstack`'s schema-version ledger was wiped
|
||||
to 0 rows while its user data stayed intact and the dashboard said fine. Postgres, by contrast,
|
||||
fails visibly (crash-loop). H3 fired — but not in its predicted shape: the prediction was a *quiet
|
||||
success*; what happens is a loud error and a silent inconsistency.
|
||||
- **R-381 (MEDIUM)** — the failure message pastes raw engine stderr into the Hungarian customer
|
||||
surface: 407 bytes for Postgres, **615 for MariaDB, whose middle is an `INSERT INTO migrations
|
||||
VALUES (…)` listing — actual table rows shown to the customer.**
|
||||
- **R-382 (LOW)** — the reconstitution's summary log omits the volume count it already has. The
|
||||
customer-facing flash names the volumes; the operator log does not.
|
||||
1. **The rollback used a dead container (fixed v0.220.1).** The DB-only start re-creates the DB
|
||||
container, so the id captured at dump time is dead by rollback time. Measured: captured
|
||||
`9adbc14f9af6`, re-created `309795897b82`, rollback timed out after 30 s — **the app was held for an
|
||||
infrastructure reason while its data was recoverable.** No unit test could see it: they all inject
|
||||
the import seam and never look at container identity.
|
||||
2. **The operator route did not take effect (fixed v0.220.2).** `--clear-restore-hold` runs as a second
|
||||
process; it cleared the file and the running controller went on refusing. Found by using it.
|
||||
|
||||
**Register: 325 236 bytes before, 330 683 after.** Ceiling was R-378; next free id is now R-383.
|
||||
## A red-proof that PASSED
|
||||
|
||||
## Also recorded
|
||||
Of nine mutations, **one did not fail its test** and is reported rather than omitted: the R-381
|
||||
behavioural test injected below `ImportDump`, so a leak reintroduced inside `ImportDump` was invisible
|
||||
to it. A guard now sits at that layer and the mutation convicts.
|
||||
|
||||
- **R-361 reproduced independently** on a second app: after the first reconstitution docmost's
|
||||
`db-dumps/` held only `pre-restore-*` files. Not re-filed — noted as corroboration.
|
||||
- **`restic check` passed** at the end: `no errors were found`, 29 snapshots.
|
||||
- **The `-db` suffix attribution is correct** for `bookstack-db` — the R-355 shape does not reproduce.
|
||||
- **Observed, not filed:** a newly deployed app is absent from the off-site set until switched on by
|
||||
hand. Plausibly deliberate; the consequence is stated so the default can be judged.
|
||||
- **A flaw in the drill's own method, recorded rather than hidden:** the first accented title was
|
||||
double-escaped by a shell chain and stored as literal ASCII. Caught by reading the stored bytes back
|
||||
as hex, and re-measured properly in Phase 1b.
|
||||
## What did not reproduce
|
||||
|
||||
## Capability map
|
||||
The undo copies rendering as app rows on the customer's backup page. The live page was read **before**
|
||||
any change: zero `pre-restore` strings while four such files sat on disk, with a positive control
|
||||
showing 8 real rows. The phantom name was real as a map **key**, never a row. Fixed as a naming defect;
|
||||
their visibility is unchanged and deliberate.
|
||||
|
||||
The 2026-08-21 narrowing of *"A customer's file survives a machine rebuild and comes back"* is now
|
||||
history: both defects it named (R-354, R-356) are closed and proven. The row records what is now
|
||||
walked — including this drill — and states plainly what is still **not** claimed: the success path is
|
||||
proven, the recovery-from-a-bad-restore path is not.
|
||||
## The golden was baked in this session, and why
|
||||
|
||||
## Teardown, three layers
|
||||
The `golden-currency` gate refused this docs push: 0.220.2 was released with no golden. **That block
|
||||
is not circular** — a golden needs the controller image, which was already pushed, not this commit —
|
||||
so the gate was satisfied by doing the work it asked for rather than bypassed with `--no-verify`.
|
||||
**No push in this session used `--no-verify`.**
|
||||
|
||||
1. `docmost` and `bookstack` were deployed by this drill and are **RETAINED** with their planted data —
|
||||
it is the evidence, and they are the only deployed members of this app class on the box. Both left
|
||||
healthy and sane.
|
||||
2. No `pvesm` "before" snapshot was taken — **said plainly rather than reconstructed.** Measured
|
||||
directly: ~233 MB of volumes inside guest 9201 (16 % of 69 GB used).
|
||||
3. **No hub-side record was created.** No customer, no appliance. Nothing to dispose of.
|
||||
Golden **0.220.2**, sha256 `cb439418c7005ce01bcb8126bb6385688c2f408c3c4649f6740001bc2c864ed5`,
|
||||
657 271 965 B, round-trip verified. All five markers hit, both negative controls at zero, both
|
||||
token-leak greps proved able to convict before their zeros were accepted. Record:
|
||||
`documentation/tests/golden-0.220.2-2026-08-22/`.
|
||||
|
||||
All phases were run. Nothing was dropped.
|
||||
## Operator follow-up
|
||||
|
||||
**Vouch** the golden — Hub → Configuration → Day-0 artifacts, a **three-field** save:
|
||||
`golden_version` **0.220.2**, `agent_version` **0.130.0**, `min_agent` **0.129.0**. **Then** raise the
|
||||
floor to **0.220.2**, last, in its own save.
|
||||
|
||||
Reference in New Issue
Block a user