SPIKE: an upgrade test that runs again — and a real defect in our own bookstack template

R-449. Until today one app upgrade out of 53 had ever been measured, by hand, and
the whole update arc was designed against that single data point.

C3 first: the negative control, whose TO image exits immediately, came back
failed. That is what makes the greens mean anything, and it cost 556s because a
negative is only honest if it waits out the full settle window.

Seven edges, three apps. All five real catalog upgrades kept the customer's data.

The finding that changes an assumption the arc was carrying: whether an upgrade
can be UNDONE is a property of the individual APP, not of upgrades. Docmost
refuses - 'corrupted migrations: previously executed migration
20260213T085259-notifications is missing' - and privatebin does not. That
reproduces the Nextcloud result on a second app by a DIFFERENT mechanism, so the
struck word 'rollback' now rests on two measurements instead of one.

The finding nobody was looking for, R-459: our own bookstack template moves
MariaDB across a major and sets no MARIADB_* env at all, so the engine logs that
the datadir upgrade it requires is being skipped, and serves anyway. The cause is
assigned rather than guessed - the app half alone produces no upgrade line, both
edges that move the engine produce it - which is exactly what decomposing E3 into
E3a and E3b was for. It also explains why E3's abort looked like it worked: the
datadir was never converted. Whether that ever breaks is NOT established, and the
row says so.

Also opened: R-460 (bookstack's file half cannot be seeded headlessly), R-461
(target-selection.md names a venue that does not exist and fences a VM that is
gone), R-462 (the widening, costed with this run's real numbers - and the cost is
dominated by fixtures, which do not amortise).

Teardown all three layers, hub checked rather than asserted. local-lvm read 30.50
percent before and after. The capability map was deliberately NOT edited: this
measured apps, not the product.

documentation/architecture/09-update-architecture.md

@@ -144,8 +144,31 @@ with a positive control proving the data is intact, only unreachable by the old
 | **ABORT** | before anything migrated | stop, put the old image back, the app runs again |
 | **RESTORE FROM A COPY** | after a migration ran | the data restore is the whole remedy |
 
-There is no third. And per §5 below, a restore's image-level undo currently has a ≤15-minute
-half-life because the syncer overwrites it (**R-441**).
+There is no third.
+
+### 4.1 MEASURED 2026-09-06 — and the abort turns out to be a property of the APP, not of upgrades
+
+`SPIKE-upgrade-test-2026-09-06.md` upgraded three real apps with real data in them and then attempted
+the abort on each. **All five real catalog upgrades kept the customer's data.** The abort did not
+behave the same way twice:
+
+| app | abort | why |
+|---|---|---|
+| **docmost** `0.25.3`→`0.95.0` | **REFUSES** | the old code finds migration ledger entries it does not know: *"corrupted migrations: previously executed migration 20260213T085259-notifications is missing"*, then *"Failed to run database migration. Exiting program."* |
+| **privatebin** `1.7.5`→`2.0.5` | **works** | file-backed, no database, no schema — a major version moves no data |
+| **bookstack** app+engine | **works, misleadingly** | only because the MariaDB datadir upgrade was skipped and never happened — see R-459 |
+
+**This puts TWO independent measurements behind the ruling above, by two unrelated mechanisms:**
+Nextcloud refused on an explicit version comparison; docmost refuses on its migration ledger. The word
+"rollback" was already struck; it is now struck on evidence rather than on one case.
+
+**And it adds a distinction this document did not have: there is no single answer to "can this update
+be undone". There are apps where it can and apps where it cannot, and the only way to know which is to
+MEASURE THAT APP.** Any design that assumes one answer for all 53 is designing against a fact that was
+checked and is false.
+
+Per §5 below, a restore's image-level undo used to have a ≤15-minute half-life because the syncer
+overwrote it (**R-441**) — closed in v0.235.0.
 
 ---
 
@@ -250,10 +273,32 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
 | **2** | **One badge says whether the app is current** — „Naprakész" / „Frissítés elérhető — N napja", from `catalog_since`. No version number. | **SHIPPED, controller v0.233.0 + catalog `69761cf` (2026-09-02)** |
 | **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
 | **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | OPEN — R-448 |
-| **5** | **An upgrade test** — prove a real one-major upgrade end to end, including the abort path. | OPEN — R-449 |
+| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
 | **6** | **A version sequence** — updates automatic within a major, a human across one; **an engine change gets its own edge.** | OPEN — R-450 |
 | **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451 |
 
+### The verdict record — the contract Slice 6 carries
+
+Decided here rather than invented twice. The harness writes one of these per edge, beside its
+evidence; Slice 6 puts the same shape in the catalog.
+
+```json
+{"harness_version": 1, "app": "bookstack",
+ "from": {"bookstack": "…:25.02.2", "bookstack-db": "mariadb:11.6"},
+ "to":   {"bookstack": "…:26.05.2", "bookstack-db": "mariadb:12.3"},
+ "verdict": "proven | failed | inconclusive",
+ "seed_read_before": true, "seed_read_after": true, "healthy_after": true,
+ "migration_observed": "verbatim log line, or null",
+ "abort": "starts-and-serves | refuses | starts-data-gone | not-attempted",
+ "abort_detail": "the refusal quoted verbatim, or null",
+ "duration_s": 0, "measured_at": "RFC3339", "evidence": "relative path"}
+```
+
+**`inconclusive` is a first-class verdict and must never be collapsed into `failed`.** "We could not
+measure it" and "it does not work" are different facts, and only one of them is about the app.
+**`migration_observed` is a quoted line, never an inference from timing** — the value of both the
+Nextcloud and the docmost findings was the exact sentence the app printed.
+
 **The rule slice 6 inherits, recorded now while it is cheap:** an engine change gets its own edge,
 never bundled with an app version bump. `bookstack` moved the application *and* MariaDB 11.6 → 12.3 in
 one commit (`0b73e5e`); that is two migrations behind one edge, and an unreadable failure when it
@@ -353,7 +398,13 @@ Version strings stay in the logs, the API and the hub.
 6. **The Update button is still unguarded.** It takes no backup, has no rollback, and can still
    attempt a multi-major jump the app will refuse (R-40). **Slice 3 did not change that and must not
    be read as having done so** — the precondition is slice 4 (R-448).
-7. **The hub does not record image tags at all.** Its report's container payload carries name, state,
+7. **An engine major can be applied without its datadir upgrade, and nothing notices.** Measured
+   2026-09-06: the catalog's own bookstack transition starts MariaDB 12.3 on an 11.6 datadir, and the
+   image logs that the required upgrade was **skipped** because the template sets no
+   `MARIADB_AUTO_UPGRADE`. The app serves. Whether that ever breaks is **not** established. **R-459.**
+8. **Only three of 53 apps have ever had an upgrade measured**, and one of them (bookstack) can only
+   be half-proven headlessly (**R-460**). The widening is **R-462**, costed with real numbers.
+9. **The hub does not record image tags at all.** Its report's container payload carries name, state,
    CPU and memory, and no image field (spike §5). So the fleet view of §6 slice 7 needs a hub-side
    change; it is not derivable from what is already reported.