SPIKE: an upgrade test that runs again — and a real defect in our own bookstack template

R-449. Until today one app upgrade out of 53 had ever been measured, by hand, and
the whole update arc was designed against that single data point.

C3 first: the negative control, whose TO image exits immediately, came back
failed. That is what makes the greens mean anything, and it cost 556s because a
negative is only honest if it waits out the full settle window.

Seven edges, three apps. All five real catalog upgrades kept the customer's data.

The finding that changes an assumption the arc was carrying: whether an upgrade
can be UNDONE is a property of the individual APP, not of upgrades. Docmost
refuses - 'corrupted migrations: previously executed migration
20260213T085259-notifications is missing' - and privatebin does not. That
reproduces the Nextcloud result on a second app by a DIFFERENT mechanism, so the
struck word 'rollback' now rests on two measurements instead of one.

The finding nobody was looking for, R-459: our own bookstack template moves
MariaDB across a major and sets no MARIADB_* env at all, so the engine logs that
the datadir upgrade it requires is being skipped, and serves anyway. The cause is
assigned rather than guessed - the app half alone produces no upgrade line, both
edges that move the engine produce it - which is exactly what decomposing E3 into
E3a and E3b was for. It also explains why E3's abort looked like it worked: the
datadir was never converted. Whether that ever breaks is NOT established, and the
row says so.

Also opened: R-460 (bookstack's file half cannot be seeded headlessly), R-461
(target-selection.md names a venue that does not exist and fences a VM that is
gone), R-462 (the widening, costed with this run's real numbers - and the cost is
dominated by fixtures, which do not amortise).

Teardown all three layers, hub checked rather than asserted. local-lvm read 30.50
percent before and after. The capability map was deliberately NOT edited: this
measured apps, not the product.

STATUS.md

@@ -1,9 +1,14 @@
 # STATUS — what works, what's broken, what's next
 
-**Updated 2026-09-06 — a restart no longer changes which version an app runs. Fixes still arrive
+**Updated 2026-09-06 (second pass) — I built a machine that upgrades a real app with real data in it
+and then asks the app whether the data is still there. Three apps, five real upgrades: the data
+survived every time. It also found a genuine problem in our own BookStack setup. ONE NEW THING NEEDS
+YOU: item 11 — how wide should I take this?**
+
+**Earlier 2026-09-06 — a restart no longer changes which version an app runs. Fixes still arrive
 every 15 minutes, and a broken app definition still repairs itself. Only the Update button moves a
-version now. Live on the HP (0.235.0). NOTHING IS WAITING ON YOU — and two old items are now closed:
-the Hetzner e-mails are answered, and the Docker Hub login is in place.**
+version now. Live on the HP (0.235.0). Two old items closed: the Hetzner e-mails are answered, and
+the Docker Hub login is in place.**
 
 **Earlier 2026-09-03 — you spotted that OpenGist had no label. You were right, and it was a real gap:
 the label only appeared on apps something had restarted. Fixed and live (0.234.0). Every app on both
@@ -31,7 +36,7 @@ not an evening's work.**
 *This section is allowed to be longer than one screen, and each item says what happens if you do
 nothing.*
 
-1. **Nothing is waiting on you.** Item 4 (the Hetzner e-mails) is answered and is being handled in a separate session. Item 7 — the safety-copy decision — is the one open question, and it is **not urgent any more**: the thing that made it urgent was that a restart could upgrade an app behind your back, and as of today it cannot. Item 10 is new and needs nothing from you. Item 5's alarm mail can now be ignored for good. Otherwise: Both problems the overnight test found are fixed and proven on
+1. **One thing is waiting on you: item 11 (how wide to take the upgrade testing).** Item 4 (the Hetzner e-mails) is answered and is being handled in a separate session. Item 7 — the safety-copy decision — is the one open question, and it is **not urgent any more**: the thing that made it urgent was that a restart could upgrade an app behind your back, and as of today it cannot. Item 10 is new and needs nothing from you. Item 5's alarm mail can now be ignored for good. Otherwise: Both problems the overnight test found are fixed and proven on
    the real machines:
    - the background job that could delete a live restore's lock now waits its turn — and the check
      that finds the next one like it is a test, not a comment, so it cannot come back quietly;
@@ -191,6 +196,41 @@ nothing.*
     version and look unwell when it is fine. **It cannot lose data — the worst case is a false
     alarm.** Freezing that file too would break the „Frissítés elérhető" label, which is a worse trade.
 
+11. **How wide should I take the upgrade testing? This is the one decision from today, and it is
+    about money and time, not about safety.**
+
+    **What I built.** A machine that installs an app, puts real data in through the app's own front
+    door, upgrades it, and then asks the app for the data back. Not "did it start" — the box has
+    already fooled us that way once.
+
+    **I also taught it to fail.** Before believing anything, I pointed it at an upgrade I knew was
+    broken. It came back red. That is why I trust the greens.
+
+    **What it found, on three apps and five real upgrades.**
+    - **The data survived every single time.** That is the good news and it is worth having.
+    - **Whether an upgrade can be UNDONE depends on the app, not on upgrades.** Docmost will not go
+      back — the old version refuses to start on the changed data. PrivateBin goes back fine. **We
+      had been assuming one answer for all 53 apps. There isn't one.**
+    - **And it found a real problem in our own BookStack setup.** Our template moves the database
+      engine to a new major version, and the engine says, in its own words, that the conversion it
+      needs is being *skipped*. It works today. **I did not measure whether it ever breaks**, and I
+      am not going to guess.
+
+    **The decision: how many of the 53 apps do I test?**
+
+    - **Only the apps that keep data in a database (~25).** Today's run showed the undo question only
+      ever bites there. **Cost: roughly a day of my time.**
+    - **All 53.** Complete, and it gives us a list nobody has. **Cost: several days**, and the reason
+      is not the machines — it is that **each app needs its own hand-written way in**, and that work
+      does not get cheaper the more you do. Two of today's three needed one, and one of them took two
+      attempts.
+
+    **My pick: the database ones first.** It answers the question that changes the product, and if it
+    goes well the rest is a decision you can take later with better numbers.
+
+    **If you do nothing:** nothing breaks. The machine is built and committed, so it does not go
+    stale, and the BookStack problem is written down and waiting either way.
+
 8. **`demo-hp`'s network setup does not match our own notes** (R-338) — the machine works, the page is
    wrong, or the other way round. **If you do nothing:** the page keeps misleading the next session,
    as it misled one by an hour.