# SPIKE — does a real app upgrade keep the customer's data, and can it be undone? (2026-09-06) > **THE ANSWER, IN ONE SENTENCE: on all five real catalog upgrades measured, the customer's data > survived — and whether the upgrade can be UNDONE is not a property of upgrades at all, it is a > property of the individual app, which is why it has to be measured per app rather than reasoned > about.** > > **AND THE FINDING NOBODY WAS LOOKING FOR, which is a defect in our own catalog:** this repo's > bookstack template moves MariaDB **11.6 → 12.3**, and MariaDB 12.3 comes up, says in its own words > that a datadir upgrade is **required**, **skips it**, and serves anyway — because the template sets > no `MARIADB_AUTO_UPGRADE`. The app works. The engine is running on a datadir it itself calls > un-upgraded. **R-459.** **Class: spike.** No controller code, no product change, no customer's box. Everything ran inside a throwaway LXC destroyed at the end (§7). **R-449** asked for "an upgrade test that runs again"; the harness is `app-catalog-felhom.eu/scripts/upgrade-test.py` and it is now a maintained script, not an audit artifact. --- ## 1. C3 first — the harness can say no **Everything below is conditional on this, so it is reported first.** C3 is an edge whose TO image is `alpine:3.20` — a real image that pulls cleanly and exits immediately. ``` verdict: failed seed_read_before: true seed_read_after: false healthy_after: false TO settled=False in 421.1s :: {"privatebin": {"status": "restarting", "health": "unhealthy"}} ``` **C3 came back RED.** A harness that cannot fail a known-broken upgrade proves nothing with its greens. It also cost the most wall-clock of any edge (556 s), because a negative is only honest if it waits out the full settle window. **C1 (the seed reads back BEFORE the upgrade) passed on every edge**, including C3 — a fixture that cannot prove itself first proves nothing after. ## 2. The verdict table | edge | app | from → to | verdict | data after | **abort** | TO settle | total | |---|---|---|---|---|---|---|---| | **C3** | privatebin | `2.0.5` → **`alpine:3.20`** | **failed** | no | starts-and-serves | 421.1 s | 556.0 s | | **C2** | privatebin | `2.0.5` → `2.0.5` (no-op) | proven | yes | starts-and-serves | 0.1 s | 6.4 s | | **E1** | privatebin | `1.7.5` → `2.0.5` (major) | **proven** | yes | **starts-and-serves** | 5.2 s | 21.5 s | | **E2** | docmost | `0.25.3` → `0.95.0` | **proven** | yes | **REFUSES** | 10.7 s | 305.1 s | | **E3** | bookstack | app `25.02.2`→`26.05.2` **+** mariadb `11.6`→`12.3` | **proven** | yes | starts-and-serves¹ | 15.7 s | 77.5 s | | **E3a** | bookstack | app only, engine held at `11.6` | **proven** | yes | starts-and-serves | 15.7 s | 71.8 s | | **E3b** | bookstack | engine only `11.6`→`12.3`, app held | **proven** | yes | starts-and-serves¹ | 0.2 s | 48.8 s | ¹ **and §4 is why that is not the good news it looks like.** ## 3. The abort is a property of the APP, not of upgrades **E2, docmost — the abort REFUSES.** The old image will not start on the migrated database. Verbatim, from the app's own log: ``` {"level":"error","context":"DatabaseMigrationService", "msg":"corrupted migrations: previously executed migration 20260213T085259-notifications is missing"} {"level":"error","context":"DatabaseMigrationService","msg":"Failed to run database migration. Exiting program."} ``` The container then crash-loops: `{"docmost": {"status": "restarting", "health": "unhealthy", "exit": 1}}` while `docmost-postgres` and `docmost-redis` stay healthy beside it. **This independently reproduces the Nextcloud finding on a second app — and by a DIFFERENT mechanism.** Nextcloud refused on an explicit version comparison; docmost refuses because its migration ledger contains entries the older code does not know about. Two apps, two unrelated causes, same outcome. **`09-update-architecture.md` §4's ruling that "rollback" is the wrong word is now supported by two independent measurements rather than one.** **E1, privatebin — the abort WORKS**, and the reason is structural: privatebin is file-backed with no database and no schema, so a major version change moves no data. Its migration-line capture is empty, which is a true negative rather than a missed observation. **So the vocabulary needs one more distinction than the arc currently has.** There is no single answer to "can an update be undone". There are apps where it can and apps where it cannot, and **the only way to know which is to measure that app**. This is the finding R-449 existed to produce. ## 4. The finding nobody was looking for — R-459 E3 is the catalog's own bookstack transition, and it moves the **database engine** across a major at the same time as the app. It came back `proven`, and its abort came back `starts-and-serves`. **Both are true and both are misleading, and the decomposition is what showed it.** Verbatim, at the moment MariaDB 12.3 first started on the 11.6 datadir: ``` [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required, but skipped due to $MARIADB_AUTO_UPGRADE setting ``` Confirmed by reading which engine actually served in each edge — not inferred: | edge | engine that served the TO step | upgrade line | |---|---|---| | E3 (app **and** engine) | `12.3.3-MariaDB-ubu2404` | **required, but skipped** | | **E3a (app only)** | `11.6.2-MariaDB-ubu2404` | **none** | | E3b (**engine only**) | `12.3.3-MariaDB-ubu2404` | **required, but skipped** | **E3a is the control that assigns the cause.** The app half alone produces no engine upgrade line at all; both edges that move the engine produce it. **This is exactly what the decomposition was for, and a bundled edge could never have said it.** `templates/bookstack/docker-compose.yml` sets **no `MARIADB_*` environment at all**, so `MARIADB_AUTO_UPGRADE` is unset and the image's entrypoint declines to run `mariadb-upgrade`. **And this is why the abort "worked":** the datadir was never converted, so MariaDB 11.6 could still read it — on the way back it says `MariaDB upgrade not required`. **The reversibility of E3 is a side-effect of an upgrade that did not fully happen.** **What is NOT established, and the row says so:** whether running 12.3 on an unconverted 11.6 datadir ever actually breaks. It did not break here. MariaDB itself calls the upgrade required; we measured that it is skipped, and we did **not** measure a consequence. **Naming the gap is the finding; the consequence is a separate measurement.** ## 5. What each edge cost — the numbers the widening must be costed with **Measured, not estimated**, which is the whole point of §15.5 of the task. | | | |---|---| | wall-clock, successful edges | **6.4 s – 305.1 s**, median **71.8 s** | | wall-clock, the failing edge | **556 s** — a negative costs ~8× a positive, because it must wait out the full settle window | | total for 7 edges | **~18 minutes** of harness time, plus ~35 minutes of build-out and two fixture iterations | | disk, 3 apps / 11 images | **5.07 GB** of images, 6.0 GB guest total | | **naive extrapolation to 53 apps** | **~90 GB of images** and, at the median, ~1 h of harness time for one edge each — **but see the caveat below, which is the real cost** | **The real cost is not the machine, it is the fixture.** Two of the three apps needed a bespoke non-browser seed route, and one of those (bookstack) needed two attempts and a discarded approach (§6). **Harness time scales with apps; fixture time scales with apps too, and it does not amortise.** Costing the widening from the 71.8 s median alone would understate it by an order of magnitude. ## 6. Fixtures — what worked, what did not, and what cannot be done at all Every seed went in through the app's **own** interface. Nothing was written into a volume or a database by hand (R-156). | app | seed route | readback route | notes | |---|---|---|---| | privatebin | its JSON paste API (HTTP POST) | HTTP GET of the same paste | **file-backed, no database — this single seed IS the file half**; there is no database half to seed | | docmost | `POST /api/auth/setup` | `POST /api/auth/login` as the seeded user | the readback deliberately uses the app's own front door, and is version-stable across the 0.25→0.95 API churn | | bookstack | `php artisan bookstack:create-admin` | `php artisan bookstack:reset-mfa --email=…` | **database half only — see below** | **Two dead ends, recorded because they cost time and will cost the next person the same:** 1. **PrivateBin refuses a malformed paste envelope** with `{"status":1,"message":"Invalid data."}`. `ct`, the IV and the salt must all be real base64. The first fixture attempt used placeholder strings and was rejected — which read like an app failure and was a harness bug. 2. **BookStack cannot be verified over HTTP without TLS.** Its `APP_URL` comes from the template as `https://${SUBDOMAIN}.${DOMAIN}`, so it marks its session and XSRF cookies **`secure`**; curl over plain http stores neither, sends neither, and **every login POST returns 419 Page Expired** — which looks exactly like a wrong password. The container serves no TLS. There is no http route to a logged-in session without changing the app's own configuration, which would be measuring a different app. **So bookstack's readback is `artisan` on both sides**, and two things were done to keep that honest: the readback command is a *different* command from the seed and must FIND the record the other one created; and **the fixture runs its own negative control on every call** — it also looks up an email that cannot exist and requires the answer `A user where email=… could not be found.` A readback that had broken into always saying "found" therefore fails instead of passing everything. **One thing genuinely cannot be done headlessly, and it is a gap, not a defect: the FILE half of bookstack.** Seeding an uploaded image or attachment needs an API token that BookStack only mints through a browser. **So for bookstack this run proves the database survived and says nothing about uploaded files.** **R-460.** **A note on the standing "gate on the exit code" rule, because this run had to break it once.** `bookstack:reset-mfa` asks for interactive confirmation, finds no TTY, and exits **1 in both cases** — for a user it found and one it did not. The exit code carries no information there, so the discriminator is the output, required positive AND with the not-found sentence required absent. Stated rather than done quietly. ## 7. Venue and teardown — all three layers Throwaway **LXC 9401 `upgrade-spike`** on `demo-hp`, Debian 13, 4 cores, 6 GB, created for this run. **Two things the runbook says about this venue are now out of date, and were worked around rather than followed blindly:** - **`/mnt/nvme-1tb` does not exist.** The 1 TB NVMe is mounted at **`/mnt/hdd_1`** and is the enrolled user-data drive. A dir storage `scratch-upg` was created **at its root** (the runbook's own rule — a subdirectory fails the agent's `exactMount` check) and removed at teardown. The guest's disk was deliberately kept **off `local-lvm`**, whose over-subscription is the runbook's real warning. - **`drill-r50` (VM 300) no longer exists.** `qm list` returns nothing on this host. The runbook's "do not destroy it" fence currently protects nothing. **R-461.** | layer | before | after | |---|---|---| | **1 — the machine** | guest 9401 created | **destroyed**; `pct list` shows only 9201. `scratch-upg` storage **removed**. Downloaded LXC template deleted. | | **2 — the host** | `local-lvm` **30.50 %**, `local` 19 595 164 KiB | `local-lvm` **30.50 % — unchanged, never touched**; `local` 19 605 084 KiB (+9.7 MB). `pct fstrim 9401` before destroy: **52.8 GiB trimmed** | | **3 — the hub** | 2 enrolled hosts | **2 enrolled hosts, 0 customers created.** Checked, not assumed: `/hosts` lists exactly `demo-felhom-8363b5` and `demo-hp-bb76ea`. **This run created no customer, no appliance and no host record.** | **No felhom-controller was in the path at any point** — raw `docker compose` throughout. The property under test belongs to the app and its images; putting the controller in the path would have confounded the two, which is the choice the persistence sweep made and stated. ## 8. What was NOT measured - **50 of 53 apps.** Three is the sample that decides whether the idea is sound, not a fleet survey. - **Whether an unconverted MariaDB datadir ever breaks** (§4). Measured that the upgrade is skipped; did not measure a consequence. - **The FILE half of bookstack and docmost.** privatebin's seed is a file; the other two prove the database only. - **Any edge with realistic data VOLUME.** Every seed here is one record. An upgrade that migrates 10 GB may behave differently, and the durations above are therefore floors, not estimates. - **Anything through felhom-controller**, deliberately (§7). ## 9. Observations — noticed, documented, not acted on 1. **`docker compose logs` only shows the containers that currently exist**, so the abort — which replaces them — erases the TO step's output from any capture taken afterwards. **The single most important line of this run survived only because it had already been extracted.** The harness now writes `to-full.log` at the TO step. Same class as R-320, one layer down. 2. **A negative edge costs ~8× a positive** because it must wait out the settle window. If the widening runs many edges, that asymmetry dominates the schedule, not the average. 3. **`bookstack:reset-mfa`'s help output disagrees with its argument parsing** in 25.02.2 — the usage line says `[options]` and a positional argument is rejected with *"No arguments expected"*, which is correct but reads as a missing feature. Upstream's problem, recorded because it cost a minute.