felhom.eu/documentation/audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md

SPIKE — is a skipped MariaDB upgrade harmless, and what does fixing it cost? (2026-09-06)

THE ANSWER IN ONE SENTENCE: the skipped conversion is STABLE but never self-resolving — the engine says a check is required every single time it starts and will go on saying it forever — and fixing it costs SEVEN SECONDS and does NOT cost the ability to go back, which is the trade this task was commissioned to price and which turns out not to exist.

OUTCOME A, qualified. Not B: converting did not kill the abort. Not C: the conversion succeeded across the multi-major jump.

AND ONE NEW INSTANCE OF THIS PROJECT'S MOST-REPEATED CLASS. After converting and going back, MariaDB's entrypoint prints MariaDB upgrade not required — reassuring, and it is not a statement that the state is sound. Asked properly, the same engine answers FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported! R-464.

Class: spike. No template changed, no controller code, no customer box. A throwaway LXC (9402 on demo-hp), destroyed at the end. Four apps carry MariaDB — bookstack, kimai, nextcloud, romm — and none of them was touched.

---

1. What was already established, and is not re-derived here

R-459 (2026-09-06): on the catalog's own transition 0b73e5e, MariaDB 12.3 starting on an 11.6 datadir logs that the conversion it requires is skipped, because templates/bookstack/docker-compose.yml sets no MARIADB_* env. The cause was assigned by decomposition — the app half alone produces no such line. R-459 explicitly did not establish whether that ever breaks. That is this document.

2. Observable 1 — mariadb_upgrade_info, at all four moments, verbatim

The file in the datadir recording the version that last completed mariadb-upgrade.

moment	state	file contents, verbatim
1	fresh 11.6 datadir	11.6.2-MariaDB (14 bytes)
2	12.3 has started on it and logged the skip	11.6.2-MariaDB — unchanged
3	after 5 restarts of 12.3 on it	11.6.2-MariaDB — still unchanged
4	comparison arm, after a conversion actually ran	12.3.3-MariaDB

The engine serving at moment 2 and 3 was 12.3.3-MariaDB (mariadb --version), so the mismatch is between what is running and what the datadir records — not an artefact of the wrong container.

3. Observable 2 — the engine's own verdict, and its exit-code semantics MEASURED not assumed

The check mode was found in the image rather than guessed: --check-if-upgrade-is-needed.

On the unconverted datadir:

$ mariadb-upgrade --check-if-upgrade-is-needed --user=root --password=$MYSQL_ROOT_PASSWORD
Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!
[exit=0]

After the conversion (comparison arm):

This installation of MariaDB is already upgraded to 12.3.3-MariaDB.
There is no need to run mariadb-upgrade again.
[exit=1]

Those two runs establish the exit-code semantics between them — 0 means the upgrade IS needed, 1 means it is not. That is a measured control rather than a reading of the documentation, and it matters because the polarity is the opposite of the usual convention.

A false start worth recording: run without credentials the command answers ERROR 1045 (28000): Access denied … FATAL ERROR: Upgrade failed with exit 1 — which looks exactly like a verdict and is an authentication failure. Taken at face value it would have produced the wrong answer with the right-looking exit code.

4. Observable 3 — it does not degrade. 5 of 5.

R-459 suggested watching for the Note to become a Warning or an Error. It never did.

restart	settled	readback	mariadb_upgrade_info	engine check	entrypoint line
1 / 5	yes	pass	11.6.2-MariaDB	Check required!	[Note] unchanged
2 / 5	yes	pass	11.6.2-MariaDB	Check required!	[Note] unchanged
3 / 5	yes	pass	11.6.2-MariaDB	Check required!	[Note] unchanged
4 / 5	yes	pass	11.6.2-MariaDB	Check required!	[Note] unchanged
5 / 5	yes	pass	11.6.2-MariaDB	Check required!	[Note] unchanged

5 of 5 restarts, readback passed each time. The severity never escalated and the record never moved.

So the state is STABLE — and permanently incomplete. It does not decay, and it does not heal. "It has not broken" and "it is fine" remain different claims, and §8 says exactly which one this run supports.

5. Observable 4 — the comparison arm, and the trade that turned out not to exist

Same edge, in a scratch copy of the template inside the guest with MARIADB_AUTO_UPGRADE=1. Nothing with MARIADB_ in it was committed.

5.1 The conversion SUCCEEDS — so this is not Outcome C

MariaDB's own guidance is one major at a time and our jump crosses several, so failure was a live possibility. It did not fail. The entrypoint, verbatim and in order:

[Note] [Entrypoint]: Starting temporary server
[Note] [Entrypoint]: Temporary server started.
[Note] [Entrypoint]: Backing up system database to system_mysql_backup_11.6.2-MariaDB.sql.zst
[Note] [Entrypoint]: Backing up complete
[Note] [Entrypoint]: Starting mariadb-upgrade
[Note] [Entrypoint]: Finished mariadb-upgrade
[Note] [Entrypoint]: Stopping temporary server

It takes its own backup first, unasked: system_mysql_backup_11.6.2-MariaDB.sql.zst, 622 437 bytes, left in the datadir.

5.2 What it costs — 7 seconds

Starting mariadb-upgrade → Finished mariadb-upgrade	17:33:24 → 17:33:31 = 7 s
whole TO step, wall clock, with conversion	35.9 s
whole TO step without it (arm 1)	~10.5 s settle
net cost	≈ 25 s of extra startup, once

MEASURED ON A NEARLY EMPTY DATABASE, and that qualification is load-bearing. mariadb-upgrade works over system tables and table metadata rather than row data, so it should scale with the number of tables rather than their size — but this run did not measure that, and 7 s must not be quoted as a fleet figure.

5.3 THE DECISION-RELEVANT FACT: converting does NOT kill the abort

This is what the arm existed for. R-459's sharpest observation was that E3's abort "worked" only because nothing had been converted — so the expectation was that a real conversion would end it.

It did not. With the datadir converted to 12.3.3-MariaDB, putting MariaDB 11.6 back:

abort: starts-and-serves        settled in 10.5 s        readback: PASS

and it survived a further restart (Up 25 seconds (healthy), mariadb_upgrade_info: 12.3.3-MariaDB).

So the trade the operator was to be asked to rule on — a correct engine you cannot go back from, versus a possibly-incorrect one you can — DOES NOT EXIST as stated. You can have both.

5.4 …but the engine says that going back is unsupported, and the entrypoint hides it

R-464, and it is the same class this whole arc keeps meeting. After the abort:

who is asked	what it says
the entrypoint, on every start	[Note] [Entrypoint]: MariaDB upgrade not required
mariadb-upgrade --check-if-upgrade-is-needed	FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!

The entrypoint compares the datadir's recorded version against its own and concludes there is nothing to do. That is true, and it is not a statement that the state is sound. Anyone building a health signal on that line would build it on a sentence that is emitted just as cheerfully for an unsupported downgrade.

So "the abort works" is an OBSERVATION that it started and served, not a claim that the resulting datadir is sound. It was exercised for one restart cycle with one seeded record. Saying more than that would repeat the mistake this document exists to correct.

6. Which outcome the evidence assigns

A	the unconverted datadir is benign	THIS ONE, qualified. Nothing degraded over 5 of 5 restarts. But the engine says a check is required every time, so "benign" holds only in the sense measured: no degradation over restarts, with one seeded record, over minutes.
B	converting works and the abort dies with it	NO. Converting works; the abort still starts and serves (§5.3). The trade does not exist.
C	converting fails, the jump is too big	NO. It succeeded, in 7 s, taking its own backup first.

What is still NOT established, and must not be read past: whether any specific MariaDB feature misbehaves on unconverted system tables. This run exercised BookStack's normal read/write path only. Establishing that is a different piece of work and is not smuggled in here.

7. The four apps, and why three of them were left alone

app	engine pin	has it moved a major?
bookstack	mariadb:12.3	yes — 11.6 → 12.3, the edge measured here
kimai	mariadb:11.6	no
nextcloud	mariadb:11.6	no
romm	mariadb:11.4	no

Only bookstack has an edge to measure. The other three carry the same engine and the same absent MARIADB_* env, so they will land in exactly this state the day their pin moves a major — which is the reason this is a fleet decision and not one template's problem. Naming them is the right treatment; measuring them today would measure nothing.

8. Cost of the run, and teardown — three layers

Guest 9402 r459-spike, Debian 13, disk on the NVMe dir storage at its root, deliberately off local-lvm.

layer	result
1 — machine	guest destroyed; pct list shows only 9201. scratch-r459 storage removed. Downloaded LXC template deleted.
2 — host	local-lvm 30.53 % before and after — never touched. local 19 630 088 → 19 631 740 KiB (+1.6 MB). pct fstrim 9402 before destroy: 35.9 GiB trimmed. Guest peak: 3.2 GB, of which 2.10 GB images.
3 — hub	Checked, not asserted: /hosts lists exactly demo-felhom-8363b5 and demo-hp-bb76ea; 0 customers. This run created no customer, no appliance and no host record.

Evidence was copied off after each arm, before any teardown: documentation/audits/r459-spike-2026-09-06/.

9. Observations — noticed, documented, not acted on

1. mariadb-upgrade without credentials returns a confident-looking failure that is an auth error. Recorded in §3 because it nearly produced the wrong answer with a plausible exit code.
2. The conversion leaves its own backup in the datadir (system_mysql_backup_*.sql.zst, 622 KB here). Harmless, and it will be picked up by any backup that copies the volume — worth knowing before someone reports an unexplained file.
3. Three shells deep (ssh → pct exec → bash → heredoc), Python heredocs are not survivable. Two scripts had to be written locally and pushed as files. Not a finding about the product; a working note that will save the next session ten minutes.