felhom.eu/REPORT.md

REPORT — SPIKE R-459: is a skipped MariaDB upgrade harmless, and what does fixing it cost? (2026-09-06)

Overwritten each session. Nothing durable lives only here.

OUTCOME A, qualified — and the trade this task was commissioned to price DOES NOT EXIST. The skipped conversion is stable but never self-resolving. Fixing it costs 7 seconds and does not cost the ability to abort, which is what B assumed. It is not C either: the conversion succeeded across the multi-major jump.

1. Confirmed baselines — none had moved

repo	task's baseline	found
app-catalog-felhom.eu	0474ce387e6f	0474ce387e6f
felhom.eu	a1a6c73fe132	a1a6c73fe132
felhom-controller	bab82c471e03	not touched

Highest R- id 462, confirmed. Minted R-463, R-464. No version bump, no release. The task's §2 table was verified and is exactly right: four MariaDB apps at the stated pins, and no MARIADB_* env in any of the 53 templates.

2. The outcome, and the evidence that assigns it

A	unconverted datadir is benign	THIS ONE, qualified — 5 of 5 restarts, no degradation. But the engine says a check is required every start, so "benign" holds only as measured: no decay over restarts, one seeded record, over minutes.
B	converting works, the abort dies with it	NO. Converting works; the abort still starts and serves.
C	converting fails, the jump is too big	NO. It succeeded in 7 s and took its own backup first.

3. Observable 1 — mariadb_upgrade_info, all four moments, verbatim

moment	state	contents
1	fresh 11.6 datadir	11.6.2-MariaDB (14 bytes)
2	12.3 started on it, skip logged	11.6.2-MariaDB — unchanged
3	after 5 restarts of 12.3	11.6.2-MariaDB — still unchanged
4	comparison arm, after conversion	12.3.3-MariaDB

The engine serving at moments 2–3 was 12.3.3-MariaDB, confirmed by mariadb --version — so the mismatch is real and not the wrong container.

4. Observable 2 — the engine's own verdict, invocation, output and exit code

Check mode located in the image, not assumed: --check-if-upgrade-is-needed.

$ docker exec bookstack-db sh -c 'mariadb-upgrade --check-if-upgrade-is-needed \
      --user=root --password=$MYSQL_ROOT_PASSWORD'
Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!
[exit=0]

After conversion:

This installation of MariaDB is already upgraded to 12.3.3-MariaDB.
There is no need to run mariadb-upgrade again.
[exit=1]

The two runs establish the exit-code semantics between them — 0 = needed, 1 = not. Measured, not read from documentation, and worth stating because the polarity is the reverse of the usual convention.

A false start, recorded because it nearly produced the wrong answer: without credentials the same command returns ERROR 1045 (28000): Access denied … FATAL ERROR: Upgrade failed with exit 1 — an authentication failure wearing the shape of a verdict. → R-464.

5. Observable 3 — the restart series: 5 of 5

restart	settled	readback	upgrade_info	engine check	entrypoint
1–5 of 5	yes, each	pass, each	11.6.2-MariaDB, unchanged	Check required!, each	[Note], never escalated

It does not degrade. It also never heals. R-459's hypothesis that the Note might become a Warning or an Error is not supported.

6. Observable 4 — the comparison arm

Scratch copy of the template inside the guest with MARIADB_AUTO_UPGRADE=1. Nothing with MARIADB_ in it was committed to the catalog.

The conversion succeeds, verbatim and in order:

[Note] [Entrypoint]: Starting temporary server
[Note] [Entrypoint]: Backing up system database to system_mysql_backup_11.6.2-MariaDB.sql.zst
[Note] [Entrypoint]: Backing up complete
[Note] [Entrypoint]: Starting mariadb-upgrade
[Note] [Entrypoint]: Finished mariadb-upgrade

Cost: mariadb-upgrade itself 17:33:24 → 17:33:31 = 7 s; whole TO step 35.9 s with it against ~10.5 s without → ≈ 25 s of extra startup, once. Measured on a nearly empty database — it works over system tables rather than row data, so it should scale with table count, but this run did not measure that and 7 s must not be quoted as a fleet figure.

The decision-relevant fact — §14.6 of the task

Converting does NOT kill the abort. With the datadir at 12.3.3-MariaDB, putting 11.6 back:

abort: starts-and-serves        settled 10.5 s        readback: PASS
survived a further restart:     Up 25 seconds (healthy),  upgrade_info: 12.3.3-MariaDB

…but the engine calls that state unsupported, and the entrypoint hides it. → R-464:

asked	answer
the entrypoint, every start	[Note] [Entrypoint]: MariaDB upgrade not required
mariadb-upgrade --check-if-upgrade-is-needed	FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!

So "the abort works" is an observation that it started and served — not a claim the datadir is sound. One restart cycle, one seeded record. Saying more would repeat the mistake this task exists to correct.

7. The harness change, and E3b re-run showing it

upgrade-test.py gained engine_state_after, reported beside the verdict and never folded into it — an unconverted datadir is not known to be a failure, so a verdict that said so would encode an unproven judgement. E3b re-run:

"verdict": "proven",
"engine_state_after": {"bookstack-db": {
  "image": "mariadb:12.3",
  "answer": "11.6.2-MariaDB| Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required! [exit=0]"}}

The exact thing the harness was blind to, now visible next to a green verdict. A PostgreSQL probe is included; it has never run against a real Postgres major (→ R-463).

8. Register — 206 open before, 208 after; closed 168, unchanged

row	disposition
R-459	NARROWED, P2 → P3, WAITING-ON-OPERATOR. Consequence measured; the expected trade does not exist. The fleet-wide env change remains the operator's call (4 apps). VIKTOR rules, CC implements
R-463	OPENED, P2-MEDIUM, CC — the PostgreSQL analogue. 11 templates, 8 on postgres:16-alpine; register grep for pg_upgrade/"postgres major" returned 0, confirmed twice. Deliberately not measured here. The two engines fail in opposite directions: MariaDB skips quietly, Postgres refuses to start — so this one cannot hide, it presents as eight apps down at once
R-464	OPENED, P3-LOW, CC — MariaDB upgrade not required is printed on an unsupported downgrade, so that line cannot be a soundness signal. Same class as "presence is not success" and R-443's HTTP 200 over a crash-looping app

9. Teardown — three layers

layer	result
1 — machine	guest 9402 destroyed; pct list shows only 9201. scratch-r459 removed. Downloaded template deleted.
2 — host	local-lvm 30.53 % before and after — never touched. local 19 630 088 → 19 631 740 KiB (+1.6 MB). pct fstrim 9402: 35.9 GiB trimmed. Guest peak 3.2 GB / 2.10 GB images.
3 — hub	Checked, not asserted: /hosts = exactly demo-felhom-8363b5, demo-hp-bb76ea; 0 customers. This run created no customer, no appliance, no host record.

Evidence off after each arm, before teardown: documentation/audits/r459-spike-2026-09-06/ (20 files, 164 KB). Scanned for secrets before commit: clean.

10. Claims in the task that turned out to be wrong, named

1. §11's venue is still stale, exactly as R-461 records. /mnt/nvme-1tb does not exist — the NVMe is at /mnt/hdd_1. The scratch storage went at its root, honouring the rule's reason; local-lvm read 30.53 % before and after. drill-r50 (VM 300) still does not exist — qm list returns nothing. Both were already filed as R-461 yesterday; this run is the second session to work around them, which is the cost that row predicts.
2. §6 Observable 2 assumed a check mode would exist but told me not to assume its flag — correct, and the caution earned its place. The flag is --check-if-upgrade-is-needed, and its exit-code polarity is the reverse of the usual convention (0 = work needed). Assuming either the name or the polarity would have inverted the headline.
3. §7's Outcome B was the expected result and it is FALSE. The abort survives a real conversion. The task was right to name three outcomes and to say "do not steer" — the run steered nowhere and landed outside the shape the brief most anticipated.
4. §9's Postgres numbers were exact: 11 templates, 8 on postgres:16-alpine, register grep 0.
5. §4.7's pointer was right — mariadb_upgrade_info was already visible in the persistence-sweep probe, and it is the observable that carried this task.

11. Observations — noticed, documented, NOT acted on

1. mariadb-upgrade without credentials returns a confident-looking failure that is an auth error. FILED: R-464, which carries it as the second half of the row.
2. The conversion leaves its own backup in the datadir (system_mysql_backup_*.sql.zst, 622 437 bytes here), which any volume-level backup will then copy. NOT-A-FINDING: it is upstream's deliberate safety net and it is harmless; recording it in the audit is enough to stop the next person reporting an unexplained file.
3. Python heredocs do not survive three shells (ssh → pct exec → bash). Two scripts had to be written locally and pushed as files. NOT-A-FINDING: it is a working technique, not a property of the product, and it is now written into the audit where the next session will meet it.