# REPORT — SPIKE R-459: is a skipped MariaDB upgrade harmless, and what does fixing it cost? (2026-09-06)

*Overwritten each session. Nothing durable lives only here.*

> **OUTCOME A, qualified — and the trade this task was commissioned to price DOES NOT EXIST.**
> The skipped conversion is **stable but never self-resolving**. Fixing it costs **7 seconds** and
> does **not** cost the ability to abort, which is what B assumed. It is not C either: the conversion
> **succeeded** across the multi-major jump.

## 1. Confirmed baselines — none had moved

| repo | task's baseline | found |
|---|---|---|
| app-catalog-felhom.eu | `0474ce387e6f` | `0474ce387e6f` |
| felhom.eu | `a1a6c73fe132` | `a1a6c73fe132` |
| felhom-controller | `bab82c471e03` | **not touched** |

Highest `R-` id **462**, confirmed. Minted **R-463**, **R-464**. No version bump, no release.
**The task's §2 table was verified and is exactly right**: four MariaDB apps at the stated pins, and
**no `MARIADB_*` env in any of the 53 templates.**

## 2. The outcome, and the evidence that assigns it

| | | |
|---|---|---|
| **A** | unconverted datadir is benign | **THIS ONE, qualified** — 5 of 5 restarts, no degradation. But the engine says a check is required *every* start, so "benign" holds only as measured: no decay over restarts, one seeded record, over minutes. |
| **B** | converting works, the abort dies with it | **NO.** Converting works; the abort still starts and serves. |
| **C** | converting fails, the jump is too big | **NO.** It succeeded in 7 s and took its own backup first. |

## 3. Observable 1 — `mariadb_upgrade_info`, all four moments, verbatim

| moment | state | contents |
|---|---|---|
| 1 | fresh 11.6 datadir | `11.6.2-MariaDB` (14 bytes) |
| 2 | 12.3 started on it, skip logged | **`11.6.2-MariaDB` — unchanged** |
| 3 | after 5 restarts of 12.3 | **`11.6.2-MariaDB` — still unchanged** |
| 4 | comparison arm, after conversion | **`12.3.3-MariaDB`** |

The engine serving at moments 2–3 was `12.3.3-MariaDB`, confirmed by `mariadb --version` — so the
mismatch is real and not the wrong container.

## 4. Observable 2 — the engine's own verdict, invocation, output and exit code

Check mode located in the image, not assumed: **`--check-if-upgrade-is-needed`**.

```
$ docker exec bookstack-db sh -c 'mariadb-upgrade --check-if-upgrade-is-needed \
      --user=root --password=$MYSQL_ROOT_PASSWORD'
Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!
[exit=0]
```

After conversion:

```
This installation of MariaDB is already upgraded to 12.3.3-MariaDB.
There is no need to run mariadb-upgrade again.
[exit=1]
```

**The two runs establish the exit-code semantics between them — 0 = needed, 1 = not.** Measured, not
read from documentation, and worth stating because the polarity is the reverse of the usual
convention.

**A false start, recorded because it nearly produced the wrong answer:** without credentials the same
command returns `ERROR 1045 (28000): Access denied … FATAL ERROR: Upgrade failed` with **exit 1** —
an authentication failure wearing the shape of a verdict. → **R-464.**

## 5. Observable 3 — the restart series: 5 of 5

| restart | settled | readback | `upgrade_info` | engine check | entrypoint |
|---|---|---|---|---|---|
| 1–5 of 5 | yes, each | **pass, each** | `11.6.2-MariaDB`, unchanged | `Check required!`, each | `[Note]`, never escalated |

**It does not degrade. It also never heals.** R-459's hypothesis that the Note might become a Warning
or an Error is **not supported**.

## 6. Observable 4 — the comparison arm

Scratch copy of the template **inside the guest** with `MARIADB_AUTO_UPGRADE=1`. **Nothing with
`MARIADB_` in it was committed to the catalog.**

**The conversion succeeds**, verbatim and in order:

```
[Note] [Entrypoint]: Starting temporary server
[Note] [Entrypoint]: Backing up system database to system_mysql_backup_11.6.2-MariaDB.sql.zst
[Note] [Entrypoint]: Backing up complete
[Note] [Entrypoint]: Starting mariadb-upgrade
[Note] [Entrypoint]: Finished mariadb-upgrade
```

**Cost:** `mariadb-upgrade` itself **17:33:24 → 17:33:31 = 7 s**; whole TO step **35.9 s** with it
against **~10.5 s** without → **≈ 25 s of extra startup, once**. **Measured on a nearly empty
database** — it works over system tables rather than row data, so it should scale with table count,
**but this run did not measure that** and 7 s must not be quoted as a fleet figure.

### The decision-relevant fact — §14.6 of the task

**Converting does NOT kill the abort.** With the datadir at `12.3.3-MariaDB`, putting **11.6** back:

```
abort: starts-and-serves        settled 10.5 s        readback: PASS
survived a further restart:     Up 25 seconds (healthy),  upgrade_info: 12.3.3-MariaDB
```

**…but the engine calls that state unsupported, and the entrypoint hides it.** → **R-464**:

| asked | answer |
|---|---|
| the entrypoint, every start | `[Note] [Entrypoint]: MariaDB upgrade not required` |
| `mariadb-upgrade --check-if-upgrade-is-needed` | `FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!` |

**So "the abort works" is an observation that it started and served — not a claim the datadir is
sound.** One restart cycle, one seeded record. Saying more would repeat the mistake this task exists
to correct.

## 7. The harness change, and E3b re-run showing it

`upgrade-test.py` gained **`engine_state_after`**, reported **beside** the verdict and never folded
into it — an unconverted datadir is not *known* to be a failure, so a verdict that said so would
encode an unproven judgement. E3b re-run:

```json
"verdict": "proven",
"engine_state_after": {"bookstack-db": {
  "image": "mariadb:12.3",
  "answer": "11.6.2-MariaDB| Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required! [exit=0]"}}
```

**The exact thing the harness was blind to, now visible next to a green verdict.** A PostgreSQL probe
is included; it has **never run against a real Postgres major** (→ R-463).

## 8. Register — 206 open before, 208 after; closed 168, unchanged

| row | disposition |
|---|---|
| **R-459** | **NARROWED, P2 → P3, WAITING-ON-OPERATOR.** Consequence measured; the expected trade does not exist. The fleet-wide env change remains the operator's call (4 apps). *VIKTOR rules, CC implements* |
| **R-463** | **OPENED**, P2-MEDIUM, CC — the PostgreSQL analogue. **11 templates, 8 on `postgres:16-alpine`**; register grep for `pg_upgrade`/"postgres major" returned **0**, confirmed twice. **Deliberately not measured here.** The two engines fail in **opposite** directions: MariaDB skips quietly, Postgres **refuses to start** — so this one cannot hide, it presents as eight apps down at once |
| **R-464** | **OPENED**, P3-LOW, CC — `MariaDB upgrade not required` is printed on an unsupported downgrade, so that line cannot be a soundness signal. Same class as "presence is not success" and R-443's HTTP 200 over a crash-looping app |

## 9. Teardown — three layers

| layer | result |
|---|---|
| **1 — machine** | guest **9402 destroyed**; `pct list` shows only 9201. `scratch-r459` **removed**. Downloaded template deleted. |
| **2 — host** | **`local-lvm` 30.53 % before and after — never touched.** `local` 19 630 088 → 19 631 740 KiB (+1.6 MB). `pct fstrim 9402`: **35.9 GiB trimmed**. Guest peak 3.2 GB / 2.10 GB images. |
| **3 — hub** | **Checked, not asserted:** `/hosts` = exactly `demo-felhom-8363b5`, `demo-hp-bb76ea`; **0 customers**. **This run created no customer, no appliance, no host record.** |

Evidence off after **each arm**, before teardown: `documentation/audits/r459-spike-2026-09-06/`
(20 files, 164 KB). Scanned for secrets before commit: clean.

## 10. Claims in the task that turned out to be wrong, named

1. **§11's venue is still stale, exactly as R-461 records. `/mnt/nvme-1tb` does not exist** — the NVMe
   is at `/mnt/hdd_1`. The scratch storage went at *its* root, honouring the rule's reason; `local-lvm`
   read 30.53 % before and after. **`drill-r50` (VM 300) still does not exist** — `qm list` returns
   nothing. Both were already filed as R-461 yesterday; this run is the second session to work around
   them, which is the cost that row predicts.
2. **§6 Observable 2 assumed a check mode would exist but told me not to assume its flag — correct,
   and the caution earned its place.** The flag is `--check-if-upgrade-is-needed`, and **its exit-code
   polarity is the reverse of the usual convention** (0 = work needed). Assuming either the name or the
   polarity would have inverted the headline.
3. **§7's Outcome B was the expected result and it is FALSE.** The abort survives a real conversion.
   The task was right to name three outcomes and to say "do not steer" — the run steered nowhere and
   landed outside the shape the brief most anticipated.
4. **§9's Postgres numbers were exact**: 11 templates, 8 on `postgres:16-alpine`, register grep 0.
5. **§4.7's pointer was right** — `mariadb_upgrade_info` was already visible in the persistence-sweep
   probe, and it is the observable that carried this task.

## 11. Observations — noticed, documented, NOT acted on

1. **`mariadb-upgrade` without credentials returns a confident-looking failure that is an auth error.**
   **FILED: R-464**, which carries it as the second half of the row.
2. **The conversion leaves its own backup in the datadir** (`system_mysql_backup_*.sql.zst`, 622 437
   bytes here), which any volume-level backup will then copy. **NOT-A-FINDING: it is upstream's
   deliberate safety net and it is harmless; recording it in the audit is enough to stop the next
   person reporting an unexplained file.**
3. **Python heredocs do not survive three shells (`ssh` → `pct exec` → `bash`).** Two scripts had to be
   written locally and pushed as files. **NOT-A-FINDING: it is a working technique, not a property of
   the product, and it is now written into the audit where the next session will meet it.**