admin
d6837d98ee
gates / gates (push) Successful in 20s
Outcome A, qualified. Not B and not C. It does not degrade: 5 of 5 restarts of 12.3 on an 11.6 datadir, readback passed every time, mariadb_upgrade_info unchanged, the entrypoint line never escalated past [Note]. It also never heals - the engine answers 'Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!' on every start and will forever. The trade R-459 was expected to produce is not real. Converting properly SUCCEEDS across the multi-major jump, takes 7 seconds, backs up the system database unasked - and putting 11.6 back afterwards STILL starts and serves the data. So the operator is being handed a cheap correction, not a choice between a correct engine and a reversible one. The exit-code polarity was measured rather than read: 0 means the upgrade IS needed, 1 means it is not. Assuming either the flag name or the polarity would have inverted the headline. And run without credentials the same command returns a confident-looking FATAL ERROR that is an auth failure. R-464: after converting and going back, the entrypoint prints 'MariaDB upgrade not required' on a state the same engine calls an unsupported downgrade. The obvious cheap instrument for R-459 would have been to grep for that line, and it would have reported fine for the broken case. R-463: the PostgreSQL analogue, deliberately NOT measured here. 11 templates, 8 on postgres:16-alpine, register grep for pg_upgrade returns zero. The two engines fail in OPPOSITE directions - MariaDB skips quietly, Postgres refuses to start - so that one cannot hide; it presents as eight apps down at once. No template changed. Teardown all three layers, hub checked rather than asserted, local-lvm 30.53 percent before and after.
182 lines
9.9 KiB
Markdown
182 lines
9.9 KiB
Markdown
# REPORT — SPIKE R-459: is a skipped MariaDB upgrade harmless, and what does fixing it cost? (2026-09-06)
|
||
|
||
*Overwritten each session. Nothing durable lives only here.*
|
||
|
||
> **OUTCOME A, qualified — and the trade this task was commissioned to price DOES NOT EXIST.**
|
||
> The skipped conversion is **stable but never self-resolving**. Fixing it costs **7 seconds** and
|
||
> does **not** cost the ability to abort, which is what B assumed. It is not C either: the conversion
|
||
> **succeeded** across the multi-major jump.
|
||
|
||
## 1. Confirmed baselines — none had moved
|
||
|
||
| repo | task's baseline | found |
|
||
|---|---|---|
|
||
| app-catalog-felhom.eu | `0474ce387e6f` | `0474ce387e6f` |
|
||
| felhom.eu | `a1a6c73fe132` | `a1a6c73fe132` |
|
||
| felhom-controller | `bab82c471e03` | **not touched** |
|
||
|
||
Highest `R-` id **462**, confirmed. Minted **R-463**, **R-464**. No version bump, no release.
|
||
**The task's §2 table was verified and is exactly right**: four MariaDB apps at the stated pins, and
|
||
**no `MARIADB_*` env in any of the 53 templates.**
|
||
|
||
## 2. The outcome, and the evidence that assigns it
|
||
|
||
| | | |
|
||
|---|---|---|
|
||
| **A** | unconverted datadir is benign | **THIS ONE, qualified** — 5 of 5 restarts, no degradation. But the engine says a check is required *every* start, so "benign" holds only as measured: no decay over restarts, one seeded record, over minutes. |
|
||
| **B** | converting works, the abort dies with it | **NO.** Converting works; the abort still starts and serves. |
|
||
| **C** | converting fails, the jump is too big | **NO.** It succeeded in 7 s and took its own backup first. |
|
||
|
||
## 3. Observable 1 — `mariadb_upgrade_info`, all four moments, verbatim
|
||
|
||
| moment | state | contents |
|
||
|---|---|---|
|
||
| 1 | fresh 11.6 datadir | `11.6.2-MariaDB` (14 bytes) |
|
||
| 2 | 12.3 started on it, skip logged | **`11.6.2-MariaDB` — unchanged** |
|
||
| 3 | after 5 restarts of 12.3 | **`11.6.2-MariaDB` — still unchanged** |
|
||
| 4 | comparison arm, after conversion | **`12.3.3-MariaDB`** |
|
||
|
||
The engine serving at moments 2–3 was `12.3.3-MariaDB`, confirmed by `mariadb --version` — so the
|
||
mismatch is real and not the wrong container.
|
||
|
||
## 4. Observable 2 — the engine's own verdict, invocation, output and exit code
|
||
|
||
Check mode located in the image, not assumed: **`--check-if-upgrade-is-needed`**.
|
||
|
||
```
|
||
$ docker exec bookstack-db sh -c 'mariadb-upgrade --check-if-upgrade-is-needed \
|
||
--user=root --password=$MYSQL_ROOT_PASSWORD'
|
||
Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!
|
||
[exit=0]
|
||
```
|
||
|
||
After conversion:
|
||
|
||
```
|
||
This installation of MariaDB is already upgraded to 12.3.3-MariaDB.
|
||
There is no need to run mariadb-upgrade again.
|
||
[exit=1]
|
||
```
|
||
|
||
**The two runs establish the exit-code semantics between them — 0 = needed, 1 = not.** Measured, not
|
||
read from documentation, and worth stating because the polarity is the reverse of the usual
|
||
convention.
|
||
|
||
**A false start, recorded because it nearly produced the wrong answer:** without credentials the same
|
||
command returns `ERROR 1045 (28000): Access denied … FATAL ERROR: Upgrade failed` with **exit 1** —
|
||
an authentication failure wearing the shape of a verdict. → **R-464.**
|
||
|
||
## 5. Observable 3 — the restart series: 5 of 5
|
||
|
||
| restart | settled | readback | `upgrade_info` | engine check | entrypoint |
|
||
|---|---|---|---|---|---|
|
||
| 1–5 of 5 | yes, each | **pass, each** | `11.6.2-MariaDB`, unchanged | `Check required!`, each | `[Note]`, never escalated |
|
||
|
||
**It does not degrade. It also never heals.** R-459's hypothesis that the Note might become a Warning
|
||
or an Error is **not supported**.
|
||
|
||
## 6. Observable 4 — the comparison arm
|
||
|
||
Scratch copy of the template **inside the guest** with `MARIADB_AUTO_UPGRADE=1`. **Nothing with
|
||
`MARIADB_` in it was committed to the catalog.**
|
||
|
||
**The conversion succeeds**, verbatim and in order:
|
||
|
||
```
|
||
[Note] [Entrypoint]: Starting temporary server
|
||
[Note] [Entrypoint]: Backing up system database to system_mysql_backup_11.6.2-MariaDB.sql.zst
|
||
[Note] [Entrypoint]: Backing up complete
|
||
[Note] [Entrypoint]: Starting mariadb-upgrade
|
||
[Note] [Entrypoint]: Finished mariadb-upgrade
|
||
```
|
||
|
||
**Cost:** `mariadb-upgrade` itself **17:33:24 → 17:33:31 = 7 s**; whole TO step **35.9 s** with it
|
||
against **~10.5 s** without → **≈ 25 s of extra startup, once**. **Measured on a nearly empty
|
||
database** — it works over system tables rather than row data, so it should scale with table count,
|
||
**but this run did not measure that** and 7 s must not be quoted as a fleet figure.
|
||
|
||
### The decision-relevant fact — §14.6 of the task
|
||
|
||
**Converting does NOT kill the abort.** With the datadir at `12.3.3-MariaDB`, putting **11.6** back:
|
||
|
||
```
|
||
abort: starts-and-serves settled 10.5 s readback: PASS
|
||
survived a further restart: Up 25 seconds (healthy), upgrade_info: 12.3.3-MariaDB
|
||
```
|
||
|
||
**…but the engine calls that state unsupported, and the entrypoint hides it.** → **R-464**:
|
||
|
||
| asked | answer |
|
||
|---|---|
|
||
| the entrypoint, every start | `[Note] [Entrypoint]: MariaDB upgrade not required` |
|
||
| `mariadb-upgrade --check-if-upgrade-is-needed` | `FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!` |
|
||
|
||
**So "the abort works" is an observation that it started and served — not a claim the datadir is
|
||
sound.** One restart cycle, one seeded record. Saying more would repeat the mistake this task exists
|
||
to correct.
|
||
|
||
## 7. The harness change, and E3b re-run showing it
|
||
|
||
`upgrade-test.py` gained **`engine_state_after`**, reported **beside** the verdict and never folded
|
||
into it — an unconverted datadir is not *known* to be a failure, so a verdict that said so would
|
||
encode an unproven judgement. E3b re-run:
|
||
|
||
```json
|
||
"verdict": "proven",
|
||
"engine_state_after": {"bookstack-db": {
|
||
"image": "mariadb:12.3",
|
||
"answer": "11.6.2-MariaDB| Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required! [exit=0]"}}
|
||
```
|
||
|
||
**The exact thing the harness was blind to, now visible next to a green verdict.** A PostgreSQL probe
|
||
is included; it has **never run against a real Postgres major** (→ R-463).
|
||
|
||
## 8. Register — 206 open before, 208 after; closed 168, unchanged
|
||
|
||
| row | disposition |
|
||
|---|---|
|
||
| **R-459** | **NARROWED, P2 → P3, WAITING-ON-OPERATOR.** Consequence measured; the expected trade does not exist. The fleet-wide env change remains the operator's call (4 apps). *VIKTOR rules, CC implements* |
|
||
| **R-463** | **OPENED**, P2-MEDIUM, CC — the PostgreSQL analogue. **11 templates, 8 on `postgres:16-alpine`**; register grep for `pg_upgrade`/"postgres major" returned **0**, confirmed twice. **Deliberately not measured here.** The two engines fail in **opposite** directions: MariaDB skips quietly, Postgres **refuses to start** — so this one cannot hide, it presents as eight apps down at once |
|
||
| **R-464** | **OPENED**, P3-LOW, CC — `MariaDB upgrade not required` is printed on an unsupported downgrade, so that line cannot be a soundness signal. Same class as "presence is not success" and R-443's HTTP 200 over a crash-looping app |
|
||
|
||
## 9. Teardown — three layers
|
||
|
||
| layer | result |
|
||
|---|---|
|
||
| **1 — machine** | guest **9402 destroyed**; `pct list` shows only 9201. `scratch-r459` **removed**. Downloaded template deleted. |
|
||
| **2 — host** | **`local-lvm` 30.53 % before and after — never touched.** `local` 19 630 088 → 19 631 740 KiB (+1.6 MB). `pct fstrim 9402`: **35.9 GiB trimmed**. Guest peak 3.2 GB / 2.10 GB images. |
|
||
| **3 — hub** | **Checked, not asserted:** `/hosts` = exactly `demo-felhom-8363b5`, `demo-hp-bb76ea`; **0 customers**. **This run created no customer, no appliance, no host record.** |
|
||
|
||
Evidence off after **each arm**, before teardown: `documentation/audits/r459-spike-2026-09-06/`
|
||
(20 files, 164 KB). Scanned for secrets before commit: clean.
|
||
|
||
## 10. Claims in the task that turned out to be wrong, named
|
||
|
||
1. **§11's venue is still stale, exactly as R-461 records. `/mnt/nvme-1tb` does not exist** — the NVMe
|
||
is at `/mnt/hdd_1`. The scratch storage went at *its* root, honouring the rule's reason; `local-lvm`
|
||
read 30.53 % before and after. **`drill-r50` (VM 300) still does not exist** — `qm list` returns
|
||
nothing. Both were already filed as R-461 yesterday; this run is the second session to work around
|
||
them, which is the cost that row predicts.
|
||
2. **§6 Observable 2 assumed a check mode would exist but told me not to assume its flag — correct,
|
||
and the caution earned its place.** The flag is `--check-if-upgrade-is-needed`, and **its exit-code
|
||
polarity is the reverse of the usual convention** (0 = work needed). Assuming either the name or the
|
||
polarity would have inverted the headline.
|
||
3. **§7's Outcome B was the expected result and it is FALSE.** The abort survives a real conversion.
|
||
The task was right to name three outcomes and to say "do not steer" — the run steered nowhere and
|
||
landed outside the shape the brief most anticipated.
|
||
4. **§9's Postgres numbers were exact**: 11 templates, 8 on `postgres:16-alpine`, register grep 0.
|
||
5. **§4.7's pointer was right** — `mariadb_upgrade_info` was already visible in the persistence-sweep
|
||
probe, and it is the observable that carried this task.
|
||
|
||
## 11. Observations — noticed, documented, NOT acted on
|
||
|
||
1. **`mariadb-upgrade` without credentials returns a confident-looking failure that is an auth error.**
|
||
**FILED: R-464**, which carries it as the second half of the row.
|
||
2. **The conversion leaves its own backup in the datadir** (`system_mysql_backup_*.sql.zst`, 622 437
|
||
bytes here), which any volume-level backup will then copy. **NOT-A-FINDING: it is upstream's
|
||
deliberate safety net and it is harmless; recording it in the audit is enough to stop the next
|
||
person reporting an unexplained file.**
|
||
3. **Python heredocs do not survive three shells (`ssh` → `pct exec` → `bash`).** Two scripts had to be
|
||
written locally and pushed as files. **NOT-A-FINDING: it is a working technique, not a property of
|
||
the product, and it is now written into the audit where the next session will meet it.**
|