admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE R-459: the skipped MariaDB conversion is stable, and the trade it implied does not exist
gates / gates (push) Successful in 20s
Details

Browse Source 
Outcome A, qualified. Not B and not C.

It does not degrade: 5 of 5 restarts of 12.3 on an 11.6 datadir, readback passed
every time, mariadb_upgrade_info unchanged, the entrypoint line never escalated
past [Note]. It also never heals - the engine answers 'Major version upgrade
detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!' on every start
and will forever.

The trade R-459 was expected to produce is not real. Converting properly SUCCEEDS
across the multi-major jump, takes 7 seconds, backs up the system database
unasked - and putting 11.6 back afterwards STILL starts and serves the data. So
the operator is being handed a cheap correction, not a choice between a correct
engine and a reversible one.

The exit-code polarity was measured rather than read: 0 means the upgrade IS
needed, 1 means it is not. Assuming either the flag name or the polarity would
have inverted the headline. And run without credentials the same command returns
a confident-looking FATAL ERROR that is an auth failure.

R-464: after converting and going back, the entrypoint prints 'MariaDB upgrade
not required' on a state the same engine calls an unsupported downgrade. The
obvious cheap instrument for R-459 would have been to grep for that line, and it
would have reported fine for the broken case.

R-463: the PostgreSQL analogue, deliberately NOT measured here. 11 templates, 8
on postgres:16-alpine, register grep for pg_upgrade returns zero. The two engines
fail in OPPOSITE directions - MariaDB skips quietly, Postgres refuses to start -
so that one cannot hide; it presents as eight apps down at once.

No template changed. Teardown all three layers, hub checked rather than asserted,
local-lvm 30.53 percent before and after.
This commit is contained in:
authentik Default Admin
2026-09-06 17:42:19 +02:00
parent a1a6c73fe1
commit d6837d98ee
26 changed files with 1551 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+132 -123
View File
@@ -1,172 +1,181 @@
# REPORT — SPIKE: an upgrade test that runs again (2026-09-06)
# REPORT — SPIKE R-459: is a skipped MariaDB upgrade harmless, and what does fixing it cost? (2026-09-06)
*Overwritten each session. Nothing durable lives only here.*
> **C3 FIRST, because everything else is conditional on it.** The negative control — an edge whose TO
> image is `alpine:3.20`, which pulls cleanly and exits immediately — came back **`failed`**
> (`healthy_after: false`, `seed_read_after: false`, `seed_read_before: true`). **The harness can say
> no, so its greens mean something.** It also cost the most wall clock of any edge, 556 s, because a
> negative is only honest if it waits out the full settle window.
> **OUTCOME A, qualified — and the trade this task was commissioned to price DOES NOT EXIST.**
> The skipped conversion is **stable but never self-resolving**. Fixing it costs **7 seconds** and
> does **not** cost the ability to abort, which is what B assumed. It is not C either: the conversion
> **succeeded** across the multi-major jump.
## 1. Confirmed baselines — none had moved
| repo | task's baseline | found |
|---|---|---|
| app-catalog-felhom.eu | `7b9b9b34a5ee` | `7b9b9b34a5ee` |
| felhom.eu | `417df06f3529` | `417df06f3529` |
| felhom-controller | `bab82c4` (v0.235.0) | **not touched** |
| app-catalog-felhom.eu | `0474ce387e6f` | `0474ce387e6f` |
| felhom.eu | `a1a6c73fe132` | `a1a6c73fe132` |
| felhom-controller | `bab82c471e03` | **not touched** |
Highest `R-` id: **458**, confirmed. Minted **R-459 … R-462**. No version bump, no release, no golden.
Highest `R-` id **462**, confirmed. Minted **R-463**, **R-464**. No version bump, no release.
**The task's §2 table was verified and is exactly right**: four MariaDB apps at the stated pins, and
**no `MARIADB_*` env in any of the 53 templates.**
## 2. The verdict record — all seven edges
## 2. The outcome, and the evidence that assigns it
Full JSON per edge in `documentation/audits/upgrade-spike-2026-09-06/evidence/<EDGE>/verdict.json`.
| | | |
|---|---|---|
| **A** | unconverted datadir is benign | **THIS ONE, qualified** — 5 of 5 restarts, no degradation. But the engine says a check is required *every* start, so "benign" holds only as measured: no decay over restarts, one seeded record, over minutes. |
| **B** | converting works, the abort dies with it | **NO.** Converting works; the abort still starts and serves. |
| **C** | converting fails, the jump is too big | **NO.** It succeeded in 7 s and took its own backup first. |
| edge | app | from → to | verdict | data after | **abort** | TO settle | total |
|---|---|---|---|---|---|---|---|
| **C3** | privatebin | `2.0.5` → **`alpine:3.20`** | **failed** | no | starts-and-serves | 421.1 s | 556.0 s |
| C2 | privatebin | `2.0.5` → `2.0.5` | proven | yes | starts-and-serves | 0.1 s | 6.4 s |
| **E1** | privatebin | `1.7.5` → `2.0.5` | **proven** | yes | **starts-and-serves** | 5.2 s | 21.5 s |
| **E2** | docmost | `0.25.3` → `0.95.0` | **proven** | yes | **REFUSES** | 10.7 s | 305.1 s |
| **E3** | bookstack | app `25.02.2`→`26.05.2` + mariadb `11.6`→`12.3` | **proven** | yes | starts-and-serves¹ | 15.7 s | 77.5 s |
| E3a | bookstack | app only, engine held | **proven** | yes | starts-and-serves | 15.7 s | 71.8 s |
| E3b | bookstack | engine only, app held | **proven** | yes | starts-and-serves¹ | 0.2 s | 48.8 s |
## 3. Observable 1 — `mariadb_upgrade_info`, all four moments, verbatim
¹ and §4 is why that is not the good news it looks like.
| moment | state | contents |
|---|---|---|
| 1 | fresh 11.6 datadir | `11.6.2-MariaDB` (14 bytes) |
| 2 | 12.3 started on it, skip logged | **`11.6.2-MariaDB` — unchanged** |
| 3 | after 5 restarts of 12.3 | **`11.6.2-MariaDB` — still unchanged** |
| 4 | comparison arm, after conversion | **`12.3.3-MariaDB`** |
**C1 passed on every edge, including C3.** A fixture that cannot prove itself first proves nothing
after.
The engine serving at moments 2–3 was `12.3.3-MariaDB`, confirmed by `mariadb --version` — so the
mismatch is real and not the wrong container.
## 3. Quoted verbatim
## 4. Observable 2 — the engine's own verdict, invocation, output and exit code
**E2's refusal — the abort of docmost:**
Check mode located in the image, not assumed: **`--check-if-upgrade-is-needed`**.
```
{"level":"error","context":"DatabaseMigrationService",
"msg":"corrupted migrations: previously executed migration 20260213T085259-notifications is missing"}
{"level":"error","context":"DatabaseMigrationService","msg":"Failed to run database migration. Exiting program."}
$ docker exec bookstack-db sh -c 'mariadb-upgrade --check-if-upgrade-is-needed \
--user=root --password=$MYSQL_ROOT_PASSWORD'
Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!
[exit=0]
```
**E2's migration, at the TO step** (six such lines, one shown):
After conversion:
```
{"level":"info","context":"DatabaseMigrationService","msg":"Migration \"20260213T085259-notifications\" executed successfully"}
This installation of MariaDB is already upgraded to 12.3.3-MariaDB.
There is no need to run mariadb-upgrade again.
[exit=1]
```
**E3/E3b, at the moment MariaDB 12.3 first started on the 11.6 datadir:**
**The two runs establish the exit-code semantics between them — 0 = needed, 1 = not.** Measured, not
read from documentation, and worth stating because the polarity is the reverse of the usual
convention.
**A false start, recorded because it nearly produced the wrong answer:** without credentials the same
command returns `ERROR 1045 (28000): Access denied … FATAL ERROR: Upgrade failed` with **exit 1** —
an authentication failure wearing the shape of a verdict. → **R-464.**
## 5. Observable 3 — the restart series: 5 of 5
| restart | settled | readback | `upgrade_info` | engine check | entrypoint |
|---|---|---|---|---|---|
| 1–5 of 5 | yes, each | **pass, each** | `11.6.2-MariaDB`, unchanged | `Check required!`, each | `[Note]`, never escalated |
**It does not degrade. It also never heals.** R-459's hypothesis that the Note might become a Warning
or an Error is **not supported**.
## 6. Observable 4 — the comparison arm
Scratch copy of the template **inside the guest** with `MARIADB_AUTO_UPGRADE=1`. **Nothing with
`MARIADB_` in it was committed to the catalog.**
**The conversion succeeds**, verbatim and in order:
```
[Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required,
but skipped due to $MARIADB_AUTO_UPGRADE setting
[Note] [Entrypoint]: Starting temporary server
[Note] [Entrypoint]: Backing up system database to system_mysql_backup_11.6.2-MariaDB.sql.zst
[Note] [Entrypoint]: Backing up complete
[Note] [Entrypoint]: Starting mariadb-upgrade
[Note] [Entrypoint]: Finished mariadb-upgrade
```
**E3a, the app half alone: no such line at all.**
**Cost:** `mariadb-upgrade` itself **17:33:24 → 17:33:31 = 7 s**; whole TO step **35.9 s** with it
against **~10.5 s** without → **≈ 25 s of extra startup, once**. **Measured on a nearly empty
database** — it works over system tables rather than row data, so it should scale with table count,
**but this run did not measure that** and 7 s must not be quoted as a fleet figure.
## 4. The two findings
### The decision-relevant fact — §14.6 of the task
**(a) Whether an upgrade can be UNDONE is a property of the app, not of upgrades.** docmost refuses;
privatebin does not. This independently reproduces the Nextcloud finding on a second app **by a
different mechanism** — Nextcloud refused on a version comparison, docmost on its migration ledger. So
`09-update-architecture.md` §4's ruling now rests on two measurements, not one. **And the arc was
carrying an assumption that there is one answer for all 53 apps. There is not.**
**Converting does NOT kill the abort.** With the datadir at `12.3.3-MariaDB`, putting **11.6** back:
**(b) R-459 — a real defect in our own catalog, found by accident.** The bookstack template moves
MariaDB across a major and sets **no `MARIADB_*` env at all**, so the image skips the datadir upgrade
it says it requires. **The cause is assigned, not guessed:** E3a (app half, engine held) produces no
upgrade line; E3 and E3b (both move the engine) produce it. **A bundled edge could never have said
which half** — which is exactly what the decomposition existed for. **It also explains why E3's abort
"worked": the datadir was never converted, so 11.6 could still read it.** Whether that ever breaks is
**not established** and the row says so.
```
abort: starts-and-serves settled 10.5 s readback: PASS
survived a further restart: Up 25 seconds (healthy), upgrade_info: 12.3.3-MariaDB
```
## 5. What it cost — measured, for costing the widening
**…but the engine calls that state unsupported, and the entrypoint hides it.** → **R-464**:
| | |
| asked | answer |
|---|---|
| successful edge | **6.4 – 305.1 s**, median **71.8 s** |
| failing edge | **556 s** — ~8× a positive |
| 7 edges total | ~18 min harness time + ~35 min build-out and two fixture iterations |
| disk, 3 apps / 11 images | **5.07 GB** images, 6.0 GB guest |
| naive extrapolation to 53 | ~90 GB, ~1 h harness time |
| the entrypoint, every start | `[Note] [Entrypoint]: MariaDB upgrade not required` |
| `mariadb-upgrade --check-if-upgrade-is-needed` | `FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!` |
**The extrapolation understates it by an order of magnitude, and that is the finding.** Two of three
apps needed a bespoke seed route, one needed two attempts and a discarded approach, and one can only
ever be half-proven. **Fixture time scales with apps and does not amortise.** → **R-462**, which asks
the operator for scope rather than proposing one.
**So "the abort works" is an observation that it started and served — not a claim the datadir is
sound.** One restart cycle, one seeded record. Saying more would repeat the mistake this task exists
to correct.
## 6. Apps with no non-browser seed route
## 7. The harness change, and E3b re-run showing it
**None was fully blocked; one is half-blocked.** privatebin and docmost have clean HTTP APIs.
**BookStack has neither an API token nor a usable HTTP login headlessly** — its template's `https`
`APP_URL` makes the session cookies `secure`, so curl over plain http gets **419 Page Expired** on
every login, which looks exactly like a wrong password. Its database half is provable through
`php artisan` (seed and readback are *different* commands, and the readback runs its own negative
control on every call). **Its FILE half cannot be seeded headlessly at all** → **R-460**. Nothing was
planted by hand anywhere.
`upgrade-test.py` gained **`engine_state_after`**, reported **beside** the verdict and never folded
into it — an unconverted datadir is not *known* to be a failure, so a verdict that said so would
encode an unproven judgement. E3b re-run:
## 7. Evidence, copied off after EACH edge
```json
"verdict": "proven",
"engine_state_after": {"bookstack-db": {
"image": "mariadb:12.3",
"answer": "11.6.2-MariaDB| Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required! [exit=0]"}}
```
`felhom.eu/documentation/audits/upgrade-spike-2026-09-06/evidence/` — 48 files, 340 KB, pulled to
DooPlex after each edge and again at the end, before any teardown. Scanned for secrets before commit:
no token, no password, no generated key. The only `spike-*` strings are seeded account names from a
guest that no longer exists.
**The exact thing the harness was blind to, now visible next to a green verdict.** A PostgreSQL probe
is included; it has **never run against a real Postgres major** (→ R-463).
## 8. Teardown — all three layers
| layer | result |
|---|---|
| **1 — machine** | guest **9401 destroyed**; `pct list` shows only 9201. `scratch-upg` storage **removed**. Downloaded LXC template deleted. |
| **2 — host** | `local-lvm` **30.50 % before and after — never touched**, which was the whole point of siting the guest off it. `local` 19 595 164 → 19 605 084 KiB (+9.7 MB). `pct fstrim 9401` before destroy: **52.8 GiB trimmed**. |
| **3 — hub** | **Checked, not asserted:** `/hosts` lists exactly `demo-felhom-8363b5` and `demo-hp-bb76ea`; 0 customers created. **This run created no customer, no appliance and no host record.** |
**No felhom-controller was in the path at any point.** Raw `docker compose` throughout.
## 9. Register — 203 open before, 206 after; closed 167 → 168
## 8. Register — 206 open before, 208 after; closed 168, unchanged
| row | disposition |
|---|---|
| **R-449** | **CLOSED** — harness built, run, and proven by a red negative control |
| **R-459** | OPENED, P2-MEDIUM — the skipped MariaDB datadir upgrade in our own bookstack template. *CC measures the consequence, VIKTOR rules on a fleet-wide env change* |
| **R-460** | OPENED, P3-LOW, CC — bookstack's file half is unprovable headlessly |
| **R-461** | OPENED, P3-LOW, CC — `target-selection.md` names a venue that does not exist and fences a fixture that is gone |
| **R-462** | OPENED, P2-MEDIUM — the widening, costed with real numbers. *VIKTOR rules on scope* |
| **R-459** | **NARROWED, P2 → P3, WAITING-ON-OPERATOR.** Consequence measured; the expected trade does not exist. The fleet-wide env change remains the operator's call (4 apps). *VIKTOR rules, CC implements* |
| **R-463** | **OPENED**, P2-MEDIUM, CC — the PostgreSQL analogue. **11 templates, 8 on `postgres:16-alpine`**; register grep for `pg_upgrade`/"postgres major" returned **0**, confirmed twice. **Deliberately not measured here.** The two engines fail in **opposite** directions: MariaDB skips quietly, Postgres **refuses to start** — so this one cannot hide, it presents as eight apps down at once |
| **R-464** | **OPENED**, P3-LOW, CC — `MariaDB upgrade not required` is printed on an unsupported downgrade, so that line cannot be a soundness signal. Same class as "presence is not success" and R-443's HTTP 200 over a crash-looping app |
## 10. The capability map was NOT edited, and that is deliberate
## 9. Teardown — three layers
**This run measured apps, not the product.** Nothing the platform can do changed: no controller code,
no version, no behaviour. Editing the map reflexively would record a capability the product did not
gain. Said here rather than left silent.
| layer | result |
|---|---|
| **1 — machine** | guest **9402 destroyed**; `pct list` shows only 9201. `scratch-r459` **removed**. Downloaded template deleted. |
| **2 — host** | **`local-lvm` 30.53 % before and after — never touched.** `local` 19 630 088 → 19 631 740 KiB (+1.6 MB). `pct fstrim 9402`: **35.9 GiB trimmed**. Guest peak 3.2 GB / 2.10 GB images. |
| **3 — hub** | **Checked, not asserted:** `/hosts` = exactly `demo-felhom-8363b5`, `demo-hp-bb76ea`; **0 customers**. **This run created no customer, no appliance, no host record.** |
## 11. Claims in the task that turned out to be wrong, named
Evidence off after **each arm**, before teardown: `documentation/audits/r459-spike-2026-09-06/`
(20 files, 164 KB). Scanned for secrets before commit: clean.
1. **§11's venue is stale, in two ways.** **`/mnt/nvme-1tb` does not exist** — the 1 TB NVMe is at
`/mnt/hdd_1`, the enrolled user-data drive, i.e. the same disk under a different path; the scratch
storage went at *its* root, honouring the rule's reason. And **`drill-r50` (VM 300) is gone** —
`qm list` returns nothing on demo-hp, so that fence protects nothing today. → **R-461**.
2. **§6's edges were all real and all resolvable.** Every one of the eleven images was verified against
its registry before use; none had to be substituted. The task asked to say so if any had.
3. **§8 expected bookstack to be "the one most likely to be hard" — correct, and for a reason the task
did not name.** The blocker was not the missing API token; it was that the template's `https`
`APP_URL` makes the session cookies `secure`, so no http login can ever work. Two independent
blockers, and only one was anticipated.
4. **§10.2's "gate on each command's own exit code" had to be broken once, deliberately and in the
open.** `bookstack:reset-mfa` exits **1 for a user it found and 1 for one it did not**, so the exit
code carries no information; the discriminator is the output, required positive with the not-found
sentence required absent. Stated in the fixture's docstring rather than done quietly.
5. **§9's verdict shape needed no change** and is now recorded in `09-update-architecture.md` §6 as the
contract Slice 6 carries.
## 10. Claims in the task that turned out to be wrong, named
## 12. Observations — noticed, documented, NOT acted on
1. **§11's venue is still stale, exactly as R-461 records. `/mnt/nvme-1tb` does not exist** — the NVMe
is at `/mnt/hdd_1`. The scratch storage went at *its* root, honouring the rule's reason; `local-lvm`
read 30.53 % before and after. **`drill-r50` (VM 300) still does not exist** — `qm list` returns
nothing. Both were already filed as R-461 yesterday; this run is the second session to work around
them, which is the cost that row predicts.
2. **§6 Observable 2 assumed a check mode would exist but told me not to assume its flag — correct,
and the caution earned its place.** The flag is `--check-if-upgrade-is-needed`, and **its exit-code
polarity is the reverse of the usual convention** (0 = work needed). Assuming either the name or the
polarity would have inverted the headline.
3. **§7's Outcome B was the expected result and it is FALSE.** The abort survives a real conversion.
The task was right to name three outcomes and to say "do not steer" — the run steered nowhere and
landed outside the shape the brief most anticipated.
4. **§9's Postgres numbers were exact**: 11 templates, 8 on `postgres:16-alpine`, register grep 0.
5. **§4.7's pointer was right** — `mariadb_upgrade_info` was already visible in the persistence-sweep
probe, and it is the observable that carried this task.
1. **`docker compose logs` only shows containers that currently exist**, so the abort erases the TO
step's output from any later capture — **the single most important line of this run survived only
because it had already been extracted.** Fixed mid-run (`to-full.log` is now written at the TO
step) and E3 re-run to get clean evidence. **FILED: R-449's closure records it; the harness change
is committed.** **NOT-A-FINDING as a separate row: it is a harness bug that was found and fixed
inside the same session, with the fix committed and the affected edge re-measured — there is no
residue for a row to track.**
2. **A negative edge costs ~8× a positive.** **NOT-A-FINDING: it is a measured cost recorded in R-462,
which is where the widening will be scheduled from; a second row would duplicate it.**
3. **`bookstack:reset-mfa`'s help says `[options]` but a positional argument is rejected with "No
arguments expected"** in 25.02.2 — correct behaviour that reads as a missing feature.
**NOT-A-FINDING: it is upstream's interface, not ours, and it costs us nothing now that the fixture
documents it.**
## 11. Observations — noticed, documented, NOT acted on
1. **`mariadb-upgrade` without credentials returns a confident-looking failure that is an auth error.**
**FILED: R-464**, which carries it as the second half of the row.
2. **The conversion leaves its own backup in the datadir** (`system_mysql_backup_*.sql.zst`, 622 437
bytes here), which any volume-level backup will then copy. **NOT-A-FINDING: it is upstream's
deliberate safety net and it is harmless; recording it in the audit is enough to stop the next
person reporting an unexplained file.**
3. **Python heredocs do not survive three shells (`ssh` → `pct exec` → `bash`).** Two scripts had to be
written locally and pushed as files. **NOT-A-FINDING: it is a working technique, not a property of
the product, and it is now written into the audit where the next session will meet it.**