felhom.eu/REPORT.md

REPORT — SPIKE: an upgrade test that runs again (2026-09-06)

Overwritten each session. Nothing durable lives only here.

C3 FIRST, because everything else is conditional on it. The negative control — an edge whose TO image is alpine:3.20, which pulls cleanly and exits immediately — came back failed (healthy_after: false, seed_read_after: false, seed_read_before: true). The harness can say no, so its greens mean something. It also cost the most wall clock of any edge, 556 s, because a negative is only honest if it waits out the full settle window.

1. Confirmed baselines — none had moved

repo	task's baseline	found
app-catalog-felhom.eu	7b9b9b34a5ee	7b9b9b34a5ee
felhom.eu	417df06f3529	417df06f3529
felhom-controller	bab82c4 (v0.235.0)	not touched

Highest R- id: 458, confirmed. Minted R-459 … R-462. No version bump, no release, no golden.

2. The verdict record — all seven edges

Full JSON per edge in documentation/audits/upgrade-spike-2026-09-06/evidence/<EDGE>/verdict.json.

edge	app	from → to	verdict	data after	abort	TO settle	total
C3	privatebin	2.0.5 → alpine:3.20	failed	no	starts-and-serves	421.1 s	556.0 s
C2	privatebin	2.0.5 → 2.0.5	proven	yes	starts-and-serves	0.1 s	6.4 s
E1	privatebin	1.7.5 → 2.0.5	proven	yes	starts-and-serves	5.2 s	21.5 s
E2	docmost	0.25.3 → 0.95.0	proven	yes	REFUSES	10.7 s	305.1 s
E3	bookstack	app 25.02.2→26.05.2 + mariadb 11.6→12.3	proven	yes	starts-and-serves¹	15.7 s	77.5 s
E3a	bookstack	app only, engine held	proven	yes	starts-and-serves	15.7 s	71.8 s
E3b	bookstack	engine only, app held	proven	yes	starts-and-serves¹	0.2 s	48.8 s

¹ and §4 is why that is not the good news it looks like.

C1 passed on every edge, including C3. A fixture that cannot prove itself first proves nothing after.

3. Quoted verbatim

E2's refusal — the abort of docmost:

{"level":"error","context":"DatabaseMigrationService",
 "msg":"corrupted migrations: previously executed migration 20260213T085259-notifications is missing"}
{"level":"error","context":"DatabaseMigrationService","msg":"Failed to run database migration. Exiting program."}

E2's migration, at the TO step (six such lines, one shown):

{"level":"info","context":"DatabaseMigrationService","msg":"Migration \"20260213T085259-notifications\" executed successfully"}

E3/E3b, at the moment MariaDB 12.3 first started on the 11.6 datadir:

[Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required,
                     but skipped due to $MARIADB_AUTO_UPGRADE setting

E3a, the app half alone: no such line at all.

4. The two findings

(a) Whether an upgrade can be UNDONE is a property of the app, not of upgrades. docmost refuses; privatebin does not. This independently reproduces the Nextcloud finding on a second app by a different mechanism — Nextcloud refused on a version comparison, docmost on its migration ledger. So 09-update-architecture.md §4's ruling now rests on two measurements, not one. And the arc was carrying an assumption that there is one answer for all 53 apps. There is not.

(b) R-459 — a real defect in our own catalog, found by accident. The bookstack template moves MariaDB across a major and sets no MARIADB_* env at all, so the image skips the datadir upgrade it says it requires. The cause is assigned, not guessed: E3a (app half, engine held) produces no upgrade line; E3 and E3b (both move the engine) produce it. A bundled edge could never have said which half — which is exactly what the decomposition existed for. It also explains why E3's abort "worked": the datadir was never converted, so 11.6 could still read it. Whether that ever breaks is not established and the row says so.

5. What it cost — measured, for costing the widening

successful edge	6.4 – 305.1 s, median 71.8 s
failing edge	556 s — ~8× a positive
7 edges total	~18 min harness time + ~35 min build-out and two fixture iterations
disk, 3 apps / 11 images	5.07 GB images, 6.0 GB guest
naive extrapolation to 53	~90 GB, ~1 h harness time

The extrapolation understates it by an order of magnitude, and that is the finding. Two of three apps needed a bespoke seed route, one needed two attempts and a discarded approach, and one can only ever be half-proven. Fixture time scales with apps and does not amortise. → R-462, which asks the operator for scope rather than proposing one.

6. Apps with no non-browser seed route

None was fully blocked; one is half-blocked. privatebin and docmost have clean HTTP APIs. BookStack has neither an API token nor a usable HTTP login headlessly — its template's https APP_URL makes the session cookies secure, so curl over plain http gets 419 Page Expired on every login, which looks exactly like a wrong password. Its database half is provable through php artisan (seed and readback are different commands, and the readback runs its own negative control on every call). Its FILE half cannot be seeded headlessly at all → R-460. Nothing was planted by hand anywhere.

7. Evidence, copied off after EACH edge

felhom.eu/documentation/audits/upgrade-spike-2026-09-06/evidence/ — 48 files, 340 KB, pulled to DooPlex after each edge and again at the end, before any teardown. Scanned for secrets before commit: no token, no password, no generated key. The only spike-* strings are seeded account names from a guest that no longer exists.

8. Teardown — all three layers

layer	result
1 — machine	guest 9401 destroyed; pct list shows only 9201. scratch-upg storage removed. Downloaded LXC template deleted.
2 — host	local-lvm 30.50 % before and after — never touched, which was the whole point of siting the guest off it. local 19 595 164 → 19 605 084 KiB (+9.7 MB). pct fstrim 9401 before destroy: 52.8 GiB trimmed.
3 — hub	Checked, not asserted: /hosts lists exactly demo-felhom-8363b5 and demo-hp-bb76ea; 0 customers created. This run created no customer, no appliance and no host record.

No felhom-controller was in the path at any point. Raw docker compose throughout.

9. Register — 203 open before, 206 after; closed 167 → 168

row	disposition
R-449	CLOSED — harness built, run, and proven by a red negative control
R-459	OPENED, P2-MEDIUM — the skipped MariaDB datadir upgrade in our own bookstack template. CC measures the consequence, VIKTOR rules on a fleet-wide env change
R-460	OPENED, P3-LOW, CC — bookstack's file half is unprovable headlessly
R-461	OPENED, P3-LOW, CC — target-selection.md names a venue that does not exist and fences a fixture that is gone
R-462	OPENED, P2-MEDIUM — the widening, costed with real numbers. VIKTOR rules on scope

10. The capability map was NOT edited, and that is deliberate

This run measured apps, not the product. Nothing the platform can do changed: no controller code, no version, no behaviour. Editing the map reflexively would record a capability the product did not gain. Said here rather than left silent.

11. Claims in the task that turned out to be wrong, named

1. §11's venue is stale, in two ways. /mnt/nvme-1tb does not exist — the 1 TB NVMe is at /mnt/hdd_1, the enrolled user-data drive, i.e. the same disk under a different path; the scratch storage went at its root, honouring the rule's reason. And drill-r50 (VM 300) is gone — qm list returns nothing on demo-hp, so that fence protects nothing today. → R-461.
2. §6's edges were all real and all resolvable. Every one of the eleven images was verified against its registry before use; none had to be substituted. The task asked to say so if any had.
3. §8 expected bookstack to be "the one most likely to be hard" — correct, and for a reason the task did not name. The blocker was not the missing API token; it was that the template's https APP_URL makes the session cookies secure, so no http login can ever work. Two independent blockers, and only one was anticipated.
4. §10.2's "gate on each command's own exit code" had to be broken once, deliberately and in the open. bookstack:reset-mfa exits 1 for a user it found and 1 for one it did not, so the exit code carries no information; the discriminator is the output, required positive with the not-found sentence required absent. Stated in the fixture's docstring rather than done quietly.
5. §9's verdict shape needed no change and is now recorded in 09-update-architecture.md §6 as the contract Slice 6 carries.

12. Observations — noticed, documented, NOT acted on

1. docker compose logs only shows containers that currently exist, so the abort erases the TO step's output from any later capture — the single most important line of this run survived only because it had already been extracted. Fixed mid-run (to-full.log is now written at the TO step) and E3 re-run to get clean evidence. FILED: R-449's closure records it; the harness change is committed. NOT-A-FINDING as a separate row: it is a harness bug that was found and fixed inside the same session, with the fix committed and the affected edge re-measured — there is no residue for a row to track.
2. A negative edge costs ~8× a positive. NOT-A-FINDING: it is a measured cost recorded in R-462, which is where the widening will be scheduled from; a second row would duplicate it.
3. bookstack:reset-mfa's help says [options] but a positional argument is rejected with "No arguments expected" in 25.02.2 — correct behaviour that reads as a missing feature. NOT-A-FINDING: it is upstream's interface, not ours, and it costs us nothing now that the fixture documents it.