felhom.eu/REPORT.md

# REPORT — MariaDB finishes its own conversion, golden 0.236.0, and goldens move to a cadence (2026-09-13)

*Overwritten each session. Nothing durable lives only here — every finding below has a register row.*

> **Everything the task asked for shipped, and every scenario A–H passed with the evidence quoted
> below.** Four templates carry `MARIADB_AUTO_UPGRADE=1`; the harness saw the conversion RUN; the
> change landed on demo-hp through the real 15-minute cycle and recreated nothing; an engine-major
> gate holds every database engine inside its major until Slice 4; golden **0.236.0** is baked,
> round-tripped, vouched and the floor raised; and `golden_currency_gate.py` reads a dated waiver.
> **Three claims in the prompt turned out wrong or imprecise — named first, in §1.**

## 1. Claims in the prompt that turned out wrong, named first

1. **"a throwaway guest on demo-hp, disk on `/mnt/nvme-1tb` at its root."** `/mnt/nvme-1tb` does not
   exist on demo-hp; the 1 TB NVMe is mounted at **`/mnt/hdd_1`** (already recorded by the two
   September spikes and R-461). The guest's disk went on a dir storage at `/mnt/hdd_1`'s root, as
   those spikes did. `target-selection.md` still names the wrong path — that is R-461, not re-filed.
2. **"the waiver is R-242's remaining half."** The gate's docstring names TWO things: the honest fix
   for a release nobody wants a golden for is "a recorded waiver, never a bypass" (built today), and
   R-242's *remaining half* is that **nothing gates the VOUCH** (unbuilt, unchanged, still open on
   R-242). The task text conflated them. The docstring, the row and this report keep them apart.
3. **"Add the bake to the nightly-session checklist."** No document by that name exists in any repo
   (`grep -rln nightly` finds none under `documentation/runbooks/`). The step went into the two
   routines that do exist: `RUNBOOK-manual-build.md` §4.2 (the cadence) and this repo's
   `CLAUDE.md` end-of-session checklist. If a nightly checklist is created later, it points at §4.2.

Also imprecise, not wrong: the runbook's vouch step says to read `MinAgent` from the golden's
controller CHANGELOG header, and **the last four headers carry no such line** — filed as **R-470**.

## 2. Confirmed baselines

| repo | before | after (pushed to `main`) |
|---|---|---|
| app-catalog-felhom.eu | `b7ef0c4a09d6` | `eec1228` templates → `bd32830` gate → `3525e35` CHANGELOG/REPORT |
| felhom.eu | `4b2e5608c227` | `ae59c31` (documents + gate + evidence) → the compression commit on top |
| felhom-controller | `155271672265` (v0.236.0) | **unchanged — no code.** Golden bake only. |

Highest R-id before: 467. Minted: **R-468** (waiver), **R-469** (engine-major rule expiry),
**R-470** (MinAgent header line), **R-471** (observations decoy hole, pre-existing).

## 3. Scenario results, with the evidence quoted

Evidence directory: `documentation/audits/r459-close-2026-09-13/` (36 files); golden:
`documentation/tests/golden-0.236.0-2026-09-13/`.

### A — the harness sees the conversion happen ✅

Throwaway LXC **9403** on demo-hp (Debian 13, Docker 29.8.0, Compose v5.5.1), harness copied from the
local commit WITH the setting. `engine_state_after.bookstack-db.answer`, verbatim, on both edges:

```
12.3.3-MariaDB | This installation of MariaDB is already upgraded to 12.3.3-MariaDB. There is no need to run mariadb-upgrade again. [exit=1]
```

| edge | verdict | seed before / after | abort |
|---|---|---|---|
| E3 (app + engine 11.6→12.3) | **proven** | true / true | starts-and-serves |
| E3b (engine half alone) | **proven** | true / true | starts-and-serves |

The entrypoint, verbatim (E3, `harness/E3/to-full.log`):

```
[Note] [Entrypoint]: Backing up system database to system_mysql_backup_11.6.2-MariaDB.sql.zst
[Note] [Entrypoint]: Backing up complete
[Note] [Entrypoint]: Starting mariadb-upgrade
Major version upgrade detected from 11.6.2-MariaDB to 12.3.3-MariaDB. Check required!
[Note] [Entrypoint]: Finished mariadb-upgrade
```

`skipped due to $MARIADB_AUTO_UPGRADE`: **0** lines in both TO logs (it appeared on every start
before this change — spike §4). Conversion 09:53:20 → 09:53:26 = **6 s**. The abort log prints
`MariaDB upgrade not required` — the R-464 trap, quoted as an observation and not as soundness.

### B — the negative control still works ✅

C3 (`privatebin/pdo:2.0.5 → alpine:3.20`): **`failed`**, `healthy_after=false`, container
`restarting`. Ran first, as the CLAUDE.md rule says.

### C — the change lands harmlessly on a live box ✅

Pushed `3525e35` at **07:58:02 UTC**; the box's previous sync was 07:54:17, so the change waited for
the real tick. `live-9201/00-baseline-before-push.txt` → `01-after-sync.txt` → `02-restart.txt`.

The sync's own lines, verbatim:

```
2026/09/13 08:09:17 sync.go:396: [INFO] [sync] Updated bookstack/docker-compose.yml
2026/09/13 08:09:17 sync.go:409: [DEBUG] [sync] bookstack: stored definition refreshed with the delivered fix
2026/09/13 08:09:17 sync.go:135: [INFO] [sync] Periodic sync: Sablonok frissítve — frissítve: bookstack, kimai, nextcloud, romm
```

Live `docker-compose.yml` fragment from 9201 after the sync:

```
  bookstack-db:
    image: mariadb:12.3
    ...
      - TZ=Europe/Budapest
      # MARIADB_AUTO_UPGRADE: on a MAJOR engine move the engine converts its own datadir (~7 s on a
      ...
      - MARIADB_AUTO_UPGRADE=1
```

**Not recreated by the sync:** both container IDs and `StartedAt` identical to the pre-push baseline
(`bookstack` `26b555d7…` started 04:21:07Z; `bookstack-db` `b02f7a09…` started 04:21:01Z), `docker ps`
`Up 4 hours (healthy)`. `applied-compose.yml` carries the setting too (fixes flow INTO the pin).

The one deliberate `POST /api/stacks/bookstack/restart` at 08:10:32 UTC → `{"ok":true,"message":"Stack
bookstack restart completed"}`, healthy after 8 s. The engine's log after the restart, verbatim:

```
2026-09-13 10:10:34+02:00 [Note] [Entrypoint]: MariaDB upgrade not required
2026-09-13 10:10:35 0 [Note] mariadbd: ready for connections.
```

and the engine asked directly (R-464): `12.3.2-MariaDB` / `This installation of MariaDB is already
upgraded to 12.3.2-MariaDB.` `[exit=1]`; `MARIADB_AUTO_UPGRADE=1` present in the running container;
`GET /login` → **200**. Nothing else on 9201 was touched; bentopdf stays.

### D — the rule exists and a gate knows it ✅

`app-catalog-felhom.eu/scripts/check-engine-major.py`, fourth row of `catalog_gates.py`, run by the
pre-push hook with `--range=<remote sha>..<local sha>`. Refusal, verbatim:

```
ENGINE-MAJOR GATE FAILED: templates/kimai/docker-compose.yml service kimai-db moves mariadb 11 -> 12 (mariadb:11.6 -> mariadb:12.3).
RULE (app-catalog CLAUDE.md, operator ruling 2026-09-13): until the Update button takes a VERIFIED BACKUP as its precondition (Slice 4, felhom.eu OPEN-ITEMS.md R-448), no template may move a database-engine image across a MAJOR version.
WHY: MariaDB sidecars now carry MARIADB_AUTO_UPGRADE=1 and WILL convert the customer's datadir on the next Update; PostgreSQL's image refuses to start on an older major's datadir (R-463). Either way this is a customer-data event with no backup in front of it.
EXPIRY: this rule is removed DELIBERATELY when R-448 ships — the removal is its own register row, not a silent edit. Until then, keep the engine within its major.
```

Pass, verbatim (this push's own range): `engine-major gate OK — no database engine crosses a major
version (rule: CLAUDE.md, until Slice 4 / R-448 ships)`. Red-proof (`engine-major-gate/redproof-7-cases.txt`):
`11.6→12.3` rc=1, `postgres:16→17` rc=1, `mariadb:lts` rc=2, `11.6→11.8` rc=0, and three decoys
(comment + `serverVersion=` env, the app's own image, README) rc=0. **CI cannot run it** — the
runner fetches at `--depth 1`; the runner announces the skip on a shallow clone (pinned by
`test_catalog_gates.py`). Same gap as R-452, not re-filed.

### E/F/G/H — the golden gate in all four waiver states ✅ (`golden-waiver-states/`)

| state | exit | the line that proves it |
|---|---|---|
| E valid waiver, golden BEHIND | **0** | `GOLDEN CURRENCY GATE ADVISORY — WAIVED, NOT CLEAN: controller v9.9.9 is released and NO golden carries it (newest bake is 9.9.7; 2 release(s) behind: 9.9.8, 9.9.9).` … `Waived by R-468 until 2026-09-26 (13 day(s) left)` |
| F expired waiver, behind | **1** | `The waiver at documentation/tests/golden-waiver.yml EXPIRED on 2026-09-12 (R-468). It ran out, as a dated waiver is meant to` |
| G valid waiver, golden UNRECORDED | **1** | `A waiver exists (R-468, until 2026-09-26) and DOES NOT COVER THIS: it covers a golden that is behind the record, never one that is unrecorded (R-385).` |
| H 15-day waiver | **2** | `the waiver … is MALFORMED — `expires:` is 15 days after `issued:` — the hard limit is 14.` |
| H decoy: a file saying only `expires` | **2** | `MALFORMED — `issued:` is absent or not a YYYY-MM-DD date` |

**Red-proof on the REAL tree, while it was still behind** (before the bake landed; R-468 row present,
a valid waiver planted): the OLD gate (`4b2e560`) → `GOLDEN CURRENCY GATE FAILED … exit=1` (it cannot
read a waiver); the NEW gate → `ADVISORY — WAIVED … 4 release(s) behind: 0.233.0, 0.234.0, 0.235.0,
0.236.0 … exit=0`. Tests: `scripts/test_golden_currency_gate.py` cases 5–15, all green
(19 cases in the file now, 4 before).

**The first waiver, as committed** (`documentation/tests/golden-waiver.yml`, issued AFTER the bake):

```yaml
issued: 2026-09-13
expires: 2026-09-27
reason: pre-customer development; goldens on a weekly cadence (operator ruling 2026-09-13)
register_row: R-468
```

Gate on the real tree now: `newest released controller : 0.236.0` / `newest golden baked : 0.236.0`
/ `waiver : VALID until 2026-09-27 (14 day(s) left)` → **OK, exit 0**. Before the bake it read
`FAILED … 4 release(s) behind`, exit 1 (`golden-waiver-states/00-real-tree-before-bake-no-waiver.txt`).

## 4. The golden (R-467) — version and vouch evidence

`GOLDEN_VERSION=0.236.0`, `GOLDEN_SHA256=58a3cc24c61dc2271f2cb508ef5a269af97005f386b671f18011258df5f958bf`,
654 115 664 B. Three readers agreed (bake log, round trip hashed from the DOWNLOADED bytes, the hub's
dropdown); `./etc/felhom-controller-image` out of the archive says `felhom-controller:0.236.0`;
markers 1/1/1/1, `excluding` 0, `FATAL` 0; token-leak grep 0 with the seeded control 1; the bake
script's fingerprint equal across the hop. Vouch: `golden_version 0.232.0 → 0.236.0`, `agent_version`
0.130.0 and `min_agent` 0.129.0 unchanged, re-read from the page, R-120 banner absent; floor
`0.232.0 → 0.236.0` re-read. **No box moved — both already ran 0.236.0 by hand**; the chain was last
exercised 2026-09-01 and nothing about it changed. This bake carried four unbaked releases and is
the last per-release one. **No `--no-verify` anywhere in this session.**

## 5. Files created / modified

**app-catalog-felhom.eu** (pushed `3525e35`): `templates/{bookstack,kimai,nextcloud,romm}/docker-compose.yml`,
`scripts/check-engine-major.py` (new), `scripts/test_gate_decoys.py` (new), `scripts/catalog_gates.py`,
`scripts/test_catalog_gates.py`, `.githooks/pre-push`, `CLAUDE.md`, `REUSE.md`, `CHANGELOG.md`, `REPORT.md`.

**felhom.eu**: `scripts/golden_currency_gate.py`, `scripts/test_golden_currency_gate.py`,
`scripts/CHANGELOG.md`, `documentation/tests/golden-waiver.yml` (new),
`documentation/tests/golden-0.236.0-2026-09-13/` (new, 12 files),
`documentation/audits/r459-close-2026-09-13/` (new, 36 files),
`documentation/runbooks/RUNBOOK-manual-build.md` (§4.2), `CLAUDE.md` (checklist step),
`documentation/architecture/09-update-architecture.md` (§3 decisions 5 and 6, §8.7, engines section),
`documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/CLOSED-ITEMS.md`, `CONTEXT.md`, `STATUS.md`, this file.

## 6. Tests

| suite | before | after |
|---|---|---|
| `app-catalog/scripts/test_catalog_gates.py` | 5 | **7**, green |
| `app-catalog/scripts/test_gate_decoys.py` | — | **7 cases**, green |
| `felhom.eu/scripts/test_golden_currency_gate.py` | 4 | **19 cases**, green |
| `felhom.eu/scripts/repo_gates.py --fast` | 14 gates | 14 gates, **all OK** |
| `felhom.eu/scripts/decoy_coverage_gate.py` on the catalog | 3 exempt | 1 covered + 3 exempt, 0 unaccounted |

No Go code changed in any repo; `go build/vet/test` not applicable.

`python3 scripts/unproven.py --summary`: 55 claims, walked 20 / partial 17 / built 14 / missing 4,
**NOT WALKED 35 of 55 — unchanged** (nothing in this session walked a capability claim).

**CI, by `head_sha`:** app-catalog job **536** for `3525e35` → `completed` / `success`. The
felhom.eu run for this push is quoted in the follow-up commit that records it.

## 7. Register — rows opened / closed, size

Closed: **R-459** (harness + live evidence), **R-467** (the bake). Narrowed: **R-242** (waiver half built;
vouch half open). Opened: **R-468** (WATCHING — the waiver; renew ≤ 14 days or bake; retire at the
first external install), **R-469** (BLOCKED on R-448 — remove the engine-major rule), **R-470**
(READY — MinAgent header line), **R-471** (READY — observations decoy hole, pre-existing).
Register size: **210 open rows / 169 closed before → 212 open / 171 closed after** (R-459 and R-467
compressed to `CLOSED-ITEMS.md` in the second commit, citing `ae59c31` for the full text; nothing deleted).
`OPEN-ITEMS.md`: 433 845 → 434988 bytes with the four new rows and two closures written out, → 429692 bytes after compression.

## 8. Evidence off the machine before every teardown

- **Harness (9403):** `evidence/` tarred and `pct pull`ed, then scp'd to DooPlex, **before**
  `pct fstrim` / `pct destroy` — `harness/teardown-9403.txt`.
- **Bake (drill VM):** `bake.log` scp'd out and markers counted **before** `pct destroy 9100`, the
  `shred -u` and the `poweroff` — `05-teardown.txt`, leftovers in the VM: 0.
- **Live (9201):** baseline captured **before** the push; after-sync and after-restart captured at
  the moment; nothing on 9201 was reverted.

## 9. Teardown — three layers

| layer | before | after |
|---|---|---|
| **1 — machines** | LXC 9403 on demo-hp (scratch dir storage `scratch-r459c` at `/mnt/hdd_1`); LXC 9100 inside the drill VM | 9403 **destroyed**, storage **removed**, template deleted, `pct list` shows only 9201; 9100 **destroyed**, drill VM **powered off**, `qemu` confirmed exited (`ps -eo comm`), disk reverted to the single `virgin` snapshot |
| **2 — hosts** | demo-hp `local-lvm` **35.32 %**, `local` 20 899 060 KiB, `/mnt/hdd_1` 5 774 620 KiB | `local-lvm` **35.32 % — never touched**; `local` 20 908 828 KiB (+9.5 MB); `/mnt/hdd_1` **5 774 620 KiB — identical**; `pct fstrim 9403` returned 55.5 GiB; guest peak 3.3 GB (2.21 GB images). DooPlex `/mnt/5_hdd` 37 % before and after |
| **3 — hub** | 2 enrolled hosts | **Checked, not asserted** (`harness/teardown-layer3-hub.txt`): `/hosts` lists exactly `demo-felhom-8363b5` and `demo-hp-bb76ea`; **this run created no customer, no appliance and no host record** — 9403 never enrolled, 9100 is the bake fixture. The hub's ONLY change is the vouch + floor, which is the deliverable. |

Helper files on demo-hp (`ctrl_pw`, `dh_user`, `dh_pat`, `upg.tgz`, `guest-setup.sh`, logs) were
`shred -u`'d / removed; the Docker Hub login inside 9403 was logged out before the guest was destroyed.

## 10. NOT live-validated — stated plainly

- **The waiver in the "behind" state on the real tree, after this push:** it is dormant (the golden is
  current). Its real-tree behaviour was proven **before** the bake (§3 E, red-proof); after today it is
  exercised the next time a release ships without a bake — which is the intended cadence.
- **The self-update chain for 0.236.0:** not re-exercised (both boxes were already there). Last proof
  2026-09-01.
- **Kimai, Nextcloud, RomM under an actual engine major move:** the setting is inert until their pins
  move, and the engine-major rule forbids that until Slice 4. Only bookstack has a measurable edge.
- **The engine-major gate in CI:** cannot run there (`--depth 1`); the hook is the enforcement.

## 11. Observations — noticed, documented, NOT acted on

1. **`felhom.eu/scripts/test_gate_decoys.py` reports a LIVE HOLE at HEAD, before this session:**
   `observations/R-419: decoy PASSED (rc=0)` — the gate scans the report's first `Observations`
   section and not an appended second one. Reproduced on a clean worktree of `4b2e560`. **FILED: R-471.**
2. **Four consecutive controller CHANGELOG headers carry no `MinAgent:` line** while the vouch runbook
   says to read it from the header. **FILED: R-470.**
3. **The stack restart recreated only the changed service** — bookstack-db got a new ID, the bookstack
   app container kept its 4-hour uptime — because the restart is compose up -d, which the update
   architecture §1.3 records as a chosen behaviour.
   **NOT-A-FINDING: documented design, and "restart completed" is accurate for a compose-level restart.**
4. **`mariadb:12.3` resolved to 12.3.3 in the harness and 12.3.2 on 9201** — the floating-pin class.
   **NOT-A-FINDING: already R-446.**
5. **`target-selection.md` names `/mnt/nvme-1tb`, which does not exist on demo-hp.** **NOT-A-FINDING:
   already R-461, and it says exactly this.**
6. **The session ran with permission prompts disabled**, which the workspace `CLAUDE.md` says not to do
   on this host. Nothing outside the workspace, `~/build`, the drill directory and the two Tier-0
   boxes was touched; every host-side act is listed in §9. **NOT-A-FINDING: an operator setting, not
   a product defect; recorded so it is not hidden.**