hub v0.88.0 — the WAL that never was (R-172)

store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is
mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams
reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and
IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode
with busy_timeout=0 for its entire life while its own source said otherwise.

Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes
a writer, so rendering an operator page blocks a host report; the hub 500s, the
agent waits its full 15-minute interval without retrying, and staleness fires at
30 minutes — two collisions is a false alarm plus an operator email. 13 collisions
in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up
two days and reconciling throughout.

The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it
while the DB was open.

Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate.
_txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a
read-then-write tx must upgrade its lock and a failed upgrade is
SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+
db.Begin() sites and they are all write paths.

Every test asserts what the DATABASE reports, never the DSN string — a string
test would have passed for the whole life of the bug. Red-proof: restoring the
shipped DSN reproduces journal_mode="delete", the missing -wal, and the live
"database is locked (5) (SQLITE_BUSY)".

Operational consequence handled: a WAL DB cannot be copied by taking hub.db
alone — a bare `cat` opens cleanly and silently omits the newest writes. The
break-glass retrieval in operations/nodes.md used exactly that; it and the
recovery-inventory note are now WAL-aware.

hub/CHANGELOG.md

@@ -1,3 +1,63 @@
+## v0.88.0 — the WAL that never was (2026-08-02, R-172)
+
+**The hub has never actually been in WAL mode.** `store.New` opened the database with
+`?_journal_mode=WAL&_busy_timeout=5000` — **mattn/go-sqlite3** syntax — while the driver is
+**modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`, `_time_format`,
+`_time_integer_format`, `_txlock` and `_inttotime`. Everything else is **ignored without an error**.
+So the hub ran in the default rollback-journal mode with `busy_timeout=0` for its entire life, while
+its own source said otherwise — a configuration asserting an invariant the code did not provide.
+
+**How it surfaced.** A false `HOST STALE` banner for `demo-felhom-8363b5` while the agent was up two
+days and reconciling normally. In rollback-journal mode a reader excludes a writer, so rendering an
+operator page can block a host report; the hub then returns **HTTP 500**, the agent logs
+`hub: report failed; keeping current interval` and **waits its full 15-minute interval**, and
+staleness fires at 30 minutes. **Two consecutive collisions = a false alarm + an operator e-mail.**
+Measured: 13 `SQLITE_BUSY` collisions in one pod lifetime, and the alarm fired twice that day
+(19:12:32 and 20:42:32 CEST) for a host that was never down.
+
+**The observable that proved it:** a 128 MB `/data/hub.db` with **no `-wal`/`-shm` file beside it
+while the database was open**. In WAL mode those files must exist.
+
+**The fix is one DSN, and each parameter earns its place:**
+
+```
+?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate
+```
+
+- `journal_mode(WAL)` — readers and one writer proceed concurrently, so a page render can no longer
+  block a report. It is a property of the database FILE, so it persists once set.
+- `busy_timeout(5000)` — writers still serialise; without a timeout SQLite returns `SQLITE_BUSY`
+  *immediately* rather than waiting.
+- `_txlock=immediate` — **the one that is easy to miss, and WAL + busy_timeout alone would not cover
+  it.** `database/sql`'s `Begin()` is DEFERRED, so a transaction that reads then writes must upgrade
+  its lock, and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**.
+  This store has 10+ `db.Begin()` sites and they are all write paths (customer delete/reset, wg,
+  appliance, pbsdr, telemetry, log bundles). Without this the fix would leave a known un-retryable
+  path open.
+
+**Every test asserts what the DATABASE reports, never the DSN string** — a test on the string would
+have passed happily for the entire life of the bug. Five tests: the runtime pragma values; the
+`-wal`/`-shm` files existing beside an open DB (the production signature, pinned); a reader not
+blocking a writer (the consequence, not the mechanism); concurrent writers waiting instead of
+erroring; and racing read-then-write transactions. Plus `TestSQLiteDriverIgnoresMattnStyleParams`, a
+guard on the ROOT CAUSE: it fails if someone "tidies" the pragmas back to the familiar mattn form,
+and skips itself with instructions if a future driver starts honouring them.
+
+**Red-proof:** restoring the shipped DSN reproduces the live failure exactly — `journal_mode = "delete"`,
+the `-wal` absent, and `a write FAILED while a read was open: database is locked (5) (SQLITE_BUSY)`.
+
+**Operational consequence, handled rather than discovered later:** a WAL database cannot be copied by
+taking `hub.db` alone — a committed transaction may still be in `hub.db-wal`, so a bare `cat` yields
+a copy that opens cleanly and **silently omits the newest writes**. That is the worst shape for a
+credential lookup, and the break-glass retrieval in `documentation/operations/nodes.md` used exactly
+that command. Both it and the `_recovery-inventory` note are now WAL-aware (copy the `-wal`, shred
+both).
+
+**Retries (options b and c in R-172) were NOT added.** With readers no longer blocking writers and
+the upgrade path covered, a `SQLITE_BUSY` reaching an HTTP handler should now be rare enough to be a
+real signal. If any appear after this, they mean something else and a retry would hide it. Revisit
+only on evidence.
+
 ## v0.87.0 — the Setup tab stops claiming a host-install version it cannot know (2026-08-02)
 
 **R-94, all three legs, closed by deletion rather than derivation.** The customer page's Setup