hub v0.88.0 — the WAL that never was (R-172)

store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is
mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams
reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and
IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode
with busy_timeout=0 for its entire life while its own source said otherwise.

Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes
a writer, so rendering an operator page blocks a host report; the hub 500s, the
agent waits its full 15-minute interval without retrying, and staleness fires at
30 minutes — two collisions is a false alarm plus an operator email. 13 collisions
in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up
two days and reconciling throughout.

The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it
while the DB was open.

Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate.
_txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a
read-then-write tx must upgrade its lock and a failed upgrade is
SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+
db.Begin() sites and they are all write paths.

Every test asserts what the DATABASE reports, never the DSN string — a string
test would have passed for the whole life of the bug. Red-proof: restoring the
shipped DSN reproduces journal_mode="delete", the missing -wal, and the live
"database is locked (5) (SQLITE_BUSY)".

Operational consequence handled: a WAL DB cannot be copied by taking hub.db
alone — a bare `cat` opens cleanly and silently omits the newest writes. The
break-glass retrieval in operations/nodes.md used exactly that; it and the
recovery-inventory note are now WAL-aware.

documentation/operations/nodes.md

@@ -116,13 +116,23 @@ root password vaulted in the hub**, `host_recovery` row `demo-hp-bb76ea` (set at
 
 Retrieval (operator-side, and **shred the copy** — that DB holds every host's secret):
 
+> **WAL-AWARE SINCE HUB v0.88.0 — copying `hub.db` ALONE is no longer safe.** The hub runs SQLite in
+> **WAL** mode (R-172), so a committed transaction may still live in `hub.db-wal` and not yet be in
+> the main file. A bare `cat /data/hub.db` therefore yields a copy that is **valid but stale** — it
+> opens cleanly and silently lacks the most recent writes, which is the worst failure shape for a
+> credential lookup. Copy the `-wal` beside it and let SQLite replay it on open.
+
 ```bash
-sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db > /tmp/x.db
+sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db     > /tmp/x.db
+sudo kubectl -n felhom-system exec <hub-pod> -- cat /data/hub.db-wal > /tmp/x.db-wal 2>/dev/null || true
 python3 -c "import sqlite3;print(sqlite3.connect('/tmp/x.db').execute(
   \"SELECT secret FROM host_recovery WHERE host_id='demo-hp-bb76ea'\").fetchone()[0])"
-shred -u /tmp/x.db
+shred -u /tmp/x.db /tmp/x.db-wal
 ```
 
+The `|| true` is deliberate: an absent `-wal` is legitimate (a freshly checkpointed database), and
+must not fail the retrieval. **Shred both files** — the WAL holds the same secrets as the DB.
+
 Then `sshpass -e ssh root@demo-hp` (sshpass is on DooPlex, not on the nodes).
 
 **This is the lockout filed as R-61**: the ISO mints a throwaway root password per build and discards