hub v0.88.0 — the WAL that never was (R-172)
gates / gates (push) Successful in 7s

store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is
mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams
reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and
IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode
with busy_timeout=0 for its entire life while its own source said otherwise.

Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes
a writer, so rendering an operator page blocks a host report; the hub 500s, the
agent waits its full 15-minute interval without retrying, and staleness fires at
30 minutes — two collisions is a false alarm plus an operator email. 13 collisions
in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up
two days and reconciling throughout.

The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it
while the DB was open.

Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate.
_txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a
read-then-write tx must upgrade its lock and a failed upgrade is
SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+
db.Begin() sites and they are all write paths.

Every test asserts what the DATABASE reports, never the DSN string — a string
test would have passed for the whole life of the bug. Red-proof: restoring the
shipped DSN reproduces journal_mode="delete", the missing -wal, and the live
"database is locked (5) (SQLITE_BUSY)".

Operational consequence handled: a WAL DB cannot be copied by taking hub.db
alone — a bare `cat` opens cleanly and silently omits the newest writes. The
break-glass retrieval in operations/nodes.md used exactly that; it and the
recovery-inventory note are now WAL-aware.
This commit is contained in:
2026-08-02 21:06:29 +02:00
parent 2c35c4204a
commit 0fc54e0122
5 changed files with 343 additions and 4 deletions
@@ -45,7 +45,10 @@ hub, guests and PBS are UTC — every timestamp below carries its zone.
the hub SQLite DB was taken with `kubectl exec … cat /data/hub.db > hub.db.copy` at **17:40:00 UTC**
(113,033,216 bytes) into the session scratchpad, and every hub-DB figure below comes from that copy.
It is a hot copy of a live database; row counts and metadata are consistent enough for an inventory
but are a snapshot of that instant, not a transactionally consistent dump. The copy is read with
but are a snapshot of that instant, not a transactionally consistent dump. **Do not reuse this
command as a recipe: since hub v0.88.0 the DB is in WAL mode (R-172), so `cat /data/hub.db` alone
yields a copy that opens cleanly and silently omits the newest writes — the `-wal` must be copied
beside it (`documentation/operations/nodes.md`).** The copy is read with
`mode=ro`. No hub table was written.
**Secret hygiene.** No secret value, key, token, password or key fingerprint is reproduced in this