store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode with busy_timeout=0 for its entire life while its own source said otherwise. Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes a writer, so rendering an operator page blocks a host report; the hub 500s, the agent waits its full 15-minute interval without retrying, and staleness fires at 30 minutes — two collisions is a false alarm plus an operator email. 13 collisions in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up two days and reconciling throughout. The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it while the DB was open. Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate. _txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+ db.Begin() sites and they are all write paths. Every test asserts what the DATABASE reports, never the DSN string — a string test would have passed for the whole life of the bug. Red-proof: restoring the shipped DSN reproduces journal_mode="delete", the missing -wal, and the live "database is locked (5) (SQLITE_BUSY)". Operational consequence handled: a WAL DB cannot be copied by taking hub.db alone — a bare `cat` opens cleanly and silently omits the newest writes. The break-glass retrieval in operations/nodes.md used exactly that; it and the recovery-inventory note are now WAL-aware.
This commit is contained in:
@@ -45,7 +45,10 @@ hub, guests and PBS are UTC — every timestamp below carries its zone.
|
||||
the hub SQLite DB was taken with `kubectl exec … cat /data/hub.db > hub.db.copy` at **17:40:00 UTC**
|
||||
(113,033,216 bytes) into the session scratchpad, and every hub-DB figure below comes from that copy.
|
||||
It is a hot copy of a live database; row counts and metadata are consistent enough for an inventory
|
||||
but are a snapshot of that instant, not a transactionally consistent dump. The copy is read with
|
||||
but are a snapshot of that instant, not a transactionally consistent dump. **Do not reuse this
|
||||
command as a recipe: since hub v0.88.0 the DB is in WAL mode (R-172), so `cat /data/hub.db` alone
|
||||
yields a copy that opens cleanly and silently omits the newest writes — the `-wal` must be copied
|
||||
beside it (`documentation/operations/nodes.md`).** The copy is read with
|
||||
`mode=ro`. No hub table was written.
|
||||
|
||||
**Secret hygiene.** No secret value, key, token, password or key fingerprint is reproduced in this
|
||||
|
||||
Reference in New Issue
Block a user