store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is
mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams
reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and
IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode
with busy_timeout=0 for its entire life while its own source said otherwise.
Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes
a writer, so rendering an operator page blocks a host report; the hub 500s, the
agent waits its full 15-minute interval without retrying, and staleness fires at
30 minutes — two collisions is a false alarm plus an operator email. 13 collisions
in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up
two days and reconciling throughout.
The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it
while the DB was open.
Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate.
_txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a
read-then-write tx must upgrade its lock and a failed upgrade is
SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+
db.Begin() sites and they are all write paths.
Every test asserts what the DATABASE reports, never the DSN string — a string
test would have passed for the whole life of the bug. Red-proof: restoring the
shipped DSN reproduces journal_mode="delete", the missing -wal, and the live
"database is locked (5) (SQLITE_BUSY)".
Operational consequence handled: a WAL DB cannot be copied by taking hub.db
alone — a bare `cat` opens cleanly and silently omits the newest writes. The
break-glass retrieval in operations/nodes.md used exactly that; it and the
recovery-inventory note are now WAL-aware.
Follow-up acting on the observations filed with runbooks/target-selection.md.
operations/nodes.md
- The demo-hp NVMe was documented "PRESENT AND UNENROLLED -- do not touch" and
listed under "What is NOT enrolled here (deliberately)". Both are FALSE and had
been for eight days: it was enrolled 2026-07-22 through the normal Tarhely flow
and is now /mnt/nvme-1tb -- the enrolled user-data drive AND the felhom-backup
target (verified live 2026-07-30: nvme0n1 -> /mnt/nvme-1tb, and
dir: felhom-backup / path /mnt/nvme-1tb / is_mountpoint 1). The fence's own
condition (join via Tarhely, not the installer, not by hand) was SATISFIED, so
the prohibition expired with it -- while still contradicting the task specs that
correctly sent drill-VM disks there. Retracted with its reason recorded, and the
caution that IS still live kept (dir storage at the mountpoint ROOT, else
exactMount fails and the storage reads disconnected forever).
- Component versions REMOVED and a note explains why: agent/controller/hub versions
change several times a day, so a number written in an inventory is wrong within
hours and then read as fact -- and the fleet is not uniform (on 2026-07-30 the two
boxes ran different agent AND different controller versions). Points at the
authorities instead: hub /hosts + /configs, felhom-agent --version, docker ps.
- Site addresses now say re-check rather than asserting one (the N100 read .162, not
the recorded .147); records that LAN literals are unreachable from DooPlex while
the boxes are away. Adds the target-selection pointer: this page is what the
hardware IS, that page is what may be done to it.
PROMPT-TEMPLATE.md -- the upstream generator of the defect
- Section 12's "Do NOT touch [the untouchable]" asked the spec author to name a
THING. Now asks for the forbidden ACT plus its REASON, with the demo-hp case as
the worked example of how a bare object-fence over-reads.
- Section 13 gains the positive counterpart, which was the actual gap: if a task
needs a machine to break, NAME IT. Listing only what is off-limits leaves the
most valuable unfenced machine as the residual choice.
runbooks/workspace-CLAUDE.md (+ the untracked root copy re-synced, verified identical)
- Host table gains a Blast radius column and the missing demo-hp row, notes felhotest
as Connection refused, and points at target-selection.md. This is the file that
loads FIRST every session, so leaving it with the old table would have undercut
the whole fix.
No code, no build, no deploy, no host reconfigured or renamed.
- B2 demo-hp + B3 demo-felhom migrated to the island (agent 0.96.0), apps
served throughout (0 container restarts), island /storage 200, LAN DNS pinned
to the LAN IP, hub reports 0.96.0. No rollback.
- capability-map 'site/network change' row PARTIAL -> PROVEN-LIVE
- ROADMAP R-50 -> SHIPPED (fleet-migrated); add R-74 (island on Peti's cluster)
- nodes.md: both boxes island-bound, agent 0.96.0
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0
nested-vm ISO through the real day-0, then ran the R-50 empirical spike:
- vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed)
- F1 replay money shot: LAN move survives on the island; LAN-literal bind
reproduces the 2026-07-20 daemon-exit bug verbatim
- dnsmasq trap confirmed live + lan_resolver.host_ip fix proven
- pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island)
- survival matrix: agent/guest/host-cold-reboot all return on the island
Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM,
REPORT overwrite.
New documentation/operations/nodes.md: HP t740 hardware, disks (SanDisk system
SSD 182195804614; Toshiba 1TB NVMe 58BS11AFT8MQ PRESENT AND UNENROLLED, still
NTFS, do not touch), the five-NIC map and the trap that cost the first install,
and the access path - no SSH key is baked, auth is the hub-vaulted G1
break-glass password.
tailscale.md gains demo-hp, the operator-lab-exception warning so a future
product-shape audit does not conclude the product ships tailscale, and the
--accept-dns evidence: the join omitted the flag, MagicDNS rewrote
/etc/resolv.conf, and it was reverted. Harmless at the vacation site, would
have bitten silently at home where split-horizon matters.
OPEN: key expiry still enabled on demo-hp (2027-01-17) - needs an admin-console
toggle or an API token; a pre-auth key cannot do it.
felhom-pve joins the tailnet as a host package (100.70.170.35, expiry
disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s
tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP
(direct-over-LAN at home, tunnel when away); felhom-pve-lan = LAN fallback.
--accept-dns=false on the host. Host converted static->DHCP (reservation
keeps .162 at home). Measured: --accept-routes hairpins the local subnet
at home -> travel-only opt-in. PBS is offsite (own WireGuard tunnel), needs
none of this. Recorded not fixed: controller<->agent plane hard-pinned to
192.168.0.162 (agent listen_addr + guest bootstrap.json) -> not yet portable
off-LAN. No code changed. Full doc: documentation/operations/tailscale.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XMForrE4c1wZxd9LukxYVt
admin