The recovery codes are shredded with the plant->find->shred->fail-to-find
control the brief asks for, and THE CONTROL PAID FOR ITSELF ON ITS FIRST RUN:
it found the Phase 0 code in ~/.config/credentials as R_CAMPAIGN_11 — a copy
this session did not create and would never have looked for. Without it, a
'codes shredded' claim would have been false. That key was removed from the
shared file with a verified diff (every other line identical, nine keys intact)
and HUB_PW re-tested at hub:200.
Consequence stated plainly rather than left to be discovered:
/home/felhom-repo.orphaned-20260805 (12 535 KB, the three Phase 0 sentinels) is
now permanently unopenable — which is what the set-aside screen promises, and
teardown removes it anyway.
Venue left WORKING and said so: ONLINE, 4 containers healthy, backup target not
degraded, off-site on 2 snapshots. Two things a future session needs: the raw
/mnt/{adatok,mentes} mounts are deliberately left unmounted (R-220's
workaround), and the appliance root credential was shredded — re-fetch it from
the hub.
REPORT-campaign11-phase24.md rather than REPORT.md, per the repo's
parallel-session rule.
No product code changed. No version bumped.
All five daily jobs fired exactly once and completed on time. The 04:15 offsite
run produced snapshot_count 1 -> 2, last_status ok, 24.9s — unprompted, on a box
rebuilt twice and set aside four hours earlier. Nothing on the must-not list
fired, including offsiteheal: that is R-218's NEGATIVE control passing (a box
with a target must not declare), which does not substitute for its positive
half, still owed.
Investigated and DISPROVED: tier2-backup completing in 118ms looked like a
silent no-op; the copy is real and on the backup drive (818.5 KB verified).
Correction to my own pre-registration: backup_run_digest is a TEST FILENAME, not
an event type. The real one is backup_run_failures, a failures digest whose
silence on a clean night is correct — reporting it as a miss would have been a
finding I invented. What survives: the offsite run emits no hub event while both
lesser tiers announce success; failures and staleness are covered elsewhere, so
it is a consistency wrinkle, recorded not filed.
Two absences answered rather than assumed: the restore-test's silence was
PRE-REGISTERED as correct (6h eval, 24h settle); the agent's whole-guest tier is
left explicitly UNRESOLVED because routine local-api calls are not logged at INFO
— a 5h search returns 0 on a box that demonstrably served such calls, so the
instrument cannot answer.
No product code changed.
Invariants across the phase boundaries: I1/I4(product)/I5/I7 held; I3 breached
twice (R-227's raw Bad Gateway, and R-220's refusal naming an impossible
action, reproduced live a third time); I6 breached twice (R-224, R-225). I2 is
recorded as untested rather than passed, because F10 could not be injected.
I4 was breached by the HARNESS, not the product — sourcing ~/.config/credentials
echoed two demo-box recovery codes into the transcript.
RTO unchanged and explicitly not re-measured: Phase 2's faults are not a
re-walk. Phase 1's 61 attended minutes stand; the unaided number stays
undefined.
Teardown recorded as OWED with all three layers named plus the off-site side —
including that the campaign's sub-account now holds TWO repos, the fresh one
and the 12 535 KB set-aside history, and that demo-felhom/demo-hp namespaces on
ep0 must not be touched.
Written BEFORE the soak so the comparison cannot be fitted afterwards. Sources
are the box's own [scheduler] Registered lines and the agent's backup tier armed
lines, read at 23:56.
MUST happen: db-dump 02:30, tier2-backup + fill-watch 03:30, metrics-prune
04:00, offbox-backup 04:15 (snapshot_count 1 -> 2), a backup_run_digest event,
reports every ~15m.
MUST NOT happen: host_stale/host_recovered, offsite_selfheal_restaged (this is
also R-218's negative control — a box WITH a target must not declare),
offbox_repo_orphaned, offsite_repo_key_changed, escrow blob SERVED, any
controller self-update.
Genuinely uncertain and recorded as such: the restore-test has a 24h settle and
every archive here is hours old, so NOT running is correct, not a gap.
The move-aside is correct and verified byte-for-byte; what follows it is not.
settings.json records orphaned_renamed_to and OrphanedRenamedTo has zero
references in any template or handler — 12.5 MB of deliberately retained
customer data at a path the box knows and never shows. Fifth instance of the
project's seam-built-but-never-wired pattern.
Campaign document gains the F7/F9/F10/F11 write-ups: F7 split (set-aside PASS,
afterwards FAIL), F9 partial (R-215's gate proven live on a narrower shape),
F10 NOT INJECTED (harness — three self-healing attempts), F11 PASS.
No product code changed.
F11 PASS ok->stale->ok, an operator mail each way, customer mail correctly
pairing-gated. DOWN (>1h) not reached — stated, not implied.
F7 the set-aside PASSES decisively: move-aside to
/home/felhom-repo.orphaned-20260805, mtime and du -s (12535 KB) and
snapshot f3d9cd67 all unchanged; a fresh empty repo beside it.
NOTHING DELETED.
the afterwards FAILS -> R-228: settings.json records
"orphaned_renamed_to" and OrphanedRenamedTo has ZERO references in
any template or handler. 12.5 MB of retained customer data at a path
the box knows and never shows. Seam built, never wired.
F9 R-215's fix PROVEN LIVE on a narrower shape: with recoveryOffer() false,
GET /recovery now 302s instead of rendering its story. The literal
never-had-offsite precondition was not staged (it needs a rebuild,
which the brief forbids before Phase 4) — recorded as partial.
F10 NOT INJECTED, harness not product. Three attempts, each with a control:
the app, then the controller's monitor, then the run itself recreate
the mandatory directory within ~1 s. Kept as an observation: at
capture the directory held only a recreated metadata.db and NOT the
customer's file, and the run still said ok.
Also reproduced live for a third time: R-220 (candidates empty; unmounting only
the raw mounts flips both drives back) — still OPEN.
Venue left WORKING: 4 containers healthy, backup target not degraded, off-site
fresh repo last_status ok, escrow escrowed.
No product code changed.
R-224 every non-code failure on the unlock path is reported as a statement
about the customer's code. F3 (hub REJECTed) and F4 (agent stopped) both
returned M4 for a CORRECT current code, in 0.056 s and 0.030 s — neither
compatible with a real unseal (~1.0 s of scrypt). The agent's err field
distinguishes fetch-failure from unseal-failure and both become one HTTP
400; the controller has no reach-failure branch. The vocabulary exists
and is wired to agentClient() construction, not to the RPC. R-216's gate
answers source=version and cannot see reachability.
R-225 the store reports 0 snapshots / 0 GB when it cannot read it, above a
card saying it holds backups. Measured over SFTP: snapshot f3d9cd67 and
12 535 KB are really there. snapshot_count/repo_size_bytes are ABSENT
from settings.json after a rebuild — unknown renders as zero.
R-226 M1, the only message that tells a customer to check their typing, is
unreachable on any box that has re-escrowed (M4's branch precedes it).
R-227 a restart mid-unlock returns a raw English "Bad Gateway" (I3).
Recorded against existing rows: R-216 §4.1 now MEASURED twice (and a correction
— SetFloor's DEBUG line can never reach the debug ring, so a restart alone
would not have produced it); R-218's live half still NOT measured, deliberately,
because the venue has a target; R-217 and R-215's fixes HELD under their faults.
R-199's back-pointer was already present in the capability map — the brief's
assumption that it was owed is stale.
No product code changed.
Phase 2 (eleven injected faults) and the §4 positives that were owed.
- §4.1 MEASURED, twice: the box's rendered GetFloor() is 0.200.0, and a
cold-started controller logs "settle-gate: GO — at/above floor 0.200.0"
against the same line reading "floor still unknown" while the hold was in
force. Also corrects the brief's plan: SetFloor's line is u.dbg(), gated on
logging.level=debug and written to the logger, so it can NEVER reach the
debug ring — a restart alone would not have produced it.
- §4.2 still NOT measured, deliberately: the venue has an off-site target, so
needsOffsiteCredential correctly returns false. Recorded, not inferred from
the unit test.
- F1 PARTIAL, F2 PASS, F3 FAIL, F4 FAIL, F5 PASS, F6 PASS, F8 PARTIAL.
F3+F4: a hub outage and a stopped agent are both rendered as "this code does
not open your package", in 0.056 s and 0.030 s — no unseal attempted. The
agent's own err field distinguishes them exactly and it is discarded at the
HTTP boundary; the R-216 capability gate answers source=version and cannot
see reachability.
- R-217's fix HOLDS under exactly its fault (F5), verified with the false-claim
strings absent and accented positive controls present.
hub/CHANGELOG.md: v0.97.1 had no heading of its own — the change was written
into the v0.97.0 entry while the deployed tag is 0.97.1. Given its own entry,
marked as added retroactively. Second occurrence of the class (agent 0.90.1).
Evidence: documentation/tests/campaign11-evidence-2026-08-05/journal-phase24.md
No product code changed.
The first vouch of agent 0.125.0 was REFUSED by R-120's gate: the artifacts form saves as
a unit, and golden 0.192.0 was older than the newest controller the fleet reports. The
golden had been stale since before controller 0.193.0, so the Day-0 manifest had been
effectively unvouchable for days and nobody had cause to notice. Correcting STATUS.md and
OPEN-ITEMS, which said "one dropdown".
Golden 0.201.0 baked in the drill VM, round-trip verified from Gitea
(e730d7cab343eb35...f007654, 658165766 B), teardown clean, 0 token hits in the saved log.
Manifest now agent=0.125.0 golden=0.201.0 min_agent=0.125.0, read back after the save.
The held floor: the HOLD is established positively (hub log + the box's own "floor still
unknown"). Its RELEASE is not — no HELD line and no dashboard reason are both absences,
the served path logs nothing by design, and the box-side positive needs a restart the
venue cannot take before Phase 2. Recorded as deduction plus a pinned test, not a live
positive.
Registers and evidence for the campaign and its fix pass.
OPEN-ITEMS: R-214..R-223. Six SHIPPED (R-215/216/217/218/219/222); three deliberately
still open and each blocks a real flow (R-214 console banner, R-220 drives unenrollable
after a rebuild, R-221 a rebuilt box cannot run the escrow ceremony); R-223 minted and
WAITING-ON-OPERATOR (vouch agent 0.125.0). R-213 and R-202 untouched.
Capability map: a new row for the customer's UNAIDED journey, recorded FAILED and staying
failed until a re-walk passes — fixes are not a journey. The existing rebuild row is
corrected where it said R-198's retention was unit-proven only: it was proven in production
on the first supersession since the fix, identity_blob retained at 572 B byte-length exact.
CLAUDE.md comment-vs-code table: eighth entry — ResolveManagedFloor, the first where the
false invariant was a GUARD rather than a comment alone.
STATUS: the headline is now "the backup promise is proved, the recovery journey is not",
and the one thing waiting on the operator.
- OPEN-ITEMS: R-193 CLOSED with both 2026-08-05 rulings (unlocking and restoring
are separate; 'I do not want the old data' moves the store aside after a double
confirmation), and the shape-(b) reasoning — WriteOffboxSecrets auto-generates a
repository password on re-apply, so the literal 'fresh data area' trigger would
have opened a window that closes by itself.
- R-213 MINTED (R-212 was and still is the highest, re-checked for the second
writer): putting files back in place, with the live-versus-backup comparison
named as its requirement. Not started, deliberately.
- capability map: the 'needs someone who knows to look' qualifier is GONE; what
remains is stated narrowly — no correct-code run through the page, the put-back
is out of scope, and the journey has not been re-walked end to end.
- 07-backup-architecture 7.0: a fifth row, and where the screen deliberately stops.
- CONTEXT: standing ruling S-34.
- STATUS: the headline change and the two things still owed as proof.
No hub change and no hub bump.
The register said 'two set-aside stores, ~1.2 GB'. Measured before touching
anything: THREE set-aside stores totalling ~1.45 GB, and the thing that was
exactly 1.2 GB was demo-felhom's LIVE felhom-repo. Matching on the size would
have deleted a working repository. The operator was shown the corrected list
and confirmed 'delete all three'.
Deleted: demo-felhom orphaned-20260717 (1.4 G) + orphaned-20260718 (3.0 M);
demo-hp orphaned-20260804 (43 M). Both LIVE repos untouched, confirmed by full
listings before and after on each account.
Proof nothing live was caught: a real off-site run on demo-hp immediately
afterwards returned status ok, orphaned false, no error, 6 snapshots.
Method note recorded for the next session: the storage box has a RESTRICTED
shell. 'test -d X && rm -rf -- X' returns 'Command not found' and does nothing
(it failed CLOSED, verified by an unchanged listing); 'rm -r <path>' as one
simple command is the working form.
- OPEN-ITEMS: R-204 all four CLOSED with both 2026-08-05 rulings recorded (the
declared-state trigger and its four-meanings-of-absence reasoning; the recovery
preview's dashboard-password exposure accepted as metadata, not content).
R-193's credential half CLOSED, screen + deletion still open. R-192 CLOSED by
REPLACEMENT. R-202 untouched.
- R-212 MINTED (R-211 was the highest, grepped): the orphaned-ciphertext deletion
HALTED at its STOP because the measured paths do not match the register — three
set-aside stores totalling ~1.45 GB, and the thing that is exactly 1.2 GB is
demo-felhom's LIVE repo. Nothing was deleted.
- capability map: all four interventions closed; the row KEEPS a qualifier for a
new reason — no step needs an operator, but there is no customer-facing recovery
screen, and the journey has not been re-walked end to end.
- 07-backup-architecture 7.0: the four-step table updated; the declaration-vs-
inference reasoning and the credential-automatic/key-customer-present split.
- CONTEXT: standing ruling S-33.
- STATUS: the headline change and the deletion STOP.
- REPORT-r204-item4.md rather than REPORT.md: a parallel session is active in this
shared clone.
Operator ruled "proceed" after reading the pre-analysis; the storageReserved
condition CC made the move conditional on was applied with it. The pre-analysis
is retained verbatim as §11-bis so the ruling can be audited against what it
was given.
MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four
independent observables: 550,891 = 550,891 filesystem entries; 448 = 448
trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is
why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical
meta.db sha256. Verified again after: identical image/tag/volume ID SETS,
cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables /
175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the
relocated store, rc=0 — not an inventory count.
k3s was never at risk, and that was established BEFORE stopping anything: it
runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and
~160 pods stayed up. Only the two jarr-* dev containers were affected.
TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part:
RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd
started normally against an absent-but-unmounted path. A typo'd guard buys
nothing and says nothing: the built-but-never-wired shape again. The guard was
therefore verified positively at the unit level (Requires= AND
After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely
absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live
Longhorn replicas and must never be unmounted, and editing fstab on a
production host risks emergency mode at boot. Result:
"Job containerd.service/start failed with result 'dependency'", is-active
inactive. It refused rather than creating an empty store on the root disk.
storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%.
NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the
move has not survived a reboot — which is this task's own standard for
validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts
/mnt/ssd_2 before containerd starts, in this host's actual ordering) is not,
and mount-ordering reasoning is exactly the class this project has been burned
by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why
/ sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot,
enabled, dry-run PASS at install — validates it automatically at the next boot,
asserting positively that /var/lib/containerd does NOT exist. Read
/var/log/felhom-store-postboot-check.log after the next boot; on PASS the
rollback copy can go and ~34 GB returns.
Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
Spike output only; no production Go code. The one shipped change rides in
homelab-manifests 6808a4b (R-205, the monitoring rule).
VERDICT: mechanism confirmed, with one correction and one refutation.
- CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict.
Proven by naming a 440 MB `go mod download` record present at build N and
absent by N+2 — not by absence of an error.
- CORRECTED: honoured ONLY in the `policy` array form. The flat form is
SILENTLY ignored — daemon starts, logs nothing, keeps its defaults.
`dockerd --validate` returned "configuration OK" for a bogus key AND for a
config that then crashed the daemon. The oracle is `docker buildx inspect`.
- REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB.
The 181.4 GB belongs to the system containerd (`root` in
/etc/containerd/config.toml).
P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem
returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY
— / went 86% -> 53% used, and Longhorn's default disk went
Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%).
P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download`
records had Usage count 1. Isolated by controlled builds — same VERSION build-arg
-> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG
GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes
the stage environment. Both Dockerfiles have it. One line each to fix -> R-208.
P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is
safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn
disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move
lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC
recommends against it unless ~80 GB is reserved on SSD2 -> R-209.
Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker
ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator),
R-211 (Prometheus has no config-reloader — rules changes have never applied
until something restarted the pod; found while verifying R-205).
Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and
its dependency stated. Header restates that R-202, the 1.2 GB ciphertext
deletion and R-198's still-unit-proven retention all REMAIN OPEN.
- capability map: the recovery row keeps its 'with a person present'
qualifier, names which crutch remains, and cites the three now gone.
- 07-backup-architecture: new 7.0 - what a customer can and cannot do
ALONE, the four steps in a table with status. This is the section a
future reader will use to answer that question.
- CONTEXT: standing ruling S-32, superseding S-31 steps 2-5.
- STATUS: rewritten to one screen per its own header; removes a corrupted
half-overwritten section left from the drill session.
- ROADMAP: R-196 and R-204 collapsed.
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
Read-only recon of the escrow -> recovery chain, from a dead node to an open
repository. No production code, no build, no version bump.
Headline: the hub's superseded-escrow retention does NOT retain the offsite
repository password. host_escrow_superseded has no identity_blob column and
demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a
supersession is the PBS datastore key, not the restic repo password. The next
escrow ceremony -- which the system tells a rebuilt box's customer to run --
destroys the last copy. Both demo boxes crossed that line on 2026-08-04.
Also established:
- the hub's blob-serving endpoints (re-enroll / restore-directive) have zero
callers anywhere: agent, hub UI, scripts, runbooks (R-199)
- POST /backup/offbox/inject-password is routed and handled but no template
contains the form (R-200)
- nothing in the recovery path has ever been exercised; the one live
round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201)
- a fail-closed mint refusal IS implementable: the report ACK already carries
escrow{identity_blob_present, restic_pw_sha256} and the controller discards
it whenever no offbox target exists
Corrections: yesterday's spike annotated (candidate (b) overturned in part --
unattended recovery is impossible, customer-present is not); capability-map
retention claim struck through and replaced with what the code does.
Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.
Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0,
reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings
running. The existing down-skip could never cover it: it reads the staleness
checker's state, which is seeded from a query over the `reports` table, so a
customer that never reported has no state at all and GetState() returns "" rather
than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is
consulted once per customer at the top of the deadline loop. The discriminator is
"was a host EVER bound", never "has a report arrived" — a box installed and never
heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof
observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the
event david sent.
Parts 0-3 (spike, NO production code for R-193/R-192):
audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot
provider password is the RECOVERABLE secret and the restic repository password is the
irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous
off-site history. Measured without touching a box, by comparing
host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed
(demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's
"lucky" 76-second recovery restored delivery and not the repository, silently, for 13h.
ReissueCredentials does NOT rotate the restic password (R-39's record and two hub
comments are wrong -> R-196); candidate (b) is not implementable against a
zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong
event. Ends in ranked options and an unanswered question for the operator.
R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
The 20-minute latch expired at 10:20:29 and the hub logged degraded -> ok
(agent_capability_recovered) at 10:30:40. Final state on both boxes: agent
0.124.1, two ACL rows on /storage/felhom-backup.
CI run 28 failed and run 29 passed on the same sha. The tag was created locally at
07:38:36Z; run 28 started 07:38:31Z (commit push) and run 29 at 07:38:41Z (tag
push). The old race was 'tag before package'; the new one is 'package before its
tag is pushed' — the window the converse probe exists to detect, and it did.
Not confirmed from the run log (the endpoint 404s), so recorded as the
timing-consistent explanation rather than a proven one. The gate passes locally
with all 7 versions installable and none untagged. Both obvious closures are the
'make the gate blind for a window' shape this row refused, so the options are to
accept a seconds-wide window or to make releases quiet to CI another way.
R-191: the gate was run first and it mattered — ep0's prune jobs have run daily
since 2026-07-27 (18 tasks, all OK). THREE of my own queries said the opposite and
all three were broken instruments (worker-type vs worker_type, prune vs prunejob,
proxmox-backup vs proxmox-backup-proxy); acting on any would have disabled the only
pruning attempt while reporting that nothing prunes.
R-190: the row now reflects the MITIGATION, not the cause, and says so. Includes
that v0.124.0's transition record failed in production while every unit test
passed, and Part 3's single pass — the token-rotation class is real and documented
in our own installer but does not fit this box (same token still authenticates, the
other three grants survived, no installer run evidenced); ruled out on this box, not
fleet-wide.
R-194: PVE caches permissions (~40s and ~16min measured), so a single permission
read is a lagging indicator and may explain when R-190's refusal surfaced.
07-backup-architecture: the offsite retention row was RIGHT and the configuration
disagreed with it — noted, since a doc that states a contract does not enforce it.
Re-issue through the designed endpoint: fresh password staged 07:11:51, settle-gate
GO, consumed 15s later at 07:12:06, offsite configured at 07:12:09 against the SAME
sub-account (275124) and repo path. Both boxes now escrowed.
CORRECTION to this session's own reading: the escrow did NOT recover by itself.
escrow_state went pending -> escrowed 15s after the apply and I inferred automation;
the operator had run the ceremony. It needed a human on both boxes — demo-hp at
07:16:02, demo-felhom at 07:20:28 (its escrow had been pending since its own rebuild
the day before). A 15-second state change is not evidence of automation, which is the
same class as reading an absent log line as success.
Snapshot continuity is NOT established: both boxes report snapshot_count 0, but the
run-history keys are ABSENT rather than zeroed — a controller that has never run an
offbox backup, not an empty repo. The 04:15 run decides it.
Operator confirms no hub-side offsite config change, so the regression was not an
action. Evidence: demo-hp's controller went 0.187.0 -> 0.192.0 at 06:12:18 with a
new config hash and the agent re-keyed its leaf three minutes earlier — a guest
rebuild. The last pre-rebuild report shows the tier fully healthy: escrowed, last
success 02:16:39Z, 15 snapshots, 40.9 MB. No offsite object in the 108 reports
since.
Mechanism: the restic credential is delivered once. demo-hp consumed its secret on
2026-07-23; the rebuilt controller has no copy and no way to request another.
demo-felhom survived the SAME rebuild only because its secret was still unconsumed
— it consumed it four seconds after its config hash changed and was reporting
offsite again 76 seconds later. That difference was luck, not design.
Also sharpens R-192: the self-heal's guard refuses when any report since the
consume carried an offbox target, but that query reads the OLDEST 500 reports —
all of which predate the rebuild. Healthy history before a rebuild is not evidence
the credential still works, which is why the automation that exists for this case
declined to act.
demo-hp has been emailing daily since 2026-08-03 06:12 UTC that '500 reports
since carry no offbox target — the credential is likely burned'. Measured against
the hub's own data: all 500 of those reports DO carry one. The credential was
consumed 2026-07-23 and applied successfully; the controller reported an offsite
object continuously until 2026-08-03 05:59:21, then it vanished at 06:12:19 and
has been absent for 108 reports. A regressed apply, not a burn.
Two defects. maybeEmitStuck builds its message from ReportsSinceConsume while
hardcoding 'carry no offbox target' and never consults OffsiteReportsSinceConsume,
so the recommended action targets a failure that did not occur. And maybeHeal
refuses SILENTLY on exactly that field ('regressed-apply shape -> operator's
call', a bare return), so there is a daily email with the wrong story, no heal,
and nothing saying why the heal declined.
The underlying condition is real: that box currently has no off-site copy of its
customer app data, since 08:12 CEST on 2026-08-03. What removed it is not
established.
Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite
540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest,
booted, verified and destroyed itself; zero 990000 guests or volumes afterwards
and both local-lvm figures returned to their pre-run values.
Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for
the first time: never-proven sorted first, each box took its HOST tier, deferred
the offsite one, and picked it up on the next evaluation six hours later. The
host-tier proofs reached the hub through R-189's merge — demo-felhom's report
carries two tiers, and the local one can only have come from disk.
The capability map's optimistic half is cashed, with its scope stated: these two
boxes, not the fleet.
Surfaced and filed rather than fixed:
- R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with
a reinstall, any logged pveum activity and any cluster-log entry ruled out.
- R-191: every weekly offsite backup uploads successfully and then fails the job
on a prune the box is deliberately not allowed to do (R-89 moved it
server-side; both boxes still arm keep_last=2).
Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the
write path (six, with the hub raising whole_guest_backup_failed at the first), and
my earlier "no restore_test_* events" was produced by grepping a 404 page.
CORRECTION 1 — the runbook annotation and the R-185 row both said the drift did
not surface as a 403 because writes go through a root path. That is WRONG.
demo-felhom's local-api backup jobs 403'd six times between 09:24 and 17:34 CEST
on exactly that storage and privilege, and the hub raised whole_guest_backup_failed
at the first with edge-triggering suppressing the rest. The impact was not only an
unreadable tier: the agent's own whole-guest backups to it were failing.
CORRECTION 2 — on this box the grant was LOST, not never issued. A vzdump by the
agent's token to that storage completed OK at 04:44:50 the same morning; the first
403 is 09:24:56. Ruled out by measurement: a host reinstall (uptime 12 days), any
pveum/ACL/user.cfg activity in syslog 04:00-10:00, any ACL entry in the cluster
log. Correlated but not established: guest 9201 was reprovisioned nine minutes
before the first failure.
R-190 files the unexplained disappearance, and notes that the new store-grant
probe detects the STATE but says nothing about the TRANSITION.
- OPEN-ITEMS: R-185 closed with the measurement, the corrected root cause (the
installer's Scenario-F reuse arm, not PVE_STORAGES), and the live sequence.
Records that demo-hp carried the same drift and was fixed too.
- capability map: the whole-guest row's HOST-tier half was OPTIMISTIC and now
says so — that tier was not merely unproven, it was unprovable on both demo
boxes, and every live proof cited was on the offsite tier.
- vzdump-target-move runbook: its item 5 predicted this; annotated (not
rewritten) with what actually happened — the create arm did grant, the reuse
arm did not, and it surfaced as a silent unreadable tier rather than the 403
the item expected, because vzdump writes through a root path.
- CONTEXT: S-21 (an empty listing cannot distinguish forbidden from newborn; the
measured trap that an ungranted path answers with INHERITED privileges) and
S-22 (the Scenario-F arm must finish the job).
- STATUS: rewritten for the operator, back to one screen.
- OPEN-ITEMS: all three rows closed with their measurements. R-189 records the
migration consequence seen live (a pre-R-189 record has no tier and is not
reportable, so upgrading does not retroactively make an old proof visible).
R-188 records both directions demonstrated and the fixture cleaned up. R-186
carries the before/after sha values including the CGO_ENABLED discrepancy.
The priority list states explicitly that R-185 is the one left open from that
group and that this session did not touch it.
- capability map: the restore-proof row now says the evidence path itself had a
gap — the proof reached the hub only because no restart intervened — and what
closed it.
- CONTEXT: S-19 (a proof is durable and reportable, a failure is neither, and
why) and S-20 (the release ordering, what each step protects, and how a binary
is verified).
- STATUS: rewritten for the operator, back to one screen.
The agent reports restore_tests[] from an IN-MEMORY store ('lost on restart; the
cadence re-populates'), and the hub report interval is 900s. A restart inside
that window drops the result. That used to self-heal within 24h because the next
cadence re-tested the tier; under per-archive due-ness the agent will NOT re-test
a proven archive, so the hub can stay ignorant until the next archive generation
— a week on the offsite tier.
Observed today: the 15:25:14 PASS on felhom-pbs (a real 14.5 GB offsite restore)
reached no host-report at all, because the agent was restarted 2m43s later for
the v0.121.1 deploy.
The persisted proof already exists — RestoreTestState.Snapshot() even carries the
comment 'for the host-report gauge' and has NO production caller: a seam built and
never wired, and an invariant asserted in a comment with no test, in one method.
on: [push] fires the gates workflow on the TAG push too, and release-agent.sh
pushes the tag before publishing (deliberately). CI can therefore run the
published-versions gate inside the window where the tag exists and the package
does not, and correctly reports 'every released agent version must be
INSTALLABLE' for a release that completes seconds later.
Measured across two releases in one session: v0.121.0 -> runs 12/13 (success,
failure) on the same sha; v0.121.1 -> runs 17/18 (failure, success) on the same
sha; and one pair both green — a race, not a rule.
It matters because R-168 made CI email on failure so a red gate cannot be missed.
A signal that cries wolf on every second correct release is how that mail becomes
something you archive unread.
- OPEN-ITEMS: R-86 CLOSED with the trap in its own wording recorded (the literal
reading is never true on a daily tier); R-87 re-ranked UP because R-86 built
most of what it waited for; R-185 (the agent cannot list demo-felhom's host
backup tier — a missing storage ACL, pre-existing), R-186 (a released binary's
sha is not reproducible from its tag), R-187 (R-115's publish leg had never
actually run) filed. R-184 was the highest ID in use.
- ROADMAP: R-86 collapsed, keeping the reasoning and correcting the shape the row
itself proposed — which would have been the never-fires version.
- 07-backup-architecture: new contract section — restore-testing is per ARCHIVE
GENERATION, with the trap and what did not change (S-1).
- 00-capability-map: the unattended restore-proof row upgraded to PROVEN-LIVE on
the 635 s due-triggered offsite run, with the restart and teardown evidence.
- CONTEXT: S-17 (the rule, the trap, the config key, the hub's derivation) and
S-18 (ep0 is Tier 2 — extends D-d's protected list to three machines).
Numbered 17/18 because S-14 and S-15 were already duplicated in the file.
- STATUS: rewritten for the operator, trimmed back to one screen.
Ships WITH agent v0.121.0, not after it. The agent now proves a tier once per
ARCHIVE GENERATION, so a weekly tier is proved weekly — in perfect health. The
flat 7-day restoreProvenStaleAfter derived its number from the 24h cadence R-86
removes, and a healthy weekly tier's proof age reaches EXACTLY 168h just before
its next proof: it sat ON the line, so any ordinary delay tipped it into a
nightly alarm about a working system.
restoreProvenWindow(tier, observed, ok):
- the tier's own archive interval, OBSERVED from reports the hub already holds
(pbs_snapshots + successful backups attributed by TARGET TYPE, slice A.4)
- x4 generations = the same tolerance the flat constant expressed
- floored at 7d (never tighter than before), capped at 12d (strictly inside the
2-week offsite retention)
- falls back to the DECLARED rhythm (26h host / 8d offsite — the thresholds the
backup-freshness checker already uses) when history is too short to observe
one; falling back to the FLOOR would recreate the false alarm on a fresh box
Kept: absence is UNKNOWN until the anchored window passes; the signal stays
edge-triggered; failed and stale remain distinct events. Every reason string now
states the window it was judged against (R-100's corollary).
Also backfills the missing v0.90.1 CHANGELOG entry (deployed since f21e7ca), and
records the operator's 2026-08-03 ruling that ep0 is Tier 2 / protected.
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.
Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.
A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.
ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.
Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.
R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.
target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.
STATUS.md 127 -> 83 lines, items rather than sentences.
R-182's direction REVERSED by Part 0's measurement. Filed yesterday as "the
reserve re-alerts on every status refresh" — too many alerts, seen at the
sending end. Measured at the receiving end: 9 events received today, 2 operator
emails sent. When two apps are refused in the same second the operator is told
about ONE; the other is dropped before LogNotification, so it leaves no row on
any channel and cannot be audited. The operator cooldown key is
customerID:eventType(+tier) and the capture-failed event carries `app` but no
`tier`, so the key has no app identifier. Same failure mode as R-97a, in a
second event type that never opted into the narrow fix. Nothing changed —
Part 0 was investigation only.
Correction owed: yesterday's report said "one recovery_unit_capture_failed per
app, HTTP 200". True of what the CONTROLLER pushed; a reader would take it as
"the operator was told about each app", which is false.
R-110 CLOSED (installer v1.23.0). Both channels moved. The spec's mechanism for
channel 2 rested on a factual error — the run-time fetches are sixteen, not
nine, and come from felhom-agent, not this repo — so no tag here could cover
them; pinned to the agent version being installed instead, on the operator's
ruling. Channel 3 needed no change: the URL never carried a ref, so no hub
change and no hub bump.
R-115 CLOSED. release-agent.sh builds, tags, publishes and verifies by an
independent download; check-published-versions.py refuses a tag with no package;
CI now runs the full gate set so it actually runs.
R-183 NEW+CLOSED: a fresh install fetched the vouched agent binary and its
sixteen config files from two different refs, and nothing compared them.
R-184 NEW: nothing stops the hub vouching a version that was never released.
The R-115 gate cannot see it — measured, the hub manifest and Gitea's package
listing are both 401 anonymously.
capability map: new PROVEN-LIVE row for the published installer channel.
STATUS.md 138 -> 127 lines.
Syncs documentation/runbooks/workspace-CLAUDE.md with the workspace root file.
Row 7: the B2 refusal claimed the previous unit was untouched; nothing-deleted
held, untouched was measured false.
Also records WHY it survived review: it passed a full green suite AND three of
its own red-proofs, because every one of them asserted the mechanism inside
captureAllRecoveryUnits and none asserted the consequence across the whole
backup run. The test that would have caught it is the one the fix ships —
fingerprint the tree before and after, and compare.
R-181 CLOSED (controller v0.193.0 + v0.193.1) and proven live on demo-hp for
BOTH reserve terms. The reserve is now a per-app, per-run ADMISSION decision
taken before the app's first write and covering all three write legs, and it
gained a size term. The refusal's wording was not weakened; the behaviour moved
so it became true, verified by sha256 tree fingerprint.
R-156 CLOSED — papra's template mounts the app's own data root. Precondition
re-measured rather than inherited (both boxes were wiped today).
Part 4, documentation only, nothing built:
- R-110 WAITING-ON-OPERATOR -> READY. Ruling: option (b), the installer's
publish channel moves to a TAG. Recorded with the condition that decides
whether it works at all — it must cover BOTH the /scripts/ git-sync and the
nine files the installer fetches from raw/branch/main.
- R-115 WAITING-ON-OPERATOR -> READY. Ruling: mechanism (b), a build-side gate
refusing to deploy or vouch an unpublished version. The third instance (agent
v0.120.0) would have silently downgraded both demo boxes while succeeding.
R-182 NEW: the periodic status refresh has no admission scope, so a refused app
re-alerts on every poll (measured: a second alert pair 13s after the run's).
Pre-existing in v0.192.0; deliberately not fixed in the R-181 task.
capability map: the local-backup row moves to PROVEN-LIVE in BOTH halves.
ROADMAP: R-165 collapses to CLOSED; R-181 collapsed into it.
07-backup-architecture.md: the reserve's contract stated as what the code
provides (S-1 — an architectural contract changed in the same session).
STATUS.md trimmed 150 -> 111 lines, "What's broken" no longer holds shipped
work, and the stale "After:" line (pointing at work that shipped on 2 August)
is fixed.
Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and
the delivery route rather than one of them twice.
demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom,
backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of
its subdirectories via fstab; one df figure and one device id (64519) on all
three paths; reboots 3/3 with the binds surviving each.
demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding
against the vouched manifest for BOTH artifacts -- 'verified sha256
54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for
the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3.
Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore,
with a planted marker returning byte-identical on each box. Ceiling measured
gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45.
R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that
row records as the bulkhead's replacement, fired live for the first time and
does refuse per app, delete nothing and alert -- but it is checked only in
captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its
'the previous unit is untouched' claim was measured false (182,272 B dump
replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181.
New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage
not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is
rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished).
No code written, no version bumps -- this was a runbook.
07-backup-architecture.md gains §7.5.1 (S-1: the contract changed in the same
session): the ceiling §7.5 describes no longer exists for a box built from
golden >= 0.192.0, the bulkhead's replacement is recorded, and R-175 is FIXED
here rather than left standing — the bound is restated as a function of mp1
and scoped to split-layout boxes, naming all three real shapes.
Capability map: new row as IMPLEMENTED, deliberately NOT proven-live, with
the missing leg named — no box has been reinstalled from the golden, and
"the golden baked" is not "a box built from it works".
R-163 CLOSED: the ceiling it recorded stops existing. R-176(a) answered by
P1; (b) WITHDRAWN, since every node is reinstalled rather than migrated.
R-178 filed for the reinstalls, which were not done this session.
CONTEXT S-13 (the variant chosen on measurement; pruning rejected with its
reason) and S-14 (prove first, then vouch — the golden is published but
deliberately unvouched, because vouching is what makes a fresh install pick
up a layout no box has been proven from).
STATUS: plain-language section; both operator questions now answered, so the
waiting-on-you item is cleared. Two older entries trimmed so the page did
not grow.
P1 PASS: a pre-merge archive (mp0+mp1, confirmed from its own vzdump log)
restore-tests clean on demo-hp with mount_parity ok in 84s. mountParity was
not touched. Limit stated: run with the current agent because the merged one
does not exist until after the STOP; the comparison is archive-vs-its-own-
restore and never consults the host layout, so it carries provided Part 2
honours its constraint not to touch the restore path. Re-run after Part 2.
P2: all three probed variants are mechanically clean — both paths writable,
ONE df figure, dockerd 3/3 reboots, /mnt propagation, and a container's
statfs(/) reporting the merged volume. The task's flagged ordering risk for
V-b did not materialise. They are separated by SCOPING instead:
V-a container sees /mnt = 8.0K but customer data sits inside Docker's
data-root, so clearing /var/lib/docker destroys every local unit
V-b container sees Docker's ENTIRE data-root under /mnt (17.9M on an
empty box), making the bootstrap's own scoping comment false
V-c neutral mount at /var/lib/felhom, both paths binds — breaks neither
V-c was probed because the measurements showed each named variant violates
a different documented invariant. It is offered as a measured option for the
operator, NOT adopted.
Teardown all three layers: 9401/9402/9403 destroyed, 5.19 GB returned, and
the hub registers verified unchanged (5 customers, 4 hosts).
R-167/R-158 shipped and proven live (controller v0.191.x, hub v0.89.0):
two new capability-map rows PROVEN-LIVE with live citations, and
07-backup-architecture.md §7.5's closing claim "nothing warns when an app
crosses the line" is now false and rewritten (S-1: an architectural
contract changed in the same session). §7.5 also gains the caveat that its
size bound is ONE BOX'S, not the fleet's.
Part 3 SPIKE (audits/SPIKE-r165-mp1-merge-2026-08-02.md): M1-M5 measured,
NO layout touched. Three findings the merge session must not re-derive:
"the layout" is not one thing (200G/50G vs 50G/20G vs 16G/8G); mp1 is a
BULKHEAD and not only a ceiling, so after the merge an overflow reaches
/var/lib/docker; the golden fails closed on the split in four places.
D-a's condition (1) is currently SATISFIED — no external box is in the
hub's register, and both demo boxes are Tier 0 and reinstallable.
Recommendation given, choice NOT made — it ends at the operator's ruling.
CONTEXT.md S-11 (D-c's routing, and why R-158's own backup_failed proposal
was overruled) and S-12 (the monitoring landed BEFORE the merge).
STATUS.md gains the plain-language section and the merge decision, with two
older entries trimmed so the page did not grow.
New rows R-174 (closed same session), R-175, R-176, R-177; each ID grepped
free before minting.
R-172's root cause was not tuning — the WAL/busy_timeout pragmas had never been
applied, because the DSN used mattn/go-sqlite3 syntax against modernc.org/sqlite,
which ignores unknown parameters without an error. Recorded that way so nobody
re-reads it as "SQLite was slow".
R-173 NEW: while establishing who copies hub.db for the WAL change, found
pvc/hub-data labelled recurring-job-group.longhorn.io/default: disabled, with
backup-daily and backup-weekly the only recurring jobs and both on the default
group — so the hub database has no volume-level backup, and it holds every box's
break-glass root password plus the escrow custody records. Filed, not fixed:
whether the exclusion is deliberate is an operator question.
The session report is REPORT-r172-hub-wal.md, not REPORT.md, per the
parallel-session rule — REPORT.md belongs to the controller session that ran
immediately before this one.
It also records, plainly, that a 60-concurrent load test I ran OOM-killed the hub
pod three times against a 256Mi limit. Not the WAL change, and not a test I
should have run against a Tier-2 box; the unit tests already proved the property.
store.New opened the DB with `?_journal_mode=WAL&_busy_timeout=5000`, which is
mattn/go-sqlite3 syntax. The driver is modernc.org/sqlite, whose applyQueryParams
reads only _pragma/_time_format/_time_integer_format/_txlock/_inttotime and
IGNORES anything else WITHOUT AN ERROR. So the hub ran in rollback-journal mode
with busy_timeout=0 for its entire life while its own source said otherwise.
Surfaced as a false HOST STALE banner: in rollback-journal mode a reader excludes
a writer, so rendering an operator page blocks a host report; the hub 500s, the
agent waits its full 15-minute interval without retrying, and staleness fires at
30 minutes — two collisions is a false alarm plus an operator email. 13 collisions
in one pod lifetime; the alarm fired twice on 2026-08-02 for a host that was up
two days and reconciling throughout.
The observable that proved it: a 128 MB /data/hub.db with no -wal/-shm beside it
while the DB was open.
Fix: ?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate.
_txlock=immediate is not optional — database/sql's Begin() is DEFERRED, so a
read-then-write tx must upgrade its lock and a failed upgrade is
SQLITE_BUSY_SNAPSHOT, which busy_timeout does NOT retry; this store has 10+
db.Begin() sites and they are all write paths.
Every test asserts what the DATABASE reports, never the DSN string — a string
test would have passed for the whole life of the bug. Red-proof: restoring the
shipped DSN reproduces journal_mode="delete", the missing -wal, and the live
"database is locked (5) (SQLITE_BUSY)".
Operational consequence handled: a WAL DB cannot be copied by taking hub.db
alone — a bare `cat` opens cleanly and silently omits the newest writes. The
break-glass retrieval in operations/nodes.md used exactly that; it and the
recovery-inventory note are now WAL-aware.
The hub's /data/hub.db (128 MB) is in rollback-journal mode, not WAL, so a UI
render can block a report write; the hub returns 500 on SQLITE_BUSY without
retrying, and the agent waits its full 15-minute interval rather than retrying.
Staleness fires at 30 minutes, so two consecutive collisions produce a false
host_stale and an operator email for a healthy host. Observed twice on
2026-08-02 while the agent was up 2 days and reconciling throughout.
Pre-existing: 13 collisions in one pod lifetime, first ~3h before that day's
controller work, though a burst of restarts amplifies it.
admin