CI checks out one repo shallow, so the golden-currency gate found no sibling controller clone, exited 2 (INCONCLUSIVE) and turned CI red on every push. Caught by pulling the run status rather than assuming it. Fixed by fetching the controller repo in CI - not by letting the gate skip, which would have been fail-open and would have left it running in neither of its automated homes. Runs 244 and 245 confirm green. Also records the reusable lesson: the pre-push hook and CI do not see the same filesystem, so a gate that reads a sibling repo can pass locally and break CI.
15 KiB
REPORT — clearing the ground before the next walk (2026-08-08)
A sibling report: REPORT.md is overwritten per-session and a parallel session shares this clone.
Golden 0.206.0 baked and VOUCHED · the R-242 gate built and shown red→green · demo-hp's stale flag
found wrong and cleared · STATUS.md 258 → 87 lines. felhom-controller and felhom-agent untouched.
1. Part 2's gate — FAILING first, then passing
Shown failing against today's state, before anything was baked. That ordering was the instruction and it is the gate's own red-proof:
newest released controller : 0.206.0
newest golden baked : 0.205.0
GOLDEN CURRENCY GATE FAILED: controller v0.206.0 is released and NO golden carries it (newest bake is 0.205.0).
A machine installed right now would receive v0.205.0 — the release is written, tested and pushed, and NOT delivered.
Entry point exit 1; summary CONVICTED: golden-currency. After the bake, the same command:
newest released controller : 0.206.0
newest golden baked : 0.206.0
golden currency gate OK
⚠ THE INTRODUCING PUSH USED --no-verify, to get past the gate's own conviction. Stated here, in
scripts/CHANGELOG.md and in the commit message rather than worked around. The alternative — baking
first so the gate had never been seen red — was explicitly rejected: a gate that has never been seen
failing has not been shown to work. Every later push in this session was clean.
What it does not catch, and why the design is what it is
It checks the BAKE, not the VOUCH. Both .githooks/pre-push and CI run repo_gates.py --fast,
which by contract selects only gates touching no network — so a hub-reading gate registered as
non-fast would run in neither, which is exactly the R-29 census failure this runner was built to
end. And the vouched version lives only in hub_settings, with no copy in git; putting one there
would create a second source of truth that can drift, and a green gate over a false claim is the
worst outcome available. So a bake without a vouch still passes. That half stays open on R-242
rather than being papered over.
It compares versions, not behaviour, so a release that changed nothing customer-visible also trips
it. Accepted deliberately, and stated because a gate that cries wolf is one people learn to bypass:
judging "customer-visible" by hand is precisely what failed three times (R-239, R-242, and this
recurrence), and the cost of a false trip is one bake — the operation the project wants routine anyway.
A waiver belongs in the register, never in a habit of --no-verify.
Inconclusive (exit 2) on an absent controller clone or an unparseable header: not knowing is never a pass.
2. Part 1 — the stale blob, all six questions
Read-only throughout. Nothing was cleared during the spike — the clearance in §3 came afterwards, on the operator's explicit approval.
Q1 — what set it, and when. MEASURED, traced to an act, to the second. At 2026-08-04 20:15:49
the hub emitted offsite_reissued and escrow_stale in the same second — an operator Re-issue,
three minutes after escrow_blob_served at 20:12:40 and 20:12:54, i.e. during the R-201 recovery
drill. That is offsite.ReissueCredentials's precautionary MarkEscrowStale call, which hub
v0.95.0 removed the next day (R-196 / R-204 item 2) for marking healthy escrows stale. So: the
drill's own Re-issue, by code that no longer exists. (Both 4-August candidates named in the task
were live that day; the events separate them.)
Q2 — is the flag correct? MEASURED: NO. The hub's blob seals
restic_pw_sha256 = 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a; the key the box
is actually using hashes to the identical value. The blob covers the key. The flag was wrong from
the moment it was set.
Q3 — what clears it? MEASURED: nothing, by itself. The only writer of stale_at = NULL is
SaveHostEscrow's ON CONFLICT — a fresh escrow ceremony, which is the one act that would
supersede the good blob. The only exit from the false alarm was the destructive act the false alarm
recommends. No timer, no self-heal, no reconciler touches it.
Q4 — who can see it? Said plainly: effectively only a database read.
| audience | what they see |
|---|---|
| the customer | a card, but stating a false reason (see below) and recommending the destructive act |
| the box | nothing — report.EscrowStatus has no Stale field, so it cannot see the flag at all |
| the operator | one page: the PBS-DR view (hub/internal/web/pbsdr.go:487) — the wrong tier for an off-site symptom |
| alerts / notifications | none. The one-shot escrow_stale event fired on 4 August and was never notified — a full notification_log census for that customer that day returns 8 rows, none of them this one. It has fired twice ever, both on 4 August |
A flag that changes behaviour, that nothing sets, and that nobody who would look for it can see. Filed as R-248 in its own right, because it is the shape this fortnight has been about.
Q5 — what else does a stale blob suppress? Enumerated from code, not assumed. (1) the ACK's
restic_pw_sha256 is withheld; (2) a pending box can never auto-confirm; (3) so every off-site
run is refused indefinitely; (4) the customer is told to create a new code; (5) NEW — v0.206.0's
shape (c) is inert, because the box records an empty hub hash and falls back to (a)/(b).
(2) and (3) did not bite demo-hp, which was already escrowed before the flag landed and has
been backing up healthily throughout — 12 snapshots, last success 2026-08-07T02:15:35Z. (4) and (5)
did.
Q6 — demo-felhom? No — stale_at empty, and it records the hub hash normally. Can a freshly
installed box reach this state? NO, and this is the answer that matters for the next walk.
MarkEscrowStale has no production caller anywhere in the tree — a full census returns only its
own definition, two comments and two test references. Nothing has set the column since hub v0.95.0
shipped on 2026-08-05. The next walk cannot meet this, by any route, unless it uses demo-hp
itself — and that box is now clear.
The sharpest finding: the box states four falsehoods and recommends the destructive act
Live on demo-hp, controller v0.206.0:
STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored recovery bundle does not cover the offsite password; create a new recovery code
Every clause is false. The hub has the hash and is withholding it; there was no supersession
(host_escrow_superseded holds no row for this host); the bundle does cover the password.
It raises EscrowStale, rendering the customer card „A letétben lévő helyreállítási csomag nem fedi a
jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."
And the cause is R-241's shape for the third time. The hub already sends escrow_stale on the
wire (json:"escrow_stale,omitempty"); the controller's struct has no matching field, so
encoding/json drops it silently. The box cannot tell withheld because flagged from genuinely
hash-less, and guesses the latter. The answer is available and discarded at the boundary — filed
as R-247, and deliberately not fixed here (§0 forbids a controller change this session).
3. The bake, the vouch, and the flag clearance
| version | 0.206.0 · sha256 c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e |
| size | 656,750,694 bytes (uncompressed 2,003,138,560) |
| MinAgent | 0.127.0 |
Round-trip verified, not trusted: fetched back (HTTP 200, byte count matches), re-hashed
independently (matches), zstd -t clean, and tar -xO ./etc/felhom-controller-image out of the
download → felhom-controller:0.206.0. That last step is the one that matters, because
GOLDEN_VERSION is derived from the tag argument and could be right over stale content. Third
witness: the hub's own dropdown lists 0.206.0 with data-sha="c85230b4…108e".
All acceptance markers pass; unit Result=success / ExecMainStatus=0; bake VM purged and reverted to
virgin; token-leak grep on the committed log 0, with the instrument proven by a planted copy
first. Evidence: documentation/tests/golden-0.206.0-2026-08-08/.
VOUCHED with the operator's approval. Verified from the stored hub_settings, not the flash:
| field | before | after |
|---|---|---|
artifact_golden_version |
0.205.0 |
0.206.0 |
artifact_golden_sha256 |
8f49b2e8…4ee8 |
c85230b4…108e |
artifact_agent_version |
0.127.0 |
0.127.0 — unchanged |
artifact_min_agent |
0.127.0 |
0.127.0 — unchanged |
artifact_wrapper_sha256 |
104db0a4…16b3 |
unchanged — carried through explicitly, because the handler clears it when omitted |
The stale flag was cleared, with the operator's approval. One row, identity-matched on host_id
and guarded on stale_at IS NOT NULL; changes() returned 1. Verified end to end: the hub
serves the hash again, the box recorded hub_escrow_key_sha256 = 8a9e33aa…d080a at 11:10:19Z, and
that is byte-identical to the key it is using — so shape (c) compares, matches, and correctly stays
silent. The false warning is gone, proven with a positive control rather than an absent line: 0
escrow-confirm lines since the restart, while 5 scheduler lines in the same window prove the box was
logging and the recorded hash proves an ACK was processed.
Method note, because it touched a production pod: the hub pod is Alpine with no sqlite3; it was
installed into the container's ephemeral writable layer — image and node untouched, gone on
restart. SQLite's own file locking coordinated the write with the live hub. An earlier attempt failed
on quoting and changed nothing, which is the fail-safe working.
4. STATUS.md, R-245, and the queue audit
Rebuilt from the register: 258 → 87 lines. Not trimmed — the 100-line "what shipped recently" log was removed outright, because restating the CHANGELOGs here is what made the page grow back.
All three named defects fixed: the "waiting on you" list no longer asks the operator to decide the
recovery screen (shipped 2026-08-05) or to approve an orphaned-backup deletion the register
records as done the same day; the stray - **Nothing.** line is gone; and the DooPlex
infrastructure work is now under its own heading — kept rather than dropped, with the reason
stated on the page: they are real asks that need the operator, but they concern the machine this is
built on, not what a customer receives. Dropping them would have lost real work.
R-245 re-filed as a decision taken, keeping the whole reasoning and gaining the condition that reopens it: quota — old set-aside history blocking new backups. A condition, not a calendar.
The queue audit. Parsing the state column exactly (grepping for the phrase over-matches rows
that merely mention it): exactly one row carried WAITING-ON-OPERATOR — R-245 — and it was the
settled one. So zero rows were genuinely waiting, and the drift was caught while it was still a
single row.
5. R-244 — still owed, now measured
A read-only census, no truncation: app_log_issues holds 1309 rows; 71 reference a torn-down
venue; of those 44 are orphans (safely deletable) and 27 are shared with a live customer and
must be de-referenced, never deleted. 1238 untouched.
Not done here, and the reason is on the row: the fix is hub code, this session's scope forbade a
hub version bump, and a hand-run SQL mutation over 71 rows — 27 needing surgical de-referencing — with
no tested code path and no red-proof is the shape that goes wrong on a live database. What it
needs: a cascade leg that removes the customer id from affected_customers / context_customer and
deletes only rows that become empty, plus a one-off sweep for the four venues already gone. The next
session starts from data rather than a guess.
6. Registers, CI, and what was not done
Opened: R-246 (the flag: wrong, traced, now cleared — the column ruling still owed), R-247 (the box states four falsehoods and recommends the destructive act; the ACK field it needs is already on the wire and dropped), R-248 (a behaviour-changing flag visible to nobody). Updated: R-242 (recurred within a day; bake half now gated, vouch half explicitly still open), R-244 (measured), R-245 (re-filed as decided). Highest register ID moves R-245 → R-248.
CI — and the gate broke it, which I caught by checking rather than assuming. Runs 241
(3ca9a7bbe6e5) and 243 (7850469d5b78) failed. 241 is expected and correct — the gate was
legitimately red at that commit, and CI saw it. 243 was not: CI checks out ONE repo, shallow, so
the gate found no sibling controller clone, exited 2 (INCONCLUSIVE) and turned CI red on every
push. A permanently-red CI is the detector-nobody-hears failure that workflow exists to prevent.
Fixed by giving the gate what it needs, not by letting it skip (6f25e02828c8): a depth-1 fetch
of the controller repo in CI, plain git, no JavaScript-action step. A skip would have been the
fail-open shape this project keeps removing — and the gate would then have run in neither of its
two automated homes. Verified green: runs 244 and 245 (6f25e02828c8) both success.
--no-verify was used exactly once, on commit 3ca9a7bbe6e5, to push past the gate's own
conviction — §9.11's explicit question, answered. Every later push was clean.
No controller or agent change, as scoped. R-247's fix is a controller change and is therefore filed rather than made.
Observations — noticed, NOT acted on
stale_atis a column with no production writer. It changes what a customer is told, and nothing can set it. Either give it an evidential setter or retire it; leaving it is leaving a trap that only a database read can spring. Recorded on R-246; not decided here.- The
escrow_staleevent type appears never to be notifiable. It fired twice and reachednotification_logneither time. I did not establish whether it is absent from the dispatcher's allow-list or merely suppressed, so I have not filed it — but if R-247 is taken up, that is worth five minutes first. - The hub's
EscrowStatus.Stalefield is serialised and has no consumer anywhere. It is dead weight on the wire until R-247 gives it one. - A new gate can break CI in a way the local run cannot show, because CI's checkout is narrower than a workstation's. That cost one red CI here and was caught only by pulling the run status. Worth remembering the next time a gate reads a sibling repo — the pre-push hook and CI do not see the same filesystem.