Files
felhom.eu/REPORT-clear-the-ground-2026-08-08.md
T
admin 10af63d0b0
gates / gates (push) Successful in 18s
report: record the CI breakage the new gate caused, and its fix
CI checks out one repo shallow, so the golden-currency gate found no sibling
controller clone, exited 2 (INCONCLUSIVE) and turned CI red on every push.
Caught by pulling the run status rather than assuming it. Fixed by fetching
the controller repo in CI - not by letting the gate skip, which would have been
fail-open and would have left it running in neither of its automated homes.
Runs 244 and 245 confirm green.

Also records the reusable lesson: the pre-push hook and CI do not see the same
filesystem, so a gate that reads a sibling repo can pass locally and break CI.
2026-08-07 13:16:32 +02:00

15 KiB

REPORT — clearing the ground before the next walk (2026-08-08)

A sibling report: REPORT.md is overwritten per-session and a parallel session shares this clone.

Golden 0.206.0 baked and VOUCHED · the R-242 gate built and shown red→green · demo-hp's stale flag found wrong and cleared · STATUS.md 258 → 87 lines. felhom-controller and felhom-agent untouched.


1. Part 2's gate — FAILING first, then passing

Shown failing against today's state, before anything was baked. That ordering was the instruction and it is the gate's own red-proof:

  newest released controller : 0.206.0
  newest golden baked        : 0.205.0

GOLDEN CURRENCY GATE FAILED: controller v0.206.0 is released and NO golden carries it (newest bake is 0.205.0).
A machine installed right now would receive v0.205.0 — the release is written, tested and pushed, and NOT delivered.

Entry point exit 1; summary CONVICTED: golden-currency. After the bake, the same command:

  newest released controller : 0.206.0
  newest golden baked        : 0.206.0
golden currency gate OK

⚠ THE INTRODUCING PUSH USED --no-verify, to get past the gate's own conviction. Stated here, in scripts/CHANGELOG.md and in the commit message rather than worked around. The alternative — baking first so the gate had never been seen red — was explicitly rejected: a gate that has never been seen failing has not been shown to work. Every later push in this session was clean.

What it does not catch, and why the design is what it is

It checks the BAKE, not the VOUCH. Both .githooks/pre-push and CI run repo_gates.py --fast, which by contract selects only gates touching no network — so a hub-reading gate registered as non-fast would run in neither, which is exactly the R-29 census failure this runner was built to end. And the vouched version lives only in hub_settings, with no copy in git; putting one there would create a second source of truth that can drift, and a green gate over a false claim is the worst outcome available. So a bake without a vouch still passes. That half stays open on R-242 rather than being papered over.

It compares versions, not behaviour, so a release that changed nothing customer-visible also trips it. Accepted deliberately, and stated because a gate that cries wolf is one people learn to bypass: judging "customer-visible" by hand is precisely what failed three times (R-239, R-242, and this recurrence), and the cost of a false trip is one bake — the operation the project wants routine anyway. A waiver belongs in the register, never in a habit of --no-verify.

Inconclusive (exit 2) on an absent controller clone or an unparseable header: not knowing is never a pass.


2. Part 1 — the stale blob, all six questions

Read-only throughout. Nothing was cleared during the spike — the clearance in §3 came afterwards, on the operator's explicit approval.

Q1 — what set it, and when. MEASURED, traced to an act, to the second. At 2026-08-04 20:15:49 the hub emitted offsite_reissued and escrow_stale in the same second — an operator Re-issue, three minutes after escrow_blob_served at 20:12:40 and 20:12:54, i.e. during the R-201 recovery drill. That is offsite.ReissueCredentials's precautionary MarkEscrowStale call, which hub v0.95.0 removed the next day (R-196 / R-204 item 2) for marking healthy escrows stale. So: the drill's own Re-issue, by code that no longer exists. (Both 4-August candidates named in the task were live that day; the events separate them.)

Q2 — is the flag correct? MEASURED: NO. The hub's blob seals restic_pw_sha256 = 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a; the key the box is actually using hashes to the identical value. The blob covers the key. The flag was wrong from the moment it was set.

Q3 — what clears it? MEASURED: nothing, by itself. The only writer of stale_at = NULL is SaveHostEscrow's ON CONFLICT — a fresh escrow ceremony, which is the one act that would supersede the good blob. The only exit from the false alarm was the destructive act the false alarm recommends. No timer, no self-heal, no reconciler touches it.

Q4 — who can see it? Said plainly: effectively only a database read.

audience what they see
the customer a card, but stating a false reason (see below) and recommending the destructive act
the box nothingreport.EscrowStatus has no Stale field, so it cannot see the flag at all
the operator one page: the PBS-DR view (hub/internal/web/pbsdr.go:487) — the wrong tier for an off-site symptom
alerts / notifications none. The one-shot escrow_stale event fired on 4 August and was never notified — a full notification_log census for that customer that day returns 8 rows, none of them this one. It has fired twice ever, both on 4 August

A flag that changes behaviour, that nothing sets, and that nobody who would look for it can see. Filed as R-248 in its own right, because it is the shape this fortnight has been about.

Q5 — what else does a stale blob suppress? Enumerated from code, not assumed. (1) the ACK's restic_pw_sha256 is withheld; (2) a pending box can never auto-confirm; (3) so every off-site run is refused indefinitely; (4) the customer is told to create a new code; (5) NEW — v0.206.0's shape (c) is inert, because the box records an empty hub hash and falls back to (a)/(b). (2) and (3) did not bite demo-hp, which was already escrowed before the flag landed and has been backing up healthily throughout — 12 snapshots, last success 2026-08-07T02:15:35Z. (4) and (5) did.

Q6 — demo-felhom? Nostale_at empty, and it records the hub hash normally. Can a freshly installed box reach this state? NO, and this is the answer that matters for the next walk. MarkEscrowStale has no production caller anywhere in the tree — a full census returns only its own definition, two comments and two test references. Nothing has set the column since hub v0.95.0 shipped on 2026-08-05. The next walk cannot meet this, by any route, unless it uses demo-hp itself — and that box is now clear.

The sharpest finding: the box states four falsehoods and recommends the destructive act

Live on demo-hp, controller v0.206.0:

STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored recovery bundle does not cover the offsite password; create a new recovery code

Every clause is false. The hub has the hash and is withholding it; there was no supersession (host_escrow_superseded holds no row for this host); the bundle does cover the password. It raises EscrowStale, rendering the customer card „A letétben lévő helyreállítási csomag nem fedi a jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."

And the cause is R-241's shape for the third time. The hub already sends escrow_stale on the wire (json:"escrow_stale,omitempty"); the controller's struct has no matching field, so encoding/json drops it silently. The box cannot tell withheld because flagged from genuinely hash-less, and guesses the latter. The answer is available and discarded at the boundary — filed as R-247, and deliberately not fixed here (§0 forbids a controller change this session).


3. The bake, the vouch, and the flag clearance

version 0.206.0 · sha256 c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e
size 656,750,694 bytes (uncompressed 2,003,138,560)
MinAgent 0.127.0

Round-trip verified, not trusted: fetched back (HTTP 200, byte count matches), re-hashed independently (matches), zstd -t clean, and tar -xO ./etc/felhom-controller-image out of the downloadfelhom-controller:0.206.0. That last step is the one that matters, because GOLDEN_VERSION is derived from the tag argument and could be right over stale content. Third witness: the hub's own dropdown lists 0.206.0 with data-sha="c85230b4…108e".

All acceptance markers pass; unit Result=success / ExecMainStatus=0; bake VM purged and reverted to virgin; token-leak grep on the committed log 0, with the instrument proven by a planted copy first. Evidence: documentation/tests/golden-0.206.0-2026-08-08/.

VOUCHED with the operator's approval. Verified from the stored hub_settings, not the flash:

field before after
artifact_golden_version 0.205.0 0.206.0
artifact_golden_sha256 8f49b2e8…4ee8 c85230b4…108e
artifact_agent_version 0.127.0 0.127.0 — unchanged
artifact_min_agent 0.127.0 0.127.0 — unchanged
artifact_wrapper_sha256 104db0a4…16b3 unchanged — carried through explicitly, because the handler clears it when omitted

The stale flag was cleared, with the operator's approval. One row, identity-matched on host_id and guarded on stale_at IS NOT NULL; changes() returned 1. Verified end to end: the hub serves the hash again, the box recorded hub_escrow_key_sha256 = 8a9e33aa…d080a at 11:10:19Z, and that is byte-identical to the key it is using — so shape (c) compares, matches, and correctly stays silent. The false warning is gone, proven with a positive control rather than an absent line: 0 escrow-confirm lines since the restart, while 5 scheduler lines in the same window prove the box was logging and the recorded hash proves an ACK was processed.

Method note, because it touched a production pod: the hub pod is Alpine with no sqlite3; it was installed into the container's ephemeral writable layer — image and node untouched, gone on restart. SQLite's own file locking coordinated the write with the live hub. An earlier attempt failed on quoting and changed nothing, which is the fail-safe working.


4. STATUS.md, R-245, and the queue audit

Rebuilt from the register: 258 → 87 lines. Not trimmed — the 100-line "what shipped recently" log was removed outright, because restating the CHANGELOGs here is what made the page grow back.

All three named defects fixed: the "waiting on you" list no longer asks the operator to decide the recovery screen (shipped 2026-08-05) or to approve an orphaned-backup deletion the register records as done the same day; the stray - **Nothing.** line is gone; and the DooPlex infrastructure work is now under its own headingkept rather than dropped, with the reason stated on the page: they are real asks that need the operator, but they concern the machine this is built on, not what a customer receives. Dropping them would have lost real work.

R-245 re-filed as a decision taken, keeping the whole reasoning and gaining the condition that reopens it: quota — old set-aside history blocking new backups. A condition, not a calendar.

The queue audit. Parsing the state column exactly (grepping for the phrase over-matches rows that merely mention it): exactly one row carried WAITING-ON-OPERATOR — R-245 — and it was the settled one. So zero rows were genuinely waiting, and the drift was caught while it was still a single row.


5. R-244 — still owed, now measured

A read-only census, no truncation: app_log_issues holds 1309 rows; 71 reference a torn-down venue; of those 44 are orphans (safely deletable) and 27 are shared with a live customer and must be de-referenced, never deleted. 1238 untouched.

Not done here, and the reason is on the row: the fix is hub code, this session's scope forbade a hub version bump, and a hand-run SQL mutation over 71 rows — 27 needing surgical de-referencing — with no tested code path and no red-proof is the shape that goes wrong on a live database. What it needs: a cascade leg that removes the customer id from affected_customers / context_customer and deletes only rows that become empty, plus a one-off sweep for the four venues already gone. The next session starts from data rather than a guess.


6. Registers, CI, and what was not done

Opened: R-246 (the flag: wrong, traced, now cleared — the column ruling still owed), R-247 (the box states four falsehoods and recommends the destructive act; the ACK field it needs is already on the wire and dropped), R-248 (a behaviour-changing flag visible to nobody). Updated: R-242 (recurred within a day; bake half now gated, vouch half explicitly still open), R-244 (measured), R-245 (re-filed as decided). Highest register ID moves R-245 → R-248.

CI — and the gate broke it, which I caught by checking rather than assuming. Runs 241 (3ca9a7bbe6e5) and 243 (7850469d5b78) failed. 241 is expected and correct — the gate was legitimately red at that commit, and CI saw it. 243 was not: CI checks out ONE repo, shallow, so the gate found no sibling controller clone, exited 2 (INCONCLUSIVE) and turned CI red on every push. A permanently-red CI is the detector-nobody-hears failure that workflow exists to prevent.

Fixed by giving the gate what it needs, not by letting it skip (6f25e02828c8): a depth-1 fetch of the controller repo in CI, plain git, no JavaScript-action step. A skip would have been the fail-open shape this project keeps removing — and the gate would then have run in neither of its two automated homes. Verified green: runs 244 and 245 (6f25e02828c8) both success.

--no-verify was used exactly once, on commit 3ca9a7bbe6e5, to push past the gate's own conviction — §9.11's explicit question, answered. Every later push was clean.

No controller or agent change, as scoped. R-247's fix is a controller change and is therefore filed rather than made.

Observations — noticed, NOT acted on

  • stale_at is a column with no production writer. It changes what a customer is told, and nothing can set it. Either give it an evidential setter or retire it; leaving it is leaving a trap that only a database read can spring. Recorded on R-246; not decided here.
  • The escrow_stale event type appears never to be notifiable. It fired twice and reached notification_log neither time. I did not establish whether it is absent from the dispatcher's allow-list or merely suppressed, so I have not filed it — but if R-247 is taken up, that is worth five minutes first.
  • The hub's EscrowStatus.Stale field is serialised and has no consumer anywhere. It is dead weight on the wire until R-247 gives it one.
  • A new gate can break CI in a way the local run cannot show, because CI's checkout is narrower than a workstation's. That cost one red CI here and was caught only by pulling the run status. Worth remembering the next time a gate reads a sibling repo — the pre-push hook and CI do not see the same filesystem.