10af63d0b0
gates / gates (push) Successful in 18s
CI checks out one repo shallow, so the golden-currency gate found no sibling controller clone, exited 2 (INCONCLUSIVE) and turned CI red on every push. Caught by pulling the run status rather than assuming it. Fixed by fetching the controller repo in CI - not by letting the gate skip, which would have been fail-open and would have left it running in neither of its automated homes. Runs 244 and 245 confirm green. Also records the reusable lesson: the pre-push hook and CI do not see the same filesystem, so a gate that reads a sibling repo can pass locally and break CI.
247 lines
15 KiB
Markdown
247 lines
15 KiB
Markdown
# REPORT — clearing the ground before the next walk (2026-08-08)
|
|
|
|
*A sibling report: `REPORT.md` is overwritten per-session and a parallel session shares this clone.*
|
|
|
|
**Golden `0.206.0` baked and VOUCHED · the R-242 gate built and shown red→green · demo-hp's stale flag
|
|
found wrong and cleared · `STATUS.md` 258 → 87 lines. `felhom-controller` and `felhom-agent` untouched.**
|
|
|
|
---
|
|
|
|
## 1. Part 2's gate — FAILING first, then passing
|
|
|
|
**Shown failing against today's state, before anything was baked.** That ordering was the instruction
|
|
and it is the gate's own red-proof:
|
|
|
|
```
|
|
newest released controller : 0.206.0
|
|
newest golden baked : 0.205.0
|
|
|
|
GOLDEN CURRENCY GATE FAILED: controller v0.206.0 is released and NO golden carries it (newest bake is 0.205.0).
|
|
A machine installed right now would receive v0.205.0 — the release is written, tested and pushed, and NOT delivered.
|
|
```
|
|
|
|
Entry point exit **1**; summary `CONVICTED: golden-currency`. After the bake, the same command:
|
|
|
|
```
|
|
newest released controller : 0.206.0
|
|
newest golden baked : 0.206.0
|
|
golden currency gate OK
|
|
```
|
|
|
|
**⚠ THE INTRODUCING PUSH USED `--no-verify`, to get past the gate's own conviction.** Stated here, in
|
|
`scripts/CHANGELOG.md` and in the commit message rather than worked around. The alternative — baking
|
|
first so the gate had never been seen red — was explicitly rejected: a gate that has never been seen
|
|
failing has not been shown to work. Every later push in this session was clean.
|
|
|
|
### What it does not catch, and why the design is what it is
|
|
|
|
**It checks the BAKE, not the VOUCH.** Both `.githooks/pre-push` **and** CI run `repo_gates.py --fast`,
|
|
which by contract selects only gates touching no network — so a hub-reading gate registered as
|
|
non-fast would run in **neither**, which is exactly the R-29 census failure this runner was built to
|
|
end. And the vouched version lives only in `hub_settings`, with no copy in git; putting one there
|
|
would create a second source of truth that can drift, and **a green gate over a false claim is the
|
|
worst outcome available**. So a bake without a vouch still passes. That half stays open on R-242
|
|
rather than being papered over.
|
|
|
|
**It compares versions, not behaviour**, so a release that changed nothing customer-visible also trips
|
|
it. **Accepted deliberately, and stated because a gate that cries wolf is one people learn to bypass:**
|
|
judging "customer-visible" by hand is precisely what failed three times (R-239, R-242, and this
|
|
recurrence), and the cost of a false trip is one bake — the operation the project wants routine anyway.
|
|
A waiver belongs in the register, never in a habit of `--no-verify`.
|
|
|
|
Inconclusive (exit 2) on an absent controller clone or an unparseable header: not knowing is never a
|
|
pass.
|
|
|
|
---
|
|
|
|
## 2. Part 1 — the stale blob, all six questions
|
|
|
|
**Read-only throughout. Nothing was cleared during the spike** — the clearance in §3 came afterwards,
|
|
on the operator's explicit approval.
|
|
|
|
**Q1 — what set it, and when. MEASURED, traced to an act, to the second.** At `2026-08-04 20:15:49`
|
|
the hub emitted `offsite_reissued` **and** `escrow_stale` in the same second — an operator **Re-issue**,
|
|
three minutes after `escrow_blob_served` at 20:12:40 and 20:12:54, i.e. during the R-201 recovery
|
|
drill. That is `offsite.ReissueCredentials`'s **precautionary** `MarkEscrowStale` call, which **hub
|
|
v0.95.0 removed the next day** (R-196 / R-204 item 2) for marking healthy escrows stale. So: **the
|
|
drill's own Re-issue, by code that no longer exists.** *(Both 4-August candidates named in the task
|
|
were live that day; the events separate them.)*
|
|
|
|
**Q2 — is the flag correct? MEASURED: NO.** The hub's blob seals
|
|
`restic_pw_sha256 = 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`; the key the box
|
|
is actually using hashes to **the identical value**. The blob covers the key. The flag was wrong from
|
|
the moment it was set.
|
|
|
|
**Q3 — what clears it? MEASURED: nothing, by itself.** The only writer of `stale_at = NULL` is
|
|
`SaveHostEscrow`'s `ON CONFLICT` — a **fresh escrow ceremony**, which is the one act that would
|
|
supersede the good blob. **The only exit from the false alarm was the destructive act the false alarm
|
|
recommends.** No timer, no self-heal, no reconciler touches it.
|
|
|
|
**Q4 — who can see it? Said plainly: effectively only a database read.**
|
|
|
|
| audience | what they see |
|
|
|---|---|
|
|
| the **customer** | a card, but stating a **false reason** (see below) and recommending the destructive act |
|
|
| the **box** | **nothing** — `report.EscrowStatus` has no `Stale` field, so it cannot see the flag at all |
|
|
| the **operator** | one page: the **PBS-DR** view (`hub/internal/web/pbsdr.go:487`) — the *wrong tier* for an off-site symptom |
|
|
| **alerts / notifications** | none. The one-shot `escrow_stale` event fired on 4 August and **was never notified** — a full `notification_log` census for that customer that day returns 8 rows, none of them this one. It has fired twice ever, both on 4 August |
|
|
|
|
**A flag that changes behaviour, that nothing sets, and that nobody who would look for it can see.**
|
|
Filed as **R-248** in its own right, because it is the shape this fortnight has been about.
|
|
|
|
**Q5 — what else does a stale blob suppress? Enumerated from code, not assumed.** (1) the ACK's
|
|
`restic_pw_sha256` is withheld; (2) a **pending** box can never auto-confirm; (3) so **every off-site
|
|
run is refused indefinitely**; (4) the customer is told to create a new code; (5) **NEW — v0.206.0's
|
|
shape (c) is inert**, because the box records an empty hub hash and falls back to (a)/(b).
|
|
**(2) and (3) did not bite `demo-hp`**, which was already `escrowed` before the flag landed and has
|
|
been backing up healthily throughout — 12 snapshots, last success `2026-08-07T02:15:35Z`. (4) and (5)
|
|
did.
|
|
|
|
**Q6 — `demo-felhom`? No** — `stale_at` empty, and it records the hub hash normally. **Can a freshly
|
|
installed box reach this state? NO, and this is the answer that matters for the next walk.**
|
|
`MarkEscrowStale` has **no production caller anywhere in the tree** — a full census returns only its
|
|
own definition, two comments and two test references. Nothing has set the column since hub v0.95.0
|
|
shipped on 2026-08-05. **The next walk cannot meet this**, by any route, unless it uses `demo-hp`
|
|
itself — and that box is now clear.
|
|
|
|
### The sharpest finding: the box states four falsehoods and recommends the destructive act
|
|
|
|
Live on `demo-hp`, controller v0.206.0:
|
|
|
|
> `STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored
|
|
> recovery bundle does not cover the offsite password; create a new recovery code`
|
|
|
|
**Every clause is false.** The hub *has* the hash and is *withholding* it; there was no supersession
|
|
(`host_escrow_superseded` holds no row for this host); the bundle **does** cover the password.
|
|
It raises `EscrowStale`, rendering the customer card *„A letétben lévő helyreállítási csomag nem fedi a
|
|
jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."*
|
|
|
|
**And the cause is R-241's shape for the third time.** The hub already sends `escrow_stale` on the
|
|
wire (`json:"escrow_stale,omitempty"`); the controller's struct has **no matching field**, so
|
|
`encoding/json` drops it silently. The box cannot tell *withheld because flagged* from *genuinely
|
|
hash-less*, and guesses the latter. **The answer is available and discarded at the boundary** — filed
|
|
as **R-247**, and deliberately **not fixed here** (§0 forbids a controller change this session).
|
|
|
|
---
|
|
|
|
## 3. The bake, the vouch, and the flag clearance
|
|
|
|
| | |
|
|
|---|---|
|
|
| version | **0.206.0** · sha256 `c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e` |
|
|
| size | **656,750,694** bytes (uncompressed 2,003,138,560) |
|
|
| MinAgent | 0.127.0 |
|
|
|
|
**Round-trip verified, not trusted:** fetched back (HTTP 200, byte count matches), re-hashed
|
|
independently (**matches**), `zstd -t` clean, and `tar -xO ./etc/felhom-controller-image` **out of the
|
|
download** → `felhom-controller:0.206.0`. That last step is the one that matters, because
|
|
`GOLDEN_VERSION` is derived from the tag argument and could be right over stale content. **Third
|
|
witness:** the hub's own dropdown lists `0.206.0` with `data-sha="c85230b4…108e"`.
|
|
|
|
All acceptance markers pass; unit `Result=success` / `ExecMainStatus=0`; bake VM purged and reverted to
|
|
`virgin`; token-leak grep on the committed log **0**, with the instrument proven by a planted copy
|
|
first. Evidence: `documentation/tests/golden-0.206.0-2026-08-08/`.
|
|
|
|
**VOUCHED with the operator's approval.** Verified from the stored `hub_settings`, not the flash:
|
|
|
|
| field | before | after |
|
|
|---|---|---|
|
|
| `artifact_golden_version` | `0.205.0` | **`0.206.0`** |
|
|
| `artifact_golden_sha256` | `8f49b2e8…4ee8` | **`c85230b4…108e`** |
|
|
| `artifact_agent_version` | `0.127.0` | `0.127.0` — unchanged |
|
|
| `artifact_min_agent` | `0.127.0` | `0.127.0` — unchanged |
|
|
| `artifact_wrapper_sha256` | `104db0a4…16b3` | unchanged — **carried through explicitly, because the handler clears it when omitted** |
|
|
|
|
**The stale flag was cleared, with the operator's approval.** One row, identity-matched on `host_id`
|
|
and guarded on `stale_at IS NOT NULL`; `changes()` returned **1**. **Verified end to end:** the hub
|
|
serves the hash again, the box recorded `hub_escrow_key_sha256 = 8a9e33aa…d080a` at `11:10:19Z`, and
|
|
that is byte-identical to the key it is using — so shape (c) compares, matches, and correctly stays
|
|
silent. **The false warning is gone, proven with a positive control rather than an absent line: 0
|
|
`escrow-confirm` lines since the restart, while 5 scheduler lines in the same window prove the box was
|
|
logging and the recorded hash proves an ACK was processed.**
|
|
|
|
*Method note, because it touched a production pod:* the hub pod is Alpine with no `sqlite3`; it was
|
|
installed into the container's **ephemeral writable layer** — image and node untouched, gone on
|
|
restart. SQLite's own file locking coordinated the write with the live hub. An earlier attempt failed
|
|
on quoting and **changed nothing**, which is the fail-safe working.
|
|
|
|
---
|
|
|
|
## 4. `STATUS.md`, R-245, and the queue audit
|
|
|
|
**Rebuilt from the register: 258 → 87 lines.** Not trimmed — the 100-line "what shipped recently" log
|
|
was removed outright, because restating the CHANGELOGs here is what made the page grow back.
|
|
|
|
**All three named defects fixed:** the "waiting on you" list no longer asks the operator to decide the
|
|
**recovery screen** (shipped 2026-08-05) or to approve an orphaned-backup deletion the register
|
|
records as **done** the same day; the stray `- **Nothing.**` line is gone; and the DooPlex
|
|
infrastructure work is now **under its own heading** — **kept rather than dropped**, with the reason
|
|
stated on the page: they are real asks that need the operator, but they concern the machine this is
|
|
built on, not what a customer receives. Dropping them would have lost real work.
|
|
|
|
**R-245 re-filed as a decision taken**, keeping the whole reasoning and gaining **the condition that
|
|
reopens it: quota — old set-aside history blocking new backups. A condition, not a calendar.**
|
|
|
|
**The queue audit.** Parsing the **state column exactly** (grepping for the phrase over-matches rows
|
|
that merely mention it): **exactly one row carried `WAITING-ON-OPERATOR` — R-245 — and it was the
|
|
settled one. So zero rows were genuinely waiting**, and the drift was caught while it was still a
|
|
single row.
|
|
|
|
---
|
|
|
|
## 5. R-244 — still owed, now measured
|
|
|
|
A read-only census, no truncation: `app_log_issues` holds **1309 rows**; **71 reference a torn-down
|
|
venue**; of those **44 are orphans** (safely deletable) and **27 are shared with a live customer** and
|
|
**must be de-referenced, never deleted**. 1238 untouched.
|
|
|
|
**Not done here, and the reason is on the row:** the fix is hub code, this session's scope forbade a
|
|
hub version bump, and a hand-run SQL mutation over 71 rows — 27 needing surgical de-referencing — with
|
|
no tested code path and no red-proof is the shape that goes wrong on a live database. **What it
|
|
needs:** a cascade leg that removes the customer id from `affected_customers` / `context_customer` and
|
|
deletes only rows that become empty, plus a one-off sweep for the four venues already gone. The next
|
|
session starts from data rather than a guess.
|
|
|
|
---
|
|
|
|
## 6. Registers, CI, and what was not done
|
|
|
|
**Opened:** **R-246** (the flag: wrong, traced, now cleared — the column ruling still owed),
|
|
**R-247** (the box states four falsehoods and recommends the destructive act; the ACK field it needs
|
|
is already on the wire and dropped), **R-248** (a behaviour-changing flag visible to nobody).
|
|
**Updated:** **R-242** (recurred within a day; bake half now gated, vouch half explicitly still open),
|
|
**R-244** (measured), **R-245** (re-filed as decided). **Highest register ID moves R-245 → R-248.**
|
|
|
|
**CI — and the gate broke it, which I caught by checking rather than assuming.** Runs **241**
|
|
(`3ca9a7bbe6e5`) and **243** (`7850469d5b78`) **failed**. 241 is expected and correct — the gate was
|
|
legitimately red at that commit, and CI saw it. **243 was not**: CI checks out ONE repo, shallow, so
|
|
the gate found no sibling controller clone, exited **2 (INCONCLUSIVE)** and turned CI red on every
|
|
push. **A permanently-red CI is the detector-nobody-hears failure that workflow exists to prevent.**
|
|
|
|
**Fixed by giving the gate what it needs, not by letting it skip** (`6f25e02828c8`): a depth-1 fetch
|
|
of the controller repo in CI, plain `git`, no JavaScript-action step. A skip would have been the
|
|
fail-open shape this project keeps removing — and the gate would then have run in **neither** of its
|
|
two automated homes. **Verified green: runs 244 and 245 (`6f25e02828c8`) both success.**
|
|
|
|
**`--no-verify` was used exactly once**, on commit `3ca9a7bbe6e5`, to push past the gate's own
|
|
conviction — §9.11's explicit question, answered. Every later push was clean.
|
|
|
|
**No controller or agent change**, as scoped. R-247's fix is a controller change and is therefore
|
|
filed rather than made.
|
|
|
|
### Observations — noticed, NOT acted on
|
|
|
|
- **`stale_at` is a column with no production writer.** It changes what a customer is told, and
|
|
nothing can set it. Either give it an evidential setter or retire it; leaving it is leaving a trap
|
|
that only a database read can spring. Recorded on R-246; not decided here.
|
|
- **The `escrow_stale` event type appears never to be notifiable.** It fired twice and reached
|
|
`notification_log` neither time. I did not establish whether it is absent from the dispatcher's
|
|
allow-list or merely suppressed, so I have not filed it — but if R-247 is taken up, that is worth
|
|
five minutes first.
|
|
- **The hub's `EscrowStatus.Stale` field is serialised and has no consumer anywhere.** It is dead
|
|
weight on the wire until R-247 gives it one.
|
|
- **A new gate can break CI in a way the local run cannot show**, because CI's checkout is narrower
|
|
than a workstation's. That cost one red CI here and was caught only by pulling the run status. Worth
|
|
remembering the next time a gate reads a sibling repo — the pre-push hook and CI do **not** see the
|
|
same filesystem.
|