golden 0.206.0 VOUCHED; demo-hp's stale flag cleared; session report
gates / gates (push) Failing after 10s

VOUCHED with the operator's approval, verified from the stored hub_settings
rather than the flash: golden_version 0.205.0 -> 0.206.0, sha
c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e.
agent_version and min_agent both stayed 0.127.0. wrapper_sha256 was carried
through explicitly, because the handler CLEARS it when omitted.

THE GATE WAS CONVICTED BEFORE THE BAKE AND IS OK AFTER IT - red to green on
the same command, which is its proof that it measures something real. It went
green on the BAKE, not the vouch; that limitation is stated in its docstring
and stays open on R-242.

THE STALE FLAG WAS WRONG AND IS CLEARED, with the operator's approval. One
row, identity-matched on host_id and guarded on stale_at IS NOT NULL;
changes() returned 1. Verified end to end, not just in the database: the hub
serves the hash again, the box recorded it at 11:10:19Z, and it is
byte-identical to the key that box is using - so shape (c) compares, matches
and correctly stays silent. The false warning is gone, PROVEN WITH A POSITIVE
CONTROL rather than an absent line: 0 escrow-confirm lines since the restart
while 5 scheduler lines in the same window prove the box was logging.

R-246 records the clearance and keeps the column ruling open: stale_at has NO
production writer, changes what a customer is told, and can be seen by nobody
who would look for it. Either give it an evidential setter or retire it.

STATUS.md finished at 87 lines (from 258). Waiting-on-you is now genuinely
empty: the base image is approved and live, and R-245 was re-filed as a
decision taken with quota as its reopening condition.

Session report: REPORT-clear-the-ground-2026-08-08.md - the six spike
questions each answered with method and measurement, Q4 said plainly (only a
database read), Q6 said loudly (a fresh box CANNOT reach this state, so the
next walk cannot meet it), and three observations noticed but not acted on.
This commit is contained in:
2026-08-07 13:13:16 +02:00
parent 721297ed5e
commit 7850469d5b
3 changed files with 241 additions and 7 deletions
+232
View File
@@ -0,0 +1,232 @@
# REPORT — clearing the ground before the next walk (2026-08-08)
*A sibling report: `REPORT.md` is overwritten per-session and a parallel session shares this clone.*
**Golden `0.206.0` baked and VOUCHED · the R-242 gate built and shown red→green · demo-hp's stale flag
found wrong and cleared · `STATUS.md` 258 → 87 lines. `felhom-controller` and `felhom-agent` untouched.**
---
## 1. Part 2's gate — FAILING first, then passing
**Shown failing against today's state, before anything was baked.** That ordering was the instruction
and it is the gate's own red-proof:
```
newest released controller : 0.206.0
newest golden baked : 0.205.0
GOLDEN CURRENCY GATE FAILED: controller v0.206.0 is released and NO golden carries it (newest bake is 0.205.0).
A machine installed right now would receive v0.205.0 — the release is written, tested and pushed, and NOT delivered.
```
Entry point exit **1**; summary `CONVICTED: golden-currency`. After the bake, the same command:
```
newest released controller : 0.206.0
newest golden baked : 0.206.0
golden currency gate OK
```
**⚠ THE INTRODUCING PUSH USED `--no-verify`, to get past the gate's own conviction.** Stated here, in
`scripts/CHANGELOG.md` and in the commit message rather than worked around. The alternative — baking
first so the gate had never been seen red — was explicitly rejected: a gate that has never been seen
failing has not been shown to work. Every later push in this session was clean.
### What it does not catch, and why the design is what it is
**It checks the BAKE, not the VOUCH.** Both `.githooks/pre-push` **and** CI run `repo_gates.py --fast`,
which by contract selects only gates touching no network — so a hub-reading gate registered as
non-fast would run in **neither**, which is exactly the R-29 census failure this runner was built to
end. And the vouched version lives only in `hub_settings`, with no copy in git; putting one there
would create a second source of truth that can drift, and **a green gate over a false claim is the
worst outcome available**. So a bake without a vouch still passes. That half stays open on R-242
rather than being papered over.
**It compares versions, not behaviour**, so a release that changed nothing customer-visible also trips
it. **Accepted deliberately, and stated because a gate that cries wolf is one people learn to bypass:**
judging "customer-visible" by hand is precisely what failed three times (R-239, R-242, and this
recurrence), and the cost of a false trip is one bake — the operation the project wants routine anyway.
A waiver belongs in the register, never in a habit of `--no-verify`.
Inconclusive (exit 2) on an absent controller clone or an unparseable header: not knowing is never a
pass.
---
## 2. Part 1 — the stale blob, all six questions
**Read-only throughout. Nothing was cleared during the spike** — the clearance in §3 came afterwards,
on the operator's explicit approval.
**Q1 — what set it, and when. MEASURED, traced to an act, to the second.** At `2026-08-04 20:15:49`
the hub emitted `offsite_reissued` **and** `escrow_stale` in the same second — an operator **Re-issue**,
three minutes after `escrow_blob_served` at 20:12:40 and 20:12:54, i.e. during the R-201 recovery
drill. That is `offsite.ReissueCredentials`'s **precautionary** `MarkEscrowStale` call, which **hub
v0.95.0 removed the next day** (R-196 / R-204 item 2) for marking healthy escrows stale. So: **the
drill's own Re-issue, by code that no longer exists.** *(Both 4-August candidates named in the task
were live that day; the events separate them.)*
**Q2 — is the flag correct? MEASURED: NO.** The hub's blob seals
`restic_pw_sha256 = 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`; the key the box
is actually using hashes to **the identical value**. The blob covers the key. The flag was wrong from
the moment it was set.
**Q3 — what clears it? MEASURED: nothing, by itself.** The only writer of `stale_at = NULL` is
`SaveHostEscrow`'s `ON CONFLICT` — a **fresh escrow ceremony**, which is the one act that would
supersede the good blob. **The only exit from the false alarm was the destructive act the false alarm
recommends.** No timer, no self-heal, no reconciler touches it.
**Q4 — who can see it? Said plainly: effectively only a database read.**
| audience | what they see |
|---|---|
| the **customer** | a card, but stating a **false reason** (see below) and recommending the destructive act |
| the **box** | **nothing**`report.EscrowStatus` has no `Stale` field, so it cannot see the flag at all |
| the **operator** | one page: the **PBS-DR** view (`hub/internal/web/pbsdr.go:487`) — the *wrong tier* for an off-site symptom |
| **alerts / notifications** | none. The one-shot `escrow_stale` event fired on 4 August and **was never notified** — a full `notification_log` census for that customer that day returns 8 rows, none of them this one. It has fired twice ever, both on 4 August |
**A flag that changes behaviour, that nothing sets, and that nobody who would look for it can see.**
Filed as **R-248** in its own right, because it is the shape this fortnight has been about.
**Q5 — what else does a stale blob suppress? Enumerated from code, not assumed.** (1) the ACK's
`restic_pw_sha256` is withheld; (2) a **pending** box can never auto-confirm; (3) so **every off-site
run is refused indefinitely**; (4) the customer is told to create a new code; (5) **NEW — v0.206.0's
shape (c) is inert**, because the box records an empty hub hash and falls back to (a)/(b).
**(2) and (3) did not bite `demo-hp`**, which was already `escrowed` before the flag landed and has
been backing up healthily throughout — 12 snapshots, last success `2026-08-07T02:15:35Z`. (4) and (5)
did.
**Q6 — `demo-felhom`? No**`stale_at` empty, and it records the hub hash normally. **Can a freshly
installed box reach this state? NO, and this is the answer that matters for the next walk.**
`MarkEscrowStale` has **no production caller anywhere in the tree** — a full census returns only its
own definition, two comments and two test references. Nothing has set the column since hub v0.95.0
shipped on 2026-08-05. **The next walk cannot meet this**, by any route, unless it uses `demo-hp`
itself — and that box is now clear.
### The sharpest finding: the box states four falsehoods and recommends the destructive act
Live on `demo-hp`, controller v0.206.0:
> `STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored
> recovery bundle does not cover the offsite password; create a new recovery code`
**Every clause is false.** The hub *has* the hash and is *withholding* it; there was no supersession
(`host_escrow_superseded` holds no row for this host); the bundle **does** cover the password.
It raises `EscrowStale`, rendering the customer card *„A letétben lévő helyreállítási csomag nem fedi a
jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."*
**And the cause is R-241's shape for the third time.** The hub already sends `escrow_stale` on the
wire (`json:"escrow_stale,omitempty"`); the controller's struct has **no matching field**, so
`encoding/json` drops it silently. The box cannot tell *withheld because flagged* from *genuinely
hash-less*, and guesses the latter. **The answer is available and discarded at the boundary** — filed
as **R-247**, and deliberately **not fixed here** (§0 forbids a controller change this session).
---
## 3. The bake, the vouch, and the flag clearance
| | |
|---|---|
| version | **0.206.0** · sha256 `c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e` |
| size | **656,750,694** bytes (uncompressed 2,003,138,560) |
| MinAgent | 0.127.0 |
**Round-trip verified, not trusted:** fetched back (HTTP 200, byte count matches), re-hashed
independently (**matches**), `zstd -t` clean, and `tar -xO ./etc/felhom-controller-image` **out of the
download** → `felhom-controller:0.206.0`. That last step is the one that matters, because
`GOLDEN_VERSION` is derived from the tag argument and could be right over stale content. **Third
witness:** the hub's own dropdown lists `0.206.0` with `data-sha="c85230b4…108e"`.
All acceptance markers pass; unit `Result=success` / `ExecMainStatus=0`; bake VM purged and reverted to
`virgin`; token-leak grep on the committed log **0**, with the instrument proven by a planted copy
first. Evidence: `documentation/tests/golden-0.206.0-2026-08-08/`.
**VOUCHED with the operator's approval.** Verified from the stored `hub_settings`, not the flash:
| field | before | after |
|---|---|---|
| `artifact_golden_version` | `0.205.0` | **`0.206.0`** |
| `artifact_golden_sha256` | `8f49b2e8…4ee8` | **`c85230b4…108e`** |
| `artifact_agent_version` | `0.127.0` | `0.127.0` — unchanged |
| `artifact_min_agent` | `0.127.0` | `0.127.0` — unchanged |
| `artifact_wrapper_sha256` | `104db0a4…16b3` | unchanged — **carried through explicitly, because the handler clears it when omitted** |
**The stale flag was cleared, with the operator's approval.** One row, identity-matched on `host_id`
and guarded on `stale_at IS NOT NULL`; `changes()` returned **1**. **Verified end to end:** the hub
serves the hash again, the box recorded `hub_escrow_key_sha256 = 8a9e33aa…d080a` at `11:10:19Z`, and
that is byte-identical to the key it is using — so shape (c) compares, matches, and correctly stays
silent. **The false warning is gone, proven with a positive control rather than an absent line: 0
`escrow-confirm` lines since the restart, while 5 scheduler lines in the same window prove the box was
logging and the recorded hash proves an ACK was processed.**
*Method note, because it touched a production pod:* the hub pod is Alpine with no `sqlite3`; it was
installed into the container's **ephemeral writable layer** — image and node untouched, gone on
restart. SQLite's own file locking coordinated the write with the live hub. An earlier attempt failed
on quoting and **changed nothing**, which is the fail-safe working.
---
## 4. `STATUS.md`, R-245, and the queue audit
**Rebuilt from the register: 258 → 87 lines.** Not trimmed — the 100-line "what shipped recently" log
was removed outright, because restating the CHANGELOGs here is what made the page grow back.
**All three named defects fixed:** the "waiting on you" list no longer asks the operator to decide the
**recovery screen** (shipped 2026-08-05) or to approve an orphaned-backup deletion the register
records as **done** the same day; the stray `- **Nothing.**` line is gone; and the DooPlex
infrastructure work is now **under its own heading****kept rather than dropped**, with the reason
stated on the page: they are real asks that need the operator, but they concern the machine this is
built on, not what a customer receives. Dropping them would have lost real work.
**R-245 re-filed as a decision taken**, keeping the whole reasoning and gaining **the condition that
reopens it: quota — old set-aside history blocking new backups. A condition, not a calendar.**
**The queue audit.** Parsing the **state column exactly** (grepping for the phrase over-matches rows
that merely mention it): **exactly one row carried `WAITING-ON-OPERATOR` — R-245 — and it was the
settled one. So zero rows were genuinely waiting**, and the drift was caught while it was still a
single row.
---
## 5. R-244 — still owed, now measured
A read-only census, no truncation: `app_log_issues` holds **1309 rows**; **71 reference a torn-down
venue**; of those **44 are orphans** (safely deletable) and **27 are shared with a live customer** and
**must be de-referenced, never deleted**. 1238 untouched.
**Not done here, and the reason is on the row:** the fix is hub code, this session's scope forbade a
hub version bump, and a hand-run SQL mutation over 71 rows — 27 needing surgical de-referencing — with
no tested code path and no red-proof is the shape that goes wrong on a live database. **What it
needs:** a cascade leg that removes the customer id from `affected_customers` / `context_customer` and
deletes only rows that become empty, plus a one-off sweep for the four venues already gone. The next
session starts from data rather than a guess.
---
## 6. Registers, CI, and what was not done
**Opened:** **R-246** (the flag: wrong, traced, now cleared — the column ruling still owed),
**R-247** (the box states four falsehoods and recommends the destructive act; the ACK field it needs
is already on the wire and dropped), **R-248** (a behaviour-changing flag visible to nobody).
**Updated:** **R-242** (recurred within a day; bake half now gated, vouch half explicitly still open),
**R-244** (measured), **R-245** (re-filed as decided). **Highest register ID moves R-245 → R-248.**
**CI:** `felhom.eu` runs to be confirmed by `head_sha` after the final push (below). **`--no-verify`
was used exactly once**, on commit `3ca9a7bbe6e5`, to push past the gate's own conviction — §9.11's
explicit question, answered.
**No controller or agent change**, as scoped. R-247's fix is a controller change and is therefore
filed rather than made.
### Observations — noticed, NOT acted on
- **`stale_at` is a column with no production writer.** It changes what a customer is told, and
nothing can set it. Either give it an evidential setter or retire it; leaving it is leaving a trap
that only a database read can spring. Recorded on R-246; not decided here.
- **The `escrow_stale` event type appears never to be notifiable.** It fired twice and reached
`notification_log` neither time. I did not establish whether it is absent from the dispatcher's
allow-list or merely suppressed, so I have not filed it — but if R-247 is taken up, that is worth
five minutes first.
- **The hub's `EscrowStatus.Stale` field is serialised and has no consumer anywhere.** It is dead
weight on the wire until R-247 gives it one.
+7 -5
View File
@@ -42,13 +42,15 @@ for byte identical — three separate times, including a filename with Hungarian
## Found today
- **A leftover flag has been silently switching off the new recovery detection on one demo machine
since 4 August.** A Re-issue during the recovery drill set it, using code we removed the next day.
- **A leftover flag had been silently switching off the new recovery detection on one demo machine
since 4 August — found, and cleared with your approval.** A Re-issue during the recovery drill set it, using code we removed the next day.
**The flag is wrong** — the sealed package does cover the key that machine is using, and the two
fingerprints match exactly. Because of it the hub withholds a figure the machine needs, so the
machine tells its owner *"create a new recovery code"* — the one act that would put their old backups
beyond reach. **A freshly installed machine cannot reach this state**, because nothing has set that
flag since 5 August. *(R-246, R-247, R-248)*
flag since 5 August. **Cleared the same day; the machine now compares its key correctly again.** What
is still owed is a ruling on the flag itself: nothing sets it, nothing can see it, and it changes
what a household is told. *(R-246, R-247, R-248)*
## What we're working on
@@ -59,8 +61,8 @@ for byte identical — three separate times, including a filename with Hungarian
## Waiting on you
- **Approve the new base image**, so every future installation carries this week's fixes. *(R-239,
R-242 — presented separately in this session.)*
- **Nothing.** The new base image was approved and is live — every future installation now carries
this week's fixes. *(R-239, R-242)*
*Nothing else is pending. R-245 — whether an undecided household is auto-abandoned after 30 days — was
**settled on 7 August** (we do not build it, and the reasoning is recorded). It has been re-filed as a
File diff suppressed because one or more lines are too long