diff --git a/REPORT-clear-the-ground-2026-08-08.md b/REPORT-clear-the-ground-2026-08-08.md new file mode 100644 index 0000000..2073c5a --- /dev/null +++ b/REPORT-clear-the-ground-2026-08-08.md @@ -0,0 +1,232 @@ +# REPORT — clearing the ground before the next walk (2026-08-08) + +*A sibling report: `REPORT.md` is overwritten per-session and a parallel session shares this clone.* + +**Golden `0.206.0` baked and VOUCHED · the R-242 gate built and shown red→green · demo-hp's stale flag +found wrong and cleared · `STATUS.md` 258 → 87 lines. `felhom-controller` and `felhom-agent` untouched.** + +--- + +## 1. Part 2's gate — FAILING first, then passing + +**Shown failing against today's state, before anything was baked.** That ordering was the instruction +and it is the gate's own red-proof: + +``` + newest released controller : 0.206.0 + newest golden baked : 0.205.0 + +GOLDEN CURRENCY GATE FAILED: controller v0.206.0 is released and NO golden carries it (newest bake is 0.205.0). +A machine installed right now would receive v0.205.0 — the release is written, tested and pushed, and NOT delivered. +``` + +Entry point exit **1**; summary `CONVICTED: golden-currency`. After the bake, the same command: + +``` + newest released controller : 0.206.0 + newest golden baked : 0.206.0 +golden currency gate OK +``` + +**⚠ THE INTRODUCING PUSH USED `--no-verify`, to get past the gate's own conviction.** Stated here, in +`scripts/CHANGELOG.md` and in the commit message rather than worked around. The alternative — baking +first so the gate had never been seen red — was explicitly rejected: a gate that has never been seen +failing has not been shown to work. Every later push in this session was clean. + +### What it does not catch, and why the design is what it is + +**It checks the BAKE, not the VOUCH.** Both `.githooks/pre-push` **and** CI run `repo_gates.py --fast`, +which by contract selects only gates touching no network — so a hub-reading gate registered as +non-fast would run in **neither**, which is exactly the R-29 census failure this runner was built to +end. And the vouched version lives only in `hub_settings`, with no copy in git; putting one there +would create a second source of truth that can drift, and **a green gate over a false claim is the +worst outcome available**. So a bake without a vouch still passes. That half stays open on R-242 +rather than being papered over. + +**It compares versions, not behaviour**, so a release that changed nothing customer-visible also trips +it. **Accepted deliberately, and stated because a gate that cries wolf is one people learn to bypass:** +judging "customer-visible" by hand is precisely what failed three times (R-239, R-242, and this +recurrence), and the cost of a false trip is one bake — the operation the project wants routine anyway. +A waiver belongs in the register, never in a habit of `--no-verify`. + +Inconclusive (exit 2) on an absent controller clone or an unparseable header: not knowing is never a +pass. + +--- + +## 2. Part 1 — the stale blob, all six questions + +**Read-only throughout. Nothing was cleared during the spike** — the clearance in §3 came afterwards, +on the operator's explicit approval. + +**Q1 — what set it, and when. MEASURED, traced to an act, to the second.** At `2026-08-04 20:15:49` +the hub emitted `offsite_reissued` **and** `escrow_stale` in the same second — an operator **Re-issue**, +three minutes after `escrow_blob_served` at 20:12:40 and 20:12:54, i.e. during the R-201 recovery +drill. That is `offsite.ReissueCredentials`'s **precautionary** `MarkEscrowStale` call, which **hub +v0.95.0 removed the next day** (R-196 / R-204 item 2) for marking healthy escrows stale. So: **the +drill's own Re-issue, by code that no longer exists.** *(Both 4-August candidates named in the task +were live that day; the events separate them.)* + +**Q2 — is the flag correct? MEASURED: NO.** The hub's blob seals +`restic_pw_sha256 = 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`; the key the box +is actually using hashes to **the identical value**. The blob covers the key. The flag was wrong from +the moment it was set. + +**Q3 — what clears it? MEASURED: nothing, by itself.** The only writer of `stale_at = NULL` is +`SaveHostEscrow`'s `ON CONFLICT` — a **fresh escrow ceremony**, which is the one act that would +supersede the good blob. **The only exit from the false alarm was the destructive act the false alarm +recommends.** No timer, no self-heal, no reconciler touches it. + +**Q4 — who can see it? Said plainly: effectively only a database read.** + +| audience | what they see | +|---|---| +| the **customer** | a card, but stating a **false reason** (see below) and recommending the destructive act | +| the **box** | **nothing** — `report.EscrowStatus` has no `Stale` field, so it cannot see the flag at all | +| the **operator** | one page: the **PBS-DR** view (`hub/internal/web/pbsdr.go:487`) — the *wrong tier* for an off-site symptom | +| **alerts / notifications** | none. The one-shot `escrow_stale` event fired on 4 August and **was never notified** — a full `notification_log` census for that customer that day returns 8 rows, none of them this one. It has fired twice ever, both on 4 August | + +**A flag that changes behaviour, that nothing sets, and that nobody who would look for it can see.** +Filed as **R-248** in its own right, because it is the shape this fortnight has been about. + +**Q5 — what else does a stale blob suppress? Enumerated from code, not assumed.** (1) the ACK's +`restic_pw_sha256` is withheld; (2) a **pending** box can never auto-confirm; (3) so **every off-site +run is refused indefinitely**; (4) the customer is told to create a new code; (5) **NEW — v0.206.0's +shape (c) is inert**, because the box records an empty hub hash and falls back to (a)/(b). +**(2) and (3) did not bite `demo-hp`**, which was already `escrowed` before the flag landed and has +been backing up healthily throughout — 12 snapshots, last success `2026-08-07T02:15:35Z`. (4) and (5) +did. + +**Q6 — `demo-felhom`? No** — `stale_at` empty, and it records the hub hash normally. **Can a freshly +installed box reach this state? NO, and this is the answer that matters for the next walk.** +`MarkEscrowStale` has **no production caller anywhere in the tree** — a full census returns only its +own definition, two comments and two test references. Nothing has set the column since hub v0.95.0 +shipped on 2026-08-05. **The next walk cannot meet this**, by any route, unless it uses `demo-hp` +itself — and that box is now clear. + +### The sharpest finding: the box states four falsehoods and recommends the destructive act + +Live on `demo-hp`, controller v0.206.0: + +> `STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored +> recovery bundle does not cover the offsite password; create a new recovery code` + +**Every clause is false.** The hub *has* the hash and is *withholding* it; there was no supersession +(`host_escrow_superseded` holds no row for this host); the bundle **does** cover the password. +It raises `EscrowStale`, rendering the customer card *„A letétben lévő helyreállítási csomag nem fedi a +jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."* + +**And the cause is R-241's shape for the third time.** The hub already sends `escrow_stale` on the +wire (`json:"escrow_stale,omitempty"`); the controller's struct has **no matching field**, so +`encoding/json` drops it silently. The box cannot tell *withheld because flagged* from *genuinely +hash-less*, and guesses the latter. **The answer is available and discarded at the boundary** — filed +as **R-247**, and deliberately **not fixed here** (§0 forbids a controller change this session). + +--- + +## 3. The bake, the vouch, and the flag clearance + +| | | +|---|---| +| version | **0.206.0** · sha256 `c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e` | +| size | **656,750,694** bytes (uncompressed 2,003,138,560) | +| MinAgent | 0.127.0 | + +**Round-trip verified, not trusted:** fetched back (HTTP 200, byte count matches), re-hashed +independently (**matches**), `zstd -t` clean, and `tar -xO ./etc/felhom-controller-image` **out of the +download** → `felhom-controller:0.206.0`. That last step is the one that matters, because +`GOLDEN_VERSION` is derived from the tag argument and could be right over stale content. **Third +witness:** the hub's own dropdown lists `0.206.0` with `data-sha="c85230b4…108e"`. + +All acceptance markers pass; unit `Result=success` / `ExecMainStatus=0`; bake VM purged and reverted to +`virgin`; token-leak grep on the committed log **0**, with the instrument proven by a planted copy +first. Evidence: `documentation/tests/golden-0.206.0-2026-08-08/`. + +**VOUCHED with the operator's approval.** Verified from the stored `hub_settings`, not the flash: + +| field | before | after | +|---|---|---| +| `artifact_golden_version` | `0.205.0` | **`0.206.0`** | +| `artifact_golden_sha256` | `8f49b2e8…4ee8` | **`c85230b4…108e`** | +| `artifact_agent_version` | `0.127.0` | `0.127.0` — unchanged | +| `artifact_min_agent` | `0.127.0` | `0.127.0` — unchanged | +| `artifact_wrapper_sha256` | `104db0a4…16b3` | unchanged — **carried through explicitly, because the handler clears it when omitted** | + +**The stale flag was cleared, with the operator's approval.** One row, identity-matched on `host_id` +and guarded on `stale_at IS NOT NULL`; `changes()` returned **1**. **Verified end to end:** the hub +serves the hash again, the box recorded `hub_escrow_key_sha256 = 8a9e33aa…d080a` at `11:10:19Z`, and +that is byte-identical to the key it is using — so shape (c) compares, matches, and correctly stays +silent. **The false warning is gone, proven with a positive control rather than an absent line: 0 +`escrow-confirm` lines since the restart, while 5 scheduler lines in the same window prove the box was +logging and the recorded hash proves an ACK was processed.** + +*Method note, because it touched a production pod:* the hub pod is Alpine with no `sqlite3`; it was +installed into the container's **ephemeral writable layer** — image and node untouched, gone on +restart. SQLite's own file locking coordinated the write with the live hub. An earlier attempt failed +on quoting and **changed nothing**, which is the fail-safe working. + +--- + +## 4. `STATUS.md`, R-245, and the queue audit + +**Rebuilt from the register: 258 → 87 lines.** Not trimmed — the 100-line "what shipped recently" log +was removed outright, because restating the CHANGELOGs here is what made the page grow back. + +**All three named defects fixed:** the "waiting on you" list no longer asks the operator to decide the +**recovery screen** (shipped 2026-08-05) or to approve an orphaned-backup deletion the register +records as **done** the same day; the stray `- **Nothing.**` line is gone; and the DooPlex +infrastructure work is now **under its own heading** — **kept rather than dropped**, with the reason +stated on the page: they are real asks that need the operator, but they concern the machine this is +built on, not what a customer receives. Dropping them would have lost real work. + +**R-245 re-filed as a decision taken**, keeping the whole reasoning and gaining **the condition that +reopens it: quota — old set-aside history blocking new backups. A condition, not a calendar.** + +**The queue audit.** Parsing the **state column exactly** (grepping for the phrase over-matches rows +that merely mention it): **exactly one row carried `WAITING-ON-OPERATOR` — R-245 — and it was the +settled one. So zero rows were genuinely waiting**, and the drift was caught while it was still a +single row. + +--- + +## 5. R-244 — still owed, now measured + +A read-only census, no truncation: `app_log_issues` holds **1309 rows**; **71 reference a torn-down +venue**; of those **44 are orphans** (safely deletable) and **27 are shared with a live customer** and +**must be de-referenced, never deleted**. 1238 untouched. + +**Not done here, and the reason is on the row:** the fix is hub code, this session's scope forbade a +hub version bump, and a hand-run SQL mutation over 71 rows — 27 needing surgical de-referencing — with +no tested code path and no red-proof is the shape that goes wrong on a live database. **What it +needs:** a cascade leg that removes the customer id from `affected_customers` / `context_customer` and +deletes only rows that become empty, plus a one-off sweep for the four venues already gone. The next +session starts from data rather than a guess. + +--- + +## 6. Registers, CI, and what was not done + +**Opened:** **R-246** (the flag: wrong, traced, now cleared — the column ruling still owed), +**R-247** (the box states four falsehoods and recommends the destructive act; the ACK field it needs +is already on the wire and dropped), **R-248** (a behaviour-changing flag visible to nobody). +**Updated:** **R-242** (recurred within a day; bake half now gated, vouch half explicitly still open), +**R-244** (measured), **R-245** (re-filed as decided). **Highest register ID moves R-245 → R-248.** + +**CI:** `felhom.eu` runs to be confirmed by `head_sha` after the final push (below). **`--no-verify` +was used exactly once**, on commit `3ca9a7bbe6e5`, to push past the gate's own conviction — §9.11's +explicit question, answered. + +**No controller or agent change**, as scoped. R-247's fix is a controller change and is therefore +filed rather than made. + +### Observations — noticed, NOT acted on + +- **`stale_at` is a column with no production writer.** It changes what a customer is told, and + nothing can set it. Either give it an evidential setter or retire it; leaving it is leaving a trap + that only a database read can spring. Recorded on R-246; not decided here. +- **The `escrow_stale` event type appears never to be notifiable.** It fired twice and reached + `notification_log` neither time. I did not establish whether it is absent from the dispatcher's + allow-list or merely suppressed, so I have not filed it — but if R-247 is taken up, that is worth + five minutes first. +- **The hub's `EscrowStatus.Stale` field is serialised and has no consumer anywhere.** It is dead + weight on the wire until R-247 gives it one. diff --git a/STATUS.md b/STATUS.md index e6497d4..c346bdf 100644 --- a/STATUS.md +++ b/STATUS.md @@ -42,13 +42,15 @@ for byte identical — three separate times, including a filename with Hungarian ## Found today -- **A leftover flag has been silently switching off the new recovery detection on one demo machine - since 4 August.** A Re-issue during the recovery drill set it, using code we removed the next day. +- **A leftover flag had been silently switching off the new recovery detection on one demo machine + since 4 August — found, and cleared with your approval.** A Re-issue during the recovery drill set it, using code we removed the next day. **The flag is wrong** — the sealed package does cover the key that machine is using, and the two fingerprints match exactly. Because of it the hub withholds a figure the machine needs, so the machine tells its owner *"create a new recovery code"* — the one act that would put their old backups beyond reach. **A freshly installed machine cannot reach this state**, because nothing has set that - flag since 5 August. *(R-246, R-247, R-248)* + flag since 5 August. **Cleared the same day; the machine now compares its key correctly again.** What + is still owed is a ruling on the flag itself: nothing sets it, nothing can see it, and it changes + what a household is told. *(R-246, R-247, R-248)* ## What we're working on @@ -59,8 +61,8 @@ for byte identical — three separate times, including a filename with Hungarian ## Waiting on you -- **Approve the new base image**, so every future installation carries this week's fixes. *(R-239, - R-242 — presented separately in this session.)* +- **Nothing.** The new base image was approved and is live — every future installation now carries + this week's fixes. *(R-239, R-242)* *Nothing else is pending. R-245 — whether an undecided household is auto-abandoned after 30 days — was **settled on 7 August** (we do not build it, and the reasoning is recorded). It has been re-filed as a diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index f2d2e3b..5a2c8b8 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -169,14 +169,14 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour | **R-241** | **The credential self-heal, succeeding, locks the customer out of their own recovery.** Measured end to end on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`). `OffsiteRecoveryOffer()` shows the recovery screen on exactly two conditions: **(a)** the box has **no** repository password — the pristine rebuilt shape — or **(b)** it has one but the inherited history will not open under it (`OffboxOrphaned()`). Overnight, unaided and exactly as designed, `offsiteheal` re-staged the one-time credential and the box's 5-minute retry **collected it and applied the tier**, writing a **fresh repository password** at 03:18Z. That makes **(a) false**. **(b)** is false too, because orphan detection only fires when a run actually tries the repository — and runs are blocked by `escrow_state: pending`. **The box therefore sits in the gap between the two conditions, and the gap is self-locking:** it cannot detect the orphan without running, cannot run without escrow, and cannot escrow without minting a NEW recovery code — which would orphan the history the customer's existing code protects. **What the customer sees:** `/` is „Indítópult" with no recovery pointer; `/recovery` **302s away**; `/backups/remote` offers „Helyreállítási kód **létrehozása**". **There is no field anywhere to enter the code they hold.** **And the operator's documented remedy also refuses** — `--recover-offsite-install` returns *„[REFUSED] a DIFFERENT repository password is already present… which history to keep is not a decision this command may take. Nothing written."*, which is correct and fail-closed and still a dead end. Recovery required moving the fresh key aside by hand and re-running the install: **three guest command lines**. **The two keys, measured:** on-disk `9b4a9a9d…` (self-heal) vs recovered-from-R `30ef574f…`. **THE DATA WAS NEVER AT RISK** — all three sentinels restored byte-identical once the right key was in place. **This is R-218's shape one level up:** that finding read *"succeeding at recovery stopped the box asking for what it still needed"*; here, succeeding at the credential self-heal stopped the box **offering** the recovery it still needed. The same walk proved the self-heal working unaided six hours earlier, and that success is what causes this. **Likely shape of the fix, not yet a decision:** the offer needs a third condition — a box holding a password it has never successfully used, while the hub holds a sealed package, is a recovery candidate — or the self-heal must not install a credential on a box whose escrow is still `pending` and whose hub blob is unconsumed. **Which of those is right is a design decision, deliberately not taken here.** **⚠ RULED 2026-08-07 by a read-only spike on the standing venue — `audits/SPIKE-r241-recovery-offer-2026-08-07.md`. IT IS A MINTING DEFECT, NOT A SCREEN-PREDICATE DEFECT, and that reverses the fix.** The screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub was holding. **Three measurements carry it.** (1) **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** It never reads `GetHubEscrowIdentityPresent()`, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and `needsOffsiteCredential()` (`:1377`), both do. **The same fact is available on three paths and used on two.** (2) **The flag was not merely available — it was the precondition of the chain that reached the minting.** The 5-minute retry job only logs when `RetryIfDeclared` fires, which requires the declaration, which requires that flag; the venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times after — **thirty minutes and six ticks before the mint at 03:18:06Z**. (3) **The box KNEW and threw it away:** at **03:28:03Z**, thirty-five minutes before the customer looked, `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) computed the exact discriminator and logged `[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`. **It is computed on every report cycle, never persisted, never surfaced.** **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session implements and must not quietly widen."* The repository key is minted in the seam between two sides that each honoured their contract, by a helper doing exactly what its doc comment says. **Fixing the predicate would paper over a box quietly making its own history unopenable.** **Recommended fix (not started, no code written): persist the discriminator the ACK already carries and add it as shape (c)** — needs no hub change and no new protocol field — **plus a `decided` latch**, because `ResetOrphanedRepo` clears `RepoState` without running a ceremony, so an `H`-mismatch discriminator alone would re-offer the screen forever to a customer who explicitly declined the old data. **Four operator decisions are stated and left unanswered in §"THE OPERATOR'S DECISION".** **✅ FIXED 2026-08-07 — controller v0.206.0 + hub v0.98.0, and the ruling above is what the fix follows.** **(1) It stops minting:** the guard is a CONJUNCTION (a package held AND no key present), so a first-time box mints exactly as before; the refusal is a HOLDING state, not a failure — the transport is still written so the recovery screen can bring the tier up the instant the key arrives (R-219), and returning an error instead would have left the hub re-staging a consumed credential for ever. New declared state `offsite.state=awaiting_recovery_key`, shown INERT to every existing hub reader from their code rather than assumed. **(2) The discriminator is persisted and drives the offer as shape (c).** §7.2 resolved deliberately: **a known difference offers however old the reading** (age is NOT gated on — gating would make a box offline from the hub silently stop offering, the very failure this removes), and **a hash never learned falls back to (a)/(b)**, because an empty hash is the hub positively saying its package seals no key rather than an unknown. **(3) Abandoning ends the question** — a 14-day countdown, visible and reversible, whose terminal step removes the set-aside store AND the sealed package together, after which shape (c) has nothing to compare and the offer falls silent **because the state is right, not because something remembers it once was not**. The two halves cannot be atomic across two machines, so it is a two-phase commit whose confirmation rides the SAME ACK that carries the request. **(4) The surface:** the full page appears **once per ENTRY into the offered state, not once ever** (an epoch — a box rebuilt months later is a new situation); three dismissal levers with three scopes, and **none removes the entry point on the backups page**. **Q7's trap does not survive:** while a recovery is outstanding „Helyreállítási kód létrehozása" is **unavailable**, not merely captioned. **§2.4 honoured:** the abandon confirmation no longer promises *„félretesszük — nem töröljük"* — it states the deletion date. **TWO REAL BUGS WERE CAUGHT BY TESTS RATHER THAN REVIEW, and both are recorded because the shape matters:** `OffboxAwaitingRecoveryKey` omitted `t.Enabled`, so a customer who had switched off-site OFF would have declared a holding state (caught by the EXISTING `TestOffsiteDeclare_DisabledTargetIsNotStranded`); and `recoveryInterrupts` returned early when the offer was false, so the FALLING edge was never recorded and the full page never came back — **the exact defect the epoch exists to fix, reintroduced inside the fix**. **Nine red-proofs, each with the mutation confirmed present in the file before its result was trusted**, including the ships-inert one (unwiring `RecordEscrowKeyHash`, which leaves everything compiling and every test passing while shape (c) reads an empty hash for ever). **NOTHING WAS DELETED ANYWHERE** — the terminal step has only ever run against injected fakes and an injected clock (§7.4). **Still open and NOT built by this:** R-242 (the release-to-golden gate) and R-245 (the automatic 30-day ending). | **FIXED 2026-08-07 — v0.206.0 / hub v0.98.0** | -| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** **⚠ IT RECURRED WITHIN A DAY, WHICH IS THE ARGUMENT FOR BUILDING IT.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08 would have received neither. Third occurrence of the shape in three days (R-111/R-115/R-120 are the older family). **✅ SHAPE (b) BUILT 2026-08-08 — `scripts/golden_currency_gate.py`, registered in `repo_gates.py` as gate 7.** **It was shown FAILING against that exact state before anything was baked**, which is its red-proof and the reason its own introducing push needed `--no-verify` (stated in the session report rather than worked around): `newest released controller : 0.206.0 / newest golden baked : 0.205.0 → CONVICTED`. **IT IS `--fast`, AND THAT FORCED ITS DESIGN:** both `.githooks/pre-push` AND CI run `repo_gates.py --fast`, so a non-fast gate would run in NEITHER — the R-29 census failure this runner exists to end. **THEREFORE IT CHECKS THE BAKE, NOT THE VOUCH**, because the vouched version lives only in the hub's `hub_settings` with no copy in git, and putting a copy there would create a second source of truth that can drift — a green gate over a false claim being the worst outcome available. **A bake without a vouch still passes: that half is NOT closed and stays on this row.** It also compares versions rather than behaviour, so a release changing nothing customer-visible trips it too — accepted deliberately, because judging that by hand is what failed three times and the cost of a false trip is one bake; a waiver belongs here, never in a habit of bypassing. | **PARTLY BUILT 2026-08-08 — the bake half is gated; the VOUCH half is not** | +| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** **⚠ IT RECURRED WITHIN A DAY, WHICH IS THE ARGUMENT FOR BUILDING IT.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08 would have received neither. Third occurrence of the shape in three days (R-111/R-115/R-120 are the older family). **✅ SHAPE (b) BUILT 2026-08-08 — `scripts/golden_currency_gate.py`, registered in `repo_gates.py` as gate 7.** **It was shown FAILING against that exact state before anything was baked**, which is its red-proof and the reason its own introducing push needed `--no-verify` (stated in the session report rather than worked around): `newest released controller : 0.206.0 / newest golden baked : 0.205.0 → CONVICTED`. **IT IS `--fast`, AND THAT FORCED ITS DESIGN:** both `.githooks/pre-push` AND CI run `repo_gates.py --fast`, so a non-fast gate would run in NEITHER — the R-29 census failure this runner exists to end. **THEREFORE IT CHECKS THE BAKE, NOT THE VOUCH**, because the vouched version lives only in the hub's `hub_settings` with no copy in git, and putting a copy there would create a second source of truth that can drift — a green gate over a false claim being the worst outcome available. **A bake without a vouch still passes: that half is NOT closed and stays on this row.** It also compares versions rather than behaviour, so a release changing nothing customer-visible trips it too — accepted deliberately, because judging that by hand is what failed three times and the cost of a false trip is one bake; a waiver belongs here, never in a habit of bypassing. **VOUCHED 2026-08-08 with the operator's approval** — golden `0.206.0` / sha `c85230b4…108e`; `agent_version` and `min_agent` both stayed `0.127.0`, and `wrapper_sha256` was carried through explicitly because the handler clears it when omitted. **The gate was CONVICTED before the bake and OK after it** — red→green on the same command, which is its proof that it measures something real. | **PARTLY BUILT 2026-08-08 — the bake half is gated; the VOUCH half is not** | | **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** **⚠ UPDATED 2026-08-07 (v0.206.0) — the STATE this row describes can no longer be entered, but the ALARM GAP is untouched and the row stays open.** R-241's mint guard means a box no longer mints a key over a sealed package, so it no longer arrives in the "escrow stuck pending against a self-minted key" state by itself. **What replaces it is a state that is VISIBLE rather than silent:** the box declares `offsite.state=awaiting_recovery_key` and the customer is offered the recovery screen. **But the hub still raises nothing for it**, and for the same three reasons: `isStale` needs `escrowed`, the delivery checker skips the `applied` shape, and `backup_failed` needs a run that never happens. **So a box whose customer never acts still stops backing up off-site with no operator signal** — the difference is that the customer can now see it and act, where before nobody could. **The remaining work is an operator-side signal for a box held in `awaiting_recovery_key` past some age**, and it is deliberately not bundled into R-241's fix. | **READY** — owner Viktor | | **R-244** | **The customer DELETE cascade leaves `app_log_issues` behind, and it is systematic across every venue ever torn down.** Found **2026-08-07** while verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a per-table query. After a cascade that logged `COMPLETE … full teardown`, **61 rows still matched `finalwalk`**. Four of the five sources are **deliberate and correct** — the cascade's own header states *"Provenance/events are NEVER wiped — audit outlives every tier"*: `events` 16, `notification_log` 14, `host_deletions` 1, `customer_resets` 1. **The fifth is a gap:** `app_log_issues` 29 rows, which the residue purge does not touch (its logged leg covers `reports`/`app_telemetry`/`app_log_tails`/`log_tail_requests`/`notif_prefs`/`selfbind_tokens`/`appliance_registrations` — not this table). **It is not a `finalwalk` quirk:** rows still reference **`c11` 40, `rewalk` 20, `part4` 24** — all three torn down 2026-08-06, whose ledger recorded *"0 occurrences"*. **That prior claim was measured with a narrower query and does not survive a full census; the correction is recorded rather than the measurement quietly redone.** **Why it was probably never written, established rather than assumed:** the table is a **fleet-wide aggregate** keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list — of the 29 `finalwalk` rows, **12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with LIVE customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted, only de-referenced.** A naive `DELETE … WHERE customer LIKE` would destroy a live customer's issue history — which is very likely why the leg does not exist, and is the reason this is not a one-line fix. **Severity is LOW and stated plainly: no secret material is involved** — app name, fingerprint, message text, counts, timestamps. What survives is a deleted customer's *identifier* inside an aggregate row. **Proposed shape:** a residue leg that (a) removes the customer id from `affected_customers`/`context_customer`, and (b) deletes rows whose `affected_customers` becomes empty; plus a one-off sweep for the four already-torn-down venues. **The general lesson is the reusable part:** *a per-table absence query is not a census.* The teardown verification is now a full-schema sweep, and that is what found this. **Not fixed** — a cascade change needs its own red-proof and this session was scoped as a spike plus two operations. Evidence: `tests/teardown-finalwalk-2026-08-07.md`. **⚠ STILL OWED, AND NOW MEASURED RATHER THAN ESTIMATED (2026-08-08 census, read-only, no truncation).** `app_log_issues` holds **1309 rows**; **71 reference a torn-down venue** (`finalwalk`, `c11`, `rewalk`, `part4`); of those **44 are ORPHANS** — they name only torn-down customers and are safely deletable — and **27 are SHARED with a live customer** (`demo-felhom`, `peti-felhom`, …) and **must be de-referenced, never deleted**. 1238 rows are untouched. **The 27 are exactly why the leg was never written**, and why a `DELETE … WHERE customer LIKE` would destroy a live customer's issue history. **What it needs, precisely:** a cascade leg that (a) removes the customer id from `affected_customers` / `context_customer`, and (b) deletes only rows whose `affected_customers` becomes empty; plus a one-off sweep for the four venues already gone. **Why it was NOT done on 2026-08-08:** the fix is hub code, and that session's scope forbade a hub version bump; a hand-run SQL mutation over 71 rows — 27 of them needing surgical de-referencing — with no tested code path and no red-proof is precisely the shape that goes wrong on a live database. **It accumulates one venue at a time, so the next walk adds to it**; the numbers above mean the next session starts from data rather than a guess. | **READY** — owner Viktor | | **R-245** | **Should a customer who never decides be auto-abandoned after 30 days? RECORDED, NOT BUILT — and the reasoning against it is recorded with it so the decision can be revisited properly.** **The operator's proposal (2026-08-07):** a box that has been offered recovery for 30 days without the customer deciding is auto-abandoned, entering the 14-day grace, so an undecided box does not sit for ever holding history nobody has claimed. **What was built instead:** the escalating reminders (1/3/7/14 days) and the operator levers `--abandon-extend` / `--abandon-stop`. **The reasoning, as settled with the operator the same day:** (1) **nobody is absent** — a box does not reinstall itself, so whoever rebuilt it was standing there and met the recovery question; the "customer away for months" case does not arise from this situation, because **a reinstall implies a person**. (2) **A customer who cannot find their code will get in touch**, which is the moment to extend or disarm by hand — so the automation would be firing at people we are already talking to, which is why the levers were the thing worth building. (3) **The cost is theirs**: the old history sits in the customer's own storage allowance, and if they are paying to keep something they have not decided about, that is their call and they feel it before we do. (4) **The real harm, if it comes, is QUOTA** — old history blocking new backups — and **that is a condition, not a calendar**. An automatic ending should trigger on the harm, with a dated warning, never on a date alone. **If this is ever built, build it that way.** **✅ RE-FILED 2026-08-08 AS A DECISION TAKEN, not a question pending.** It sat in the operator's queue as `WAITING-ON-OPERATOR` for a day, and **nothing was actually pending** — the operator and the reviewer settled it on 2026-08-07: it is **not built**, the levers were built instead, and the whole reasoning above is the record of why. A settled decision parked in a queue is a queue nobody trusts, and an audit of every `WAITING-ON-OPERATOR` row the same day found this was the ONLY one — so the drift was caught while it was still a single row. **THE CONDITION THAT REOPENS IT, which the reasoning already names: QUOTA — old set-aside history blocking new backups.** Not a calendar. If a customer's retained history ever refuses a new backup, revisit this with a dated warning that triggers on the refusal; until then it stays decided. | **DECIDED 2026-08-07 — not built; reopens on quota** | -| **R-246** | **A leftover staleness flag has silently disabled the new recovery discriminator on `demo-hp` since 2026-08-04, and the flag is WRONG.** Found by a read-only spike, 2026-08-08. **Q1 — traced to an act, to the second:** at `2026-08-04 20:15:49` the hub emitted `offsite_reissued` and `escrow_stale` in the same second — an operator **Re-issue** pressed during the R-201 drill, three minutes after `escrow_blob_served` at 20:12:40/20:12:54. That was `offsite.ReissueCredentials`'s **precautionary** `MarkEscrowStale` call, which **hub v0.95.0 REMOVED the very next day** (R-196 / R-204 item 2) precisely because it marked healthy escrows stale. **Q2 — the flag is wrong, measured on both sides:** the hub's blob seals `restic_pw_sha256 = 8a9e33aa4da6769c…d080a`, and the key the box is actually using hashes to **the identical value**. The blob covers the key. **Q3 — nothing clears it by itself:** the ONLY writer of `stale_at = NULL` is `SaveHostEscrow`'s `ON CONFLICT` — i.e. a fresh escrow ceremony, **which is the one act that would supersede the good blob**. So the only exit from a false alarm is the destructive act the false alarm recommends. **Q5 — the blast radius, enumerated rather than assumed:** (1) `GetEscrowStatusForCustomer` withholds `restic_pw_sha256` from the ACK; (2) a **pending** box can never auto-confirm, so (3) **every off-site run is refused indefinitely** — neither bites `demo-hp`, which was already `escrowed` and is backing up healthily (12 snapshots, last success 2026-08-07T02:15:35Z); (4) the customer is told to create a new code; (5) **NEW — v0.206.0's shape (c) is inert**, because the box records an empty hub hash and falls back to (a)/(b), so the recovery screen would stay silent even if recovery were needed. **Q6 — a fresh box CANNOT reach this state:** `MarkEscrowStale` has **no production caller anywhere in the tree** (census: only its own definition, two comments and two test references). **The next walk cannot meet it.** **NOT CLEARED, deliberately** — Q2's answer says the fix is to clear this instance, but whether to also stop the column being settable at all is a separate ruling, and the spike was scoped read-only. **What it needs:** an operator decision to clear `stale_at` for `demo-hp-bb76ea` (a one-row UPDATE), and a ruling on whether the column keeps a live setter. | **READY** — owner Viktor | +| **R-246** | **A leftover staleness flag has silently disabled the new recovery discriminator on `demo-hp` since 2026-08-04, and the flag is WRONG.** Found by a read-only spike, 2026-08-08. **Q1 — traced to an act, to the second:** at `2026-08-04 20:15:49` the hub emitted `offsite_reissued` and `escrow_stale` in the same second — an operator **Re-issue** pressed during the R-201 drill, three minutes after `escrow_blob_served` at 20:12:40/20:12:54. That was `offsite.ReissueCredentials`'s **precautionary** `MarkEscrowStale` call, which **hub v0.95.0 REMOVED the very next day** (R-196 / R-204 item 2) precisely because it marked healthy escrows stale. **Q2 — the flag is wrong, measured on both sides:** the hub's blob seals `restic_pw_sha256 = 8a9e33aa4da6769c…d080a`, and the key the box is actually using hashes to **the identical value**. The blob covers the key. **Q3 — nothing clears it by itself:** the ONLY writer of `stale_at = NULL` is `SaveHostEscrow`'s `ON CONFLICT` — i.e. a fresh escrow ceremony, **which is the one act that would supersede the good blob**. So the only exit from a false alarm is the destructive act the false alarm recommends. **Q5 — the blast radius, enumerated rather than assumed:** (1) `GetEscrowStatusForCustomer` withholds `restic_pw_sha256` from the ACK; (2) a **pending** box can never auto-confirm, so (3) **every off-site run is refused indefinitely** — neither bites `demo-hp`, which was already `escrowed` and is backing up healthily (12 snapshots, last success 2026-08-07T02:15:35Z); (4) the customer is told to create a new code; (5) **NEW — v0.206.0's shape (c) is inert**, because the box records an empty hub hash and falls back to (a)/(b), so the recovery screen would stay silent even if recovery were needed. **Q6 — a fresh box CANNOT reach this state:** `MarkEscrowStale` has **no production caller anywhere in the tree** (census: only its own definition, two comments and two test references). **The next walk cannot meet it.** **NOT CLEARED, deliberately** — Q2's answer says the fix is to clear this instance, but whether to also stop the column being settable at all is a separate ruling, and the spike was scoped read-only. **✅ THE FLAG IS CLEARED — operator-approved and applied 2026-08-08.** One row, identity-matched on `host_id` and guarded on `stale_at IS NOT NULL`; `changes()` returned **1**. **Verified end to end, not just in the database:** the hub now serves the hash again, the box recorded `hub_escrow_key_sha256 = 8a9e33aa4da6769c…d080a` at `11:10:19Z`, and that is **byte-identical to the key it is using** — so shape (c) compares, matches, and correctly stays silent. **The false stale warning is gone, proven with a positive control** rather than an absent line: **0** `escrow-confirm` lines since the restart while **5** scheduler lines in the same window prove the box was logging, and the recorded hash proves an ACK was processed. *(Method note: the hub pod is Alpine with no `sqlite3`; it was installed into the container's ephemeral writable layer — image and node untouched, gone on restart. SQLite's own file locking coordinated the write with the live hub; an earlier attempt failed cleanly on quoting and changed nothing, which is the fail-safe working.)* **STILL OPEN under this ID: the ruling on whether `stale_at` keeps a live setter at all.** It currently has NO production caller, so the column is write-only-by-accident — a field that changes behaviour, that nothing sets and nothing can see (R-248). Either give it an evidential setter or retire it; **do not leave it as a trap that only a database read can spring.** | **READY — flag cleared; the column ruling is still owed** — owner Viktor | | **R-247** | **The box is being told something false, in its own words, and it recommends the destructive act.** `demo-hp`, live, on controller v0.206.0: *"STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored recovery bundle does not cover the offsite password; create a new recovery code"*. **Every clause is false.** The hub HAS the hash and is *withholding* it (R-246); there was no supersession (`host_escrow_superseded` has no row for this host); and the bundle **does** cover the password — the hashes match exactly. It raises `EscrowStale`, which renders the customer-facing card *„A letétben lévő helyreállítási csomag nem fedi a jelenlegi távoli mentési jelszót. Hozzon létre új helyreállítási kódot."* **THE CAUSE IS A FACT ON THE WIRE THAT THE BOX THROWS AWAY.** The hub already sends `escrow_stale` in the ACK (`json:"escrow_stale,omitempty"`), and the controller's `report.EscrowStatus` **has no matching field**, so `encoding/json` drops it silently. The box therefore cannot distinguish *withheld because flagged stale* from *genuinely hash-less*, and guesses the latter. **This is R-241's shape for the third time: the answer is available, and it is discarded at the boundary.** **The fix is small and is NOT made here** (§0 forbids a controller change this session): add the field, and say the true thing — or say nothing, since on an ESCROWED box with matching hashes there is nothing wrong to report. | **READY** — owner Viktor | | **R-248** | **A flag that changes behaviour is visible to nobody who would look for it.** Q4 of the 2026-08-08 spike, answered plainly. **The customer** sees only a derived card stating a false reason (R-247). **The box** cannot see it at all (R-247's dropped field). **The operator** can see it on exactly ONE page — the **PBS-DR** view (`hub/internal/web/pbsdr.go:487`, `v.EscrowStale = escrow.StaleAt != ""`) — which is the wrong tier for this symptom: an operator investigating an OFF-SITE problem has no reason to open a PBS-DR page. **No alert, no report field, no off-site surface.** The one-shot `escrow_stale` event fired on 2026-08-04 and **was never notified** (a full census of `notification_log` for that customer that day returns 8 rows, none of them this one); it has fired twice ever, both on 4 August. **So the practical answer is: only a database read.** That is a finding in its own right — a flag that silently changes behaviour and cannot be seen is the shape this fortnight has been about (R-241's discarded comparison, R-228's unread field, R-243's unobserved state). **What it needs:** surface `stale_at` on the off-site operator surface and in the host report, or stop using a field nobody can observe to change what a customer is told. | **READY** — owner Viktor |