R-241 FIXED: registers, capability map, STATUS, hub CHANGELOG v0.98.0
gates / gates (push) Successful in 14s
gates / gates (push) Successful in 14s
R-241 closed against controller v0.206.0 + hub v0.98.0, following the spike's ruling rather than the obvious reading. The row records what the fix does AND the two real bugs the tests caught rather than review - a missing t.Enabled (caught by an EXISTING test) and a missing falling-edge sync that reintroduced the very defect the epoch exists to fix. R-243 UPDATED, not closed: the STATE it describes can no longer be entered (the mint guard), and what replaces it is VISIBLE rather than silent - the box declares awaiting_recovery_key and the customer is offered the screen. But the ALARM GAP is untouched, for the same three reasons, so a box whose customer never acts still stops backing up with no operator signal. The remaining work is an operator-side signal for a box held past some age, deliberately not bundled into R-241's fix. R-245 NEW - WAITING-ON-OPERATOR, recorded and NOT built: should an undecided customer be auto-abandoned after 30 days? The operator's proposal is recorded WITH the reasoning against it, so the decision can be revisited properly: a reinstall implies a person, so nobody is absent; a customer who cannot find their code gets in touch, which is why the operator LEVERS were the thing worth building; the cost is the customer's own storage allowance; and the real harm is QUOTA, which is a condition, not a calendar. If it is ever built, build it to trigger on the harm with a dated warning, never on a date alone. The capability map's recovery-journey row STAYS FAIL. These are fixes, not a walk - nothing here walked a customer end to end, and the row goes green only when one completes with no operator intervention AND a byte-identical sentinel. R-214, R-202 and R-240 are still open. STATUS compressed rather than extended, per its own one-screen rule, and the "rebuilding throws away the off-site history" line corrected: the cause is fixed, so leaving it as a live defect would be false. hub CHANGELOG v0.98.0 for the superseded-package purge. Highest register ID moves R-244 -> R-245.
This commit is contained in:
1 parent
ac4b2a4ba9
commit
9657334fb7
4 files changed
+69
-39
No files matched your search
@@ -167,13 +167,15 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
|
||||
| **R-239** | **The fixes are written, tested, pushed — and a machine installed tonight gets none of them.** Measured on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`): a fresh install from the published ISO landed on agent **0.127.0** (vouched, **no hand upgrade** — that half works) and controller **0.203.0**, while the newest released controller is **0.205.0**. The vouched golden still bakes 0.203.0, so a new box has neither **R-237** (v0.204.0 — the restore list driven by the store) nor **R-234** (v0.205.0 — the skipped-app verdict and the single-flight message). **Proven from the customer's side on that box, not inferred:** T3 — with the future-backup toggle switched off, the restore page showed **0** entries for an app whose snapshot demonstrably exists, and the wizard redirected away; T2 — two off-site runs back to back both answered „A távoli mentés elindult", the second as a SUCCESS flash, which is exactly the sequence that made a customer read a stale green tick as covering an app they had just selected. **Not a regression and not a code defect — a delivery gap.** The remedy is already documented: bake a golden carrying the current controller and vouch it as a three-field change (`golden_version` + `agent_version` + `min_agent`), per `runbooks/RUNBOOK-manual-build.md` §4.1. **Until that happens, every closure claimed for R-234 and R-237 is true of the code and false of the fleet** — which is the distinction this register exists to keep. **✅ CLOSED 2026-08-07 — golden `0.205.0` baked, published, round-trip verified and VOUCHED (operator-approved this session).** Evidence: `tests/golden-0.205.0-2026-08-07/`. sha256 `8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8`, 656,937,561 bytes. **Verified by round trip rather than trusted:** the published bytes were fetched back, re-hashed (match), `zstd -t`'d clean, and `./etc/felhom-controller-image` was read **out of the downloaded archive** → `felhom-controller:0.205.0` — the step that matters, because `GOLDEN_VERSION` is derived from the tag argument and could have been right over stale content. A **third independent witness**: the hub's own Day-0 dropdown lists `0.205.0` with `data-sha="8f49b2e8…"`. **In the event it was a ONE-field change, not three:** `golden_version` 0.203.0→0.205.0 (plus its derived sha); `agent_version` and `min_agent` both stayed **0.127.0**, because the new golden's MinAgent is also 0.127.0 — read from live `hub_settings` before and after, not assumed. `wrapper_sha256` was carried through explicitly: the handler reads it from the form and **clears it when omitted**, so a headless POST that forgets it silently drops the PBS-DR wrapper hash. **The R-120 gate passed exactly** — the newest controller the fleet reports is 0.205.0 (demo-hp), so a 0.204.0 golden would have been REFUSED. Vouching is reversible by re-selecting the previous values; a bake never deletes an older golden's package. **The systemic half is NOT closed** — nothing enforces that a release reaches a golden → **R-242**. | **CLOSED 2026-08-07** |
|
||||
| **R-240** | **A backup that covered nothing calls itself „Sikeres".** On a configured box with no app selected for off-site backup, a run reports status `ok` with the warning „Sikeres — nincs mentésre jelölt alkalmazás" — *successful* immediately beside *nothing is selected*. Measured as T4 on the final walk, 2026-08-07; flagged once before (2026-08-06) and deliberately not touched then, because the task that noticed it forbade changing that path. **It is the same rhetorical shape the project has spent a fortnight removing** — R-203's *a warning beside a success is read as a success*, R-234's *„✓ Rendben" over an app that was skipped*, R-225's *unknown rendered as zero* — one notch weaker each time, and this is the weakest and last of them. The state itself is honest and must stay `ok`: an unconfigured box reporting `incomplete` forever is its own defect, pinned by a test. **The defect is the word „Sikeres", not the verdict.** Wording such as „Nincs mentésre jelölt alkalmazás — ez a futás semmit nem mentett" says the same thing without congratulating the customer on it. | **READY** — owner Viktor |
|
||||
|
||||
| **R-241** | **The credential self-heal, succeeding, locks the customer out of their own recovery.** Measured end to end on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`). `OffsiteRecoveryOffer()` shows the recovery screen on exactly two conditions: **(a)** the box has **no** repository password — the pristine rebuilt shape — or **(b)** it has one but the inherited history will not open under it (`OffboxOrphaned()`). Overnight, unaided and exactly as designed, `offsiteheal` re-staged the one-time credential and the box's 5-minute retry **collected it and applied the tier**, writing a **fresh repository password** at 03:18Z. That makes **(a) false**. **(b)** is false too, because orphan detection only fires when a run actually tries the repository — and runs are blocked by `escrow_state: pending`. **The box therefore sits in the gap between the two conditions, and the gap is self-locking:** it cannot detect the orphan without running, cannot run without escrow, and cannot escrow without minting a NEW recovery code — which would orphan the history the customer's existing code protects. **What the customer sees:** `/` is „Indítópult" with no recovery pointer; `/recovery` **302s away**; `/backups/remote` offers „Helyreállítási kód **létrehozása**". **There is no field anywhere to enter the code they hold.** **And the operator's documented remedy also refuses** — `--recover-offsite-install` returns *„[REFUSED] a DIFFERENT repository password is already present… which history to keep is not a decision this command may take. Nothing written."*, which is correct and fail-closed and still a dead end. Recovery required moving the fresh key aside by hand and re-running the install: **three guest command lines**. **The two keys, measured:** on-disk `9b4a9a9d…` (self-heal) vs recovered-from-R `30ef574f…`. **THE DATA WAS NEVER AT RISK** — all three sentinels restored byte-identical once the right key was in place. **This is R-218's shape one level up:** that finding read *"succeeding at recovery stopped the box asking for what it still needed"*; here, succeeding at the credential self-heal stopped the box **offering** the recovery it still needed. The same walk proved the self-heal working unaided six hours earlier, and that success is what causes this. **Likely shape of the fix, not yet a decision:** the offer needs a third condition — a box holding a password it has never successfully used, while the hub holds a sealed package, is a recovery candidate — or the self-heal must not install a credential on a box whose escrow is still `pending` and whose hub blob is unconsumed. **Which of those is right is a design decision, deliberately not taken here.** **⚠ RULED 2026-08-07 by a read-only spike on the standing venue — `audits/SPIKE-r241-recovery-offer-2026-08-07.md`. IT IS A MINTING DEFECT, NOT A SCREEN-PREDICATE DEFECT, and that reverses the fix.** The screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub was holding. **Three measurements carry it.** (1) **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** It never reads `GetHubEscrowIdentityPresent()`, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and `needsOffsiteCredential()` (`:1377`), both do. **The same fact is available on three paths and used on two.** (2) **The flag was not merely available — it was the precondition of the chain that reached the minting.** The 5-minute retry job only logs when `RetryIfDeclared` fires, which requires the declaration, which requires that flag; the venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times after — **thirty minutes and six ticks before the mint at 03:18:06Z**. (3) **The box KNEW and threw it away:** at **03:28:03Z**, thirty-five minutes before the customer looked, `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) computed the exact discriminator and logged `[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`. **It is computed on every report cycle, never persisted, never surfaced.** **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session implements and must not quietly widen."* The repository key is minted in the seam between two sides that each honoured their contract, by a helper doing exactly what its doc comment says. **Fixing the predicate would paper over a box quietly making its own history unopenable.** **Recommended fix (not started, no code written): persist the discriminator the ACK already carries and add it as shapeLine truncated
|
||||
| **R-241** | **The credential self-heal, succeeding, locks the customer out of their own recovery.** Measured end to end on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`). `OffsiteRecoveryOffer()` shows the recovery screen on exactly two conditions: **(a)** the box has **no** repository password — the pristine rebuilt shape — or **(b)** it has one but the inherited history will not open under it (`OffboxOrphaned()`). Overnight, unaided and exactly as designed, `offsiteheal` re-staged the one-time credential and the box's 5-minute retry **collected it and applied the tier**, writing a **fresh repository password** at 03:18Z. That makes **(a) false**. **(b)** is false too, because orphan detection only fires when a run actually tries the repository — and runs are blocked by `escrow_state: pending`. **The box therefore sits in the gap between the two conditions, and the gap is self-locking:** it cannot detect the orphan without running, cannot run without escrow, and cannot escrow without minting a NEW recovery code — which would orphan the history the customer's existing code protects. **What the customer sees:** `/` is „Indítópult" with no recovery pointer; `/recovery` **302s away**; `/backups/remote` offers „Helyreállítási kód **létrehozása**". **There is no field anywhere to enter the code they hold.** **And the operator's documented remedy also refuses** — `--recover-offsite-install` returns *„[REFUSED] a DIFFERENT repository password is already present… which history to keep is not a decision this command may take. Nothing written."*, which is correct and fail-closed and still a dead end. Recovery required moving the fresh key aside by hand and re-running the install: **three guest command lines**. **The two keys, measured:** on-disk `9b4a9a9d…` (self-heal) vs recovered-from-R `30ef574f…`. **THE DATA WAS NEVER AT RISK** — all three sentinels restored byte-identical once the right key was in place. **This is R-218's shape one level up:** that finding read *"succeeding at recovery stopped the box asking for what it still needed"*; here, succeeding at the credential self-heal stopped the box **offering** the recovery it still needed. The same walk proved the self-heal working unaided six hours earlier, and that success is what causes this. **Likely shape of the fix, not yet a decision:** the offer needs a third condition — a box holding a password it has never successfully used, while the hub holds a sealed package, is a recovery candidate — or the self-heal must not install a credential on a box whose escrow is still `pending` and whose hub blob is unconsumed. **Which of those is right is a design decision, deliberately not taken here.** **⚠ RULED 2026-08-07 by a read-only spike on the standing venue — `audits/SPIKE-r241-recovery-offer-2026-08-07.md`. IT IS A MINTING DEFECT, NOT A SCREEN-PREDICATE DEFECT, and that reverses the fix.** The screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub was holding. **Three measurements carry it.** (1) **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** It never reads `GetHubEscrowIdentityPresent()`, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and `needsOffsiteCredential()` (`:1377`), both do. **The same fact is available on three paths and used on two.** (2) **The flag was not merely available — it was the precondition of the chain that reached the minting.** The 5-minute retry job only logs when `RetryIfDeclared` fires, which requires the declaration, which requires that flag; the venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times after — **thirty minutes and six ticks before the mint at 03:18:06Z**. (3) **The box KNEW and threw it away:** at **03:28:03Z**, thirty-five minutes before the customer looked, `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) computed the exact discriminator and logged `[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`. **It is computed on every report cycle, never persisted, never surfaced.** **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session implements and must not quietly widen."* The repository key is minted in the seam between two sides that each honoured their contract, by a helper doing exactly what its doc comment says. **Fixing the predicate would paper over a box quietly making its own history unopenable.** **Recommended fix (not started, no code written): persist the discriminator the ACK already carries and add it as shapeLine truncated
|
||||
|
||||
| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** | **READY — recorded, not built** — owner Viktor |
|
||||
| **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** | **READY** — owner Viktor |
|
||||
| **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** **⚠ UPDATED 2026-08-07 (v0.206.0) — the STATE this row describes can no longer be entered, but the ALARM GAP is untouched and the row stays open.** R-241's mint guard means a box no longer mints a key over a sealed package, so it no longer arrives in the "escrow stuck pending against a self-minted key" state by itself. **What replaces it is a state that is VISIBLE rather than silent:** the box declares `offsite.state=awaiting_recovery_key` and the customer is offered the recovery screen. **But the hub still raises nothing for it**, and for the same three reasons: `isStale` needs `escrowed`, the delivery checker skips the `applied` shape, and `backup_failed` needs a run that never happens. **So a box whose customer never acts still stops backing up off-site with no operator signal** — the difference is that the customer can now see it and act, where before nobody could. **The remaining work is an operator-side signal for a box held in `awaiting_recovery_key` past some age**, and it is deliberately not bundled into R-241's fix. | **READY** — owner Viktor |
|
||||
|
||||
| **R-244** | **The customer DELETE cascade leaves `app_log_issues` behind, and it is systematic across every venue ever torn down.** Found **2026-08-07** while verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a per-table query. After a cascade that logged `COMPLETE … full teardown`, **61 rows still matched `finalwalk`**. Four of the five sources are **deliberate and correct** — the cascade's own header states *"Provenance/events are NEVER wiped — audit outlives every tier"*: `events` 16, `notification_log` 14, `host_deletions` 1, `customer_resets` 1. **The fifth is a gap:** `app_log_issues` 29 rows, which the residue purge does not touch (its logged leg covers `reports`/`app_telemetry`/`app_log_tails`/`log_tail_requests`/`notif_prefs`/`selfbind_tokens`/`appliance_registrations` — not this table). **It is not a `finalwalk` quirk:** rows still reference **`c11` 40, `rewalk` 20, `part4` 24** — all three torn down 2026-08-06, whose ledger recorded *"0 occurrences"*. **That prior claim was measured with a narrower query and does not survive a full census; the correction is recorded rather than the measurement quietly redone.** **Why it was probably never written, established rather than assumed:** the table is a **fleet-wide aggregate** keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list — of the 29 `finalwalk` rows, **12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with LIVE customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted, only de-referenced.** A naive `DELETE … WHERE customer LIKE` would destroy a live customer's issue history — which is very likely why the leg does not exist, and is the reason this is not a one-line fix. **Severity is LOW and stated plainly: no secret material is involved** — app name, fingerprint, message text, counts, timestamps. What survives is a deleted customer's *identifier* inside an aggregate row. **Proposed shape:** a residue leg that (a) removes the customer id from `affected_customers`/`context_customer`, and (b) deletes rows whose `affected_customers` becomes empty; plus a one-off sweep for the four already-torn-down venues. **The general lesson is the reusable part:** *a per-table absence query is not a census.* The teardown verification is now a full-schema sweep, and that is what found this. **Not fixed** — a cascade change needs its own red-proof and this session was scoped as a spike plus two operations. Evidence: `tests/teardown-finalwalk-2026-08-07.md`. | **READY** — owner Viktor |
|
||||
|
||||
| **R-245** | **Should a customer who never decides be auto-abandoned after 30 days? RECORDED, NOT BUILT — and the reasoning against it is recorded with it so the decision can be revisited properly.** **The operator's proposal (2026-08-07):** a box that has been offered recovery for 30 days without the customer deciding is auto-abandoned, entering the 14-day grace, so an undecided box does not sit for ever holding history nobody has claimed. **What was built instead:** the escalating reminders (1/3/7/14 days) and the operator levers `--abandon-extend` / `--abandon-stop`. **The reasoning, as settled with the operator the same day:** (1) **nobody is absent** — a box does not reinstall itself, so whoever rebuilt it was standing there and met the recovery question; the "customer away for months" case does not arise from this situation, because **a reinstall implies a person**. (2) **A customer who cannot find their code will get in touch**, which is the moment to extend or disarm by hand — so the automation would be firing at people we are already talking to, which is why the levers were the thing worth building. (3) **The cost is theirs**: the old history sits in the customer's own storage allowance, and if they are paying to keep something they have not decided about, that is their call and they feel it before we do. (4) **The real harm, if it comes, is QUOTA** — old history blocking new backups — and **that is a condition, not a calendar**. An automatic ending should trigger on the harm, with a dated warning, never on a date alone. **If this is ever built, build it that way.** **Do not implement it without a fresh ruling.** | **WAITING-ON-OPERATOR** — owner Viktor |
|
||||
|
||||
**Recorded against existing rows by Phase 2:**
|
||||
|
||||
- **R-216 — §4.1 is now MEASURED, not deduced.** The previous session could only offer two absences.
|
||||
|
||||
Reference in new issue
Block a user