diff --git a/documentation/audits/SPIKE-r241-recovery-offer-2026-08-07.md b/documentation/audits/SPIKE-r241-recovery-offer-2026-08-07.md new file mode 100644 index 0000000..15604bb --- /dev/null +++ b/documentation/audits/SPIKE-r241-recovery-offer-2026-08-07.md @@ -0,0 +1,519 @@ +# SPIKE R-241 — why the recovery screen retired itself + +**2026-08-07 · READ-ONLY · no product code written · venue read before anything else touched the fleet** + +**Venue:** `demo-hp` VM **324 `finalwalk-appliance`**, guest LXC 9201, hub customer `finalwalk`, host +`finalwalk-ed05d6`. Read in the state the overnight walk left it. **Nothing on it was changed.** + +**Evidence for the events themselves:** `documentation/tests/finalwalk-r201-2026-08-07/journal.md`. +Everything below that is labelled *measured* was measured **in this session, from the venue**, not +copied from that journal. Where the journal is the only witness (the customer-side observations, which +happened before this session) it is cited as such and marked **[walk]**. + +--- + +## THE RULING, first + +> ### R-241 is a **MINTING defect**, not a screen-predicate defect. + +The screen was telling the truth. At the moment the customer looked there genuinely was nothing +recoverable under the key the box was holding, because **the box had minted that key itself, thirty +minutes earlier, over the top of a sealed package it already knew the hub was holding for it.** + +Fixing the screen's predicate would make the screen appear — and would leave in place a box that +quietly makes its own backup history unopenable. **The predicate is not where this is fixed.** + +The reviewer's hypothesis in the task was correct, and it is now measured rather than read off two +call sites. Two independent findings carry it: + +1. **The minting does not consult the flag** — and the flag was not merely *available*, it was the + **precondition of the chain that reached the minting**. §Q3. +2. **The hub's self-heal explicitly disclaims doing this**, in its own package doc: *"credential + automatic, key customer-present — the ruling this session implements and must not quietly widen."* + The minting widens exactly that ruling, from the box's side, where neither side is looking. §Q6. + +And the box **knew**. §Q5: ten minutes after minting, it computed the exact discriminator and wrote +the answer to its own log. + +--- + +## The measured timeline + +All times UTC, from the venue's own controller log (`docker logs felhom-controller`, guest 9201) and +from file mtimes in the controller's data volume. Guest is UTC; the appliance is CEST. + +| time | event | source | +|---|---|---| +| 2026-08-06 22:17:02 | escrow ceremony seals the repository key `30ef574f…`; hub stores it, fingerprint `81:dc:91:…:68:d5` | hub `host_escrow`, **measured** | +| 02:40:31 | guest 9201 destroyed `--purge` | [walk] | +| 02:43:28 | Day-0 provision SUCCESS; rebuilt guest, controller **0.203.0** | [walk] | +| 02:43:02 | controller starts; `settle-gate: awaiting floor knowledge (first report ACK)` | log **measured** | +| 02:43:12 | `settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)`; reconcile finds no unconsumed password | log **measured** | +| **02:48:03** | **`credential retry: … (the box still declares a need; retrying)`** — the first tick that proves the declaration is live | log **measured** | +| 02:53 · 02:58 · 03:03 · 03:08 · 03:13 | same line, five more times | log **measured** | +| 02:59 onward | hub: `offsite-delivery: finalwalk: self-heal REFUSED — the box DECLARES offsite.state=needs_credential` | [walk] | +| 03:15 | hub `offsiteheal` re-stages the stored one-time secret, unaided | [walk] | +| **03:18:06** | `offsite configured for u629488-sub4@…:/home/felhom-repo (pending key escrow)` + `credential retry: the staged credential was collected and the tier applied` — **and `WriteOffboxSecrets` mints `9b4a9a9d…`** | log **measured**; key hash **measured** | +| **03:28:03** | **`[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`** | log **measured** | +| ~04:00 | the customer logs in. `/` → „Indítópult". No recovery pointer. `/recovery` 302s away. | [walk] | +| 04:41:15 | operator moves the self-heal key aside, places the recovered key `30ef574f…` | mtime **measured** | +| 04:43:03 | `[INFO] [escrow-confirm] hub-verified: the escrow covers the current repo password (hash 30ef574fe492…) — EscrowState auto-confirmed escrowed` | log **measured** | + +### The two keys, measured on the venue this session + +``` +offbox/repo_password sha256 = 30ef574fe492a43f89bf1a5071c44e89f51c44c6b08ebcb184b320a2634fad75 (mtime 04:41:15) +offbox/repo_password.selfheal-aside sha256 = 9b4a9a9dcec7898e7544f35b18470aac77c3d9064e5d3a302897617fa62edd65 (mtime 03:18:06) +``` + +Both reproduce the journal's figures **exactly**, computed independently here. The values were never +read, printed or moved — only hashed in place inside the guest. + +### The hub's side, measured this session + +``` +host_escrow finalwalk-ed05d6 + key_fingerprint 81:dc:91:ce:a1:d0:50:3a:…:09:68:d5 + created_at 2026-08-06T22:17:02Z + restic_pw_sha256 30ef574fe492a43f89bf1a5071c44e89f51c44c6b08ebcb184b320a2634fad75 + stale_at NULL + identity_blob 572 bytes (present — R-198 material) +host_escrow_superseded finalwalk-ed05d6 → NO ROWS +``` + +So the hub held — and still holds — a sealed package covering `30ef574f…`, the key that opens the +history at `ep0`. The hash the box logged at 03:28:03 as *"hub hash 30ef574fe492…"* is this row. +**Both sides of the mismatch are now confirmed from independent sources.** + +--- + +## Q1 — what did the screen evaluate, and to what? + +**Measured (state), [walk] (the customer-side observation).** + +`recovery_handlers.go:34` → `backup.OffsiteRecoveryOffer()` (`offbox.go:1411`): + +```go +if !m.settings.GetHubEscrowIdentityPresent() { return false } // fact 1 +if _, ok := m.OffboxRepoPasswordHash(); !ok { return true } // shape (a) +return m.OffboxOrphaned() // shape (b) +``` + +At the moment the customer looked, the three inputs were: + +| input | value | how established | +|---|---|---| +| `hub_escrow_identity_present` | **true** | measured in `settings.json`; and see Q3 — it must have been true from 02:48:03 at the latest | +| repository password on disk | **present** (`9b4a9a9d…`) | measured: `repo_password.selfheal-aside`, mtime 03:18:06 — the file that *was* `repo_password` until the operator set it aside | +| `RepoState == "orphaned"` | **false** (field absent) | measured: `settings.json` carries no `repo_state`; and Q2 — no run ever probed | + +So `OffsiteRecoveryOffer()` = `true && !(no password) && false` = **false**. + +`recoveryPageHandler` (`recovery_handlers.go:79`) then does exactly what it is written to do: + +```go +if !s.recoveryOffer() { http.Redirect(w, r, "/backups/remote", http.StatusFound); return } +``` + +**That is the 302 the walk observed.** It is the R-215 gate, shipped in v0.201.0 to stop the page +asserting a recovery story on a box that has none — working correctly, on a box that had one. + +**The state today is different and must not be mistaken for the failed state.** The venue now reads +`escrow_state: "escrowed"` with `repo_password` = `30ef574f…`, because the operator's 04:41 +intervention made the local key match the hub's package and the auto-confirm flipped it at 04:43. +`OffsiteRecoveryOffer()` is **still false today, and correctly so** — the box now holds the right key. +The failed state survives only in the aside file, the mtimes and the log. + +--- + +## Q2 — can shape (b) ever fire on a box in this state? + +**Answer: no. It is structurally unreachable, and the block is upstream of the detection.** + +`RepoState = "orphaned"` has exactly **one** producer: `markOrphaned()` (`offbox.go:270`), called only +from `ensureOffboxRepo()` (`offbox.go:692, :697`). `ensureOffboxRepo` is called only from +`runOffboxInternal` (`:1162`) and `offboxPruneOnly` (`:1494`), both inside a run. + +And every run passes this gate **first** (`offbox.go:743`, *before* any repo contact): + +```go +if !m.offboxEscrowed() { + m.logger.Printf("[INFO] [offbox] skipped — pending key escrow (…)") + return nil +} +``` + +`offboxEscrowed()` is `EscrowState == "escrowed"`. The box was `pending` — and **could not stop being +pending**, because the auto-confirm flips pending→escrowed only when the hub's hash equals the local +hash, and those were `30ef574f…` vs `9b4a9a9d…`. The box said so itself at 03:28:03: *"staying +pending"*. + +**So the loop is closed in both directions, and it is self-locking exactly as the walk described:** + +> no escrow → no run → no repo probe → no orphan state → shape (b) false. +> And: fresh key ≠ hub's key → auto-confirm never flips → no escrow. Forever. + +**Positive control, because an absent log line is not evidence.** The controller log for the whole +post-rebuild window (02:43→06:43) contains **zero `[offbox]` lines** — but that alone would be equally +consistent with "the scheduler died". It did not: `agent-channel-health` ran **241** times, +`stack-scan` 120, `offsite-credential-retry` 48, `hub-report` 16. The scheduler was alive throughout. +`offbox-backup` is a **`sched.Daily` leg** (`main.go:937`), not a periodic job — its slot is ~02:15 +controller time, which fell **before** the 02:40 destruction, so the next was ~02:15 tomorrow. No +scheduled run was due, none was requested, and had one been, line 743 would have returned before the +probe. **The absence is explained, not merely observed.** + +### Is the escrow block itself right — should a run be prevented while an escrow is pending? + +**Yes, and it should not be touched.** It is the fork-4 atomicity guarantee: no offsite ciphertext may +exist that nobody can decrypt. Relaxing it to let orphan detection run would create exactly the +un-recoverable ciphertext the gate exists to prevent, and would be a much worse defect than the one it +fixed. **Shape (b) is not reachable by loosening this gate, and no option below proposes it.** + +--- + +## Q3 — does `WriteOffboxSecrets` consult the hub's sealed-package flag? + +**Answer: NO. And the flag was not merely available — it was the precondition of the chain that +reached the minting.** + +The minting, in full (`offbox.go:410-419`): + +```go +// Auto-generate the repo password once (0600), never log it. +if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) { + pw, gerr := generateOffboxPassword() + … + os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600) +} +``` + +**One input: does the file exist.** No settings read, no `GetHubEscrowIdentityPresent()`, no hub hash, +no escrow state. The `Manager` has `m.settings` in hand three lines away and does not ask it. + +Compare the two neighbours that *do* ask, both in the same file: + +| call site | consults the flag? | +|---|---| +| `OffsiteRecoveryOffer()` — `offbox.go:1412` | **yes**, `GetHubEscrowIdentityPresent()` is fact 1 | +| `needsOffsiteCredential()` — `offbox.go:1377` | **yes**, same call | +| `WriteOffboxSecrets()` — `offbox.go:411` | **no** | + +**The same fact is available on all three paths and used on two.** That is the shape the task +predicted, and it is confirmed. + +### Was the flag available at 03:18:06? Yes — provably, and this is the strongest single measurement. + +Not by timestamp comparison, but by the product's own control flow. The 5-minute retry job +(`main.go:787-795`) only does work when the box declares: + +```go +attempted, err := offsiteBridge.RetryIfDeclared(ctx, func() bool { + st := backupMgr.OffboxReportStatus() + return st != nil && st.State == backup.OffsiteStateNeedsCredential +}) +if !attempted { return nil } // no log line at all +``` + +`OffboxReportStatus()` returns that state only via `needsOffsiteCredential()` — which **returns false +unless `GetHubEscrowIdentityPresent()` is true** (`offbox.go:1377-1380`). + +The venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03**, and +five more times before the mint. **That line is unreachable unless the flag was already true.** + +> **The flag was true at 02:48:03 — thirty minutes and six ticks before the minting at 03:18:06.** +> The minting is the terminal step of a loop whose *entry condition* is "the hub is holding a sealed +> package for us". The box walked that whole loop and then minted a key as though it had never asked. + +--- + +## Q4 — the state table + +`P` = hub holds a sealed package (`hub_escrow_identity_present`) · `K` = repository password on disk · +`O` = `RepoState=="orphaned"` · `H` = hub's ACK `restic_pw_sha256` vs local hash · `S` = +`hub_escrow_superseded_present`. + +Today's predicate: **`Offer = P && (!K || O)`**. + +| # | state | P | K | O | H | today | correct? | should be | +|---|---|---|---|---|---|---|---|---| +| 1 | **never had off-site backups** | ✗ | ✗ | ✗ | n/a | no offer | ✅ | no offer — nothing to recover | +| 2 | **pristine rebuild**, credential not yet arrived | ✓ | ✗ | ✗ | n/a | **offer** (shape a) | ✅ | offer — but this window is ~15–35 min and closes by itself | +| 3 | **self-healed with a fresh key, hub holds the older package** ← **THE VENUE** | ✓ | ✓ | ✗ | **mismatch** | **no offer** | ❌ | **offer** — this is R-241 | +| 4 | **healthy**, key matches the package | ✓ | ✓ | ✗ | match | no offer | ✅ | no offer | +| 5 | **re-escrowed** — customer made a new code after a self-heal; old package retained (`S`=✓) | ✓ | ✓ | ✗ | match | no offer | ⚠ | no offer from the *current* package — but the retained older history is unreachable and unmentioned (R-199 read path unbuilt, R-222 message only) | +| 6 | **orphaned** — a run proved the repo will not open | ✓ | ✓ | ✓ | mismatch | **offer** (shape b) | ✅ | offer | +| 7 | **customer set the old data aside** (`ResetOrphanedRepo`) | ✓ | ✓ | ✗ (cleared) | **mismatch, until a new ceremony runs** | no offer | ⚠ | **no offer** — and see the hazard below | + +### Two things this table makes visible that the task did not ask for + +**(i) Row 7 is the trap in the obvious fix.** `resetOrphanedRepo` (`offbox.go:291-332`) moves the old +repo aside, re-inits under the *current* passphrase and clears `RepoState` — it does **not** run a new +escrow ceremony. So the hub's package still seals the *old* key and `H` stays **mismatch**. A +discriminator that offers the screen on `H`-mismatch alone would **re-offer the recovery screen +forever to a customer who has explicitly said they do not want the old data.** Any fix needs a +*decided* latch, or the reset flow must carry the ceremony with it. *(Reasoned from code; not measured +— reaching row 7 requires a run, which requires escrow, which the venue never had.)* + +**(ii) Row 3's box cannot self-heal out of the state either, and NOTHING ALARMS.** It is not just that +the screen is missing: `escrow_state` is stuck `pending` forever, so **off-site backups never run +again**. The box is not only unrecoverable-by-the-customer, it is **silently not backing up**. + +All three signals that could catch it are excluded, each for its own reason — verified in the hub: + +| signal | why it does not fire | +|---|---| +| `offsite_stale` | `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`. Its own comment: *"Pending/disabled = normal onboarding, never stale."* A row-3 box is classified as **a box still being set up** — forever. | +| `offsite_delivery_stuck` | `monitor/offsite_delivery.go:91` skips the `applied` shape. The credential **was** consumed and the target **is** present in reports, so delivery is `applied` and correct. | +| `backup_failed` | never fires — the run returns `nil` at the escrow gate (`offbox.go:743-746`) before touching anything. Nothing fails, so nothing reports a failure. | + +**Each exclusion is individually right. Together they leave the state unobserved.** The walk found the +recovery journey broken; the same state also silently stops the product's core promise, and that has +not been filed. **See R-243 below.** + +--- + +## Q5 — is there a discriminator that answers the real question directly? + +**Answer: YES. It already exists, it is already computed on every report cycle, and the result is +thrown away.** + +The question — *does the hub hold a sealed package whose key differs from the one this box is using?* +— is precisely: + +``` +es.IdentityBlobPresent && es.ResticPwSHA256 != "" && localHash != es.ResticPwSHA256 +``` + +Both halves are present and wired today: + +- the hub sends `restic_pw_sha256` on every report ACK (`report/escrow_confirm.go:21`) — the hash of + the password the sealed blob **covers**; +- the box has `LocalHash()` (`offbox.go:539`, `OffboxRepoPasswordHash`). + +And `EscrowAutoConfirmer.Reconcile` **already performs exactly this comparison**, at +`escrow_confirm.go:154`. On the venue it produced, at **03:28:03**: + +> `[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) — run the escrow ceremony …; staying pending` + +**The box computed the right answer to the right question, thirty-five minutes before the customer +looked, and wrote it to a log.** It did not persist it, did not surface it, and the recovery screen — +evaluating in the same process — went on asking a question that could not see it. + +### Would it give the right answer in every row of Q4? + +**No — not on its own.** Honest assessment, row by row: + +| row | `H`-mismatch discriminator alone | +|---|---| +| 1 | ✅ correct — `P` false, no ACK escrow object, no offer | +| 2 | ⚠ **needs shape (a) kept** — there is no local key, so there is no hash to mismatch. `LocalHash()` returns `ok=false` and `reconcileEscrowed` returns early. Shape (a) is still required. | +| 3 | ✅ **correct, and it is the fix** | +| 4 | ✅ correct — hashes match, no offer | +| 5 | ✅ correct for the current package; says nothing about the retained one (unchanged, R-199) | +| 6 | ✅ correct — mismatch is exactly why the repo would not open | +| 7 | ❌ **wrong** — see hazard (i). Re-offers forever after the customer declined. | + +**So the discriminator is necessary and nearly sufficient: it fixes rows 3 and 6, must keep shape (a) +for row 2, and needs one additional latch for row 7.** + +There is a second, stronger check — `CheckOffsiteKeyRecoverable` +(`offbox_recovery_check.go`, R-200) — which recovers the key through the agent and compares hashes. +**It is not usable here:** it requires R, the customer's recovery code, which is the very thing the +screen exists to ask for. It is the post-code diagnostic, not the pre-code discriminator. + +--- + +## Q6 — what did the self-heal intend? + +**Answer: the transport credential, and explicitly not key material. The minting is a side-effect of +the path, not an intent of either side.** + +The hub's own package doc (`hub/internal/offsiteheal/reconciler.go`, header) is unusually explicit: + +> **WHAT IT DELIBERATELY DOES NOT DO:** it never runs, or asks for, an escrow ceremony. A credential is +> replaceable; the recovery code is not, because only the customer holds it. **Credential automatic, +> key customer-present** — the ruling this session implements and must not quietly widen. + +And, on why an already-keyed box cannot reach it at all: + +> It cannot reach this reconciler at all, because the controller's declaration predicate requires the +> repository password to be **ABSENT** (`backup.needsOffsiteCredential`). + +The hub therefore acts on a stated request whose stated meaning is *"I have no repository password"*, +re-stages a **transport** secret, makes **no provider call and mints no new password**, and hands back. +The box then applies it through `ApplyOffsiteTarget` → `WriteOffboxSecrets`, and **that** — a function +whose job is writing an SSH key and a known_hosts line — mints a 256-bit repository key on the way past +because a file was missing. + +> **Neither side intended it and neither side can see it.** The hub ruled "key customer-present" and +> honoured it. The box honoured the transport contract. The repository key was created in the seam +> between them, by a helper doing the thing its doc comment says it does. That is why this survived +> review: every individual comment is accurate. + +**This is R-218's shape one level up, exactly as the walk said** — and the escalation is real: R-218 +was *"succeeding at recovery stopped the box asking for what it still needed"*; this is *"succeeding at +the credential self-heal stopped the box being able to recover at all"*. + +--- + +## Q7 — the „Helyreállítási kód létrehozása" button + +**This was flagged as possibly the most important thing in the spike. The finding is a genuine +near-miss, and the thing that stops it being a disaster is R-198 — shipped for exactly this, and it +holds.** + +**What the customer saw** [walk]: `/backups/remote` offering **„Helyreállítási kód szükséges … → +Helyreállítási kód létrehozása"**. **Measured:** that is `backups_remote.html:132-141`, the branch +`{{else if and .OffboxConfigured (ne .Offbox.EscrowState "escrowed")}}` — which is precisely the venue's +state (configured, `pending`). It links to `/backup/escrow`. + +**It was the only affirmative action on the page.** The recovery screen was gone, `/recovery` 302'd +away, and this was what remained. + +### What it actually does, traced + +*(Reasoned from code and the hub schema. **I did not press it** — it is a state change and it would +have destroyed the evidence this spike exists to read. Marked as such deliberately.)* + +A new ceremony seals the **current** key — `9b4a9a9d…`, the self-heal's — under a **new** recovery code, +and uploads it. On the hub, `SaveHostEscrow` (`store.go:2779`) finds a current row sealing a +**different** hash and calls `demoteCurrentEscrowTx` **before** overwriting: + +```sql +INSERT INTO host_escrow_superseded (host_id, blob, …, identity_blob) +SELECT host_id, blob, …, identity_blob FROM host_escrow WHERE host_id = ? +``` + +**Copy-before-delete, and since R-198 the `identity_blob` is copied too** — that column is *"where the +offsite restic REPOSITORY password lives"*, and its omission for two months is the defect R-198 fixed. + +### So: how close did last night come to a customer destroying their own history? + +**Not as close as it looks — and closer than anyone should be comfortable with.** + +| | | +|---|---| +| the ciphertext at `ep0` | **untouched.** Nothing in this path writes to the repository. | +| the sealed package protecting it | **retained, not destroyed** — demoted to `host_escrow_superseded` with its `identity_blob`. R-198 holds. | +| the customer's held `R` | **still opens that retained package.** The material survives. | +| **but** the read path for a retained package | **UNBUILT** — R-199's inventory. The product has no way to use it. | +| and the recovery engine | **fails closed against the current package.** `R` would now produce the R-222 message: *„a régebbi csomagot… nem töröltük — megőrizzük, megnyitni viszont innen egyelőre nem lehet."* | + +> **The honest answer: the button does not destroy the data. It converts a situation that was one +> screen away from self-service recovery into one that needs an operator, a database and a read path +> that does not exist yet.** Nothing is lost; everything becomes unreachable by any shipped path. +> R-198 is the whole margin, and R-198 shipped nine days ago. + +**One further consequence, which cuts the other way and is worth stating** (reasoned from code): after +a new ceremony `escrow_state` flips to `escrowed`, so runs unblock; the first run's `ensureOffboxRepo` +probes a repo keyed under `30ef574f…` with a local key of `9b4a9a9d…`, classifies **`orphaned`**, and — +the box being claimed — calls `markOrphaned()`. `OffboxOrphaned()` becomes true and **shape (b) fires: +the recovery screen comes back.** But by then the code that screen accepts is the *new* one, and the +customer's held `R` opens only the retained package the screen cannot read. + +**So the button re-enables the screen and invalidates the code it asks for, in the same act.** That is +a worse trap than a plain dead end, because it looks like progress. + +--- + +## What I could not establish + +Named, per rule 9. + +1. **Row 7 of Q4 and the post-button behaviour in Q7 are reasoned from code, not measured.** Both + require a state change on the venue (a run, or pressing the button). Reaching either would have + destroyed the evidence for everything above. **If the operator wants them measured, that is a + separate scoped session with its own fixture** — and it should be, because hazard (i) shapes the fix. +2. **The exact wall-clock at which `hub_escrow_identity_present` first became true** is not measured — + `SetHubEscrowIdentityPresent` writes only on change and logs nothing. What *is* established is the + bound that matters: **true at or before 02:48:03**, thirty minutes before the mint, by the control- + flow argument in Q3. A tighter figure would add nothing. +3. **Whether the customer would in fact have pressed the button.** Unknowable. What is measured is that + it was the only affirmative action offered. +4. **`applied_marker`** (`offbox/applied_marker`, 64 bytes, mtime 03:18:06) — noted, not investigated; + it plays no part in any predicate above. + +--- + +## Options, ranked + +**No code was written. No option below has been started.** + +### Option A — persist the discriminator and add it as shape (c) ★ RECOMMENDED + +Have `EscrowAutoConfirmer.Reconcile` **persist** the comparison it already makes (a new settings mirror +beside `hub_escrow_identity_present`, same last-write-wins discipline), and add a third shape to +`OffsiteRecoveryOffer`: *the hub's package covers a different key than the one we hold.* + +- **Fixes rows 3 and 6**; keeps shape (a) for row 2; needs a *decided* latch for row 7. +- **Uses only facts already on the wire** — no hub change, no new ACK field, no protocol version. +- The fact is already computed on every cycle; this stops discarding it. +- Shape (b) can then be **retired or kept as a corroborator** — it becomes redundant for row 6, and it + is the shape that could never fire on the state that needed it. +- **Cost:** one settings field, one predicate branch, one latch, and the tests that pin them. + +### Option B — do not mint when the hub holds a package for us + +Make `WriteOffboxSecrets` refuse to auto-generate when `GetHubEscrowIdentityPresent()` is true, and +leave the box in shape (a) — the state the recovery screen was designed for. + +- **Attacks the root cause directly**, and is the smallest change that makes the walk's headline false. +- **But** it changes a helper that six call sites and eleven tests depend on, and it makes + `ApplyOffsiteTarget` return a target with no usable key — a state `OffboxConfigured()` currently + treats as not-configured, which would need tracing before it could be trusted. +- **Strictly better as a companion to A than as a substitute.** On its own it leaves row 6 to shape (b), + which is the shape that failed. + +### Option C — A and B together ★ the honest end state + +B stops the box creating the problem; A makes the box able to describe the problem when it exists +anyway (a key minted before B ships, a hand-injected key, a restored volume). **This is where the fix +should land eventually.** It is listed separately because doing both at once is a larger change than +one session should make against a defect this subtle, and A alone already restores the journey. + +### Option D — surface it to the operator only + +Turn the 03:28 WARN into a hub event and an operator alert; change nothing customer-facing. + +- **Cheapest, and insufficient.** It makes the failure visible on a dashboard while leaving the + customer with no route to their data. It is worth doing *as well* — the state in Q4 hazard (ii) + silently stops off-site backups and nothing alarms — but it is not a fix for R-241. + +### Rejected outright + +**Loosening the escrow-pending run gate so orphan detection can run.** It would create un-recoverable +offsite ciphertext, which is the exact hazard fork-4 exists to prevent. Named here so it is not +rediscovered as clever. See Q2. + +--- + +## THE OPERATOR'S DECISION — owed, and deliberately not taken here + +> **1. Which option, and in what order?** The recommendation is **A now, B next, converging on C**. +> Taking A alone leaves boxes able to mint a redundant key that the screen will then correctly offer to +> fix — safe, but noisier than necessary. +> +> **2. Row 7's latch — what counts as "the customer has decided"?** A persisted decision flag, or +> making `ResetOrphanedRepo` carry the new ceremony with it. **This shapes A's design and A should not +> be started before it is answered**, because it is the one way A can regress a shipped flow. +> +> **3. Is Q4 hazard (ii) — a box in row 3 silently stops backing up, with nothing alarming — worth its +> own fix, or does it fall out of A?** It falls out of A *for boxes that recover*. It does not alarm for +> boxes that do not. Filed as **R-243**. +> +> **4. Should Q7's post-button behaviour and Q4 row 7 be measured before the fix is written?** They are +> reasoned from code here. Measuring them needs a fresh fixture and its own session. + +**These are stated and left unanswered, per §3.4.** + +--- + +## Register + +| ID | | +|---|---| +| **R-241** | **RULED — minting defect.** Body updated in `OPEN-ITEMS.md` with this spike's evidence. Not fixed; no code written. | +| **R-242** | **NEW** — a controller release that changes customer-visible behaviour is not delivered until a golden carries it, and nothing enforces that (§4.1 of the task; recorded, not built). | +| **R-243** | **NEW** — a box in Q4 row 3 is stuck `escrow_state: pending` forever, so **off-site backups silently never run again**, and **no alarm of any kind fires**: `isStale` requires `escrowed`, `offsite_delivery_stuck` skips the `applied` shape, and `backup_failed` needs a run that never happens. Three correct exclusions leaving one state unobserved. Found by this spike; not part of the walk's finding. | + +Highest register ID moves **R-241 → R-243**. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 7602b50..d31b1c3 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -167,7 +167,10 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour | **R-239** | **The fixes are written, tested, pushed — and a machine installed tonight gets none of them.** Measured on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`): a fresh install from the published ISO landed on agent **0.127.0** (vouched, **no hand upgrade** — that half works) and controller **0.203.0**, while the newest released controller is **0.205.0**. The vouched golden still bakes 0.203.0, so a new box has neither **R-237** (v0.204.0 — the restore list driven by the store) nor **R-234** (v0.205.0 — the skipped-app verdict and the single-flight message). **Proven from the customer's side on that box, not inferred:** T3 — with the future-backup toggle switched off, the restore page showed **0** entries for an app whose snapshot demonstrably exists, and the wizard redirected away; T2 — two off-site runs back to back both answered „A távoli mentés elindult", the second as a SUCCESS flash, which is exactly the sequence that made a customer read a stale green tick as covering an app they had just selected. **Not a regression and not a code defect — a delivery gap.** The remedy is already documented: bake a golden carrying the current controller and vouch it as a three-field change (`golden_version` + `agent_version` + `min_agent`), per `runbooks/RUNBOOK-manual-build.md` §4.1. **Until that happens, every closure claimed for R-234 and R-237 is true of the code and false of the fleet** — which is the distinction this register exists to keep. | **READY** — owner Viktor | | **R-240** | **A backup that covered nothing calls itself „Sikeres".** On a configured box with no app selected for off-site backup, a run reports status `ok` with the warning „Sikeres — nincs mentésre jelölt alkalmazás" — *successful* immediately beside *nothing is selected*. Measured as T4 on the final walk, 2026-08-07; flagged once before (2026-08-06) and deliberately not touched then, because the task that noticed it forbade changing that path. **It is the same rhetorical shape the project has spent a fortnight removing** — R-203's *a warning beside a success is read as a success*, R-234's *„✓ Rendben" over an app that was skipped*, R-225's *unknown rendered as zero* — one notch weaker each time, and this is the weakest and last of them. The state itself is honest and must stay `ok`: an unconfigured box reporting `incomplete` forever is its own defect, pinned by a test. **The defect is the word „Sikeres", not the verdict.** Wording such as „Nincs mentésre jelölt alkalmazás — ez a futás semmit nem mentett" says the same thing without congratulating the customer on it. | **READY** — owner Viktor | -| **R-241** | **The credential self-heal, succeeding, locks the customer out of their own recovery.** Measured end to end on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`). `OffsiteRecoveryOffer()` shows the recovery screen on exactly two conditions: **(a)** the box has **no** repository password — the pristine rebuilt shape — or **(b)** it has one but the inherited history will not open under it (`OffboxOrphaned()`). Overnight, unaided and exactly as designed, `offsiteheal` re-staged the one-time credential and the box's 5-minute retry **collected it and applied the tier**, writing a **fresh repository password** at 03:18Z. That makes **(a) false**. **(b)** is false too, because orphan detection only fires when a run actually tries the repository — and runs are blocked by `escrow_state: pending`. **The box therefore sits in the gap between the two conditions, and the gap is self-locking:** it cannot detect the orphan without running, cannot run without escrow, and cannot escrow without minting a NEW recovery code — which would orphan the history the customer's existing code protects. **What the customer sees:** `/` is „Indítópult" with no recovery pointer; `/recovery` **302s away**; `/backups/remote` offers „Helyreállítási kód **létrehozása**". **There is no field anywhere to enter the code they hold.** **And the operator's documented remedy also refuses** — `--recover-offsite-install` returns *„[REFUSED] a DIFFERENT repository password is already present… which history to keep is not a decision this command may take. Nothing written."*, which is correct and fail-closed and still a dead end. Recovery required moving the fresh key aside by hand and re-running the install: **three guest command lines**. **The two keys, measured:** on-disk `9b4a9a9d…` (self-heal) vs recovered-from-R `30ef574f…`. **THE DATA WAS NEVER AT RISK** — all three sentinels restored byte-identical once the right key was in place. **This is R-218's shape one level up:** that finding read *"succeeding at recovery stopped the box asking for what it still needed"*; here, succeeding at the credential self-heal stopped the box **offering** the recovery it still needed. The same walk proved the self-heal working unaided six hours earlier, and that success is what causes this. **Likely shape of the fix, not yet a decision:** the offer needs a third condition — a box holding a password it has never successfully used, while the hub holds a sealed package, is a recovery candidate — or the self-heal must not install a credential on a box whose escrow is still `pending` and whose hub blob is unconsumed. **Which of those is right is a design decision, deliberately not taken here.** | **READY** — owner Viktor | +| **R-241** | **The credential self-heal, succeeding, locks the customer out of their own recovery.** Measured end to end on the final walk (2026-08-07, `tests/finalwalk-r201-2026-08-07/journal.md`). `OffsiteRecoveryOffer()` shows the recovery screen on exactly two conditions: **(a)** the box has **no** repository password — the pristine rebuilt shape — or **(b)** it has one but the inherited history will not open under it (`OffboxOrphaned()`). Overnight, unaided and exactly as designed, `offsiteheal` re-staged the one-time credential and the box's 5-minute retry **collected it and applied the tier**, writing a **fresh repository password** at 03:18Z. That makes **(a) false**. **(b)** is false too, because orphan detection only fires when a run actually tries the repository — and runs are blocked by `escrow_state: pending`. **The box therefore sits in the gap between the two conditions, and the gap is self-locking:** it cannot detect the orphan without running, cannot run without escrow, and cannot escrow without minting a NEW recovery code — which would orphan the history the customer's existing code protects. **What the customer sees:** `/` is „Indítópult" with no recovery pointer; `/recovery` **302s away**; `/backups/remote` offers „Helyreállítási kód **létrehozása**". **There is no field anywhere to enter the code they hold.** **And the operator's documented remedy also refuses** — `--recover-offsite-install` returns *„[REFUSED] a DIFFERENT repository password is already present… which history to keep is not a decision this command may take. Nothing written."*, which is correct and fail-closed and still a dead end. Recovery required moving the fresh key aside by hand and re-running the install: **three guest command lines**. **The two keys, measured:** on-disk `9b4a9a9d…` (self-heal) vs recovered-from-R `30ef574f…`. **THE DATA WAS NEVER AT RISK** — all three sentinels restored byte-identical once the right key was in place. **This is R-218's shape one level up:** that finding read *"succeeding at recovery stopped the box asking for what it still needed"*; here, succeeding at the credential self-heal stopped the box **offering** the recovery it still needed. The same walk proved the self-heal working unaided six hours earlier, and that success is what causes this. **Likely shape of the fix, not yet a decision:** the offer needs a third condition — a box holding a password it has never successfully used, while the hub holds a sealed package, is a recovery candidate — or the self-heal must not install a credential on a box whose escrow is still `pending` and whose hub blob is unconsumed. **Which of those is right is a design decision, deliberately not taken here.** **⚠ RULED 2026-08-07 by a read-only spike on the standing venue — `audits/SPIKE-r241-recovery-offer-2026-08-07.md`. IT IS A MINTING DEFECT, NOT A SCREEN-PREDICATE DEFECT, and that reverses the fix.** The screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because the box minted that key itself over the top of a sealed package it already knew the hub was holding. **Three measurements carry it.** (1) **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** It never reads `GetHubEscrowIdentityPresent()`, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and `needsOffsiteCredential()` (`:1377`), both do. **The same fact is available on three paths and used on two.** (2) **The flag was not merely available — it was the precondition of the chain that reached the minting.** The 5-minute retry job only logs when `RetryIfDeclared` fires, which requires the declaration, which requires that flag; the venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times after — **thirty minutes and six ticks before the mint at 03:18:06Z**. (3) **The box KNEW and threw it away:** at **03:28:03Z**, thirty-five minutes before the customer looked, `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) computed the exact discriminator and logged `[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`. **It is computed on every report cycle, never persisted, never surfaced.** **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session implements and must not quietly widen."* The repository key is minted in the seam between two sides that each honoured their contract, by a helper doing exactly what its doc comment says. **Fixing the predicate would paper over a box quietly making its own history unopenable.** **Recommended fix (not started, no code written): persist the discriminator the ACK already carries and add it as shape (c)** — needs no hub change and no new protocol field — **plus a `decided` latch**, because `ResetOrphanedRepo` clears `RepoState` without running a ceremony, so an `H`-mismatch discriminator alone would re-offer the screen forever to a customer who explicitly declined the old data. **Four operator decisions are stated and left unanswered in §"THE OPERATOR'S DECISION".** | **READY — ruled, unfixed** — owner Viktor | + +| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** | **READY — recorded, not built** — owner Viktor | +| **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** | **READY** — owner Viktor | **Recorded against existing rows by Phase 2:**