docs: R-196 closed, R-204 items 1-3 closed, item 4 open (R-193)

- OPEN-ITEMS: R-196 CLOSED; R-204 items 1-3 CLOSED with item 4 named and
  its dependency stated. Header restates that R-202, the 1.2 GB ciphertext
  deletion and R-198's still-unit-proven retention all REMAIN OPEN.
- capability map: the recovery row keeps its 'with a person present'
  qualifier, names which crutch remains, and cites the three now gone.
- 07-backup-architecture: new 7.0 - what a customer can and cannot do
  ALONE, the four steps in a table with status. This is the section a
  future reader will use to answer that question.
- CONTEXT: standing ruling S-32, superseding S-31 steps 2-5.
- STATUS: rewritten to one screen per its own header; removes a corrupted
  half-overwritten section left from the drill session.
- ROADMAP: R-196 and R-204 collapsed.

documentation/architecture/07-backup-architecture.md

@@ -351,6 +351,26 @@ about whether the bytes can be turned back into a working system, and a system c
 completely while having no executable recovery route for a given failure. That is not a hypothetical
 here — §8 has rows where it is the actual state.
 
+### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
+
+> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
+> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
+> byte-identical. **It passed with an operator present, and four manual interventions stood between
+> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
+> are recorded here because this is the section a future reader will use to answer *"can the customer
+> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
+
+| # | The step | Why it stopped a customer | Status |
+|---|---|---|---|
+| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
+| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed` → `OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
+| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
+| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** is required. | **OPEN → R-193.** This is a design decision (the one-shot credential model), not a defect, and it needs an operator ruling. **It is the reason the capability-map row keeps its "with a person present" qualifier.** |
+
+> **[FACT] The honest current answer:** three of the four are gone. **A customer alone still cannot
+> complete a rebuild recovery**, because step 4 requires the operator. Everything after step 4 —
+> claim, restore, verify — is now self-service.
+
 ### 7.1 The dependency graph
 
 > **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below

documentation/backlog/ROADMAP.md

@@ -59,14 +59,14 @@
 | R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
 | R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
 | R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
-| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **comments corrected (hub v0.93.0); the BEHAVIOUR stays open** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping |
+| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **CLOSED — behaviour shipped hub v0.95.0 (2026-08-05)** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping **Collapsed 2026-08-05.** The comment correction landed in v0.93.0; the BEHAVIOUR landed in v0.95.0 with R-204 item 2 — the precautionary `MarkEscrowStale` and its `escrow_stale` event are gone from `ReissueCredentials`. Removing a signal was safe because the real case is measured twice over, and the mark was **blinding** one of them: a stale flag empties `restic_pw_sha256` from the ACK, which is what the controller's per-ACK re-check compares against. |
 | R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
 | R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
 | R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
 | R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
 | R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
 | R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
-| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **OPEN — the wall the 2026-08-04 night drill hit** | Blocks R-201's verdict. Fix is most likely R-196's (mark stale on evidence, not on a Re-issue) or letting a verified recovered key confirm the escrow without a ceremony |
+| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ITEMS 1–3 SHIPPED (controller v0.198.0 + hub v0.95.0, 2026-08-05); ITEM 4 OPEN → R-193** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill |
 | R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
 | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
 | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |