CAMPAIGN-11 Phase 4: pre-registered expectations, both directions, before the window

Written BEFORE the soak so the comparison cannot be fitted afterwards. Sources
are the box's own [scheduler] Registered lines and the agent's backup tier armed
lines, read at 23:56.

MUST happen: db-dump 02:30, tier2-backup + fill-watch 03:30, metrics-prune
04:00, offbox-backup 04:15 (snapshot_count 1 -> 2), a backup_run_digest event,
reports every ~15m.
MUST NOT happen: host_stale/host_recovered, offsite_selfheal_restaged (this is
also R-218's negative control — a box WITH a target must not declare),
offbox_repo_orphaned, offsite_repo_key_changed, escrow blob SERVED, any
controller self-update.
Genuinely uncertain and recorded as such: the restore-test has a 24h settle and
every archive here is hours old, so NOT running is correct, not a gap.

.githooks/pre-push

@@ -29,6 +29,41 @@ root=$(git rev-parse --show-toplevel 2>/dev/null) || {
 }
 cd "$root" || exit 1
 
+# ── WORKSPACE-ROOT ASSERTION (2026-08-05, R-204 rider) ───────────────────────────────────────────
+# Refuse a push from a clone outside the felhom workspace.
+#
+# WHY THIS IS A HOOK AND NOT A LINE IN A DOCUMENT: the workspace root is ALREADY written down, in
+# documentation/runbooks/workspace-CLAUDE.md and in the workspace-root CLAUDE.md ("stay inside it"),
+# and work drifted into a home directory anyway. A rule that has failed once as a reminder is not
+# fixed by writing it down again — it has to be asserted where it can bite.
+#
+# A PUSH IS THE RIGHT TRIGGER, deliberately: throwaway clones under /tmp for probes and red-proofs
+# never push, so nothing legitimate breaks. Reads and builds elsewhere stay unaffected.
+#
+# Symlinks are resolved on BOTH sides before comparison, so a symlinked path neither falsely passes
+# nor falsely fails. If the workspace root does not exist on this machine the check is SKIPPED, not
+# failed — this hook must not brick a legitimate clone on a different host.
+#
+# The only bypass is the documented `git push --no-verify`, whose use is already reportable.
+FELHOM_WORKSPACE_ROOT=/mnt/5_hdd/felhom.eu
+if [ -d "$FELHOM_WORKSPACE_ROOT" ]; then
+    ws_real=$(cd "$FELHOM_WORKSPACE_ROOT" 2>/dev/null && pwd -P) || ws_real=""
+    root_real=$(pwd -P) || root_real=""
+    if [ -n "$ws_real" ] && [ -n "$root_real" ]; then
+        case "$root_real/" in
+            "$ws_real"/*) : ;;  # inside the workspace — proceed
+            *)
+                echo "pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace." >&2
+                echo "          clone:    $root_real" >&2
+                echo "          expected: under $ws_real (repos live in $ws_real/git/<repo>)" >&2
+                echo "          Work in the workspace clone, or bypass with 'git push --no-verify'" >&2
+                echo "          and state that you did in the session report." >&2
+                exit 1
+                ;;
+        esac
+    fi
+fi
+
 if ! command -v python3 >/dev/null 2>&1; then
     echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
     echo "          pass by default. Install python3, or push with --no-verify and say so." >&2

CONTEXT.md

@@ -17,6 +17,342 @@
 
 ## Standing rulings
 
+**S-34 — UNLOCKING AND RESTORING ARE SEPARATE. The recovery screen shipped (2026-08-05, controller
+v0.200.0, R-193 CLOSED). Read with S-32 and S-33; together they close the whole customer journey up to
+the listing.**
+
+**THE RULINGS (operator, 2026-08-05), recorded as given:**
+
+1. **Unlocking and restoring are separate.** This screen unlocks only. Restore stays per-app, in the
+   backups area. **A screen that unlocks and then offers to overwrite is two decisions wearing one
+   button.** Putting files back is its own item (→ **R-213**), and its stated requirement is a
+   live-versus-backup comparison — the customer must see what would change before anything is
+   overwritten.
+2. **"I do not want the old data" MOVES THE STORE ASIDE**, after double-checking with the customer.
+
+*Mechanism facts a future session needs:*
+
+- **ONE CORE, TWO CALLERS.** `backup.RecoverInstallCore` is the only fetch→unseal→compare→install path.
+  `RecoverAndInstall` is now a thin CLI wrapper (exit codes and printed lines byte-identical — every
+  pre-existing CLI test passed unchanged) and the web handler calls the same function. **Asserted from
+  source by AST on BOTH sides**, plus a third test that the routes and the landing-page interception
+  exist. Two implementations of the one operation that can permanently lose a customer's data would
+  drift, and only one would be tested.
+- **THE TRIGGER HAS TWO SHAPES, and the second is the one that matters in practice.**
+  `backup.OffsiteRecoveryOffer` = the hub holds a package **AND** (no repository password **OR** the
+  tier is orphaned). The literal "no repository password" alone is a window that CLOSES BY ITSELF:
+  `WriteOffboxSecrets` auto-generates a password on re-apply — R-193's own orphaning mechanism — and
+  since hub v0.96.0's credential self-heal the re-apply happens within ~15–30 minutes. Shape (b) is
+  also the state the shipped move-aside requires, which is what lets the discard choice reach the
+  existing handler instead of needing a new one.
+- **CLAIMED IS PART OF THE PREDICATE.** A legacy-open box (no password anywhere) reaches `ServeHTTP`
+  through `RequireAuth`'s pass-through, so without an explicit `authEnabled()` check the interception
+  fires for an unauthenticated visitor. Caught by a test, not by review.
+- **"Most nem" suppresses the FULL PAGE ONLY.** The backups-area entry point is bound to
+  `recoveryOffer`, never to the postpone flag. A one-shot notice a flustered person clicks past is a
+  notice that never happened.
+- **The code is handled no more loosely than on the CLI:** POST body only (`PostFormValue`), never
+  logged, never persisted, never echoed, cleared on every path, `no-store`, `autocomplete="off"`.
+  **NO LOCKOUT, deliberately** — the code is a ten-word phrase and locking a customer out of their own
+  data for a typo is worse than anything a lockout prevents; failures are logged locally without the code.
+- **NOT YET PROVEN LIVE: a CORRECT code through the page.** No recovery code was kept for demo-felhom's
+  orphaned history (`RECON-offsite-dr-chain-2026-08-04.md` §12.3) and demo-hp's is operator-held. The
+  live run exercised handler → agent → hub fetch → age KDF and stopped at the unseal.
+
+**S-33 — THE BOX DECLARES, THE HUB ANSWERS. R-204 item 4 / R-193's credential half closed
+(2026-08-05, controller v0.199.0 + hub v0.96.0). Read with S-32; together they close all four of the
+drill's manual interventions.**
+
+**THE RULING AND ITS REASONING (operator, 2026-08-05).** Automate the rebuilt box's credential, and
+**the trigger is a state the BOX DECLARES — never the hub inferring from a report.** From the hub an
+ABSENT off-site object has FOUR meanings — never configured, mid-restart, a transient config read
+failure, rebuilt-and-stranded — and the hub cannot tell them apart. The box can, from two local facts:
+its data area is fresh (no repository password) AND the hub holds a sealed recovery package for it
+(the ACK's `identity_blob_present`). **Both are required.** Freshness alone is a box that never had
+off-site backups — declaring on it would make the whole fleet ask for credentials.
+
+**SECOND RULING RECORDED:** the dashboard-password exposure on the future recovery-preview screen is
+**metadata** (backup dates, app names), not content, and is **ACCEPTED**.
+
+*Mechanism facts a future session needs:*
+
+- **The ACK field used to be discarded.** `EscrowAutoConfirmer.Reconcile` returns early when the box is
+  neither pending nor escrowed — exactly a rebuilt box — so `identity_blob_present` was thrown away
+  every cycle. It is now recorded FIRST, before every gate, in the confirmer itself (already the one
+  place the ACK's escrow object arrives, and already wired; a second consumer would be a second wiring
+  point, and this project's built-but-never-wired count is six). Last-write-wins, not set-only: a
+  customer RESET must be able to turn the declaration back off.
+- **The declaration is inert to every existing reader, established from their code:** it carries
+  `enabled:false` and zero sizes, and `OffsiteChecker.isStale` returns early on `!Enabled` while
+  `fillBand` returns OK on a zero quota/size. A configured box's report JSON is byte-identical to
+  v0.198.0's — no `state` key at all.
+- **`reportHasOffsite` had to be tightened, and it is the ninth comment-vs-code entry — caught before
+  it shipped.** Its comment asserted *"presence == applied-on-the-box"*. Left as bare presence it would
+  have read a request for help as proof the tier was applied. It now requires `enabled:true`, provably
+  a no-op for every pre-v0.199.0 report shape.
+- **Restage before mint is possible because the stored value SURVIVES a consume** —
+  `ConsumeOneTimeSecret` stamps `consumed_at` and nothing else, and `value` is `TEXT NOT NULL`. This was
+  established from the schema, NOT inherited from the PBS analogy; the two secrets are different objects
+  and assuming a shared shape is how two sessions confused the credentials.
+- **The debounce is TWO DISTINCT REPORTS, derived from the ~15-minute report cadence** — one would act
+  on a blip, three would leave a stranded customer waiting ~45 minutes. It counts fresh evidence, not
+  reconciler ticks, so the 5-minute sweep cannot shorten it.
+- **§8.4's deliberately-unhealed state exists and is excluded UPSTREAM, by construction:** the regressed
+  shape (a box that lost its target while still holding its repository password) cannot reach the
+  reconciler, because the declaration predicate requires the repository password to be absent.
+- **R-192's counting guard is REPLACED, not repaired.** A declaration outranks both inferred shapes and
+  the delivery checker stands down with a record — two mechanisms healing one customer would
+  double-issue, and that one can only mint.
+- **NEVER widen this to the ceremony. Credential automatic, key customer-present.**
+
+**S-32 — THREE OF S-31's FOUR MANUAL STEPS ARE CLOSED (2026-08-05, R-204 items 1–3 / R-196).
+controller v0.198.0 + hub v0.95.0. Read this BEFORE S-31 — it supersedes S-31's steps 2–5.**
+
+- **Step 1 (the reset code) — CLOSED.** `web.effectiveClaimCode` now calls `settings.ReloadClaimCode()`
+  before applying the settings-vs-config precedence, so a code minted by the SEPARATE
+  `--print-reset-code` process is visible without a restart. **The precedence rule is unchanged** —
+  the defect was the freshness of the settings value, not which source wins. It is a READ-THROUGH on
+  the claim path only, deliberately **not** a watcher, a signal handler or a TTL: a TTL would leave a
+  window in which a superseded code still works, which is worse than the bug. `effectiveClaimCode`
+  now returns an error and **every caller fails closed** (`claimGateActive` keeps the gate UP; an
+  absent settings file is NOT an error — a pre-first-save box legitimately falls back to the
+  controller.yaml bake). **S-31's "restart the controller between minting and claiming" is obsolete.**
+- **Steps 2–4 (the false staleness) — CLOSED.** `offsite.ReissueCredentials` no longer calls
+  `MarkEscrowStale` and no longer emits `escrow_stale`; `offsite_reissued` is untouched. **The
+  mechanism that made this expensive, recorded because it is not obvious:** `stale_at` makes
+  `GetEscrowStatusForCustomer` WITHHOLD `restic_pw_sha256` — which is the exact value the controller's
+  Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo
+  password on every ACK. **The precautionary mark was blinding the measurement that would have told
+  the truth**, leaving the box able to report only the hash-LESS reason, which is false. Coverage
+  after removal: that per-ACK comparison (continuous, box-side) plus R-197's `offsite_repo_key_changed`
+  (edge-triggered, hub-side, at a supersession). **`MarkEscrowStale` is KEPT with no caller** — the
+  flag is still read by the ACK, the operator card and the PBS-DR view, and the right way to set it is
+  a future caller that has MEASURED a change; `TestEscrowStaleMechanism_StillWithholdsAndClears` stops
+  it decaying meanwhile. **The manual `/backup/offbox/confirm-escrow` workaround is no longer needed
+  after a Re-issue.**
+- **Step 5 (the restore default) — CLOSED.** `restoreScratchOutcomeMsg` states, for `mode=unit`, what
+  came back, that the customer's own files did NOT, and the step that gets them; the wizard's intent
+  card 1 states its scope before the choice. **The `mode=full` size gate is untouched** and pinned as
+  unchanged. **The default stays `unit`:** all three wizard forms set `mode` explicitly, so changing
+  it would alter nothing the customer sees while silently changing a hand-crafted mode-less POST.
+- **WHAT REMAINS is S-31's step 1-before-the-list: a rebuilt box still cannot obtain an off-site
+  credential unaided** (the one-time password was consumed by its predecessor), so an operator
+  Re-issue is still required. That is **R-193**, it needs an operator ruling, and it is why the
+  capability-map recovery row keeps its *"with a person present"* qualifier.
+- **Live-validated on demo boxes, endpoint-level** (no browser on DooPlex): reset codes on
+  demo-felhom 9201 with `restarts=0` across the mints; the unit-restore outcome and the full-restore
+  size-gate reveal on demo-hp 9201, using `privatebin` so the drill's `calibre-web` scratch was not
+  touched. **The Part 2 change was NOT fired live on demo-hp** — a Re-issue there was out of scope.
+
+**S-31 — THE DRILL PASSED: a customer's file survives a machine rebuild and comes back byte-identical.
+The capability is proven; the customer JOURNEY is four undocumented manual steps (2026-08-04, R-201/R-204).**
+
+**Proven on hardware, end to end:** demo-hp's controller data volume was destroyed and the sentinel
+deleted from disk. The recovery code produced `8a9e33aa4da6…` (byte-identical to the pre-wipe on-disk
+key AND the hub's independent record); it installed on the bare box; the **existing repository opened**
+(`repo_state: null`, 3 snapshots, 42 026 B = the pre-wipe size — not a fresh history); and the customer
+restore flow returned the sentinel at `643166269103a25c…`, **byte-identical**. `identity_blob` was
+unchanged throughout and **no ceremony was run at any point** (superseded rows still 2).
+
+**But it took four manual steps that are in no design document (R-204), all measured:**
+1. a rebuilt controller cannot configure its off-site tier — the one-time password was consumed by its
+   predecessor (`no unconsumed offsite password`, R-193);
+2. the Re-issue that fixes that sets `stale_at` **while `restic_pw_sha256` is unchanged** (R-196);
+3. a stale escrow makes the hub withhold the hash from the ACK → `EscrowAutoConfirmer` can never flip
+   `pending → escrowed` → `OffboxRunnable` refuses every run;
+4. it was cleared with the **manual** confirm (`/backup/offbox/confirm-escrow`), NOT a ceremony — a
+   ceremony would have superseded the identity blob and destroyed the recovered key;
+5. and the restore's **default `mode=unit` returns the recovery unit, not the customer's files** —
+   `mode=full` is what returns documents. A customer following the obvious path gets the wrong thing
+   and is told nothing.
+
+**Before any of it, a rebuilt box is unclaimed** and the claim gate intercepts every controller
+endpoint. **The local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to
+`settings.json` while the running controller keeps its old copy in memory, so the code never matches —
+**restart the controller between minting and claiming.**
+
+*Facts a future session needs:*
+- **A guest rebuild in this fleet is a controller-DATA-VOLUME loss, not a guest reprovision.** The
+  2026-08-03 incident R-193 is filed against ran with guest 9201 up throughout — no `pct destroy`, no
+  `pct restore`, no `--selftest=provision`. Reproduce it that way.
+- **A good snapshot is not durable against a later bad run on the same day.** `forget --keep-daily 7
+  --group-by host,tags` keeps one per tag per day; a later, worse snapshot evicts a good one.
+- **Never run a ceremony while a recovery is in flight** — it supersedes the identity blob. Under
+  v0.93.0 the old blob is retained, but nothing serves a superseded blob back (R-199).
+
+**S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.**
+
+It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run
+reported `ok` (**R-203**): the deploy-time `${USERDATA_PATH}` root and the backup-time `NamespaceRoot`
+disagree for an app whose `HDD_PATH` equals `system_data_path` — and the same compose used both roots.
+Wiping would have destroyed the only copy of the sentinel and proven nothing.
+
+*Established live on the way, all firsts, and all worth carrying:*
+- **A rebuilt box's off-site run REFUSES** — `repo_state: orphaned`, the orphan card, and
+  `offbox_repo_orphaned` at the hub. It does **not** silently start a fresh history. Closes R-193's Q3.
+- **The orphan reset works**: move-aside to `/home/felhom-repo.orphaned-20260804`, never delete.
+- **demo-hp's pre-rebuild off-site history is permanently unrecoverable** — its key is in superseded
+  row id 3 with `identity_blob` NULL, superseded **four hours before** v0.93.0 fixed the retention.
+- **Neither pre-existing off-site-toggled app has a restorable file leg** — both are named-volume-only,
+  which the tier tars but the customer restore never unpacks. Any drill needing a recoverable file
+  needs an app with a **mandatory userdata path**; `calibre-web` is now deployed on demo-hp as that
+  fixture, with the sentinel in place.
+
+*Still not established, unchanged:* **no file has ever been restored from an off-site backup after a
+wipe**, and Part 0's install path (controller v0.196.0) has never run against a live recovery.
+
+**S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR
+path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller
+v0.195.0).**
+
+**The decision, so it is never re-litigated from scratch.** Self-service recovery needs the box to
+obtain its own sealed package. The only retrieval that existed was operator-gated behind recovery mode
+and the global key (`dr.go`), which cannot serve a customer-present flow. `GET /hosts/{id}/escrow` now
+serves that ONE object to its authenticated owner.
+
+- **Why it is safe:** the blob is opaque; the hub has never held R and has no decrypt path; a wrong
+  code fails closed at age's scrypt KDF (measured live — `exit status 1`, no plaintext, nothing
+  written); the caller already authenticates as this host for its report, desired state, WG
+  registration and PBS token, so this adds no new identity, only a new object.
+- **What it changes, which is the whole of the trade:** obtaining the blob used to require the
+  OPERATOR to arm recovery mode. It now needs only that box's credential. **Fewer parties are
+  required.** The mitigation is that the capability is AUDITED, not silent: every retrieval raises
+  `escrow_blob_served` (warning, operator-only), written before the bytes leave.
+- **The override is one constant.** `api.escrowSelfServiceRetrieval` — false re-imposes the
+  recovery-mode requirement on this endpoint and changes nothing else. Do not scatter the condition.
+- **`handleReEnroll` / `handleGetRestoreDirective` are UNTOUCHED** and must stay so: they rotate the
+  host API key and serve the K-escrow and the directive as well. Different object, different gate.
+  Pinned by `TestEscrowGet_OperatorDRPathUnchanged`.
+
+**Facts a future session needs before touching this area:**
+- **The unseal lives in the AGENT and must stay there.** `age` is an agent runtime dependency
+  (`/usr/bin/age`, present on both demo hosts) and is deliberately absent from the controller image;
+  the blob is host-scoped and its only writer is the agent under the per-host key.
+- **The agent returns ONE field.** Not the tunnel token, not the PBS token, not the WG key — the
+  controller is a trust tier down. Narrowing cost nothing and is not recoverable later.
+- **R's rules:** in memory for one call, cleared on every path, never on disk, never in argv, never
+  logged, never echoed. The controller CLI reads it from STDIN for exactly that reason. Verified live:
+  0 log lines, 0 files, 0 leftover staging dirs, with a positive control proving the search worked.
+- **The chain today: links 1–8 walked, 9–11 not.** The KEY comes back. Nothing installs it, reopens a
+  repository with it, or restores a file — R-200's remaining half and R-201.
+
+**S-28 — the escrow retention now covers the OFF-SITE data key, and customer-present recovery is the
+accepted design, which makes that retention load-bearing (2026-08-04, R-198/R-197; hub v0.93.0).**
+
+**The contract changed, and this is the sentence to carry:** when an escrow blob supersedes another,
+`demoteCurrentEscrowTx` retains **both** sealed keys — the K-escrow `blob` (PBS datastore key) **and**
+`identity_blob` (the age-wrapped bundle carrying the restic **repository** password). Until v0.93.0 it
+retained only the first, so the retention preserved the whole-guest key and destroyed the off-site data
+key, and **the destroying act was the escrow ceremony a rebuilt box asks its customer to run.**
+
+*Why this is load-bearing rather than tidy:* the operator ruled on 2026-08-04 that **candidate (c) is
+refused — no repository password is retained on the Proxmox host.** With no host-retained copy, the
+**customer-present recovery path is the only way back from a rebuild**, and it runs entirely through
+the retained identity blob. A session that touches escrow custody is touching the single copy.
+
+*Facts a future session needs before touching this area:*
+- **The ordering is load-bearing and invisible from the routine.** `SaveHostDRBundle` writes
+  `identity_blob` **after** `SaveHostEscrow` returns (the escrow PUT, `api/handler.go`), so at demote
+  time `host_escrow` still holds the OLD identity blob. Invert that and the retained bytes become the
+  NEW blob filed under the OLD hash — recoverable-looking and wrong. Pinned by
+  `TestSaveHostEscrow_RetainsIdentityBlob`.
+- **`demoteCurrentEscrowTx` is shared by re-escrow and host-delete.** Both have a test; fixing one
+  caller instead of the routine is a demonstrated red-proof, not a hypothetical.
+- **Nothing was backfilled and nothing could be** — pre-v0.93.0 retained rows have no identity blob and
+  their source rows are already overwritten. Both demo boxes' pre-2026-08-04 repository passwords are
+  gone permanently, recovery code or not.
+- **A retained key is not a recovery.** The hub's blob-serving endpoints have no client (R-199), the
+  injection seam has no form (R-200), and the path has never been exercised (R-201). v0.93.0 makes the
+  key survive; it does not hand it back.
+- **`offsite_repo_key_changed`** (R-197, warning, operator-only, edge-triggered on a supersession where
+  both hashes are known and differ) is the evidential signal that a box's off-site data key moved. It
+  carries **no hash value**. `MarkEscrowStale` is **precautionary**, not evidential — see S-26(a).
+
+**S-27 — a customer with NO machine ever bound is UNKNOWN, silently; one that was bound and went quiet
+still alarms (2026-08-04, R-195; hub v0.92.0).** Operator ruling, implemented as
+`store.HasEverBoundHost` (live `hosts` row OR `host_deletions` tombstone) consulted once at the top of
+`CheckBackupDeadlines`' loop. **The discriminator is "was a host EVER bound", never "has a report
+arrived"** — a box that was installed and never phoned home is a real fault. Fail-**open** on a read
+error; the deferral is logged with its own counter; the R-81 anchored-verdict structure is untouched.
+
+*The mechanism is worth carrying, because the guard that should have covered this was keyed off the
+wrong thing:* the existing down-skip reads `StalenessChecker.GetState()`, whose map is seeded from
+`store.GetCustomers()` — **a query over the `reports` table**. A customer with zero reports is in no
+row, gets no state, and `GetState()` returns `""` not `"down"`, so **the skip misses exactly the
+customer it would most obviously cover.** `david` (created 2026-08-01, no machine) e-mailed an
+`expected_dbdump_missed` ERROR three mornings running; `peti-felhom` — active, host deleted
+2026-07-15 — does not, because its 482 old reports make it `down`. Generalise it: **a "skip the dead"
+guard built on evidence of life cannot see something that was never alive.**
+
+**S-26 — the one-shot secret is the recoverable one; the irreplaceable one is minted fresh on every
+guest rebuild (2026-08-04, R-193 spike — `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`).
+No code shipped for it; the decision is the operator's.**
+
+Two sentences, because they are what every future session needs before touching this area:
+
+- **Q1 — what is one-shot.** Of the three secrets in the offsite path, the hub's **one-time
+  provider password** is one-shot but freely re-issuable, the box's **SFTP key** is regenerated on
+  every apply by design, and the **restic repository password** — the DATA key, which the agent's own
+  source calls *"irreplaceable"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the only one
+  nothing can restage.
+- **Q2 — what a rebuild costs.** A rebuilt controller **mints a brand-new repository password**
+  (`WriteOffboxSecrets`, `offbox.go:392`, generates whenever `<DataDir>/offbox/repo_password` is
+  absent) and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly
+  one caller in the whole repo, a web form a human pastes into — so **every guest rebuild orphans the
+  previous off-site history.**
+
+*Measured without touching a box, and the method is reusable:* `host_escrow.restic_pw_sha256` vs
+`host_escrow_superseded.restic_pw_sha256`. demo-hp `8e03eddf…`→`8a9e33aa…` (15 snapshots / 40.9 MB
+orphaned); demo-felhom `48741892…`→`c60c8bc7…` (**36 snapshots / 1.14 GB**). **demo-felhom is the half
+that matters:** R-193 recorded it as having survived the rebuild by luck, and it did — its *delivery*
+recovered in 76 s off a stale staged secret. **Its repository did not**, and nothing said so for 13 h.
+*Luck restored the plumbing, not the data.*
+
+*Three corrections that must not be re-inherited.* **(a)** `ReissueCredentials` does **not** rotate the
+restic password — R-39's record, `offsite.go:198-201` and `api/handler.go:1067-1069` all say it does,
+and all three are wrong (→ **R-196**, the eighth entry in `CLAUDE.md`'s table). **(b)** Candidate (b),
+*"recoverable from escrow at re-bootstrap"*, is **not implementable** — the escrow is R-wrapped and
+zero-knowledge (D6), so only a customer-present ceremony can open it, which is the manual form that
+already exists. **(c)** Candidate (a), *"the hub restages automatically"*, **already exists** as
+`reissueOnReenroll`'s F3 leg and is wired to the wrong event — it sits behind `handleHostEnroll`'s
+mint-once-reuse short-circuit, and a **guest** rebuild leaves the `hosts` row intact. **Shipping it
+first would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** —
+strictly worse than the current loud failure.
+
+*The candidate nobody had named,* and the only one aimed at the actual harm: **the agent survives a
+guest rebuild**, already receives the repo password over the pinned local API
+(`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after
+the ceremony. Retaining and serving it back needs no new seam. Its price is one real trade: a copy of
+the data key at rest on the Proxmox host. **That trade is the operator's to make and the spike does not
+make it.**
+
+**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).**
+R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks"
+ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not
+follow**, so every weekly run uploaded fine and then failed the job on a refused prune. The offsite
+tier now writes `keep_last: 0`, which the agent's existing guard already reads as *never prune from
+the box*; the local tier is untouched. A gate asserts it. **Before switching it off, ep0 was read:**
+both namespaces have a prune job at 03:30 keep-last 2 that has run daily since 2026-07-27 (18 tasks,
+all OK). **If that ever stops, `keep_last: 0` is unbounded growth** — check ep0's prune jobs before
+assuming the offsite tier is retained.
+
+**S-25 — a lost storage grant repairs itself, and the repair is REPORTED (2026-08-04, R-190; agent
+v0.124.1).** On a missing grant the agent runs the existing root wrapper (`felhom-backup-target-apply
+grant <id>`, already sudoers-permitted for any id) and re-reads once — the pbsdr R-22 shape. Bounded
+at one attempt per tier per hour.
+
+*The record is the point, and getting it wrong is easy.* A repair that leaves only `ok` behind
+destroys the only evidence a permission vanished. v0.124.0 reported degraded for "one cycle" and
+**that failed in production while every unit test passed**: `probeAll` is called independently by the
+self-check log and by the collector, the repairing call was the log's, and the host-report three
+seconds later sent `ok`. The fix is a latch on **time** (20 min > the 900 s report interval), so at
+least one report must carry it. The explanation rides `Feature`, because that is the field the hub
+interpolates into the operator's e-mail — `Reason` does not travel. **Nothing new was built:** the
+hub's existing ok→degraded→ok edge is the channel.
+
+*Caveat measured live:* **PVE caches permissions** (~40 s and ~16 min observed), so detection lags the
+loss and a single permission read is a lagging indicator → R-194.
+
 **S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
 (2026-08-04). Scope: those two boxes, not the fleet.**
 

REPORT-r204-item4.md

@@ -0,0 +1,219 @@
+# REPORT — R-204 item 4 / R-193 credential half (hub v0.96.0), 2026-08-05
+
+**A rebuilt box asks for its credential back, and the hub answers.** The last of the four manual
+interventions the 2026-08-04 drill needed. Controller half: `felhom-controller` v0.199.0.
+
+> **Written as `REPORT-r204-item4.md`, not `REPORT.md`.** A PARALLEL SESSION is active in this shared
+> clone — it committed `c917251` (the R-205…R-211 disk containment) between my baseline read and my
+> first commit. Per `CLAUDE.md`'s parallel-session rule the second session never touches the shared
+> `REPORT.md`. Explicit per-file staging was used throughout; verified after the fact that none of my
+> three commits carries a foreign file.
+
+## 1. Baselines, re-read on arrival — with a drift
+
+| Repo | Expected | Found |
+|---|---|---|
+| `felhom.eu` | `0dbd954fec90` / hub v0.95.0 | **DRIFTED to `ee9d9bf`** — one docs-only commit ahead (R-205…R-211 spike output). Hub code, `manifests/hub.yaml` and the deployed image were all still v0.95.0, so the drift did not affect the work. A second foreign docs commit (`c917251`) landed mid-session. |
+| `felhom-controller` | `68f195676b91` / v0.198.0 | exact match, tree clean |
+| `felhom-agent` | v0.125.0 | `3f5f61b`, clean (rider only, **no version bump**) |
+| `app-catalog-felhom.eu` | n/a | `122bbee`, clean (rider only, **no version bump**) |
+
+**The task's "highest register ID in use: R-204" was STALE.** Grepped before minting, as instructed:
+the highest is **R-211**. This session mints **R-212** (§8 below).
+
+## 2. Part 2.0 FIRST — does the stored value survive a consume? **YES.**
+
+Established from the schema and the code, deliberately **not** from the PBS analogy:
+
+- `one_time_secrets` declares `value TEXT NOT NULL` (store.go, the CREATE TABLE).
+- `ConsumeOneTimeSecret` runs `UPDATE one_time_secrets SET consumed_at = datetime('now')` — **it
+  touches nothing else.** The value column is never cleared or overwritten.
+
+**So restage-before-mint is possible**, and `RestageOneTimeSecret` mirrors `RestageHostPBSSecret`:
+clear the consumed flag, never touch the value, never bump a generation, return false when no row
+exists so the caller escalates.
+
+This is asserted rather than assumed by `TestRestageOneTimeSecret_ReArmsTheSameValue`, which checks
+the **same** value comes back — so a future hardening that cleared the column fails loudly instead of
+silently turning every rebuild into an external mint. The two secrets are different objects with
+different lifecycles, and assuming a shared shape is how two earlier sessions confused the credentials.
+
+## 3. The declared state — name, shape, and how a pre-upgrade hub treats it
+
+**Name:** `needs_credential`, on the report's existing `offsite` object (not a new top-level field).
+Constants: `backup.OffsiteStateNeedsCredential` / `offsiteheal.StateNeedsCredential`, pinned to each
+other by `TestDeclaredStateStringMatchesTheReconciler`.
+
+**Shape, as received by the hub live** (report id 16743):
+
+```json
+{"enabled": false, "escrow_state": "", "state": "needs_credential",
+ "snapshot_count": 0, "repo_size_bytes": 0, "quota_gb": 0}
+```
+
+**A pre-upgrade hub treats it as INERT, established from the readers' code rather than assumed:**
+
+| Reader | Behaviour on the declaration | Why |
+|---|---|---|
+| `OffsiteChecker.isStale` | no staleness alarm | returns early on `!off.Enabled` |
+| `OffsiteChecker.fillBand` | `bandOK` | returns OK on a zero quota/size |
+| `encoding/json` | ignores `state` | unknown field, not an error |
+| **`reportHasOffsite`** | **would have MISREAD it** | see below — the one reader that needed changing |
+
+**A configured box's report JSON is byte-identical to v0.198.0's** — `state` is `omitempty` and is
+never set on a configured box (`TestOffsiteDeclare_ConfiguredBoxJSONIsUnchanged`).
+
+## 4. The debounce: TWO distinct reports, derived not chosen
+
+The controller reports every ~15 minutes. Two distinct declarations mean the state survived a full
+report cycle, and a restart, a slow first report or a transient config read all resolve well inside
+one. **One** would act on a blip; **three** would leave a genuinely stranded customer waiting ~45
+minutes for the one thing they cannot obtain any other way. The reconciler's own sweep is 5 minutes —
+deliberately faster than the report cadence so it adds no latency of its own — and the debounce counts
+**fresh evidence, not ticks**, so a fast sweep cannot shorten it.
+
+## 5. §8.4 — is there a deliberately-unhealed state here? **Yes, and it is excluded upstream.**
+
+`pbsdrheal` refuses to heal `verify_failed` because re-staging would not help it and it must stay
+loud. The off-site analogue is the **regressed** shape: a box that HAD a working tier and lost its
+target while still holding its repository password. Re-arming a credential would not help it either.
+
+**It cannot reach this reconciler at all** — the controller's declaration predicate requires the
+repository password to be **absent**, so a box that still holds one never declares. The unhealable
+case is excluded *by construction*, upstream, rather than filtered out in a switch. Stated explicitly
+because "there is nothing like that here" is usually wrong, and this was checked rather than assumed.
+
+## 6. Files modified
+
+| File | Change |
+|---|---|
+| `hub/internal/store/store.go` | **new** `RestageOneTimeSecret`, **new** `LatestReportOffsiteDeclaration`; `reportHasOffsite` tightened to require `enabled:true` (+ its comment corrected) |
+| `hub/internal/offsiteheal/reconciler.go` | **new package** — the reconciler |
+| `hub/internal/offsiteheal/{reconciler,wiring}_test.go` | **new** — Scenarios A–F + the wiring AST test |
+| `hub/internal/store/offsite_restage_test.go` | **new** — the value-survives proof + the `reportHasOffsite` equivalence table |
+| `hub/internal/monitor/offsite_delivery.go` | `shapeDeclared` outranks both inferred shapes; `maybeHeal` stands down with a record |
+| `hub/internal/monitor/offsite_declared_test.go` | **new** — the demo-hp shape + the cross-package string pin |
+| `hub/cmd/hub/main.go` | wires + runs the reconciler (`OFFSITEHEAL_ONLY_CUSTOMER` for a supervised rollout) |
+| `.githooks/pre-push` | the workspace-root assertion (rider) |
+| `manifests/hub.yaml` | image `0.95.0` → `0.96.0` |
+
+**Commits on `main`:** `f62a115` (hub half) · `fe1e816` (rider) · `b2462b8` (CHANGELOG) ·
+`4114c5f` (manifest). **Deployed:** `felhom-hub:0.96.0`, ArgoCD **Synced/Healthy**, `deploy/hub`
+rolled out, startup log carries `offsite credential self-heal reconciler started (interval 5m,
+debounce 2 reports)`.
+
+## 7. Tests and red-proofs
+
+Green gate: `go build ./... && go vet ./... && go test ./...` in `hub/` — **rc=0**.
+`python3 scripts/repo_gates.py --fast` — **all five gates OK**.
+
+| Scenario | Test | Result | Red-proof — what was mutated | Outcome |
+|---|---|---|---|---|
+| A+D | `TestScenarioAD_DeclaringBoxIsRestagedNotMinted` | PASS | inverted `heal()` so Reissue runs before Restage | **FAILED** — *"a sustained declaration was not served: restages=0"* |
+| B | `TestScenarioB_SilentBoxIsNeverActedOn` | PASS | — | — |
+| C | `TestScenarioC_HealthyBoxIsAPureNoOp` | PASS | — | — |
+| E | `TestScenarioE_NothingToRestageEscalatesOnce` | PASS | — | — |
+| F | `TestScenarioF_BlipIsAbsorbedByTheDebounce` | PASS | `debounceReportsDefault` 2 → 1 | **FAILED** — *"a one-report blip triggered a credential action"* (and Scenario A+D also failed, *"acted on a SINGLE declaration"*) |
+| H | `TestMainWiresTheOffsiteHealReconciler` | PASS | (AST; comments dropped) | — |
+| — | `TestRestageOneTimeSecret_ReArmsTheSameValue` | PASS | — | — |
+| — | `TestReportHasOffsite_EnabledOnly` | PASS | — | — |
+| — | `TestShapeOf_DeclarationOutranksBothInferredShapes` | PASS | — | — |
+| — | `TestDisabledDescriptorIsNeverHealed`, `TestBlockedCustomerIsNeverHealed`, `TestRestageErrorDoesNotEscalate` | PASS | — | — |
+
+**A test that first passed for the wrong reason, caught and fixed.** `TestBlockedCustomerIsNeverHealed`
+initially seeded `Status: "blocked"` through `SaveCustomerConfig`, whose INSERT does not carry the
+column — so the customer was never actually blocked and the assertion would have been vacuous. It now
+goes through `SetCustomerConfigStatus` **and asserts `IsCustomerBlocked` before proceeding**.
+
+## 8. Part 4 — STOPPED, corrected, then COMPLETED with the operator's confirmation
+
+§8.7: *"If the paths do not match R-193's record exactly, STOP. A near-match on a protected endpoint
+is not a match."* **They do not match.**
+
+Measured read-only over SFTP, using each box's own credential, from inside its guest:
+
+| Customer | Path | Size | What it is |
+|---|---|---|---|
+| demo-felhom (`u629488-sub1`) | `/home/felhom-repo` | **1.2 G** | **LIVE** — the configured `repo_path`. Unopenable by the box (R-193), but NOT a set-aside store |
+| demo-felhom | `/home/felhom-repo.orphaned-20260717` | **1.4 G** | set aside |
+| demo-felhom | `/home/felhom-repo.orphaned-20260718` | **3.0 M** | set aside |
+| demo-hp (`u629488-sub3`) | `/home/felhom-repo` | **582 K** | **LIVE** |
+| demo-hp | `/home/felhom-repo.orphaned-20260804` | **43 M** | set aside |
+
+The ruling says *"~1.2 GB across the two demo boxes, in set-aside stores"*. Reality: **three** set-aside
+stores totalling **~1.45 GB** — and **the figure that matches ~1.2 GB is demo-felhom's LIVE
+`felhom-repo`**. Had the size been used to identify the target, the live repository would have been
+deleted. Filed as **R-212**, and the operator was asked with the corrected list.
+
+**The operator confirmed *delete all three*, and all three were deleted.**
+
+| Account | Deleted | Freed |
+|---|---|---|
+| `u629488-sub1` | `felhom-repo.orphaned-20260718` | 3.0 M |
+| `u629488-sub1` | `felhom-repo.orphaned-20260717` | 1.4 G |
+| `u629488-sub3` | `felhom-repo.orphaned-20260804` | 43 M |
+
+**AFTER, on each account, a full listing:** `u629488-sub1` holds `.ssh` + `felhom-repo` (**1.2 G**,
+live); `u629488-sub3` holds `.ssh` + `felhom-repo` (**582 K**, live). **Nothing outside the three named
+paths was touched**, and no prune job, datastore or live repository was involved.
+
+**Proof nothing live was caught:** a REAL off-site run triggered on demo-hp immediately afterwards
+(`POST /backup/offbox/run`, authenticated + CSRF) returned `status: "ok"`, `orphaned: false`,
+`last_error: ""`, `last_run: 2026-08-05T09:14:02Z`, `last_duration: 1m24s`, 6 snapshots.
+
+**METHOD NOTE, worth carrying forward.** The Hetzner storage box runs a **restricted shell**: no shell
+operators, no `test`, no GNU long flags. The first attempt used `test -d X && rm -rf -- X` and got
+*"Command not found. Use 'help' to get a list of available commands."* — **it failed CLOSED, verified
+by a byte-identical before/after listing.** `rm -r <path>` issued as ONE simple command is the working
+form, and the smallest store was deleted first to confirm the syntax before the 1.4 GB one.
+
+## 9. Live validation
+
+| # | What | Observable |
+|---|---|---|
+| 1 | **A healthy box: no action, no events** | demo-hp reported healthy throughout (`enabled:true`, no `state` key — report id 16742). Zero `offsite_selfheal_*` rows in the hub DB. **Honest limit:** the reconciler is silent by design on a healthy sweep, so there is no per-tick positive observable; what I have is the startup line proving `Run` was entered, the DB showing no events, and Scenario C. |
+| 2 | **The declared state, produced live without wiping a box** | demo-felhom 9201 arranged **reversibly** into the stranded shape (settings + `offbox/` backed up first; the `offbox` key removed and `repo_password` moved aside). Report **id=16743** reached the hub carrying `{enabled:false, state:"needs_credential", quota_gb:0, repo_size_bytes:0}`. Restored the same minute; report **id=16744** is healthy again. **The single declaration was absorbed by the debounce — no self-heal event fired** — which is Scenario F demonstrated on live infrastructure rather than in a fake. |
+| 2b | **The ACK field is no longer discarded** | Both demo boxes' `settings.json` now carry `hub_escrow_identity_present = true` — the recorder working on a HEALTHY box, which is the case that used to return early. |
+| 3 | **A push from outside the workspace is refused** | A scratch clone at `/tmp/.../outside-clone`: `pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace`, naming `/mnt/5_hdd/felhom.eu`, **before the gates run**. **Red-proof:** with the assertion removed the same push **succeeded** (`rc=0`, new branch on a throwaway bare remote). In-workspace pushes ran normally all session. |
+| 4 | **Part 4** | **Completed after the STOP and a corrected list.** Full listings before and after on both accounts; both LIVE repos intact; a real off-site run on demo-hp succeeded immediately afterwards (`status: ok`, `last_run 09:14:02Z`). |
+
+**Not fired live: the hub actually re-staging a credential.** Doing so would have re-applied
+demo-felhom's off-site target mid-session and changed the very state Part 4's listing describes. It is
+proven by Scenarios A/D/E with the store primitive proven separately against a real SQLite database.
+
+**Teardown:** the scratch clone and throwaway remote are removed; demo-felhom's `settings.json` and
+`offbox/` restored from the backup taken first (verified: `enabled=True`, `escrow_state=escrowed`,
+`repo_password present=True`); the backup copy remains at `/root/r204-backup` on the guest for
+traceability. Nothing else was provisioned.
+
+## 10. Registers
+
+- **R-204 — ALL FOUR ITEMS CLOSED.** Both 2026-08-05 rulings recorded on the row: the declared-state
+  trigger with its four-meanings-of-absence reasoning, and that the recovery preview's
+  dashboard-password exposure is **metadata, not content, and accepted**.
+- **R-193 — credential half CLOSED.** Still open under this ID: the customer-facing recovery **screen**,
+  and the ciphertext deletion (now R-212).
+- **R-192 — CLOSED, guard half by REPLACEMENT.** The counting inference is outranked by the declaration.
+- **R-202 — untouched, still open.**
+- **R-212 — NEW** (R-211 was the highest; grepped): the halted deletion, with the full measured listing.
+
+## 11. CI
+
+felhom-agent **0404f60**, app-catalog **ee2c810**, felhom-controller **992803c**, felhom.eu
+**4114c5f** — run IDs and conclusions confirmed in the controller's `REPORT.md` §11 and re-checked at
+session end. **`--no-verify` was NOT used**; every push ran the pre-push gate, including the new
+workspace-root assertion.
+
+## 12. Observations — noticed, NOT acted on
+
+- **A second, accidental barrier exists outside the workspace and should not be relied on:**
+  `felhom-agent`'s `reuse-refs` gate FAILS in a clone outside the workspace because the shared
+  `reuse_refs_check.py` lives in the `felhom.eu` sibling. That is why the first Scenario-G red-proof
+  had to be redone with `app-catalog-felhom.eu`, whose fast gate is self-contained. It is incidental,
+  repo-specific and not a substitute for the assertion.
+- **`offsite_credential_restaged` (the R-71c event) has still never fired for any customer.** The new
+  reconciler emits its own `offsite_selfheal_*` events, so the old one may now be permanently dead —
+  worth a deliberate look rather than leaving two event families for one concern.
+- **demo-felhom's live `felhom-repo` is 1.2 GB of ciphertext its own box cannot open** (0 snapshots,
+  `last_status: error`). It is not a set-aside store, so it is out of R-212's scope as filed — but it is
+  the largest single block of unrecoverable data in the fleet and nothing currently plans its disposal.

REPORT.md

@@ -1,173 +1,160 @@
-# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
+# REPORT — R-196 / R-204 item 2 (hub v0.95.0), 2026-08-05
 
-**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
-and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
-box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
-`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
-source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
+**A re-issue no longer marks a healthy escrow stale.** One behaviour change, one register closed, and
+the coverage claim proved rather than assumed. The controller's half of R-204 (items 1 and 3) is
+`felhom-controller` v0.198.0.
 
-**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
+## 1. Baselines, re-read on arrival
 
----
-
-## 1. Preconditions
-
-| # | demo-felhom | demo-hp |
+| Repo | Expected | Found |
 |---|---|---|
-| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
-| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
-| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
-| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
-| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
-| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
-| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
+| `felhom.eu` | `2a7ac03c4726` / hub v0.94.0 (deployed `felhom-hub:0.94.0`) | **exact match**, tree clean, `HEAD == origin/main` |
 
-## 2. The due verdicts before the run, quoted
+**§3.2's landmark had DRIFTED, and the drift changed the work.** The task described
+`offsite.go:222-231` under a known-consequence comment saying the mark was made on a false premise.
+That comment had already been rewritten by the R-196 comment-correction commit, and the version on
+`main` gave a **non-false** ground for the mark: *"the box's re-apply may mint a fresh repository
+password (it does exactly that whenever `<DataDir>/offbox/repo_password` is absent — the
+guest-rebuild shape)"*. So the question was no longer "delete a comment's lie" but "is the shape it
+guards actually covered elsewhere?" — which is Scenario D, and §8.2 says to stop and report if it is
+not. It is; §3 below is the evidence.
 
-```
-demo-felhom  tier=felhom-backup due=true  archive="…2026_08_02-04_42_14.tar.zst"
-               reason: newest settled archive … has not been proven; nothing proven on this tier yet
-             tier=felhom-pbs    due=true  archive="…2026-07-28T04:49:43Z"
-               reason: … has not been proven (last proven archive was a different one)
+## 2. What changed
 
-demo-hp      tier=felhom-backup due=true  archive="…2026_08_02-04_49_29.tar.zst"
-             tier=felhom-pbs    due=true  archive="…2026-07-28T19:19:45Z"
-```
+`offsite.ReissueCredentials` no longer calls `MarkEscrowStale` and no longer emits the `escrow_stale`
+event. **`offsite_reissued` is untouched** and still fires on every re-issue. The known-consequence
+comment is rewritten to record what was done, when, and why — with the disagreement below stated in
+it rather than absorbed.
 
-**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
+**What the mark actually cost, established mechanically rather than asserted** (this is why it was a
+blocker and not a nit):
 
-## 3. The runs — all four SCHEDULED, none triggered
+1. `stale_at` set → `GetEscrowStatusForCustomer` **withholds** `restic_pw_sha256` from the report ACK.
+2. With no hash, the controller's SLICE-3 auto-confirm returns early and cannot flip
+   `pending → escrowed`.
+3. `OffboxRunnable() = OffboxConfigured() && EscrowState == "escrowed"` → **every off-site backup
+   refused**, indefinitely, on a box whose repository key was never in doubt.
+4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
+   supersedes the sealed blob. **During a recovery that is the one act that would have destroyed the
+   key just recovered.**
 
-| box | tier | due at | archive | result |
-|---|---|---|---|---|
-| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
-| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
-| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
-| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
+## 3. Scenario D — the evidence that the removed marking is covered
 
-**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
-only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
-which start nothing.
+The mark was precautionary and aimed at ONE shape: a re-issue followed by a box that mints a fresh
+repository password (the guest-rebuild shape, where `offbox/repo_password` is absent). That shape is
+measured in two independent places, and **the mark was blinding one of them**:
 
-**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
-proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
-evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
-is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
+- **Continuous, box-side — the real coverage.** `report.EscrowAutoConfirmer.reconcileEscrowed`
+  (controller) compares the ACK's sealed `restic_pw_sha256` against the box's CURRENT local repo
+  password on **every report ACK**, raising the stale flag, the customer card and the
+  „create a new recovery code" CTA on a mismatch. That is a **measurement**, not a guess, and it is
+  continuous rather than edge-triggered. Pinned by the controller's
+  `TestEscrowStale_MismatchWarnsOnceAndFlags` — re-run green this session.
+  **And step 1 above was blinding it:** a stale flag empties the very hash that comparison needs, so
+  the box could only reach the hash-LESS branch and report *"the hub's current blob carries NO
+  password hash"* — which is false. Removing the mark restores the true signal.
+- **Edge-triggered, hub-side.** R-197's `offsite_repo_key_changed` fires from
+  `api.maybeEmitRepoKeyChanged` on a proven hash difference across a supersession and pages the
+  operator. **Red-proved:** removing the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut`
+  made `TestEscrowPut_ChangedRepoKey_RaisesSignal` fail with *"the repository key demonstrably changed
+  and NO signal was raised"*, while the two silence tests stayed green.
 
-**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
-tier an ordinary recovery uses is also the cheapest to prove.
+**Disagreement recorded, per the R-96 standing rule.** Scenario D as written asks that a real key
+change also *"mark the escrow stale"*. **It must not, and nothing was changed to make it.** The hub
+learns of a real change at the instant a supersession **seals the new password** — i.e. when the
+escrow is at its freshest. Marking it stale there would ask the customer for a ceremony to fix the
+ceremony that just ran. The correct consequence at that instant is the operator alarm, which is
+exactly what R-197 already does. This is recorded in the code comment, the CHANGELOG and OPEN-ITEMS,
+not only here.
 
-## 4. What the runs left behind
+## 4. `MarkEscrowStale` is kept with no caller — deliberately
 
-**Persisted state — v3, naming the archive, the tier and what was verified:**
+Per task §5 it was not to be modified, and it is not deleted either. The `stale_at` flag remains live
+and correct — read by the ACK, the operator config card and the PBS-DR view — and the right way to
+set it is a **future EVIDENTIAL caller** that has measured a key change rather than guessed at one.
+Its doc comment now says so plainly instead of naming a caller that no longer exists, and
+`TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying to inert while
+nothing writes it (the seam-built-but-never-wired shape, in reverse).
 
-```json
-demo-felhom  felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
-                              proven_at 2026-08-03T22:56:45Z}
-             felhom-pbs    → {archive …2026-07-28T04:49:43Z,        tier pbs,   proven_at 2026-08-04T05:04:21Z}
-demo-hp      felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
-             felhom-pbs    → {archive …2026-07-28T19:19:45Z,        tier pbs,   proven_at 2026-08-04T06:10:39Z}
-```
+The schema comment and `EscrowStatus.Stale`'s comment were corrected the same way — each of the three
+previously asserted a writer that is now gone.
 
-**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
-latest report holds **two** entries, one per tier; the `local` one can only have come from the
-persisted state, because the in-memory store held only that morning's offsite run:
+## 5. Files modified
 
-```
-demo-felhom  tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
-             tier=pbs   pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z          tested_at=2026-08-04T05:04:21Z
-demo-hp      tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
-```
+| File | Change |
+|---|---|
+| `hub/internal/offsite/offsite.go` | the pessimistic `MarkEscrowStale` + `escrow_stale` event removed; comment rewritten to record the change, the coverage and the disagreement |
+| `hub/internal/offsite/offsite_test.go` | `TestReissue_InvalidatesEscrow` **replaced in place by its exact inverse** `TestReissue_DoesNotMarkAHealthyEscrowStale`; new `TestEscrowStaleMechanism_StillWithholdsAndClears` |
+| `hub/internal/store/store.go` | three comments corrected (`MarkEscrowStale`, the `stale_at` schema note, `EscrowStatus.Stale`) — each had named a writer that no longer exists |
+| `manifests/hub.yaml` | image tag `0.94.0` → `0.95.0` |
+| `hub/CHANGELOG.md`, `CONTEXT.md`, `STATUS.md`, `documentation/…` | v0.95.0 entry; ruling **S-32**; the register and architecture updates below |
 
-**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
+**Commits on `main`:** `d1a8edb` (behaviour + tests + comments) · `5c7d671` (CHANGELOG) ·
+`975a690` (manifest bump).
 
-```
-tier=felhom-pbs  due=false  proven="…2026-07-28T04:49:43Z"
-  reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
-```
+**Deploy:** built + pushed `felhom-hub:0.95.0`, bumped `manifests/hub.yaml`, pushed, then a
+**deliberate ArgoCD hard-refresh + sync** (auto-sync stays off; no `kubectl set image` anywhere).
+Result: app `felhom` **Synced / Healthy**, `deploy/hub` rolled out, running image
+`gitea.dooplex.hu/admin/felhom-hub:0.95.0`, startup log clean (offsite provisioning, pool-box checker,
+PBS-DR reconciler and all six host checkers initialised; `Listening on :8080`).
 
-…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
-That is not a defect: it is "proved daily, on its own archive", visible one day later.
+## 6. Tests and red-proofs
 
-## 5. Teardown — three layers, per box
+Green gate: `cd hub && go build ./... && go vet ./... && go test ./...` — **rc=0**.
+`python3 scripts/repo_gates.py --fast` — **all five gates OK**.
 
-| layer | demo-felhom | demo-hp |
-|---|---|---|
-| the machine | `pct list` → **0** entries for 990000 | **0** |
-| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
-| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
+| Test | Result | Red-proof — what was mutated | Outcome |
+|---|---|---|---|
+| `TestReissue_DoesNotMarkAHealthyEscrowStale` (C) | PASS | restored the pessimistic `MarkEscrowStale` block in `ReissueCredentials`, exactly as it was | **FAILED** — *"a re-issue marked a HEALTHY escrow stale…"* |
+| `TestEscrowStaleMechanism_StillWithholdsAndClears` | PASS | same mutation | **stayed GREEN** — correctly: the mutation restores a *caller*, not a break in the mechanism. That split is the evidence Scenario C's assertion is about the caller and not the flag. |
+| `TestEscrowPut_ChangedRepoKey_RaisesSignal` (D) | PASS | removed the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut` | **FAILED** — *"the repository key demonstrably changed and NO signal was raised"* |
+| `TestEscrowPut_UnchangedRepoKey_Silent`, `TestEscrowPut_HashlessSupersession_NoSignal` | PASS | same | stayed green — the detector's silence branches are independent |
+| controller `TestEscrowStale_MismatchWarnsOnceAndFlags` | PASS | — (cited as the continuous-coverage pin) | — |
 
-Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
+Scenario C asserts the **consequence** (the ACK still carries the hash, so auto-confirm can proceed)
+rather than the mechanism (that a function was not called), because the hash is what the drill's
+blockage actually turned on. It also asserts that `offsite_reissued` still fires — removing a false
+alarm must not remove the true notice.
 
-## 6. What the run surfaced — three findings, two of them corrections to my own record
+## 7. Live validation
 
-### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
+**Per task §12 point 5, a live re-issue was NOT run, and must not have been on demo-hp** — it would
+have been a credential rotation on the box holding the drill's evidence. Part 2 is proved by test and
+by the deployment being live and healthy. The controller-side halves of R-204 were validated live and
+are reported in `felhom-controller/REPORT.md`.
 
-A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
-2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
-~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
-(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
-Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
-**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
-and v0.123.0's probe detects the *state* but says nothing about the *transition*.
+## 8. Register and documentation
 
-### R-191 (new) — every weekly offsite backup reports FAILED although it worked
+- **`OPEN-ITEMS.md`** — **R-196 → CLOSED (hub v0.95.0)**; **R-204 → items 1–3 CLOSED, item 4 OPEN
+  (→ R-193)** with its dependency named. The header block is updated and states explicitly that
+  **R-202**, **the ~1.2 GB orphaned-ciphertext deletion** and **R-198's retention (still UNIT-PROVEN
+  ONLY — the second deliberate wipe is the next item)** all **remain open**, so nothing is presumed
+  closed by association. R-201 is recorded as PASSED. **R-204 is still the highest ID; nothing new
+  was minted.**
+- **`architecture/00-capability-map.md`** — the recovery row now says three of the four crutches are
+  gone, names the fixes and their evidence, and states that **item 4 (R-193) is the one that remains**
+  and is why the row **keeps its "with a person present" qualifier**. The **R-199 back-pointer was
+  already present** on the adjacent key-recovery row (added when that row was last corrected), so it
+  needed no further action — verified, not assumed.
+- **`architecture/07-backup-architecture.md`** — **new §7.0, "What a customer can and cannot do
+  ALONE"**: the four steps in a table with what each cost and its status, plus the honest current
+  answer. This is the section a future reader will use to answer the question.
+- **`documentation/backlog/ROADMAP.md`** — R-196 and R-204 collapsed per the coupling rule.
+- **`CONTEXT.md`** — new standing ruling **S-32**, which supersedes S-31's steps 2–5 and carries the
+  blinding mechanism, the fail-closed rule and the "no TTL" reasoning forward.
+- **`STATUS.md`** — rewritten to **one screen** (191 → ~90 lines) per its own header. It also had a
+  corrupted, half-overwritten "What we're working on" section left from the drill session, which is
+  now gone. Next item stated as the retention drill.
 
-demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
-incrementally) and the job then failed on the prune —
-`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
-errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
-(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
-the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
-must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
-— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
-(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
-namespaces.
-
-### Two corrections to yesterday's record
-
-1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
-   **six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
-   `whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
-   said backups kept landing because writes go through a root path — wrong, and now corrected in the
-   runbook and on the row.
-2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
-   I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
-   since the R-86 work — and one **more** since, below.
-
-**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
-2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
-host tier had never been proven, and the hub said so on its own, without being asked. It has not
-re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
-
-## 7. The capability map
-
-The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
-boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
-teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
-is untested and untouched, and one box proving something does not make it a fleet property.
-
-## 8. Registers
-
-- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
-  correction in §6.
-- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
-  the highest in use).
-- `ROADMAP.md` holds none of these rows, so nothing to collapse.
-- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
-  "What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
+**CI:** felhom.eu runs **154** (`5c7d671`, code) and **155** (`975a690`, manifest) — both success.
+`--no-verify` was **not** used; the pre-push gate ran and passed on every push.
 
 ## 9. Observations — noticed, NOT acted on
 
-- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
-  returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
-  warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
-  the restore pointed at `/mnt/nvme-1tb` instead.
-- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
-  none". Nothing depends on that assumption now, but the operations notes still carry it.
-- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
-  role — a 2.7× difference worth understanding before either is used to size anything.
-- **Both waiters this session produced no output** despite the runs completing; the evidence was
-  gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
-  instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
-  the hub's database, not on the waiters.
+- **`allowedEventTypes` still lists `escrow_stale`**, which after this change has **no producer** in
+  either repo. It is inert rather than harmful; removing an allowlist entry is a behaviour change and
+  is out of this session's scope.
+- `MarkEscrowStale` is now dead code by call-graph. Kept on purpose (§4 above) — but if a future
+  session's linter or cleanup pass proposes deleting it, the reason it exists is in its doc comment
+  and in the test that exercises it.
+- `/` on DooPlex is at **86%** used — under the 90% abort line, but worth watching before large builds.

REUSE.md

@@ -75,8 +75,11 @@
 | `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
 | `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). |
 | `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
+| `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. |
 | `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
 | `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
+| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
+| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
 | `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
 | `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
 | `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |

STATUS.md

@@ -1,6 +1,6 @@
 # STATUS — what works, what's broken, what's next
 
-**Updated 2026-08-03.**
+**Updated 2026-08-05.**
 
 > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
 > page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
@@ -13,77 +13,144 @@
 A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
 who sets their own password. They install apps from a catalogue of fifty-three, share files over the
 home network, and open apps from a launcher or a shared link. Backups run on their own to three
-places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
-restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
-box six times, everything returned every time, and an app switched off deliberately stayed off.
-Proven end to end on real hardware.
+places — the machine's drive, a second drive, and an encrypted off-site copy.
+
+**The backup promise is proved. The recovery JOURNEY is not.** On 5 August we built a brand-new
+machine from the published disc, gave it three marked files, destroyed it, and tried to get them back
+**the way a household would** — no shortcuts, no command line. The files came back **byte for byte
+identical**, all three, including one with Hungarian accents in its name. But **the journey needed us
+four times**, and the very first thing the machine did was tell the customer their correct recovery
+code was wrong. *(CAMPAIGN 11)*
 
 ## What's broken
 
-- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
-  3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
-  the machine picked the new password up in 15 seconds and reattached to the same repository, and you
-  escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
-  once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
-  The other machine survived the same rebuild only because an unused password happened to be waiting
-  for it. *(R-193)*
-- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
-  this declines without saying why. The message says the password was never applied; it was, on
-  23 July, and worked for eleven days. *(R-192)*
-- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
-  trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
-  you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
-  setting per machine fixes it. *(R-191)*
-- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
-  also delete it. A daily snapshot is armed as a stopgap.
-  *(R-95, R-87)*
+- **A machine installed today would tell its owner their correct recovery code is wrong.** The
+  recovery screen needs a newer in-house service than a new machine is given; when it asked and got
+  nothing, it blamed the customer's typing. **We fixed the lie today** — it now says plainly that the
+  *machine* cannot do this yet, and never accuses anyone. **It still needs one click from you to
+  actually work on new machines** (below). *(R-216, R-223)*
+- **Three things a rebuilt machine still cannot do by itself.** Its owner cannot re-attach their own
+  drives, so no app can be put back on its data; it cannot create a new recovery code at all; and the
+  screen at the machine itself never stops showing a stale pairing code. Each is understood, measured
+  and written down — none is fixed yet. *(R-220, R-221, R-214)*
+- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
+  that encrypts its own off-site backups, and a rebuilt machine invents a new one. **The good news:
+  the old key really is kept now — we proved it on a real machine today, for the first time**, and a
+  changed key raises an alarm the same day. *(R-193, R-198)*
+- **The kept older backups cannot be opened yet.** We keep the previous sealed package, and there is
+  no way to open it. A customer holding exactly the right code for it used to be told they had
+  mistyped; today the screen names the situation honestly instead — but it still cannot open it, and
+  it does not pretend otherwise. *(R-222, R-202)*
+- **The off-site copy can be erased by the machine that made it.** A daily snapshot is armed as a
+  stopgap. *(R-95, R-87)*
 
 ## What shipped recently
 
-- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
-  demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
-  checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
-  had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
-  never been tested on either machine. Both also proved their off-site copy on the same night, one
-  after the other rather than at once, which is the machine deciding for itself what to do first.
-  *(closes the last open half of R-86/R-185)*
-- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
-  demo machines. The permission was one command; the silence was the real fault, and the machine now
-  checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
-- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
-  data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
-  restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
-  day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
-  written to disk with the result and survives. This was caught happening, not predicted: a real
-  14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
-  fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
-  published after the binary, and a new check catches the opposite mistake so nothing is traded away.
-  **(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
-  approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
+- **2026-08-05 (late)** — **New machines now get current software again — and the disc image had been
+  quietly un-updatable for days.** Approving the newer in-house service turned out to be impossible on
+  its own: the system correctly refuses to publish a set where the pre-built machine image is older
+  than the software the fleet already runs, and that image had been behind since late July. **So the
+  image was rebuilt and both were published together.** A machine installed from now on lands on
+  current software and can open a recovery package on day one. *(R-223)*
 
+- **2026-08-05 (evening)** — **Four sentences where there was one, and none of them blames you.**
+  "We did not accept your recovery code" used to appear when the code was wrong, when the machine
+  could not ask, when the store could not be read, and when the customer held the code for an older
+  backup we still keep. Only the first is the customer's doing. Also today: succeeding at recovery no
+  longer switches off the machine's own request for the thing it still needs; the screen now finishes
+  the job and shows what is in the backups instead of promising a list it could never produce; and the
+  recovery page can no longer be reached on a machine that never had backups.
+  *(R-216, R-217, R-218, R-219, R-222, R-215)*
+
+- **2026-08-05** — **The recovery screen: a customer whose machine was rebuilt is now told, and shown
+  how.** Until today they had everything needed to get their data back and no way to find out — the
+  only route was a command line. The screen unlocks the backups and lists what is in them; it does
+  **not** restore anything, because unlocking and restoring are two different decisions and mixing
+  them would turn one clear moment into a wizard. Three ways out, none of them a dismiss button — and
+  the „most nem" option keeps the route to the data permanently visible in the backups area, because
+  a notice someone clicks past once is a notice that never happened. *(R-193)*
+
+- **2026-08-05** — **The orphaned backups are deleted — and the list you were given was wrong, which
+  is why you were asked again.** You had approved "about 1.2 GB in two set-aside stores". Measured
+  before touching anything: there were **three** set-aside stores totalling **~1.45 GB** — and the
+  thing that was exactly 1.2 GB was demo-felhom's **live** store. Matching on the size would have
+  deleted a working backup. With the corrected list confirmed, all three were removed and both live
+  stores left alone; a real off-site backup ran successfully straight afterwards to prove nothing
+  working had been caught. *(R-212)*
+
+- **2026-08-05** — **A rebuilt machine now asks for its storage credential, and the hub gives it back.**
+  The machine says plainly what it needs — it can tell it has been rebuilt, because its data area is
+  empty *and* the hub is holding a sealed recovery package for it — instead of leaving the hub to guess
+  from a silence that has four possible meanings. The hub waits long enough to be sure it is not a
+  restart, then **re-uses the credential it already holds** before creating a new one at the storage
+  provider. **The recovery ceremony stays manual, deliberately:** a credential can be replaced, your
+  recovery code cannot. *(R-204 item 4, R-193)*
+
+- **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer
+  is unstuck.** The cause was leftover working data from building our own software — 157 GB of it,
+  growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.**
+  The storage layer had already stopped accepting new copies of any volume onto that disk; that is
+  fixed the same day. A **30 GB ceiling** is now in place and was **proved to work by deliberately
+  overfilling it and watching it evict** — not by assuming the setting took. Two things that failed
+  quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
+  and the weekly cleanup is still forbidden from touching the thing that grows (next session).
+  *(R-205 … R-211)*
+- **2026-08-05** — **The build data now lives on the second SSD, moved with nothing lost.** All 345
+  images, every saved volume and both development databases came through identical — checked before
+  the original was touched and again afterwards, and confirmed by running a real build on the moved
+  copy. **The server's own services never went down:** Gitea, the registry, the hub and the backup
+  system run on a separate system and stayed up throughout. The second SSD now also **reserves 80 GB**
+  for this, so the storage layer can no longer quietly claim the space and repeat what happened to the
+  first disk. The old copy is kept as the way back until the machine next restarts. *(R-209)*
+- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
+  a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
+  restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)*
+- **2026-08-04 (night)** — **The drill PASSED**, end to end, on real hardware. *(R-201)*
+- **2026-08-04** — The folder-left-out-of-the-backup problem fixed both halves: the app and its backup
+  look in the same directory, and a backup that misses a folder marked essential reports *incomplete*
+  instead of success. *(R-203)*
+- **2026-08-04** — The hub now keeps the off-site backup key when a machine re-seals, instead of only
+  the whole-machine one, and a machine can fetch its own sealed package back. *(R-198, R-199)*
+- **2026-08-04** — The daily false alarm about David is gone; a vanished permission now repairs itself
+  **and says it had to**; the weekly off-site backup stopped reporting failure after a successful
+  upload. *(R-195, R-190, R-191)*
 
 ## What we're working on
 
-- **Now:** nothing outstanding.
-- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
-  tests unattended. Most of the machinery it needed arrived with the restore-test change below.
-  *(R-87)*
-- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
+- **Next: the retention proof.** The hub keeping the old sealed key when a machine re-seals is the one
+  remaining link that has **never run outside a test**. Proving it needs a second deliberate wipe on
+  the spare demo machine, and it is its own procedure. *(R-198)*
+- **Then:** the orphaned-backup deletion you asked for (below), and the off-site copy the machine can
+  still erase. *(R-193, R-95)*
 
 ## Waiting on you
 
+
+- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
+  not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
+  through a restart. It works right now and nothing was lost, but a restart is the one test that
+  matters for this kind of change, and it has not happened. **I made it check itself:** whenever the
+  machine next starts, for any reason, it writes a plain PASS or FAIL line to
+  `/var/log/felhom-store-postboot-check.log`. **If it says PASS, the old copy can be deleted and 34 GB
+  comes back.** Until then I have deliberately kept that old copy, which is the only quick way back —
+  it is why the disk sits at 54% rather than lower. *(R-209a)*
+- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
+  62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
+  (controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
+  was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
+  199 GB now free, so this is about clutter, not space. *(R-210)*
+- **Nothing.**
+- **The recovery screen you described has been priced, and it can be built.** A freshly installed
+  machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
+  show what would come back before doing anything. One thing to weigh, deliberately not decided: that
+  screen is reachable by anyone with the household's dashboard password, and the preview reveals
+  backup dates and app names. *(R-193)*
+- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
+  across the two demo machines, in set-aside stores nobody can open and nothing prunes. It wants its
+  own session rather than riding along with other work. *(R-193)*
+- *(decided 4 Aug)* You chose **not** to keep a copy of the backup key on the Proxmox host, which
+  makes the customer's own recovery code the only route back from a rebuild. *(R-193)*
 - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
   session log; nothing suggests anyone else saw it. *(R-132)*
-- **One small question, not urgent.** The automatic check cannot see which version you have told
-  machines to install, only which ones exist. Closing that needs either a password given to the build
-  server or a check inside the hub itself. *(R-184)*
-- **Nothing else.**
-
-## Changed since last update
-
-- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
-  overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
-  weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
-- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
-  that could vanish on a restart (seen happening), a release that emailed a failure for a release
-  that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
+- **One small question, not urgent.** The automatic version check cannot see which version you have
+  told machines to install, only which ones exist. *(R-184)*

documentation/architecture/00-capability-map.md

@@ -36,9 +36,13 @@
 | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`scripts/iso/felhom-bootstrap.sh` `network_gate`/`sweep_nics`, `build-felhom-iso.sh` rootpw emission) | **PROVEN-LIVE (nested drill — nested ≠ metal: metal proof rides the next real multi-NIC install)** | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — dead-NIC install from the virgin v1.24.0 ISO baked the 192.168.100.2 fallback (WITH a dead default gateway), the R-59 screen painted on the console (screendump captured), and after the cable move the box swept to the working NIC, re-leased and **self-registered at the hub unaided in under a minute**; the drill also caught + fixed the stale-fallback-route trap (flush before the bounded dhclient) and verified the emitted rootpw against the installed box's shadow hash | R-59 ships as a first-boot gate, not an install-time abort (recorded deviation — the fallback is the auto-installer's own, initrd hook out of scope); sweep is structurally first-boot-only (`state.json` gate + unit done-flag condition); a box past install-start gets the screen but its interfaces are never touched |
 | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
 | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
-| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
-| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant  **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). |
-| **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
+| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ the claim as written was FALSE for the offsite tier for two months; **FIXED in hub v0.93.0 (2026-08-04), and the row below states what ships TODAY:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`) — and **since v0.93.0 the retained row carries `identity_blob` as well as the K-escrow blob**, so a previous passphrase does now stay recoverable with the recovery code that sealed it. **What was wrong until v0.93.0, recorded because it is the ninth entry in `CLAUDE.md`'s comment-vs-code table and the first that was also customer-facing copy:** `host_escrow_superseded` had no `identity_blob` column and `demoteCurrentEscrowTx` did not copy one, so what survived a supersession was the **PBS datastore key only** — never the restic repository password, which lives in `identity_blob`. The destroying act was the **escrow ceremony** a rebuilt box asks its customer to run. Measured live 2026-08-04, before the fix: both current rows held `blob`=383 B **and** `identity_blob`=572 B; both retained rows held `blob`=383 B only. → **R-198** (SHIPPED), evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **THREE SCOPE LIMITS THIS ROW MUST NOT BE READ PAST.** (1) **Nothing was backfilled and nothing could be** — rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; both demo boxes' pre-2026-08-04 repository passwords are gone permanently. (2) **A retained key is not a restore — but as of 2026-08-04 evening it IS a recovered key.** See the row below. (3) The customer-facing orphan card still promises recoverability unconditionally (R-202, gate hit, card untouched). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test |
+
+| **The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code** | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE (2026-08-04)** | On demo-felhom, through the real endpoints end to end: the box fetched its own sealed blob from the hub with its own per-host credential (hub log: *escrow blob SERVED … 572 opaque bytes, self_scope=true*), the agent unsealed it with the customer's recovery code, and the extracted repository password's sha256 was **byte-identical** to the one on disk — `c60c8bc737a6…`, which is ALSO the hash the hub had independently stored, so three sources agree. Five minutes earlier the same path with a WRONG code failed closed at age's KDF with nothing written, which proves links 6 and 7 ran independently of the success. R was searched for afterwards and found in 0 log lines and 0 files, with a positive control confirming the search would have found it. Evidence: per-repo CHANGELOGs; `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 links 6–8 | **WHAT THIS ROW DOES NOT CLAIM, stated because the previous over-claim here was struck out four hours earlier.** It covers the KEY, not the DATA. **R-199 BACK-POINTER (omitted when this row was written): the recovery chain's link inventory and the per-link status live in `audits/RECON-offsite-dr-chain-2026-08-04.md` §3; links 1–8 are walked, 9–11 are not.** **Re-confirmed 2026-08-04 evening by an attempt to prove the DATA half:** the R-201 drill was prepared on demo-hp and **halted before the wipe** — the sentinel file was not in the off-site snapshot (R-203), so the wipe would have destroyed it and proven nothing. **No file has still ever been restored from an off-site backup after a wipe** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). The install half of the chain (controller v0.196.0 `--recover-offsite-install`, R-200) is likewise unit-proven only — it has never run against a live recovery. A recovered password has never been **installed** (the diagnostic compares and refuses to write, by design), no existing repository has ever been **reopened** under one, and **no file has ever been restored** from an off-site history via a recovered key. Links 9–11 of the chain are open (R-200's remaining half, R-201). The proof also used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run || DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant  **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent Line truncated
+
+| **A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end** | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE (2026-08-04 night drill)** | demo-hp's controller data volume was destroyed and the sentinel deleted from disk. The customer's recovery code then produced `8a9e33aa4da6…` — **byte-identical** to the pre-wipe on-disk key and to the hub's independent record — and installed cleanly on the bare box. `identity_blob` was unchanged across the wipe (572 B, `updated_at` still 11:11:37). Evidence: `audits/DRILL-r201-night-run-2026-08-04.md` §2 | **WHAT IT DOES AND DOES NOT CLAIM.** PROVEN: after a real rebuild the key recovers byte-identical, the EXISTING repository opens (3 snapshots, 42 026 B — the pre-wipe size; not a fresh history), and the customer restore flow returns the file byte-identical. **ALL FOUR MANUAL INTERVENTIONS ARE CLOSED, AND THE CUSTOMER IS NOW OFFERED THE RECOVERY (2026-08-05). ONE QUALIFIER REMAINS AND IT IS NOT THE OLD ONE — state it precisely.** The 2026-08-04 drill needed four interventions between "the key is recoverable" and "the file is back", none of them in any design document (R-204). **Items 1–3 closed in controller v0.198.0 + hub v0.95.0** (the reset code works first time; a Re-issue no longer marks a healthy escrow stale; a unit restore states it returned the app's definition and database and NOT the customer's files). **Item 4 closed in controller v0.199.0 + hub v0.96.0** (a rebuilt box DECLARES `offsite.state=needs_credential` and `internal/offsiteheal` re-arms the stored credential before minting). **AND THE "needs someone who knows to look" HALF IS NOW GONE TOO (controller v0.200.0, R-193):** a full-page recovery screen takes over the landing pages while the hub holds a sealed package this box cannot open, explains that nobody can replace a lost recovery code, takes the code, opens the repository and lists what is in it — proven live on demo-felhom, which is genuinely in that state (`/launcher` → 302 `/recovery`; a wrong code refused with nothing written, three times, no lockout; the code found in no file, no container log and no debug ring, with a planted-copy positive control that first exposed a mis-aimed sweep). **WHAT THE QUALIFIER IS NOW, and it is narrower:** (1) **the final unlock has never been driven with a CORRECT code through the page** — no recovery code was kept for demo-felhom's orphaned history and demo-hp's is operator-held out of band, so the live run exercised the whole chain (handler → agent → hub fetch → age KDF) and stopped at the unseal; the install and listing rest on tests. (2) **putting files back in place is deliberately NOT part of this** — restore stays per-app, and the step after the listing is R-213. (3) **the journey has still not been re-walked end to end since these fixes** — the closures are proven individually, not as one uninterrupted run. That re-walk is one more drill and it is what is owed. Scope: `demo-hp`, a controller-data-volume rebuild — **NOT** a total host loss, and **NOT** a guest reprovision. **⚠ THE RE-WALK RAN ON 2026-08-05 (CAMPAIGN-11) AND IT FAILED. THIS ROW STAYS AS IT IS — it records a key recovering and a file returning, which remains true — but the JOURNEY row below is the honest verdict and this row must not be read as covering it.** The campaign built a throwaway appliance from the published ISO, destroyed it, and walked the customer's route: **the data half PASSED** (three sentinels byte-identical, including a 12 MB binary and a non-ASCII Hungarian filename, out of the pre-wipe snapshot in 16 s) and **the journey half FAILED** — four operator interventions, three needing root on the appliance, and the first of them was the machine telling the customer their perfectly correct recovery code was wrong (R-216). **R-198's retention is NO LONGER unit-proven — it was PROVEN IN PRODUCTION on 2026-08-05** (CAMPAIGN-11 Phase 3): the first supersession since the fix retained `identity_blob` at **572 B, byte-length exact**, carrying the OLD key `626e424670248db3` while the current row moved to the newly minted `e11a6c542b73477a`; `offsite_repo_key_changed` and `escrow_superseded` both fired at the instant of supersession, and the next off-site run REFUSED as `orphaned` rather than starting a fresh history. Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` |
+| **The customer's own UNAIDED recovery journey, end to end** | controller v0.201.0, hub v0.97.1, agent v0.125.0 | **FAILED (2026-08-05, CAMPAIGN-11 Phase 1) — and it stays FAILED until a re-walk passes** | A throwaway appliance was built from the published ISO on demo-hp, claimed, given three sentinels, escrowed, backed up off-site, then destroyed — guest purged and both enrolled drives wiped. The rebuild and the customer's route were then walked with **no command line inside the guest**. **Data: PASS.** All three sentinels byte-identical (`beb9175d…`, `7c8cb0ad…`, `e012e76f…`), including a 12 MB binary and `C11-őrszem-ékezetes-árvíztűrő.txt` — the Hungarian filename survived disk → restic → SFTP → Storage Box → restore → disk intact. Restored in **16 s** out of the pre-wipe snapshot `f3d9cd67`, non-destructively. **Journey: FAIL.** Four interventions, three of them root on the appliance: R-216 (a correct code called wrong), R-218 (the store never opened; the box stopped asking), R-219 (the promised listing structurally could not render), R-220 (the app could not be redeployed — its drives were unenrollable). Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` | **WHAT IT DOES NOT CLAIM, and what the v0.201.0/v0.97.1 fixes do and do not change.** Six of the nine findings are fixed and each is red-proofed (R-216, R-218, R-219, R-217, R-222, R-215) — but **fixes are not a journey.** This row goes green only when the walk is repeated end to end and completes with no operator intervention. **Three findings are deliberately still open and each is a live blocker for some flow:** **R-214** (the console never stops showing a stale pairing code), **R-220** (after a rebuild the customer's drives cannot be re-enrolled — the deploy refuses and the wizard's list is empty, so no app can be redeployed onto its own data drive), **R-221** (a rebuilt box cannot run the escrow ceremony at all, because the pbsdr convergence marker survives on the host while the seeded `escrow.pbs_storage_id` is rewritten out of `agent.json`). **And R-216's fix does not by itself make a NEW box work:** with the hub guard corrected but the Day-0 manifest still vouching agent 0.120.0, a new box is **HELD, not served** — it stops being lied to, which is better, but the feature does not work for it until agent 0.125.0 is vouched. That is an operator decision |
+| **Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts** | controller v0.197.0 | **PROVEN-LIVE (2026-08-04) — and this row was OPTIMISTIC before it** | On demo-hp, an app on the **system-data fallback** bound `/mnt/sys_drive/userdata/media/books` while the capture set looked in `/mnt/sys_drive/felhom-data/userdata/media/books`: the declared-mandatory directory was in **no** snapshot and the run reported `ok` (R-203). After v0.197.0 the capture log reads `1 mandatory path(s)` and the file is **listed inside the snapshot** — `restic ls -l latest --tag calibre-web` → `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt`. Evidence: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 and the v0.197.0 CHANGELOG | **WHAT IT DOES NOT CLAIM.** It covers CAPTURE, not RESTORE: **no file has ever been restored from an off-site backup after a wipe** (R-201, ready to resume). It also does not claim the enrolled-drive layout was ever wrong — it was not, which is precisely why this survived. And a run that still misses a mandatory directory now reports **`incomplete`** rather than `ok`, so this row's guarantee is one the status can express || **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** |
 | **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-0Line truncated
 | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **R-25b CLOSED (hub v0.69.0, 2026-07-21):** the Danger-zone DELETE is now the guided full-teardown cascade that runs this very sequence as its middle leg — see the row below |
 | **Customer DELETE cascade** (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **UNIT-PROVEN; live leg PENDING** | `hub/internal/web/customer_delete_test.go` — leg ORDER observed from inside leg 2 (hosts already gone, customer row still present, custody still retained); 9 fail-closed gate cases each asserting zero mutations + zero external calls + no journal row; resume-after-external-failure converges; `purgeEscrow` custody semantics; preview leaks no secret. **5 red-proofs** (ack gate, stale-preview gate, ONLINE-host gate, leg order inverted, `purgeEscrow=true`) | Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run |
@@ -76,7 +80,7 @@
 | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded **DEGRADED** rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | `E2D-fresh-vm-2026-07-29` C1 (real 1.22.0 install, rc=0, `Day-0 provision SUCCESS`) + C2 (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort) | Case A (a second drive already present at install) has never fired naturally — only Case B has |
 | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 **DB replay route → `07-backup-architecture.md` §8 row 3** |
 | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 4, 5.** The matrix records that the copy's `recovery-unit/` mirror is read by no path (→ R-102) |
-| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect) |
+| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect). **2026-08-04 (R-193/R-197, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`) — the row is NOT overclaiming and the guard is not the gap; the CADENCE is.** This row already recorded that a recreated data volume orphans the repo, and the spike confirms the mechanism at source: `WriteOffboxSecrets` (`offbox.go:392`) mints a fresh 256-bit repo password whenever `<DataDir>/offbox/repo_password` is absent, and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly one caller, a web form a human pastes into. What was NOT recorded is that this now fires on an **ordinary, planned, unattended guest rebuild**, on every box: measured on BOTH demo boxes 2026-08-03/04 by comparing `host_escrow.restic_pw_sha256` against `host_escrow_superseded.restic_pw_sha256` (demo-hp `8e03eddf…`→`8a9e33aa…`, demo-felhom `48741892…`→`c60c8bc7…`), orphaning **15 snapshots / 40.9 MB** and **36 snapshots / 1.14 GB** respectively. **demo-felhom is the important half:** it kept its DELIVERY (a stale staged secret restored the target in 76 s) and lost its REPOSITORY anyway, with **nothing marking the escrow stale for 13 h** — `escrow_stale` is wired to `ReissueCredentials`, the one path that does NOT change the repo password (**R-196**), and absent from the rebuild path that does. Status unchanged: the classify-and-move-aside guard remains PROVEN-LIVE and correct, and is predicted (not yet measured) to refuse the 2026-08-05 run on both boxes rather than start a silent fresh history |
 | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** (2026-07-20) | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e **Route + RTO → `07-backup-architecture.md` §8 rows 3, 4** — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107) |
 | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
 | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured |

documentation/architecture/03-host-agent.md

@@ -378,6 +378,60 @@ nicety.
   **loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
   honest signup-time tradeoff, not a hidden one.
 
+#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
+
+A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
+the current `host_escrow` row into `host_escrow_superseded` **before** overwriting it, so a previous
+generation stays openable with the recovery code that sealed it. The same routine
+(`demoteCurrentEscrowTx`) is used when a host is deleted, which **demotes** custody rather than
+destroying it. The hub never decrypts either table.
+
+**The contract as originally shipped was incomplete, and the gap is worth recording because it was
+invisible for two months.** An escrow generation is **two** sealed artifacts — the K-escrow `blob`
+(the PBS datastore key `K`) and `identity_blob` (the age-wrapped `IdentityBundle`, which carries the
+offsite **restic repository password** as well as the tunnel/PBS tokens and the WG key). Retention
+covered only the first. So a supersession preserved the whole-guest key and destroyed the off-site
+data key — and because the copy happens as the new blob lands, **the destroying act was the ceremony
+itself**, i.e. the step a rebuilt box asks its customer to perform. Corrected in hub v0.93.0: the
+retained row carries both.
+
+**Two consequences that belong in this doc rather than a changelog:**
+
+- **Generational custody is now the recovery precondition for the offsite tier.** The operator ruled
+  on 2026-08-04 that no repository password is retained on the Proxmox host (R-193 candidate (c),
+  refused). With no host-side copy, a rebuilt box's only route back to its previous off-site history
+  is *the customer's older recovery code opening the retained identity blob* — which exists only if
+  the supersession that replaced it happened on hub v0.93.0 or later.
+- **Retention is custody, not recovery, and this doc must not be read as claiming otherwise.** The
+  hub's restore-mode serving of these blobs (§ slice 10D) has **no client** in the agent, the
+  controller's recovered-password injection seam has **no user interface**, and no part of the chain
+  has ever been exercised outside a unit test — the one live identity round-trip (2026-06-10)
+  predates the `ResticRepoPassword` field. Tracked as R-199/R-200/R-201; evidence
+  `documentation/audits/RECON-offsite-dr-chain-2026-08-04.md`.
+
+#### Serving a blob back to its own box (hub v0.94.0 + agent v0.125.0, R-199)
+
+Retention without retrieval is custody, not recovery. Since 2026-08-04 a host reads its own sealed
+identity blob from `GET /api/v1/hosts/{host_id}/escrow` — self-scoped by the per-host key, the mirror
+of the PUT that stored it — and opens it locally with the customer's recovery code via the agent's
+`POST /escrow/recover-offsite-password`, which returns **only** the offsite repository password.
+
+**This is a deliberate widening of who can obtain the ciphertext**, recorded in `CONTEXT.md` S-29 and
+in the handler itself: it previously took an operator arming recovery mode, and now takes only that
+box's own credential. The blob remains unopenable without R, and every retrieval raises an
+operator-visible `escrow_blob_served` event. The operator-driven re-enroll/restore-directive path is
+unchanged and keeps its recovery-mode gate.
+
+**Proven end to end on demo-felhom, 2026-08-04:** the recovered repository password's sha256 was
+byte-identical to the one on the box, and to the hash the hub had independently recorded. **The chain
+stops there** — nothing installs a recovered password, reopens a repository with it, or restores a
+file.
+
+**The honesty property above is unchanged and now has a second edge:** losing `R` and the box makes
+the offsite backups unrecoverable by anyone — *and* an older generation's data is recoverable only
+with **that generation's** recovery code, which the customer must still hold. A current recovery code
+does not open a previous generation's blob.
+
 ## 9. Provisioning & DR flows
 
 **Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs

documentation/architecture/07-backup-architecture.md

@@ -278,7 +278,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what
 | **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
 | **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
 | **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
-| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) |
+| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) |
+
+> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
+
 
 **[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
 offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
@@ -348,6 +351,51 @@ about whether the bytes can be turned back into a working system, and a system c
 completely while having no executable recovery route for a given failure. That is not a hypothetical
 here — §8 has rows where it is the actual state.
 
+### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
+
+> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
+> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
+> byte-identical. **It passed with an operator present, and four manual interventions stood between
+> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
+> are recorded here because this is the section a future reader will use to answer *"can the customer
+> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
+
+| # | The step | Why it stopped a customer | Status |
+|---|---|---|---|
+| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
+| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed` → `OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
+| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
+| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** was required. | **CLOSED — controller v0.199.0 + hub v0.96.0** (2026-08-05, operator ruling: automate it, and **the trigger is a state the BOX DECLARES**). The box now reports `offsite.state=needs_credential` when two local facts hold together — a fresh data area AND a hub-held recovery package — and the hub's `internal/offsiteheal` re-arms the stored one-time secret, minting only when there is nothing to re-arm. **Deliberately NOT automated: the escrow ceremony.** A credential is replaceable; the recovery code is not. |
+
+> **[FACT] The honest current answer, updated 2026-08-05 (controller v0.200.0): all four are gone AND
+> the customer is now offered the recovery.** A full-page screen meets the owner of a rebuilt box while
+> the hub holds a sealed package the box cannot open: it explains the situation, says plainly that
+> **nobody can replace a lost recovery code**, takes the code, opens the repository and **lists what is
+> in it** — apps, dates, sizes. No step requires an operator, and no step requires a command line.
+>
+> **WHERE THAT STOPS, and it is a real stop.** The screen **unlocks and only unlocks**. It restores
+> nothing. Putting files back is per-app, lives in the backups area, and the step after the listing —
+> a customer seeing what would change before anything is overwritten — is **not built** (→ R-213). A
+> screen that unlocks and then offers to overwrite is two decisions wearing one button, which is why
+> the operator ruled them separate.
+>
+> **What is still owed as evidence:** the final unlock has never been driven with a CORRECT code
+> through the page (no code was kept for demo-felhom's orphaned history; demo-hp's is operator-held),
+> so the live run exercised handler → agent → hub fetch → age KDF and stopped at the unseal. **And the
+> whole journey has not been re-walked end to end since these fixes** — the closures are proven
+> individually, not as one uninterrupted run. That re-walk is one more drill.
+>
+> **WHY THE TRIGGER FOR STEP 4 IS A DECLARATION, recorded here because it is the design and not an
+> implementation detail:** from the hub, an ABSENT off-site object means *never configured*,
+> *mid-restart*, *a transient config read failure* OR *rebuilt and stranded*, and the hub cannot
+> distinguish them. The BOX can, from two local facts it holds with certainty. So the box states its
+> condition and the hub acts on a stated request — never on a silence. Both facts are required:
+> freshness alone is a box that never had off-site backups, and an escrow alone is a healthy box.
+>
+> **The split that is deliberate and must not be widened: CREDENTIAL AUTOMATIC, KEY CUSTOMER-PRESENT.**
+> A credential is transport and is replaceable; the recovery code is not, because only the customer
+> holds it. Nothing in this chain runs, or asks for, an escrow ceremony.
+
 ### 7.1 The dependency graph
 
 > **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
@@ -941,3 +989,4 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
 - It does not estimate a single RTO or RPO. Every blank in §8 is a real gap.
 - It does not answer §11. Those are the operator's.
 - It does not claim ratification.
+| 5 | **A customer had no way to BEGIN — the only route was a command line** | Everything above was self-service, and nothing told the owner of a rebuilt box that a sealed package was waiting or how to open it. | **CLOSED — controller v0.200.0** (R-193). A full-page recovery screen, shown while the hub holds a package this box cannot open; it unlocks and lists, and **restores nothing** (→ R-213 for the put-back). |

documentation/audits/CAMPAIGN-11-recovery-journey-2026-08-05.md

@@ -0,0 +1,433 @@
+# CAMPAIGN 11 — the recovery journey (2026-08-05 → 08-06)
+
+**Four phases. Phase 1 (the clean journey) FAILED on the journey and PASSED on the data. Phase 3 (key
+supersession) PASSED on its central question. Phase 2 (eleven injected faults) and Phase 4 (an
+unattended soak) ran overnight on 2026-08-05/06 and are reported here for the first time.**
+
+**Findings: R-214 … R-223 from Phases 1 and 3** (seven since fixed and shipped), **plus R-224 … R-228
+from Phase 2.** **Three suspicions investigated — two confirmed, one DISPROVED.** **Five harness
+faults, separated from the product's.**
+
+Evidence: `../tests/campaign11-evidence-2026-08-05/` — `journal.md` (Phases 0, 1, 3) and
+`journal-phase24.md` (Phases 2 and 4, every observable in the order taken).
+
+> **The one-line answer to the question this campaign was built to ask.** A Hungarian household whose
+> machine is rebuilt **gets their data back only if an operator is standing next to them.** The
+> cryptography, the retention and the transport all work and are now proven live. **What fails is
+> being told the truth**: on this box, four different situations — a mistyped code, a hub outage, a
+> stopped agent, and a correct code for a retained earlier package — produce **one** message, and
+> three of the four are wrong.
+
+---
+
+## 1. Venue and baselines
+
+| | |
+|---|---|
+| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill host. Reached **by SSH key, first try** — R-129 stands |
+| VM | **321 `c11-appliance`** — q35/OVMF, 4 cores, 8 GB, `cpu=host` |
+| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` |
+| Storage | **`c11-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT** (a subdirectory fails the agent's `exactMount` check; Campaign 10 §1) |
+| Appliance | `192.168.0.105`, hostname `c11`, `felhom-agent 0.125.0` |
+| Guest | LXC **9201**, **`192.168.0.106`** — DHCP, and it MOVED between phases (`.207` → `.227` → `.106`) |
+| Hub customer | **`c11`** "Campaign 11", domain `c11.felhom.eu`, DR tier ON, off-site ON |
+| Host id | **`c11-36d660`** |
+| Untouched | `drill-r50` (VM 300, stopped), guest 9201 on both demo boxes, `local-lvm`, `felhom-backup`, every other hub customer |
+
+### Baselines — every value re-read fresh at the start of Phase 2
+
+| What | Value | How |
+|---|---|---|
+| `felhom-controller` `main` | **v0.201.0** @ `05cf352a2f17` | `HEAD` == `origin/main`, tree clean |
+| `felhom-agent` `main` | **v0.125.0** @ `0404f60e6a7b` | same |
+| `felhom.eu` `main` | @ `3a539ea5306a` | same |
+| hub, LIVE | **`felhom-hub:0.97.1`** | `kubectl … get deploy hub -o jsonpath` |
+| Day-0 manifest | agent **0.125.0** · golden **0.201.0** · `min_agent` **0.125.0** | hub `/configuration`, `selected` options |
+| c11 floor | **`v0.200.0 (override)`**; every other customer `v0.156.0` | hub `/configs` |
+| Highest register ID | **R-223** | fresh `grep -rhoE 'R-[0-9]{1,3}'` over all four repos |
+
+**All four cited commits match the brief exactly**, and the manifest matches the golden rebake the
+Phase-1/3 journal records. **Blast radius of the per-customer floor is still zero, measured** — the
+other five customers all read `v0.156.0`.
+
+### The hub CHANGELOG/deployed mismatch — diagnosed more precisely than filed
+
+The brief records *"`0.97.1` shipped without an entry"*. **The content was not missing; its heading
+was.** Commit `a7f1d27` wrote the change into the **v0.97.0** entry and `79e31ac` bumped the manifest
+to `0.97.1`, so a reader matching the running tag against `hub/CHANGELOG.md` found no `v0.97.1`
+heading at all. **Fixed in this session** — the paragraph now has its own entry, marked as added
+retroactively. Second occurrence of the class (the first was agent `0.90.1`).
+
+---
+
+## 2. Scope, and what was deliberately not isolated
+
+Campaign 10 could run with Tier 3 OFF; **a campaign about off-site recovery cannot.** Off-site
+hard-requires the DR tier, which provisions on **ep0** (Tier 2). The Phase-0 operator ruling stands
+and is restated here rather than quietly inherited: **ep0 and the Hetzner Storage Box are written to,
+additively** — a PBS namespace and token, a WireGuard peer, and a Storage Box sub-account, all created
+on the ordinary customer path. **Nothing existing is modified or deleted.** The brief's I7 wording
+("ep0 read-only") was relaxed by that ruling, not widened by this session.
+
+**Phase 2 added no new external writes.** Its faults are network blocks, a service stop, a container
+restart and a VM shutdown — all on the campaign's own appliance, all reverted, each with a positive
+control proving the fault was real.
+
+---
+
+## 3. The two positives that were owed (brief §4)
+
+### §4.1 — is the floor actually SERVED? **MEASURED. YES. Twice, independently.**
+
+The previous session recorded *"no HELD line and no held reason"* — **two absences** — and correctly
+refused to call that a measurement. Both positives were taken here.
+
+**(a) The box's own rendered state.** `/settings` → „Automatikus frissítés – **Minimális verzió
+(üzemeltető) `0.200.0`**". That value is `s.updater.GetFloor()`, and `u.floor` has exactly one writer
+— `SetFloor`, whose only non-test caller is the report-ACK handler. **Both hold branches of
+`ResolveManagedFloor` set `Floor = ""`** (`store.go:2086`, `:2095`), pinned by
+`managed_floor_test.go:94`. A non-empty floor on the box therefore proves an ACK carried one.
+
+**(b) A cold-started process saying it out loud.** F8's restart produced the decisive A/B — the same
+log line, same box, same code, before and after the golden rebake:
+
+```
+15:58:35  settle-gate: GO — floor still unknown after 1m30s               ← hold IN FORCE
+21:06:16  settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0)       ← hold RELEASED
+```
+
+**The hub half, as a positive rather than an absence:** `managed floor HELD for c11 …` fired every
+15 minutes from 17:57:11 to **22:12:06** and then stopped — and the silence is backed by a liveness
+control (the hub logged `host-report from demo-hp-bb76ea` at 22:44:41 and `wgsync: pushed 5 peers` at
+22:44:57, so it was demonstrably still logging).
+
+> **A correction to the brief's own plan.** It says to take this at DEBUG during F8's restart.
+> **A restart alone would not have produced it**: `SetFloor`'s line is `u.dbg(...)`, gated on a private
+> flag set from `cfg.Logging.Level == "debug"`, and it writes to the logger — **never to the logx debug
+> ring**, so it cannot appear in `/api/debug/logs` at any level. Confirmed live: 4 000 ring entries
+> spanning the release window contain no `SetFloor` line, **with a level census run first** (1196
+> DEBUG / 2802 INFO / 2 WARN) so the absence was known to be structural rather than evidential.
+
+### §4.2 — R-218's live half: **still NOT measured, deliberately, with the reason**
+
+The fix is present and readable — `needsOffsiteCredential` now retires the declaration on the
+**target**, not on the key:
+
+```go
+if t != nil { return false }   // a TARGET exists — not a rebuild
+```
+
+**The venue cannot exercise it.** c11 has a target (`applied_marker` present since 14:57), so
+`t != nil` and the box correctly does **not** declare; declaring here would be the bug. The state that
+exercises the fix — hub identity blob present, key placed, **no target** — is shape (a), which the
+venue held during Phase 1 and does not hold now. F7's set-aside was examined as a route to it and
+**does not produce it either** (§5, F7).
+
+**Recorded as still not measured rather than inferred from the unit test.** It needs one rebuild,
+which the brief forbids before Phase 4.
+
+---
+
+## 4. Phase 2 — the eleven faults
+
+**Judged on the message, not the outcome.** Every fault carries a positive control proving the fault
+was real; **every control that failed is reported as a failed control, not as a result.**
+
+### The four messages (`internal/web/recovery_handlers.go`, v0.201.0)
+
+| | fires when | first words |
+|---|---|---|
+| **M1** | unseal failed **and** no superseded package | „A megadott helyreállítási kódot **nem fogadtuk el**. Ellenőrizd, hogy mind a tíz szót…" |
+| **M2** | the capability gate says the agent cannot do it (R-216) | „Ez a gép **még nem tudja megnyitni** a mentéseidet… **A kódoddal semmi baj**…" |
+| **M3** | unlocked, inventory unreadable (R-217) | „**A kulcs visszakerült**, de a mentések listáját most nem sikerült beolvasni…" |
+| **M4** | unseal failed **and** a superseded package is retained (R-222) | „**Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk** ehhez a géphez…" |
+
+**The venue retains a superseded package**, so M4's branch — tested **before** M1 on the failure path
+— is live for every failed unlock. That single ordering fact produces two of the four new findings.
+
+### Results
+
+| | Fault | Right answer | Result |
+|---|---|---|---|
+| **F1** | wrong code ×3 | refused, **M1 only**, no lockout, nothing written | **PARTIAL** — refused ✅, no lockout ✅, nothing written ✅, but **M4, not M1** → **R-226** |
+| **F2** | no code at all | states nobody can recover it; offers set-aside | **PASS** |
+| **F3** | hub unreachable | names **the hub**, never the code | **FAIL** — M4. → **R-224** |
+| **F4** | agent stopped | **M2**, the R-216 fix under pressure | **FAIL** — M4. → **R-224** |
+| **F5** | store unreachable after a successful unlock | unlock counts; „could not be read", not „opened with content" | **PASS** — R-217's fix holds |
+| **F6** | „Most nem", return later | entry point survives; unlock still works | **PASS** |
+| **F7** | set aside, then a change of mind | set aside, never deleted; honest afterwards | **SPLIT** — set-aside **PASS**, verified byte-for-byte; the afterwards **FAIL** → **R-228** |
+| **F8** | controller restarted mid-unlock | no half-state; the screen says which | **PARTIAL** — no half-state ✅, but a raw English `Bad Gateway` → **R-227** |
+| **F9** | a box that never had off-site | nothing, incl. a direct `GET /recovery` | **PARTIAL PASS** — R-215's gate proven live on a narrower shape; the literal precondition was not staged |
+| **F10** | app with a mandatory data path missing | run reports `incomplete`, names the app, digest carries it | **NOT INJECTED** — harness. Three attempts, each self-healed |
+| **F11** | box offline a whole reporting window | hub and box agree once it returns | **PASS** — both directions fired, each with an operator mail |
+
+### F7 — the set-aside, and the afterwards
+
+**The move-aside is exactly what it claims.** Both confirmations state the consequences first
+(*„félretesszük — nem töröljük"*, *„a gép új, üres mentési tárolót kezd"*, *„ez az oldal többé nem
+jelenik meg"*), and the result measured against the Storage Box:
+
+```
+BEFORE  /home/felhom-repo                    mtime 13:13   snapshot f3d9cd67   du -s 12535
+AFTER   /home/felhom-repo.orphaned-20260805  mtime 13:13   snapshot f3d9cd67   du -s 12535   ← untouched
+        /home/felhom-repo                    mtime 21:32   (empty)                           ← fresh
+```
+
+**Nothing deleted.** Then the customer changes their mind — and finds nothing: `GET /recovery` 302s,
+`POST /recovery/unlock` 302s in 0.028 s with no message, and `/backups/remote` never mentions the
+set-aside history at all, though the box records its exact path. → **R-228**
+
+### F9 — what was and was not proven
+
+The literal precondition (a box that never had off-site backups) needs a rebuild, which the brief
+forbids before Phase 4, so it was **not staged**. But **the assertion that failed in Phase 1 was
+tested and passes**: Phase 1's defect was that `GET /recovery` never asked the predicate. After F7
+drives `recoveryOffer()` false, `GET /recovery` → **302 `/backups/remote`**. **R-215's fix is proven
+live.** The `GetHubEscrowIdentityPresent()==false` arm remains covered by tests only.
+
+### F10 — not injected, and why that is a harness result
+
+Three attempts, each with a control confirming the directory was genuinely absent — and each
+self-healed before capture: (1) the app recreated it; (2) `docker stop` was undone by **the
+controller's own monitor restarting the stack**; (3) with the stack stopped through the controller
+API, the directory reappeared at **21:51:38.65**, coincident with the run's own start. All three runs
+reported `ok` with `1 mandatory path(s)`, and R-203's stat-gap correctly never fired, because by the
+time `os.Stat` ran the path existed.
+
+**The state F10 describes is not reachable on a deployed app of this kind.** Recorded as **harness,
+not product**.
+
+**One observation kept, with its evidence:** at capture the directory held only a recreated
+`metadata.db` and **not** the customer's `F10-SENTINEL.txt`, and the run still said `ok`. The verdict
+is about a path's *presence*, not its *content* — correct as designed, and it means an `ok` off-site
+run can immediately follow the loss of everything that path contained.
+
+### F11 — the dead-man's switch, both ways
+
+```
+23:28:11  ok → stale (host_stale)        + Operator email SENT
+23:29:59  Received report from c11        ← the box returns unaided
+23:30:11  stale → ok (host_recovered)     + Operator email SENT
+23:30:11  customer mail skipped — no unanswered customer down mail (pairing miss)
+```
+
+The customer mail was correctly **withheld** by the pairing gate under a real outage. **I5 holds** —
+box and hub agree after the return, judged after a full report cycle rather than from one read.
+**Not reached:** `STALE → DOWN` (>1 h); the outage was ended once both transitions had fired because
+Phase 4 needed the venue back.
+
+### The timing discriminator used throughout
+
+`age`'s scrypt makes a real unseal cost ~1 s. **Phase 1's headline was diagnosed by a 0.134 s
+response**, and the same instrument separates every fault below:
+
+| | elapsed | what it means |
+|---|---|---|
+| F1 wrong code | **1.194 / 1.004 / 1.014 s** | a real unseal was attempted and failed |
+| F3 hub down | **0.0556 s** | **no unseal attempted** — it failed before the KDF |
+| F4 agent down | **0.0299 s** | **no unseal attempted** |
+| F5 store down | **1.198 s** | a real unseal, which SUCCEEDED |
+
+**The customer sees the same sentence for the 1.0 s case and the 0.03 s case.**
+
+---
+
+## 5. Findings
+
+### R-224 — every non-code failure on the unlock path is reported as a statement about the code
+
+**F3 and F4 are one defect with two faces**, and it is Phase 1's headline finding relocated from the
+version channel to the transport.
+
+| | injected (control) | customer sees | machine's own log |
+|---|---|---|---|
+| **F3** | hub REJECTed (`302` → `exit 7`) | **M4** | `fetching the sealed bundle: hub: transport error: … no route to host` |
+| **F4** | `systemctl stop felhom-agent` (`:8443` gone) | **M4** | `dial tcp 169.254.253.1:8443: connect: connection refused` |
+
+In both the **correct, current** recovery code was entered, so the only possible cause of failure was
+the injected fault.
+
+**The discriminator exists and is discarded at the HTTP boundary.** The agent's own `err` separates
+the cases exactly —
+
+```
+wrong code : "escrow: the recovery code did not unwrap the identity escrow …"
+hub down   : "escrow: fetching the sealed bundle: hub: transport error: … no route to host"
+```
+
+— but both return **HTTP 400** under one merged sentence (*"the recovery code did not open the sealed
+bundle, **or the bundle could not be fetched**"*), the agent's own `msg=` collapses them too, and the
+controller's failure path has **no branch for "could not ask / could not reach"** at all. `rerr != nil`
+falls straight into the code/package messages.
+
+**Why R-216's gate did not catch F4**, measured rather than reasoned — from the box's debug ring:
+
+```
+[web] recovery capability gate: offsite_key_recovery=yes (source=version)
+```
+
+**`source=version`.** The gate answers from the *known agent version* without probe traffic, which is
+correct for the question it was built for (*is this agent too old?*) and cannot answer the question it
+is being used for (*can this machine ask right now?*). **A dead agent of the right version sails
+through it** — and the failure that follows is attributed to the code, which R-216's own comment says
+must never happen: *"An attempt that cannot succeed must never be made, because its failure is
+attributed to the code."*
+
+**Consequence.** During any hub outage or agent restart, a customer holding a perfect recovery code is
+told it does not open their package and is routed to customer support about *older* backups. **I6** —
+an unreachable service reported as a fact about the code.
+
+### R-225 — the store reports `0 snapshots · 0 GB` when it cannot read it, beside a card saying it holds backups
+
+Found while checking F6's *"the listing is coherent"* clause. `/backups/remote` renders, **on one
+screen**:
+
+```
+Tároló méret · 0 pillanatkép            Tárhelykeret: 0 / 50 GB (0%)
+```
+
+immediately above:
+
+> „**A távoli tároló másik kulccsal készült mentéseket tartalmaz.** … **A meglévő mentések nem
+> sérültek**…"
+
+**Ground truth, measured directly against the Storage Box over SFTP** — a read-only listing, no
+decryption, using the box's own transport credential:
+
+```
+/home/felhom-repo/snapshots:
+  f3d9cd67d539c00359f0454ea7a782f45beac8406c55275bee5a6886afa8d791   (Aug 5 13:14)
+/home/felhom-repo:   du -s → 12535 KB          /home/felhom-repo/keys:   exactly ONE key
+```
+
+That is the Phase 0 snapshot holding **all three sentinels** — the customer's only surviving copy —
+matching the journal's `repo_size_bytes 12 611 522`.
+
+**Mechanism, from the box's own state:** after the Phase 3 rebuild the `offbox` block in
+`settings.json` carries **no `snapshot_count` and no `repo_size_bytes` key at all**. The values are
+*unknown*, and unknown renders as the zero value.
+
+**This is R-217's defect class in a second location** — a field whose zero is indistinguishable from a
+real measurement, defaulted past on an unknown path. `OffsiteInventory.Empty` exists precisely because
+*"len(Apps)==0 is also what a failed read looks like"*. **I6.**
+
+**I5 checked and NOT breached:** the hub's `/offsite` shows `Campaign 11 · 0.0 GB`, but that is 12.5 MB
+rounded to one decimal of a GB and the pool total (`Used 3.8 GB`) is consistent. The two views do not
+disagree; **both understate, for different reasons**, and only the box's snapshot **count** — an
+integer — is false.
+
+### R-226 — M1, the only message that tells a customer to check their typing, is unreachable on any box that has re-escrowed
+
+The failure path tests M4's condition **before** M1's:
+
+```go
+if present, at := s.recoverySuperseded(); present { …M4…; return }
+…M1…
+```
+
+So on every box the hub keeps an earlier package for, a **genuinely mistyped code** produces M4 —
+which is hedged (*„**Ha** egy korábbi kódot adtál meg"*) and states two true facts, but offers no hint
+to re-check the ten words and routes the customer to support about older backups.
+
+**Measured:** F1's three wrong-code attempts each returned M4, each after a real ~1 s unseal.
+
+**Why it matters rather than being a nicety:** the population that has re-escrowed is exactly the
+population that has just been handed a new recovery code and is most likely to be typing one. R-222's
+fix removed one conflation (correct-earlier-code read as a mistype) and introduced another
+(mistype read as a correct-earlier-code) **on the same branch**.
+
+### R-227 — a restart mid-unlock returns a raw English `Bad Gateway`
+
+F8 restarted the controller at T+0.7 s, inside the unseal window (control: the container's `StartedAt`
+moved). The customer got:
+
+```
+HTTP 502 · "Bad Gateway"
+```
+
+A raw upstream error, in **English**, from traefik. It names no reason, offers no action, and says
+nothing about whether the key was installed. **I3** — *"every refusal names a reason a person can act
+on, in Hungarian, with no raw error."* Low severity (the window is ~1 s wide) and recorded rather than
+inflated.
+
+### R-228 — the set-aside history becomes invisible the moment it is set aside
+
+The move-aside is correct and was verified byte-for-byte (§F7). What follows it is not.
+
+```json
+"orphaned_renamed_to": "/home/felhom-repo.orphaned-20260805"
+```
+
+The box records the exact path. And:
+
+```
+grep -rn "OrphanedRenamedTo" internal/web/templates/ internal/web/*.go   →   (no hits)
+```
+
+The field is **written and read by nobody**. `/backups/remote` after the set-aside contains no
+occurrence of the path, „félretéve", „régi előzmény" or any equivalent — with instrument controls
+passing (`felhom-repo` → 2, „letétbe helyezve" → 1), so the page and the matcher both work.
+
+> **12.5 MB of the customer's deliberately retained data sits at a path the box knows and never
+> shows.** Its only mention is a flash message on the redirect, gone on the next click.
+
+**The project's own "seam built but never wired" pattern** — the fifth recorded instance — landing on
+the one promise the set-aside screen makes. The fix is a plain statement that an earlier history is
+set aside and not deleted; **it must not promise the history can be reopened**, because R-222 means it
+cannot be, and that is exactly the conditional promise R-202's gate exists to prevent.
+
+---
+
+## 6. What behaved correctly
+
+**A campaign that reports only what broke is half a campaign.** These were tested and held.
+
+| | |
+|---|---|
+| **R-217's fix** | F5: with the store blocked after a successful unlock, the page rendered M3 and **no listing block at all**. The four false-claim strings („A tároló megnyílt", „van benne tartalom", „nem tudtuk alkalmazásokhoz rendelni") are absent — verified in UTF-8 with two accented positive controls present, after the documented accented-substring false-zero trap was accounted for |
+| **No lockout** | F1: three wrong codes, three identical responses, no rate limit, no refusal to try again — as `§8.4 NO LOCKOUT` documents |
+| **Nothing written on failure** | F1: all four `/data/offbox` files byte-identical, **mtimes frozen at 14:57:35** |
+| **I4** | the entered code appears in **no** log, file or page. Sweep run with a planted-canary control first; the only hits were the harness's own script |
+| **The „Most nem" asymmetry** | F6: the full page stops interrupting `/launcher` and `/dashboard`, while `/recovery` stays 200 and the backups-area entry point survives — bound to the offer, never to the postpone flag |
+| **No half-state** | F8: a restart mid-unlock left the key files and `settings.json` coherent; the controller returned healthy in 40 s |
+| **R-215's fix** | the `GET /recovery` gate is present in v0.201.0 and consults the same predicate as the POST sibling |
+| **R-198's retention, in the UI** | the hub host page reads „Key Escrow: present · **1 superseded escrow blob(s) retained**" |
+| **The empty submission** | F2: „Add meg a helyreállítási kódot." in 0.027 s — no unseal, and no blame attached to a code never given |
+
+---
+
+## 7. Harness faults, separated from the product's
+
+**Four, all mine, none a product defect** — recorded because Campaign 10's §4d is the format and
+because two of them nearly produced false findings.
+
+1. **`source ~/.config/credentials` echoes secrets.** The file holds keys with hyphens
+   (`R_DEMO-HP=…`) that bash cannot assign, and the `command not found` error **prints the value**.
+   Two demo-box recovery codes were printed this way before the helper was rewritten to `grep` the one
+   key it needs. **A live I4 hazard for any session that sources that file** — worth a memory, not a
+   register row.
+2. **Quoted values in that file** — a bare `cut -d=` keeps the quotes and yields the wrong secret; the
+   hub returned `302` until they were stripped. Already in project memory; re-confirmed.
+3. **The recovery page carries no `<meta name="csrf-token">`** — it renders a hidden `_csrf` input
+   (`data["CSRFField"]`). The first F1 run produced three `403`s and `CSRF-LEN-0`; **the harness's own
+   length check caught it**, and the controller's log named the reason word (`token mismatch`).
+   Harness, not product.
+4. **Two failed reachability controls, reported as failures.** The F5 probe first used `nc`, which the
+   controller container does not have, so blocked and unblocked printed the same fallback; the second
+   attempt used the IPv6 address `getent hosts` prefers, and **the container has no IPv6 route at
+   all**. Only a `/dev/tcp` probe against the IPv4 address (`91.98.242.176`) discriminated
+   `TCP-OPEN` → `TCP-CLOSED`. **Two readings that looked like results and were instrument failures.**
+
+---
+
+## 8. Suspicions investigated
+
+| | verdict |
+|---|---|
+| *"The floor is still held — there is no HELD line and no held reason"* | **DISPROVED.** Both absences were real, and both are explained: the hub stops logging when it stops holding, and `SetFloor`'s DEBUG line cannot reach the debug ring at all. The floor is served, measured two ways (§3) |
+| *"The hub and the box disagree about the store's contents (I5)"* | **DISPROVED.** Both understate; the hub's `0.0 GB` is rounding of 12.5 MB. Only the box's snapshot **count** is false → R-225, which is an I6 finding, not an I5 one |
+| *"`GET /recovery` still renders on a box it should not"* | **CONFIRMED FIXED** — the gate is present in v0.201.0 (§9, F9) |
+
+---
+
+*(Sections 9–12 — F7, F9, F10, F11, Phase 4, invariants, teardown and hygiene — follow below.)*

documentation/audits/DRILL-r201-night-run-2026-08-04.md

@@ -0,0 +1,265 @@
+# DRILL — R-201, the night run: **PASS. The customer's file came back, byte-identical.**
+
+**Date:** 2026-08-04, 21:30–23:15 · **Box:** `demo-hp` (Tier 0) · unattended, then **finished with the
+operator present**. The box is up, healthy, and fully re-armed.
+
+> **THE VERDICT: PASS.**
+>
+> ```
+> pre-wipe   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
+> restored   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
+> ```
+>
+> A machine was destroyed on purpose — controller data volume gone, the customer's file deleted from
+> disk — and **the file came back byte-identical**, through the customer's own recovery code and the
+> customer's own restore flow. **This is the first time the Felhom backup story has been proved end to
+> end.**
+>
+> The key recovered byte-identical (`8a9e33aa4da6…`), the existing repository **opened rather than
+> being re-initialised** (`repo_state: null`, 3 snapshots, 42 026 B — the pre-wipe size exactly), and
+> the pre-wipe snapshot `e6132ae5` was still there with the file inside it.
+>
+> **It took four undocumented steps to get there**, each measured, and they are the other half of the
+> result — see §3. A customer hitting this alone today would not get their data back.
+
+---
+
+## 1. The verdict — **PASS**
+
+| | |
+|---|---|
+| sentinel sha256, pre-wipe (step 3) | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
+| sentinel sha256, restored (step 10) | **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** |
+| **match** | **YES — byte-identical** |
+| snapshot count at step 9 | **3** — the pre-wipe count. **NOT 1.** No fresh history was started |
+| repo size at step 9 | **42 026 B** — the pre-wipe size exactly |
+| `repo_state` | **`null`** — the repository OPENED. Not orphaned |
+
+The restore ran through the customer flow (`POST /backup/offbox/restore`, `mode=full`, two-step
+confirm), out of snapshot `e6132ae5` — the one taken **before** the wipe.
+
+**A detail worth keeping:** the first restore attempt used the default `mode=unit`, which recovers the
+recovery unit only and **not** the userdata leg. `mode=full` is what returns the customer's files. A
+customer told to "restore from off-site" would, on the default, get their app definition back and not
+their documents — and nothing in that outcome says so.
+
+## 2. What was proven, in order, on hardware
+
+### Step 6 — the wipe
+
+The controller's data volume was destroyed and the bootstrap re-ran. Fresh data directory, everything
+stamped `2026-08-04 20:00:2x`:
+
+```
+encryption.key   32 B   2026-08-04 20:00:21     ← brand new: every pre-wipe app secret is undecryptable
+claimed = None                                   ← the fresh-install signal
+offbox  = null                                   ← no off-site target
+```
+
+**Faithful to the incident, and deliberately so.** The 2026-08-03 rebuild that R-193 is filed against
+was **not** a guest reprovision — the journal shows guest 9201 running continuously through that
+window, with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the
+controller and its data volume. Reproducing *that* is what the drill needs; a guest destroy plus an
+unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.
+
+### Step 7 — the assertion that keeps recovery possible: **PASSED**
+
+```
+host_escrow (demo-hp-bb76ea), AFTER the wipe:
+  identity_blob    = 572 bytes          ← unchanged
+  restic_pw_sha256 = 8a9e33aa4da6…      ← unchanged
+  updated_at       = 2026-08-04 11:11:37 ← unchanged; nothing re-escrowed
+  stale_at         = NULL
+```
+
+**No ceremony was run and nothing re-escrowed itself.** The sealed key survived the rebuild untouched.
+
+### Step 8 — **THE KEY CAME BACK**
+
+```
+=== offsite key recovery check (R-200) — compares, never installs ===
+  recovered sha256: 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a
+  [FAIL] there is no repository password on this box to compare against
+  (the recovery itself SUCCEEDED — this box simply has no local key. That is the
+   rebuilt-box shape, where the next step is to INSTALL rather than compare.)
+```
+
+Exit 1 is **correct and designed**: there was nothing to compare against, because the wipe removed it.
+The recovered hash matches the pre-wipe on-disk key and the hub's own record — three independent
+sources agreeing, one of them recovered through the full chain (hub → agent → unseal → extract) on a
+box that had just lost everything it knew.
+
+### Step 9a — installed cleanly
+
+```
+=== offsite key recovery INSTALL (R-200) ===
+  on-disk   sha256: (none — this box has no repository password)
+  recovered sha256: 8a9e33aa4da6…
+  [INSTALLED] the recovered repository password is in place and reads back identical.
+```
+
+The "installed" branch — the rebuilt-box shape v0.196.0 was written for — took its first real run.
+
+### Step 9b — the apply kept it
+
+After the target was reconfigured, the on-disk key was still `8a9e33aa4da6…`. `WriteOffboxSecrets`
+found the file present and kept it, exactly as documented.
+
+---
+
+## 3. The wall — three blockers, each measured
+
+### (a) R-193, reconfirmed live: a rebuilt controller cannot configure its off-site tier
+
+```
+[INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.197.0)
+[WARN] [offsite-apply] reconcile: offsite-apply: consume one-time password:
+       no unconsumed offsite password (already consumed or none provisioned)
+       (retries on next config refresh/restart)
+```
+
+The ledger, measured: `demo-hp` one-time secret created `07:11:51`, consumed `07:12:06` — **by the
+previous controller**. The rebuilt one has nothing to consume and retries forever. Exactly the state
+demo-hp sat in for 25 hours on 2026-08-03.
+
+**Remedy:** an operator Re-issue. Performed here through the designed endpoint
+(`POST /configs/demo-hp/offsite-reissue`, HTTP 303), after which the target configured normally.
+
+### (b) R-196, measured live — and it lands squarely on the recovery path
+
+```
+host_escrow (demo-hp-bb76ea), after the Re-issue:
+  stale_at         = 2026-08-04 20:15:49   ← set
+  restic_pw_sha256 = 8a9e33aa4da6…         ← UNCHANGED. The key did not move.
+```
+
+**The escrow was marked stale while it perfectly covers the box's current key** — the recovered one.
+That is R-196's false staleness, and here it is not a cosmetic lie: a stale escrow makes the hub
+**withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending →
+escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses to run any off-site backup**.
+
+> **The remedy for (a) disables the recovery it was needed for.** The only documented way to clear a
+> stale escrow is a fresh ceremony — which supersedes the identity blob, i.e. **destroys the very key
+> being recovered**. Under hub v0.93.0 the old blob is now *retained*, but nothing can serve a
+> superseded blob back (R-199's inventory: the endpoint serves the CURRENT row only).
+
+**Superseded rows still number 2** — no ceremony was run tonight. The key is intact.
+
+### (c) The claim gate — undocumented as a recovery step
+
+```
+[DEBUG] [web] claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)
+```
+
+A rebuilt box is **unclaimed** (`claimed = None`, fresh `settings.json`), and the claim gate correctly
+intercepts every non-claim route. **So no controller endpoint can be driven at all** — not the manual
+escrow confirm, not the backup trigger — until the customer re-claims the box. Both POSTs returned 302
+to the claim page and neither reached its handler; `last_run` stayed null, which is why the earlier
+reading of "the run was refused" needed this second look to be accurate.
+
+This is correct behaviour and it is **not wrong** — but it is a step in the customer's recovery journey
+that appears in no design document, and it comes *before* anything else can happen.
+
+---
+
+## 4. How it was finished, and the fourth undocumented step
+
+The unattended run stopped at the wall in §3 rather than improvising further at 22:35 (§8.10). With the
+operator present the remaining path was walked, and it took **four steps a customer would have to
+discover**:
+
+1. **Re-issue** the offsite credential (hub, operator-only) — otherwise the box can never configure its
+   tier. §3(a).
+2. **Re-claim the box.** The local escape hatch (`--print-reset-code`) mints a code — **and does not
+   work on its own**: it writes the new hash to `settings.json` while the RUNNING controller keeps its
+   old copy in memory, so `effectiveClaimCode()` never sees it and every attempt returns
+   *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming.** Two
+   claim attempts failed on this before it was diagnosed; the generation counter reached 6.
+3. **Confirm the escrow** manually (`/backup/offbox/confirm-escrow`) → `escrowed`, so `OffboxRunnable`
+   permits a run. **Not a ceremony** — the identity blob was never superseded (still 2 retained rows).
+4. **Restore with `mode=full`.** The default `mode=unit` returns the recovery unit and **not** the
+   customer's files.
+
+**None of these four is in any design document, and the drill needed all four.**
+
+## 5. §5's five conditions, as recorded before the wipe
+
+| # | Condition | Evidence |
+|---|---|---|
+| 1 | §4 confirmed, sentinel **listed by name** | snapshot `e6132ae5` (19:36:26) — `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt` |
+| 2 | deliberate rollback archive, **verified** | `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B; **full zstd stream read OK** (4 867 573 760 B uncompressed); the sentinel confirmed **inside** it |
+| 3 | §3's option | see §6 |
+| 4 | demo-felhom untouched | `health=ok`, `escrow_state=escrowed`, not touched at any point |
+| 5 | free space | nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 % |
+
+**A precondition had drifted and was repaired before the wipe, not worked around.** The staged
+snapshot no longer contained the sentinel: the afternoon's `[WARN] mandatory data path missing`
+experiment produced a *later* same-day calibre-web snapshot, and restic's `forget --keep-daily 7
+--group-by host,tags` had pruned the good one in its favour. The fixture on disk was correct, so one
+backup re-established it and it was re-verified by listing. **Lesson worth carrying: a good snapshot is
+not durable against a later bad run on the same day.**
+
+## 6. §3 — the recovery code
+
+**Option B as already in place, with a strict improvement: no new copy was created, so nothing needed
+shredding.**
+
+The operator placed `R_DEMO-HP` in their own `~/.config/credentials` on DooPlex (mode `0600`) two
+sessions ago, deliberately, for this purpose. This session read it from there and **piped it to stdin**
+for each of the two invocations that needed it. It was never an argument, never exported, never written
+to a second file, and never logged.
+
+**Nothing was shredded, and that is the point:** the operator's own permanent store is theirs, not a
+drill artefact, and destroying it would have destroyed their record. Because no additional copy was
+made, there is nothing left behind to prove gone — a stronger position than option B's
+create-then-shred.
+
+**Verified afterwards** with the planted-copy positive control, exactly as on 2026-08-04 afternoon —
+see §8.
+
+---
+
+## 7. The state the box is left in — healthy and re-armed
+
+```
+controller   felhom-controller:0.197.0, healthy, CLAIMED
+apps         privatebin opengist calibre-web filebrowser cloudflared traefik  — all serving
+escrow_state escrowed          (no ceremony was run; identity_blob untouched)
+repo key     8a9e33aa4da6…     THE RECOVERED KEY — and the repository opens with it
+off-site     3 snapshots, 42 026 B, repo_state null; all three apps re-toggled for off-site
+sentinel     restored to its live location, sha256 643166269103a25c… — verified
+```
+
+The verification copy under `backups/offsite-restore/calibre-web/` is **left in place as evidence**.
+
+## 8. R persisted nowhere — searched, with a positive control
+
+Swept the agent journal, the controller container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`,
+`/root` on demo-hp for the recovery code: **0 hits**, and **0** leftover `felhom-idesc-*` unseal
+staging directories. A planted copy was found by the same sweep (**1**) and not found after shredding
+(**0**), so the instrument is shown sensitive rather than assumed to be.
+
+---
+
+## 9. Teardown — three layers
+
+| layer | state |
+|---|---|
+| the guest | **Nothing torn down.** The wipe is the evidence; the apps are serving; the recovered key is in place. The scratch band is empty — no restore-test guest was created tonight. |
+| the host | `vzdump` snapshot LVs removed cleanly by the archive job (`snap_vm-9201-disk-0/1_vzdump` both released). One new archive, 1.6 GB, on `nvme-1tb` (883 GB free). Nothing deleted. |
+| the hub | **No new customer records.** `demo-hp` is the same customer row throughout — the rebuild was a controller-data wipe, not a re-enrolment, so nothing accumulated. One Re-issue staged a fresh one-time secret (consumed at 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran. |
+
+**Nothing was deleted on the storage endpoint**, including the ~1.2 GB of previously-orphaned
+ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.
+
+## 10. Part 2 — not run
+
+Its gate is *"the drill PASSED; its evidence is written down; and there is time."* The first two are
+now met. **The third was not** — it was 23:15 after a full destructive cycle, and Part 2 is a second
+wipe. It is deliberately left for a session that starts fresh, with drill 1's result already recorded
+so a second wipe cannot overwrite it.
+
+**R-198's retention therefore remains unit-proven only.** Nothing has yet superseded a key in
+production, so the retention's live behaviour is still unobserved — and it is now the last unproven
+link in the chain this drill exercised. The check is cheap: wipe again, do NOT recover, let a ceremony
+seal a different password, then read the superseded row's `identity_blob`.

documentation/audits/DRILL-r201-offsite-recovery-2026-08-04.md

@@ -0,0 +1,311 @@
+# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE**
+
+**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.**
+**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the
+verdict unobtainable — and that defect is worth more than the drill.
+
+> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the
+> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one
+> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup
+> succeeded. → **R-203**
+>
+> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it
+> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a
+> sentinel file on disk.
+
+---
+
+## 1. The verdict — not reached, and why that is the correct outcome
+
+The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that
+the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have:
+
+- destroyed the sentinel, which exists only on that box;
+- proven nothing about recovery, because there would be nothing to recover;
+- and done so *after* the point of no return.
+
+The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That
+means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was
+one the runbook's P1–P6 table did not contain, because nobody knew to look for it.
+
+**Sentinel sha256 (step 3), recorded and still on the box:**
+`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
+at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B).
+
+---
+
+## 2. R-203 — the defect that halted the drill
+
+**Measured, twice, on the live box.**
+
+| what | path | exists? |
+|---|---|---|
+| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) |
+| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
+
+The controller's own log, verbatim:
+
+```
+[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
+       /mnt/sys_drive/felhom-data/userdata/media/books
+[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s))
+[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
+```
+
+**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing
+customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was
+dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the
+snapshot* — and the code even has the right words for it in a WARN nobody reads.
+
+**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)`
+(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path**
+(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`,
+`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed
+as **`<HDD_PATH>/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive`
+and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories.
+
+**And the same compose file used BOTH roots.** From `docker inspect calibre-web`:
+
+```
+bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest     ← felhom-data root
+bind /mnt/sys_drive/userdata/media/books               -> /calibre-library      ← NO felhom-data root
+```
+
+`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one
+template, two roots.
+
+### What is measured and what is not — stated because the scope changes the fix
+
+- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run
+  reports `ok`; the only signal is a container-log WARN.
+- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because
+  demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app
+  namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted**
+  (HTTP 202) rather than refused.
+
+**Either branch is a defect, which is why this is filed regardless:**
+
+- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for
+  every app deployed on it, and their mandatory directories are silently unprotected;
+- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it
+  refused the NAS one call earlier, and the refusal that exists is not reaching this case.
+
+**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two
+pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they
+own is in named volumes — so they are unaffected, and their snapshots are real.
+
+---
+
+## 3. Preconditions, each measured
+
+| # | Precondition | Result |
+|---|---|---|
+| **P1** | operator holds the recovery code | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* |
+| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. |
+| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill** — `identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). |
+| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. |
+| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. |
+| **P6** | space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. |
+
+---
+
+## 4. Step-by-step, with every observable
+
+### Step 1 — starting state (hub, read-only)
+
+`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`.
+Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of
+a controller that has never run an off-site backup in this lifetime.
+
+### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction
+
+The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next
+off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting
+a fresh history, but **refusing**. Measured here, twice over.
+
+**Read-only probe first** (`restic cat config` with the current key, writes nothing):
+
+```
+Fatal: wrong password or no key found
+```
+
+— the exact string `classifyResticProbe` maps to `"orphaned"`.
+
+**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`):
+
+```
+[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
+       no-longer-available key; runs will skip until reset
+[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
+[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult …
+```
+
+`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card
+rendered on `/backups/remote`, and the event reached the hub (HTTP 200).
+
+> **The system stopped and said so. It did not quietly start a new history over the old one.**
+> That closes R-193's open Q3 — and it is the good half of this month's story.
+
+**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written
+under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow
+row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`,
+**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a
+recovery code.
+
+### Step 1c — the reset (operator-authorised)
+
+The orphan card's own reset, confirmed by the operator during the session:
+
+```
+[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
+       move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init
+[INFO] [offbox] orphaned repo reset complete — old history set aside at
+       /home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized
+[INFO] Event pushed: offbox_repo_reset (info)
+```
+
+**Nothing was deleted.** The reset path had never run in anger before; it works.
+
+### Steps 2–3 — the recovery code and the sentinel
+
+The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured
+hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z`
+(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated
+them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is
+v0.93.0's Scenario E, live.
+
+**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate:
+neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both
+keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore
+flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable
+by design.
+
+`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a
+single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202),
+toggled for off-site, and a Tier-1 recovery unit captured
+(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`).
+
+Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B.
+
+### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong**
+
+```
+last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
+snapshot_count 3 · repo_size_bytes 30 636 · repo_state null
+```
+
+Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2.
+
+### Steps 5–11 — NOT RUN
+
+Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is
+moot given §2. Steps 6–11 (archive, wipe, reinstall, recover, install, restore, compare) were **not
+started**. The §7 STOP was never reached, because the drill failed its own precondition first.
+
+---
+
+## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half)
+
+`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract
+path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered
+password via `InjectOffboxPassword`.
+
+- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes
+  nothing. A single interactive prompt would have had to share stdin with R.
+- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape),
+  *unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present;
+  installing would clobber the key the current repository is encrypted under — exit 2, no force offered).
+- It re-reads the file after writing rather than trusting the call's return.
+
+**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK.
+Removing the confirmation gate makes the dry run write the password and fails
+`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence
+test carries a **positive control** — a planted copy of the code is found by the sweep, then removed
+and not found — because an absence check is worth only what its sensitivity is.
+
+**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted
+before step 9. Its unit proof stands; its live proof does not exist.
+
+---
+
+## 6. What this drill did and did not establish
+
+**Established, live, for the first time:**
+
+1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted
+   third outcome, measured. It does not silently start a fresh history.
+2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed.
+3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot
+   while the run reports `ok`** (R-203).
+4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four
+   hours before the fix that would have kept it.
+
+**NOT established — and unchanged from before this session:**
+
+- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still
+  open, and its pass condition is unchanged.
+- Part 0's install path has never run against a live recovery.
+- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session
+  superseded a key, so nothing exercised it.
+
+---
+
+## 7. State left behind, and teardown
+
+**Deliberately not torn down** — this is evidence, and the box is better off than it was:
+
+| layer | state |
+|---|---|
+| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. |
+| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. |
+| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. |
+| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. |
+
+**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed,
+and §8.3 of the runbook forbids riding it along with a drill. The reset added
+`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside.
+
+**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to
+confirm the key exists in the credentials file; no unseal was performed on demo-hp.
+
+---
+
+## 7b. UPDATE, same day — the blocker is FIXED and the drill is ready to resume
+
+Controller **v0.197.0** shipped both halves of R-203:
+
+- **the paths agree.** On demo-hp the live bind moved from `/mnt/sys_drive/userdata/media/books` to
+  `/mnt/sys_drive/felhom-data/userdata/media/books` — the directory the capture set looks in. The
+  capture log went from `0 mandatory path(s)` to **`1 mandatory path(s)`**.
+- **`ok` means it.** A run that cannot capture a MANDATORY directory now reports **`incomplete`**,
+  names the app and the folders, and raises the operator digest — instead of `ok` with a warning
+  beside it.
+
+**And the sentinel is in the snapshot, listed by name:**
+
+```
+$ restic ls -l latest --tag calibre-web
+-rw-r--r--  1000  1000  181  2026-08-04 12:53:06  /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
+```
+
+sha256 **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** — byte-identical to §1,
+verified after the fix's migration moved the file to the corrected directory.
+
+**One live check could NOT be reproduced, and is recorded rather than claimed.** Hiding the books
+directory to watch the `incomplete` verdict fire on hardware did not work: the running container's
+bind mount **recreated** the directory, so `os.Stat` succeeded and there was no gap. That is itself
+worth knowing — a bind-mounted directory cannot easily be "missing" while its app runs, so the
+mandatory-gap condition arises in practice when the path resolves somewhere the app never binds (the
+R-203 case), not when a live app's own directory vanishes. The verdict is proven by the run-level
+test, which drives the real `RunOffboxBackup` and asserts both `incomplete` and the operator signal.
+The fixture was restored and the sentinel re-verified at the same hash.
+
+## 8. To resume the drill
+
+1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can
+   survive a wipe and the drill cannot reach its verdict.
+2. Re-run steps 4–5 and confirm the sentinel IS in the snapshot — by listing it, not by a green status.
+3. Then P4 (the deliberate archive), the §7 STOP, and steps 6–11 as written.
+
+Everything else is already in place: the code, the versions, the recovery code, the working repository,
+the file-leg app and the sentinel.

documentation/audits/RECON-offsite-dr-chain-2026-08-04.md

@@ -0,0 +1,531 @@
+# RECON — can a customer actually get their backups back?
+
+**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
+**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.
+
+> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
+> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
+> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
+> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
+> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
+> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
+> because the remedy closes it.
+
+---
+
+## 1. Confirmed baselines (re-read on arrival)
+
+| Repo | `main` @ commit | tree | Version |
+|------|-----------------|------|---------|
+| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
+| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
+| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |
+
+All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.
+
+**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
+R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
+`CONTEXT.md`. New items in this session therefore start at **R-198**.
+
+---
+
+## 2. Instruments — what each can and cannot answer
+
+| # | Instrument | Exact read | Known limit |
+|---|---|---|---|
+| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
+| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) |
+| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
+| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
+| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |
+
+**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
+write of any kind, nothing touched on the storage endpoint.
+
+---
+
+## 3. Q1 — the chain, link by link
+
+From a dead node to an open repository. **Status vocabulary as specified.**
+
+| # | Link | Component / function | Who invokes it | Status |
+|---|---|---|---|---|
+| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
+| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
+| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
+| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
+| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
+| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
+| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
+| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
+| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
+| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
+| 11 | a backup or restore proves it | — | — | **never** |
+
+### The chain in one sentence
+
+Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually
+manual, and link 6 has no client at all. Link 11 has never happened.
+
+---
+
+## 4. Q2 — where the chain stops today
+
+**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**
+
+Two distinct stopping points, and the order matters:
+
+**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
+still seals the *old* password. Recovery is possible in principle, and a person must:
+
+1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
+   or skip it entirely, because nothing consumes the endpoint it gates;
+2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
+3. copy the blob to a machine with the agent binary and `age`;
+4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
+5. open `bundle.json` by hand and read `restic_repo_password` out of it;
+6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
+   session + CSRF) — there is no form;
+7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
+8. trigger a run and hope `restic cat config` succeeds.
+
+Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
+to be present with R, and one (step 6) with no user interface at all.
+
+**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
+rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
+`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
+has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
+**no procedure exists**, with or without R, at any level of effort. The password is gone.
+
+Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
+`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
+snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.
+
+---
+
+## 5. Q3 — can a rebuilt controller know it should recover?
+
+### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**
+
+**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
+`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
+are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
+about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
+reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
+exactly the right line, but cannot distinguish "never configured" from "rebuilt".
+
+**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
+(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
+(`internal/report/escrow_confirm.go:19-23`):
+
+```go
+type EscrowStatus struct {
+	IdentityBlobPresent bool   `json:"identity_blob_present"`
+	ResticPwSHA256      string `json:"restic_pw_sha256"`
+	CreatedAt           string `json:"created_at"`
+}
+```
+
+That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
+every report cycle, on a channel that already exists, with no new API and no new secret.
+
+**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
+immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
+non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
+there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
+exactly **one** reference to the ACK field in the whole controller
+(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).
+
+**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:
+
+- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
+  caches the hub's claim-code state into `settings.json` idempotently by generation,
+  `internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
+  which is the correct shape here too);
+- at the mint branch, refuse when the password file is absent **and** the cached status says
+  `identity_blob_present && restic_pw_sha256 != ""`.
+
+**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
+ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
+`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
+design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
+status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
+nothing on a partial failure, `offsiteapply.go:1-6`).
+
+**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
+9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):
+
+| file | mtime |
+|---|---|
+| the data directory's parent | `2026-08-03 07:17:25` |
+| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
+| `offbox/` | `2026-08-03 07:18:02` |
+| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |
+
+The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
+Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
+rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4
+reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
+submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.
+
+> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
+> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
+> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).
+
+---
+
+## 6. Q4 — what the stale-escrow detector actually covers
+
+### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**
+
+**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
+that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
+event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
+and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
+detectors; only one of them was ever claimed to be misplaced.
+
+**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
+with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
+true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
+is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:
+
+```
+[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
+(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
+```
+
+**So: confirmed, it fires.** The detector is not absent from the mint path.
+
+**Three limits, each of which matters more than the fact that it fires.**
+
+1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
+   flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
+   `:130-165`); the pending branch sets no display flag. The operator learns nothing.
+2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
+   `SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it
+   → **the old repo password ceases to exist**. The detector correctly identifies the situation and
+   then tells the customer to close the door.
+3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
+   deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
+   identical warning.
+
+So: **less needs building than the prompt feared for the *detection*, and more needs building for the
+*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
+signal or a recovery.
+
+---
+
+## 7. Q5 — is the superseded-escrow retention real, and what does it retain?
+
+### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**
+
+**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):
+
+```sql
+INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
+SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
+FROM host_escrow WHERE host_id = ?
+```
+
+`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):
+
+| table | columns |
+|---|---|
+| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
+| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |
+
+**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
+**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
+(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
+`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
+`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
+else.
+
+**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**
+
+- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
+- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
+- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.
+
+**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
+production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
+for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
+comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
+this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.
+
+**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:
+
+> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
+> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
+> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*
+
+The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
+own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):
+
+> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
+> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
+> kóddal később visszaállítható lehet."*
+
+**For the offsite restic repository — the incident it names — that is false.** The retained blob does
+not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
+asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
+→ **R-198**, and the capability-map correction in §9.
+
+---
+
+## 8. Q6 — has any part of this ever been exercised?
+
+**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
+identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.
+
+### Proven
+
+| what | evidence | scope of the proof |
+|---|---|---|
+| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
+| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
+| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
+| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |
+
+### Never exercised — named in those words
+
+- **A blob has never been served to a box.** Link 6 has no client (§3).
+- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
+  caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
+  explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
+  run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
+- **A recovered repo password has never been injected into a controller.**
+- **An existing offsite repository has never been reopened with a recovered password.**
+- **No restore of any kind has ever been performed from a recovered secret.**
+
+`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
+*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
+operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
+exercised destructively**"*. **That document is right and this session confirms it.** What it did not
+catch — because it was written from `host_escrow`, not from the supersession path — is Q5.
+
+**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
+**retention** claim quoted in §7. See §9.
+
+---
+
+## 9. Q7 — the operator's ruled shape, assessed row by row
+
+> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
+> blob, then the controller should yell that recovery is available, and provide a form for the customer
+> to enter the recovery key. After unlocking the blob, the controller should show what will be
+> recovered before proceeding.*
+
+| The ruling requires | Exists? | What it needs, and what it costs |
+|---|---|---|
+| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
+| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
+| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
+| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
+| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
+| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
+| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |
+
+### Things the ruling implies that do not exist, priced
+
+| missing piece | cost | note |
+|---|---|---|
+| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
+| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
+| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
+| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
+| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
+| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
+| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |
+
+### The security question the ruling raises — stated, not answered
+
+**What gates the surface today.** The controller dashboard is behind `RequireAuth`
+(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
+`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
+logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
+the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
+password**, on a controller reachable at whatever address that box's dashboard is reachable at.
+
+**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
+snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
+the repository size.** That is a map of the household's digital life, and it is precisely the metadata
+the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.
+
+**Three specifics the operator should weigh, without this session choosing:**
+
+1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
+   (~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
+   online guessing is not a route. But **anyone with the dashboard password can already restore live
+   offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is
+   confined to the *orphaned/previous* repository, which today nobody can read at all.
+2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
+   dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
+   fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
+   and should rate-limit like the login path does.
+3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
+   root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
+   set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
+   the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
+   smaller and time-bounded form**, and it should be decided alongside it rather than separately.
+
+---
+
+## 10. Q8 — the drill design
+
+**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
+no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
+the full circle. **This section is the design; the drill is not run in this session.**
+
+### Target and preconditions
+
+| | |
+|---|---|
+| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
+| **Guest** | the customer guest on that host (rebuild target) |
+| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
+| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
+| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
+| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |
+
+### The cycle
+
+| # | Step | Who | Observable that must be recorded |
+|---|---|---|---|
+| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
+| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
+| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
+| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
+| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
+| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
+| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
+| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
+| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
+| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
+| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |
+
+### The single observation that makes this a FAILURE rather than a pass
+
+> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**
+
+A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
+not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
+pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
+customer's data came back. Only the sha256 comparison does.
+
+### Secondary failures worth recording separately (each is a finding, not a pass/fail)
+
+- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
+  closes without a human act (would be new and serious).
+- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
+- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
+  the transport, not the key.
+
+### What could make it fail for uninteresting reasons
+
+The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
+(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
+step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
+`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
+its GO line rather than retrying).
+
+### Rollback, and whether the loss is acceptable
+
+**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
+whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
+explicitly disposable, and it is the box the target-selection runbook names for drills. **The
+rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
+is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
+**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
+do not rely on the nightly.
+
+**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
+box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.
+
+---
+
+## 11. Every query used
+
+| # | Source | Query / read | Value read |
+|---|---|---|---|
+| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
+| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
+| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
+| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
+| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
+| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
+| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
+| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
+| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
+| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
+| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
+| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
+| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
+| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
+| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
+| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
+| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
+| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |
+
+---
+
+## 12. What could not be established
+
+Named, not guessed.
+
+1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
+   runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
+   Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
+2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
+   (I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
+   credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
+   identical on both boxes, which is suggestive, not decisive.
+3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
+   the endpoint needs the SFTP credential, which was not touched.
+4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
+   holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
+   have carried the identity blob anyway).
+5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
+   agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
+   demo host.
+
+---
+
+## 13. Observations — noticed, not acted on
+
+1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
+   pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
+   dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
+   week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
+   **not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
+2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
+   means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
+3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
+   a trap for anyone reusing it as a read-only preview primitive (§9).
+4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
+   the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
+   nothing currently calls — worth a second look before anything starts calling it.
+
+---
+
+## 14. The operator's decisions — stated, and NOT answered here
+
+1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
+   *"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
+   history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
+   that walks the current chain will pass or fail on a chain that is missing a link everyone believed
+   was there.
+2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
+   **unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
+   passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
+   indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
+   act on it and states it as the operator's.
+3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
+   that every rebuild starts a new repository. **This session adds one input to that decision:** the
+   "accept it, provided the system says so loudly" branch currently rests on a retention feature that
+   does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
+   stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
+   to be re-described honestly to the customer.
+</content>
+</invoke>

documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md

@@ -0,0 +1,826 @@
+# SPIKE — DooPlex build-cache containment (cap at 30 GB, relocate to SSD2)
+
+**Date:** 2026-08-05 · **Host:** DooPlex (192.168.0.180) — **Tier 2, protected**, not a drill target
+**Class:** Spike. §1, §2, §3, §4, §13-verify, §15 apply; §5–§12 skipped (no production Go code written).
+**Anchor:** `dooplex_report.md` (2026-08-05 read-only disk audit). Every number in it was treated as a
+claim to re-measure, not a fact to carry forward.
+
+> Unit convention: **GB = decimal (10⁹)**, because `df -B1`, `du -B1` and Prometheus emit raw bytes.
+> `docker buildx inspect` renders **GiB** (2³⁰) — where that matters it is labelled.
+
+---
+
+## Verdict
+
+**The mechanism is confirmed, with one correction and one refutation, and the follow-up is safe to spec.**
+
+- **Confirmed:** a `builder.gc` policy in `/etc/docker/daemon.json` **is** honoured under the containerd
+  worker, and it **does** evict. Proven by naming a 440 MB cache record present at one build and absent
+  two builds later — not by the absence of an error.
+- **Corrected:** it is honoured **only in the `policy` array form**. The flat form
+  (`{"gc": {"reservedSpace": …}}`) is **silently ignored** — the daemon starts, logs nothing, and keeps
+  its built-in defaults. `dockerd --validate` is worthless here: it returned `configuration OK` for a
+  bogus key **and** for a config that then crashed the daemon.
+- **Refuted:** the audit's proposal to change Docker's `data-root` would **not** move the build cache. It
+  moves 0.62 GB. The 181.4 GB belongs to the **system containerd**, relocated via `root` in
+  `/etc/containerd/config.toml`.
+- **The prune is done and it was the whole rescue.** `/` went **86% → 53% used**, and Longhorn's default
+  disk went **`Schedulable=False (DiskPressure)` → `Schedulable=True`**. The live consequence is cleared.
+- **The largest single finding is not the cap.** 90.5 GB of the 157 GB was one layer re-downloading on
+  every build, and the root cause is a **one-line Dockerfile ordering defect** — `ARG VERSION` declared
+  above the module-download step. Fixing it removes the source; the cap only bounds the symptom.
+- **P6 (the SSD2 move) was NOT executed** — it stops at the operator, as specified. The pre-analysis
+  says the move is safe as measured but carries a real Longhorn risk, and the prune removed its urgency.
+
+---
+
+## 1. Confirmed baselines
+
+| Repo | `main` @ commit at spike start | Drift vs the task's stated baseline |
+|---|---|---|
+| `homelab-manifests` | `5188058d0e43` | none — matched |
+| `felhom.eu` | `0dbd954fec90` | none — matched |
+
+Both trees clean (`git status --porcelain` empty) and `HEAD == origin/main` before any commit.
+
+**Register:** highest allocated at baseline was **R-204**, re-confirmed in `OPEN-ITEMS.md`. **R-205**
+onward were free and are allocated here (R-205 … R-211).
+
+**One path correction:** `homelab-manifests` is **not** in `/mnt/5_hdd/felhom.eu/git/` with the other
+repos — it lives at **`/home/kisfenyo/git/homelab-manifests`**.
+
+---
+
+## 2. P0 — arrival state, expected vs measured
+
+Measured live before anything was changed. **Values that matched are recorded, not omitted.**
+
+| Fact | Expected (audit) | **Measured** | |
+|---|---|---|---|
+| Root fs size | 477,393,182,720 B | **477,393,182,720 B** | match |
+| Root fs used | 387.3 GB | **387,332,755,456 B** | match (+10.8 MB in 2 h) |
+| Root fs avail | 65.7 GB | **65,734,934,528 B** | match |
+| Root fs percent | 86% | **86%** | match |
+| Docker server version | 29.6.2 | **29.6.2** (build `3d80467`) | match |
+| Storage driver | overlayfs, containerd snapshotter | **`overlayfs`, `driver-type: io.containerd.snapshotter.v1`** | match |
+| Docker Root Dir | `/var/lib/docker`, ~0.6 GB | **`/var/lib/docker`, 618,541,056 B** | match |
+| containerd root | `/var/lib/containerd`, ~181.4 GB | **181,394,223,104 B** | match |
+| Build cache total / reclaimable | 157 GB / 147.8 GB | **157 GB / 147.8 GB**, 1446 records | match |
+| Images / containers | 345 images, 2 running | **345 unique** (350 tag lines), **2 running** | match |
+| `/etc/docker/daemon.json` | does not exist | **does not exist** (`/etc/docker/` exists, empty) | match |
+| `/etc/buildkit/` | does not exist | **does not exist** | match |
+| SSD2 device / mount / fs / free | ~240 GB, ~203 GB free | **`/dev/sde2`, `/mnt/ssd_2`, ext4, 235,148,750,848 B fs, 203,034,353,664 B free** | match |
+| SSD1 Longhorn | `Schedulable=False (DiskPressure)` | **`Schedulable=False`, `DiskPressure`, 18.85% avail** | match |
+| HDD2 Longhorn | also False | **`Schedulable=False`, `DiskPressure`, 24.91% avail** | match |
+| Housekeeping timer | active, last Sun 2026-08-02 | **active**, last **2026-08-02 05:35:10 CEST**, next 08-09, `ExecMainStatus=0` | match |
+
+**Prohibition honoured:** `node-housekeeping.sh` was **not run**, in any mode. Its state was read from
+systemd, from the textfile on disk, and from Prometheus. The textfile's mtime is still
+`2026-08-02 05:35:11` — the 8-week reclaim history is intact.
+
+**What the audit did not measure — the SSD2 Longhorn record** (`Crucial-SSD-240G`,
+`/mnt/ssd_2/data/longhorn`):
+
+| field | value |
+|---|---:|
+| `storageMaximum` | 235,148,750,848 |
+| `storageAvailable` | 214,958,080,000 (**91.41%**) |
+| `storageScheduled` | 139,586,437,120 |
+| `storageReserved` | **0** |
+| condition | **`Schedulable=True`** |
+| replicas | **12** |
+
+**Two things stand out and both matter for P6.** SSD2 is the **only** Longhorn disk reserving **zero**
+(HDD2 and HDD4 each reserve 500 GB). And `storageScheduled` (139.6 GB) exceeds actual usage
+(`df`: 20,094,939,136 B) by **6.9×** — Longhorn is heavily overcommitted on this disk.
+
+---
+
+## 3. P1 — which daemon owns the bytes
+
+> **The single configuration change that relocates the build cache is `root = "<path>"` in
+> `/etc/containerd/config.toml`, owned by `containerd.service` — the SYSTEM containerd
+> (PID 2698677, socket `/run/containerd/containerd.sock`).**
+
+**The audit's `data-root` proposal is refuted, and the split is measured:**
+
+| tree | size | moved by `data-root`? |
+|---|---:|---|
+| `/var/lib/docker` | **618,541,056 B** (volumes 321 MB, **buildkit metadata 282 MB**, containers 14 MB) | yes |
+| `/var/lib/containerd` | **181,394,223,104 B** (overlayfs snapshots **170,357,870,592 B**, content 10.98 GB) | **no** |
+
+`dockerd` runs as `/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock` with **no
+systemd drop-ins**. BuildKit keeps its *record database* under Docker Root Dir (0.28 GB) but its *layer
+bytes* in containerd's snapshotter — which is why `data-root` looks plausible and is wrong.
+
+In `/etc/containerd/config.toml` the `root` key is **present but commented out**, so it defaults to
+`/var/lib/containerd`. Uncommenting it is the whole change.
+
+**k3s is unaffected — confirmed, not assumed.** k3s runs a **separate** containerd instance with its own
+socket (`/run/k3s/containerd/containerd.sock`) and its own root
+(`/var/lib/rancher/k3s/agent/containerd`, **79,669,473,280 B**). Nothing in this spike touches it.
+
+---
+
+## 4. P2 — which GC key schema this release accepts
+
+**`dockerd --validate` exists on 29.6.2 and is not an oracle for this subtree.** Measured:
+
+| candidate | `--validate` | reality |
+|---|---|---|
+| `defaultKeepStorage` (old flat form) | `configuration OK` | — |
+| `policy` array | `configuration OK` | **honoured** |
+| `reservedSpace` / `maxUsedSpace` / `minFreeSpace` flat | `configuration OK` | **silently ignored** |
+| **`totallyBogusKey`** | **`configuration OK`** | proves the validator does no field checking |
+| `policy` with `"filter": [three values]` | **`configuration OK`** | **crashed the daemon on restart** |
+
+> **The real oracle is `docker buildx inspect`, which renders the *effective* policy.** That is an
+> assertion of effect; `--validate` and "no error in the log" are both assertions of absence, and both
+> were measured to lie here.
+
+**The flat form is silently ignored.** With `{"gc":{"enabled":true,"reservedSpace":"1GB","maxUsedSpace":"2GB","minFreeSpace":"10GB"}}`
+applied and the daemon restarted, `buildx inspect` still reported the built-in defaults, and
+`journalctl -u docker.service` carried **no warning, no deprecation notice and no error**.
+
+**The `filter` key takes ONE value per policy entry, not an array.** An array is fatal at startup:
+
+```
+error initializing buildkit: error creating buildkit instance: filters expect only one value
+```
+
+The daemon then entered a systemd restart loop. **This is also how the two `unless-stopped` dev
+containers ended up stopped** — and they did **not** come back when the daemon later started cleanly;
+they needed a manual `docker start`. A follow-up that templates this file must **restart and verify**,
+never validate-and-assume.
+
+**Documented statement on containerd-worker applicability:** none found in `dockerd --help` or the
+installed docs. Recorded as **unknown from documentation**, and settled empirically by P5.
+
+**The default policy is not "no policy".** Before any change, `buildx inspect` reported built-in rules
+with **Reserved 41.91 GiB / Max Used 331.6 GiB / Min Free 82.89 GiB**. The audit's "there is no BuildKit
+GC policy on this host" is *almost* right — there was one, sized so large it could never bind: the cache
+at 157 GB was less than half the 331.6 GiB ceiling.
+
+### The validated `daemon.json`, verbatim (final state, left in place deliberately)
+
+```json
+{
+  "builder": {
+    "gc": {
+      "enabled": true,
+      "policy": [
+        { "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" },
+        { "all": true, "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" }
+      ]
+    }
+  }
+}
+```
+
+Effective policy confirmed live (note the **GiB** rendering — `30GB` in JSON becomes a 30 GiB rule):
+
+```
+GC Policy rule#0:  All: false  Reserved Space: 10GiB  Max Used Space: 30GiB  Min Free Space: 20GiB
+GC Policy rule#1:  All: true   Reserved Space: 10GiB  Max Used Space: 30GiB  Min Free Space: 20GiB
+```
+
+**No deprecation warning was emitted** for any accepted form.
+
+---
+
+## 5. P3 — the prune: claimed, actual, settle
+
+Pre-checked: no build in progress, P0 and P7 captured first.
+
+| | value |
+|---|---:|
+| **Claimed by `docker builder prune -af`** | **156.9 GB** (1444 records deleted, 86 s wall-clock) |
+| **Actual filesystem reclaim** | **150,353,829,888 B (150.35 GB)** |
+| **Time to settle** | **synchronous — full reclaim visible at t+0** |
+
+Settle poll (t+0 already showed +150,354,477,056 B; two consecutive readings <50 MB apart at t+60s;
+the residual −0.4 MB and −0.3 MB steps are ordinary background writes, not lagging reclaim):
+
+| t | avail | step delta |
+|---|---:|---:|
+| 0 s | 215,947,522,048 | — |
+| 30 s | 215,947,141,120 | −380,928 |
+| 60 s | 215,946,874,880 | −266,240 |
+
+**The 6.5 GB gap between claimed and actual is not an error** — it is the layers still shared with
+retained images (`Shared: 8.6 GB` before the prune). **A weekly job must measure the filesystem, not
+trust the command's `Total:`.**
+
+**Unlike containerd's image GC, this returns bytes synchronously**, so the follow-up needs no
+`settle_imagefs` equivalent — but the settle poll is what *established* that, and it cost 60 seconds.
+
+| | before | after |
+|---|---:|---:|
+| `/` used | 387,474,644,992 B (**86%**) | 240,301,133,824 B (**53%**) |
+| `/` avail | 65,593,044,992 B | 215,946,874,880 B |
+| `/var/lib/containerd` | 181,537,714,176 B | 31,160,053,760 B |
+| build cache | 156.9 GB / 1443 records | **0 B / 0 records** |
+| images | 345 | **345** (untouched) |
+| dev containers | 2 healthy | **2 healthy** |
+
+### The live consequence, cleared
+
+Longhorn re-evaluated within 60 s:
+
+| disk | before | after |
+|---|---|---|
+| `default-disk-…` (SSD1, `/var/lib/longhorn/`) | `Schedulable=False (DiskPressure)`, 89,967,820,800 B, **18.85%** | **`Schedulable=True`**, 240,228,761,600 B, **50.32%** |
+| `HDD2` | `Schedulable=False (DiskPressure)`, 24.91% | **unchanged — still False, 24.91%** |
+
+**HDD2 is untouched by this work and remains unschedulable.** It is a separate item, stated here so it
+is not read as fixed by association.
+
+---
+
+## 6. P5 — proving the ceiling is enforced
+
+**Rollback stated before the first mutation** and never needed beyond the recovery below:
+`sudo rm /etc/docker/daemon.json && sudo systemctl restart docker` — the file did not exist, so removal
+is a complete revert.
+
+Test ceiling **2 GB** (below one build's working set, deliberately, so eviction is reachable).
+
+### Build-by-build totals
+
+| step | build cache total | reclaimable | records |
+|---|---:|---:|---:|
+| baseline (post-prune) | 0 B | 0 B | 0 |
+| controller 1 | 2.583 GB | 2.583 GB | 22 |
+| hub 1 | **3.703 GB** ← peak | 3.703 GB | 37 |
+| controller 2 | 3.525 GB | 3.525 GB | 37 |
+| hub 2 | 3.051 GB | 3.051 GB | 38 |
+| controller 3 | 2.817 GB | 2.817 GB | 36 |
+| hub 3 | **2.782 GB** | 2.782 GB | 38 |
+
+**The total stops rising and trends down** — the cap binds. Records plateau at 36–38.
+
+### The named eviction — the assertion that matters
+
+Set-differencing the record IDs, **4 records present at controller-build-1 were absent by
+controller-build-3**, including the exact class that accumulated 208 copies:
+
+| ID | size | created | description |
+|---|---:|---|---|
+| **`9ak8egnxrg6rkibic2zo44u1`** | **440 MB** | 07:28:34 | **`mount / from exec /bin/sh -c go mod download \|\| true`** |
+| `kkic530irq4em04kywzwrsjp` | 214.6 MB | 07:28:37 | `… go build -ldflags …` |
+| `yvityhta4wppwa8cxs525c3g` | 58.83 MB | 07:28:36 | `… go mod tidy` |
+| `mvttjfrimnt1ttt0ogry75ue` | 49.8 MB | 07:28:35 | `[builder 5/7] COPY . .` |
+
+Across the whole sequence: `h1→h3` evicted **17** records, `c2→c3` evicted **10**.
+
+### Tolerance, stated honestly
+
+Steady state settled at **2.78–2.82 GB against a 2 GiB (2.147 GB) rule — about 30% over**, with a peak
+of **3.70 GB (~72% over)** mid-sequence.
+
+**This overshoot is an artefact of the test, not a property of the cap, and should not be extrapolated to
+30 GB.** A single controller build produces ~2.6 GB of cache in one shot, so a 2 GB ceiling was set
+*below one build's working set* — GC cannot get under it without deleting the cache it just created. At
+30 GB the same working set is ~9% of the ceiling and GC has ample room.
+
+### GC is build-triggered, not timer-driven
+
+Six minutes idle, polled each minute: **total unchanged at 2.782 GB, records unchanged at 38.** The
+cache does not shrink on its own. A burst of builds can overshoot and *stay* overshot until the next
+build — which is the argument for keeping a weekly `docker builder prune` as a backstop (R-206), rather
+than relying on the cap alone.
+
+### Then the ruled 30 GB
+
+Applied, restarted, and confirmed by the effective policy (§4). Both dev containers came back healthy
+and **their data is intact — asserted positively, not by absence of complaint**:
+
+| observable | before | after |
+|---|---:|---:|
+| postgres `count(*) from pg_database` | 4 | **4** |
+| postgres `pg_database_size('jarr_dev')` | 175,135,767 | **175,135,767** |
+| postgres public tables | 31 | **31** |
+| redis `DBSIZE` | 2437 | **2437** |
+| docker volumes | 8 | **8, identical names** |
+
+---
+
+## 7. P7 — why the module-download layer re-runs
+
+**Root cause found and proven: `ARG VERSION` / `ARG GIT_COMMIT` are declared in the builder stage
+*above* the module-download step, so a new version value invalidates every `RUN` beneath them.**
+
+### The evidence chain
+
+**1. Every record was used exactly once.** All **208** `go mod download` records (206 at audit time, +2
+from this spike's builds) carried **`Usage count: 1`**, spanning 2026-07-06 → 2026-08-05, totalling
+91.36 GB at 438–440 MB each. Not one was ever reused.
+
+**2. Two builds through `build.sh`, byte-identical tree, both missed.**
+
+| step | build 1 | build 2 |
+|---|---|---|
+| `#8 WORKDIR /build` | CACHED | CACHED |
+| `#9 COPY go.mod ./` | **CACHED** | **CACHED** |
+| **`#10 RUN go mod download \|\| true`** | **executed** | **executed** |
+| `#11 COPY . .` | executed | executed |
+| `#12 RUN go mod tidy` | executed | executed |
+| `#13 RUN … go build …` | executed | executed |
+| `#14/#15/#16` (stage-1 `RUN`s) | CACHED | CACHED |
+
+`go.mod` was **byte-identical** (`d8ded3f0798c…`) in the repo and in the workspace, mtime unchanged
+(`2026-07-24 11:54:08`).
+
+**3. `build.sh`'s churn was ruled out by fingerprinting.** Content-hashing all 737 files across
+`rm -rf` + `cp -a` + host-side `go mod tidy`: **`tidy` changes nothing**, and the reproduced tree is
+**content-identical** to the live workspace. `go.mod`'s inode number was even identical across the churn.
+
+**4. A direct `docker build` on the untouched workspace cached *everything*** — including
+`RUN go mod download` — proving BuildKit's cache is healthy and the Dockerfile is cacheable.
+
+**5. The decisive isolation.** Same workspace, same command, only the build-arg value varied:
+
+| run | `--build-arg VERSION` | `COPY go.mod ./` | **`RUN go mod download`** |
+|---|---|---|---|
+| E1 | **new** (`spike-p7e`) | CACHED | **executed** |
+| E2 | **same** (`spike-p7e`) | CACHED | **CACHED** |
+
+**The mechanism, and it explains the `#9 CACHED / #10 executed` signature exactly:** a `COPY`'s cache key
+is content-based and carries no environment, so it hits; a `RUN`'s key includes the stage **environment**,
+which contains every `ARG` declared above it. Since every real build passes a fresh `VERSION` (and often
+a fresh `GIT_COMMIT`), the module-download layer is invalidated **on every build, without exception** —
+206 builds, 206 retained 440 MB snapshots, 90.5 GB.
+
+**Both Dockerfiles have it.** `felhom.eu/hub/Dockerfile` declares `ARG VERSION` / `ARG BUILD_TIME` above
+`COPY go.mod go.sum*` → `RUN go mod download`. **And because both produce byte-identical `buildx du`
+description strings, the 208 records are a COMBINED count** and must not be attributed to one project.
+
+**Fix shape — not applied here, filed as R-208:** move the `ARG` declarations down to just above the
+final `go build`. One line per Dockerfile. **This is worth more than the cap and the move combined:** the
+cap bounds the symptom at 30 GB; this removes the source.
+
+---
+
+## 8. P4 — the image store, enumerated (nothing deleted)
+
+345 unique images across 350 tag lines (5 double-tagged), **0 dangling**. Only two are held by running
+containers: `redis:7-alpine` and `postgres:16-alpine`. `docker image prune -a` would remove **343**.
+
+**Size the prize before spending a decision on it.** Per-tag sizes sum to 139.29 GB, but that
+double-counts shared layers. `docker system df` puts the **real dedup'd footprint at 31.02 GB, of which
+27.02 GB is reclaimable** — roughly a fifth of what the build-cache prune already returned.
+
+| GB (per-tag sum) | tags | repository |
+|---:|---:|---|
+| 118.44 | **207** | `gitea.dooplex.hu/admin/felhom-controller` |
+| 7.25 | **107** | `gitea.dooplex.hu/admin/felhom-hub` |
+| 4.34 | 5 | `gitea.dooplex.hu/admin/jarr` |
+| 1.69 | 1 | `ghcr.io/seanmorley15/adventurelog-backend` |
+| 0.94 | 1 | `ghcr.io/papra-hq/papra` |
+| 0.72 | 1 | `codewithcj/sparkyfitness_server` |
+| 0.72 | 1 | `postgis/postgis` |
+| 0.59 | 3 | `gitea.dooplex.hu/admin/sparkyfitness-export` |
+| (23 more, each ≤0.53 GB) | | |
+
+### Registry presence — which are recoverable
+
+Queried against the live Gitea registry v2 API:
+
+| repo | tags in registry | tags local | **local-only (NOT re-pullable)** |
+|---|---:|---:|---:|
+| `felhom-controller` | 76 | 207 | **131** |
+| `felhom-hub` | 45 | 107 | **62** |
+
+**The local-only tags are all old.** Controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`; hub
+`0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0`. **Everything from controller `0.136.0` and hub `0.58.0`
+upward is in the registry** and therefore recoverable by `docker pull`.
+
+**This is the list the operator still owes a ruling on (R-210).** CC's view: not worth doing for the
+space — ~27 GB against 199 GB now free. The only real benefit is dropping unrecoverable clutter.
+
+---
+
+## 9. P8 — what else writes to Docker
+
+**The bounded list of build triggers on the host Docker daemon is: interactive/manual runs only.**
+
+- **18 systemd timers** enumerated — none invokes docker. (`node-housekeeping`, `dooplex-backup`,
+  `apt-daily*`, `sysstat*`, `logrotate`, `fstrim`, `e2scrub_all`, `smart-fancontrol`, …)
+- **Root crontab: empty** (comments only). **User crontab: one entry**, `sync-hetzner-backups.sh`,
+  writes under `/mnt/5_hdd`.
+- **`/etc/cron.d`: 4 real files** — anacron, e2scrub_all, sysstat, zfsutils. None touches docker.
+
+**The hole the audit left open is now closed with positive evidence.** The audit concluded "no CI runner
+exists on this host" from systemd/process/crontab alone — but a Gitea Actions runner **does** exist as a
+**cluster pod** (R-168, 2026-08-02). It cannot reach the host daemon:
+
+- **162 pods swept across all namespaces: `0` mount any `docker`/`containerd` hostPath.**
+- `act-runner` (`gitea-system`, image `felhom-act-runner:0.1.0`) has one volume — a PVC at `/data`.
+  `privileged: false`, `allowPrivilegeEscalation: false`, **all capabilities dropped**, no docker socket.
+- Its Dockerfile states the absence is deliberate: *"DELIBERATELY NOT INSTALLED: a Go toolchain, Node, or
+  Docker … There is no container runtime in here on purpose."*
+- `gates.yml` runs exactly one command — `python3 scripts/<repo>_gates.py --fast` — plus a git fetch and
+  a failure-only Resend alarm. **No `docker build` anywhere.**
+
+**So the audit's ~7 builds/day attribution to interactive use is confirmed**, and now on a complete
+enumeration rather than a partial one.
+
+---
+
+## 10. P9 — the monitoring rule fix (shipped)
+
+**File:** `homelab-manifests/mon-system/prometheus-rules.yaml`, alert
+`RootFsPressureDespiteHousekeeping`. **Commit `6808a4b`.**
+
+**Change:** `and` → **`and on(instance)`**. A bare `and` requires identical label sets; the left vector
+carries `{device,fstype,instance,job,mountpoint}` and the right — a node_exporter *textfile* metric —
+carries only `{instance,job}`. Nothing ever matched.
+
+### Red-proof (required, and both numbers recorded)
+
+**Threshold substituted `0.25` → `0.5`, and the reason is stated rather than hidden:** P3's prune had
+already taken `/` to 44.6% available, so the shipped threshold's left side is now legitimately false. The
+defect is in **label matching**, not in the threshold, so the proof is run at a threshold where both
+sides are individually true — which is exactly the condition the original rule was meant to catch.
+
+| expression | `result_count` |
+|---|---:|
+| left side alone (`avail/size < 0.5`) | **1** |
+| right side alone (`time() − last_success < 1209600`) | **1** |
+| **as shipped — bare `and`** | **0** |
+| **fixed — `and on(instance)`** | **1** |
+
+For completeness, at the shipped 0.25 threshold **both** forms return 0 today — because the left side is
+false now, not because the fix failed.
+
+**Annotation rewritten.** It pointed at Longhorn replicas/snapshots (~10% of the growth). It now names
+`docker buildx du` first, records the 30 GB cap in `/etc/docker/daemon.json`, and notes that a reverted
+or unapplied cap looks exactly like this alert firing — so the alert text carries its own diagnosis.
+
+### Deployment — GitOps, no `kubectl set image`, no live ConfigMap edit
+
+ArgoCD app `monitoring` (auto-sync **off**): hard-refresh → `OutOfSync` at `6808a4be54a7` → deliberate
+sync → **`Synced` / `Healthy`**, operation phase **`Succeeded`**.
+
+**Verified in Prometheus, not assumed from ArgoCD:**
+
+```
+query : … < 0.25 and on (instance) time() - node_housekeeping_last_success_timestamp_seconds < 1209600
+state : inactive     health : ok     lastError : (none)
+```
+
+`inactive` is now the **correct** answer — `/` really is fine at 44.6% available.
+
+> **A near-miss worth recording: I invented a full commit SHA from ArgoCD's truncated display and the
+> sync failed with `upload-pack: not our ref`.** It failed loudly and cost one retry. It would not have
+> failed loudly had the fabricated SHA happened to exist.
+
+### The finding P9 turned up on the way — R-211
+
+**There is no config-reloader.** The `mon-system/prometheus` Deployment runs **one** container
+(`prom/prometheus:v3.12.0`) with no reloader sidecar. After the sync the updated rules file was present
+**inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus rules
+API **still served the old expression for 4+ minutes**, with no error anywhere. It took effect only after
+an explicit `POST /-/reload` (HTTP 200; `--web.enable-lifecycle` was already set).
+
+**The consequence is general:** every rule edit in this repo has silently not applied until something
+happened to restart the pod. `Synced/Healthy` was true and beside the point. Filed as **R-211**.
+
+---
+
+## 11. P6 — the SSD2 move: **EXECUTED 2026-08-05 on operator authorisation**
+
+> **Executed after the operator ruled "proceed", having read the pre-analysis below. The
+> `storageReserved` condition CC named was applied with it. The operator subsequently ruled
+> explicitly: DO NOT REBOOT DooPlex — so the reboot validation is DEFERRED, and what that leaves
+> unproven is stated in §11.9 rather than glossed.**
+
+### 11.0 Execution record
+
+**Blast radius, established before anything was stopped:** k3s runs a **separate** containerd
+(`/run/k3s/containerd/containerd.sock`), so **Gitea, the registry, the hub, PBS, Longhorn and all
+~160 pods stayed up throughout**. The only workloads affected were the two `jarr-*` dev containers.
+`ctr -n moby containers list` confirmed exactly those two on the system containerd.
+
+| # | step | result |
+|---|---|---|
+| 1 | stop `docker.service`, `docker.socket`, `containerd.service` | all `inactive` |
+| 2 | `mkdir /mnt/ssd_2/containerd` (mode 711, root:root) | ok |
+| 3 | `rsync -aHAX --numeric-ids` | **550,891 files, 33,276,370,041 B, 4 m 58 s** |
+| 4 | `mv /var/lib/containerd → /var/lib/containerd.pre-move-2026-08-05` | **moved aside, never deleted** |
+| 5 | `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` (backup kept) | one-line diff |
+| 6 | `RequiresMountsFor=/mnt/ssd_2` drop-ins on **both** `containerd.service` and `docker.service` | resolved into both units |
+| 7 | start + verify | see below |
+| 8 | `storageReserved = 80 GB` on SSD2 | applied |
+
+**`-X` is load-bearing** — overlayfs snapshots carry `trusted.overlay.*` xattrs, and losing them
+would silently corrupt layer stacking.
+
+### 11.1 Copy verified BEFORE the original was touched
+
+Four independent observables, all matching:
+
+| check | source | destination |
+|---|---:|---:|
+| filesystem entries | 550,891 | **550,891** |
+| `trusted.overlay` xattrs | 448 | **448** |
+| files with link count >1 (hardlinks) | 37,243 | **37,243** |
+| `io.containerd.metadata.v1.bolt/meta.db` sha256 | `c60ecfb4ee134255…` | **`c60ecfb4ee134255…`** |
+
+(Byte totals differ by +14 MB — ext4 block allocation, not content.)
+
+### 11.2 Nothing was lost
+
+| | pre-move | post-move |
+|---|---:|---:|
+| unique images | 345 | **345 — identical ID set** |
+| tag lines | 350 | **350 — identical set** |
+| docker volumes | 8 | **8 — identical set** |
+| build cache | 2.782 GB / 38 records | **2.782 GB / 38 records** |
+| GC ceiling | 30 GiB | **30 GiB, still in force** |
+| postgres databases | 4 | **4** |
+| postgres `jarr_dev` bytes | 175,135,767 | **175,135,767** |
+| postgres public tables | 31 | **31** |
+| redis `DBSIZE` | 2437 | **2437** |
+
+**End-to-end proof, not just inventory:** a real `./build.sh` run against the relocated store
+completed `rc=0`, produced an image, and grew the cache 2.782 → 4.012 GB. The scratch tag was removed
+and `:latest` restored to the anchor digest afterwards.
+
+### 11.3 The store is demonstrably on SSD2
+
+```
+containerd config dump → root = '/mnt/ssd_2/containerd'
+du /mnt/ssd_2/containerd → 35,500,183,552 B
+/var/lib/containerd     → does not exist
+```
+
+### 11.4 The guard — and a trap found while proving it
+
+**First attempt did NOT refuse, and that is a finding worth more than the test.** With
+`RequiresMountsFor=/mnt/absent-guard-test-2026-08-05` — a path with **no mount unit at all** —
+containerd **started normally**. `RequiresMountsFor` on a path systemd has no mount unit for is a
+**silent no-op**. So a typo in the guard path buys nothing and says nothing: the same
+*built-but-never-wired* shape this project has shipped repeatedly.
+
+**So the guard was verified positively, at the unit level:**
+
+```
+containerd.service  Requires=mnt-ssd_2.mount   After=mnt-ssd_2.mount
+docker.service      Requires=mnt-ssd_2.mount
+```
+
+**Then the refusal itself was proven with a genuinely absent DEVICE.** `/mnt/ssd_2` hosts **12 live
+Longhorn replicas** and must never be unmounted, and editing `fstab` on a production host risks
+emergency mode at boot — so a **temporary synthetic `.mount` unit** pointing at a nonexistent UUID
+was used instead, with a temporary `RequiresMountsFor` on the real `containerd.service`:
+
+```
+A dependency job for containerd.service failed.
+systemd: Dependency failed for containerd.service - containerd container runtime.
+systemd: containerd.service: Job containerd.service/start failed with result 'dependency'.
+containerd is-active: inactive
+```
+
+**containerd REFUSED to start rather than creating an empty store on the root disk** — exactly the
+required behaviour, and exactly the signature the ep0 runbook describes. All test artifacts were
+removed; only `10-ssd2-mount.conf` remains on each unit and the guard resolves to `/mnt/ssd_2` alone.
+
+### 11.5 Longhorn — the condition CC made the move conditional on
+
+`storageReserved` on SSD2 raised **0 → 80,000,000,000 B**, covering the containerd tree so Longhorn's
+scheduler can no longer oversubscribe into the space the store needs.
+
+| disk | available | reserved | `Schedulable` |
+|---|---:|---:|---|
+| **Crucial-SSD-240G (SSD2)** | 179,516,211,200 (**76.34%**) | **80,000,000,000** | **True** |
+| default-disk (SSD1) | 236,978,176,000 (49.64%) | 143,217,954,816 | **True** |
+| HDD4 | 1,793,798,963,200 (60.82%) | 536,870,912,000 | True |
+| HDD2 | 2,470,864,486,400 (24.91%) | 536,870,912,000 | **False** — pre-existing, untouched |
+
+SSD2 remains schedulable with the reserve applied. **Rollback: patch `storageReserved` back to 0.**
+
+### 11.6 A standing post-boot assertion was installed
+
+`felhom-store-postboot-check.service` (oneshot, `After=docker.service`, enabled) runs
+`/usr/local/sbin/felhom-store-postboot-check.sh` at **every** boot and logs to
+`/var/log/felhom-store-postboot-check.log`. It asserts, with **positive** observables:
+
+- `/mnt/ssd_2` is mounted;
+- containerd's effective `root` is `/mnt/ssd_2/containerd`;
+- **`/var/lib/containerd` does NOT exist** — i.e. no empty store was recreated on the root disk, which
+  is the precise trap the guard exists to prevent;
+- docker sees ≥100 images;
+- both dev containers are running.
+
+Dry-run at install time: **RESULT: PASS** on all six lines. It was proven working *before* being
+relied upon.
+
+### 11.7 Disk position after the move
+
+| | value |
+|---|---:|
+| `/` | 224 GB used, **199 GB free, 54%** |
+| `/mnt/ssd_2` | 52 GB used, **157 GB free, 25%** |
+| store on SSD2 | 35,500,183,552 B |
+| **rollback copy retained on `/`** | **34,299,498,496 B** |
+
+**`/` did not drop below 54% because the rollback copy is deliberately still there.** Reclaiming it
+returns ~34 GB — see §11.9.
+
+### 11.8 Rollback, complete and one sequence
+
+1. `sudo systemctl stop docker docker.socket containerd`
+2. `sudo mv /var/lib/containerd.pre-move-2026-08-05 /var/lib/containerd`
+3. `sudo cp /etc/containerd/config.toml.pre-move-2026-08-05 /etc/containerd/config.toml`
+4. `sudo rm -rf /etc/systemd/system/{containerd,docker}.service.d/10-ssd2-mount.conf`
+5. `sudo systemctl disable --now felhom-store-postboot-check.service && sudo rm /etc/systemd/system/felhom-store-postboot-check.service /usr/local/sbin/felhom-store-postboot-check.sh`
+6. `sudo kubectl patch nodes.longhorn.io -n longhorn-system dooplex --type merge -p '{"spec":{"disks":{"Crucial-SSD-240G":{"storageReserved":0}}}}'`
+7. `sudo systemctl daemon-reload && sudo systemctl start containerd docker`
+8. `sudo rm -rf /mnt/ssd_2/containerd`
+
+### 11.9 **NOT PROVEN: the reboot. Stated, not glossed.**
+
+**The operator ruled explicitly that DooPlex must not be rebooted, so the reboot validation did not
+happen.** Uptime is unbroken (7 weeks 6 days, since 2026-06-10) — verified, not assumed.
+
+**By this task's own standard — *"a move that has not survived a reboot has not been validated"* — the
+move is NOT fully validated,** and it would be dishonest to close it as if it were. Precisely:
+
+- **PROVEN (mechanism):** the guard is wired into both units (`Requires=` + `After=mnt-ssd_2.mount`),
+  and containerd **refuses to start** when a required mount's device is genuinely absent.
+- **NOT PROVEN (consequence):** that a real boot sequence mounts `/mnt/ssd_2` before containerd starts
+  *on this host, in the real ordering*. Mount-ordering reasoning is exactly the class this project has
+  been burned by, and the mechanism proof does not substitute for it. `CLAUDE.md` prefers the
+  consequence over the mechanism; here only the mechanism is available.
+
+**Two consequences follow, and both are deliberate:**
+
+1. **The rollback copy `/var/lib/containerd.pre-move-2026-08-05` (34.3 GB) STAYS** until a reboot has
+   validated the move. Deleting it now would trade a cheap 34 GB for the only cheap way back.
+2. **The next reboot — whenever it happens, planned or not — validates this automatically.** The
+   post-boot service writes `RESULT: PASS` or `RESULT: FAIL` to
+   `/var/log/felhom-store-postboot-check.log`. **Read that file after the next boot**; on `PASS`, the
+   rollback copy can be removed and ~34 GB returned to `/`.
+
+Tracked as **R-209a**.
+
+---
+
+## 11-bis. P6 — the pre-analysis the ruling was made on
+
+*(Retained as written, before execution, so the ruling can be audited against what it was given.)*
+
+**First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck
+SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4,
+CC's disagreement is stated rather than absorbed: **the cap alone resolved the problem the move was meant
+to solve, and the move carries a Longhorn risk the cap does not.**
+
+### Mechanism — three candidates
+
+| | mechanism | survives package upgrade | expressible in Ansible | visible reading the host cold |
+|---|---|---|---|---|
+| **(a)** | **`root = "/mnt/ssd_2/…"` in `/etc/containerd/config.toml`** | **yes** — dpkg conffile, local edits preserved | **yes** — a template, the file is already managed-shaped | **yes** — one line in the daemon's own config |
+| (b) | systemd mount unit / fstab bind onto `/var/lib/containerd` | yes | yes | **weak** — invisible unless you check `findmnt` |
+| (c) | symlink | fragile | yes | **weak**, and overlayfs mount semantics through a symlink are unproven here |
+
+**Recommendation: (a).** It is the only one that a person reading the host cold can *see*, and the key
+already exists in the file (commented out).
+
+### The boot-order trap, and its guard
+
+If containerd or Docker starts before `/mnt/ssd_2` is mounted, the daemon **creates an empty store on the
+root disk** and 345 images appear to have vanished.
+
+**Guard:** `RequiresMountsFor=/mnt/ssd_2` on **`containerd.service`** and **`docker.service`**. Neither
+has it today — `containerd.service` carries only `After=network.target dbus.service`.
+
+> **`RequiresMountsFor` RE-MOUNTS; it does not refuse.** systemd pulls the mount unit in and *starts* it,
+> so an unmounted-but-available device is silently re-mounted and everything looks fine. It refuses only
+> when the device is genuinely **unavailable**. **Testing it by unmounting proves nothing** — it must be
+> tested by pointing at an absent device. This is the ep0 datastore lesson
+> (`RUNBOOK-ep0-datastore-volume-2026-07-27.md`), earned on this project.
+
+**And a move that has not survived a reboot has not been validated.** Mount-ordering reasoning is exactly
+the class this project has been burned by.
+
+### The Longhorn consequence — the part that decides it
+
+`Crucial-SSD-240G`, `storageMaximum` 235,148,750,848, **25% floor = 58,787,187,712**:
+
+| scenario | resulting `storageAvailable` | as % | vs the 25% floor |
+|---|---:|---:|---:|
+| today, untouched | 214,958,080,000 | 91.41% | +66.4 pp |
+| move cache only (30 GB) | 184,958,080,000 | 78.66% | **+53.7 pp** |
+| move whole containerd tree today (34.3 GB) | 180,658,569,216 | 76.83% | +51.8 pp |
+| **move at steady state (~31.5 GB images + 30 GB cache ≈ 65 GB)** | 149,958,080,000 | **63.77%** | **+38.8 pp** |
+
+**As measured, the move is safe by a wide margin — nowhere near the 10-pp caution band.**
+
+**But the worst case is not comfortable, and it is not hypothetical enough to omit.** `storageScheduled`
+on SSD2 is **139,586,437,120** while actual usage is **20,094,939,136** — a 6.9× overcommit. If those 12
+volumes ever inflate to their scheduled size:
+
+| scenario | available | as % | vs floor |
+|---|---:|---:|---|
+| + cache only (30 GB) | 65,562,313,728 | 27.88% | +2.9 pp — inside the caution band |
+| **+ steady state (65 GB)** | 30,562,313,728 | **13.00%** | **−12.0 pp → `Schedulable=False`** |
+
+That is precisely the failure that just took SSD1 out of service.
+
+> **Recommendation: proceed only together with setting Longhorn's `storageReserved` on SSD2 to cover the
+> containerd tree (~80 GB).** SSD2 reserving **zero** while HDD2 and HDD4 each reserve 500 GB is an
+> anomaly in its own right, and it is what lets Longhorn oversubscribe into the space the cache needs.
+> Without that, CC recommends **not moving** — the cap has already done the job.
+
+### The sequence, prepared and not executed
+
+Each step with its rollback. **Do not run without the operator.**
+
+| # | step | rollback |
+|---|---|---|
+| 1 | `systemctl stop docker docker.socket containerd` | `systemctl start containerd docker` |
+| 2 | `mkdir -p /mnt/ssd_2/containerd` | `rmdir` |
+| 3 | `rsync -aHAX --numeric-ids /var/lib/containerd/ /mnt/ssd_2/containerd/` (small now — ~34 GB, and ~2.8 GB of it is cache) | delete the copy; the original is untouched |
+| 4 | `mv /var/lib/containerd /var/lib/containerd.pre-move` (**move aside, never delete**) | `mv` back |
+| 5 | set `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` | restore the commented line |
+| 6 | drop-in `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`; `systemctl daemon-reload` | delete the drop-ins |
+| 7 | `systemctl start containerd docker`; **verify `docker images` = 345, both dev containers up, `du` confirms the store's real path** | steps 5→4→1 in reverse |
+| 8 | set SSD2 `storageReserved` ≈ 80 GB in the Longhorn node CR | set back to 0 |
+| 9 | **`reboot`, then re-verify everything in step 7** | as above |
+| 10 | test the guard by pointing at an **absent** device — an unmount alone proves nothing | restore fstab |
+| 11 | only after a clean reboot + a full build: `rm -rf /var/lib/containerd.pre-move` | — none; do this last |
+
+---
+
+## 12. Teardown
+
+**Nothing was provisioned.** No machine, no VM, no LXC guest, no hub record, no customer, no Longhorn
+volume, no PVC. There is nothing to decommission.
+
+**Cleaned up:**
+
+- The six candidate `daemon.json` files written for P2 (in the session scratch dir, never in
+  `/etc/docker/`) — **removed**.
+- The scratch workspace fingerprint tree (`ws_sim`) — **removed**.
+- **Nine scratch image tags** created by the probe builds — `felhom-spike:{p7c,e1,e2}`,
+  `felhom-controller:{spike-p7a,spike-p7b,spike-p7d,evict-c1,evict-c2,evict-c3}`,
+  `felhom-hub:{evict-h1,evict-h2,evict-h3}` — **all removed**.
+- `felhom-controller:latest` **restored** to `sha256:bdcd4a4ae8e006f1…` (0.198.0), byte-matching the
+  anchor captured before the first build. Image count back to **345**, the P0 baseline.
+
+**Intentional state left in place, recorded here so it does not read as a leftover:**
+**`/etc/docker/daemon.json` exists**, with the content in §4. It did not exist before this session.
+Removing it plus a daemon restart is a complete revert.
+
+**Not cleaned up, deliberately:** the ~2.78 GB of build cache from the eviction sequence. It is the
+working cache under a live 30 GB policy, not scratch.
+
+---
+
+## 13. Final state
+
+| | at arrival | **now** |
+|---|---:|---:|
+| `/` used | 387,332,755,456 B (**86%**) | **240,301,133,824 B (54%)** |
+| `/` available | 65,734,934,528 B | **212,766,556,160 B** |
+| `/var/lib/containerd` | 181,394,223,104 B | **34,299,510,784 B** |
+| build cache | 157 GB / 1446 records | **2.782 GB / 38 records, capped at 30 GiB** |
+| images | 345 | **345** |
+| dev containers | 2 healthy | **2 healthy, data verified intact** |
+| SSD1 Longhorn | **`Schedulable=False (DiskPressure)`** | **`Schedulable=True`, 50.32%** |
+| `RootFsPressureDespiteHousekeeping` | **dead — could never fire** | **live, `health=ok`, red-proven** |
+| growth driver | unbounded, ~5.25 GB/day | **capped; source diagnosed (R-208)** |
+
+---
+
+## 14. Observations — noticed, deliberately not acted on
+
+1. **`build.sh` exists in two copies** — `felhom-controller/controller/build.sh` (in git) and
+   `/mnt/5_hdd/felhom.eu/build/felhom-controller/build.sh` (the operative one). They differ by **one
+   comment line**. The hub solved this with a symlink into the repo; the controller has not. Not fixed —
+   out of scope, but it is a drift channel.
+2. **`build.sh` tags `:latest` on every build**, including throwaway version tags. A spike build silently
+   moved `latest` off 0.198.0; it was restored from an anchor captured beforehand. A `--no-latest` flag,
+   or tagging `latest` only on `--push`, would remove the trap.
+3. **The default BuildKit GC policy is derived from disk size**, so a bigger root disk would have raised
+   the 331.6 GiB ceiling too. The cap needs to be explicit no matter how much disk is added.
+4. **`node-housekeeping.sh`'s own `avail_bytes()` uses `df … | tail -1`** — the truncating-instrument
+   shape this project has been bitten by twice. It is *correct* here (`df` on one path emits exactly one
+   data row), so it was left alone, but it is worth knowing it is there.
+5. **HDD2 remains `Schedulable=False (DiskPressure)`** at 24.91%, a hair under the 25% floor with 2.47 TB
+   free and its 500 GB reserve satisfied. Untouched by this work, and not fixed by it.
+6. **The `mon-system` Prometheus has `--web.enable-admin-api`** as well as `--web.enable-lifecycle`. The
+   admin API permits series deletion. Noted, not acted on.
+7. **Longhorn on SSD2 is overcommitted 6.9×** (139.6 GB scheduled vs 20.1 GB used). That is normal for
+   thin provisioning, but combined with `storageReserved = 0` it is what makes the P6 worst case reach
+   the floor.
+
+---
+
+## 15. Register
+
+Allocated this session: **R-205** (CLOSED — the monitoring rule, shipped), **R-206** (Ansible: cap +
+prune + narrowed Docker ban), **R-207** (the `DRY_RUN` guard), **R-208** (the `ARG`-ordering cache miss —
+the largest remaining win), **R-209** (WAITING-ON-OPERATOR — whether to move to SSD2 at all),
+**R-210** (WAITING-ON-OPERATOR — the image list), **R-211** (no Prometheus config-reloader).

documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md

@@ -0,0 +1,540 @@
+# SPIKE — what is actually one-shot, and what a guest rebuild really costs
+
+**Date:** 2026-08-04 · **Items:** R-193 (rebuild drops offsite), R-192 (the alert says the opposite of
+what it measured) · **Class:** spike — **no production code shipped for R-193 or R-192**
+**Baselines read:** `felhom.eu` @ `f456835bbcc4` (hub v0.91.1) · `felhom-controller` @ `0887fd67` ·
+`felhom-agent` @ `856a127`
+
+> **This document ends in ranked options and a STOP.** It establishes facts; it does not choose.
+> The operator's question is stated, unanswered, at the end.
+
+---
+
+> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8
+>
+> A follow-up recon traced the escrow→recovery chain link by link
+> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull
+> in opposite directions.**
+>
+> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in
+> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is
+> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back —
+> customer-present, not unattended — is a real design, and the operator has since ruled on its shape.
+> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a
+> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row.
+>
+> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked
+> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx`
+> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained
+> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The
+> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony
+> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is
+> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and
+> 07:20:08. → **R-198**.
+>
+> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but
+> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as
+> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is
+> not complete without R-198.
+>
+> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d)
+> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is
+> still the decisive one.
+
+---
+
+## 0. Why this exists
+
+Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
+session reports described the same mechanism in contradictory terms:
+
+- R-39's record: Re-issue *"rotates the restic password and makes the escrow stale"*.
+- R-193's own resolution note: the one-time password is *"only the transport credential used once to
+  install the box's own SSH key"*.
+
+Those two worlds imply very different fates for the fifteen existing snapshots. This spike settles it
+**from source first, live state second**, so a measurement can contradict the code rather than be
+interpreted by it.
+
+**The headline, before the detail:** *both prior claims are wrong in the way that matters.* Re-issue
+does **not** touch the restic repository password — but a **guest rebuild does**, silently, on every
+box, with nothing marking the escrow stale and nothing telling anyone. Both demo boxes lost repository
+continuity on 2026-08-03/04. The one that "recovered by luck" lost it too.
+
+---
+
+## 1. The instruments, and what each one can and cannot answer
+
+Stated up front so a wrong instrument is visible here rather than inside a conclusion (§8 rule 2).
+
+| # | Instrument | Exact query / read | Known limit |
+|---|---|---|---|
+| I1 | hub SQLite snapshot | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale at copy time (mtime 08:34 vs wal 10:45 CEST). Copying only the main file would have produced confident answers from a stale file — the exact R-3 shape. |
+| I1-check | freshness proof | `PRAGMA integrity_check` → `ok`; `MAX(host_reports.received_at)` = `2026-08-04 08:45:38`, `datetime('now')` = `08:49:46` | **Positive observable**: newest row 4 min old. Not "the query returned no error". |
+| I2 | `host_escrow.restic_pw_sha256` + `host_escrow_superseded.restic_pw_sha256` | see §2 | sha256 of a 256-bit random secret — non-reversible, safe to compare. Answers *"did the repo password change"* **without touching a box or a credential**. |
+| I3 | `reports.report_json → $.offsite` | grouped distinct-object query with first/last seen | The report echoes the box's own target state; absence of the key = no target configured. |
+| I4 | `one_time_secrets` | `SELECT customer_id, created_at, consumed_at` | **`customer_id` is the PRIMARY KEY** — one row per customer, last-write-wins. It **cannot** answer "how many credentials were delivered"; a Re-issue destroys the prior row. demo-hp's 2026-07-23 consume is no longer in this table. |
+| I5 | source | `hub/internal/offsite/*`, `hub/internal/api/*`, `controller/internal/{backup,offsiteapply}`, `agent/internal/escrow` | — |
+
+**Not used, deliberately:** no `ReissueCredentials`, no ceremony, no config change, no deletion, no
+write to any box. Nothing on either box or on the storage endpoint was altered by this session.
+
+---
+
+## Q2 — Does a rebuilt controller reuse the escrowed repository password, or mint a new one?
+
+### **RULING: it MINTS A NEW ONE. The escrowed password is never consulted on any automatic path.**
+
+**Method — source.** The apply-bridge's terminal step is
+`Enabler.ConfigureOffbox(...)` (`controller/internal/offsiteapply/offsiteapply.go:221,256`), which
+reaches `Manager.ApplyOffsiteTarget` → `WriteOffboxSecrets`
+(`controller/internal/backup/offbox.go:370`). Quoted verbatim, `offbox.go:392-401`:
+
+```go
+// Auto-generate the repo password once (0600), never log it.
+if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
+    pw, gerr := generateOffboxPassword()
+    ...
+    if werr := os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600); werr != nil {
+```
+
+`offboxPwPath()` is `<DataDir>/offbox/repo_password` (`offbox.go:363`). A rebuilt guest has a fresh
+data dir, so the file is absent, so **a fresh 256-bit password is minted**.
+
+**The only path that recovers the old one is `InjectOffboxPassword`** (`offbox.go:541`), whose sole
+caller in the entire repo is `offboxInjectPasswordHandler`
+(`controller/internal/web/offbox_handlers.go:189`) — a **web form where a human pastes the recovered
+password**. Verified by census: `grep -rn "InjectOffboxPassword" --include=*.go .` → three hits, one
+definition, one caller, one comment. **The apply-bridge never calls it.**
+
+**Method — live, and it is decisive without touching a box (I2).** The hub already stores the sha256
+of the escrowed repo password, so the question is answerable as a hash comparison:
+
+| host | superseded (pre-rebuild) | current (post-rebuild) | verdict |
+|---|---|---|---|
+| `demo-hp-bb76ea` | `8e03eddf9ff7…` created `2026-07-23T10:01:17Z` | `8a9e33aa4da6…` created `2026-08-04T07:15:36Z` | **CHANGED** |
+| `demo-felhom-8363b5` | `48741892f0ef…` created `2026-07-21T08:38:33Z` | `c60c8bc737a6…` created `2026-08-04T07:20:07Z` | **CHANGED** |
+
+**Both boxes minted a new repository password.** demo-hp's pre-rebuild escrow (`8e03…`, sealed eight
+minutes after its 2026-07-23 09:53:41 apply) is the key to its 15 snapshots. That key now exists
+**only inside a superseded, R-wrapped escrow blob** that the hub cannot open.
+
+### **This is bigger than R-193 as filed, and it hits the box R-193 called lucky.**
+
+R-193 records demo-felhom as having *"survived the SAME rebuild by luck"* — 76 seconds of downtime.
+Measured (I3), demo-felhom's pre-rebuild offsite object was:
+
+```
+last_run 2026-08-03T02:17:44Z · last_status ok · snapshot_count 36 · repo_size_bytes 1 136 685 919
+```
+
+**36 snapshots, 1.14 GB.** Since 2026-08-03 07:19:10 UTC it has reported `snapshot_count: 0,
+repo_size_bytes: 0` in every one of 109 reports, and its repo password hash changed.
+
+> **The luck recovered DELIVERY. It did not recover the REPOSITORY.** The staged secret restored the
+> transport in 76 seconds and the box then minted a brand-new repo password anyway. The contrast
+> R-193 draws between the two boxes is real for the *credential*, and **false for the data**: both
+> boxes lost repository continuity, one loudly and one silently. The silent one is worse.
+
+---
+
+## Q3 — What did the 04:15 run report?
+
+### **RULING: UNMEASURED — the decisive run has not happened yet. And the binary the question offers is the wrong one; source says the answer will be NEITHER 15 nor 1.**
+
+**Why it has not happened.** The scheduled off-box run fires at ~02:15 UTC (04:15 CEST). Measured
+from I3 — `last_run` values `2026-08-02T02:15:52Z`, `2026-08-03T02:16:39Z` on demo-hp;
+`2026-08-01T02:17:54Z`, `2026-08-02T02:17:01Z`, `2026-08-03T02:17:44Z` on demo-felhom. **Daily, both
+boxes.** Then:
+
+| box | why 2026-08-04 02:15 UTC produced nothing |
+|---|---|
+| demo-hp | no offsite target existed at all — the `offsite` key was **ABSENT** from every report between `2026-08-03 06:12:19` and `2026-08-04 07:12:01` UTC (111 reports). The re-issue landed at 07:15:47, five hours after the window. |
+| demo-felhom | the target existed but `escrow_state: pending` from `2026-08-03 07:19:10` to `2026-08-04 07:20:13`. `OffboxRunnable` requires configured **AND** escrowed (`offbox.go:564-569`), so the fork-4 gate blocked the run. The ceremony landed at 07:20:28. |
+
+Both boxes became runnable only this morning. **The decisive run is 2026-08-05 ~02:15 UTC on both.**
+Reported as unmeasured rather than inferred.
+
+### What source says will happen — and it is a third outcome
+
+Same sub-account, same repo path, new password. Measured coordinates (I3, `$.dr_recipe`):
+`u629488-sub3.your-storagebox.de:/home/felhom-repo` — **unchanged** across the whole incident, and
+`repoPath` is a compile-time constant (`hub/internal/offsite/offsite.go:85`). So the next run opens an
+**existing repo with the wrong key**, which the codebase already has a name for
+(`controller/internal/backup/offbox.go:67-93`):
+
+```go
+// ErrOffboxOrphaned is the sentinel returned when the offsite repo exists but is keyed under a
+// passphrase this controller no longer has (the reinstall shape) ...
+case strings.Contains(s, "wrong password or no key found"):
+    return "orphaned"
+```
+
+and `ensureOffboxRepo` (`offbox.go:663-679`) branches on claim state:
+
+```go
+if !m.settings.GetClaimed() {  // UNCLAIMED → auto move-aside + re-init
+...
+m.markOrphaned()
+return ErrOffboxOrphaned      // CLAIMED → skip the run, show the orphan card
+```
+
+Both boxes report `claimed: 1` (I3, `$.claimed`, latest report each). **So the predicted outcome is:
+the run REFUSES with `ErrOffboxOrphaned`, produces no snapshot at all, and waits for the customer to
+confirm a reset.** Not 15 (reattached) and not 1 (silently fresh) — a **third** outcome the question's
+framing did not contain. That is the good news half: the system is not going to quietly start a new
+history over the old one. It will stop and say so.
+
+**Consequence if the customer confirms the reset:** `resetOrphanedRepo` (`offbox.go:270-314`) moves the
+old repo aside and re-inits — *"move-aside, not deleted"*. The 40.9 MB + 1.14 GB of old ciphertext
+survives, unreadable without the superseded escrow, **and keeps consuming the 50 GB soft quota
+indefinitely.** Nothing prunes a moved-aside restic repo (S-24 covers ep0's PBS namespaces, a
+different tier).
+
+**To measure it tomorrow:** re-read `$.offsite.last_status` / `last_error` on both boxes after
+02:20 UTC, and `$.offsite.repo_state` for the orphan flag. **Predicted `ErrOffboxOrphaned`, not a
+snapshot count.** Whoever picks this up should record which of the three actually occurred — a
+prediction from source is not a measurement.
+
+---
+
+## Q1 — What is one-shot, and what is not?
+
+Three secrets, and they have almost nothing in common.
+
+| | **① storage-provider password** | **② the box's SFTP key** | **③ restic repository password** |
+|---|---|---|---|
+| **What it is** | the Hetzner sub-account / box password | ed25519 keypair for `sftp` transport | the **data** key — restic's repo encryption |
+| **Generated by** | **hub** — `genPassword()` (`offsite.go:481`) | **controller** — `KeyGen.Generate()` (`offsiteapply.go:234`) | **controller** — `generateOffboxPassword()`, 32 random bytes hex (`offbox.go:406`) |
+| **Stored where** | `one_time_secrets` (hub, plaintext, **one row per customer**) | `<DataDir>/offbox/ssh_key` 0600 (box only) | `<DataDir>/offbox/repo_password` 0600 (box) + inside the R-wrapped escrow blob (hub, opaque) |
+| **Delivered how** | `GET …/offsite/consume-password`, **served exactly once** then marked consumed (`api/offsite.go:9-31`) | never delivered — installed onto the provider by ssh-copy-id using ① | never delivered anywhere; it never leaves the box except into the escrow ceremony |
+| **Re-issuable?** | **YES** — `ReissueCredentials` resets it at the provider, any time, operator-initiated | **YES** — regenerated on every full apply; `dr_recipe.go:45` states it plainly: *"the SFTP access key is regenerated at DR"* | **NO automatic path.** Recoverable only by unsealing the escrow with the customer's recovery code and pasting it into `offboxInjectPasswordHandler` |
+| **What a guest rebuild does to it** | nothing (it lives on the hub / at the provider) | destroys it; a fresh one is minted and installed — **harmless** | **destroys it; a fresh one is minted — CATASTROPHIC for the existing repo** |
+| **Escrowed?** | no | no | **yes** — `IdentityBundle.ResticRepoPassword` |
+
+The agent's own source names the asymmetry, and it is the sentence the whole spike turns on
+(`felhom-agent/internal/escrow/identity.go:35-39`):
+
+> `ResticRepoPassword` … **It is the DATA key for the offsite tier — irreplaceable (unlike the SFTP
+> access key, which is regenerable at DR).**
+
+**The one-shot thing (①) is the recoverable one. The irreplaceable thing (③) is the one nothing
+re-stages.** R-193 was filed against ①. The damage is in ③.
+
+**Instrument caveat, worth carrying (I4):** `one_time_secrets` has `customer_id` as PRIMARY KEY, and
+`SaveOneTimeSecret` is last-write-wins **by design** (the R-39(a) guard comment in
+`monitor/offsite_delivery.go:26-30` depends on it). So the delivery ledger holds only the newest
+credential per customer: demo-hp's 2026-07-23 consume, on which R-192's whole diagnosis rests, was
+**overwritten** by the 2026-08-04 Re-issue and is no longer readable from that table.
+
+---
+
+## Q4 — Which claim about Re-issue is true?
+
+### **RULING: `ReissueCredentials` touches ONLY the provider credential (①). It does NOT touch the restic repository password. R-39's record is wrong — and the hub's own comments repeat the same wrong claim in three places.**
+
+**Method — source, quoted rather than summarised.** `ReissueCredentials`
+(`hub/internal/offsite/offsite.go:150-228`) does exactly four things:
+
+1. `genPassword()`;
+2. `ResetSubaccountPassword` / `ResetBoxPassword` + `WaitAction` — **the provider account password**;
+3. `Store.SaveOneTimeSecret(customerID, pw)` — stage ① for delivery;
+4. bookkeeping: `MarkEscrowStale` + `offsite_reissued` + `escrow_stale` events.
+
+There is **no reference to a restic password anywhere in the function**, and none is possible — the
+repo password is generated on the box and never leaves it except into the escrow blob. The hub does
+not hold it in any openable form (`api/handler.go`: *"The hub stores the bytes and NEVER decrypts them
+(it has no recovery code)"*).
+
+**But step 4 is justified by a claim that is false**, `offsite.go:198-201`:
+
+> `// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing`
+> `// key-escrow blob — which sealed the OLD password — is now STALE.`
+
+The escrow *is* marked stale; the stated reason for marking it is not something this function does.
+The same false premise appears twice more: `api/handler.go:1067-1069` (*"The re-issuer resets the
+restic repo password, which makes the OLD escrow blob stale"*) and `OPEN-ITEMS.md`'s R-193 row, which
+inherited it from R-39.
+
+> **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does
+> not provide** — and the first where the comment is not merely unenforced but factually describes a
+> different function. It survived because it reads as settled and because its *effect* (a stale escrow)
+> is real, so nobody checked the *cause*.
+
+**Consequence, and it is a live defect, not a documentation nit.** On the ordinary Re-issue shape —
+a consumed-but-failed install on a box that still has its `repo_password` file — the box re-applies,
+`WriteOffboxSecrets` finds the file present and **keeps it**, and the repo password is unchanged. The
+hub has nonetheless told the customer, in Hungarian, that their recovery escrow is stale and asked
+them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** → **R-196**.
+
+**Corroboration against what actually happened on demo-hp yesterday.** The escrow went `pending` after
+the Re-issue and the task asks: if the repo password was untouched, what explains that? **It is fully
+explained, and not by the Re-issue.** Two independent mechanisms, both hub-side-invisible:
+
+1. `ApplyOffsiteTarget` (`offbox.go:482-494`) carries `EscrowState` over from the *existing* target —
+   but the rebuilt guest had **no existing target**, so `cur` was nil, so the fallthrough
+   `if tgt.EscrowState != "escrowed" { tgt.EscrowState = "pending" }` fired.
+2. The box had minted a **new** repo password moments earlier (Q2), so `pending` was *correct* — the
+   new key genuinely was not escrowed.
+
+**The Re-issue's `escrow_stale` event fired at 07:11:51 for the wrong reason and happened to be true
+for a different one.** Coincidence, not correctness. The proof that these are independent is
+**demo-felhom**: it received **no Re-issue at all** (measured — its only `escrow_stale` /
+`offsite_reissued` events in the entire history are dated `2026-07-21 08:29:29`), its repo password
+changed anyway, and **nothing marked its escrow stale for thirteen hours.**
+
+> **The mechanism is exactly inverted.** `escrow_stale` is wired to the one path that does **not**
+> change the repo password, and absent from the path that **does**.
+
+---
+
+## Q5 — Why did one box recover itself and the other not?
+
+### **RULING: confirmed from the ledger — an unconsumed staged secret, thirteen days old. And the recovery was partial in a way the narrative missed.**
+
+**Method — I4, the `one_time_secrets` ledger, no values read:**
+
+| customer | `created_at` | `consumed_at` |
+|---|---|---|
+| `demo-felhom` | `2026-07-21 08:29:29` | `2026-08-03 07:17:58` |
+| `demo-hp` | `2026-08-04 07:11:51` | `2026-08-04 07:12:06` |
+
+demo-felhom's row was **created 2026-07-21 and sat UNCONSUMED for 13 days** — traceable to a Re-issue
+at that exact timestamp (its `offsite_reissued` + `escrow_stale` event pair, `2026-07-21 08:29:29`).
+When the rebuild dropped its target on 2026-08-03, the fresh controller consumed the spare at 07:17:58
+and was reporting `offsite` again by 07:19:10. **76 seconds, on a credential nobody had planned to
+leave there.** Confirmed from the ledger, not the narrative.
+
+**What would have happened without one:** exactly demo-hp — `consume` → **404** (`api/offsite.go:20-22`,
+`no unconsumed offsite password`), the apply-bridge returns
+`offsite-apply: consume one-time password: …` and retries forever, and the tier stays absent until an
+operator acts. demo-hp sat there **25 hours** (`2026-08-03 06:12:19` → `2026-08-04 07:12:01`).
+
+**The correction to the contrast (see Q2):** demo-felhom's spare restored *delivery* only. Its 36
+snapshots / 1.14 GB were orphaned by the same rebuild. **Neither box's data survived; one box's
+plumbing did.**
+
+*Not established:* demo-felhom's escrow generation before 2026-07-21. `host_escrow_superseded` holds
+only two rows in the whole database (ids 3 and 4, one per box, both superseded 2026-08-04), so whether
+the 2026-07-21 Re-issue's ceremony re-sealed an *unchanged* password — the predicted false-staleness
+shape — **cannot be shown from this data**. Stated as unknown rather than asserted.
+
+---
+
+## Q6 — Why is delivery one-shot at all?
+
+### **RULING: the design's reason is that ① is a LIVE PROVIDER PASSWORD, not a bootstrap token; and the honest answer to the security question is that an automatic restage weakens EXPOSURE WINDOW, not authentication.**
+
+**The design's own reasons, from source:**
+
+- `api/offsite.go:9-12` — *"serves the one-time transient offsite password to the controller **EXACTLY
+  ONCE** … The value is returned once then marked consumed — a second call 404s. NEVER logged."*
+- `offsite.go:145-149` — *"the **EXPLICIT** operator recovery for a consumed-password dead-end … It is
+  **NOT** implicit rotation: ProvisionOffsite never calls this."*
+- `offsiteapply.go:1-6` — the apply-bridge is *"idempotent (a descriptor hash marker prevents
+  **re-consuming a spent password**)"*, i.e. the design treats a wasted consume as a real hazard, which
+  R-71a's whole settle-gate exists to prevent.
+
+**The security half, stated rather than skipped.** The question is: *could a machine that is not the
+real one obtain a credential via an automatic restage?*
+
+**What stands there today is authentication, and a restage does not remove it.** The consume endpoint
+is gated by `checkAuthCustomer` (`api/offsite.go:14-18`) — **the customer's API key, the same
+credential as the config pull**. Anything that could consume a restaged password can already pull the
+customer's whole config. So an automatic restage **grants no new access to an unauthenticated party**.
+
+**What it genuinely weakens is the exposure window.** Today a live Hetzner sub-account password is
+fetchable only in the seconds after an operator deliberately staged one. Under an automatic restage it
+becomes fetchable **whenever the box reports no offsite target** — a condition an attacker holding the
+API key can arguably induce. The credential's blast radius at the provider is the customer's own
+sub-account (label-scoped; `ReissueCredentials` refuses unless the label lookup finds **exactly one**,
+`offsite.go:164,180`), so this is a widening of *when*, not of *what*.
+
+**And one further guard must survive any change** — R-39(a), `monitor/offsite_delivery.go:26-30`:
+`SaveOneTimeSecret` is last-write-wins, so restaging on top of an **unconsumed** secret clobbers a
+password a box may be about to consume. Any automatic restage must re-read the row immediately before
+acting and refuse unless it is still a **consumed** row. That guard is correct and is not what R-192 is
+about.
+
+---
+
+## Q7 — R-192's guard: exactly what it reads, and why it read wrong
+
+### **RULING: both halves confirmed, with numbers. The guard reads the OLDEST 500 reports since the consume, and for demo-hp all 500 predated the rebuild by six days.**
+
+**The query, quoted** (`hub/internal/store/store.go:987`):
+
+```sql
+SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500
+```
+
+`ORDER BY id` ascending with `LIMIT 500` = **the oldest 500**, not the newest.
+
+**Reproduced against the live data (I1)**, with demo-hp's real consume anchor `2026-07-23 09:53:41`
+(recovered from R-192's record — it is no longer in `one_time_secrets`, see I4):
+
+| measurement | value |
+|---|---|
+| `total` (what the guard sees) | **500** |
+| `withOffsite` (what the guard sees) | **500** |
+| oldest report in that window | `2026-07-23 09:53:47` |
+| newest report in that window | **`2026-07-28 11:17:40`** |
+| true `total` over the same window, no LIMIT | 1174 |
+| true `withOffsite` over the same window, no LIMIT | 1063 (⇒ 111 without — matching the 111 ABSENT reports in I3) |
+
+**The whole 500-report evidence set ends 2026-07-28 — six days before the 2026-08-03 rebuild.**
+
+**Defect (a) — the message is exactly inverted.** `maybeEmitStuck`
+(`monitor/offsite_delivery.go:~110`) interpolates `status.ReportsSinceConsume` into a string that
+hardcodes the phrase *"carry no offbox target"*, and never reads `OffsiteReportsSinceConsume`. The
+live e-mail, quoted in full from the events table:
+
+> *"Offsite delivery stuck: one-time password consumed 284h19m0s ago and **500 report(s) since carry
+> no offbox target** — the credential is likely burned (apply died between consume and persist).
+> Re-issue delivers a fresh one."*
+
+`OffsiteReportsSinceConsume` for that same call was **500**. Every one of them carried a target. The
+message states the precise negation of its own measurement, and prescribes a remedy for a failure mode
+that did not occur.
+
+**Defect (b) — the heal refuses silently, on stale evidence.** `maybeHeal` returns bare on
+`status.OffsiteReportsSinceConsume != 0` (it was 500), with **no log line**. `offsite_credential_restaged`
+has never fired for any customer — confirmed: zero rows of that event type in the database.
+
+**What the guard SHOULD discriminate on — the shape, not the code.** The guard's intent is right:
+*"there is offbox evidence, so this is a regressed apply, not a burned credential — the operator's
+call."* Its error is that **it asks the question of the wrong time period**. Offbox evidence from
+before a rebuild is not evidence that the credential still works. The discriminator must be
+**recency-bounded and rebuild-aware**: judge on evidence *after the newest `controller_started` /
+config-hash change*, or on the latest N reports — never on "everything since the consume". And a
+refusal must **name its reason in the log**, because "we chose not to heal" and "the heal never ran"
+must not look identical (the S-16 rule, one layer down).
+
+**Do not fix it in isolation.** Under Q2's finding, a successful auto-restage on demo-hp would have
+restored the transport and the box would *still* have minted a new repo password. **The heal cannot
+protect the data; it can only protect the plumbing.** Whatever shape it takes must say so.
+
+---
+
+## Q8 — What each candidate would cost
+
+**A fourth option exists and is the important one**, because (a) and (b) both address ① and Q2 shows
+the damage is in ③.
+
+### (a) The hub restages automatically when a re-enrolled box reports no offsite
+
+**A version of this ALREADY EXISTS and is wired to the wrong event.** `reissueOnReenroll`
+(`hub/internal/api/handler.go:1051-1084`), leg **F3**:
+
+> *"F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
+> one-time password only ever reached the OLD controller)."*
+
+It is called from `handleHostEnroll` — but **after** a mint-once-reuse short-circuit
+(`handler.go:1004-1016`): `if existing != nil { …return… }`. **A guest rebuild leaves the `hosts` row
+intact** (measured: `demo-hp-bb76ea` is the same host_id throughout the incident; only the agent leaf
+re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the existing credential and
+**F3 is never reached.**
+
+- **Changes:** hub only. Either lower the trigger from host-enrollment to a controller-level rebuild
+  signal (the `config_hash` change + `controller_started` pair the hub already receives and already
+  logs), or fix R-192's guard so the existing R-71c self-heal can act.
+- **Weakens:** the exposure window in Q6. Needs the R-39(a) unconsumed-secret guard kept intact.
+- **Does NOT solve:** ③. The box still mints a new repo password and still orphans the repo. **This
+  option restores the plumbing to a box whose data key is already gone.**
+- **Cost:** small. **Value against the actual harm: near zero.**
+
+### (b) The credential becomes recoverable from escrow at re-bootstrap
+
+> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.**
+> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a
+> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape.
+> The recon also found that the chain this option would use is clientless at three links and lossy at
+> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4).
+
+- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
+  **R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
+  (`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
+  customer's recovery code.** Any design that says "recover it at re-bootstrap" is describing a
+  customer-present ceremony, i.e. what the existing manual `offboxInjectPasswordHandler` already is.
+- **Does NOT solve:** an unattended rebuild, which is the whole scenario.
+- **Cost: not implementable as specified.**
+
+### (c) — NOT PREVIOUSLY NAMED — the agent retains the repo password across a guest rebuild
+
+The agent lives **on the Proxmox host** and survives a guest rebuild. It already receives the repo
+password today: the controller pushes it over the pinned local API
+(`POST /escrow/stage-secret` → `agent/internal/localapi/escrow_stage.go`), and the agent writes it to
+the fixed 0600 path `/var/lib/felhom-agent/escrow-stage/restic_repo_password`
+(`agent/internal/escrow/identity.go:44-46`) **transiently — wiped by the ceremony**
+(`WipeStagedResticPassword`). The seam, the transport, the pinning and the file are all already built.
+
+- **Changes:** agent (retain rather than wipe; serve back over the same pinned local API) + controller
+  (on a fresh data dir, ask the agent before minting — the `InjectOffboxPassword` seam already exists
+  and is exactly the right shape).
+- **Weakens:** the escrow's *zero-knowledge* posture in one specific way — the irreplaceable data key
+  would now sit at rest on the Proxmox host, which the operator has root on. **Measured against D6,
+  that is not a new exposure in principle** (D6 already states plainly that *"the operator cannot read
+  customer data"* was never the security property, and the operator holds root on every box) — but it
+  IS a new copy, and copies are the thing R-133 is already open about for the hub DB. **This is a real
+  trade and it is the operator's to make.**
+- **Solves:** exactly the failure that occurred, unattended, on every rebuild, without any hub
+  involvement and without any credential leaving the premises.
+- **Cost:** medium. Two repos, one existing seam each.
+
+### (d) Do neither — keep the manual Re-issue, and fix what LIES
+
+- **Changes:** R-192's message + refusal log; correct the three false comments about what Re-issue
+  rotates; and — the part that actually matters — **make a repo-password change VISIBLE**. The hub
+  already receives `restic_pw_sha256` on every escrow upload and already stores it. Comparing the new
+  hash against the superseded one is a two-line verdict, and it is the signal that was missing on
+  demo-felhom for thirteen hours → **R-197**. (Note the shape: **this spike answered its own hardest
+  question with a comparison the hub could be making automatically and is not.** Both values were
+  already in the database; nothing read them.)
+- **Weakens:** nothing.
+- **Does NOT solve:** the data loss. Every rebuild still orphans the repo; the operator simply finds
+  out the same day instead of never.
+- **Cost:** small.
+
+### Ranked, with a recommendation
+
+| rank | option | why |
+|---|---|---|
+| **1** | **(d) now — the honesty pass** | It is cheap, it weakens nothing, and **it is the only option that helps the incident already in flight.** The alarm that lies is worse than no alarm; the repo-password-changed detector is the one signal whose absence let demo-felhom lose 1.14 GB of history silently. Ship this regardless of what is decided about the rest. |
+| **2** | **(c) — agent-retained repo password** | The only candidate that addresses ③, which is where the harm is. Every seam it needs exists. It costs one deliberate trade-off the operator must actually make. |
+| **3** | **(a) — auto-restage** | Worth doing **after** (c), never instead of it. On its own it restores the transport to a box whose data key is already gone, and would have made both boxes look healthy on 2026-08-04 while the snapshots were orphaned — a *strictly worse* outcome than the current loud failure. |
+| **4** | **(b) — recover from escrow at re-bootstrap** | Not implementable as stated; the escrow is zero-knowledge by design. |
+
+**Recommendation: ship (d), then decide (c).** And do not ship (a) first — it would have hidden this.
+
+---
+
+## What could not be established
+
+Named rather than guessed (§8 rule 4).
+
+1. **The 2026-08-05 02:15 UTC run's actual outcome** (Q3). Predicted `ErrOffboxOrphaned` from source +
+   claim state; **not measured**.
+2. **Whether the orphaned ciphertext still exists at the provider.** 40.9 MB (demo-hp) + 1.14 GB
+   (demo-felhom) are *presumed* present at `/home/felhom-repo` — reading the endpoint needs the SFTP
+   credential, which this session did not touch. The Hetzner API has no directory-listing surface.
+3. **Whether demo-felhom's 2026-07-21 Re-issue re-sealed an unchanged password** (the predicted
+   false-staleness shape, Q4/Q5). Only two `host_escrow_superseded` rows exist in the whole database;
+   the generation before 2026-07-21 is not retained.
+4. **What removed demo-hp's offsite target at the rebuild** is fully explained (fresh data dir), but
+   **why the guest was rebuilt on both boxes on consecutive mornings** is outside this spike.
+
+---
+
+## The operator's question — stated, and NOT answered here
+
+> **The irreplaceable secret in the offsite tier is the restic repository password. It is generated on
+> the box, it is destroyed by a guest rebuild, and the only copy that survives is sealed under the
+> customer's recovery code — which nothing but a human can open. Every machine is going to be
+> reinstalled.**
+>
+> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, where
+> you already have root — or do you want it to stay sealed under the customer's recovery code and
+> accept that every rebuild starts a new repository and orphans the old one, provided the system says
+> so loudly on the day it happens?**
+
+Option (c) is the first. Option (d) alone is the second. **This spike does not choose.**

documentation/backlog/OPEN-ITEMS.md

@@ -7,8 +7,134 @@ a non-overwritten `REPORT-<topic>.md` sibling instead (`CLAUDE.md:82-87`), of wh
 
 State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row has an owner.
 
+## Operator rulings — 2026-08-04
+
+Recorded here because a ruling that lives only in a conversation binds nobody (the R-96 standing rule).
+
+1. **Run the recovery drill, after R-198.** R-198 shipped in hub v0.93.0; **the drill is the next
+   session.** Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10 — demo-hp, ~3–4 h, a recovery
+   code created and KEPT, a sentinel file, wipe, reinstall, recover, and **pass = a byte-identical
+   sha256, not "the repository opened"**. → R-201
+2. **Delete the orphaned ciphertext** (~1.2 GB across the two demo boxes, in set-aside restic stores
+   nothing prunes). **STILL OWED — not done in v0.93.0.** It is a destructive act on a protected
+   endpoint and belongs to a session that is scoped for it, not to a release that ships a schema
+   change. → R-193
+3. **Accept the risk on R-193 candidate (c)** — no repository password retained on the Proxmox host.
+   **This is what makes R-198 load-bearing rather than tidy:** with no host-retained copy, the
+   customer-present recovery path is the ONLY way back from a rebuild, and that path runs entirely
+   through the retained identity blob. → R-193, R-199, R-200, R-201
+
+~~**Still open and untouched by v0.93.0:** R-199, R-200, R-201.~~ **UPDATED 2026-08-04 (evening), after
+hub v0.94.0 + agent v0.125.0 + controller v0.195.0:**
+
+- **R-199 — CLOSED, proven on hardware.** Chain links 6–8 are assembled and walked. The offsite
+  repository password came back out of the sealed bundle **byte-identical** to the one on disk
+  (`c60c8bc737a6…` from three independent sources: the box's file, the recovered bundle, and the hash
+  the hub already stored).
+- **R-200 — the plumbing half shipped**, the customer-facing form did not, deliberately.
+- **R-201 — PASSED 2026-08-04 (night run).** A customer's file survived a machine rebuild and came
+  back byte-identical, through the customer's own restore flow. **It passed only because a person was
+  there:** four manual interventions stood between the recovered key and the restored file, none of
+  them in any design document → R-204.
+- **R-202 — untouched.** The orphan card still promises recoverability unconditionally.
+- **The orphaned ciphertext deletion (~1.2 GB) is STILL OWED** — ruling 2, above.
+
+**UPDATED 2026-08-05, after controller v0.198.0 + hub v0.95.0 (R-204):**
+
+- **R-204 items 1–3 — CLOSED.** The reset code works without a restart; a re-issue no longer marks a
+  healthy escrow stale (→ **R-196 CLOSED**); a unit restore states what it did NOT restore.
+- **R-204 item 4 — OPEN and unstarted:** a rebuilt box cannot obtain an off-site credential unaided.
+  It needs an operator ruling on the one-shot credential design → **R-193**.
+- **Still open and untouched by this session, stated so nothing is presumed closed by association:**
+  **R-202** (the orphan card's unconditional promise), **the ~1.2 GB orphaned-ciphertext deletion**
+  (ruling 2 — still owed, still needs its own scoped session), and **R-198's retention, which remains
+  UNIT-PROVEN ONLY** — nothing has superseded a key in production, and proving it needs a SECOND
+  deliberate wipe. That retention drill is the next item, and it is not this session's.
+
+v0.93.0 made the key survive. v0.94.0/v0.125.0/v0.195.0 make it come back. v0.197.0 got the file into
+the snapshot and the drill got it out again. **v0.198.0/v0.95.0 remove three of the four crutches the
+drill needed — the fourth is R-193, and until it goes the recovery is still operator-assisted.**
+
+## CAMPAIGN 11 — the recovery journey, 2026-08-05
+
+**The whole journey was walked end to end for the first time, on a throwaway appliance built from the
+published ISO. The data came back byte-identical; the journey did not exist.** **Ten findings from
+Phases 1 and 3, R-214 … R-223** (seven fixed in controller **v0.201.0** + hub **v0.97.0/0.97.1**;
+**three deliberately still open**, each blocking a real flow), **plus five from Phase 2's injected
+faults, R-224 … R-228.** Evidence: `tests/campaign11-evidence-2026-08-05/journal.md` (Phases 0/1/3)
+and `journal-phase24.md` (Phases 2/4). Campaign document:
+`audits/CAMPAIGN-11-recovery-journey-2026-08-05.md`.
+
+> **Phase 2's verdict in one line.** The cryptography, the retention and the transport all work and
+> are now proven live. **What fails is being told the truth:** a mistyped code, a hub outage, a
+> stopped agent and a correct code for a retained earlier package all produce **one** message, and
+> three of the four are wrong.
+
+| ID | What | State |
+|---|---|---|
+| **R-216** | **A correct recovery code was reported to the customer as wrong.** The unseal needs agent v0.125.0; on an older agent the route 404s and the unlock was attempted anyway, producing *„…nem fogadtuk el. Ellenőrizd, hogy mind a tíz szót pontosan…"* in 0.134 s. **The default state** — the Day-0 manifest vouches 0.120.0, and a reinstall actively DOWNGRADES a hand-fixed box back to it. The hub's own guard could not catch it: `ResolveManagedFloor` compared against the GOLDEN's MinAgent while serving a FLOOR that pointed elsewhere (the 8th entry in `CLAUDE.md`'s comment-vs-code table, and the first where the false invariant was a guard) | **SHIPPED** (controller v0.201.0 + hub v0.97.0/0.97.1) — **but see R-223**: the feature does not work on a NEW box until the manifest vouches agent 0.125.0. Until then such a box is correctly HELD, not lied to |
+| **R-218** | **Succeeding at recovery stopped the box asking for what it still needed.** `needsOffsiteCredential` short-circuited on a repository password existing — and installing one is the recovery screen's whole job. 32 s after the hub re-staged the credential, the customer's success switched the mechanism off; the hub held an unconsumed credential the box had no reason to collect, and nothing ever asked again | **SHIPPED** (controller v0.201.0) — the declaration now stops when the TIER WORKS, not when a key exists |
+| **R-219** | **The listing the screen promises could never render on the shape it exists for.** Listing needs a target; a target cannot exist without a repository password; shape (a) is defined by having none. And placing the key flipped the offer false, so the unlock response was the only chance and was guaranteed not to contain it | **SHIPPED** (controller v0.201.0) — the unlock now places the key, brings the tier up, then lists |
+| **R-217** | **An unreadable store reported as "opened, with unattributable content".** The failure path passed `backup.OffsiteInventory{}`, whose `Empty=false` the template read as `InvUntagged`. The field built to prevent exactly this names the hazard in its own doc comment | **SHIPPED** (controller v0.201.0) — opened / empty / unreadable are three distinguishable states |
+| **R-222** | **Reaching for a RETAINED earlier package read as a wrong code.** The engine is right (it fails closed against the current package); the message was not. Proven live: the correct code for the orphaned history got *„check your ten words"* | **SHIPPED** (controller v0.201.0 + hub v0.97.0) — the ACK carries `superseded_present`/`superseded_at` and the screen names the situation. **It states what the hub knows and promises nothing** — the read path is still unbuilt (R-199's inventory) |
+| **R-215** | **`GET /recovery` rendered the recovery story on a box that never had off-site backups.** The predicate was right and the page never asked it; the POST sibling and the backups-area template both gated the same sentence correctly | **SHIPPED** (controller v0.201.0) |
+| **R-214** | **The physical console never stops asking to be paired.** Half an hour after `Day-0 provision SUCCESS`, with the host ONLINE, the console still showed the pairing banner and a stale code — on a screen whose own text promises *„Ez a képernyő magától frissül"*. Census: exactly two `/dev/console` writers in the whole day-0 path, both in the pairing loop; `felhom-host-install.sh` writes to the console not at all | **OPEN — NOT FIXED** |
+| **R-220** | **After a rebuild the customer's drives cannot be re-enrolled, and the refusal names an impossible action.** The deploy refuses (*„Válasszon a listából csatlakoztatott meghajtót"*) and the list is empty: `claim.go:84` treats a device mounted outside `/mnt/felhom-drives` as claimed, and the raw `/mnt/<name>` mount that enrolment itself creates survives the guest rebuild while the controller's registry does not. **Red-proved**: unmounting only the raw mounts flipped `attach: []` → both drives. **This is the state Campaign 10 reached by hand and recorded as its own harness error; the product's rebuild path now arrives there.** Breaches I3 | **OPEN — NOT FIXED** |
+| **R-221** | **A rebuilt box cannot run the escrow ceremony at all.** The preflight refuses on `escrow.pbs_storage_id`, which the pbsdr bridge seeds into `agent.json` only via `finishConverged`. The convergence marker lives on the HOST and survives a guest rebuild; `agent.json` is rewritten by the installer. Unchanged descriptor → same hash → early return → the seed never runs into a config that no longer has it. **Red-proved**: moving only the marker aside seeded it instantly (`grep -c escrow`: 0 → 1). A real blocker for re-escrow, which is exactly what a rebuilt box must do | **OPEN — NOT FIXED** |
+| **R-223** | **The Day-0 manifest vouched agent 0.120.0 while the recovery feature needs 0.125.0 — and a reinstall DOWNGRADES a box that was fixed by hand.** Verbatim from the second reinstall: `agent (existing): felhom-agent 0.125.0` → `manifest: agent v0.120.0` → `installed /usr/local/bin/felhom-agent (felhom-agent 0.120.0)`. So every rebuild re-broke the recovery path — the one event that makes the feature necessary. **⚠ AND IT WAS NOT A DROPDOWN.** The first vouch attempt was REFUSED by R-120's gate (`configs.go:1162`): *"golden 0.192.0 is older than the newest controller the fleet reports (0.201.0)"*. **The artifacts form saves as a unit, so the agent could not be vouched while the golden was stale — and the golden had been stale since controller 0.193.0, meaning the Day-0 manifest had been effectively UNVOUCHABLE for days and nobody had cause to notice.** The real remedy was a golden rebake | **CLOSED 2026-08-05.** Golden **0.201.0** baked in the drill VM (658 165 766 B, sha `e730d7cab343eb35…f007654`, **round-trip verified from Gitea**), then manifest set in one save: `agent=0.125.0 golden=0.201.0 min_agent=0.125.0`. A fresh install now lands on current agent AND current controller |
+
+### Phase 2 — the injected faults, 2026-08-05/06 (unattended)
+
+Eleven faults, each judged on **the message, not the outcome**, each with a positive control proving
+the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/journal-phase24.md`.
+
+| ID | What | State |
+|---|---|---|
+| **R-224** | **Every non-code failure on the unlock path is reported to the customer as a statement about their code.** Two faults, one defect. **F3** — the hub REJECTed (control: `302` → `exit 7`) and the **correct current** code returned M4 (*„Ez a kód nem nyitja meg azt a csomagot…"*) in **0.0556 s**. **F4** — `felhom-agent` stopped (control: `:8443` gone) and the same correct code returned M4 in **0.0299 s**. Neither elapsed time is compatible with a real unseal (~1.0 s of scrypt; F1's wrong-code attempts took 1.194/1.004/1.014 s), so **no unseal was attempted in either case** — the failure was upstream and was attributed to the code anyway. **The discriminator exists and is thrown away**: the agent's own `err` reads `fetching the sealed bundle: hub: transport error: … no route to host` (F3) versus `the recovery code did not unwrap the identity escrow` (F1), but both return **HTTP 400** under one merged sentence (*"…or the bundle could not be fetched"*), and the controller's failure path has **no branch for "could not ask / could not reach"**. **⚠ The vocabulary already exists and is wired to the wrong condition** — „A gép házon belüli kapcsolata most nem elérhető" fires only when `s.agentClient()` fails to *construct*, never when the RPC fails. **And R-216's gate cannot catch it**: the box's own ring reads `recovery capability gate: offsite_key_recovery=yes (source=version)` — the gate discriminates the agent's **age**, not its **reachability**, so a dead agent of the right version sails through the guard whose own comment says *"An attempt that cannot succeed must never be made, because its failure is attributed to the code."* **This is Phase 1's headline finding relocated from the version channel to the transport.** Breaches **I6** | **OPEN — NOT FIXED.** Fix shape: classify `rerr` (transport/fetch vs unseal) at the agent's HTTP boundary and give the controller a reach-failure branch. Do NOT widen the capability gate into a liveness probe on the unlock path — a probe that must succeed before an unlock adds a second thing that can fail |
+| **R-225** | **The remote store reports `0 pillanatkép · 0 / 50 GB` when the box cannot read it — directly above a card stating the store holds backups.** `/backups/remote` renders both at once. **Ground truth, measured over SFTP with the box's own credential (read-only, no decryption):** `/home/felhom-repo/snapshots` holds **`f3d9cd67…`** — the Phase 0 snapshot with all three sentinels, the customer's only surviving copy — and `du -s` returns **12 535 KB**, matching the journal's `repo_size_bytes 12 611 522`. **Mechanism, from the box's own state:** after the rebuild the `offbox` block carries **no `snapshot_count` and no `repo_size_bytes` key at all** — the values are *unknown* and unknown renders as the zero value. **This is R-217's defect class in a second location**, and `OffsiteInventory.Empty` exists precisely because *"len(Apps)==0 is also what a failed read looks like"*. **I5 checked and NOT breached** — the hub's `0.0 GB` is 12.5 MB rounded, and the pool total is consistent; only the box's snapshot **count**, an integer, is false. Breaches **I6** | **OPEN — NOT FIXED.** A count the box has never successfully taken must render as unknown („—"), never as `0` |
+| **R-226** | **M1 — the only message that tells a customer to check their typing — is unreachable on any box that has re-escrowed.** The failure path tests M4's condition *before* M1's (`if present, at := s.recoverySuperseded(); present { …M4…; return }`), so on every box the hub keeps an earlier package for, a **genuinely mistyped code** produces M4. **Measured**: F1's three wrong-code attempts each returned M4 after a real ~1 s unseal. M4 is hedged (*„**Ha** egy korábbi kódot adtál meg"*) and states two true facts, but it offers no hint to re-check the ten words and routes the customer to support about *older* backups. **The population that has re-escrowed is exactly the population that has just been handed a new recovery code and is most likely to be typing one.** R-222's fix removed one conflation and introduced another on the same branch | **OPEN — NOT FIXED.** Fix shape: M4 should name both possibilities (a mistype OR an earlier code), since from the engine's side they genuinely are indistinguishable — which is the honest thing to say |
+| **R-228** | **After „I do not want the old data", the set-aside history becomes invisible — the box records where it is and shows it to nobody.** The move-aside itself is **correct and was verified byte-for-byte**: `/home/felhom-repo` → `/home/felhom-repo.orphaned-20260805` with its mtime, its `du -s` (**12 535 KB**) and snapshot **`f3d9cd67`** all unchanged, and a fresh empty repo initialised beside it. **Nothing was deleted.** But `settings.json` then carries `"orphaned_renamed_to": "/home/felhom-repo.orphaned-20260805"` and a census returns **zero** references to `OrphanedRenamedTo` in any template or web handler — the field is written and read by nobody. `/backups/remote` after the set-aside contains no occurrence of the path, „félretéve", „régi előzmény" or any equivalent (instrument controls: `felhom-repo` → 2, „letétbe helyezve" → 1). **12.5 MB of the customer's deliberately retained data sits at a path the box knows and never shows**; its only mention is a flash message on the redirect, gone on the next click. Meanwhile `GET /recovery` → 302 and `POST /recovery/unlock` → 302 with no message, so a customer who changes their mind gets **no explanation at all** (correctly, not a typing accusation — but not an explanation either). **The project's own "seam built but never wired" pattern**, landing on the one promise the set-aside screen makes | **OPEN — NOT FIXED.** Fix shape: surface `OrphanedRenamedTo` on `/backups/remote` as a plain statement that an earlier history is set aside and not deleted. **Do NOT promise it can be reopened** — R-222 means it cannot be, and R-202's gate is precisely about not making that conditional promise |
+| **R-227** | **A controller restart mid-unlock returns a raw English `Bad Gateway`.** F8 restarted the container at T+0.7 s, inside the unseal window (control: `StartedAt` moved). The customer got **HTTP 502 / „Bad Gateway"** from traefik — a raw upstream error, in English, naming no reason and saying nothing about whether the key was installed. **The state half is clean**: the four `/data/offbox` files stayed byte-identical with mtimes frozen, and the controller returned healthy in 40 s. Breaches **I3** | **OPEN — NOT FIXED.** Low severity — the window is ~1 s wide. Recorded rather than inflated |
+
+**Recorded against existing rows by Phase 2:**
+
+- **R-216 — §4.1 is now MEASURED, not deduced.** The previous session could only offer two absences.
+  The box's own `/settings` renders „Minimális verzió (üzemeltető) **0.200.0**" (`GetFloor()`, whose
+  only writer is the report-ACK handler; **both hold branches serve `Floor=""`**, pinned by
+  `managed_floor_test.go:94`), and a **cold-started** controller logs
+  `settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0)` against the same line reading
+  `floor still unknown after 1m30s` while the hold was in force. The hub's HELD lines ran every
+  15 min to 22:12:06 and stopped, with a liveness control proving the hub kept logging. **The floor
+  is served.**
+- **⚠ A correction to how that positive was to be taken.** `SetFloor`'s line is `u.dbg(...)`, gated on
+  `cfg.Logging.Level == "debug"` and written to the **logger** — it can **never** reach the logx debug
+  ring, so it cannot appear in `/api/debug/logs` at any level. A controller restart alone would not
+  have produced it. Confirmed with a level census on the ring first (1196 DEBUG / 2802 INFO / 2 WARN),
+  so the absence was known to be structural rather than evidential.
+- **R-218 — the live half is STILL NOT MEASURED, deliberately.** The fix is present and correct
+  (`needsOffsiteCredential` now retires on the **target**, not the key), but the venue **has** a
+  target, so the box correctly does not declare; declaring here would be the bug. The state that
+  exercises it is shape (a), which the venue no longer holds. **Recorded as not measured rather than
+  inferred from the unit test.**
+- **R-217 — its fix HELD under exactly its fault** (F5): with the store blocked after a successful
+  unlock, the page rendered M3 and **no listing block at all**; the three false-claim strings are
+  absent, verified in UTF-8 with accented positive controls present.
+- **R-215 — its fix is present** and the `GET /recovery` gate consults the same predicate as the POST
+  sibling.
+- **R-199's back-pointer in `architecture/00-capability-map.md` was already added** — the brief lists
+  it as owed; it is present on the escrow-recovery row, explicitly labelled as the omitted
+  back-pointer. **No action taken; the brief's assumption was stale.**
+
+**Untouched by this session, stated so nothing is presumed closed by association:** **R-213** (putting
+files back — the half the recovery screen deliberately does not do) and **R-202** (the orphan card's
+unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing cost of).
+
+
+
 | ID | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|
+| **R-213** | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | Design the live-vs-backup comparison, then the put-back flow on top of it. Do NOT fold it into the recovery screen | Operator + CC |
+| **R-212** | **The orphaned-ciphertext deletion HALTED: the stores on the storage box do not match this register's record.** Ruling 2 (2026-08-04) says *"delete the orphaned ciphertext, ~1.2 GB across the two demo boxes, in set-aside stores nothing prunes"*. **Measured 2026-08-05 over SFTP with each box's own credential, read-only:** demo-felhom (`u629488-sub1`) holds `felhom-repo` **1.2 G**, `felhom-repo.orphaned-20260717` **1.4 G**, `felhom-repo.orphaned-20260718` **3.0 M**; demo-hp (`u629488-sub3`) holds `felhom-repo` **582 K**, `felhom-repo.orphaned-20260804` **43 M**. So there are **THREE** set-aside stores totalling **~1.45 GB**, not two totalling ~1.2 GB — and **the figure that DOES match ~1.2 GB is demo-felhom's LIVE `felhom-repo`**, which is the configured `repo_path` and must not be deleted (it is unopenable by the box, which is R-193's point, but it is not a set-aside store) | **CLOSED 2026-08-05 — all three deleted after the operator confirmed the corrected list** | an exact path list, confirmed by the operator | Confirm which of the three `*.orphaned-*` paths to delete. **Nothing was deleted**; the listing above is the complete measured state | **DONE 2026-08-05.** The operator was shown the measured listing, confirmed *delete all three*, and all three `*.orphaned-*` stores were removed: demo-felhom `felhom-repo.orphaned-20260717` (1.4 G) + `felhom-repo.orphaned-20260718` (3.0 M), demo-hp `felhom-repo.orphaned-20260804` (43 M). **~1.45 GB freed.** **Both LIVE `felhom-repo` stores are untouched** (demo-felhom 1.2 G, demo-hp 582 K) — confirmed by a full listing before AND after on each account. **Proof nothing live was caught:** a REAL off-site run on demo-hp immediately afterwards returned `status: ok`, `orphaned: false`, `last_error: ""`, `last_run 2026-08-05T09:14:02Z`, 6 snapshots. **METHOD NOTE for the next session:** the Hetzner storage box has a RESTRICTED shell — no shell operators, no `test`, no GNU long flags. `rm -r <path>` as ONE simple command works; `test -d X && rm -rf -- X` returns *"Command not found"* and silently does nothing, which is how the first attempt failed CLOSED (verified by an unchanged listing). |
 | **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — |
 | **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — |
 | **E-2d** | **Prove E-2 on a fresh VM** — a real `felhom-host-install.sh` 1.22.0 run, Case B naturally, a claimable customer, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **CLOSED — PARTIALLY PROVEN** (2026-07-29) | — | **C1, C2 proven** (`audits/E2D-fresh-vm-2026-07-29.md`); **C3, C4 proven live** (`audits/SESSION-C-2026-07-29.md`); **C5 FAILED → R-116** — the gate fires and an alarm reaches the hub, but it is the generic event, so the alarm and its recovery cannot be paired. **R-116 is the single named open leg**; per the Session-C runbook §9, decided in advance, a failed claim closes the item as partially proven rather than triggering a re-run. Both audits carry the full record — the `local-lvm` fence, the ISO/PAIRING derivation, the Phase 0 answers, the per-claim observables and the teardown evidence — and are the place to read it, not this cell. **The arc's actual definition of done is R-106 + R-109, R-108 and D5**, none of which this detour touched | CC |
@@ -31,12 +157,23 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
 | **R-185** | **The agent cannot see the host backup tier's archives on demo-felhom — the PVE token has no ACL on `/storage/felhom-backup`, so the content listing returns EMPTY where root sees three archives.** Found 2026-08-03 while live-validating R-86. `pveum acl list` grants `FelhomAgentStore` on `/storage/{local,local-lvm,felhom-pbs}` and **not** on `felhom-backup`, which is the box's actual `local_backup_target`. Verified three ways: `pvesh` as root lists 3 archives (6.1–6.3 GB, 08-01/02/03); the same endpoint with the agent's token returns `{"data":[]}`; and `local` — which HAS a grant — returns its archives through the same token | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.123.0**, installer **1.24.0**) | — | **Pre-existing and independent of R-86** (it is a property of the ACL, and the R-85 rotation had the same blindness). **Consequences:** the host tier has never been restore-testable on that box, and R-85's *"an empty tier is skipped, not failed"* rule made that silent. **The part worth fixing is the silence, not only the grant:** a permission-blinded tier is today INDISTINGUISHABLE from a newborn one — both report *"no settled archive yet"* — which is this project's own absence-is-not-evidence rule failing in a new place. The agent already knows better: it RECORDS successful backups to that target, so *"I wrote archives here and the tier lists none"* is a contradiction it can detect and should say loudly. **Do not fix by widening the token blind:** decide whether the host-install ACL set should follow `local_backup_target` (it currently hardcodes `local`), which is where the drift began **BOTH HALVES FIXED, AND THE ROOT CAUSE WAS NOT WHERE IT LOOKED.** **(1) The silence.** `Client.Permissions` asks `GET /access/permissions?path=/storage/<target>` **as the agent's own token**, and one `capability.Status` per configured tier reports it — composed around the sudo prober the way the pool-read check already is, with the `Status` wire shape untouched so the hub's existing critical-degraded alert applies with **no hub change**. **MEASURED BEFORE IT WAS DESIGNED AGAINST, and the obvious reading is wrong:** an ungranted path answers neither empty nor 403 but with the privileges INHERITED from the box-wide `/` grant — `felhom-pbs` → `{Datastore.Allocate:1, Datastore.AllocateSpace:1}` vs `felhom-backup` → `{Sys.Audit:1, SDN.Use:1, Datastore.Audit:1}`. A probe asking *"is the path present?"* or *"does it have Datastore.Audit?"* would report the blinded storage HEALTHY; the probe tests `Datastore.AllocateSpace` and a red-proof pins that. The probed set comes from the box's own `BackupTiers()`, never a fixed list — a hardcoded probe list is the defect reproduced inside the fix. Critical (the hub alerts only on critical), **except** the `local` fallback target, which is reported but does not page: turning host-install's own documented DEGRADED configuration into an alert is how a signal becomes something an operator archives unread. It never looks at content, so it cannot alarm on a newborn tier by construction; it never reports ok when it could not ask. **(2) The installer.** The row and the task both assumed `PVE_STORAGES` was the culprit. It is not: `configure_backup_target`'s **Case A** arm creates the storage and grants in the same breath, so a box that builds its own target was always fine. The **Scenario-F** arm — *"the target already exists, leave it exactly as it is"* — returned **without granting**, which is why a box whose `felhom-backup` pre-dated the install (the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token could not read. That arm now ensures the ACL through the same guarded wrapper; the storage DEFINITION is still untouched, so Scenario F is unviolated, and `pveum acl modify` is idempotent. `$BACKUP_TARGET_ID` is deliberately still NOT in `PVE_STORAGES` — that list is granted in step 4/5, before the target is resolved in step 6, and `--acl-storages` entries are preflight-checked for existence. A **gate** now asserts every arm that resolves the target also grants on it (red-proved: *resolves in 2 place(s) but grants in only 1*). **LIVE, in order:** the three-way observation reproduced unchanged (root 3 archives / token `{"data":[]}` / `local` lists through the same token) → agent v0.123.0 deployed → **`capability DEGRADED … pve:store-grant:felhom-backup … lacks Datastore.AllocateSpace … INVISIBLE to the agent and it is never restore-tested`, critical=true**, and the hub raised `agent_capability_degraded` **and emailed the operator** → grant applied (user AND token) → the token lists **3 archives** → `ok=70 total=70 degraded=0` and the hub logged `degraded → ok (agent_capability_recovered)` → **the host tier is a due-check candidate for the first time on that box**, correctly picking the 08-02 archive because the 08-03 one has not settled 24 h. **demo-hp CARRIES THE SAME DRIFT Line truncated
 | **R-186** | **A released agent binary's sha256 cannot be reproduced from its tag.** `release-agent.sh` builds at step 3 and tags at step 4, so Go's VCS stamp records a PSEUDO-version (`v0.120.1-0.20260803130452-4d825910…`) in the published bytes, while any rebuild after the tag exists stamps `v0.121.0` — a different binary. Measured 2026-08-03 on v0.121.0: published `b2128f3c…` (14 081 336 B) vs rebuild-at-tag `8302e396…` (14 077 240 B), identical source, identical toolchain, 4 096 bytes apart | **CLOSED — SHIPPED + MEASURED 2026-08-03** (agent **v0.122.0**) | — | **Why it matters:** the sha the operator vouches is the one thing tying a machine to a binary, and today nobody can independently rebuild it to check. **The build order is deliberate** (the script's own comment: a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it), so the fix is not to swap the steps blind. Candidates: `-buildvcs=false` or `-trimpath` for a version-stable stamp, or tag-then-build with the tag deleted on a failed publish. **Mitigation used this session:** the DEPLOYED binary is the PUBLISHED artifact, downloaded from Gitea — not a local rebuild — so the running bytes are the vouchable ones **FIXED by removing the cause, not by sequencing around it.** `-buildvcs=false` drops the VCS stamp (nothing in the repo reads it — no `ReadBuildInfo` caller) and `-trimpath` removes absolute build paths, so the same source produces the same bytes whether or not the tag exists yet and regardless of checkout directory. **Measured before the change**, same source, same toolchain, one commit: default flags with no tag `18f4a495…` 14 085 464 B (mod `v0.121.2-0.2026…`) vs default flags tagged `4a38f394…` 14 085 440 B (mod `v0.121.99`) — different bytes; with the new flags, `7ffcdf1d…` 14 064 574 B **identical both ways**. **Measured after the change on the real release:** published `d5f294e5…` 14 076 649 B = rebuild-at-tag #1 = rebuild-at-tag #2, **all three identical**. **A second discrepancy fell out of it:** `publish-agent.sh`'s fallback build forced `CGO_ENABLED=0` and therefore produced a binary **74 KB smaller** (13 990 236 B vs 14 064 574 B) than the release path built for the same version — one version name, two binaries, decided by which entry point ran. Both paths now build identically, with a comment in each naming the other. **The operator can now check a published sha without being there:** the exact command is in `felhom-agent/CLAUDE.md`. | CC |
 | **R-187** | **R-115's one-command release had never actually run its publish leg — the first real use died there.** `scripts/publish-agent.sh` has been mode `0644` since it was created (2026-06-28), because every earlier caller invoked it as `bash scripts/publish-agent.sh`; `release-agent.sh` (written 2026-08-03) called it directly and got `Permission denied` on v0.121.0's release | **CLOSED — SHIPPED 2026-08-03** (`felhom-agent`) | — | **Fixed both ways in one commit:** the executable bit restored, and the caller changed to `bash "$REPO_ROOT/scripts/publish-agent.sh"` so the release no longer depends on a file mode — the kind of thing a checkout, an archive or a copy silently loses again. **The lesson is R-115's own, one level up:** the mechanism written to make a step unforgettable was itself never exercised end-to-end, so it failed the first time it mattered. A mechanism that has not been RUN is a note with better formatting | CC |
-| **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v<ver>` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. | CC |
+| **R-188** | **Every agent release has a ~50 % chance of emailing the operator a CI failure for a release that is correct.** `on: [push]` fires the gates workflow on the **tag** push too, and `release-agent.sh` pushes the tag BEFORE publishing the package (deliberately — a tag with no package is caught by `check-published-versions.py`, a package with no tag is invisible to it). So CI can run the published-versions gate inside the seconds-long window where `v<ver>` exists and the package does not, and it correctly reports *"every released agent version must be INSTALLABLE"* — for a release that completes normally a moment later | **CLOSED — SHIPPED 2026-08-03** (agent **v0.122.0**) | — | **MEASURED, 2026-08-03, across two releases in one session:** v0.121.0 → runs #12 (id 79) **success** and #13 (id 81) **failure** on the same sha `4618169`; v0.121.1 → #17 (id 88) **failure** and #18 (id 89) **success** on the same sha `cd1b087`; and v0.121.0's second pair (#14/#15, `4d82591`) both **success** — i.e. it is a race, not a rule. Local gate afterwards: all three versions verify. **Why it matters more than it looks:** R-168 made CI email the operator on failure precisely so a red gate cannot be missed. A signal that cries wolf on every second correct release is how that mail becomes something you archive unread — the R-29 failure mode one layer up, and R-100's corollary in a new place (an alarm whose text is true but whose trigger is not the thing you would act on). **Fix shapes, cheapest first:** have the workflow skip tag refs (`if: !startsWith(github.ref, 'refs/tags/')` — the commit push already covers the content, and the tag adds no new code); or make `check-published-versions.py` tolerate a tag younger than ~2 minutes; or publish before tagging, which the release script's own comment argues against. **Do not "fix" it by quieting the alarm** **FIXED by moving ONLY the tag push** (build → tag locally → publish → push tag). The tag is still created before anything is published, so the build and the tag still describe the same commit; it becomes VISIBLE — to CI, and to any `raw/tag/…` fetch — only once the package is downloadable. **The invariant the old order protected was not traded away:** `check-published-versions.py` now asserts the converse directly — *no published version may be missing its tag* — as a BOUNDED probe of the frontier (where a failed tag push leaves an orphan) and of patch gaps, printing its probe set on every run because a check whose coverage is invisible reads as a guarantee it is not making. The package listing api still answers **401** without a token (re-measured 2026-08-03), so absence still cannot be enumerated and the script says so. **Both directions demonstrated:** a real published-but-untagged fixture (0.121.2, published then deleted — HTTP 201 then 204, absence re-verified by a 404) made the gate FAIL and print the one-line recovery; removing the converse block made it pass over that same orphan. A tag with no package still fails, shown against a local stand-in rather than by pushing a probe tag — pushing one would have emailed the operator a TRUE alarm about a fixture, which is the same attention cost this row removes. **A half-done release is now loud:** publish-succeeded-then-push-failed dies naming the exact recovery command, and a FAILED publish removes the local-only tag so the release can simply be retried instead of colliding with step 2's re-release guard. **Scenario E, live:** v0.122.0's own release produced CI runs **21 (task id 96)** and **22 (task id 97)** on the same sha — **both green**, where the same shape produced one red run on each of the two previous releases. **THE REORDER MOVED THE RACE RATHER THAN REMOVING IT — observed 2026-08-04, and it is narrower but real.** Releasing v0.124.0 produced CI run **28 (id 115) FAILED** and run **29 (id 116) SUCCESS** on the same sha `257c4d85`. Timing: the tag was created locally at **07:38:36Z**; run 28 started **07:38:31Z** (the COMMIT push, five seconds earlier) and run 29 at **07:38:41Z** (the tag push). The old race was *tag visible before its package*; the new one is *package published before its tag is pushed* — the exact window the converse probe was added to detect, and it detected it. **NOT CONFIRMED from the run log** (Gitea's run-log endpoint 404s for this runner), so this is the timing-consistent explanation rather than a proven one; the gate passes locally with all 7 released versions installable and none untagged. **What it means in practice:** a release can still produce one red run and one operator e-mail, now from the opposite side. The window is seconds rather than the previous coin-flip, and it is self-clearing. **If it is worth closing:** have the workflow skip runs whose ref is a branch push occurring inside a release, or have the converse probe tolerate a package younger than ~2 minutes — but note that BOTH are the 'make the gate blind for a window' sLine truncated
 | **R-189** | **A passing restore-test can be invisible to the hub forever — and R-86 made that window a week instead of a day.** The agent reports `restore_tests[]` from `internal/backup.Store`, which is **in-memory** (its own comment: *"lost on restart; the cadence re-populates"*), and the hub's report interval is 900 s. An agent restart inside that window drops the result. That used to self-heal within 24 h because the next cadence re-tested the same tier; **under per-archive due-ness the agent will NOT re-test a proven archive**, so the hub can stay ignorant of a proof until the NEXT archive generation — a week on the offsite tier. **Observed live, 2026-08-03:** the 15:25:14 PASS on `felhom-pbs` (a real 14.5 GB offsite restore) reached **no** host-report — the hub logged `0 restore-tests` at 15:28:00 and 15:32:14 — because the agent was restarted at 15:27:57 for the v0.121.1 deploy, 2 m 43 s after the run and before the next report | **CLOSED — SHIPPED + PROVEN-LIVE 2026-08-03** (agent **v0.122.0**) | — | **The persisted proof already exists and is not reported.** `RestoreTestState` (on disk, per tier, R-86: with the archive id) survives exactly this, and `RestoreTestState.Snapshot()` carries the doc comment *"for the host-report gauge"* — **and has no production caller at all** (grep, 2026-08-03). That is this project's own *seam built but never wired* class, fifth instance, and a comment asserting an invariant with no test pinning it, both in one method. **Severity is bounded, and worth stating so it is not over-ranked:** the hub scans its retained window, archives keep arriving, and R-86's per-tier window for the offsite tier is 12 days against a 7-day archive rhythm — so ONE lost report is tolerated. What is not tolerable is that the loss is **silent** and that the box will not repeat the work. **Fix shape:** report the persisted per-tier proof (target, archive, proven_at) alongside the in-memory latest run, and let the hub's `lastProvenPerTier` read it — a report-contract change, so it touches `internal/hub/report.go` + `contract_test.go` + the hub's checker, and it is a spec, not a one-liner **FIXED.** `RestoreTestState` now stores the **tier** and what was **verified** beside the archive (recorded at proof time, from the run's own result — deriving them later would need a storage lookup that can fail on the one path where failing means mislabelling a proof), exposes `ProvenRestoreTests`, and `Collector.SetProvenRestoreTests` merges it with the in-memory result. **Merge rule: one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news and lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (two entries would read at the hub as two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **It refuses to lie:** a record missing the archive OR the tier produces NO entry, and run mechanics (scratch VMID, duration) are not re-invented — an absent duration is not a claim, a fabricated one would be. **The asymmetry is now written where it will be read:** only successes are persisted, because a success SUPPRESSES future work (a proven archive is never re-tested, so a lost proof leaves the box quietly less tested than it believes) while a failure CAUSES future work and heals itself at the next evaluation. **`Store`'s comment was corrected in place** — *"lost on restart; the cadence re-populates"* was true under a timer and became false the moment R-86 shipped; leaving it is how the next reader concludes this is handled. **The seam that had no caller is now pinned by an AST test** (`TestMainWiresTheDurableRestoreTestProof`), because `Snapshot()` claimed a *"host-report gauge"* in its doc comment for weeks with nothing calling it. **MIGRATION, stated because it is visible on the live box:** a pre-R-189 record carries an archive but no tier and is therefore NOT reportable — upgrading does not retroactively make an old proof visible to the hub; the tier's next real proof fills it in. Observed on demo-felhom immediately after the upgrade: still `0 restore-tests` until a fresh run. | CC |
-| **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/<vmid>` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/<id>` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had | CC |
-| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **OPEN** | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking | CC |
-| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. | CC |
-| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. **The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained. Record the answer here. | CC |
+| **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **MITIGATION SHIPPED 2026-08-04** (agent **v0.124.0 → v0.124.1**) — **MECHANISM STILL OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/<vmid>` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/<id>` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had **THE ROW NOW REFLECTS THE MITIGATION, NOT THE CAUSE — stated plainly because the two are different things.** The box is resilient; the loss is still unexplained. **Mitigation:** when the store-grant probe finds the grant absent, the agent runs the EXISTING root wrapper (`felhom-backup-target-apply grant <id>`) and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including its restraint. **No new privileged surface:** the sudoers vector `grant *` already covers any storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants BOTH user and token. The verb existed, was permitted, and had only ever been called at storage CREATION — the *built but never wired* shape in a verb rather than a seam, this project's seventh instance. Bounded at one attempt per tier per hour (a storage can be unreadable for reasons an ACL cannot fix; re-granting every cycle is a repair loop wearing a fix's clothes). **THE RECORD IS THE HALF THIS ROW IS ABOUT, and v0.124.0 got it wrong in production while every unit test passed.** It reported degraded for *one cycle* — meaning the probe call that repaired. But `probeAll` is invoked INDEPENDENTLY by the self-check log and by the collector building a host-report: on the box the repairing call was the log's (`09:39:34`, journal shows the repair and `degraded=1`) and the report three seconds later found the grant present and sent **`ok`**. The agent's journal had the record, the hub had nothing, and the operator would have learned nothing — the exact silence this row exists for, re-created inside its own mitigation. **v0.124.1** replaces it with a latch on TIME (20 min > the 900 s report interval), so at least one report must carry it. **PROVEN LIVE, twice, on demo-felhom** (grant deleted by hand, both rows): agent logs `store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190) target=felhom-backup privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read`; the ACL rows return; and on v0.124.1 the **host-report at 08:00:30Z carried `status=degraded`** with the explanation and the hub raised `agent_capability_degraded` and **e-mailed the operator** at 08:00:40. Nothing new was built to carry it — the hub's existing ok→degraded→ok edge is the channel, and the text rides `Feature` because that is the field the hub interpolates into the e-mail (`Reason` does not travel). **PART 3 — the single bounded pass at the mechanism, with the negatives named.** The lead §8.6 nominated is real as a CLASS and is documented in our own installer: *"`pveum user token remove` purges the token's ACL, so re-applying post-rotate is mandatory"*. **It does NOT fit this box.** A rotation purges ALL of the token's ACLs and mints a NEW secret; demo-felhom's token still authenticates with the same secret (`--selftest` OK), it retained its other three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced (no 2026-08-03 install log; host uptime 12 days at thLine truncated
+| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **CLOSED — SHIPPED 2026-08-04** (installer **1.25.0**; both live boxes corrected) | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking **THE GATE WAS RUN FIRST, AND IT MATTERED.** Before disabling anything, ep0 was read (read-only, Tier 2): prune jobs `prune-demo-felhom` and `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, `schedule 03:30`, `keep-last 2`, comment *"R-82 retention keep-last=2, server-side (box tokens are write-only)"* — and they have run **every day since 2026-07-27: 18 tasks, all `status=OK`**. The newest task log reads `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` / `keep ct/9201/2026-07-27…` / `keep ct/9201/2026-07-28…` / `TASK OK`. Retention happens, and it happens there. **A METHODOLOGICAL WARNING WORTH MORE THAN THE FIX.** Three separate queries said the OPPOSITE — *no prune jobs have ever run* — and **all three were broken instruments**: `worker-type` where the field is `worker_type`; the value `prune` where the worker type is `prunejob`; and `journalctl -u proxmox-backup` where the unit is `proxmox-backup-proxy`. A fourth reading (3 snapshots under keep-last 2) was mis-framed by CC and self-corrected — the third snapshot had landed AFTER that day's 03:30 window. Acting on any of them would have disabled the only pruning ATTEMPT while reporting that nothing prunes: a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. **The gate is what caught it, and only because it demanded evidence rather than a verdict.** **Shipped:** installer **1.25.0** writes `keep_last: 0` on the offsite tier (the agent's existing guard `allowPBSPrune = !primary && keep_last > 0` already reads that as *never prune from the box* — no agent change), the justifying paragraph is rewritten to say where retention lives and cite R-89, and `hostinstall_gates.py` asserts it (red-proved: pinning `keep_last: 2` back fails the gate). **Both live boxes corrected in their own config** — `backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false` on demo-felhom and demo-hp, with the local tier untouched at `keep_last=3`. Served over HTTPS at `1.25.0` with `"keep_last":0` in the served bytes. **STILL TO OBSERVE:** the next weekly offsite run completing OK end-to-end. The change removes the failing step; the *schedule* proving it is next week's event, and this row should carry that line when it happens. | CC |
+| **R-194** | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC |
+| **R-195** | ~~**A customer with no machine ever bound e-mailed an `expected_dbdump_missed` ERROR every morning.** `david` — a real prospective customer whose record was created 2026-08-01 16:51:49 with **hosts=0, host_deletions=0, host_reports=0, reports=0** — raised the alarm at 03:00 UTC on 08-02, 08-03 and 08-04~~ | **SHIPPED** (hub **v0.92.0**, 2026-08-04) | — | **The mechanism, and it is the interesting half: the skip that protects every other silent customer is keyed off having reported at least once.** `CheckBackupDeadlines`' down-skip reads `StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — a query **over the `reports` table**. A customer with zero reports appears in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. The skip therefore misses exactly the customer it would most obviously cover. (Corroborated live: `peti-felhom` is active with a host deleted 2026-07-15 and **does not** alarm — it has 482 old reports, so it is `down` and skipped.) The backup half was already safe (`reportJSON == ""` → skip); the DB-dump half had no guard at all. **Fix:** `store.HasEverBoundHost(customerID)` = a live `hosts` row **OR** a `host_deletions` tombstone, consulted once per customer at the top of the deadline loop. **The discriminator is deliberately "was a host EVER bound", NOT "has a report arrived"** — a box that was installed and went silent is a real fault and must keep alarming; that is the case the change could break and it has its own test. Fail-**open** on a read error (an unreadable binding must never SUPPRESS a real alarm), and the deferral is LOGGED with its own counter (the v0.73.0 Part-7 precedent: a quiet check must not look like a check that did not run). The anchored-verdict structure is untouched. **Red-proof observed:** deleting the guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` — verbatim the event `david` sent three mornings running. `david`'s record was NOT modified; the record was correct and the alarm was what was wrong | — |
+| **R-196** | **`escrow_stale` is wired to the ONE path that does not change the repo password, and absent from the path that does.** `ReissueCredentials` (`hub/internal/offsite/offsite.go:150-228`) resets **only** the Hetzner sub-account/box password and stages a fresh one-time secret — it contains no reference to a restic password and *cannot*, since that password is generated on the box and never leaves it except into the R-wrapped escrow. Yet it calls `MarkEscrowStale` on the stated grounds that *"the restic repo password just changed"* (`offsite.go:198-201`), and the same false premise is repeated at `api/handler.go:1067-1069` and in R-39's record | **CLOSED 2026-08-05 — hub v0.95.0** | — | **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — and the first where the comment factually describes a *different function*. It survived because its EFFECT (a stale escrow) is real, so nobody checked its CAUSE. **The live defect, not a documentation nit:** on the ordinary Re-issue shape (a consumed-but-failed install on a box that still holds its `repo_password` file) the box re-applies, `WriteOffboxSecrets` finds the file present and **keeps it**, the repo password is unchanged — and the hub has told the customer in Hungarian that their recovery escrow is stale and asked them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** **The inverse is the worse half and is R-193's:** demo-felhom's repo password *did* change on 2026-08-03 with **no Re-issue anywhere in its history** (its only `escrow_stale`/`offsite_reissued` pair is dated 2026-07-21 08:29:29) and therefore **nothing marked its escrow stale for 13 h**. **Fix shape:** mark the escrow stale on the evidence that it IS stale — a changed `restic_pw_sha256` (→ R-197) — not on a Re-issue; and correct all three comments in the same commit. **Not established, so not asserted:** whether the 2026-07-21 Re-issue on demo-felhom re-sealed an unchanged password (the predicted false-staleness shape) — `host_escrow_superseded` holds only two rows in the whole DB, both from 2026-08-04, so the prior generation is not retained. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q4 **FIVE instances, not three** — the spec expected three and named two; a census found five: `offsite/offsite.go` (the `MarkEscrowStale` justification), `api/handler.go` (the F3 re-enroll comment) and **three in `store/store.go`** (the `stale_at` ALTER comment, the `MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). All five now state what the code does, name the correction, and cite the recon; the staleness mark is documented as **precautionary** (the box's re-apply may mint a fresh repository password — the guest-rebuild shape) rather than evidential, with R-197's measured signal named as the evidential one. **THE BEHAVIOUR IS UNCHANGED AND THIS ROW STAYS OPEN:** on the ordinary re-issue shape — a box that still holds its `repo_password` file — the password does not change and the hub still marks a healthy escrow stale and asks the customer for an unnecessary ceremony. That is a behaviour change and must not ride a comment-correction release; it is also now **more** consequential than when filed, because under R-198 an unnecessary ceremony is no longer harmless bookkeeping — it supersedes a blob. Fix shape unchanged: mark stale on the evidence that it IS stale (R-197's hash comparison), not on a re-issue. **MEASURED LIVE 2026-08-04 night, in the flow where it does real damage.** After the Re-issue a rebuilt box needs to configure its off-site tier, `stale_at` was set at `2026-08-04 20:15:49` **while `restic_pw_sha256` was unchanged** — the escrow perfectly covered the box's current (recovered) key. The consequence is not cosmetic: a stale escrow makes the hub withhold the hash from the ACK, `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` refuses every off-site run. **The only documented way out is a fresh ceremony, which supersedes the identity blob and destroys the key being recovered.** This row's fix — mark the escrow stale on the evidence that it IS stale (a changed `restic_pw_sha256`, R-197's comparison), not on a Re-issue — is now on the critical path for R-201/R-204, not a tidy-up. Full chain: `audits/DRILL-r201-night-run-2026-08-04.md` §3(b) | CC **CLOSED (hub v0.95.0, 2026-08-05).** The pessimistic `MarkEscrowStale` and its `escrow_stale` event are GONE from `ReissueCredentials`; `offsite_reissued` is untouched. The removal was not a softening — nothing now guesses at re-issue time. **The case the mark guessed at is measured in two places, and one of them the mark was BLINDING:** a stale flag empties `restic_pw_sha256` from the report ACK, which is the exact value the controller's Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo password every cycle — so oLine truncated
+| **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) **SHIPPED.** `SaveHostEscrow` returns the hash it replaced; `handleHostEscrowPut` raises **`offsite_repo_key_changed`** when both hashes are known and differ. **Edge-triggered** (once per supersession, never per report — the dispatcher owns cooldown), **operator-only** (registered in `notify.operatorOnlyEvents` in the same commit that mints the type, because a missing `customerMessages` entry is NOT a block — the v0.78.0 defect), and **no hash value travels** in the message or the details. The in-between shapes (a first-ever hash, a hash-less supersession) are LOGGED rather than dropped, so *"we chose not to alarm"* and *"the check did not run"* never look identical. **Severity = warning, chosen for the world v0.93.0 creates:** before R-198 a changed key meant the previous history was unopenable by anyone ever, which would have argued for `error`; from v0.93.0 the superseding ceremony retains the old identity blob, so the fact is *"this customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss. `warning` also routes (the dispatcher treats `info` as an intentional non-notify). Driven through the real endpoint in test, not by calling the emitter. **Red-proof observed:** removing the comparison from the escrow PUT → the changed-key scenario fails with *"the repository key demonstrably changed and NO signal was raised"* while the unchanged-key scenario still passes. | CC |
+| **R-198** | **The hub's superseded-escrow retention does NOT retain the offsite repository password — and the ceremony the system tells the customer to run is what destroys the last copy.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`. `blob` is the **K-escrow** (the PBS datastore key, PBS-native scrypt); the **restic repo password lives in `identity_blob`** (`felhom-agent/internal/escrow/identity.go:34-39`, age-wrapped `IdentityBundle`). Measured live: both hosts' current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B and nothing else | **SHIPPED** (hub **v0.93.0**, 2026-08-04) | — | **This is the NINTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does not provide, and the first that is ALSO customer-facing copy.** The claim appears three times: the schema comment (`store.go:370-375`, *"so the old passphrase stays customer-R-recoverable … turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable'"*), the capability map's escrow row, and — in Hungarian, to the customer, on the orphan card — `controller/internal/web/templates/backups_remote.html:66,69` (*„a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek"*). **For the offsite restic repository, the incident it names, all three are false.** **Why it is worse than a missing column:** a rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, `offbox.go:570`), the card says „Helyreállítási kód szükséges", and the controller's own detector logs *"run the escrow ceremony"* (`report/escrow_confirm.go:100`) — so **the prescribed remedy is the act that overwrites `host_escrow.identity_blob` and loses the old password forever**. Both demo boxes crossed that line on **2026-08-04 at 07:15:36 (demo-hp) and 07:20:08 (demo-felhom)**. **This is an independent, stronger reason the 51 orphaned demo snapshots are unrecoverable than "nobody kept the recovery codes" — keeping R would not have helped.** **Fix shape (small):** add `identity_blob` to the superseded table and to `demoteCurrentEscrowTx`'s SELECT list; pin it with a test that asserts the CONSEQUENCE (a superseded row can still yield a repo password) rather than the mechanism. **Then correct all three claims in the same commit** — including the Hungarian card, which must not promise what the system cannot do. **Prerequisite for R-193's (d)-alone branch** and for the drill. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §7 **SHIPPED.** `host_escrow_superseded` gains `identity_blob` (CREATE + additive `ALTER TABLE`) and `demoteCurrentEscrowTx` carries it, so **both** callers — re-escrow and host-delete demotion — are fixed by one change to the routine its own comment calls *"THE ONE escrow row-copy routine"*. `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob is reachable from Go at all. The table comment now records that the ruling stated above it was not met and what that cost. **Tests assert the CONSEQUENCE, which is why the existing one stayed green:** `TestSaveHostEscrow_RetainsSuperseded` asserted that a retained row exists carrying the old K-blob and passed throughout; `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still yield a repository password, and pins the load-bearing ordering (the identity blob is written by `SaveHostDRBundle` AFTER `SaveHostEscrow`, so the demote sees the PREVIOUS generation — if that inverts, the retained bytes would be the new blob filed under the old hash, recoverable-looking and wrong). `TestDeleteHost_DemotesIdentityBlob` proves the shared routine through its other caller. **Red-proofs, both observed failing:** dropping `identity_blob` from the copy (production behaviour ≤ v0.92.0) fails BOTH scenarios; fixing only the re-escrow caller fails the delete scenario while the re-escrow one passes — the §8.2 mistake, demonstrated rather than asserted. **Nothing was backfillable and it was CHECKED, not deduced:** rows superseded before v0.93.0 were written without the blob and their source rows are already overwritten; the live database holds exactly 2 retained rows, both from 2026-08-04, both `identity_blob` NULL. **Who the fix protects, measured live:** 2 of 2 hosts with a current escrow carry an identity blob (`demo-felhom-8363b5`, `demo-hp-bb76ea`) — their NEXT ceremony now retains a recoverable off-site key instead of destroying one. **STILL UNIT-PROVEN ONLY after the 2026-08-04 night drill.** Part 2 (wipe again, do NOT recover, let a ceremony seal a DIFFERENT password, then inspect the superseded row's `identity_blob`) was gated on the first drill passing and **did not run** — the verdict was not reached, and a second wipe wouldLine truncated
+| **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **SHIPPED + PROVEN-LIVE 2026-08-04** (hub **v0.94.0**, agent **v0.125.0**) | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts/<id>/escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 **LINKS 6, 7 AND 8 ARE ASSEMBLED AND WALKED.** **Link 6:** `GET /api/v1/hosts/{host_id}/escrow` — the box-authenticated MIRROR of the PUT that stores the blob, self-scoped by the per-host key (global may read any, the same asymmetry the PUT has). The operator-driven DR endpoints are UNTOUCHED and pinned by a test that exercises them with recovery mode off and on. **Link 7:** `POST /escrow/recover-offsite-password` on the agent's pinned local API gives `UnwrapIdentityBundle` its first production caller in two months. **Link 8:** it extracts and returns **only** the repository password (not the tunnel token, not the PBS token, not the WG key — the controller is a trust tier down). **PROVEN ON HARDWARE, demo-felhom, 2026-08-04 13:49 CEST:** on-disk `c60c8bc737a6…` vs recovered `c60c8bc737a6…` — **MATCH**, and the same hash the hub independently stores as `restic_pw_sha256`, so three sources agree. **Scenario B proven live 5 minutes earlier** with a deliberately wrong code: hub served the blob (572 B, `self_scope=true`), agent logged *the recovery code did not unwrap the identity escrow … exit status 1*, nothing written — which also proves links 6 and 7 ran independently of the success. **Scenario E proven live:** both retrievals raised `escrow_blob_served` (warning, operator-only); the first mailed the operator, the second was cooldown-suppressed AND that suppression is itself recorded; the customer leg reads `skipped/operator_only` on both. **R persisted nowhere, searched not claimed:** 0 lines in the agent journal, 0 in the controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover `felhom-idesc-*` staging dirs, and the staged-secret dir empty — with a **positive control** (a planted copy was found: 1, then 0 after removal) so the sweep is a measurement rather than an unfalsifiable absence. **THE §8.2 TRADE, made deliberately and recorded in the handler:** obtaining the blob used to require the operator to arm recovery mode; it now needs only the box's own credential. They still cannot open it (the hub never held R; a wrong code fails closed at age's scrypt KDF). `escrowSelfServiceRetrieval` is a single named constant — flipping it to false re-imposes recovery mode and changes nothing else, so the operator can overrule the trade for the cost of a boolean. **Red-proofs observed:** removing the ownership check served host B's blob to host A; removing the audit record made the retrieval silent; returning `PBSToken` instead of `ResticRepoPassword` yielded a plausible bundle with a non-matching key; commenting the `Options.EscrowRecovery` wiring failed the AST seam test. **Seam discipline:** the wiring is asserted by walking `main` → `runDaemon` → `buildLocalAPIServer` and checking the composite literal, not by `strings.Contains` — links 6 and 7 were two of this project's six built-but-never-wired instances and the fix must not become the seventh | CC |
+| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **PLUMBING COMPLETE** (controller **v0.196.0**); **the FORM is not built — still open** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 **THE DIAGNOSTIC HALF IS DONE AND IT ANSWERED THE QUESTION.** `--recover-offsite-check` is a `docker exec` escape hatch in the shape of `--print-reset-code`: R on **STDIN** (never argv, never `ps`, never shell history, never a transcript), fetch+unseal via the agent, and a verdict of **two sha256 hashes**. **It compares and never installs** — the recovered password is not written to `offbox/repo_password`; a test asserts the data dir is byte-unchanged and its red-proof (adding the install call) fails it. Confirmed live: `repo_password` mtime still `2026-08-03 07:18:02` after the successful check at `2026-08-04 11:49`. **Exit codes are load-bearing** — `0` match, `2` a clean MISMATCH, `1` a step failed; "it failed" and "it worked and disagreed" must never share a status because only one is a finding about the system. A box with no local password reports distinctly (the rebuilt-box shape, where the next step is to INSTALL rather than compare). **WHAT IS NOT BUILT, deliberately:** no card, no form, no preview, no customer-facing text — building an interface on top of a chain nobody had walked is how the preceding three weeks went wrong. **What remains for this row:** link 9 (the recovered password placed so `WriteOffboxSecrets` keeps it) and the customer-facing shape the operator ruled on 2026-08-04 (yell → R form → preview → proceed), which is now priced against a chain that exists rather than one that is assumed **PART 0 SHIPPED 2026-08-04 (v0.196.0):** `--recover-offsite-install` is the sibling of the check — same fetch/unseal path, same STDIN discipline for R — and it **places** the recovered password via `InjectOffboxPassword`. The confirmation is a SECOND invocation (`--confirm-install`): without it, both hashes print and nothing is written, so the operator sees the comparison before a write is possible. Three outcomes named distinctly: **installed** (no local password — the rebuilt-box shape), **unchanged** (identical key, nothing written), **refused** (a DIFFERENT key present — installing would clobber the key the current repository is encrypted under; exit 2, no force offered). It re-reads the file after writing rather than trusting the call. **Red-proof observed:** removing the confirmation gate makes the dry run write the password. The R-persistence test carries a **positive control** (a planted copy found, then removed and not found). **NOT YET EXERCISED AGAINST A LIVE RECOVERY** — the R-201 drill halted before step 9, so this is unit-proven only. **What remains for this row:** the customer-facing shape the operator ruled on 2026-08-04 (yell → recovery-code form → preview → proceed) | CC |
+| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **PASSED + PROVEN-LIVE 2026-08-04** — the customer file came back byte-identical | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there **MATERIALLY ADVANCED 2026-08-04, AND THE DISTINCTION MATTERS.** What was proven on hardware is that the offsite repository **password** comes back out of the sealed bundle byte-identical (R-199). What has STILL never happened: a recovered password **installed**, an existing repository **reopened** under one, and a **file restored** from it. **The drill's pass condition is unchanged and is not this** — it is a byte-identical sha256 of a sentinel file after a wipe and reinstall, and "the store opened" was explicitly ruled insufficient. Design: `audits/RECON-offsite-dr-chain-2026-08-04.md` §10. **The drill is now cheaper and better-founded than when it was designed:** links 6–8 are walked, so a failure during the drill can be localised instead of being an undifferentiated "recovery did not work", and R-198 means the ceremony that creates the drill's recovery code no longer destroys the key it is meant to protect **THE DRILL RAN AND DID NOT REACH ITS VERDICT, and stopping was the correct call** (`audits/DRILL-r201-offsite-recovery-2026-08-04.md`). Steps 1–4 completed; the wipe never happened; **nothing irreversible was done**. It halted because **the sentinel file was not in the off-site snapshot** (R-203) — wiping would have destroyed the only copy and proven nothing. **Sentinel sha256 `643166269103a25c…`, still on the box.** **What the attempt established live, all of it new:** (1) a rebuilt box's off-site run **refuses** with the orphan card and pushes `offbox_repo_orphaned` — the spike's predicted third outcome, measured for the first time, and it does NOT silently start a fresh history (this closes R-193's Q3); (2) the **orphan reset works** — move-aside to `/home/felhom-repo.orphaned-20260804`, never delete, fresh repo initialised, `offbox_repo_reset` pushed (operator-authorised during the session); (3) demo-hp's pre-rebuild history is **permanently unrecoverable** — its key sits in superseded row id 3 with `identity_blob` NULL, superseded 07:15:36, **four hours before v0.93.0 fixed the retention**; (4) **a precondition the runbook did not contain:** neither pre-existing off-site-toggled app has a restorable file leg — both are named-volume-only, which the off-site tier tars but the customer restore never unpacks — so a file-leg app (`calibre-web`, mandatory `userdata: media/books`) had to be deployed, and it is now in place as the fixture. **TO RESUME:** fix or scope R-203, re-run steps 4–5 and confirm the sentinel IS in the snapshot **by listing it, not by a green status**, then P4 + the STOP + steps 6–11. Everything else is already staged: versions, recovery code, working repository, file-leg app, sentinel. **The pass condition is unchanged** — a byte-identical sentinel sha256, not "the store opened" **UNBLOCKED 2026-08-04 by R-203 (controller v0.197.0).** The sentinel now lands in the off-site snapshot and is listed there by name and size — the thing whose absence halted the drill. **Everything the resumed run needs is already in place on demo-hp:** agent v0.125.0, controller v0.197.0, the recovery code held by the operator (`R_DEMO-HP`), a working off-site repository (3 snapshots), `calibre-web` deployed with a mandatory userdata path, and the sentinel at sha256 `643Line truncated
+| **R-202** | **The orphan card promises the customer their old backups "may later be restorable with the matching recovery code" — and after v0.93.0 that is true for supersessions from now on and FALSE for anything already orphaned.** `controller/internal/web/templates/backups_remote.html:66,69` states it unconditionally, in Hungarian, on the one surface where being wrong costs most | **OPEN — Part 5 hit its gate 2026-08-04; the card is UNTOUCHED and the sentence is still live** | R-199/R-201 (which generation an orphaned repo belongs to is not knowable to the box today) | — | **THE GATE, and why it was hit rather than squeezed past.** The condition was: ship it iff the hub can tell a box what it needs with **one** additional boolean on the escrow ACK it already sends. The hub *can* cheaply compute *"≥1 retained blob for this host carries an identity blob"* — one correlated predicate in `GetEscrowStatusForCustomer`, and the controller even has the right seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer the card's question.** The card renders on `RepoState == "orphaned"`, and the promise is about *the key THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned remote belongs to; a box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified, which is strictly worse on a customer-facing card than today's hedged one. **What would actually make it truthful** is knowing the orphaned repo's generation, which is the same knowledge R-199/R-201's unassembled chain needs. **Interim exposure, stated rather than buried:** the sentence remains live and remains false for both demo boxes. **Cheapest honest interim** (not taken here — it is a customer-copy change and the gate said leave it alone): drop the recoverability clause and say only that the old history is set aside and not deleted, which is true unconditionally | CC + operator |
+| **R-203** | **A customer-declared MANDATORY data directory was silently absent from the off-site snapshot while the run reported `ok`.** Measured live on demo-hp 2026-08-04: `calibre-web` declares `userdata: media/books class: mandatory`; its live bind is `/mnt/sys_drive/userdata/media/books` (the sentinel file was there), while the off-site capture set looked for `/mnt/sys_drive/felhom-data/userdata/media/books`, which does not exist. Result: `[WARN] mandatory data path missing on disk, skipped from offsite`, then `backed up calibre-web (…, 0 mandatory path(s))` and `backup OK: 3 app(s), 3 snapshot(s)` — `last_status: ok`, `last_success` stamped, nothing customer-visible, nothing hub-visible | **SHIPPED + PROVEN-LIVE 2026-08-04** (controller **v0.197.0**) | — | **THE MECHANISM, from source.** `NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment **when the drive IS the system data path** (`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`, `backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is `<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive` and an app deployed at `HDD_PATH=/mnt/sys_drive`, the two produce different directories. **The same compose used BOTH roots**, from one deploy: `${IMPORT_PATH}` → `/mnt/sys_drive/felhom-data/userdata/import/calibre` (with the segment), `${USERDATA_PATH}` → `/mnt/sys_drive/userdata/media/books` (without). **WHAT IS MEASURED vs NOT, because it changes the fix.** MEASURED: the paths disagree, the mandatory directory is absent from the snapshot, the run says `ok`, and the only signal is a container-log WARN. **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a SUPPORTED choice — it was used because demo-hp's only registered drive (`Felhom-Share`) is a NAS and was **correctly refused** as an app namespace (R-108 working as designed), while `/mnt/sys_drive` was **accepted** (HTTP 202). **EITHER BRANCH IS A DEFECT:** if the system drive is a supported app namespace, userdata resolution is wrong for every app on it and their mandatory directories are silently unprotected; if it is not supported, the deploy accepted a namespace it should have refused one call after refusing the NAS. **NOT a general off-site failure:** `opengist` and `privatebin` declare no mandatory userdata paths (everything of theirs is in named volumes), so they are unaffected and their snapshots are real. **Fix shape:** make the two roots one function, whichever is right — and make a skipped MANDATORY path a customer/hub-visible signal rather than a WARN, because `ok` with a missing mandatory directory is this project's own *a path the customer thinks is protected is not in the snapshot* shape. Source: `audits/DRILL-r201-offsite-recovery-2026-08-04.md` §2 **BOTH HALVES SHIPPED.** **(1) The paths.** `appbackup`'s helpers take a NAMESPACE ROOT; the census found **FIVE** bare-drive-path callers, not the four the spec named — the fifth is the **FileBrowser mount builder** (`web/handlers.go`), i.e. the customer's own file browser would have shown the wrong directory on a non-enrolled path (latent: the system drive is deliberately never a registered `StoragePath`). The rule now has **ONE expression** (`appbackup.NamespaceRootFor` / `IsEnrolledDrive`); there were already **two** copies and **they differed** — `backup.Manager.namespaceRoot` compared without `filepath.Clean`, `stacks.Manager.inGuest` with it, so a trailing slash from config would have flipped the mode in one package and not the other. `ComputeFabBuckets` now receives the namespace root, which is what `ComputeCaptureSet` has always received, so the export and the backup describe the same directories by construction. **(2) The verdict.** `last_status` gains **`incomplete`** — minted, because `ok`|`error`|`running` had nothing meaning *"it ran, and this app is not fully protected"*. **Not `error`:** the rest of the run worked, so `SnapshotCount` and the `LastSuccess` anchor still record what WAS captured. It reaches the operator through the **existing** per-run digest (`backup_run_failures`) — a new event type would be a two-repo change and the hub drops anything outside `allowedEventTypes`. **§8.4's narrowing is a NO-OP and no customer warning disappears:** `TierOffsite`'s `tierKeeps()` already admits mandatory only, demonstrated by widening the tier filter alone and watching the class check hold the line. **THE SPEC'S §8.3 RISK DOES NOT EXIST, and this is the correction owed:** `ExportDataMounts` lives in `delete.go` but is **export-only** — its single production caller is the `.fab` adapter, nothing deletes on its result, and the delete path's own guard `ProtectedHDDPaths` is layout-agnostic by construction (it protects BOTH `<hdd>/…` and `<hdd>/felhom-data/…`). It shipped as its own commit anyway. **PROVEN LIVE on demo-hp:** the bind moved `/mnt/sys_drive/userdata/Line truncated
+| **R-204** | **A rebuilt box can recover its off-site key and still cannot use it: the remedy that reconfigures the tier is the thing that blocks the recovery.** Measured end to end on demo-hp during the 2026-08-04 night drill, after a real controller-data wipe | **ALL FOUR ITEMS CLOSED 2026-08-05** (items 1–3 controller v0.198.0 + hub v0.95.0; item 4 controller v0.199.0 + hub v0.96.0) | — | **THE CHAIN, each link measured.** (1) A rebuilt controller **cannot configure its off-site target at all**: `offsite-apply: consume one-time password: no unconsumed offsite password` — the previous controller consumed it (ledger: created `07:11:51`, consumed `07:12:06`). That is R-193, reconfirmed live. (2) The documented remedy is an operator **Re-issue**, which works — and **sets `stale_at` on the escrow** (measured: `2026-08-04 20:15:49`) **while `restic_pw_sha256` is unchanged**, i.e. R-196's false staleness. (3) A stale escrow makes the hub **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending → escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses every off-site run**. (4) The only documented way to clear a stale escrow is **a fresh ceremony — which supersedes the identity blob and destroys the key being recovered.** **So the recovery and its precondition are mutually exclusive as built.** The key comes back (proven — `8a9e33aa4da6…` recovered byte-identical after the wipe, and installed) and then cannot be used to open the repository. **(5) A FOURTH link, undocumented anywhere:** a rebuilt box is **unclaimed**, and the claim gate correctly intercepts every non-claim route (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`), so **no controller endpoint responds at all** until the customer re-claims. That step appears in no design document and comes first. **Fix shape, not decided here:** either the Re-issue stops marking the escrow stale on evidence it does not have (R-196's own fix), or a recovered-and-verified key is allowed to confirm the escrow without a ceremony — the hash comparison that `EscrowAutoConfirmer` already performs is exactly the evidence needed, and it is being withheld precisely when it would be conclusive. **Do NOT fix by widening `OffboxRunnable`** — the atomicity guarantee it enforces (no un-recoverable ciphertext) is the reason the escrow exists. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3 **ALL FOUR WALKED AND MEASURED during the passing 2026-08-04 drill, so this row is now a known-good manual runbook AND the gap list.** (1) **R-193:** a rebuilt controller cannot configure its off-site tier — `no unconsumed offsite password` (ledger: created `07:11:51`, consumed `07:12:06` by its predecessor). Remedy: operator Re-issue. (2) **The claim gate:** a rebuilt box is unclaimed, so the gate intercepts EVERY controller endpoint (`claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)`) — a first step of every recovery that appears in no design document. **And the local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to `settings.json` while the RUNNING controller keeps its old copy in memory, so `effectiveClaimCode()` never sees it and the claim fails with *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming** — two attempts failed before this was diagnosed. (3) **R-196:** the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged → the hub withholds the hash → auto-confirm can never fire → `OffboxRunnable` refuses every run. Cleared here with the **manual** confirm (`/backup/offbox/confirm-escrow`) — NOT a ceremony, which would have superseded the identity blob and destroyed the recovered key. (4) **`mode=unit` is the restore default and returns the recovery unit, NOT the userdata leg.** A customer told to "restore from off-site" gets their app definition back and not their documents, and nothing in that outcome says so. **Fix priorities, in the order they hurt:** (4) is a customer-facing trap on the last step; (3) is R-196's fix; (2) needs the escape hatch to reload settings (or the claim state to survive a rebuild); (1) is R-193's restage. Source: `audits/DRILL-r201-night-run-2026-08-04.md` §3–§4 | CC + operator **OUTCOME 2026-08-05 — controller v0.198.0 + hub v0.95.0.** **Item 1 (the reset code needs a restart) — CLOSED, proven live.** `effectiveClaimCode` reads through to the persisted claim state, so a code minted by the separate `--print-reset-code` process is seen without a restart; the precedence rule between settings and config is unchanged. Read-through, not a TTL: a TTL leaves a window in which a superseded code still works, and that is the mutation `TestClaimCode_SupersededByASecondMint_RefusedImmediately` kills. Fails closed on an unreadable state. **Live on demo-felhom 9201, nothing restarted (`restarts=0`, container older than both mints): the Line truncated
+| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **CLOSED 2026-08-05 — the guard's scoping half closed BY REPLACEMENT** (hub v0.96.0) | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the samLine truncated
+| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **CLOSED 2026-08-05 — controller v0.200.0** (credential half v0.199.0/v0.96.0; the recovery SCREEN v0.200.0) | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`feLine truncated
 | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
 | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
 | **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |
@@ -122,6 +259,14 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
 | **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — |
 | **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — |
 | **R-169** | **CI can only report, because there is no gate in the road.** Every felhom repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand at. R-168's runner therefore notices a broken push *after* it has landed | **WAITING-ON-OPERATOR** (a working-style decision, not a defect) | an operator ruling | Making CI *blocking* requires two things this task deliberately did NOT do, because both change how the operator works and that is not a task's call: **(a)** branch protection on `main`, and **(b)** a pull-request workflow instead of direct-to-`main` pushes. The cost is real — every change would need a PR, which for a single-operator project may be worse than the disease. **The current arrangement is two nets, and it is not nothing**: `.githooks/pre-push` REFUSES locally, and R-168's runner NOTICES when that hook was skipped or was never armed in a clone, and emails. The honest gap is the window between a `--no-verify` push landing and the operator reading the alarm. Decide only if that window ever actually costs something | operator |
+| **R-205** | ~~**`RootFsPressureDespiteHousekeeping` can never fire** — a bare PromQL `and` across mismatched label sets~~ | **CLOSED — SHIPPED + RED-PROVEN LIVE** (`homelab-manifests` `6808a4b`, 2026-08-05) | — | **`and` → `and on(instance)`.** Left vector carries `{device,fstype,instance,job,mountpoint}`; the right is a node_exporter *textfile* metric carrying `{instance,job}`. Nothing ever matched, so the alert returned the empty vector unconditionally — `health=ok`, `inactive`, no error, **dead from the day it was written**. It stayed silent through the entire 2026-07 build-cache incident that took `/` to 86% used and put Longhorn's default disk into `Schedulable=False (DiskPressure)`. **Red-proof against the live Prometheus API, both sides individually true** (threshold substituted `0.25`→`0.5` because P3's prune had already cleared the incident; the defect is in label matching, not the threshold): bare `and` → **`result_count 0`**, `and on(instance)` → **`result_count 1`**. **The annotation was also wrong and is rewritten** — it sent the reader to Longhorn replicas/snapshots (~10% of the growth) instead of the unbounded build cache (157 GB, ~95%); it now names `docker buildx du` first and records the 30 GB cap so a reverted cap is diagnosable from the alert text. ArgoCD `monitoring` Synced/Healthy at `6808a4b`; rule live and `health=ok`. Evidence: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | — |
+| **R-206** | **The build-cache cap and the weekly prune exist only as a hand-edited `/etc/docker/daemon.json` on DooPlex — not in Ansible, so a rebuild loses them.** The `node_housekeeping` role must also carry the prune, which today it is forbidden to run | **READY (M) — NEW 2026-08-05** | — | **The spike validated the recipe; this row builds it.** Three parts. **(a) Template `/etc/docker/daemon.json`** with the **`policy` array** form — **the flat form (`{"gc":{"reservedSpace":…}}`) is SILENTLY IGNORED**, measured: the daemon starts, logs nothing, and `docker buildx inspect` still reports the built-in defaults. **The oracle is `docker buildx inspect`, never `dockerd --validate`** — the validator returned `configuration OK` for a bogus key AND for the config that then **crashed the daemon** (`filter` takes one value per policy entry, not an array; `error initializing buildkit: filters expect only one value`). **(b) Narrow the role's Docker ban** (`node-housekeeping.sh.j2:10-14`) to permit exactly `docker builder prune -af` and nothing else — the ban's stated premise ("Docker here runs only unrelated jarr-* dev containers") is obsolete: the growth is Felhom Go build cache. **The measured prune is SYNCHRONOUS** (150.35 GB back at t+0, two consecutive polls <1 MB apart within 60 s) — **unlike containerd's image GC, so it needs no `settle_imagefs` equivalent**, but it MUST measure the filesystem rather than trust the command: `prune` claimed **156.9 GB** and the filesystem returned **150.35 GB**, the 6.5 GB gap being layers still shared with images. **(c) A restart-safety note in the role:** a bad `daemon.json` takes the daemon down AND leaves the `unless-stopped` dev containers stopped — they needed a manual `docker start` — so the role must restart-and-verify, not validate-and-assume. Recipe + every measurement: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | CC |
+| **R-207** | **`DRY_RUN=1` on `node-housekeeping.sh` is NOT non-mutating — it destroys the metric history it is supposed to let you inspect** | **READY (S) — NEW 2026-08-05** | — | `write_metrics()` (`node-housekeeping.sh.j2:119-150`) has **no `DRY_RUN` guard at all** — `DRY_RUN` appears in it only inside a log line — and it is called from an **unconditional `EXIT` trap** (`:150`). A dry run therefore atomically renames over the live node_exporter textfile, overwriting `node_housekeeping_last_success_timestamp_seconds` with *now* and `reclaimed_bytes` with ~0 — **resetting the staleness clock `HousekeepingStale` watches and erasing the 8-week reclaim history**. Confirmed by reading the source in both the 2026-08-05 audit and this spike; **neither run executed it**, so the history survives. Fix: guard `write_metrics` on `DRY_RUN`, or have the trap skip it. Pairs naturally with R-206 (same file, same role) | CC |
+| **R-208** | **Every Felhom Go build re-downloads its modules because `ARG VERSION` sits ABOVE the module-download layer — ~440 MB of dead cache per build, 90.5 GB of the 157 GB** | **READY (S) — NEW 2026-08-05, ROOT CAUSE PROVEN** | — | **Measured, not inferred.** All **208** retained `go mod download` records carried **`Usage count: 1`** — not one was ever reused in ~a month of builds. The mechanism was isolated by four controlled builds: an unchanged tree rebuilt with the **same** `--build-arg VERSION` → `RUN go mod download` **CACHED**; the **same** tree with a **new** `VERSION` → **executed**. `COPY go.mod ./` stays CACHED either way, which is the tell: a `COPY`'s key is content-based, while a `RUN`'s key includes the stage **environment**, and `ARG VERSION`/`ARG GIT_COMMIT` are declared *before* the download in `felhom-controller/controller/Dockerfile`. Since every real build passes a fresh version, the layer is invalidated **every single time**. **`felhom.eu/hub/Dockerfile` has the identical defect** (`ARG VERSION`/`ARG BUILD_TIME` above `COPY go.mod go.sum*` → `RUN go mod download`) — and because both Dockerfiles produce byte-identical `buildx du` description strings, the 208 records are a COMBINED count and must not be attributed to one project. **Fix shape (one line each, not applied here):** move the `ARG VERSION`/`ARG GIT_COMMIT`/`ARG BUILD_TIME` declarations down to just above the final `go build`. **Worth more than the cap and the move combined** — the cap bounds the symptom, this removes the source. `build.sh`'s `rm -rf` + `cp -a` and its host-side `go mod tidy` were **ruled out by fingerprinting**: the tree is byte-identical across runs and `tidy` is a no-op | CC |
+| **R-209** | ~~**Should the containerd store move to SSD2 at all?**~~ | **EXECUTED 2026-08-05 on operator ruling — reboot validation DEFERRED → R-209a** | — | **Operator ruled "proceed" having read the pre-analysis; CC's `storageReserved` condition was applied with it.** Moved with **zero loss, verified on four independent observables BEFORE the original was touched** (550,891 entries = 550,891; **448 = 448 `trusted.overlay` xattrs**; 37,243 = 37,243 hardlinks; byte-identical `meta.db` sha256) and again after (identical image/tag/volume ID **sets**, cache 2.782 GB/38 records, pg 4 DBs / 31 tables / 175,135,767 B, redis 2437). **`-X` is load-bearing** — overlayfs stacking rides `trusted.overlay.*`. End-to-end proof was a **real build** on the relocated store, `rc=0`. **k3s was never at risk and this was established before stopping anything:** it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up; only the two `jarr-*` dev containers were affected. `storageReserved` on SSD2 **0 → 80 GB**, still `Schedulable=True` at 76.34%. **A TRAP was found while proving the guard, and it is the reusable part: `RequiresMountsFor` on a path with NO mount unit is a SILENT NO-OP** — containerd started normally against an absent-but-unmounted path, so a typo'd guard buys nothing and says nothing (the *built-but-never-wired* shape again). The guard was therefore verified positively at the unit level (`Requires=` **and** `After=mnt-ssd_2.mount` on both units), and refusal was then proven with a genuinely absent **device** — via a temporary synthetic `.mount` unit, because `/mnt/ssd_2` hosts 12 live Longhorn replicas and must never be unmounted, and editing `fstab` on a production host risks emergency mode at boot: `Job containerd.service/start failed with result 'dependency'`, `is-active: inactive`. **Rollback is one documented sequence** (audit §11.8); the pre-move tree is **moved aside, not deleted**. Evidence: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §11 | — |
+| **R-209a** | **The SSD2 move has NOT survived a reboot, so by this project's own standard it is not fully validated** | **WATCHING — NEW 2026-08-05** | the next DooPlex reboot | **Operator ruled explicitly: do NOT reboot DooPlex.** Uptime verified unbroken (7 weeks 6 days, since 2026-06-10). **The distinction is stated rather than glossed: the MECHANISM is proven** — the guard is wired into both units and containerd refuses to start when a required mount's device is absent — **but the CONSEQUENCE is not**: that a real boot mounts `/mnt/ssd_2` before containerd starts, in this host's actual ordering. Mount-ordering reasoning is precisely the class this project has been burned by (`RequiresMountsFor` RE-MOUNTS rather than refusing — the ep0 lesson), and `CLAUDE.md` prefers a consequence assertion over a mechanism one. **Two deliberate consequences: (1)** the rollback copy `/var/lib/containerd.pre-move-2026-08-05` (**34.3 GB on `/`**) **STAYS** until a reboot validates — which is why `/` sits at 54% and not lower; deleting it now would trade a cheap 34 GB for the only cheap way back. **(2)** validation is **automatic and needs no one to remember it**: `felhom-store-postboot-check.service` (oneshot, enabled, dry-run PASS at install) runs at **every** boot and writes `RESULT: PASS`/`FAIL` to `/var/log/felhom-store-postboot-check.log`, asserting positively that `/mnt/ssd_2` is mounted, that containerd's root is on it, that **`/var/lib/containerd` does NOT exist** (the empty-store trap), that ≥100 images are visible and that both dev containers run. **Next action: after the next reboot — planned or not — read that file; on PASS, `rm -rf /var/lib/containerd.pre-move-2026-08-05` returns ~34 GB to `/`** | operator + CC | **P3's prune already removed the urgency: `/` went 86% → 53% used and SSD1's Longhorn disk went `Schedulable=False (DiskPressure)` → `Schedulable=True` (18.85% → 50.32% available).** The move was ruled "cap then move"; the cap is in and the pressure is gone, so this is now a deliberate choice rather than a rescue. **The numbers, measured (`Crucial-SSD-240G`, `/mnt/ssd_2/data/longhorn`, `storageMaximum` 235,148,750,848):** available today **214,958,080,000 (91.41%)**; 25% floor **58,787,187,712**. Moving the whole containerd tree at steady state (~31.5 GB images + ≤30 GB cache ≈ 65 GB) leaves **63.77%, i.e. +38.8 pp above the floor — comfortably safe as measured.** **But `storageScheduled` on SSD2 is 139,586,437,120 while `df` says only 20,094,939,136 is actually used** — Longhorn has overcommitted 6.9× — and if those volumes ever inflate to their scheduled size, the same disk lands at **13.00%, i.e. 12 pp BELOW the floor → `Schedulable=False`**, which is exactly the failure that just took SSD1 out. **Recommendation: do the move only together with setting `storageReserved` on SSD2 to cover the containerd tree (~80 GB); SSD2 reserving zero while HDD2 and HDD4 each reserve 500 GB is an anomaly in its own right.** Mechanism, if it goes ahead: **containerd's `root` in `/etc/containerd/config.toml`** (the key is present but commented out) — **not** Docker's `data-root`, which would move only 0.62 GB. Guard: `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`, remembering that **`RequiresMountsFor` RE-MOUNTS rather than refusing** ([[ep0-datastore-volume-move-2026-07-27]]) — so it must be tested with a genuinely absent device, and the move is not validated until it has survived a **reboot**. Full pre-analysis: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §P6 | operator |
+| **R-210** | **Which of the 345 local images may be deleted — 131 controller tags and 62 hub tags exist ONLY on this box and are not recoverable by `docker pull`** | **WAITING-ON-OPERATOR** — NEW 2026-08-05 | an operator ruling | **Nothing was deleted; this is a list, not an action.** The registry was queried directly: `felhom-controller` has **76** tags in Gitea vs **207** locally, `felhom-hub` **45** vs **107**. The **131 + 62 local-only tags are all OLD** — controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`, hub `0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0` — while everything from controller `0.136.0` and hub `0.58.0` upward IS in the registry and therefore re-pullable. **Size the prize honestly before spending a decision on it:** per-tag sizes sum to 139.29 GB, but that double-counts shared layers — `docker system df` puts the **real** dedup'd image footprint at **31.02 GB with 27.02 GB reclaimable**, i.e. an order of magnitude less than the build cache P3 already returned. `docker image prune -a` would remove 343 of 345 (only `redis:7-alpine` and `postgres:16-alpine` are held by running containers). **CC's view: not worth doing for the space** — it buys ~27 GB against 199 GB now free, and its only real benefit is dropping unrecoverable clutter | operator |
+| **R-211** | **Prometheus has no config-reloader — a rules change reaches the pod and is never read** | **READY (S) — NEW 2026-08-05** | — | Found while verifying R-205 rather than by looking for it. The `mon-system/prometheus` Deployment runs **one** container (`prom/prometheus:v3.12.0`) with **no `configmap-reload`/`prometheus-config-reloader` sidecar**. After the ArgoCD sync the updated `node-housekeeping-alerts.yml` was present **inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus **rules API still served the old expression** — for **4+ minutes**, with no error anywhere. It only took effect after an explicit `POST /-/reload`. **The consequence is general, not specific to R-205: every rule edit in this repo since the stack was built has silently not applied until something happened to restart the pod** — so "committed and synced" has never meant "in force", and ArgoCD reporting `Synced/Healthy` is true and beside the point. `--web.enable-lifecycle` IS already set, so the fix is small: add a reloader sidecar watching the ConfigMap, or a `checksum/config` pod annotation so a rules change rolls the pod. **Same class as the four *built-but-never-wired* seams** — the control exists, nothing walks it | CC |
 
 ## Why the TOP READY rows rank this way
 

documentation/backlog/ROADMAP.md

@@ -26,7 +26,7 @@
 | R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
 | R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
 | R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
-| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
+| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path. **SCOPE-CORRECTED 2026-08-04 (R-193 spike, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): "auto-fire on re-enrollment" means HOST re-enrollment, and only when the host record was DELETED first.** `reissueOnReenroll` sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`, `api/handler.go:1004-1016`), so a **guest** rebuild — which leaves the `hosts` row intact — never reaches F2 or F3. Measured live: demo-hp's guest was rebuilt 2026-08-03, `demo-hp-bb76ea` was unchanged throughout, F3 did not fire, and the box had **no off-site copy for 25 hours** until an operator ran Re-issue by hand. Peti's clean-slate (a real host delete + re-enroll) still walks the supported path; **the arc does not cover a guest rebuild, and that is the gap R-193 owns.** Note also that F3 alone would NOT have saved the data — the rebuilt controller mints a fresh restic repo password regardless (spike Q2) |
 | R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
 | R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
 | R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |
@@ -57,6 +57,18 @@
 | R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
 | R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
 | R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
+| R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
+| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
+| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **CLOSED — behaviour shipped hub v0.95.0 (2026-08-05)** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping **Collapsed 2026-08-05.** The comment correction landed in v0.93.0; the BEHAVIOUR landed in v0.95.0 with R-204 item 2 — the precautionary `MarkEscrowStale` and its `escrow_stale` event are gone from `ReissueCredentials`. Removing a signal was safe because the real case is measured twice over, and the mark was **blinding** one of them: a stale flag empties `restic_pw_sha256` from the ACK, which is what the controller's per-ACK re-check compares against. |
+| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
+| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
+| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
+| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
+| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
+| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
+| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ALL FOUR ITEMS SHIPPED (controller v0.199.0 + hub v0.96.0, 2026-08-05)** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill **Item 4 shipped 2026-08-05 by DECLARATION, per the operator ruling: the box states its condition, the hub answers; an absence has four meanings and the hub cannot tell them apart. `internal/offsiteheal` mirrors `pbsdrheal` (declared states only, two-distinct-report debounce, restage before mint, healthy = no-op). Closes R-193's credential half and R-192's guard half (by replacement). **The capability-map row keeps a qualifier for a NEW reason** — no step needs an operator any more, but there is no customer-facing recovery SCREEN, so the experience still requires someone who knows to look. |
+| R-193 | **A rebuilt box's off-site history was unreachable, and later reachable only from a command line** | L | **CLOSED (2026-08-05)** — credential half controller v0.199.0 + hub v0.96.0; the customer-facing SCREEN controller v0.200.0 | Collapsed per the coupling rule. Three legs: (a) the hub's retention keeps the sealed key (R-198); (b) a rebuilt box gets its credential back by DECLARING it needs one (R-204 item 4); (c) **the recovery screen** — a full page that unlocks the repository and lists what is in it, and **restores nothing**. Operator ruling: unlocking and restoring are SEPARATE, and the put-back needs a live-versus-backup comparison first (→ R-213). Flips the capability-map row's *needs someone who knows to look* qualifier; what remains there is one end-to-end re-walk and a correct-code run through the page |
+| R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
 | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
 | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
 | R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |

documentation/runbooks/workspace-CLAUDE.md

@@ -178,7 +178,7 @@ turns a true alarm into one the operator dismisses.
 
 ### A comment asserting an invariant needs a test pinning it, or it is a wish
 
-**Seven instances in this project have shipped guarantees the code did not provide** — each survived
+**Eight instances in this project have shipped guarantees the code did not provide** — each survived
 review because the comment read as settled:
 
 | # | Comment | What it claimed | What the code did |
@@ -190,6 +190,7 @@ review because the comment read as settled:
 | 5 | `inflight.go` | *"a caller that cannot acquire DEFERS"* | the backup caller recorded a failure and paged the operator (F-A1) |
 | 6 | `quiesce.go` | the agent's 409 *prevents* "a spurious failure" | on the start path it produced one (F-A1) |
 | 7 | `recovery_unit.go` B2 refusal (R-181) | *"the previous unit is untouched and NOTHING was deleted"* | *nothing deleted* held; **untouched was measured false** — the floor was checked ONLY in `captureAllRecoveryUnits`, while the two dump legs wrote the bulk into the same tree first and unguarded, so a 182,272 B tar became 2,147,666,432 B under a manifest that had not moved |
+| 8 | `ResolveManagedFloor` (R-216) | *"never push a controller past the agent it depends on"* | it compared the box's agent against the **golden's** MinAgent while serving a **floor** that could point elsewhere. Raise a floor above the vouched golden — which the day-0 runbook recommends and a per-customer override makes trivial — and the guard checks a version it is not serving. Measured live 2026-08-05: golden 0.192.0/MinAgent 0.113.0, floor 0.200.0, agent 0.120.0 → served, and the box landed on a controller needing 0.125.0. Its customer was then told their correct recovery code was wrong. **The first entry in this table where the false invariant was a GUARD, not a comment alone.** Fixed hub v0.97.0: a floor above the vouched golden is HELD, with its own reason |
 
 Three of these (4, 5/6 and 7) were found **on live hardware**, not by review or unit tests — #4 had a
 green, red-proofed test suite over a production path that was broken two independent ways, and #7

documentation/tests/campaign11-evidence-2026-08-05/journal-phase24.md

@@ -0,0 +1,707 @@
+# CAMPAIGN 11 — Phases 2 and 4 journal
+
+Every observable, in the order taken. **Unattended session, started 2026-08-05 22:38 CEST.**
+
+Clocks, stated once because three are in play and they differ:
+**demo-hp and the appliance = CEST · the guest (LXC 9201) = UTC · the hub = CEST** (the hub's own log
+lines read `22:12:06` at a wall-clock of `22:48`, so the hub logs local time, *not* UTC as the
+project memory's "hub=UTC" note says — see the hygiene section).
+
+---
+
+## Pre-flight — baselines re-read on arrival
+
+| What | Value | How read |
+|---|---|---|
+| `felhom-controller` `main` | **v0.201.0** @ `05cf352a2f17` | `git rev-parse HEAD` == `origin/main`, tree clean |
+| `felhom-agent` `main` | **v0.125.0** @ `0404f60e6a7b` | same |
+| `felhom.eu` `main` | @ `3a539ea5306a` | same |
+| hub, LIVE | `gitea.dooplex.hu/admin/felhom-hub:0.97.1` | `kubectl -n felhom-system get deploy hub -o jsonpath=…` |
+| Day-0 manifest | agent **0.125.0** (`f7d8339b…7016b3`) · golden **0.201.0** (`e730d7ca…f007654`) · `min_agent` **0.125.0** | hub `/configuration`, the `selected` options |
+| Highest register ID | **R-223** | fresh `grep -rhoE 'R-[0-9]{1,3}' --include=*.md` over all four repos |
+
+**All four cited commits match the brief exactly.** The manifest matches the journal's rebake record.
+
+### Venue, re-confirmed alive
+
+| | |
+|---|---|
+| Host | `demo-hp` (HP t740), Tier 0. Reached **by SSH key, first try** — R-129 stands |
+| VM | **321 `c11-appliance`**, `running`, uptime of host 13 d |
+| Appliance | `192.168.0.105`, hostname `c11`, `felhom-agent 0.125.0`, guest 9201 `running` |
+| Guest | **`192.168.0.106`** (DHCP — was `.207` in Phase 0 and `.227` in Phase 1; it moves) |
+| Containers | `felhom-controller:0.201.0` (healthy) · `filebrowser` (healthy) · `traefik` |
+| Hub | `c11-36d660` **ONLINE**, agent `0.125.0`, 1/1 guests |
+| Floor | c11 `v0.200.0 (override)`; **every other customer `v0.156.0`** — blast radius still zero |
+| Untouched | `drill-r50` (VM 300, stopped), guest 9201 on both demo boxes, every other hub customer |
+
+**`calibre-web` is NOT deployed.** Phase 3 destroyed the guest and never redeployed it. The venue's
+current shape is therefore *a rebuilt box that has completed a second escrow ceremony*, which is
+exactly what Phase 2's F-series needs.
+
+### Harness — the access path, and two credential traps hit on the way in
+
+Transport: **DooPlex → (Tailscale) → demo-hp → `192.168.0.105` (appliance) → `pct exec 9201`**, and
+for HTTP, **curl run on demo-hp**, which shares `192.168.0.0/24` with the guest. No server logic is
+skipped. **Harness, not product.** (The journal's Phase-0/1 SOCKS proxy was not needed; running the
+client on demo-hp is the same transport with fewer moving parts.)
+
+Two traps, both recorded because each cost a wrong reading first:
+
+1. **Never `source ~/.config/credentials`.** It holds keys with hyphens (`R_DEMO-HP=…`) that bash
+   cannot assign, and the resulting `command not found` error **echoes the secret value** into the
+   transcript. Two demo-box recovery codes were printed this way before the helper was rewritten to
+   `grep` the single key it needs. Nothing was written to a file and the values are not reproduced
+   here — but the mechanism is a live I4 hazard for any session that sources that file.
+2. **Values in that file are quoted**, so a bare `cut -d=` keeps the quotes and yields the wrong
+   secret — the hub returned `302` until the quotes were stripped. Already in project memory
+   (`credentials-file-values-are-quoted`); re-confirmed.
+
+---
+
+## §4 — the two positives that were owed
+
+### §4.1 — is the floor actually SERVED? **MEASURED. YES.**
+
+The previous session recorded *"no HELD line and no held reason"* — two absences — and correctly
+refused to call that a measurement. Here is the positive.
+
+**The box's own view, rendered from `GetFloor()`** (`/settings`, the „Rendszer" subpage):
+
+> Verzió és frissítés — Jelenlegi verzió **0.201.0** · Legújabb verzió **0.201.0 — naprakész** ·
+> Utolsó ellenőrzés `2026-08-05T15:57:36Z` · Automatikus frissítés – **Minimális verzió
+> (üzemeltető) `0.200.0` — a rendszer automatikusan erre a verzióra frissít, ha régebbi** ·
+> Utolsó frissítés Sikeres (**0.192.0 → 0.200.0**)
+
+**Why that value is proof and not decoration**, established from source rather than assumed:
+
+- `data["ControllerFloor"] = s.updater.GetFloor()` (`internal/web/handlers.go:1454`).
+- `u.floor` has **exactly one writer** — `SetFloor` (`selfupdate/updater.go:556`); the only
+  non-test caller is the report-ACK handler `updater.SetFloor(resp.MinControllerVersion)`
+  (`cmd/controller/main.go:696`).
+- **Both hold branches of `ResolveManagedFloor` set `d.Floor = ""`** (`store.go:2086` and `:2095`),
+  and `managed_floor_test.go:94` pins it (`!fd.Held || fd.Floor != ""`).
+
+So a **non-empty** floor on the box can only have come from an ACK that carried one. A held floor is
+served as the empty string and would render as none.
+
+**The temporal half, which rules out a stale pre-hold value:**
+
+| time (CEST) | observable | source |
+|---|---|---|
+| 17:57:05 | controller starts | guest log `15:57:05Z` |
+| 17:57:11 | `managed floor HELD for c11 … (controller floor withheld)` | hub log |
+| 17:58:35 | `[offsite-apply] settle-gate: GO — floor still unknown after 1m30s` | **the box's own** log |
+| 17:57 → **22:12:06**, every 15 min | `managed floor HELD for c11` ×18 | hub log |
+| after 22:12:06 | **no HELD line** | hub log |
+| 22:48 | box renders floor **`0.200.0`** | `/settings` |
+
+At start the box's floor was empty **by its own account**, and the hub withheld it for the next four
+hours. The `0.200.0` it holds now can therefore only have arrived on an ACK **after the vouch
+released the hold**. That is a measurement, not arithmetic.
+
+**The absence is backed by a liveness positive** (rule 6): the HELD lines stop at 22:12:06 while the
+hub demonstrably keeps logging — `22:44:41 host-report from demo-hp-bb76ea`, `22:44:57 wgsync: pushed
+5 peers`. Two further report cycles (22:27, 22:42) passed with no HELD line.
+
+#### And a correction to the brief's own plan for §4.1
+
+The brief says to take this positive during **F8**'s controller restart, *"at DEBUG if necessary"*.
+**A restart would not have produced it.** `SetFloor`'s log line is `u.dbg(...)`
+(`updater.go:554`), and `u.dbg` is gated on a private `u.debug` flag set from
+`cfg.Logging.Level == "debug"` (`main.go:478`) — it writes to the **logger**, never to the logx debug
+ring, so it cannot appear in `/api/debug/logs` at any level.
+
+Verified live: the debug ring was pulled (4 000 entries, `17:03:08Z → 20:46:46Z` — a window that
+**covers** the 20:12→20:27 UTC release), and it contains **no** `SetFloor` line. **Positive control
+on the instrument first**: the same ring carries `1196 DEBUG / 2802 INFO / 2 WARN` entries, so the
+ring works and the level is not filtered — the absence is structural, not evidential.
+
+> This is the same class the brief warns about, caught on the instrument rather than the subject: had
+> the ring been read without the level census, "no `SetFloor` line after a restart" would have looked
+> exactly like "the floor is still held".
+
+**§4.1 verdict: the floor IS being served, measured from the box's own state.** Recorded against
+R-216's row.
+
+
+---
+
+## §4.2 — R-218's live half: deferred, with the reason
+
+R-218's fix is present and readable in `backup/offbox.go:1257`:
+
+```go
+func (m *Manager) needsOffsiteCredential(t *settings.OffboxTarget) bool {
+    if t != nil { return false }   // a TARGET exists — not a rebuild
+    if m.settings == nil || !m.settings.GetHubEscrowIdentityPresent() { return false }
+    return true
+}
+```
+
+The short-circuit on `OffboxRepoPasswordHash()` that caused the deadlock is **gone** — the declaration
+now retires on the *target*, not on the key.
+
+**The live half cannot be taken in the venue's current state**, and the reason is the fix itself: c11
+**has** a target (`host`/`user`/`port`/`repo_path` all set, `applied_marker` present since 14:57), so
+`t != nil` and the box correctly does **not** declare. Declaring here would be the bug.
+
+Reaching the state that exercises it needs a box with the hub's identity blob, a placed key and **no
+target** — which the venue last held during Phase 1 and does not hold now. **Recorded as still not
+measured, deliberately, rather than reported as measured from the unit test.** F7's set-aside was
+examined as a candidate route and is reported in F7 below.
+
+---
+
+## Phase 2 — the faults
+
+Order: reversible first. **Every fault carries a positive control proving the fault was real**, and
+every control that failed is reported as a failed control rather than a result.
+
+### The four messages, as identified in `internal/web/recovery_handlers.go` (v0.201.0)
+
+| | trigger | first words |
+|---|---|---|
+| **M1** | the unseal failed **and** the hub holds no superseded package | „A megadott helyreállítási kódot **nem fogadtuk el**. Ellenőrizd, hogy mind a tíz szót…" |
+| **M2** | the capability gate says the agent cannot do it (R-216) | „Ez a gép **még nem tudja megnyitni** a mentéseidet… **A kódoddal semmi baj**…" |
+| **M3** | unlocked, inventory unreadable (R-217) | „A kulcs visszakerült, **de a mentések listáját most nem sikerült beolvasni**…" |
+| **M4** | the unseal failed **and** a superseded package is retained (R-222) | „**Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk** ehhez a géphez…" |
+
+**The venue holds a superseded package** (`hub_escrow_superseded_present: true`, `…_at 2026-08-05
+15:03:14`), so the M4 branch — which is tested *before* M1 on the failure path — is live for every
+failed unlock. That single fact shapes F1, F3 and F4 below.
+
+### F1 — a wrong recovery code, three times
+
+Injected: a ten-word hyphenated code that is neither real code, POSTed three times.
+
+| attempt | HTTP | elapsed | message |
+|---|---|---|---|
+| 1 | 200 | **1.194 s** | M4 |
+| 2 | 200 | **1.004 s** | M4 |
+| 3 | 200 | **1.014 s** | M4 |
+
+- **Refused each time** ✅ · **no lockout** ✅ (three identical responses; the handler documents
+  `NO LOCKOUT (§8.4)` by design) · **HTTP 200, no error page** ✅
+- **~1.0–1.2 s is the positive tell that a REAL unseal was attempted** — `age`'s scrypt cost, visible.
+  Phase 1's 0.134 s is what "no attempt was made" looks like. This discriminator is used throughout.
+- **Nothing was written** ✅ — all four `/data/offbox` files byte-identical with **mtimes unchanged at
+  14:57:35**:
+
+```
+846ac6e78a71e868  2026-08-05 14:57:35  applied_marker
+75dbbc9be6909912  2026-08-05 14:57:35  known_hosts
+e11a6c542b73477a  2026-08-05 14:57:35  repo_password     ← the Phase-3 key, unchanged
+85579eb0033832a0  2026-08-05 14:57:35  ssh_key
+```
+
+- **I4 held** ✅ — a sweep for three fragments of the entered code across the controller's data dir,
+  `/var/log`, `/tmp` and `docker logs` returned **product hits: 0**. **Both controls passed first**:
+  a planted canary was found by the same sweep, and `docker logs … | grep -c recovery` returned 6.
+  The only hits were in the harness's own script (`/tmp/.cc.sh`) — a self-match, cleaned up.
+
+**The deviation from the brief's expected answer: message 4, not message 1.** F1's stated right
+answer is *"message 1 only"*. On this box M1 is **unreachable**, because the M4 branch precedes it and
+fires whenever a superseded package exists.
+
+Judged on the message rather than the outcome: M4 is *hedged* („**Ha** egy korábbi kódot adtál meg"),
+it does not accuse the customer of mistyping, and it states two true facts. But a customer who simply
+**fumbled their current code** is given no hint to re-check their ten words and is instead routed to
+customer support about *older* backups. **M1 — the only message that tells a customer to check their
+typing — cannot be reached on any box that has re-escrowed**, which is exactly the population most
+likely to be typing a recovery code. Recorded as a finding.
+
+### F2 — the customer has no code at all — **PASS**
+
+Two halves, both taken.
+
+- **The screen** states it plainly, unprompted:
+  > „**A helyreállítási kódot senki nem tudja pótolni** — sem a Felhom, sem az ügyfélszolgálat, sem az
+  > üzemeltető. Ez szándékos… Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé."
+- **It offers the set-aside path rather than a dead end** — „…**nem kérem vissza a korábbi adatokat**"
+  is rendered, because `CanSetAside` (= `OffboxOrphaned()`) is true here. ✅
+- **An empty submission** returns „**Add meg a helyreállítási kódot.**" in **0.027 s** — no unseal
+  attempted, and no blame attached to a code that was never given. ✅
+
+### F3 — the hub unreachable when the code is entered — **FAIL**
+
+Injected on the appliance: `iptables -I OUTPUT 1 -d 37.191.56.193 -j REJECT`.
+**Control: hub `http=302` before → `http=000, exit=7` after.** The fault is real.
+
+Then the **correct current recovery code** was entered, so the only possible cause of failure is the hub.
+
+```
+HTTP 200 · elapsed 0.0556 s          ← far below the ~1.0 s scrypt cost: NO unseal was attempted
+```
+
+> **M4** — „Ez a kód nem nyitja meg azt a csomagot, amit most őrzünk ehhez a géphez. Ha egy korábbi
+> kódot adtál meg: a géped azóta új mentési kulcsot kapott…"
+
+**The code was perfect and current. The hub was down. The customer is told their code does not open
+their package and is sent to customer support about older backups.** The hub is never named. **I6
+breach** — an unreachable service reported as a fact about the code.
+
+**And the machine knew.** The agent's own journal distinguishes the two cases exactly:
+
+```
+F1 (wrong code):  err="escrow: the recovery code did not unwrap the identity escrow
+                       (wrong recovery code, or a corrupt blob): exit status 1"
+F3 (hub down):    err="escrow: fetching the sealed bundle: hub: transport error:
+                       Get \"https://hub.felhom.eu/api/v1/hosts/c11-36d660/escrow\":
+                       dial tcp 37.191.56.193:443: connect: no route to host"
+```
+
+The discriminator exists in the agent's `err` field and is **discarded at the HTTP boundary**: both
+return **HTTP 400** with one merged sentence — *"the recovery code did not open the sealed bundle,
+**or the bundle could not be fetched**"* — and the agent's own `msg=` collapses them too
+(*"FAILED (wrong recovery code, or the blob could not be fetched)"*). The controller then has no
+branch for "could not reach", so `rerr != nil` falls into the code/package messages.
+
+**Reverted**: rule deleted, chain empty, hub `http=302` again.
+
+### F4 — the agent stopped when the code is entered — **FAIL**
+
+Injected: `systemctl stop felhom-agent`. **Control: `is-active` = `active` → `inactive`, and no
+listener on `:8443`.** Real.
+
+```
+HTTP 200 · elapsed 0.0299 s          ← again no unseal
+```
+
+> **M4** — the same sentence as F1 and F3.
+
+**The stated right answer is M2** — *"names the machine's inability to ask, the R-216 fix under
+pressure"*. M2 did **not** fire. The controller log names the truth the customer never sees:
+
+```
+recovery: unlock failed: agentapi: POST /escrow/recover-offsite-password:
+  Post "https://169.254.253.1:8443/escrow/recover-offsite-password":
+  dial tcp 169.254.253.1:8443: connect: connection refused
+```
+
+**Why the R-216 gate did not catch it, measured rather than reasoned** — the box's own debug ring:
+
+```
+[web] recovery capability gate: offsite_key_recovery=yes (source=version)
+```
+
+**`source=version`.** `recoverySupport` asks `netFeatures.SupportsWithSource(...)`, which answers from
+the **known agent version** without probe traffic — by design, and correct for the question it was
+built to answer. But the question it is being used for here is *"can this machine ask right now?"*,
+and a version number cannot answer that. The gate discriminates **age**, not **reachability**; a dead
+agent of the right version sails through it, and the failure that follows is attributed to the code —
+the precise outcome R-216's comment says *"must never"* happen (*"An attempt that cannot succeed must
+never be made, because its failure is attributed to the code"*).
+
+**F3 and F4 are one defect with two faces**: on the unlock failure path there is no branch for *the
+machine could not ask or could not reach*, so every non-code cause is rendered as a statement about
+the code or the package. This is Phase 1's headline finding, relocated from the version channel to the
+transport.
+
+**Reverted**: agent `active`, `:8443` listening, `felhom-agent 0.125.0`.
+
+### F5 — the store unreachable after a successful unlock — **PASS**
+
+The first attempt's control **failed and is reported as a failed control**: the probe used `nc`, which
+the controller container does not have, so blocked and unblocked both printed the same fallback. A
+second attempt used the IPv6 address `getent hosts` returned — and the container has **no IPv6 route
+at all**, so that too could not discriminate. The container resolves the store to **IPv4
+`91.98.242.176`**, and only a `/dev/tcp` probe on that address separated the states:
+
+```
+BEFORE: TCP-OPEN     INJECT: iptables REJECT (OUTPUT + FORWARD)     AFTER: TCP-CLOSED
+```
+
+Only then was the fault real. With the **correct current code**:
+
+```
+HTTP 200 · elapsed 1.198 s           ← a REAL unseal
+```
+
+> „**A kulcs visszakerült**, de a mentések listáját most nem sikerült beolvasni. A mentéseid
+> nincsenek veszélyben — nézd meg a Biztonsági mentés oldalt néhány perc múlva."
+
+- **The unlock still counts** ✅ · **M3, the "could not be read" shape** ✅
+- **R-217's false claim is genuinely absent** ✅ — verified in Python on the local file with UTF-8
+  matching, *after* the accented-substring trap was accounted for (the project's own documented false-
+  zero hazard):
+
+```
+MUST BE ABSENT   "A tároló megnyílt" 0 · "van benne tartalom" 0 · "nem tudtuk alkalmazásokhoz rendelni" 0
+CONTROLS PRESENT "A kulcs visszakerült" 1 · "nézd meg a Biztonsági mentés oldalt" 1
+```
+
+**R-217's fix holds under exactly the fault it was written for.**
+
+**Reverted**: rules deleted, `TCP-OPEN` again.
+
+### F6 — „Most nem", then return — **PASS**
+
+Baseline first (the interception is on `/launcher` and `/dashboard`, not `/`):
+
+```
+BEFORE   GET /          -> 302 /launcher
+         GET /launcher  -> 302 /recovery        ← the full-page interruption
+         GET /dashboard -> 302 /recovery
+         GET /recovery  -> 200
+```
+
+`POST /recovery/postpone` → **302 → /launcher**, and:
+
+```
+AFTER    GET /launcher  -> 200                  ← no longer interrupted
+         GET /dashboard -> 200
+         GET /recovery  -> 200                  ← still reachable, permanently
+```
+
+**The backups-area entry point SURVIVES** ✅ — `/backups/remote` still carries exactly one `/recovery`
+link and the sealed-package sentence, before and after:
+
+> „A korábbi, házon kívüli mentéseid visszaszerezhetők… **Adatok visszaszerzése**"
+
+That asymmetry — full page silenced, entry point permanent — is what the handler documents, and it
+holds. **And the unlock still works after postponing** (measured: HTTP 200, 3.214 s, a real unseal).
+
+
+### F8 — the controller restarted mid-unlock — **state PASS, message FAIL**
+
+Injected: an unlock started, then `docker restart felhom-controller` at **T+0.7 s**, inside the
+~1.1 s unseal window. **Control: the container's `StartedAt` moved `2026-08-05T15:57:04Z` →
+`2026-08-05T21:06:05Z`.** The restart really landed mid-request.
+
+What the customer got:
+
+```
+HTTP 502 · elapsed 1.914 s ·  body: "Bad Gateway"
+```
+
+- **No half-state** ✅ — after the restart the four `/data/offbox` files are byte-identical with
+  **mtimes still 14:57:35**, and `settings.json` differs from the pre-F8 snapshot in exactly one
+  field, `postponed: True`, which F6 set. The controller returned **healthy in 40 s**.
+- **„the screen says which" — it does not** ❌. The customer is shown **`Bad Gateway`**: a raw
+  upstream error, in **English**, from traefik. It names no reason, offers no action, and says
+  nothing about whether the key was installed. **I3 breach** — *"every refusal names a reason a
+  person can act on, in Hungarian, with no raw error."*
+
+> **What F8 does NOT establish, stated so the pass is not overclaimed.** On this venue the unlock
+> resolves to `RecoverUnchanged` (the recovered password is byte-identical to the one already on
+> disk), so `InjectOffboxPassword` — the only writing branch — **was never reached**. F8 therefore
+> shows that a mid-request restart leaves state coherent and returns a raw 502; it does **not**
+> establish that the password *write* is atomic. Testing that needs shape (a), which the venue does
+> not hold. Recorded as a gap, not as a pass.
+
+#### §4.1 again — and this time the box says it out loud
+
+The brief expected the floor positive here, and F8's restart delivered a better one than `/settings`:
+a **cold-started process** stating the floor it was served, ten seconds in. The same log line, same
+box, same code, before and after the vouch:
+
+```
+15:58:35  [offsite-apply] settle-gate: GO — floor still unknown after 1m30s      ← hold IN FORCE
+...
+21:06:06  [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK)
+21:06:16  [offsite-apply] settle-gate: GO — at/above floor 0.200.0 (we are 0.201.0),
+                                        no managed update running                 ← hold RELEASED
+```
+
+**§4.1 is therefore measured twice, independently**: the rendered `GetFloor()` value, and this
+box-side line from a fresh process that had no prior in-memory state to inherit.
+
+---
+
+## The finding F6 turned up on the way past — the store reports 0 when it cannot read
+
+Not one of the eleven; found while checking F6's *"the listing is coherent"* clause on
+`/backups/remote`. It is reported because a suspicion checked and confirmed is a result.
+
+The page renders, **on one screen**:
+
+```
+Tároló méret · 0 pillanatkép                    Tárhelykeret: 0 / 50 GB (0%)
+```
+
+and, immediately below:
+
+> „**A távoli tároló másik kulccsal készült mentéseket tartalmaz.** A távoli tárhelyen lévő mentések
+> egy korábbi, már nem elérhető kulccsal készültek… **A meglévő mentések nem sérültek**…"
+
+**Ground truth, measured directly against the Storage Box over SFTP — a read-only listing, no
+decryption, using the box's own transport credential:**
+
+```
+/home/felhom-repo/snapshots:
+  -rw------- 253  Aug  5 13:14  f3d9cd67d539c00359f0454ea7a782f45beac8406c55275bee5a6886afa8d791
+/home/felhom-repo:  du -s → 12535 (KB)
+/home/felhom-repo/keys:  exactly ONE key
+```
+
+That is **snapshot `f3d9cd67`** — the Phase 0 snapshot holding all three sentinels, byte-size and
+timestamp matching the journal's `repo_size_bytes 12 611 522` and its `13:13:32Z` capture.
+
+> **One snapshot and ~12.5 MB of the customer's only surviving copy, rendered as `0 pillanatkép`
+> and `0 / 50 GB`, on the page the customer opens to find out whether their backups still exist —
+> directly above a paragraph telling them the store contains backups.**
+
+**Mechanism, from the box's own state rather than inferred.** After the Phase 3 rebuild the
+controller's `settings.json` `offbox` block carries **no `snapshot_count` and no `repo_size_bytes`
+key at all** (the full block is quoted in the pre-flight section above). The values are *unknown*,
+and unknown renders as the zero value.
+
+This is **R-217's defect class in a second location**: a field whose zero value is indistinguishable
+from a real measurement, defaulted past on a failure/unknown path. `OffsiteInventory.Empty` exists
+precisely because *"len(Apps)==0 is also what a failed read looks like"* — the same reasoning applies
+to a snapshot count the box has never successfully taken. **I6**: an absence reported as a fact.
+
+**I5 checked and NOT breached** — the hub's `/offsite` page shows `Campaign 11 · 0.0 GB · 50 GB`, but
+that is 12.5 MB rounded to one decimal of a GB, and the pool total (`Used 3.8 GB`) is consistent. The
+hub and the box do not disagree; **both** understate, for different reasons. Only the box's
+**snapshot count** — an integer, not a rounding — is false.
+
+
+### F11 — the box offline for a whole reporting window — **PASS**
+
+Injected: `qm shutdown 321` at **23:07:45**, restored **23:29:12** (≈21½ min). Control: `qm status` →
+`stopped`, and the last report before it was 23:06:12.
+
+**Both directions fired, and each sent an operator mail:**
+
+```
+23:28:11  Host staleness: c11-36d660 ok → stale (host_stale)      + Operator email SENT
+23:29:59  Received report from c11 (2514 bytes)                    ← the box returns, unaided
+23:30:11  Host staleness: c11-36d660 stale → ok (host_recovered)   + Operator email SENT
+23:30:11  Recovery host_recovered for c11: customer mail skipped — no unanswered customer down
+          mail (pairing miss)
+```
+
+- **The dead-man's switch fires and clears itself.** `ONLINE → STALE → ONLINE`, hub-side, with no
+  operator action.
+- **The customer mail was correctly withheld** — pairing-gated, exactly as the power-outage audit
+  records. The gate held under a real outage.
+- **I5 holds**: after the return the box's `offbox` block and the hub's view agree (nothing changed
+  across the outage, and both say so). Judged **after** a full report cycle rather than from a single
+  read — the Phase 0 lesson.
+
+**What was NOT reached, stated rather than implied:** the box never went to `DOWN` (>1 h) — the
+outage was ended once STALE and the recovery mail had both fired, because Phase 4 needed the venue
+back. The `STALE → DOWN` escalation is untested.
+
+### F7 — „I do not want the old data", then a change of mind — **the set-aside PASSES, the afterwards does NOT**
+
+**Pre-state, measured on the Storage Box** (read-only SFTP listing, no decryption):
+
+```
+/home           felhom-repo (mtime Aug 5 13:13)          ← exactly one repo
+/felhom-repo/snapshots   f3d9cd67d539c003…afa8d791       ← the Phase 0 history, 3 sentinels
+/felhom-repo             du -s → 12535 KB
+```
+
+**Confirmation 1** (`/recovery?setaside=1`) states all four consequences before anything happens:
+
+> „a korábbi mentéseket **félretesszük — nem töröljük**; a helyreállítási kód nélkül **többé nem
+> lesznek megnyithatók**; a gép **új, üres mentési tárolót kezd**; **ez az oldal többé nem jelenik
+> meg**."
+
+**Confirmation 2** (`POST /backup/offbox/reset` `confirm=1`) → **302**, flash
+*„Új távoli mentés indítása folyamatban — a régi előzmény félretéve (nem törölve)."*
+
+```
+[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
+       move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260805, then re-init
+```
+
+**NOTHING WAS DELETED — the assertion, verified against the remote:**
+
+```
+/home   felhom-repo                     (mtime 21:32)   ← fresh, re-inited
+        felhom-repo.orphaned-20260805   (mtime 13:13)   ← UNCHANGED mtime
+/felhom-repo.orphaned-20260805/snapshots   f3d9cd67…    ← the Phase 0 history, still there
+/felhom-repo.orphaned-20260805             du -s → 12535 KB   ← byte-for-byte the pre-state
+/felhom-repo/snapshots                     (empty)
+```
+
+**Set aside, never deleted. PASS**, and the two confirmations are honest about what they will do.
+
+#### The change of mind — and this is where it fails
+
+The customer returns wanting the old data. What they find:
+
+```
+GET  /recovery         -> 302 /backups/remote      ← the screen is GONE
+GET  /launcher         -> 200                      ← no interruption
+POST /recovery/unlock  -> 302, 0.028 s             ← the handler's gate; NO unseal, NO message
+```
+
+The brief's expected answer is *"message 4, not a typing accusation"*. **There is no message at all** —
+which is not a typing accusation, but is also not an explanation.
+
+**And the set-aside history is invisible.** The box knows exactly where it is:
+
+```json
+"orphaned_renamed_to": "/home/felhom-repo.orphaned-20260805"
+```
+
+`/backups/remote` after the set-aside contains **zero** occurrences of `orphaned-20260805`,
+„félretéve", „régi előzmény" or any equivalent (instrument controls: `felhom-repo` → 2,
+„letétbe helyezve" → 1, so the page and the matcher both work). And a census of the whole web layer:
+
+```
+grep -rn "OrphanedRenamedTo" internal/web/templates/ internal/web/*.go   →   (no hits)
+```
+
+> **The field is written and read by nobody.** 12.5 MB of the customer's retained data sits on the
+> Storage Box at a path the box records and never shows. The only mention it ever gets is a flash
+> message on the redirect, which is gone on the next click.
+
+This is the project's own recurring **"seam built but never wired"** pattern, and it lands on the one
+promise the set-aside screen makes — *we set it aside, we do not delete it*. → **R-228**
+
+*(Correct and recorded as a pass: the recovery screen retiring itself is exactly what confirmation 1
+promised — „ez az oldal többé nem jelenik meg" — so its disappearance is declared, not sprung.)*
+
+### F9 — a box that never had off-site backups — **the gate PROVEN LIVE, on a narrower shape**
+
+F9 as literally specified needs a box that never had off-site backups. **The venue cannot be that box
+without being rebuilt, which the brief forbids before Phase 4** — so the literal precondition was not
+staged.
+
+**But the assertion that FAILED in Phase 1 was tested and it now passes.** Phase 1's failure was not
+about the precondition; it was that `GET /recovery` **never asked the predicate** and rendered its
+story regardless (R-215). F7's set-aside drives `recoveryOffer()` false (a repository password exists
+and the tier is no longer orphaned), and:
+
+```
+GET /recovery  ->  302  /backups/remote        ← the page consults the predicate
+POST /recovery/unlock -> 302, no unseal        ← the POST sibling was already gated
+```
+
+**R-215's fix is proven live.** What is NOT covered: the `GetHubEscrowIdentityPresent()==false` arm
+(a box the hub holds nothing for). Recorded as partial, not claimed as full.
+
+### F10 — an app with a mandatory data path missing — **COULD NOT BE INJECTED. Three attempts, each with its control.**
+
+Setup, through the customer's own endpoints:
+
+- **R-220 reproduced live, a third time** — `GET /api/disks/candidates` returned
+  `initialize: [], attach: []` while both drives sat mounted at **both** `/mnt/felhom-drives/<name>`
+  and the raw `/mnt/<name>`. **Its red-proof reproduced too**: unmounting only the raw mounts flipped
+  the scan to both drives with `fstype: ext4, data_bearing: true`. *(Deviation: an operator action on
+  the appliance host, unavoidable while R-220 is open, and it doubles as the red-proof.)*
+- Both drives re-enrolled via `POST /api/storage/attach` → `{"registered":true,"where":"/mnt/felhom-drives/adatok"}`.
+- `calibre-web` deployed via `POST /api/stacks/calibre-web/deploy` (**202**, the same 409→202 flip
+  Phase 1 recorded), healthy in 39 s, bound at `/mnt/felhom-drives/adatok/userdata/media/books` —
+  its declared-mandatory path.
+- Baseline off-site run: **`last_status: ok`, 1 snapshot**, log `backed up calibre-web (…, 1 mandatory
+  path(s))`.
+
+> **A clean confirmation of R-225's mechanism, taken here:** after this first *successful* run
+> `settings.json` gained `"snapshot_count": 1` and `"repo_size_bytes": 27371`. Those keys were
+> **absent** before it — which is exactly why the page rendered `0 pillanatkép · 0 GB`.
+
+**The three injection attempts:**
+
+| # | method | control | why it failed |
+|---|---|---|---|
+| 1 | `mv` the directory aside, app running | ABSENT confirmed | the **app** recreated it (mtime 21:45) before capture |
+| 2 | `docker stop calibre-web`, then `rm -rf` | ABSENT confirmed, held 25 s | the **controller's monitor auto-restarted the stack**; the app recreated it |
+| 3 | stop the stack **through the controller API**, then `rm -rf` | ABSENT confirmed, held 25 s | the directory reappeared at **21:51:38.65** — coincident with the run's own start (21:51:38) — and the stack was found `Up` again a minute later |
+
+All three runs reported **`ok`** with `1 mandatory path(s)`, and **the R-203 stat-gap WARN
+(`mandatory data path missing on disk, skipped from offsite`) never fired** — correctly, because by
+the time `os.Stat` ran the path existed again.
+
+**Verdict: harness, not product. F10 is recorded as NOT INJECTED**, because the state it describes is
+not reachable on a *deployed* app of this kind: the app (or the monitor that keeps it up) recreates
+its own data directory within a second, every time.
+
+**But one observation is worth keeping, with its evidence.** At capture time the directory held:
+
+```
+metadata.db  metadata.db-shm  metadata.db-wal        ← recreated, 21:51
+(no F10-SENTINEL.txt — the customer's file, written at 21:39)
+```
+
+**The run reported `ok`.** The verdict is about the *presence* of a declared path, not its *content* —
+which is correct as designed and could hardly be otherwise — but it means **an `ok` off-site run can
+immediately follow the loss of everything that path contained.** Not filed as a defect; recorded
+because "the run said ok" is exactly the reassurance R-203 was filed to make trustworthy, and this is
+its edge.
+
+*(The customer's sentinel was restored into the live library afterwards and the harness copy removed.)*
+
+---
+
+## Venue state at the end of Phase 2 — **WORKING**
+
+| | |
+|---|---|
+| Containers | `calibre-web` healthy · `filebrowser` healthy · `felhom-controller:0.201.0` healthy · `traefik` up |
+| Drives | both enrolled; backup target `{"degraded":false,"known":true,"label":"mentes","target":"felhom-backup"}` — the I5/I6 baseline restored |
+| Off-site | fresh repo, `last_status: ok`, `last_success 2026-08-05T21:52:43Z`, 1 snapshot, 26.4 KB, `escrow_state: escrowed` |
+| Set aside | `/home/felhom-repo.orphaned-20260805` — the Phase 0 history, 12 535 KB, untouched |
+| Recovery screen | retired (correct — the set-aside declared it would be) |
+| Deviations left in place | the raw `/mnt/adatok` and `/mnt/mentes` mounts remain **unmounted** (R-220's workaround); the stable `/mnt/felhom-drives/*` mounts are what everything uses |
+
+
+---
+
+# Phase 4 — leave it alone and watch
+
+**Window: 2026-08-05 23:56 CEST → 2026-08-06 ~04:45 CEST.** The venue is not touched: no endpoint is
+driven, no service restarted, nothing configured. Only logs and the hub are read.
+
+## Pre-registered expectations — written BEFORE the window, so the comparison is not fitted afterwards
+
+**A soak that reports only "nothing broke" has measured nothing.** Both directions are named here in
+advance. Sources: the box's own `[scheduler] Registered …` lines and the agent's own `backup tier
+armed` lines, read at 23:56.
+
+### The box's own schedule (controller, restarted 23:29:53 CEST after F11)
+
+| when | job |
+|---|---|
+| **02:30** | `db-dump` |
+| **03:30** | `tier2-backup` · `fill-watch` |
+| **04:00** | `metrics-prune` |
+| **04:15** | `offbox-backup` ← **the one this campaign cares about** |
+| every 15 m | `hub-report` |
+| every 1 m / 5 m / 2 m | `agent-channel-health` · `system-health` · `stack-scan` |
+| every 10 s / 30 s | `status-refresh`, `health-probes` · `deadapp-check`, `ring-spill` |
+| every 6 h | `selfupdate-check` · `disk-health-check` |
+
+### The agent's schedule (host domain)
+
+```
+backup tier armed  target=felhom-backup  cadence=24h    keep_last=3  primary=true
+backup tier armed  target=felhom-pbs     cadence=168h   keep_last=0  primary=false
+backup: restore-test scheduler starting (per-archive due-check)  eval_interval=6h  settle=24h
+```
+
+### MUST happen (an absence here is a finding)
+
+1. Reports keep arriving every ~15 min; the host stays **ONLINE** all night.
+2. **`db-dump` at 02:30**, **`tier2-backup` + `fill-watch` at 03:30**, **`metrics-prune` at 04:00**.
+3. **`offbox-backup` at 04:15** runs unprompted and succeeds — `snapshot_count` **1 → 2**,
+   `last_status: ok`, `last_success` stamped in the 04:15 window.
+4. A **`backup_run_digest`** event reaches the hub after a backup run (this is the "operator digest";
+   it is event-driven per run, **not** a nightly cron — checked in the hub source rather than assumed).
+5. The agent's **`felhom-backup`** tier (24 h cadence) takes a whole-guest archive when it comes due.
+
+### MUST NOT happen (firing here is equally a finding)
+
+1. **No `host_stale` / `host_recovered`** — the box is up and staying up.
+2. **No `offsite_selfheal_restaged`** — the box has a target, so `needsOffsiteCredential` returns
+   false and it must not declare. *(This is also §4.2's negative control: if the box DOES declare
+   tonight with a target present, R-218's fix is wrong in the other direction.)*
+3. **No `offbox_repo_orphaned`** — the repo is fresh and written under the key the box holds.
+4. **No `offsite_repo_key_changed`** — nothing re-escrows tonight.
+5. **No `escrow blob SERVED`** — nobody is driving a recovery.
+6. **No controller self-update** — `auto-update: false`, and the box is at 0.201.0 with floor 0.200.0.
+
+### Genuinely uncertain, recorded as open questions rather than predictions
+
+- **The restore-test** evaluates every 6 h with a **24 h settle**, and every archive on this box is
+  hours old. **Expected NOT to run** — and if it does not, that is correct behaviour, not a gap. The
+  distinction matters because "no restore-test fired" would otherwise read as a failure.
+- Whether the agent's local tier fires tonight depends on when its 24 h cadence last elapsed on a box
+  built today.
+

hub/CHANGELOG.md

@@ -1,3 +1,381 @@
+## v0.97.1 — the held-floor REASON must match the hold's CAUSE (2026-08-05, CAMPAIGN-11 follow-on)
+
+> **Entry added retroactively 2026-08-05 (CAMPAIGN-11 Phase 2).** The change shipped and **is** the
+> deployed image (`felhom-hub:0.97.1`, commits `a7f1d27` + `79e31ac`), but it was written into the
+> **v0.97.0** entry instead of its own — so a reader matching the running tag against this file found
+> no `v0.97.1` heading at all. Second occurrence of a version deployed without its own entry (the
+> first was agent `0.90.1`). Nothing about the change is new here; only its heading is.
+
+Adding a second hold reason and leaving both surfaces printing the first is `CLAUDE.md`'s corollary,
+and it happened here: the freshly deployed hub logged
+
+```
+managed floor HELD for c11: agent "0.125.0" < MinAgent 0.113.0
+```
+
+— a comparison that is **FALSE** (0.125.0 is *above* 0.113.0) for a box held because its floor sat
+above the vouched golden. Both the ACK log line and the Hosts-dashboard `HeldReason` now come from one
+`ManagedFloorDecision.HoldReason()`, with `RequestedFloor` kept so the message can name the floor that
+was asked for (`Floor` is cleared to `""` on a hold, and a reason naming an empty floor tells the
+operator nothing). `TestResolveManagedFloor_HoldReasonMatchesTheCause` pins each reason to its cause.
+**A true alarm that misdescribes itself is one the operator learns to dismiss.**
+
+**Confirmed live in CAMPAIGN-11 Phase 2**: the corrected sentence is what the hub logged for c11 every
+15 minutes until the golden rebake released the hold —
+*"held: floor 0.200.0 is ABOVE the vouched golden 0.192.0, so its agent requirement is unknown — vouch
+a golden carrying the floor's controller (publish-train rule 1)"* — and it stopped, as it should, once
+golden 0.201.0 was vouched.
+
+## v0.97.0 — the floor stops being served past the agent it depends on (2026-08-05, CAMPAIGN-11)
+
+**R-216, the hub half.** `ResolveManagedFloor` exists to *"never push a controller past the agent it
+depends on"* — its own comment says so — and it compared the box's agent against
+`ArtifactManifest.MinAgent`, which by ITS own comment is *"the MINIMUM host-agent version this
+GOLDEN's controller requires"*. Those two sentences contradict each other, and
+`publish-train-rules.md` rule 3 states the rule about the **floor's** controller, not the golden's.
+
+They are the same number only while the floor sits at or below the golden — the arrangement rule 5's
+ISO gate assumes. Raise a floor above the vouched golden (which the day-0 runbook explicitly
+recommends after a golden rebuild, and which a per-customer override makes trivial) and the guard
+compares against a version it is not serving.
+
+**Measured live 2026-08-05 (CAMPAIGN-11 Phase 1):** golden 0.192.0 with MinAgent 0.113.0, a
+per-customer floor of 0.200.0, a box on agent 0.120.0. `0.120.0 ≥ 0.113.0`, so the floor was served
+and the box was pushed onto a controller needing agent 0.125.0 — whose customer was then told their
+correct recovery code was wrong. **This is the tenth entry in `CLAUDE.md`'s comment-vs-code table, and
+the first where the false invariant was a guard rather than a comment alone.**
+
+**The fix, and why nobody types a number twice:** the hub cannot know the agent requirement of a
+controller version it was never told about, so it must not pretend to. A floor ABOVE the vouched
+golden is now **HELD**, with its own reason (`HeldBeyondGolden`), reusing Part D's existing dashboard
+visibility — a held box is visible, never silently stale. The vouched MinAgent keeps its exact
+meaning; the guard simply stops applying it to versions it does not describe. The operator's remedy is
+the one the publish train already prescribes: vouch a golden carrying the floor's controller
+(rule 1, *manifest before floor*).
+
+⚠ **What this does NOT do:** with the guard corrected but the day-0 agent unchanged, a new box is
+**held, not served**. It stops being lied to; the recovery feature still does not work for it until
+agent 0.125.0 is vouched in the Day-0 manifest. That is an operator decision, not a code change.
+
+Unchanged: an uncoupled release (no MinAgent vouched) is served exactly as before, above the golden or
+not — a fleet must not be frozen by a comparison that now over-fires. An unparseable golden degrades to
+the previous behaviour rather than gating.
+
+**R-222.** The report ACK's `escrow` object gains `superseded_present` / `superseded_at`: whether the
+hub is ALSO keeping an EARLIER sealed package for this host, and when it was set aside. It exists for
+one message — the recovery screen cannot otherwise tell a genuinely wrong code from a code that is
+RIGHT about a retained earlier package, because the unseal fails closed against the current package
+either way. **Only rows that actually carry an identity blob count**: the pre-v0.93.0 rows are NULL and
+retain nothing the screen could be talking about. It is a boolean and a timestamp; it grants **no read
+path**, which is still unbuilt (R-199's inventory).
+
+Tests: `managed_floor_test.go` — the campaign's exact numbers, held-regardless-of-agent, Scenario C
+(uncoupled untouched), floor==golden unchanged, unparseable golden. **Red-proof: removing the
+floor-above-golden branch reproduces the campaign's measurement exactly.**
+
+## v0.96.0 — the hub answers a rebuilt box's request (2026-08-05, R-204 item 4 / R-193 / R-192)
+
+The hub half of the last manual intervention the 2026-08-04 drill needed. The box declares
+(controller v0.199.0); this answers.
+
+### `internal/offsiteheal` — the sibling of `pbsdrheal`
+
+Same shape, same restraint: **declared states only**, a debounce, **restage → escalate**, a distinct
+audit event per remediation, and a healthy box is a pure no-op. It never blind-timer-reissues and
+never re-runs a provisioning step.
+
+**The debounce is TWO distinct reports, derived rather than chosen.** The controller reports every
+~15 minutes, so two distinct declarations mean the state survived a full report cycle — a restart, a
+slow first report or a transient config read all resolve well inside one. One report would act on a
+blip; three would leave a stranded customer waiting ~45 minutes for the one thing they cannot obtain
+any other way. The reconciler's own tick (5 min) is deliberately faster than the report cadence so it
+adds no latency of its own; the debounce counts **fresh evidence, not ticks**.
+
+**§8.4 — is there a deliberately-unhealed state, as `pbsdrheal` has in `verify_failed`?** Yes, and it
+is excluded **by construction** rather than by a case in the switch — worth stating plainly, because
+"there is nothing like that here" is usually wrong. The analogue is the **regressed** shape: a box
+that had a working tier and lost its target while still holding its repository password. Re-arming a
+credential would not help it. It cannot reach this reconciler at all, because the controller's
+declaration predicate requires the repository password to be **absent**.
+
+### Restage before mint — and the fact it rests on
+
+`store.RestageOneTimeSecret` clears the consumed flag without touching the value, mirroring
+`RestageHostPBSSecret`. **That the value survives a consume was established from this schema and from
+`ConsumeOneTimeSecret` — which stamps `consumed_at` and nothing else — not inherited from the PBS
+analogy.** The two secrets are different objects with different lifecycles, and assuming a shared
+shape is how two earlier sessions confused the credentials. `TestRestageOneTimeSecret_ReArmsTheSameValue`
+asserts the **same** value comes back, so a future "hardening" that cleared the column would fail
+loudly instead of silently turning every rebuild into an external mint.
+
+### `reportHasOffsite` tightened — the ninth comment-vs-code entry, caught before it shipped
+
+Its comment asserted *"the report builder attaches `offsite` only when the box actually has an offbox
+target configured, so presence == applied-on-the-box"*. Controller v0.199.0 breaks that premise
+deliberately. Left as bare presence, this would have read a **request for help** as proof the tier was
+**applied** — making `DeliveryStateFor` return `DeliveryApplied` for exactly the stranded boxes, and
+suppressing their stuck event. It now requires `enabled:true`, which is **provably a no-op for every
+report shape that existed before** (an attached object has always carried `enabled:true`), and the
+equivalence is measured in `TestReportHasOffsite_EnabledOnly` rather than argued.
+
+### R-192's guard half — CLOSED BY REPLACEMENT, not repaired
+
+The delivery checker decided the shape by counting how many of the **oldest 500** reports after a
+consume carried an offbox target. On demo-hp all 500 predated the rebuild, so it confidently reported
+the *regressed* shape and declined to heal — for 108 reports, while the box sat stranded. A
+declaration needs no window, no count and no inference, so it **outranks both inferred shapes**
+(`shapeDeclared`), the stuck e-mail says what was declared instead of prescribing Re-issue, and
+`maybeHeal` **stands down with a record** because `offsiteheal` owns the remediation. Two mechanisms
+healing one customer would double-issue — and this one can only mint, so it would also skip the cheap
+path.
+
+### What is deliberately NOT automated
+
+**No escrow ceremony is ever run or requested.** A credential is replaceable; the recovery code is
+not, because only the customer holds it. **Credential automatic, key customer-present.**
+
+**On security, weighed rather than skipped:** a re-issued credential is transport only. Anyone able to
+forge the declaration already controls the box, and the box already holds its own installed access
+key. The data stays sealed under the recovery code throughout.
+
+### Rider — the pre-push hook refuses a clone outside the workspace
+
+One assertion, identical in all four repos. See the controller's v0.199.0 entry.
+
+## v0.95.0 — a re-issue no longer marks a healthy escrow stale (2026-08-05, R-196 / R-204 item 2)
+
+`ReissueCredentials` called `MarkEscrowStale` on every re-issue that found an escrow row. The mark was
+**precautionary, not evidential**: it guessed that the box's re-apply MIGHT mint a fresh repository
+password. On the ordinary re-issue shape — a box that still holds its `<DataDir>/offbox/repo_password`
+— the password does not change, so it marked a HEALTHY escrow stale. The 2026-08-04 recovery drill
+(R-201) is what promoted this from a nit to a blocker.
+
+**What the mark actually did**, because "asks for an unnecessary ceremony" understates it:
+
+1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report ACK.
+2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed.
+3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → **every off-site backup
+   refused, indefinitely**, on a box whose key was never in doubt.
+4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
+   supersedes the sealed blob. During a recovery that is the one act that would have destroyed the key
+   just recovered.
+
+A precautionary flag that stops the data protection it is guarding is not conservative.
+
+**Removing it leaves no gap — the case it guessed at is measured elsewhere, and better.**
+
+- **Continuous, box-side:** the controller compares the ACK's sealed hash against its CURRENT local repo
+  password on every report ACK (`report.EscrowAutoConfirmer.reconcileEscrowed`, the Scenario-F
+  re-check). In the guest-rebuild shape — the only shape where a re-issue is followed by a fresh
+  repository password — that mismatches within one report cycle and raises the stale card plus the
+  „create a new recovery code" CTA. Pinned by the controller's `TestEscrowStale_MismatchWarnsOnceAndFlags`.
+  **And the mark was BLINDING it:** by emptying the hash (step 1) it removed the very value that
+  comparison needs, leaving the box able to report only the hash-LESS reason, which is false.
+- **Edge-triggered, hub-side:** R-197's `offsite_repo_key_changed` fires on a proven hash difference
+  across a supersession and pages the operator. Pinned by `TestEscrowPut_ChangedRepoKey_RaisesSignal`;
+  red-proved by removing the `maybeEmitRepoKeyChanged` call.
+
+`offsite_reissued` is unchanged and still always fires — the customer must still learn that the
+credential moved. Only the stale marking and its `escrow_stale` event are gone.
+
+**`MarkEscrowStale` is kept, not deleted, and now has no caller** — deliberately. The `stale_at` flag
+is still live and correct (read by the ACK, the operator card and the PBS-DR view); the right way to
+set it is a future caller that has MEASURED a key change rather than guessed at one. Its doc now says
+so, and `TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying while
+nothing writes it.
+
+**Disagreement recorded** (the R-96 standing rule): the task's Scenario D asked that a real key change
+also "mark the escrow stale". It must not, and nothing was changed to make it — the hub learns of a
+real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow is freshest.
+Marking it stale there would ask for a ceremony to fix the ceremony that just ran. The correct
+consequence at that instant is the operator alarm, which is what R-197 already does.
+
+`TestReissue_InvalidatesEscrow` is replaced in place by `TestReissue_DoesNotMarkAHealthyEscrowStale`,
+its exact inverse, so the reversal is visible in `git log -L`. The old test was not wrong about the
+effect it measured; it was wrong about the cause, and it pinned the defect.
+
+## v0.94.0 — a box can fetch its own sealed recovery package (2026-08-04, R-199 link 6)
+
+**Chain link 6 had no client.** The hub has served the identity blob since slice 10D, from
+`handleReEnroll` / `handleGetRestoreDirective` — gated on operator-armed recovery mode and the global
+key. Nothing in the agent, the hub UI, any script or any runbook ever called either. The only
+documented retrieval was `sqlite3 … writefile(…)` by hand on a `kubectl cp`-ed database.
+
+**`GET /api/v1/hosts/{host_id}/escrow`** — the box-authenticated MIRROR of the PUT that put the blob
+there. Self-scoped: a per-host key reads only its own; the global key may read any, the same asymmetry
+the PUT has. A host with no sealed bundle gets `200 {present:false}` — a clean answer, because a 404
+is indistinguishable from an unknown host and a bare empty 200 from a zero-length blob, and neither
+of those is what "no ceremony has run yet" means.
+
+**THE TRADE, RECORDED IN THE HANDLER RATHER THAN INFERRED.** Before this, obtaining the blob required
+the OPERATOR to arm recovery mode. Now whoever controls a rebuilt box can obtain it with that box's
+own credential. That is a real reduction in the number of parties required, and it is the whole of the
+trade — they still cannot open it, because the hub has never held R and a wrong code fails closed at
+age's scrypt KDF. **The mitigation is that the capability is audited rather than silent:** every
+successful retrieval raises `escrow_blob_served` (warning, operator-only), recorded before the bytes
+leave. A silent capability on this object is the shape the last two weeks were spent removing.
+
+`escrowSelfServiceRetrieval` is a single named constant — the §8.2/§8.3 decision point. Flipping it to
+false re-imposes the recovery-mode requirement on this endpoint and changes nothing else, so the
+operator can overrule the trade at the cost of a boolean rather than a redesign.
+
+**The operator-driven DR path is untouched** — same gate, same behaviour, pinned by a test that
+exercises re-enroll and restore-directive with recovery mode off and on. Red-proofs observed: removing
+the ownership check makes a cross-host read succeed (host A served host B's blob); removing the audit
+record makes the retrieval silent.
+
+**Not in this release:** the customer-facing flow. No card, no form, no preview — those are designed on
+ground that has been walked, and R-200/R-201 are that walk.
+
+## v0.93.0 — the retention keeps the key it was built to keep, and three things stop lying (2026-08-04, R-198/R-197/R-196/R-192)
+
+### R-198 — the superseded-escrow retention was preserving the wrong key, and the ceremony was destroying the right one
+
+`host_escrow_superseded` shipped in v0.60.0 with a `blob` column — the **K-escrow**, i.e. the PBS
+datastore encryption key. `identity_blob` was added to `host_escrow` **later**, by the slice-10D
+`ALTER TABLE`, and **never to the retained table**. The offsite **restic repository password** — the
+data key for every off-site app-data snapshot — lives inside `identity_blob`. So
+`demoteCurrentEscrowTx`, whose own comment calls it *"THE ONE escrow row-copy routine"*, retained the
+whole-guest key and silently dropped the off-site data key: **precisely the secret the reinstall-orphan
+incident the feature was built for was about.**
+
+**What makes it the most serious defect this project has recorded is not the missing column — it is
+which act destroys the key.** The copy happens as the new blob overwrites the old, so the destroying
+act is the **escrow ceremony**: the exact thing a rebuilt box tells its customer to run, on a card
+that promises in Hungarian that the old backups stay recoverable with their recovery code. Both demo
+boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08), and their previous repository passwords
+are now unrecoverable whether or not anyone kept the recovery codes.
+
+- `host_escrow_superseded` gains `identity_blob BLOB` (in the CREATE for fresh databases **and** as an
+  additive `ALTER TABLE`, following the existing idiom; tolerated on re-run, changes no existing row).
+- `demoteCurrentEscrowTx` carries it — so **both** callers are fixed by one change: the re-escrow path
+  (`SaveHostEscrow`) and the host-delete demotion (`DeleteHost`).
+- `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob
+  is reachable from Go at all. `CountCurrentEscrowWithIdentity` is the census of who the fix protects.
+- The comment above the table now records that the ruling stated there was not met, and what it cost.
+- **Nothing is backfillable and the code says so:** rows superseded before this were written without
+  the identity blob and their source rows are already overwritten.
+
+**Tests assert the CONSEQUENCE, not the mechanism** — the pre-existing
+`TestSaveHostEscrow_RetainsSuperseded` was green throughout, because it asserted that a retained row
+exists with the old K-blob. `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still
+yield a repository password, and pins the load-bearing ordering (the identity blob is written **after**
+`SaveHostEscrow`, so the demote sees the previous generation). `TestDeleteHost_DemotesIdentityBlob`
+proves the shared routine through its **other** caller — a shared routine tested through one caller is
+how a fix gets believed on a path nobody exercised. Red-proofs: dropping the column from the copy fails
+both; fixing only the re-escrow caller fails the delete one while the re-escrow one passes.
+
+### R-197 — the hub held both halves of the evidence and compared them nowhere
+
+`restic_pw_sha256` is stored on the current escrow and carried to the retained row on every
+supersession. Comparing the two is what answered the 2026-08-04 recon's hardest question in one query —
+and nothing in the hub did it. demo-felhom's repository password changed on 2026-08-03, orphaning 36
+snapshots / 1.14 GB, and **no event, e-mail, card or log line said so for thirteen hours.**
+
+`SaveHostEscrow` now returns the hash it replaced, and `handleHostEscrowPut` raises
+**`offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession) when both
+hashes are known and differ. The in-between shapes are logged rather than dropped, so *"we chose not to
+alarm"* and *"the check did not run"* never look identical. **No hash value travels** — not in the
+message, not in the details; the event names the host, the customer and the retained count.
+
+**Severity chosen for the world this release creates, not the one it inherits:** before R-198 a changed
+key meant the previous history was unopenable by anyone, ever, which would have argued for `error`.
+From v0.93.0 the superseding ceremony retains the old identity blob, so the situation is *"this
+customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss.
+`warning` also routes: the dispatcher treats `info` as an intentional non-notify.
+
+### R-192 (partially) — the alert describes what was measured, and a refusal leaves a trace
+
+demo-hp e-mailed this daily from 2026-08-03: *"one-time password consumed 284h19m0s ago and 500
+report(s) since carry no offbox target — the credential is likely burned … Re-issue delivers a fresh
+one."* `OffsiteReportsSinceConsume` for that same call was **500**: every one of those reports carried a
+target. The message stated the precise negation of its own measurement and prescribed a remedy for a
+failure mode that had not occurred — while the self-heal correctly declined, **in complete silence**.
+
+- The one stuck state is now reported as the two situations it actually covers, **burned** and
+  **regressed**, each stating its own measurement and carrying its own recommendation (the regressed
+  text explicitly withdraws Re-issue and points at what removes an offbox target — a guest rebuild,
+  R-193). `offsite_reports_since_consume` is in the details for the first time.
+- **The counting window is named in the text.** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT
+  500` — the **oldest** 500 reports after the consume — so on a long-lived box the counts describe the
+  start of the window, not the present. **That guard defect stays OPEN (R-192)** because its correct
+  shape depends on the recovery chain that is not yet assembled (R-199/R-200/R-201); naming the window
+  is how it stays visible instead of being laundered into a confident sentence.
+- **Every self-heal refusal now leaves a record** — a `notification_log` row on the operator channel
+  with status `refused` and its reason, the R-182 suppressed-e-mail precedent. It rides the stuck
+  event's 24 h cadence, so it sits beside the e-mail it explains rather than accumulating per tick.
+  **The guard's logic is unchanged**: the set of situations in which the heal fires is what it was.
+- `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither
+  was ever registered, and neither has a `customerMessages` entry — which is **not** a block, so a
+  customer with a configured recipient was in line for an English e-mail about one-time passwords being
+  *"likely burned"*. Narrowing only; the operator channel is untouched.
+
+### R-196 — five comments corrected (the spec expected three)
+
+`ReissueCredentials` resets the **provider account** password. It does not and cannot touch the restic
+repository password, which is generated on the box and never leaves it except sealed under R. Five
+places in the hub said otherwise: `offsite/offsite.go` (the `MarkEscrowStale` justification),
+`api/handler.go` (the F3 re-enroll comment), and three in `store/store.go` (the `stale_at` ALTER, the
+`MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). **That false belief sent two separate
+investigations the wrong way in one day.** All five now say what the code does, name the correction and
+cite the recon; the staleness mark is documented as **precautionary**, not evidential. **The behaviour
+is unchanged and R-196 stays OPEN** — the false-staleness alarm it causes is a behaviour change and
+must not ride a comment-correction release.
+
+### Not in this release
+
+**R-199, R-200 and R-201 remain open** — the hub's blob-serving endpoints still have no client, the
+password-injection seam still has no form, and nothing in the recovery path has ever been exercised.
+**Part 5 hit its gate and the orphan card is untouched** — see R-202: a single ACK boolean can say
+whether a retained recoverable blob exists, but not whether one covers *the key this orphaned
+repository was written under*, and a conditional promise that is still capable of being false is worse
+on that surface than a hedged sentence.
+
+## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
+
+`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
+— `hosts=0`, `host_deletions=0`, `host_reports=0`, `reports=0` — and it e-mailed an
+`expected_dbdump_missed` **error** at 03:00 UTC on 2026-08-02, 08-03 and 08-04. Nothing has ever been
+expected of that customer, so the honest verdict is UNKNOWN, not missed.
+
+**The mechanism is the interesting half, because the guard that should have covered this is keyed off
+the wrong thing.** `CheckBackupDeadlines` already skips nodes that are `down`, and that skip reads
+`StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — **a query over the
+`reports` table**. A customer with zero reports is in no row, gets no state at all, and `GetState()`
+returns `""` rather than `"down"`. **The skip misses exactly the customer it would most obviously
+cover.** Corroborated on live data: `peti-felhom` is active with a host deleted 2026-07-15 and does
+*not* alarm, because it has 482 old reports and is therefore `down`. The backup half was already safe
+(`reportJSON == ""` → skip); the DB-dump half had no guard at all.
+
+- **`store.HasEverBoundHost(customerID)`** — a live `hosts` row **OR** a `host_deletions` tombstone.
+  Consulted once per customer at the top of the deadline loop; a false verdict skips the customer with
+  an INFO line and its own counter in the summary.
+- **The discriminator is "was a host EVER bound", NOT "has a report arrived", and that is the whole
+  design.** A box that was installed, bound and then went silent is a real fault and must keep
+  alarming — it has a `hosts` row, so it is judged. Only a customer that never had a machine is
+  UNKNOWN. The tombstone is included for the same reason: a customer whose host was deleted *had* one,
+  and that judgement belongs to the staleness checker, not to this predicate.
+- **Fail-OPEN on a read error.** An unreadable binding must never SUPPRESS a real alarm; it logs and
+  judges anyway.
+- **The anchored-verdict structure is untouched** (R-81/v0.73.0). This is the same invariant — absence
+  is UNKNOWN until an anchor elapses — applied one level up, at the question of whether there is a
+  subject at all.
+
+**Tests, including the one that matters.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`
+is a real shape — a machine installed and never phoned home — and it is indistinguishable from `david`
+on every signal except the one the guard reads. If the guard is ever "simplified" to key off report
+presence, customer age or a name pattern, it goes red. Plus a bound-then-went-quiet case, a
+deleted-host case, and the predicate's own three inputs. **Companion red-proof observed:** removing the
+guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` —
+verbatim the event `david` sent three mornings running. Restored after.
+
+**`david`'s record was not modified.** The record is correct; the alarm was what was wrong.
+
+**Shipped alongside a spike that deliberately shipped NO code** —
+`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md` (R-193/R-192): what is one-shot
+in the offsite path, and what a guest rebuild really costs. It ends in ranked options and a question
+for the operator; R-196 and R-197 were minted from it and are not implemented here.
+
 ## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
 
 **Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's

hub/cmd/hub/main.go

@@ -20,14 +20,15 @@ import (
 	"gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/mailrelay"
-	"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
-	"gitea.dooplex.hu/admin/felhom-hub/internal/pbsdrheal"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/offsiteheal"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/pbsdrheal"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/poke"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
-	"gitea.dooplex.hu/admin/felhom-hub/internal/web"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/web"
 	"gitea.dooplex.hu/admin/felhom-hub/internal/wgsync"
 	"gopkg.in/yaml.v3"
 )
@@ -290,8 +291,8 @@ func main() {
 	webServer := web.New(dataStore, cfg.Auth.PasswordHash, cfg.API.ReportAPIKey, Version, staleThreshold, logger)
 	webServer.SetTemplateFetcher(templateFetcher)
 	webServer.SetAssetManager(assetsMgr)
-	webServer.SetClaimEngine(claimEngine)      // v0.50.0 — Setup-tab claim chip + resend button
-	webServer.SetSelfBindMailer(dispatcher)    // v0.66.0 (R-27) — customer self-bind link button (sibling of claim mailer)
+	webServer.SetClaimEngine(claimEngine)   // v0.50.0 — Setup-tab claim chip + resend button
+	webServer.SetSelfBindMailer(dispatcher) // v0.66.0 (R-27) — customer self-bind link button (sibling of claim mailer)
 	// Day-0 artifact version dropdowns: let the operator pick a version and have the hub derive the
 	// sha256 from Gitea (no hand-copied checksums). Reuses the registry creds; degrades to manual text
 	// entry when they're absent.
@@ -510,6 +511,23 @@ func main() {
 		logger.Printf("[INFO] PBS-DR self-heal reconciler started (interval 5m)")
 	}
 
+	// Off-site credential self-heal reconciler (internal/offsiteheal, R-193 / R-204 item 4). The
+	// sibling of the block above, for the last of the four manual interventions the 2026-08-04 drill
+	// needed: a REBUILT box has no off-site credential because its predecessor spent the one-time
+	// password. It acts ONLY on a state the box DECLARES (an absence has four meanings and the hub
+	// cannot tell them apart; the box can), sustained across two distinct reports, re-staging the
+	// stored secret before ever minting. A healthy box is a pure no-op. OFFSITEHEAL_ONLY_CUSTOMER
+	// scopes a supervised first rollout (empty = whole fleet — the steady state).
+	{
+		offsiteReconciler := offsiteheal.NewReconciler(dataStore, offsiteheal.NewActions(dataStore, webServer), logger)
+		if only := os.Getenv("OFFSITEHEAL_ONLY_CUSTOMER"); only != "" {
+			offsiteReconciler.RestrictToCustomer(only)
+			logger.Printf("[INFO] offsite credential self-heal RESTRICTED to customer %s (supervised rollout scope)", only)
+		}
+		go offsiteReconciler.Run(ctx)
+		logger.Printf("[INFO] offsite credential self-heal reconciler started (interval 5m, debounce 2 reports)")
+	}
+
 	// Session cleanup — removes expired sessions every hour
 	go webServer.CleanupSessions(ctx)
 

hub/internal/api/escrow_get_test.go

@@ -0,0 +1,203 @@
+package api
+
+import (
+	"encoding/base64"
+	"encoding/json"
+	"net/http"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
+)
+
+// R-199 (hub v0.94.0) — the box-authenticated retrieval of a host's OWN sealed identity blob.
+// This is a new surface on the most sensitive object in the system; these tests exist to pin the
+// three properties that make it defensible: it is self-scoped, it is honest when there is nothing to
+// serve, and it is never silent.
+
+func seedEscrowedHost(t *testing.T, st *store.Store, hostID, customerID, apiKey string, identity []byte) {
+	t.Helper()
+	if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: apiKey}); err != nil {
+		t.Fatal(err)
+	}
+	if _, _, err := st.SaveHostEscrow(hostID, []byte("k-escrow"), "fp", "zero_knowledge", "2026-08-04T11:00:00Z", "SHA"); err != nil {
+		t.Fatal(err)
+	}
+	if len(identity) > 0 {
+		if err := st.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
+// The happy path: a host reads back exactly the bytes it uploaded, verbatim.
+func TestEscrowGet_ServesOwnBlobVerbatim(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	identity := []byte("\x00\x01age-wrapped-identity-bundle\xff")
+	seedEscrowedHost(t, st, "h1", "c1", "HKEY", identity)
+
+	rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
+	if rr.Code != http.StatusOK {
+		t.Fatalf("GET escrow = %d, want 200 (%s)", rr.Code, rr.Body.String())
+	}
+	var out struct {
+		HostID  string `json:"host_id"`
+		Present bool   `json:"present"`
+		B64     string `json:"identity_escrow_b64"`
+	}
+	if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
+		t.Fatal(err)
+	}
+	if !out.Present || out.HostID != "h1" {
+		t.Fatalf("unexpected envelope: %+v", out)
+	}
+	got, err := base64.StdEncoding.DecodeString(out.B64)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if string(got) != string(identity) {
+		t.Fatal("the served blob is not the stored blob — the hub must return ciphertext verbatim")
+	}
+}
+
+// Scenario C — a box can fetch ONLY its own blob.
+// RED-PROOF: delete the `!isGlobal && authHostID != pathHostID` check in handleHostEscrowGet →
+// cross-host retrieval succeeds → this FAILS. Without that line, any host key is a fleet-wide reader
+// of every customer's sealed bundle.
+func TestEscrowGet_CrossHostRefused(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	seedEscrowedHost(t, st, "hostA", "custA", "KEY-A", []byte("A-identity"))
+	seedEscrowedHost(t, st, "hostB", "custB", "KEY-B", []byte("B-identity-SECRET"))
+
+	rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "KEY-A", "")
+	if rr.Code != http.StatusForbidden {
+		t.Fatalf("host A reading host B's blob = %d, want 403 — a cross-host read of a sealed bundle "+
+			"must be impossible on every code path (body: %s)", rr.Code, rr.Body.String())
+	}
+	if b := rr.Body.String(); len(b) > 0 && (contains(b, "B-identity") || contains(b, base64.StdEncoding.EncodeToString([]byte("B-identity-SECRET")))) {
+		t.Fatal("the refusal body leaked the other host's blob")
+	}
+	// Unauthenticated is refused too, and does not leak which hosts exist.
+	if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "", ""); rr.Code != http.StatusUnauthorized {
+		t.Fatalf("unauthenticated GET = %d, want 401", rr.Code)
+	}
+	// The global operator key MAY read any — the same asymmetry the PUT has.
+	if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", globalKey, ""); rr.Code != http.StatusOK {
+		t.Fatalf("global key GET = %d, want 200", rr.Code)
+	}
+}
+
+func contains(hay, needle string) bool {
+	return len(needle) > 0 && len(hay) >= len(needle) && (func() bool {
+		for i := 0; i+len(needle) <= len(hay); i++ {
+			if hay[i:i+len(needle)] == needle {
+				return true
+			}
+		}
+		return false
+	})()
+}
+
+// Scenario D — a host with no sealed bundle gets a CLEAN answer: 200 with present=false. Not a 404
+// (indistinguishable from an unknown host), not an empty 200 without the flag (indistinguishable
+// from a zero-length blob). Both of those read as a fault to a caller whose situation is simply
+// "no ceremony has run yet".
+func TestEscrowGet_NoBlobIsCleanNone(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	seedEscrowedHost(t, st, "h1", "c1", "HKEY", nil) // K-escrow only, no identity blob
+
+	rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
+	if rr.Code != http.StatusOK {
+		t.Fatalf("no-blob GET = %d, want 200 with present=false", rr.Code)
+	}
+	var out struct {
+		Present bool   `json:"present"`
+		B64     string `json:"identity_escrow_b64"`
+	}
+	json.Unmarshal(rr.Body.Bytes(), &out)
+	if out.Present || out.B64 != "" {
+		t.Fatalf("a host with no bundle must report present=false and no bytes, got %+v", out)
+	}
+	// An unknown host is a DIFFERENT answer — the two must not collapse into one.
+	if rr := do(h, http.MethodGet, "/hosts/nope/escrow", globalKey, ""); rr.Code != http.StatusNotFound {
+		t.Fatalf("unknown host = %d, want 404 (distinct from a known host with no bundle)", rr.Code)
+	}
+}
+
+// Scenario E — every successful retrieval leaves a durable record naming the host.
+// RED-PROOF: remove the SaveEvent call in handleHostEscrowGet → no event → this FAILS. A silent
+// capability on this object is the whole reason §8.2's trade is acceptable at all.
+func TestEscrowGet_LeavesARecord(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
+
+	if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
+		t.Fatalf("GET = %d", rr.Code)
+	}
+	ev, err := st.GetLatestEventByType("c1", eventEscrowBlobServed)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if ev == nil {
+		t.Fatal("R-199: a sealed recovery blob was served and NOTHING recorded it — the audit row is " +
+			"the mitigation that makes this endpoint's trade acceptable")
+	}
+	if ev.Severity != "warning" {
+		t.Errorf("severity = %q, want warning: info is an intentional non-notify, so the operator would "+
+			"never hear that the recovery path was used", ev.Severity)
+	}
+	if !contains(ev.Message, "h1") {
+		t.Errorf("the record must name the host, got %q", ev.Message)
+	}
+	// The record must not carry the blob itself — it is an audit row, not a second copy.
+	if contains(ev.Message, "age-wrapped") || contains(ev.DetailsJSON, "age-wrapped") ||
+		contains(ev.DetailsJSON, base64.StdEncoding.EncodeToString([]byte("age-wrapped"))) {
+		t.Errorf("the audit record embedded the blob: msg=%q details=%q", ev.Message, ev.DetailsJSON)
+	}
+	// A "no blob" answer is NOT a retrieval and must not raise one.
+	seedEscrowedHost(t, st, "h2", "c2", "KEY2", nil)
+	do(h, http.MethodGet, "/hosts/h2/escrow", "KEY2", "")
+	if ev, _ := st.GetLatestEventByType("c2", eventEscrowBlobServed); ev != nil {
+		t.Fatal("a present=false answer served no blob and must raise no retrieval record")
+	}
+}
+
+// Operator-tier by construction, registered in the same commit that mints the type.
+func TestEscrowBlobServed_IsOperatorOnly(t *testing.T) {
+	if !notify.IsOperatorOnly(eventEscrowBlobServed) {
+		t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
+			"about their sealed recovery bundle being handed out", eventEscrowBlobServed)
+	}
+}
+
+// Scenario G — the operator-driven DR path is UNTOUCHED: same gate, same behaviour. The new sibling
+// must not loosen it, and must not share it.
+func TestEscrowGet_OperatorDRPathUnchanged(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
+
+	// re-enroll without recovery mode: still refused.
+	if rr := do(h, http.MethodPost, "/hosts/h1/re-enroll", "HKEY", `{"new_api_key":"x"}`); rr.Code != http.StatusForbidden {
+		t.Fatalf("re-enroll without recovery mode = %d, want 403 — the new endpoint must not loosen it", rr.Code)
+	}
+	// restore-directive without recovery mode: still refused.
+	if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusForbidden {
+		t.Fatalf("restore-directive without recovery mode = %d, want 403", rr.Code)
+	}
+	// …and arming it still works, i.e. the old path is functional, not merely closed.
+	if err := st.SetRecoveryMode("h1", time.Now().UTC().Add(10*time.Minute)); err != nil {
+		t.Fatal(err)
+	}
+	if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusOK {
+		t.Fatalf("restore-directive WITH recovery mode = %d, want 200", rr.Code)
+	}
+	// The new endpoint does NOT depend on recovery mode (that is the §8.2 trade, made explicit here so
+	// a future change to escrowSelfServiceRetrieval is visible as a test change).
+	if err := st.ClearRecoveryMode("h1"); err != nil {
+		t.Fatal(err)
+	}
+	if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
+		t.Fatalf("self-service escrow GET with recovery mode OFF = %d, want 200 (escrowSelfServiceRetrieval=true)", rr.Code)
+	}
+}

hub/internal/api/escrow_repo_key_changed_test.go

@@ -0,0 +1,146 @@
+package api
+
+import (
+	"encoding/base64"
+	"encoding/json"
+	"net/http"
+	"strings"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
+)
+
+// R-197 (v0.93.0) — a changed offsite repository key must be REPORTED. Both halves of the comparison
+// (host_escrow.restic_pw_sha256 and the row it replaces) have been in the database since SLICE 3 and
+// nothing read them: demo-felhom's repository password changed on 2026-08-03, orphaning 36 snapshots /
+// 1.14 GB, and no event, e-mail, card or log line said so for thirteen hours.
+//
+// Driven through the real endpoint (PUT /hosts/<id>/escrow), not by calling the emitter, so the wiring
+// is part of what is proven.
+
+// escrowBodyWithHash is escrowBody plus the sealed-password hash and an identity blob — the fork-4
+// upload shape. Hash VALUES here are test fixtures, never real secrets.
+func escrowBodyWithHash(blob []byte, resticPwSHA, identity string) string {
+	m := map[string]string{
+		"blob_b64":         base64.StdEncoding.EncodeToString(blob),
+		"key_fingerprint":  "ab:cd:ef",
+		"posture":          "zero_knowledge",
+		"created_at":       "2026-08-04T05:00:00Z",
+		"restic_pw_sha256": resticPwSHA,
+	}
+	if identity != "" {
+		m["identity_blob_b64"] = base64.StdEncoding.EncodeToString([]byte(identity))
+	}
+	b, _ := json.Marshal(m)
+	return string(b)
+}
+
+// Scenario D — a new escrow sealing a DIFFERENT repo password raises exactly one operator signal,
+// naming the host, carrying NO hash value.
+// RED-PROOF: remove the maybeEmitRepoKeyChanged call from handleHostEscrowPut (or make its predicate
+// always fall through) → no event → this FAILS.
+func TestEscrowPut_ChangedRepoKey_RaisesSignal(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
+
+	if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-gen1")); rr.Code != http.StatusOK {
+		t.Fatalf("first PUT = %d (%s)", rr.Code, rr.Body.String())
+	}
+	// Generation 1 is onboarding, not a change — nothing may fire yet.
+	if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
+		t.Fatalf("a FIRST escrow must not raise %s: %+v", eventRepoKeyChanged, ev)
+	}
+
+	if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_GEN2", "age-gen2")); rr.Code != http.StatusOK {
+		t.Fatalf("second PUT = %d (%s)", rr.Code, rr.Body.String())
+	}
+
+	ev, err := st.GetLatestEventByType("c1", eventRepoKeyChanged)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if ev == nil {
+		t.Fatal("R-197: the repository key demonstrably changed and NO signal was raised — this is the " +
+			"thirteen-hour silence that let demo-felhom lose 1.14 GB unremarked")
+	}
+	if ev.Severity != "warning" {
+		t.Errorf("severity = %q, want warning (info is an intentional non-notify — the operator would not be told)", ev.Severity)
+	}
+	if !strings.Contains(ev.Message, "h1") {
+		t.Errorf("message must name the host, got %q", ev.Message)
+	}
+	// NO HASH VALUE may travel — not in the message, not in the details.
+	for _, forbidden := range []string{"SHA_GEN1", "SHA_GEN2"} {
+		if strings.Contains(ev.Message, forbidden) || strings.Contains(ev.DetailsJSON, forbidden) {
+			t.Errorf("a repo-password hash value leaked into the event (%s): msg=%q details=%q", forbidden, ev.Message, ev.DetailsJSON)
+		}
+	}
+	// Exactly one, not one per anything.
+	evs, _ := st.GetRecentEvents("c1", 50)
+	n := 0
+	for _, e := range evs {
+		if e.EventType == eventRepoKeyChanged {
+			n++
+		}
+	}
+	if n != 1 {
+		t.Fatalf("%s fired %d times for one supersession, want 1 (edge-triggered)", eventRepoKeyChanged, n)
+	}
+}
+
+// Scenario E — an UNCHANGED key says nothing. A customer re-running a ceremony for good reasons must
+// not be punished for it with an alarm.
+func TestEscrowPut_UnchangedRepoKey_Silent(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
+
+	do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_SAME", "age-1"))
+	do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_SAME", "age-2"))
+
+	if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
+		t.Fatalf("a same-hash re-ceremony must be SILENT, got: %+v", ev)
+	}
+	// And it must not have created a superseded row either (pre-existing contract, re-asserted here
+	// because Scenario E's silence would also be produced by the supersede branch never running).
+	if n, _ := st.CountSupersededEscrow("h1"); n != 0 {
+		t.Fatalf("same-hash re-upload created %d superseded row(s), want 0", n)
+	}
+}
+
+// The in-between shapes must not be silent-and-indistinguishable either: a hash-less NEW blob cannot
+// show whether the key changed, and saying nothing at all would look identical to "unchanged".
+// Asserted at the level the code offers — no event (correct: nothing was measured) and no crash.
+func TestEscrowPut_HashlessSupersession_NoSignal(t *testing.T) {
+	h, st, _ := newTestHandler(t)
+	st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
+
+	do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-1"))
+	if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "", "age-2")); rr.Code != http.StatusOK {
+		t.Fatalf("hash-less supersession PUT = %d", rr.Code)
+	}
+	if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
+		t.Fatalf("a hash-less new blob cannot EVIDENCE a change; it must not claim one: %+v", ev)
+	}
+	// The retention still ran (the hashes differ, so this IS a supersession) — the key is kept even
+	// though the change cannot be evidenced.
+	if n, _ := st.CountSupersededEscrow("h1"); n != 1 {
+		t.Fatalf("superseded rows = %d, want 1", n)
+	}
+}
+
+// R-97a's lesson, pinned in the SAME commit that mints the type: an operator-tier event that is not in
+// operatorOnlyEvents reaches customers as raw English, because a missing customerMessages entry is NOT
+// a block. Checked here rather than in notify so the type's two registers are asserted together.
+func TestRepoKeyChanged_IsOperatorOnly(t *testing.T) {
+	if !notify.IsOperatorOnly(eventRepoKeyChanged) {
+		t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
+			"about escrow custody", eventRepoKeyChanged)
+	}
+	// The two R-192 types have the same property and the same reason (see the register's comment).
+	for _, et := range []string{"offsite_delivery_stuck", "offsite_credential_restaged"} {
+		if !notify.IsOperatorOnly(et) {
+			t.Errorf("%s is not registered operator-only", et)
+		}
+	}
+}

hub/internal/api/escrow_test.go

@@ -155,7 +155,7 @@ func TestReportACK_EscrowStatus(t *testing.T) {
 	}
 
 	// escrow row with identity blob + hash → the ACK carries all three fields
-	if _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
+	if _, _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
 		t.Fatal(err)
 	}
 	if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {

hub/internal/api/handler.go

@@ -239,6 +239,12 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
 		hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
 		h.handleHostEscrowPut(w, r, hostID)
+	// R-199 (v0.94.0): the box-authenticated MIRROR of the PUT above — a host reads back its own
+	// opaque identity blob so it can be unsealed with the customer's recovery code. Distinct from the
+	// operator-driven DR path in dr.go, which stays exactly as it is (see handleHostEscrowGet).
+	case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
+		hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
+		h.handleHostEscrowGet(w, r, hostID)
 	// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
 	// operator retrieves it via the /admin/ path (global key only).
 	case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
@@ -534,8 +540,10 @@ func (h *Handler) handleReport(w http.ResponseWriter, r *http.Request) {
 	if fd := h.store.ResolveManagedFloor(payload.CustomerID); fd.Floor != "" {
 		resp["min_controller_version"] = fd.Floor
 	} else if fd.Held {
-		h.logger.Printf("[INFO] managed floor HELD for %s: agent %q < MinAgent %s (controller floor withheld)",
-			payload.CustomerID, fd.AgentVersion, fd.MinAgent)
+		// ONE sentence, from the decision itself — the two hold reasons must never drift apart
+		// across this line and the dashboard (see ManagedFloorDecision.HoldReason).
+		h.logger.Printf("[INFO] managed floor HELD for %s: %s (controller floor withheld)",
+			payload.CustomerID, fd.HoldReason())
 	}
 	if h.latestVersion != nil {
 		if latest := h.latestVersion.LatestVersion(); latest != "" {
@@ -1065,9 +1073,13 @@ func (h *Handler) reissueOnReenroll(cc *store.CustomerConfig) {
 		}
 	}
 	// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
-	// one-time password only ever reached the OLD controller). The re-issuer resets the restic repo
-	// password, which makes the OLD escrow blob stale — the offsite provisioner invalidates the
-	// escrow (2.3) and emits both events. Skips silently when offsite isn't provisioned/enabled.
+	// one-time password only ever reached the OLD controller). ⚠ CORRECTED 2026-08-04 (R-196): this
+	// used to claim "the re-issuer resets the restic repo password, which makes the OLD escrow blob
+	// stale". It does not and cannot — the re-issuer resets the PROVIDER account password; the
+	// repository password is generated on the box and never leaves it except sealed under R. The
+	// provisioner does mark the escrow stale, but PRECAUTIONARILY (see the reasoning at
+	// offsite.ReissueCredentials), not because this call rotated anything.
+	// Skips silently when offsite isn't provisioned/enabled.
 	if h.offsiteReissuer != nil {
 		if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
 			h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
@@ -1141,7 +1153,7 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
 	// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
 	// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
 	// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
-	superseded, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
+	superseded, prevPwSHA, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
 	if serr != nil {
 		h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
 		http.Error(w, "Internal error", http.StatusInternalServerError)
@@ -1157,6 +1169,10 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
 				"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
 				h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
 			}
+			// R-197: the box's offsite DATA key demonstrably changed. Both halves of that comparison
+			// have been stored since SLICE 3 and nothing read them — demo-felhom's key changed on
+			// 2026-08-03 and nothing said so for thirteen hours.
+			h.maybeEmitRepoKeyChanged(host.CustomerID, pathHostID, prevPwSHA, req.ResticPwSHA256, n)
 		}
 	}
 	// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
@@ -1185,6 +1201,188 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
 	w.Write([]byte(`{"status":"ok"}`))
 }
 
+// eventRepoKeyChanged (R-197) — the box's offsite restic REPOSITORY password changed, proven by the
+// hub's own stored hashes. Hub-internal (not in allowedEventTypes, like escrow_superseded) and
+// registered operator-only in notify.operatorOnlyEvents.
+const eventRepoKeyChanged = "offsite_repo_key_changed"
+
+// eventEscrowBlobServed (R-199) — a host retrieved its own sealed identity blob. Hub-internal,
+// operator-only. See handleHostEscrowGet for why every retrieval is loud.
+const eventEscrowBlobServed = "escrow_blob_served"
+
+// escrowSelfServiceRetrieval is THE SINGLE DECISION POINT for the §8.2/§8.3 trade (R-199).
+//
+//	true  (§8.2, shipped v0.94.0) — a host may read its own blob whenever it authenticates as itself.
+//	false (§8.3, the fallback)    — the same read additionally requires operator-armed recovery mode.
+//
+// It is one condition on purpose: the operator may overrule the trade below, and switching must cost a
+// boolean rather than a redesign. Everything else in the recovery chain is identical either way.
+const escrowSelfServiceRetrieval = true
+
+// handleHostEscrowGet serves a host its OWN opaque identity-escrow blob (R-199, v0.94.0).
+//
+// WHAT THIS GIVES OUT, WHY IT IS SAFE, AND WHAT IT CHANGES ABOUT WHO IS REQUIRED — recorded here so the
+// next reader finds the trade rather than inferring it (the dr.go header convention).
+//
+// WHAT: the age-wrapped `IdentityBundle` — opaque ciphertext. It carries the offsite restic repository
+// password, the tunnel token, the PBS token and the WG key. The hub stores these bytes and has no
+// decrypt path; the recovery code R that opens them exists only in the customer's hands.
+//
+// WHY IT IS SAFE TO GIVE OUT: the blob is useless without R (age scrypt + ChaCha20-Poly1305; a wrong R
+// fails closed at the KDF, never to a plausible-but-wrong bundle), and a 10-word EFF code carries ~129
+// bits. The caller already authenticates as this host for its report, its desired state, its WG
+// registration and its PBS token — this adds no new identity, only a new object, and it is the exact
+// MIRROR of the PUT above, which is how the blob got here in the first place.
+//
+// WHAT IT CHANGES, STATED PLAINLY BECAUSE IT IS THE WHOLE OF THE TRADE: before this, obtaining the blob
+// required the OPERATOR to arm recovery mode with the global key (dr.go). Now whoever controls a
+// rebuilt box can obtain it with that box's own credential. That is a real reduction in the number of
+// parties required. They still cannot open it. The mitigation is that the capability is AUDITED rather
+// than silent: every successful retrieval raises an operator event (below), because a silent capability
+// is the shape this project has spent two weeks removing.
+//
+// THE OPERATOR-DRIVEN DR PATH IS UNTOUCHED. `handleReEnroll` / `handleGetRestoreDirective` keep their
+// recovery-mode gate and their global-key arming, and they serve the K-escrow and the directive as
+// well. This endpoint serves ONE object to ONE authenticated owner. Do not merge them.
+func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pathHostID string) {
+	authHostID, _, isGlobal, ok := h.checkAuthHost(r)
+	if !ok {
+		http.Error(w, "Unauthorized", http.StatusUnauthorized)
+		return
+	}
+	if pathHostID == "" {
+		http.Error(w, "Missing host_id", http.StatusBadRequest)
+		return
+	}
+	// SELF-SCOPED: a per-host key reads only its OWN escrow. The global operator key may read any —
+	// the same asymmetry the PUT has. Without this line any host key is a fleet-wide blob reader.
+	if !isGlobal && authHostID != pathHostID {
+		h.logger.Printf("[WARN] escrow GET REFUSED: host %s asked for %s's blob (self-scope)", authHostID, pathHostID)
+		http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden)
+		return
+	}
+	host, err := h.store.GetHost(pathHostID)
+	if err != nil {
+		http.Error(w, "Internal error", http.StatusInternalServerError)
+		return
+	}
+	if host == nil {
+		http.Error(w, "Unknown host_id", http.StatusNotFound)
+		return
+	}
+	// §8.3 fallback lives here and nowhere else.
+	if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) {
+		h.logger.Printf("[WARN] escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID)
+		http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden)
+		return
+	}
+
+	bundle, berr := h.store.GetHostDRBundle(pathHostID)
+	if berr != nil {
+		h.logger.Printf("[ERROR] escrow GET for %s: %v", pathHostID, berr)
+		http.Error(w, "Internal error", http.StatusInternalServerError)
+		return
+	}
+	// A host with no sealed bundle gets a CLEAN ANSWER, not a fault: 200 with present=false. A 404
+	// here would be indistinguishable from an unknown host, and an empty 200 without the flag would be
+	// indistinguishable from a zero-length blob — both read as "something is broken" to a caller whose
+	// situation is simply "no ceremony has run yet".
+	if bundle == nil || len(bundle.IdentityBlob) == 0 {
+		h.logger.Printf("[INFO] escrow GET for %s: no identity blob stored (no ceremony has run)", pathHostID)
+		writeJSON(w, http.StatusOK, map[string]any{"host_id": pathHostID, "present": false, "identity_escrow_b64": ""})
+		return
+	}
+
+	// THE MITIGATION (§8.2). Recorded BEFORE the bytes leave, so a retrieval cannot be served without
+	// its audit row; a save failure is logged and does NOT block the response (the blob is opaque and
+	// refusing it would break a recovery over an audit hiccup — but the log line always exists).
+	//
+	// SEVERITY = warning, i.e. it reaches the operator by e-mail. Retrieval is not routine today: it
+	// happens during a recovery and nowhere else. IF a customer-facing self-service flow ever makes it
+	// routine, revisit this — but revisit it deliberately, do not let it decay to info because the
+	// mail became annoying.
+	if host.CustomerID != "" {
+		msg := fmt.Sprintf("Recovery blob served: host %s retrieved its own sealed identity escrow (%d opaque bytes). "+
+			"This is the recovery path in use — the blob cannot be opened without the customer's recovery code, which the hub never holds. "+
+			"If no recovery is in progress on that box, investigate.", pathHostID, len(bundle.IdentityBlob))
+		details, _ := json.Marshal(map[string]any{
+			"host_id":    pathHostID,
+			"blob_bytes": len(bundle.IdentityBlob),
+			"self_scope": !isGlobal,
+		})
+		if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil {
+			h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr)
+		} else if h.dispatcher != nil {
+			go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub")
+		}
+	}
+	h.logger.Printf("[WARN] escrow blob SERVED to host %s (%d opaque bytes, self_scope=%v) — recovery path in use",
+		pathHostID, len(bundle.IdentityBlob), !isGlobal)
+	writeJSON(w, http.StatusOK, map[string]any{
+		"host_id":             pathHostID,
+		"present":             true,
+		"identity_escrow_b64": base64.StdEncoding.EncodeToString(bundle.IdentityBlob),
+	})
+}
+
+// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
+// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
+// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
+// changed on 2026-08-03, orphaning 36 snapshots / 1.14 GB, and no event, e-mail, card or log line
+// said so for thirteen hours — the comparison that eventually found it is this one
+// (audits/RECON-offsite-dr-chain-2026-08-04.md, R-197).
+//
+// THE PREDICATE IS DELIBERATELY NARROW: both hashes known AND different. A first-ever hash (prev "")
+// is onboarding, not a change; a hash-less supersession (now "") cannot show a change happened; an
+// identical hash is a re-ceremony of the SAME password, which is a normal healthy act and must stay
+// silent or a customer is punished for re-running a ceremony. The in-between shapes are LOGGED rather
+// than dropped, so "we chose not to alarm" and "the check did not run" never look identical.
+//
+// SEVERITY = warning, chosen for the world v0.93.0 creates rather than the one it inherits. Before
+// R-198 a changed key meant the previous history was unopenable by anyone, ever — that would have
+// argued for error. From v0.93.0 the superseding ceremony RETAINS the old identity blob, so the
+// previous history stays recoverable with the recovery code that sealed it: the situation is "this
+// customer's off-site history now depends on an older recovery code", which is operator-actionable
+// (check the orphan card, expect a fresh repository) and is not a loss. warning also routes: the
+// dispatcher notifies on warning/error/critical and treats info as an intentional non-notify, and
+// the whole point of this row is that the operator learns on the day.
+//
+// EDGE-TRIGGERED: called only from the superseded branch of the escrow PUT, i.e. once per
+// supersession, never per report. No timer lives here — the dispatcher owns cooldown.
+//
+// NO HASH VALUE TRAVELS. The message and the details name the host, the customer and the retained
+// count only. The hashes are non-reversible, but a hash is still a fingerprint of a live secret and
+// this project's rule is that values do not leave the store.
+func (h *Handler) maybeEmitRepoKeyChanged(customerID, hostID, prevSHA, newSHA string, retained int) {
+	switch {
+	case prevSHA == "":
+		h.logger.Printf("[INFO] escrow for host %s: no previous repo-password hash recorded (first hash, or a legacy hash-less blob) — repo-key-change check not applicable", hostID)
+		return
+	case newSHA == "":
+		h.logger.Printf("[WARN] escrow for host %s: the NEW blob carries no repo-password hash (hash-less supersession) — whether the repository key changed CANNOT be determined from the hub's data", hostID)
+		return
+	case prevSHA == newSHA:
+		return // same password re-sealed: a healthy re-ceremony (Scenario E — silence is correct)
+	}
+	msg := fmt.Sprintf("Offsite repository key CHANGED for host %s: the new escrow seals a different repository password than the one it replaced. "+
+		"The previous off-site history is no longer opened by this box's current key. The superseding blob was retained (%d held), so that history stays recoverable with the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an orphan card, and expect the next backup to start a fresh history.",
+		hostID, retained)
+	details, _ := json.Marshal(map[string]any{
+		"host_id":        hostID,
+		"retained_count": retained,
+		"repo_key":       "changed", // never the hash values
+	})
+	if _, err := h.store.SaveEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub"); err != nil {
+		h.logger.Printf("[WARN] %s event save failed for %s: %v", eventRepoKeyChanged, hostID, err)
+		return // audit row first: an e-mail without its event row lies (the OffsiteChecker convention)
+	}
+	h.logger.Printf("[WARN] offsite repository key CHANGED for host %s (customer %s) — previous history now depends on the superseded recovery code; %d retained blob(s)",
+		hostID, customerID, retained)
+	if h.dispatcher != nil {
+		go h.dispatcher.ProcessEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub")
+	}
+}
+
 // handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
 // SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
 // host api_key. The secret is stored at rest and NEVER logged (only the username + a length are

hub/internal/monitor/deadline.go

@@ -336,7 +336,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
 	midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
 	sinceUTC := midnightBudapest.UTC()
 
-	var backupMissed, dbdumpMissed, skipped, deferred int
+	var backupMissed, dbdumpMissed, skipped, deferred, unbound int
 
 	for _, id := range customerIDs {
 		// Skip nodes that are down — they already have staleness events
@@ -350,6 +350,37 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
 			continue
 		}
 
+		// ── R-195: a customer with NO machine EVER bound is UNKNOWN, not missed ────────────────
+		//
+		// Both verdicts below ask "did the thing we expect every day happen?". For a customer
+		// that has never had a machine bound, nothing has ever been expected, so the honest
+		// answer is UNKNOWN — the same invariant assessBackupFreshness states above, applied one
+		// level up, at the question of whether there is a subject at all.
+		//
+		// The discriminator is "was a host EVER bound", NOT "has a report arrived". That is the
+		// case this check must not break: a box that was installed, bound, and then went silent
+		// has a real fault and must keep alarming. It is bound, so it is judged.
+		//
+		// WHY THIS WAS REACHABLE AT ALL, measured 2026-08-04: the down-skip above is what
+		// protects every other silent customer, and it reads the staleness checker's state — which
+		// is seeded from the `reports` table (store.GetCustomers). A customer that has NEVER
+		// reported appears in no report row, so it gets no staleness state at all and GetState()
+		// returns "" rather than "down". The skip misses exactly the customer it would most
+		// obviously cover, and the DB-dump half below then fires every night: `david`, a
+		// prospective customer whose record was created 2026-08-01 with no machine ever bound,
+		// e-mailed an expected_dbdump_missed ERROR at 03:00 UTC on three consecutive days.
+		//
+		// Fail-open on a read error: an unreadable binding must never SUPPRESS a real alarm.
+		if bound, berr := s.HasEverBoundHost(id); berr != nil {
+			logger.Printf("[WARN] Deadline check: failed to read host binding for %s (judging anyway): %v", id, berr)
+		} else if !bound {
+			// Visible, per the v0.73.0 Part-7 precedent below: a quiet check must never be
+			// indistinguishable from a check that did not run. Once daily, one line per customer.
+			logger.Printf("[INFO] Deadline check: %s has no host EVER bound — all deadline verdicts UNKNOWN (no alarm)", id)
+			unbound++
+			continue
+		}
+
 		// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
 		// the authoritative offsite-backup signal post-slice-8C.
 		reportJSON, rerr := s.GetLatestHostReportJSON(id)
@@ -423,6 +454,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
 		}
 	}
 
-	logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
-		len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
+	logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down), %d unknown (no host ever bound)",
+		len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped, unbound)
 }

hub/internal/monitor/deadline_unbound_test.go

@@ -0,0 +1,148 @@
+package monitor
+
+import (
+	"io"
+	"log"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
+)
+
+// R-195 — a customer with NO machine EVER bound must not alarm; a customer WITH one must.
+//
+// Origin, measured on the live hub 2026-08-04: `david` is a prospective customer whose record was
+// created 2026-08-01 16:51:49 with no host ever bound (hosts=0, host_deletions=0, host_reports=0,
+// reports=0). It e-mailed an `expected_dbdump_missed` ERROR at 03:00 UTC on 08-02, 08-03 and 08-04.
+//
+// The mechanism, established at source: the down-skip in CheckBackupDeadlines reads
+// StalenessChecker.GetState(), whose map is seeded from store.GetCustomers() — a query over the
+// `reports` table. A customer with zero reports is in no row, so it has no state, and GetState()
+// returns "" rather than "down". The skip that protects every other silent customer misses the one
+// that never reported at all.
+//
+// These tests pin BOTH halves. A suite that only proved the silence would pass against an
+// implementation that never alarms, which is strictly worse than the defect it replaces.
+
+// newUnboundStore creates a store holding ONE active customer and NO host row at all.
+func newUnboundStore(t *testing.T) *store.Store {
+	t.Helper()
+	st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
+	if err != nil {
+		t.Fatalf("store.New: %v", err)
+	}
+	t.Cleanup(func() { st.Close() })
+	if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"}); err != nil {
+		t.Fatalf("SaveCustomerConfig: %v", err)
+	}
+	return st
+}
+
+// ── Half 1: the silence ────────────────────────────────────────────────────────────────────────
+
+// TestCheckBackupDeadlines_NeverBoundHost_Silent is the david case.
+//
+// COMPANION RED-PROOF (observed): deleting the HasEverBoundHost guard from CheckBackupDeadlines
+// makes this test fail with
+//
+//	deadline_unbound_test.go: a customer with NO host ever bound must raise NOTHING;
+//	got [expected_dbdump_missed]
+//
+// which is verbatim the event `david` e-mailed three mornings running. Restored after.
+func TestCheckBackupDeadlines_NeverBoundHost_Silent(t *testing.T) {
+	st := newUnboundStore(t)
+	// No UpsertHost, no host-report, no db_dump_completed event — nothing has ever been expected.
+	got := runDeadline(t, st)
+	if len(got) != 0 {
+		t.Fatalf("a customer with NO host ever bound must raise NOTHING; got %v", got)
+	}
+}
+
+// TestHasEverBoundHost pins the predicate itself across its three inputs, because the whole
+// behaviour above turns on it and a predicate that answered `false` for everything would make the
+// test above pass while silencing the entire fleet.
+func TestHasEverBoundHost(t *testing.T) {
+	st := newUnboundStore(t)
+
+	if bound, err := st.HasEverBoundHost("c1"); err != nil || bound {
+		t.Fatalf("no host rows → want (false,nil); got (%v,%v)", bound, err)
+	}
+
+	if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
+		t.Fatalf("UpsertHost: %v", err)
+	}
+	if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
+		t.Fatalf("live host row → want (true,nil); got (%v,%v)", bound, err)
+	}
+
+	// An unknown customer is never bound — the predicate must not answer from another customer's rows.
+	if bound, err := st.HasEverBoundHost("nobody"); err != nil || bound {
+		t.Fatalf("unknown customer → want (false,nil); got (%v,%v)", bound, err)
+	}
+}
+
+// ── Half 2: THE RED-PROOF THAT MATTERS — a bound machine must still alarm ───────────────────────
+
+// TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms is the case the change could break,
+// and it is a real shape: a machine that was installed and bound and never phoned home. It has a
+// `hosts` row and zero reports — indistinguishable from `david` on every signal EXCEPT the one the
+// guard discriminates on. If the guard is ever "simplified" to key off report presence, customer
+// age or a name pattern, this test goes red.
+func TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms(t *testing.T) {
+	st := newUnboundStore(t)
+	if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
+		t.Fatalf("UpsertHost: %v", err)
+	}
+	// No host-report and no db_dump_completed — the box was bound and never said anything.
+	got := runDeadline(t, st)
+	if !has(got, "expected_dbdump_missed") {
+		t.Fatalf("a BOUND machine that never reported is a real fault and must still alarm; got %v", got)
+	}
+}
+
+// TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms: the machine reported once, days ago,
+// and stopped. Nothing about the guard may suppress that.
+//
+// The staleness checker's down-skip is nil here (runDeadline passes nil), which is deliberate: it
+// isolates THIS guard. In production a genuinely down node is skipped by staleness and gets its own
+// node_down event — that path is unchanged and is not what this test is about.
+func TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms(t *testing.T) {
+	st := newUnboundStore(t)
+	if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
+		t.Fatalf("UpsertHost: %v", err)
+	}
+	// One host-report whose newest backup evidence is 9 days old, and a db dump that last
+	// completed 5 days ago (i.e. not since midnight).
+	report := hostReportJSON(t, [][2]string{{rfc(-9 * 24 * time.Hour), "ok"}}, nil)
+	if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
+		t.Fatalf("SaveHostReport: %v", err)
+	}
+	got := runDeadline(t, st)
+	if !has(got, "expected_backup_missed") {
+		t.Fatalf("a bound machine that went quiet with stale backups must still raise expected_backup_missed; got %v", got)
+	}
+	if !has(got, "expected_dbdump_missed") {
+		t.Fatalf("a bound machine that went quiet must still raise expected_dbdump_missed; got %v", got)
+	}
+}
+
+// TestCheckBackupDeadlines_DeletedHost_StillJudged: the host row is gone but a tombstone remains
+// (peti-felhom's live shape). The customer HAD a machine, so this check must not take over the
+// judgement — it hands off to the staleness down-skip exactly as before the change.
+func TestCheckBackupDeadlines_DeletedHost_StillJudged(t *testing.T) {
+	st := newUnboundStore(t)
+	if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
+		t.Fatalf("UpsertHost: %v", err)
+	}
+	if err := st.DeleteHost("h1", false); err != nil {
+		t.Fatalf("DeleteHost: %v", err)
+	}
+	if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
+		t.Fatalf("a DELETED host is still a machine that was once bound → want (true,nil); got (%v,%v)", bound, err)
+	}
+	got := runDeadline(t, st)
+	if !has(got, "expected_dbdump_missed") {
+		t.Fatalf("a customer whose host was deleted was still bound and stays judged here; got %v", got)
+	}
+}

hub/internal/monitor/offsite_declared_test.go

@@ -0,0 +1,47 @@
+package monitor
+
+import (
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/offsiteheal"
+)
+
+// R-192's guard half, closed by REPLACEMENT (R-204 item 4).
+//
+// The counting guard inferred the situation from how many of the OLDEST 500 reports after a consume
+// carried an offbox target. On demo-hp all 500 predated the rebuild, so the checker confidently
+// reported the REGRESSED shape and declined to heal — for 108 reports, while the box sat stranded.
+// A declaration needs no window and no count, and it OUTRANKS both inferred shapes.
+
+func TestShapeOf_DeclarationOutranksBothInferredShapes(t *testing.T) {
+	cases := []struct {
+		name     string
+		status   offsite.DeliveryStatus
+		declared bool
+		want     deliveryShape
+	}{
+		{"burned, undeclared", offsite.DeliveryStatus{OffsiteReportsSinceConsume: 0, ReportsSinceConsume: 9}, false, shapeBurned},
+		{"regressed, undeclared", offsite.DeliveryStatus{OffsiteReportsSinceConsume: 500, ReportsSinceConsume: 500}, false, shapeRegressed},
+		// THE demo-hp SHAPE: the counts say "regressed" from a window that predates the rebuild, and
+		// the box says it needs a credential. The declaration wins.
+		{"regressed counts BUT the box declares", offsite.DeliveryStatus{OffsiteReportsSinceConsume: 500, ReportsSinceConsume: 500}, true, shapeDeclared},
+		{"burned counts AND the box declares", offsite.DeliveryStatus{OffsiteReportsSinceConsume: 0, ReportsSinceConsume: 9}, true, shapeDeclared},
+	}
+	for _, tc := range cases {
+		if got := shapeOf(tc.status, tc.declared); got != tc.want {
+			t.Errorf("%s: shapeOf = %q, want %q", tc.name, got, tc.want)
+		}
+	}
+}
+
+// The declared-state string is duplicated in three packages (the controller declares it, this checker
+// recognises it, the reconciler acts on it). A silent drift between them would make the whole feature
+// inert with every test still green — so the two hub-side copies are pinned to each other here. The
+// controller's copy is pinned by its own report-shape test and by the live validation.
+func TestDeclaredStateStringMatchesTheReconciler(t *testing.T) {
+	if declaredNeedsCredential != offsiteheal.StateNeedsCredential {
+		t.Fatalf("declared-state drift: monitor has %q, offsiteheal has %q — the checker would never stand down and both mechanisms would heal the same customer",
+			declaredNeedsCredential, offsiteheal.StateNeedsCredential)
+	}
+}

hub/internal/monitor/offsite_delivery.go

@@ -94,40 +94,151 @@ func (c *OffsiteDeliveryChecker) Check() {
 		if age < stuckAfter {
 			continue // normal convergence window
 		}
-		c.maybeEmitStuck(cfg.CustomerID, status, age)
-		c.maybeHeal(cfg.CustomerID, status)
+		// R-204 item 4 / R-192's guard half: does the BOX declare that it needs a credential? A
+		// declaration is stronger evidence than anything this checker can infer, and it is owned by
+		// internal/offsiteheal — see maybeHeal. A read error is treated as "no declaration", which is
+		// the conservative direction: this checker keeps its pre-v0.96.0 behaviour rather than
+		// silently standing down.
+		declared := false
+		if _, _, state, derr := c.store.LatestReportOffsiteDeclaration(cfg.CustomerID); derr != nil {
+			c.logger.Printf("[WARN] offsite-delivery: %s: read declaration: %v (treating as undeclared)", cfg.CustomerID, derr)
+		} else {
+			declared = state == declaredNeedsCredential
+		}
+		emitted := c.maybeEmitStuck(cfg.CustomerID, status, age, declared)
+		c.maybeHeal(cfg.CustomerID, status, emitted, declared)
 	}
 }
 
-// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer.
-func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) {
+// deliveryShape names the two situations the ONE stuck state actually covers. They need different
+// text and different advice, and conflating them is R-192's defect (a).
+type deliveryShape string
+
+const (
+	// shapeBurned — NO report since the consume carried an offbox target: the apply never persisted.
+	// Re-issue is the indicated action.
+	shapeBurned deliveryShape = "burned"
+	// shapeRegressed — reports since the consume DID carry an offbox target and the latest does not:
+	// the credential worked and the target was later lost (a guest rebuild does exactly this, R-193).
+	// Re-issue is NOT indicated; it treats a symptom whose cause is elsewhere.
+	shapeRegressed deliveryShape = "regressed"
+	// shapeDeclared — the BOX itself says it needs a credential (controller >= v0.199.0). This
+	// OUTRANKS both shapes above and is R-192's guard half closed by REPLACEMENT rather than repair:
+	// the counting guard inferred the situation from how many of the OLDEST 500 reports after the
+	// consume carried an offbox target — all of which predate a rebuild, which is why demo-hp sat
+	// stranded for 108 reports while the checker confidently reported the regressed shape. A
+	// declaration needs no window, no count and no inference. The remediation belongs to
+	// internal/offsiteheal; this checker stands down and says so.
+	shapeDeclared deliveryShape = "declared"
+)
+
+// declaredNeedsCredential mirrors backup.OffsiteStateNeedsCredential / offsiteheal.StateNeedsCredential.
+// Duplicated as a literal rather than imported to avoid a monitor→offsiteheal dependency; the three
+// are pinned together by TestDeclaredStateStringMatchesTheReconciler.
+const declaredNeedsCredential = "needs_credential"
+
+func shapeOf(status offsite.DeliveryStatus, declared bool) deliveryShape {
+	if declared {
+		return shapeDeclared
+	}
+	if status.OffsiteReportsSinceConsume == 0 {
+		return shapeBurned
+	}
+	return shapeRegressed
+}
+
+// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. Returns
+// whether it emitted, so the heal's refusal record rides the same cadence rather than inventing one.
+//
+// R-192 defect (a), fixed here: the message used to interpolate ReportsSinceConsume (the TOTAL) into
+// a hardcoded phrase "report(s) since carry no offbox target", and never consulted
+// OffsiteReportsSinceConsume — the field that says the opposite. On demo-hp it stated, daily, that
+// 500 reports carried no offbox target when all 500 of them did, and prescribed Re-issue for a
+// failure mode that had not occurred. The message now STATES WHAT WAS MEASURED and lets the operator
+// read it; the recommendation follows the shape rather than being hardcoded.
+//
+// THE WINDOW IS NAMED ON PURPOSE. CountReportsOffsiteSince reads `ORDER BY id LIMIT 500` — the OLDEST
+// 500 reports after the consume, not the newest — so on a long-lived customer these counts describe
+// the beginning of the window and not the present. That is a real scoping defect (R-192's other half)
+// and it stays OPEN because its correct shape depends on the recovery chain that is not yet
+// assembled (R-199/R-200/R-201). Naming the window in the text is how it stays visible instead of
+// being laundered into a confident sentence — an instrument that can silently mis-scope its results
+// must say so where it reports them.
+func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration, declared bool) bool {
 	last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
 	if err != nil {
 		c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
-		return
+		return false
 	}
 	if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
-		return
+		return false
+	}
+	shape := shapeOf(status, declared)
+	var msg string
+	switch shape {
+	case shapeDeclared:
+		msg = fmt.Sprintf("Offsite delivery stuck (DECLARED by the box): the one-time password was consumed %s ago and the box now reports `offsite.state=%s` — it has been rebuilt, holds no repository password, and the hub is holding a sealed recovery package for it. No inference was needed: the box said so. The offsite self-heal reconciler owns this remediation (it re-arms the stored credential before minting a new one); no operator action is indicated unless this repeats.",
+			age.Round(time.Minute), declaredNeedsCredential)
+	case shapeBurned:
+		msg = fmt.Sprintf("Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, NONE carried an offbox target, and the latest report carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.",
+			age.Round(time.Minute), status.ReportsSinceConsume)
+	default:
+		msg = fmt.Sprintf("Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, %d DID carry an offbox target — and the latest report carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).",
+			age.Round(time.Minute), status.ReportsSinceConsume, status.OffsiteReportsSinceConsume)
 	}
-	msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.",
-		age.Round(time.Minute), status.ReportsSinceConsume)
 	details, _ := json.Marshal(map[string]any{
-		"state":                 string(status.State),
-		"consumed_at":           status.Since.UTC().Format(time.RFC3339),
-		"reports_since_consume": status.ReportsSinceConsume,
+		"state":                         string(status.State),
+		"shape":                         string(shape),
+		"consumed_at":                   status.Since.UTC().Format(time.RFC3339),
+		"reports_since_consume":         status.ReportsSinceConsume,
+		"offsite_reports_since_consume": status.OffsiteReportsSinceConsume,
+		"count_window":                  "oldest 500 reports after consumed_at (R-192, open)",
 	})
 	c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
+	return true
 }
 
 // maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
 // consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
 // one heal per healCooldown — and the R-39(a) guard holds at act time.
-func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) {
+//
+// R-192 defect (b), fixed here: every refusal above the "not configured" line now leaves a RECORD.
+// The regressed-shape branch used to be a bare `return`, so the operator received a daily e-mail with
+// the wrong story, no heal, and nothing anywhere saying why the heal declined — "we chose not to act"
+// and "the heal never ran" looked identical. `offsite_credential_restaged` has never fired for any
+// customer, and until now that fact was indistinguishable from the checker being dead.
+//
+// The record is a notification_log row (the dispatcher's suppressed-operator-e-mail precedent, R-182:
+// a decision not to act is written down on the channel it would have used). It rides `recordRefusal`
+// — true only when the stuck event was emitted this pass — so it appears once per stuckCooldown
+// beside the e-mail it explains, rather than once per monitor tick. The GUARD ITSELF IS UNCHANGED:
+// the set of situations in which the heal fires is byte-for-byte what it was; only the silence is
+// gone. The two conditions are split into separate branches solely so each refusal can name its own
+// reason.
+func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus, recordRefusal bool, declared bool) {
 	if c.reissuer == nil {
+		return // no provisioner configured: the heal does not exist on this hub, so there is nothing to explain
+	}
+	// R-204 item 4: a DECLARING box belongs to internal/offsiteheal, which re-arms the stored
+	// credential before minting a new one. Two mechanisms healing the same customer would double-issue
+	// — and this one can only mint, so it would also skip the cheap path. Stand down, with a record:
+	// "we chose not to act" and "the heal never ran" must not look identical (R-192 defect (b)).
+	if declared {
+		c.recordHealRefusal(customerID, recordRefusal,
+			"the box DECLARES offsite.state=needs_credential; internal/offsiteheal owns this remediation (it re-stages the stored credential before minting). A second mechanism minting here would double-issue.")
 		return
 	}
-	if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 {
-		return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call
+	if status.OffsiteReportsSinceConsume != 0 {
+		c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
+			"regressed-apply shape: %d of the first %d report(s) after the consume DID carry an offbox target, so a burned credential is ruled out — a restage would treat a symptom whose cause is elsewhere. Operator's call (R-193).",
+			status.OffsiteReportsSinceConsume, status.ReportsSinceConsume))
+		return
+	}
+	if status.ReportsSinceConsume < healMinReports {
+		c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
+			"only %d report(s) since the consume (need %d): the box has not reported enough for the burned shape to be unambiguous.",
+			status.ReportsSinceConsume, healMinReports))
+		return
 	}
 	last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
 	if err != nil {
@@ -146,8 +257,8 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
 		return
 	}
 	if info == nil || info.ConsumedAt.IsZero() {
-		c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)",
-			customerID, secretShape(info))
+		c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
+			"R-39(a) guard: the secret row is now %s — restaging over an unconsumed secret would clobber a password the box may be about to consume.", secretShape(info)))
 		return
 	}
 	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
@@ -167,6 +278,22 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
 		string(details))
 }
 
+// recordHealRefusal makes a decision NOT to self-heal visible. Always logs; additionally writes a
+// notification_log row on the "operator" channel with status "refused" when `record` is set (the
+// stuck event was emitted this pass), so the refusal sits next to the e-mail that prompted the
+// question. A LogNotification failure is logged, never swallowed, and never blocks the refusal — the
+// refusal is the primary effect.
+func (c *OffsiteDeliveryChecker) recordHealRefusal(customerID string, record bool, reason string) {
+	c.logger.Printf("[INFO] offsite-delivery: %s: self-heal REFUSED — %s", customerID, reason)
+	if !record {
+		return
+	}
+	if err := c.store.LogNotification(customerID, eventCredentialRestaged, "warning",
+		"Automatic offsite credential restage was NOT performed.", "refused", reason, "operator"); err != nil {
+		c.logger.Printf("[WARN] offsite-delivery: %s: could not record the heal refusal: %v", customerID, err)
+	}
+}
+
 func secretShape(info *store.OneTimeSecretInfo) string {
 	if info == nil {
 		return "absent"

hub/internal/monitor/offsite_delivery_honesty_test.go

@@ -0,0 +1,223 @@
+package monitor
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
+)
+
+// R-192 (v0.93.0) — the alert must describe what was MEASURED, and a refusal to self-heal must leave a
+// trace.
+//
+// What was wrong, quoted from the live e-mail demo-hp sent daily from 2026-08-03: "one-time password
+// consumed 284h19m0s ago and 500 report(s) since carry no offbox target — the credential is likely
+// burned … Re-issue delivers a fresh one." OffsiteReportsSinceConsume for that same call was 500:
+// every one of those reports carried a target. The message stated the precise negation of its own
+// measurement and prescribed a remedy for a failure mode that had not occurred — while the self-heal
+// correctly declined, in complete silence.
+
+// regressedFixture puts c1 into demo-hp's real shape: consumed >1h ago, reports WITH an offbox target
+// after the consume, then reports WITHOUT — the latest carrying none.
+//
+// The report timestamps get jitter deliberately: a perfectly regular series lands exactly on
+// boundaries and survives mutations that a realistic one catches.
+func (h dtHarness) regressedFixture(t *testing.T, withOffsite, without int) {
+	t.Helper()
+	if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
+		t.Fatal(err)
+	}
+	consumed := time.Now().UTC().Add(-30 * time.Hour).Format("2006-01-02 15:04:05")
+	staged := time.Now().UTC().Add(-31 * time.Hour).Format("2006-01-02 15:04:05")
+	if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil {
+		t.Fatal(err)
+	}
+	jitter := []int{0, 3, 1, 7, 2, 5, 11, 4}
+	for i := 0; i < withOffsite; i++ {
+		if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
+			t.Fatal(err)
+		}
+		time.Sleep(time.Duration(jitter[i%len(jitter)]) * time.Millisecond)
+	}
+	for i := 0; i < without; i++ {
+		if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
+			t.Fatal(err)
+		}
+		time.Sleep(time.Duration(jitter[(i+3)%len(jitter)]) * time.Millisecond)
+	}
+}
+
+func (h dtHarness) refusalRows(t *testing.T) []store.NotificationLogEntry {
+	t.Helper()
+	all, err := h.st.GetRecentNotifications("c1", 50)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var out []store.NotificationLogEntry
+	for _, n := range all {
+		if n.Status == "refused" {
+			out = append(out, n)
+		}
+	}
+	return out
+}
+
+// Scenario F — the message states the measured state, not a denied cause.
+// RED-PROOF: restore the old single hardcoded sentence (interpolate ReportsSinceConsume into
+// "report(s) since carry no offbox target") → the regressed assertions FAIL.
+func TestDeliveryChecker_RegressedShape_MessageMatchesMeasurement(t *testing.T) {
+	h := newDTHarness(t, true)
+	h.regressedFixture(t, 6, 2)
+
+	h.checker.Check()
+
+	saved := h.savedEvents(t, "offsite_delivery_stuck")
+	if len(saved) != 1 {
+		t.Fatalf("stuck events = %d, want 1", len(saved))
+	}
+	msg := saved[0].Message
+
+	// It must NOT assert the thing the data contradicts.
+	if strings.Contains(msg, "carry no offbox target") || strings.Contains(msg, "likely burned") {
+		t.Fatalf("R-192(a): the message still asserts a burned credential on a REGRESSED shape — this is "+
+			"the sentence that stated the negation of its own measurement:\n%s", msg)
+	}
+	// It must name the shape it detected and the measurement behind it.
+	if !strings.Contains(msg, "REGRESSED") {
+		t.Errorf("the message must name the detected shape, got:\n%s", msg)
+	}
+	if !strings.Contains(msg, "6 DID carry an offbox target") {
+		t.Errorf("the message must state the measured offbox-bearing count (6), got:\n%s", msg)
+	}
+	// It must not prescribe the action for a failure mode that did not occur.
+	if !strings.Contains(msg, "Re-issue is NOT the indicated action") {
+		t.Errorf("the message must withdraw the Re-issue recommendation for this shape, got:\n%s", msg)
+	}
+	// The counting window is named, because CountReportsOffsiteSince reads the OLDEST 500 — an
+	// instrument that can silently mis-scope must say so where it reports (R-192's other half, open).
+	if !strings.Contains(msg, "500") {
+		t.Errorf("the message must name its counting window, got:\n%s", msg)
+	}
+
+	var details map[string]any
+	if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil {
+		t.Fatal(err)
+	}
+	if details["shape"] != "regressed" {
+		t.Errorf("details.shape = %v, want regressed", details["shape"])
+	}
+	// The field whose absence was defect (a) is now carried.
+	if got, ok := details["offsite_reports_since_consume"].(float64); !ok || int(got) != 6 {
+		t.Errorf("details must carry offsite_reports_since_consume=6, got %v", details["offsite_reports_since_consume"])
+	}
+}
+
+// The burned shape keeps its own (correct) text and its recommendation — the fix must not blunt the
+// case that was always right.
+func TestDeliveryChecker_BurnedShape_KeepsItsRecommendation(t *testing.T) {
+	h := newDTHarness(t, false)
+	h.burnedFixture(t, 5)
+
+	h.checker.Check()
+
+	saved := h.savedEvents(t, "offsite_delivery_stuck")
+	if len(saved) != 1 {
+		t.Fatalf("stuck events = %d, want 1", len(saved))
+	}
+	msg := saved[0].Message
+	if !strings.Contains(msg, "BURNED") || !strings.Contains(msg, "NONE carried an offbox target") {
+		t.Errorf("the burned shape must state its own measurement, got:\n%s", msg)
+	}
+	if !strings.Contains(msg, "Re-issue delivers a fresh one") {
+		t.Errorf("the burned shape must keep its recommendation, got:\n%s", msg)
+	}
+	var details map[string]any
+	_ = json.Unmarshal([]byte(saved[0].DetailsJSON), &details)
+	if details["shape"] != "burned" {
+		t.Errorf("details.shape = %v, want burned", details["shape"])
+	}
+}
+
+// Scenario G — a refusal to self-heal leaves a durable record naming its reason.
+// RED-PROOF: replace the recordHealRefusal call in the regressed branch with a bare `return`
+// (production behaviour up to v0.92.0) → no row → this FAILS.
+func TestDeliveryChecker_HealRefusal_LeavesARecord(t *testing.T) {
+	h := newDTHarness(t, true)
+	h.regressedFixture(t, 6, 2)
+
+	h.checker.Check()
+
+	if h.reissuer.count() != 0 {
+		t.Fatal("the heal must NOT fire on a regressed shape — the guard is unchanged by this fix")
+	}
+	rows := h.refusalRows(t)
+	if len(rows) != 1 {
+		t.Fatalf("R-192(b): the heal declined and left %d record(s), want 1 — 'we chose not to act' and "+
+			"'the heal never ran' must not look identical", len(rows))
+	}
+	r := rows[0]
+	if r.Channel != "operator" {
+		t.Errorf("refusal channel = %q, want operator (the channel the heal's own event would use)", r.Channel)
+	}
+	if !strings.Contains(r.ErrorMessage, "regressed-apply shape") {
+		t.Errorf("the record must name WHY, got %q", r.ErrorMessage)
+	}
+	if !strings.Contains(r.ErrorMessage, "6 of the first 8") {
+		t.Errorf("the record must carry the measurement it refused on, got %q", r.ErrorMessage)
+	}
+}
+
+// The R-39(a) guard's refusal is recorded too — it was the one refusal that already logged, and it
+// still must not be the only one that does.
+//
+// Reaching it needs the real TOCTOU, not a pre-staged row: staging BEFORE Check() makes the derive
+// itself return staged_awaiting_consume and the pass never reaches the heal at all. Same technique as
+// TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed — the operator stages mid-tick, on the
+// stuck event's dispatch, which runs after the derive and before maybeHeal.
+func TestDeliveryChecker_R39aRefusal_LeavesARecord(t *testing.T) {
+	h := newDTHarness(t, true)
+	h.burnedFixture(t, 5)
+
+	base := h.checker.onEvent
+	h.checker.onEvent = func(cid, et, sev, msg, det, src string) {
+		if et == "offsite_delivery_stuck" {
+			if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil {
+				t.Errorf("mid-tick stage: %v", err)
+			}
+			if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-23 12:00:00", ""); err != nil {
+				t.Errorf("mid-tick stamp: %v", err)
+			}
+		}
+		base(cid, et, sev, msg, det, src)
+	}
+
+	h.checker.Check()
+
+	if h.reissuer.count() != 0 {
+		t.Fatal("R-39(a): the heal must never restage over an unconsumed secret")
+	}
+	rows := h.refusalRows(t)
+	if len(rows) != 1 {
+		t.Fatalf("the R-39(a) refusal left %d record(s), want 1", len(rows))
+	}
+	if !strings.Contains(rows[0].ErrorMessage, "R-39(a)") {
+		t.Errorf("the record must name the guard, got %q", rows[0].ErrorMessage)
+	}
+}
+
+// The refusal record rides the stuck event's 24h cadence: it explains the e-mail the operator just
+// received, and must not accumulate one row per monitor tick.
+func TestDeliveryChecker_HealRefusal_RidesTheStuckCadence(t *testing.T) {
+	h := newDTHarness(t, true)
+	h.regressedFixture(t, 6, 2)
+
+	h.checker.Check()
+	h.checker.Check() // the stuck event is swallowed by its cooldown; so is the refusal record
+	h.checker.Check()
+
+	if n := len(h.refusalRows(t)); n != 1 {
+		t.Fatalf("refusal records = %d after three passes, want 1 (one per stuck e-mail, not per tick)", n)
+	}
+}

hub/internal/monitor/offsite_neverran_test.go

@@ -102,7 +102,7 @@ func TestOffsiteStale_NeverRanEscrowAnchorWins(t *testing.T) {
 		t.Fatal(err)
 	}
 	// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
-	if _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
+	if _, _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
 		t.Fatal(err)
 	}
 	saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))

hub/internal/notify/dispatcher.go

@@ -421,6 +421,29 @@ var operatorOnlyEvents = map[string]bool{
 	// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
 	// once (v0.78.0) and the comment above records it.
 	"backup_run_failures": true,
+	// R-197 (v0.93.0). "The sealed offsite repository key changed" is a custody fact about escrow
+	// blobs. A customer can take no action on it — the remedy is the operator's inspection of the
+	// off-site tier — and the text is operator-grade English naming host ids and retained-blob
+	// counts. Listed here in the SAME commit that mints the type: an operator-tier type that is not
+	// registered here reaches customers as raw English, because a missing customerMessages entry is
+	// NOT a block (the v0.78.0 defect recorded above).
+	"offsite_repo_key_changed": true,
+	// R-192 (v0.93.0). These two predate the register and were never added to it — a real gap, not a
+	// tidy-up. `offsite_delivery_stuck` is severity warning, has no customerMessages entry, and
+	// therefore fell through to FormatCustomerEmail's raw-English fallback: a customer whose box hit
+	// the stuck shape was in line for an English e-mail about one-time passwords being "likely
+	// burned". Measured on the live hub: notification_log holds operator rows for demo-hp and no
+	// customer rows — which is NOT evidence the leg is blocked (it is equally consistent with no
+	// configured recipient), so the register makes it structural instead of incidental. Narrowing
+	// only: the operator channel is untouched.
+	"offsite_delivery_stuck":      true,
+	"offsite_credential_restaged": true,
+	// R-199 (v0.94.0). A host retrieved its own sealed recovery blob. Operator-tier by construction:
+	// it names host ids and opaque byte counts, the customer can take no action on it, and its whole
+	// purpose is that the operator sees a capability being used. Registered in the same commit that
+	// mints the type — an operator-tier type absent from this register reaches customers as raw
+	// English (the v0.78.0 defect recorded above).
+	"escrow_blob_served": true,
 }
 
 // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the

hub/internal/offsite/offsite.go

@@ -195,35 +195,60 @@ func (p *Provisioner) ReissueCredentials(ctx context.Context, customerID, typ st
 		return fmt.Errorf("offsite: store re-issued one-time password: %w", err)
 	}
 
-	// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing
-	// key-escrow blob — which sealed the OLD password — is now STALE. A recovery code minted against
-	// it would decrypt a password that no longer opens the repo. Mark the escrow stale so the hub
-	// stops advertising "ceremony done" and the customer's escrow wizard is offered again; a fresh
-	// ceremony seals the new password and clears the flag. Every credential change also emits a
-	// visible customer event (offsite_reissued always; escrow_stale only when a blob was invalidated).
-	// Best-effort: the password reset already succeeded — a bookkeeping failure here must not fail it.
-	escrowStaled := false
-	if host, herr := p.Store.GetHostByCustomer(customerID); herr == nil && host != nil {
-		if esc, eerr := p.Store.GetHostEscrow(host.HostID); eerr == nil && esc != nil {
-			if serr := p.Store.MarkEscrowStale(host.HostID); serr != nil {
-				p.logf("[offsite] WARN mark-escrow-stale for %s: %v", customerID, serr)
-			} else {
-				escrowStaled = true
-			}
-		}
-	}
+	// v0.57.0 (2.3, the escrow-honesty fix). ⚠ ITS STATED REASON WAS FALSE AND IS CORRECTED HERE
+	// (R-196, 2026-08-04). The comment used to read "the restic repo password just changed" — this
+	// function does NOT change it and CANNOT: the repository password is generated on the box
+	// (controller `WriteOffboxSecrets`) and never leaves it except sealed inside the R-wrapped escrow.
+	// What this function changes is the PROVIDER account password (①), which is a different secret at
+	// a different layer. The false premise survived because the EFFECT (a stale escrow) is real, so
+	// nobody checked the CAUSE — and it sent two separate investigations the wrong way in one day
+	// (audits/SPIKE-offsite-credential-recovery-2026-08-04.md Q4).
+	//
+	// ⚠ THE MARK IS GONE (R-196 / R-204 item 2, hub v0.95.0, 2026-08-05). What used to stand here —
+	// a `MarkEscrowStale` on every re-issue that found an escrow row, plus an `escrow_stale` customer
+	// event — was PRECAUTIONARY, not evidential: it guessed that the box's re-apply MIGHT mint a fresh
+	// repository password. On the ordinary re-issue shape (a box that still holds its
+	// `<DataDir>/offbox/repo_password`) the password does not change, so it marked a HEALTHY escrow
+	// stale. The 2026-08-04 recovery drill (R-201) is what promoted this from a nit to a blocker.
+	//
+	// WHAT THE MARK ACTUALLY DID, mechanically, because "it asked for an unnecessary ceremony"
+	// understates it by a lot:
+	//   1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report
+	//      ACK (store.go, the v0.57.0 rule).
+	//   2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed
+	//      (report.EscrowAutoConfirmer.Reconcile returns early on an empty hash).
+	//   3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → EVERY off-site backup
+	//      is refused, indefinitely, on a box whose key was never in doubt.
+	//   4. The customer is told to re-run the recovery ceremony — which mints a NEW recovery code and
+	//      supersedes the sealed blob. During a recovery that is the one act that would have destroyed
+	//      the key just recovered.
+	// A precautionary flag that stops the data-protection it is guarding is not conservative.
+	//
+	// WHY REMOVING IT LEAVES NO GAP — the case it guessed at is MEASURED elsewhere, and better:
+	//   • Continuous, box-side: the controller compares the ACK's sealed hash against its CURRENT
+	//     local repo password on EVERY report ACK (`reconcileEscrowed`, the Scenario-F re-check). In
+	//     the guest-rebuild shape — the only shape where a re-issue is followed by a fresh repository
+	//     password — that comparison mismatches within one report cycle and raises the stale card plus
+	//     the „create a new recovery code" CTA. It is a measurement, not a guess.
+	//     AND THE MARK WAS BLINDING IT: by emptying the hash (step 1 above) it removed the very value
+	//     that comparison needs, so the box could only report the hash-LESS reason, which is false.
+	//   • Edge-triggered, hub-side: R-197's `offsite_repo_key_changed` fires on a proven hash
+	//     difference across a supersession (api.maybeEmitRepoKeyChanged) and pages the operator.
+	//
+	// DISAGREEMENT RECORDED, per the R-96 standing rule: the task's Scenario D asks that a real key
+	// change "marks the escrow stale". It must NOT, and nothing here was changed to make it: the hub
+	// learns of a real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow
+	// is freshest. Marking it stale there would ask for a ceremony to fix the ceremony that just ran.
+	// The correct consequence at that instant is the operator alarm, which is what R-197 does.
+	//
+	// `offsite_reissued` is UNCHANGED and still always fires — the customer must still learn that the
+	// credential moved. Best-effort: the password reset already succeeded, so a bookkeeping failure
+	// here must not fail it.
 	if _, serr := p.Store.SaveEvent(customerID, "offsite_reissued", "info",
 		"Az offsite (házon kívüli) mentési hozzáférést újra kiadtuk — az új egyszeri jelszót a vezérlő a következő frissítéskor átveszi.",
 		"", "hub"); serr != nil {
 		p.logf("[offsite] WARN save offsite_reissued event for %s: %v", customerID, serr)
 	}
-	if escrowStaled {
-		if _, serr := p.Store.SaveEvent(customerID, "escrow_stale", "warning",
-			"A helyreállítási kulcs-letét elavult az offsite jelszó cseréje miatt — futtasd le újra a helyreállítási szertartást (Biztonsági mentés → Helyreállítás).",
-			"", "hub"); serr != nil {
-			p.logf("[offsite] WARN save escrow_stale event for %s: %v", customerID, serr)
-		}
-	}
 	return nil
 }
 

hub/internal/offsite/offsite_test.go

@@ -211,12 +211,27 @@ func TestReissue_RefusesAmbiguousLookup(t *testing.T) {
 	}
 }
 
-// v0.57.0 (2.3, escrow honesty) — re-issuing offsite credentials INVALIDATES the key-escrow blob:
-// the blob sealed the OLD repo password, so a recovery code minted against it would decrypt a
-// password that no longer opens the repo. RED-PROOF (Scenario C): on pre-fix code (no MarkEscrowStale
-// in ReissueCredentials + no stale plumbing) the hub keeps advertising the escrow as current after a
-// re-issue and keeps serving its restic-hash for auto-confirm — this test asserts it does NEITHER.
-func TestReissue_InvalidatesEscrow(t *testing.T) {
+// SCENARIO C (R-196 / R-204 item 2, hub v0.95.0) — re-issuing the offsite PROVIDER credential must
+// LEAVE A HEALTHY ESCROW ALONE.
+//
+// THIS TEST REPLACES `TestReissue_InvalidatesEscrow`, which asserted the opposite and was wrong on a
+// false premise: a re-issue resets the storage-provider account password and cannot touch the restic
+// REPOSITORY password, which is generated on the box and never leaves it except sealed under R. The
+// old assertion looked right because the EFFECT it checked (a stale escrow) was real — nobody checked
+// the CAUSE. Kept as one test rather than deleted so the inversion is visible in `git log -L`.
+//
+// WHAT IT ASSERTS IS THE CONSEQUENCE, not the mechanism: not "MarkEscrowStale was not called" but
+// "the ACK still carries the hash", because the hash is what the controller's auto-confirm needs and
+// its absence is what blocked every off-site backup during the 2026-08-04 drill.
+//
+// RED-PROOF: restore the marking — put back
+//
+//	if host, _ := p.Store.GetHostByCustomer(customerID); host != nil {
+//	    if esc, _ := p.Store.GetHostEscrow(host.HostID); esc != nil { p.Store.MarkEscrowStale(host.HostID) }
+//	}
+//
+// in ReissueCredentials → es.Stale becomes true, the hash is withheld, and this test fails on both.
+func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
 	p, _, st := newTestProvisioner(t)
 	const cust = "cust-esc"
 	if _, err := p.ProvisionOffsite(context.Background(), cust, Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil {
@@ -226,10 +241,9 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
 	if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
+	if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
 		t.Fatal(err)
 	}
-	// Before re-issue: current escrow — the hub serves the sealed hash and is NOT stale.
 	es, err := st.GetEscrowStatusForCustomer(cust)
 	if err != nil || es == nil {
 		t.Fatalf("escrow status (before): %v", err)
@@ -238,25 +252,56 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
 		t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256)
 	}
 
-	// Re-issue the offsite credential — the repo password just changed under the sealed blob.
+	// Re-issue the offsite PROVIDER credential. The box still holds its repo_password file, so the
+	// repository password does NOT change — this is the ordinary re-issue shape.
 	if err := p.ReissueCredentials(context.Background(), cust, "shared"); err != nil {
 		t.Fatalf("reissue: %v", err)
 	}
 
-	// After: the escrow is STALE and the restic-hash is WITHHELD (no auto-confirm against a dead key).
 	es, err = st.GetEscrowStatusForCustomer(cust)
 	if err != nil || es == nil {
 		t.Fatalf("escrow status (after): %v", err)
 	}
-	if !es.Stale {
-		t.Fatal("RED-PROOF: escrow must be STALE after an offsite re-issue (the hub was advertising ceremony-done against a key the repo no longer accepts)")
+	if es.Stale {
+		t.Fatal("a re-issue marked a HEALTHY escrow stale — this blocks every off-site backup and asks the customer for a ceremony that would supersede a good key (R-196)")
 	}
-	if es.ResticPwSHA256 != "" {
-		t.Fatalf("a stale escrow must WITHHOLD the restic hash to inhibit auto-confirm, got %q", es.ResticPwSHA256)
+	if es.ResticPwSHA256 != "OLDHASH" {
+		t.Fatalf("the sealed hash must keep flowing to the ACK so the controller can auto-confirm; got %q", es.ResticPwSHA256)
 	}
 
+	// The customer is still TOLD the credential moved — removing the false alarm must not remove the
+	// true notice. `offsite_reissued` is untouched by this change.
+	ev, eerr := st.GetLatestEventByType(cust, "offsite_reissued")
+	if eerr != nil || ev == nil {
+		t.Fatalf("offsite_reissued must still fire on every re-issue (err=%v ev=%+v)", eerr, ev)
+	}
+	// …and the false alarm is gone.
+	if bad, _ := st.GetLatestEventByType(cust, "escrow_stale"); bad != nil {
+		t.Fatalf("a re-issue must not raise escrow_stale on a healthy escrow: %+v", bad)
+	}
+}
+
+// The other half of Scenario C: the stale MECHANISM is intact and still does its job when something
+// legitimately sets it. Without this, "nothing marks stale any more" could decay into "stale no longer
+// works", and the next evidential caller would ship inert (the seam-built-but-never-wired shape).
+func TestEscrowStaleMechanism_StillWithholdsAndClears(t *testing.T) {
+	_, _, st := newTestProvisioner(t)
+	const cust = "cust-stale-mech"
+	if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
+		t.Fatal(err)
+	}
+	if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
+		t.Fatal(err)
+	}
+	if err := st.MarkEscrowStale(cust + "-01"); err != nil {
+		t.Fatalf("MarkEscrowStale: %v", err)
+	}
+	es, _ := st.GetEscrowStatusForCustomer(cust)
+	if es == nil || !es.Stale || es.ResticPwSHA256 != "" {
+		t.Fatalf("a marked escrow must read stale AND withhold the hash: %+v", es)
+	}
 	// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
-	if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
+	if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
 		t.Fatal(err)
 	}
 	es, _ = st.GetEscrowStatusForCustomer(cust)

hub/internal/offsiteheal/reconciler.go

@@ -0,0 +1,284 @@
+// Package offsiteheal is the hub-side OFF-SITE credential self-heal reconciler (R-193 / R-204 item 4,
+// hub v0.96.0). It is the sibling of internal/pbsdrheal, deliberately: same shape, same restraint,
+// same rule that one thing is left loud rather than healed.
+//
+// THE PROBLEM. The 2026-08-04 drill (R-201) proved a customer's file survives a machine rebuild and
+// comes back — with a person present for four manual interventions. Three were closed in controller
+// v0.198.0 / hub v0.95.0. The fourth is this one: a REBUILT box has no off-site credential of its own,
+// because the one-time provider password was spent by its predecessor. Everything downstream is
+// self-service; nothing gets the box past that first step.
+//
+// WHY THE TRIGGER IS A DECLARATION AND NOT AN INFERENCE (the operator ruling, 2026-08-05, and the
+// whole design). From the hub, an ABSENT off-site object has FOUR meanings — never configured,
+// mid-restart, a transient config read failure, and rebuilt-and-stranded — and the hub cannot tell
+// them apart. The BOX can, from two local facts it holds with certainty: its data area is fresh (no
+// repository password) AND the hub is holding a sealed recovery package for it. So the box says so,
+// in its ordinary report, and this reconciler acts on a stated request rather than on a silence.
+// That is also why R-192's counting guard is REPLACED rather than repaired: it inferred the same
+// thing by counting reports over the OLDEST 500 after a consume, all of which predate a rebuild.
+//
+// WHAT IT DOES, once a declaration has held across a debounce: RE-STAGES the customer's stored
+// one-time secret (store.RestageOneTimeSecret — no provider call, no new password) and escalates to
+// the existing Re-issue ONLY when there is nothing stored to re-arm. It NEVER blind-timer-reissues
+// and NEVER re-runs a provisioning step. A box that is not declaring is a pure no-op.
+//
+// WHAT IT DELIBERATELY DOES NOT DO: it never runs, or asks for, an escrow ceremony. A credential is
+// replaceable; the recovery code is not, because only the customer holds it. Credential automatic,
+// key customer-present — the ruling this session implements and must not quietly widen.
+package offsiteheal
+
+import (
+	"context"
+	"log"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
+	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
+)
+
+// StateNeedsCredential is the ONE state acted on — the string controller v0.199.0 declares on its
+// report's `offsite` object (backup.OffsiteStateNeedsCredential). Everything else, including "" (an
+// older controller, a healthy box, or a box with no off-site object at all), is a no-op.
+//
+// THE §8.4 QUESTION — IS THERE AN EQUIVALENT OF pbsdrheal's DELIBERATELY-UNHEALED `verify_failed`?
+// Yes, and it is handled by CONSTRUCTION rather than by a case here, which is worth stating plainly
+// because "there is nothing like that here" is usually wrong. The analogue is the REGRESSED shape:
+// a box that HAD a working off-site tier and lost its target while still holding its repository
+// password. Re-arming a credential would not help it — its problem is whatever removed the target —
+// and R-192's existing checker already refuses to heal that shape for exactly this reason. It cannot
+// reach this reconciler at all, because the controller's declaration predicate requires the
+// repository password to be ABSENT (backup.needsOffsiteCredential). So the unhealable case is
+// excluded upstream, by the declaration itself, rather than filtered out here.
+const StateNeedsCredential = "needs_credential"
+
+// Audit event types (store.SaveEvent; hub-internal, not gated by allowedEventTypes — the pbsdr_*
+// precedent). Distinct per remediation so the operator sees exactly what was auto-done.
+const (
+	eventRestaged = "offsite_selfheal_restaged" // re-armed the stored secret (routine, no provider call)
+	eventReissued = "offsite_selfheal_reissued" // nothing stored to re-arm → minted a fresh credential
+)
+
+// debounceReportsDefault — how many DISTINCT reports must carry the declaration before acting.
+//
+// TWO, and the interval is derived rather than chosen. The controller reports every ~15 minutes, so
+// two distinct declarations mean the state has survived at least one full report cycle: a restart, a
+// slow first report or a transient config read cannot produce it, because each of those resolves
+// well inside one cycle. One report would act on a blip; three would leave a genuinely stranded
+// customer waiting ~45 minutes for a credential they cannot obtain any other way. The reconciler's
+// own tick (below) is deliberately FASTER than the report cadence so it never adds latency of its
+// own — the debounce is counted in fresh evidence, not in ticks.
+const debounceReportsDefault = 2
+
+// tickIntervalDefault — how often the fleet is swept. Shorter than the 15-minute report cadence on
+// purpose (see above); a sweep that finds nothing writes nothing.
+const tickIntervalDefault = 5 * time.Minute
+
+// Actions is the mutation seam — fakes in tests count calls without touching a provider. Restage
+// flips a stored secret's consumed flag (false when NO row exists → the caller escalates); Reissue
+// mints a fresh provider credential and stores a fresh consume-once secret.
+type Actions interface {
+	Restage(customerID string) (restaged bool, err error)
+	Reissue(ctx context.Context, customerID string) error
+}
+
+// Reissuer is satisfied by *web.Server (its ReissueOffsiteForCustomer) — the same indirection
+// pbsdrheal uses, so the escalation IS the designed Re-issue path and not a sibling mechanism.
+type Reissuer interface {
+	ReissueOffsiteForCustomer(ctx context.Context, customerID string) error
+}
+
+type storeActions struct {
+	st       *store.Store
+	reissuer Reissuer
+}
+
+func (a storeActions) Restage(customerID string) (bool, error) {
+	return a.st.RestageOneTimeSecret(customerID)
+}
+func (a storeActions) Reissue(ctx context.Context, customerID string) error {
+	return a.reissuer.ReissueOffsiteForCustomer(ctx, customerID)
+}
+
+// NewActions builds the production mutation seam.
+func NewActions(st *store.Store, reissuer Reissuer) Actions { return storeActions{st: st, reissuer: reissuer} }
+
+// debounceState tracks, per customer, the last DISTINCT report seen and how many consecutive
+// distinct reports have carried the declaration.
+type debounceState struct {
+	reportID int64
+	state    string
+	streak   int
+}
+
+// Reconciler re-arms stranded boxes. DECLARATIVE + IDEMPOTENT: a tick over a healthy fleet writes
+// nothing. It reads the hub DB — never the box.
+type Reconciler struct {
+	store           *store.Store
+	act             Actions
+	interval        time.Duration
+	debounceReports int
+	onlyCustomer    string // "" = whole fleet; non-empty restricts the work set (supervised rollout)
+	trigger         chan struct{}
+	logger          *log.Logger
+
+	deb map[string]debounceState
+}
+
+// NewReconciler builds the reconciler with the derived defaults.
+func NewReconciler(st *store.Store, act Actions, logger *log.Logger) *Reconciler {
+	if logger == nil {
+		logger = log.Default()
+	}
+	return &Reconciler{
+		store:           st,
+		act:             act,
+		interval:        tickIntervalDefault,
+		debounceReports: debounceReportsDefault,
+		trigger:         make(chan struct{}, 1),
+		logger:          logger,
+		deb:             map[string]debounceState{},
+	}
+}
+
+// RestrictToCustomer scopes the work set to one customer (empty = whole fleet) for a supervised first
+// rollout — the pbsdrheal precedent. Set before Run.
+func (r *Reconciler) RestrictToCustomer(customerID string) { r.onlyCustomer = customerID }
+
+// SetDebounceReports overrides the debounce (tests). Values < 1 are ignored.
+func (r *Reconciler) SetDebounceReports(n int) {
+	if n >= 1 {
+		r.debounceReports = n
+	}
+}
+
+// Trigger requests an immediate reconcile. Non-blocking.
+func (r *Reconciler) Trigger() {
+	select {
+	case r.trigger <- struct{}{}:
+	default:
+	}
+}
+
+// Run loops until ctx is done. Never exits on an error.
+func (r *Reconciler) Run(ctx context.Context) {
+	ticker := time.NewTicker(r.interval)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-r.trigger:
+		case <-ticker.C:
+		}
+		r.ReconcileOnce(ctx)
+	}
+}
+
+// ReconcileOnce sweeps the fleet once. Exported so tests drive it directly instead of racing a
+// ticker, and so the wiring can trigger it. Errors are logged and retried next tick; a read failure
+// means this reconciler does NOTHING rather than acting on a partial view.
+func (r *Reconciler) ReconcileOnce(ctx context.Context) {
+	configs, err := r.store.ListCustomerConfigs()
+	if err != nil {
+		r.logger.Printf("[ERROR] offsiteheal: list customer configs: %v (retry next tick)", err)
+		return
+	}
+	seen := make(map[string]bool, len(configs))
+	for _, cfg := range configs {
+		if r.onlyCustomer != "" && cfg.CustomerID != r.onlyCustomer {
+			continue
+		}
+		if cfg.Status != "active" {
+			continue // blocked/inactive customers are never healed
+		}
+		// The descriptor must still say the customer HAS an off-site tier. A deliberately disabled
+		// tier must not be re-credentialed behind the operator's back; an unparseable config never
+		// drives a heal (the config UI owns that failure).
+		d, derr := offsite.ReadDescriptor(cfg.ConfigJSON)
+		if derr != nil || d == nil || !d.Enabled {
+			delete(r.deb, cfg.CustomerID)
+			continue
+		}
+		found, reportID, state, err := r.store.LatestReportOffsiteDeclaration(cfg.CustomerID)
+		if err != nil {
+			r.logger.Printf("[ERROR] offsiteheal: %s: read latest declaration: %v (retry next tick)", cfg.CustomerID, err)
+			continue // never act on a partial view
+		}
+		seen[cfg.CustomerID] = true
+		if !found || state != StateNeedsCredential {
+			delete(r.deb, cfg.CustomerID) // healthy, silent, or an older controller → pure no-op
+			continue
+		}
+		if !r.confirm(cfg.CustomerID, reportID, state) {
+			continue // one declaration is not evidence — wait for a second distinct report
+		}
+		r.heal(ctx, cfg.CustomerID, reportID, state)
+	}
+	for c := range r.deb {
+		if !seen[c] {
+			delete(r.deb, c)
+		}
+	}
+}
+
+// confirm advances the per-customer debounce and reports whether the declaration has held across
+// >= debounceReports DISTINCT reports. A re-observed same report never advances the streak — the
+// debounce counts fresh evidence, not reconciler ticks, so a fast tick cannot shorten it.
+func (r *Reconciler) confirm(customerID string, reportID int64, state string) bool {
+	st := r.deb[customerID]
+	if reportID != st.reportID {
+		if state == st.state {
+			st.streak++
+		} else {
+			st.streak = 1
+		}
+		st.state = state
+		st.reportID = reportID
+		r.deb[customerID] = st
+	}
+	return st.streak >= r.debounceReports
+}
+
+// heal re-arms the stored secret; with nothing stored, escalates to a fresh mint.
+//
+// RESTAGE BEFORE MINT, because re-arming costs no provider call and converges in one report tick,
+// while a mint is external churn for a credential the hub already holds. The stored value survives a
+// consume (store.RestageOneTimeSecret documents how that was established), which is what makes the
+// cheap path possible at all.
+func (r *Reconciler) heal(ctx context.Context, customerID string, reportID int64, state string) {
+	restaged, err := r.act.Restage(customerID)
+	if err != nil {
+		r.logger.Printf("[ERROR] offsiteheal: re-stage %s: %v (retry next tick)", customerID, err)
+		return
+	}
+	if restaged {
+		r.logger.Printf("[INFO] offsiteheal: re-staged the stored one-time offsite secret for customer %s (declared %s across %d reports) — the box re-consumes on its next cycle; no provider credential was minted",
+			customerID, state, r.debounceReports)
+		r.event(customerID, eventRestaged, "info",
+			"Offsite self-heal: a rebuilt box asked for its storage credential and the stored one-time password was re-armed. No new credential was created at the storage provider.")
+		r.resetAfterHeal(customerID, reportID, state)
+		return
+	}
+	r.logger.Printf("[INFO] offsiteheal: customer %s declares %s with NO stored one-time secret — escalating to Re-issue", customerID, state)
+	if err := r.act.Reissue(ctx, customerID); err != nil {
+		r.logger.Printf("[ERROR] offsiteheal: re-issue for customer %s: %v (retry next tick)", customerID, err)
+		return
+	}
+	r.event(customerID, eventReissued, "warning",
+		"Offsite self-heal: a rebuilt box asked for its storage credential and the hub had none stored, so fresh credentials were issued at the storage provider.")
+	r.resetAfterHeal(customerID, reportID, state)
+}
+
+// resetAfterHeal clears the streak (keeping the report id) so the SAME report cannot re-heal on the
+// next tick — a fresh report must re-confirm the box is still stranded before acting again. This is
+// what makes Scenario E's "one mint, not repeated" true.
+func (r *Reconciler) resetAfterHeal(customerID string, reportID int64, state string) {
+	r.deb[customerID] = debounceState{reportID: reportID, state: state, streak: 0}
+}
+
+// event records an audit row; a failure to write it must never break the heal loop.
+func (r *Reconciler) event(customerID, eventType, severity, message string) {
+	if _, err := r.store.SaveEvent(customerID, eventType, severity, message, "", "hub"); err != nil {
+		r.logger.Printf("[WARN] offsiteheal: audit event %s for %s not stored: %v", eventType, customerID, err)
+	}
+}

hub/internal/offsiteheal/reconciler_test.go

@@ -0,0 +1,310 @@
+package offsiteheal
+
+// Non-hollow reconciler tests mapping 1:1 to the task's integration scenarios A–F. A REAL store
+// (t.TempDir sqlite) supplies the work set, the customer configs and the reports; a FAKE Actions seam
+// counts restage/reissue calls without touching a storage provider. ReconcileOnce is driven directly
+// for determinism — the debounce is exercised by feeding DISTINCT REPORTS, never by sleeping, because
+// the debounce counts fresh evidence rather than elapsed time.
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"log"
+	"path/filepath"
+	"sync"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-hub/internal/store"
+)
+
+type fakeActions struct {
+	mu            sync.Mutex
+	restaged      []string
+	reissued      []string
+	restageResult bool // does a stored secret row exist?
+	restageErr    error
+	reissueErr    error
+}
+
+func (f *fakeActions) Restage(customerID string) (bool, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.restaged = append(f.restaged, customerID)
+	return f.restageResult, f.restageErr
+}
+
+func (f *fakeActions) Reissue(_ context.Context, customerID string) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.reissued = append(f.reissued, customerID)
+	return f.reissueErr
+}
+
+func (f *fakeActions) restages() int { f.mu.Lock(); defer f.mu.Unlock(); return len(f.restaged) }
+func (f *fakeActions) reissues() int { f.mu.Lock(); defer f.mu.Unlock(); return len(f.reissued) }
+
+func newHealStore(t *testing.T) *store.Store {
+	t.Helper()
+	st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
+	if err != nil {
+		t.Fatalf("store.New: %v", err)
+	}
+	t.Cleanup(func() { st.Close() })
+	return st
+}
+
+// seedCustomer writes an ACTIVE customer whose offsite descriptor is enabled (or not).
+func seedCustomer(t *testing.T, st *store.Store, customerID string, offsiteEnabled bool) {
+	t.Helper()
+	cfgJSON := `{"offsite":{"enabled":false}}`
+	if offsiteEnabled {
+		cfgJSON = `{"offsite":{"enabled":true,"type":"shared","host":"x.your-storagebox.de","user":"u","port":23,"repo_path":"/home/felhom-repo"}}`
+	}
+	if err := st.SaveCustomerConfig(&store.CustomerConfig{
+		CustomerID: customerID, CustomerName: customerID, Domain: customerID + ".hu",
+		APIKey: "k-" + customerID, ConfigJSON: cfgJSON,
+	}); err != nil {
+		t.Fatalf("SaveCustomerConfig %s: %v", customerID, err)
+	}
+}
+
+// pushReport writes ONE controller report. state=="" → a healthy configured box (enabled offsite
+// object); otherwise the declaration shape controller v0.199.0 emits. Each call is a DISTINCT report,
+// which is the debounce's currency.
+func pushReport(t *testing.T, st *store.Store, customerID, state string, seq int) {
+	t.Helper()
+	var off map[string]any
+	if state == "" {
+		off = map[string]any{"enabled": true, "escrow_state": "escrowed", "last_status": "ok"}
+	} else {
+		off = map[string]any{"enabled": false, "state": state}
+	}
+	b, err := json.Marshal(map[string]any{
+		"controller_version": fmt.Sprintf("0.199.0-%d", seq), // makes each row distinguishable
+		"offsite":            off,
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := st.SaveReport(customerID, b); err != nil {
+		t.Fatalf("SaveReport %s: %v", customerID, err)
+	}
+}
+
+// pushReportNoOffsite writes a report with NO offsite object at all — the pre-v0.199.0 shape a
+// stranded box used to send, and the shape a box that never had off-site backups still sends.
+func pushReportNoOffsite(t *testing.T, st *store.Store, customerID string, seq int) {
+	t.Helper()
+	b, _ := json.Marshal(map[string]any{"controller_version": fmt.Sprintf("0.198.0-%d", seq)})
+	if err := st.SaveReport(customerID, b); err != nil {
+		t.Fatalf("SaveReport %s: %v", customerID, err)
+	}
+}
+
+func newRec(st *store.Store, act Actions) *Reconciler {
+	return NewReconciler(st, act, log.New(io.Discard, "", 0))
+}
+
+func countEvents(t *testing.T, st *store.Store, customerID, eventType string) int {
+	t.Helper()
+	ev, err := st.GetLatestEventByType(customerID, eventType)
+	if err != nil {
+		t.Fatalf("GetLatestEventByType: %v", err)
+	}
+	if ev == nil {
+		return 0
+	}
+	return 1
+}
+
+// SCENARIO A + D — a declaring box is served, and the STORED secret is re-armed rather than a fresh
+// credential minted.
+//
+// RED-PROOF (D): invert the order in heal() so Reissue is called before Restage — the assertion
+// "no mint" fails, and the test names the cost: external churn for a credential the hub already has.
+func TestScenarioAD_DeclaringBoxIsRestagedNotMinted(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: true} // a stored secret exists
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", true)
+
+	pushReport(t, st, "c1", StateNeedsCredential, 1)
+	r.ReconcileOnce(context.Background())
+	if act.restages() != 0 || act.reissues() != 0 {
+		t.Fatalf("acted on a SINGLE declaration — the debounce did not hold (restages=%d reissues=%d)", act.restages(), act.reissues())
+	}
+
+	pushReport(t, st, "c1", StateNeedsCredential, 2) // a second DISTINCT report
+	r.ReconcileOnce(context.Background())
+
+	if act.restages() != 1 {
+		t.Fatalf("a sustained declaration was not served: restages=%d", act.restages())
+	}
+	if act.reissues() != 0 {
+		t.Fatalf("a fresh credential was minted although a stored one could be re-armed — external churn for nothing (reissues=%d)", act.reissues())
+	}
+	if countEvents(t, st, "c1", eventRestaged) != 1 {
+		t.Error("no offsite_selfheal_restaged audit event — an automatic remediation must be visible")
+	}
+	if countEvents(t, st, "c1", eventReissued) != 0 {
+		t.Error("a reissue event was recorded for a restage")
+	}
+}
+
+// SCENARIO B — a box that never had off-site backups says nothing, so nothing happens. (The
+// declaration itself is the controller's guard; here we assert the hub side is a no-op on silence.)
+func TestScenarioB_SilentBoxIsNeverActedOn(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: true}
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", true)
+
+	for i := 1; i <= 4; i++ {
+		pushReportNoOffsite(t, st, "c1", i)
+		r.ReconcileOnce(context.Background())
+	}
+	if act.restages() != 0 || act.reissues() != 0 {
+		t.Fatalf("a silent box was acted on: restages=%d reissues=%d", act.restages(), act.reissues())
+	}
+}
+
+// SCENARIO C — a healthy box is a PURE no-op, repeatedly. No restage, no mint, no event.
+func TestScenarioC_HealthyBoxIsAPureNoOp(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: true}
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", true)
+
+	for i := 1; i <= 5; i++ {
+		pushReport(t, st, "c1", "", i) // healthy: enabled offsite object, no declaration
+		r.ReconcileOnce(context.Background())
+	}
+	if act.restages() != 0 || act.reissues() != 0 {
+		t.Fatalf("a healthy box was acted on: restages=%d reissues=%d", act.restages(), act.reissues())
+	}
+	if countEvents(t, st, "c1", eventRestaged)+countEvents(t, st, "c1", eventReissued) != 0 {
+		t.Error("a healthy box produced a self-heal event")
+	}
+}
+
+// SCENARIO E — nothing to re-arm escalates to a mint, ONCE, and does not repeat while the staged
+// secret sits unconsumed (a fresh report must re-confirm before acting again).
+func TestScenarioE_NothingToRestageEscalatesOnce(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: false} // NO stored secret
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", true)
+
+	pushReport(t, st, "c1", StateNeedsCredential, 1)
+	r.ReconcileOnce(context.Background())
+	pushReport(t, st, "c1", StateNeedsCredential, 2)
+	r.ReconcileOnce(context.Background())
+
+	if act.reissues() != 1 {
+		t.Fatalf("escalation did not happen exactly once: reissues=%d", act.reissues())
+	}
+	if countEvents(t, st, "c1", eventReissued) != 1 {
+		t.Error("no offsite_selfheal_reissued audit event")
+	}
+
+	// Repeated ticks on the SAME report must not mint again — this is the "a mint on every tick"
+	// failure the sibling's resetAfterHeal exists to prevent.
+	for i := 0; i < 5; i++ {
+		r.ReconcileOnce(context.Background())
+	}
+	if act.reissues() != 1 {
+		t.Fatalf("minted repeatedly on an unchanged report: reissues=%d", act.reissues())
+	}
+}
+
+// SCENARIO F — a BLIP is absorbed: one declaring report followed by a healthy one triggers nothing.
+//
+// RED-PROOF: set debounceReports to 1 (i.e. remove the debounce) — the blip acts, and a credential is
+// churned by what was only a restart.
+func TestScenarioF_BlipIsAbsorbedByTheDebounce(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: true}
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", true)
+
+	pushReport(t, st, "c1", StateNeedsCredential, 1) // the blip
+	r.ReconcileOnce(context.Background())
+	pushReport(t, st, "c1", "", 2) // healthy again
+	r.ReconcileOnce(context.Background())
+
+	if act.restages() != 0 || act.reissues() != 0 {
+		t.Fatalf("a one-report blip triggered a credential action: restages=%d reissues=%d", act.restages(), act.reissues())
+	}
+
+	// And the streak must have been FORGOTTEN, not merely paused: a single later declaration must
+	// still not act.
+	pushReport(t, st, "c1", StateNeedsCredential, 3)
+	r.ReconcileOnce(context.Background())
+	if act.restages() != 0 {
+		t.Fatal("the debounce streak survived a healthy report — a flapping box would be healed on every other cycle")
+	}
+}
+
+// A DISABLED offsite descriptor is the operator's own choice and must never be re-credentialed,
+// even if the box declares (e.g. an old declaration left in the newest report).
+func TestDisabledDescriptorIsNeverHealed(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: true}
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", false) // offsite disabled in the config
+
+	for i := 1; i <= 4; i++ {
+		pushReport(t, st, "c1", StateNeedsCredential, i)
+		r.ReconcileOnce(context.Background())
+	}
+	if act.restages() != 0 || act.reissues() != 0 {
+		t.Fatalf("a customer whose offsite is DISABLED was re-credentialed: restages=%d reissues=%d", act.restages(), act.reissues())
+	}
+}
+
+// A blocked customer is never healed.
+func TestBlockedCustomerIsNeverHealed(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: true}
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", true)
+	// SaveCustomerConfig always writes status 'active' (its INSERT does not carry the column), so the
+	// block must go through the real setter — asserted below, because a test that silently failed to
+	// block would pass for the wrong reason.
+	if err := st.SetCustomerConfigStatus("c1", "blocked"); err != nil {
+		t.Fatalf("SetCustomerConfigStatus: %v", err)
+	}
+	if !st.IsCustomerBlocked("c1") {
+		t.Fatal("precondition: the customer is not actually blocked — this test would pass vacuously")
+	}
+	for i := 1; i <= 4; i++ {
+		pushReport(t, st, "c1", StateNeedsCredential, i)
+		r.ReconcileOnce(context.Background())
+	}
+	if act.restages() != 0 || act.reissues() != 0 {
+		t.Fatalf("a BLOCKED customer was healed: restages=%d reissues=%d", act.restages(), act.reissues())
+	}
+}
+
+// A restage ERROR must not escalate to a mint — the reconciler does nothing and retries. Acting on a
+// failed read is the "never act on a partial view" rule.
+func TestRestageErrorDoesNotEscalate(t *testing.T) {
+	st := newHealStore(t)
+	act := &fakeActions{restageResult: true, restageErr: fmt.Errorf("db is busy")}
+	r := newRec(st, act)
+	seedCustomer(t, st, "c1", true)
+
+	pushReport(t, st, "c1", StateNeedsCredential, 1)
+	r.ReconcileOnce(context.Background())
+	pushReport(t, st, "c1", StateNeedsCredential, 2)
+	r.ReconcileOnce(context.Background())
+
+	if act.reissues() != 0 {
+		t.Fatalf("a failed re-stage escalated to an external mint: reissues=%d", act.reissues())
+	}
+	if countEvents(t, st, "c1", eventRestaged) != 0 {
+		t.Error("a failed re-stage recorded a success event")
+	}
+}

hub/internal/offsiteheal/wiring_test.go

@@ -0,0 +1,58 @@
+package offsiteheal
+
+import (
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"testing"
+)
+
+// SCENARIO H — the reconciler is WIRED, asserted from main.go's source.
+//
+// Every test in this package passes on a reconciler that main.go never starts. That is this
+// project's most-repeated failure shape: six features built and never wired, one of them an off-site
+// restage event that existed and never fired once. The whole of R-204 item 4 is worth nothing if
+// `Run` is not called.
+//
+// It walks the AST rather than grepping, because a commented-out call still contains the string, and
+// it parses with comments DROPPED so a commented `go rec.Run(ctx)` cannot satisfy it.
+//
+// RED-PROOF: comment out the `go offsiteReconciler.Run(ctx)` line in cmd/hub/main.go → this fails.
+func TestMainWiresTheOffsiteHealReconciler(t *testing.T) {
+	const mainPath = "../../cmd/hub/main.go"
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, mainPath, nil, 0) // comments dropped on purpose
+	if err != nil {
+		t.Fatalf("parse %s: %v — the reconciler's wiring is now unasserted", mainPath, err)
+	}
+
+	var sawConstruct, sawRun bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		call, ok := n.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		sel, ok := call.Fun.(*ast.SelectorExpr)
+		if !ok {
+			return true
+		}
+		pkg, isIdent := sel.X.(*ast.Ident)
+		// offsiteheal.NewReconciler(...)
+		if isIdent && pkg.Name == "offsiteheal" && sel.Sel.Name == "NewReconciler" {
+			sawConstruct = true
+		}
+		// <something>.Run(ctx) — the receiver is a local variable, so match on the method name and
+		// confirm the construction separately. Narrow enough: this file has one Run per reconciler.
+		if sel.Sel.Name == "Run" && isIdent && pkg.Name == "offsiteReconciler" {
+			sawRun = true
+		}
+		return true
+	})
+
+	if !sawConstruct {
+		t.Fatal("cmd/hub/main.go never calls offsiteheal.NewReconciler — R-204 item 4 ships inert")
+	}
+	if !sawRun {
+		t.Fatal("the offsite self-heal reconciler is CONSTRUCTED but never Run — a stranded box would declare forever and nothing would answer")
+	}
+}

hub/internal/store/customer_reset_test.go

@@ -25,10 +25,10 @@ func seedRetainedBlob(t *testing.T, st *Store, customerID string) {
 	if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil {
 		t.Fatalf("upsert host: %v", err)
 	}
-	if _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
+	if _, _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
 		t.Fatalf("escrow A: %v", err)
 	}
-	if _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
+	if _, _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
 		t.Fatalf("escrow B: %v", err)
 	}
 	if err := st.DeleteHost(hostID, true); err != nil {

hub/internal/store/escrow_identity_retention_test.go

@@ -0,0 +1,168 @@
+package store
+
+import (
+	"bytes"
+	"testing"
+)
+
+// R-198 (v0.93.0) — the retention must keep the OFF-SITE data key, not only the PBS key.
+//
+// WHAT WAS BROKEN, and why these tests are the ones that would have caught it: host_escrow_superseded
+// shipped with `blob` (the K-escrow / PBS datastore key) and `identity_blob` was added to host_escrow
+// by a later ALTER and never to the retained table. The restic REPOSITORY password lives inside
+// identity_blob. So every supersession retained the whole-guest key and destroyed the off-site data
+// key — the exact secret the retention exists to preserve — and the destroying act is the escrow
+// ceremony a rebuilt box asks its customer to run.
+//
+// The pre-existing TestSaveHostEscrow_RetainsSuperseded was GREEN throughout, because it asserts the
+// MECHANISM (a retained row exists, with the old K-blob) and not the CONSEQUENCE (the retained row can
+// still yield a repository password). These assert the consequence.
+
+// Scenario A — a re-escrow retains BOTH sealed keys.
+// RED-PROOF: drop `identity_blob` from demoteCurrentEscrowTx's INSERT/SELECT (production behaviour up
+// to v0.92.0) → the retained row's identity blob is nil → this FAILS.
+func TestSaveHostEscrow_RetainsIdentityBlob(t *testing.T) {
+	st := newTestStore(t)
+	const h = "hid-1"
+	oldIdentity := []byte("age-blob-sealing-REPO-PASSWORD-OLD")
+	newIdentity := []byte("age-blob-sealing-REPO-PASSWORD-NEW")
+
+	// Generation 1: the K-escrow, then the identity blob — the real order the escrow PUT uses
+	// (SaveHostEscrow, then SaveHostDRBundle).
+	if _, _, err := st.SaveHostEscrow(h, []byte("k-blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
+		t.Fatal(err)
+	}
+	if err := st.SaveHostDRBundle(h, oldIdentity, `{"gen":1}`); err != nil {
+		t.Fatal(err)
+	}
+
+	// Generation 2 with a DIFFERENT sealed repo password → supersede.
+	sup, prev, err := st.SaveHostEscrow(h, []byte("k-blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !sup {
+		t.Fatal("a different-passphrase upload must supersede")
+	}
+	if prev != "SHA_OLD" {
+		t.Fatalf("prevResticPwSHA256 = %q, want SHA_OLD (R-197 needs the replaced hash)", prev)
+	}
+	if err := st.SaveHostDRBundle(h, newIdentity, `{"gen":2}`); err != nil {
+		t.Fatal(err)
+	}
+
+	retained, err := st.ListSupersededEscrow(h)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(retained) != 1 {
+		t.Fatalf("retained rows = %d, want 1", len(retained))
+	}
+	// THE ASSERTION THIS WHOLE ITEM IS ABOUT: the off-site data key survived the supersession.
+	if retained[0].IdentityBlob == nil {
+		t.Fatal("R-198: the retained row carries NO identity blob — the off-site repository password " +
+			"was destroyed by the ceremony that was supposed to preserve it")
+	}
+	// And it is the PREVIOUS generation's blob, not the one that replaced it. This pins the ordering
+	// dependency named on demoteCurrentEscrowTx: the identity blob is written AFTER SaveHostEscrow, so
+	// the demote sees the old one. If that order ever inverts, the retained bytes would silently be
+	// the new blob filed under the old hash — recoverable-looking and wrong.
+	if !bytes.Equal(retained[0].IdentityBlob, oldIdentity) {
+		t.Fatalf("retained identity blob is not the PREVIOUS generation (got %q) — the demote ran after the overwrite",
+			retained[0].IdentityBlob)
+	}
+	if retained[0].ResticPwSHA256 != "SHA_OLD" || string(retained[0].Blob) != "k-blob-old" {
+		t.Fatalf("retained row is not the old generation: %+v", retained[0])
+	}
+	// Current row unchanged in behaviour: the NEW generation, both blobs.
+	if bundle, berr := st.GetHostDRBundle(h); berr != nil || bundle == nil || !bytes.Equal(bundle.IdentityBlob, newIdentity) {
+		t.Fatalf("current identity blob is not the new one: %+v (%v)", bundle, berr)
+	}
+}
+
+// Scenario B — deleting a host demotes BOTH blobs too. demoteCurrentEscrowTx is shared by the
+// re-escrow path and the host-delete path; a shared routine proven through one caller is how a fix
+// gets believed on a path nobody exercised.
+// RED-PROOF: fix only the re-escrow caller (e.g. carry the column in SaveHostEscrow's own SQL instead
+// of in the shared routine) → this FAILS while Scenario A passes.
+func TestDeleteHost_DemotesIdentityBlob(t *testing.T) {
+	s := newTestStore(t)
+	const hostID, cust = "hid-del", "cust-del"
+	identity := []byte("age-blob-sealing-REPO-PASSWORD")
+	if err := s.UpsertHost(&Host{HostID: hostID, CustomerID: cust, APIKey: "k"}); err != nil {
+		t.Fatal(err)
+	}
+	if _, _, err := s.SaveHostEscrow(hostID, []byte("k-blob"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
+		t.Fatal(err)
+	}
+	if err := s.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
+		t.Fatal(err)
+	}
+
+	if err := s.DeleteHost(hostID, true); err != nil {
+		t.Fatalf("DeleteHost: %v", err)
+	}
+	retained, err := s.ListSupersededEscrow(hostID)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(retained) != 1 {
+		t.Fatalf("demoted rows = %d, want 1", len(retained))
+	}
+	if !bytes.Equal(retained[0].IdentityBlob, identity) {
+		t.Fatalf("R-198: host delete demoted custody WITHOUT the identity blob (got %q) — the off-site "+
+			"repository password was destroyed by a host delete", retained[0].IdentityBlob)
+	}
+}
+
+// Scenario C — a legacy host whose current escrow has NO identity blob supersedes cleanly. The column
+// is nullable on purpose: a NOT NULL constraint here would make the fix block a ceremony, which is a
+// worse failure than the one it repairs.
+func TestSaveHostEscrow_SupersedesWithoutIdentityBlob(t *testing.T) {
+	st := newTestStore(t)
+	const h = "hid-legacy"
+	if _, _, err := st.SaveHostEscrow(h, []byte("k-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
+		t.Fatal(err)
+	}
+	// No SaveHostDRBundle — a slice-7-era upload.
+	sup, prev, err := st.SaveHostEscrow(h, []byte("k-new"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
+	if err != nil {
+		t.Fatalf("a supersession of an identity-less escrow must not fail: %v", err)
+	}
+	if !sup || prev != "SHA_OLD" {
+		t.Fatalf("superseded=%v prev=%q, want true/SHA_OLD", sup, prev)
+	}
+	retained, err := st.ListSupersededEscrow(h)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(retained) != 1 {
+		t.Fatalf("retained rows = %d, want 1", len(retained))
+	}
+	if retained[0].IdentityBlob != nil {
+		t.Fatalf("a legacy row must retain a NULL identity blob, got %q", retained[0].IdentityBlob)
+	}
+}
+
+// CountCurrentEscrowWithIdentity is the census §8.1 asks for: which hosts hold an identity blob today
+// and are therefore protected from the next ceremony by this fix. Asserted rather than eyeballed,
+// because the report quotes its numbers.
+func TestCountCurrentEscrowWithIdentity(t *testing.T) {
+	st := newTestStore(t)
+	if _, _, err := st.SaveHostEscrow("with-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA1"); err != nil {
+		t.Fatal(err)
+	}
+	if err := st.SaveHostDRBundle("with-id", []byte("age-blob"), `{}`); err != nil {
+		t.Fatal(err)
+	}
+	if _, _, err := st.SaveHostEscrow("without-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA2"); err != nil {
+		t.Fatal(err)
+	}
+	total, withIdentity, err := st.CountCurrentEscrowWithIdentity()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if total != 2 || withIdentity != 1 {
+		t.Fatalf("census = %d/%d, want 2 total / 1 with identity", withIdentity, total)
+	}
+}

hub/internal/store/escrow_superseded_test.go

@@ -11,7 +11,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
 	const h = "h1"
 
 	// 1st upload (P_old) — nothing to supersede.
-	sup, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
+	sup, _, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -20,7 +20,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
 	}
 
 	// 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old.
-	sup, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
+	sup, _, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -45,7 +45,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
 	}
 
 	// 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row.
-	sup, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
+	sup, _, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
 	if err != nil {
 		t.Fatal(err)
 	}

hub/internal/store/host_delete_demote_test.go

@@ -15,10 +15,10 @@ func TestDeleteHost_DemotesEscrowNeverDestroys(t *testing.T) {
 		t.Fatal(err)
 	}
 	// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
-	if _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
+	if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
+	if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
 		t.Fatal(err)
 	}
 	if n, _ := s.CountSupersededEscrow(hostID); n != 1 {

hub/internal/store/host_delete_test.go

@@ -57,7 +57,7 @@ func seedHostWithArtifacts(t *testing.T, s *Store, hostID, customerID string) {
 		"PK-"+hostID, "ip-"+hostID, hostID); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
+	if _, _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
 		t.Fatal(err)
 	}
 }

hub/internal/store/managed_floor_test.go

@@ -1,6 +1,9 @@
 package store
 
-import "testing"
+import (
+	"strings"
+	"testing"
+)
 
 // setHostAgent enrolls a host for customerID and records its agent version (the reality column).
 func setHostAgent(t *testing.T, s *Store, customerID, hostID, agentVer string) {
@@ -93,3 +96,136 @@ func TestResolveManagedFloor(t *testing.T) {
 		}
 	})
 }
+
+// ── R-216: the floor may not be served ABOVE the version the manifest describes ─────────────────
+//
+// `publish-train-rules.md` rule 3 is about the FLOOR's controller; the implementation compared
+// against the GOLDEN's MinAgent. Those are the same number only while the floor sits at or below the
+// golden. Measured live 2026-08-05 (CAMPAIGN-11 Phase 1): golden 0.192.0 / MinAgent 0.113.0, a
+// per-customer floor of 0.200.0, a box on agent 0.120.0 — 0.120.0 ≥ 0.113.0, so the floor was served
+// and the box was pushed onto a controller needing agent 0.125.0. Its customer was then told their
+// correct recovery code was wrong.
+//
+// RED-PROOF: delete the `semver.Compare(d.Floor, d.GoldenVersion) > 0` branch from
+// ResolveManagedFloor and TestResolveManagedFloor_R216_FloorAboveGolden FAILS — the box is served
+// past its agent again, reproducing the campaign's measurement exactly. Demonstrated failing before
+// these tests were kept.
+func TestResolveManagedFloor_R216_FloorAboveGolden(t *testing.T) {
+	t.Run("the campaign's exact numbers → HELD, not served", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.200.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "0.192.0", MinAgent: "0.113.0"})
+		setHostAgent(t, s, "c11", "c11-36d660", "0.120.0")
+
+		fd := s.ResolveManagedFloor("c11")
+		if !fd.Held || fd.Floor != "" {
+			t.Fatalf("R-216 RETURNED: a floor ABOVE the vouched golden was served, so the hub pushed a controller whose agent requirement it does not know: %+v", fd)
+		}
+		if !fd.HeldBeyondGolden {
+			t.Errorf("the hold must be distinguishable from the ordinary below-MinAgent hold (they need different operator text): %+v", fd)
+		}
+		if fd.GoldenVersion != "0.192.0" {
+			t.Errorf("the decision must carry the golden it compared against: %+v", fd)
+		}
+	})
+
+	// The hold is about the FLOOR being unknown, not about the agent — so it holds even for a box
+	// whose agent comfortably exceeds the manifest's MinAgent. That is the whole point: the hub cannot
+	// know what the served version needs.
+	t.Run("held even when the agent clears the manifest's MinAgent", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.200.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "0.192.0", MinAgent: "0.113.0"})
+		setHostAgent(t, s, "c1", "h1", "0.125.0")
+
+		if fd := s.ResolveManagedFloor("c1"); !fd.Held {
+			t.Fatalf("a floor above the golden must hold regardless of the agent: %+v", fd)
+		}
+	})
+
+	// SCENARIO C — an uncoupled release is untouched. A fleet must not be frozen by a comparison that
+	// now over-fires: with no MinAgent vouched there is no gating at all, above the golden or not.
+	t.Run("uncoupled release above the golden → still served (Scenario C)", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.200.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "0.192.0", MinAgent: ""})
+		setHostAgent(t, s, "c1", "h1", "0.70.0")
+
+		fd := s.ResolveManagedFloor("c1")
+		if fd.Held || fd.Floor != "0.200.0" {
+			t.Fatalf("an uncoupled release must be served exactly as before: %+v", fd)
+		}
+	})
+
+	// The ordinary arrangement — floor at or below the golden — is completely unchanged.
+	t.Run("floor at the golden → unchanged behaviour", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.192.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "0.192.0", MinAgent: "0.113.0"})
+		setHostAgent(t, s, "c1", "h1", "0.120.0")
+
+		fd := s.ResolveManagedFloor("c1")
+		if fd.Held || fd.Floor != "0.192.0" {
+			t.Fatalf("floor == golden with a satisfied MinAgent must serve: %+v", fd)
+		}
+	})
+
+	// An unreadable golden must not gate the fleet — degrade to the pre-existing behaviour.
+	t.Run("unparseable golden → falls through to the agent comparison", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.200.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "", MinAgent: "0.113.0"})
+		setHostAgent(t, s, "c1", "h1", "0.120.0")
+
+		fd := s.ResolveManagedFloor("c1")
+		if fd.Held || fd.Floor != "0.200.0" {
+			t.Fatalf("an unknown golden must not freeze the fleet: %+v", fd)
+		}
+	})
+}
+
+// The hold REASON must match the hold CAUSE. This is `CLAUDE.md`'s corollary, and it caught a real
+// slip in this very session: v0.97.0 added a second hold reason, and for one deploy the live hub
+// logged `managed floor HELD for c11: agent "0.125.0" < MinAgent 0.113.0` — a comparison that is
+// FALSE, for a box held because its floor sat above the golden. A true alarm that misdescribes itself
+// is one the operator learns to dismiss.
+func TestResolveManagedFloor_HoldReasonMatchesTheCause(t *testing.T) {
+	t.Run("beyond-golden hold names the floor and the golden, not the agent", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.200.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "0.192.0", MinAgent: "0.113.0"})
+		setHostAgent(t, s, "c11", "c11-36d660", "0.125.0")
+
+		reason := s.ResolveManagedFloor("c11").HoldReason()
+		if strings.Contains(reason, "< MinAgent") {
+			t.Fatalf("the beyond-golden hold claims an agent comparison that is FALSE here (0.125.0 > 0.113.0): %q", reason)
+		}
+		for _, want := range []string{"0.200.0", "0.192.0", "ABOVE"} {
+			if !strings.Contains(reason, want) {
+				t.Errorf("hold reason must name %q; got %q", want, reason)
+			}
+		}
+	})
+
+	t.Run("below-MinAgent hold still names the agent comparison", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.113.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "0.113.0", MinAgent: "0.81.0"})
+		setHostAgent(t, s, "c1", "h1", "0.79.0")
+
+		reason := s.ResolveManagedFloor("c1").HoldReason()
+		if !strings.Contains(reason, "0.79.0") || !strings.Contains(reason, "MinAgent 0.81.0") {
+			t.Fatalf("the below-MinAgent hold must name the agent comparison; got %q", reason)
+		}
+	})
+
+	t.Run("not held → no reason", func(t *testing.T) {
+		s := newTestStore(t)
+		_ = s.SetGlobalMinControllerVersion("0.113.0")
+		_ = s.SetArtifactManifest(ArtifactManifest{GoldenVersion: "0.113.0", MinAgent: ""})
+		setHostAgent(t, s, "c1", "h1", "0.70.0")
+		if r := s.ResolveManagedFloor("c1").HoldReason(); r != "" {
+			t.Fatalf("a served floor must have no hold reason, got %q", r)
+		}
+	})
+}

hub/internal/store/offsite_restage_test.go

@@ -0,0 +1,177 @@
+package store
+
+import (
+	"database/sql"
+	"errors"
+	"io"
+	"log"
+	"path/filepath"
+	"testing"
+)
+
+func newRestageStore(t *testing.T) *Store {
+	t.Helper()
+	st, err := New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
+	if err != nil {
+		t.Fatalf("store.New: %v", err)
+	}
+	t.Cleanup(func() { st.Close() })
+	return st
+}
+
+// R-204 item 4 / R-193 — THE FACT THE WHOLE DESIGN RESTS ON: the one-time offsite secret's VALUE
+// survives consumption, so it can be re-armed without any storage-provider call.
+//
+// This is asserted rather than inherited from the PBS analogy on purpose. The two secrets are
+// different objects with different lifecycles, and assuming a shared shape is exactly how two earlier
+// sessions confused the two credentials. If a future change ever clears `value` on consume — a
+// perfectly reasonable-looking hardening — restage-before-mint becomes silently impossible and every
+// rebuild turns into an external mint. This test fails the moment that happens.
+func TestRestageOneTimeSecret_ReArmsTheSameValue(t *testing.T) {
+	st := newRestageStore(t)
+	const cust, pw = "c1", "the-one-time-password"
+
+	if err := st.SaveOneTimeSecret(cust, pw); err != nil {
+		t.Fatal(err)
+	}
+	got, err := st.ConsumeOneTimeSecret(cust)
+	if err != nil || got != pw {
+		t.Fatalf("first consume: got %q err=%v", got, err)
+	}
+	// Single-use holds before the restage.
+	if _, err := st.ConsumeOneTimeSecret(cust); !errors.Is(err, sql.ErrNoRows) {
+		t.Fatalf("a consumed secret was served twice: err=%v", err)
+	}
+
+	restaged, err := st.RestageOneTimeSecret(cust)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !restaged {
+		t.Fatal("an existing secret row reported nothing to re-stage")
+	}
+
+	// THE ASSERTION: the SAME value comes back. Not "a value" — the same one, which is what makes
+	// this a re-arm rather than a silent mint.
+	again, err := st.ConsumeOneTimeSecret(cust)
+	if err != nil {
+		t.Fatalf("consume after restage: %v", err)
+	}
+	if again != pw {
+		t.Fatalf("the re-armed secret changed value: got %q want %q — the value did NOT survive the consume, so restage-before-mint is impossible and the reconciler must mint instead", again, pw)
+	}
+	// And it is single-use again.
+	if _, err := st.ConsumeOneTimeSecret(cust); !errors.Is(err, sql.ErrNoRows) {
+		t.Fatalf("the re-armed secret was not single-use: err=%v", err)
+	}
+}
+
+// No row → restaged=false, so the caller escalates to a fresh mint instead of silently doing nothing.
+func TestRestageOneTimeSecret_NoRowReportsFalse(t *testing.T) {
+	st := newRestageStore(t)
+	restaged, err := st.RestageOneTimeSecret("nobody")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if restaged {
+		t.Fatal("re-staging a customer with no stored secret claimed success — the caller would never escalate")
+	}
+}
+
+// A restage must not touch the timestamps' meaning beyond the consumed flag: created_at stays put, so
+// the delivery-state age math is unchanged.
+func TestRestageOneTimeSecret_ClearsOnlyTheConsumedFlag(t *testing.T) {
+	st := newRestageStore(t)
+	const cust = "c1"
+	if err := st.SaveOneTimeSecret(cust, "pw"); err != nil {
+		t.Fatal(err)
+	}
+	before, err := st.GetOneTimeSecretInfo(cust)
+	if err != nil || before == nil {
+		t.Fatalf("info before: %v", err)
+	}
+	if _, err := st.ConsumeOneTimeSecret(cust); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := st.RestageOneTimeSecret(cust); err != nil {
+		t.Fatal(err)
+	}
+	after, err := st.GetOneTimeSecretInfo(cust)
+	if err != nil || after == nil {
+		t.Fatalf("info after: %v", err)
+	}
+	if !after.ConsumedAt.IsZero() {
+		t.Errorf("consumed_at was not cleared: %v", after.ConsumedAt)
+	}
+	if !after.CreatedAt.Equal(before.CreatedAt) {
+		t.Errorf("created_at moved on a restage: %v → %v (the delivery-state age math would jump)", before.CreatedAt, after.CreatedAt)
+	}
+}
+
+// reportHasOffsite is TIGHTENED in v0.96.0 to require enabled:true, because controller v0.199.0 sends
+// an offsite object with enabled:false to ASK for a credential. This test pins BOTH halves: the
+// tightening works, AND it is a no-op for every report shape that existed before — the equivalence is
+// measured here rather than argued in a comment.
+func TestReportHasOffsite_EnabledOnly(t *testing.T) {
+	cases := []struct {
+		name   string
+		report string
+		want   bool
+	}{
+		// The PRE-v0.199.0 shapes. These are the equivalence claim: every object the old controller
+		// could ever attach carried enabled:true, so the tightening changes none of them.
+		{"configured box (pre-0.199 shape)", `{"offsite":{"enabled":true,"escrow_state":"escrowed","last_status":"ok"}}`, true},
+		{"configured box, minimal", `{"offsite":{"enabled":true}}`, true},
+		{"no offsite object", `{"controller_version":"0.198.0"}`, false},
+		{"explicit null", `{"offsite":null}`, false},
+		{"unparseable", `{not json`, false},
+
+		// The v0.199.0 DECLARATION — a request for help, not evidence of an applied tier.
+		{"stranded rebuild declaring", `{"offsite":{"enabled":false,"state":"needs_credential"}}`, false},
+		{"disabled object without a state", `{"offsite":{"enabled":false}}`, false},
+	}
+	for _, tc := range cases {
+		if got := reportHasOffsite(tc.report); got != tc.want {
+			t.Errorf("%s: reportHasOffsite = %v, want %v — a declaration read as an applied tier makes DeliveryStateFor say the stranded box is fine", tc.name, got, tc.want)
+		}
+	}
+}
+
+// The declaration is readable as such, and an older controller's report yields "" rather than an error.
+func TestLatestReportOffsiteDeclaration(t *testing.T) {
+	st := newRestageStore(t)
+	const cust = "c1"
+
+	found, _, state, err := st.LatestReportOffsiteDeclaration(cust)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if found {
+		t.Fatal("a customer with no reports was reported as found")
+	}
+
+	if err := st.SaveReport(cust, []byte(`{"controller_version":"0.198.0"}`)); err != nil {
+		t.Fatal(err)
+	}
+	found, id1, state, err := st.LatestReportOffsiteDeclaration(cust)
+	if err != nil || !found {
+		t.Fatalf("found=%v err=%v", found, err)
+	}
+	if state != "" {
+		t.Errorf("an old controller's report declared %q", state)
+	}
+
+	if err := st.SaveReport(cust, []byte(`{"offsite":{"enabled":false,"state":"needs_credential"}}`)); err != nil {
+		t.Fatal(err)
+	}
+	found, id2, state, err := st.LatestReportOffsiteDeclaration(cust)
+	if err != nil || !found {
+		t.Fatalf("found=%v err=%v", found, err)
+	}
+	if state != "needs_credential" {
+		t.Errorf("declared state = %q, want needs_credential", state)
+	}
+	if id2 <= id1 {
+		t.Errorf("the report id did not advance (%d → %d) — the debounce counts distinct reports and would never confirm", id1, id2)
+	}
+}

hub/internal/store/store.go

@@ -374,6 +374,22 @@ func (s *Store) migrate() error {
 		-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
 		-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
 		-- restore-serving read host_escrow (the CURRENT row) — never this table.
+		--
+		-- THE RULING ABOVE WAS NOT MET FOR TWO MONTHS, AND THIS IS THE RECORD OF IT (R-198, fixed
+		-- v0.93.0). This table shipped with the blob column — the K-escrow, i.e. the PBS datastore key
+		-- — and identity_blob was added to host_escrow LATER (the slice-10D ALTER below), never here.
+		-- The offsite restic REPOSITORY password lives in identity_blob, not in blob. So the retention
+		-- preserved the whole-guest key and silently dropped the off-site data key: precisely the
+		-- secret the reinstall-orphan incident was about. Worse, the copy happens as the new blob
+		-- overwrites the old, so the destroying act was the ESCROW CEREMONY — the exact thing a
+		-- rebuilt box asks its customer to run, on a card promising the old backups stay recoverable.
+		-- Both demo boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08) and their previous
+		-- repository passwords are unrecoverable, recovery code or not.
+		-- identity_blob is now carried (see demoteCurrentEscrowTx, which is still THE ONE row-copy
+		-- routine). Pinned by TestSaveHostEscrow_RetainsIdentityBlob and
+		-- TestDeleteHost_DemotesIdentityBlob — the routine is proven through BOTH of its callers,
+		-- because a shared routine tested through one caller is how a fix gets believed on a path
+		-- nobody exercised. Evidence: audits/RECON-offsite-dr-chain-2026-08-04.md §7.
 		CREATE TABLE IF NOT EXISTS host_escrow_superseded (
 			id               INTEGER PRIMARY KEY AUTOINCREMENT,
 			host_id          TEXT NOT NULL,
@@ -382,7 +398,8 @@ func (s *Store) migrate() error {
 			posture          TEXT NOT NULL DEFAULT '',
 			created_at       DATETIME NOT NULL,
 			restic_pw_sha256 TEXT NOT NULL DEFAULT '',
-			superseded_at    DATETIME NOT NULL DEFAULT (datetime('now'))
+			superseded_at    DATETIME NOT NULL DEFAULT (datetime('now')),
+			identity_blob    BLOB
 		);
 		CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
 
@@ -414,16 +431,28 @@ func (s *Store) migrate() error {
 	s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
 	s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
 
+	// R-198 (v0.93.0) — the SAME column on the RETAINED table. It was added above and not here, and
+	// that omission is what made the retention keep the wrong key for two months (see the comment on
+	// host_escrow_superseded). Additive and tolerated on re-run, exactly like the lines above; it
+	// changes no existing row. Rows superseded BEFORE this ships were written without the identity
+	// blob and their source rows are already overwritten — there is nothing to backfill, and the
+	// v0.93.0 report records that as a looked-at fact rather than a deduction.
+	s.db.Exec(`ALTER TABLE host_escrow_superseded ADD COLUMN identity_blob BLOB`)
+
 	// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
 	// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
 	// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
 	// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
 	s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
 
-	// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite repo
-	// password is re-issued: the blob then seals a password that no longer opens the repo, so the
-	// hub must stop advertising "ceremony done" and withhold the (now non-matching) restic_pw_sha256
-	// from the auto-confirm ACK. NULL = current; a fresh ceremony (SaveHostEscrow) clears it.
+	// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at marks an escrow blob that may no
+	// longer cover the box's live repository password. While set, the hub stops advertising "ceremony
+	// done" and withholds restic_pw_sha256 from the auto-confirm ACK. NULL = current; a fresh ceremony
+	// (SaveHostEscrow) clears it.
+	// ⚠ NOTHING SETS IT as of v0.95.0 (R-196 / R-204 item 2). The only writer was the PRECAUTIONARY
+	// mark on offsite re-issue, which guessed rather than measured and so blocked off-site backups on
+	// boxes whose key had not changed. The column and its readers stay; see MarkEscrowStale for what a
+	// legitimate future writer would have to prove first.
 	s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
 
 	// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
@@ -949,11 +978,27 @@ func (s *Store) SaveReport(customerID string, reportJSON []byte) error {
 	return err
 }
 
-// reportOffsitePresence is the minimal parse for "does this controller report carry an offsite
-// status object" — the R-70 delivery-state signal. The report builder attaches `offsite` only when
-// the box actually has an offbox target configured, so presence == applied-on-the-box.
+// reportOffsitePresence is the minimal parse for "does this controller report show an offsite tier
+// that is actually CONFIGURED on the box" — the R-70 delivery-state signal, i.e. applied-on-the-box.
+//
+// ⚠ TIGHTENED 2026-08-05 (hub v0.96.0, R-204 item 4), and the reason is exactly the trap this
+// project keeps recording. The predicate used to be bare PRESENCE, under a comment asserting *"the
+// report builder attaches `offsite` only when the box actually has an offbox target configured, so
+// presence == applied-on-the-box"*. Controller v0.199.0 breaks that premise deliberately: a REBUILT
+// box now attaches an offsite object carrying `enabled:false` and a declared `state` in order to ASK
+// for a credential. Left as presence-only, this function would have read that request for help as
+// proof the tier was applied — turning `DeliveryStateFor` into DeliveryApplied for the exact boxes
+// that are stranded, and suppressing the stuck event for them.
+//
+// The tightening is `enabled == true`, and it is PROVABLY A NO-OP for every report shape that exists
+// today: `backup.OffboxReportStatus` returned nil unless `t != nil && t.Enabled`, so an attached
+// object has ALWAYS carried `enabled:true`. Pinned by
+// TestReportHasOffsite_EnabledOnly — including a case asserting the pre-v0.199.0 shape still reads
+// true, so the equivalence is measured rather than argued.
 type reportOffsitePresence struct {
-	Offsite json.RawMessage `json:"offsite"`
+	Offsite *struct {
+		Enabled bool `json:"enabled"`
+	} `json:"offsite"`
 }
 
 func reportHasOffsite(reportJSON string) bool {
@@ -961,7 +1006,7 @@ func reportHasOffsite(reportJSON string) bool {
 	if err := json.Unmarshal([]byte(reportJSON), &p); err != nil {
 		return false // unparseable report → no offsite evidence
 	}
-	return len(p.Offsite) > 0 && string(p.Offsite) != "null"
+	return p.Offsite != nil && p.Offsite.Enabled
 }
 
 // LatestReportOffsitePresence reports whether the customer's most recent controller report exists
@@ -1386,6 +1431,67 @@ func (s *Store) ConsumeOneTimeSecret(customerID string) (string, error) {
 	return value, nil
 }
 
+// RestageOneTimeSecret re-arms an ALREADY-STORED one-time offsite secret for re-consumption by
+// clearing its consumed flag — WITHOUT changing the secret value, WITHOUT inserting a row, and
+// WITHOUT any storage-provider call. It is the off-site sibling of RestageHostPBSSecret (pbsdr.go),
+// and it exists because a REBUILT box has no credential of its own: its predecessor spent the
+// one-time password (R-193 / R-204 item 4).
+//
+// IT IS POSSIBLE AT ALL ONLY BECAUSE THE VALUE SURVIVES A CONSUME, and that was established from
+// this file rather than assumed from the PBS analogy (the two secrets are different objects with
+// different lifecycles, and assuming a shared shape is how two sessions confused the credentials):
+// ConsumeOneTimeSecret sets `consumed_at` and NOTHING ELSE — the `value` column is never cleared or
+// overwritten, and the schema declares it `TEXT NOT NULL`. So the row still holds a serviceable
+// password after consumption, and re-arming it costs no external call and converges in one report
+// tick. Pinned by TestRestageOneTimeSecret_ReArmsTheSameValue.
+//
+// Returns restaged=true when a row existed (its consumed flag is now cleared, so
+// ConsumeOneTimeSecret will serve it once more); restaged=false when NO secret is stored for the
+// customer — the caller must then escalate to a fresh mint (ReissueCredentials). The value is never
+// read or logged here.
+func (s *Store) RestageOneTimeSecret(customerID string) (bool, error) {
+	res, err := s.db.Exec(`UPDATE one_time_secrets SET consumed_at = NULL WHERE customer_id = ?`, customerID)
+	if err != nil {
+		return false, err
+	}
+	n, err := res.RowsAffected()
+	if err != nil {
+		return false, err
+	}
+	return n > 0, nil
+}
+
+// LatestReportOffsiteDeclaration returns the id of the customer's newest controller report and the
+// state DECLARED on its `offsite` object (controller >= v0.199.0; "" on anything older, on a report
+// with no offsite object, and on an unparseable one).
+//
+// The report id is the debounce currency: the reconciler counts DISTINCT reports, not its own ticks,
+// so a stuck box is confirmed by fresh evidence rather than by the passage of time.
+//
+// found=false when the customer has no reports at all — distinguished from "a report that declares
+// nothing" so the caller never treats an absent report as a healthy one.
+func (s *Store) LatestReportOffsiteDeclaration(customerID string) (found bool, reportID int64, state string, err error) {
+	var id int64
+	var reportJSON string
+	err = s.db.QueryRow(`SELECT id, report_json FROM reports WHERE customer_id = ? ORDER BY id DESC LIMIT 1`,
+		customerID).Scan(&id, &reportJSON)
+	if err == sql.ErrNoRows {
+		return false, 0, "", nil
+	}
+	if err != nil {
+		return false, 0, "", err
+	}
+	var p struct {
+		Offsite *struct {
+			State string `json:"state"`
+		} `json:"offsite"`
+	}
+	if json.Unmarshal([]byte(reportJSON), &p) == nil && p.Offsite != nil {
+		state = p.Offsite.State
+	}
+	return true, id, state, nil
+}
+
 // OneTimeSecretInfo is the delivery-state metadata of a customer's one-time offsite secret —
 // timestamps ONLY, the value column is deliberately never selected (R-70 detector input; the
 // customer-scoped sibling of PBSDRHealStates' SecretUnconsumedFor).
@@ -1863,32 +1969,123 @@ func (s *Store) EffectiveMinControllerVersion(customerID string) string {
 // to actually serve this customer's controller, whether it is being HELD (and why), and the inputs.
 type ManagedFloorDecision struct {
 	Floor        string // the controller-version floor to SERVE ("" = serve none)
-	Held         bool   // true = the floor is withheld because the box's agent is below MinAgent
+	Held         bool   // true = the floor is withheld (see HeldBeyondGolden for which of the two reasons)
 	AgentVersion string // the box's reported agent version ("" = unknown → held when MinAgent is set)
-	MinAgent     string // the manifest's MinAgent for the current golden ("" = uncoupled, no gating)
+	// RequestedFloor is the floor that was asked for, kept for the message: Floor is cleared to "" on
+	// a hold, and a reason naming an empty floor tells the operator nothing.
+	RequestedFloor string
+	// MinAgent is the manifest's vouched MinAgent — the agent requirement of the GOLDEN's controller
+	// (`ArtifactManifest.MinAgent`). ⚠ It describes the golden, NOT necessarily the version this
+	// decision is about; that gap is what HeldBeyondGolden closes. Empty = uncoupled, no gating.
+	MinAgent string
+	// GoldenVersion is the vouched golden's controller version — the version MinAgent describes.
+	GoldenVersion string
+	// HeldBeyondGolden distinguishes the two hold reasons, because they need different operator text:
+	// false = the box's agent is below MinAgent (the original Part D hold); true = the FLOOR points
+	// ABOVE the vouched golden, so the manifest's MinAgent does not describe the version being served
+	// and the hub does not know its agent requirement (R-216).
+	HeldBeyondGolden bool
 }
 
-// ResolveManagedFloor decides the controller-version floor to serve a customer, HOLDING it when the
-// golden the floor points at requires a newer host agent than the box currently runs (Part D — the
-// hub-enforced "agent BEFORE controller floor" rule). Logic:
+// HoldReason is the ONE operator-facing sentence for a held floor, so the two hold reasons can never
+// drift apart across the log line and the dashboard again.
+//
+// ⚠ It exists because v0.97.0 introduced a second reason and, for one deploy, both surfaces still
+// printed the first: the live hub logged `managed floor HELD for c11: agent "0.125.0" < MinAgent
+// 0.113.0` — a comparison that is FALSE (0.125.0 is above 0.113.0) for a box held because its floor
+// sat above the golden. That is `CLAUDE.md`'s corollary exactly: when a verdict changes which field it
+// counts from, the alarm text has to change with it, or a true alarm reads as one to dismiss.
+// Returns "" when not held.
+func (d ManagedFloorDecision) HoldReason() string {
+	switch {
+	case !d.Held:
+		return ""
+	case d.HeldBeyondGolden:
+		return fmt.Sprintf("held: floor %s is ABOVE the vouched golden %s, so its agent requirement is unknown — vouch a golden carrying the floor's controller (publish-train rule 1)",
+			d.EffectiveFloorForMessage(), goldenOrUnknown(d.GoldenVersion))
+	default:
+		return fmt.Sprintf("held: agent %s < MinAgent %s", agentOrUnknownVersion(d.AgentVersion), d.MinAgent)
+	}
+}
+
+// EffectiveFloorForMessage reports the floor that was REQUESTED, for the message — Floor is cleared to
+// "" when held, and a message naming an empty floor tells the operator nothing.
+func (d ManagedFloorDecision) EffectiveFloorForMessage() string {
+	if d.RequestedFloor != "" {
+		return d.RequestedFloor
+	}
+	return d.Floor
+}
+
+func goldenOrUnknown(v string) string {
+	if v == "" {
+		return "(none vouched)"
+	}
+	return v
+}
+
+func agentOrUnknownVersion(v string) string {
+	if v == "" {
+		return "(unknown)"
+	}
+	return v
+}
+
+// ResolveManagedFloor decides the controller-version floor to serve a customer, HOLDING it rather
+// than pushing a controller past the agent it depends on (Part D — the hub-enforced "agent BEFORE
+// controller floor" rule).
+//
+// ⚠ R-216, and the reason this function changed. `publish-train-rules.md` rule 3 states the rule
+// about the FLOOR's controller: *"The floor may not effectively push a box past a controller whose
+// MinAgent that box's agent does not yet meet."* The implementation compared against
+// `ArtifactManifest.MinAgent`, which by its own doc comment is *"the MINIMUM host-agent version this
+// GOLDEN's controller requires"*. Those are the same number only while the floor sits at or below the
+// golden — the arrangement rule 5's ISO gate assumes. Raise a floor above the vouched golden (which
+// the day-0 runbook explicitly recommends after a golden rebuild, and which a per-customer override
+// makes trivial) and the guard compares against a version it is not serving.
+//
+// Measured live on 2026-08-05 (CAMPAIGN-11 Phase 1): golden 0.192.0 with MinAgent 0.113.0, a
+// per-customer floor of 0.200.0, and a box on agent 0.120.0. 0.120.0 ≥ 0.113.0, so the floor was
+// served — and the box was pushed onto a controller needing agent 0.125.0. Its customer was then
+// told their correct recovery code was wrong. The guard whose comment says it exists to *"never push
+// a controller past its agent"* had just done exactly that.
+//
+// THE FIX, and why it needs no new operator input: the hub cannot know the agent requirement of a
+// controller version it was never told about, so it must not pretend to. A floor ABOVE the vouched
+// golden is served blind today; from v0.97.0 it is HELD, with its own reason. Nobody types a number
+// twice — the existing vouched MinAgent keeps its exact meaning, and the guard simply stops applying
+// it to versions it does not describe. The operator's remedy is the one the publish train already
+// prescribes: vouch a golden that carries the floor's controller (rule 1, "manifest before floor").
+//
+// Logic:
 //   - effective floor "" → nothing to serve (no floor configured);
 //   - manifest MinAgent "" → UNCOUPLED release: serve the floor as-is (no agent gating);
+//   - floor ABOVE the vouched golden's controller → HOLD (requirement unknown) + flag;
 //   - agent_version known AND ≥ MinAgent → serve the floor;
 //   - agent_version below MinAgent, OR unknown/unparseable → HOLD (serve no directive) + flag.
 //
 // A held box is VISIBLE (the dashboard renders the reason), never silently stale.
 func (s *Store) ResolveManagedFloor(customerID string) ManagedFloorDecision {
 	d := ManagedFloorDecision{Floor: s.EffectiveMinControllerVersion(customerID)}
+	d.RequestedFloor = d.Floor
 	if d.Floor == "" {
 		return d
 	}
-	d.MinAgent = s.GetArtifactManifest().MinAgent
+	man := s.GetArtifactManifest()
+	d.MinAgent, d.GoldenVersion = man.MinAgent, man.GoldenVersion
 	if d.MinAgent == "" {
-		return d // uncoupled release — no agent gate
+		return d // uncoupled release — no agent gate (Scenario C: unchanged from before)
 	}
 	if h, err := s.GetHostByCustomer(customerID); err == nil && h != nil {
 		d.AgentVersion = h.AgentVersion
 	}
+	// R-216: the floor names a controller the manifest does not describe → the agent requirement is
+	// UNKNOWN, so hold. Only when both versions parse — an unreadable golden must not gate the fleet.
+	if semver.Valid(d.Floor) && semver.Valid(d.GoldenVersion) && semver.Compare(d.Floor, d.GoldenVersion) > 0 {
+		d.Held, d.HeldBeyondGolden = true, true
+		d.Floor = ""
+		return d
+	}
 	if d.AgentVersion != "" && semver.Valid(d.AgentVersion) && semver.Valid(d.MinAgent) &&
 		semver.Compare(d.AgentVersion, d.MinAgent) >= 0 {
 		return d // agent is new enough — serve the floor
@@ -2262,6 +2459,28 @@ func (s *Store) GetHostByCustomer(customerID string) (*Host, error) {
 	return h, err
 }
 
+// HasEverBoundHost reports whether a machine was EVER bound to this customer — a live row in
+// `hosts` OR a tombstone in `host_deletions`. It answers "was anything ever expected of this
+// customer", which is the question the deadline verdicts actually need (R-195).
+//
+// It is deliberately NOT "has a report arrived", and the distinction is the whole point: a box
+// that was installed, bound, and then went silent IS bound, and its silence is a real fault that
+// must keep alarming. Only a customer that never had a machine at all is UNKNOWN.
+//
+// `host_deletions` is included because a customer whose host was removed HAD one — the deadline
+// caller reaches its down-skip for that shape, and this predicate must not quietly take over a
+// judgement the staleness checker owns.
+func (s *Store) HasEverBoundHost(customerID string) (bool, error) {
+	var n int
+	if err := s.db.QueryRow(
+		`SELECT EXISTS(SELECT 1 FROM hosts WHERE customer_id = ?)
+		     OR EXISTS(SELECT 1 FROM host_deletions WHERE customer_id = ?)`,
+		customerID, customerID).Scan(&n); err != nil {
+		return false, err
+	}
+	return n != 0, nil
+}
+
 // ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
 // customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
 // Uses the idx_hosts_customer index.
@@ -2507,6 +2726,11 @@ type HostEscrow struct {
 	// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
 	// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
 	StaleAt string
+	// IdentityBlob (R-198, v0.93.0) — the age-wrapped identity bundle, which is where the offsite
+	// restic REPOSITORY password lives. Populated by ListSupersededEscrow so a retained blob is
+	// reachable from Go at all; nil for pre-v0.93.0 retained rows and for hosts that never uploaded
+	// one. Opaque: useless without the customer's recovery code, which the hub never holds.
+	IdentityBlob []byte
 }
 
 // SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
@@ -2516,16 +2740,35 @@ type HostEscrow struct {
 // host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
 // (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
 // superseded row.
+//
+// R-197 (v0.93.0): it also returns prevResticPwSHA256 — the hash the row being replaced sealed ("" when
+// no row existed, or when it was a legacy hash-less blob). Both halves of "did the box's offsite DATA
+// key change?" have been in this database since SLICE 3 and NOTHING compared them; demo-felhom's key
+// changed on 2026-08-03 and no signal of any kind fired for thirteen hours. The comparison is the
+// caller's (api.handleHostEscrowPut) because the event needs the customer id; returning the value is
+// this function's part. The VALUE is a non-reversible hash of a 256-bit random secret and is never
+// logged, mailed or written to a report — see the caller.
 // demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
 // host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
 // routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
 // overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
 // instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
 // The hub never decrypts; custody is unchanged.
+//
+// R-198 (v0.93.0): `identity_blob` is copied too. It was omitted from this SELECT for two months, so
+// every supersession retained the PBS datastore key and destroyed the offsite restic repository
+// password — the one secret the retention exists to preserve. Copying more opaque bytes gains the hub
+// NO knowledge: it still has no recovery code and no decrypt path.
+//
+// ORDERING THIS DEPENDS ON, stated because it is load-bearing and invisible from here: the identity
+// blob is written by SaveHostDRBundle AFTER SaveHostEscrow returns (api/handler.go, the escrow PUT),
+// so at demote time host_escrow still holds the OLD identity blob. If that order ever changes, this
+// routine silently retains the NEW blob under the OLD blob's hash — pinned by
+// TestSaveHostEscrow_RetainsIdentityBlob, which asserts the retained bytes are the previous ones.
 func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
 	res, err := tx.Exec(`
-		INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
-		SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
+		INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob)
+		SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now'), identity_blob
 		FROM host_escrow WHERE host_id = ?`, hostID)
 	if err != nil {
 		return 0, err
@@ -2533,10 +2776,10 @@ func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
 	return res.RowsAffected()
 }
 
-func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, err error) {
+func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, prevResticPwSHA256 string, err error) {
 	tx, err := s.db.Begin()
 	if err != nil {
-		return false, err
+		return false, "", err
 	}
 	defer func() {
 		if err != nil {
@@ -2555,11 +2798,14 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
 		exists = false
 	default:
 		err = scanErr
-		return false, err
+		return false, "", err
+	}
+	if exists {
+		prevResticPwSHA256 = curSHA // R-197: the caller compares; "" = no row or a legacy hash-less blob
 	}
 	if exists && curSHA != resticPwSHA256 {
 		if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
-			return false, err
+			return false, prevResticPwSHA256, err
 		}
 		superseded = true
 	}
@@ -2576,10 +2822,10 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
 			stale_at = NULL,
 			updated_at = datetime('now')`,
 		hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
-		return false, err
+		return false, prevResticPwSHA256, err
 	}
 	err = tx.Commit()
-	return superseded, err
+	return superseded, prevResticPwSHA256, err
 }
 
 // CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
@@ -2594,7 +2840,7 @@ func (s *Store) CountSupersededEscrow(hostID string) (int, error) {
 // first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
 func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
 	rows, err := s.db.Query(`
-		SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at
+		SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob
 		FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
 	if err != nil {
 		return nil, err
@@ -2603,7 +2849,7 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
 	var out []HostEscrow
 	for rows.Next() {
 		var e HostEscrow
-		if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt); err != nil {
+		if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt, &e.IdentityBlob); err != nil {
 			return nil, err
 		}
 		out = append(out, e)
@@ -2611,9 +2857,33 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
 	return out, rows.Err()
 }
 
-// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite repo
-// password is re-issued, because the blob then seals a password that no longer opens the repo. No-op
-// when no escrow row exists; idempotent (only stamps the first re-issue since the last ceremony; a
+// CountCurrentEscrowWithIdentity returns how many hosts hold a CURRENT escrow row, and how many of
+// those carry an identity blob — i.e. the population whose offsite repository password R-198's fix
+// now protects from the next ceremony, and the remainder for whom there is nothing to protect
+// because no identity blob was ever uploaded. Read-only; no blob or hash leaves this call.
+func (s *Store) CountCurrentEscrowWithIdentity() (total, withIdentity int, err error) {
+	err = s.db.QueryRow(`
+		SELECT COUNT(*), COALESCE(SUM(identity_blob IS NOT NULL), 0) FROM host_escrow`).Scan(&total, &withIdentity)
+	return total, withIdentity, err
+}
+
+// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3).
+//
+// ⚠ IT HAS NO CALLER as of hub v0.95.0 (R-196 / R-204 item 2), and that is deliberate, not an
+// oversight. Its ONE caller was `offsite.ReissueCredentials`, which called it on every re-issue on
+// the PRECAUTIONARY grounds that the box's re-apply might mint a fresh repository password. It
+// usually does not, so the call marked healthy escrows stale — and because a stale flag WITHHOLDS
+// restic_pw_sha256 from the ACK (GetEscrowStatusForCustomer, below), it blocked every off-site backup
+// on those boxes and asked the customer for a ceremony that would supersede a perfectly good key. The
+// full reasoning, and the two measured signals that cover the real case, are at
+// offsite.ReissueCredentials.
+//
+// KEPT, not deleted, because the FLAG is still live and correct — `stale_at` is read by the ACK, the
+// operator card and the PBS-DR view, and a future EVIDENTIAL caller (one that has measured a key
+// change rather than guessed at one) is the right way to set it. Pinned by
+// TestReissue_DoesNotMarkAHealthyEscrowStale: if a caller reappears without that evidence, it fails.
+//
+// No-op when no escrow row exists; idempotent (only stamps the first mark since the last ceremony; a
 // fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
 func (s *Store) MarkEscrowStale(hostID string) error {
 	_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
@@ -2643,10 +2913,30 @@ type EscrowStatus struct {
 	IdentityBlobPresent bool   `json:"identity_blob_present"`
 	ResticPwSHA256      string `json:"restic_pw_sha256,omitempty"`
 	CreatedAt           string `json:"created_at,omitempty"`
-	// Stale (v0.57.0, 2.3) — true when the offsite password was re-issued after the blob was sealed.
+	// Stale (v0.57.0, 2.3) — true when `stale_at` is stamped on the host's escrow row. ⚠ SINCE hub
+	// v0.95.0 (R-196 / R-204 item 2) NOTHING STAMPS IT: the one caller was the precautionary
+	// re-issue mark, and it was removed for marking healthy escrows stale. See MarkEscrowStale.
 	// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
-	// a hash that no longer matches the live repo password — the ceremony must run again.
+	// a hash that may no longer match the live repo password — the ceremony must run again. That
+	// withholding is exactly why the precautionary caller had to go: it BLINDED the controller's own
+	// hash comparison, which is the measurement that actually detects a changed repository password.
 	Stale bool `json:"escrow_stale,omitempty"`
+	// SupersededPresent (hub v0.97.0, R-222) — this host has at least one RETAINED superseded escrow
+	// row carrying an identity blob. It exists so the recovery screen can distinguish two situations
+	// that are otherwise identical from the box's side: a genuinely wrong recovery code, and a code
+	// that is right about an EARLIER sealed package the hub is deliberately keeping.
+	//
+	// Measured on 2026-08-05 (CAMPAIGN-11 Phase 3, step 7): the customer entered the correct code for
+	// the orphaned history, the unseal failed closed against the CURRENT package — correctly — and the
+	// screen told them to check their typing. Nothing on the box could have known better.
+	//
+	// It is a BOOLEAN, deliberately. It says "an earlier package is kept"; it does not serve one, does
+	// not say whose code opens it, and grants no read path — that link is still unbuilt (R-199's
+	// inventory) and this field must not be mistaken for it.
+	SupersededPresent bool `json:"superseded_present,omitempty"`
+	// SupersededAt (hub v0.97.0, R-222) — when the most recent supersession happened (RFC3339-ish, as
+	// stored). Empty when none. Non-secret: a timestamp.
+	SupersededAt string `json:"superseded_at,omitempty"`
 }
 
 // GetEscrowStatusForCustomer returns the escrow status of the customer's host (nil if the customer has no
@@ -2674,6 +2964,18 @@ func (s *Store) GetEscrowStatusForCustomer(customerID string) (*EscrowStatus, er
 		st.Stale = true
 		st.ResticPwSHA256 = ""
 	}
+	// R-222 (v0.97.0): does a RETAINED earlier package exist for this host? Only rows that actually
+	// carry an identity blob count — the pre-v0.93.0 rows have `identity_blob` NULL and retain nothing
+	// the recovery screen could ever be talking about, so counting them would make the screen offer an
+	// explanation that is false for exactly the boxes hurt by the original defect.
+	var supersededAt sql.NullString
+	if err := s.db.QueryRow(`
+		SELECT MAX(sup.superseded_at)
+		FROM host_escrow_superseded sup JOIN hosts h ON h.host_id = sup.host_id
+		WHERE h.customer_id = ? AND sup.identity_blob IS NOT NULL`, customerID).Scan(&supersededAt); err == nil && supersededAt.Valid {
+		st.SupersededPresent = true
+		st.SupersededAt = supersededAt.String
+	}
 	return &st, nil
 }
 

hub/internal/web/customer_delete_test.go

@@ -74,11 +74,11 @@ func seedDeletable(t *testing.T, st *store.Store, customerID string) string {
 	if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
 		t.Fatalf("seed host: %v", err)
 	}
-	if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
+	if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
 		t.Fatalf("seed escrow A: %v", err)
 	}
 	// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
-	if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
+	if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
 		t.Fatalf("seed escrow B: %v", err)
 	}
 	if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {

hub/internal/web/customer_reset_test.go

@@ -37,10 +37,10 @@ func seedResettable(t *testing.T, st *store.Store, customerID string) {
 	if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil {
 		t.Fatalf("seed host: %v", err)
 	}
-	if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
+	if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
 		t.Fatalf("seed escrow A: %v", err)
 	}
-	if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
+	if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
 		t.Fatalf("seed escrow B: %v", err)
 	}
 	if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions

hub/internal/web/hosts.go

@@ -4,7 +4,6 @@ import (
 	"database/sql"
 	"encoding/json"
 	"errors"
-	"fmt"
 	"net/http"
 	"sort"
 	"strings"
@@ -401,7 +400,7 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) {
 		// never silently stale.
 		if fd := s.store.ResolveManagedFloor(h.CustomerID); fd.Held {
 			row.FloorHeld = true
-			row.HeldReason = fmt.Sprintf("held: agent %s < MinAgent %s", agentOrUnknown(fd.AgentVersion), fd.MinAgent)
+			row.HeldReason = fd.HoldReason()
 		}
 
 		// Guest counts from the reality table (per-host accurate).

hub/internal/web/hosts_delete_test.go

@@ -64,7 +64,7 @@ func TestHostDelete_EscrowAckRequired(t *testing.T) {
 	if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
+	if _, _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
 		t.Fatal(err)
 	}
 
@@ -110,7 +110,7 @@ func TestHostDelete_ImpactJSON(t *testing.T) {
 	if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
+	if _, _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
 		t.Fatal(err)
 	}
 	if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100),
@@ -159,7 +159,7 @@ func TestHostDelete_HappyPath(t *testing.T) {
 	if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
+	if _, _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
 		t.Fatal(err)
 	}
 	rr := postHostDelete(t, s, "dr-drill", url.Values{

hub/internal/web/hosts_test.go

@@ -130,7 +130,7 @@ func TestHandleHostDetail(t *testing.T) {
 		t.Fatal(err)
 	}
 	// DR + escrow present (escrow row must exist before the DR bundle UPDATE).
-	if _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
+	if _, _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
 		t.Fatal(err)
 	}
 	if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil {

hub/internal/web/pbsdr_test.go

@@ -404,7 +404,7 @@ func TestPBSDR_F14AutoReissueOnAckedDeletion(t *testing.T) {
 	if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
 		t.Fatal(err)
 	}
-	if _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
+	if _, _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
 		t.Fatal(err)
 	}
 	if err := st.DeleteHost("peti-00-dead", true); err != nil {

manifests/hub.yaml

@@ -125,7 +125,7 @@ spec:
     spec:
       containers:
         - name: hub
-          image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
+          image: gitea.dooplex.hu/admin/felhom-hub:0.97.1
           ports:
             - containerPort: 8080
               name: http

manifests/webpage.yaml

@@ -324,7 +324,7 @@ spec:
           image: registry.k8s.io/git-sync/git-sync:v4.4.0
           args:
             - --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
-            - --ref=installer-v1.24.0
+            - --ref=installer-v1.25.0
             - --root=/git-scripts
             - --link=current
             - --period=30s
@@ -369,7 +369,7 @@ spec:
           image: registry.k8s.io/git-sync/git-sync:v4.4.0
           args:
             - --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
-            - --ref=installer-v1.24.0
+            - --ref=installer-v1.25.0
             - --root=/git-scripts
             - --link=current
             - --one-time