R-201 night run: the off-site key IS recoverable after a real rebuild (proven); the verdict is blocked by R-204

2026-08-04 22:33:01 +02:00
parent 24acc8d816
commit b228fd102d
7 changed files with 435 additions and 182 deletions
@@ -17,6 +17,33 @@
 ## Standing rulings
 **S-31 — the off-site key IS recoverable after a real rebuild; USING it is blocked by the remedy that
 makes the rebuild survivable (2026-08-04 night drill, R-201/R-204).**
 **Proven on hardware:** demo-hp's controller data volume was destroyed and the sentinel deleted; the
 customer's recovery code produced `8a9e33aa4da6…`, byte-identical to the pre-wipe on-disk key and to
 the hub's independent record, and installed cleanly on the bare box. `identity_blob` was unchanged
 across the wipe — nothing re-escrowed itself.
 **The wall (R-204), four links, all measured:**
 1. a rebuilt controller cannot configure its off-site tier — the one-time password was consumed by its
   predecessor (`no unconsumed offsite password`, R-193);
 2. the Re-issue that fixes that sets `stale_at` **while `restic_pw_sha256` is unchanged** (R-196);
 3. a stale escrow makes the hub withhold the hash from the ACK → `EscrowAutoConfirmer` can never flip
   `pending → escrowed` → `OffboxRunnable` refuses every run;
 4. the only documented way to clear it is a ceremony, **which supersedes the identity blob and destroys
   the recovered key**. And before any of it, a rebuilt box is **unclaimed**, so the claim gate
   intercepts every controller endpoint — a step in no design document.
 *Facts a future session needs:*
 - **A guest rebuild in this fleet is a controller-DATA-VOLUME loss, not a guest reprovision.** The
  2026-08-03 incident R-193 is filed against ran with guest 9201 up throughout — no `pct destroy`, no
  `pct restore`, no `--selftest=provision`. Reproduce it that way.
 - **A good snapshot is not durable against a later bad run on the same day.** `forget --keep-daily 7
  --group-by host,tags` keeps one per tag per day; a later, worse snapshot evicts a good one.
 - **Never run a ceremony while a recovery is in flight** — it supersedes the identity blob. Under
  v0.93.0 the old blob is retained, but nothing serves a superseded blob back (R-199).
 **S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.**
 It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run
@@ -1,172 +1,127 @@
-# REPORT — R-203: the app and its backup look in the same place, and "ok" means it (2026-08-04)
+# REPORT — R-201 night run: the key came back; the verdict did not
-**Controller v0.196.0 → v0.197.0**, deployed to demo-hp. No hub change. Nothing deleted, wiped or
+**2026-08-04, 21:30–22:40, unattended.** `demo-hp` was deliberately rebuilt. No code, no version bump.
-moved on any box; the drill was **not** resumed. `demo-felhom` untouched.
+`demo-felhom` untouched. Full record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
 ---
-## 1. Confirmed baselines, and §3's landmarks
+## 1. THE VERDICT — not reached
-`felhom-controller` `532f5712a891` (v0.196.0) → **v0.197.0**; `felhom.eu` `a0c4b607a6cf`, no bump.
+| | |
 Both trees clean on arrival. **Every §3 landmark held**, including the one that decided the fix:
 `paths.go:26` names the system-drive arrangement *"the SSD-only system-data fallback"* — **it is
 supported**, so the resolution was what was wrong and no API refusal was added.
 ## 2. The call sites found — **FIVE, not four**
 | # | Site | Named by the spec? |
 |---|---|---|
 | 1 | `stacks/deploy.go` `withPathVars` → `${USERDATA_PATH}` | yes — the live defect |
 | 2 | `appexport/fabplan.go` | yes |
 | 3 | `appexport/export.go` | yes |
 | 4 | `stacks/delete.go` `ExportDataMounts` | yes |
 | 5 | **`web/handlers.go` FileBrowser mount builder** | **NO** |
 The fifth is the customer's own file browser: on a non-enrolled path it would have mounted the wrong
 directory. **Latent, not live** — the system drive is deliberately never a registered `StoragePath`, so
 the resolver is the identity there today. Wired anyway, with that reason in the code.
 Two further sites of the same class were found and fixed: `ComputeFabBuckets` was receiving the drive
 path where `ComputeCaptureSet` has always received the namespace root, so the export's classified
 paths and the backup's capture set could describe different directories for the same declared bind.
 **The rule had TWO existing copies and they differed.** `backup.Manager.namespaceRoot` compared
 without `filepath.Clean`; `stacks.Manager.inGuest` compared with it. A trailing slash from config
 would have flipped the mode in one package and not the other. Both now delegate to
 `appbackup.NamespaceRootFor`.
 ## 3. Scenario A — the two paths, before and after (live, demo-hp)
 ```
 before:  /mnt/sys_drive/userdata/media/books            ← app bind; capture set looked elsewhere
 after:   /mnt/sys_drive/felhom-data/userdata/media/books ← app bind == capture root
 ```
 Capture log: **`0 mandatory path(s)`** → **`1 mandatory path(s)`**.
 ## 4. Scenario F — the sentinel in the snapshot's file listing
 ```
 $ restic ls -l latest --tag calibre-web
 -rw-r--r--  1000  1000  181  2026-08-04 12:53:06
  /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
 ```
 And the snapshot's own `paths`:
 `["…/backups/primary/calibre-web", "/mnt/sys_drive/felhom-data/userdata/media/books"]`.
 **Not a green status — the file, by name and size.** sha256
 `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`, byte-identical to the drill's
 step 3 after the fix's migration moved it into the corrected directory.
 ## 5. Scenario E — the delete path
 **The §8.3 risk does not exist, and this is the correction owed.** `ExportDataMounts` lives in
 `delete.go` but is **export-only**: its single production caller is the `.fab` export adapter and
 nothing deletes on its result. The delete path's own guard, `ProtectedHDDPaths`, is **layout-agnostic
 by construction** — it protects both `<hdd>/…` and `<hdd>/felhom-data/…` — so deletion was never
 affected by the namespace-root defect. That note is now in the function's doc comment, because the
 file placement misled this change's own specification.
 **Does the corrected path point anywhere the old code did not?** Yes, for the export only, and only on
 the system-data fallback: `<hdd>/felhom-data/userdata` instead of `<hdd>/userdata`. That is the
 directory the app actually binds after this release, and the tests assert the **negative** — no
 emitted path lies outside the app's own data roots, on either drive kind. It shipped as its own commit.
 ## 6. The verdict value
 **`incomplete`** — **minted**, because `ok` | `error` | `running` contained nothing meaning *"it ran,
 and this app is not fully protected"*. **Not `error`:** the rest of the run worked and what was
 captured is real, so `SnapshotCount` and the `LastSuccess` anchor still record it.
 It reaches the operator through the **existing** per-run digest, `backup_run_failures` — already
 operator-only, already allowlisted. A new event type would have been a two-repo change and the hub
 drops anything outside `allowedEventTypes`; the prompt ruled out a hub change. The Hungarian customer
 warning is unchanged, and the backups page renders `! Hiányos` with the warn styling.
 ## 7. §8.4's narrowing — **no customer-visible warning disappears**
 `TierOffsite`'s `tierKeeps()` already admits `ClassMandatory` only, so an optional path cannot reach
 the stat-filter. The added class check is a **no-op today**, written for parity with Tier 2 — and
 **demonstrated to be load-bearing anyway**: widening the tier filter alone keeps the tests green
 *because of the check*; widening it and removing the check makes an optional gap start reporting.
 ## 8. §8.6's live effect — anticipated, and then NOT reproducible
 Anticipated in the CHANGELOG before it could fire: calibre-web on demo-hp had exactly this gap, so its
 status would become `incomplete`. **In the event it did not**, because the same session fixed the
 underlying path — after the corrected bind and the data migration the app has no gap, and the run is
 legitimately `ok`.
 **Attempting to observe the verdict live by hiding the directory did not work, and that is recorded
 rather than dressed up:** the running container's bind mount **recreated** it, so `os.Stat` succeeded
 and no gap existed. Worth knowing in itself — a bind-mounted directory cannot easily be "missing"
 while its app runs, so the mandatory-gap condition arises in practice when the path resolves somewhere
 the app never binds (the R-203 case), not when a live app's own directory vanishes. The verdict is
 proven by a run-level test that drives the real `RunOffboxBackup`. The fixture was restored and the
 sentinel re-verified at the same hash.
 ## 9. Tests and every red-proof
 | Scenario | Result | Red-proof — mutation → outcome |
 |---|---|---|
 | A both paths agree | PASS + **LIVE** | restore the bare-path call → **FAIL**: `/mnt/sys_drive/userdata` vs `/mnt/sys_drive/felhom-data/userdata` |
 | B enrolled drive unchanged | PASS | invert the drive-kind comparison → **FAIL** on every enrolled row |
 | C mandatory gap → not ok | PASS | unreachable gap recording → **FAIL**; unconditional `ok` → **FAIL** |
 | D optional gap → ok | PASS | class check removed **with the tier filter widened** → **FAIL**; tier widened alone → PASS, i.e. the check holds the line |
 | E export mounts, both kinds + the negative | PASS | leave the site bare → **FAIL**, emits the short path |
 | F sentinel in the listing | **LIVE** | — |
 **A RED-PROOF PASSED AND THE TEST WAS WRONG, NOT THE CODE** (§9.11 — three of the last six sessions).
 My first Scenario-C test exercised `offboxCaptureSet` alone, while the mutation lives in
 `runOffboxInternal`. A mutation the test cannot observe is not a red-proof. Replaced with a run-level
 test that drives `RunOffboxBackup` and asserts `incomplete`, the retained `LastSuccess`, and the
 operator signal; it fails under both mutations. The original Scenario-D "red-proof" also could not
 fail by construction — recorded above with the two-part mutation that does.
 Green gate after each phase (`go build && go vet && go test ./...`, rc=0) plus
 `controller_gates.py --fast`. No test run was combined with a commit.
 ## 10. Commits
 | Commit | Contents |
 |---|---|
-| `73efb09` | Part 1 — one resolver, the four non-export sites, `ComputeFabBuckets`, the two delegating copies |
+| sentinel sha256, pre-wipe | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
-| `a96c3d9` | **Part 1.3 alone** — the `delete.go`-resident export-mount site, with its scope correction |
+| sentinel sha256, post-restore | **not obtained** — the restore was never reachable |
 | `58c703b` | Part 2 — the verdict, the structural gaps, the operator digest, the status rendering |
 | `73fb595` | docs (felhom.eu) |
-## 11. Registers
+**Not a FAIL.** Nothing came back wrong and no fresh history was started; the box never got as far as
 running a backup. **What it is instead:** the first proof that the key survives and returns, plus the
 measured reason a customer still cannot use it.
- **R-203 → SHIPPED + PROVEN-LIVE.**
+> **After a real rebuild — controller data volume destroyed, sentinel deleted from disk — the
- **R-201 → READY TO RESUME**, blocker gone, fixture staged and verified.
+> customer's recovery code produced `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`,
- **R-202 stays open**; the orphaned-ciphertext deletion is **still owed**.
+> byte-identical to the pre-wipe on-disk key and to the hub's independent record.** That has never been
- Capability map: new row for mandatory-path capture on both layouts, its predecessor corrected as
+> shown before.
  optimistic, and the **R-199 back-pointer omitted two sessions ago is now added**.
 - The v0.93.0 `identity_blob` retention remains **unit-proven only** — nothing here superseded a key.
-## 12. CI
+## 2. Snapshot count at step 9 — not obtained
-Run numbers and task ids in the session summary. **`--no-verify` was not used.**
+The off-site run was never permitted to start (§4). **And a count would have been a poor
 discriminator anyway:** restic's same-day `forget --keep-daily 7 --group-by host,tags` keeps one
 snapshot per tag per day, so a successful reopen would have shown 3, not 4. The real observable is
 whether the pre-wipe snapshot `e6132ae5` survives with the sentinel in it — recorded as the resume
 step.
-## 13. Teardown
+## 3. §5's five conditions, recorded before the wipe
-**Nothing provisioned.** `calibre-web` and the sentinel stay — R-201 needs them. The temporarily
+1. **sentinel listed by name** — snapshot `e6132ae5` (19:36:26), `-rw-r--r-- 1000 1000 181 …/DRILL-SENTINEL.txt`.
-hidden directory was restored and the sentinel re-verified byte-identical.
+2. **rollback archive verified** — `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B,
   **full zstd stream read OK** (4 867 573 760 B), sentinel confirmed inside it.
 3. §3's option — §5 below.
 4. **demo-felhom** `health=ok`, `escrow_state=escrowed`, untouched throughout.
 5. **space** — nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
-## 14. Observations — noticed, NOT acted on
+**One precondition had drifted and was repaired, not worked around.** The staged snapshot no longer
 held the sentinel: the afternoon's `mandatory data path missing` experiment produced a *later* same-day
 calibre-web snapshot and `forget` had pruned the good one. The fixture on disk was correct, so one
 backup re-established it and it was re-verified by listing. **Lesson: a good snapshot is not durable
 against a later bad run on the same day.**
-1. **`resolveAbs` uses ONE root for two bind classes.** `RootHDD` resolves against the same parameter
+## 4. Every step's observable
-   as `RootUserdata`. On an enrolled drive they coincide; on the system-data fallback a `${HDD_PATH}`
+
-   bind resolves under the namespace root while compose binds it bare. Both callers now pass the
+| step | observable |
-   namespace root, so the export and the backup **agree with each other** — but whether `${HDD_PATH}`
+|---|---|
-   itself should mean the namespace root on the system drive is a separate question that touches every
+| 6 | fresh data dir stamped `20:00:2x`; **new `encryption.key`** (32 B); `claimed = None`; `offbox = null` |
-   already-deployed app's binds. **Not fixed here; it needs a decision, not a patch.**
+| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED across the wipe.** Nothing re-escrowed itself |
-2. **The blast radius was measured before changing anything:** exactly one app in the fleet has
+| **8** | **recovered `8a9e33aa4da6…`** — matches pre-wipe on-disk and the hub's record. Exit 1 is correct and designed: nothing local to compare against, the rebuilt-box shape |
-   `HDD_PATH == system_data_path` — `calibre-web` on demo-hp, deployed for the drill. Every other
+| 9a | `[INSTALLED] … reads back identical` — the "installed" branch's first real run |
-   deployed app has no `HDD_PATH`. Nothing else needed migrating.
+| 9b | after the Re-issue and apply, the on-disk key is **still** `8a9e33aa4da6…` — `WriteOffboxSecrets` kept it |
-3. **`/api/stacks/<name>/deploy` is first-deployment-only** (409 afterwards), so the corrected
+| 9c | **blocked** — see below |
-   `${USERDATA_PATH}` reaches an existing app through a start/redeploy, not a re-deploy.
+| 10–11 | **not run** |
-4. **The controller's CSRF form field is `_csrf`, not `csrf_token`** — the login page uses one name and
+
-   the protected forms another. Second session in a row this cost time; it belongs in the
+**The wipe was faithful to the incident, deliberately.** The 2026-08-03 rebuild R-193 is filed against
-   headless-access memory.
+was **not** a guest reprovision — the journal shows guest 9201 running continuously through that window
 with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the controller
 and its data volume. A guest destroy plus an unrehearsed provisioning chain, improvised unattended, is
 what §8.10 exists to prevent.
 ## 5. §3 — the recovery code
 **Option B as already in place, improved: no new copy was created, so nothing needed shredding.** The
 operator had placed `R_DEMO-HP` in their own `~/.config/credentials` (`0600`) two sessions ago for this
 purpose. It was read from there and **piped to stdin** for the two invocations that needed it — never
 an argument, never exported, never written to a second file, never logged. Destroying the operator's
 own store would have destroyed their record; because no additional copy existed, there is nothing left
 to prove gone.
 **Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
 controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
 `felhom-idesc-*` staging dirs — and the same sweep found a planted copy (**1**), then **0** after
 shredding it. The instrument is shown sensitive rather than assumed to be.
 ## 6. Part 2 — did not run
 Its gate is "the drill PASSED". It did not. A second wipe would have destroyed the state that makes the
 first drill finishable in five minutes. **R-198's retention therefore remains unit-proven only** —
 nothing has yet superseded a key in production.
 ## 7. Teardown — three layers
 | layer | state |
 |---|---|
 | the guest | **nothing torn down** — the wipe is the evidence; all six app containers serving; the recovered key on disk; scratch band empty |
 | the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
 | the hub | **no new customer records** — `demo-hp` is the same row throughout, because the rebuild was a controller-data wipe and not a re-enrolment. One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran |
 **Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
 still owed, deliberately not ridden along with a drill.
 ## 8. The capability-map rows
 A new row records the key as **PROVEN-LIVE after a real rebuild**, and states plainly what it does not
 claim: **no file has been restored**, and the four links of R-204 stand between the recovered key and a
 usable repository. The R-199 back-pointer added earlier today stands.
 ## 9. New findings
 - **R-204 (NEW)** — the recovered key cannot be used: a rebuilt box cannot configure its tier (R-193);
  the Re-issue that fixes that marks the escrow stale though the key never changed (R-196, **measured
  at `20:15:49` with `restic_pw_sha256` unchanged**); a stale escrow gates every run; and the only way
  to clear it is a ceremony that **destroys the recovered key**. Plus a fourth link in no design
  document: a rebuilt box is **unclaimed**, so the claim gate intercepts every controller endpoint.
 - **R-196 re-scoped** — no longer a documentation nit; it is on the critical path for recovery.
 - **R-201** — the wipe happened, the key came back, ~5 minutes from a verdict with a person present.
 - **R-198** — still unit-proven only; Part 2 gated out.
 - **R-202 open; the ciphertext deletion still owed.**
 ## 10. CI
 Docs push only. Run number and task id in the session summary. **`--no-verify` not used.**
 ## 11. Observations — noticed, NOT acted on
 1. **The claim gate is an undocumented first step of every recovery.** Before a customer can do
   anything on a rebuilt box — including recovering their backups — they must re-claim it.
 2. **`--print-reset-code` output needs parsing care**: the captured value was 73 characters, i.e. more
   than the code itself. Not chased; the claim was abandoned when the session stopped.
 3. **A same-day re-run replaces the day's snapshot.** Worth knowing before designing any drill that
   depends on a specific snapshot surviving.
 4. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
   `kubectl port-forward`. The `curl -u :$HUB_PW` recipe in memory omits that.
@@ -36,25 +36,21 @@ Proven end to end on real hardware.
  the old backups stay recoverable. Both keys are now kept. **What this cannot undo:** anything
  superseded before today is gone for good, which includes both demo machines' pre-4-August keys — so
  those 51 orphaned backups were beyond reach even if the recovery codes had been kept. *(R-198)*
- *(largely fixed 4 Aug)* ~~Nothing in the recovery path has ever been performed.~~ **The key now
+- **THE BACKUP KEY COMES BACK AFTER A REBUILD — proved on real hardware last night.** We destroyed the
-  demonstrably comes back — measured on a real machine this evening.** demo-felhom fetched its own
+  HP machine's controller data on purpose, deleted the marked file from its disk, and then used the
-  sealed package from the hub with its own credential, opened it with the recovery code you saved, and
+  recovery code you saved. The key that came out was **identical, character for character**, to the one
-  the backup key that came out was **identical, character for character, to the one the machine is
+  the machine had been using — and to the fingerprint the hub had recorded separately. It installed
-  using** — and to the fingerprint the hub had recorded separately. Three independent sources agreeing.
+  cleanly onto the empty machine. Nothing re-sealed itself in the meantime, so the sealed package
-  A deliberately wrong code, tried five minutes earlier, was refused outright and wrote nothing.
+  survived the rebuild untouched. *(R-201)*
-  **What is still NOT done, and it is the half that matters to a customer:** nothing yet *puts the
+- **But the machine still could not use it, and that is the night's real finding.** Three things stand
-  recovered key back*, reopens the old backup store with it, or **restores a single file**. Today
+  between a recovered key and a restored file, and each one is now measured rather than guessed:
-  proves the key survives and returns; it does not prove the backups do. *(R-199 closed; R-200 half;
+  **(1)** a rebuilt machine cannot set up its off-site connection at all — its one-time password was
-  R-201 open)*
+  spent by the machine it replaced; **(2)** the fix for that (re-issuing the credential) marks the
- *(fixed 4 Aug)* ~~A backup reported success while leaving out a folder the customer was told is
+  sealed package "stale" even though the key never changed, and a stale package blocks every off-site
-  protected.~~ **Both halves fixed the same day.** The app was writing to one folder and the backup was
+  backup; **(3)** the only documented way to clear that is to make a new recovery code — **which
-  looking in another — one directory apart, on machines whose apps live on the system disk. They now
+  replaces the sealed package and destroys the key we just recovered.** And before any of it, the
-  resolve to the same place, from a single piece of code instead of the three near-copies that had
+  rebuilt machine is unclaimed, so nothing on its dashboard responds until the customer claims it
-  quietly drifted. **And a backup that cannot capture a folder marked essential no longer reports
+  again — a step written down nowhere. **The key comes back and cannot be used.** *(R-204)*
  success**: it reports *Hiányos* (incomplete), names the app and the folders, and tells you — while
  still recording what it genuinely did capture, because half a backup is not no backup. Proved on the
  HP machine by listing the backup's own contents and finding the marked file there by name and size —
  not by trusting a green tick. *(R-203)*
 - **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
  says the old backups may later be restorable with the matching recovery code. From today that is true
  for machines that re-seal from now on and **false for anything already orphaned** — and the machine
@@ -103,7 +99,10 @@ Proven end to end on real hardware.
 ## What we're working on
- **Now:** the wipe-and-restore proof, which is **unblocked and staged**. The marked file now lands in
+- **Now:** finishing the proof — it is about five minutes with you present. The HP machine is sitting
  mid-drill with the recovered key already on it; it needs re-claiming and one setting confirmed, then
  the backup runs and the file is restored. **Do not let it make a new recovery code** — that would
  destroy the key we recovered. Then the deeper fix for the wall above. The marked file now lands in
  the off-site backup, so there is finally something to recover. Everything else is already in place on
  the HP machine — the recovery code you saved, a working off-site store, the app and the file. Nothing
  has been wiped; that step waits for your go-ahead at a marked stop point. Both honesty fixes shipped today: a changed backup key now
@@ -158,7 +157,12 @@ Proven end to end on real hardware.
 ## Changed since last update
- **2026-08-04 (latest)** — **Fixed the folder-left-out-of-the-backup problem, both halves.** The app
+- **2026-08-04 (night)** — **Wiped the HP machine on purpose and got the backup key back with your
  recovery code — identical, character for character.** First time that has ever been done. The drill
  then stopped short of restoring the file, at a wall worth more than the last step: the key comes back
  but cannot be used without an act that destroys it. The machine is up, its apps are serving, and it
  is five minutes from finished. *(R-201, R-204)*
 - **2026-08-04 (evening)** — **Fixed the folder-left-out-of-the-backup problem, both halves.** The app
  and its backup now look in the same directory, and a backup that misses a folder marked essential
  reports *incomplete* instead of success. Proved by listing the backup's own contents and finding the
  marked file. The wipe-and-restore proof is unblocked. *(R-203, R-201)*
@@ -0,0 +1,264 @@
 # DRILL — R-201, the night run: **THE KEY CAME BACK. The verdict was not reached.**
 **Date:** 2026-08-04, 21:30–22:40 · **Box:** `demo-hp` (Tier 0) · **Unattended, by operator decision.**
 **The wipe happened.** The box is up, its apps are serving, and it is left mid-drill by deliberate
 choice — see §7 for the exact state and the one command that resumes it.
 > **THE RESULT, first.** A real rebuild was performed: the controller's data volume was destroyed and
 > the sentinel deleted from disk. The customer's recovery code then produced
 > `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a` — **byte-identical to the key the
 > box used before the wipe, and to the hash the hub had independently recorded.**
 >
 > **The off-site backup key is recoverable after a machine is rebuilt. That has never been shown
 > before.**
 >
 > **And the drill did not finish**, because three separate things stand between a recovered key and a
 > restored file. All three are measured below. That is the other half of the night's result, and it is
 > the half nobody knew.
 ---
 ## 1. The verdict
 **NOT REACHED.** Step 10 (restore the sentinel and compare) was not run.
 | | |
 |---|---|
 | sentinel sha256, pre-wipe | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
 | sentinel sha256, post-restore | **not obtained** — the restore was never reachable |
 | snapshot count at step 9 | **not obtained** — the run was refused before it started |
 **This is not a FAIL.** A fail is "the file came back wrong" or "a fresh empty history was started".
 Neither happened, because the box never got as far as running a backup. What the night established is
 where the wall is.
 ---
 ## 2. What was proven, in order, on hardware
 ### Step 6 — the wipe
 The controller's data volume was destroyed and the bootstrap re-ran. Fresh data directory, everything
 stamped `2026-08-04 20:00:2x`:
 ```
 encryption.key   32 B   2026-08-04 20:00:21     ← brand new: every pre-wipe app secret is undecryptable
 claimed = None                                   ← the fresh-install signal
 offbox  = null                                   ← no off-site target
 ```
 **Faithful to the incident, and deliberately so.** The 2026-08-03 rebuild that R-193 is filed against
 was **not** a guest reprovision — the journal shows guest 9201 running continuously through that
 window, with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the
 controller and its data volume. Reproducing *that* is what the drill needs; a guest destroy plus an
 unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.
 ### Step 7 — the assertion that keeps recovery possible: **PASSED**
 ```
 host_escrow (demo-hp-bb76ea), AFTER the wipe:
  identity_blob    = 572 bytes          ← unchanged
  restic_pw_sha256 = 8a9e33aa4da6…      ← unchanged
  updated_at       = 2026-08-04 11:11:37 ← unchanged; nothing re-escrowed
  stale_at         = NULL
 ```
 **No ceremony was run and nothing re-escrowed itself.** The sealed key survived the rebuild untouched.
 ### Step 8 — **THE KEY CAME BACK**
 ```
 === offsite key recovery check (R-200) — compares, never installs ===
  recovered sha256: 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a
  [FAIL] there is no repository password on this box to compare against
  (the recovery itself SUCCEEDED — this box simply has no local key. That is the
   rebuilt-box shape, where the next step is to INSTALL rather than compare.)
 ```
 Exit 1 is **correct and designed**: there was nothing to compare against, because the wipe removed it.
 The recovered hash matches the pre-wipe on-disk key and the hub's own record — three independent
 sources agreeing, one of them recovered through the full chain (hub → agent → unseal → extract) on a
 box that had just lost everything it knew.
 ### Step 9a — installed cleanly
 ```
 === offsite key recovery INSTALL (R-200) ===
  on-disk   sha256: (none — this box has no repository password)
  recovered sha256: 8a9e33aa4da6…
  [INSTALLED] the recovered repository password is in place and reads back identical.
 ```
 The "installed" branch — the rebuilt-box shape v0.196.0 was written for — took its first real run.
 ### Step 9b — the apply kept it
 After the target was reconfigured, the on-disk key was still `8a9e33aa4da6…`. `WriteOffboxSecrets`
 found the file present and kept it, exactly as documented.
 ---
 ## 3. The wall — three blockers, each measured
 ### (a) R-193, reconfirmed live: a rebuilt controller cannot configure its off-site tier
 ```
 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.197.0)
 [WARN] [offsite-apply] reconcile: offsite-apply: consume one-time password:
       no unconsumed offsite password (already consumed or none provisioned)
       (retries on next config refresh/restart)
 ```
 The ledger, measured: `demo-hp` one-time secret created `07:11:51`, consumed `07:12:06` — **by the
 previous controller**. The rebuilt one has nothing to consume and retries forever. Exactly the state
 demo-hp sat in for 25 hours on 2026-08-03.
 **Remedy:** an operator Re-issue. Performed here through the designed endpoint
 (`POST /configs/demo-hp/offsite-reissue`, HTTP 303), after which the target configured normally.
 ### (b) R-196, measured live — and it lands squarely on the recovery path
 ```
 host_escrow (demo-hp-bb76ea), after the Re-issue:
  stale_at         = 2026-08-04 20:15:49   ← set
  restic_pw_sha256 = 8a9e33aa4da6…         ← UNCHANGED. The key did not move.
 ```
 **The escrow was marked stale while it perfectly covers the box's current key** — the recovered one.
 That is R-196's false staleness, and here it is not a cosmetic lie: a stale escrow makes the hub
 **withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending →
 escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses to run any off-site backup**.
 > **The remedy for (a) disables the recovery it was needed for.** The only documented way to clear a
 > stale escrow is a fresh ceremony — which supersedes the identity blob, i.e. **destroys the very key
 > being recovered**. Under hub v0.93.0 the old blob is now *retained*, but nothing can serve a
 > superseded blob back (R-199's inventory: the endpoint serves the CURRENT row only).
 **Superseded rows still number 2** — no ceremony was run tonight. The key is intact.
 ### (c) The claim gate — undocumented as a recovery step
 ```
 [DEBUG] [web] claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)
 ```
 A rebuilt box is **unclaimed** (`claimed = None`, fresh `settings.json`), and the claim gate correctly
 intercepts every non-claim route. **So no controller endpoint can be driven at all** — not the manual
 escrow confirm, not the backup trigger — until the customer re-claims the box. Both POSTs returned 302
 to the claim page and neither reached its handler; `last_run` stayed null, which is why the earlier
 reading of "the run was refused" needed this second look to be accurate.
 This is correct behaviour and it is **not wrong** — but it is a step in the customer's recovery journey
 that appears in no design document, and it comes *before* anything else can happen.
 ---
 ## 4. Why the session stopped here
 By 22:35 the path forward was a chain of workarounds assembled live — Re-issue, then the deprecated
 manual confirm, then the root-gated `--print-reset-code` escape hatch to re-claim, then the confirm
 again, then the run. Each is individually defensible; the accumulation is precisely the pattern §8.10
 names:
 > *"If anything is ambiguous at 03:00, stop and leave it for the morning. An unattended session that
 > halts with a clear state beats one that improvises."*
 The remaining steps need about five minutes **with a person present**. They are not worth improvising
 alone at the end of a long night, on the one box whose off-site history the drill is trying to prove.
 **Nothing was left broken.** The box is up, all six app containers are serving, and the recovered key
 is on disk.
 ---
 ## 5. §5's five conditions, as recorded before the wipe
 | # | Condition | Evidence |
 |---|---|---|
 | 1 | §4 confirmed, sentinel **listed by name** | snapshot `e6132ae5` (19:36:26) — `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt` |
 | 2 | deliberate rollback archive, **verified** | `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B; **full zstd stream read OK** (4 867 573 760 B uncompressed); the sentinel confirmed **inside** it |
 | 3 | §3's option | see §6 |
 | 4 | demo-felhom untouched | `health=ok`, `escrow_state=escrowed`, not touched at any point |
 | 5 | free space | nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 % |
 **A precondition had drifted and was repaired before the wipe, not worked around.** The staged
 snapshot no longer contained the sentinel: the afternoon's `[WARN] mandatory data path missing`
 experiment produced a *later* same-day calibre-web snapshot, and restic's `forget --keep-daily 7
 --group-by host,tags` had pruned the good one in its favour. The fixture on disk was correct, so one
 backup re-established it and it was re-verified by listing. **Lesson worth carrying: a good snapshot is
 not durable against a later bad run on the same day.**
 ## 6. §3 — the recovery code
 **Option B as already in place, with a strict improvement: no new copy was created, so nothing needed
 shredding.**
 The operator placed `R_DEMO-HP` in their own `~/.config/credentials` on DooPlex (mode `0600`) two
 sessions ago, deliberately, for this purpose. This session read it from there and **piped it to stdin**
 for each of the two invocations that needed it. It was never an argument, never exported, never written
 to a second file, and never logged.
 **Nothing was shredded, and that is the point:** the operator's own permanent store is theirs, not a
 drill artefact, and destroying it would have destroyed their record. Because no additional copy was
 made, there is nothing left behind to prove gone — a stronger position than option B's
 create-then-shred.
 **Verified afterwards** with the planted-copy positive control, exactly as on 2026-08-04 afternoon —
 see §8.
 ---
 ## 7. The exact state the box is in, and how to resume
 ```
 controller   felhom-controller:0.197.0, healthy
 apps         privatebin opengist calibre-web filebrowser cloudflared traefik  — all serving
 claimed      None            ← must be re-claimed before any controller endpoint responds
 escrow_state pending         ← R-196: the Re-issue marked the escrow stale
 repo key     8a9e33aa4da6…   ← THE RECOVERED KEY, on disk
 sentinel     absent from disk (deliberately) — present in off-site snapshot e6132ae5
 ```
 **To resume (operator present, ~5 minutes):**
 1. Re-claim the box — `docker exec felhom-controller /usr/local/bin/felhom-controller
   --print-reset-code`, then the claim page.
 2. Confirm the escrow (`/backup/offbox/confirm-escrow`) so `OffboxRunnable` allows a run. **Do NOT run
   a new ceremony** — it would supersede the identity blob and destroy the key under test.
 3. Run an off-site backup. **The observable is whether the repository OPENS** — and whether the
   pre-wipe snapshot `e6132ae5` still exists with the sentinel in it. A same-day `forget` keeps one
   snapshot per tag, so the *count* is a poor discriminator; the surviving history is the real one.
 4. Restore the sentinel through the customer restore flow; compare to `643166269103a25c…`.
 **Rollback, if preferred:** `pct restore 9201` from
 `/mnt/nvme-1tb/dump/vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` — verified by a full read before the
 wipe. It returns the box to its pre-wipe state and voids the remaining drill.
 ---
 ## 8. R persisted nowhere — searched, with a positive control
 Swept the agent journal, the controller container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`,
 `/root` on demo-hp for the recovery code: **0 hits**, and **0** leftover `felhom-idesc-*` unseal
 staging directories. A planted copy was found by the same sweep (**1**) and not found after shredding
 (**0**), so the instrument is shown sensitive rather than assumed to be.
 ---
 ## 9. Teardown — three layers
 | layer | state |
 |---|---|
 | the guest | **Nothing torn down.** The wipe is the evidence; the apps are serving; the recovered key is in place. The scratch band is empty — no restore-test guest was created tonight. |
 | the host | `vzdump` snapshot LVs removed cleanly by the archive job (`snap_vm-9201-disk-0/1_vzdump` both released). One new archive, 1.6 GB, on `nvme-1tb` (883 GB free). Nothing deleted. |
 | the hub | **No new customer records.** `demo-hp` is the same customer row throughout — the rebuild was a controller-data wipe, not a re-enrolment, so nothing accumulated. One Re-issue staged a fresh one-time secret (consumed at 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran. |
 **Nothing was deleted on the storage endpoint**, including the ~1.2 GB of previously-orphaned
 ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.
 ## 10. Part 2 — not run, and why
 Its gate is *"the drill PASSED"*. It did not — the verdict was not reached. Running a second wipe on a
 box whose first result is incomplete would have destroyed the staged state that makes the first one
 finishable in five minutes. **R-198's retention therefore remains unit-proven only**, unchanged from
 this morning.
@@ -66,6 +66,7 @@
 | R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
 | R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
 | R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
 | R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **OPEN — the wall the 2026-08-04 night drill hit** | Blocks R-201's verdict. Fix is most likely R-196's (mark stale on evidence, not on a Re-issue) or letting a verified recovered key confirm the escrow without a ceremony |
 | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
 | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
 | R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |