# DRILL — R-201, the night run: **PASS. The customer's file came back, byte-identical.**

**Date:** 2026-08-04, 21:30–23:15 · **Box:** `demo-hp` (Tier 0) · unattended, then **finished with the
operator present**. The box is up, healthy, and fully re-armed.

> **THE VERDICT: PASS.**
>
> ```
> pre-wipe   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
> restored   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
> ```
>
> A machine was destroyed on purpose — controller data volume gone, the customer's file deleted from
> disk — and **the file came back byte-identical**, through the customer's own recovery code and the
> customer's own restore flow. **This is the first time the Felhom backup story has been proved end to
> end.**
>
> The key recovered byte-identical (`8a9e33aa4da6…`), the existing repository **opened rather than
> being re-initialised** (`repo_state: null`, 3 snapshots, 42 026 B — the pre-wipe size exactly), and
> the pre-wipe snapshot `e6132ae5` was still there with the file inside it.
>
> **It took four undocumented steps to get there**, each measured, and they are the other half of the
> result — see §3. A customer hitting this alone today would not get their data back.

---

## 1. The verdict — **PASS**

| | |
|---|---|
| sentinel sha256, pre-wipe (step 3) | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
| sentinel sha256, restored (step 10) | **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** |
| **match** | **YES — byte-identical** |
| snapshot count at step 9 | **3** — the pre-wipe count. **NOT 1.** No fresh history was started |
| repo size at step 9 | **42 026 B** — the pre-wipe size exactly |
| `repo_state` | **`null`** — the repository OPENED. Not orphaned |

The restore ran through the customer flow (`POST /backup/offbox/restore`, `mode=full`, two-step
confirm), out of snapshot `e6132ae5` — the one taken **before** the wipe.

**A detail worth keeping:** the first restore attempt used the default `mode=unit`, which recovers the
recovery unit only and **not** the userdata leg. `mode=full` is what returns the customer's files. A
customer told to "restore from off-site" would, on the default, get their app definition back and not
their documents — and nothing in that outcome says so.

## 2. What was proven, in order, on hardware

### Step 6 — the wipe

The controller's data volume was destroyed and the bootstrap re-ran. Fresh data directory, everything
stamped `2026-08-04 20:00:2x`:

```
encryption.key   32 B   2026-08-04 20:00:21     ← brand new: every pre-wipe app secret is undecryptable
claimed = None                                   ← the fresh-install signal
offbox  = null                                   ← no off-site target
```

**Faithful to the incident, and deliberately so.** The 2026-08-03 rebuild that R-193 is filed against
was **not** a guest reprovision — the journal shows guest 9201 running continuously through that
window, with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the
controller and its data volume. Reproducing *that* is what the drill needs; a guest destroy plus an
unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.

### Step 7 — the assertion that keeps recovery possible: **PASSED**

```
host_escrow (demo-hp-bb76ea), AFTER the wipe:
  identity_blob    = 572 bytes          ← unchanged
  restic_pw_sha256 = 8a9e33aa4da6…      ← unchanged
  updated_at       = 2026-08-04 11:11:37 ← unchanged; nothing re-escrowed
  stale_at         = NULL
```

**No ceremony was run and nothing re-escrowed itself.** The sealed key survived the rebuild untouched.

### Step 8 — **THE KEY CAME BACK**

```
=== offsite key recovery check (R-200) — compares, never installs ===
  recovered sha256: 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a
  [FAIL] there is no repository password on this box to compare against
  (the recovery itself SUCCEEDED — this box simply has no local key. That is the
   rebuilt-box shape, where the next step is to INSTALL rather than compare.)
```

Exit 1 is **correct and designed**: there was nothing to compare against, because the wipe removed it.
The recovered hash matches the pre-wipe on-disk key and the hub's own record — three independent
sources agreeing, one of them recovered through the full chain (hub → agent → unseal → extract) on a
box that had just lost everything it knew.

### Step 9a — installed cleanly

```
=== offsite key recovery INSTALL (R-200) ===
  on-disk   sha256: (none — this box has no repository password)
  recovered sha256: 8a9e33aa4da6…
  [INSTALLED] the recovered repository password is in place and reads back identical.
```

The "installed" branch — the rebuilt-box shape v0.196.0 was written for — took its first real run.

### Step 9b — the apply kept it

After the target was reconfigured, the on-disk key was still `8a9e33aa4da6…`. `WriteOffboxSecrets`
found the file present and kept it, exactly as documented.

---

## 3. The wall — three blockers, each measured

### (a) R-193, reconfirmed live: a rebuilt controller cannot configure its off-site tier

```
[INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.197.0)
[WARN] [offsite-apply] reconcile: offsite-apply: consume one-time password:
       no unconsumed offsite password (already consumed or none provisioned)
       (retries on next config refresh/restart)
```

The ledger, measured: `demo-hp` one-time secret created `07:11:51`, consumed `07:12:06` — **by the
previous controller**. The rebuilt one has nothing to consume and retries forever. Exactly the state
demo-hp sat in for 25 hours on 2026-08-03.

**Remedy:** an operator Re-issue. Performed here through the designed endpoint
(`POST /configs/demo-hp/offsite-reissue`, HTTP 303), after which the target configured normally.

### (b) R-196, measured live — and it lands squarely on the recovery path

```
host_escrow (demo-hp-bb76ea), after the Re-issue:
  stale_at         = 2026-08-04 20:15:49   ← set
  restic_pw_sha256 = 8a9e33aa4da6…         ← UNCHANGED. The key did not move.
```

**The escrow was marked stale while it perfectly covers the box's current key** — the recovered one.
That is R-196's false staleness, and here it is not a cosmetic lie: a stale escrow makes the hub
**withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending →
escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses to run any off-site backup**.

> **The remedy for (a) disables the recovery it was needed for.** The only documented way to clear a
> stale escrow is a fresh ceremony — which supersedes the identity blob, i.e. **destroys the very key
> being recovered**. Under hub v0.93.0 the old blob is now *retained*, but nothing can serve a
> superseded blob back (R-199's inventory: the endpoint serves the CURRENT row only).

**Superseded rows still number 2** — no ceremony was run tonight. The key is intact.

### (c) The claim gate — undocumented as a recovery step

```
[DEBUG] [web] claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)
```

A rebuilt box is **unclaimed** (`claimed = None`, fresh `settings.json`), and the claim gate correctly
intercepts every non-claim route. **So no controller endpoint can be driven at all** — not the manual
escrow confirm, not the backup trigger — until the customer re-claims the box. Both POSTs returned 302
to the claim page and neither reached its handler; `last_run` stayed null, which is why the earlier
reading of "the run was refused" needed this second look to be accurate.

This is correct behaviour and it is **not wrong** — but it is a step in the customer's recovery journey
that appears in no design document, and it comes *before* anything else can happen.

---

## 4. How it was finished, and the fourth undocumented step

The unattended run stopped at the wall in §3 rather than improvising further at 22:35 (§8.10). With the
operator present the remaining path was walked, and it took **four steps a customer would have to
discover**:

1. **Re-issue** the offsite credential (hub, operator-only) — otherwise the box can never configure its
   tier. §3(a).
2. **Re-claim the box.** The local escape hatch (`--print-reset-code`) mints a code — **and does not
   work on its own**: it writes the new hash to `settings.json` while the RUNNING controller keeps its
   old copy in memory, so `effectiveClaimCode()` never sees it and every attempt returns
   *"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming.** Two
   claim attempts failed on this before it was diagnosed; the generation counter reached 6.
3. **Confirm the escrow** manually (`/backup/offbox/confirm-escrow`) → `escrowed`, so `OffboxRunnable`
   permits a run. **Not a ceremony** — the identity blob was never superseded (still 2 retained rows).
4. **Restore with `mode=full`.** The default `mode=unit` returns the recovery unit and **not** the
   customer's files.

**None of these four is in any design document, and the drill needed all four.**

## 5. §5's five conditions, as recorded before the wipe

| # | Condition | Evidence |
|---|---|---|
| 1 | §4 confirmed, sentinel **listed by name** | snapshot `e6132ae5` (19:36:26) — `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt` |
| 2 | deliberate rollback archive, **verified** | `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B; **full zstd stream read OK** (4 867 573 760 B uncompressed); the sentinel confirmed **inside** it |
| 3 | §3's option | see §6 |
| 4 | demo-felhom untouched | `health=ok`, `escrow_state=escrowed`, not touched at any point |
| 5 | free space | nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 % |

**A precondition had drifted and was repaired before the wipe, not worked around.** The staged
snapshot no longer contained the sentinel: the afternoon's `[WARN] mandatory data path missing`
experiment produced a *later* same-day calibre-web snapshot, and restic's `forget --keep-daily 7
--group-by host,tags` had pruned the good one in its favour. The fixture on disk was correct, so one
backup re-established it and it was re-verified by listing. **Lesson worth carrying: a good snapshot is
not durable against a later bad run on the same day.**

## 6. §3 — the recovery code

**Option B as already in place, with a strict improvement: no new copy was created, so nothing needed
shredding.**

The operator placed `R_DEMO-HP` in their own `~/.config/credentials` on DooPlex (mode `0600`) two
sessions ago, deliberately, for this purpose. This session read it from there and **piped it to stdin**
for each of the two invocations that needed it. It was never an argument, never exported, never written
to a second file, and never logged.

**Nothing was shredded, and that is the point:** the operator's own permanent store is theirs, not a
drill artefact, and destroying it would have destroyed their record. Because no additional copy was
made, there is nothing left behind to prove gone — a stronger position than option B's
create-then-shred.

**Verified afterwards** with the planted-copy positive control, exactly as on 2026-08-04 afternoon —
see §8.

---

## 7. The state the box is left in — healthy and re-armed

```
controller   felhom-controller:0.197.0, healthy, CLAIMED
apps         privatebin opengist calibre-web filebrowser cloudflared traefik  — all serving
escrow_state escrowed          (no ceremony was run; identity_blob untouched)
repo key     8a9e33aa4da6…     THE RECOVERED KEY — and the repository opens with it
off-site     3 snapshots, 42 026 B, repo_state null; all three apps re-toggled for off-site
sentinel     restored to its live location, sha256 643166269103a25c… — verified
```

The verification copy under `backups/offsite-restore/calibre-web/` is **left in place as evidence**.

## 8. R persisted nowhere — searched, with a positive control

Swept the agent journal, the controller container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`,
`/root` on demo-hp for the recovery code: **0 hits**, and **0** leftover `felhom-idesc-*` unseal
staging directories. A planted copy was found by the same sweep (**1**) and not found after shredding
(**0**), so the instrument is shown sensitive rather than assumed to be.

---

## 9. Teardown — three layers

| layer | state |
|---|---|
| the guest | **Nothing torn down.** The wipe is the evidence; the apps are serving; the recovered key is in place. The scratch band is empty — no restore-test guest was created tonight. |
| the host | `vzdump` snapshot LVs removed cleanly by the archive job (`snap_vm-9201-disk-0/1_vzdump` both released). One new archive, 1.6 GB, on `nvme-1tb` (883 GB free). Nothing deleted. |
| the hub | **No new customer records.** `demo-hp` is the same customer row throughout — the rebuild was a controller-data wipe, not a re-enrolment, so nothing accumulated. One Re-issue staged a fresh one-time secret (consumed at 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran. |

**Nothing was deleted on the storage endpoint**, including the ~1.2 GB of previously-orphaned
ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.

## 10. Part 2 — not run

Its gate is *"the drill PASSED; its evidence is written down; and there is time."* The first two are
now met. **The third was not** — it was 23:15 after a full destructive cycle, and Part 2 is a second
wipe. It is deliberately left for a session that starts fresh, with drill 1's result already recorded
so a second wipe cannot overwrite it.

**R-198's retention therefore remains unit-proven only.** Nothing has yet superseded a key in
production, so the retention's live behaviour is still unobserved — and it is now the last unproven
link in the chain this drill exercised. The check is cheap: wipe again, do NOT recover, let a ceremony
seal a different password, then read the superseded row's `identity_blob`.
