R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
This commit is contained in:
@@ -1,38 +1,47 @@
|
||||
# DRILL — R-201, the night run: **THE KEY CAME BACK. The verdict was not reached.**
|
||||
# DRILL — R-201, the night run: **PASS. The customer's file came back, byte-identical.**
|
||||
|
||||
**Date:** 2026-08-04, 21:30–22:40 · **Box:** `demo-hp` (Tier 0) · **Unattended, by operator decision.**
|
||||
**The wipe happened.** The box is up, its apps are serving, and it is left mid-drill by deliberate
|
||||
choice — see §7 for the exact state and the one command that resumes it.
|
||||
**Date:** 2026-08-04, 21:30–23:15 · **Box:** `demo-hp` (Tier 0) · unattended, then **finished with the
|
||||
operator present**. The box is up, healthy, and fully re-armed.
|
||||
|
||||
> **THE RESULT, first.** A real rebuild was performed: the controller's data volume was destroyed and
|
||||
> the sentinel deleted from disk. The customer's recovery code then produced
|
||||
> `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a` — **byte-identical to the key the
|
||||
> box used before the wipe, and to the hash the hub had independently recorded.**
|
||||
> **THE VERDICT: PASS.**
|
||||
>
|
||||
> **The off-site backup key is recoverable after a machine is rebuilt. That has never been shown
|
||||
> before.**
|
||||
> ```
|
||||
> pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
> restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||
> ```
|
||||
>
|
||||
> **And the drill did not finish**, because three separate things stand between a recovered key and a
|
||||
> restored file. All three are measured below. That is the other half of the night's result, and it is
|
||||
> the half nobody knew.
|
||||
> A machine was destroyed on purpose — controller data volume gone, the customer's file deleted from
|
||||
> disk — and **the file came back byte-identical**, through the customer's own recovery code and the
|
||||
> customer's own restore flow. **This is the first time the Felhom backup story has been proved end to
|
||||
> end.**
|
||||
>
|
||||
> The key recovered byte-identical (`8a9e33aa4da6…`), the existing repository **opened rather than
|
||||
> being re-initialised** (`repo_state: null`, 3 snapshots, 42 026 B — the pre-wipe size exactly), and
|
||||
> the pre-wipe snapshot `e6132ae5` was still there with the file inside it.
|
||||
>
|
||||
> **It took four undocumented steps to get there**, each measured, and they are the other half of the
|
||||
> result — see §3. A customer hitting this alone today would not get their data back.
|
||||
|
||||
---
|
||||
|
||||
## 1. The verdict
|
||||
|
||||
**NOT REACHED.** Step 10 (restore the sentinel and compare) was not run.
|
||||
## 1. The verdict — **PASS**
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| sentinel sha256, pre-wipe | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
|
||||
| sentinel sha256, post-restore | **not obtained** — the restore was never reachable |
|
||||
| snapshot count at step 9 | **not obtained** — the run was refused before it started |
|
||||
| sentinel sha256, pre-wipe (step 3) | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
|
||||
| sentinel sha256, restored (step 10) | **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** |
|
||||
| **match** | **YES — byte-identical** |
|
||||
| snapshot count at step 9 | **3** — the pre-wipe count. **NOT 1.** No fresh history was started |
|
||||
| repo size at step 9 | **42 026 B** — the pre-wipe size exactly |
|
||||
| `repo_state` | **`null`** — the repository OPENED. Not orphaned |
|
||||
|
||||
**This is not a FAIL.** A fail is "the file came back wrong" or "a fresh empty history was started".
|
||||
Neither happened, because the box never got as far as running a backup. What the night established is
|
||||
where the wall is.
|
||||
The restore ran through the customer flow (`POST /backup/offbox/restore`, `mode=full`, two-step
|
||||
confirm), out of snapshot `e6132ae5` — the one taken **before** the wipe.
|
||||
|
||||
---
|
||||
**A detail worth keeping:** the first restore attempt used the default `mode=unit`, which recovers the
|
||||
recovery unit only and **not** the userdata leg. `mode=full` is what returns the customer's files. A
|
||||
customer told to "restore from off-site" would, on the default, get their app definition back and not
|
||||
their documents — and nothing in that outcome says so.
|
||||
|
||||
## 2. What was proven, in order, on hardware
|
||||
|
||||
@@ -153,23 +162,25 @@ that appears in no design document, and it comes *before* anything else can happ
|
||||
|
||||
---
|
||||
|
||||
## 4. Why the session stopped here
|
||||
## 4. How it was finished, and the fourth undocumented step
|
||||
|
||||
By 22:35 the path forward was a chain of workarounds assembled live — Re-issue, then the deprecated
|
||||
manual confirm, then the root-gated `--print-reset-code` escape hatch to re-claim, then the confirm
|
||||
again, then the run. Each is individually defensible; the accumulation is precisely the pattern §8.10
|
||||
names:
|
||||
The unattended run stopped at the wall in §3 rather than improvising further at 22:35 (§8.10). With the
|
||||
operator present the remaining path was walked, and it took **four steps a customer would have to
|
||||
discover**:
|
||||
|
||||
> *"If anything is ambiguous at 03:00, stop and leave it for the morning. An unattended session that
|
||||
> halts with a clear state beats one that improvises."*
|
||||
1. **Re-issue** the offsite credential (hub, operator-only) — otherwise the box can never configure its
|
||||
tier. §3(a).
|
||||
2. **Re-claim the box.** The local escape hatch (`--print-reset-code`) mints a code — **and does not
|
||||
work on its own**: it writes the new hash to `settings.json` while the RUNNING controller keeps its
|
||||
old copy in memory, so `effectiveClaimCode()` never sees it and every attempt returns
|
||||
*"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming.** Two
|
||||
claim attempts failed on this before it was diagnosed; the generation counter reached 6.
|
||||
3. **Confirm the escrow** manually (`/backup/offbox/confirm-escrow`) → `escrowed`, so `OffboxRunnable`
|
||||
permits a run. **Not a ceremony** — the identity blob was never superseded (still 2 retained rows).
|
||||
4. **Restore with `mode=full`.** The default `mode=unit` returns the recovery unit and **not** the
|
||||
customer's files.
|
||||
|
||||
The remaining steps need about five minutes **with a person present**. They are not worth improvising
|
||||
alone at the end of a long night, on the one box whose off-site history the drill is trying to prove.
|
||||
|
||||
**Nothing was left broken.** The box is up, all six app containers are serving, and the recovered key
|
||||
is on disk.
|
||||
|
||||
---
|
||||
**None of these four is in any design document, and the drill needed all four.**
|
||||
|
||||
## 5. §5's five conditions, as recorded before the wipe
|
||||
|
||||
@@ -208,33 +219,18 @@ see §8.
|
||||
|
||||
---
|
||||
|
||||
## 7. The exact state the box is in, and how to resume
|
||||
## 7. The state the box is left in — healthy and re-armed
|
||||
|
||||
```
|
||||
controller felhom-controller:0.197.0, healthy
|
||||
controller felhom-controller:0.197.0, healthy, CLAIMED
|
||||
apps privatebin opengist calibre-web filebrowser cloudflared traefik — all serving
|
||||
claimed None ← must be re-claimed before any controller endpoint responds
|
||||
escrow_state pending ← R-196: the Re-issue marked the escrow stale
|
||||
repo key 8a9e33aa4da6… ← THE RECOVERED KEY, on disk
|
||||
sentinel absent from disk (deliberately) — present in off-site snapshot e6132ae5
|
||||
escrow_state escrowed (no ceremony was run; identity_blob untouched)
|
||||
repo key 8a9e33aa4da6… THE RECOVERED KEY — and the repository opens with it
|
||||
off-site 3 snapshots, 42 026 B, repo_state null; all three apps re-toggled for off-site
|
||||
sentinel restored to its live location, sha256 643166269103a25c… — verified
|
||||
```
|
||||
|
||||
**To resume (operator present, ~5 minutes):**
|
||||
|
||||
1. Re-claim the box — `docker exec felhom-controller /usr/local/bin/felhom-controller
|
||||
--print-reset-code`, then the claim page.
|
||||
2. Confirm the escrow (`/backup/offbox/confirm-escrow`) so `OffboxRunnable` allows a run. **Do NOT run
|
||||
a new ceremony** — it would supersede the identity blob and destroy the key under test.
|
||||
3. Run an off-site backup. **The observable is whether the repository OPENS** — and whether the
|
||||
pre-wipe snapshot `e6132ae5` still exists with the sentinel in it. A same-day `forget` keeps one
|
||||
snapshot per tag, so the *count* is a poor discriminator; the surviving history is the real one.
|
||||
4. Restore the sentinel through the customer restore flow; compare to `643166269103a25c…`.
|
||||
|
||||
**Rollback, if preferred:** `pct restore 9201` from
|
||||
`/mnt/nvme-1tb/dump/vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` — verified by a full read before the
|
||||
wipe. It returns the box to its pre-wipe state and voids the remaining drill.
|
||||
|
||||
---
|
||||
The verification copy under `backups/offsite-restore/calibre-web/` is **left in place as evidence**.
|
||||
|
||||
## 8. R persisted nowhere — searched, with a positive control
|
||||
|
||||
@@ -256,9 +252,14 @@ staging directories. A planted copy was found by the same sweep (**1**) and not
|
||||
**Nothing was deleted on the storage endpoint**, including the ~1.2 GB of previously-orphaned
|
||||
ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.
|
||||
|
||||
## 10. Part 2 — not run, and why
|
||||
## 10. Part 2 — not run
|
||||
|
||||
Its gate is *"the drill PASSED"*. It did not — the verdict was not reached. Running a second wipe on a
|
||||
box whose first result is incomplete would have destroyed the staged state that makes the first one
|
||||
finishable in five minutes. **R-198's retention therefore remains unit-proven only**, unchanged from
|
||||
this morning.
|
||||
Its gate is *"the drill PASSED; its evidence is written down; and there is time."* The first two are
|
||||
now met. **The third was not** — it was 23:15 after a full destructive cycle, and Part 2 is a second
|
||||
wipe. It is deliberately left for a session that starts fresh, with drill 1's result already recorded
|
||||
so a second wipe cannot overwrite it.
|
||||
|
||||
**R-198's retention therefore remains unit-proven only.** Nothing has yet superseded a key in
|
||||
production, so the retention's live behaviour is still unobserved — and it is now the last unproven
|
||||
link in the chain this drill exercised. The check is cheap: wipe again, do NOT recover, let a ceremony
|
||||
seal a different password, then read the superseded row's `identity_blob`.
|
||||
|
||||
Reference in New Issue
Block a user