125 lines
7.2 KiB
Markdown
125 lines
7.2 KiB
Markdown
# REPORT — R-201: **THE DRILL PASSED.** A customer's file survived a machine rebuild
|
||
|
||
**2026-08-04, 21:30–23:15** · `demo-hp` deliberately rebuilt · no code, no version bump ·
|
||
`demo-felhom` untouched. Record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
|
||
|
||
---
|
||
|
||
## 1. THE VERDICT — PASS
|
||
|
||
```
|
||
pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||
restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||
```
|
||
|
||
**Byte-identical.** The controller's data volume was destroyed and the file deleted from disk; the
|
||
customer's recovery code, the customer's restore flow, and the file came back. **The Felhom backup
|
||
story is proved end to end for the first time.**
|
||
|
||
## 2. Snapshot count at step 9 — **3, not 1**
|
||
|
||
`repo_state: null` (the repository **opened**, not orphaned), `snapshot_count: 3`, `repo_size_bytes:
|
||
42 026` — the pre-wipe size exactly. The pre-wipe snapshot `e6132ae5` was still present with the file
|
||
inside it. **No fresh history was started over the old one** — the failure this drill was designed to
|
||
catch did not occur.
|
||
|
||
## 3. §5's five conditions, recorded before the wipe
|
||
|
||
1. sentinel **listed by name** in snapshot `e6132ae5` (19:36:26), 181 B.
|
||
2. rollback archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` (1 606 765 083 B) **verified by a
|
||
full zstd stream read** (4 867 573 760 B) with the sentinel confirmed inside.
|
||
3. §3's option — §4 below.
|
||
4. `demo-felhom` `health=ok`, `escrow_state=escrowed`, untouched throughout.
|
||
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
|
||
|
||
**One precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the
|
||
sentinel because the afternoon's experiment produced a *later* same-day snapshot and `forget
|
||
--keep-daily 7 --group-by host,tags` pruned the good one. One backup re-established it, re-verified by
|
||
listing. **A good snapshot is not durable against a later bad run on the same day.**
|
||
|
||
## 4. §3 — the recovery code
|
||
|
||
**Option B as already in place, improved: no new copy was made, so nothing needed shredding.** The
|
||
operator's own `~/.config/credentials` (`0600`) was the source; the code was **piped to stdin** for
|
||
each invocation — never an argument, never exported, never a second file, never logged. Destroying the
|
||
operator's own store would have destroyed their record; because no extra copy existed, there is nothing
|
||
left to prove gone.
|
||
|
||
**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
|
||
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
|
||
`felhom-idesc-*` dirs — and the same sweep found a planted copy (**1**), then **0** after shredding it.
|
||
|
||
## 5. Every step's observable
|
||
|
||
| step | observable |
|
||
|---|---|
|
||
| 6 | fresh data dir `20:00:2x`; new `encryption.key`; `claimed = None`; `offbox = null` |
|
||
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED.** Nothing re-escrowed itself |
|
||
| **8** | **recovered `8a9e33aa4da6…`** — matches the pre-wipe on-disk key and the hub's record |
|
||
| 9a | `[INSTALLED] … reads back identical` |
|
||
| 9b | after Re-issue + apply, the on-disk key is **still** the recovered one |
|
||
| 9c | **repository OPENED** — `repo_state: null`, 3 snapshots, 42 026 B |
|
||
| **10** | **restored sha256 byte-identical** |
|
||
| 11 | this record |
|
||
|
||
Superseded escrow rows remained **2** throughout — **no ceremony was run at any point.**
|
||
|
||
## 6. Part 2 — not run
|
||
|
||
Gate: *"the drill PASSED; its evidence is written down; and there is time."* The first two are met; the
|
||
third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left
|
||
for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it.
|
||
**R-198's retention remains unit-proven only** — nothing has yet superseded a key in production, and it
|
||
is now the last unproven link in this chain.
|
||
|
||
## 7. Teardown — three layers
|
||
|
||
| layer | state |
|
||
|---|---|
|
||
| the guest | **healthy and re-armed**: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is **kept as evidence**. Scratch band empty |
|
||
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
|
||
| the hub | **no new customer records** — `demo-hp` is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. **The two superseded escrow rows are unchanged** |
|
||
|
||
**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
|
||
still owed, deliberately not ridden along with a drill.
|
||
|
||
## 8. The capability-map row
|
||
|
||
Now reads *"A customer's file survives a machine rebuild and comes back — the whole off-site story, end
|
||
to end"*, **PROVEN-LIVE**. What it deliberately does **not** claim: the journey is manual and
|
||
undocumented (R-204); the scope is `demo-hp` and a **controller-data** rebuild, **not** a total host
|
||
loss and **not** a guest reprovision; and R-198's retention is still unit-proven.
|
||
|
||
## 9. New findings — R-204, expanded into the gap list
|
||
|
||
The four steps between a recovered key and a restored file, all measured while walking them:
|
||
|
||
1. **R-193** — a rebuilt controller cannot configure its off-site tier (`no unconsumed offsite
|
||
password`; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
|
||
2. **The claim gate** — a rebuilt box is unclaimed, so *every* controller endpoint is intercepted. And
|
||
**the local escape hatch does not work unaided**: `--print-reset-code` writes the new hash to
|
||
`settings.json` while the running controller keeps its old copy in memory, so `effectiveClaimCode()`
|
||
never sees it. **Restart the controller between minting and claiming** — two claim attempts failed
|
||
before this was diagnosed.
|
||
3. **R-196** — the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged, which
|
||
gates every off-site run. Cleared with the **manual** confirm, never a ceremony.
|
||
4. **`mode=unit` is the restore default** and returns the recovery unit, **not** the userdata leg. A
|
||
customer told to "restore from off-site" gets their app definition and not their documents, and
|
||
nothing in the outcome says so. **The worst of the four**, because it fails silently at the last
|
||
step.
|
||
|
||
## 10. CI
|
||
|
||
Docs push only; run number and task id in the session summary. **`--no-verify` not used.**
|
||
|
||
## 11. Observations — noticed, NOT acted on
|
||
|
||
1. **A same-day re-run replaces the day's snapshot** (`forget --keep-daily 7 --group-by host,tags`).
|
||
Any drill depending on a specific snapshot surviving must account for it.
|
||
2. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
|
||
`kubectl port-forward`; the memory's `curl -u :$HUB_PW` recipe omits that.
|
||
3. **`--print-reset-code` mints a new generation on every invocation** — three wasted generations here
|
||
(4, 5, 6) while diagnosing the in-memory staleness.
|
||
4. **The restore wrote into `backups/offsite-restore/<app>/` mirroring the full absolute path** — deep
|
||
but unambiguous; worth knowing before writing customer-facing copy about where files land.
|