admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

R-201 PASSED: a customer's file survived a machine rebuild and came back byte-identical
gates / gates (push) Successful in 7s
Details

Browse Source
This commit is contained in:
authentik Default Admin
2026-08-04 23:18:47 +02:00
parent b228fd102d
commit 2a7ac03c47
7 changed files with 194 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+86 -89
View File
@@ -1,127 +1,124 @@
# REPORT — R-201 night run: the key came back; the verdict did not
# REPORT — R-201: **THE DRILL PASSED.** A customer's file survived a machine rebuild
**2026-08-04, 21:30–22:40, unattended.** `demo-hp` was deliberately rebuilt. No code, no version bump.
`demo-felhom` untouched. Full record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
**2026-08-04, 21:30–23:15** · `demo-hp` deliberately rebuilt · no code, no version bump ·
`demo-felhom` untouched. Record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
---
## 1. THE VERDICT — not reached
## 1. THE VERDICT — PASS
| | |
|---|---|
| sentinel sha256, pre-wipe | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
| sentinel sha256, post-restore | **not obtained** — the restore was never reachable |
```
pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
```
**Not a FAIL.** Nothing came back wrong and no fresh history was started; the box never got as far as
running a backup. **What it is instead:** the first proof that the key survives and returns, plus the
measured reason a customer still cannot use it.
**Byte-identical.** The controller's data volume was destroyed and the file deleted from disk; the
customer's recovery code, the customer's restore flow, and the file came back. **The Felhom backup
story is proved end to end for the first time.**
> **After a real rebuild — controller data volume destroyed, sentinel deleted from disk — the
> customer's recovery code produced `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`,
> byte-identical to the pre-wipe on-disk key and to the hub's independent record.** That has never been
> shown before.
## 2. Snapshot count at step 9 — **3, not 1**
## 2. Snapshot count at step 9 — not obtained
The off-site run was never permitted to start (§4). **And a count would have been a poor
discriminator anyway:** restic's same-day `forget --keep-daily 7 --group-by host,tags` keeps one
snapshot per tag per day, so a successful reopen would have shown 3, not 4. The real observable is
whether the pre-wipe snapshot `e6132ae5` survives with the sentinel in it — recorded as the resume
step.
`repo_state: null` (the repository **opened**, not orphaned), `snapshot_count: 3`, `repo_size_bytes:
42 026` — the pre-wipe size exactly. The pre-wipe snapshot `e6132ae5` was still present with the file
inside it. **No fresh history was started over the old one** — the failure this drill was designed to
catch did not occur.
## 3. §5's five conditions, recorded before the wipe
1. **sentinel listed by name** — snapshot `e6132ae5` (19:36:26), `-rw-r--r-- 1000 1000 181 …/DRILL-SENTINEL.txt`.
2. **rollback archive verified** — `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B,
**full zstd stream read OK** (4 867 573 760 B), sentinel confirmed inside it.
3. §3's option — §5 below.
4. **demo-felhom** `health=ok`, `escrow_state=escrowed`, untouched throughout.
5. **space** — nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
1. sentinel **listed by name** in snapshot `e6132ae5` (19:36:26), 181 B.
2. rollback archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` (1 606 765 083 B) **verified by a
full zstd stream read** (4 867 573 760 B) with the sentinel confirmed inside.
3. §3's option — §4 below.
4. `demo-felhom` `health=ok`, `escrow_state=escrowed`, untouched throughout.
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
**One precondition had drifted and was repaired, not worked around.** The staged snapshot no longer
held the sentinel: the afternoon's `mandatory data path missing` experiment produced a *later* same-day
calibre-web snapshot and `forget` had pruned the good one. The fixture on disk was correct, so one
backup re-established it and it was re-verified by listing. **Lesson: a good snapshot is not durable
against a later bad run on the same day.**
**One precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the
sentinel because the afternoon's experiment produced a *later* same-day snapshot and `forget
--keep-daily 7 --group-by host,tags` pruned the good one. One backup re-established it, re-verified by
listing. **A good snapshot is not durable against a later bad run on the same day.**
## 4. Every step's observable
## 4. §3 — the recovery code
| step | observable |
|---|---|
| 6 | fresh data dir stamped `20:00:2x`; **new `encryption.key`** (32 B); `claimed = None`; `offbox = null` |
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED across the wipe.** Nothing re-escrowed itself |
| **8** | **recovered `8a9e33aa4da6…`** — matches pre-wipe on-disk and the hub's record. Exit 1 is correct and designed: nothing local to compare against, the rebuilt-box shape |
| 9a | `[INSTALLED] … reads back identical` — the "installed" branch's first real run |
| 9b | after the Re-issue and apply, the on-disk key is **still** `8a9e33aa4da6…` — `WriteOffboxSecrets` kept it |
| 9c | **blocked** — see below |
| 10–11 | **not run** |
**The wipe was faithful to the incident, deliberately.** The 2026-08-03 rebuild R-193 is filed against
was **not** a guest reprovision — the journal shows guest 9201 running continuously through that window
with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the controller
and its data volume. A guest destroy plus an unrehearsed provisioning chain, improvised unattended, is
what §8.10 exists to prevent.
## 5. §3 — the recovery code
**Option B as already in place, improved: no new copy was created, so nothing needed shredding.** The
operator had placed `R_DEMO-HP` in their own `~/.config/credentials` (`0600`) two sessions ago for this
purpose. It was read from there and **piped to stdin** for the two invocations that needed it — never
an argument, never exported, never written to a second file, never logged. Destroying the operator's
own store would have destroyed their record; because no additional copy existed, there is nothing left
to prove gone.
**Option B as already in place, improved: no new copy was made, so nothing needed shredding.** The
operator's own `~/.config/credentials` (`0600`) was the source; the code was **piped to stdin** for
each invocation — never an argument, never exported, never a second file, never logged. Destroying the
operator's own store would have destroyed their record; because no extra copy existed, there is nothing
left to prove gone.
**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
`felhom-idesc-*` staging dirs — and the same sweep found a planted copy (**1**), then **0** after
shredding it. The instrument is shown sensitive rather than assumed to be.
`felhom-idesc-*` dirs — and the same sweep found a planted copy (**1**), then **0** after shredding it.
## 6. Part 2 — did not run
## 5. Every step's observable
Its gate is "the drill PASSED". It did not. A second wipe would have destroyed the state that makes the
first drill finishable in five minutes. **R-198's retention therefore remains unit-proven only** —
nothing has yet superseded a key in production.
| step | observable |
|---|---|
| 6 | fresh data dir `20:00:2x`; new `encryption.key`; `claimed = None`; `offbox = null` |
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED.** Nothing re-escrowed itself |
| **8** | **recovered `8a9e33aa4da6…`** — matches the pre-wipe on-disk key and the hub's record |
| 9a | `[INSTALLED] … reads back identical` |
| 9b | after Re-issue + apply, the on-disk key is **still** the recovered one |
| 9c | **repository OPENED** — `repo_state: null`, 3 snapshots, 42 026 B |
| **10** | **restored sha256 byte-identical** |
| 11 | this record |
Superseded escrow rows remained **2** throughout — **no ceremony was run at any point.**
## 6. Part 2 — not run
Gate: *"the drill PASSED; its evidence is written down; and there is time."* The first two are met; the
third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left
for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it.
**R-198's retention remains unit-proven only** — nothing has yet superseded a key in production, and it
is now the last unproven link in this chain.
## 7. Teardown — three layers
| layer | state |
|---|---|
| the guest | **nothing torn down** — the wipe is the evidence; all six app containers serving; the recovered key on disk; scratch band empty |
| the guest | **healthy and re-armed**: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is **kept as evidence**. Scratch band empty |
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
| the hub | **no new customer records** — `demo-hp` is the same row throughout, because the rebuild was a controller-data wipe and not a re-enrolment. One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran |
| the hub | **no new customer records** — `demo-hp` is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. **The two superseded escrow rows are unchanged** |
**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
still owed, deliberately not ridden along with a drill.
## 8. The capability-map rows
## 8. The capability-map row
A new row records the key as **PROVEN-LIVE after a real rebuild**, and states plainly what it does not
claim: **no file has been restored**, and the four links of R-204 stand between the recovered key and a
usable repository. The R-199 back-pointer added earlier today stands.
Now reads *"A customer's file survives a machine rebuild and comes back — the whole off-site story, end
to end"*, **PROVEN-LIVE**. What it deliberately does **not** claim: the journey is manual and
undocumented (R-204); the scope is `demo-hp` and a **controller-data** rebuild, **not** a total host
loss and **not** a guest reprovision; and R-198's retention is still unit-proven.
## 9. New findings
## 9. New findings — R-204, expanded into the gap list
- **R-204 (NEW)** — the recovered key cannot be used: a rebuilt box cannot configure its tier (R-193);
the Re-issue that fixes that marks the escrow stale though the key never changed (R-196, **measured
at `20:15:49` with `restic_pw_sha256` unchanged**); a stale escrow gates every run; and the only way
to clear it is a ceremony that **destroys the recovered key**. Plus a fourth link in no design
document: a rebuilt box is **unclaimed**, so the claim gate intercepts every controller endpoint.
- **R-196 re-scoped** — no longer a documentation nit; it is on the critical path for recovery.
- **R-201** — the wipe happened, the key came back, ~5 minutes from a verdict with a person present.
- **R-198** — still unit-proven only; Part 2 gated out.
- **R-202 open; the ciphertext deletion still owed.**
The four steps between a recovered key and a restored file, all measured while walking them:
1. **R-193** — a rebuilt controller cannot configure its off-site tier (`no unconsumed offsite
password`; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
2. **The claim gate** — a rebuilt box is unclaimed, so *every* controller endpoint is intercepted. And
**the local escape hatch does not work unaided**: `--print-reset-code` writes the new hash to
`settings.json` while the running controller keeps its old copy in memory, so `effectiveClaimCode()`
never sees it. **Restart the controller between minting and claiming** — two claim attempts failed
before this was diagnosed.
3. **R-196** — the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged, which
gates every off-site run. Cleared with the **manual** confirm, never a ceremony.
4. **`mode=unit` is the restore default** and returns the recovery unit, **not** the userdata leg. A
customer told to "restore from off-site" gets their app definition and not their documents, and
nothing in the outcome says so. **The worst of the four**, because it fails silently at the last
step.
## 10. CI
Docs push only. Run number and task id in the session summary. **`--no-verify` not used.**
Docs push only; run number and task id in the session summary. **`--no-verify` not used.**
## 11. Observations — noticed, NOT acted on
1. **The claim gate is an undocumented first step of every recovery.** Before a customer can do
anything on a rebuilt box — including recovering their backups — they must re-claim it.
2. **`--print-reset-code` output needs parsing care**: the captured value was 73 characters, i.e. more
than the code itself. Not chased; the claim was abandoned when the session stopped.
3. **A same-day re-run replaces the day's snapshot.** Worth knowing before designing any drill that
depends on a specific snapshot surviving.
4. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
`kubectl port-forward`. The `curl -u :$HUB_PW` recipe in memory omits that.
1. **A same-day re-run replaces the day's snapshot** (`forget --keep-daily 7 --group-by host,tags`).
Any drill depending on a specific snapshot surviving must account for it.
2. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
`kubectl port-forward`; the memory's `curl -u :$HUB_PW` recipe omits that.
3. **`--print-reset-code` mints a new generation on every invocation** — three wasted generations here
(4, 5, 6) while diagnosing the in-memory staleness.
4. **The restore wrote into `backups/offsite-restore/<app>/` mirroring the full absolute path** — deep
but unambiguous; worth knowing before writing customer-facing copy about where files land.