14 KiB
DRILL — R-201, the night run: PASS. The customer's file came back, byte-identical.
Date: 2026-08-04, 21:30–23:15 · Box: demo-hp (Tier 0) · unattended, then finished with the
operator present. The box is up, healthy, and fully re-armed.
THE VERDICT: PASS.
pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7cA machine was destroyed on purpose — controller data volume gone, the customer's file deleted from disk — and the file came back byte-identical, through the customer's own recovery code and the customer's own restore flow. This is the first time the Felhom backup story has been proved end to end.
The key recovered byte-identical (
8a9e33aa4da6…), the existing repository opened rather than being re-initialised (repo_state: null, 3 snapshots, 42 026 B — the pre-wipe size exactly), and the pre-wipe snapshote6132ae5was still there with the file inside it.It took four undocumented steps to get there, each measured, and they are the other half of the result — see §3. A customer hitting this alone today would not get their data back.
1. The verdict — PASS
| sentinel sha256, pre-wipe (step 3) | 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c |
| sentinel sha256, restored (step 10) | 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c |
| match | YES — byte-identical |
| snapshot count at step 9 | 3 — the pre-wipe count. NOT 1. No fresh history was started |
| repo size at step 9 | 42 026 B — the pre-wipe size exactly |
repo_state |
null — the repository OPENED. Not orphaned |
The restore ran through the customer flow (POST /backup/offbox/restore, mode=full, two-step
confirm), out of snapshot e6132ae5 — the one taken before the wipe.
A detail worth keeping: the first restore attempt used the default mode=unit, which recovers the
recovery unit only and not the userdata leg. mode=full is what returns the customer's files. A
customer told to "restore from off-site" would, on the default, get their app definition back and not
their documents — and nothing in that outcome says so.
2. What was proven, in order, on hardware
Step 6 — the wipe
The controller's data volume was destroyed and the bootstrap re-ran. Fresh data directory, everything
stamped 2026-08-04 20:00:2x:
encryption.key 32 B 2026-08-04 20:00:21 ← brand new: every pre-wipe app secret is undecryptable
claimed = None ← the fresh-install signal
offbox = null ← no off-site target
Faithful to the incident, and deliberately so. The 2026-08-03 rebuild that R-193 is filed against
was not a guest reprovision — the journal shows guest 9201 running continuously through that
window, with no pct destroy, no pct restore and no --selftest=provision. What changed was the
controller and its data volume. Reproducing that is what the drill needs; a guest destroy plus an
unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.
Step 7 — the assertion that keeps recovery possible: PASSED
host_escrow (demo-hp-bb76ea), AFTER the wipe:
identity_blob = 572 bytes ← unchanged
restic_pw_sha256 = 8a9e33aa4da6… ← unchanged
updated_at = 2026-08-04 11:11:37 ← unchanged; nothing re-escrowed
stale_at = NULL
No ceremony was run and nothing re-escrowed itself. The sealed key survived the rebuild untouched.
Step 8 — THE KEY CAME BACK
=== offsite key recovery check (R-200) — compares, never installs ===
recovered sha256: 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a
[FAIL] there is no repository password on this box to compare against
(the recovery itself SUCCEEDED — this box simply has no local key. That is the
rebuilt-box shape, where the next step is to INSTALL rather than compare.)
Exit 1 is correct and designed: there was nothing to compare against, because the wipe removed it. The recovered hash matches the pre-wipe on-disk key and the hub's own record — three independent sources agreeing, one of them recovered through the full chain (hub → agent → unseal → extract) on a box that had just lost everything it knew.
Step 9a — installed cleanly
=== offsite key recovery INSTALL (R-200) ===
on-disk sha256: (none — this box has no repository password)
recovered sha256: 8a9e33aa4da6…
[INSTALLED] the recovered repository password is in place and reads back identical.
The "installed" branch — the rebuilt-box shape v0.196.0 was written for — took its first real run.
Step 9b — the apply kept it
After the target was reconfigured, the on-disk key was still 8a9e33aa4da6…. WriteOffboxSecrets
found the file present and kept it, exactly as documented.
3. The wall — three blockers, each measured
(a) R-193, reconfirmed live: a rebuilt controller cannot configure its off-site tier
[INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.197.0)
[WARN] [offsite-apply] reconcile: offsite-apply: consume one-time password:
no unconsumed offsite password (already consumed or none provisioned)
(retries on next config refresh/restart)
The ledger, measured: demo-hp one-time secret created 07:11:51, consumed 07:12:06 — by the
previous controller. The rebuilt one has nothing to consume and retries forever. Exactly the state
demo-hp sat in for 25 hours on 2026-08-03.
Remedy: an operator Re-issue. Performed here through the designed endpoint
(POST /configs/demo-hp/offsite-reissue, HTTP 303), after which the target configured normally.
(b) R-196, measured live — and it lands squarely on the recovery path
host_escrow (demo-hp-bb76ea), after the Re-issue:
stale_at = 2026-08-04 20:15:49 ← set
restic_pw_sha256 = 8a9e33aa4da6… ← UNCHANGED. The key did not move.
The escrow was marked stale while it perfectly covers the box's current key — the recovered one.
That is R-196's false staleness, and here it is not a cosmetic lie: a stale escrow makes the hub
withhold the hash from the report ACK, so EscrowAutoConfirmer can never flip pending → escrowed, and OffboxRunnable (configured && escrowed) refuses to run any off-site backup.
The remedy for (a) disables the recovery it was needed for. The only documented way to clear a stale escrow is a fresh ceremony — which supersedes the identity blob, i.e. destroys the very key being recovered. Under hub v0.93.0 the old blob is now retained, but nothing can serve a superseded blob back (R-199's inventory: the endpoint serves the CURRENT row only).
Superseded rows still number 2 — no ceremony was run tonight. The key is intact.
(c) The claim gate — undocumented as a recovery step
[DEBUG] [web] claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)
A rebuilt box is unclaimed (claimed = None, fresh settings.json), and the claim gate correctly
intercepts every non-claim route. So no controller endpoint can be driven at all — not the manual
escrow confirm, not the backup trigger — until the customer re-claims the box. Both POSTs returned 302
to the claim page and neither reached its handler; last_run stayed null, which is why the earlier
reading of "the run was refused" needed this second look to be accurate.
This is correct behaviour and it is not wrong — but it is a step in the customer's recovery journey that appears in no design document, and it comes before anything else can happen.
4. How it was finished, and the fourth undocumented step
The unattended run stopped at the wall in §3 rather than improvising further at 22:35 (§8.10). With the operator present the remaining path was walked, and it took four steps a customer would have to discover:
- Re-issue the offsite credential (hub, operator-only) — otherwise the box can never configure its tier. §3(a).
- Re-claim the box. The local escape hatch (
--print-reset-code) mints a code — and does not work on its own: it writes the new hash tosettings.jsonwhile the RUNNING controller keeps its old copy in memory, soeffectiveClaimCode()never sees it and every attempt returns "Hibás vagy lejárt kód". The controller must be restarted between minting and claiming. Two claim attempts failed on this before it was diagnosed; the generation counter reached 6. - Confirm the escrow manually (
/backup/offbox/confirm-escrow) →escrowed, soOffboxRunnablepermits a run. Not a ceremony — the identity blob was never superseded (still 2 retained rows). - Restore with
mode=full. The defaultmode=unitreturns the recovery unit and not the customer's files.
None of these four is in any design document, and the drill needed all four.
5. §5's five conditions, as recorded before the wipe
| # | Condition | Evidence |
|---|---|---|
| 1 | §4 confirmed, sentinel listed by name | snapshot e6132ae5 (19:36:26) — -rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt |
| 2 | deliberate rollback archive, verified | vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst, 1 606 765 083 B; full zstd stream read OK (4 867 573 760 B uncompressed); the sentinel confirmed inside it |
| 3 | §3's option | see §6 |
| 4 | demo-felhom untouched | health=ok, escrow_state=escrowed, not touched at any point |
| 5 | free space | nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 % |
A precondition had drifted and was repaired before the wipe, not worked around. The staged
snapshot no longer contained the sentinel: the afternoon's [WARN] mandatory data path missing
experiment produced a later same-day calibre-web snapshot, and restic's forget --keep-daily 7 --group-by host,tags had pruned the good one in its favour. The fixture on disk was correct, so one
backup re-established it and it was re-verified by listing. Lesson worth carrying: a good snapshot is
not durable against a later bad run on the same day.
6. §3 — the recovery code
Option B as already in place, with a strict improvement: no new copy was created, so nothing needed shredding.
The operator placed R_DEMO-HP in their own ~/.config/credentials on DooPlex (mode 0600) two
sessions ago, deliberately, for this purpose. This session read it from there and piped it to stdin
for each of the two invocations that needed it. It was never an argument, never exported, never written
to a second file, and never logged.
Nothing was shredded, and that is the point: the operator's own permanent store is theirs, not a drill artefact, and destroying it would have destroyed their record. Because no additional copy was made, there is nothing left behind to prove gone — a stronger position than option B's create-then-shred.
Verified afterwards with the planted-copy positive control, exactly as on 2026-08-04 afternoon — see §8.
7. The state the box is left in — healthy and re-armed
controller felhom-controller:0.197.0, healthy, CLAIMED
apps privatebin opengist calibre-web filebrowser cloudflared traefik — all serving
escrow_state escrowed (no ceremony was run; identity_blob untouched)
repo key 8a9e33aa4da6… THE RECOVERED KEY — and the repository opens with it
off-site 3 snapshots, 42 026 B, repo_state null; all three apps re-toggled for off-site
sentinel restored to its live location, sha256 643166269103a25c… — verified
The verification copy under backups/offsite-restore/calibre-web/ is left in place as evidence.
8. R persisted nowhere — searched, with a positive control
Swept the agent journal, the controller container log, and /tmp, /var/tmp, /var/lib/felhom-agent,
/root on demo-hp for the recovery code: 0 hits, and 0 leftover felhom-idesc-* unseal
staging directories. A planted copy was found by the same sweep (1) and not found after shredding
(0), so the instrument is shown sensitive rather than assumed to be.
9. Teardown — three layers
| layer | state |
|---|---|
| the guest | Nothing torn down. The wipe is the evidence; the apps are serving; the recovered key is in place. The scratch band is empty — no restore-test guest was created tonight. |
| the host | vzdump snapshot LVs removed cleanly by the archive job (snap_vm-9201-disk-0/1_vzdump both released). One new archive, 1.6 GB, on nvme-1tb (883 GB free). Nothing deleted. |
| the hub | No new customer records. demo-hp is the same customer row throughout — the rebuild was a controller-data wipe, not a re-enrolment, so nothing accumulated. One Re-issue staged a fresh one-time secret (consumed at 20:16) and set stale_at. The two superseded escrow rows are unchanged — no ceremony ran. |
Nothing was deleted on the storage endpoint, including the ~1.2 GB of previously-orphaned ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.
10. Part 2 — not run
Its gate is "the drill PASSED; its evidence is written down; and there is time." The first two are now met. The third was not — it was 23:15 after a full destructive cycle, and Part 2 is a second wipe. It is deliberately left for a session that starts fresh, with drill 1's result already recorded so a second wipe cannot overwrite it.
R-198's retention therefore remains unit-proven only. Nothing has yet superseded a key in
production, so the retention's live behaviour is still unobserved — and it is now the last unproven
link in the chain this drill exercised. The check is cheap: wipe again, do NOT recover, let a ceremony
seal a different password, then read the superseded row's identity_blob.