74 lines
4.9 KiB
Markdown
74 lines
4.9 KiB
Markdown
# RUNBOOK — Escrow ceremony (customer recovery-code R)
|
|
|
|
<!--
|
|
The zero-knowledge custody ceremony: seals the customer's unrecoverable secrets (PBS key K where
|
|
present; the IdentityBundle — restic offsite password, tunnel/PBS tokens, WG key) under a fresh
|
|
customer-held recovery code R. The hub stores only the opaque blob + a non-reversible password hash;
|
|
neither Felhom nor a hub compromise can open it. Cross-validated live: fork-4 validation (blob →
|
|
recover → decrypted the real repo) and the offsite arc (auto-confirm on hash match).
|
|
-->
|
|
|
|
## When a ceremony runs
|
|
1. **Onboarding** (with the PBS tier): seals K + the identity secrets.
|
|
2. **Offsite enable** (fork-4 rule): the restic repo password MUST enter a CURRENT R-blob before any
|
|
offsite backup runs — the controller stages the password to the agent at enable and the run-gate holds
|
|
`pending` until the hub-verified auto-confirm.
|
|
3. **Any new unrecoverable secret / suspected R exposure**: re-ceremony (R cannot be retro-fitted —
|
|
it is never retained).
|
|
|
|
## Prerequisites (check BEFORE scheduling with the customer)
|
|
- **Agent version:** ≥ v0.79.0 (the ceremony records `restic_pw_sha256` — older agents produce a blob
|
|
auto-confirm can never match). **No-PBS hosts** (BYO without the PBS tier): ≥ **v0.80.0**
|
|
(identity-only mode; below that the ceremony hard-requires the PBS key and refuses).
|
|
- **K gate (PBS hosts):** `escrow.pbs_storage_id` set and the key file present
|
|
(`cfg.Backup.PBSEncKeyPath(<id>)`).
|
|
- **Staged secret (offsite):** offsite enabled → `EscrowState="pending"` on the controller and the staged
|
|
password file present on the agent (`escrow-stage/restic_repo_password` — pushed automatically at
|
|
enable; re-saving the offsite config re-pushes it if missing).
|
|
- The customer is present/reachable: R is displayed ONCE and must land in their hands.
|
|
|
|
## The ceremony (ONE command, on the host, as the agent's operator context)
|
|
```bash
|
|
felhom-agent --selftest=escrow-create --upload
|
|
```
|
|
- `--storage <pbs-storage-id>` only if `escrow.pbs_storage_id` isn't configured. No-PBS hosts (agent
|
|
≥0.80.0): omit — identity-only engages automatically.
|
|
- What it does, in order: generates a fresh **R** (EFF-wordlist passphrase; entropy printed) → seals K
|
|
(if present, via proxmox-backup-client re-key) and the IdentityBundle (age-under-R; the staged restic
|
|
password auto-injected; the live WG key auto-captured if present) → **self-verifies by recovering its
|
|
own blob** ("an escrow you haven't recovered isn't an escrow") → uploads the opaque blob +
|
|
`restic_pw_sha256` to the hub → **wipes the staged secret**.
|
|
- Output hygiene: the ONLY secret ever displayed is R. Nothing else in the output is sensitive; the
|
|
command is safe to run over a screen-share EXCEPT the R moment.
|
|
|
|
## R handling — the rules to say out loud to the customer
|
|
1. R is displayed **once**; the system keeps no copy anywhere. Write it down / password manager /
|
|
safe — NOT on the server it protects.
|
|
2. **A new ceremony's R SUPERSEDES the old one.** Replace the stored code; the old blob remains valid
|
|
only for pre-existing history.
|
|
3. R + the hub blob = full recovery after total loss (see Recovery below). Losing R does not lose data
|
|
while the box lives — but after a disaster, an offsite copy without R is permanently sealed. That is
|
|
the zero-knowledge trade, stated honestly.
|
|
|
|
## After the ceremony — hands off (auto-confirm)
|
|
Within one report cycle the hub serves the blob's presence + hash in the ACK; the controller matches
|
|
`sha256(local repo password)` and flips `pending → escrowed` on its own, wiping the agent-staged copy.
|
|
**Verify remotely:** the customer page / report shows `escrow_state: "escrowed"` with zero manual clicks;
|
|
the next scheduled offsite run proceeds and snapshots appear. Hash-mismatch warning instead → the blob
|
|
does not cover the CURRENT password (stale blob) → re-run the ceremony; do NOT manual-confirm around it.
|
|
(The manual confirm endpoint exists solely as a deprecated fallback for pre-hash legacy blobs.)
|
|
|
|
## Recovery (the other half — operator-assisted, sketch)
|
|
Total loss → the customer's R + the hub blob:
|
|
`felhom-agent --selftest=identity-consume --blob <blob> --keydest <0600 scratch>` with R → recovers the
|
|
bundle (restic password, tokens; K restored via its own path on PBS hosts). Offsite data: hub re-issues
|
|
SFTP credentials (Re-issue button — access is regenerable; the ENCRYPTION key is what R guards) → inject
|
|
the recovered password on the fresh controller (`POST /backup/offbox/inject-password`, honored by the
|
|
secrets writer) → restore through the real endpoints. Live-proven end-to-end 2026-07-09 (recovered
|
|
password opened the production repo).
|
|
|
|
## Do NOT
|
|
- Run a ceremony unattended or pipe its output to a log (R would land in it).
|
|
- Manual-confirm a hash-mismatch; skip the self-verify; retain or transmit R on Felhom systems.
|
|
- Treat the SFTP credential as escrow-worthy (it's regenerable access; only the ENCRYPTION secrets ride R).
|