RUNBOOK — Escrow ceremony (customer recovery-code R)

When a ceremony runs

Onboarding (with the PBS tier): seals K + the identity secrets.
Offsite enable (fork-4 rule): the restic repo password MUST enter a CURRENT R-blob before any offsite backup runs — the controller stages the password to the agent at enable and the run-gate holds pending until the hub-verified auto-confirm.
Any new unrecoverable secret / suspected R exposure: re-ceremony (R cannot be retro-fitted — it is never retained).

Prerequisites (check BEFORE scheduling with the customer)

Agent version: ≥ v0.79.0 (the ceremony records restic_pw_sha256 — older agents produce a blob auto-confirm can never match). No-PBS hosts (BYO without the PBS tier): ≥ v0.80.0 (identity-only mode; below that the ceremony hard-requires the PBS key and refuses).
K gate (PBS hosts): escrow.pbs_storage_id set and the key file present (cfg.Backup.PBSEncKeyPath(<id>)).
Staged secret (offsite): offsite enabled → EscrowState="pending" on the controller and the staged password file present on the agent (escrow-stage/restic_repo_password — pushed automatically at enable; re-saving the offsite config re-pushes it if missing).
The customer is present/reachable: R is displayed ONCE and must land in their hands.

The ceremony (ONE command, on the host, as the agent's operator context)

felhom-agent --selftest=escrow-create --upload

--storage <pbs-storage-id> only if escrow.pbs_storage_id isn't configured. No-PBS hosts (agent ≥0.80.0): omit — identity-only engages automatically.
What it does, in order: generates a fresh R (EFF-wordlist passphrase; entropy printed) → seals K (if present, via proxmox-backup-client re-key) and the IdentityBundle (age-under-R; the staged restic password auto-injected; the live WG key auto-captured if present) → self-verifies by recovering its own blob ("an escrow you haven't recovered isn't an escrow") → uploads the opaque blob + restic_pw_sha256 to the hub → wipes the staged secret.
Output hygiene: the ONLY secret ever displayed is R. Nothing else in the output is sensitive; the command is safe to run over a screen-share EXCEPT the R moment.

R handling — the rules to say out loud to the customer

R is displayed once; the system keeps no copy anywhere. Write it down / password manager / safe — NOT on the server it protects.
A new ceremony's R SUPERSEDES the old one. Replace the stored code; the old blob remains valid only for pre-existing history.
R + the hub blob = full recovery after total loss (see Recovery below). Losing R does not lose data while the box lives — but after a disaster, an offsite copy without R is permanently sealed. That is the zero-knowledge trade, stated honestly.

After the ceremony — hands off (auto-confirm)

Within one report cycle the hub serves the blob's presence + hash in the ACK; the controller matches sha256(local repo password) and flips pending → escrowed on its own, wiping the agent-staged copy. Verify remotely: the customer page / report shows escrow_state: "escrowed" with zero manual clicks; the next scheduled offsite run proceeds and snapshots appear. Hash-mismatch warning instead → the blob does not cover the CURRENT password (stale blob) → re-run the ceremony; do NOT manual-confirm around it. (The manual confirm endpoint exists solely as a deprecated fallback for pre-hash legacy blobs.)

Recovery (the other half — operator-assisted, sketch)

Total loss → the customer's R + the hub blob: felhom-agent --selftest=identity-consume --blob <blob> --keydest <0600 scratch> with R → recovers the bundle (restic password, tokens; K restored via its own path on PBS hosts). Offsite data: hub re-issues SFTP credentials (Re-issue button — access is regenerable; the ENCRYPTION key is what R guards) → inject the recovered password on the fresh controller (POST /backup/offbox/inject-password, honored by the secrets writer) → restore through the real endpoints. Live-proven end-to-end 2026-07-09 (recovered password opened the production repo).

Do NOT

Run a ceremony unattended or pipe its output to a log (R would land in it).
Manual-confirm a hash-mismatch; skip the self-verify; retain or transmit R on Felhom systems.
Treat the SFTP credential as escrow-worthy (it's regenerable access; only the ENCRYPTION secrets ride R).

4.9 KiB Raw Blame History