felhom.eu/documentation/runbooks/RUNBOOK-escrow-ceremony.md

RUNBOOK — Escrow ceremony (customer recovery-code R)

When a ceremony runs

1. Onboarding (with the PBS tier): seals K + the identity secrets.
2. Offsite enable (fork-4 rule): the restic repo password MUST enter a CURRENT R-blob before any offsite backup runs — the controller stages the password to the agent at enable and the run-gate holds pending until the hub-verified auto-confirm.
3. Any new unrecoverable secret / suspected R exposure: re-ceremony (R cannot be retro-fitted — it is never retained).

Prerequisites (check BEFORE scheduling with the customer)

• Agent version: ≥ v0.79.0 (the ceremony records restic_pw_sha256 — older agents produce a blob auto-confirm can never match).
• ⚠ No-PBS hosts: the ceremony CANNOT run. The previously documented "identity-only mode (≥ v0.80.0)" was NEVER implemented — v0.80.0's actual feature was seeding escrow.pbs_storage_id on PBS hosts. escrow-create hard-requires a PBS storage id + its key file (drill-proven 2026-07-12, finding F-6 of DRILL-day0-vm-2026-07-12.md). Until identity-only ships, a box MUST have the PBS DR tier (which itself requires the WG tunnel peer first) before any escrow/offsite arc can complete.
• Host packages: age must be installed (identity wrap dependency; NOT installed by host-install as of v1.14.0 — drill finding F-10). apt-get install -y age.
• K gate (PBS hosts): escrow.pbs_storage_id set and the key file present (cfg.Backup.PBSEncKeyPath(<id>)).
• Staged secret (offsite): offsite enabled → EscrowState="pending" on the controller and the staged password file present on the agent (escrow-stage/restic_repo_password — pushed automatically at enable; re-saving the offsite config re-pushes it if missing).
• The customer is present/reachable: R is displayed ONCE and must land in their hands.

The ceremony (ONE command, on the host, as the agent's operator context)

felhom-agent --selftest=escrow-create --upload

• --storage <pbs-storage-id> only if escrow.pbs_storage_id isn't configured (the PBS DR apply-bridge seeds it automatically). No-PBS hosts: see the prerequisite warning above — the ceremony refuses without a PBS key; identity-only mode does not exist yet.
• What it does, in order: generates a fresh R (EFF-wordlist passphrase; entropy printed) → seals K (if present, via proxmox-backup-client re-key) and the IdentityBundle (age-under-R; the staged restic password auto-injected; the live WG key auto-captured if present) → self-verifies by recovering its own blob ("an escrow you haven't recovered isn't an escrow") → uploads the opaque blob + restic_pw_sha256 to the hub → wipes the staged secret.
• Output hygiene: the ONLY secret ever displayed is R. Nothing else in the output is sensitive; the command is safe to run over a screen-share EXCEPT the R moment.

R handling — the rules to say out loud to the customer

1. R is displayed once; the system keeps no copy anywhere. Write it down / password manager / safe — NOT on the server it protects.
2. A new ceremony's R SUPERSEDES the old one. Replace the stored code; the old blob remains valid only for pre-existing history.
3. R + the hub blob = full recovery after total loss (see Recovery below). Losing R does not lose data while the box lives — but after a disaster, an offsite copy without R is permanently sealed. That is the zero-knowledge trade, stated honestly.

After the ceremony — hands off (auto-confirm)

Within one report cycle the hub serves the blob's presence + hash in the ACK; the controller matches sha256(local repo password) and flips pending → escrowed on its own, wiping the agent-staged copy. Verify remotely: the customer page / report shows escrow_state: "escrowed" with zero manual clicks; the next scheduled offsite run proceeds and snapshots appear. Hash-mismatch warning instead → the blob does not cover the CURRENT password (stale blob) → re-run the ceremony; do NOT manual-confirm around it. (The manual confirm endpoint exists solely as a deprecated fallback for pre-hash legacy blobs.)

Recovery (the other half — operator-assisted, sketch)

Total loss → the customer's R + the hub blob: felhom-agent --selftest=identity-consume --blob <blob> --keydest <0600 scratch> with R → recovers the bundle (restic password, tokens; K restored via its own path on PBS hosts). Offsite data: hub re-issues SFTP credentials (Re-issue button — access is regenerable; the ENCRYPTION key is what R guards) → inject the recovered password on the fresh controller (POST /backup/offbox/inject-password, honored by the secrets writer) → restore through the real endpoints. Live-proven end-to-end 2026-07-09 (recovered password opened the production repo).

Do NOT

• Run a ceremony unattended or pipe its output to a log (R would land in it).
• Manual-confirm a hash-mismatch; skip the self-verify; retain or transmit R on Felhom systems.
• Treat the SFTP credential as escrow-worthy (it's regenerable access; only the ENCRYPTION secrets ride R).