felhom.eu/documentation/runbooks/RUNBOOK-escrow-ceremony.md

# RUNBOOK — Escrow ceremony (customer recovery-code R)

<!--
  The zero-knowledge custody ceremony: seals the customer's unrecoverable secrets (PBS key K where
  present; the IdentityBundle — restic offsite password, tunnel/PBS tokens, WG key) under a fresh
  customer-held recovery code R. The hub stores only the opaque blob + a non-reversible password hash;
  neither Felhom nor a hub compromise can open it. Cross-validated live: fork-4 validation (blob →
  recover → decrypted the real repo) and the offsite arc (auto-confirm on hash match).
-->

## When a ceremony runs
1. **Onboarding** (with the PBS tier): seals K + the identity secrets.
2. **Offsite enable** (fork-4 rule): the restic repo password MUST enter a CURRENT R-blob before any
   offsite backup runs — the controller stages the password to the agent at enable and the run-gate holds
   `pending` until the hub-verified auto-confirm.
3. **Any new unrecoverable secret / suspected R exposure**: re-ceremony (R cannot be retro-fitted —
   it is never retained).

## Prerequisites (check BEFORE scheduling with the customer)
- **Agent version:** ≥ v0.79.0 (the ceremony records `restic_pw_sha256` — older agents produce a blob
  auto-confirm can never match).
- **⚠ No-PBS hosts: the ceremony CANNOT run.** The previously documented "identity-only mode
  (≥ v0.80.0)" was NEVER implemented — v0.80.0's actual feature was seeding `escrow.pbs_storage_id`
  on PBS hosts. `escrow-create` hard-requires a PBS storage id + its key file (drill-proven
  2026-07-12, finding F-6 of DRILL-day0-vm-2026-07-12.md). Until identity-only ships, a box MUST
  have the PBS DR tier (which itself requires the WG tunnel peer first) before any escrow/offsite
  arc can complete.
- **Host packages:** `age` must be installed (identity wrap dependency; NOT installed by
  host-install as of v1.14.0 — drill finding F-10). `apt-get install -y age`.
- **K gate (PBS hosts):** `escrow.pbs_storage_id` set and the key file present
  (`cfg.Backup.PBSEncKeyPath(<id>)`).
- **Staged secret (offsite):** offsite enabled → `EscrowState="pending"` on the controller and the staged
  password file present on the agent (`escrow-stage/restic_repo_password` — pushed automatically at
  enable; re-saving the offsite config re-pushes it if missing).
- The customer is present/reachable: R is displayed ONCE and must land in their hands.

## The ceremony (ONE command, on the host, as the agent's operator context)
```bash
felhom-agent --selftest=escrow-create --upload
```
- `--storage <pbs-storage-id>` only if `escrow.pbs_storage_id` isn't configured (the PBS DR
  apply-bridge seeds it automatically). No-PBS hosts: see the prerequisite warning above — the
  ceremony refuses without a PBS key; identity-only mode does not exist yet.
- What it does, in order: generates a fresh **R** (EFF-wordlist passphrase; entropy printed) → seals K
  (if present, via proxmox-backup-client re-key) and the IdentityBundle (age-under-R; the staged restic
  password auto-injected; the live WG key auto-captured if present) → **self-verifies by recovering its
  own blob** ("an escrow you haven't recovered isn't an escrow") → uploads the opaque blob +
  `restic_pw_sha256` to the hub → **wipes the staged secret**.
- Output hygiene: the ONLY secret ever displayed is R. Nothing else in the output is sensitive; the
  command is safe to run over a screen-share EXCEPT the R moment.

## R handling — the rules to say out loud to the customer
1. R is displayed **once**; the system keeps no copy anywhere. Write it down / password manager /
   safe — NOT on the server it protects.
2. **A new ceremony's R SUPERSEDES the old one.** Replace the stored code; the old blob remains valid
   only for pre-existing history.
3. R + the hub blob = full recovery after total loss (see Recovery below). Losing R does not lose data
   while the box lives — but after a disaster, an offsite copy without R is permanently sealed. That is
   the zero-knowledge trade, stated honestly.

## After the ceremony — hands off (auto-confirm)
Within one report cycle the hub serves the blob's presence + hash in the ACK; the controller matches
`sha256(local repo password)` and flips `pending → escrowed` on its own, wiping the agent-staged copy.
**Verify remotely:** the customer page / report shows `escrow_state: "escrowed"` with zero manual clicks;
the next scheduled offsite run proceeds and snapshots appear. Hash-mismatch warning instead → the blob
does not cover the CURRENT password (stale blob) → re-run the ceremony; do NOT manual-confirm around it.
(The manual confirm endpoint exists solely as a deprecated fallback for pre-hash legacy blobs.)

## Recovery (the other half — operator-assisted, sketch)
Total loss → the customer's R + the hub blob:
`felhom-agent --selftest=identity-consume --blob <blob> --keydest <0600 scratch>` with R → recovers the
bundle (restic password, tokens; K restored via its own path on PBS hosts). Offsite data: hub re-issues
SFTP credentials (Re-issue button — access is regenerable; the ENCRYPTION key is what R guards) → inject
the recovered password on the fresh controller (`POST /backup/offbox/inject-password`, honored by the
secrets writer) → restore through the real endpoints. Live-proven end-to-end 2026-07-09 (recovered
password opened the production repo).

## Do NOT
- Run a ceremony unattended or pipe its output to a log (R would land in it).
- Manual-confirm a hash-mismatch; skip the self-verify; retain or transmit R on Felhom systems.
- Treat the SFTP credential as escrow-worthy (it's regenerable access; only the ENCRYPTION secrets ride R).