88 lines
6.1 KiB
Markdown
88 lines
6.1 KiB
Markdown
# RUNBOOK — Escrow ceremony (customer recovery-code R)
|
|
|
|
<!--
|
|
The zero-knowledge custody ceremony: seals the customer's unrecoverable secrets (PBS key K where
|
|
present; the IdentityBundle — restic offsite password, tunnel/PBS tokens, WG key) under a fresh
|
|
customer-held recovery code R. The hub stores only the opaque blob + a non-reversible password hash;
|
|
neither Felhom nor a hub compromise can open it. Cross-validated live: fork-4 validation (blob →
|
|
recover → decrypted the real repo) and the offsite arc (auto-confirm on hash match).
|
|
-->
|
|
|
|
## When a ceremony runs
|
|
1. **Onboarding** (with the PBS tier): seals K + the identity secrets.
|
|
2. **Offsite enable** (fork-4 rule): the restic repo password MUST enter a CURRENT R-blob before any
|
|
offsite backup runs — the controller stages the password to the agent at enable and the run-gate holds
|
|
`pending` until the hub-verified auto-confirm.
|
|
3. **Any new unrecoverable secret / suspected R exposure**: re-ceremony (R cannot be retro-fitted —
|
|
it is never retained).
|
|
|
|
## Prerequisites (check BEFORE scheduling with the customer)
|
|
- **Agent version:** ≥ v0.79.0 (the ceremony records `restic_pw_sha256` — older agents produce a blob
|
|
auto-confirm can never match).
|
|
- **⚠ No-PBS hosts: the ceremony CANNOT run — and that is now POLICY, not a gap.** The previously
|
|
documented "identity-only mode (≥ v0.80.0)" was NEVER implemented — v0.80.0's actual feature was
|
|
seeding `escrow.pbs_storage_id` on PBS hosts. `escrow-create` hard-requires a PBS storage id +
|
|
its key file (drill-proven 2026-07-12, finding F-6 of DRILL-day0-vm-2026-07-12.md).
|
|
**Operator decision 2026-07-12 (DR-tier-by-default, decision 4): identity-only escrow is
|
|
formally PARKED — F-6 is closed by policy, not by implementation.** Every install ships the
|
|
full DR plumbing (host-install ≥ v1.15.0); activation is the hub's per-customer DR-tier flag
|
|
(default ON), and offsite backup REQUIRES that flag (hub v0.51.0 enforces the coupling:
|
|
"Offsite backup requires the DR tier"). A box therefore always reaches the ceremony THROUGH
|
|
the tier: WG peer → PBS-DR descriptor applied → ceremony. Revisit identity-only ONLY if a
|
|
customer class demands lean-tier offsite — this line is here so a future session knows the
|
|
absence is a decision, not an oversight.
|
|
- **Host packages:** `age` must be installed (identity wrap dependency; installed by host-install
|
|
since v1.15.0 — drill finding F-10 closed. Pre-v1.15.0 boxes: `apt-get install -y age`).
|
|
- **K gate (PBS hosts):** `escrow.pbs_storage_id` set and the key file present
|
|
(`cfg.Backup.PBSEncKeyPath(<id>)`).
|
|
- **Staged secret (offsite):** offsite enabled → `EscrowState="pending"` on the controller and the staged
|
|
password file present on the agent (`escrow-stage/restic_repo_password` — pushed automatically at
|
|
enable; re-saving the offsite config re-pushes it if missing).
|
|
- The customer is present/reachable: R is displayed ONCE and must land in their hands.
|
|
|
|
## The ceremony (ONE command, on the host, as the agent's operator context)
|
|
```bash
|
|
felhom-agent --selftest=escrow-create --upload
|
|
```
|
|
- `--storage <pbs-storage-id>` only if `escrow.pbs_storage_id` isn't configured (the PBS DR
|
|
apply-bridge seeds it automatically). No-PBS hosts: see the prerequisite warning above — the
|
|
ceremony refuses without a PBS key; identity-only mode does not exist yet.
|
|
- What it does, in order: generates a fresh **R** (EFF-wordlist passphrase; entropy printed) → seals K
|
|
(if present, via proxmox-backup-client re-key) and the IdentityBundle (age-under-R; the staged restic
|
|
password auto-injected; the live WG key auto-captured if present) → **self-verifies by recovering its
|
|
own blob** ("an escrow you haven't recovered isn't an escrow") → uploads the opaque blob +
|
|
`restic_pw_sha256` to the hub → **wipes the staged secret**.
|
|
- Output hygiene: the ONLY secret ever displayed is R. Nothing else in the output is sensitive; the
|
|
command is safe to run over a screen-share EXCEPT the R moment.
|
|
|
|
## R handling — the rules to say out loud to the customer
|
|
1. R is displayed **once**; the system keeps no copy anywhere. Write it down / password manager /
|
|
safe — NOT on the server it protects.
|
|
2. **A new ceremony's R SUPERSEDES the old one.** Replace the stored code; the old blob remains valid
|
|
only for pre-existing history.
|
|
3. R + the hub blob = full recovery after total loss (see Recovery below). Losing R does not lose data
|
|
while the box lives — but after a disaster, an offsite copy without R is permanently sealed. That is
|
|
the zero-knowledge trade, stated honestly.
|
|
|
|
## After the ceremony — hands off (auto-confirm)
|
|
Within one report cycle the hub serves the blob's presence + hash in the ACK; the controller matches
|
|
`sha256(local repo password)` and flips `pending → escrowed` on its own, wiping the agent-staged copy.
|
|
**Verify remotely:** the customer page / report shows `escrow_state: "escrowed"` with zero manual clicks;
|
|
the next scheduled offsite run proceeds and snapshots appear. Hash-mismatch warning instead → the blob
|
|
does not cover the CURRENT password (stale blob) → re-run the ceremony; do NOT manual-confirm around it.
|
|
(The manual confirm endpoint exists solely as a deprecated fallback for pre-hash legacy blobs.)
|
|
|
|
## Recovery (the other half — operator-assisted, sketch)
|
|
Total loss → the customer's R + the hub blob:
|
|
`felhom-agent --selftest=identity-consume --blob <blob> --keydest <0600 scratch>` with R → recovers the
|
|
bundle (restic password, tokens; K restored via its own path on PBS hosts). Offsite data: hub re-issues
|
|
SFTP credentials (Re-issue button — access is regenerable; the ENCRYPTION key is what R guards) → inject
|
|
the recovered password on the fresh controller (`POST /backup/offbox/inject-password`, honored by the
|
|
secrets writer) → restore through the real endpoints. Live-proven end-to-end 2026-07-09 (recovered
|
|
password opened the production repo).
|
|
|
|
## Do NOT
|
|
- Run a ceremony unattended or pipe its output to a log (R would land in it).
|
|
- Manual-confirm a hash-mismatch; skip the self-verify; retain or transmit R on Felhom systems.
|
|
- Treat the SFTP credential as escrow-worthy (it's regenerable access; only the ENCRYPTION secrets ride R).
|