felhom.eu/documentation/runbooks/break-glass-sheet.md

Break-glass sheet — the keys that must exist outside DooPlex (R-923)

A template. It holds NO key value, and none may ever be written into this file, a commit, a log or a chat. Print this page, then write or stick each value onto the paper copy only. Keep the paper away from home (DooPlex is at home: a fire takes both). Why each key is here and what it opens: total-loss-of-dooplex.md.

A key in the password manager is not enough: Vaultwarden runs on DooPlex (operator ruling, 2026-10-09).

How to print a key without it landing anywhere

Run these from your own workstation (not through Claude Code, not through !). Each command writes one file on the workstation; open it, print it, then destroy the file. Nothing is stored on DooPlex or in any log.

D=kisfenyo@192.168.0.180        # DooPlex
# S1 and S2 — PBS keys: Proxmox's own paper form (text + QR code)
ssh -t $D 'sudo proxmox-backup-client key paperkey /etc/felhom-dooplex-offsite/enc.key --output-format html --subject "S1 DooPlex off-site key"' > s1.html
ssh -t $D 'sudo proxmox-backup-client key paperkey /etc/felhom-hub-backup/enc.key --output-format html --subject "S2 Hub-DB off-site key"'  > s2.html
# S4, S5, S7 — one line each
ssh -t $D 'sudo kubectl -n felhom-system get secret offsite-secret-key -o jsonpath="{.data.OFFSITE_SECRET_KEY}" | base64 -d' > s4.txt
ssh -t $D 'sudo cat /etc/backup/restic-password'            > s5.txt
ssh -t $D 'sudo cat /etc/felhom-hub-backup/token-restore'    > s7.txt
# S6 — the signing keys (OpenSSH text, ~7 lines each)
ssh $D 'cat /mnt/5_hdd/felhom.eu/felhom-rec-recovery'     > s6-recovery.txt
ssh $D 'cat /mnt/5_hdd/felhom.eu/felhom-op-operational'   > s6-operational.txt
# open each, print, check the print is readable, then:
shred -u s1.html s2.html s4.txt s5.txt s7.txt s6-recovery.txt s6-operational.txt

On Windows without shred: delete the files and empty the recycle bin. The -t adds a carriage return to the captured line on some systems; strip it when typing the value back.

Check a print later without exposing it: for S1/S2 the only reliable check is one restore with a key file rebuilt from the printed data field (hub-DB runbook Step 0 note) — key show cannot check a rebuilt key.

---

The sheet (print from here)

FELHOM — break-glass keys. Printed on: ____________ Stored at: ______________________________

#	Key	Public fingerprint (to match the right key)	Value — write or stick here
S1	DooPlex off-site key — opens Gitea, the password manager, the k8s Secrets export (ep0 operator, host/dooplex-gitea)	PBS key 93:03:bf:d7:1f:4c:9e:fe…	data: ______________________________________________
S2	Hub-DB off-site key — opens the hub database (ep0 operator, host/dooplex-hub)	PBS key b2:19:bf:36:3b:97:3d:6c…	data: ______________________________________________
S4	Hub seal key OFFSITE_SECRET_KEY (64 hex)	—	______________________________________________________
S5	DooPlex restic / Secrets-export passphrase	—	______________________________________________________
S6a	Signing RECOVERY key felhom-rec-recovery (also in the S1 copy since 2026-10-09; the paper is the copy that needs nothing else)	SHA256:/ixgTesZqykAGJpFUUd4kLAiHFgKOkYFLNC3AQXWP+k	(staple the printout)
S6b	Signing OPERATIONAL key felhom-op-operational	SHA256:7YqN4rXO08yixTeOO+UtQ8jHyIGycICuctQgRYVGnWw	(staple the printout)
S7	ep0 read-only token dooplex-hub@pbs!restore	—	______________________________________________________
S8	Hetzner account: login e-mail · password · two-factor recovery codes	—	______________________________________________________
S11	Cloudflare account: login · password · two-factor recovery codes	—	______________________________________________________
S12	Gmail felhom.eu@gmail.com: password · two-factor recovery codes	—	______________________________________________________
S3	Vaultwarden master password — in your head; write it here only if you decide to	—	______________________________________________________

Not secret, needed with the keys:

• ep0: 167.233.158.164 (Hetzner, felhom-hetzner); PBS datastore felhom-offsite, namespace operator; its certificate fingerprint c6:07:28:3f:5b:7b:5a:41:90:28:d7:ca:4f:37:14:70:56:39:2e:2f:0b:71:e8:06:ca:60:4a:d5:56:5f:3c:fd.
• Restore order: total-loss-of-dooplex.md (in the restored Gitea, repo felhom.eu, documentation/runbooks/). Print that page too — the runbook itself is inside the copy it explains how to open.

Re-print when a key above is rotated, and once a year.