felhom.eu/REPORT-break-the-circle-2026-10-09.md

REPORT — break the circle: the password manager off-site, every recovery key listed, one sheet to print (2026-10-09)

Part	What	Result
A	Vaultwarden's data in the nightly off-site copy	DONE, LIVE (operator yes in chat) — run by hand, restore test, throwaway start
B	Recovery-key inventory + printable sheet + paper walk	DONE — runbooks/total-loss-of-dooplex.md, runbooks/break-glass-sheet.md
C	R-922 A, a quiet restored hub, R-921	BUILT, committed, NOT released (hub + controller); R-921 partly

Register: before 135 · after 138 · opened 2 (R-923, R-924) · closed 0. A parallel session added R-925 meanwhile.

Rulings recorded first: (1) the operator's finding → R-923 (P2); "off DooPlex" corrected in the hub-DB runbook (Step 0 and §3), gitea-restore.md, secrets.md, STATUS; (2) R-922 → option A, recorded in the row.

Part A — Vaultwarden (vaultwarden-system, 1.37.3, 7.4 MB)

• Method: Vaultwarden's own vaultwarden backup (SQLite VACUUM INTO, consistent while it runs; its backup guidance), plus rsa_key.pem and — when present — attachments/, sends/, config.json. The one backup file it writes into /data is copied out and removed (also on refusal). Same archive, key and write-only token as Gitea.
• Live: first run 2026-10-09T10:02Z: vaultwarden: integrity ok, 1 user(s), pushed in 13 s; /data clean after.
• Restore test (Sunday job, run by hand): 27 969 files match, 10 repos pass git fsck, Vaultwarden 1 user(s) / 797 item(s).
• Throwaway (bench 9401): row counts only — users 1, ciphers 797, folders 5, twofactor 0, sso_users 1; a Vaultwarden 1.37.3 started on the copy, /alive 200, no route out. Deleted (0 containers, 0 volumes, bench stopped); DooPlex scratch shredded. No login, no item opened, the master password never asked for.
• Tests: 26 (6 new), green with GNU, BusyBox and the fake sqlite3; 4 red-proofs (audits/dooplex-survival-2026-10-09/vaultwarden/). The tests found a defect before the live run (the optional-file listing failed when the last file was absent).
• Seen, not changed: Vaultwarden warns its ADMIN_TOKEN is plain text (in R-923).

Part B — what a total loss needs

18 secrets/logins listed with where each lives today. Exists only on DooPlex (lost with it): S1 the DooPlex off-site key, S2 the hub-DB key (otherwise only in Vaultwarden, which is behind S1), S6 the signing keys (in no backup at all, no passphrase — R-924), S7 the restore token (re-mintable). Circles: S8 Hetzner and S11 Cloudflare logins if kept only in Vaultwarden. Checked: the nightly Secrets export opens with the DooPlex restic passphrase (S5) and holds all eight Secrets the hub uses (names only read). Paper walk: with a filled sheet and the master password, no Felhom recovery step needs DooPlex; without it, S1, S6 and S8 block.

Part C — code (waits for the next release)

• Hub (d55c590c, e5b9151e, CI 1605 and 1606 success): R-922 A (email_cleared deletes the notification address; F12 guard kept); MAIL-HOLD (<data_dir>/MAIL-HOLD → no mail on any path, dropped not queued, banner, release button) — now step 1 of the hub restore runbook; two log lines no longer print the address; SECURITY: /preferences and /notify accepted any box's key for any household — now 403 (red-proved).
• Controller (ec9b997, CI 1607 success): R-922 sends email_cleared after a household's clear; R-921 pre-check — no stop while another tier's job is in flight. Not covered: the agent's host-wide busy lock (the case measured on demo-hp) — no agent endpoint serves it; needs an agent field (R-921 LEFT).
• Release order: hub first, then the controller (the controller sends a field an older hub ignores).
• Built by two helpers under this brief's fences; reviewed, tested again and committed here. One stash of mine briefly held a helper's uncommitted work (about a minute; nothing lost).

Teardown

Machine: bench 9401 — 0 containers, 0 volumes, image removed, stopped as it was. Host: DooPlex scratch dirs shredded. Hub: provisioned nothing. Vaultwarden: its one test backup file removed by name.

Done after the report: R-924 option A (operator, 2026-10-09)

The three signing key files and their public halves ride the nightly copy (felhom.eu 96f68f1b, CI 1614 success). Live run 12:57Z: signing: 3 key(s) with their public halves, pushed in 22 s; restore test: 3 signing key(s) match (audits/dooplex-survival-2026-10-09/signing/). 4 new tests, 3 red-proofs. R-924 narrowed: left is printing them.

Decisions for the operator (as asked; 1 is answered — A)

1. R-924 — the signing keys. (A) Print both on the sheet AND add them to the nightly encrypted copy (CC changes the DooPlex job with your yes). (B) Print only the recovery key and keep it away from home. Pick A. If you do nothing: after a loss of DooPlex no box accepts a signed update, bundle or OS step until it is re-enrolled on site.
2. Print the sheet (runbooks/break-glass-sheet.md, commands at its top, run from your workstation). If you do nothing: the off-site copies exist but cannot be opened after a loss of DooPlex.