14 KiB
Runbook — total loss of DooPlex: bring the business back on new hardware (R-232, R-923)
Written 2026-10-09. Not walked end to end. Its parts were walked separately on throwaways: Gitea (
gitea-restore.md), the hub (RUNBOOK-hub-db-offsite-backup.md§3), Vaultwarden (rows + start,audits/dooplex-survival-2026-10-09/vaultwarden/). This page names secrets, never values. The printable page that holds the few values that must exist outside DooPlex isbreak-glass-sheet.md.
What a total loss means, and what survives
DooPlex (192.168.0.180, the home server) holds: Gitea (all code + the container registry), the hub (k3s), Vaultwarden (the operator's password manager), DooPlex's own PBS, every local backup, the signing keys, the CI runner, the build tree. Assume all of it is gone, and the house with it.
| Survives (off DooPlex) | What it holds | Opened by |
|---|---|---|
ep0 (Hetzner, 167.233.158.164), PBS felhom-offsite, namespace operator |
host/dooplex-hub — the hub database, nightly; host/dooplex-gitea — Gitea (repos, DB dump, app.ini), Vaultwarden (DB + rsa_key.pem), DooPlex's nightly k8s Secrets export (GPG), nightly |
S1 (+ S2 for the hub DB), a PBS token or ep0 root |
| ep0, the households' namespaces | each household's whole-box copies | each household's own key (escrow) — not the operator's |
| Hetzner Storage Boxes | each household's restic copy | each household's own password (in the box, and in its whole-box copies) |
| The boxes themselves | they keep running and backing up; they cannot report, update or mail until the hub is back | — |
| The operator's workstation (if it has them) | the WireGuard peer to ep0 (10.77.0.250), maybe one of ep0's two root SSH keys, maybe a Bitwarden client's offline vault cache |
the operator |
Lost with DooPlex and NOT in any copy (measured 2026-10-09): (the container registry was here until 2026-10-23: the released images and agent packages now ride a nightly copy to ep0 — D33, „Restore the release copy" below; the rest of the registry — older tags, rc builds, non-Felhom images — is still lost); (the operator signing keys were here until 2026-10-09 afternoon; they now ride the nightly copy — R-924); DooPlex's own SSH key (one of ep0's two root keys);
the build tree and drills (/mnt/5_hdd/felhom.eu/{build,drills}); .claude-memory (same-disk restic only); the
local restic repos; Prometheus history; DooPlex's other homelab apps (not Felhom; Longhorn on DooPlex only).
The secrets a full recovery needs — where each lives today
S = must exist outside DooPlex (on the sheet). "DooPlex only" = a total loss loses it unless the sheet carries it. "Vaultwarden" = the password manager on DooPlex — a key kept only there is not off DooPlex (operator ruling, 2026-10-09); it comes back only after S1 and the master password.
| # | Secret / login | Needed for | Lives today | Lost in a total loss? |
|---|---|---|---|---|
| S1 | DooPlex off-site key (/etc/felhom-dooplex-offsite/enc.key, its paper data field) |
opening host/dooplex-gitea: Gitea, Vaultwarden, the Secrets export |
DooPlex; Vaultwarden if the operator saved it there | YES unless on paper — and it is the key to everything else |
| S2 | Hub-DB off-site key (/etc/felhom-hub-backup/enc.key, data field) |
opening host/dooplex-hub |
DooPlex; Vaultwarden (saved 2026-10-05) | yes unless on paper (Vaultwarden is behind S1 + S3) |
| S3 | Vaultwarden master password (user count 1; no two-factor rows, SSO-linked to Authentik but password login allowed) | every item in the password manager | the operator's head | no — if remembered |
| S4 | Hub seal key OFFSITE_SECRET_KEY |
the hub opening its console passwords and off-site passwords | DooPlex (Secret/offsite-secret-key); inside the Secrets export (behind S1 + S5); Vaultwarden |
yes unless on paper or S1+S5 work |
| S5 | DooPlex restic/GPG passphrase (/etc/backup/restic-password) |
opening the nightly Secrets export (secrets/*.gpg: every k8s Secret — Resend, report-api, Hetzner token, the hub's SSH keys, gitea-creds, TLS …) |
DooPlex (sdb1); "an offline copy, out of band" (operator, recon 2026-08-06 — form unknown to CC). Checked 2026-10-09: the newest export opens with it (symmetric GPG, AES256) and holds 428 Secrets, among them all eight the hub uses (names only read) |
depends on that offline copy |
| S6 | Signing recovery key felhom-rec-recovery (and the operational key felhom-op-operational) |
signing agent updates / config bundles / OS steps for every box; the recovery key authorizes rotating the operational key | DooPlex, no passphrase; since 2026-10-09 (R-924 option A) also in the nightly copy host/dooplex-gitea (behind S1) and, once printed, on the sheet |
no, if S1 or the printout survives — otherwise every box needs on-site re-enrollment (04 §4) |
| S7 | ep0 read-only restore token dooplex-hub@pbs!restore |
reading both copies without ep0 root | DooPlex only (/etc/felhom-hub-backup/token-restore) |
yes — but ep0 root can mint a new one (hub-DB runbook Step 2) |
| S8 | Hetzner account login + its two-factor | ep0's console (rescue, root reset), ep0 itself, Storage Boxes, the Hetzner API token | the operator; probably Vaultwarden | circle: if only in Vaultwarden, it is behind the copy it is needed to reach — paper or head |
| S9 | ep0 root SSH | the PBS tunnel; minting tokens | ep0 authorized_keys: DooPlex's key (lost) + one more key, owner not recorded (operator workstation?) |
partly — Hetzner console (S8) is the fallback |
| S10 | WireGuard operator peer (10.77.0.250) |
reaching ep0's PBS (10.77.0.1:8007) without SSH |
the operator's workstation | no, if the workstation survives |
| S11 | Cloudflare account login (+2FA) | DNS of felhom.eu and dooplex.hu (both on Cloudflare); pointing hub.felhom.eu at the new home; households' tunnels |
the operator; Vaultwarden | circle unless in head/paper |
| S12 | Gmail felhom.eu@gmail.com (+2FA) |
the @felhom.eu catch-all inbox; account recovery for other services |
the operator | — |
| S13 | Domain registrar(s) for felhom.eu, dooplex.hu; No-IP (dooplex.hopto.org, the home address CNAME) |
moving DNS / the home address | the operator; Vaultwarden | — |
| S14 | Resend account / API key | all mail | Secrets export (S1+S5); Vaultwarden; the Resend dashboard (re-mint) | no — re-mint at Resend |
| S15 | Gitea admin password; registry credentials (gitea-creds) |
Gitea UI; image push/pull | the restored Gitea DB (password hash) + Vaultwarden; the Secrets export | no, after S1 (+S3) |
| S16 | The hub operator password | the hub UI | the restored hub DB (hash); Vaultwarden — never in CC's store (operator-only since 2026-10-15, 09 §3 decision 251; CC holds the scoped token HUB_TOKEN in its secret store (S19) instead, which a restored hub DB keeps as its hash) |
no, after S2 (+S3) |
| S17 | Authentik admin | SSO for Vaultwarden and DooPlex apps | Authentik's DB (Longhorn on DooPlex, local PostgreSQL dumps only) | yes — not in any copy; Vaultwarden does not need it (password login) |
| S18 | Households' recovery codes, PBS keys, restic passwords | households' restores | the households / escrow / the boxes | not the operator's |
| S19 | CC secrets age key (~/.config/sops/age/keys.txt on DooPlex, 0600; public recipient age1aw83ej37x9sck89xwt856ry0553kf5sng3w544an4et54yjdx5qq79utrv) |
opening admin/cc-secrets store/cc.sops.yaml — every secret CC uses (sec; cc-secrets/ARCHITECTURE.md) |
DooPlex; Vaultwarden „CC secrets age key" (operator, 2026-10-17); the encrypted store itself is in Gitea (behind S1) | no, after S1 + S3 — unless also on the break-glass sheet |
The order of steps (new hardware)
- A machine and the sheet. Any Linux machine with Docker; for the hub also k3s. Install
proxmox-backup-client. - Reach ep0's PBS. Either the operator's WireGuard peer (S10) →
10.77.0.1:8007; or ep0 root SSH (S9) and the tunnelssh -N -L 127.0.0.1:18007:127.0.0.1:8007 root@167.233.158.164; or the Hetzner console (S8) → root → add a new SSH key. Fingerprint pin:proxmox-backup-manager cert infoon ep0. - A read token. S7 from the sheet, or mint a new one on ep0 (hub-DB runbook Step 2:
token/restore, ACLDatastoreReaderon/datastore/felhom-offsite/operator). - Restore
host/dooplex-giteawith S1 (key file rebuilt from thedatafield — hub-DB runbook Step 0 note). - Vaultwarden first (it holds the rest):
docker run -v <out>/vaultwarden:/data vaultwarden/server:<version>(DOMAIN,SIGNUPS_ALLOWED=false); log in with the e-mail and S3. Or, if a Bitwarden client still holds an offline copy, use that. Every later login comes from here. - The Secrets export:
gpg --batch --passphrase-file <S5 file> -d secrets-<stamp>.yaml.gpg > secrets.yaml— every k8s Secret of the old cluster (shred it after use). - Gitea:
gitea-restore.md(on a normal network this time). Then put the released images and agent packages back into its registry from ep0 — „Restore the release copy" below (same digests; no rebuild). Rebuild from code (RUNBOOK-manual-build.md) only what the copy does not hold. - The hub: k3s, then
RUNBOOK-hub-db-offsite-backup.md§3 with S2 and S4 — start it with mail held (§3 step 1: theMAIL-HOLDmarker — shipped in hub 0.145.0, 2026-10-10). - DNS: in Cloudflare (S11) point
hub.felhom.eu(today a CNAME todooplex.hopto.org) andgitea.dooplex.huat the new place. The boxes reconnect by themselves: their API keys are in the restored hub DB. - Signing: the keys are in the restored copy (
signing/) and on the sheet (S6) — put them back and sign as before. Without it, no box accepts an agent update or a bundle until it is re-enrolled on site (04§4). - Backups again: reinstall
scripts/hub-db-backup/,scripts/dooplex-offsite/andscripts/release-copy/from the restored code, the tunnel and the tokens (the release copy needs a new read-only registry token and a new SSH key in ep0'sfelhom-relcopyauthorized_keys), and check the next night's copies.
Restore the release copy (D33, since 2026-10-23)
What it is. Every night at 01:40 DooPlex copies the newest 3 release tags of felhom-controller, felhom-hub and
felhom-samba, and the newest 5 agent versions (binary + config bundle), to ep0 /srv/felhom-release-copy/data
(scripts/release-copy/). Images are in skopeo's dir: layout — the registry's own bytes — so a push back gives the
same digest as the release. Nothing in it is secret; it is not encrypted. Alarm: FelhomReleaseCopyStale.
- Get it off ep0 (ep0 root — step 2):
rsync -a root@167.233.158.164:/srv/felhom-release-copy/data/ ./release-copy/(or work on ep0 itself). Thencd release-copy && sha256sum -c --quiet SHA256SUMS— every line must pass. - Images back into the new registry, one per tag (log in to the new Gitea first:
skopeo login <gitea host>):The two hashes on each line must be equal.for d in images/*/*; do repo=${d#images/}; repo=${repo%%/*}; tag=${d##*/} skopeo copy --all --preserve-digests "dir:$d" "docker://<gitea host>/admin/$repo:$tag" echo "$repo:$tag $(skopeo inspect --raw "docker://<gitea host>/admin/$repo:$tag" | sha256sum | cut -c1-64) want $(cut -c8- "$d/DIGEST")" done - Agent packages back (Gitea's generic registry, a token with package write):
Check each download's sha256 against
for d in agent/*; do v=${d#agent/}; for f in felhom-agent felhom-config-bundle.json; do curl -fsS -u admin:"$(cat token-file)" -X PUT --upload-file "$d/$f" \ "https://<gitea host>/api/packages/admin/generic/felhom-agent/$v/$f"; done; doneagent/<v>/<file>.sha256. - Keep the registry's name (
gitea.dooplex.hu, step 9's DNS) — the boxes pullgitea.dooplex.hu/admin/…by name.
Proven 2026-10-23 (audits/day-2026-10-23/08-release-copy-D33.txt): the copy was pulled back from ep0, every
sha256 matched, every image re-verified against its digest, and controller 0.340.0 from it loaded on a scratch box
(image ID = its config digest; a container ran). Not yet walked: step 2 into a real new registry, and step 3.
Walked on paper (2026-10-09): what still needs DooPlex if the operator has only the sheet, memory and a new machine
| Step | Needs | From the sheet / memory? |
|---|---|---|
| reach ep0 | WireGuard peer or ep0 root or Hetzner login | WireGuard peer is on the workstation, not on the sheet; ep0 root key: one of two keys is DooPlex's; Hetzner login (S8) must be on the sheet |
| PBS fingerprint | ep0's cert fingerprint | not secret; on the sheet as a line, or read on ep0 |
| a read token | S7 or ep0 root | on the sheet (S7), or via S8 |
| open the Gitea/Vaultwarden copy | S1 | sheet |
| open the vault | S3 | memory |
| open the Secrets export | S5 | sheet (today: "offline copy", form unknown) |
| the hub's seal key | S4 (or S1+S5) | sheet, or via S5 |
| the hub DB | S2 (or Vaultwarden) | sheet, or via S3 |
| sign a box update | S6 | sheet — today exists on DooPlex only |
| hub, controller, samba images; agent packages | the release copy on ep0 (D33, nightly since 2026-10-23) | ep0 root (S8/S9) → „Restore the release copy" — no secret, no rebuild |
felhom.eu website + the installer script, ISO host |
Cloudflare (S11): since 2026-10-30 (D35 / R-1228) the site and /scripts/felhom-host-install.sh are served by Cloudflare Pages, the ISO by R2 — both keep answering with DooPlex gone; a CHANGE to them needs the repo (Gitea copy) and scripts/publish-site-cloudflare.sh. A new install still needs DooPlex for two things: the hub (hub.felhom.eu: the claim, the box's config, its first report) and Gitea (gitea.dooplex.hu: the agent package and the golden). The contact form (api.felhom.eu) stops; the page's mailto: link works |
S11 on the sheet; the build tree is rebuilt from code |
| Authentik (S17) | not in any copy | not needed for the business; DooPlex homelab apps that use it are out of scope |
Result: with a filled sheet (S1, S2, S4, S5, S6, S7, S8, S11 + the ep0 fingerprint) and the master password in memory, nothing in the Felhom recovery still needs DooPlex. Without the sheet, three things block: S1 (no copy opens), S6 (no box can be signed for), S8 (ep0 cannot be reached once DooPlex's key is gone, unless the workstation's WireGuard peer survives).