10 KiB
10 KiB
DooPlex survival survey — night 2026-10-22 → 23 (read only)
Night brief Part 5. Nothing on DooPlex was changed. Every number below was read on DooPlex tonight (unit state,
journal lines, du, kubectl get, DNS, HTTP status) — raw readings in readings/. Secrets are named, never read:
no key file, token file or password was opened. ep0 was not contacted (the nightly jobs' own journal lines are the
evidence of what reached it). Read with runbooks/total-loss-of-dooplex.md (the order of steps) and the 2026-10-09
plan (audits/dooplex-survival-2026-10-09/PLAN.md, which built the nightly copy).
1. What lives on DooPlex, and what saves it
| Thing | Where it lives | What backs it up | Where the copy is | Last copy (measured) | Who can restore | How long |
|---|---|---|---|---|---|---|
Gitea — 12 repositories (845 MB), DB in CNPG, app.ini |
PVC gitea-data (Longhorn, sdb1) |
(1) 6-hourly pg_dump + Longhorn backup, both on sda1 (same box); (2) felhom-dooplex-offsite nightly 00:20 |
ep0, namespace operator, host/dooplex-gitea, encrypted (key S1) |
2026-10-22 00:22, 736 MiB (425 MiB new), 42 s, success | the operator with S1 + the read token S7 (or ep0 root) | restore ~1 min + start ~1 min on a ready machine (walked 2026-10-09 Part C); on new hardware add k3s/Docker install — hours |
| Weekly proof of that copy | DooPlex timer, Sunday 05:30 | — | — | 2026-10-18: 27 739 files match, 11 repositories pass git fsck, gitea.dump readable, Vaultwarden 1 user / 810 items |
— | — |
| Container registry + generic packages (controller/hub images, agent packages) | /data/gitea/packages in gitea-data, 38.6 GB |
Longhorn backup on sda1 only |
nowhere off DooPlex (left out on purpose 2026-10-09: „rebuilt from the code") | — | rebuild from the restored code on a build machine | never timed |
Hub + its DB (hub.db ~493 MiB) |
PVC hub-data (Longhorn) |
Longhorn backup (same box); felhom-hub-db-backup nightly 02:30 |
ep0 operator, host/dooplex-hub, encrypted (key S2); seal key S4 needed to open its sealed passwords |
2026-10-22 02:31, 493 MiB (24 MiB compressed), 6 s, success | the operator with S2 + S4 + S7 | ~5 min on a bench with k3s ready (2026-10-09 Part D) |
| Weekly proof of that copy | Sunday 04:31 | — | — | 2026-10-18: integrity ok, 4 hosts, 4 sealed console passwords, 0 readable | — | — |
| Operator signing keys (3 files, 0600, no passphrase) | /mnt/5_hdd/felhom.eu/ on sda1 (the same disk as the local backup tree) |
inside the nightly host/dooplex-gitea copy (signing/; a missing key refuses the push) |
ep0, behind S1; paper (S6) if printed | with the Gitea copy (2026-10-22) | the operator with S1, or the printout | minutes |
| Password manager (Vaultwarden, 1 user, 810 items) | PVC vaultwarden-data (Longhorn) |
Longhorn backup (same box); nightly copy (its own vaultwarden backup + rsa_key.pem) |
ep0, behind S1; the vault itself opens with the master password S3 | with the Gitea copy (2026-10-22) | the operator with S1 + S3 | started once on a throwaway (2026-10-09) |
| k8s Secrets (every Secret, GPG-encrypted with S5) | nightly export /mnt/5_hdd/backup/secrets (136 MB, 30 days) |
the same file rides the nightly copy | ep0 behind S1, then S5 | 2026-10-22 | the operator with S1 + S5 | minutes |
CI (act-runner pod, PVC act-runner-data) |
k3s | nothing needed: re-registered from Gitea | — | — | the operator | ~15 min |
Website, installer script, hub.felhom.eu, gitea.dooplex.hu |
DooPlex, behind the home line (all four resolve to the home address; measured) | the website and installer are code in the Gitea copy | ep0 (as code) | — | the operator + a DNS change at Cloudflare (S11) | after Gitea + k3s |
| The ISO | Cloudflare R2 (iso.felhom.eu) |
— | not on DooPlex | — | — | survives |
| ArgoCD | k3s; Applications from homelab-manifests (in the Gitea copy) |
its Secrets in the export | ep0 behind S1 + S5 | — | the operator | ~30 min; the per-app auto-sync settings are set on the live objects, not in git (several apps differ) |
| DooPlex's own local backups (R-232) | /mnt/5_hdd/backup on sda1: Longhorn 117 GB, data 53 GB, PostgreSQL dumps 2.1 GB, ep0-copy 21 GB, … |
dooplex-backup nightly 03:00 |
the same box | 2026-10-22 03:16, „All backup components completed successfully" | — | — |
Restore tests: both weekly tests passed on 2026-10-18 (their next run is Sunday 2026-10-25). Since then the pod lists 12 repositories against the test's 11; the nightly push refuses a copy with fewer repositories than the pod lists, and it succeeded, so the twelfth is in the copy — the next Sunday test will show it.
2. total-loss-of-dooplex.md, step by step — works today / stale / never tried
| Step | Verdict | Why |
|---|---|---|
| 1. A machine and the sheet | depends | the sheet's printing is the operator's (R-923, R-924 „LEFT"); CC cannot see whether it exists |
| 2. Reach ep0's PBS | works from DooPlex (the nightly jobs use the tunnel); never tried from new hardware (WireGuard peer, Hetzner console) | |
| 3. A read token (S7) | works today | the weekly restore tests use it |
4. Restore host/dooplex-gitea with S1 |
works today | the weekly test restores and checks it (2026-10-18) |
| 5. Vaultwarden first | works (once) | started on a throwaway 2026-10-09; its rows are checked weekly |
| 6. The Secrets export with S5 | works (once) | opened 2026-10-09; the form of S5's offline copy is unknown to CC |
| 7. Gitea, then rebuild the images | Gitea: works (once, 2026-10-09). Image rebuild: never tried | the registry is not copied |
| 8. The hub with S2 + S4, mail held | works (once, 2026-10-09 Part D) | stale text: it says the mail hold comes „from the next hub release" — it shipped in hub 0.145.0 (2026-10-10); fixed tonight |
| 9. DNS at Cloudflare | never tried | |
| 10. Signing | keys readable (the weekly test derives each public half) — signing from the restored keys never tried | |
| 11. Backups again | never tried on new hardware | install.sh exists in the code |
3. The gaps — two options each, with CC's pick
| # | Gap | Option A (cost: money / operator hours / risk) | Option B (cost) | CC's pick |
|---|---|---|---|---|
| G1 | The paper sheet — S1 opens every off-site copy (Gitea, Vaultwarden, signing keys, Secrets); without it no copy opens (R-923, R-924) | print break-glass-sheet.md filled (S1, S2, S4, S5, S6, S7, S8, S11, ep0 fingerprint) and store it outside the home, sealed — 0 € / ~30 min / paper can be lost or read |
the key files on an encrypted USB stick kept outside the home + its passphrase in the head — ~10 € / ~1 h / one more secret to remember, a stick can die | A — it is the written plan; nothing else breaks the circle |
| G2 | Signing keys without a passphrase (R-924) | keep as the custody ruling (2026-09-16) says until the first sale; they ride the S1-encrypted copy — 0 € / 0 h / a DooPlex break-in can sign | a passphrase on the operational key and the recovery key off DooPlex — 0 € / ~2 h / every signed job then needs the operator; no unattended signing | A now; revisit at the first sale (the ruling stands) |
| G3 | The password manager holds keys to DooPlex's own copies (R-923) | the sheet (G1) breaks the circle — no extra cost | a monthly encrypted export to the operator's laptop — 0 € / 10 min a month / easy to forget | A (same act as G1) |
| G4 | DooPlex's backups sit inside the same box (R-232) — the Felhom parts are off-site now; the homelab apps, Authentik, Prometheus are not | accept for the homelab (not Felhom) — 0 € / 0 h / a homelab loss is total | restic of the chosen /mnt/5_hdd/backup sets to a Storage Box sub-account — ~4 €/month / ~2 h / ~170 GB first upload |
A for Felhom's scope; the homelab is the operator's call |
| G5 | Hub DB copy is up to 24 h old — a box that enrolled or re-escrowed after 02:30 has no escrow row in a restored hub | keep nightly — 0 € / 0 h / up to a day of hub changes lost | push every 6 h (same job, same token, same prune) — 0 € / ~30 min (a DooPlex change, operator's yes) / ~4 × 24 MB a day on ep0 | B |
| G6 | Gitea repositories off-site — covered nightly, tested weekly | keep — 0 € | a second off-site mirror (Codeberg/GitHub private) — 0 € / ~1 h / code at a third party | A |
| G7 | The registry and the agent packages exist only on DooPlex — every box update, every reinstall and every new install pull from them | copy only what a box can be offered (the released controller/hub tags, the agent packages, the infra images) nightly to ep0 operator — 0 € / ~2 h CC + the operator's yes for a DooPlex change / a few GB on ep0 |
a timed rebuild-from-code drill, then accept the outage — 0 € / ~3 h once / a rebuilt image is not the same digest (the „a version is pushed once" rule) | A |
| G8 | Website, installer script and the hub all on the home line — a DooPlex loss stops new installs and the public site | accept: Gitea and the hub come back first, then the site — 0 € / hours on the day / no installs meanwhile | serve the static site and the installer tag from Cloudflare (as the ISO already is) — 0 € / ~2 h / a second publish path to keep true | B, after the release week |
| G9 | ArgoCD's per-app sync settings live only on the cluster | write them into homelab-manifests — 0 € / ~30 min / none |
re-set by hand after a loss from the memory notes — 0 € / ~1 h on the day | A |
| G10 | S5's offline copy: form unknown to CC | put S5 on the sheet (G1) — 0 € | re-encrypt the Secrets export with an S1-derived key — 0 € / ~1 h / a DooPlex change | A |
The three biggest: G1 (no sheet → nothing opens), G7 (no registry copy → no update or install after a loss), G8 (the website, installer and hub on one home line → new installs stop for hours or days).
4. What tomorrow's hour decides (also in STATUS.md)
- G1 — print the sheet now? (CC: yes)
- G7 — copy the release images and agent packages to ep0 nightly? (CC: yes)
- G5 — hub DB every 6 hours? (CC: yes)
- G8 — the site and installer on Cloudflare after the release week? (CC: yes, later)