felhom.eu/documentation/audits/dooplex-survival-2026-10-22/PLAN.md

# DooPlex survival survey — night 2026-10-22 → 23 (read only)

Night brief Part 5. **Nothing on DooPlex was changed.** Every number below was read on DooPlex tonight (unit state,
journal lines, `du`, `kubectl get`, DNS, HTTP status) — raw readings in `readings/`. Secrets are named, never read:
no key file, token file or password was opened. ep0 was not contacted (the nightly jobs' own journal lines are the
evidence of what reached it). Read with `runbooks/total-loss-of-dooplex.md` (the order of steps) and the 2026-10-09
plan (`audits/dooplex-survival-2026-10-09/PLAN.md`, which built the nightly copy).

## 1. What lives on DooPlex, and what saves it

| Thing | Where it lives | What backs it up | Where the copy is | Last copy (measured) | Who can restore | How long |
|---|---|---|---|---|---|---|
| **Gitea** — 12 repositories (845 MB), DB in CNPG, `app.ini` | PVC `gitea-data` (Longhorn, `sdb1`) | (1) 6-hourly `pg_dump` + Longhorn backup, both on `sda1` (same box); (2) `felhom-dooplex-offsite` nightly 00:20 | ep0, namespace `operator`, `host/dooplex-gitea`, encrypted (key **S1**) | 2026-10-22 00:22, 736 MiB (425 MiB new), 42 s, success | the operator with S1 + the read token S7 (or ep0 root) | restore ~1 min + start ~1 min on a ready machine (walked 2026-10-09 Part C); on new hardware add k3s/Docker install — hours |
| **Weekly proof of that copy** | DooPlex timer, Sunday 05:30 | — | — | 2026-10-18: 27 739 files match, **11** repositories pass `git fsck`, `gitea.dump` readable, Vaultwarden 1 user / 810 items | — | — |
| **Container registry + generic packages** (controller/hub images, agent packages) | `/data/gitea/packages` in `gitea-data`, **38.6 GB** | Longhorn backup on `sda1` only | **nowhere off DooPlex** (left out on purpose 2026-10-09: „rebuilt from the code") | — | rebuild from the restored code on a build machine | never timed |
| **Hub + its DB** (`hub.db` ~493 MiB) | PVC `hub-data` (Longhorn) | Longhorn backup (same box); `felhom-hub-db-backup` nightly 02:30 | ep0 `operator`, `host/dooplex-hub`, encrypted (key **S2**); seal key **S4** needed to open its sealed passwords | 2026-10-22 02:31, 493 MiB (24 MiB compressed), 6 s, success | the operator with S2 + S4 + S7 | ~5 min on a bench with k3s ready (2026-10-09 Part D) |
| **Weekly proof of that copy** | Sunday 04:31 | — | — | 2026-10-18: integrity ok, 4 hosts, 4 sealed console passwords, 0 readable | — | — |
| **Operator signing keys** (3 files, 0600, no passphrase) | `/mnt/5_hdd/felhom.eu/` on `sda1` (the same disk as the local backup tree) | inside the nightly `host/dooplex-gitea` copy (`signing/`; a missing key refuses the push) | ep0, behind S1; paper (S6) **if printed** | with the Gitea copy (2026-10-22) | the operator with S1, or the printout | minutes |
| **Password manager** (Vaultwarden, 1 user, 810 items) | PVC `vaultwarden-data` (Longhorn) | Longhorn backup (same box); nightly copy (its own `vaultwarden backup` + `rsa_key.pem`) | ep0, behind S1; the vault itself opens with the master password S3 | with the Gitea copy (2026-10-22) | the operator with S1 + S3 | started once on a throwaway (2026-10-09) |
| **k8s Secrets** (every Secret, GPG-encrypted with S5) | nightly export `/mnt/5_hdd/backup/secrets` (136 MB, 30 days) | the same file rides the nightly copy | ep0 behind S1, then S5 | 2026-10-22 | the operator with S1 + S5 | minutes |
| **CI** (`act-runner` pod, PVC `act-runner-data`) | k3s | nothing needed: re-registered from Gitea | — | — | the operator | ~15 min |
| **Website, installer script, `hub.felhom.eu`, `gitea.dooplex.hu`** | DooPlex, behind the home line (all four resolve to the home address; measured) | the website and installer are code in the Gitea copy | ep0 (as code) | — | the operator + a DNS change at Cloudflare (S11) | after Gitea + k3s |
| **The ISO** | Cloudflare R2 (`iso.felhom.eu`) | — | not on DooPlex | — | — | survives |
| **ArgoCD** | k3s; Applications from `homelab-manifests` (in the Gitea copy) | its Secrets in the export | ep0 behind S1 + S5 | — | the operator | ~30 min; **the per-app auto-sync settings are set on the live objects, not in git** (several apps differ) |
| **DooPlex's own local backups** (R-232) | `/mnt/5_hdd/backup` on `sda1`: Longhorn 117 GB, `data` 53 GB, PostgreSQL dumps 2.1 GB, `ep0-copy` 21 GB, … | `dooplex-backup` nightly 03:00 | **the same box** | 2026-10-22 03:16, „All backup components completed successfully" | — | — |

**Restore tests:** both weekly tests passed on 2026-10-18 (their next run is Sunday 2026-10-25). Since then the pod
lists **12** repositories against the test's 11; the nightly push refuses a copy with fewer repositories than the pod
lists, and it succeeded, so the twelfth is in the copy — the next Sunday test will show it.

## 2. `total-loss-of-dooplex.md`, step by step — works today / stale / never tried

| Step | Verdict | Why |
|---|---|---|
| 1. A machine and the sheet | **depends** | the sheet's printing is the operator's (R-923, R-924 „LEFT"); CC cannot see whether it exists |
| 2. Reach ep0's PBS | works **from DooPlex** (the nightly jobs use the tunnel); **never tried** from new hardware (WireGuard peer, Hetzner console) | |
| 3. A read token (S7) | works today | the weekly restore tests use it |
| 4. Restore `host/dooplex-gitea` with S1 | works today | the weekly test restores and checks it (2026-10-18) |
| 5. Vaultwarden first | works (once) | started on a throwaway 2026-10-09; its rows are checked weekly |
| 6. The Secrets export with S5 | works (once) | opened 2026-10-09; **the form of S5's offline copy is unknown to CC** |
| 7. Gitea, then rebuild the images | Gitea: works (once, 2026-10-09). Image rebuild: **never tried** | the registry is not copied |
| 8. The hub with S2 + S4, mail held | works (once, 2026-10-09 Part D) | **stale text:** it says the mail hold comes „from the next hub release" — it shipped in hub 0.145.0 (2026-10-10); fixed tonight |
| 9. DNS at Cloudflare | **never tried** | |
| 10. Signing | keys readable (the weekly test derives each public half) — **signing from the restored keys never tried** | |
| 11. Backups again | **never tried** on new hardware | `install.sh` exists in the code |

## 3. The gaps — two options each, with CC's pick

| # | Gap | Option A (cost: money / operator hours / risk) | Option B (cost) | CC's pick |
|---|---|---|---|---|
| G1 | **The paper sheet** — S1 opens every off-site copy (Gitea, Vaultwarden, signing keys, Secrets); without it no copy opens (R-923, R-924) | print `break-glass-sheet.md` filled (S1, S2, S4, S5, S6, S7, S8, S11, ep0 fingerprint) and store it outside the home, sealed — 0 € / ~30 min / paper can be lost or read | the key files on an encrypted USB stick kept outside the home + its passphrase in the head — ~10 € / ~1 h / one more secret to remember, a stick can die | **A** — it is the written plan; nothing else breaks the circle |
| G2 | **Signing keys without a passphrase** (R-924) | keep as the custody ruling (2026-09-16) says until the first sale; they ride the S1-encrypted copy — 0 € / 0 h / a DooPlex break-in can sign | a passphrase on the operational key and the recovery key off DooPlex — 0 € / ~2 h / every signed job then needs the operator; no unattended signing | **A** now; revisit at the first sale (the ruling stands) |
| G3 | **The password manager holds keys to DooPlex's own copies** (R-923) | the sheet (G1) breaks the circle — no extra cost | a monthly encrypted export to the operator's laptop — 0 € / 10 min a month / easy to forget | **A** (same act as G1) |
| G4 | **DooPlex's backups sit inside the same box** (R-232) — the Felhom parts are off-site now; the homelab apps, Authentik, Prometheus are not | accept for the homelab (not Felhom) — 0 € / 0 h / a homelab loss is total | restic of the chosen `/mnt/5_hdd/backup` sets to a Storage Box sub-account — ~4 €/month / ~2 h / ~170 GB first upload | **A** for Felhom's scope; the homelab is the operator's call |
| G5 | **Hub DB copy is up to 24 h old** — a box that enrolled or re-escrowed after 02:30 has no escrow row in a restored hub | keep nightly — 0 € / 0 h / up to a day of hub changes lost | push every 6 h (same job, same token, same prune) — 0 € / ~30 min (a DooPlex change, operator's yes) / ~4 × 24 MB a day on ep0 | **B** |
| G6 | **Gitea repositories off-site** — covered nightly, tested weekly | keep — 0 € | a second off-site mirror (Codeberg/GitHub private) — 0 € / ~1 h / code at a third party | **A** |
| G7 | **The registry and the agent packages exist only on DooPlex** — every box update, every reinstall and every new install pull from them | copy only what a box can be offered (the released controller/hub tags, the agent packages, the infra images) nightly to ep0 `operator` — 0 € / ~2 h CC + the operator's yes for a DooPlex change / a few GB on ep0 | a timed rebuild-from-code drill, then accept the outage — 0 € / ~3 h once / a rebuilt image is not the same digest (the „a version is pushed once" rule) | **A** |
| G8 | **Website, installer script and the hub all on the home line** — a DooPlex loss stops new installs and the public site | accept: Gitea and the hub come back first, then the site — 0 € / hours on the day / no installs meanwhile | serve the static site and the installer tag from Cloudflare (as the ISO already is) — 0 € / ~2 h / a second publish path to keep true | **B**, after the release week |
| G9 | **ArgoCD's per-app sync settings live only on the cluster** | write them into `homelab-manifests` — 0 € / ~30 min / none | re-set by hand after a loss from the memory notes — 0 € / ~1 h on the day | **A** |
| G10 | **S5's offline copy: form unknown to CC** | put S5 on the sheet (G1) — 0 € | re-encrypt the Secrets export with an S1-derived key — 0 € / ~1 h / a DooPlex change | **A** |

**The three biggest:** G1 (no sheet → nothing opens), G7 (no registry copy → no update or install after a loss),
G8 (the website, installer and hub on one home line → new installs stop for hours or days).

## 4. What tomorrow's hour decides (also in `STATUS.md`)

1. G1 — print the sheet now? (CC: yes)
2. G7 — copy the release images and agent packages to ep0 nightly? (CC: yes)
3. G5 — hub DB every 6 hours? (CC: yes)
4. G8 — the site and installer on Cloudflare after the release week? (CC: yes, later)