felhom.eu/REPORT-dooplex-survival-2026-10-09.md

REPORT — can the business survive losing DooPlex? (2026-10-09)

Part	What	Result
A	Measure; plan	DONE — audits/dooplex-survival-2026-10-09/PLAN.md; operator said yes in chat
B	Nightly encrypted copy of Gitea + secrets to ep0	DONE, LIVE — first push 57 s; read back; alarms in force; failure mail proven
C	Gitea restored into a throwaway	DONE — 10/10 repos, main = live, a file byte for byte, a login; deleted
D	Hub restored into a throwaway k3s (R-173 steps 4–5)	DONE — customers 4/4, hosts 4/4, console passwords 4/4; deleted; R-173 closed
E	R-861 (a), R-518, the mail-address row	R-861 closed, R-518 closed (+ R-921), R-922 filed

Register: before 135 · after 135 · opened 2 (R-921, R-922) · closed 3 (R-173, R-861, R-518). A parallel session added one row meanwhile (the file read 136 before this session's edit).

Architecture read before any claim: 07-backup-architecture.md (trust model, encryption), 06-offsite-connectivity.md, 01-topology-and-trust.md, runbooks/target-selection.md, R-232 + its recon, the R-173 runbook.

Part A — measured (read only)

• Gitea 1.26.2: repositories 614 MB (10 repos), database gitea 64 MB live = 6.2 MB as a dump (~150 KB/day), app.ini 2 KB (holds Gitea's secrets). Registry 27.7 GB — left out on purpose (images rebuild from code).
• Secrets set: three GPG files per night, 4.4 MB, flat.
• Correction to the recon (§7): the API mirror holds ONE repo (homelab-manifests). The product repos had one backup only: Longhorn, retain=1, on DooPlex.
• ep0 operator: 83 GB free of 98 GB; the hub-DB's write-only token already reaches it. Nothing to create on ep0.

Part B — what was installed on DooPlex (operator yes)

• /usr/local/sbin/felhom-dooplex-offsite (+ restore test, + felhom-backup-failmail), from scripts/dooplex-offsite/ at felhom.eu 1707c928/02a54e26. Timers: daily 00:20 and Sun 05:30.
• New key /etc/felhom-dooplex-offsite/enc.key (root 0600, --kdf none, never printed). Tokens: the hub-DB's, read in place.
• OnFailure=felhom-backup-failmail@%n.service on the two new units and the two hub-DB units.
• homelab-manifests 691db39: DooplexGiteaOffsiteStale (26 h) + DooplexGiteaRestoreTestStale (8 days), absent() included; only the rules ConfigMap synced; /-/reload 200; /api/v1/rules reads both ok.
• Consistency choice: Gitea's docs say stop it for a consistent backup. Not chosen — a nightly stop costs CI and the registry. Instead: the newest complete database dump (taken first), then the files; the restore test runs git fsck on every repo.
• Proof: first push 544 MB in 57 s; restore test with the read-only token: 27 805 files match, 10 repos pass git fsck; the push token asked to forget a snapshot → permission check failed; ep0's disk shows the new group.
• Found and fixed live: the failure-mail script died under set -u on the shared config's unset variable (the first dry run sent no mail). Red test, fix, second dry run → Resend id, mail in the inbox 08:17:00Z.
• Security review findings, both fixed and tested: links in the pod's archive are refused; root git fsck never reads a repo's own config (red-proved with a config git refuses).
• Tests: 20, green with GNU and BusyBox tools; red-proofs for 7 checks (partB/red-proof.txt); promtool green + 2 reds.

Part C — Gitea from ep0 into a throwaway (the bench, LXC 9401 on demo-hp)

Restored on DooPlex with the read-only token (22 s), streamed to the bench without the secrets files, Postgres 17.2 and Gitea 1.26.2 on an --internal Docker network (no route out: wget gitea.com → bad address). 10/10 repos; three product main equal live; felhom.eu one commit behind — that commit (02a54e26) was pushed 3 min after the copy and the copy's 1707c928 is its parent; CLAUDE.md sha256 equal; throwaway admin logged in. Runbook: runbooks/gitea-restore.md.

Part D — the hub into a throwaway k3s (same bench)

k3s v1.33.6 in a Docker container on an internal network (two fixes needed: the cgroup v2 move, --flannel-iface eth0). Hub image of the live digest, by file. Runbook §3 steps 4–5 as written. Start-up: console passwords sealed at rest (0 legacy …). Customers 4/4 and hosts 4/4 equal live (GET /configs, /hosts, live read GET-only). 4/4 reveals → 32-char passwords (length only); second channel hubdb-check: 4/4 with the saved key, 0/4 random. Found: the restored hub tried to mail two households a pending kernel notice within a minute — only the cut network stopped it; operator-mail-off does not. Runbook §3 corrected (and: three more Secrets are not optional).

Teardown (three layers)

• Machine (bench 9401): containers 0, volumes 0, networks back to the original four, images removed, /root as before, stopped again (it was stopped). Found: Postgres and k3s leave anonymous volumes holding the data; 1 + 8 found by counting, each checked by creation time, removed by name (no prune).
• Host (DooPlex): both scratch dirs shredded and removed; the seal-key and password files shredded.
• Hub: provisioned nothing. The live hub was only read (GET).

Part E

• R-861 (a) CLOSED: all three 0.304.0 swaps went through felhom-priv-apply controller-image (sudo log + wrapper log
  • agent journal) and the guests run 0.304.0 (Docker). No tee that day.
• R-518 CLOSED: 2026-10-08 on demo-hp both tiers ran, each with its own stop under ~2 min. A third stop of ~1 min with no copy happened when the off-site tier answered BUSY → R-921 (P3, needs a release).
• R-922 filed (P2): a household's cleared mail address stays on the hub (the no-clobber guard). Rule: operator.

Other

• CI: 1707c928, 02a54e26 first failed with every step red and no runner log (R-887, dropped jobs); re-run → success. 994826ec success. I pushed the second commit before reading the first run — the rule says check first.
• unproven.py --summary: not walked 35 of 55 (unchanged).
• Instruction files: none edited.

For the operator

1. Save the new key's paper copy (at your own terminal, not through ! here): sudo proxmox-backup-client key paperkey /etc/felhom-dooplex-offsite/enc.key --output-format text → the data field into the password manager as „DooPlex off-site (Gitea) key". If you do nothing: after a DooPlex loss the copy on ep0 cannot be opened. Pick: do it today.
2. R-922 — a household clears its mail address: (A) the hub drops the address when the household clears it (the controller says so explicitly); (B) keep it, and write the keep time into the privacy notice. Pick A. If you do nothing: the address stays on the hub with no end date.