felhom.eu/documentation/audits/dooplex-survival-2026-10-09/PLAN.md

PLAN — an encrypted off-site copy of Gitea and DooPlex's secrets (R-232 (b), (h))

2026-10-09, Part A of the DooPlex-survival brief. Everything below was measured read-only; nothing was created. Raw readings: partA/readings.txt.

What is measured

Thing	Size	Growth	Where it lives today
Gitea repositories (/data/git/repositories, 10 repos under admin/)	614 MB	~600 MB in 8 months (oldest product repo 2026-02-11); felhom.eu.git 188 MB	Longhorn PVC gitea-data on sdb1; Longhorn backup on sda1, retain=1. Nothing else.
Gitea database (CNPG postgresql, db gitea)	64 MB live, 6.2 MB as a pg_dump -Fc	5.62 → 6.22 MB in 4 days (~150 KB/day)	6-hourly dumps in /mnt/5_hdd/backup/postgresql/dumps/ (same disk as every backup)
Gitea config app.ini (holds SECRET_KEY, INTERNAL_TOKEN, JWT secrets)	2 KB	—	in the PVC only
Gitea registry (/data/gitea/packages)	27.7 GB	—	Left out on purpose: the images rebuild from the code.
DooPlex secrets set: the nightly GPG files (secrets-, configmaps-, by-namespace-*.gpg)	4.4 MB per night	flat (2,055,542 → 2,056,090 B in 12 days)	sda1 only, 30 days

Correction to the 2026-08-06 recon (§7): it said the Gitea repositories are "covered twice". The API mirror (backup/homelab-manifests/git-mirrors/) holds one repository, homelab-manifests.git (7.1 MB). The four product repositories have exactly one backup: the Longhorn copy with retain=1, on the same machine.

The two places measured

	ep0, namespace operator (where the hub DB goes)	A new Hetzner Storage Box sub-account
Free space	83 GB of 98 GB (/mnt/pbs-datastore, 16 % used)	depends on the box; not measured
Write-only key	Already exists. Token dooplex-hub@pbs!push has DatastoreBackup on /operator only: it cannot forget a snapshot or list the households (measured 2026-10-05). A second read-only token !restore exists for the restore test.	the R-820 append-only pin; the sub-account shell can still rm unless the pin is right
Changes needed there	None. A new backup group (host/dooplex-gitea) in the same namespace needs no new token, ACL or prune job. The existing prune job prune-operator-hubdb (keep-daily 14, keep-weekly 8) covers every group in operator.	a new sub-account + key custody
Cost	0 € extra	a sub-account (same Storage Box fee)
Return copy	ep0's nightly pull to DooPlex (ep0-copy, 05:00) copies it back too — ciphertext only	none

Pick: ep0 operator. No change on ep0 at all, the same write-only token as the hub DB, the same prune and the same alarm family. Worst case on ep0: a git repack every night makes each copy new chunks — 22 kept copies × 0.65 GB ≈ 14 GB, inside the 83 GB free.

What runs

A script and a timer on DooPlex, felhom-dooplex-offsite (versioned in felhom.eu/scripts/dooplex-offsite/, installed by its install.sh, tests by hand like the hub DB's), daily 00:20 (after the 00:00 database dump, before the hub DB at 02:30):

1. The database first: the newest complete gitea.dump + globals.sql from the 6-hourly dumps (folder with SUCCESS; refused if older than 7 h). This is PostgreSQL's own consistent snapshot.
2. Then the files, read from the running Gitea pod with tar over kubectl exec: /data/git/repositories, /data/git/lfs, /data/gitea/conf/app.ini, attachments, avatars, jwt. Not packages, logs, indexers, queues, tmp.
3. Then the secrets set: the newest night's three .gpg files (already GPG-encrypted with DooPlex's restic passphrase, which the operator holds offline).
4. A MANIFEST.sha256 of everything, then proxmox-backup-client backup → host/dooplex-gitea in operator, --crypt-mode encrypt with a new key /etc/felhom-dooplex-offsite/enc.key.
5. Stage shredded; a success timestamp (written only after the push returns 0) for Prometheus.

Why not Gitea's own gitea dump: Gitea's documentation says the instance "must be shutdown during backup" for consistency, and gitea dump does not stop it either. A nightly stop costs CI and the registry a gap and adds a restart that must come back. Instead the order makes the copy safe: the database is older than the repositories, so every commit the database names is in the copy (git writes objects before it moves a ref). A repository newer than its database is the normal state after a push and Gitea reads it. The restore test catches the rest: it runs git fsck on every repository.

Encryption and the key

• New key, generated on DooPlex as root: proxmox-backup-client key create --kdf none → 0600 file in /etc/felhom-dooplex-offsite/, never printed by CC, never in chat.
• The operator copies its paper form into the password manager at his own terminal (not through ! here): sudo proxmox-backup-client key paperkey /etc/felhom-dooplex-offsite/enc.key --output-format text — the data field is enough (as for the hub DB key, runbook Step 0).
• ep0 never sees the key. The secrets files inside are encrypted twice (GPG + PBS).

Retention and who can delete

• ep0's existing prune-operator-hubdb (03:45): 14 daily + 8 weekly, per group. ep0's GC frees chunks.
• Can delete: only ep0's root (and that prune job). The push token cannot forget a snapshot; the restore token cannot write. DooPlex's root cannot delete on ep0 with what it holds.

Alarms and failure mail

• DooplexGiteaOffsiteStale (26 h, absent() included) and DooplexGiteaRestoreTestStale (8 days), same group as HubDBBackupStale, proven with promtool test rules and a red-proof.
• Failure mail: an OnFailure= unit that calls the R-232 (a) notify_failure (Resend → admin@felhom.eu). The same OnFailure= is added to the two hub-DB units. Proven with one dry failure (a unit that fails on purpose, no backup).

The restore test

• Weekly, Sunday 05:30, with the read-only token: restore the newest copy to a temp dir, check MANIFEST.sha256, git fsck every repository, pg_restore --list gitea.dump, then shred; success timestamp.
• Once now (Part C): restore into a throwaway machine, start Gitea there with no outside network, compare every product repository's main with live Gitea, read one file byte for byte, log in with a throwaway admin. Steps become runbooks/gitea-restore.md. The throwaway and every copy are deleted.

What changes on DooPlex (needs the operator's yes)

1. /usr/local/sbin/felhom-dooplex-offsite + felhom-dooplex-offsite-restore-test (scripts, from git).
2. Units: felhom-dooplex-offsite.{service,timer}, felhom-dooplex-offsite-restore-test.{service,timer}, felhom-backup-failmail@.service; one OnFailure= line added to the two hub-DB services.
3. /etc/felhom-dooplex-offsite/ (root 0700): env (no secret) and the new enc.key. The tokens are the existing /etc/felhom-hub-backup/token-push and token-restore (read, not copied).
4. Two alarm rules in homelab-manifests prometheus-rules + the Prometheus reload.
5. One manual run, one manual restore test, one dry failure mail.

On ep0: nothing. Reads only (the snapshot list).