adminandClaude Opus 5.5
1707c928a9
gates / gates (push) Successful in 5m4s
Part A plan + readings, Part E read-backs (R-861 a, R-518) in audits/dooplex-survival-2026-10-09/. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
98 lines
7.1 KiB
Markdown
98 lines
7.1 KiB
Markdown
# PLAN — an encrypted off-site copy of Gitea and DooPlex's secrets (R-232 (b), (h))
|
||
|
||
2026-10-09, Part A of the DooPlex-survival brief. Everything below was **measured read-only**; nothing was created.
|
||
Raw readings: `partA/readings.txt`.
|
||
|
||
## What is measured
|
||
|
||
| Thing | Size | Growth | Where it lives today |
|
||
|---|---|---|---|
|
||
| Gitea repositories (`/data/git/repositories`, 10 repos under `admin/`) | **614 MB** | ~600 MB in 8 months (oldest product repo 2026-02-11); felhom.eu.git 188 MB | Longhorn PVC `gitea-data` on `sdb1`; Longhorn backup on `sda1`, **retain=1**. Nothing else. |
|
||
| Gitea database (CNPG `postgresql`, db `gitea`) | 64 MB live, **6.2 MB** as a `pg_dump -Fc` | 5.62 → 6.22 MB in 4 days (~150 KB/day) | 6-hourly dumps in `/mnt/5_hdd/backup/postgresql/dumps/` (same disk as every backup) |
|
||
| Gitea config `app.ini` (holds `SECRET_KEY`, `INTERNAL_TOKEN`, JWT secrets) | 2 KB | — | in the PVC only |
|
||
| Gitea registry (`/data/gitea/packages`) | 27.7 GB | — | **Left out on purpose:** the images rebuild from the code. |
|
||
| DooPlex secrets set: the nightly GPG files (`secrets-`, `configmaps-`, `by-namespace-*.gpg`) | **4.4 MB per night** | flat (2,055,542 → 2,056,090 B in 12 days) | `sda1` only, 30 days |
|
||
|
||
**Correction to the 2026-08-06 recon (§7):** it said the Gitea repositories are "covered twice". The API mirror
|
||
(`backup/homelab-manifests/git-mirrors/`) holds **one** repository, `homelab-manifests.git` (7.1 MB). The four
|
||
product repositories have exactly one backup: the Longhorn copy with retain=1, on the same machine.
|
||
|
||
## The two places measured
|
||
|
||
| | ep0, namespace `operator` (where the hub DB goes) | A new Hetzner Storage Box sub-account |
|
||
|---|---|---|
|
||
| Free space | 83 GB of 98 GB (`/mnt/pbs-datastore`, 16 % used) | depends on the box; not measured |
|
||
| Write-only key | **Already exists.** Token `dooplex-hub@pbs!push` has `DatastoreBackup` on `/operator` only: it cannot forget a snapshot or list the households (measured 2026-10-05). A second read-only token `!restore` exists for the restore test. | the R-820 append-only pin; the sub-account shell can still `rm` unless the pin is right |
|
||
| Changes needed there | **None.** A new backup group (`host/dooplex-gitea`) in the same namespace needs no new token, ACL or prune job. The existing prune job `prune-operator-hubdb` (keep-daily 14, keep-weekly 8) covers every group in `operator`. | a new sub-account + key custody |
|
||
| Cost | 0 € extra | a sub-account (same Storage Box fee) |
|
||
| Return copy | ep0's nightly pull to DooPlex (`ep0-copy`, 05:00) copies it back too — ciphertext only | none |
|
||
|
||
**Pick: ep0 `operator`.** No change on ep0 at all, the same write-only token as the hub DB, the same prune and
|
||
the same alarm family. Worst case on ep0: a git repack every night makes each copy new chunks — 22 kept copies ×
|
||
0.65 GB ≈ 14 GB, inside the 83 GB free.
|
||
|
||
## What runs
|
||
|
||
A script and a timer on DooPlex, `felhom-dooplex-offsite` (versioned in `felhom.eu/scripts/dooplex-offsite/`,
|
||
installed by its `install.sh`, tests by hand like the hub DB's), **daily 00:20** (after the 00:00 database dump,
|
||
before the hub DB at 02:30):
|
||
|
||
1. **The database first:** the newest **complete** `gitea.dump` + `globals.sql` from the 6-hourly dumps (folder with
|
||
`SUCCESS`; refused if older than 7 h). This is PostgreSQL's own consistent snapshot.
|
||
2. **Then the files**, read from the running Gitea pod with `tar` over `kubectl exec`: `/data/git/repositories`,
|
||
`/data/git/lfs`, `/data/gitea/conf/app.ini`, attachments, avatars, `jwt`. **Not** packages, logs, indexers, queues, tmp.
|
||
3. **Then the secrets set:** the newest night's three `.gpg` files (already GPG-encrypted with DooPlex's restic
|
||
passphrase, which the operator holds offline).
|
||
4. A `MANIFEST.sha256` of everything, then `proxmox-backup-client backup` → `host/dooplex-gitea` in `operator`,
|
||
`--crypt-mode encrypt` with a **new key** `/etc/felhom-dooplex-offsite/enc.key`.
|
||
5. Stage shredded; a success timestamp (written only after the push returns 0) for Prometheus.
|
||
|
||
**Why not Gitea's own `gitea dump`:** Gitea's documentation says the instance "must be shutdown during backup" for
|
||
consistency, and `gitea dump` does not stop it either. A nightly stop costs CI and the registry a gap and adds a
|
||
restart that must come back. Instead the order makes the copy safe: the database is older than the repositories, so
|
||
every commit the database names is in the copy (git writes objects before it moves a ref). A repository newer than
|
||
its database is the normal state after a push and Gitea reads it. **The restore test catches the rest:** it runs
|
||
`git fsck` on every repository.
|
||
|
||
## Encryption and the key
|
||
|
||
- New key, generated on DooPlex as root: `proxmox-backup-client key create --kdf none` → `0600` file in
|
||
`/etc/felhom-dooplex-offsite/`, never printed by CC, never in chat.
|
||
- **The operator copies its paper form into the password manager** at his own terminal (not through `!` here):
|
||
`sudo proxmox-backup-client key paperkey /etc/felhom-dooplex-offsite/enc.key --output-format text` — the `data`
|
||
field is enough (as for the hub DB key, runbook Step 0).
|
||
- ep0 never sees the key. The secrets files inside are encrypted twice (GPG + PBS).
|
||
|
||
## Retention and who can delete
|
||
|
||
- ep0's existing `prune-operator-hubdb` (03:45): 14 daily + 8 weekly, per group. ep0's GC frees chunks.
|
||
- **Can delete:** only ep0's root (and that prune job). The push token cannot forget a snapshot; the restore token
|
||
cannot write. DooPlex's root cannot delete on ep0 with what it holds.
|
||
|
||
## Alarms and failure mail
|
||
|
||
- `DooplexGiteaOffsiteStale` (26 h, `absent()` included) and `DooplexGiteaRestoreTestStale` (8 days), same group as
|
||
`HubDBBackupStale`, proven with `promtool test rules` and a red-proof.
|
||
- Failure mail: an `OnFailure=` unit that calls the R-232 (a) `notify_failure` (Resend → admin@felhom.eu). The same
|
||
`OnFailure=` is added to the two hub-DB units. Proven with one dry failure (a unit that fails on purpose, no backup).
|
||
|
||
## The restore test
|
||
|
||
- **Weekly, Sunday 05:30**, with the read-only token: restore the newest copy to a temp dir, check `MANIFEST.sha256`,
|
||
`git fsck` every repository, `pg_restore --list gitea.dump`, then shred; success timestamp.
|
||
- **Once now (Part C):** restore into a throwaway machine, start Gitea there with no outside network, compare every
|
||
product repository's `main` with live Gitea, read one file byte for byte, log in with a throwaway admin. Steps
|
||
become `runbooks/gitea-restore.md`. The throwaway and every copy are deleted.
|
||
|
||
## What changes on DooPlex (needs the operator's yes)
|
||
|
||
1. `/usr/local/sbin/felhom-dooplex-offsite` + `felhom-dooplex-offsite-restore-test` (scripts, from git).
|
||
2. Units: `felhom-dooplex-offsite.{service,timer}`, `felhom-dooplex-offsite-restore-test.{service,timer}`,
|
||
`felhom-backup-failmail@.service`; one `OnFailure=` line added to the two hub-DB services.
|
||
3. `/etc/felhom-dooplex-offsite/` (root 0700): `env` (no secret) and the new `enc.key`. The tokens are the existing
|
||
`/etc/felhom-hub-backup/token-push` and `token-restore` (read, not copied).
|
||
4. Two alarm rules in `homelab-manifests` `prometheus-rules` + the Prometheus reload.
|
||
5. One manual run, one manual restore test, one dry failure mail.
|
||
|
||
**On ep0: nothing.** Reads only (the snapshot list).
|