felhom.eu/documentation/runbooks/gitea-restore.md

Runbook — bring Gitea back from the off-site copy on ep0 (R-232)

TESTED 2026-10-09 into a throwaway (the bench, LXC 9401 on demo-hp): 10 of 10 repositories listed, the four product repositories' main equal to live Gitea (one was one commit behind: that commit was pushed three minutes after the copy, and the copy's commit is its parent), one file byte for byte, a throwaway admin logged in. Restore from ep0: 544 MB in 22 s. Evidence: audits/dooplex-survival-2026-10-09/partC/. The copy itself: audits/dooplex-survival-2026-10-09/PLAN.md and scripts/dooplex-offsite/.

What the copy holds

One encrypted archive dooplex.pxar per night in ep0's PBS, namespace operator, group host/dooplex-gitea (14 daily + 8 weekly kept). Inside:

Path	What
db/gitea.dump	pg_dump -Fc of the gitea database (PostgreSQL 17.2), taken BEFORE the files
db/globals.sql, db/DUMP-FOLDER	all roles of the CNPG cluster (password hashes — not needed for this restore); which dump
gitea/git/repositories/<owner>/<repo>.git	the bare repositories
gitea/git/lfs, gitea/gitea/{attachments,avatars,repo-avatars,jwt}	the rest of Gitea's data
gitea/gitea/conf/app.ini	the config, with Gitea's secrets (SECRET_KEY, INTERNAL_TOKEN, JWT, the DB password)
secrets/*.gpg	DooPlex's nightly k8s Secrets/ConfigMaps export, GPG-encrypted with DooPlex's restic passphrase
MANIFEST.sha256, REPOS	a checksum of every file; the repository count

Not in it: the container registry (/data/gitea/packages, 27.7 GB). The images rebuild from the code.

What you need

• The key: the data field of the paper key — S1 on the break-glass sheet (break-glass-sheet.md). Not the password manager: it runs on DooPlex and is itself inside this copy (R-923). Write {"kdf": null, "created": "2026-01-01T00:00:00+00:00", "modified": "2026-01-01T00:00:00+00:00", "data": "<data>"} to enc.key (root, umask 077). On DooPlex it is /etc/felhom-dooplex-offsite/enc.key.
• A read-only token for dooplex-hub@pbs!restore (DooPlex: /etc/felhom-hub-backup/token-restore), or ep0 root to mint one (RUNBOOK-hub-db-offsite-backup.md Step 2).
• A route to ep0's PBS (127.0.0.1:18007 through DooPlex's tunnel, or ep0's 8007 over the WireGuard).
• A machine with Docker. For the secrets files: DooPlex's restic passphrase (operator, offline).

Steps

1. Restore the newest copy (any machine with proxmox-backup-client):

   export PBS_PASSWORD_FILE=<token-restore file> PBS_FINGERPRINT=<ep0 cert fingerprint, /etc/felhom-hub-backup/env>
   R='dooplex-hub@pbs!restore@<ep0 PBS>:felhom-offsite'
   proxmox-backup-client snapshot list host/dooplex-gitea --ns operator --repository "$R"      # pick the newest
   umask 077; proxmox-backup-client restore host/dooplex-gitea/<time> dooplex.pxar ./out --ns operator --keyfile enc.key --repository "$R"
   (cd out && sha256sum -c MANIFEST.sha256 >/dev/null && echo manifest OK)
   

2. The database. pg_restore must be 17 (the dump is from 17.2):

   docker network create --internal gr-net        # a test: no route out. A real rebuild: a normal network
   docker run -d --name gr-db --network gr-net -e POSTGRES_PASSWORD=<pw> postgres:17.2
   docker exec -i gr-db psql -U postgres -c "CREATE ROLE gitea LOGIN PASSWORD '<gitea pw>'" -c "CREATE DATABASE gitea OWNER gitea"
   docker cp out/db/gitea.dump gr-db:/tmp/ && docker exec gr-db pg_restore -U postgres -d gitea --no-owner --role=gitea --exit-on-error /tmp/gitea.dump
   

   On a rebuilt DooPlex: restore into the CNPG cluster's gitea database instead (same pg_restore line).
3. The config. In out/gitea/gitea/conf/app.ini, [database]: HOST → the new database, PASSWD → <gitea pw>. For a test also [mailer] ENABLED = false. Keep every other key — SECRET_KEY and INTERNAL_TOKEN must be the old ones or Gitea cannot read its own stored secrets (2FA, tokens).
4. Start Gitea on the data, owned by uid 1000 (the image's git user):

   chown -R 1000:1000 out/gitea
   docker run -d --name gr-gitea --network gr-net -v "$PWD/out/gitea:/data" gitea/gitea:<the version live ran>
   docker exec gr-gitea wget -q -O - http://127.0.0.1:3000/api/healthz      # "status": "pass"
   

5. Check it (a throwaway admin for a test; the real admin's password works on a real rebuild):

   docker exec -u git gr-gitea gitea admin user create --admin --username restore-check --password <pw> \
     --email restore-check@example.invalid --must-change-password=false
   # /api/v1/repos/search?limit=50&private=true  → the count equals out/REPOS
   # /api/v1/repos/admin/<repo>/branches/main     → equals the last known main (git ls-remote of any clone)
   # /api/v1/repos/admin/felhom.eu/raw/CLAUDE.md?ref=<main> | sha256sum  → equals `git show <main>:CLAUDE.md | sha256sum`
   

6. A test ends with teardown — the copy holds Gitea's secrets:

   docker rm -f gr-gitea gr-db; docker network rm gr-net; docker rmi postgres:17.2 gitea/gitea:<ver>
   docker volume ls      # ⚠ postgres leaves an ANONYMOUS volume holding the restored database — remove it BY NAME
   find out -type f \( -name app.ini -o -name gitea.dump -o -name globals.sql -o -name '*.gpg' \) -exec shred -u {} +; rm -rf out enc.key
   

   Never docker volume prune: on a shared machine it deletes other volumes too.

Gotchas found on 2026-10-09

• The anonymous Postgres volume survives docker rm -f (no -v). It held the restored database; found by counting volumes after the teardown, removed by name.
• pg_restore --no-owner --role=gitea: the dump's objects belong to gitea in the source too, but --no-owner avoids needing every role from globals.sql.
• The weekly restore test on DooPlex (felhom-dooplex-offsite-restore-test, Sun 05:30) checks the manifest, git fsck on every repository and pg_restore --list — it does not start Gitea. This runbook is the full test.