felhom.eu/documentation/runbooks/gitea-restore.md

# Runbook — bring Gitea back from the off-site copy on ep0 (R-232)

> **TESTED 2026-10-09** into a throwaway (the bench, LXC 9401 on demo-hp): 10 of 10 repositories listed, the four
> product repositories' `main` equal to live Gitea (one was one commit behind: that commit was pushed three minutes
> after the copy, and the copy's commit is its parent), one file byte for byte, a throwaway admin logged in. Restore
> from ep0: 544 MB in 22 s. Evidence: `audits/dooplex-survival-2026-10-09/partC/`. The copy itself:
> `audits/dooplex-survival-2026-10-09/PLAN.md` and `scripts/dooplex-offsite/`.

## What the copy holds

One encrypted archive `dooplex.pxar` per night in ep0's PBS, namespace `operator`, group `host/dooplex-gitea`
(14 daily + 8 weekly kept). Inside:

| Path | What |
|---|---|
| `db/gitea.dump` | `pg_dump -Fc` of the `gitea` database (PostgreSQL 17.2), taken BEFORE the files |
| `db/globals.sql`, `db/DUMP-FOLDER` | all roles of the CNPG cluster (password hashes — not needed for this restore); which dump |
| `gitea/git/repositories/<owner>/<repo>.git` | the bare repositories |
| `gitea/git/lfs`, `gitea/gitea/{attachments,avatars,repo-avatars,jwt}` | the rest of Gitea's data |
| `gitea/gitea/conf/app.ini` | the config, **with Gitea's secrets** (`SECRET_KEY`, `INTERNAL_TOKEN`, JWT, the DB password) |
| `secrets/*.gpg` | DooPlex's nightly k8s Secrets/ConfigMaps export, GPG-encrypted with DooPlex's restic passphrase |
| `MANIFEST.sha256`, `REPOS` | a checksum of every file; the repository count |

**Not in it:** the container registry (`/data/gitea/packages`, 27.7 GB). The images rebuild from the code.

## What you need

- **The key**: the `data` field of the paper key — **S1 on the break-glass sheet** (`break-glass-sheet.md`). Not the password manager: it runs on DooPlex and is itself inside this copy (R-923). Write
  `{"kdf": null, "created": "2026-01-01T00:00:00+00:00", "modified": "2026-01-01T00:00:00+00:00", "data": "<data>"}`
  to `enc.key` (root, `umask 077`). On DooPlex it is `/etc/felhom-dooplex-offsite/enc.key`.
- **A read-only token** for `dooplex-hub@pbs!restore` (DooPlex: `/etc/felhom-hub-backup/token-restore`), or ep0 root to
  mint one (`RUNBOOK-hub-db-offsite-backup.md` Step 2).
- **A route to ep0's PBS** (`127.0.0.1:18007` through DooPlex's tunnel, or ep0's 8007 over the WireGuard).
- A machine with Docker. For the secrets files: DooPlex's restic passphrase (operator, offline).

## Steps

1. **Restore the newest copy** (any machine with `proxmox-backup-client`):
   ```bash
   export PBS_PASSWORD_FILE=<token-restore file> PBS_FINGERPRINT=<ep0 cert fingerprint, /etc/felhom-hub-backup/env>
   R='dooplex-hub@pbs!restore@<ep0 PBS>:felhom-offsite'
   proxmox-backup-client snapshot list host/dooplex-gitea --ns operator --repository "$R"      # pick the newest
   umask 077; proxmox-backup-client restore host/dooplex-gitea/<time> dooplex.pxar ./out --ns operator --keyfile enc.key --repository "$R"
   (cd out && sha256sum -c MANIFEST.sha256 >/dev/null && echo manifest OK)
   ```
2. **The database.** `pg_restore` must be 17 (the dump is from 17.2):
   ```bash
   docker network create --internal gr-net        # a test: no route out. A real rebuild: a normal network
   docker run -d --name gr-db --network gr-net -e POSTGRES_PASSWORD=<pw> postgres:17.2
   docker exec -i gr-db psql -U postgres -c "CREATE ROLE gitea LOGIN PASSWORD '<gitea pw>'" -c "CREATE DATABASE gitea OWNER gitea"
   docker cp out/db/gitea.dump gr-db:/tmp/ && docker exec gr-db pg_restore -U postgres -d gitea --no-owner --role=gitea --exit-on-error /tmp/gitea.dump
   ```
   On a rebuilt DooPlex: restore into the CNPG cluster's `gitea` database instead (same `pg_restore` line).
3. **The config.** In `out/gitea/gitea/conf/app.ini`, `[database]`: `HOST` → the new database, `PASSWD` → `<gitea pw>`.
   For a test also `[mailer] ENABLED = false`. Keep every other key — `SECRET_KEY` and `INTERNAL_TOKEN` must be the old
   ones or Gitea cannot read its own stored secrets (2FA, tokens).
4. **Start Gitea** on the data, owned by uid 1000 (the image's `git` user):
   ```bash
   chown -R 1000:1000 out/gitea
   docker run -d --name gr-gitea --network gr-net -v "$PWD/out/gitea:/data" gitea/gitea:<the version live ran>
   docker exec gr-gitea wget -q -O - http://127.0.0.1:3000/api/healthz      # "status": "pass"
   ```
5. **Check it** (a throwaway admin for a test; the real admin's password works on a real rebuild):
   ```bash
   docker exec -u git gr-gitea gitea admin user create --admin --username restore-check --password <pw> \
     --email restore-check@example.invalid --must-change-password=false
   # /api/v1/repos/search?limit=50&private=true  → the count equals out/REPOS
   # /api/v1/repos/admin/<repo>/branches/main     → equals the last known main (git ls-remote of any clone)
   # /api/v1/repos/admin/felhom.eu/raw/CLAUDE.md?ref=<main> | sha256sum  → equals `git show <main>:CLAUDE.md | sha256sum`
   ```
6. **A test ends with teardown** — the copy holds Gitea's secrets:
   ```bash
   docker rm -f gr-gitea gr-db; docker network rm gr-net; docker rmi postgres:17.2 gitea/gitea:<ver>
   docker volume ls      # ⚠ postgres leaves an ANONYMOUS volume holding the restored database — remove it BY NAME
   find out -type f \( -name app.ini -o -name gitea.dump -o -name globals.sql -o -name '*.gpg' \) -exec shred -u {} +; rm -rf out enc.key
   ```
   Never `docker volume prune`: on a shared machine it deletes other volumes too.

## Gotchas found on 2026-10-09

- **The anonymous Postgres volume** survives `docker rm -f` (no `-v`). It held the restored database; found by counting
  volumes after the teardown, removed by name.
- `pg_restore --no-owner --role=gitea`: the dump's objects belong to `gitea` in the source too, but `--no-owner` avoids
  needing every role from `globals.sql`.
- The weekly restore test on DooPlex (`felhom-dooplex-offsite-restore-test`, Sun 05:30) checks the manifest, `git fsck`
  on every repository and `pg_restore --list` — it does not start Gitea. This runbook is the full test.