# Fix and proof, 2026-10-24T10:58:25Z

## homelab-manifests 531023d: CronJob reads postgres-superuser; ArgoCD sync of CronJob/postgresql-backup only
env after sync: PGUSER=postgres-superuser PGPASSWORD=postgres-superuser

## by-hand job postgresql-backup-manual-1043 (kubectl create job --from=cronjob/postgresql-backup)
total 22064
drwxr-xr-x  2 26 tape    4096 Oct 24 12:43 .
drwxr-xr-x 25 26 tape    4096 Oct 24 12:43 ..
-rw-r--r--  1 26 tape 5297319 Oct 24 12:43 authentik.dump
-rw-r--r--  1 26 tape  711473 Oct 24 12:43 calcom.dump
-rw-r--r--  1 26 tape   10258 Oct 24 12:43 calcom_saml.dump
-rw-r--r--  1 26 tape 8598648 Oct 24 12:43 gitea.dump
-rw-r--r--  1 26 tape    4016 Oct 24 12:43 globals.sql
-rw-r--r--  1 26 tape   90743 Oct 24 12:43 guacamole_db.dump
-rw-r--r--  1 26 tape 1367243 Oct 24 12:43 homarr.dump
-rw-r--r--  1 26 tape 3624770 Oct 24 12:43 outline.dump
-rw-r--r--  1 26 tape 1224943 Oct 24 12:43 paperless.dump
-rw-r--r--  1 26 tape    2754 Oct 24 12:43 postgres.dump
-rw-r--r--  1 26 tape       0 Oct 24 12:43 SUCCESS
-rw-r--r--  1 26 tape  594915 Oct 24 12:43 tandoor.dump
-rw-r--r--  1 26 tape  648505 Oct 24 12:43 wger.dump
-rw-r--r--  1 26 tape  382029 Oct 24 12:43 zipline.dump
gitea.dump pg_restore --list rc=0 entries=1114
authentik.dump pg_restore --list rc=0 entries=1660
paperless.dump pg_restore --list rc=0 entries=753
CronJob lastSuccessfulTime: 2026-10-24T10:43:55Z

## felhom-dooplex-offsite by hand + restore test
Oct 24 12:48:40 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: database: 20261024-104341, 8598648 bytes, 4 min old
Oct 24 12:49:30 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: files: 12 repositories, 813 MB
Oct 24 12:49:30 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: secrets: 3 file(s) of 20261024_031421
Oct 24 12:49:32 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: vaultwarden: integrity ok, 1 user(s), files: db.sqlite3 rsa_key.pem
Oct 24 12:49:32 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: signing: 3 key(s) with their public halves
Oct 24 12:50:20 dooplex felhom-dooplex-offsite[509023]: dooplex.pxar: had to backup 422.032 MiB of 750.918 MiB (compressed 413.468 MiB) in 43.59 s (average 9.681 MiB/s)
Oct 24 12:50:20 dooplex felhom-dooplex-offsite[509023]: dooplex.pxar: backup was done incrementally, reused 328.886 MiB (43.8%)
Oct 24 12:50:22 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: pushed to ep0 (ns operator, host/dooplex-gitea) in 46 s
Oct 24 12:50:22 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: success signal written
Oct 24 12:50:23 dooplex systemd[1]: Finished felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232).
Oct 24 12:50:40 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: restoring host/dooplex-gitea/2026-10-24T10:49:36Z
Oct 24 12:51:07 dooplex felhom-dooplex-offsite-restore-test[522066]: restore complete (750.918 MiB processed in 27.1s, average 27.745 MiB/s)
Oct 24 12:51:33 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: checked: 26456 files match the manifest, 12 repositories pass git fsck, gitea.dump readable, 3 secrets file(s), Vaultwarden 1 user(s) / 811 item(s), 3 signing key(s) match
Oct 24 12:51:33 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: success signal written
Oct 24 12:51:34 dooplex systemd[1]: Finished felhom-dooplex-offsite-restore-test.service - Felhom: restore-test the Gitea + secrets copy on ep0 (R-232).

## ep0 (read only): host/dooplex-gitea snapshot 2026-10-24T10:49:36Z present (catalog.pcat1.didx, dooplex.pxar.didx, index.json.blob, 10:50Z)

## alarms cleared (Prometheus ALERTS, 30 s step)
PostgresqlDumpStale last firing sample 10:44:00Z; DooplexGiteaOffsiteStale last firing sample 10:51:00Z
metric ages at 10:51Z: offsite 83 s, dump 470 s; Alertmanager /api/v2/alerts = [] at 10:51:52Z

## the restic sweep: dooplex-backup.timer 03:14 local; 2026-10-24 03:15:48 snapshot a801930d of /mnt/5_hdd/backup/postgresql/dumps ok; not run by hand (apply_retention would forget today's snapshot); the next run is the proof (dated check)
