# Part 1+2 diagnosis, read 2026-10-24T10:39:09Z

## felhom-dooplex-offsite journal 2026-10-24
Oct 24 00:20:48 dooplex systemd[1]: Starting felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232)...
Oct 24 00:20:48 dooplex felhom-dooplex-offsite[1455158]: felhom-dooplex-offsite: FAILED: newest complete dump 20261023-100001 is 12 h old (limit 7 h) — the dump CronJob stopped
Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Main process exited, code=exited, status=1/FAILURE
Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Failed with result 'exit-code'.
Oct 24 00:20:48 dooplex systemd[1]: Failed to start felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232).
Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Triggering OnFailure= dependencies.

## postgresql-backup jobs
NAME                         STATUS     COMPLETIONS   DURATION   AGE
postgresql-backup-29878440   Complete   1/1           17s        36h
postgresql-backup-29878800   Complete   1/1           17s        30h
postgresql-backup-29879160   Complete   1/1           13s        24h
postgresql-backup-29879880   Failed     0/1           12h        12h
postgresql-backup-29880240   Failed     0/1           6h39m      6h39m
postgresql-backup-29880600   Failed     0/1           39m        39m

## CronJob status
lastSchedule=2026-10-24T10:00:00Z lastSuccess=2026-10-23T10:00:13Z

## failed dump folders (globals.sql only, 0 bytes)
total 8
drwxr-xr-x  2 26 tape 4096 Oct 23 18:00 .
drwxr-xr-x 29 26 tape 4096 Oct 24 12:00 ..
-rw-r--r--  1 26 tape    0 Oct 23 18:00 globals.sql
total 8
drwxr-xr-x  2 26 tape 4096 Oct 24 12:00 .
drwxr-xr-x 29 26 tape 4096 Oct 24 12:00 ..
-rw-r--r--  1 26 tape    0 Oct 24 12:00 globals.sql

## database log: failed logins in 24 h (user, client, count) — first 2026-10-23 16:00:01 UTC; all four clients are backup job pods
message: password authentication failed for user "postgres" (sql_state 28P01)

## which secret's password does role postgres accept (scratchpad pgpw-check.sh; prints ok/refused only)
postgresql-superuser: refused   (the CNPG superuser secret; the CronJob reads this)
postgres-superuser: ok          (the bootstrap.initdb secret; owner=postgres)

## cluster status.secretsResourceVersion
{"applicationSecretVersion":"4427397","clientCaSecretVersion":"80646921","replicationSecretVersion":"80646924","serverCaSecretVersion":"80646921","serverSecretVersion":"80646922","superuserSecretVersion":"59321453"}

## instance start: postgresql-2 instance manager up 2026-10-23T13:50:14Z (window reboot); last good dump 10:00Z, first bad 16:00Z

## ruled out
tunnel: active since 2026-10-23 13:41:29Z (one 'No route to host' at boot, restart after 31 s); PBS via 127.0.0.1:18007 -> 200
nft inet hm040_blocklist: ioc_ipv4 = { 5.255.109.74, 5.255.115.190 } only; ep0 167.233.158.164 not in it
disk: Filesystem      Size  Used Avail Use% Mounted on
disk: /dev/sda1       9.1T  3.2T  5.4T  38% /mnt/5_hdd
disk: /dev/sdb1       445G  239G  184G  57% /
overlap: the job died at 00:20:48 in 0 s on the dump-age check, before any network use; release copy 01:40, hub copy 02:30 (UTC+2) do not overlap
