dooplex-offsite: failure mail survives the shared config's unset variable (found by the live dry run); Part B push + restore evidence
gates / gates (push) Successful in 5m6s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-10-09 10:15:10 +02:00
parent 1707c928a9
commit 02a54e26f5
8 changed files with 103 additions and 3 deletions
+2 -1
View File
@@ -9,7 +9,8 @@
- `felhom-dooplex-offsite-restore-test` (Sun 05:30, read-only token): manifest check, repo count, `git fsck` on every
repository through a scratch repo with a known config (the pod's `config` files are never read by root git),
`pg_restore --list`, secrets present.
- `felhom-backup-failmail@.service`: `OnFailure=` mail through R-232 (a)'s `notify_failure`; also added to both
- `felhom-backup-failmail@.service`: `OnFailure=` mail through R-232 (a)'s `notify_failure` (no `set -u`: the live dry run
found it dying on the shared config's unset `NOTIFY_WEBHOOK_URL`; red-proved); also added to both
hub-DB units (`scripts/hub-db-backup/*.service`).
- 20 tests (`test_dooplex_offsite.py`; fakes for kubectl/PBS/pg_restore, real git/tar), green with GNU and BusyBox
tools; red-proofs for 6 checks in `audits/dooplex-survival-2026-10-09/partB/red-proof.txt`.
@@ -2,7 +2,7 @@
# felhom-backup-failmail — mail admin@ that a Felhom backup unit on DooPlex failed (R-232 (a) route, reused).
# Called by felhom-backup-failmail@<unit>.service, which the backup units name in OnFailure=.
# Reuses notify_failure from /opt/backup/scripts/backup-config.sh (Resend; never prints the key; never fails).
set -u
# No `set -u`: backup-config.sh reads unset variables (NOTIFY_WEBHOOK_URL), which killed the first dry run.
UNIT=${1:?unit name}
CONFIG=${FELHOM_FAILMAIL_CONFIG:-/opt/backup/scripts/backup-config.sh}
# shellcheck source=/dev/null
@@ -313,7 +313,10 @@ class FailMail(unittest.TestCase):
t = tempfile.mkdtemp()
try:
cfg = os.path.join(t, "cfg.sh"); out = os.path.join(t, "out")
open(cfg, "w").write('notify_failure() { echo "$1" > %s; return 0; }\n' % out)
# Like the real backup-config.sh, it reads a variable that is unset (NOTIFY_WEBHOOK_URL is commented out
# there): the 2026-10-09 dry run found the mail script under `set -u` dying on exactly that line.
open(cfg, "w").write('notify_failure() { if [ -n "${NOTIFY_WEBHOOK_URL_UNSET_IN_TEST}" ]; then :; fi; '
'echo "$1" > %s; return 0; }\n' % out)
r = subprocess.run([FAILMAIL, "felhom-dooplex-offsite.service"], env=dict(os.environ, FELHOM_FAILMAIL_CONFIG=cfg),
capture_output=True, text=True)
self.assertEqual(r.returncode, 0, r.stderr)