dooplex-offsite: failure mail survives the shared config's unset variable (found by the live dry run); Part B push + restore evidence
gates / gates (push) Successful in 5m6s
gates / gates (push) Successful in 5m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -9,7 +9,8 @@
|
||||
- `felhom-dooplex-offsite-restore-test` (Sun 05:30, read-only token): manifest check, repo count, `git fsck` on every
|
||||
repository through a scratch repo with a known config (the pod's `config` files are never read by root git),
|
||||
`pg_restore --list`, secrets present.
|
||||
- `felhom-backup-failmail@.service`: `OnFailure=` mail through R-232 (a)'s `notify_failure`; also added to both
|
||||
- `felhom-backup-failmail@.service`: `OnFailure=` mail through R-232 (a)'s `notify_failure` (no `set -u`: the live dry run
|
||||
found it dying on the shared config's unset `NOTIFY_WEBHOOK_URL`; red-proved); also added to both
|
||||
hub-DB units (`scripts/hub-db-backup/*.service`).
|
||||
- 20 tests (`test_dooplex_offsite.py`; fakes for kubectl/PBS/pg_restore, real git/tar), green with GNU and BusyBox
|
||||
tools; red-proofs for 6 checks in `audits/dooplex-survival-2026-10-09/partB/red-proof.txt`.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
# felhom-backup-failmail — mail admin@ that a Felhom backup unit on DooPlex failed (R-232 (a) route, reused).
|
||||
# Called by felhom-backup-failmail@<unit>.service, which the backup units name in OnFailure=.
|
||||
# Reuses notify_failure from /opt/backup/scripts/backup-config.sh (Resend; never prints the key; never fails).
|
||||
set -u
|
||||
# No `set -u`: backup-config.sh reads unset variables (NOTIFY_WEBHOOK_URL), which killed the first dry run.
|
||||
UNIT=${1:?unit name}
|
||||
CONFIG=${FELHOM_FAILMAIL_CONFIG:-/opt/backup/scripts/backup-config.sh}
|
||||
# shellcheck source=/dev/null
|
||||
|
||||
@@ -313,7 +313,10 @@ class FailMail(unittest.TestCase):
|
||||
t = tempfile.mkdtemp()
|
||||
try:
|
||||
cfg = os.path.join(t, "cfg.sh"); out = os.path.join(t, "out")
|
||||
open(cfg, "w").write('notify_failure() { echo "$1" > %s; return 0; }\n' % out)
|
||||
# Like the real backup-config.sh, it reads a variable that is unset (NOTIFY_WEBHOOK_URL is commented out
|
||||
# there): the 2026-10-09 dry run found the mail script under `set -u` dying on exactly that line.
|
||||
open(cfg, "w").write('notify_failure() { if [ -n "${NOTIFY_WEBHOOK_URL_UNSET_IN_TEST}" ]; then :; fi; '
|
||||
'echo "$1" > %s; return 0; }\n' % out)
|
||||
r = subprocess.run([FAILMAIL, "felhom-dooplex-offsite.service"], env=dict(os.environ, FELHOM_FAILMAIL_CONFIG=cfg),
|
||||
capture_output=True, text=True)
|
||||
self.assertEqual(r.returncode, 0, r.stderr)
|
||||
|
||||
Reference in New Issue
Block a user