dooplex-offsite: failure mail survives the shared config's unset variable (found by the live dry run); Part B push + restore evidence
gates / gates (push) Successful in 5m6s
gates / gates (push) Successful in 5m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,12 @@
|
||||
## 2026-10-09T08:14:02Z dry failure: transient unit /bin/false with the same OnFailure= line
|
||||
-- No entries --
|
||||
## (transient units do not expand %n, so the instance is spelled out; the real units use %n — systemctl show above)
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: Started felhom-failmail-dryrun-r232.service - [systemd-run] /bin/false.
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: felhom-failmail-dryrun-r232.service: Main process exited, code=exited, status=1/FAILURE
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: felhom-failmail-dryrun-r232.service: Failed with result 'exit-code'.
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: felhom-failmail-dryrun-r232.service: Triggering OnFailure= dependencies.
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: Starting felhom-backup-failmail@felhom-failmail-dryrun-r232.service.service - Felhom: mail admin@ that felhom-failmail-dryrun-r232.service failed (R-232)...
|
||||
2026-10-09T10:14:08+02:00 dooplex felhom-backup-failmail[3030045]: /opt/backup/scripts/backup-config.sh: line 143: NOTIFY_WEBHOOK_URL: unbound variable
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: felhom-backup-failmail@felhom-failmail-dryrun-r232.service.service: Main process exited, code=exited, status=1/FAILURE
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: felhom-backup-failmail@felhom-failmail-dryrun-r232.service.service: Failed with result 'exit-code'.
|
||||
2026-10-09T10:14:08+02:00 dooplex systemd[1]: Failed to start felhom-backup-failmail@felhom-failmail-dryrun-r232.service.service - Felhom: mail admin@ that felhom-failmail-dryrun-r232.service failed (R-232).
|
||||
@@ -0,0 +1,16 @@
|
||||
## 2026-10-09T08:10:21Z install (felhom.eu 1707c928)
|
||||
install.sh: installed; timers NOT enabled (see the header)
|
||||
## key
|
||||
key_rc=0
|
||||
total 24
|
||||
drwx------ 2 root root 4096 Oct 9 10:10 .
|
||||
drwxr-xr-x 133 root root 12288 Oct 9 10:10 ..
|
||||
-rw------- 1 root root 255 Oct 9 10:10 enc.key
|
||||
-rw------- 1 root root 268 Oct 9 10:10 env
|
||||
600 root
|
||||
PBS_REPOSITORY_PUSH='dooplex-hub@pbs!push@127.0.0.1:18007:felhom-offsite'
|
||||
PBS_REPOSITORY_RESTORE='dooplex-hub@pbs!restore@127.0.0.1:18007:felhom-offsite'
|
||||
PBS_FINGERPRINT=<pinned, as hub-backup>
|
||||
OnFailure=felhom-backup-failmail@felhom-hub-db-backup.service.service
|
||||
|
||||
OnFailure=felhom-backup-failmail@felhom-dooplex-offsite.service.service
|
||||
@@ -0,0 +1,26 @@
|
||||
2026-10-09T10:10:34+02:00 dooplex systemd[1]: Starting felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232)...
|
||||
2026-10-09T10:10:35+02:00 dooplex felhom-dooplex-offsite[2995184]: felhom-dooplex-offsite: database: 20261009-040001, 6218173 bytes, 250 min old
|
||||
2026-10-09T10:11:53+02:00 dooplex felhom-dooplex-offsite[2995184]: felhom-dooplex-offsite: files: 10 repositories, 614 MB
|
||||
2026-10-09T10:11:53+02:00 dooplex felhom-dooplex-offsite[2995184]: felhom-dooplex-offsite: secrets: 3 file(s) of 20261009_031011
|
||||
2026-10-09T10:11:56+02:00 dooplex felhom-dooplex-offsite[3006280]: Starting backup: [operator]:host/dooplex-gitea/2026-10-09T08:11:56Z
|
||||
2026-10-09T10:11:56+02:00 dooplex felhom-dooplex-offsite[3006280]: Client name: dooplex
|
||||
2026-10-09T10:11:56+02:00 dooplex felhom-dooplex-offsite[3006280]: Starting backup protocol: Fri Oct 9 10:11:56 2026
|
||||
2026-10-09T10:11:56+02:00 dooplex felhom-dooplex-offsite[3006280]: Using encryption key from '/etc/felhom-dooplex-offsite/enc.key'..
|
||||
2026-10-09T10:11:56+02:00 dooplex felhom-dooplex-offsite[3006280]: Encryption key fingerprint: 93:03:bf:d7:1f:4c:9e:fe
|
||||
2026-10-09T10:11:56+02:00 dooplex felhom-dooplex-offsite[3006280]: No previous manifest available.
|
||||
2026-10-09T10:11:56+02:00 dooplex felhom-dooplex-offsite[3006280]: Upload directory '/var/lib/felhom-dooplex-offsite/stage/root' to 'dooplex-hub@pbs!push@127.0.0.1:18007:felhom-offsite' as dooplex.pxar.didx
|
||||
2026-10-09T10:12:51+02:00 dooplex felhom-dooplex-offsite[3006280]: dooplex.pxar: had to backup 540.664 MiB of 544.173 MiB (compressed 531.953 MiB) in 54.26 s (average 9.964 MiB/s)
|
||||
2026-10-09T10:12:51+02:00 dooplex felhom-dooplex-offsite[3006280]: dooplex.pxar: backup was done incrementally, reused 3.508 MiB (0.6%)
|
||||
2026-10-09T10:12:51+02:00 dooplex felhom-dooplex-offsite[3006280]: Uploaded backup catalog (1.25 MiB)
|
||||
2026-10-09T10:12:53+02:00 dooplex felhom-dooplex-offsite[3006280]: Duration: 56.87s
|
||||
2026-10-09T10:12:53+02:00 dooplex felhom-dooplex-offsite[3006280]: End Time: Fri Oct 9 10:12:53 2026
|
||||
2026-10-09T10:12:53+02:00 dooplex felhom-dooplex-offsite[2995184]: felhom-dooplex-offsite: pushed to ep0 (ns operator, host/dooplex-gitea) in 57 s
|
||||
2026-10-09T10:12:53+02:00 dooplex felhom-dooplex-offsite[2995184]: felhom-dooplex-offsite: success signal written
|
||||
2026-10-09T10:12:54+02:00 dooplex systemd[1]: felhom-dooplex-offsite.service: Deactivated successfully.
|
||||
2026-10-09T10:12:54+02:00 dooplex systemd[1]: Finished felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232).
|
||||
2026-10-09T10:12:54+02:00 dooplex systemd[1]: felhom-dooplex-offsite.service: Consumed 24.466s CPU time, 1.2G memory peak.
|
||||
# HELP felhom_dooplex_offsite_last_success_timestamp_seconds Last successful push of Gitea + DooPlex secrets to ep0 (R-232).
|
||||
# TYPE felhom_dooplex_offsite_last_success_timestamp_seconds gauge
|
||||
felhom_dooplex_offsite_last_success_timestamp_seconds 1791533573
|
||||
felhom_dooplex_offsite_last_success_bytes 566109337
|
||||
felhom_dooplex_offsite_last_success_repositories 10
|
||||
@@ -29,3 +29,11 @@ FAILED (failures=1)
|
||||
### GREEN, full suite
|
||||
Ran 20 tests in 35.065s
|
||||
OK
|
||||
### RED: failmail under set -u, config reads an unset variable
|
||||
FAIL: test_failmail_calls_notify_failure_with_the_unit_name (__main__.FailMail.test_failmail_calls_notify_failure_with_the_unit_name)
|
||||
AssertionError: 1 != 0 : /tmp/tmp48157fhy/cfg.sh: line 1: NOTIFY_WEBHOOK_URL_UNSET_IN_TEST: unbound variable
|
||||
Ran 2 tests in 0.007s
|
||||
FAILED (failures=1)
|
||||
### GREEN
|
||||
Ran 20 tests in 35.056s
|
||||
OK
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
2026-10-09T10:13:03+02:00 dooplex systemd[1]: Starting felhom-dooplex-offsite-restore-test.service - Felhom: restore-test the Gitea + secrets copy on ep0 (R-232)...
|
||||
2026-10-09T10:13:03+02:00 dooplex felhom-dooplex-offsite-restore-test[3016447]: felhom-dooplex-offsite-restore-test: restoring host/dooplex-gitea/2026-10-09T08:11:56Z
|
||||
2026-10-09T10:13:04+02:00 dooplex felhom-dooplex-offsite-restore-test[3016610]: Using encryption key from '/etc/felhom-dooplex-offsite/enc.key'..
|
||||
2026-10-09T10:13:04+02:00 dooplex felhom-dooplex-offsite-restore-test[3016610]: Fingerprint: 93:03:bf:d7:1f:4c:9e:fe
|
||||
2026-10-09T10:13:09+02:00 dooplex felhom-dooplex-offsite-restore-test[3016610]: progress 23% (126.51 MiB of 544.173 MiB in 5.3s, 24.049 MiB/s)
|
||||
2026-10-09T10:13:19+02:00 dooplex felhom-dooplex-offsite-restore-test[3016610]: progress 74% (404.7 MiB of 544.173 MiB in 15.3s, 27.637 MiB/s)
|
||||
2026-10-09T10:13:24+02:00 dooplex felhom-dooplex-offsite-restore-test[3016610]: restore complete (544.173 MiB processed in 20s, average 27.217 MiB/s)
|
||||
2026-10-09T10:13:43+02:00 dooplex felhom-dooplex-offsite-restore-test[3016447]: felhom-dooplex-offsite-restore-test: checked: 27805 files match the manifest, 10 repositories pass git fsck, gitea.dump readable, 3 secrets file(s)
|
||||
2026-10-09T10:13:43+02:00 dooplex felhom-dooplex-offsite-restore-test[3016447]: felhom-dooplex-offsite-restore-test: success signal written
|
||||
2026-10-09T10:13:44+02:00 dooplex systemd[1]: felhom-dooplex-offsite-restore-test.service: Deactivated successfully.
|
||||
2026-10-09T10:13:44+02:00 dooplex systemd[1]: Finished felhom-dooplex-offsite-restore-test.service - Felhom: restore-test the Gitea + secrets copy on ep0 (R-232).
|
||||
2026-10-09T10:13:44+02:00 dooplex systemd[1]: felhom-dooplex-offsite-restore-test.service: Consumed 29.822s CPU time, 834.6M memory peak.
|
||||
# HELP felhom_dooplex_offsite_restore_test_last_success_timestamp_seconds Last successful restore test of the Gitea + secrets copy on ep0 (R-232).
|
||||
# TYPE felhom_dooplex_offsite_restore_test_last_success_timestamp_seconds gauge
|
||||
felhom_dooplex_offsite_restore_test_last_success_timestamp_seconds 1791533623
|
||||
## ep0 filesystem (second channel, read-only ssh)
|
||||
/mnt/pbs-datastore/ns/operator/host/:
|
||||
total 16
|
||||
drwxr-xr-x 4 backup backup 4096 Oct 9 08:11 .
|
||||
drwxr-xr-x 3 backup backup 4096 Oct 5 13:50 ..
|
||||
drwxr-xr-x 3 backup backup 4096 Oct 9 08:11 dooplex-gitea
|
||||
drwxr-xr-x 7 backup backup 4096 Oct 9 00:31 dooplex-hub
|
||||
|
||||
/mnt/pbs-datastore/ns/operator/host/dooplex-gitea/:
|
||||
total 16
|
||||
drwxr-xr-x 3 backup backup 4096 Oct 9 08:11 .
|
||||
drwxr-xr-x 4 backup backup 4096 Oct 9 08:11 ..
|
||||
drwxr-xr-x 2 backup backup 4096 Oct 9 08:12 2026-10-09T08:11:56Z
|
||||
-rw-r--r-- 1 backup backup 21 Oct 9 08:11 owner
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/sdb 98G 16G 83G 16% /mnt/pbs-datastore
|
||||
## restore token cannot write / push token cannot forget
|
||||
Error: permission check failed - missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/operator
|
||||
forget_rc=0
|
||||
Reference in New Issue
Block a user