R-886 opened (336); alarm proven end to end; REPORT-hub-db-offsite-2026-10-05

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 16:25:48 +02:00
parent afba622fb6
commit 53d8131b20
5 changed files with 151 additions and 4 deletions
@@ -3,3 +3,13 @@ backup_freshness.prom
 fan_metrics.prom
 felhom_hub_db_restore.prom
 node_housekeeping.prom
+## 2026-10-05T14:23:16Z HubDBBackupStale FIRING (activeAt 13:52:06Z; fired 14:22); Alertmanager: 1 active alert, receiver email-notifications (to: the operator's address); email notifications_total=5 failed_total=0 since the pod started 13:20:48Z
+## recovery push: unit Result=success
+felhom-hub-db-backup: snapshot hub-20261005T123037Z.db, 112 min old
+felhom-hub-db-backup: checked: 369807360 bytes, integrity ok, 4 host(s)
+felhom-hub-db-backup: pushed hub-20261005T123037Z.db to ep0 (ns operator) in 2 s
+felhom-hub-db-backup: success signal written
+felhom_hub_db_backup_last_success_timestamp_seconds 1791210207
+felhom_hub_db_backup_last_success_bytes 369807360
+## 2026-10-05T14:23:42Z HubDBBackupStale state=inactive
+email notifications_total now: 6