admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Decisions 157-158 recorded; R-892 corrected (VM 341), R-894 filed (unreadable off-site storage reads due after an agent restart); Part C/E/F evidence; nodes.md: the Tester 1 box; 11 §5.8: the memory-kill check
gates / gates (push) Successful in 2m47s
Details

Browse Source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
authentik Default Admin
2026-10-06 19:18:14 +02:00
parent eed1dbd80c
commit 01c4a5d2a9
30 changed files with 845 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/readback-2026-10-07/C/C1-demo-hp-agent-pbs-lines.txt
+133
View File
@@ -0,0 +1,133 @@
Sep 22 04:29:29 demo-hp felhom-agent[1195783]: time=2026-09-22T04:29:29.642+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-15T04:11:51Z) is already proven; local: newest settled archive (landed 2026-09-20T04:43:47Z) is already proven"
Sep 22 06:12:46 demo-hp felhom-agent[1195783]: time=2026-09-22T06:12:46.231+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1790050365164112326
Sep 22 06:16:22 demo-hp felhom-agent[1195783]: time=2026-09-22T06:16:22.067+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z size_bytes=18418647089 uncovered_volumes=2
Sep 22 06:16:22 demo-hp felhom-agent[1195783]: time=2026-09-22T06:16:22.067+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1790050365164112326 archive=felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z
Sep 22 06:58:40 demo-hp felhom-agent[1195783]: time=2026-09-22T06:58:40.652+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst size_bytes=6267269544 uncovered_volumes=2
Sep 22 06:58:40 demo-hp felhom-agent[1195783]: time=2026-09-22T06:58:40.652+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790052768442108602 archive=local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst
Sep 22 10:29:29 demo-hp felhom-agent[1195783]: time=2026-09-22T10:29:29.346+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_09_21-06_47_30.tar.zst landed=2026-09-21T04:47:30Z reason="newest settled archive (landed 2026-09-21T04:47:30Z) has not been proven (last proven archive was a dif
Sep 22 10:34:49 demo-hp felhom-agent[1195783]: time=2026-09-22T10:34:49.417+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_09_21-06_47_30.tar.zst duration_s=320.064619459
Sep 22 16:29:29 demo-hp felhom-agent[1195783]: time=2026-09-22T16:29:29.663+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-15T04:11:51Z) is already proven; local: newest settled archive (landed 2026-09-21T04:47:30Z) is already proven"
Sep 22 22:29:29 demo-hp felhom-agent[1195783]: time=2026-09-22T22:29:29.659+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-15T04:11:51Z) is already proven; local: newest settled archive (landed 2026-09-21T04:47:30Z) is already proven"
Sep 23 04:29:29 demo-hp felhom-agent[1195783]: time=2026-09-23T04:29:29.655+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-15T04:11:51Z) is already proven; local: newest settled archive (landed 2026-09-21T04:47:30Z) is already proven"
Sep 23 07:02:53 demo-hp felhom-agent[1195783]: time=2026-09-23T07:02:53.924+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_23-06_55_25.tar.zst size_bytes=7417540996 uncovered_volumes=2
Sep 23 07:02:53 demo-hp felhom-agent[1195783]: time=2026-09-23T07:02:53.924+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790139325067362818 archive=local:backup/vzdump-lxc-9201-2026_09_23-06_55_25.tar.zst
Sep 23 10:29:29 demo-hp felhom-agent[1195783]: time=2026-09-23T10:29:29.358+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z landed=2026-09-22T04:12:45Z reason="newest settled archive (landed 2026-09-22T04:12:45Z) has not been proven (last proven archive was a differen
Sep 23 10:37:41 demo-hp felhom-agent[1195783]: time=2026-09-23T10:37:41.197+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=felhom-pbs:backup/ct/9201/2026-09-22T04:12:45Z duration_s=491.830022319
Sep 23 16:29:29 demo-hp felhom-agent[1195783]: time=2026-09-23T16:29:29.364+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst landed=2026-09-22T04:52:48Z reason="newest settled archive (landed 2026-09-22T04:52:48Z) has not been proven (last proven archive was a dif
Sep 23 16:35:15 demo-hp felhom-agent[1195783]: time=2026-09-23T16:35:15.142+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_09_22-06_52_48.tar.zst duration_s=345.775360191
Sep 23 22:29:29 demo-hp felhom-agent[1195783]: time=2026-09-23T22:29:29.632+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-22T04:12:45Z) is already proven; local: newest settled archive (landed 2026-09-22T04:52:48Z) is already proven"
Sep 24 04:29:29 demo-hp felhom-agent[1195783]: time=2026-09-24T04:29:29.641+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-22T04:12:45Z) is already proven; local: newest settled archive (landed 2026-09-22T04:52:48Z) is already proven"
Sep 24 07:05:52 demo-hp felhom-agent[1195783]: time=2026-09-24T07:05:52.441+02:00 level=ERROR msg="local-api: backup job failed" vmid=9201 target=local job=backup-9201-1790225968108254908 err="backup: vzdump task vmid 9201: proxmox: task UPID:demo-hp:001C2729:11475EED:6AB4AE30:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"job errors\""
Sep 24 07:24:21 demo-hp felhom-agent[1195783]: time=2026-09-24T07:24:21.352+02:00 level=ERROR msg="local-api: backup job failed" vmid=9201 target=local job=backup-9201-1790227073555655065 err="backup: vzdump task vmid 9201: proxmox: task UPID:demo-hp:001D29F4:11490EBD:6AB4B281:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"job errors\""
Sep 24 07:49:16 demo-hp felhom-agent[1195783]: time=2026-09-24T07:49:16.829+02:00 level=ERROR msg="local-api: backup job failed" vmid=9201 target=local job=backup-9201-1790228574136769346 err="backup: vzdump task vmid 9201: proxmox: task UPID:demo-hp:001E7D73:114B58E7:6AB4B85E:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"job errors\""
Sep 24 08:22:57 demo-hp felhom-agent[1195783]: time=2026-09-24T08:22:57.653+02:00 level=ERROR msg="local-api: backup job failed" vmid=9201 target=local job=backup-9201-1790230975578595274 err="backup: vzdump task vmid 9201: proxmox: task UPID:demo-hp:0020B26F:114F02F7:6AB4C1BF:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"job errors\""
Sep 24 10:29:29 demo-hp felhom-agent[1195783]: time=2026-09-24T10:29:29.327+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_09_23-06_55_25.tar.zst landed=2026-09-23T04:55:25Z reason="newest settled archive (landed 2026-09-23T04:55:25Z) has not been proven (last proven archive was a dif
Sep 24 10:36:30 demo-hp felhom-agent[1195783]: time=2026-09-24T10:36:30.331+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=local:backup/vzdump-lxc-9201-2026_09_23-06_55_25.tar.zst err="reconcile: restore-test start task: proxmox: task UPID:demo-hp:002789C4:115B3A8C:6AB4E106:vzstart:990000:felhom-agent@pve!agent: failed: exitstatus \"startup for container '990000' failed\""
Sep 24 12:47:34 demo-hp felhom-agent[1195783]: time=2026-09-24T12:47:34.646+02:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content -> HTTP 500: {\"data\":null,\"message\":\"felhom-pbs: error fetching datastores - 500 Can't connect to 10.77.
Sep 24 13:10:04 demo-hp felhom-agent[3134463]: time=2026-09-24T13:10:04.268+02:00 level=INFO msg="stale-lock: removed dangling vzdump snapshot" vmid=9201
Sep 24 22:06:23 demo-hp felhom-agent[756657]: time=2026-09-24T22:06:23.065+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst size_bytes=8182759056 uncovered_volumes=2
Sep 24 22:06:23 demo-hp felhom-agent[756657]: time=2026-09-24T22:06:23.065+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790279965190292701 archive=local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst
Sep 24 22:06:26 demo-hp felhom-agent[756657]: time=2026-09-24T22:06:26.321+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1790280385256228451
Sep 24 22:06:45 demo-hp felhom-agent[756657]: time=2026-09-24T22:06:45.066+02:00 level=WARN msg="backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup" target=felhom-pbs vmid=9201 volid=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z size_bytes=1 reason="size 1 B is below the 1048576 B plausibility floor — an aborted/incomplete archive, not a succe
Sep 24 22:07:45 demo-hp felhom-agent[756657]: time=2026-09-24T22:07:45.839+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:felhom-pbs
Sep 24 22:10:57 demo-hp felhom-agent[756657]: time=2026-09-24T22:10:57.339+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z size_bytes=23236593106 uncovered_volumes=2
Sep 24 22:10:57 demo-hp felhom-agent[756657]: time=2026-09-24T22:10:57.339+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1790280385256228451 archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z
Sep 25 22:57:53 demo-hp felhom-agent[995644]: time=2026-09-25T22:57:53.343+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different
Sep 25 22:57:54 demo-hp felhom-agent[995644]: time=2026-09-25T22:57:54.955+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620982 avail_bytes=23108536763 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has
Sep 25 22:57:54 demo-hp felhom-agent[995644]: time=2026-09-25T22:57:54.955+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.5 GiB"
Sep 26 04:38:34 demo-hp felhom-agent[995644]: time=2026-09-26T04:38:34.211+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_26-04_30_46.tar.zst size_bytes=8384567204 uncovered_volumes=2
Sep 26 04:38:34 demo-hp felhom-agent[995644]: time=2026-09-26T04:38:34.211+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790389845916353490 archive=local:backup/vzdump-lxc-9201-2026_09_26-04_30_46.tar.zst
Sep 26 04:57:53 demo-hp felhom-agent[995644]: time=2026-09-26T04:57:53.350+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different
Sep 26 04:57:55 demo-hp felhom-agent[995644]: time=2026-09-26T04:57:55.007+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620981 avail_bytes=23079614940 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has
Sep 26 04:57:55 demo-hp felhom-agent[995644]: time=2026-09-26T04:57:55.007+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.5 GiB"
Sep 26 10:57:53 demo-hp felhom-agent[995644]: time=2026-09-26T10:57:53.346+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different
Sep 26 10:57:54 demo-hp felhom-agent[995644]: time=2026-09-26T10:57:54.951+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620982 avail_bytes=23068046210 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has
Sep 26 10:57:54 demo-hp felhom-agent[995644]: time=2026-09-26T10:57:54.951+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.5 GiB"
Sep 26 16:57:53 demo-hp felhom-agent[995644]: time=2026-09-26T16:57:53.359+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different
Sep 26 16:57:55 demo-hp felhom-agent[995644]: time=2026-09-26T16:57:55.016+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620982 avail_bytes=23056477481 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has
Sep 26 16:57:55 demo-hp felhom-agent[995644]: time=2026-09-26T16:57:55.017+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.5 GiB"
Sep 26 22:57:53 demo-hp felhom-agent[995644]: time=2026-09-26T22:57:53.370+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different
Sep 26 22:57:55 demo-hp felhom-agent[995644]: time=2026-09-26T22:57:55.013+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620982 avail_bytes=23044908752 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has
Sep 26 22:57:55 demo-hp felhom-agent[995644]: time=2026-09-26T22:57:55.013+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.5 GiB"
Sep 27 04:43:29 demo-hp felhom-agent[995644]: time=2026-09-27T04:43:29.138+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst size_bytes=8380885161 uncovered_volumes=2
Sep 27 04:43:29 demo-hp felhom-agent[995644]: time=2026-09-27T04:43:29.138+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790476546353426458 archive=local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst
Sep 27 04:57:53 demo-hp felhom-agent[995644]: time=2026-09-27T04:57:53.352+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different
Sep 27 04:57:54 demo-hp felhom-agent[995644]: time=2026-09-27T04:57:54.985+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620981 avail_bytes=22663140685 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has
Sep 27 04:57:54 demo-hp felhom-agent[995644]: time=2026-09-27T04:57:54.985+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.1 GiB"
Sep 27 10:57:53 demo-hp felhom-agent[995644]: time=2026-09-27T10:57:53.351+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a different
Sep 27 10:57:54 demo-hp felhom-agent[995644]: time=2026-09-27T10:57:54.968+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620981 avail_bytes=22657356320 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has
Sep 27 10:57:54 demo-hp felhom-agent[995644]: time=2026-09-27T10:57:54.968+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.1 GiB"
Sep 27 20:13:17 demo-hp felhom-agent[1370926]: time=2026-09-27T20:13:17.853+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a differen
Sep 27 20:13:19 demo-hp felhom-agent[1370926]: time=2026-09-27T20:13:19.466+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620981 avail_bytes=22657356320 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, ha
Sep 27 20:13:19 demo-hp felhom-agent[1370926]: time=2026-09-27T20:13:19.467+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.1 GiB"
Sep 28 02:13:17 demo-hp felhom-agent[1370926]: time=2026-09-28T02:13:17.870+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a differen
Sep 28 02:13:19 demo-hp felhom-agent[1370926]: time=2026-09-28T02:13:19.630+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620982 avail_bytes=22657356320 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, ha
Sep 28 02:13:19 demo-hp felhom-agent[1370926]: time=2026-09-28T02:13:19.630+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 21.1 GiB"
Sep 28 04:44:58 demo-hp felhom-agent[1370926]: time=2026-09-28T04:44:58.683+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_28-04_37_06.tar.zst size_bytes=8825181955 uncovered_volumes=2
Sep 28 04:44:58 demo-hp felhom-agent[1370926]: time=2026-09-28T04:44:58.683+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790563025947283530 archive=local:backup/vzdump-lxc-9201-2026_09_28-04_37_06.tar.zst
Sep 28 08:13:17 demo-hp felhom-agent[1370926]: time=2026-09-28T08:13:17.866+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a differen
Sep 28 08:13:19 demo-hp felhom-agent[1370926]: time=2026-09-28T08:13:19.655+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620982 avail_bytes=21801270353 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, ha
Sep 28 08:13:19 demo-hp felhom-agent[1370926]: time=2026-09-28T08:13:19.655+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 20.3 GiB"
Sep 28 14:13:17 demo-hp felhom-agent[1370926]: time=2026-09-28T14:13:17.848+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a differen
Sep 28 14:13:19 demo-hp felhom-agent[1370926]: time=2026-09-28T14:13:19.532+02:00 level=WARN msg="restore-test SKIPPED by the space preflight (R-672) — nothing was created" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z storage=local-lvm required_bytes=33252620982 avail_bytes=25653657207 reason="not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, ha
Sep 28 14:13:19 demo-hp felhom-agent[1370926]: time=2026-09-28T14:13:19.533+02:00 level=ERROR msg="backup: scheduled restore-test FAILED" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z err="skipped: not enough space on local-lvm: restoring 21.6 GiB (pbs snapshot size) needs 31.0 GiB free, has 23.9 GiB"
Sep 28 22:25:23 demo-hp felhom-agent[2355282]: time=2026-09-28T22:25:23.861+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z landed=2026-09-24T20:06:25Z reason="newest settled archive (landed 2026-09-24T20:06:25Z) has not been proven (last proven archive was a differen
Sep 28 22:33:59 demo-hp felhom-agent[2355282]: time=2026-09-28T22:33:59.872+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=felhom-pbs:backup/ct/9201/2026-09-24T20:06:25Z duration_s=516.008133181
Sep 29 04:25:24 demo-hp felhom-agent[2355282]: time=2026-09-29T04:25:24.198+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven"
Sep 29 04:50:53 demo-hp felhom-agent[2355282]: time=2026-09-29T04:50:53.815+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_29-04_42_11.tar.zst size_bytes=9516315002 uncovered_volumes=2
Sep 29 04:50:53 demo-hp felhom-agent[2355282]: time=2026-09-29T04:50:53.815+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790649730829915100 archive=local:backup/vzdump-lxc-9201-2026_09_29-04_42_11.tar.zst
Sep 29 10:25:24 demo-hp felhom-agent[2355282]: time=2026-09-29T10:25:24.184+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven"
Sep 29 16:25:24 demo-hp felhom-agent[2355282]: time=2026-09-29T16:25:24.219+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven"
Sep 29 22:25:24 demo-hp felhom-agent[2355282]: time=2026-09-29T22:25:24.194+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven"
Sep 30 04:25:24 demo-hp felhom-agent[2355282]: time=2026-09-30T04:25:24.194+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven"
Sep 30 10:25:23 demo-hp felhom-agent[2355282]: time=2026-09-30T10:25:23.855+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_09_29-04_42_11.tar.zst landed=2026-09-29T02:42:11Z reason="newest settled archive (landed 2026-09-29T02:42:11Z) has not been proven (last proven archive was a dif
Sep 30 10:27:48 demo-hp felhom-agent[2355282]: time=2026-09-30T10:27:48.373+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_09_29-04_42_11.tar.zst duration_s=144.515446972
Sep 30 13:47:12 demo-hp felhom-agent[2135127]: time=2026-09-30T13:47:12.666+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_09_30-13_37_38.tar.zst size_bytes=9978959837 uncovered_volumes=2
Sep 30 13:47:12 demo-hp felhom-agent[2135127]: time=2026-09-30T13:47:12.666+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790768258239656621 archive=local:backup/vzdump-lxc-9201-2026_09_30-13_37_38.tar.zst
Sep 30 16:40:32 demo-hp felhom-agent[2135127]: time=2026-09-30T16:40:32.449+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Sep 30 22:40:32 demo-hp felhom-agent[2135127]: time=2026-09-30T22:40:32.547+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Oct 01 04:40:32 demo-hp felhom-agent[2135127]: time=2026-10-01T04:40:32.442+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Oct 01 08:11:08 demo-hp felhom-agent[2135127]: time=2026-10-01T08:11:08.130+02:00 level=WARN msg="local-api: storage view unavailable for the backup-target presence check — assuming present" target=felhom-pbs err="storage: NodeStorage: proxmox: GET /nodes/demo-hp/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage\": context canceled"
Oct 01 08:11:08 demo-hp felhom-agent[2135127]: time=2026-10-01T08:11:08.130+02:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/felhom-pbs/content\": context canceled"
Oct 01 10:40:32 demo-hp felhom-agent[2135127]: time=2026-10-01T10:40:32.422+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Oct 01 16:40:32 demo-hp felhom-agent[2135127]: time=2026-10-01T16:40:32.078+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_09_30-13_37_38.tar.zst landed=2026-09-30T11:37:38Z reason="newest settled archive (landed 2026-09-30T11:37:38Z) has not been proven (last proven archive was a dif
Oct 01 16:43:11 demo-hp felhom-agent[2135127]: time=2026-10-01T16:43:11.704+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_09_30-13_37_38.tar.zst duration_s=159.623154174
Oct 01 22:15:25 demo-hp felhom-agent[2135127]: time=2026-10-01T22:15:25.563+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_01-22_10_48.tar.zst size_bytes=4681052311 uncovered_volumes=2
Oct 01 22:15:25 demo-hp felhom-agent[2135127]: time=2026-10-01T22:15:25.563+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790885448142698430 archive=local:backup/vzdump-lxc-9201-2026_10_01-22_10_48.tar.zst
Oct 01 22:15:30 demo-hp felhom-agent[2135127]: time=2026-10-01T22:15:30.441+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1790885728244905026
Oct 01 22:18:31 demo-hp felhom-agent[2135127]: time=2026-10-01T22:18:31.284+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-10-01T20:15:29Z size_bytes=15223817181 uncovered_volumes=2
Oct 01 22:18:31 demo-hp felhom-agent[2135127]: time=2026-10-01T22:18:31.284+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1790885728244905026 archive=felhom-pbs:backup/ct/9201/2026-10-01T20:15:29Z
Oct 01 22:40:32 demo-hp felhom-agent[2135127]: time=2026-10-01T22:40:32.400+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Oct 02 04:40:32 demo-hp felhom-agent[2135127]: time=2026-10-02T04:40:32.387+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Oct 02 10:40:32 demo-hp felhom-agent[2135127]: time=2026-10-02T10:40:32.460+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Oct 02 16:40:32 demo-hp felhom-agent[2135127]: time=2026-10-02T16:40:32.406+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-09-24T20:06:25Z) is already proven; local: no settled archive yet — nothing to prove (newborn or still settling)"
Oct 02 22:40:32 demo-hp felhom-agent[2135127]: time=2026-10-02T22:40:32.065+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-10-01T20:15:29Z landed=2026-10-01T20:15:29Z reason="newest settled archive (landed 2026-10-01T20:15:29Z) has not been proven (last proven archive was a differen
Oct 02 22:46:02 demo-hp felhom-agent[2135127]: time=2026-10-02T22:46:02.535+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=felhom-pbs:backup/ct/9201/2026-10-01T20:15:29Z duration_s=330.466115271
Oct 03 04:38:20 demo-hp felhom-agent[2135127]: time=2026-10-03T04:38:20.266+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_03-04_33_33.tar.zst size_bytes=4674947408 uncovered_volumes=2
Oct 03 04:38:20 demo-hp felhom-agent[2135127]: time=2026-10-03T04:38:20.266+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1790994812809499660 archive=local:backup/vzdump-lxc-9201-2026_10_03-04_33_33.tar.zst
Oct 03 04:40:32 demo-hp felhom-agent[2135127]: time=2026-10-03T04:40:32.407+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 03 10:40:32 demo-hp felhom-agent[2135127]: time=2026-10-03T10:40:32.425+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 03 16:40:32 demo-hp felhom-agent[2135127]: time=2026-10-03T16:40:32.455+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 03 22:40:32 demo-hp felhom-agent[2135127]: time=2026-10-03T22:40:32.411+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 04 04:39:41 demo-hp felhom-agent[2135127]: time=2026-10-04T04:39:41.809+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst size_bytes=4679944932 uncovered_volumes=2
Oct 04 04:39:41 demo-hp felhom-agent[2135127]: time=2026-10-04T04:39:41.809+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1791081294394153713 archive=local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst
Oct 04 04:40:32 demo-hp felhom-agent[2135127]: time=2026-10-04T04:40:32.412+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 05 01:49:22 demo-hp felhom-agent[447632]: time=2026-10-05T01:49:22.786+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 05 04:40:16 demo-hp felhom-agent[447632]: time=2026-10-05T04:40:16.647+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_05-04_35_24.tar.zst size_bytes=4882309520 uncovered_volumes=2
Oct 05 04:40:16 demo-hp felhom-agent[447632]: time=2026-10-05T04:40:16.647+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1791167723831350602 archive=local:backup/vzdump-lxc-9201-2026_10_05-04_35_24.tar.zst
Oct 05 06:25:36 demo-hp felhom-agent[2031788]: time=2026-10-05T06:25:36.366+02:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content -> HTTP 500: {\"data\":null,\"message\":\"felhom-pbs: error fetching datastores - 500 Can't connect to 10.77.
Oct 05 06:26:23 demo-hp felhom-agent[2031788]: time=2026-10-05T06:26:23.344+02:00 level=ERROR msg="local-api: backup job failed" vmid=9201 target=felhom-pbs job=backup-9201-felhom-pbs-1791174359051131871 err="backup: vzdump task vmid 9201: proxmox: task UPID:demo-hp:0022DC8C:00483400:6AC326DF:vzdump:9201:felhom-agent@pve!agent: failed: exitstatus \"could not activate storage 'felhom-pbs': felhom-p
Oct 05 06:30:36 demo-hp felhom-agent[2031788]: time=2026-10-05T06:30:36.395+02:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content -> HTTP 500: {\"message\":\"felhom-pbs: error fetching datastores - 500 Can't connect to 10.77.0.1:8007 (Conn
Oct 05 06:35:36 demo-hp felhom-agent[2031788]: time=2026-10-05T06:35:36.377+02:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content -> HTTP 500: {\"data\":null,\"message\":\"felhom-pbs: error fetching datastores - 500 Can't connect to 10.77.
Oct 05 10:26:59 demo-hp felhom-agent[1323]: time=2026-10-05T10:26:59.170+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 05 11:24:01 demo-hp felhom-agent[1323]: time=2026-10-05T11:24:01.884+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_05-11_19_30.tar.zst size_bytes=4880501245 uncovered_volumes=2
Oct 05 11:24:01 demo-hp felhom-agent[1323]: time=2026-10-05T11:24:01.884+02:00 level=INFO msg="local-api: backup job complete" vmid=9201 target=local job=backup-9201-1791191969558324187 archive=local:backup/vzdump-lxc-9201-2026_10_05-11_19_30.tar.zst
Oct 05 11:24:09 demo-hp felhom-agent[1323]: time=2026-10-05T11:24:09.634+02:00 level=INFO msg="local-api: backup refused — a heavy operation is already in flight" vmid=9201 requested_target=felhom-pbs busy=backup:local
Oct 05 12:57:08 demo-hp felhom-agent[453394]: time=2026-10-05T12:57:08.743+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 05 18:57:09 demo-hp felhom-agent[453394]: time=2026-10-05T18:57:09.755+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 05 19:42:24 demo-hp felhom-agent[1625894]: time=2026-10-05T19:42:24.219+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 06 01:12:37 demo-hp felhom-agent[2556733]: time=2026-10-06T01:12:37.224+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 06 07:12:38 demo-hp felhom-agent[2556733]: time=2026-10-06T07:12:38.110+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="local: no settled archive yet — nothing to prove (newborn or still settling); felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven"
Oct 06 12:21:17 demo-hp felhom-agent[297545]: time=2026-10-06T12:21:17.652+02:00 level=INFO msg="backup: restore-test tier is DUE (per-archive; oldest-proven first among due tiers)" target=local archive=local:backup/vzdump-lxc-9201-2026_10_05-11_19_30.tar.zst landed=2026-10-05T09:19:30Z reason="newest settled archive (landed 2026-10-05T09:19:30Z) has not been proven (last proven archive was a diff
Oct 06 12:22:46 demo-hp felhom-agent[297545]: time=2026-10-06T12:22:46.987+02:00 level=INFO msg="backup: scheduled restore-test passed" archive=local:backup/vzdump-lxc-9201-2026_10_05-11_19_30.tar.zst duration_s=89.331269652
Oct 06 18:22:47 demo-hp felhom-agent[297545]: time=2026-10-06T18:22:47.786+02:00 level=INFO msg="backup: restore-test evaluated — nothing due" verdicts="felhom-pbs: newest settled archive (landed 2026-10-01T20:15:29Z) is already proven; local: newest settled archive (landed 2026-10-05T09:19:30Z) is already proven"
documentation/audits/readback-2026-10-07/C/C2-ep0-snapshot-listing.txt
+14
View File
@@ -0,0 +1,14 @@
Tester-2 ct 9201 2026-10-04T16:31:13Z 2096481498 ok
<string>:5: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
demo-felhom ct 9201 2026-09-22T04:12:20Z 5840921438 ok
demo-felhom ct 9201 2026-09-29T04:16:43Z 6956268440 ok
demo-felhom ct 9201 2026-10-06T04:21:09Z 2749038678 ok
<string>:5: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
<string>:5: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
demo-hp ct 9201 2026-09-24T20:06:25Z 23236593218 ok
demo-hp ct 9201 2026-10-01T20:15:29Z 15223817295 ok
operator host dooplex-hub 2026-10-05T14:23:25Z 369808250 ok
operator host dooplex-hub 2026-10-06T00:32:21Z 372921211 ok
<string>:5: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
tester-1 ct 9201 2026-10-04T19:57:18Z 4913374015 ok
<string>:5: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
documentation/audits/readback-2026-10-07/C/C3-hub-events-demo-hp-oct05.txt
+48
View File
@@ -0,0 +1,48 @@
events columns: ['id', 'customer_id', 'event_type', 'severity', 'message', 'details_json', 'source', 'created_at']
('2026-10-05 04:27:31', 'whole_guest_backup_failed', 'error', 'Whole-guest backup FAILED on the felhom-pbs tier — retrying with backoff (next attempt in 15m0s)')
('2026-10-05 05:16:22', 'controller_updated', 'info', 'Controller frissítve: 0.293.0 → 0.294.0')
('2026-10-05 05:16:27', 'controller_started', 'info', 'Controller elindult (0.294.0)')
('2026-10-05 06:09:05', 'os_update_applied', 'info', 'System security fixes installed on the box (13 package(s)).')
('2026-10-05 06:11:02', 'os_update_applied', 'info', 'System security fixes installed on the box (13 package(s)).')
('2026-10-05 06:16:39', 'controller_started', 'info', 'Controller elindult (0.294.0)')
('2026-10-05 06:18:03', 'os_update_failed', 'error', 'OS update (guest) on demo-hp-bb76ea failed: {"dpkg_audit":"clean","rc":100,"tail":["E: dpkg was interrupted, you must manually run \'sudo dpkg --configure -a\' to correct the problem."]} no health readi')
('2026-10-05 06:19:35', 'os_update_applied', 'info', 'System security fixes installed on the box (12 package(s)).')
('2026-10-05 06:29:57', 'host_restarted_after_crash', 'info', 'Your box restarted by itself after an unexpected stop. You do not need to do anything.')
('2026-10-05 06:29:57', 'host_crash_restart', 'warning', 'demo-hp-bb76ea restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T06:14:33Z — 4 such restart(s) in 24 h. kernel.panic now 10.')
('2026-10-05 07:29:43', 'controller_updated', 'info', 'Controller frissítve: 0.294.0 → 0.295.0')
('2026-10-05 07:29:48', 'controller_started', 'info', 'Controller elindult (0.295.0)')
('2026-10-05 07:58:50', 'controller_started', 'info', 'Controller elindult (0.295.0)')
('2026-10-05 08:00:42', 'os_update_applied', 'info', 'System security fixes installed on the box (12 package(s)).')
('2026-10-05 08:12:06', 'host_restarted_after_crash', 'info', 'Your box restarted by itself after an unexpected stop. You do not need to do anything.')
('2026-10-05 08:12:06', 'host_crash_restart', 'warning', 'demo-hp-bb76ea restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T07:56:41Z — 5 such restart(s) in 24 h. kernel.panic now 10.')
('2026-10-05 09:16:15', 'recovery_credential_revealed', 'info', 'Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás).')
('2026-10-05 10:25:08', 'controller_updated', 'info', 'Controller frissítve: 0.295.0 → 0.296.0')
('2026-10-05 10:25:13', 'controller_started', 'info', 'Controller elindult (0.296.0)')
('2026-10-05 10:28:08', 'agent_capability_degraded', 'warning', 'Host demo-hp-bb76ea: agent privileged capability degraded — wg-conf-install (impairs: wg-felhom conf install (root content check, R-861))')
('2026-10-05 11:13:08', 'agent_capability_recovered', 'info', 'Host demo-hp-bb76ea: agent privileged capabilities recovered (all required grants restored)')
== notification_log demo-hp 2026-10-05:
['id', 'customer_id', 'event_type', 'severity', 'message', 'status', 'error_message', 'created_at', 'channel']
(1084, 'demo-hp', 'whole_guest_backup_failed', 'error', 'Whole-guest backup FAILED on the felhom-pbs tier — retrying with backoff (next attempt in 15m0s)', 'failed', 'sending request: Post "https://api.resend.com/emails": context deadline exceeded (Client.Timeout exceeded while awaiting headers)', '
(1085, 'demo-hp', 'whole_guest_backup_failed', 'error', 'Whole-guest backup FAILED on the felhom-pbs tier — retrying with backoff (next attempt in 15m0s)', 'skipped', 'operator_only', '2026-10-05 04:27:41', 'customer')
(1086, 'demo-hp', 'os_update_failed', 'error', 'OS update (guest) on demo-hp-bb76ea failed: {"dpkg_audit":"clean","rc":100,"tail":["E: dpkg was interrupted, you must manually run \'sudo dpkg --configure -a\' to correct the problem."]} no health reading', 'sent', '', '2026-10-05 06:18:03', 'operator'
(1087, 'demo-hp', 'os_update_failed', 'error', 'OS update (guest) on demo-hp-bb76ea failed: {"dpkg_audit":"clean","rc":100,"tail":["E: dpkg was interrupted, you must manually run \'sudo dpkg --configure -a\' to correct the problem."]} no health reading', 'skipped', 'operator_only', '2026-10-05 06:18
(1088, 'demo-hp', 'host_crash_restart', 'warning', 'demo-hp-bb76ea restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T06:14:33Z — 4 such restart(s) in 24 h. kernel.panic now 10.', 'sent', '', '2026-10-05 06:29:57', 'operator')
(1089, 'demo-hp', 'host_crash_restart', 'warning', 'demo-hp-bb76ea restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T06:14:33Z — 4 such restart(s) in 24 h. kernel.panic now 10.', 'skipped', 'operator_only', '2026-10-05 06:29:57', 'customer')
(1090, 'tester-1', 'node_stale', 'warning', 'No report received for 45m', 'sent', '', '2026-10-05 06:46:41', 'operator')
(1091, 'tester-1', 'host_stale', 'warning', 'Host tester-1-d70be4: no report for 45m', 'sent', '', '2026-10-05 06:56:41', 'operator')
(1092, 'tester-1', 'node_down', 'error', 'No report received for 1h30m', 'sent', '', '2026-10-05 07:31:41', 'operator')
(1093, 'tester-1', 'host_recovered', 'info', 'Host tester-1-d70be4: reports resumed (was stale for 35m)', 'sent', '', '2026-10-05 07:31:42', 'operator')
(1094, 'tester-1', 'host_crash_restart', 'warning', 'tester-1-d70be4 restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T07:31:20Z — 1 such restart(s) in 24 h. kernel.panic now 10.', 'sent', '', '2026-10-05 07:33:12', 'operator')
(1095, 'tester-1', 'host_crash_restart', 'warning', 'tester-1-d70be4 restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T07:31:20Z — 1 such restart(s) in 24 h. kernel.panic now 10.', 'skipped', 'operator_only', '2026-10-05 07:33:12', 'customer')
(1096, 'tester-1', 'node_recovered', 'info', 'Reports resumed (was down for 46m)', 'sent', '', '2026-10-05 07:33:41', 'operator')
(1097, 'Tester-2', 'host_down', 'error', 'Host Tester-2-be8404: no report for 13h45m', 'sent', '', '2026-10-05 07:51:34', 'operator')
(1098, 'demo-hp', 'host_crash_restart', 'warning', 'demo-hp-bb76ea restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T07:56:41Z — 5 such restart(s) in 24 h. kernel.panic now 10.', 'sent', '', '2026-10-05 08:12:06', 'operator')
(1099, 'demo-hp', 'host_crash_restart', 'warning', 'demo-hp-bb76ea restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T07:56:41Z — 5 such restart(s) in 24 h. kernel.panic now 10.', 'skipped', 'operator_only', '2026-10-05 08:12:06', 'customer')
(1100, 'tester-1', 'host_crash_guard_tripped', 'error', 'The crash guard of tester-1-d70be4 TRIPPED at 2026-10-05T07:57:17Z: 2 unclean boots within 60 minutes — the next crash leaves the box off (limit 3). The next crash leaves the box OFF until someone switches it on. Re-arm: `felhom-crash-guard re
(1101, 'tester-1', 'host_crash_guard_tripped', 'error', 'The crash guard of tester-1-d70be4 TRIPPED at 2026-10-05T07:57:17Z: 2 unclean boots within 60 minutes — the next crash leaves the box off (limit 3). The next crash leaves the box OFF until someone switches it on. Re-arm: `felhom-crash-guard re
(1102, 'tester-1', 'host_crash_restart', 'warning', 'tester-1-d70be4 restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T07:57:12Z — 2 such restart(s) in 24 h. kernel.panic now 0.', 'sent', '', '2026-10-05 08:12:32', 'operator')
(1103, 'tester-1', 'host_crash_restart', 'warning', 'tester-1-d70be4 restarted by itself after an unexpected stop (a kernel crash, a power cut or a hard reset) at 2026-10-05T07:57:12Z — 2 such restart(s) in 24 h. kernel.panic now 0.', 'skipped', 'operator_only', '2026-10-05 08:12:32', 'customer')
(1104, 'Tester-2', 'host_down', 'error', 'Host Tester-2-be8404: no report for 15h10m', 'sent', '', '2026-10-05 09:16:10', 'operator')
(1105, 'demo-hp', 'agent_capability_degraded', 'warning', 'Host demo-hp-bb76ea: agent privileged capability degraded — wg-conf-install (impairs: wg-felhom conf install (root content check, R-861))', 'sent', '', '2026-10-05 10:28:08', 'operator')
(1106, 'tester-1', 'agent_capability_degraded', 'warning', 'Host tester-1-d70be4: agent privileged capability degraded — wg-conf-install (impairs: wg-felhom conf install (root content check, R-861))', 'sent', '', '2026-10-05 10:28:09', 'operator')
(1107, 'demo-felhom', 'agent_capability_degraded', 'warning', 'Host demo-felhom-8363b5: agent privileged capability degraded — wg-conf-install (impairs: wg-felhom conf install (root content check, R-861))', 'sent', '', '2026-10-05 10:39:09', 'operator')
documentation/audits/readback-2026-10-07/C/C4-hub-failed-mails.txt
+6
View File
@@ -0,0 +1,6 @@
all notification rows: (1112, '2026-02-16 18:46:11', '2026-10-06 09:56:51')
by channel/status: [('customer', 'sent', 155), ('customer', 'skipped', 166), ('operator', 'failed', 1), ('operator', 'recorded', 22), ('operator', 'refused', 7), ('operator', 'sent', 691), ('operator', 'suppressed', 70)]
failed OPERATOR mails, by severity: [('error', 1)]
failed operator mails, error/critical, newest 15:
('2026-10-05 04:27:41', 'demo-hp', 'whole_guest_backup_failed', 'error', 'sending request: Post "https://api.resend.com/emails": context deadline exceeded (Client.Timeout exceeded whil')
failed CUSTOMER mails: (0,)
documentation/audits/readback-2026-10-07/C/hub-go-test.txt
+27
View File
@@ -0,0 +1,27 @@
ok gitea.dooplex.hu/admin/felhom-hub/cmd/hub 0.077s
ok gitea.dooplex.hu/admin/felhom-hub/cmd/hubdb-check 0.092s
ok gitea.dooplex.hu/admin/felhom-hub/internal/api 5.785s
? gitea.dooplex.hu/admin/felhom-hub/internal/assets [no test files]
ok gitea.dooplex.hu/admin/felhom-hub/internal/claim 3.336s
ok gitea.dooplex.hu/admin/felhom-hub/internal/cloudflare 0.008s
ok gitea.dooplex.hu/admin/felhom-hub/internal/configgen 0.031s
ok gitea.dooplex.hu/admin/felhom-hub/internal/dbsnap 0.154s
ok gitea.dooplex.hu/admin/felhom-hub/internal/gitea 0.013s
ok gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi 0.053s
ok gitea.dooplex.hu/admin/felhom-hub/internal/i18n 0.009s
ok gitea.dooplex.hu/admin/felhom-hub/internal/intent 0.092s
ok gitea.dooplex.hu/admin/felhom-hub/internal/mailrelay 0.009s
ok gitea.dooplex.hu/admin/felhom-hub/internal/monitor 38.751s
ok gitea.dooplex.hu/admin/felhom-hub/internal/notify 1.624s
ok gitea.dooplex.hu/admin/felhom-hub/internal/offsite 0.761s
ok gitea.dooplex.hu/admin/felhom-hub/internal/offsiteheal 0.269s
ok gitea.dooplex.hu/admin/felhom-hub/internal/offsitekeys 0.245s
ok gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 1.325s
ok gitea.dooplex.hu/admin/felhom-hub/internal/pbsdrheal 0.622s
ok gitea.dooplex.hu/admin/felhom-hub/internal/poke 0.009s
ok gitea.dooplex.hu/admin/felhom-hub/internal/semver 0.007s
ok gitea.dooplex.hu/admin/felhom-hub/internal/store 4.224s
ok gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts 0.005s
ok gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync 0.042s
ok gitea.dooplex.hu/admin/felhom-hub/internal/web 25.654s
ok gitea.dooplex.hu/admin/felhom-hub/internal/wgsync 0.237s
documentation/audits/readback-2026-10-07/C/red-operator-mail-retry.txt
+12
View File
@@ -0,0 +1,12 @@
$ go test ./internal/notify/ -run TestOperatorMailRetry -v (processOperator without the retryOperatorEmail call)
=== RUN TestOperatorMailRetry_FailedOperatorMailIsRetried
r895_operator_retry_test.go:34: the failed mail was never sent again (tries=1)
--- FAIL: TestOperatorMailRetry_FailedOperatorMailIsRetried (0.03s)
=== RUN TestOperatorMailRetry_RetriesAreBounded
r895_operator_retry_test.go:69: tries=1, want 4
--- FAIL: TestOperatorMailRetry_RetriesAreBounded (0.03s)
=== RUN TestOperatorMailRetry_SuccessIsNotRetried
--- PASS: TestOperatorMailRetry_SuccessIsNotRetried (0.03s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-hub/internal/notify 0.104s
FAIL