admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9c1d05d3600a078a578410c6eb3d298fbf025c4c
felhom.eu/documentation/tests/campaign10-evidence-2026-07-31/state/journal.tsv
T
admin 405a795e32 Campaign 10: RESOLVED — the backup_target_* silence was transient and self-recovered; SQLITE_BUSY drops are absorbed by retry 
Both halves of the disposition were run and neither survived as a finding.

The I1/I1-pair violations cluster at cycles 31-33 and nowhere else across 39
cycles; c34-c39 are clean, so it recovered with no intervention. Final tally I1
37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION. On the quiesced box one slow
detach with 4 minutes either side produced a perfect pair. And the alarming
false-healthy (mentes bound=False while degraded=false) does not survive
quiescence - I had been reading the two halves at different instants of a detach.
No R-n.

Separately cleared: the hub's SQLITE_BUSY event drops. 7 in 24h including one for
the real customer demo-felhom, and the hub does return 500 with notification
dispatch only after a successful save - so a lost event would be a lost alarm. But
the controller retries 3 times and ZERO events exhausted their attempts; the
07:04:39 drop landed at 07:04:42. Nothing lost. Only cosmetic note: the ERROR line
reads like data loss and is not.
2026-08-02 07:10:49 +02:00

89 KiB
Raw Blame History

1cycleutcinvariantverdictdetail
212026-08-01T17:53:31ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
312026-08-01T17:53:34ZI3PASSadatok bound_under_parent=False while absent
412026-08-01T17:55:47ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
512026-08-01T17:55:49ZI4PASSguest init boot before=1785606256 after=1785606256 (must be equal)
612026-08-01T17:57:25ZKILL-AGENTPASSagent active after kill=True; subsequent backup ok=True
712026-08-01T18:00:35ZI7PASSapp=rallly want=C10-C001-A-195822 got=C10-C001-A-195822 snap=helyi restore_ok=True rto=41.9s
812026-08-01T18:02:51ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
912026-08-01T18:02:54ZI3PASSmentes bound_under_parent=False while absent
1012026-08-01T18:05:06ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
1112026-08-01T18:05:09ZI4PASSguest init boot before=1785606256 after=1785606256 (must be equal)
1212026-08-01T18:05:53ZREDEPLOYPASSgrafana redeployed
1312026-08-01T18:06:43ZKILL-CTRLPASScontroller recovered after kill
1412026-08-01T18:06:46ZI5PASSdegraded=False absent_events=0 target=felhom-backup
1512026-08-01T18:06:46ZI6PASShealthy state must carry no degraded message; got=None
1612026-08-01T18:06:48ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
1712026-08-01T18:06:48ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
1812026-08-01T18:06:50ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
1912026-08-01T18:06:52ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
2012026-08-01T18:06:54ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
2112026-08-01T18:06:56ZI11PASSleaked=[]
2222026-08-01T18:07:42ZREDEPLOYPASSgrafana redeployed
2322026-08-01T18:09:58ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
2422026-08-01T18:10:00ZI3PASSadatok bound_under_parent=False while absent
2522026-08-01T18:12:13ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
2622026-08-01T18:12:16ZI4PASSguest init boot before=1785606256 after=1785606256 (must be equal)
2722026-08-01T18:13:06ZKILL-CTRLPASScontroller recovered after kill
2822026-08-01T18:17:24ZHARD-RESETPASSVM returned=True apps_ready=True canaries_intact=True (a hard reset mid-write must not corrupt them)
2922026-08-01T18:19:37ZI7PASSapp=rallly want=C10-C002-A-201726 got=C10-C002-A-201726 snap=helyi restore_ok=True rto=41.6s
3022026-08-01T18:21:52ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
3122026-08-01T18:21:55ZI3PASSmentes bound_under_parent=False while absent
3222026-08-01T18:24:08ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
3322026-08-01T18:24:11ZI4PASSguest init boot before=1785608199 after=1785608199 (must be equal)
3422026-08-01T18:24:13ZI5PASSdegraded=False absent_events=0 target=felhom-backup
3522026-08-01T18:24:13ZI6PASShealthy state must carry no degraded message; got=None
3622026-08-01T18:24:15ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
3722026-08-01T18:24:15ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
3822026-08-01T18:24:17ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
3922026-08-01T18:24:19ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
4022026-08-01T18:24:21ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
4122026-08-01T18:24:24ZI11PASSleaked=[]
4232026-08-01T18:25:39ZREDEPLOYPASShomebox redeployed
4332026-08-01T18:27:55ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
4432026-08-01T18:27:58ZI3PASSmentes bound_under_parent=False while absent
4532026-08-01T18:30:11ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
4632026-08-01T18:30:13ZI4PASSguest init boot before=1785608199 after=1785608199 (must be equal)
4732026-08-01T18:31:29ZREBOOT-VMPASScontroller back after full VM reboot
4832026-08-01T18:33:45ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
4932026-08-01T18:33:48ZI3PASSadatok bound_under_parent=False while absent
5032026-08-01T18:36:01ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
5132026-08-01T18:36:03ZI4PASSguest init boot before=1785609078 after=1785609078 (must be equal)
5232026-08-01T18:36:53ZKILL-CTRLPASScontroller recovered after kill
5332026-08-01T18:39:14ZI7PASSapp=rallly want=C10-C003-A-203656 got=C10-C003-A-203656 snap=helyi restore_ok=True rto=45.0s
5432026-08-01T18:39:17ZI5PASSdegraded=False absent_events=0 target=felhom-backup
5532026-08-01T18:39:17ZI6PASShealthy state must carry no degraded message; got=None
5632026-08-01T18:39:19ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
5732026-08-01T18:39:19ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
5832026-08-01T18:39:21ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
5932026-08-01T18:39:23ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
6032026-08-01T18:39:25ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
6132026-08-01T18:39:28ZI11PASSleaked=[]
6242026-08-01T18:41:46ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
6342026-08-01T18:41:49ZI3PASSadatok bound_under_parent=False while absent
6442026-08-01T18:44:02ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
6542026-08-01T18:44:05ZI4PASSguest init boot before=1785609078 after=1785609078 (must be equal)
6642026-08-01T18:44:49ZREDEPLOYPASSgrafana redeployed
6742026-08-01T18:46:00ZREBOOTPASScontroller back after guest reboot
6842026-08-01T18:48:31ZI7PASSapp=rallly want=C10-C004-A-204616 got=C10-C004-A-204616 snap=helyi restore_ok=True rto=42.1s
6942026-08-01T18:50:46ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
7042026-08-01T18:50:49ZI3PASSmentes bound_under_parent=False while absent
7142026-08-01T18:53:02ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
7242026-08-01T18:53:04ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
7342026-08-01T18:53:53ZKILL-CTRLPASScontroller recovered after kill
7442026-08-01T18:53:56ZI5PASSdegraded=False absent_events=0 target=felhom-backup
7542026-08-01T18:53:56ZI6PASShealthy state must carry no degraded message; got=None
7642026-08-01T18:53:58ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
7742026-08-01T18:53:58ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
7842026-08-01T18:54:00ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
7942026-08-01T18:54:02ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
8042026-08-01T18:54:04ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
8142026-08-01T18:54:07ZI11PASSleaked=[]
8252026-08-01T18:56:26ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
8352026-08-01T18:56:29ZI3PASSadatok bound_under_parent=False while absent
8452026-08-01T18:58:42ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
8552026-08-01T18:58:45ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
8652026-08-01T18:59:56ZCONCURRENCYPASSbackup+restore raced; restore-status op=None running=False (recorded, not asserted)
8752026-08-01T19:00:41ZREDEPLOYPASShomebox redeployed
8852026-08-01T19:01:33ZKILL-CTRLPASScontroller recovered after kill
8952026-08-01T19:03:46ZI7PASSapp=rallly want=C10-C005-A-210135 got=C10-C005-A-210135 snap=helyi restore_ok=True rto=42.1s
9052026-08-01T19:06:02ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
9152026-08-01T19:06:05ZI3PASSmentes bound_under_parent=False while absent
9252026-08-01T19:08:20ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
9352026-08-01T19:08:26ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
9452026-08-01T19:08:32ZI5PASSdegraded=False absent_events=0 target=felhom-backup
9552026-08-01T19:08:32ZI6PASShealthy state must carry no degraded message; got=None
9652026-08-01T19:08:34ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
9752026-08-01T19:08:34ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
9852026-08-01T19:08:37ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
9952026-08-01T19:08:40ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
10052026-08-01T19:08:43ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
10152026-08-01T19:08:49ZI11PASSleaked=[]
10262026-08-01T19:11:26ZI7PASSapp=rallly want=C10-C006-A-210908 got=C10-C006-A-210908 snap=helyi restore_ok=True rto=41.6s
10362026-08-01T19:13:42ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
10462026-08-01T19:13:45ZI3PASSmentes bound_under_parent=False while absent
10562026-08-01T19:15:58ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
10662026-08-01T19:16:00ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
10762026-08-01T19:19:03ZI1-under-loadPASStarget pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
10862026-08-01T19:21:21ZI1-under-load-recoverPASSafter return degraded=False target=felhom-backup
10962026-08-01T19:22:11ZKILL-CTRLPASScontroller recovered after kill
11062026-08-01T19:22:54ZREDEPLOYPASSgrafana redeployed
11162026-08-01T19:25:10ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
11262026-08-01T19:25:13ZI3PASSadatok bound_under_parent=False while absent
11362026-08-01T19:27:26ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
11462026-08-01T19:27:28ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
11562026-08-01T19:27:30ZI5PASSdegraded=False absent_events=0 target=felhom-backup
11662026-08-01T19:27:30ZI6PASShealthy state must carry no degraded message; got=None
11762026-08-01T19:27:32ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
11862026-08-01T19:27:32ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
11962026-08-01T19:27:34ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
12062026-08-01T19:27:36ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
12162026-08-01T19:27:38ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
12262026-08-01T19:27:40ZI11PASSleaked=[]
12372026-08-01T19:28:27ZREDEPLOYPASShomebox redeployed
12472026-08-01T19:29:16ZKILL-CTRLPASScontroller recovered after kill
12572026-08-01T19:31:20ZI7PASSapp=rallly want=C10-C007-A-212918 got=C10-C007-A-212918 snap=helyi restore_ok=True rto=41.4s
12672026-08-01T19:34:12ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
12772026-08-01T19:34:15ZI3PASSmentes bound_under_parent=False while absent
12872026-08-01T19:36:28ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
12972026-08-01T19:36:31ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
13072026-08-01T19:38:47ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
13172026-08-01T19:38:50ZI3PASSadatok bound_under_parent=False while absent
13272026-08-01T19:41:03ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
13372026-08-01T19:41:05ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
13472026-08-01T19:41:37ZI5PASSdegraded=False absent_events=0 target=felhom-backup
13572026-08-01T19:41:37ZI6PASShealthy state must carry no degraded message; got=None
13672026-08-01T19:41:39ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
13772026-08-01T19:41:39ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
13872026-08-01T19:41:41ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
13972026-08-01T19:41:43ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
14072026-08-01T19:41:46ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
14172026-08-01T19:41:48ZI11PASSleaked=[]
14282026-08-01T19:42:35ZREDEPLOYPASSgrafana redeployed
14382026-08-01T19:43:23ZKILL-CTRLPASScontroller recovered after kill
14482026-08-01T19:45:27ZI7PASSapp=rallly want=C10-C008-A-214325 got=C10-C008-A-214325 snap=helyi restore_ok=True rto=41.4s
14582026-08-01T19:48:12ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
14682026-08-01T19:48:15ZI3PASSmentes bound_under_parent=False while absent
14782026-08-01T19:50:28ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
14882026-08-01T19:50:31ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
14982026-08-01T19:53:07ZI4-abortPASSaborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
15082026-08-01T19:53:11ZI3-abortPASScalibre-web must be stopped while its namespace is aborted; running=False
15182026-08-01T19:56:01ZI4-abort-recoverPASSafter re-seat: bound=True app_up=True
15282026-08-01T19:58:17ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
15382026-08-01T19:58:21ZI3PASSadatok bound_under_parent=False while absent
15482026-08-01T20:00:34ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
15582026-08-01T20:00:36ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
15682026-08-01T20:00:39ZI5PASSdegraded=False absent_events=0 target=felhom-backup
15782026-08-01T20:00:39ZI6PASShealthy state must carry no degraded message; got=None
15882026-08-01T20:00:41ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
15982026-08-01T20:00:41ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
16082026-08-01T20:00:43ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
16182026-08-01T20:00:45ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
16282026-08-01T20:00:47ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
16382026-08-01T20:00:49ZI11PASSleaked=[]
16492026-08-01T20:01:35ZREDEPLOYPASShomebox redeployed
16592026-08-01T20:02:24ZKILL-CTRLPASScontroller recovered after kill
16692026-08-01T20:03:50ZKILL-AGENTPASSagent active after kill=True; subsequent backup ok=True
16792026-08-01T20:06:05ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
16892026-08-01T20:06:08ZI3PASSadatok bound_under_parent=False while absent
16992026-08-01T20:08:21ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
17092026-08-01T20:08:24ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
17192026-08-01T20:10:40ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
17292026-08-01T20:10:43ZI3PASSmentes bound_under_parent=False while absent
17392026-08-01T20:12:56ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
17492026-08-01T20:12:58ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
17592026-08-01T20:15:13ZI7PASSapp=rallly want=C10-C009-A-221300 got=C10-C009-A-221300 snap=helyi restore_ok=True rto=41.9s
17692026-08-01T20:15:45ZI5PASSdegraded=False absent_events=0 target=felhom-backup
17792026-08-01T20:15:45ZI6PASShealthy state must carry no degraded message; got=None
17892026-08-01T20:15:47ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
17992026-08-01T20:15:47ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
18092026-08-01T20:15:49ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
18192026-08-01T20:15:51ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
18292026-08-01T20:15:53ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
18392026-08-01T20:15:56ZI11PASSleaked=[]
184102026-08-01T20:18:13ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
185102026-08-01T20:18:16ZI3PASSmentes bound_under_parent=False while absent
186102026-08-01T20:20:29ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
187102026-08-01T20:20:32ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
188102026-08-01T20:22:48ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
189102026-08-01T20:22:51ZI3PASSadatok bound_under_parent=False while absent
190102026-08-01T20:25:04ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
191102026-08-01T20:25:06ZI4PASSguest init boot before=1785609940 after=1785609940 (must be equal)
192102026-08-01T20:38:40ZHARD-RESETVIOLATIONVM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
193102026-08-01T20:39:25ZREDEPLOYPASSgrafana redeployed
194102026-08-01T20:49:32ZI7-SKIPPASSapps not ready before seed — atom skipped, NOT counted as a violation
195102026-08-01T20:50:51ZKILL-CTRLPASScontroller recovered after kill
196102026-08-01T20:50:54ZI5PASSdegraded=False absent_events=0 target=felhom-backup
197102026-08-01T20:50:54ZI6PASShealthy state must carry no degraded message; got=None
198102026-08-01T20:50:56ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
199102026-08-01T20:50:56ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
200102026-08-01T20:50:58ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
201102026-08-01T20:51:00ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
202102026-08-01T20:51:02ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
203102026-08-01T20:51:05ZI11PASSleaked=[]
204112026-08-01T20:51:57ZKILL-CTRLPASScontroller recovered after kill
205112026-08-01T20:53:13ZREBOOT-VMPASScontroller back after full VM reboot
206112026-08-01T20:55:59ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
207112026-08-01T20:56:02ZI3PASSmentes bound_under_parent=False while absent
208112026-08-01T20:58:15ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
209112026-08-01T20:58:25ZI4PASSguest init boot before=1785617581 after=1785617581 (must be equal)
210112026-08-01T21:00:50ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
211112026-08-01T21:00:56ZI3PASSadatok bound_under_parent=False while absent
212112026-08-01T21:03:09ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
213112026-08-01T21:03:12ZI4PASSguest init boot before=1785617581 after=1785617581 (must be equal)
214112026-08-01T21:06:54ZI7PASSapp=rallly want=C10-C011-A-230315 got=C10-C011-A-230315 snap=helyi restore_ok=True rto=49.6s
215112026-08-01T21:08:02ZREDEPLOYPASShomebox redeployed
216112026-08-01T21:08:06ZI5PASSdegraded=False absent_events=0 target=felhom-backup
217112026-08-01T21:08:06ZI6PASShealthy state must carry no degraded message; got=None
218112026-08-01T21:08:14ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
219112026-08-01T21:08:14ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
220112026-08-01T21:08:20ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
221112026-08-01T21:08:24ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
222112026-08-01T21:08:27ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
223112026-08-01T21:08:30ZI11PASSleaked=[]
224122026-08-01T21:10:07ZREDEPLOYPASShomebox redeployed
225122026-08-01T21:12:29ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
226122026-08-01T21:12:32ZI3PASSadatok bound_under_parent=False while absent
227122026-08-01T21:14:45ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
228122026-08-01T21:14:48ZI4PASSguest init boot before=1785617581 after=1785617581 (must be equal)
229122026-08-01T21:15:37ZKILL-CTRLPASScontroller recovered after kill
230122026-08-01T21:17:53ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
231122026-08-01T21:17:56ZI3PASSmentes bound_under_parent=False while absent
232122026-08-01T21:20:10ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
233122026-08-01T21:20:12ZI4PASSguest init boot before=1785617581 after=1785617581 (must be equal)
234122026-08-01T21:22:18ZI7PASSapp=rallly want=C10-C012-A-232015 got=C10-C012-A-232015 snap=helyi restore_ok=True rto=41.4s
235122026-08-01T21:22:58ZREBOOTPASScontroller back after guest reboot
236122026-08-01T21:23:01ZI5PASSdegraded=False absent_events=0 target=felhom-backup
237122026-08-01T21:23:01ZI6PASShealthy state must carry no degraded message; got=None
238122026-08-01T21:23:03ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
239122026-08-01T21:23:03ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
240122026-08-01T21:23:05ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
241122026-08-01T21:23:07ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
242122026-08-01T21:23:09ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
243122026-08-01T21:23:11ZI11PASSleaked=[]
244132026-08-01T21:23:57ZREDEPLOYPASSgrafana redeployed
245132026-08-01T21:25:28ZKILL-CTRLPASScontroller recovered after kill
246132026-08-01T21:27:44ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
247132026-08-01T21:27:47ZI3PASSmentes bound_under_parent=False while absent
248132026-08-01T21:30:00ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
249132026-08-01T21:30:02ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
250132026-08-01T21:30:41ZCONCURRENCYPASSbackup+restore raced; restore-status op=None running=False (recorded, not asserted)
251132026-08-01T21:32:43ZI7PASSapp=rallly want=C10-C013-A-233043 got=C10-C013-A-233043 snap=helyi restore_ok=True rto=29.1s
252132026-08-01T21:34:59ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
253132026-08-01T21:35:02ZI3PASSadatok bound_under_parent=False while absent
254132026-08-01T21:37:15ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
255132026-08-01T21:37:17ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
256132026-08-01T21:37:20ZI5PASSdegraded=False absent_events=0 target=felhom-backup
257132026-08-01T21:37:20ZI6PASShealthy state must carry no degraded message; got=None
258132026-08-01T21:37:22ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
259132026-08-01T21:37:22ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
260132026-08-01T21:37:24ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
261132026-08-01T21:37:26ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
262132026-08-01T21:37:28ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
263132026-08-01T21:37:30ZI11PASSleaked=[]
264142026-08-01T21:40:29ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
265142026-08-01T21:40:32ZI3PASSmentes bound_under_parent=False while absent
266142026-08-01T21:42:45ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
267142026-08-01T21:42:48ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
268142026-08-01T21:45:09ZI1-under-loadPASStarget pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
269142026-08-01T21:47:27ZI1-under-load-recoverPASSafter return degraded=False target=felhom-backup
270142026-08-01T21:49:43ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
271142026-08-01T21:49:46ZI3PASSadatok bound_under_parent=False while absent
272142026-08-01T21:51:59ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
273142026-08-01T21:52:02ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
274142026-08-01T21:53:51ZI7PASSapp=rallly want=C10-C014-A-235204 got=C10-C014-A-235204 snap=helyi restore_ok=True rto=29.1s
275142026-08-01T21:54:40ZKILL-CTRLPASScontroller recovered after kill
276142026-08-01T21:55:25ZREDEPLOYPASShomebox redeployed
277142026-08-01T21:55:28ZI5PASSdegraded=False absent_events=0 target=felhom-backup
278142026-08-01T21:55:28ZI6PASShealthy state must carry no degraded message; got=None
279142026-08-01T21:55:29ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
280142026-08-01T21:55:29ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
281142026-08-01T21:55:31ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
282142026-08-01T21:55:33ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
283142026-08-01T21:55:35ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
284142026-08-01T21:55:37ZI11PASSleaked=[]
285152026-08-01T21:57:55ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
286152026-08-01T21:57:58ZI3PASSmentes bound_under_parent=False while absent
287152026-08-01T22:00:11ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
288152026-08-01T22:00:13ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
289152026-08-01T22:02:58ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
290152026-08-01T22:03:01ZI3PASSadatok bound_under_parent=False while absent
291152026-08-01T22:05:14ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
292152026-08-01T22:05:16ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
293152026-08-01T22:06:06ZKILL-CTRLPASScontroller recovered after kill
294152026-08-01T22:07:38ZREDEPLOYPASSgrafana redeployed
295152026-08-01T22:09:50ZI7PASSapp=rallly want=C10-C015-A-000739 got=C10-C015-A-000739 snap=helyi restore_ok=True rto=41.6s
296152026-08-01T22:09:53ZI5PASSdegraded=False absent_events=0 target=felhom-backup
297152026-08-01T22:09:53ZI6PASShealthy state must carry no degraded message; got=None
298152026-08-01T22:09:55ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
299152026-08-01T22:09:55ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
300152026-08-01T22:09:57ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
301152026-08-01T22:09:59ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
302152026-08-01T22:10:01ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
303152026-08-01T22:10:03ZI11PASSleaked=[]
304162026-08-01T22:12:21ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
305162026-08-01T22:12:24ZI3PASSmentes bound_under_parent=False while absent
306162026-08-01T22:14:37ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
307162026-08-01T22:14:39ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
308162026-08-01T22:16:55ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
309162026-08-01T22:16:58ZI3PASSadatok bound_under_parent=False while absent
310162026-08-01T22:19:10ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
311162026-08-01T22:19:12ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
312162026-08-01T22:21:48ZI4-abortPASSaborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
313162026-08-01T22:21:52ZI3-abortPASScalibre-web must be stopped while its namespace is aborted; running=False
314162026-08-01T22:24:43ZI4-abort-recoverPASSafter re-seat: bound=True app_up=True
315162026-08-01T22:25:32ZKILL-CTRLPASScontroller recovered after kill
316162026-08-01T22:26:57ZREDEPLOYPASSgrafana redeployed
317162026-08-01T22:28:56ZI7PASSapp=rallly want=C10-C016-A-002659 got=C10-C016-A-002659 snap=helyi restore_ok=True rto=40.9s
318162026-08-01T22:28:59ZI5PASSdegraded=False absent_events=0 target=felhom-backup
319162026-08-01T22:28:59ZI6PASShealthy state must carry no degraded message; got=None
320162026-08-01T22:29:00ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
321162026-08-01T22:29:00ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
322162026-08-01T22:29:02ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
323162026-08-01T22:29:04ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
324162026-08-01T22:29:06ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
325162026-08-01T22:29:08ZI11PASSleaked=[]
326172026-08-01T22:30:45ZKILL-AGENTPASSagent active after kill=True; subsequent backup ok=True
327172026-08-01T22:33:38ZI7PASSapp=rallly want=C10-C017-A-003140 got=C10-C017-A-003140 snap=helyi restore_ok=True rto=41.0s
328172026-08-01T22:35:53ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
329172026-08-01T22:35:56ZI3PASSadatok bound_under_parent=False while absent
330172026-08-01T22:38:09ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
331172026-08-01T22:38:11ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
332172026-08-01T22:39:01ZKILL-CTRLPASScontroller recovered after kill
333172026-08-01T22:39:45ZREDEPLOYPASSgrafana redeployed
334172026-08-01T22:42:00ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
335172026-08-01T22:42:03ZI3PASSmentes bound_under_parent=False while absent
336172026-08-01T22:44:16ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
337172026-08-01T22:44:19ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
338172026-08-01T22:44:22ZI5PASSdegraded=False absent_events=0 target=felhom-backup
339172026-08-01T22:44:22ZI6PASShealthy state must carry no degraded message; got=None
340172026-08-01T22:44:23ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
341172026-08-01T22:44:23ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
342172026-08-01T22:44:25ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
343172026-08-01T22:44:27ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
344172026-08-01T22:44:29ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
345172026-08-01T22:44:32ZI11PASSleaked=[]
346182026-08-01T22:47:31ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
347182026-08-01T22:47:33ZI3PASSmentes bound_under_parent=False while absent
348182026-08-01T22:49:46ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
349182026-08-01T22:49:48ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
350182026-08-01T22:52:04ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
351182026-08-01T22:52:07ZI3PASSadatok bound_under_parent=False while absent
352182026-08-01T22:54:20ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
353182026-08-01T22:54:23ZI4PASSguest init boot before=1785619357 after=1785619357 (must be equal)
354182026-08-01T22:55:06ZREDEPLOYPASSgrafana redeployed
355182026-08-01T22:57:17ZI7PASSapp=rallly want=C10-C018-A-005508 got=C10-C018-A-005508 snap=helyi restore_ok=True rto=41.4s
356182026-08-01T22:58:07ZKILL-CTRLPASScontroller recovered after kill
357182026-08-01T23:02:05ZHARD-RESETPASSVM returned=True apps_ready=True canaries_intact=True (a hard reset mid-write must not corrupt them)
358182026-08-01T23:02:08ZI5PASSdegraded=False absent_events=0 target=felhom-backup
359182026-08-01T23:02:08ZI6PASShealthy state must carry no degraded message; got=None
360182026-08-01T23:02:10ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
361182026-08-01T23:02:10ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
362182026-08-01T23:02:12ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
363182026-08-01T23:02:14ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
364182026-08-01T23:02:16ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
365182026-08-01T23:02:18ZI11PASSleaked=[]
366192026-08-01T23:03:51ZKILL-CTRLPASScontroller recovered after kill
367192026-08-01T23:06:06ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
368192026-08-01T23:06:09ZI3PASSadatok bound_under_parent=False while absent
369192026-08-01T23:08:22ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
370192026-08-01T23:08:25ZI4PASSguest init boot before=1785625268 after=1785625268 (must be equal)
371192026-08-01T23:09:40ZREBOOT-VMPASScontroller back after full VM reboot
372192026-08-01T23:11:56ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
373192026-08-01T23:11:59ZI3PASSmentes bound_under_parent=False while absent
374192026-08-01T23:14:11ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
375192026-08-01T23:14:14ZI4PASSguest init boot before=1785625770 after=1785625770 (must be equal)
376192026-08-01T23:14:58ZREDEPLOYPASSgrafana redeployed
377192026-08-01T23:17:09ZI7PASSapp=rallly want=C10-C019-A-011500 got=C10-C019-A-011500 snap=helyi restore_ok=True rto=41.6s
378192026-08-01T23:17:11ZI5PASSdegraded=False absent_events=0 target=felhom-backup
379192026-08-01T23:17:11ZI6PASShealthy state must carry no degraded message; got=None
380192026-08-01T23:17:13ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
381192026-08-01T23:17:13ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
382192026-08-01T23:17:15ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
383192026-08-01T23:17:17ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
384192026-08-01T23:17:19ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
385192026-08-01T23:17:21ZI11PASSleaked=[]
386202026-08-01T23:18:02ZREBOOTPASScontroller back after guest reboot
387202026-08-01T23:20:26ZI7PASSapp=rallly want=C10-C020-A-011816 got=C10-C020-A-011816 snap=helyi restore_ok=True rto=41.0s
388202026-08-01T23:21:10ZREDEPLOYPASSgrafana redeployed
389202026-08-01T23:24:07ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
390202026-08-01T23:24:10ZI3PASSadatok bound_under_parent=False while absent
391202026-08-01T23:26:23ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
392202026-08-01T23:26:25ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
393202026-08-01T23:27:19ZKILL-CTRLPASScontroller recovered after kill
394202026-08-01T23:29:38ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
395202026-08-01T23:29:40ZI3PASSmentes bound_under_parent=False while absent
396202026-08-01T23:31:53ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
397202026-08-01T23:31:56ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
398202026-08-01T23:31:58ZI5PASSdegraded=False absent_events=0 target=felhom-backup
399202026-08-01T23:31:58ZI6PASShealthy state must carry no degraded message; got=None
400202026-08-01T23:32:00ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
401202026-08-01T23:32:00ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
402202026-08-01T23:32:02ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
403202026-08-01T23:32:04ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
404202026-08-01T23:32:06ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
405202026-08-01T23:32:08ZI11PASSleaked=[]
406212026-08-01T23:33:35ZREDEPLOYPASShomebox redeployed
407212026-08-01T23:35:35ZI7PASSapp=rallly want=C10-C021-A-013337 got=C10-C021-A-013337 snap=helyi restore_ok=True rto=41.3s
408212026-08-01T23:37:50ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
409212026-08-01T23:37:53ZI3PASSmentes bound_under_parent=False while absent
410212026-08-01T23:40:06ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
411212026-08-01T23:40:08ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
412212026-08-01T23:40:47ZCONCURRENCYPASSbackup+restore raced; restore-status op=restore running=False (recorded, not asserted)
413212026-08-01T23:41:36ZKILL-CTRLPASScontroller recovered after kill
414212026-08-01T23:43:52ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
415212026-08-01T23:43:55ZI3PASSadatok bound_under_parent=False while absent
416212026-08-01T23:46:08ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
417212026-08-01T23:46:10ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
418212026-08-01T23:46:12ZI5PASSdegraded=False absent_events=0 target=felhom-backup
419212026-08-01T23:46:12ZI6PASShealthy state must carry no degraded message; got=None
420212026-08-01T23:46:14ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
421212026-08-01T23:46:14ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
422212026-08-01T23:46:16ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
423212026-08-01T23:46:18ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
424212026-08-01T23:46:20ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
425212026-08-01T23:46:22ZI11PASSleaked=[]
426222026-08-01T23:47:08ZREDEPLOYPASSgrafana redeployed
427222026-08-01T23:49:24ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
428222026-08-01T23:49:27ZI3PASSmentes bound_under_parent=False while absent
429222026-08-01T23:51:39ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
430222026-08-01T23:51:41ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
431222026-08-01T23:54:03ZI1-under-loadPASStarget pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
432222026-08-01T23:56:20ZI1-under-load-recoverPASSafter return degraded=False target=felhom-backup
433222026-08-01T23:59:01ZI7PASSapp=rallly want=C10-C022-A-015704 got=C10-C022-A-015704 snap=helyi restore_ok=True rto=40.8s
434222026-08-01T23:59:50ZKILL-CTRLPASScontroller recovered after kill
435222026-08-02T00:02:06ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
436222026-08-02T00:02:09ZI3PASSadatok bound_under_parent=False while absent
437222026-08-02T00:04:22ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
438222026-08-02T00:04:24ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
439222026-08-02T00:04:27ZI5PASSdegraded=False absent_events=0 target=felhom-backup
440222026-08-02T00:04:27ZI6PASShealthy state must carry no degraded message; got=None
441222026-08-02T00:04:28ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
442222026-08-02T00:04:28ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
443222026-08-02T00:04:30ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
444222026-08-02T00:04:32ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
445222026-08-02T00:04:34ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
446222026-08-02T00:04:36ZI11PASSleaked=[]
447232026-08-02T00:07:31ZI7PASSapp=rallly want=C10-C023-A-020527 got=C10-C023-A-020527 snap=helyi restore_ok=True rto=42.8s
448232026-08-02T00:09:48ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
449232026-08-02T00:09:50ZI3PASSadatok bound_under_parent=False while absent
450232026-08-02T00:12:03ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
451232026-08-02T00:12:07ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
452232026-08-02T00:12:56ZKILL-CTRLPASScontroller recovered after kill
453232026-08-02T00:13:41ZREDEPLOYPASShomebox redeployed
454232026-08-02T00:15:58ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
455232026-08-02T00:16:01ZI3PASSmentes bound_under_parent=False while absent
456232026-08-02T00:18:15ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
457232026-08-02T00:18:18ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
458232026-08-02T00:18:52ZI5PASSdegraded=False absent_events=0 target=felhom-backup
459232026-08-02T00:18:52ZI6PASShealthy state must carry no degraded message; got=None
460232026-08-02T00:18:54ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
461232026-08-02T00:18:54ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
462232026-08-02T00:18:56ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
463232026-08-02T00:18:59ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
464232026-08-02T00:19:01ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
465232026-08-02T00:19:04ZI11PASSleaked=[]
466242026-08-02T00:21:50ZI7PASSapp=rallly want=C10-C024-A-021951 got=C10-C024-A-021951 snap=helyi restore_ok=True rto=30.9s
467242026-08-02T00:24:06ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
468242026-08-02T00:24:09ZI3PASSmentes bound_under_parent=False while absent
469242026-08-02T00:26:22ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
470242026-08-02T00:26:25ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
471242026-08-02T00:27:15ZKILL-CTRLPASScontroller recovered after kill
472242026-08-02T00:29:31ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
473242026-08-02T00:29:34ZI3PASSadatok bound_under_parent=False while absent
474242026-08-02T00:31:47ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
475242026-08-02T00:31:50ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
476242026-08-02T00:32:33ZREDEPLOYPASShomebox redeployed
477242026-08-02T00:35:09ZI4-abortPASSaborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
478242026-08-02T00:35:13ZI3-abortPASScalibre-web must be stopped while its namespace is aborted; running=False
479242026-08-02T00:38:03ZI4-abort-recoverPASSafter re-seat: bound=True app_up=True
480242026-08-02T00:38:06ZI5PASSdegraded=False absent_events=0 target=felhom-backup
481242026-08-02T00:38:06ZI6PASShealthy state must carry no degraded message; got=None
482242026-08-02T00:38:08ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
483242026-08-02T00:38:08ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
484242026-08-02T00:38:10ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
485242026-08-02T00:38:12ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
486242026-08-02T00:38:13ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
487242026-08-02T00:38:15ZI11PASSleaked=[]
488252026-08-02T00:40:33ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
489252026-08-02T00:40:36ZI3PASSmentes bound_under_parent=False while absent
490252026-08-02T00:42:49ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
491252026-08-02T00:42:52ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
492252026-08-02T00:45:08ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
493252026-08-02T00:45:10ZI3PASSadatok bound_under_parent=False while absent
494252026-08-02T00:47:23ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
495252026-08-02T00:47:26ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
496252026-08-02T00:48:44ZREDEPLOYPASSgrafana redeployed
497252026-08-02T00:51:12ZI7PASSapp=rallly want=C10-C025-A-024848 got=C10-C025-A-024848 snap=helyi restore_ok=True rto=34.4s
498252026-08-02T00:52:41ZKILL-AGENTPASSagent active after kill=True; subsequent backup ok=True
499252026-08-02T00:53:31ZKILL-CTRLPASScontroller recovered after kill
500252026-08-02T00:53:34ZI5PASSdegraded=False absent_events=0 target=felhom-backup
501252026-08-02T00:53:34ZI6PASShealthy state must carry no degraded message; got=None
502252026-08-02T00:53:36ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
503252026-08-02T00:53:36ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
504252026-08-02T00:53:37ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
505252026-08-02T00:53:39ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
506252026-08-02T00:53:41ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
507252026-08-02T00:53:44ZI11PASSleaked=[]
508262026-08-02T00:55:16ZKILL-CTRLPASScontroller recovered after kill
509262026-08-02T00:57:32ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
510262026-08-02T00:57:35ZI3PASSmentes bound_under_parent=False while absent
511262026-08-02T00:59:48ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
512262026-08-02T00:59:50ZI4PASSguest init boot before=1785626262 after=1785626262 (must be equal)
513262026-08-02T01:02:01ZI7PASSapp=rallly want=C10-C026-A-025952 got=C10-C026-A-025952 snap=helyi restore_ok=True rto=41.4s
514262026-08-02T01:15:31ZHARD-RESETVIOLATIONVM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
515262026-08-02T01:17:47ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
516262026-08-02T01:17:51ZI3PASSadatok bound_under_parent=False while absent
517262026-08-02T01:20:04ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
518262026-08-02T01:20:06ZI4PASSguest init boot before=1785632702 after=1785632702 (must be equal)
519262026-08-02T01:20:50ZREDEPLOYPASSgrafana redeployed
520262026-08-02T01:20:53ZI5PASSdegraded=False absent_events=0 target=felhom-backup
521262026-08-02T01:20:53ZI6PASShealthy state must carry no degraded message; got=None
522262026-08-02T01:20:55ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
523262026-08-02T01:20:55ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
524262026-08-02T01:20:57ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
525262026-08-02T01:20:58ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
526262026-08-02T01:21:00ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
527262026-08-02T01:21:02ZI11PASSleaked=[]
528272026-08-02T01:22:48ZREBOOT-VMPASScontroller back after full VM reboot
529272026-08-02T01:25:04ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
530272026-08-02T01:25:07ZI3PASSadatok bound_under_parent=False while absent
531272026-08-02T01:27:20ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
532272026-08-02T01:27:22ZI4PASSguest init boot before=1785633757 after=1785633757 (must be equal)
533272026-08-02T01:28:06ZREDEPLOYPASShomebox redeployed
534272026-08-02T01:30:21ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
535272026-08-02T01:30:23ZI3PASSmentes bound_under_parent=False while absent
536272026-08-02T01:32:36ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
537272026-08-02T01:32:39ZI4PASSguest init boot before=1785633757 after=1785633757 (must be equal)
538272026-08-02T01:33:28ZKILL-CTRLPASScontroller recovered after kill
539272026-08-02T01:35:29ZI7PASSapp=rallly want=C10-C027-A-033330 got=C10-C027-A-033330 snap=helyi restore_ok=True rto=41.0s
540272026-08-02T01:35:31ZI5PASSdegraded=False absent_events=0 target=felhom-backup
541272026-08-02T01:35:31ZI6PASShealthy state must carry no degraded message; got=None
542272026-08-02T01:35:33ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
543272026-08-02T01:35:33ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
544272026-08-02T01:35:35ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
545272026-08-02T01:35:37ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
546272026-08-02T01:35:39ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
547272026-08-02T01:35:41ZI11PASSleaked=[]
548282026-08-02T01:37:59ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
549282026-08-02T01:38:02ZI3PASSadatok bound_under_parent=False while absent
550282026-08-02T01:40:15ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
551282026-08-02T01:40:17ZI4PASSguest init boot before=1785633757 after=1785633757 (must be equal)
552282026-08-02T01:41:30ZREDEPLOYPASShomebox redeployed
553282026-08-02T01:43:46ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
554282026-08-02T01:43:49ZI3PASSmentes bound_under_parent=False while absent
555282026-08-02T01:46:02ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
556282026-08-02T01:46:04ZI4PASSguest init boot before=1785633757 after=1785633757 (must be equal)
557282026-08-02T01:48:17ZI7PASSapp=rallly want=C10-C028-A-034606 got=C10-C028-A-034606 snap=helyi restore_ok=True rto=41.8s
558282026-08-02T01:49:06ZKILL-CTRLPASScontroller recovered after kill
559282026-08-02T01:49:45ZREBOOTPASScontroller back after guest reboot
560282026-08-02T01:49:48ZI5PASSdegraded=False absent_events=0 target=felhom-backup
561282026-08-02T01:49:48ZI6PASShealthy state must carry no degraded message; got=None
562282026-08-02T01:49:50ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
563282026-08-02T01:49:50ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
564282026-08-02T01:49:52ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
565282026-08-02T01:49:54ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
566282026-08-02T01:49:56ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
567282026-08-02T01:49:58ZI11PASSleaked=[]
568292026-08-02T01:50:50ZCONCURRENCYPASSbackup+restore raced; restore-status op=None running=False (recorded, not asserted)
569292026-08-02T01:51:39ZKILL-CTRLPASScontroller recovered after kill
570292026-08-02T01:54:37ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
571292026-08-02T01:54:40ZI3PASSadatok bound_under_parent=False while absent
572292026-08-02T01:56:53ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
573292026-08-02T01:56:55ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
574292026-08-02T01:58:53ZI7PASSapp=rallly want=C10-C029-A-035657 got=C10-C029-A-035657 snap=helyi restore_ok=True rto=41.2s
575292026-08-02T01:59:37ZREDEPLOYPASSgrafana redeployed
576292026-08-02T02:01:53ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
577292026-08-02T02:01:55ZI3PASSmentes bound_under_parent=False while absent
578292026-08-02T02:04:08ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
579292026-08-02T02:04:10ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
580292026-08-02T02:04:13ZI5PASSdegraded=False absent_events=0 target=felhom-backup
581292026-08-02T02:04:13ZI6PASShealthy state must carry no degraded message; got=None
582292026-08-02T02:04:15ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
583292026-08-02T02:04:15ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
584292026-08-02T02:04:16ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
585292026-08-02T02:04:18ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
586292026-08-02T02:04:20ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
587292026-08-02T02:04:22ZI11PASSleaked=[]
588302026-08-02T02:05:08ZREDEPLOYPASSgrafana redeployed
589302026-08-02T02:07:28ZI1-under-loadPASStarget pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
590302026-08-02T02:09:46ZI1-under-load-recoverPASSafter return degraded=False target=felhom-backup
591302026-08-02T02:10:35ZKILL-CTRLPASScontroller recovered after kill
592302026-08-02T02:12:51ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
593302026-08-02T02:12:53ZI3PASSmentes bound_under_parent=False while absent
594302026-08-02T02:15:06ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
595302026-08-02T02:15:09ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
596302026-08-02T02:17:54ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
597302026-08-02T02:17:56ZI3PASSadatok bound_under_parent=False while absent
598302026-08-02T02:20:09ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
599302026-08-02T02:20:12ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
600302026-08-02T02:22:22ZI7PASSapp=rallly want=C10-C030-A-042014 got=C10-C030-A-042014 snap=helyi restore_ok=True rto=41.2s
601302026-08-02T02:22:25ZI5PASSdegraded=False absent_events=0 target=felhom-backup
602302026-08-02T02:22:25ZI6PASShealthy state must carry no degraded message; got=None
603302026-08-02T02:22:27ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
604302026-08-02T02:22:27ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
605302026-08-02T02:22:29ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
606302026-08-02T02:22:31ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
607302026-08-02T02:22:33ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
608302026-08-02T02:22:35ZI11PASSleaked=[]
609312026-08-02T02:25:29ZI7PASSapp=rallly want=C10-C031-A-042320 got=C10-C031-A-042320 snap=helyi restore_ok=True rto=41.4s
610312026-08-02T02:26:18ZKILL-CTRLPASScontroller recovered after kill
611312026-08-02T02:29:21ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
612312026-08-02T02:29:24ZI3PASSadatok bound_under_parent=False while absent
613312026-08-02T02:31:37ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
614312026-08-02T02:31:40ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
615312026-08-02T02:32:23ZREDEPLOYPASSgrafana redeployed
616312026-08-02T02:34:39ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
617312026-08-02T02:34:42ZI3PASSmentes bound_under_parent=False while absent
618312026-08-02T02:36:55ZI1-pairVIOLATIONmentes return -> got=None sev=None (want backup_target_restored/info)
619312026-08-02T02:36:57ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
620312026-08-02T02:37:00ZI5PASSdegraded=False absent_events=0 target=felhom-backup
621312026-08-02T02:37:00ZI6PASShealthy state must carry no degraded message; got=None
622312026-08-02T02:37:01ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
623312026-08-02T02:37:01ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
624312026-08-02T02:37:03ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
625312026-08-02T02:37:05ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
626312026-08-02T02:37:07ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
627312026-08-02T02:37:09ZI11PASSleaked=[]
628322026-08-02T02:39:27ZI1VIOLATIONmentes absent -> got=None sev=None (want backup_target_absent/error)
629322026-08-02T02:39:29ZI3PASSmentes bound_under_parent=False while absent
630322026-08-02T02:41:42ZI1-pairVIOLATIONmentes return -> got=None sev=None (want backup_target_restored/info)
631322026-08-02T02:41:45ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
632322026-08-02T02:44:00ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
633322026-08-02T02:44:03ZI3PASSadatok bound_under_parent=False while absent
634322026-08-02T02:46:16ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
635322026-08-02T02:46:18ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
636322026-08-02T02:47:43ZREDEPLOYPASSgrafana redeployed
637322026-08-02T02:50:23ZI4-abortPASSaborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
638322026-08-02T02:50:27ZI3-abortPASScalibre-web must be stopped while its namespace is aborted; running=False
639322026-08-02T02:53:17ZI4-abort-recoverPASSafter re-seat: bound=True app_up=True
640322026-08-02T02:55:16ZI7PASSapp=rallly want=C10-C032-A-045319 got=C10-C032-A-045319 snap=helyi restore_ok=True rto=41.4s
641322026-08-02T02:56:06ZKILL-CTRLPASScontroller recovered after kill
642322026-08-02T02:56:09ZI5PASSdegraded=False absent_events=0 target=felhom-backup
643322026-08-02T02:56:09ZI6PASShealthy state must carry no degraded message; got=None
644322026-08-02T02:56:11ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
645322026-08-02T02:56:11ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
646322026-08-02T02:56:12ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
647322026-08-02T02:56:14ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
648322026-08-02T02:56:16ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
649322026-08-02T02:56:18ZI11PASSleaked=[]
650332026-08-02T02:57:10ZKILL-CTRLPASScontroller recovered after kill
651332026-08-02T02:58:34ZKILL-AGENTPASSagent active after kill=True; subsequent backup ok=True
652332026-08-02T03:00:45ZI7PASSapp=rallly want=C10-C033-A-045848 got=C10-C033-A-045848 snap=helyi restore_ok=True rto=40.7s
653332026-08-02T03:01:29ZREDEPLOYVIOLATIONgrafana redeployed
654332026-08-02T03:03:46ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
655332026-08-02T03:03:48ZI3PASSadatok bound_under_parent=False while absent
656332026-08-02T03:06:00ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
657332026-08-02T03:06:03ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
658332026-08-02T03:08:18ZI1VIOLATIONmentes absent -> got=None sev=None (want backup_target_absent/error)
659332026-08-02T03:08:21ZI3PASSmentes bound_under_parent=False while absent
660332026-08-02T03:10:34ZI1-pairVIOLATIONmentes return -> got=None sev=None (want backup_target_restored/info)
661332026-08-02T03:10:37ZI4PASSguest init boot before=1785635365 after=1785635365 (must be equal)
662332026-08-02T03:11:09ZI5PASSdegraded=False absent_events=0 target=felhom-backup
663332026-08-02T03:11:09ZI6PASShealthy state must carry no degraded message; got=None
664332026-08-02T03:11:11ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
665332026-08-02T03:11:11ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
666332026-08-02T03:11:12ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
667332026-08-02T03:11:15ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
668332026-08-02T03:11:16ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
669332026-08-02T03:11:18ZI11PASSleaked=[]
670342026-08-02T03:24:56ZHARD-RESETVIOLATIONVM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
671342026-08-02T03:25:40ZREDEPLOYPASSgrafana redeployed
672342026-08-02T03:27:53ZI7PASSapp=rallly want=C10-C034-A-052542 got=C10-C034-A-052542 snap=helyi restore_ok=True rto=41.6s
673342026-08-02T03:30:08ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
674342026-08-02T03:30:11ZI3PASSadatok bound_under_parent=False while absent
675342026-08-02T03:32:24ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
676342026-08-02T03:32:27ZI4PASSguest init boot before=1785640463 after=1785640463 (must be equal)
677342026-08-02T03:33:16ZKILL-CTRLPASScontroller recovered after kill
678342026-08-02T03:36:14ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
679342026-08-02T03:36:16ZI3PASSmentes bound_under_parent=False while absent
680342026-08-02T03:38:28ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
681342026-08-02T03:38:31ZI4PASSguest init boot before=1785640463 after=1785640463 (must be equal)
682342026-08-02T03:38:33ZI5PASSdegraded=False absent_events=0 target=felhom-backup
683342026-08-02T03:38:33ZI6PASShealthy state must carry no degraded message; got=None
684342026-08-02T03:38:35ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
685342026-08-02T03:38:35ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
686342026-08-02T03:38:37ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
687342026-08-02T03:38:39ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
688342026-08-02T03:38:41ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
689342026-08-02T03:38:43ZI11PASSleaked=[]
690352026-08-02T03:40:56ZI7PASSapp=rallly want=C10-C035-A-053847 got=C10-C035-A-053847 snap=helyi restore_ok=True rto=41.4s
691352026-08-02T03:41:39ZREDEPLOYPASSgrafana redeployed
692352026-08-02T03:42:28ZKILL-CTRLPASScontroller recovered after kill
693352026-08-02T03:44:44ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
694352026-08-02T03:44:47ZI3PASSadatok bound_under_parent=False while absent
695352026-08-02T03:47:00ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
696352026-08-02T03:47:02ZI4PASSguest init boot before=1785640463 after=1785640463 (must be equal)
697352026-08-02T03:49:17ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
698352026-08-02T03:49:20ZI3PASSmentes bound_under_parent=False while absent
699352026-08-02T03:51:34ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
700352026-08-02T03:51:36ZI4PASSguest init boot before=1785640463 after=1785640463 (must be equal)
701352026-08-02T03:53:33ZREBOOT-VMPASScontroller back after full VM reboot
702352026-08-02T03:53:36ZI5PASSdegraded=False absent_events=0 target=felhom-backup
703352026-08-02T03:53:36ZI6PASShealthy state must carry no degraded message; got=None
704352026-08-02T03:53:38ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
705352026-08-02T03:53:38ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
706352026-08-02T03:53:40ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
707352026-08-02T03:53:42ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
708352026-08-02T03:53:44ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
709352026-08-02T03:53:47ZI11PASSleaked=[]
710362026-08-02T03:56:04ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
711362026-08-02T03:56:07ZI3PASSadatok bound_under_parent=False while absent
712362026-08-02T03:58:20ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
713362026-08-02T03:58:22ZI4PASSguest init boot before=1785642801 after=1785642801 (must be equal)
714362026-08-02T03:59:06ZREDEPLOYPASSgrafana redeployed
715362026-08-02T04:01:15ZI7PASSapp=rallly want=C10-C036-A-055908 got=C10-C036-A-055908 snap=helyi restore_ok=True rto=41.4s
716362026-08-02T04:02:05ZKILL-CTRLPASScontroller recovered after kill
717362026-08-02T04:04:20ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
718362026-08-02T04:04:24ZI3PASSmentes bound_under_parent=False while absent
719362026-08-02T04:06:36ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
720362026-08-02T04:06:39ZI4PASSguest init boot before=1785642801 after=1785642801 (must be equal)
721362026-08-02T04:07:59ZREBOOTPASScontroller back after guest reboot
722362026-08-02T04:08:02ZI5PASSdegraded=False absent_events=0 target=felhom-backup
723362026-08-02T04:08:02ZI6PASShealthy state must carry no degraded message; got=None
724362026-08-02T04:08:04ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
725362026-08-02T04:08:04ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
726362026-08-02T04:08:06ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
727362026-08-02T04:08:08ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
728362026-08-02T04:08:09ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
729362026-08-02T04:08:12ZI11PASSleaked=[]
730372026-08-02T04:09:03ZKILL-CTRLPASScontroller recovered after kill
731372026-08-02T04:11:18ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
732372026-08-02T04:11:21ZI3PASSadatok bound_under_parent=False while absent
733372026-08-02T04:13:34ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
734372026-08-02T04:13:36ZI4PASSguest init boot before=1785643659 after=1785643659 (must be equal)
735372026-08-02T04:15:52ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
736372026-08-02T04:15:54ZI3PASSmentes bound_under_parent=False while absent
737372026-08-02T04:18:07ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
738372026-08-02T04:18:09ZI4PASSguest init boot before=1785643659 after=1785643659 (must be equal)
739372026-08-02T04:20:20ZI7PASSapp=rallly want=C10-C037-A-061811 got=C10-C037-A-061811 snap=helyi restore_ok=True rto=41.5s
740372026-08-02T04:20:58ZCONCURRENCYPASSbackup+restore raced; restore-status op=restore running=False (recorded, not asserted)
741372026-08-02T04:21:41ZREDEPLOYPASSgrafana redeployed
742372026-08-02T04:22:25ZI5PASSdegraded=False absent_events=0 target=felhom-backup
743372026-08-02T04:22:25ZI6PASShealthy state must carry no degraded message; got=None
744372026-08-02T04:22:26ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
745372026-08-02T04:22:26ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
746372026-08-02T04:22:28ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
747372026-08-02T04:22:30ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
748372026-08-02T04:22:32ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
749372026-08-02T04:22:34ZI11PASSleaked=[]
750382026-08-02T04:23:20ZREDEPLOYPASSgrafana redeployed
751382026-08-02T04:25:36ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
752382026-08-02T04:25:39ZI3PASSadatok bound_under_parent=False while absent
753382026-08-02T04:27:52ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
754382026-08-02T04:27:54ZI4PASSguest init boot before=1785643659 after=1785643659 (must be equal)
755382026-08-02T04:30:05ZI7PASSapp=rallly want=C10-C038-A-062756 got=C10-C038-A-062756 snap=helyi restore_ok=True rto=41.6s
756382026-08-02T04:32:21ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
757382026-08-02T04:32:24ZI3PASSmentes bound_under_parent=False while absent
758382026-08-02T04:34:37ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
759382026-08-02T04:34:39ZI4PASSguest init boot before=1785643659 after=1785643659 (must be equal)
760382026-08-02T04:37:00ZI1-under-loadPASStarget pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
761382026-08-02T04:39:18ZI1-under-load-recoverPASSafter return degraded=False target=felhom-backup
762382026-08-02T04:40:07ZKILL-CTRLPASScontroller recovered after kill
763382026-08-02T04:40:39ZI5PASSdegraded=False absent_events=0 target=felhom-backup
764382026-08-02T04:40:39ZI6PASShealthy state must carry no degraded message; got=None
765382026-08-02T04:40:41ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
766382026-08-02T04:40:41ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
767382026-08-02T04:40:43ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
768382026-08-02T04:40:45ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
769382026-08-02T04:40:47ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
770382026-08-02T04:40:50ZI11PASSleaked=[]
771392026-08-02T04:42:23ZKILL-CTRLPASScontroller recovered after kill
772392026-08-02T04:44:25ZI7PASSapp=rallly want=C10-C039-A-064225 got=C10-C039-A-064225 snap=helyi restore_ok=True rto=41.6s
773392026-08-02T04:46:41ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
774392026-08-02T04:46:44ZI3PASSmentes bound_under_parent=False while absent
775392026-08-02T04:48:57ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
776392026-08-02T04:49:00ZI4PASSguest init boot before=1785643659 after=1785643659 (must be equal)
777392026-08-02T04:52:12ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
778392026-08-02T04:52:16ZI3PASSadatok bound_under_parent=False while absent
779392026-08-02T04:54:30ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
780392026-08-02T04:54:32ZI4PASSguest init boot before=1785643659 after=1785643659 (must be equal)
781392026-08-02T04:55:16ZREDEPLOYPASSgrafana redeployed
782392026-08-02T04:55:19ZI5PASSdegraded=False absent_events=0 target=felhom-backup
783392026-08-02T04:55:19ZI6PASShealthy state must carry no degraded message; got=None
784392026-08-02T04:55:21ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
785392026-08-02T04:55:21ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
786392026-08-02T04:55:23ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
787392026-08-02T04:55:25ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
788392026-08-02T04:55:27ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
789392026-08-02T04:55:29ZI11PASSleaked=[]
Reference in New Issue View Git Blame Copy Permalink