admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
259f48aedc05e38c00a67674357efbce6e692e2f
felhom.eu/documentation/tests/campaign10-evidence-2026-07-31/state/journal-run2a.tsv
T
admin 9ca57e591b Campaign 10: two run-2a violations were HARNESS defects, not product defects — fixed 
Run 2a hit its first two violations at cycle 10 and BOTH trace to my harness, not
the product. Recorded in full because a check that fails for the wrong reason is
as corrosive as one that passes for the wrong reason.

  HARD-RESET  VM returned=True canaries_intact=False
  I7          want=C10-C010-A-194530 got=C10-C009-A-192929 restore_ok=True

Root cause, evidenced: the cc_proof table's highest row is C10-C009-A — there is
NO C010-A row at all, so the seed never landed. The hard-reset atom ran earlier in
the same cycle and left rallly Exited(255); atom_restore_verify called seed() and
never checked its return value, so an unwritten generation became a fake stale
2026-08-01 19:51:57 +02:00

22 KiB
Raw Blame History

1cycleutcinvariantverdictdetail
212026-08-01T15:13:03ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
312026-08-01T15:13:06ZI3PASSadatok bound_under_parent=False while absent
412026-08-01T15:15:18ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
512026-08-01T15:15:21ZI4PASSguest init boot before=1785586762 after=1785586762 (must be equal)
612026-08-01T15:16:45ZKILL-AGENTPASSagent active after kill=True; subsequent backup ok=True
712026-08-01T15:19:24ZI7PASSapp=rallly want=C10-C001-A-171728 got=C10-C001-A-171728 snap=helyi restore_ok=True rto=41.5s
812026-08-01T15:21:39ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
912026-08-01T15:21:42ZI3PASSmentes bound_under_parent=False while absent
1012026-08-01T15:23:55ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
1112026-08-01T15:23:58ZI4PASSguest init boot before=1785586762 after=1785586762 (must be equal)
1212026-08-01T15:24:42ZREDEPLOYPASSgrafana redeployed
1312026-08-01T15:25:31ZKILL-CTRLPASScontroller recovered after kill
1412026-08-01T15:25:34ZI5PASSdegraded=False absent_events=0 target=felhom-backup
1512026-08-01T15:25:34ZI6PASShealthy state must carry no degraded message; got=None
1612026-08-01T15:25:36ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
1712026-08-01T15:25:36ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
1812026-08-01T15:25:38ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
1912026-08-01T15:25:40ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
2012026-08-01T15:25:42ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
2112026-08-01T15:25:44ZI11PASSleaked=[]
2222026-08-01T15:26:30ZREDEPLOYPASShomebox redeployed
2322026-08-01T15:28:46ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
2422026-08-01T15:28:49ZI3PASSadatok bound_under_parent=False while absent
2522026-08-01T15:31:01ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
2622026-08-01T15:31:04ZI4PASSguest init boot before=1785586762 after=1785586762 (must be equal)
2722026-08-01T15:31:53ZKILL-CTRLPASScontroller recovered after kill
2822026-08-01T15:35:56ZHARD-RESETPASSVM returned=True canaries_intact=True (a hard reset mid-write must not corrupt them)
2922026-08-01T15:37:51ZI7PASSapp=rallly want=C10-C002-A-173556 got=C10-C002-A-173556 snap=helyi restore_ok=True rto=41.8s
3022026-08-01T15:40:07ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
3122026-08-01T15:40:10ZI3PASSmentes bound_under_parent=False while absent
3222026-08-01T15:42:23ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
3322026-08-01T15:42:25ZI4PASSguest init boot before=1785598524 after=1785598524 (must be equal)
3422026-08-01T15:42:27ZI5PASSdegraded=False absent_events=0 target=felhom-backup
3522026-08-01T15:42:27ZI6PASShealthy state must carry no degraded message; got=None
3622026-08-01T15:42:29ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
3722026-08-01T15:42:29ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
3822026-08-01T15:42:31ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
3922026-08-01T15:42:33ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
4022026-08-01T15:42:35ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
4122026-08-01T15:42:37ZI11PASSleaked=[]
4232026-08-01T15:43:53ZREDEPLOYPASSgrafana redeployed
4332026-08-01T15:46:09ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
4432026-08-01T15:46:12ZI3PASSmentes bound_under_parent=False while absent
4532026-08-01T15:48:25ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
4632026-08-01T15:48:27ZI4PASSguest init boot before=1785598524 after=1785598524 (must be equal)
4732026-08-01T15:49:46ZREBOOT-VMPASScontroller back after full VM reboot
4832026-08-01T15:52:03ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
4932026-08-01T15:52:05ZI3PASSadatok bound_under_parent=False while absent
5032026-08-01T15:54:18ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
5132026-08-01T15:54:20ZI4PASSguest init boot before=1785599371 after=1785599371 (must be equal)
5232026-08-01T15:55:09ZKILL-CTRLPASScontroller recovered after kill
5332026-08-01T15:56:52ZI7PASSapp=rallly want=C10-C003-A-175509 got=C10-C003-A-175509 snap=helyi restore_ok=True rto=29.4s
5432026-08-01T15:56:55ZI5PASSdegraded=False absent_events=0 target=felhom-backup
5532026-08-01T15:56:55ZI6PASShealthy state must carry no degraded message; got=None
5632026-08-01T15:56:56ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
5732026-08-01T15:56:56ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
5832026-08-01T15:56:58ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
5932026-08-01T15:57:00ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
6032026-08-01T15:57:03ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
6132026-08-01T15:57:05ZI11PASSleaked=[]
6242026-08-01T15:59:23ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
6342026-08-01T15:59:26ZI3PASSadatok bound_under_parent=False while absent
6442026-08-01T16:01:39ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
6542026-08-01T16:01:42ZI4PASSguest init boot before=1785599371 after=1785599371 (must be equal)
6642026-08-01T16:02:25ZREDEPLOYPASShomebox redeployed
6742026-08-01T16:03:33ZREBOOTPASScontroller back after guest reboot
6842026-08-01T16:05:30ZI7PASSapp=rallly want=C10-C004-A-180333 got=C10-C004-A-180333 snap=helyi restore_ok=True rto=41.2s
6942026-08-01T16:07:46ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
7042026-08-01T16:07:49ZI3PASSmentes bound_under_parent=False while absent
7142026-08-01T16:10:02ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
7242026-08-01T16:10:05ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
7342026-08-01T16:10:55ZKILL-CTRLPASScontroller recovered after kill
7442026-08-01T16:10:57ZI5PASSdegraded=False absent_events=0 target=felhom-backup
7542026-08-01T16:10:57ZI6PASShealthy state must carry no degraded message; got=None
7642026-08-01T16:10:59ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
7742026-08-01T16:10:59ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
7842026-08-01T16:11:01ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
7942026-08-01T16:11:03ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
8042026-08-01T16:11:05ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
8142026-08-01T16:11:07ZI11PASSleaked=[]
8252026-08-01T16:13:25ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
8352026-08-01T16:13:28ZI3PASSadatok bound_under_parent=False while absent
8452026-08-01T16:15:41ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
8552026-08-01T16:15:44ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
8652026-08-01T16:16:55ZCONCURRENCYPASSbackup+restore raced; restore-status op=None running=False (recorded, not asserted)
8752026-08-01T16:17:39ZREDEPLOYPASSgrafana redeployed
8852026-08-01T16:18:29ZKILL-CTRLPASScontroller recovered after kill
8952026-08-01T16:20:14ZI7PASSapp=rallly want=C10-C005-A-181829 got=C10-C005-A-181829 snap=helyi restore_ok=True rto=43.7s
9052026-08-01T16:22:30ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
9152026-08-01T16:22:33ZI3PASSmentes bound_under_parent=False while absent
9252026-08-01T16:24:46ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
9352026-08-01T16:24:48ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
9452026-08-01T16:24:51ZI5PASSdegraded=False absent_events=0 target=felhom-backup
9552026-08-01T16:24:51ZI6PASShealthy state must carry no degraded message; got=None
9652026-08-01T16:24:53ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
9752026-08-01T16:24:53ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
9852026-08-01T16:24:56ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
9952026-08-01T16:24:58ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
10052026-08-01T16:25:00ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
10152026-08-01T16:25:02ZI11PASSleaked=[]
10262026-08-01T16:26:50ZI7PASSapp=rallly want=C10-C006-A-182504 got=C10-C006-A-182504 snap=helyi restore_ok=True rto=41.9s
10362026-08-01T16:29:05ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
10462026-08-01T16:29:07ZI3PASSmentes bound_under_parent=False while absent
10562026-08-01T16:31:20ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
10662026-08-01T16:31:23ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
10762026-08-01T16:34:16ZI1-under-loadPASStarget pulled DURING a backup -> got=backup_target_absent (want backup_target_absent)
10862026-08-01T16:36:33ZI1-under-load-recoverPASSafter return degraded=False target=felhom-backup
10962026-08-01T16:37:23ZKILL-CTRLPASScontroller recovered after kill
11062026-08-01T16:38:08ZREDEPLOYPASShomebox redeployed
11162026-08-01T16:40:25ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
11262026-08-01T16:40:28ZI3PASSadatok bound_under_parent=False while absent
11362026-08-01T16:42:40ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
11462026-08-01T16:42:43ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
11562026-08-01T16:42:46ZI5PASSdegraded=False absent_events=0 target=felhom-backup
11662026-08-01T16:42:46ZI6PASShealthy state must carry no degraded message; got=None
11762026-08-01T16:42:48ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
11862026-08-01T16:42:48ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
11962026-08-01T16:42:50ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
12062026-08-01T16:42:52ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
12162026-08-01T16:42:55ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
12262026-08-01T16:42:58ZI11PASSleaked=[]
12372026-08-01T16:43:50ZREDEPLOYPASSgrafana redeployed
12472026-08-01T16:44:40ZKILL-CTRLPASScontroller recovered after kill
12572026-08-01T16:46:32ZI7PASSapp=rallly want=C10-C007-A-184440 got=C10-C007-A-184440 snap=helyi restore_ok=True rto=33.4s
12672026-08-01T16:49:29ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
12772026-08-01T16:49:33ZI3PASSmentes bound_under_parent=False while absent
12872026-08-01T16:51:46ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
12972026-08-01T16:51:49ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
13072026-08-01T16:54:06ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
13172026-08-01T16:54:09ZI3PASSadatok bound_under_parent=False while absent
13272026-08-01T16:56:23ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
13372026-08-01T16:56:28ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
13472026-08-01T16:57:08ZI5PASSdegraded=False absent_events=0 target=felhom-backup
13572026-08-01T16:57:08ZI6PASShealthy state must carry no degraded message; got=None
13672026-08-01T16:57:11ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
13772026-08-01T16:57:11ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
13872026-08-01T16:57:13ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
13972026-08-01T16:57:17ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
14072026-08-01T16:57:22ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
14172026-08-01T16:57:24ZI11PASSleaked=[]
14282026-08-01T16:58:14ZREDEPLOYPASShomebox redeployed
14382026-08-01T16:59:05ZKILL-CTRLPASScontroller recovered after kill
14482026-08-01T17:01:12ZI7PASSapp=rallly want=C10-C008-A-185905 got=C10-C008-A-185905 snap=helyi restore_ok=True rto=46.6s
14582026-08-01T17:04:16ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
14682026-08-01T17:04:25ZI3PASSmentes bound_under_parent=False while absent
14782026-08-01T17:06:41ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
14882026-08-01T17:06:45ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
14982026-08-01T17:09:24ZI4-abortPASSaborted-in-place: bound=False (want False), device still present=True, opts= rw,relatime,abort,emergency_ro 0 0
15082026-08-01T17:09:31ZI3-abortPASScalibre-web must be stopped while its namespace is aborted; running=False
15182026-08-01T17:12:29ZI4-abort-recoverPASSafter re-seat: bound=True app_up=True
15282026-08-01T17:14:47ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
15382026-08-01T17:14:51ZI3PASSadatok bound_under_parent=False while absent
15482026-08-01T17:17:03ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
15582026-08-01T17:17:06ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
15682026-08-01T17:17:08ZI5PASSdegraded=False absent_events=0 target=felhom-backup
15782026-08-01T17:17:08ZI6PASShealthy state must carry no degraded message; got=None
15882026-08-01T17:17:10ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
15982026-08-01T17:17:10ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
16082026-08-01T17:17:12ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
16182026-08-01T17:17:15ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
16282026-08-01T17:17:17ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
16382026-08-01T17:17:19ZI11PASSleaked=[]
16492026-08-01T17:18:05ZREDEPLOYPASSgrafana redeployed
16592026-08-01T17:18:56ZKILL-CTRLPASScontroller recovered after kill
16692026-08-01T17:20:21ZKILL-AGENTPASSagent active after kill=True; subsequent backup ok=True
16792026-08-01T17:22:37ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
16892026-08-01T17:22:40ZI3PASSadatok bound_under_parent=False while absent
16992026-08-01T17:24:53ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
17092026-08-01T17:24:56ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
17192026-08-01T17:27:12ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
17292026-08-01T17:27:14ZI3PASSmentes bound_under_parent=False while absent
17392026-08-01T17:29:27ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
17492026-08-01T17:29:29ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
17592026-08-01T17:31:11ZI7PASSapp=rallly want=C10-C009-A-192929 got=C10-C009-A-192929 snap=helyi restore_ok=True rto=41.1s
17692026-08-01T17:31:55ZI5PASSdegraded=False absent_events=0 target=felhom-backup
17792026-08-01T17:31:55ZI6PASShealthy state must carry no degraded message; got=None
17892026-08-01T17:31:57ZI10PASSrallly/SECRET_PASSWORD type=secret must travel; in_unit=True
17992026-08-01T17:31:57ZI10PASSrallly/DB_PASSWORD type=secret must travel; in_unit=True
18092026-08-01T17:31:59ZI10PASShomebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
18192026-08-01T17:32:00ZI10PASSgrafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
18292026-08-01T17:32:02ZI10PASSpapra/AUTH_SECRET type=secret must travel; in_unit=True
18392026-08-01T17:32:05ZI11PASSleaked=[]
184102026-08-01T17:34:22ZI1PASSmentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
185102026-08-01T17:34:25ZI3PASSmentes bound_under_parent=False while absent
186102026-08-01T17:36:38ZI1-pairPASSmentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
187102026-08-01T17:36:41ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
188102026-08-01T17:38:57ZI2PASSadatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
189102026-08-01T17:38:59ZI3PASSadatok bound_under_parent=False while absent
190102026-08-01T17:41:12ZI2-pairPASSadatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
191102026-08-01T17:41:14ZI4PASSguest init boot before=1785600193 after=1785600193 (must be equal)
192102026-08-01T17:44:47ZHARD-RESETVIOLATIONVM returned=True canaries_intact=False (a hard reset mid-write must not corrupt them)
193102026-08-01T17:45:30ZREDEPLOYPASShomebox redeployed
194102026-08-01T17:47:12ZI7VIOLATIONapp=rallly want=C10-C010-A-194530 got=C10-C009-A-192929 snap=helyi restore_ok=True rto=41.0s
Reference in New Issue View Git Blame Copy Permalink