admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f7dbc335aca400a515e4e5e2c3092f7185f5abfa
felhom.eu/documentation/tests/campaign10-evidence-2026-07-31/state/runner.log
T
admin 405a795e32 Campaign 10: RESOLVED — the backup_target_* silence was transient and self-recovered; SQLITE_BUSY drops are absorbed by retry 
Both halves of the disposition were run and neither survived as a finding.

The I1/I1-pair violations cluster at cycles 31-33 and nowhere else across 39
cycles; c34-c39 are clean, so it recovered with no intervention. Final tally I1
37 PASS / 2 VIOLATION, I1-pair 36 PASS / 3 VIOLATION. On the quiesced box one slow
detach with 4 minutes either side produced a perfect pair. And the alarming
false-healthy (mentes bound=False while degraded=false) does not survive
quiescence - I had been reading the two halves at different instants of a detach.
No R-n.

Separately cleared: the hub's SQLITE_BUSY event drops. 7 in 24h including one for
the real customer demo-felhom, and the hub does return 500 with notification
dispatch only after a successful save - so a lost event would be a lost alarm. But
the controller retries 3 times and ZERO events exhausted their attempts; the
07:04:39 drop landed at 07:04:42. Nothing lost. Only cosmetic note: the ERROR line
reads like data loss and is not.
2026-08-02 07:10:49 +02:00

99 lines
9.2 KiB
Plaintext
Raw Blame History

19:51:13 === Campaign 10 Phase B start — max 45 cycles, deadline 06:51 ===
19:51:15 cycle 1: detach_nontarget -> kill_agent_mid_backup -> backup -> restore_verify -> detach_target -> redeploy_app -> kill_controller
20:06:56 cycle 1 done — violations so far: 0
20:06:58 cycle 2: redeploy_app -> detach_nontarget -> kill_controller -> backup -> hard_reset_mid_write -> restore_verify -> detach_target
20:24:24 cycle 2 done — violations so far: 0
20:24:26 cycle 3: backup -> redeploy_app -> detach_target -> reboot_vm -> detach_nontarget -> kill_controller -> restore_verify
20:39:28 cycle 3 done — violations so far: 0
20:39:30 cycle 4: detach_nontarget -> redeploy_app -> backup -> reboot_guest -> restore_verify -> detach_target -> kill_controller
20:54:07 cycle 4 done — violations so far: 0
20:54:09 cycle 5: detach_nontarget -> backup -> concurrent_backup_restore -> redeploy_app -> kill_controller -> restore_verify -> detach_target
21:08:49 cycle 5 done — violations so far: 0
21:08:59 cycle 6: restore_verify -> detach_target -> backup -> concurrent_backup_detach -> kill_controller -> redeploy_app -> detach_nontarget
21:27:40 cycle 6 done — violations so far: 0
21:27:42 cycle 7: redeploy_app -> kill_controller -> restore_verify -> fill_drive -> detach_target -> detach_nontarget -> backup
21:41:48 cycle 7 done — violations so far: 0
21:41:50 cycle 8: redeploy_app -> kill_controller -> restore_verify -> backup -> detach_target -> abort_fs_inplace -> detach_nontarget
22:00:49 cycle 8 done — violations so far: 0
22:00:51 cycle 9: redeploy_app -> kill_controller -> kill_agent_mid_backup -> detach_nontarget -> detach_target -> restore_verify -> backup
22:15:56 cycle 9 done — violations so far: 0
22:15:58 cycle 10: detach_target -> detach_nontarget -> hard_reset_mid_write -> redeploy_app -> restore_verify -> backup -> kill_controller
22:38:40 !! HARD-RESET VIOLATION: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
22:51:05 cycle 10 done — violations so far: 1
22:51:07 cycle 11: kill_controller -> reboot_vm -> backup -> detach_target -> detach_nontarget -> restore_verify -> redeploy_app
23:08:30 cycle 11 done — violations so far: 1
23:08:34 cycle 12: backup -> redeploy_app -> detach_nontarget -> kill_controller -> detach_target -> restore_verify -> reboot_guest
23:23:11 cycle 12 done — violations so far: 1
23:23:13 cycle 13: redeploy_app -> backup -> kill_controller -> detach_target -> concurrent_backup_restore -> restore_verify -> detach_nontarget
23:37:30 cycle 13 done — violations so far: 1
23:37:32 cycle 14: backup -> detach_target -> concurrent_backup_detach -> detach_nontarget -> restore_verify -> kill_controller -> redeploy_app
23:55:37 cycle 14 done — violations so far: 1
23:55:39 cycle 15: detach_target -> backup -> detach_nontarget -> kill_controller -> fill_drive -> redeploy_app -> restore_verify
00:10:03 cycle 15 done — violations so far: 1
00:10:05 cycle 16: detach_target -> detach_nontarget -> abort_fs_inplace -> kill_controller -> backup -> redeploy_app -> restore_verify
00:29:08 cycle 16 done — violations so far: 1
00:29:10 cycle 17: kill_agent_mid_backup -> backup -> restore_verify -> detach_nontarget -> kill_controller -> redeploy_app -> detach_target
00:44:32 cycle 17 done — violations so far: 1
00:44:34 cycle 18: backup -> detach_target -> detach_nontarget -> redeploy_app -> restore_verify -> kill_controller -> hard_reset_mid_write
01:02:18 cycle 18 done — violations so far: 1
01:02:20 cycle 19: backup -> kill_controller -> detach_nontarget -> reboot_vm -> detach_target -> redeploy_app -> restore_verify
01:17:21 cycle 19 done — violations so far: 1
01:17:23 cycle 20: reboot_guest -> restore_verify -> redeploy_app -> backup -> detach_nontarget -> kill_controller -> detach_target
01:32:08 cycle 20 done — violations so far: 1
01:32:10 cycle 21: backup -> redeploy_app -> restore_verify -> detach_target -> concurrent_backup_restore -> kill_controller -> detach_nontarget
01:46:22 cycle 21 done — violations so far: 1
01:46:24 cycle 22: redeploy_app -> detach_target -> concurrent_backup_detach -> backup -> restore_verify -> kill_controller -> detach_nontarget
02:04:36 cycle 22 done — violations so far: 1
02:04:38 cycle 23: fill_drive -> restore_verify -> detach_nontarget -> kill_controller -> redeploy_app -> detach_target -> backup
02:19:04 cycle 23 done — violations so far: 1
02:19:06 cycle 24: backup -> restore_verify -> detach_target -> kill_controller -> detach_nontarget -> redeploy_app -> abort_fs_inplace
02:38:15 cycle 24 done — violations so far: 1
02:38:17 cycle 25: detach_target -> detach_nontarget -> backup -> redeploy_app -> restore_verify -> kill_agent_mid_backup -> kill_controller
02:53:44 cycle 25 done — violations so far: 1
02:53:46 cycle 26: backup -> kill_controller -> detach_target -> restore_verify -> hard_reset_mid_write -> detach_nontarget -> redeploy_app
03:15:31 !! HARD-RESET VIOLATION: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
03:21:02 cycle 26 done — violations so far: 2
03:21:04 cycle 27: backup -> reboot_vm -> detach_nontarget -> redeploy_app -> detach_target -> kill_controller -> restore_verify
03:35:41 cycle 27 done — violations so far: 2
03:35:43 cycle 28: detach_nontarget -> backup -> redeploy_app -> detach_target -> restore_verify -> kill_controller -> reboot_guest
03:49:58 cycle 28 done — violations so far: 2
03:50:00 cycle 29: concurrent_backup_restore -> kill_controller -> backup -> detach_nontarget -> restore_verify -> redeploy_app -> detach_target
04:04:22 cycle 29 done — violations so far: 2
04:04:24 cycle 30: redeploy_app -> concurrent_backup_detach -> kill_controller -> detach_target -> backup -> detach_nontarget -> restore_verify
04:22:35 cycle 30 done — violations so far: 2
04:22:37 cycle 31: backup -> restore_verify -> kill_controller -> fill_drive -> detach_nontarget -> redeploy_app -> detach_target
04:36:55 !! I1-pair VIOLATION: mentes return -> got=None sev=None (want backup_target_restored/info)
04:37:09 cycle 31 done — violations so far: 3
04:37:11 cycle 32: detach_target -> detach_nontarget -> backup -> redeploy_app -> abort_fs_inplace -> restore_verify -> kill_controller
04:39:27 !! I1 VIOLATION: mentes absent -> got=None sev=None (want backup_target_absent/error)
04:41:42 !! I1-pair VIOLATION: mentes return -> got=None sev=None (want backup_target_restored/info)
04:56:18 cycle 32 done — violations so far: 5
04:56:20 cycle 33: kill_controller -> kill_agent_mid_backup -> restore_verify -> redeploy_app -> detach_nontarget -> detach_target -> backup
05:01:29 !! REDEPLOY VIOLATION: grafana redeployed
05:08:18 !! I1 VIOLATION: mentes absent -> got=None sev=None (want backup_target_absent/error)
05:10:34 !! I1-pair VIOLATION: mentes return -> got=None sev=None (want backup_target_restored/info)
05:11:18 cycle 33 done — violations so far: 8
05:11:21 cycle 34: hard_reset_mid_write -> redeploy_app -> restore_verify -> detach_nontarget -> kill_controller -> backup -> detach_target
05:24:56 !! HARD-RESET VIOLATION: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
05:38:43 cycle 34 done — violations so far: 9
05:38:45 cycle 35: restore_verify -> redeploy_app -> kill_controller -> detach_nontarget -> detach_target -> backup -> reboot_vm
05:53:47 cycle 35 done — violations so far: 9
05:53:49 cycle 36: detach_nontarget -> redeploy_app -> restore_verify -> kill_controller -> detach_target -> backup -> reboot_guest
06:08:12 cycle 36 done — violations so far: 9
06:08:14 cycle 37: kill_controller -> detach_nontarget -> detach_target -> restore_verify -> concurrent_backup_restore -> redeploy_app -> backup
06:22:34 cycle 37 done — violations so far: 9
06:22:36 cycle 38: redeploy_app -> detach_nontarget -> restore_verify -> detach_target -> concurrent_backup_detach -> kill_controller -> backup
06:40:50 cycle 38 done — violations so far: 9
06:40:52 cycle 39: backup -> kill_controller -> restore_verify -> detach_target -> fill_drive -> detach_nontarget -> redeploy_app
06:55:29 cycle 39 done — violations so far: 9
06:55:29 === finished after 39 cycles, 9 violations ===
06:55:29 cycle 10 HARD-RESET: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
06:55:29 cycle 26 HARD-RESET: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
06:55:29 cycle 31 I1-pair: mentes return -> got=None sev=None (want backup_target_restored/info)
06:55:29 cycle 32 I1: mentes absent -> got=None sev=None (want backup_target_absent/error)
06:55:29 cycle 32 I1-pair: mentes return -> got=None sev=None (want backup_target_restored/info)
06:55:29 cycle 33 REDEPLOY: grafana redeployed
06:55:29 cycle 33 I1: mentes absent -> got=None sev=None (want backup_target_absent/error)
06:55:29 cycle 33 I1-pair: mentes return -> got=None sev=None (want backup_target_restored/info)
06:55:29 cycle 34 HARD-RESET: VM returned=True apps_ready=False canaries_intact=None — apps did not return within 10 min (that IS the failure)
Reference in New Issue View Git Blame Copy Permalink