From 2d64ee7241a319e32c2e74af48559bc82b104689 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 04:44:51 +0200 Subject: [PATCH] =?UTF-8?q?Campaign=2010:=20OPEN=20observation=20=E2=80=94?= =?UTF-8?q?=20backup=5Ftarget=5F*=20pair=20went=20silent=20under=20rapid?= =?UTF-8?q?=20cycling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three I1/I1-pair violations in ~5 minutes, all "expected event absent". Recorded as an OPEN observation, NOT a finding: the system was mid-abuse when it was seen, and a verdict taken on a system being hammered is worth little. Established: it is not hub-side suppression and not a truncated log. The hub pod has 0 restarts over 43h and the controller's own log matches it line for line, so the events were never emitted. It is specific to the backup_target_* pair - the generic storage_disconnected/reconnected pair for the other drive kept firing normally throughout the same window. Also sampled, and the more serious half if it survives quiescence: mentes reads bound_under_parent=False while the backup-target state simultaneously reports degraded=false. Those cannot both be right - a false healthy on the backup target is I5/I6's failure mode. NOT established: whether the pair recovers once cycling stops (the harness detaches every ~2 min; a customer does not), whether the 02:25:37 controller restart is implicated, and whether the degraded=false sample was transient. Disposition written into the doc: after the run ends, quiesce with both drives attached, then do ONE slow detach/reattach and see whether the pair fires. That distinguishes "does not survive rapid cycling" from "the target alarm has silently stopped working", which would be severe. --- .../OPEN-backup-target-pair-went-silent.md | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 documentation/tests/campaign10-evidence-2026-07-31/OPEN-backup-target-pair-went-silent.md diff --git a/documentation/tests/campaign10-evidence-2026-07-31/OPEN-backup-target-pair-went-silent.md b/documentation/tests/campaign10-evidence-2026-07-31/OPEN-backup-target-pair-went-silent.md new file mode 100644 index 0000000..5013d0a --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/OPEN-backup-target-pair-went-silent.md @@ -0,0 +1,64 @@ +# OPEN OBSERVATION — the `backup_target_*` pair stopped firing under rapid detach/reattach cycling + +**Not filed as a finding yet.** The system was still being hammered when this was observed, and a +verdict taken on a system mid-abuse is worth little. The disposition below says exactly what would +settle it. + +## What was observed (2026-08-02, run 2b, cycles ~30-32) + +Three violations in ~5 minutes, all of the same shape — the expected event simply absent: + + 04:36:55 I1-pair mentes return -> got=None sev=None (want backup_target_restored/info) + 04:39:27 I1 mentes absent -> got=None sev=None (want backup_target_absent/error) + 04:41:42 I1-pair mentes return -> got=None sev=None (want backup_target_restored/info) + +## It is NOT hub-side suppression, and NOT a truncated log + +The hub pod has **0 restarts over 43 h**, so its log is intact, and the controller's own log matches +the hub's line for line — the events were **never emitted**, not emitted-and-dropped: + + controller 02:32:34 Event pushed: backup_target_absent (error) — … Mentes … + controller 02:32:38 Event pushed: whole_guest_backup_failed (error) — … felhom-backup tier … + controller 02:35:32 Event pushed: health_degraded (warning) … + controller 02:42:08 Event pushed: storage_disconnected (error) — … Adatok <-- generic path STILL WORKS + (no backup_target_restored, and no second backup_target_absent, at any point after 02:32:34) + +So this is controller-side emission, and it is **specific to the `backup_target_*` pair**: the generic +`storage_disconnected` / `storage_reconnected` pair for the *other* drive kept firing normally +throughout the same window (02:16:02, 02:18:21, 02:27:38, 02:29:35, 02:42:08). + +## The internally inconsistent state alongside it + +Sampled while it was happening: + + backup-target: {"degraded": false, "known": true, "label": "Mentes", "target": "felhom-backup"} + drives: adatok bound_under_parent=False backup_target=None + mentes bound_under_parent=False backup_target=True + +`mentes` is the backup target, reads **not bound**, and the target state simultaneously reports +**`degraded: false`**. Those two cannot both be right. If that survives quiescence it is the more +serious half of this observation — a **false healthy** on the backup target, which is I5/I6's +failure mode and worse than a missing event. + +## What is NOT yet established + +- Whether the pair recovers once the cycling stops. The harness detaches and reattaches roughly every + two minutes; a real customer does not. Rapid cycling may be outside the design envelope, in which + case this is a robustness note rather than a defect. +- Whether the controller restart at 02:25:37 is implicated. In this controller instance there was + **1 `backup_target_absent` and 0 `backup_target_restored`**; the previous instance emitted complete + pairs (02:13:21 restored). That is suggestive, not established. +- Whether the `degraded: false` reading was a transient mid-flux sample or a stuck state. + +## Disposition — how to settle it + +After run 2b ends, let the box **quiesce with both drives attached**, then: + +1. confirm `bound_under_parent` returns to `True` for both drives and the target still reads healthy — + if it does not, the false-healthy is real and this becomes a finding; +2. do **one** slow detach/reattach of `mentes` with several minutes either side, and check the pair + fires. If it does, the observation is "the pair does not survive rapid cycling"; if it does not, + the target alarm has silently stopped working, which is severe — an alarm that never fires again + after its first firing is worse than no alarm. + +Only then does it earn an `R-n` or a retraction.