Campaign 10: OPEN observation — backup_target_* pair went silent under rapid cycling

Three I1/I1-pair violations in ~5 minutes, all "expected event absent". Recorded as an OPEN observation, NOT a finding: the system was mid-abuse when it was seen, and a verdict taken on a system being hammered is worth little. Established: it is not hub-side suppression and not a truncated log. The hub pod has 0 restarts over 43h and the controller's own log matches it line for line, so the events were never emitted. It is specific to the backup_target_* pair - the generic storage_disconnected/reconnected pair for the other drive kept firing normally throughout the same window. Also sampled, and the more serious half if it survives quiescence: mentes reads bound_under_parent=False while the backup-target state simultaneously reports degraded=false. Those cannot both be right - a false healthy on the backup target is I5/I6's failure mode. NOT established: whether the pair recovers once cycling stops (the harness detaches every ~2 min; a customer does not), whether the 02:25:37 controller restart is implicated, and whether the degraded=false sample was transient. Disposition written into the doc: after the run ends, quiesce with both drives attached, then do ONE slow detach/reattach and see whether the pair fires. That distinguishes "does not survive rapid cycling" from "the target alarm has silently stopped working", which would be severe.
2026-08-02 04:44:51 +02:00
parent 3d4c5365c1
commit 2d64ee7241
1 changed files with 64 additions and 0 deletions
@@ -0,0 +1,64 @@
+# OPEN OBSERVATION — the `backup_target_*` pair stopped firing under rapid detach/reattach cycling
+
+**Not filed as a finding yet.** The system was still being hammered when this was observed, and a
+verdict taken on a system mid-abuse is worth little. The disposition below says exactly what would
+settle it.
+
+## What was observed (2026-08-02, run 2b, cycles ~30-32)
+
+Three violations in ~5 minutes, all of the same shape — the expected event simply absent:
+
+    04:36:55  I1-pair  mentes return -> got=None sev=None   (want backup_target_restored/info)
+    04:39:27  I1       mentes absent -> got=None sev=None   (want backup_target_absent/error)
+    04:41:42  I1-pair  mentes return -> got=None sev=None   (want backup_target_restored/info)
+
+## It is NOT hub-side suppression, and NOT a truncated log
+
+The hub pod has **0 restarts over 43 h**, so its log is intact, and the controller's own log matches
+the hub's line for line — the events were **never emitted**, not emitted-and-dropped:
+
+    controller  02:32:34  Event pushed: backup_target_absent (error) — … Mentes …
+    controller  02:32:38  Event pushed: whole_guest_backup_failed (error) — … felhom-backup tier …
+    controller  02:35:32  Event pushed: health_degraded (warning) …
+    controller  02:42:08  Event pushed: storage_disconnected (error) — … Adatok      <-- generic path STILL WORKS
+    (no backup_target_restored, and no second backup_target_absent, at any point after 02:32:34)
+
+So this is controller-side emission, and it is **specific to the `backup_target_*` pair**: the generic
+`storage_disconnected` / `storage_reconnected` pair for the *other* drive kept firing normally
+throughout the same window (02:16:02, 02:18:21, 02:27:38, 02:29:35, 02:42:08).
+
+## The internally inconsistent state alongside it
+
+Sampled while it was happening:
+
+    backup-target:  {"degraded": false, "known": true, "label": "Mentes", "target": "felhom-backup"}
+    drives:         adatok  bound_under_parent=False  backup_target=None
+                    mentes  bound_under_parent=False  backup_target=True
+
+`mentes` is the backup target, reads **not bound**, and the target state simultaneously reports
+**`degraded: false`**. Those two cannot both be right. If that survives quiescence it is the more
+serious half of this observation — a **false healthy** on the backup target, which is I5/I6's
+failure mode and worse than a missing event.
+
+## What is NOT yet established
+
+- Whether the pair recovers once the cycling stops. The harness detaches and reattaches roughly every
+  two minutes; a real customer does not. Rapid cycling may be outside the design envelope, in which
+  case this is a robustness note rather than a defect.
+- Whether the controller restart at 02:25:37 is implicated. In this controller instance there was
+  **1 `backup_target_absent` and 0 `backup_target_restored`**; the previous instance emitted complete
+  pairs (02:13:21 restored). That is suggestive, not established.
+- Whether the `degraded: false` reading was a transient mid-flux sample or a stuck state.
+
+## Disposition — how to settle it
+
+After run 2b ends, let the box **quiesce with both drives attached**, then:
+
+1. confirm `bound_under_parent` returns to `True` for both drives and the target still reads healthy —
+   if it does not, the false-healthy is real and this becomes a finding;
+2. do **one** slow detach/reattach of `mentes` with several minutes either side, and check the pair
+   fires. If it does, the observation is "the pair does not survive rapid cycling"; if it does not,
+   the target alarm has silently stopped working, which is severe — an alarm that never fires again
+   after its first firing is worse than no alarm.
+
+Only then does it earn an `R-n` or a retraction.