Files

T

admin 2d64ee7241 Campaign 10: OPEN observation — backup_target_* pair went silent under rapid cycling

Three I1/I1-pair violations in ~5 minutes, all "expected event absent". Recorded as
an OPEN observation, NOT a finding: the system was mid-abuse when it was seen, and
a verdict taken on a system being hammered is worth little.

Established: it is not hub-side suppression and not a truncated log. The hub pod
has 0 restarts over 43h and the controller's own log matches it line for line, so
the events were never emitted. It is specific to the backup_target_* pair - the
generic storage_disconnected/reconnected pair for the other drive kept firing
normally throughout the same window.

Also sampled, and the more serious half if it survives quiescence: mentes reads
bound_under_parent=False while the backup-target state simultaneously reports
degraded=false. Those cannot both be right - a false healthy on the backup target
is I5/I6's failure mode.

NOT established: whether the pair recovers once cycling stops (the harness detaches
every ~2 min; a customer does not), whether the 02:25:37 controller restart is
implicated, and whether the degraded=false sample was transient.

Disposition written into the doc: after the run ends, quiesce with both drives
attached, then do ONE slow detach/reattach and see whether the pair fires. That
distinguishes "does not survive rapid cycling" from "the target alarm has silently
stopped working", which would be severe.

2026-08-02 04:44:51 +02:00

3.7 KiB

Raw Blame History

OPEN OBSERVATION — the `backup_target_*` pair stopped firing under rapid detach/reattach cycling

Not filed as a finding yet. The system was still being hammered when this was observed, and a verdict taken on a system mid-abuse is worth little. The disposition below says exactly what would settle it.

What was observed (2026-08-02, run 2b, cycles ~30-32)

Three violations in ~5 minutes, all of the same shape — the expected event simply absent:

04:36:55  I1-pair  mentes return -> got=None sev=None   (want backup_target_restored/info)
04:39:27  I1       mentes absent -> got=None sev=None   (want backup_target_absent/error)
04:41:42  I1-pair  mentes return -> got=None sev=None   (want backup_target_restored/info)

It is NOT hub-side suppression, and NOT a truncated log

The hub pod has 0 restarts over 43 h, so its log is intact, and the controller's own log matches the hub's line for line — the events were never emitted, not emitted-and-dropped:

controller  02:32:34  Event pushed: backup_target_absent (error) — … Mentes …
controller  02:32:38  Event pushed: whole_guest_backup_failed (error) — … felhom-backup tier …
controller  02:35:32  Event pushed: health_degraded (warning) …
controller  02:42:08  Event pushed: storage_disconnected (error) — … Adatok      <-- generic path STILL WORKS
(no backup_target_restored, and no second backup_target_absent, at any point after 02:32:34)

So this is controller-side emission, and it is specific to the backup_target_* pair: the generic storage_disconnected / storage_reconnected pair for the other drive kept firing normally throughout the same window (02:16:02, 02:18:21, 02:27:38, 02:29:35, 02:42:08).

The internally inconsistent state alongside it

Sampled while it was happening:

backup-target:  {"degraded": false, "known": true, "label": "Mentes", "target": "felhom-backup"}
drives:         adatok  bound_under_parent=False  backup_target=None
                mentes  bound_under_parent=False  backup_target=True

mentes is the backup target, reads not bound, and the target state simultaneously reports degraded: false. Those two cannot both be right. If that survives quiescence it is the more serious half of this observation — a false healthy on the backup target, which is I5/I6's failure mode and worse than a missing event.

What is NOT yet established

Whether the pair recovers once the cycling stops. The harness detaches and reattaches roughly every two minutes; a real customer does not. Rapid cycling may be outside the design envelope, in which case this is a robustness note rather than a defect.
Whether the controller restart at 02:25:37 is implicated. In this controller instance there was 1 backup_target_absent and 0 backup_target_restored; the previous instance emitted complete pairs (02:13:21 restored). That is suggestive, not established.
Whether the degraded: false reading was a transient mid-flux sample or a stuck state.

Disposition — how to settle it

After run 2b ends, let the box quiesce with both drives attached, then:

confirm bound_under_parent returns to True for both drives and the target still reads healthy — if it does not, the false-healthy is real and this becomes a finding;
do one slow detach/reattach of mentes with several minutes either side, and check the pair fires. If it does, the observation is "the pair does not survive rapid cycling"; if it does not, the target alarm has silently stopped working, which is severe — an alarm that never fires again after its first firing is worse than no alarm.

Only then does it earn an R-n or a retraction.

3.7 KiB Raw Blame History

OPEN OBSERVATION — the backup_target_* pair stopped firing under rapid detach/reattach cycling