admin
2d64ee7241
Three I1/I1-pair violations in ~5 minutes, all "expected event absent". Recorded as an OPEN observation, NOT a finding: the system was mid-abuse when it was seen, and a verdict taken on a system being hammered is worth little. Established: it is not hub-side suppression and not a truncated log. The hub pod has 0 restarts over 43h and the controller's own log matches it line for line, so the events were never emitted. It is specific to the backup_target_* pair - the generic storage_disconnected/reconnected pair for the other drive kept firing normally throughout the same window. Also sampled, and the more serious half if it survives quiescence: mentes reads bound_under_parent=False while the backup-target state simultaneously reports degraded=false. Those cannot both be right - a false healthy on the backup target is I5/I6's failure mode. NOT established: whether the pair recovers once cycling stops (the harness detaches every ~2 min; a customer does not), whether the 02:25:37 controller restart is implicated, and whether the degraded=false sample was transient. Disposition written into the doc: after the run ends, quiesce with both drives attached, then do ONE slow detach/reattach and see whether the pair fires. That distinguishes "does not survive rapid cycling" from "the target alarm has silently stopped working", which would be severe.
65 lines
3.7 KiB
Markdown
65 lines
3.7 KiB
Markdown
# OPEN OBSERVATION — the `backup_target_*` pair stopped firing under rapid detach/reattach cycling
|
|
|
|
**Not filed as a finding yet.** The system was still being hammered when this was observed, and a
|
|
verdict taken on a system mid-abuse is worth little. The disposition below says exactly what would
|
|
settle it.
|
|
|
|
## What was observed (2026-08-02, run 2b, cycles ~30-32)
|
|
|
|
Three violations in ~5 minutes, all of the same shape — the expected event simply absent:
|
|
|
|
04:36:55 I1-pair mentes return -> got=None sev=None (want backup_target_restored/info)
|
|
04:39:27 I1 mentes absent -> got=None sev=None (want backup_target_absent/error)
|
|
04:41:42 I1-pair mentes return -> got=None sev=None (want backup_target_restored/info)
|
|
|
|
## It is NOT hub-side suppression, and NOT a truncated log
|
|
|
|
The hub pod has **0 restarts over 43 h**, so its log is intact, and the controller's own log matches
|
|
the hub's line for line — the events were **never emitted**, not emitted-and-dropped:
|
|
|
|
controller 02:32:34 Event pushed: backup_target_absent (error) — … Mentes …
|
|
controller 02:32:38 Event pushed: whole_guest_backup_failed (error) — … felhom-backup tier …
|
|
controller 02:35:32 Event pushed: health_degraded (warning) …
|
|
controller 02:42:08 Event pushed: storage_disconnected (error) — … Adatok <-- generic path STILL WORKS
|
|
(no backup_target_restored, and no second backup_target_absent, at any point after 02:32:34)
|
|
|
|
So this is controller-side emission, and it is **specific to the `backup_target_*` pair**: the generic
|
|
`storage_disconnected` / `storage_reconnected` pair for the *other* drive kept firing normally
|
|
throughout the same window (02:16:02, 02:18:21, 02:27:38, 02:29:35, 02:42:08).
|
|
|
|
## The internally inconsistent state alongside it
|
|
|
|
Sampled while it was happening:
|
|
|
|
backup-target: {"degraded": false, "known": true, "label": "Mentes", "target": "felhom-backup"}
|
|
drives: adatok bound_under_parent=False backup_target=None
|
|
mentes bound_under_parent=False backup_target=True
|
|
|
|
`mentes` is the backup target, reads **not bound**, and the target state simultaneously reports
|
|
**`degraded: false`**. Those two cannot both be right. If that survives quiescence it is the more
|
|
serious half of this observation — a **false healthy** on the backup target, which is I5/I6's
|
|
failure mode and worse than a missing event.
|
|
|
|
## What is NOT yet established
|
|
|
|
- Whether the pair recovers once the cycling stops. The harness detaches and reattaches roughly every
|
|
two minutes; a real customer does not. Rapid cycling may be outside the design envelope, in which
|
|
case this is a robustness note rather than a defect.
|
|
- Whether the controller restart at 02:25:37 is implicated. In this controller instance there was
|
|
**1 `backup_target_absent` and 0 `backup_target_restored`**; the previous instance emitted complete
|
|
pairs (02:13:21 restored). That is suggestive, not established.
|
|
- Whether the `degraded: false` reading was a transient mid-flux sample or a stuck state.
|
|
|
|
## Disposition — how to settle it
|
|
|
|
After run 2b ends, let the box **quiesce with both drives attached**, then:
|
|
|
|
1. confirm `bound_under_parent` returns to `True` for both drives and the target still reads healthy —
|
|
if it does not, the false-healthy is real and this becomes a finding;
|
|
2. do **one** slow detach/reattach of `mentes` with several minutes either side, and check the pair
|
|
fires. If it does, the observation is "the pair does not survive rapid cycling"; if it does not,
|
|
the target alarm has silently stopped working, which is severe — an alarm that never fires again
|
|
after its first firing is worse than no alarm.
|
|
|
|
Only then does it earn an `R-n` or a retraction.
|