v0.184.0 — E-2b + Part 5: wire the drive-absent alarm that was never called
NotifyStorageDisconnected and NotifyStorageReconnected were defined and called from NOWHERE. Registered in allowedEventTypes, in DefaultEnabledEvents, and given a Hungarian message on the hub -- and never invoked. A drive going absent produced apps stopped, a WARN log and a UI badge, then silence on every channel. Verified against the gitignored-cmd/ trap with a positive control. Fifth instance of this class, found by E-2 Phase 0 rather than by a failure. A drive that is ONLY a backup target has no apps to stop, so it was silent twice. ReconcileDriveGates now calls both halves. When the absent drive is the whole-guest backup target it raises the more specific backup_target_absent (error) instead -- never both, since two mails for one event trains people to ignore the channel -- and recovers as backup_target_restored (info, the existing pairing-gated pattern; severityNotifies NOT widened). The recovery mirrors the alarm's choice or the operator cannot match them. Which drive is the target comes from the AGENT (/disks backup_target, >= 0.112.0), not from our StoragePath.BackupTarget: that is customer INTENT, and on the two boxes migrated by hand in E-1 the intent was never recorded while the drive really is the target. An older agent omits the field -> false -> generic alarm, never a wrong one. Before this an absent backup target had NO prompt signal: the tier stays DUE (targetStoragePresent checks name presence, never reachability), so the only evidence was its own failure at the next due cycle, up to ~24h away. The R-100 shape. Tests observe the WIRE, not a mock, because the failure class is "nothing arrives": a real Notifier posts to an httptest hub and the test asserts the event type and severity that actually went out. MinAgent: 0.112.0 Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
This commit is contained in:
@@ -286,6 +286,15 @@ func (s *Server) ReconcileDriveGates() {
|
||||
s.logger.Printf("[WARN] [gate] mark disconnected %s: %v", a.Path, err)
|
||||
}
|
||||
s.logger.Printf("[WARN] [gate] drive ABSENT %s — stopped+blocked %d app(s): %v", a.Path, len(stopped), stopped)
|
||||
// E-2b: THE SEAM THAT WAS NEVER WIRED. NotifyStorageDisconnected existed, was registered in
|
||||
// allowedEventTypes + DefaultEnabledEvents + the hub's Hungarian customerMessages — and was
|
||||
// called from nowhere, so a drive going absent produced a log line and silence on every
|
||||
// channel. A drive that is ONLY a backup target has no apps to stop, so it was silent twice
|
||||
// over. Fifth instance of this class in the project; found by E-2's Phase 0.
|
||||
//
|
||||
// E-2 Part 5: when the absent drive is the BACKUP TARGET, that is the more specific and more
|
||||
// urgent fact, so it gets its own event rather than being folded into the generic one.
|
||||
s.notifyDriveAbsent(a.Path, stopped, driveTargetByPath(resp.Disks))
|
||||
go s.SyncFileBrowserMounts()
|
||||
case a.Return:
|
||||
if a.Raw != "" {
|
||||
@@ -304,6 +313,9 @@ func (s *Server) ReconcileDriveGates() {
|
||||
s.logger.Printf("[WARN] [gate] clear disconnected %s: %v", a.Path, err)
|
||||
}
|
||||
s.logger.Printf("[INFO] [gate] drive RETURNED %s — re-attached + restarted gate-stopped apps", a.Path)
|
||||
// E-2b: the recovery half. An operator told a drive vanished must be told it came back —
|
||||
// otherwise the alarm is a dead end and the next one is trusted less.
|
||||
s.notifyDriveReturned(a.Path, driveTargetByPath(resp.Disks))
|
||||
go s.SyncFileBrowserMounts()
|
||||
}
|
||||
}
|
||||
@@ -582,3 +594,59 @@ func (s *Server) gateWhere(w http.ResponseWriter, r *http.Request) (string, bool
|
||||
}
|
||||
return where, true
|
||||
}
|
||||
|
||||
// driveTargetByPath maps host mount path → the agent's backup-target flag. The agent is the authority
|
||||
// (E-2): our own StoragePath.BackupTarget is customer intent, and on the two hand-migrated boxes that
|
||||
// intent was never recorded while the drive really is the target. An older agent omits the field, so
|
||||
// every entry is false and we degrade to the generic disconnect alarm — never a wrong one.
|
||||
func driveTargetByPath(disks []agentapi.DiskInfo) map[string]bool {
|
||||
out := make(map[string]bool, len(disks))
|
||||
for _, d := range disks {
|
||||
if d.MountPath != "" {
|
||||
out[d.MountPath] = d.BackupTarget
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// storageLabelFor returns the customer-facing label for a registered path, falling back to the path
|
||||
// itself. An alarm that names a device node the customer has never seen is not actionable.
|
||||
func (s *Server) storageLabelFor(path string) string {
|
||||
for _, sp := range s.settings.GetStoragePaths() {
|
||||
if sp.Path == path && strings.TrimSpace(sp.Label) != "" {
|
||||
return sp.Label
|
||||
}
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
// notifyDriveAbsent raises the right alarm for a drive that vanished: the backup-target-specific one
|
||||
// when it holds the whole-guest backup, the generic one otherwise. Never both — two emails for one
|
||||
// event trains people to ignore the channel.
|
||||
func (s *Server) notifyDriveAbsent(path string, stopped []string, isTarget map[string]bool) {
|
||||
if s.notifier == nil {
|
||||
return
|
||||
}
|
||||
label := s.storageLabelFor(path)
|
||||
if isTarget[path] {
|
||||
s.logger.Printf("[ERROR] [gate] the ABSENT drive %s is the WHOLE-GUEST BACKUP TARGET — the system backup cannot run until it returns", path)
|
||||
s.notifier.NotifyBackupTargetAbsent(label, path)
|
||||
return
|
||||
}
|
||||
s.notifier.NotifyStorageDisconnected(label, stopped)
|
||||
}
|
||||
|
||||
// notifyDriveReturned is the recovery counterpart, and it must mirror notifyDriveAbsent's choice or
|
||||
// the pairing breaks: a target that alarmed as backup_target_absent has to recover as
|
||||
// backup_target_restored, not as a generic reconnect the operator cannot match to the original.
|
||||
func (s *Server) notifyDriveReturned(path string, isTarget map[string]bool) {
|
||||
if s.notifier == nil {
|
||||
return
|
||||
}
|
||||
label := s.storageLabelFor(path)
|
||||
if isTarget[path] {
|
||||
s.notifier.NotifyBackupTargetRestored(label, path)
|
||||
return
|
||||
}
|
||||
s.notifier.NotifyStorageReconnected(label)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user