v0.222.0: ask whether a supervised member is DEAD before whether one is UNHEALTHY (R-384), and stop promising an undo copy nobody looked for (R-383)
gates / gates (push) Successful in 11s
gates / gates (push) Successful in 11s
R-384. aggregateState returned StateUnhealthy the moment unhealthy > 0, and the R-51 mixed-case block that asks "is a supervised member dead?" sat below it. A two-container app whose database exits goes unhealthy BECAUSE it cannot reach that database - so the symptom the dead database causes was what suppressed the alarm for it. unhealthy is not a down state, so classifyRunStates never marked the app down and app_start_failed never fired. Measured live on demo-hp 2026-08-22: bookstack-db stopped at 21:27:01 and the F-OBS heartbeat printed "0 currently down" throughout. R-51's 18-hour immich failure, back through a different door. Two things moved, and either alone leaves the defect standing: the supervised test is hoisted above the unhealthy/starting/restarting returns, and "some members are up" now counts ANY member not in the down bucket. The old guard was running > 0, which made the R-51 block unreachable in exactly the case it was written for. IsDownState is byte-identical - unhealthy stays excluded, because an unhealthy container is running and folding it in reintroduces the flapping that exclusion exists to stop. No new state was minted. Only the ORDER changed. The priority comment was rewritten because it asserted an ordering the code no longer has. Three subtests in TestAggregateState_UnchangedBranches were AMENDED: they asserted an unhealthy/starting/restarting member beat an exited peer on unless-stopped, which pinned the defect as settled behaviour. They keep their intent with the down member given a benign policy. R-383. The double-failure message said the previous state's backup EXISTS, built from the returned path without asking the filesystem - and a missing file is one of the two ways that rollback fails. undoCopyPhrase now describes the copy from disk: present, partial, missing (still naming where it should be), or never written. Zero-length counts as missing. Test count 1494 -> 1504. Four red-proofs planted, four seen failing; the two halves of R-384 convict independently.
This commit is contained in:
@@ -150,6 +150,54 @@ func (s safetyDumpSet) First() string {
|
||||
return s.Files[0].Path
|
||||
}
|
||||
|
||||
// undoCopyPhrase is the sentence the DOUBLE-FAILURE message uses to describe the customer's undo
|
||||
// copy — and it says what is TRUE, which is the whole of R-383.
|
||||
//
|
||||
// THE BUG THIS EXISTS TO KILL. The double-failure branch ended with „a korábbi állapot mentése
|
||||
// megvan: <file>" — *the previous state's backup EXISTS* — built from the path `writeSafetyDump`
|
||||
// returned and WITHOUT ever asking the filesystem. But one of the two ways `rollbackSafetyDump` fails
|
||||
// is that the file is not there, so in exactly the case that sentence is printed it is most likely to
|
||||
// be false. Measured twice live, on v0.220.2 and v0.221.1.
|
||||
//
|
||||
// A false reassurance is worse than no sentence: it is read at the moment the customer is deciding
|
||||
// whether their data is recoverable, and it points support at a file that is not there.
|
||||
//
|
||||
// WHY NOT SIMPLY DROP THE FILENAME. R-351's lesson: a refusal that names nothing forces a person to
|
||||
// remember what the product already knows. The operator needs the path either way — to fetch the
|
||||
// file, or to look for it. So the absent case still names WHERE it should have been, and says
|
||||
// plainly that it is not there.
|
||||
//
|
||||
// The check is `os.Stat`, deliberately not a readability or integrity test: this runs at the end of a
|
||||
// failed restore on a machine that may be unwell, and the honest claim available here is presence.
|
||||
// A zero-length file is reported as MISSING — a 0-byte dump restores nothing, and calling it present
|
||||
// is the same false reassurance one step smaller.
|
||||
func undoCopyPhrase(set safetyDumpSet) string {
|
||||
var present, absent []string
|
||||
for _, f := range set.Files {
|
||||
if f.Path == "" {
|
||||
continue
|
||||
}
|
||||
if st, err := os.Stat(f.Path); err == nil && !st.IsDir() && st.Size() > 0 {
|
||||
present = append(present, filepath.Base(f.Path))
|
||||
continue
|
||||
}
|
||||
absent = append(absent, filepath.Base(f.Path))
|
||||
}
|
||||
switch {
|
||||
case len(present) > 0 && len(absent) == 0:
|
||||
return "a korábbi állapot mentése megvan: " + strings.Join(present, ", ")
|
||||
case len(present) > 0:
|
||||
// Partial: name both halves. An app with two databases whose undo is half there is a
|
||||
// different situation from either whole one, and support must not have to guess which.
|
||||
return "a korábbi állapot mentése RÉSZBEN van meg — megvan: " + strings.Join(present, ", ") +
|
||||
"; HIÁNYZIK: " + strings.Join(absent, ", ")
|
||||
case len(absent) > 0:
|
||||
return "a korábbi állapot mentését NEM találjuk a helyén (" + strings.Join(absent, ", ") + ")"
|
||||
default:
|
||||
return "a korábbi állapotról nem készült menthető másolat"
|
||||
}
|
||||
}
|
||||
|
||||
// writeSafetyDump dumps every live database of stack into the app's unit db-dumps dir under the
|
||||
// `pre-restore-` prefix, and returns the SET it wrote. Returns (zero, nil) when the app has no
|
||||
// database at all — a no-DB app has nothing to undo and must flow exactly as it did before
|
||||
@@ -704,9 +752,12 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack
|
||||
// operator ruling, 2026-08-22.
|
||||
m.logger.Printf("[ERROR] [offbox] %s: ROLLBACK ALSO FAILED (%v) — holding the app stopped; replay error was: %v", stack, rbErr, iErr)
|
||||
m.holdAppAfterFailedRollback(stack, iErr, rbErr)
|
||||
// R-383: the undo copy is DESCRIBED FROM DISK, never from the path alone. See
|
||||
// undoCopyPhrase — this sentence used to assert the file existed in exactly the
|
||||
// branch where a missing file is one of the two causes.
|
||||
return res, fmt.Errorf("a(z) %s adatbázisának visszaállítása sikertelen, és a korábbi állapot visszatöltése sem sikerült. "+
|
||||
"Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk, hogy az adatai ne sérüljenek tovább. "+
|
||||
"Vedd fel velünk a kapcsolatot — a korábbi állapot mentése megvan: %s", stack, filepath.Base(safety))
|
||||
"Vedd fel velünk a kapcsolatot — %s", stack, undoCopyPhrase(safetySet))
|
||||
}
|
||||
res.RolledBack = true
|
||||
if sErr := restartStack(); sErr != nil {
|
||||
|
||||
Reference in New Issue
Block a user