feat(D53): the hub learns which apps are held — app_held / app_hold_cleared, daily repeat
gates / gates (push) Successful in 54s
gates / gates (push) Successful in 54s
internal/holdreport: one observer of the hold store, ticked every minute; app_held for a new or changed hold (stack_name, reason, since, why), app_hold_cleared when gone (also while the controller was down: settings.reported_holds), every hold again daily and at start. Severity info. RestoreHold.Why/WireReason (holdWhy calls it). 8 tests, red-proved (inert Tick; unwired scheduler line). Commit only (felhom.eu 08 §10).
This commit is contained in:
@@ -1,3 +1,20 @@
|
||||
## Unreleased (next: v0.346.0) — D53: the hub learns which apps are held
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). Operator ruling D53 (2026-10-26, felhom.eu `09` §3 decision 265): operator-only
|
||||
reminders first. Needs hub with `app_held` / `app_hold_cleared` in `allowedEventTypes` (felhom.eu, same day) — an older
|
||||
hub answers 400 to both and nothing else changes.
|
||||
|
||||
- `internal/holdreport`: ONE observer of the hold store (`settings.RestoreHolds`), ticked by the scheduler every minute
|
||||
(`hold-report`). `app_held` for a new or changed hold (details `stack_name`, `reason` — the empty legacy reason is sent
|
||||
as `restore_failed` — `since`, `why`), `app_hold_cleared` when a hold is gone; every hold is sent again once a day and
|
||||
at each start (`repeat: true`). The last-reported set is kept in `settings.reported_holds`, so a hold cleared while the
|
||||
controller was down is still reported cleared. Both severity info (the hub mails nothing on receipt).
|
||||
- `settings.RestoreHold.Why()` / `WireReason()`: the hold sentence moved out of `cmd/controller` (`holdWhy` calls it), so
|
||||
`--restore-holds` and the hub's list say the same thing.
|
||||
- Tests `internal/holdreport/holdreport_test.go` (8): the report field, the daily repeat, cleared once, cleared while
|
||||
down, a changed reason, the wire reason, the wiring in main, the record across a reload. Red-proved with an inert
|
||||
`Tick` (all six behaviour tests failed) and with the scheduler line commented out (the wiring test failed).
|
||||
|
||||
## v0.345.0 (2026-10-26) — the 50 % rule owns qBittorrent's NORMAL limit only (R-1292) — attended release run (afternoon)
|
||||
|
||||
Live before the release: Tester 1 on `0.345.0-rc.1` with a throwaway qBittorrent, alternative speed ON (the household's
|
||||
|
||||
@@ -84,6 +84,7 @@
|
||||
| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `<ns>/backups/restore-undo/<app>/` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` |
|
||||
| `CrashPutBack` + `Manager.persistUndoForCrash` + `AppStopGuard.SetUndo` (D22 / R-1195, v0.338.0) | controller/internal/backup/restore_crash_putback.go | `NewCrashPutBack(stop, writeDef, logger)`; `PutBackInterruptedRestore(stack, undoDir) error` | A restore cut by a power loss is put back at start: the copy (volumes + `definition/` + `undo.json`) is recorded in the app-stop marker once whole; `Recover` pours it back BEFORE the restart | Call `persistUndoForCrash` right after `copyLiveVolumes` in any new restore path. `Recover` runs before the backup manager exists, so the put-back is wired in main.go with `stackMgr.StopStack` and the stack adapter's definition writer. A failed put-back = a FAILED recovery (app not started, marker kept). The household reads `note.restore.interrupted_put_back` (`LoadRestoreRecordAfterRecovery`). Pinned by `TestR1195_*` |
|
||||
| `tier2CaptureSetFull(...) (legs, missing, warns, classified)` + `keptCopyRels` + `dropKeptCopiesUnder` (R-1252, unreleased) | controller/internal/backup/tier2_capture.go, tier2_kept.go | `classified` = the legs come from a VALID backup block | **The second-drive reconcile deletes only under a valid classification**; a rejected or absent block keeps every dest dir; kept copies (D25) are always in the keep set; a mirror deleted by Remove drops its kept records | A new reconcile caller must take `classified` and the kept rels too — never reconcile against the legacy legs. A rejected block is told by `Manager.BackupBlockRejected` (stacks; via the adapter's optional `BackupBlockRejected` method) and signalled once a day (`SetTier2SettingsUnreadableNotify`). Pinned by `r1252_rejected_block_keeps_copies_test.go` |
|
||||
| `holdreport.Reporter` + `settings.RestoreHold.Why` / `WireReason` (D53, unreleased) | controller/internal/holdreport/holdreport.go · controller/internal/settings/settings.go | `New(store, notifier.PushEvent, logger)`; `Tick(now)` from the scheduler | Tells the hub every hold in place (`app_held`), every hold gone (`app_hold_cleared`), each again daily | **One observer of the hold store — never a send inside a hold writer.** A new hold writer needs nothing here. The hold sentence is `RestoreHold.Why()` (one per reason) — never a second copy. Pinned by `internal/holdreport/holdreport_test.go` |
|
||||
| `AppStopGuard.ResumePutBack` / `ResumeWaitingPutBacks` + `HoldReasonPutBackPending` (R-1251, unreleased) | controller/internal/backup/appstop_putback_wait.go | `ResumePutBack(stack) bool` (true = nothing owed, the caller may start); `ResumeWaitingPutBacks() []string` (puts back, then starts through the gated starter) | A cut restore's copy on a drive that is not bound is WAITING: Recover writes the restore hold `putback_pending` (copy in `SafetyDump`) and starts nothing; the put-back runs when the drive binds, before any start | **Any new path that starts an app when its drive arrives calls `ResumePutBack` first** (the boot recreate and `restartStacks` do, in web via `Server.resumePutBack`). Every other start path is covered by the hold store it already honours — do not add a second check. A failed put-back (at start or on resume) writes the D20 hold (`restore_mixed`) and drops the app from the marker. Needs `SetHolds` + `SetUndoReachable` before `Recover` (main.go). Pinned by `r1251_putback_waits_test.go`, `TestR1251_DriveStartPathsPutBackBeforeTheyStart`, `TestR1251_MainWiresThePutBackWait` |
|
||||
| `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned |
|
||||
| `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces |
|
||||
|
||||
@@ -2283,6 +2283,10 @@ that folder is never a dead end, and an install never runs into it silently (R-6
|
||||
- **R-1286 (v0.344.0).** A held app says WHY: each hold reason (a failed restore, an unhealthy stop by the crash-loop
|
||||
guard, …) has its own sentence in the start gate, the boot sweep and `--restore-holds` (an unhealthy stop used to
|
||||
read as a failed restore).
|
||||
- **D53 (next release, felhom.eu `08` §10).** The hub learns which apps are held: `internal/holdreport` reads the hold
|
||||
store every minute and sends `app_held` (stack, reason, since, the hold's sentence) for a new or changed hold and
|
||||
`app_hold_cleared` when one is gone — also one cleared while the controller was down (`settings.reported_holds`) — and
|
||||
repeats every hold daily and at start. Both are severity info; the hub's operator reminder after a grace is the mail.
|
||||
- **R-690 (fixed here).** The removed-app restore (R-487) never found a unit on a DATA drive — it asked
|
||||
`GetStackComposePath`, true for every catalog app — and restored with no env. It now asks `isStackDeployed`.
|
||||
- `.felhom.yml` **`after_load: {service, user, command: [...]}`** — one command run with `docker compose exec -T`
|
||||
|
||||
@@ -40,6 +40,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/crashboot"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/crypto"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/holdreport"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/infra"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/integrations"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/mailrelay"
|
||||
@@ -1063,6 +1064,14 @@ func main() {
|
||||
go func() { defer aubBusy.Unlock(); aubRunner.Tick(time.Now()) }() // a run can take minutes; never block the scheduler
|
||||
return nil
|
||||
})
|
||||
// D53 (felhom.eu `08` §10): tell the hub which apps are held — app_held for a new or changed hold, app_hold_cleared
|
||||
// when one is gone, every hold again once a day and after each start. One observer of the hold store, so every
|
||||
// writer is covered. Pinned by internal/holdreport.
|
||||
holdReporter := holdreport.New(sett, notifier.PushEvent, logger)
|
||||
sched.Every("hold-report", time.Minute, func(ctx context.Context) error {
|
||||
holdReporter.Tick(time.Now())
|
||||
return nil
|
||||
})
|
||||
sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error {
|
||||
if bootGrace.Within(time.Now()) {
|
||||
return nil // still inside the startup settle window (90 s, or ~15 min after a crash boot)
|
||||
@@ -2562,22 +2571,7 @@ type driveStartGate struct {
|
||||
// stop read as „held after a failed restore whose rollback also failed" for 11 days on demo-hp, so the operator could
|
||||
// not tell a crash loop from a broken restore). Pinned by TestR1286_EveryHoldReasonSaysWhatItIs.
|
||||
func holdWhy(h settings.RestoreHold) string {
|
||||
switch h.Reason {
|
||||
case settings.HoldReasonUpdateFailed:
|
||||
return "held after a failed update (" + h.At + ") — restore it from its backup to start it"
|
||||
case settings.HoldReasonUnhealthyStop:
|
||||
kind := h.UnhealthyKind
|
||||
if kind == "" {
|
||||
kind = "unhealthy"
|
||||
}
|
||||
return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try"
|
||||
case settings.HoldReasonRestoreMixed:
|
||||
return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it"
|
||||
case settings.HoldReasonPutBackPending:
|
||||
return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it"
|
||||
default:
|
||||
return "held after a failed restore whose rollback also failed (" + h.At + ") — clear the hold to start it"
|
||||
}
|
||||
return h.Why() // one sentence per reason, shared with the hub's „Held apps" list (D53)
|
||||
}
|
||||
|
||||
func (g driveStartGate) MayStart(stackName string) (bool, string) {
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
// Package holdreport tells the hub which apps are held (D53, felhom.eu `08` §10).
|
||||
//
|
||||
// A held app is stopped on purpose and stays stopped until a person acts. Until D53 the hub knew nothing about it:
|
||||
// the crash-loop stop sent one event, the other holds sent none, and demo-hp's Kimai sat held for 11 days unseen
|
||||
// (R-1291). The legacy report is frozen, so the event channel carries the holds:
|
||||
//
|
||||
// - app_held when a hold appears or changes (stack, reason, since, the hold's sentence);
|
||||
// - app_hold_cleared when it is gone — also when it went while the controller was down, because the set last
|
||||
// reported is kept in settings (ReportedHolds), not in memory;
|
||||
// - every hold still in place is sent again once a day and at every controller start, because an event can be
|
||||
// lost. The hub treats a repeat as idempotent.
|
||||
//
|
||||
// ONE OBSERVER, NOT A HOOK PER WRITER. Seven code paths write a hold and four clear one. A send in each of them is
|
||||
// the shape where the one path nobody found is the next silent hold; this package reads the hold store itself, so a
|
||||
// new writer is covered the day it is added. Both events are severity info: the hub mails nothing on receipt. Its
|
||||
// reminder after the grace per reason is the operator's mail, and it never goes to the household (D53).
|
||||
//
|
||||
// Pinned by holdreport_test.go.
|
||||
package holdreport
|
||||
|
||||
import (
|
||||
"log"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// Event types — the hub's allowedEventTypes carries both (the coupling rule).
|
||||
const (
|
||||
EventHeld = "app_held"
|
||||
EventCleared = "app_hold_cleared"
|
||||
)
|
||||
|
||||
// ResendEvery is how often every hold still in place is sent again.
|
||||
const ResendEvery = 24 * time.Hour
|
||||
|
||||
// Store is what the reporter needs of the settings.
|
||||
type Store interface {
|
||||
ListRestoreHolds() []settings.RestoreHold
|
||||
GetReportedHolds() map[string]string
|
||||
SetReportedHolds(map[string]string) error
|
||||
}
|
||||
|
||||
// Push sends one event to the hub (notify.Notifier.PushEvent).
|
||||
type Push func(eventType, severity, message string, details interface{})
|
||||
|
||||
// Details is the payload of both events.
|
||||
type Details struct {
|
||||
StackName string `json:"stack_name"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Since string `json:"since,omitempty"`
|
||||
Why string `json:"why,omitempty"`
|
||||
Repeat bool `json:"repeat,omitempty"` // a daily or start-up re-send, not a new hold
|
||||
}
|
||||
|
||||
// Reporter compares the hold store with what was last reported. Tick is called by the scheduler.
|
||||
type Reporter struct {
|
||||
store Store
|
||||
push Push
|
||||
logger *log.Logger
|
||||
lastFull time.Time
|
||||
}
|
||||
|
||||
// New builds a reporter. The first Tick re-sends every hold (a controller start).
|
||||
func New(store Store, push Push, logger *log.Logger) *Reporter {
|
||||
return &Reporter{store: store, push: push, logger: logger}
|
||||
}
|
||||
|
||||
func key(h settings.RestoreHold) string { return h.WireReason() + "|" + h.At }
|
||||
|
||||
// Tick sends what changed since the last tick, and every hold again when a day has passed.
|
||||
func (r *Reporter) Tick(now time.Time) {
|
||||
if r.store == nil || r.push == nil {
|
||||
return
|
||||
}
|
||||
holds := r.store.ListRestoreHolds()
|
||||
sort.Slice(holds, func(i, j int) bool { return holds[i].Stack < holds[j].Stack })
|
||||
prev := r.store.GetReportedHolds()
|
||||
full := r.lastFull.IsZero() || now.Sub(r.lastFull) >= ResendEvery
|
||||
|
||||
cur := make(map[string]string, len(holds))
|
||||
sent, cleared := 0, 0
|
||||
for _, h := range holds {
|
||||
k := key(h)
|
||||
cur[h.Stack] = k
|
||||
known, seen := prev[h.Stack]
|
||||
if seen && known == k && !full {
|
||||
continue
|
||||
}
|
||||
repeat := seen && known == k
|
||||
r.push(EventHeld, "info", "App "+h.Stack+" is held: "+h.Why(),
|
||||
Details{StackName: h.Stack, Reason: h.WireReason(), Since: h.At, Why: h.Why(), Repeat: repeat})
|
||||
sent++
|
||||
if !repeat {
|
||||
r.logf("[INFO] [holdreport] %s held (%s since %s) — reported to the hub", h.Stack, h.WireReason(), h.At)
|
||||
}
|
||||
}
|
||||
gone := make([]string, 0)
|
||||
for stack := range prev {
|
||||
if _, ok := cur[stack]; !ok {
|
||||
gone = append(gone, stack)
|
||||
}
|
||||
}
|
||||
sort.Strings(gone)
|
||||
for _, stack := range gone {
|
||||
r.push(EventCleared, "info", "App "+stack+" is no longer held", Details{StackName: stack})
|
||||
cleared++
|
||||
r.logf("[INFO] [holdreport] %s hold cleared — reported to the hub", stack)
|
||||
}
|
||||
if full {
|
||||
r.lastFull = now
|
||||
r.logf("[DEBUG] [holdreport] full re-send: %d hold(s) sent, %d cleared", sent, cleared)
|
||||
}
|
||||
if !sameMap(prev, cur) {
|
||||
if err := r.store.SetReportedHolds(cur); err != nil {
|
||||
// Not fatal: the next tick sends the same difference again, which the hub reads as idempotent.
|
||||
r.logf("[WARN] [holdreport] could not save the reported holds: %v — the next tick repeats the difference", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func sameMap(a, b map[string]string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for k, v := range a {
|
||||
if b[k] != v {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (r *Reporter) logf(format string, args ...interface{}) {
|
||||
if r.logger != nil {
|
||||
r.logger.Printf(format, args...)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
package holdreport
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
type fakeStore struct {
|
||||
holds map[string]settings.RestoreHold
|
||||
reported map[string]string
|
||||
}
|
||||
|
||||
func (f *fakeStore) ListRestoreHolds() []settings.RestoreHold {
|
||||
out := []settings.RestoreHold{}
|
||||
for _, h := range f.holds {
|
||||
out = append(out, h)
|
||||
}
|
||||
return out
|
||||
}
|
||||
func (f *fakeStore) GetReportedHolds() map[string]string {
|
||||
out := map[string]string{}
|
||||
for k, v := range f.reported {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
func (f *fakeStore) SetReportedHolds(m map[string]string) error { f.reported = m; return nil }
|
||||
|
||||
type sent struct {
|
||||
typ, sev string
|
||||
d Details
|
||||
}
|
||||
|
||||
func rec(out *[]sent) Push {
|
||||
return func(t, sev, msg string, d interface{}) { *out = append(*out, sent{t, sev, d.(Details)}) }
|
||||
}
|
||||
|
||||
var t0 = time.Date(2026, 10, 26, 18, 0, 0, 0, time.UTC)
|
||||
|
||||
// The report field (D53): a new hold is sent once as app_held with its stack, reason and since, severity info (the hub
|
||||
// mails nothing on receipt; its reminder after the grace is the mail).
|
||||
func TestD53_ANewHoldIsReportedWithStackReasonAndSince(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{
|
||||
"kimai": {Stack: "kimai", At: "2026-10-14T08:00:00Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"},
|
||||
}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
if len(out) != 1 || out[0].typ != EventHeld || out[0].sev != "info" {
|
||||
t.Fatalf("want one info app_held, got %+v", out)
|
||||
}
|
||||
d := out[0].d
|
||||
if d.StackName != "kimai" || d.Reason != "unhealthy_stop" || d.Since != "2026-10-14T08:00:00Z" || d.Why == "" {
|
||||
t.Fatalf("payload: %+v", d)
|
||||
}
|
||||
if st.reported["kimai"] != "unhealthy_stop|2026-10-14T08:00:00Z" {
|
||||
t.Fatalf("reported record: %v", st.reported)
|
||||
}
|
||||
// The next tick inside a day sends nothing.
|
||||
out = nil
|
||||
r.Tick(t0.Add(time.Minute))
|
||||
if len(out) != 0 {
|
||||
t.Fatalf("an unchanged hold was sent again inside a day: %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// An event can be lost: every hold still in place is sent again after a day, marked as a repeat.
|
||||
func TestD53_AHoldIsSentAgainEveryDay(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonUpdateFailed}}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
out = nil
|
||||
r.Tick(t0.Add(ResendEvery))
|
||||
if len(out) != 1 || out[0].typ != EventHeld || !out[0].d.Repeat {
|
||||
t.Fatalf("want one repeat app_held after a day, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A hold cleared by a person ends it: one app_hold_cleared, then nothing.
|
||||
func TestD53_AClearedHoldSendsClearedOnce(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonRestoreMixed}}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
delete(st.holds, "a")
|
||||
out = nil
|
||||
r.Tick(t0.Add(time.Minute))
|
||||
if len(out) != 1 || out[0].typ != EventCleared || out[0].d.StackName != "a" || out[0].sev != "info" {
|
||||
t.Fatalf("want one info app_hold_cleared for a, got %+v", out)
|
||||
}
|
||||
if len(st.reported) != 0 {
|
||||
t.Fatalf("the reported record still names the cleared hold: %v", st.reported)
|
||||
}
|
||||
out = nil
|
||||
r.Tick(t0.Add(2 * time.Minute))
|
||||
r.Tick(t0.Add(ResendEvery + time.Hour))
|
||||
if len(out) != 0 {
|
||||
t.Fatalf("a cleared hold was reported again: %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A hold cleared while the controller was down is still reported cleared: the record lives in settings.
|
||||
func TestD53_AHoldClearedWhileDownIsReportedAtStart(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{}, reported: map[string]string{"gone": "update_failed|2026-10-01T00:00:00Z"}}
|
||||
var out []sent
|
||||
New(st, rec(&out), nil).Tick(t0)
|
||||
if len(out) != 1 || out[0].typ != EventCleared || out[0].d.StackName != "gone" {
|
||||
t.Fatalf("want app_hold_cleared for gone at start, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A hold whose reason changes (a put-back that failed becomes a mixed restore) is a new hold for the hub.
|
||||
func TestD53_AChangedReasonIsReportedAgain(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonPutBackPending}}}
|
||||
var out []sent
|
||||
r := New(st, rec(&out), nil)
|
||||
r.Tick(t0)
|
||||
st.holds["a"] = settings.RestoreHold{Stack: "a", At: "2026-10-20T01:00:00Z", Reason: settings.HoldReasonRestoreMixed}
|
||||
out = nil
|
||||
r.Tick(t0.Add(time.Minute))
|
||||
if len(out) != 1 || out[0].typ != EventHeld || out[0].d.Reason != "restore_mixed" || out[0].d.Repeat {
|
||||
t.Fatalf("want a new (not repeat) app_held restore_mixed, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// The legacy empty reason is a restore that failed with its rollback: the hub reads it as restore_failed.
|
||||
func TestD53_TheEmptyReasonIsRestoreFailedOnTheWire(t *testing.T) {
|
||||
st := &fakeStore{holds: map[string]settings.RestoreHold{"db": {Stack: "db", At: "2026-10-20T00:00:00Z"}}}
|
||||
var out []sent
|
||||
New(st, rec(&out), nil).Tick(t0)
|
||||
if len(out) != 1 || out[0].d.Reason != "restore_failed" {
|
||||
t.Fatalf("want reason restore_failed, got %+v", out)
|
||||
}
|
||||
}
|
||||
|
||||
// The seam is wired: main builds the reporter on the real settings and notifier and ticks it from the scheduler. A
|
||||
// reporter nobody ticks is the inert-seam shape (four instances in this project). A commented-out line does not count.
|
||||
func TestD53_TheReporterIsWiredInMain(t *testing.T) {
|
||||
raw, err := os.ReadFile(filepath.Join("..", "..", "cmd", "controller", "main.go"))
|
||||
if err != nil {
|
||||
t.Fatalf("cannot read main.go: %v", err)
|
||||
}
|
||||
live := []string{}
|
||||
for _, l := range strings.Split(string(raw), "\n") {
|
||||
if !strings.HasPrefix(strings.TrimSpace(l), "//") {
|
||||
live = append(live, l)
|
||||
}
|
||||
}
|
||||
src := strings.Join(live, "\n")
|
||||
for _, want := range []string{"holdreport.New(sett, notifier.PushEvent, logger)", `sched.Every("hold-report"`, "holdReporter.Tick("} {
|
||||
if !strings.Contains(src, want) {
|
||||
t.Errorf("main.go lacks %q — the hold report is not wired", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The reported record survives a restart: it is what lets a hold cleared while the controller was down be reported.
|
||||
func TestD53_TheReportedRecordSurvivesAReload(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "settings.json")
|
||||
s, err := settings.Load(p, log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.SetReportedHolds(map[string]string{"kimai": "unhealthy_stop|2026-10-14T08:00:00Z"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s2, err := settings.Load(p, log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := s2.GetReportedHolds()["kimai"]; got != "unhealthy_stop|2026-10-14T08:00:00Z" {
|
||||
t.Fatalf("after a reload the record reads %q", got)
|
||||
}
|
||||
}
|
||||
@@ -232,6 +232,10 @@ type Settings struct {
|
||||
// out-of-memory storm (RFC3339). Survives the hold being lifted by Start, so a second stop within
|
||||
// 24 h is known to be a second one.
|
||||
UnhealthyStops map[string]string `json:"unhealthy_stops,omitempty"`
|
||||
// ReportedHolds (D53, `08` §10) — the holds last told to the hub as app_held: stack → „reason|at". The hold
|
||||
// reporter (internal/holdreport) compares it with RestoreHolds to send app_held for a new or changed hold and
|
||||
// app_hold_cleared for one that is gone — also one cleared while the controller was down, because it is kept here.
|
||||
ReportedHolds map[string]string `json:"reported_holds,omitempty"`
|
||||
|
||||
// Cross-drive restic repo password (auto-generated on first use)
|
||||
CrossDriveResticPassword string `json:"cross_drive_restic_password,omitempty"`
|
||||
@@ -1998,6 +2002,35 @@ const (
|
||||
HoldReasonPutBackPending = "putback_pending"
|
||||
)
|
||||
|
||||
// WireReason is the hold's reason as the hub reads it (D53): the empty legacy value is named restore_failed.
|
||||
func (h RestoreHold) WireReason() string {
|
||||
if h.Reason == HoldReasonRestoreFailed {
|
||||
return "restore_failed"
|
||||
}
|
||||
return h.Reason
|
||||
}
|
||||
|
||||
// Why says which operation put a hold in place and what lifts it — one sentence per reason (R-1286). English,
|
||||
// operator-facing: the controller's own -list-restore-holds output and the hub's „Held apps" list (D53) both use it.
|
||||
func (h RestoreHold) Why() string {
|
||||
switch h.Reason {
|
||||
case HoldReasonUpdateFailed:
|
||||
return "held after a failed update (" + h.At + ") — restore it from its backup to start it"
|
||||
case HoldReasonUnhealthyStop:
|
||||
kind := h.UnhealthyKind
|
||||
if kind == "" {
|
||||
kind = "unhealthy"
|
||||
}
|
||||
return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try"
|
||||
case HoldReasonRestoreMixed:
|
||||
return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it"
|
||||
case HoldReasonPutBackPending:
|
||||
return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it"
|
||||
default:
|
||||
return "held after a failed restore whose rollback also failed (" + h.At + ") — clear the hold to start it"
|
||||
}
|
||||
}
|
||||
|
||||
// SetRestoreHold records a hold. Modelled on SetDisconnected: a condition, plus what it is holding.
|
||||
func (s *Settings) SetRestoreHold(h RestoreHold) error {
|
||||
s.mu.Lock()
|
||||
@@ -2031,6 +2064,32 @@ func (s *Settings) ListRestoreHolds() []RestoreHold {
|
||||
return out
|
||||
}
|
||||
|
||||
// GetReportedHolds returns a copy of the holds last reported to the hub (D53): stack → „reason|at".
|
||||
func (s *Settings) GetReportedHolds() map[string]string {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
out := make(map[string]string, len(s.ReportedHolds))
|
||||
for k, v := range s.ReportedHolds {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SetReportedHolds replaces the record of the holds reported to the hub (D53). An empty map removes the field.
|
||||
func (s *Settings) SetReportedHolds(m map[string]string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if len(m) == 0 {
|
||||
s.ReportedHolds = nil
|
||||
} else {
|
||||
s.ReportedHolds = make(map[string]string, len(m))
|
||||
for k, v := range m {
|
||||
s.ReportedHolds[k] = v
|
||||
}
|
||||
}
|
||||
return s.save()
|
||||
}
|
||||
|
||||
// ClearRestoreHold removes a hold. Returns false when there was none, so a caller can tell "cleared"
|
||||
// from "there was nothing to clear" rather than reporting success either way.
|
||||
func (s *Settings) ClearRestoreHold(stack string) (bool, error) {
|
||||
|
||||
Reference in New Issue
Block a user