diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a0d3b69..10ba3432 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,20 @@ +## Unreleased (next: v0.346.0) — D53: the hub learns which apps are held + +**MinAgent: 0.158.1** (unchanged). Operator ruling D53 (2026-10-26, felhom.eu `09` §3 decision 265): operator-only +reminders first. Needs hub with `app_held` / `app_hold_cleared` in `allowedEventTypes` (felhom.eu, same day) — an older +hub answers 400 to both and nothing else changes. + +- `internal/holdreport`: ONE observer of the hold store (`settings.RestoreHolds`), ticked by the scheduler every minute + (`hold-report`). `app_held` for a new or changed hold (details `stack_name`, `reason` — the empty legacy reason is sent + as `restore_failed` — `since`, `why`), `app_hold_cleared` when a hold is gone; every hold is sent again once a day and + at each start (`repeat: true`). The last-reported set is kept in `settings.reported_holds`, so a hold cleared while the + controller was down is still reported cleared. Both severity info (the hub mails nothing on receipt). +- `settings.RestoreHold.Why()` / `WireReason()`: the hold sentence moved out of `cmd/controller` (`holdWhy` calls it), so + `--restore-holds` and the hub's list say the same thing. +- Tests `internal/holdreport/holdreport_test.go` (8): the report field, the daily repeat, cleared once, cleared while + down, a changed reason, the wire reason, the wiring in main, the record across a reload. Red-proved with an inert + `Tick` (all six behaviour tests failed) and with the scheduler line commented out (the wiring test failed). + ## v0.345.0 (2026-10-26) — the 50 % rule owns qBittorrent's NORMAL limit only (R-1292) — attended release run (afternoon) Live before the release: Tester 1 on `0.345.0-rc.1` with a throwaway qBittorrent, alternative speed ON (the household's diff --git a/REUSE.md b/REUSE.md index 1964dd58..44dce7dc 100644 --- a/REUSE.md +++ b/REUSE.md @@ -84,6 +84,7 @@ | `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE restore undo — off-site (v0.336.0) and, since D22 (unreleased), the LOCAL unit restore (`RestoreFromRecoveryUnitAtWith`, so also the second drive's and a kept load): the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `/backups/restore-undo//` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` and `r1177_local_putback_test.go` | | `CrashPutBack` + `Manager.persistUndoForCrash` + `AppStopGuard.SetUndo` (D22 / R-1195, v0.338.0) | controller/internal/backup/restore_crash_putback.go | `NewCrashPutBack(stop, writeDef, logger)`; `PutBackInterruptedRestore(stack, undoDir) error` | A restore cut by a power loss is put back at start: the copy (volumes + `definition/` + `undo.json`) is recorded in the app-stop marker once whole; `Recover` pours it back BEFORE the restart | Call `persistUndoForCrash` right after `copyLiveVolumes` in any new restore path. `Recover` runs before the backup manager exists, so the put-back is wired in main.go with `stackMgr.StopStack` and the stack adapter's definition writer. A failed put-back = a FAILED recovery (app not started, marker kept). The household reads `note.restore.interrupted_put_back` (`LoadRestoreRecordAfterRecovery`). Pinned by `TestR1195_*` | | `tier2CaptureSetFull(...) (legs, missing, warns, classified)` + `keptCopyRels` + `dropKeptCopiesUnder` (R-1252, unreleased) | controller/internal/backup/tier2_capture.go, tier2_kept.go | `classified` = the legs come from a VALID backup block | **The second-drive reconcile deletes only under a valid classification**; a rejected or absent block keeps every dest dir; kept copies (D25) are always in the keep set; a mirror deleted by Remove drops its kept records | A new reconcile caller must take `classified` and the kept rels too — never reconcile against the legacy legs. A rejected block is told by `Manager.BackupBlockRejected` (stacks; via the adapter's optional `BackupBlockRejected` method) and signalled once a day (`SetTier2SettingsUnreadableNotify`). Pinned by `r1252_rejected_block_keeps_copies_test.go` | +| `holdreport.Reporter` + `settings.RestoreHold.Why` / `WireReason` (D53, unreleased) | controller/internal/holdreport/holdreport.go · controller/internal/settings/settings.go | `New(store, notifier.PushEvent, logger)`; `Tick(now)` from the scheduler | Tells the hub every hold in place (`app_held`), every hold gone (`app_hold_cleared`), each again daily | **One observer of the hold store — never a send inside a hold writer.** A new hold writer needs nothing here. The hold sentence is `RestoreHold.Why()` (one per reason) — never a second copy. Pinned by `internal/holdreport/holdreport_test.go` | | `AppStopGuard.ResumePutBack` / `ResumeWaitingPutBacks` + `HoldReasonPutBackPending` (R-1251, unreleased) | controller/internal/backup/appstop_putback_wait.go | `ResumePutBack(stack) bool` (true = nothing owed, the caller may start); `ResumeWaitingPutBacks() []string` (puts back, then starts through the gated starter) | A cut restore's copy on a drive that is not bound is WAITING: Recover writes the restore hold `putback_pending` (copy in `SafetyDump`) and starts nothing; the put-back runs when the drive binds, before any start | **Any new path that starts an app when its drive arrives calls `ResumePutBack` first** (the boot recreate and `restartStacks` do, in web via `Server.resumePutBack`). Every other start path is covered by the hold store it already honours — do not add a second check. A failed put-back (at start or on resume) writes the D20 hold (`restore_mixed`) and drops the app from the marker. Needs `SetHolds` + `SetUndoReachable` before `Recover` (main.go). Pinned by `r1251_putback_waits_test.go`, `TestR1251_DriveStartPathsPutBackBeforeTheyStart`, `TestR1251_MainWiresThePutBackWait` | | `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned | | `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces | diff --git a/controller/README.md b/controller/README.md index 926c5764..9b06d78d 100644 --- a/controller/README.md +++ b/controller/README.md @@ -2283,6 +2283,10 @@ that folder is never a dead end, and an install never runs into it silently (R-6 - **R-1286 (v0.344.0).** A held app says WHY: each hold reason (a failed restore, an unhealthy stop by the crash-loop guard, …) has its own sentence in the start gate, the boot sweep and `--restore-holds` (an unhealthy stop used to read as a failed restore). +- **D53 (next release, felhom.eu `08` §10).** The hub learns which apps are held: `internal/holdreport` reads the hold + store every minute and sends `app_held` (stack, reason, since, the hold's sentence) for a new or changed hold and + `app_hold_cleared` when one is gone — also one cleared while the controller was down (`settings.reported_holds`) — and + repeats every hold daily and at start. Both are severity info; the hub's operator reminder after a grace is the mail. - **R-690 (fixed here).** The removed-app restore (R-487) never found a unit on a DATA drive — it asked `GetStackComposePath`, true for every catalog app — and restored with no env. It now asks `isStackDeployed`. - `.felhom.yml` **`after_load: {service, user, command: [...]}`** — one command run with `docker compose exec -T` diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index a7dd69f4..fea6d5e4 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -40,6 +40,7 @@ import ( "gitea.dooplex.hu/admin/felhom-controller/internal/crashboot" "gitea.dooplex.hu/admin/felhom-controller/internal/crypto" "gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch" + "gitea.dooplex.hu/admin/felhom-controller/internal/holdreport" "gitea.dooplex.hu/admin/felhom-controller/internal/infra" "gitea.dooplex.hu/admin/felhom-controller/internal/integrations" "gitea.dooplex.hu/admin/felhom-controller/internal/mailrelay" @@ -1063,6 +1064,14 @@ func main() { go func() { defer aubBusy.Unlock(); aubRunner.Tick(time.Now()) }() // a run can take minutes; never block the scheduler return nil }) + // D53 (felhom.eu `08` §10): tell the hub which apps are held — app_held for a new or changed hold, app_hold_cleared + // when one is gone, every hold again once a day and after each start. One observer of the hold store, so every + // writer is covered. Pinned by internal/holdreport. + holdReporter := holdreport.New(sett, notifier.PushEvent, logger) + sched.Every("hold-report", time.Minute, func(ctx context.Context) error { + holdReporter.Tick(time.Now()) + return nil + }) sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error { if bootGrace.Within(time.Now()) { return nil // still inside the startup settle window (90 s, or ~15 min after a crash boot) @@ -2562,22 +2571,7 @@ type driveStartGate struct { // stop read as „held after a failed restore whose rollback also failed" for 11 days on demo-hp, so the operator could // not tell a crash loop from a broken restore). Pinned by TestR1286_EveryHoldReasonSaysWhatItIs. func holdWhy(h settings.RestoreHold) string { - switch h.Reason { - case settings.HoldReasonUpdateFailed: - return "held after a failed update (" + h.At + ") — restore it from its backup to start it" - case settings.HoldReasonUnhealthyStop: - kind := h.UnhealthyKind - if kind == "" { - kind = "unhealthy" - } - return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try" - case settings.HoldReasonRestoreMixed: - return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it" - case settings.HoldReasonPutBackPending: - return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it" - default: - return "held after a failed restore whose rollback also failed (" + h.At + ") — clear the hold to start it" - } + return h.Why() // one sentence per reason, shared with the hub's „Held apps" list (D53) } func (g driveStartGate) MayStart(stackName string) (bool, string) { diff --git a/controller/internal/holdreport/holdreport.go b/controller/internal/holdreport/holdreport.go new file mode 100644 index 00000000..3ab060c0 --- /dev/null +++ b/controller/internal/holdreport/holdreport.go @@ -0,0 +1,139 @@ +// Package holdreport tells the hub which apps are held (D53, felhom.eu `08` §10). +// +// A held app is stopped on purpose and stays stopped until a person acts. Until D53 the hub knew nothing about it: +// the crash-loop stop sent one event, the other holds sent none, and demo-hp's Kimai sat held for 11 days unseen +// (R-1291). The legacy report is frozen, so the event channel carries the holds: +// +// - app_held when a hold appears or changes (stack, reason, since, the hold's sentence); +// - app_hold_cleared when it is gone — also when it went while the controller was down, because the set last +// reported is kept in settings (ReportedHolds), not in memory; +// - every hold still in place is sent again once a day and at every controller start, because an event can be +// lost. The hub treats a repeat as idempotent. +// +// ONE OBSERVER, NOT A HOOK PER WRITER. Seven code paths write a hold and four clear one. A send in each of them is +// the shape where the one path nobody found is the next silent hold; this package reads the hold store itself, so a +// new writer is covered the day it is added. Both events are severity info: the hub mails nothing on receipt. Its +// reminder after the grace per reason is the operator's mail, and it never goes to the household (D53). +// +// Pinned by holdreport_test.go. +package holdreport + +import ( + "log" + "sort" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/settings" +) + +// Event types — the hub's allowedEventTypes carries both (the coupling rule). +const ( + EventHeld = "app_held" + EventCleared = "app_hold_cleared" +) + +// ResendEvery is how often every hold still in place is sent again. +const ResendEvery = 24 * time.Hour + +// Store is what the reporter needs of the settings. +type Store interface { + ListRestoreHolds() []settings.RestoreHold + GetReportedHolds() map[string]string + SetReportedHolds(map[string]string) error +} + +// Push sends one event to the hub (notify.Notifier.PushEvent). +type Push func(eventType, severity, message string, details interface{}) + +// Details is the payload of both events. +type Details struct { + StackName string `json:"stack_name"` + Reason string `json:"reason,omitempty"` + Since string `json:"since,omitempty"` + Why string `json:"why,omitempty"` + Repeat bool `json:"repeat,omitempty"` // a daily or start-up re-send, not a new hold +} + +// Reporter compares the hold store with what was last reported. Tick is called by the scheduler. +type Reporter struct { + store Store + push Push + logger *log.Logger + lastFull time.Time +} + +// New builds a reporter. The first Tick re-sends every hold (a controller start). +func New(store Store, push Push, logger *log.Logger) *Reporter { + return &Reporter{store: store, push: push, logger: logger} +} + +func key(h settings.RestoreHold) string { return h.WireReason() + "|" + h.At } + +// Tick sends what changed since the last tick, and every hold again when a day has passed. +func (r *Reporter) Tick(now time.Time) { + if r.store == nil || r.push == nil { + return + } + holds := r.store.ListRestoreHolds() + sort.Slice(holds, func(i, j int) bool { return holds[i].Stack < holds[j].Stack }) + prev := r.store.GetReportedHolds() + full := r.lastFull.IsZero() || now.Sub(r.lastFull) >= ResendEvery + + cur := make(map[string]string, len(holds)) + sent, cleared := 0, 0 + for _, h := range holds { + k := key(h) + cur[h.Stack] = k + known, seen := prev[h.Stack] + if seen && known == k && !full { + continue + } + repeat := seen && known == k + r.push(EventHeld, "info", "App "+h.Stack+" is held: "+h.Why(), + Details{StackName: h.Stack, Reason: h.WireReason(), Since: h.At, Why: h.Why(), Repeat: repeat}) + sent++ + if !repeat { + r.logf("[INFO] [holdreport] %s held (%s since %s) — reported to the hub", h.Stack, h.WireReason(), h.At) + } + } + gone := make([]string, 0) + for stack := range prev { + if _, ok := cur[stack]; !ok { + gone = append(gone, stack) + } + } + sort.Strings(gone) + for _, stack := range gone { + r.push(EventCleared, "info", "App "+stack+" is no longer held", Details{StackName: stack}) + cleared++ + r.logf("[INFO] [holdreport] %s hold cleared — reported to the hub", stack) + } + if full { + r.lastFull = now + r.logf("[DEBUG] [holdreport] full re-send: %d hold(s) sent, %d cleared", sent, cleared) + } + if !sameMap(prev, cur) { + if err := r.store.SetReportedHolds(cur); err != nil { + // Not fatal: the next tick sends the same difference again, which the hub reads as idempotent. + r.logf("[WARN] [holdreport] could not save the reported holds: %v — the next tick repeats the difference", err) + } + } +} + +func sameMap(a, b map[string]string) bool { + if len(a) != len(b) { + return false + } + for k, v := range a { + if b[k] != v { + return false + } + } + return true +} + +func (r *Reporter) logf(format string, args ...interface{}) { + if r.logger != nil { + r.logger.Printf(format, args...) + } +} diff --git a/controller/internal/holdreport/holdreport_test.go b/controller/internal/holdreport/holdreport_test.go new file mode 100644 index 00000000..5cf5e3bf --- /dev/null +++ b/controller/internal/holdreport/holdreport_test.go @@ -0,0 +1,182 @@ +package holdreport + +import ( + "io" + "log" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/settings" +) + +type fakeStore struct { + holds map[string]settings.RestoreHold + reported map[string]string +} + +func (f *fakeStore) ListRestoreHolds() []settings.RestoreHold { + out := []settings.RestoreHold{} + for _, h := range f.holds { + out = append(out, h) + } + return out +} +func (f *fakeStore) GetReportedHolds() map[string]string { + out := map[string]string{} + for k, v := range f.reported { + out[k] = v + } + return out +} +func (f *fakeStore) SetReportedHolds(m map[string]string) error { f.reported = m; return nil } + +type sent struct { + typ, sev string + d Details +} + +func rec(out *[]sent) Push { + return func(t, sev, msg string, d interface{}) { *out = append(*out, sent{t, sev, d.(Details)}) } +} + +var t0 = time.Date(2026, 10, 26, 18, 0, 0, 0, time.UTC) + +// The report field (D53): a new hold is sent once as app_held with its stack, reason and since, severity info (the hub +// mails nothing on receipt; its reminder after the grace is the mail). +func TestD53_ANewHoldIsReportedWithStackReasonAndSince(t *testing.T) { + st := &fakeStore{holds: map[string]settings.RestoreHold{ + "kimai": {Stack: "kimai", At: "2026-10-14T08:00:00Z", Reason: settings.HoldReasonUnhealthyStop, UnhealthyKind: "crash_loop"}, + }} + var out []sent + r := New(st, rec(&out), nil) + r.Tick(t0) + if len(out) != 1 || out[0].typ != EventHeld || out[0].sev != "info" { + t.Fatalf("want one info app_held, got %+v", out) + } + d := out[0].d + if d.StackName != "kimai" || d.Reason != "unhealthy_stop" || d.Since != "2026-10-14T08:00:00Z" || d.Why == "" { + t.Fatalf("payload: %+v", d) + } + if st.reported["kimai"] != "unhealthy_stop|2026-10-14T08:00:00Z" { + t.Fatalf("reported record: %v", st.reported) + } + // The next tick inside a day sends nothing. + out = nil + r.Tick(t0.Add(time.Minute)) + if len(out) != 0 { + t.Fatalf("an unchanged hold was sent again inside a day: %+v", out) + } +} + +// An event can be lost: every hold still in place is sent again after a day, marked as a repeat. +func TestD53_AHoldIsSentAgainEveryDay(t *testing.T) { + st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonUpdateFailed}}} + var out []sent + r := New(st, rec(&out), nil) + r.Tick(t0) + out = nil + r.Tick(t0.Add(ResendEvery)) + if len(out) != 1 || out[0].typ != EventHeld || !out[0].d.Repeat { + t.Fatalf("want one repeat app_held after a day, got %+v", out) + } +} + +// A hold cleared by a person ends it: one app_hold_cleared, then nothing. +func TestD53_AClearedHoldSendsClearedOnce(t *testing.T) { + st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonRestoreMixed}}} + var out []sent + r := New(st, rec(&out), nil) + r.Tick(t0) + delete(st.holds, "a") + out = nil + r.Tick(t0.Add(time.Minute)) + if len(out) != 1 || out[0].typ != EventCleared || out[0].d.StackName != "a" || out[0].sev != "info" { + t.Fatalf("want one info app_hold_cleared for a, got %+v", out) + } + if len(st.reported) != 0 { + t.Fatalf("the reported record still names the cleared hold: %v", st.reported) + } + out = nil + r.Tick(t0.Add(2 * time.Minute)) + r.Tick(t0.Add(ResendEvery + time.Hour)) + if len(out) != 0 { + t.Fatalf("a cleared hold was reported again: %+v", out) + } +} + +// A hold cleared while the controller was down is still reported cleared: the record lives in settings. +func TestD53_AHoldClearedWhileDownIsReportedAtStart(t *testing.T) { + st := &fakeStore{holds: map[string]settings.RestoreHold{}, reported: map[string]string{"gone": "update_failed|2026-10-01T00:00:00Z"}} + var out []sent + New(st, rec(&out), nil).Tick(t0) + if len(out) != 1 || out[0].typ != EventCleared || out[0].d.StackName != "gone" { + t.Fatalf("want app_hold_cleared for gone at start, got %+v", out) + } +} + +// A hold whose reason changes (a put-back that failed becomes a mixed restore) is a new hold for the hub. +func TestD53_AChangedReasonIsReportedAgain(t *testing.T) { + st := &fakeStore{holds: map[string]settings.RestoreHold{"a": {Stack: "a", At: "2026-10-20T00:00:00Z", Reason: settings.HoldReasonPutBackPending}}} + var out []sent + r := New(st, rec(&out), nil) + r.Tick(t0) + st.holds["a"] = settings.RestoreHold{Stack: "a", At: "2026-10-20T01:00:00Z", Reason: settings.HoldReasonRestoreMixed} + out = nil + r.Tick(t0.Add(time.Minute)) + if len(out) != 1 || out[0].typ != EventHeld || out[0].d.Reason != "restore_mixed" || out[0].d.Repeat { + t.Fatalf("want a new (not repeat) app_held restore_mixed, got %+v", out) + } +} + +// The legacy empty reason is a restore that failed with its rollback: the hub reads it as restore_failed. +func TestD53_TheEmptyReasonIsRestoreFailedOnTheWire(t *testing.T) { + st := &fakeStore{holds: map[string]settings.RestoreHold{"db": {Stack: "db", At: "2026-10-20T00:00:00Z"}}} + var out []sent + New(st, rec(&out), nil).Tick(t0) + if len(out) != 1 || out[0].d.Reason != "restore_failed" { + t.Fatalf("want reason restore_failed, got %+v", out) + } +} + +// The seam is wired: main builds the reporter on the real settings and notifier and ticks it from the scheduler. A +// reporter nobody ticks is the inert-seam shape (four instances in this project). A commented-out line does not count. +func TestD53_TheReporterIsWiredInMain(t *testing.T) { + raw, err := os.ReadFile(filepath.Join("..", "..", "cmd", "controller", "main.go")) + if err != nil { + t.Fatalf("cannot read main.go: %v", err) + } + live := []string{} + for _, l := range strings.Split(string(raw), "\n") { + if !strings.HasPrefix(strings.TrimSpace(l), "//") { + live = append(live, l) + } + } + src := strings.Join(live, "\n") + for _, want := range []string{"holdreport.New(sett, notifier.PushEvent, logger)", `sched.Every("hold-report"`, "holdReporter.Tick("} { + if !strings.Contains(src, want) { + t.Errorf("main.go lacks %q — the hold report is not wired", want) + } + } +} + +// The reported record survives a restart: it is what lets a hold cleared while the controller was down be reported. +func TestD53_TheReportedRecordSurvivesAReload(t *testing.T) { + p := filepath.Join(t.TempDir(), "settings.json") + s, err := settings.Load(p, log.New(io.Discard, "", 0)) + if err != nil { + t.Fatal(err) + } + if err := s.SetReportedHolds(map[string]string{"kimai": "unhealthy_stop|2026-10-14T08:00:00Z"}); err != nil { + t.Fatal(err) + } + s2, err := settings.Load(p, log.New(io.Discard, "", 0)) + if err != nil { + t.Fatal(err) + } + if got := s2.GetReportedHolds()["kimai"]; got != "unhealthy_stop|2026-10-14T08:00:00Z" { + t.Fatalf("after a reload the record reads %q", got) + } +} diff --git a/controller/internal/settings/settings.go b/controller/internal/settings/settings.go index c3d0ec50..e16326b8 100644 --- a/controller/internal/settings/settings.go +++ b/controller/internal/settings/settings.go @@ -232,6 +232,10 @@ type Settings struct { // out-of-memory storm (RFC3339). Survives the hold being lifted by Start, so a second stop within // 24 h is known to be a second one. UnhealthyStops map[string]string `json:"unhealthy_stops,omitempty"` + // ReportedHolds (D53, `08` §10) — the holds last told to the hub as app_held: stack → „reason|at". The hold + // reporter (internal/holdreport) compares it with RestoreHolds to send app_held for a new or changed hold and + // app_hold_cleared for one that is gone — also one cleared while the controller was down, because it is kept here. + ReportedHolds map[string]string `json:"reported_holds,omitempty"` // Cross-drive restic repo password (auto-generated on first use) CrossDriveResticPassword string `json:"cross_drive_restic_password,omitempty"` @@ -1998,6 +2002,35 @@ const ( HoldReasonPutBackPending = "putback_pending" ) +// WireReason is the hold's reason as the hub reads it (D53): the empty legacy value is named restore_failed. +func (h RestoreHold) WireReason() string { + if h.Reason == HoldReasonRestoreFailed { + return "restore_failed" + } + return h.Reason +} + +// Why says which operation put a hold in place and what lifts it — one sentence per reason (R-1286). English, +// operator-facing: the controller's own -list-restore-holds output and the hub's „Held apps" list (D53) both use it. +func (h RestoreHold) Why() string { + switch h.Reason { + case HoldReasonUpdateFailed: + return "held after a failed update (" + h.At + ") — restore it from its backup to start it" + case HoldReasonUnhealthyStop: + kind := h.UnhealthyKind + if kind == "" { + kind = "unhealthy" + } + return "stopped by the box for a " + strings.ReplaceAll(kind, "_", " ") + " (" + h.At + ") — the household's Start gives it one more try" + case HoldReasonRestoreMixed: + return "held after an off-site restore failed its database replay (" + h.At + ") — clear the hold to start it" + case HoldReasonPutBackPending: + return "held until the pre-restore copy is poured back (" + h.At + ") — the put-back starts it" + default: + return "held after a failed restore whose rollback also failed (" + h.At + ") — clear the hold to start it" + } +} + // SetRestoreHold records a hold. Modelled on SetDisconnected: a condition, plus what it is holding. func (s *Settings) SetRestoreHold(h RestoreHold) error { s.mu.Lock() @@ -2031,6 +2064,32 @@ func (s *Settings) ListRestoreHolds() []RestoreHold { return out } +// GetReportedHolds returns a copy of the holds last reported to the hub (D53): stack → „reason|at". +func (s *Settings) GetReportedHolds() map[string]string { + s.mu.RLock() + defer s.mu.RUnlock() + out := make(map[string]string, len(s.ReportedHolds)) + for k, v := range s.ReportedHolds { + out[k] = v + } + return out +} + +// SetReportedHolds replaces the record of the holds reported to the hub (D53). An empty map removes the field. +func (s *Settings) SetReportedHolds(m map[string]string) error { + s.mu.Lock() + defer s.mu.Unlock() + if len(m) == 0 { + s.ReportedHolds = nil + } else { + s.ReportedHolds = make(map[string]string, len(m)) + for k, v := range m { + s.ReportedHolds[k] = v + } + } + return s.save() +} + // ClearRestoreHold removes a hold. Returns false when there was none, so a caller can tell "cleared" // from "there was nothing to clear" rather than reporting success either way. func (s *Settings) ClearRestoreHold(stack string) (bool, error) {