From 635c33d381918f31a1228c477bf5caad9959e7b1 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 5 Oct 2026 09:27:16 +0200 Subject: [PATCH] v0.295.0: a box that was off at its backup time catches up once (R-871, decision 109); the missed-backup banner (decision 110); a late daily timer after a host suspend is skipped Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- CHANGELOG.md | 29 + REUSE.md | 2 + controller/README.md | 12 + controller/cmd/controller/main.go | 96 ++- .../controller/r871_catchup_wiring_test.go | 63 ++ controller/internal/i18n/locales/en.json | 8 + controller/internal/i18n/locales/hu.json | 8 + controller/internal/metrics/store.go | 28 + controller/internal/nightchain/banner.go | 125 ++++ controller/internal/nightchain/banner_test.go | 123 ++++ controller/internal/nightchain/nightchain.go | 365 +++++++++++ .../internal/nightchain/nightchain_test.go | 248 ++++++++ controller/internal/notify/notifier.go | 6 + controller/internal/quiesce/quiesce.go | 26 + .../internal/quiesce/r871_catchup_test.go | 30 + controller/internal/scheduler/scheduler.go | 54 +- .../internal/scheduler/scheduler_test.go | 58 ++ controller/internal/web/i18n_parity_test.go | 10 + .../internal/web/missed_backup_banner.go | 113 ++++ .../web/r871_missed_backup_banner_test.go | 59 ++ controller/internal/web/server.go | 9 + controller/internal/web/templates/layout.html | 18 + .../i18n_parity/launcher_missed_backup.html | 596 ++++++++++++++++++ controller/scripts/i18n_go_keys.json | 6 + 24 files changed, 2074 insertions(+), 18 deletions(-) create mode 100644 controller/cmd/controller/r871_catchup_wiring_test.go create mode 100644 controller/internal/nightchain/banner.go create mode 100644 controller/internal/nightchain/banner_test.go create mode 100644 controller/internal/nightchain/nightchain.go create mode 100644 controller/internal/nightchain/nightchain_test.go create mode 100644 controller/internal/quiesce/r871_catchup_test.go create mode 100644 controller/internal/web/missed_backup_banner.go create mode 100644 controller/internal/web/r871_missed_backup_banner_test.go create mode 100644 controller/internal/web/testdata/i18n_parity/launcher_missed_backup.html diff --git a/CHANGELOG.md b/CHANGELOG.md index 016d265..b8a01a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,32 @@ +## v0.295.0 — a box that was off at its backup time catches up once; the household sees why in a banner (R-871, `09` decisions 109–110) (2026-10-05) + +**MinAgent: 0.131.0** (unchanged). New household strings: the timeline line `event.backup_catchup_done` and the banner +(`banner.missed_backup.*`, `layout.missed_backup_*`), Hungarian and English. + +- **The catch-up (decision 109).** New `internal/nightchain`: a persisted night ledger records when each backup leg + (database dump, second copy, off-site copy) last ran to its end. On a controller start and on a host resume, a leg + that missed its last scheduled time is made up ONCE, 15 minutes later, in the night's order, through the same + wrapped leg bodies the scheduled jobs run (`withLeg`, `catchUpLegs`) — never the app-update leg, never a Docker + step. Several missed nights = one catch-up; a daytime restart after a normal night = none; a power cut mid-chain = + only the legs that did not end; a leg due within 30 min is left to its normal run; the ledger is seeded at the first + start of this release (no catch-up at the upgrade). One lock for every leg. The whole-guest backup defers while a + catch-up runs (`quiesce.SetCatchUpFn`) and the catch-up waits for a quiesce. Measured before the fix on 9202: off + across a 09:05 window, on at 09:08 → `db-dump scheduled for 2026-10-06 09:05`, nothing ran. +- **A host suspend.** A daily job whose timer fires more than 60 min after its wall-clock time is SKIPPED + (`scheduler.DailyLateLimit`) — Go timers run on CLOCK_MONOTONIC, which stops while suspended, so the app-update leg + would otherwise start at noon. A resume watch (wall vs monotonic clock, every minute) triggers the catch-up. +- **The banner (decision 110, the operator's idea).** When the last database dump (and the off-site copy, when + configured) is over 26 h old, every page of a logged-in household says when the last backup was, "the box was off + at backup time (02:30)" when the metrics record shows no sample then, and suggests the latest hour the box is + usually on (5 of the last 7 days, for 3 hours). A button opens the backup-time setting; it never changes the time. + Closing it (`POST /backups/missed-banner/dismiss`) lasts until the next missed backup time (durable, in the ledger); + a successful night removes it. New `MetricsStore.SampleCount` / `HourSampleCounts`. +- The off-site leg's "no target" log line no longer says "the update leg runs now" (the catch-up runs that body too). +- Tests: `internal/nightchain` (`TestCatchUp_*`, `TestResumeWatch_*`, `TestBanner_*`), `TestDaily_LateFireIsSkipped`, + `TestR871_ScheduledCycleWaitsForCatchUp`, `TestR871_CatchUpWiring`, `TestR871_CatchUpRunsNoUpdateLeg`, + `TestR871_Banner*` (real pages + the real dismiss route), parity fixture `launcher_missed_backup`. Red-proofs: + `felhom.eu/documentation/audits/catchup-2026-10-05/part{A,B}/red-proofs.txt`. + ## v0.294.0 — the off-site clean-up deletes honest old copies; a first install survives the image clean-up; failed compose logs its reason; the move-aside line names its destination (R-867, R-863, R-864, R-869) (2026-10-05) **MinAgent: 0.131.0** (unchanged). No new household string. diff --git a/REUSE.md b/REUSE.md index 592a512..75fa8d7 100644 --- a/REUSE.md +++ b/REUSE.md @@ -32,6 +32,8 @@ | `stacks.OpenInstallHold` / `installHoldTick` (v0.284.0, R-741) | controller/internal/stacks/install_hold.go | `(name, by)` / `()` | An `after_install` app held behind the setup gate's door until its known login is replaced | **Written before the first start**, like the gate; opens on `after_install` success or the household's "I changed it"; the door (`SetupGateHost`) reads holds first | | `stacks.RetainImagesAfterUpdate` / `RetainImagesAfterRemove` / `RunImageRetentionOnce` (→ `(deleted, done)` since v0.294.0) / `RunImageRetentionOnceUntilDone` · seam `imageDocker` (v0.284.0, decision 53) | controller/internal/stacks/image_retention.go | `(name, previous)` / `(name, repos)` / `()` | Deletes an app's images older than its running + previous one | **The keep set is box-wide and read at delete time** (containers, installed composes, installed/previous records); exact id, never forced or pruned; skipped while any update runs AND while any image-pulling compose command runs (R-863, `dockerexec.TryImageCleanup`); the one-time pass writes its marker only after a pass that RAN; tests use the `imageDocker` seam, never Docker | | `dockerexec.BeginImageWork(args)` / `TryImageCleanup()` / `ImagePulling(args)` (v0.294.0, R-863) | controller/internal/dockerexec/imagework.go | `defer dockerexec.BeginImageWork(args)()` | **Every NEW place that runs `compose up/pull/create/run` must hold it** (the stacks compose helpers, appexport's restore and the FileBrowser sync already do) | An image pulled by compose is named by no container until compose creates one; a clean-up pass in that window deleted it (measured, R-863). The pass takes the lock exclusively without waiting; pulling commands wait for a running pass (seconds) | +| `nightchain.Ledger` / `CatchUp` / `ResumeWatch` / `ComputeBanner` (v0.295.0, R-871) | controller/internal/nightchain/ | `Open(path, now)`, `MarkEnded(leg, t)`, `Missed(now, W, loc)`, `Evaluate(ctx, why)` | "was a night missed?" and making it up; the missed-backup banner's rule | The ledger is an ATTEMPT record (ran to its end) — never evidence a backup exists; a NEW nightly backup leg must be wrapped by `withLeg` in main.go and listed in `nightchain.Order`, or the catch-up never makes it up; never add the update leg to `catchUpLegs` | +| `scheduler.DailyLateLimit` (v0.295.0) | controller/internal/scheduler/scheduler.go | — | a daily timer firing > 60 min late (host suspend) is skipped | do not "fix" a late fire by running it: the app-update leg would run at noon | | `stacks.RetainControllerImages(ControllerImageRecord)` · `selfupdate.UpdateState.RecordedPrevious` (v0.285.0, decision 56) | controller/internal/stacks/controller_image_retention.go | `({Repo, Running, Previous})` | Deletes controller images older than the running + previous one | The previous comes from the SWAP RECORD (success onto the running version), version order only as the fallback; versions above the running one and non-version tags are kept; skipped while the controller swaps itself; same `imageDocker` seam | | `stacks.CloseSignupNow` / `CloseSignupOffered` / `applyNativeLock` (v0.282.0, decisions 47/49) | controller/internal/stacks/after_setup.go | `(name)` | The app's own sign-up switch after the setup; "close sign-up now" for an app installed before the rule | **Check the installed compose reads the variable** (an old install carries the old compose until its next update) — never record a lock that is not there. One run per app at a time (`nativeLockBusy`) | | `backup.judgeCopy` / `HollowCopies` / `SetHollowCopyNotify` (Part D, v0.279.0) | controller/internal/backup/hollow_watch.go | `(app, tier, unitDir)` | A RUNNING app whose newest copy holds no data → operator digest once/day + page sentence | Uses `unitCarriesData` (the manifest, never size); a stopped held app is never flagged | diff --git a/controller/README.md b/controller/README.md index eeba595..ec26565 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1116,6 +1116,18 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, self-sufficient backup** — any single tier can fully restore an app. +**A box that was off at its backup time catches up (v0.295.0, R-871, `09` decisions 109–110).** +`internal/nightchain`. The night ledger (`/night-ledger.json`) records when each nightly backup leg (database +dump, second copy, off-site copy) last ran to its end. On a controller start or a host resume, a leg that missed its +last scheduled time is made up ONCE, 15 minutes later, in the night's order — backup legs only, never the app-update +leg or a Docker step. Several missed nights are one catch-up; a scheduled leg and a catch-up never overlap; the +whole-guest backup and the catch-up wait for each other. A daily job whose timer fires over 60 minutes late (a host +suspend) is skipped, not run late. The household gets one timeline line (`backup_catchup_done`). +**The missed-backup banner:** when the last daily backup is over 26 h old, every page says when it was, that the box +was off at the backup time (from the metrics record, one sample a minute), and suggests a time when the box is usually +on — it never changes the time itself. Closing it lasts until the next missed backup time; a successful night removes +it. Design: `felhom.eu/documentation/architecture/07-backup-architecture.md` §6.1.1. + **The restore carries the customer's own previous answers (v0.217.0, R-351).** `internal/backup/offbox_placement.go`. Every recovery unit's `manifest.json` records `drive` and `namespace_root`, and its `compose/app.yaml` records `SUBDOMAIN`/`DOMAIN`. Until v0.217.0 nothing read diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 280468c..2cc96a8 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -43,6 +43,7 @@ import ( "gitea.dooplex.hu/admin/felhom-controller/internal/mailrelay" "gitea.dooplex.hu/admin/felhom-controller/internal/metrics" "gitea.dooplex.hu/admin/felhom-controller/internal/monitor" + "gitea.dooplex.hu/admin/felhom-controller/internal/nightchain" "gitea.dooplex.hu/admin/felhom-controller/internal/notify" "gitea.dooplex.hu/admin/felhom-controller/internal/offsiteapply" "gitea.dooplex.hu/admin/felhom-controller/internal/quiesce" @@ -1110,7 +1111,30 @@ func main() { // App-data backup: daily database dumps. Disk-tier (restic snapshots, // cross-drive, integrity check, infra backup) has moved to the host agent. - sched.Daily("db-dump", dbLeg, func(ctx context.Context) error { + // R-871 (v0.295.0, `09` decision 109): the night ledger records when each leg ran to its end, so a box that + // was off at W makes the night up ONCE when it comes back (internal/nightchain). Every leg body is wrapped + // once and shared by the scheduled run and the catch-up — the two can never drift apart. + nightLedger, nerr := nightchain.Open(filepath.Join(cfg.Paths.DataDir, "night-ledger.json"), time.Now()) + if nerr != nil { + logger.Printf("[WARN] [catch-up] the night ledger cannot be read — no catch-up on this box until it can: %v", nerr) + } + webNightLedger = nightLedger + withLeg := func(leg nightchain.Leg, fn func(context.Context) error) func(context.Context) error { + return func(ctx context.Context) error { + if nightLedger == nil { + return fn(ctx) + } + nightLedger.LegLock.Lock() // a scheduled leg and a catch-up never run at the same time + defer nightLedger.LegLock.Unlock() + err := fn(ctx) + nightLedger.MarkEnded(leg, time.Now()) + if leg == nightchain.LegDBDump && err == nil { + nightLedger.MarkDBDumpOK(time.Now()) + } + return err + } + } + dbDumpLeg := withLeg(nightchain.LegDBDump, func(ctx context.Context) error { err := backupMgr.RunDBDumps(ctx) if err != nil { notifier.NotifyDBDumpFailed("Adatbázis mentés sikertelen", err.Error()) @@ -1119,6 +1143,7 @@ func main() { } return err }) + sched.Daily("db-dump", dbLeg, dbDumpLeg) // ── R-218, THE CONSUME HALF (v0.203.0) ──────────────────────────────────────────────── // @@ -1204,10 +1229,11 @@ func main() { }) } }) - sched.Daily("tier2-backup", tier2Leg, func(ctx context.Context) error { + tier2LegFn := withLeg(nightchain.LegTier2, func(ctx context.Context) error { backupMgr.RunAllTier2() return nil }) + sched.Daily("tier2-backup", tier2Leg, tier2LegFn) // Off-box (NAS) restic-SFTP backup (Part B): the off-site leg. A failure (incl. a fail-fast dead-NAS // error) alerts the operator via the allowlisted backup_failed event. Daily after Tier 2. @@ -1353,16 +1379,49 @@ func main() { // off-site job — it runs when this function's off-site half has ended, on every path: configured // or not, ok, failed, any of RunOffboxBackup's early returns, even a panic. A box with no off-site // target runs it at W+105m. One call site, no signal to go stale. - sched.Daily("offbox-backup", offboxLeg, func(ctx context.Context) error { - return chainUpdateLeg(ctx, func(ctx context.Context) error { - t := sett.GetOffboxTarget() - if t == nil || !t.Enabled || t.Schedule != "daily" || !backupMgr.OffboxConfigured() { - logger.Printf("[INFO] [offbox] no scheduled off-site target on this box — the off-site leg does nothing; the update leg runs now") - return nil // not configured / not scheduled - } - return backupMgr.RunOffboxBackup(ctx) - }, func(ctx context.Context) { stackMgr.RunUpdateLeg(ctx, "after-offsite") }) + offsiteOnly := withLeg(nightchain.LegOffsite, func(ctx context.Context) error { + t := sett.GetOffboxTarget() + if t == nil || !t.Enabled || t.Schedule != "daily" || !backupMgr.OffboxConfigured() { + logger.Printf("[INFO] [offbox] no scheduled off-site target on this box — the off-site leg does nothing") + return nil // not configured / not scheduled + } + return backupMgr.RunOffboxBackup(ctx) }) + sched.Daily("offbox-backup", offboxLeg, func(ctx context.Context) error { + return chainUpdateLeg(ctx, offsiteOnly, func(ctx context.Context) { stackMgr.RunUpdateLeg(ctx, "after-offsite") }) + }) + // R-871: the catch-up — the three BACKUP legs only (no update leg: it restarts apps and waits for a real + // night). Triggered at start and on a host resume; it waits 15 minutes, and it waits for a whole-guest + // backup (and that backup waits for it). Pinned by internal/nightchain + TestR871_CatchUpWiring. + if nightLedger != nil { + catchUp := &nightchain.CatchUp{ + Ledger: nightLedger, + Window: func() string { + return backupwindow.EffectiveWindow(sett.GetBackupWindowStart(), cfg.Backup.DBDumpSchedule) + }, + Legs: catchUpLegs(dbDumpLeg, tier2LegFn, offsiteOnly), + QuiesceBusy: func() bool { + return quiesceLoop != nil && len(quiesceLoop.SuppressedStacks()) > 0 + }, + Event: func(missedAt time.Time) { + notifier.NotifyBackupCatchUp(missedAt.In(budapestLoc()).Format("15:04")) + }, + Logger: logger, + Loc: budapestLoc(), + } + if quiesceLoop != nil { + quiesceLoop.SetCatchUpFn(catchUp.Running) + } + catchUp.Evaluate(ctx, "controller start") + resumeTick := time.NewTicker(time.Minute) + go nightchain.ResumeWatch(ctx, resumeTick.C, + func() time.Time { return time.Now().Round(0) }, + func() time.Duration { return time.Since(startTime) }, + 5*time.Minute, + func(slept time.Duration) { + catchUp.Evaluate(ctx, fmt.Sprintf("host resume (suspended ~%s)", slept.Round(time.Minute))) + }) + } // R-241 — the abandonment terminal step. DAILY and not on the backup leg, deliberately: it must // run on a box whose off-site tier is NOT configured for runs (an abandoning box may be sitting // with escrow pending), and tying it to the backup leg would make the deletion depend on a @@ -1815,6 +1874,8 @@ func main() { // --- Initialize web server --- webServer := web.NewServer(cfg, stackMgr, cpuCollector, backupMgr, sched, sett, alertMgr, notifier, updater, logger, Version) + // R-871 (decision 110): the missed-backup banner reads the night ledger and the box's own on/off record. + webServer.SetMissedBackupBanner(webNightLedger, metricsStore) // Migration done-hook: a decommission-initiated migration finalizes the source decommission on // success (soft-mark + agent). Wire it before RecoverMigration so a resumed one still finalizes. stackMgr.SetMigrationDoneHook(webServer.OnMigrationDone) @@ -3910,6 +3971,19 @@ func stopUnhealthyApps(logger *log.Logger, d unhealthyDeps, ooms []stacks.OOMCon // returned nil (ran, or had nothing to do), returned an error, or panicked. The off-site half's error is // returned (the scheduler logs it); a panic becomes an error. `09` §6.4.2 point 2; pinned by // TestChainUpdateLeg_EveryPath. +// catchUpLegs is the catch-up's leg table (R-871): the three backup legs and NOTHING else — no update leg, no +// Docker step. A function so the table is one reviewable place; pinned by TestR871_CatchUpWiring. +func catchUpLegs(db, tier2, offsite func(context.Context) error) map[nightchain.Leg]func(context.Context) error { + return map[nightchain.Leg]func(context.Context) error{ + nightchain.LegDBDump: db, + nightchain.LegTier2: tier2, + nightchain.LegOffsite: offsite, + } +} + +// webNightLedger hands the ledger to the web server (the missed-backup banner); nil when backups are off. +var webNightLedger *nightchain.Ledger + func chainUpdateLeg(ctx context.Context, offsite func(context.Context) error, leg func(context.Context)) (err error) { defer func() { if r := recover(); r != nil { diff --git a/controller/cmd/controller/r871_catchup_wiring_test.go b/controller/cmd/controller/r871_catchup_wiring_test.go new file mode 100644 index 0000000..89c54de --- /dev/null +++ b/controller/cmd/controller/r871_catchup_wiring_test.go @@ -0,0 +1,63 @@ +package main + +import ( + "context" + "go/ast" + "os" + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/nightchain" +) + +// TestR871_CatchUpWiring — v0.295.0 (`09` decision 109): the catch-up is built, triggered at start and on a host +// resume, and holds the whole-guest cycle — the seam-built-but-never-wired class (four earlier instances). +// COMPANION RED-PROOF: delete the `catchUp.Evaluate(ctx, "controller start")` line → this fails. +func TestR871_CatchUpWiring(t *testing.T) { + lines, _, _ := slice4CallLines(t) + for _, call := range []string{"ResumeWatch", "SetCatchUpFn", "catchUpLegs", "NotifyBackupCatchUp"} { + if len(lines[call]) == 0 { + t.Errorf("%s is never called in main.go — the catch-up is built but not wired", call) + } + } + // Both triggers, by their reason argument: the start (a literal) and the resume (inside ResumeWatch's callback). + _, f, _ := slice4CallLines(t) + start := 0 + ast.Inspect(f, func(n ast.Node) bool { + if call, ok := n.(*ast.CallExpr); ok { + if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Evaluate" && len(call.Args) == 2 { + if lit, ok := call.Args[1].(*ast.BasicLit); ok && lit.Value == `"controller start"` { + start++ + } + } + } + return true + }) + if start != 1 { + t.Errorf("the catch-up's START trigger (Evaluate(ctx, \"controller start\")) is called %d times, want 1", start) + } +} + +// The catch-up's leg table holds the three BACKUP legs and nothing else, and the off-site body it shares with the +// night never calls the update leg (that is chained only around the SCHEDULED off-site job). +// COMPANION RED-PROOF: put `stackMgr.RunUpdateLeg` inside offsiteOnly's body → "the update leg is inside". +func TestR871_CatchUpRunsNoUpdateLeg(t *testing.T) { + noop := func(context.Context) error { return nil } + legs := catchUpLegs(noop, noop, noop) + if len(legs) != 3 || legs[nightchain.LegDBDump] == nil || legs[nightchain.LegTier2] == nil || legs[nightchain.LegOffsite] == nil { + t.Fatalf("catchUpLegs = %v, want exactly db-dump, tier2, offsite", legs) + } + src, err := os.ReadFile("main.go") + if err != nil { + t.Fatal(err) + } + s := string(src) + i := strings.Index(s, "offsiteOnly := withLeg(nightchain.LegOffsite,") + j := strings.Index(s[i:], "sched.Daily(\"offbox-backup\"") + if i < 0 || j < 0 { + t.Fatal("the shared off-site leg body was not found in main.go") + } + if body := s[i : i+j]; strings.Contains(body, "RunUpdateLeg") || strings.Contains(body, "chainUpdateLeg") { + t.Fatal("the update leg is inside the off-site body the catch-up runs") + } +} diff --git a/controller/internal/i18n/locales/en.json b/controller/internal/i18n/locales/en.json index 4be88fd..f85b932 100644 --- a/controller/internal/i18n/locales/en.json +++ b/controller/internal/i18n/locales/en.json @@ -1374,6 +1374,14 @@ "event.app_update_undone": "The update of %s at %s did not work. The box put back the previous version and its data automatically — nothing was lost, and there is nothing you need to do.", "event.backup_target_absent": "The whole-system backup drive is not available: %s (%s)", "event.backup_target_restored": "The whole-system backup drive is available again: %s (%s)", + "event.backup_catchup_done": "Missed backup made now: the box was off at %s, so the backup ran when it came back on.", + "banner.missed_backup.never": "No daily backup has been made yet.", + "banner.missed_backup.last": "Your last backup was %d day(s) ago.", + "banner.missed_backup.off_at": "The box was off at backup time (%s).", + "banner.missed_backup.choose": "Choose a time when the box is usually on.", + "banner.missed_backup.suggest": "Choose a time when the box is usually on — for example %s.", + "layout.missed_backup_change": "Change the backup time", + "layout.missed_backup_close": "Close", "event.controller_started": "Controller started (%s)", "event.controller_update_failed": "Controller update failed: %s → %s", "event.controller_updated": "Controller updated: %s → %s", diff --git a/controller/internal/i18n/locales/hu.json b/controller/internal/i18n/locales/hu.json index 557c3a2..041f451 100644 --- a/controller/internal/i18n/locales/hu.json +++ b/controller/internal/i18n/locales/hu.json @@ -1365,6 +1365,14 @@ "event.app_update_undone": "A(z) %s frissítése %s-kor nem sikerült. A doboz automatikusan visszaállította az előző változatot és az adatokat — semmi nem veszett el, nincs teendőd.", "event.backup_target_absent": "A rendszermentés meghajtója nem érhető el: %s (%s)", "event.backup_target_restored": "A rendszermentés meghajtója újra elérhető: %s (%s)", + "event.backup_catchup_done": "Kimaradt mentés pótolva: a doboz ki volt kapcsolva %s-kor, a mentés most elkészült.", + "banner.missed_backup.never": "Még nem készült napi mentés.", + "banner.missed_backup.last": "A legutóbbi mentés %d nappal ezelőtt készült.", + "banner.missed_backup.off_at": "A doboz ki volt kapcsolva a mentés idején (%s).", + "banner.missed_backup.choose": "Válassz egy olyan időpontot, amikor a doboz általában be van kapcsolva.", + "banner.missed_backup.suggest": "Válassz egy olyan időpontot, amikor a doboz általában be van kapcsolva — például %s.", + "layout.missed_backup_change": "Mentési idő módosítása", + "layout.missed_backup_close": "Bezárás", "event.controller_started": "Controller elindult (%s)", "event.controller_update_failed": "Controller frissítés sikertelen: %s → %s", "event.controller_updated": "Controller frissítve: %s → %s", diff --git a/controller/internal/metrics/store.go b/controller/internal/metrics/store.go index 178c859..43be00e 100644 --- a/controller/internal/metrics/store.go +++ b/controller/internal/metrics/store.go @@ -329,3 +329,31 @@ func (s *MetricsStore) Prune(olderThan time.Duration) (int64, error) { return total, nil } + +// SampleCount is how many system samples (one a minute) exist in [from, to) — the box's own record of whether it +// was on (R-871 banner: "the box was off at 02:30"). +func (s *MetricsStore) SampleCount(from, to time.Time) (int, error) { + var n int + err := s.db.QueryRow(`SELECT COUNT(*) FROM system_metrics WHERE ts >= ? AND ts < ?`, from.Unix(), to.Unix()).Scan(&n) + return n, err +} + +// HourSampleCounts returns the number of system samples per UTC hour start (unix seconds) in [from, to) — the +// banner's "when is this box usually on" pattern (R-871). At most 24×7 rows for a week. +func (s *MetricsStore) HourSampleCounts(from, to time.Time) (map[int64]int, error) { + rows, err := s.db.Query(`SELECT (ts / 3600) * 3600 AS h, COUNT(*) FROM system_metrics WHERE ts >= ? AND ts < ? GROUP BY h`, from.Unix(), to.Unix()) + if err != nil { + return nil, err + } + defer rows.Close() + out := map[int64]int{} + for rows.Next() { + var h int64 + var n int + if err := rows.Scan(&h, &n); err != nil { + return nil, err + } + out[h] = n + } + return out, rows.Err() +} diff --git a/controller/internal/nightchain/banner.go b/controller/internal/nightchain/banner.go new file mode 100644 index 0000000..2fc28c6 --- /dev/null +++ b/controller/internal/nightchain/banner.go @@ -0,0 +1,125 @@ +package nightchain + +import ( + "fmt" + "time" +) + +// ── The missed-backup banner (R-871, `09` decision 110 — the operator's idea) ────────────────────────────────── +// +// The household's dashboard says it plainly when a daily backup is missing: when the last one was, that the box +// was off at the backup time (when the box's own record says so), and a suggested different time. It NEVER changes +// the time by itself. The household can close it; it stays closed until the NEXT missed backup time, and it goes +// away by itself after a successful night. +// +// "The box's own record of when it was on" is the controller's system-metrics table: one sample a minute, kept 30 +// days (internal/metrics). An hour with at least 30 samples counts as "on" in that hour. +// +// Pinned by banner_test.go (TestBanner_*). + +// StaleAfter: a daily tier with no success for this long shows the banner. 26 h = one night (24 h) plus the chain's +// own two hours, the same line as the hub's backupStaleAfter — so the household and the operator see the same +// fact at the same time. Decided by CC — operator may reverse. +const StaleAfter = 26 * time.Hour + +// suggestMinDays: an hour is "usually on" when the box was on in it on at least this many of the last 7 days. +const suggestMinDays = 5 + +// HourDays counts, per Budapest hour of day, on how many of the last 7 days the box was on in that hour. +type HourDays [24]int + +// BannerInput is everything the banner rule reads. +type BannerInput struct { + Now time.Time + Window string // W, Budapest HH:MM + Loc *time.Location + SeededAt time.Time // the ledger's seed: before it nothing is known + DBDumpOK time.Time // the last successful database dump + OffsiteConfigured bool + OffsiteOK time.Time // the off-site tier's LastSuccess + Dismissed time.Time // the missed W instant the household closed + OnAtW func(t time.Time) (on bool, known bool) // was the box on at t (from the metrics record) + Hours *HourDays // nil = no record +} + +// Banner is what the dashboard shows. +type Banner struct { + Show bool + LastBackup time.Time // zero = never + DaysAgo int + OffAt string // "02:30" when the record says the box was off at the last backup time; "" = unknown/was on + Suggest string // "21:00", or "" (no clear pattern — say only "choose a time when the box is usually on") + MissedAt time.Time // the last backup time that was missed (what a dismissal records) +} + +// ComputeBanner is the PURE rule. +func ComputeBanner(in BannerInput) Banner { + last := in.DBDumpOK + if in.OffsiteConfigured && (in.OffsiteOK.IsZero() || in.OffsiteOK.Before(last)) { + last = in.OffsiteOK + } + known := last + if known.Before(in.SeededAt) { + known = in.SeededAt // nothing before the seed is judged (an upgrade must not show a banner at once) + } + if in.Now.Sub(known) <= StaleAfter { + return Banner{} + } + w, ok := LastInstant(in.Now, in.Window, in.Loc) + if !ok { + return Banner{} + } + if !w.After(in.Dismissed) { + return Banner{} // closed by the household, and no backup time has been missed since + } + b := Banner{Show: true, LastBackup: last, MissedAt: w} + if !last.IsZero() { + b.DaysAgo = int(in.Now.Sub(last).Hours() / 24) + } + if in.OnAtW != nil { + if on, known := in.OnAtW(w); known && !on { + b.OffAt = w.In(in.Loc).Format("15:04") + } + } + if in.Hours != nil { + b.Suggest = Suggest(*in.Hours, w.In(in.Loc).Hour()) + } + return b +} + +// Suggest proposes a window start: the LATEST hour H such that H, H+1 and H+2 were all "usually on" (the chain +// needs about two hours: the dump at W, the second copy at W+1h, the off-site copy at W+1h45m). Latest, because a +// later evening disturbs the household least. "" when there is no such hour, and "" when the CURRENT window's +// span is already usually on (a new time would not help; the miss has another cause). Decided by CC — operator +// may reverse. +func Suggest(h HourDays, currentHour int) string { + usual := func(start int) bool { + for k := 0; k < 3; k++ { + if h[(start+k)%24] < suggestMinDays { + return false + } + } + return true + } + if usual(currentHour) { + return "" + } + for start := 23; start >= 0; start-- { + if usual(start) { + return fmt.Sprintf("%02d:00", start) + } + } + return "" +} + +// HourDaysFrom folds per-UTC-hour sample counts (unix hour start → samples) into "on how many of the days was the +// box on in this Budapest hour": an hour with at least 30 one-minute samples counts as on. +func HourDaysFrom(counts map[int64]int, loc *time.Location) HourDays { + var h HourDays + for start, n := range counts { + if n >= 30 { + h[time.Unix(start, 0).In(loc).Hour()]++ + } + } + return h +} diff --git a/controller/internal/nightchain/banner_test.go b/controller/internal/nightchain/banner_test.go new file mode 100644 index 0000000..8ca0c80 --- /dev/null +++ b/controller/internal/nightchain/banner_test.go @@ -0,0 +1,123 @@ +package nightchain + +import ( + "testing" + "time" +) + +// R-871 / decision 110 — the banner's rules. Red-proofs: catchup-2026-10-05/partB/red-proofs.txt. + +func laptop() *HourDays { // on 18:00–23:59 every day of the week (Tester 2's shape) + var h HourDays + for k := 18; k <= 23; k++ { + h[k] = 7 + } + return &h +} + +func offAt(on bool) func(time.Time) (bool, bool) { + return func(time.Time) (bool, bool) { return on, true } +} + +func base(now time.Time) BannerInput { + return BannerInput{Now: now, Window: "02:30", Loc: bud, SeededAt: at(1, 12, 0), DBDumpOK: at(3, 2, 31), + OnAtW: offAt(false), Hours: laptop()} +} + +// Shown: the last dump is 2+ days old, the box was off at 02:30, and the record suggests 21:00. +// RED-PROOF: return Banner{} when stale (the pre-v0.295.0 silence) → "not shown". +func TestBanner_ShownWithReasonAndSuggestion(t *testing.T) { + b := ComputeBanner(base(at(5, 19, 0))) + if !b.Show { + t.Fatal("not shown although the last backup is 2 days old") + } + if b.OffAt != "02:30" || b.Suggest != "21:00" || b.DaysAgo != 2 { + t.Fatalf("banner = %+v, want off at 02:30, suggest 21:00, 2 days ago", b) + } +} + +// Not shown: a backup within the last 26 h. +func TestBanner_NotShownWhenFresh(t *testing.T) { + in := base(at(5, 19, 0)) + in.DBDumpOK = at(5, 2, 31) + if b := ComputeBanner(in); b.Show { + t.Fatalf("shown although the last dump is 16 h old: %+v", b) + } +} + +// Not shown right after an upgrade (the ledger was seeded less than 26 h ago and knows no dump). +func TestBanner_NotShownBeforeTheRecordKnows(t *testing.T) { + in := base(at(5, 19, 0)) + in.SeededAt, in.DBDumpOK = at(5, 9, 0), time.Time{} + if b := ComputeBanner(in); b.Show { + t.Fatalf("shown on the day of the upgrade: %+v", b) + } +} + +// Dismissed: closed until the NEXT missed backup time, then back. +// RED-PROOF: drop the Dismissed check → "a closed banner came back without a new miss". +func TestBanner_DismissedThenBackAfterANewMiss(t *testing.T) { + in := base(at(5, 19, 0)) + b := ComputeBanner(in) + in.Dismissed = b.MissedAt // the household closed it (02:30 on the 5th) + if ComputeBanner(in).Show { + t.Fatal("a closed banner came back without a new miss") + } + in.Now = at(5, 23, 0) // same evening — still the same miss + if ComputeBanner(in).Show { + t.Fatal("a closed banner came back the same evening") + } + in.Now = at(6, 19, 0) // off again at 02:30 on the 6th + if !ComputeBanner(in).Show { + t.Fatal("the banner did not come back after the next missed backup time") + } +} + +// Gone by itself after a successful night (e.g. the catch-up ran). +func TestBanner_GoneAfterASuccess(t *testing.T) { + in := base(at(5, 19, 0)) + in.DBDumpOK = at(5, 18, 20) // the catch-up's dump + if ComputeBanner(in).Show { + t.Fatal("still shown after a successful backup") + } +} + +// The off-site tier counts when configured: a fresh dump with a 3-day-old off-site copy still shows. +func TestBanner_OffsiteTierCounts(t *testing.T) { + in := base(at(5, 19, 0)) + in.DBDumpOK = at(5, 18, 20) + in.OffsiteConfigured, in.OffsiteOK = true, at(2, 4, 20) + b := ComputeBanner(in) + if !b.Show || b.DaysAgo != 3 { + t.Fatalf("banner = %+v, want shown with the off-site copy's 3 days", b) + } +} + +// No clear pattern → no suggestion ("choose a time when the box is usually on"); box ON at W → no "off at". +// RED-PROOF: let Suggest return the first start without checking the days → a time is suggested from noise. +func TestBanner_NoPatternNoSuggestion(t *testing.T) { + in := base(at(5, 19, 0)) + var noisy HourDays + for k := 18; k <= 23; k++ { + noisy[k] = 2 // on some evenings — 2 of 7 days is not "usually" + } + in.Hours, in.OnAtW = &noisy, offAt(true) + b := ComputeBanner(in) + if !b.Show || b.Suggest != "" || b.OffAt != "" { + t.Fatalf("banner = %+v, want shown, no suggestion, no 'off at'", b) + } +} + +// A box usually ON at its window gets no new time (the miss has another cause). +func TestBanner_UsuallyOnAtWNoSuggestion(t *testing.T) { + var always HourDays + for k := range always { + always[k] = 7 + } + if s := Suggest(always, 2); s != "" { + t.Fatalf("suggested %q for a box that is on at its window", s) + } + if s := Suggest(*laptop(), 2); s != "21:00" { + t.Fatalf("laptop suggestion %q, want 21:00", s) + } +} diff --git a/controller/internal/nightchain/nightchain.go b/controller/internal/nightchain/nightchain.go new file mode 100644 index 0000000..b2da325 --- /dev/null +++ b/controller/internal/nightchain/nightchain.go @@ -0,0 +1,365 @@ +// Package nightchain makes a missed night's backups up once, when the box comes back (R-871, `09` §3 decision 109, +// design `07` §6.1.1). +// +// THE PROBLEM, MEASURED (2026-10-05 Part F spike, Tester 2 — a laptop switched off at night): the controller's +// daily jobs always schedule the NEXT future time (scheduler.nextDailyRun), and nothing remembers that a night was +// missed. A box that is off at its window W never gets its database dumps, its second copy or its off-site copy — +// for ever, with no alarm. +// +// THE MECHANISM: +// - A LEDGER (`night-ledger.json` in the data dir) records when each of the three nightly backup legs last RAN TO +// ITS END. It is an attempt record and is used for ONE question only — "was this leg's last scheduled time +// missed?" — never as evidence that a backup exists (that is LastSuccess's job, R-100). A leg that ran and +// failed was NOT missed: failures have their own alarms. +// - On a controller START and on a host RESUME (a suspended laptop: Go timers run on CLOCK_MONOTONIC, which does +// not count suspended time), Evaluate asks the ledger which legs missed their last scheduled time. If any did, +// ONE catch-up is scheduled 15 minutes later (apps settle; a box switched on and off again at once does nothing). +// - The catch-up runs ONLY the backup legs, in their normal order (database dump, second copy, off-site copy). +// It never runs app updates or Docker steps — they restart apps and wait for a real night. +// - Several missed nights are ONE catch-up (the question is about the LAST scheduled time only). A normal night +// followed by a daytime restart is none. A power cut in the middle of the chain makes up only the legs that did +// not end. +// - A leg whose own scheduled time is less than 30 minutes away is left to its normal run. +// - Every leg — scheduled or catch-up — holds one mutex (`Ledger.LegLock`), so a catch-up and a scheduled leg +// never run at the same time. The whole-guest backup (agent-side, quiesce) waits while a catch-up runs +// (quiesce.Options.CatchUpFn) and the catch-up waits while a quiesce holds the apps (QuiesceBusy). +// +// A ledger that does not exist yet (the first start on this release) is SEEDED at that moment and nothing before +// it counts as missed — an upgrade must not set off a surprise catch-up on every box. +// +// Pinned by nightchain_test.go (TestCatchUp_*), each rule with a red-proof. +package nightchain + +import ( + "context" + "encoding/json" + "fmt" + "log" + "os" + "path/filepath" + "sort" + "strings" + "sync" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/backupwindow" +) + +// Leg is one nightly backup leg. +type Leg string + +const ( + LegDBDump Leg = "db-dump" + LegTier2 Leg = "tier2" + LegOffsite Leg = "offsite" +) + +// Order is the night chain's own order (07 §6.1). +var Order = []Leg{LegDBDump, LegTier2, LegOffsite} + +const ( + // DefaultDelay: the catch-up waits this long after the trigger (decision 109's "soon after it is switched on"). + DefaultDelay = 15 * time.Minute + // leaveToNormal: a leg whose own scheduled time is closer than this runs normally, not in the catch-up. + leaveToNormal = 30 * time.Minute + // quiesceWaitMax: how long a catch-up waits for a whole-guest backup that holds the apps. + quiesceWaitMax = 2 * time.Hour +) + +type ledgerData struct { + SeededAt time.Time `json:"seeded_at"` + Ended map[Leg]time.Time `json:"ended"` // ATTEMPT record: the leg ran to its end (success or failure) + // DBDumpOK is the last SUCCESSFUL database dump (the banner's evidence; the off-site tier has its own + // LastSuccess in settings). Written only on success (R-100's rule). + DBDumpOK time.Time `json:"db_dump_ok,omitempty"` + LastCatchUp time.Time `json:"last_catch_up,omitempty"` + BannerDismissed time.Time `json:"banner_dismissed_through,omitempty"` // the missed W instant the household closed +} + +// Ledger is the persisted night record. +type Ledger struct { + mu sync.Mutex + path string + d ledgerData + // LegLock is held by every leg run, scheduled or catch-up. + LegLock sync.Mutex +} + +// Open reads the ledger, or seeds a new one at now. +func Open(path string, now time.Time) (*Ledger, error) { + l := &Ledger{path: path, d: ledgerData{Ended: map[Leg]time.Time{}}} + b, err := os.ReadFile(path) + switch { + case err == nil: + if jerr := json.Unmarshal(b, &l.d); jerr != nil { + return nil, fmt.Errorf("nightchain: ledger unreadable: %w", jerr) + } + if l.d.Ended == nil { + l.d.Ended = map[Leg]time.Time{} + } + case os.IsNotExist(err): + l.d.SeededAt = now.UTC() + if serr := l.saveLocked(); serr != nil { + return nil, serr + } + default: + return nil, fmt.Errorf("nightchain: ledger: %w", err) + } + return l, nil +} + +func (l *Ledger) saveLocked() error { + b, _ := json.MarshalIndent(l.d, "", " ") + if err := os.MkdirAll(filepath.Dir(l.path), 0o755); err != nil { + return err + } + tmp := l.path + ".tmp" + if err := os.WriteFile(tmp, b, 0o644); err != nil { + return err + } + return os.Rename(tmp, l.path) +} + +// MarkEnded records that a leg ran to its end (success or failure). +func (l *Ledger) MarkEnded(leg Leg, at time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + l.d.Ended[leg] = at.UTC() + _ = l.saveLocked() +} + +// MarkDBDumpOK records a successful database dump. +func (l *Ledger) MarkDBDumpOK(at time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + l.d.DBDumpOK = at.UTC() + _ = l.saveLocked() +} + +func (l *Ledger) markCatchUp(at time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + l.d.LastCatchUp = at.UTC() + _ = l.saveLocked() +} + +// DismissBanner records that the household closed the banner for every miss up to `through`. +func (l *Ledger) DismissBanner(through time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + l.d.BannerDismissed = through.UTC() + _ = l.saveLocked() +} + +// Snapshot returns a copy of the record (for the banner and the debug page). +func (l *Ledger) Snapshot() (seeded, dbDumpOK, lastCatchUp, dismissed time.Time, ended map[Leg]time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + ended = map[Leg]time.Time{} + for k, v := range l.d.Ended { + ended[k] = v + } + return l.d.SeededAt, l.d.DBDumpOK, l.d.LastCatchUp, l.d.BannerDismissed, ended +} + +// legOffsets: each leg's time relative to W — taken from backupwindow.LegTimes so the two can never disagree. +func legTimes(window string) map[Leg]string { + db, t2, off := backupwindow.LegTimes(window) + return map[Leg]string{LegDBDump: db, LegTier2: t2, LegOffsite: off} +} + +// LastInstant is the most recent occurrence of the Budapest clock time hhmm at or before now. +func LastInstant(now time.Time, hhmm string, loc *time.Location) (time.Time, bool) { + min, err := backupwindow.ParseHHMM(hhmm) + if err != nil { + return time.Time{}, false + } + n := now.In(loc) + t := time.Date(n.Year(), n.Month(), n.Day(), min/60, min%60, 0, 0, loc) + if t.After(n) { + t = time.Date(n.Year(), n.Month(), n.Day()-1, min/60, min%60, 0, 0, loc) + } + return t, true +} + +// NextInstant is the next occurrence of hhmm strictly after now. +func NextInstant(now time.Time, hhmm string, loc *time.Location) (time.Time, bool) { + last, ok := LastInstant(now, hhmm, loc) + if !ok { + return time.Time{}, false + } + return time.Date(last.Year(), last.Month(), last.Day()+1, last.Hour(), last.Minute(), 0, 0, loc), true +} + +// Missed is the PURE rule: the legs (in chain order) whose last scheduled time passed after the ledger was seeded +// and that have not run to their end since — minus a leg whose next scheduled time is under 30 minutes away. +// lastW is the most recent missed leg instant (the banner's "the box was off at …"). +func (l *Ledger) Missed(now time.Time, window string, loc *time.Location) (legs []Leg, lastW time.Time) { + l.mu.Lock() + defer l.mu.Unlock() + lt := legTimes(window) + for _, leg := range Order { + inst, ok := LastInstant(now, lt[leg], loc) + if !ok || !inst.After(l.d.SeededAt) { + continue + } + if !l.d.Ended[leg].Before(inst) { + continue // ran to its end at or after its last scheduled time + } + if next, ok := NextInstant(now, lt[leg], loc); ok && next.Sub(now) < leaveToNormal { + continue // its normal run is about to happen + } + legs = append(legs, leg) + if inst.After(lastW) { + lastW = inst + } + } + return legs, lastW +} + +// CatchUp runs at most one catch-up at a time. +type CatchUp struct { + Ledger *Ledger + Window func() string // the household's backup window W (settings > config > 02:30) + Legs map[Leg]func(context.Context) error // the leg bodies — WITHOUT the update leg + QuiesceBusy func() bool // a whole-guest backup holds the apps now + Event func(missedAt time.Time) // the household's timeline line + Logger *log.Logger + Delay time.Duration + Loc *time.Location + Now func() time.Time + // Sleep waits d or until ctx ends (false). A seam for tests. + Sleep func(ctx context.Context, d time.Duration) bool + + mu sync.Mutex + pending bool + running bool +} + +func (c *CatchUp) now() time.Time { + if c.Now != nil { + return c.Now() + } + return time.Now() +} + +func (c *CatchUp) sleep(ctx context.Context, d time.Duration) bool { + if c.Sleep != nil { + return c.Sleep(ctx, d) + } + t := time.NewTimer(d) + defer t.Stop() + select { + case <-ctx.Done(): + return false + case <-t.C: + return true + } +} + +func (c *CatchUp) delay() time.Duration { + if c.Delay > 0 { + return c.Delay + } + return DefaultDelay +} + +// Running reports whether a catch-up is running its legs now (the whole-guest backup waits for it). +func (c *CatchUp) Running() bool { + c.mu.Lock() + defer c.mu.Unlock() + return c.running +} + +// Evaluate is the trigger (controller start, host resume). It schedules ONE catch-up when a leg was missed and +// none is pending; it returns what it scheduled (nil = nothing missed or one already pending). The work runs in +// its own goroutine; Wait-style callers use EvaluateSync. +func (c *CatchUp) Evaluate(ctx context.Context, why string) []Leg { + legs, lastW := c.Ledger.Missed(c.now(), c.Window(), c.Loc) + if len(legs) == 0 { + c.Logger.Printf("[INFO] [catch-up] %s: no backup leg missed its last scheduled time — nothing to make up", why) + return nil + } + c.mu.Lock() + if c.pending || c.running { + c.mu.Unlock() + c.Logger.Printf("[INFO] [catch-up] %s: missed %v, but a catch-up is already scheduled", why, legs) + return nil + } + c.pending = true + c.mu.Unlock() + c.Logger.Printf("[INFO] [catch-up] %s: the box missed %v (last scheduled %s) — ONE catch-up in %s (backup legs only; app updates wait for a real night)", + why, legs, lastW.In(c.Loc).Format("2006-01-02 15:04"), c.delay()) + go c.run(ctx, why) + return legs +} + +// run waits, re-reads the ledger (a normal run may have happened meanwhile), waits out a whole-guest backup, then +// runs the still-missed legs in chain order. +func (c *CatchUp) run(ctx context.Context, why string) { + defer func() { + c.mu.Lock() + c.pending, c.running = false, false + c.mu.Unlock() + }() + if !c.sleep(ctx, c.delay()) { + return + } + legs, lastW := c.Ledger.Missed(c.now(), c.Window(), c.Loc) + if len(legs) == 0 { + c.Logger.Printf("[INFO] [catch-up] (%s) nothing is missed any more — not run", why) + return + } + for waited := time.Duration(0); c.QuiesceBusy != nil && c.QuiesceBusy(); waited += time.Minute { + if waited >= quiesceWaitMax { + c.Logger.Printf("[WARN] [catch-up] (%s) a whole-guest backup held the apps for %s — catch-up dropped; the next start or resume tries again", why, waited) + return + } + if !c.sleep(ctx, time.Minute) { + return + } + } + c.mu.Lock() + c.running = true + c.mu.Unlock() + start := c.now() + var names []string + for _, leg := range legs { + fn := c.Legs[leg] + if fn == nil { + continue + } + c.Logger.Printf("[INFO] [catch-up] running the missed %s leg", leg) + if err := fn(ctx); err != nil { + c.Logger.Printf("[WARN] [catch-up] the %s leg ended with an error (its own alarm reports it): %v", leg, err) + } + names = append(names, string(leg)) + } + c.Ledger.markCatchUp(c.now()) + sort.Strings(names) + c.Logger.Printf("[INFO] [catch-up] done: %s in %s (missed at %s)", strings.Join(names, ", "), + c.now().Sub(start).Round(time.Second), lastW.In(c.Loc).Format("2006-01-02 15:04")) + if c.Event != nil { + c.Event(lastW) + } +} + +// ResumeWatch calls onResume when, between two ticks, the WALL clock advanced more than gap beyond the MONOTONIC +// clock — the shape of a host suspend/resume: Go's timers and monotonic readings run on CLOCK_MONOTONIC, which does +// not count suspended time, while the wall clock does. Seams: wall() is the wall clock, mono() a monotonic elapsed +// duration (production: time.Now().Round(0) and time.Since(a fixed start)). Pinned by TestResumeWatch_*. +func ResumeWatch(ctx context.Context, tick <-chan time.Time, wall func() time.Time, mono func() time.Duration, gap time.Duration, onResume func(slept time.Duration)) { + pw, pm := wall(), mono() + for { + select { + case <-ctx.Done(): + return + case <-tick: + } + cw, cm := wall(), mono() + if d := cw.Sub(pw) - (cm - pm); d > gap { + onResume(d) + } + pw, pm = cw, cm + } +} diff --git a/controller/internal/nightchain/nightchain_test.go b/controller/internal/nightchain/nightchain_test.go new file mode 100644 index 0000000..30ada6c --- /dev/null +++ b/controller/internal/nightchain/nightchain_test.go @@ -0,0 +1,248 @@ +package nightchain + +import ( + "context" + "io" + "log" + "path/filepath" + "sync" + "testing" + "time" +) + +// R-871 (`09` decision 109). Every rule of the catch-up, each with a COMPANION RED-PROOF recorded in +// felhom.eu/documentation/audits/catchup-2026-10-05/partA/red-proofs.txt. + +var bud = func() *time.Location { l, _ := time.LoadLocation("Europe/Budapest"); return l }() + +func at(day, hh, mm int) time.Time { return time.Date(2026, 10, day, hh, mm, 0, 0, bud) } + +type harness struct { + l *Ledger + c *CatchUp + mu sync.Mutex + ran []Leg + events []time.Time + slept []time.Duration + now time.Time + busy int // QuiesceBusy answers true this many times + done chan struct{} +} + +func newHarness(t *testing.T, seeded, now time.Time) *harness { + t.Helper() + l, err := Open(filepath.Join(t.TempDir(), "night-ledger.json"), seeded) + if err != nil { + t.Fatal(err) + } + h := &harness{l: l, now: now, done: make(chan struct{}, 4)} + leg := func(g Leg) func(context.Context) error { + return func(context.Context) error { + h.mu.Lock() + h.ran = append(h.ran, g) + h.mu.Unlock() + l.MarkEnded(g, h.clock()) + return nil + } + } + h.c = &CatchUp{ + Ledger: l, Window: func() string { return "02:30" }, Loc: bud, + Legs: map[Leg]func(context.Context) error{LegDBDump: leg(LegDBDump), LegTier2: leg(LegTier2), LegOffsite: leg(LegOffsite)}, + Logger: log.New(io.Discard, "", 0), + Now: h.clock, + Sleep: func(_ context.Context, d time.Duration) bool { + h.mu.Lock() + h.slept = append(h.slept, d) + h.now = h.now.Add(d) + h.mu.Unlock() + return true + }, + QuiesceBusy: func() bool { + h.mu.Lock() + defer h.mu.Unlock() + if h.busy > 0 { + h.busy-- + return true + } + return false + }, + Event: func(w time.Time) { + h.mu.Lock() + h.events = append(h.events, w) + h.mu.Unlock() + h.done <- struct{}{} + }, + } + return h +} + +func (h *harness) clock() time.Time { h.mu.Lock(); defer h.mu.Unlock(); return h.now } + +func (h *harness) wait(t *testing.T) { + t.Helper() + select { + case <-h.done: + case <-time.After(3 * time.Second): + t.Fatal("the catch-up never finished") + } +} + +// Off at W (02:30 on the 5th), switched on at 09:00: ONE catch-up, 15 minutes later, all three legs in chain order. +// RED-PROOF: make Missed return nothing (the pre-v0.295.0 behaviour) → "no catch-up was scheduled". +func TestCatchUp_OffAtWThenOn_OneCatchUpAfter15Min(t *testing.T) { + h := newHarness(t, at(1, 12, 0), at(5, 9, 0)) + h.l.MarkEnded(LegDBDump, at(4, 2, 31)) // the night of the 4th ran + h.l.MarkEnded(LegTier2, at(4, 3, 31)) + h.l.MarkEnded(LegOffsite, at(4, 4, 20)) + legs := h.c.Evaluate(context.Background(), "start") + if len(legs) != 3 { + t.Fatalf("no catch-up was scheduled for a box off across W (missed %v)", legs) + } + h.wait(t) + if len(h.slept) == 0 || h.slept[0] != DefaultDelay { + t.Fatalf("the catch-up must wait %s first, slept %v", DefaultDelay, h.slept) + } + if got := []Leg{LegDBDump, LegTier2, LegOffsite}; len(h.ran) != 3 || h.ran[0] != got[0] || h.ran[1] != got[1] || h.ran[2] != got[2] { + t.Fatalf("legs ran %v, want the chain order %v", h.ran, got) + } + if len(h.events) != 1 || !h.events[0].Equal(at(5, 4, 15)) { + t.Fatalf("household line %v, want one for the missed 04:15 leg", h.events) + } +} + +// On at W (the night ran) then a daytime restart: no catch-up. +// RED-PROOF: drop the `!Ended.Before(inst)` check → the restart makes up a night that ran. +func TestCatchUp_NightRan_DaytimeRestart_None(t *testing.T) { + h := newHarness(t, at(1, 12, 0), at(5, 11, 0)) + h.l.MarkEnded(LegDBDump, at(5, 2, 31)) + h.l.MarkEnded(LegTier2, at(5, 3, 31)) + h.l.MarkEnded(LegOffsite, at(5, 4, 20)) + if legs := h.c.Evaluate(context.Background(), "start"); legs != nil { + t.Fatalf("a normal night was made up again: %v", legs) + } +} + +// Two missed nights: ONE catch-up, each leg once. +func TestCatchUp_TwoMissedNights_One(t *testing.T) { + h := newHarness(t, at(1, 12, 0), at(6, 18, 0)) + h.l.MarkEnded(LegDBDump, at(4, 2, 31)) + h.l.MarkEnded(LegTier2, at(4, 3, 31)) + h.l.MarkEnded(LegOffsite, at(4, 4, 20)) + h.c.Evaluate(context.Background(), "start") + if again := h.c.Evaluate(context.Background(), "resume"); again != nil { + t.Fatalf("a second catch-up was scheduled while one was pending: %v", again) + } + h.wait(t) + if len(h.ran) != 3 { + t.Fatalf("two missed nights ran %v — want each leg once", h.ran) + } + if legs := h.c.Evaluate(context.Background(), "start"); legs != nil { + t.Fatalf("after the catch-up nothing is missed any more, got %v", legs) + } +} + +// A power cut in the middle of the chain: only the legs that did not end. +func TestCatchUp_PowerCutMidChain_FinishesTheRest(t *testing.T) { + h := newHarness(t, at(1, 12, 0), at(5, 10, 0)) + h.l.MarkEnded(LegDBDump, at(5, 2, 31)) // the dump ended, then the power went + h.l.MarkEnded(LegTier2, at(4, 3, 31)) + h.l.MarkEnded(LegOffsite, at(4, 4, 20)) + h.c.Evaluate(context.Background(), "start") + h.wait(t) + if len(h.ran) != 2 || h.ran[0] != LegTier2 || h.ran[1] != LegOffsite { + t.Fatalf("ran %v, want only tier2 + offsite", h.ran) + } +} + +// The first start on this release seeds the ledger: nothing before it is "missed" (no surprise catch-up on upgrade). +func TestCatchUp_FreshLedger_None(t *testing.T) { + h := newHarness(t, at(5, 9, 0), at(5, 9, 0)) + if legs := h.c.Evaluate(context.Background(), "start"); legs != nil { + t.Fatalf("a freshly seeded ledger made up %v", legs) + } +} + +// Started at 02:10 (the dump is due at 02:30): the dump is left to its normal run. +func TestCatchUp_LegAboutToRun_LeftToNormal(t *testing.T) { + h := newHarness(t, at(1, 12, 0), at(5, 2, 10)) + h.l.MarkEnded(LegDBDump, at(3, 2, 31)) + h.l.MarkEnded(LegTier2, at(3, 3, 31)) + h.l.MarkEnded(LegOffsite, at(3, 4, 20)) + legs, _ := h.l.Missed(at(5, 2, 10), "02:30", bud) + for _, l := range legs { + if l == LegDBDump { + t.Fatalf("the dump due in 20 minutes was put in the catch-up: %v", legs) + } + } + if len(legs) != 2 { + t.Fatalf("missed %v, want tier2 + offsite of the night of the 4th", legs) + } +} + +// App updates are never in a catch-up: only the chain's backup legs are run, whatever else Legs holds. +// RED-PROOF: iterate c.Legs instead of the missed chain legs → "update" runs. +func TestCatchUp_NeverRunsAnythingButBackupLegs(t *testing.T) { + h := newHarness(t, at(1, 12, 0), at(5, 9, 0)) + updateRan := false + h.c.Legs["update"] = func(context.Context) error { updateRan = true; return nil } + h.c.Evaluate(context.Background(), "start") + h.wait(t) + if updateRan { + t.Fatal("an app update ran inside a catch-up") + } + for _, l := range Order { + if l == "update" || l == "docker" { + t.Fatalf("the chain order names a non-backup leg %q", l) + } + } +} + +// A whole-guest backup holding the apps: the catch-up waits for it. +// RED-PROOF: drop the QuiesceBusy loop → the legs run while busy (ran before the waits). +func TestCatchUp_WaitsForAWholeGuestBackup(t *testing.T) { + h := newHarness(t, at(1, 12, 0), at(5, 9, 0)) + h.busy = 3 + h.c.Evaluate(context.Background(), "start") + h.wait(t) + if len(h.slept) != 4 || h.slept[1] != time.Minute { + t.Fatalf("slept %v — want the 15 min delay then 3 one-minute waits for the quiesce", h.slept) + } + if len(h.ran) != 3 { + t.Fatalf("ran %v", h.ran) + } +} + +// A host resume is a trigger too: the wall clock jumps ahead of the monotonic clock. +// RED-PROOF: compare wall with wall → the suspend is never seen. +func TestResumeWatch_SeesASuspend(t *testing.T) { + wall := at(5, 1, 0) + var mono time.Duration + tick := make(chan time.Time) + got := make(chan time.Duration, 2) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + var mu sync.Mutex + go ResumeWatch(ctx, tick, func() time.Time { mu.Lock(); defer mu.Unlock(); return wall }, + func() time.Duration { mu.Lock(); defer mu.Unlock(); return mono }, 5*time.Minute, func(d time.Duration) { got <- d }) + step := func(w, m time.Duration) { + mu.Lock() + wall, mono = wall.Add(w), mono+m + mu.Unlock() + tick <- time.Time{} + } + step(time.Minute, time.Minute) // a normal minute + step(7*time.Hour, time.Minute) // suspended 02:00 → 09:00: wall +7h, monotonic +1 min + select { + case d := <-got: + if d < 6*time.Hour { + t.Fatalf("slept %s, want ~7h", d) + } + case <-time.After(2 * time.Second): + t.Fatal("a 7-hour suspend was not seen") + } + select { + case d := <-got: + t.Fatalf("a normal minute was read as a resume (%s)", d) + default: + } +} diff --git a/controller/internal/notify/notifier.go b/controller/internal/notify/notifier.go index 160e211..ac7ecd1 100644 --- a/controller/internal/notify/notifier.go +++ b/controller/internal/notify/notifier.go @@ -541,6 +541,12 @@ func (n *Notifier) NotifyControllerUpdated(fromVer, toVer string, success bool) n.pushEventMsg("controller_updated", severity, key, details, fromVer, toVer) } +// NotifyBackupCatchUp (R-871, v0.295.0) — the household's timeline line after a catch-up made a missed night's +// backups: "the box was off at ; the missed backup is made now". Info: recorded, never mailed. +func (n *Notifier) NotifyBackupCatchUp(missedAt string) { + n.pushEventMsg("backup_catchup_done", "info", "event.backup_catchup_done", map[string]string{"missed_at": missedAt}, missedAt) +} + // NotifyControllerStarted sends a controller startup event. // details may include self-test summary (e.g., {"selftest_pass": 8, "selftest_warn": 1, "selftest_fail": 0}). func (n *Notifier) NotifyControllerStarted(version string, details map[string]interface{}) { diff --git a/controller/internal/quiesce/quiesce.go b/controller/internal/quiesce/quiesce.go index 45a3e65..7ab0504 100644 --- a/controller/internal/quiesce/quiesce.go +++ b/controller/internal/quiesce/quiesce.go @@ -100,6 +100,9 @@ type Loop struct { windowStartFn func() string cadence time.Duration updateLegFn func() (active, stepRunning bool) + // catchUpMu/catchUpFn (R-871, v0.295.0): a running catch-up (internal/nightchain) holds the scheduled cycle. + catchUpMu sync.Mutex + catchUpFn func() bool // mu single-flights the quiesce cycle across the scheduled loop AND the manual trigger, so the // two can never stop the same stacks concurrently (the persisted marker covers crash-safety across // restarts; this covers concurrency within the process — which a manual trigger introduces). @@ -235,6 +238,15 @@ func (l *Loop) runOnce(ctx context.Context) error { return nil } + // R-871 (v0.295.0): a catch-up of a missed night is making its database dumps and copies right now — the + // whole-guest backup would stop the apps under it. SCHEDULED path only (TriggerNow is the household's press); + // the catch-up waits for a quiesce the same way (nightchain.CatchUp.QuiesceBusy), so neither starves: the + // catch-up is minutes, and this is re-asked at the next poll. Pinned by TestR871_ScheduledCycleWaitsForCatchUp. + if l.catchUpRunning() { + l.logger.Printf("[INFO] [quiesce] full-system backup due, but a catch-up of a missed night is running — deferring to the next poll (R-871)") + return nil + } + // Window gate (Part 3) — SCHEDULED path only. TriggerNow calls quiesceAndPoll directly and is // never gated. Disabled when no window fn is wired (pre-v0.168.0 behavior). // @@ -842,3 +854,17 @@ func updateLegDefers(fn func() (bool, bool), now time.Time, windowStart string) // legStepGraceMin bounds how long past W+5h the gate waits for a step already in flight. const legStepGraceMin = 30 + +// SetCatchUpFn wires the catch-up's "running" state (R-871). Safe to call after Run has started. +func (l *Loop) SetCatchUpFn(fn func() bool) { + l.catchUpMu.Lock() + defer l.catchUpMu.Unlock() + l.catchUpFn = fn +} + +func (l *Loop) catchUpRunning() bool { + l.catchUpMu.Lock() + fn := l.catchUpFn + l.catchUpMu.Unlock() + return fn != nil && fn() +} diff --git a/controller/internal/quiesce/r871_catchup_test.go b/controller/internal/quiesce/r871_catchup_test.go new file mode 100644 index 0000000..d286418 --- /dev/null +++ b/controller/internal/quiesce/r871_catchup_test.go @@ -0,0 +1,30 @@ +package quiesce + +import ( + "context" + "testing" +) + +// R-871 (v0.295.0): a scheduled whole-guest cycle waits while a catch-up of a missed night runs (it would stop the +// apps under the catch-up's database dumps), and runs at the next poll once the catch-up has ended. +// COMPANION RED-PROOF: drop the catchUpRunning() check in runOnce → "the backup started while a catch-up ran". +func TestR871_ScheduledCycleWaitsForCatchUp(t *testing.T) { + be := &fakeBackend{due: true, dueAge: i64(20 * 3600), phases: []string{"done"}} + st := &fakeStacks{running: []string{"nextcloud"}} + l := windowLoop(t, be, st, "02:30", atBudapest(5, 0)) // inside [W+2h, W+6h) + running := true + l.SetCatchUpFn(func() bool { return running }) + if err := l.runOnce(context.Background()); err != nil { + t.Fatalf("runOnce: %v", err) + } + if be.startCalls != 0 || len(st.stoppedNames()) != 0 { + t.Fatalf("the backup started while a catch-up ran: start=%d stopped=%v", be.startCalls, st.stoppedNames()) + } + running = false + if err := l.runOnce(context.Background()); err != nil { + t.Fatalf("runOnce: %v", err) + } + if be.startCalls != 1 { + t.Fatalf("after the catch-up the backup must run at the next poll; start=%d", be.startCalls) + } +} diff --git a/controller/internal/scheduler/scheduler.go b/controller/internal/scheduler/scheduler.go index 34faf7c..ed34233 100644 --- a/controller/internal/scheduler/scheduler.go +++ b/controller/internal/scheduler/scheduler.go @@ -53,6 +53,23 @@ type Scheduler struct { cancel context.CancelFunc wg sync.WaitGroup started bool + // nowFn / afterFn are seams for the late-fire guard's tests (nil = time.Now / time.After). + nowFn func() time.Time + afterFn func(d time.Duration) <-chan time.Time +} + +// DailyLateLimit (R-871, v0.295.0): a daily job whose timer fires more than this after its scheduled wall-clock +// time is SKIPPED, not run late. Go's timers run on CLOCK_MONOTONIC, which does not count a host suspend, so on a +// laptop that slept through the night the 02:30 timer fires hours late — and would start the app-update leg at +// noon while the household uses the box. A skipped backup leg is made up by the catch-up (internal/nightchain), +// which the resume watch triggers; the update leg waits for a real night. Pinned by TestDaily_LateFireIsSkipped. +const DailyLateLimit = 60 * time.Minute + +func (s *Scheduler) now() time.Time { + if s.nowFn != nil { + return s.nowFn() + } + return time.Now() } // SetDebug enables or disables debug logging. @@ -289,8 +306,8 @@ func (s *Scheduler) runDailyJob(job *Job) { schedule := job.Schedule s.mu.Unlock() - nextRun := nextDailyRun(schedule) - waitDuration := time.Until(nextRun) + nextRun := nextDailyRunAt(schedule, s.now()) + waitDuration := nextRun.Sub(s.now()) if waitDuration < 0 { waitDuration = 0 @@ -298,18 +315,36 @@ func (s *Scheduler) runDailyJob(job *Job) { s.dbg("daily job %s: next run at %s (waiting %s)", job.Name, nextRun.Format("2006-01-02 15:04:05 MST"), waitDuration.Round(time.Second)) - timer := time.NewTimer(waitDuration) + var fire <-chan time.Time + var timer *time.Timer + if s.afterFn != nil { + fire = s.afterFn(waitDuration) + } else { + timer = time.NewTimer(waitDuration) + fire = timer.C + } + stop := func() { + if timer != nil { + timer.Stop() + } + } select { case <-s.ctx.Done(): - timer.Stop() + stop() s.dbg("daily job %s: context cancelled, stopping", job.Name) return case <-job.resched: // Runtime reschedule: abandon the current timer and recompute against the new Schedule. - timer.Stop() + stop() s.dbg("daily job %s: rescheduled — recomputing next run", job.Name) continue - case <-timer.C: + case <-fire: + // The wall clock, not the timer, decides: a host suspend stops the timer's clock (R-871). + if late := s.now().Round(0).Sub(nextRun); late > DailyLateLimit { + s.logger.Printf("[INFO] [scheduler] Daily job %s SKIPPED: its timer fired %s after %s (the host was suspended) — not run late; a missed backup leg is made up by the catch-up, everything else waits for its next time", + job.Name, late.Round(time.Minute), nextRun.Format("2006-01-02 15:04 MST")) + continue + } s.executeJob(job, false) } } @@ -387,6 +422,11 @@ func NextDailyRun(timeStr string) time.Time { // nextDailyRun calculates the next occurrence of the daily schedule in Europe/Budapest timezone. func nextDailyRun(timeStr string) time.Time { + return nextDailyRunAt(timeStr, time.Now()) +} + +// nextDailyRunAt is nextDailyRun for a given "now" (the scheduler's clock seam). +func nextDailyRunAt(timeStr string, at time.Time) time.Time { hour, min, err := parseDailyTime(timeStr) if err != nil { // Should not happen — validated at registration @@ -395,7 +435,7 @@ func nextDailyRun(timeStr string) time.Time { loc := getBudapestLocation() - now := time.Now().In(loc) + now := at.In(loc) next := time.Date(now.Year(), now.Month(), now.Day(), hour, min, 0, 0, loc) // If the time has already passed today, schedule for tomorrow. diff --git a/controller/internal/scheduler/scheduler_test.go b/controller/internal/scheduler/scheduler_test.go index 705fd98..751e0e2 100644 --- a/controller/internal/scheduler/scheduler_test.go +++ b/controller/internal/scheduler/scheduler_test.go @@ -4,6 +4,7 @@ import ( "context" "io" "log" + "sync" "testing" "time" ) @@ -79,3 +80,60 @@ func TestUpdateDaily_GoroutineConsumesReschedule(t *testing.T) { // drained by the goroutine → immediacy works } } + +// R-871 (v0.295.0): a daily timer that fires long after its wall-clock time (a host suspend: the timer's clock stops) +// SKIPS the job; one that fires on time runs it. +// COMPANION RED-PROOF: drop the `late > DailyLateLimit` branch → "a 6-hour-late fire ran the job". +func TestDaily_LateFireIsSkipped(t *testing.T) { + loc := getBudapestLocation() + for _, tc := range []struct { + name string + firedAt time.Time + wantRun bool + }{ + {"on time", time.Date(2026, 10, 5, 2, 30, 1, 0, loc), true}, + {"after a suspend", time.Date(2026, 10, 5, 8, 31, 0, 0, loc), false}, + } { + t.Run(tc.name, func(t *testing.T) { + s := discardScheduler() + var mu sync.Mutex + now := time.Date(2026, 10, 5, 1, 0, 0, 0, loc) + s.nowFn = func() time.Time { mu.Lock(); defer mu.Unlock(); return now } + fired := make(chan time.Time, 1) + armed := make(chan struct{}, 1) + calls := 0 + s.afterFn = func(time.Duration) <-chan time.Time { + mu.Lock() + calls++ + first := calls == 1 + mu.Unlock() + if !first { + return make(chan time.Time) // never: park after the first fire + } + armed <- struct{}{} // the next run (02:30) is computed against 01:00 + return fired + } + ran := make(chan struct{}, 1) + s.Daily("offbox-backup", "02:30", func(context.Context) error { ran <- struct{}{}; return nil }) + ctx, cancel := context.WithCancel(context.Background()) + s.Start(ctx) + <-armed + mu.Lock() + now = tc.firedAt + mu.Unlock() + fired <- tc.firedAt + select { + case <-ran: + if !tc.wantRun { + t.Fatal("a 6-hour-late fire ran the job (the app-update leg would start at noon)") + } + case <-time.After(300 * time.Millisecond): + if tc.wantRun { + t.Fatal("an on-time fire did not run the job") + } + } + cancel() + s.Stop() + }) + } +} diff --git a/controller/internal/web/i18n_parity_test.go b/controller/internal/web/i18n_parity_test.go index bcea836..0a3d523 100644 --- a/controller/internal/web/i18n_parity_test.go +++ b/controller/internal/web/i18n_parity_test.go @@ -109,6 +109,16 @@ func i18nCases() []i18nCase { d["EscrowBannerBack"] = "/launcher" return d }}, + // R-871 (v0.295.0, decision 110): the missed-backup banner. Its two sentences are computed in Go per reader + // language (missed_backup_banner.go); here the Hungarian ones, as the hu render receives them. + {"launcher_missed_backup", "launcher", func() map[string]interface{} { + d := i18nLayoutData("launcher", "Indítópult") + d["MissedBackupBanner"] = true + d["MissedBackupMessage"] = "A legutóbbi mentés 2 nappal ezelőtt készült. A doboz ki volt kapcsolva a mentés idején (02:30)." + d["MissedBackupHint"] = "Válassz egy olyan időpontot, amikor a doboz általában be van kapcsolva — például 21:00." + d["MissedBackupBack"] = "/launcher" + return d + }}, {"launcher_nopassword", "launcher", func() map[string]interface{} { d := i18nLayoutData("launcher", "Indítópult") d["RecoveryBanner"] = true diff --git a/controller/internal/web/missed_backup_banner.go b/controller/internal/web/missed_backup_banner.go new file mode 100644 index 0000000..19b29a7 --- /dev/null +++ b/controller/internal/web/missed_backup_banner.go @@ -0,0 +1,113 @@ +package web + +import ( + "net/http" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/metrics" + "gitea.dooplex.hu/admin/felhom-controller/internal/nightchain" +) + +// The missed-backup banner (R-871, `09` decision 110 — the operator's idea). The rule is the pure +// nightchain.ComputeBanner; this file only gathers its inputs and renders. It hangs off executeTemplate like the +// escrow bar (R-543), so it reaches every dashboard page of a logged-in household. Unlike that bar its dismissal is +// DURABLE (the night ledger): closed until the NEXT missed backup time, gone by itself after a successful night. + +// SetMissedBackupBanner wires the night ledger and the metrics record (nil ledger = no banner: backups off). +func (s *Server) SetMissedBackupBanner(l *nightchain.Ledger, ms *metrics.MetricsStore) { + s.nightLedger, s.metricsRecord = l, ms +} + +// missedBackupBanner computes the banner for now (nil = nothing to show). +func (s *Server) missedBackupBanner(now time.Time) *nightchain.Banner { + if s.nightLedger == nil || s.settings == nil { + return nil + } + loc := budapestLocation() + seeded, dumpOK, _, dismissed, _ := s.nightLedger.Snapshot() + in := nightchain.BannerInput{ + Now: now, Window: s.effectiveBackupWindow(), Loc: loc, + SeededAt: seeded, DBDumpOK: dumpOK, Dismissed: dismissed, + } + if s.backupMgr != nil && s.backupMgr.OffboxConfigured() { + if t := s.settings.GetOffboxTarget(); t != nil && t.Enabled { + in.OffsiteConfigured = true + if ok, err := time.Parse(time.RFC3339, t.LastSuccess); err == nil { + in.OffsiteOK = ok + } + } + } + if s.metricsRecord != nil { + ms := s.metricsRecord + in.OnAtW = func(w time.Time) (bool, bool) { + n, err := ms.SampleCount(w.Add(-5*time.Minute), w.Add(5*time.Minute)) + if err != nil { + return false, false + } + return n > 0, true + } + if counts, err := ms.HourSampleCounts(now.Add(-7*24*time.Hour), now); err == nil { + h := nightchain.HourDaysFrom(counts, loc) + in.Hours = &h + } + } + b := nightchain.ComputeBanner(in) + if !b.Show { + return nil + } + return &b +} + +// addMissedBackupBanner is called from executeTemplate for every authenticated page. +func (s *Server) addMissedBackupBanner(data map[string]interface{}, r *http.Request) { + if data == nil || r == nil || !s.hasAdminSession(r) { + return + } + b := s.missedBackupBanner(time.Now()) + if b == nil { + return + } + lang := s.langFor(r) + msg := s.i18n.Msg(lang, "banner.missed_backup.never") + if !b.LastBackup.IsZero() { + msg = s.i18n.Msgf(lang, "banner.missed_backup.last", b.DaysAgo) + } + if b.OffAt != "" { + msg += " " + s.i18n.Msgf(lang, "banner.missed_backup.off_at", b.OffAt) + } + hint := s.i18n.Msg(lang, "banner.missed_backup.choose") + if b.Suggest != "" { + hint = s.i18n.Msgf(lang, "banner.missed_backup.suggest", b.Suggest) + } + data["MissedBackupBanner"] = true + data["MissedBackupMessage"] = msg + data["MissedBackupHint"] = hint + data["MissedBackupBack"] = r.URL.Path + if data["CSRFField"] == nil { + data["CSRFField"] = s.csrfField(r) + } + if s.isDebug() { + s.logger.Printf("[DEBUG] [web] missed-backup banner: rendered on %s (last=%s off_at=%q suggest=%q)", r.URL.Path, + b.LastBackup.Format(time.RFC3339), b.OffAt, b.Suggest) + } +} + +// missedBackupBannerDismissHandler (POST /backups/missed-banner/dismiss) closes the banner until the NEXT missed +// backup time — durable, in the night ledger. +func (s *Server) missedBackupBannerDismissHandler(w http.ResponseWriter, r *http.Request) { + if b := s.missedBackupBanner(time.Now()); b != nil && s.nightLedger != nil { + s.nightLedger.DismissBanner(b.MissedAt) + s.logger.Printf("[INFO] [web] missed-backup banner: closed by the household until the next missed backup time (after %s)", + b.MissedAt.Format(time.RFC3339)) + } + http.Redirect(w, r, redirectBackTo(r, "/launcher"), http.StatusFound) +} + +// budapestLocation is the backup window's clock. +func budapestLocation() *time.Location { + loc, err := time.LoadLocation("Europe/Budapest") + if err != nil { + return time.Local + } + return loc +} diff --git a/controller/internal/web/r871_missed_backup_banner_test.go b/controller/internal/web/r871_missed_backup_banner_test.go new file mode 100644 index 0000000..725eec6 --- /dev/null +++ b/controller/internal/web/r871_missed_backup_banner_test.go @@ -0,0 +1,59 @@ +package web + +import ( + "net/url" + "path/filepath" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/nightchain" +) + +// R-871 / `09` decision 110 — the missed-backup banner through the REAL pages (ServeHTTP → executeTemplate) and the +// REAL dismiss route. The rule itself is pinned in internal/nightchain/banner_test.go. +// COMPANION RED-PROOF: remove `s.addMissedBackupBanner(data, r)` from executeTemplate → "not shown on /launcher". + +const missedBannerID = `id="missed-backup-banner"` + +func missedServer(t *testing.T, lastDump time.Time) (*Server, *nightchain.Ledger) { + t.Helper() + s := newDashboardServer(t, time.Time{}) + l, err := nightchain.Open(filepath.Join(t.TempDir(), "night-ledger.json"), time.Now().Add(-10*24*time.Hour)) + if err != nil { + t.Fatal(err) + } + if !lastDump.IsZero() { + l.MarkDBDumpOK(lastDump) + } + s.SetMissedBackupBanner(l, nil) + return s, l +} + +func TestR871_BannerShownThenClosedUntilTheNextMiss(t *testing.T) { + s, l := missedServer(t, time.Now().Add(-3*24*time.Hour)) + body := getPage(t, s, "/launcher").Body.String() + if !strings.Contains(body, missedBannerID) { + t.Fatal("not shown on /launcher although the last backup is 3 days old") + } + if !strings.Contains(body, "A legutóbbi mentés 3 nappal ezelőtt készült.") || !strings.Contains(body, `href="/backups#window_start"`) { + t.Fatalf("the banner lacks its sentence or its button:\n%s", body[strings.Index(body, missedBannerID):][:600]) + } + rec := postForm(t, s, "/backups/missed-banner/dismiss", url.Values{"back": {"/launcher"}}) + if rec.Code/100 != 3 { + t.Fatalf("dismiss = %d", rec.Code) + } + if strings.Contains(getPage(t, s, "/launcher").Body.String(), missedBannerID) { + t.Fatal("still shown after the household closed it") + } + if _, _, _, dismissed, _ := l.Snapshot(); dismissed.IsZero() { + t.Fatal("the dismissal was not recorded in the ledger (it would not survive a restart)") + } +} + +func TestR871_BannerNotShownAfterASuccessfulNight(t *testing.T) { + s, _ := missedServer(t, time.Now().Add(-2*time.Hour)) + if strings.Contains(getPage(t, s, "/launcher").Body.String(), missedBannerID) { + t.Fatal("shown although the last backup is 2 hours old") + } +} diff --git a/controller/internal/web/server.go b/controller/internal/web/server.go index d826213..dada471 100644 --- a/controller/internal/web/server.go +++ b/controller/internal/web/server.go @@ -7,6 +7,8 @@ import ( "encoding/hex" "fmt" "gitea.dooplex.hu/admin/felhom-controller/internal/family" + "gitea.dooplex.hu/admin/felhom-controller/internal/metrics" + "gitea.dooplex.hu/admin/felhom-controller/internal/nightchain" "html/template" "io" "io/fs" @@ -36,6 +38,10 @@ import ( ) type Server struct { + // R-871: the missed-backup banner's inputs (SetMissedBackupBanner); nil = no banner. + nightLedger *nightchain.Ledger + metricsRecord *metrics.MetricsStore + cfg *config.Config stackMgr *stacks.Manager cpuCollector *system.CPUCollector @@ -765,6 +771,8 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { case path == "/backup/shares/place" && r.Method == http.MethodPost: s.sharesPlaceHandler(w, r) // Controller-driven escrow ceremony wizard (v0.127.0): the customer-facing R flow. + case path == "/backups/missed-banner/dismiss" && r.Method == http.MethodPost: + s.missedBackupBannerDismissHandler(w, r) case path == "/backup/escrow/banner/dismiss" && r.Method == http.MethodPost: s.escrowBannerDismissHandler(w, r) case path == "/backup/escrow" && r.Method == http.MethodGet: @@ -1005,6 +1013,7 @@ func (s *Server) executeTemplate(w http.ResponseWriter, r *http.Request, name st // through here (TestI18nDirectRenderPagesHaveNoAdminChrome); the session check inside is a second // fence for the guest share page, which has no admin session. s.addEscrowBanner(data, r) + s.addMissedBackupBanner(data, r) // R-871 (decision 110) lang := s.langFor(r) s.addLanguageData(data, r, lang) var buf bytes.Buffer diff --git a/controller/internal/web/templates/layout.html b/controller/internal/web/templates/layout.html index 684407a..dc05395 100644 --- a/controller/internal/web/templates/layout.html +++ b/controller/internal/web/templates/layout.html @@ -178,6 +178,24 @@ until the household creates its recovery code (zero-knowledge escrow: their code is the only key). The pause is the design; what was missing was ASKING. Same mechanism as the R-241 bar above — dismissable for the visit, back at the next one, gone for good when the escrow completes. */}} +{{/* R-871 (v0.295.0, `09` decision 110) — the missed-backup banner: when the last daily backup is over 26 h old, + why (the box was off at the backup time, from the box's own record) and a suggested time. Closing it lasts until + the NEXT missed backup time (durable, the night ledger); a successful night removes it. Never changes the time. */ -}} +{{if .MissedBackupBanner}} +
+
+ + {{.MissedBackupMessage}} {{.MissedBackupHint}} + + {{T "layout.missed_backup_change"}} +
+ {{.CSRFField}} + +
+ +
+
+{{end -}} {{if .EscrowBanner}}
diff --git a/controller/internal/web/testdata/i18n_parity/launcher_missed_backup.html b/controller/internal/web/testdata/i18n_parity/launcher_missed_backup.html new file mode 100644 index 0000000..3f98bde --- /dev/null +++ b/controller/internal/web/testdata/i18n_parity/launcher_missed_backup.html @@ -0,0 +1,596 @@ + + + + + + + + Indítópult — Felhom.eu + + + + + + + + +
+ + +
+ + +
+ + + + + +
+
+ + A legutóbbi mentés 2 nappal ezelőtt készült. A doboz ki volt kapcsolva a mentés idején (02:30). Válassz egy olyan időpontot, amikor a doboz általában be van kapcsolva — például 21:00. + + Mentési idő módosítása +
+ + +
+ +
+
+ + + + +
+

Indítópult

+ example.hu + +
+ + + + +
+

Még nincs telepített alkalmazás.

+

Alkalmazások telepítése

+
+ + + + + + + +
+ + + + diff --git a/controller/scripts/i18n_go_keys.json b/controller/scripts/i18n_go_keys.json index f97f3d4..12b8cfb 100644 --- a/controller/scripts/i18n_go_keys.json +++ b/controller/scripts/i18n_go_keys.json @@ -1,6 +1,12 @@ { "_comment": "Localisation slice 2 (R-557). key -> the base-commit Go literal it replaced, or the ORDERED list of literals a concatenation joined. Checked by scripts/i18n_go_parity.py against scripts/i18n_go_base.json, which is frozen at 736f54b49610 (the base commit of release v0.252.0). A key whose Hungarian text is not byte-identical to what the Go code said fails the gate.", "_preexisting": { + "banner.missed_backup.never": "BORN AS A KEY, v0.295.0 (R-871, `09` decision 110) -- a NEW sentence of the missed-backup banner, never a Go literal; pinned in Hungarian by TestR871_BannerShownThenClosedUntilTheNextMiss and the parity fixture launcher_missed_backup.", + "banner.missed_backup.last": "BORN AS A KEY, v0.295.0 (R-871, `09` decision 110) -- a NEW sentence of the missed-backup banner, never a Go literal; pinned in Hungarian by TestR871_BannerShownThenClosedUntilTheNextMiss and the parity fixture launcher_missed_backup.", + "banner.missed_backup.off_at": "BORN AS A KEY, v0.295.0 (R-871, `09` decision 110) -- a NEW sentence of the missed-backup banner, never a Go literal; pinned in Hungarian by TestR871_BannerShownThenClosedUntilTheNextMiss and the parity fixture launcher_missed_backup.", + "banner.missed_backup.choose": "BORN AS A KEY, v0.295.0 (R-871, `09` decision 110) -- a NEW sentence of the missed-backup banner, never a Go literal; pinned in Hungarian by TestR871_BannerShownThenClosedUntilTheNextMiss and the parity fixture launcher_missed_backup.", + "banner.missed_backup.suggest": "BORN AS A KEY, v0.295.0 (R-871, `09` decision 110) -- a NEW sentence of the missed-backup banner, never a Go literal; pinned in Hungarian by TestR871_BannerShownThenClosedUntilTheNextMiss and the parity fixture launcher_missed_backup.", + "event.backup_catchup_done": "BORN AS A KEY, v0.295.0 (R-871, `09` decision 109) -- the catch-up's NEW timeline line, never a Go literal; sent by Notifier.NotifyBackupCatchUp (wiring pinned by TestR871_CatchUpWiring).", "badge.lifecycle.abandoned": "R-589 (v0.258.0) -- localeFuncs; the Hungarian form stays in templateFuncMap, pinned by TestLocaleFuncsHungarianBundleMatchesFuncMap", "badge.lifecycle.abandoned.title": "R-589 (v0.258.0) -- localeFuncs; the Hungarian form stays in templateFuncMap, pinned by TestLocaleFuncsHungarianBundleMatchesFuncMap", "badge.update.ahead.title": "BORN AS A KEY, v0.260.0 (R-524) -- a NEW sentence, never a Go literal, so there is nothing in the base capture to measure it against. Pinned in both languages by TestUpdateBadgeFollowsTheLanguage.",