R-359 + R-397: the off-site store gets checked, and the advertised check becomes real
gates / gates (push) Successful in 12s

Nothing ever verified that the off-site copies are still readable. The
whole-guest tier has verify jobs; the tier holding the customer's documents and
photos had none -- the complete set of restic verbs this controller used
contained no `check`. We would have found out at restore time, with a customer
waiting. On 2026-08-21 a deliberately damaged pack was caught at once by plain
`restic check`; we had never run it.

R-397: NotifyIntegrityOK/NotifyIntegrityFailed existed with no caller, the hub
allowlists both event types and carries the Hungarian text for both, the
settings checkbox exists, and the debug button posts to /api/debug/backup/
integrity. Everything was built except the part that runs. SIXTH instance of
that shape in this project.

THE HAZARD SHAPES THE WHOLE DESIGN. resticStep self-heals a crash lock by
running `unlock --remove-all` and retrying, and its own comment records why that
is safe: every caller holds the in-process single-flight mutex, so any lock it
meets is stale. A check that did not take that flag could meet a LIVE prune's
lock from this same box, remove it, and retry over the top of it. So the check
TAKES THE FLAG and SKIPS rather than waits -- waiting would pin the nightly
backup behind it, and a skip costs nothing because due-ness makes tomorrow try
again. TestR359_SkipsWhenRunningFlagHeld asserts the NON-EFFECTS: restic never
invoked, `unlock` never in any argv. Its red-proof prints the real thing --
restic running `check` while the flag was held.

DUE-NESS, NOT A WEEKDAY. Daily job, weekly behaviour: "is the last successful
check older than 7 days?" not "is it Sunday?". R-341 is exactly the other shape,
a dated check quietly missed and never caught up. No Weekly primitive added.

THREE OUTCOMES, NOT TWO. Skipped, Unreachable and failed are different facts.
"I could not look" is not "I looked and it is broken" -- R-339 already owns
reachability, and a second alarm for the same fact trains the operator to
discount the one alarm that means the backups are damaged. A timeout is
unreachable, never damage. A failure advances due-ness (a broken store must not
be re-checked nightly); a skip and an unreachable store do not.

Success is severity `info`, which severityNotifies DROPS -- it mails NOBODY, by
design. A weekly success e-mail is how people stop reading their alerts.

The customer gets a SENTENCE; restic's words go to the log, truncated (R-379:
615 bytes of raw database text reached a customer once). read-data-subset ships
OFF and a malformed value is refused at read time rather than handed to restic,
where one typo would fail the whole check.

Published on OffboxReportStatus, NOT on report.BackupReport's IntegrityOK --
those were retired by R-331 YESTERDAY and TestBackupReport_DeadFieldsStayZero
still passes unmodified.

Also: the monitoring page stopped promising a Sunday job that never existed, and
the debug button got its dispatch case.

PART 0 WAS NOT BUILT, AND R-398 WAS MY OWN MISTAKE. The seam it asked for
already exists: offboxRunner/SetOffboxRunner/m.runner() has been injectable
since the off-site tier shipped, and other tests drive restic-backed paths
through it. A resticStepFn seam would have been WORSE here -- it would replace
the `unlock --remove-all` escalation and hide it from the assertions that must
see it. R-358's AST ordering test is converted to a real execution test instead,
which immediately surfaced something the AST walk could not: unlockStale
legitimately runs before the restore.

Four red-proofs, each printing the pre-fix behaviour. Green gate: 28 packages,
rc 0. All 12 controller gates OK.
This commit is contained in:
2026-08-30 21:03:29 +02:00
parent e64c84aef8
commit 0d52a42c17
13 changed files with 1365 additions and 52 deletions
+53
View File
@@ -17,6 +17,7 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
"gitea.dooplex.hu/admin/felhom-controller/internal/appexport"
"gitea.dooplex.hu/admin/felhom-controller/internal/monitor"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/report"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
@@ -29,6 +30,15 @@ type DebugCallbacks struct {
HubConnectivityTest func() (statusCode int, latencyMs int64, err error)
GiteaConnectivityTest func() (statusCode int, latencyMs int64, err error)
GetTelemetryPreview func() ([]report.AppTelemetry, error)
// RunIntegrityCheck (R-359/R-397) runs the off-site integrity check SYNCHRONOUSLY and returns what
// it did. It is a callback rather than a direct call because the one implementation lives in
// main.go, where the manager and the notifier are both in scope — and there must be exactly one:
// a hand-run that diverged from the scheduled run is how a guard gets bypassed "because the
// operator asked for it", which is the specific hazard this feature is shaped around.
//
// `force` is the ONLY difference between the two callers. It skips the due-ness question and
// nothing else — every guard, above all the single-writer flag, applies identically.
RunIntegrityCheck func(force bool) backup.IntegrityResult
}
// debugPageHandler renders the debug dashboard page.
@@ -55,6 +65,12 @@ func (s *Server) handleDebugAPI(w http.ResponseWriter, r *http.Request) {
// Section 3: Backup testing (app-data only; disk-tier moved to host agent)
case subpath == "backup/dbdump" && r.Method == http.MethodPost:
s.debugTriggerDBDump(w, r)
// R-397 — the button at debug.html:83 has posted here since it was added and NOTHING answered.
// Verified 2026-08-30: this dispatch had no such case, so pressing „Restic integritás" did
// nothing at all. Seventh instance of built-but-never-wired in this project; filed as R-400 in its
// own right rather than disappearing inside this change.
case subpath == "backup/integrity" && r.Method == http.MethodPost:
s.debugRunIntegrityCheck(w, r)
// Section 5: Hub & connectivity
case subpath == "hub/push" && r.Method == http.MethodPost:
@@ -388,6 +404,43 @@ func (s *Server) debugTriggerDBDump(w http.ResponseWriter, r *http.Request) {
writeDebugJSON(w, http.StatusOK, true, "DB dump elindítva", nil)
}
// debugRunIntegrityCheck runs the off-site integrity check by hand (R-359/R-397).
//
// SYNCHRONOUS on purpose, unlike the DB-dump button beside it: the operator pressed this to learn an
// ANSWER, and a fire-and-forget that returns „elindítva" would leave them reading logs for the result.
// A structure check is seconds-to-minutes; its own timeout bounds it.
//
// Due-ness is IGNORED — "run it now" is the whole point of a button. Every other guard is intact,
// including the single-writer flag, so a hand-run during a backup SKIPS exactly as the scheduled one
// would. That is asserted by TestR397_DebugRunStillHonoursTheRunningFlag, because "the operator asked
// for it" is precisely the reasoning that would reintroduce the hazard.
func (s *Server) debugRunIntegrityCheck(w http.ResponseWriter, r *http.Request) {
if s.debugCallbacks == nil || s.debugCallbacks.RunIntegrityCheck == nil {
writeDebugJSON(w, http.StatusNotImplemented, false, "Nem bekötött", nil)
return
}
res := s.debugCallbacks.RunIntegrityCheck(true)
data := map[string]interface{}{
"ok": res.OK,
"skipped": res.Skipped,
"skip_reason": res.SkipReason,
"unreachable": res.Unreachable,
"duration_ms": res.Duration.Milliseconds(),
"read_data_subset": res.ReadDataSubset,
}
switch {
case res.Skipped:
writeDebugJSON(w, http.StatusOK, true, "Kihagyva: "+res.SkipReason, data)
case res.Unreachable:
// NOT reported as a failure: the store could not be opened, so nothing was concluded about it.
writeDebugJSON(w, http.StatusOK, true, "A tároló nem érhető el — az ellenőrzés nem futott le", data)
case res.OK:
writeDebugJSON(w, http.StatusOK, true, "Az ellenőrzés rendben lezajlott", data)
default:
writeDebugJSON(w, http.StatusOK, false, "Az ellenőrzés hibát talált a tárolóban", data)
}
}
// ── Section 5: Hub & connectivity ───────────────────────────────────
func (s *Server) debugHubPush(w http.ResponseWriter, r *http.Request) {
+13 -1
View File
@@ -805,7 +805,7 @@ func (s *Server) monitoringHandler(w http.ResponseWriter, r *http.Request) {
{"Label": "Rendszer allapot", "Icon": "system", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.SystemHealth), "Schedule": "5 percenkent"},
{"Label": "Adatbazis mentes", "Icon": "db", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.DBDump), "Schedule": "Naponta " + s.cfg.Backup.DBDumpSchedule},
{"Label": "Biztonsagi mentes", "Icon": "backup", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.Backup), "Schedule": "Naponta " + s.cfg.Backup.ResticSchedule},
{"Label": "Mentes integritas", "Icon": "integrity", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.BackupIntegrity), "Schedule": "Hetente (vasarnap)"},
{"Label": "Mentes integritas", "Icon": "integrity", "Configured": isPingConfigured(s.cfg.Monitoring.PingUUIDs.BackupIntegrity), "Schedule": monitoringIntegritySchedule},
}
allConfigured := true
for _, p := range pings {
@@ -1564,6 +1564,18 @@ const (
unitRestoreNoneReturnedMsgFmt = "A(z) %s: FIGYELEM — a mentés %d adatkötetet és %d adatbázis-mentést sorol fel, de egyik sem állt vissza. Az adataid változatlanok maradtak. Kérj segítséget, mielőtt újra próbálod."
)
// monitoringIntegritySchedule (R-359) describes what the off-site integrity job actually does.
//
// Until v0.227.0 this read „Hetente (vasarnap)" and described a check that DID NOT EXIST — the page
// told the operator a weekly integrity check ran while nothing in the product ever called restic's
// `check`. It now describes the job that was built, and the distinction is not pedantry: the job is
// due-ness based, so a box that was switched off on its check day is checked the next day it is on,
// and a page promising a fixed weekday would be a second untrue sentence replacing the first.
//
// ASCII deliberately, matching the surrounding rows — and §9's accented-grep rule: a zero result on
// accented text is suspect before the software is (R-364).
const monitoringIntegritySchedule = "Hetente, kimarado ellenorzest potol"
// C9-F1 customer-facing strings. Kept as named constants, not inlined, because both are asserted
// verbatim by tests — a silent edit to either is the way an honest message drifts back into a
// comforting one.
@@ -0,0 +1,144 @@
package web
import (
"encoding/json"
"io"
"log"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// ── R-397 / R-400 — the button that posted to nothing ────────────────────────────────────────────
//
// `debug.html:83` has carried a „Restic integritás" button posting to /api/debug/backup/integrity
// since it was added. Verified 2026-08-30: `handleDebugAPI` had no such case, so pressing it did
// nothing at all — no error, no result, no log line. Seventh instance of built-but-never-wired in this
// project, and filed as R-400 in its own right rather than disappearing inside this change.
func newDebugServer(t *testing.T, cb *DebugCallbacks) *Server {
t.Helper()
cfg := &config.Config{}
cfg.Paths.DataDir = t.TempDir()
return &Server{cfg: cfg, logger: log.New(io.Discard, "", 0), debugCallbacks: cb}
}
// postIntegrity returns the response, the ENVELOPE, and the nested `data` payload. The two are
// separate on purpose: `ok` at the envelope level means "the request was handled", while the payload's
// own `ok` means "the store passed". Conflating them is how an unreachable store would read as a
// failed check.
func postIntegrity(t *testing.T, s *Server) (*httptest.ResponseRecorder, map[string]interface{}, map[string]interface{}) {
t.Helper()
req := httptest.NewRequest(http.MethodPost, "/api/debug/backup/integrity", nil)
w := httptest.NewRecorder()
s.handleDebugAPI(w, req)
var env map[string]interface{}
_ = json.Unmarshal(w.Body.Bytes(), &env)
data, _ := env["data"].(map[string]interface{})
if data == nil {
data = map[string]interface{}{}
}
return w, env, data
}
func TestR397_DebugRouteExists(t *testing.T) {
var called bool
s := newDebugServer(t, &DebugCallbacks{
RunIntegrityCheck: func(force bool) backup.IntegrityResult {
called = true
if !force {
t.Error("the hand-run did not force — a button that respects due-ness does nothing on " +
"six days out of seven, which is indistinguishable from the button being dead")
}
return backup.IntegrityResult{OK: true}
},
})
w, env, _ := postIntegrity(t, s)
if !called {
t.Fatal("POST /api/debug/backup/integrity was not dispatched — this is the defect: the button " +
"has posted here since it was added and nothing answered")
}
if w.Code != http.StatusOK {
t.Fatalf("status = %d", w.Code)
}
if ok, _ := env["ok"].(bool); !ok {
t.Errorf("a passing check did not report success: %v", env)
}
}
func TestR397_DebugRunStillHonoursTheRunningFlag(t *testing.T) {
// THE POINT. "The operator asked for it" is precisely the reasoning that would bypass the
// single-writer flag and let the check meet a live prune's lock. force skips due-ness and NOTHING
// else; the skip must still surface as a skip.
s := newDebugServer(t, &DebugCallbacks{
RunIntegrityCheck: func(bool) backup.IntegrityResult {
return backup.IntegrityResult{Skipped: true, SkipReason: "a backup or restore is already running"}
},
})
_, env, data := postIntegrity(t, s)
msg, _ := env["message"].(string)
if !strings.Contains(msg, "Kihagyva") {
t.Fatalf("a hand-run during a live operation did not report a skip: %v", env)
}
if skipped, _ := data["skipped"].(bool); !skipped {
t.Errorf("the payload did not carry skipped=true: %v", data)
}
if ok, _ := data["ok"].(bool); ok {
t.Error("a skip was reported as a passing CHECK — nothing was checked")
}
}
func TestR397_DebugRunSeparatesUnreachableFromDamage(t *testing.T) {
// "I could not look" and "I looked and it is broken" must not read the same on the operator's
// screen either — the same rule the notifier path obeys.
s := newDebugServer(t, &DebugCallbacks{
RunIntegrityCheck: func(bool) backup.IntegrityResult {
return backup.IntegrityResult{Unreachable: true}
},
})
_, env, data := postIntegrity(t, s)
// The REQUEST succeeded (we handled it) but the CHECK did not pass — and must not read as damage.
if ok, _ := env["ok"].(bool); !ok {
t.Error("an unreachable store was reported as a FAILED check — nothing was looked at, and " +
"telling an operator their store is damaged when it was merely unreachable is the alarm " +
"that trains them to ignore the real one")
}
if unreach, _ := data["unreachable"].(bool); !unreach {
t.Errorf("the payload did not distinguish unreachable: %v", data)
}
if ok, _ := data["ok"].(bool); ok {
t.Error("an unreachable store reported a passing check")
}
}
func TestR397_DebugRouteUnwiredSaysSo(t *testing.T) {
// The nil-callback case must be loud, not silently successful — that is the failure shape the
// button itself had.
s := newDebugServer(t, &DebugCallbacks{})
w, _, _ := postIntegrity(t, s)
if w.Code != http.StatusNotImplemented {
t.Fatalf("an unwired callback returned %d, want 501", w.Code)
}
}
// TestR359_MonitoringPageNoLongerPromisesASundayJob — §9 rule 9: the source line is ASCII (no
// accents), so this asserts on ASCII fragments and carries both controls.
func TestR359_MonitoringPageNoLongerPromisesASundayJob(t *testing.T) {
const src = "controller/internal/web/handlers.go"
_ = src
// POSITIVE CONTROL: the label the row is about is still present in the code under test.
if !strings.Contains(monitoringIntegritySchedule, "Hetente") {
t.Fatalf("positive control failed — the schedule string does not mention a weekly cadence: %q", monitoringIntegritySchedule)
}
// NEGATIVE CONTROL + the assertion: it no longer names a weekday, because the job is due-ness based
// and a box that was off on Sunday is checked on Monday.
if strings.Contains(strings.ToLower(monitoringIntegritySchedule), "vasarnap") {
t.Fatalf("the monitoring page still promises a Sunday job, which is not what was built: %q", monitoringIntegritySchedule)
}
}