v0.227.1: the damage classifier matched restic's ordinary progress output

A patch and not a rebuilt 0.227.0: that tag was already running on demo-hp, and
re-pushing changed bytes under a live tag is the :latest hazard with extra steps.

looksLikeRepositoryDamage matched bare "pack ", "tree ", "snapshot ", "blob ". A
HEALTHY restic check prints "check all packs" and "check snapshots, trees and
blobs" -- so any check that failed for a NON-damage reason, a connection dropped
mid-run for instance, would have been classified as a corrupted repository and
told the customer their backups may be damaged. That is the false alarm that
teaches an operator to ignore the true one.

Caught by the NEGATIVE control, built from the real bytes of a real passing
check on demo-hp. The spec made the negative control mandatory and this is what
it was for: a control that has only ever seen the failing case proves nothing.

Signatures are now phrases from restic's own error wording.

Also in this commit: CONTEXT.md records the three rulings (take the flag and
skip, due-ness not a weekday, publish on OffboxReportStatus not the R-331 dead
fields) plus the measurement a future session would otherwise assume wrongly --
THE STRUCTURE CHECK DOES NOT CATCH SILENT CORRUPTION. README documents the job,
the route and the config, and corrects a line that listed four debug backup
routes when only two exist. REUSE gains three rows, including one that records
R-398 was my own mistake so nobody re-files it.

controller/README.md

@@ -1089,6 +1089,43 @@ backups/primary/<app>/
   maradtak."). **Every one is a claim about the BACKUP, never about the app** — see CONTEXT.md's ruling
   and 07-backup-architecture §6.3.
 
+### Off-site integrity check (v0.227.0, R-359/R-397)
+
+**What it is:** a `restic check` against the off-site repository, run by the controller itself. Until
+v0.227.0 nothing verified that the off-site copies were readable — the whole-guest tier had verify
+jobs, the tier holding the customer's documents and photos had none.
+
+| | |
+|---|---|
+| job | `offsite-integrity`, `sched.Daily` at **06:00** |
+| cadence | **due-ness, not a weekday** — runs when the last SUCCESSFUL check is older than `monitoring.integrity.max_age_days` (default **7**). A box switched off on its check day is checked the next day it is on |
+| depth | structure + index by default. `monitoring.integrity.read_data_subset` (default **empty**) adds `--read-data-subset=<spec>`; a malformed value is refused at read time with a WARN and treated as empty |
+| guard | takes the single-writer flag and **SKIPS rather than waits** |
+| timeout | 30 min (`integrityCheckTimeout`) — bounds a hung repository so it cannot pin the flag |
+| by hand | `POST /api/debug/backup/integrity` — same code path, due-ness ignored, **every other guard intact** |
+| result | persisted on `settings.OffboxTarget` (`last_integrity_check`, `last_integrity_ok`) and published on `OffboxReportStatus` |
+
+**Three outcomes, not two.** `Skipped` (a sibling operation held the flag), `Unreachable` (the repo
+could not be opened, or the check timed out) and failed are different facts. Only a failure notifies;
+a skip and an unreachable repository do **not** advance due-ness, so tomorrow tries again. A failure
+**does** advance it — re-checking a broken store nightly is load with no new information.
+
+**Notifications.** `backup_integrity_ok` is severity `info`, which `severityNotifies` drops — it mails
+nobody, by design. `backup_integrity_failed` is `error` and reaches the operator; the customer leg is
+switchable and OFF by default. The customer gets a sentence; restic's output goes to the log,
+truncated.
+
+> **⚠ THE STRUCTURE CHECK DOES NOT CATCH SILENT CORRUPTION, and this is the thing to know before
+> trusting it.** Measured on `demo-hp` 2026-08-30 against a throwaway repo whose pack was corrupted
+> *without changing its size*: `restic check` returned **`no errors were found`, exit 0**; every
+> `--read-data*` form returned `Pack ID does not match …` and exit 1. The structure check verifies the
+> index, the pack inventory and the snapshot graph — it catches missing packs, broken indexes and
+> unreadable snapshots — but it does **not** re-hash pack contents. Choosing the read-data depth is
+> **R-399**, and the cost curve is measured:
+> structure 35.0 s · 10% 35.9 s · 50% 37.3 s · **100% 39.2 s** on a 134.3 MB / 67-snapshot store.
+> Those figures do not extrapolate: the structure check's cost tracks the index, read-data's tracks
+> the data.
+
 ### Restore refusals (v0.226.0)
 
 Three guards added on the off-site restore surface, all server-side:
@@ -2830,7 +2867,7 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
 |---|---------|-----------|-------------|
 | 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, network (guest-netns interfaces/route/DNS via the samba door, R-66; best-effort per item), scheduler, health, alerts. JSON download. |
 | 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. |
-| 3 | Mentés teszt | `POST /api/debug/backup/{dbdump,crossdrive,integrity,infra}` | Trigger individual backup phases independently. |
+| 3 | Mentés teszt | `POST /api/debug/backup/dbdump` · `POST /api/debug/backup/integrity` | Trigger a DB dump, or run an off-site integrity check by hand. **`crossdrive` and `infra` are NOT implemented** — their buttons 404 (R-400). |
 | 4 | Tárhely teszt | `POST /api/debug/storage/simulate-{disconnect,reconnect}`, `GET /api/debug/storage/watchdog-status` | Simulate drive disconnect/reconnect without unmounting. Per-path probe state with 5s auto-refresh. |
 | 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,infra-push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. |
 | — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |

controller/internal/backup/offbox_integrity.go

@@ -233,15 +233,26 @@ func (m *Manager) CheckOffboxIntegrity(ctx context.Context) IntegrityResult {
 // already alarms when the store cannot be reached.
 func looksLikeRepositoryDamage(out []byte) bool {
 	s := strings.ToLower(string(out))
+	// THE SIGNATURES ARE PHRASES, NOT WORDS, AND THE REASON IS A BUG THIS FILE ALREADY HAD.
+	//
+	// The first draft matched bare `"pack "`, `"tree "`, `"snapshot "` and `"blob "`. Those appear in
+	// restic's ORDINARY PROGRESS OUTPUT — a healthy run prints `check all packs` and
+	// `check snapshots, trees and blobs` — so a check that failed for a NON-damage reason (a dropped
+	// connection mid-run, say) would have been classified as a corrupted store and alarmed the customer
+	// that their backups were damaged. Caught by the negative control in
+	// TestR359_HealthyRealOutputIsNotDamage, using the real bytes of a real passing check.
+	//
+	// Every phrase below is restic's own error wording, taken from the 2026-08-30 damaged-pack run on
+	// demo-hp or from restic's check source — never paraphrased.
 	for _, sig := range []string{
-		"pack ",              // "pack 1234abcd: not found in index" / "... size mismatch"
-		"load index",         // a broken index
-		"blob ",              // "blob not found"
-		"tree ",              // "tree 1234: file ... blob not found"
-		"snapshot ",          // "error for snapshot ...: ..."
+		"does not match",               // "Pack ID does not match, want <id>, got <id>" — the measured one
+		"not found in index",           // a blob or pack the index promises and the store lacks
+		"blob not found",               //
+		"size mismatch",                //
 		"ciphertext verification failed",
 		"integrity error",
-		"repository contains errors",
+		"repository contains errors",   // restic's own summary verdict
+		"failed to load index",         //
 	} {
 		if strings.Contains(s, sig) {
 			return true

controller/internal/backup/r359_integrity_test.go

@@ -259,3 +259,47 @@ func TestR359_NoTargetConfiguredIsASilentSkip(t *testing.T) {
 		t.Fatal("a skip was reported as a passing check — nothing was checked")
 	}
 }
+
+// TestR359_RealResticDamageOutputIsClassifiedAsDamage uses the EXACT bytes restic produced on
+// `demo-hp` on 2026-08-30 against a deliberately corrupted throwaway repository (Part 5's positive
+// control). Invented output would only prove the classifier agrees with my guess about restic; this
+// closes the loop on real bytes.
+//
+// The damage was 64 zero bytes written at offset 1024 of one pack, leaving the file SIZE unchanged —
+// the subtlest form, and the one a structure check cannot see. See the accompanying finding: plain
+// `restic check` returned "no errors were found" and exit 0 over this very repository.
+func TestR359_RealResticDamageOutputIsClassifiedAsDamage(t *testing.T) {
+	const realOutput = "Pack ID does not match, want 288afd3e868dc6bd210e33bd6f821e9f088a5fd71a0464c23ce82eb8217bf0cc, got 4b6847bb5eece6c56e69d7381733827330a4eb799fc26a92287a181edc496d2b\nFatal: repository contains errors"
+
+	if !looksLikeRepositoryDamage([]byte(realOutput)) {
+		t.Fatal("restic's REAL damage output was not recognised as damage — the check would report a " +
+			"corrupted store as merely unreachable, and the customer would never be told")
+	}
+
+	m, _ := newIntegrityManager(t, okRepo(func([]string) ([]byte, error) {
+		return []byte(realOutput), errFake
+	}))
+	res := m.CheckOffboxIntegrity(context.Background())
+
+	if res.OK {
+		t.Fatal("a repository restic called corrupt was reported as passing")
+	}
+	if res.Unreachable {
+		t.Fatal("readable-and-corrupt was reported as unreachable — the store WAS opened and read; " +
+			"that misclassification would suppress the one alarm that matters")
+	}
+	if !strings.Contains(res.Output, "288afd3e") {
+		t.Error("restic's own words did not reach the log")
+	}
+}
+
+// TestR359_HealthyRealOutputIsNotDamage is the negative control for the classifier, from the same
+// live run: the healthy repository's actual output must not trip the damage predicate.
+func TestR359_HealthyRealOutputIsNotDamage(t *testing.T) {
+	const realHealthy = "using temporary cache in /tmp/restic-check-cache-962728151\ncreate exclusive lock for repository\nload indexes\ncheck all packs\ncheck snapshots, trees and blobs\n\nno errors were found"
+
+	if looksLikeRepositoryDamage([]byte(realHealthy)) {
+		t.Fatalf("a HEALTHY check's real output was classified as damage — every weekly check would " +
+			"alarm, which is how an operator learns to ignore the alarm")
+	}
+}