v0.227.1: the damage classifier matched restic's ordinary progress output
gates / gates (push) Successful in 11s

A patch and not a rebuilt 0.227.0: that tag was already running on demo-hp, and
re-pushing changed bytes under a live tag is the :latest hazard with extra steps.

looksLikeRepositoryDamage matched bare "pack ", "tree ", "snapshot ", "blob ". A
HEALTHY restic check prints "check all packs" and "check snapshots, trees and
blobs" -- so any check that failed for a NON-damage reason, a connection dropped
mid-run for instance, would have been classified as a corrupted repository and
told the customer their backups may be damaged. That is the false alarm that
teaches an operator to ignore the true one.

Caught by the NEGATIVE control, built from the real bytes of a real passing
check on demo-hp. The spec made the negative control mandatory and this is what
it was for: a control that has only ever seen the failing case proves nothing.

Signatures are now phrases from restic's own error wording.

Also in this commit: CONTEXT.md records the three rulings (take the flag and
skip, due-ness not a weekday, publish on OffboxReportStatus not the R-331 dead
fields) plus the measurement a future session would otherwise assume wrongly --
THE STRUCTURE CHECK DOES NOT CATCH SILENT CORRUPTION. README documents the job,
the route and the config, and corrects a line that listed four debug backup
routes when only two exist. REUSE gains three rows, including one that records
R-398 was my own mistake so nobody re-files it.
This commit is contained in:
2026-08-30 21:22:23 +02:00
parent 0d52a42c17
commit 45770f2282
6 changed files with 270 additions and 8 deletions
+38 -1
View File
@@ -1089,6 +1089,43 @@ backups/primary/<app>/
maradtak."). **Every one is a claim about the BACKUP, never about the app** — see CONTEXT.md's ruling
and 07-backup-architecture §6.3.
### Off-site integrity check (v0.227.0, R-359/R-397)
**What it is:** a `restic check` against the off-site repository, run by the controller itself. Until
v0.227.0 nothing verified that the off-site copies were readable — the whole-guest tier had verify
jobs, the tier holding the customer's documents and photos had none.
| | |
|---|---|
| job | `offsite-integrity`, `sched.Daily` at **06:00** |
| cadence | **due-ness, not a weekday** — runs when the last SUCCESSFUL check is older than `monitoring.integrity.max_age_days` (default **7**). A box switched off on its check day is checked the next day it is on |
| depth | structure + index by default. `monitoring.integrity.read_data_subset` (default **empty**) adds `--read-data-subset=<spec>`; a malformed value is refused at read time with a WARN and treated as empty |
| guard | takes the single-writer flag and **SKIPS rather than waits** |
| timeout | 30 min (`integrityCheckTimeout`) — bounds a hung repository so it cannot pin the flag |
| by hand | `POST /api/debug/backup/integrity` — same code path, due-ness ignored, **every other guard intact** |
| result | persisted on `settings.OffboxTarget` (`last_integrity_check`, `last_integrity_ok`) and published on `OffboxReportStatus` |
**Three outcomes, not two.** `Skipped` (a sibling operation held the flag), `Unreachable` (the repo
could not be opened, or the check timed out) and failed are different facts. Only a failure notifies;
a skip and an unreachable repository do **not** advance due-ness, so tomorrow tries again. A failure
**does** advance it — re-checking a broken store nightly is load with no new information.
**Notifications.** `backup_integrity_ok` is severity `info`, which `severityNotifies` drops — it mails
nobody, by design. `backup_integrity_failed` is `error` and reaches the operator; the customer leg is
switchable and OFF by default. The customer gets a sentence; restic's output goes to the log,
truncated.
> **⚠ THE STRUCTURE CHECK DOES NOT CATCH SILENT CORRUPTION, and this is the thing to know before
> trusting it.** Measured on `demo-hp` 2026-08-30 against a throwaway repo whose pack was corrupted
> *without changing its size*: `restic check` returned **`no errors were found`, exit 0**; every
> `--read-data*` form returned `Pack ID does not match …` and exit 1. The structure check verifies the
> index, the pack inventory and the snapshot graph — it catches missing packs, broken indexes and
> unreadable snapshots — but it does **not** re-hash pack contents. Choosing the read-data depth is
> **R-399**, and the cost curve is measured:
> structure 35.0 s · 10% 35.9 s · 50% 37.3 s · **100% 39.2 s** on a 134.3 MB / 67-snapshot store.
> Those figures do not extrapolate: the structure check's cost tracks the index, read-data's tracks
> the data.
### Restore refusals (v0.226.0)
Three guards added on the off-site restore surface, all server-side:
@@ -2830,7 +2867,7 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
|---|---------|-----------|-------------|
| 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, network (guest-netns interfaces/route/DNS via the samba door, R-66; best-effort per item), scheduler, health, alerts. JSON download. |
| 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. |
| 3 | Mentés teszt | `POST /api/debug/backup/{dbdump,crossdrive,integrity,infra}` | Trigger individual backup phases independently. |
| 3 | Mentés teszt | `POST /api/debug/backup/dbdump` · `POST /api/debug/backup/integrity` | Trigger a DB dump, or run an off-site integrity check by hand. **`crossdrive` and `infra` are NOT implemented** — their buttons 404 (R-400). |
| 4 | Tárhely teszt | `POST /api/debug/storage/simulate-{disconnect,reconnect}`, `GET /api/debug/storage/watchdog-status` | Simulate drive disconnect/reconnect without unmounting. Per-path probe state with 5s auto-refresh. |
| 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,infra-push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. |
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |