v0.227.1: the damage classifier matched restic's ordinary progress output
gates / gates (push) Successful in 11s
gates / gates (push) Successful in 11s
A patch and not a rebuilt 0.227.0: that tag was already running on demo-hp, and re-pushing changed bytes under a live tag is the :latest hazard with extra steps. looksLikeRepositoryDamage matched bare "pack ", "tree ", "snapshot ", "blob ". A HEALTHY restic check prints "check all packs" and "check snapshots, trees and blobs" -- so any check that failed for a NON-damage reason, a connection dropped mid-run for instance, would have been classified as a corrupted repository and told the customer their backups may be damaged. That is the false alarm that teaches an operator to ignore the true one. Caught by the NEGATIVE control, built from the real bytes of a real passing check on demo-hp. The spec made the negative control mandatory and this is what it was for: a control that has only ever seen the failing case proves nothing. Signatures are now phrases from restic's own error wording. Also in this commit: CONTEXT.md records the three rulings (take the flag and skip, due-ness not a weekday, publish on OffboxReportStatus not the R-331 dead fields) plus the measurement a future session would otherwise assume wrongly -- THE STRUCTURE CHECK DOES NOT CATCH SILENT CORRUPTION. README documents the job, the route and the config, and corrects a line that listed four debug backup routes when only two exist. REUSE gains three rows, including one that records R-398 was my own mistake so nobody re-files it.
This commit is contained in:
+38
-1
@@ -1089,6 +1089,43 @@ backups/primary/<app>/
|
||||
maradtak."). **Every one is a claim about the BACKUP, never about the app** — see CONTEXT.md's ruling
|
||||
and 07-backup-architecture §6.3.
|
||||
|
||||
### Off-site integrity check (v0.227.0, R-359/R-397)
|
||||
|
||||
**What it is:** a `restic check` against the off-site repository, run by the controller itself. Until
|
||||
v0.227.0 nothing verified that the off-site copies were readable — the whole-guest tier had verify
|
||||
jobs, the tier holding the customer's documents and photos had none.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| job | `offsite-integrity`, `sched.Daily` at **06:00** |
|
||||
| cadence | **due-ness, not a weekday** — runs when the last SUCCESSFUL check is older than `monitoring.integrity.max_age_days` (default **7**). A box switched off on its check day is checked the next day it is on |
|
||||
| depth | structure + index by default. `monitoring.integrity.read_data_subset` (default **empty**) adds `--read-data-subset=<spec>`; a malformed value is refused at read time with a WARN and treated as empty |
|
||||
| guard | takes the single-writer flag and **SKIPS rather than waits** |
|
||||
| timeout | 30 min (`integrityCheckTimeout`) — bounds a hung repository so it cannot pin the flag |
|
||||
| by hand | `POST /api/debug/backup/integrity` — same code path, due-ness ignored, **every other guard intact** |
|
||||
| result | persisted on `settings.OffboxTarget` (`last_integrity_check`, `last_integrity_ok`) and published on `OffboxReportStatus` |
|
||||
|
||||
**Three outcomes, not two.** `Skipped` (a sibling operation held the flag), `Unreachable` (the repo
|
||||
could not be opened, or the check timed out) and failed are different facts. Only a failure notifies;
|
||||
a skip and an unreachable repository do **not** advance due-ness, so tomorrow tries again. A failure
|
||||
**does** advance it — re-checking a broken store nightly is load with no new information.
|
||||
|
||||
**Notifications.** `backup_integrity_ok` is severity `info`, which `severityNotifies` drops — it mails
|
||||
nobody, by design. `backup_integrity_failed` is `error` and reaches the operator; the customer leg is
|
||||
switchable and OFF by default. The customer gets a sentence; restic's output goes to the log,
|
||||
truncated.
|
||||
|
||||
> **⚠ THE STRUCTURE CHECK DOES NOT CATCH SILENT CORRUPTION, and this is the thing to know before
|
||||
> trusting it.** Measured on `demo-hp` 2026-08-30 against a throwaway repo whose pack was corrupted
|
||||
> *without changing its size*: `restic check` returned **`no errors were found`, exit 0**; every
|
||||
> `--read-data*` form returned `Pack ID does not match …` and exit 1. The structure check verifies the
|
||||
> index, the pack inventory and the snapshot graph — it catches missing packs, broken indexes and
|
||||
> unreadable snapshots — but it does **not** re-hash pack contents. Choosing the read-data depth is
|
||||
> **R-399**, and the cost curve is measured:
|
||||
> structure 35.0 s · 10% 35.9 s · 50% 37.3 s · **100% 39.2 s** on a 134.3 MB / 67-snapshot store.
|
||||
> Those figures do not extrapolate: the structure check's cost tracks the index, read-data's tracks
|
||||
> the data.
|
||||
|
||||
### Restore refusals (v0.226.0)
|
||||
|
||||
Three guards added on the off-site restore surface, all server-side:
|
||||
@@ -2830,7 +2867,7 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
|
||||
|---|---------|-----------|-------------|
|
||||
| 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, network (guest-netns interfaces/route/DNS via the samba door, R-66; best-effort per item), scheduler, health, alerts. JSON download. |
|
||||
| 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. |
|
||||
| 3 | Mentés teszt | `POST /api/debug/backup/{dbdump,crossdrive,integrity,infra}` | Trigger individual backup phases independently. |
|
||||
| 3 | Mentés teszt | `POST /api/debug/backup/dbdump` · `POST /api/debug/backup/integrity` | Trigger a DB dump, or run an off-site integrity check by hand. **`crossdrive` and `infra` are NOT implemented** — their buttons 404 (R-400). |
|
||||
| 4 | Tárhely teszt | `POST /api/debug/storage/simulate-{disconnect,reconnect}`, `GET /api/debug/storage/watchdog-status` | Simulate drive disconnect/reconnect without unmounting. Per-path probe state with 5s auto-refresh. |
|
||||
| 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,infra-push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. |
|
||||
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
|
||||
|
||||
Reference in New Issue
Block a user