v0.228.0 — the off-site check reads the data; the debug page stops lying (R-399 + R-400)
gates / gates (push) Successful in 12s

R-399: monitoring.integrity.read_data_subset defaults to 100%. A pack damaged
without changing its size made plain `restic check` report "no errors were found"
on demo-hp 2026-08-30; every read-data form caught it. Cost on that 134 MB store:
35.0s structure vs 39.2s at 100%. "off" (any case) is the off token; empty means
not-configured, therefore the default; a malformed value falls back to the DEFAULT,
never to structure. A completed check over 5 minutes logs a WARN naming the
duration, the depth and R-401 — operator log only, no hub event, no depth change.
The depth is now recorded with the verdict (LastIntegrityDepth; empty = NOT
RECORDED, never "structure").

R-400: 24 debug-page references, 17 dispatched, 7 dead — three of which fetched on
page LOAD, so those panels were permanently blank. backup/crossdrive implemented;
backup/infra, hub/infra-push, dr/infra-status, storage/watchdog-status and both
storage/simulate-* deleted with their panels and JavaScript.
scripts/debug_route_gate.py fails in both directions and is registered after the
seven were resolved. 18 referenced, 18 dispatched, none orphaned.

Corrections: the dead-field warning in report/types.go said the controller runs no
integrity check and the notifiers are called from nowhere — both false since
v0.227.0. controller.yaml.example gains its missing integrity: block.
integrityCheckTimeout's "ships OFF" comment rewritten.
This commit is contained in:
2026-08-31 10:24:29 +02:00
parent 300d7e87d7
commit 3c49dc8ea4
23 changed files with 1144 additions and 175 deletions
+10 -7
View File
@@ -1099,11 +1099,13 @@ jobs, the tier holding the customer's documents and photos had none.
|---|---|
| job | `offsite-integrity`, `sched.Daily` at **06:00** |
| cadence | **due-ness, not a weekday** — runs when the last SUCCESSFUL check is older than `monitoring.integrity.max_age_days` (default **7**). A box switched off on its check day is checked the next day it is on |
| depth | structure + index by default. `monitoring.integrity.read_data_subset` (default **empty**) adds `--read-data-subset=<spec>`; a malformed value is refused at read time with a WARN and treated as empty |
| depth | **`--read-data-subset=100%` by default since v0.228.0 (R-399)** — the check re-reads and re-hashes every stored byte, not just the catalogue. `monitoring.integrity.read_data_subset`: absent or empty = the default; **`off`** (any case) = structure and index only; any form restic accepts (`10%`, `1/7`, `50M`) = itself. A malformed value WARNs and falls back to **the default**, never to `off` — a typo must not quietly remove the protection |
| why full depth | **the structure check does not detect a size-preserving pack corruption.** Measured on `demo-hp` 2026-08-30: a pack was damaged without changing its size, plain `restic check` reported `no errors were found` and exited clean, every read-data form caught it. Cost on that 134 MB store: 35.0 s structure vs **39.2 s** at 100% |
| slow notice | a **completed** check over `integritySlowNoticeThreshold` (**5 min**) logs a WARN naming the duration, the depth and **R-401**. Operator log only — no hub event, no customer alarm, and it never changes the depth by itself. A skip or an unreachable store never warns: neither has a duration to judge |
| guard | takes the single-writer flag and **SKIPS rather than waits** |
| timeout | 30 min (`integrityCheckTimeout`) — bounds a hung repository so it cannot pin the flag |
| by hand | `POST /api/debug/backup/integrity` — same code path, due-ness ignored, **every other guard intact** |
| result | persisted on `settings.OffboxTarget` (`last_integrity_check`, `last_integrity_ok`) and published on `OffboxReportStatus` |
| result | persisted on `settings.OffboxTarget` (`last_integrity_check`, `last_integrity_ok`, **`last_integrity_depth`** — v0.228.0) and published on `OffboxReportStatus`. Depth empty = NOT RECORDED (a pre-0.228.0 controller), never "structure" |
**Three outcomes, not two.** `Skipped` (a sibling operation held the flag), `Unreachable` (the repo
could not be opened, or the check timed out) and failed are different facts. Only a failure notifies;
@@ -2859,7 +2861,9 @@ The Hub serves three asset types per app:
### 13. Debug Mode
When `logging.level: "debug"` is set in `controller.yaml`, the controller exposes a full diagnostic dashboard at `/debug` with 9 testing sections. All debug endpoints are gated — at `info` level, the sidebar link disappears and all `/api/debug/*` routes return 404.
When `logging.level: "debug"` is set in `controller.yaml`, the controller exposes a full diagnostic dashboard at `/debug`. All debug endpoints are gated — at `info` level, the sidebar link disappears and all `/api/debug/*` routes return 404.
**R-400 (v0.228.0): the table below is now MECHANICALLY pinned to the dispatcher.** `controller/scripts/debug_route_gate.py` compares every `/api/debug/...` reference in `debug.html` against every `subpath ==` case in `handler_debug.go` and fails on either difference. Before it existed the page referenced 24 addresses and 17 were answered; three of the seven dead ones fetched on page LOAD, so whole panels had been permanently blank. Six controls were deleted and one (`backup/crossdrive`) implemented — the "Tárhely teszt" section went entirely, which is why the section numbers below skip 4.
#### Debug Page Sections
@@ -2867,12 +2871,11 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
|---|---------|-----------|-------------|
| 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, network (guest-netns interfaces/route/DNS via the samba door, R-66; best-effort per item), scheduler, health, alerts. JSON download. |
| 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. |
| 3 | Mentés teszt | `POST /api/debug/backup/dbdump` · `POST /api/debug/backup/integrity` | Trigger a DB dump, or run an off-site integrity check by hand. **`crossdrive` and `infra` are NOT implemented** — their buttons 404 (R-400). |
| 4 | Tárhely teszt | `POST /api/debug/storage/simulate-{disconnect,reconnect}`, `GET /api/debug/storage/watchdog-status` | Simulate drive disconnect/reconnect without unmounting. Per-path probe state with 5s auto-refresh. |
| 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,infra-push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. |
| 3 | Mentés teszt | `POST /api/debug/backup/dbdump` · `POST /api/debug/backup/crossdrive` · `POST /api/debug/backup/integrity` | Trigger a DB dump, run the Tier-2 (cross-drive) sweep over every deployed HDD-backed app, or run an off-site integrity check by hand. `crossdrive` is asynchronous and answers with the app list it started for; `integrity` is synchronous and answers with the verdict. **`backup/infra` was DELETED (R-400)** — the disk-tier infra backup moved to the host agent in slice 8C and nothing in this repo backs it. |
| 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. **`hub/infra-push` was DELETED (R-400)** — `Pusher.PushInfraBackup` was removed 2026-06-16. |
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
| 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. |
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). |
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup` | Trigger setup mode via marker file (requires "RESET"). **`dr/infra-status` and its panel were DELETED (R-400)** — it rendered the two retired infra-backup mechanisms above, and it fetched on page LOAD, so the panel had been permanently blank. |
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
#### Key Implementation Details