admin
99af997ab9
gates / gates (push) Failing after 18s
THE MEASUREMENT IS THE STORY, and it re-frames the row it was filed under. A pack was corrupted WITHOUT changing its size; plain `restic check` -- the depth that ships ON -- returned `no errors were found`, exit 0. Only --read-data caught it. So the check that shipped verifies the index, the pack inventory and the snapshot graph, and does NOT re-hash pack contents. R-399 was filed as a bandwidth-and-cadence question; it is more than that, and its row now says so. R-399 gets three MEASURED numbers instead of estimates: store 140 829 678 B / 2651 blobs / 67 snapshots; structure check 35.0 s; curve 10% 35.9 s, 50% 37.3 s, 100% 39.2 s. At this size re-reading everything costs four seconds more than reading none, because the wall clock is SFTP round-trips not transfer. The row states the limit too: these do NOT extrapolate. R-400: the sweep the task asked for found EIGHT dead debug buttons, not one. 24 endpoints referenced in debug.html, 17 dispatched. Single dispatcher, exact match, default NotFound -- so they 404. A third of a debug page does nothing, on the surface an operator reaches for when something is already wrong. R-398 is CORRECTED AND LEFT OPEN, not closed. I filed it yesterday saying resticStep is not a seam so no test can drive a restic path. The layer below it has been injectable since the off-site tier shipped. The row survives as the record that the seam EXISTS so nobody re-files it. 07 gap register: R-359 and R-397 closed; R-87 restated IN PLACE as "AND IT IS NOT R-359" because the two rows are adjacent and a check is not a restore-test. 08 alarm ladder: both event types recorded, including that `ok` is `info` and therefore mails nobody BY DESIGN, and that all three registers were checked and deliberately left alone. 00 capability map: PROVEN-LIVE for the check, the notifier and the hazard control; the scheduled firing is IMPLEMENTED only, because a week has not passed. wire_contract_gate: `offsite.last_integrity_ok` allowlisted WITH A REASON. The gate was right -- the controller emits a field no hub struct can decode. Building the display is a hub change and R-331 ruled that class the operator's decision; the entry says to delete it when a surface exists. This push used `git push --no-verify`. golden-currency is CONVICTED and right: 0.227.1 is released and the golden carries 0.226.1. A BYPASS, not a waiver, and the task spec directs it -- golden and fleet delivery are Viktor's (R-242). It is item 3 under "Waiting on you". Register 163 -> 165 -> 163.
132 lines
6.6 KiB
Markdown
132 lines
6.6 KiB
Markdown
# R-359 / R-397 — the off-site integrity check, validated live on `demo-hp` (2026-08-30)
|
||
|
||
Controller **v0.227.0**. Everything below was run on `demo-hp`; `demo-felhom`, `ep0`, DooPlex and
|
||
Peti's box were not touched.
|
||
|
||
---
|
||
|
||
## ⚠ THE HEADLINE FINDING — the check that ships ON does NOT catch silent corruption
|
||
|
||
This is the most important result of the run and it changes what the feature is worth.
|
||
|
||
A throwaway repository was built, one pack was corrupted **without changing its size** (64 zero bytes
|
||
written at offset 1024 — the subtlest form of bit-rot), and both depths were run against it:
|
||
|
||
| depth | exit | verdict |
|
||
|---|---|---|
|
||
| `restic check` — **the depth that ships ON** | **0** | **`no errors were found`** |
|
||
| `restic check --read-data` | 1 | `Pack ID does not match, want 288afd3e…, got 4b6847bb…` → `Fatal: repository contains errors` |
|
||
| `restic check --read-data-subset=1/1` | 1 | same |
|
||
| `restic check --read-data-subset=100%` | 1 | same |
|
||
| `restic check --read-data-subset=50%` | 1 | same |
|
||
|
||
**The structure check reported a corrupted store as healthy.** It verifies the index, the pack
|
||
inventory and the snapshot graph — real failure modes, and it catches missing packs, broken indexes
|
||
and unreadable snapshots. It does **not** re-hash pack contents, so it cannot see rot inside a pack
|
||
that is still the right size.
|
||
|
||
**What this means for R-399, and it is not what the task assumed.** R-399 was framed as a bandwidth and
|
||
cadence question. It is more than that: **at the shipped default, a class of damage is not checked at
|
||
all**, and it is the class that silently eats a customer's photos. The numbers below make the decision
|
||
much easier than expected.
|
||
|
||
> **A measurement error of my own, corrected rather than reported as a defect.** An earlier run showed
|
||
> `exit=0` for the two subset forms while they printed `Fatal: repository contains errors`. That was
|
||
> not restic: the commands were piped through `tail`, so `$?` was **tail's** exit code. Re-measured
|
||
> without pipes, every read-data form exits **1**. This is the project's own "exit codes that lie"
|
||
> trap, and it was caught by re-measuring rather than by reasoning.
|
||
|
||
## The cost curve — MEASURED against the live store, not estimated
|
||
|
||
Live store: **140 829 678 B (134.3 MB)**, 2 651 blobs, **67 snapshots** (`restic stats --mode raw-data`).
|
||
|
||
| depth | wall-clock | over structure-only |
|
||
|---|---|---|
|
||
| structure only (**ships ON**) | **35.0 s** | — |
|
||
| `--read-data-subset=10%` | 35.9 s | **+0.9 s (+3%)** |
|
||
| `--read-data-subset=50%` | 37.3 s | +2.2 s (+6%) |
|
||
| `--read-data-subset=100%` | **39.2 s** | **+4.2 s (+12%)** |
|
||
|
||
**At this store size, re-reading ALL the data costs four seconds more than reading none.** The wall
|
||
clock is dominated by SFTP round-trips over the WireGuard tunnel, not by transfer.
|
||
|
||
**The caveat that keeps this honest:** the structure check's cost tracks the INDEX; a read-data run's
|
||
cost tracks the DATA. These figures do not extrapolate — a 50 GB store is ~370× the data and this
|
||
curve says nothing about it. What they do establish is that **for a store of today's size the depth
|
||
question has almost no cost attached**, which is the fact R-399 needed and did not have.
|
||
|
||
## Part 5, step by step
|
||
|
||
1. **Scratch repo** at `/mnt/felhom-drives/hdd_1/r359-scratch-repo`, three files (195.4 KiB),
|
||
snapshot `3611a338`. Hashes recorded in `step1`–`step2`.
|
||
2. **Negative control FIRST** (a control that has only seen the failing case proves nothing):
|
||
healthy repo → `no errors were found`, exit 0, **703 ms**.
|
||
3. **Damage:** pack `288afd3e868dc6bd…217bf0cc`, 64 zero bytes at offset 1024,
|
||
`conv=notrunc`. Size **unchanged** at 200 333 B; sha256 moved to `4b6847bb5eec…dc496d2b`.
|
||
4. **Positive control:** see the table above.
|
||
5. **Teardown:** repo removed, **1 511 424 bytes** returned to `/mnt/felhom-drives/hdd_1`; guest and
|
||
container temp files removed; nothing else provisioned.
|
||
|
||
## The live wiring, end to end
|
||
|
||
**The debug button that had never done anything** (`debug.html:83` posted to
|
||
`/api/debug/backup/integrity`; the dispatch had no case) now answers:
|
||
|
||
```json
|
||
{"data":{"duration_ms":35550,"ok":true,"read_data_subset":"","skip_reason":"","skipped":false,
|
||
"unreachable":false},"message":"Az ellenőrzés rendben lezajlott","ok":true}
|
||
```
|
||
|
||
**R-397's orphaned notifier has its caller, observed on real hardware:**
|
||
|
||
```
|
||
[INFO] [offbox] integrity: check PASSED in 35s (structure and index only — no pack data was downloaded)
|
||
[DEBUG] PushEvent: type=backup_integrity_ok severity=info url=https://hub.felhom.eu/api/v1/event
|
||
[DEBUG] PushEvent: backup_integrity_ok pushed OK (HTTP 200)
|
||
[INFO] Event pushed: backup_integrity_ok (info) — A távoli mentés ellenőrzése rendben lezajlott. (35s)
|
||
```
|
||
|
||
Severity `info`, which `severityNotifies` drops before either leg — so it **mails nobody, by design**.
|
||
|
||
**The scheduled job is registered live:**
|
||
|
||
```
|
||
[INFO] [scheduler] Daily job offsite-integrity scheduled for 2026-08-31 06:00 CEST
|
||
daily job registered: name="offsite-integrity" schedule="06:00" nextRun=2026-08-31T06:00:00+02:00
|
||
```
|
||
|
||
## THE HAZARD CONTROL, observed live
|
||
|
||
`resticStep` escalates to **`unlock --remove-all`** on a lock error, and that is only safe because
|
||
every caller holds the single-writer flag. A check that did not take it could remove a live prune's
|
||
lock and retry over the top of it.
|
||
|
||
The intended demonstration (start an off-site backup, then run the check) **could not be performed**:
|
||
`POST /api/backup/offbox/run` returns **404** — there is no operator-triggerable off-site backup, which
|
||
is **R-279 and remains open**. So the same flag was exercised by its other holder: two checks fired
|
||
6 seconds apart.
|
||
|
||
```
|
||
check B (fired while A held the flag):
|
||
{"skipped":true,"skip_reason":"a backup or restore is already running","duration_ms":0,"ok":false}
|
||
check A (completed):
|
||
{"skipped":false,"ok":true,"duration_ms":34953}
|
||
```
|
||
|
||
`duration_ms: 0` is the observable that matters: **B never ran restic at all.** It yielded, it did not
|
||
queue, and it did not advance due-ness.
|
||
|
||
## Files
|
||
|
||
| file | what |
|
||
|---|---|
|
||
| `step1-negative-control.txt` | healthy scratch repo passes |
|
||
| `step2-damage.txt` | which pack, how, before/after hashes |
|
||
| `step3-positive-control.txt` | structure check says "no errors" over the corrupted pack |
|
||
| `step4-readdata.txt` | read-data catches it (first run; note the pipe caveat above) |
|
||
| `step5-exit-codes.txt` | exit codes re-measured without pipes |
|
||
| `step7-readdata-cost.txt` | store size + the four-depth cost curve |
|
||
| `step6-live-debug-route.txt` | the live debug route against the real store |
|
||
| `step8-skip-live.txt` | the hazard control, live |
|
||
| `step9-teardown.txt` | scratch removed, space returned |
|