v0.228.0 — the off-site check reads the data; the debug page stops lying (R-399 + R-400)
gates / gates (push) Successful in 12s

R-399: monitoring.integrity.read_data_subset defaults to 100%. A pack damaged
without changing its size made plain `restic check` report "no errors were found"
on demo-hp 2026-08-30; every read-data form caught it. Cost on that 134 MB store:
35.0s structure vs 39.2s at 100%. "off" (any case) is the off token; empty means
not-configured, therefore the default; a malformed value falls back to the DEFAULT,
never to structure. A completed check over 5 minutes logs a WARN naming the
duration, the depth and R-401 — operator log only, no hub event, no depth change.
The depth is now recorded with the verdict (LastIntegrityDepth; empty = NOT
RECORDED, never "structure").

R-400: 24 debug-page references, 17 dispatched, 7 dead — three of which fetched on
page LOAD, so those panels were permanently blank. backup/crossdrive implemented;
backup/infra, hub/infra-push, dr/infra-status, storage/watchdog-status and both
storage/simulate-* deleted with their panels and JavaScript.
scripts/debug_route_gate.py fails in both directions and is registered after the
seven were resolved. 18 referenced, 18 dispatched, none orphaned.

Corrections: the dead-field warning in report/types.go said the controller runs no
integrity check and the notifiers are called from nowhere — both false since
v0.227.0. controller.yaml.example gains its missing integrity: block.
integrityCheckTimeout's "ships OFF" comment rewritten.
This commit is contained in:
2026-08-31 10:24:29 +02:00
parent 300d7e87d7
commit 3c49dc8ea4
23 changed files with 1144 additions and 175 deletions
+6 -4
View File
@@ -7,10 +7,12 @@ paths: ["controller/**/*.go", "controller/**/*.html", "controller/**/*.css", "co
## The ONE entry point ## The ONE entry point
**Run `python3 controller/scripts/controller_gates.py` (from `controller/`) after ANY change in this **Run `python3 controller/scripts/controller_gates.py` (from `controller/`) after ANY change in this
repo.** It runs all seven local gates — `template_id_gate`, `emoji_gate`, `native_confirm_gate`, repo.** It runs the local gates — `template_id_gate`, `emoji_gate`, `native_confirm_gate`,
`offbox_rename_gate`, `app_row_dedup_gate`, `mojibake_gate`, `docker_run_volume_path_gate` — plus `offbox_rename_gate`, `app_row_dedup_gate`, `mojibake_gate`, `docker_run_volume_path_gate`,
`reuse_refs_check` and `instructions_gate` on the repo root, streaming each gate's own output and `secret_in_markup_gate`, `retrieval_promise_gate`, `debug_route_gate` — plus `reuse_refs_check`,
exiting non-zero if any fails. `instructions_gate` and `observations_gate` on the repo root, streaming each gate's own output and
exiting non-zero if any fails. **The runner's `GATES` table is the list; this sentence is a pointer to
it, not a second copy** — it has already drifted once (it said "seven" while nine were registered).
- `--fast` selects the gates that touch no network and no container runtime; today that is all of them. - `--fast` selects the gates that touch no network and no container runtime; today that is all of them.
- **A missing gate script is a FAILURE, never a skip.** - **A missing gate script is a FAILURE, never a skip.**
+83
View File
@@ -1,3 +1,86 @@
## v0.228.0 — the check reads the data, and the debug page stops lying (2026-08-31, R-399 + R-400)
**MinAgent: 0.129.0** (unchanged)
### The off-site check now re-reads the stored data (R-399)
`monitoring.integrity.read_data_subset` **defaults to `100%`**. Every box whose `controller.yaml` has
no `integrity:` block — which is every box in the fleet — now downloads and re-hashes the whole
off-site store on its weekly check instead of reading only the catalogue.
**The fact that forced it:** on `demo-hp`, 2026-08-30, a pack was damaged **without changing its
size**. Plain `restic check` — the structure-and-index check every box ran — reported
`no errors were found` and exited clean. Every `--read-data*` form caught it. A structurally-verified
store is one whose rot is found at restore time, with a customer waiting.
**The cost**, same store and day (140 829 678 B / 2 651 blobs / 67 snapshots): structure 35.0 s,
10% 35.9 s, 50% 37.3 s, **100% 39.2 s**.
- **`off` (any case) is the new off token.** Absent or empty means *not configured*, therefore the
default; a setting with no off switch is not a setting, and emptiness cannot mean both things.
- **A malformed value falls back to the DEFAULT, never to structure.** Downgrading on a typo would
silently remove the protection this row adds — R-357's shape, a guard that opens quietly.
- **A completed check over 5 minutes logs a WARN** naming the duration, the depth and R-401. Operator
log only: no hub event, no customer alarm, and it never changes the depth by itself. A skip or an
unreachable store never warns — neither has a duration to judge. The threshold is deliberately
imprecise (≈7.6× the only full-depth number that exists) because a notice changes no behaviour,
while a precise number invented from one measurement on one 134 MB store would not.
- **The depth is now recorded with the verdict** — `settings.OffboxTarget.LastIntegrityDepth` and
`OffboxReportStatus.LastIntegrityDepth`. Empty means NOT RECORDED (a pre-0.228.0 controller), never
"structure": absence means the box cannot answer, following the `StatsKnown` precedent beside it.
- **R-401 filed with a TRIGGER, not a date:** revisit the depth when the timing WARN fires on any box.
No rotation schedule, size threshold or bandwidth budget is built here — every one of those would be
a number invented from a single data point.
**R-87 (the restic tier is never restore-tested) stays OPEN.** Reading the bytes back out of the store
is not a restore.
### The debug page stops lying (R-400)
The shipped debug page referenced **24** `/api/debug/...` addresses and the dispatcher answered **17**.
Seven controls did nothing — and three of those seven were not buttons at all: `dr/infra-status` and
both `storage/watchdog-status` calls fetch on page **load**, so whole panels had been permanently
blank and nobody had to click anything to be misled. This is the page an operator opens when something
is already wrong.
| control | disposition | why |
|---|---|---|
| `backup/crossdrive` | **IMPLEMENTED** | `Manager.RunTier2` is live; only the route was missing |
| `backup/infra` | **DELETED** | the disk-tier infra backup moved to the host agent in slice 8C |
| `hub/infra-push` | **DELETED** | `Pusher.PushInfraBackup` was removed 2026-06-16 (it pushed plaintext secrets) |
| `dr/infra-status` | **DELETED** | it rendered the two retired mechanisms above; fetched on page load |
| `storage/watchdog-status` | **DELETED** | the slice-8C watchdog is retired; the drive-gate reconcile replaced it and publishes no such status. Fetched on page load, twice |
| `storage/simulate-disconnect` | **DELETED** | no backing capability, and it WRITES storage state — a button that fakes a drive disconnect on a customer's machine is a foot-gun |
| `storage/simulate-reconnect` | **DELETED** | same |
Each deletion took its panel and its JavaScript with it; the "Tárhely teszt" section went entirely.
A panel left behind renders nothing forever, which is how this class hides.
**`controller/scripts/debug_route_gate.py` makes the class impossible.** Two lists and a difference: it
fails when the template references an address the dispatcher lacks, **and** when the dispatcher answers
one nothing references. Registered in `controller_gates.py` **after** the seven were resolved — a
registered-but-failing gate refuses every push. Ten lines on purpose. Red-proofed in both directions.
Tree after the change: **18 referenced addresses, 18 dispatched, none orphaned.**
### Three corrections
- `internal/report/types.go` — the dead-field warning block said *"the controller runs no integrity
check, and `NotifyIntegrityOK`/`NotifyIntegrityFailed` … are called from nowhere"*. **Both halves
became false in v0.227.0.** The fields stay dead and unrendered (`TestBackupReport_DeadFieldsStayZero`
still passes unmodified); only the *reason* changed.
- `configs/controller.yaml.example` had **no `integrity:` block at all** — an operator could not
discover the setting exists. Added, with both keys, the default, the off token and the measurement.
- `internal/backup/offbox_integrity.go` — `integrityCheckTimeout`'s comment said read-data "ships OFF
… whoever turns it on must revisit this number". Rewritten: it is now the number a large store meets
first, and the slow notice exists to say so long before it does.
### Superseded tests
Two R-359 tests asserted the ruling this release reverses, and are replaced rather than weakened.
`TestR359_StructureCheckPassesNoReadDataFlag` → `TestR399_AbsentConfigRunsFullDepth`.
`TestR359_MalformedReadDataSubsetIsTreatedAsOff` → `TestR399_MalformedFallsBackToTheDefault` (its NAME
was the defect: treating a typo as "off" is the quiet downgrade). Both are recorded in place, so a
later reader does not re-derive the old ruling from an absence.
## v0.227.1 — the damage classifier matched restic's ordinary progress output (2026-08-30, R-359 follow-on) ## v0.227.1 — the damage classifier matched restic's ordinary progress output (2026-08-30, R-359 follow-on)
**MinAgent: 0.129.0** (unchanged) **MinAgent: 0.129.0** (unchanged)
+40 -1
View File
@@ -7,7 +7,46 @@
> >
> Ask Claude Code: "Please update CONTEXT.md with what we did today" > Ask Claude Code: "Please update CONTEXT.md with what we did today"
Last updated: 2026-08-30 (v0.227.1 — R-359/R-397: the off-site store gets checked) Last updated: 2026-08-31 (v0.228.0 — R-399/R-400: the check reads the data, the debug page stops lying)
> **2026-08-31 — v0.228.0. TWO RULINGS, recorded so neither is re-litigated.**
>
> **1. The off-site integrity check ships at FULL depth — `--read-data-subset=100%` — and `off` is the
> way back.** Viktor's ruling, 2026-08-31, taken on a measurement rather than a claim: on `demo-hp`,
> 2026-08-30, a pack damaged **without changing its size** made plain `restic check` report
> `no errors were found` and exit clean; every read-data form caught it. Cost on that store
> (140 829 678 B / 2 651 blobs / 67 snapshots): structure 35.0 s, 10% 35.9 s, 50% 37.3 s, 100% 39.2 s.
>
> Three consequences that are decided, not open:
> - **Empty means "not configured", therefore the default.** It does NOT mean off. `off` (any case) is
> the off token, and it exists because a setting with no off switch is not a setting.
> - **A malformed value falls back to the DEFAULT, never to structure.** Falling back to structure
> would silently remove the protection on a typo — R-357's shape, a guard that opens quietly.
> - **The default lives in `internal/backup/offbox_integrity.go`, NOT in `config.applyDefaults`.** Both
> integrity defaults are resolved in one accessor each, beside the argument that justifies them;
> symmetry with the other `Monitoring` defaults is worth less than that.
>
> **The thing a future session will get wrong: there is exactly ONE data point, on a 134 MB store.**
> The cost curves are governed by different quantities — structure tracks the index, read-data tracks
> the data — so nothing here extrapolates. That is why v0.228.0 ships a *notice* (a WARN over 5 minutes
> naming R-401) and NOT a rotation schedule, a size threshold or a bandwidth budget. Every one of those
> would be a number invented from one measurement, which is the shape of the four production designs
> this project has already specced against nothing. **R-401's trigger is that WARN firing on any box,
> not a calendar date.**
>
> **2. Implement or delete FIRST, register the gate SECOND.** R-400 found seven debug-page controls
> with no handler. The gate that makes that impossible (`controller/scripts/debug_route_gate.py`) was
> written and registered only after all seven were resolved — a registered-but-failing gate refuses
> every push, exactly as `instructions_gate` established. The gate is deliberately ten lines: two lists
> and a difference, in both directions, because a cleverer gate needs maintaining and an unmaintained
> gate is how the class hides in the first place. **Keep `handleDebugAPI`'s exact-match switch with its
> `NotFound` default** — a prefix match would have made the original defect invisible instead of merely
> silent.
>
> **The shape worth remembering is worse than "seven dead buttons":** three of the seven fetched on
> page LOAD, so those panels were permanently blank on the page an operator opens when something is
> already wrong.
> **2026-08-30 — v0.227.0/v0.227.1. THREE RULINGS, recorded so none is re-litigated.** > **2026-08-30 — v0.227.0/v0.227.1. THREE RULINGS, recorded so none is re-litigated.**
> >
+1
View File
@@ -310,6 +310,7 @@ Cross-repo edges:
## 5. Extension points (where new features plug in) ## 5. Extension points (where new features plug in)
- **New storage web endpoint**: switch in `ServeStorageAPI` (controller/internal/web/storage_handlers.go); disk ops in `ServeDiskAPI` (controller/internal/web/agent_disk_handlers.go); backup in `ServeBackupAPI`; export in `ServeExportAPI`; debug in `handleDebugAPI` (debug-mode gated). - **New storage web endpoint**: switch in `ServeStorageAPI` (controller/internal/web/storage_handlers.go); disk ops in `ServeDiskAPI` (controller/internal/web/agent_disk_handlers.go); backup in `ServeBackupAPI`; export in `ServeExportAPI`; debug in `handleDebugAPI` (debug-mode gated).
- **New debug-page control (R-400, v0.228.0)**: the control in `controller/internal/web/templates/debug.html` AND the `subpath == "…"` case in `controller/internal/web/handler_debug.go` are ONE change — `controller/scripts/debug_route_gate.py` fails on either half alone, in both directions (a reference with no case, and a case with no reference). Keep the dispatcher's EXACT-match switch with its `http.NotFound` default; a prefix match is what would have hidden the original defect. Handler shape: `debugTriggerDBDump` for a fire-and-forget trigger, `debugRunIntegrityCheck` for one the operator pressed to learn an ANSWER (synchronous). **Before this gate the page referenced 24 addresses and 17 were answered, and three of the seven dead ones fetched on page LOAD** — those panels were permanently blank on the page an operator opens when something is already wrong.
- **New REST endpoint**: path dispatch in `Router.ServeHTTP` (controller/internal/api/router.go); use `writeJSON` + `limitBody`. - **New REST endpoint**: path dispatch in `Router.ServeHTTP` (controller/internal/api/router.go); use `writeJSON` + `limitBody`.
- **New background job**: `sched.Every`/`sched.Daily` registration block in controller/cmd/controller/main.go. - **New background job**: `sched.Every`/`sched.Daily` registration block in controller/cmd/controller/main.go.
- **New template function**: `Server.templateFuncMap` (controller/internal/web/funcmap.go) — obey v2 state-suffix vocabulary. - **New template function**: `Server.templateFuncMap` (controller/internal/web/funcmap.go) — obey v2 state-suffix vocabulary.
+10 -7
View File
@@ -1099,11 +1099,13 @@ jobs, the tier holding the customer's documents and photos had none.
|---|---| |---|---|
| job | `offsite-integrity`, `sched.Daily` at **06:00** | | job | `offsite-integrity`, `sched.Daily` at **06:00** |
| cadence | **due-ness, not a weekday** — runs when the last SUCCESSFUL check is older than `monitoring.integrity.max_age_days` (default **7**). A box switched off on its check day is checked the next day it is on | | cadence | **due-ness, not a weekday** — runs when the last SUCCESSFUL check is older than `monitoring.integrity.max_age_days` (default **7**). A box switched off on its check day is checked the next day it is on |
| depth | structure + index by default. `monitoring.integrity.read_data_subset` (default **empty**) adds `--read-data-subset=<spec>`; a malformed value is refused at read time with a WARN and treated as empty | | depth | **`--read-data-subset=100%` by default since v0.228.0 (R-399)** — the check re-reads and re-hashes every stored byte, not just the catalogue. `monitoring.integrity.read_data_subset`: absent or empty = the default; **`off`** (any case) = structure and index only; any form restic accepts (`10%`, `1/7`, `50M`) = itself. A malformed value WARNs and falls back to **the default**, never to `off` — a typo must not quietly remove the protection |
| why full depth | **the structure check does not detect a size-preserving pack corruption.** Measured on `demo-hp` 2026-08-30: a pack was damaged without changing its size, plain `restic check` reported `no errors were found` and exited clean, every read-data form caught it. Cost on that 134 MB store: 35.0 s structure vs **39.2 s** at 100% |
| slow notice | a **completed** check over `integritySlowNoticeThreshold` (**5 min**) logs a WARN naming the duration, the depth and **R-401**. Operator log only — no hub event, no customer alarm, and it never changes the depth by itself. A skip or an unreachable store never warns: neither has a duration to judge |
| guard | takes the single-writer flag and **SKIPS rather than waits** | | guard | takes the single-writer flag and **SKIPS rather than waits** |
| timeout | 30 min (`integrityCheckTimeout`) — bounds a hung repository so it cannot pin the flag | | timeout | 30 min (`integrityCheckTimeout`) — bounds a hung repository so it cannot pin the flag |
| by hand | `POST /api/debug/backup/integrity` — same code path, due-ness ignored, **every other guard intact** | | by hand | `POST /api/debug/backup/integrity` — same code path, due-ness ignored, **every other guard intact** |
| result | persisted on `settings.OffboxTarget` (`last_integrity_check`, `last_integrity_ok`) and published on `OffboxReportStatus` | | result | persisted on `settings.OffboxTarget` (`last_integrity_check`, `last_integrity_ok`, **`last_integrity_depth`** — v0.228.0) and published on `OffboxReportStatus`. Depth empty = NOT RECORDED (a pre-0.228.0 controller), never "structure" |
**Three outcomes, not two.** `Skipped` (a sibling operation held the flag), `Unreachable` (the repo **Three outcomes, not two.** `Skipped` (a sibling operation held the flag), `Unreachable` (the repo
could not be opened, or the check timed out) and failed are different facts. Only a failure notifies; could not be opened, or the check timed out) and failed are different facts. Only a failure notifies;
@@ -2859,7 +2861,9 @@ The Hub serves three asset types per app:
### 13. Debug Mode ### 13. Debug Mode
When `logging.level: "debug"` is set in `controller.yaml`, the controller exposes a full diagnostic dashboard at `/debug` with 9 testing sections. All debug endpoints are gated — at `info` level, the sidebar link disappears and all `/api/debug/*` routes return 404. When `logging.level: "debug"` is set in `controller.yaml`, the controller exposes a full diagnostic dashboard at `/debug`. All debug endpoints are gated — at `info` level, the sidebar link disappears and all `/api/debug/*` routes return 404.
**R-400 (v0.228.0): the table below is now MECHANICALLY pinned to the dispatcher.** `controller/scripts/debug_route_gate.py` compares every `/api/debug/...` reference in `debug.html` against every `subpath ==` case in `handler_debug.go` and fails on either difference. Before it existed the page referenced 24 addresses and 17 were answered; three of the seven dead ones fetched on page LOAD, so whole panels had been permanently blank. Six controls were deleted and one (`backup/crossdrive`) implemented — the "Tárhely teszt" section went entirely, which is why the section numbers below skip 4.
#### Debug Page Sections #### Debug Page Sections
@@ -2867,12 +2871,11 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
|---|---------|-----------|-------------| |---|---------|-----------|-------------|
| 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, network (guest-netns interfaces/route/DNS via the samba door, R-66; best-effort per item), scheduler, health, alerts. JSON download. | | 1 | Rendszer diagnosztika | `GET /api/debug/dump` | Full state dump: controller info, storage, stacks, network (guest-netns interfaces/route/DNS via the samba door, R-66; best-effort per item), scheduler, health, alerts. JSON download. |
| 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. | | 2 | Értesítés teszt | `POST /api/debug/event/test`, `GET /api/debug/event/history` | Send test events with configurable type/severity, view event history ring buffer. |
| 3 | Mentés teszt | `POST /api/debug/backup/dbdump` · `POST /api/debug/backup/integrity` | Trigger a DB dump, or run an off-site integrity check by hand. **`crossdrive` and `infra` are NOT implemented** — their buttons 404 (R-400). | | 3 | Mentés teszt | `POST /api/debug/backup/dbdump` · `POST /api/debug/backup/crossdrive` · `POST /api/debug/backup/integrity` | Trigger a DB dump, run the Tier-2 (cross-drive) sweep over every deployed HDD-backed app, or run an off-site integrity check by hand. `crossdrive` is asynchronous and answers with the app list it started for; `integrity` is synchronous and answers with the verdict. **`backup/infra` was DELETED (R-400)** — the disk-tier infra backup moved to the host agent in slice 8C and nothing in this repo backs it. |
| 4 | Tárhely teszt | `POST /api/debug/storage/simulate-{disconnect,reconnect}`, `GET /api/debug/storage/watchdog-status` | Simulate drive disconnect/reconnect without unmounting. Per-path probe state with 5s auto-refresh. | | 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. **`hub/infra-push` was DELETED (R-400)** — `Pusher.PushInfraBackup` was removed 2026-06-16. |
| 5 | Hub & Kapcsolatok | `POST /api/debug/hub/{push,infra-push,test-connectivity,preferences-sync}`, `POST /api/debug/gitea/test-connectivity` | Test Hub/Gitea connectivity with latency. Push reports and sync preferences. |
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. | | — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
| 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. | | 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. |
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). | | 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup` | Trigger setup mode via marker file (requires "RESET"). **`dr/infra-status` and its panel were DELETED (R-400)** — it rendered the two retired infra-backup mechanisms above, and it fetched on page LOAD, so the panel had been permanently blank. |
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). | | 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
#### Key Implementation Details #### Key Implementation Details
+2 -2
View File
@@ -3134,7 +3134,7 @@ func runOffsiteIntegrityCheck(ctx context.Context, mgr *backup.Manager, n *notif
// one alarm that means the customer's backups are damaged. // one alarm that means the customer's backups are damaged.
return res return res
case res.OK: case res.OK:
mgr.RecordIntegrityOutcome(res.RanAt, true) mgr.RecordIntegrityVerdict(res)
// severity `info`, which severityNotifies DROPS before either leg — so this mails NOBODY, by // severity `info`, which severityNotifies DROPS before either leg — so this mails NOBODY, by
// design (08 §6.1). A weekly success e-mail is how people stop reading their alerts. It is // design (08 §6.1). A weekly success e-mail is how people stop reading their alerts. It is
// still pushed, because the hub stores it and the event stream is where "was it checked?" is // still pushed, because the hub stores it and the event stream is where "was it checked?" is
@@ -3142,7 +3142,7 @@ func runOffsiteIntegrityCheck(ctx context.Context, mgr *backup.Manager, n *notif
n.NotifyIntegrityOK(integrityOKMsg(res)) n.NotifyIntegrityOK(integrityOKMsg(res))
return res return res
default: default:
mgr.RecordIntegrityOutcome(res.RanAt, false) mgr.RecordIntegrityVerdict(res)
// A FAILING store advances due-ness too: re-checking a broken repository every night is load // A FAILING store advances due-ness too: re-checking a broken repository every night is load
// with no new information, and the hourly operator cooldown already governs the mail. // with no new information, and the hourly operator cooldown already governs the mail.
// //
@@ -0,0 +1,71 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"testing"
)
// B6 — R-399/R-401: SLOWNESS RAISES NO HUB EVENT. This asserts a NON-EFFECT.
//
// A "this took a while" event type would cost the severity contract, the grain table and three
// registers, and 08 §6.2's coarse-by-default rule points the other way. Worse, this project's
// dispatcher coerces any severity outside {info,warning,error,critical} to `info` and mails nobody
// while still returning 200 — so an event added carelessly here would be indistinguishable from one
// that works.
//
// AST, not strings.Contains: a commented-out call still contains the string, and this repo has a
// recorded case of a text-based wiring test passing the very red-proof it existed to fail.
func TestR399_SlownessRaisesNoHubEvent(t *testing.T) {
body := funcBody(t, "runOffsiteIntegrityCheck")
// Every notifier method the one integrity caller invokes.
var notifies []string
ast.Inspect(body, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
if len(sel.Sel.Name) >= 6 && sel.Sel.Name[:6] == "Notify" {
notifies = append(notifies, sel.Sel.Name)
}
return true
})
// POSITIVE CONTROL. If this list is empty the test proves nothing — it would pass just as happily
// against a file it failed to walk.
allowed := map[string]bool{"NotifyIntegrityOK": true, "NotifyIntegrityFailed": true}
if len(notifies) == 0 {
t.Fatal("no Notify* call was found in runOffsiteIntegrityCheck — the walk found nothing, so " +
"the non-effect below would be asserted against an empty set and would pass vacuously")
}
for _, name := range notifies {
if !allowed[name] {
t.Errorf("runOffsiteIntegrityCheck calls %s — slowness and depth are OPERATOR notices in "+
"the log, and the only two customer-facing notifications this job may raise are the "+
"integrity pass and the integrity failure", name)
}
}
}
// funcBody returns the body of a named top-level function in main.go.
func funcBody(t *testing.T, name string) *ast.BlockStmt {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
for _, decl := range f.Decls {
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Name.Name == name && fn.Body != nil {
return fn.Body
}
}
t.Fatalf("func %s not found in main.go", name)
return nil
}
@@ -83,6 +83,21 @@ monitoring:
# ping_uuids: (deprecated — monitoring is now handled by the Hub event system) # ping_uuids: (deprecated — monitoring is now handled by the Hub event system)
system_health_interval: "5m" system_health_interval: "5m"
health_check_schedule: "06:00" health_check_schedule: "06:00"
# Off-site (restic) store integrity check — R-359 / R-399.
integrity:
max_age_days: 7 # A MAX AGE, not a weekday: the job runs daily and asks "is the last
# successful check older than this?", so a box that was off on its check
# day is checked the next day it is on.
read_data_subset: "" # HOW DEEP the check looks. Empty or absent = the default, 100% — every
# stored byte is downloaded and re-hashed. Set "off" to check the
# structure and index only; set "10%", "1/7" or "50M" for a partial
# re-read. A value restic does not accept WARNs and falls back to the
# default, never to "off".
# The default is 100% because the structure check does NOT detect a
# size-preserving pack corruption: measured on a real store 2026-08-30,
# `restic check` reported "no errors were found" over a damaged pack that
# every read-data form caught. Cost on that 134 MB store: 35.0 s at
# structure depth, 39.2 s at 100%.
thresholds: thresholds:
disk_warn_percent: 80 disk_warn_percent: 80
disk_crit_percent: 90 disk_crit_percent: 90
+7
View File
@@ -1469,6 +1469,12 @@ type OffboxReportStatus struct {
// timestamp is the field that says whether the bool means anything at all. // timestamp is the field that says whether the bool means anything at all.
LastIntegrityCheck string `json:"last_integrity_check,omitempty"` // RFC3339 LastIntegrityCheck string `json:"last_integrity_check,omitempty"` // RFC3339
LastIntegrityOK bool `json:"last_integrity_ok,omitempty"` LastIntegrityOK bool `json:"last_integrity_ok,omitempty"`
// LastIntegrityDepth (R-399) is how deep that verdict looked — "structure", or the subset that was
// re-read ("100%"). THIRD field, same argument as the two above and as StatsKnown: a hub that shows
// "checked, OK" without the depth shows the same words for a check that re-read every byte and one
// that only read the index, and those are different news. Absent = the box cannot answer (a
// controller older than v0.228.0), never "structure".
LastIntegrityDepth string `json:"last_integrity_depth,omitempty"`
} }
// OffsiteStateNeedsCredential is the ONE declared state (v0.199.0, R-204 item 4 / R-193): this box // OffsiteStateNeedsCredential is the ONE declared state (v0.199.0, R-204 item 4 / R-193): this box
@@ -1662,6 +1668,7 @@ func (m *Manager) OffboxReportStatus() *OffboxReportStatus {
StatsKnown: t.StatsKnown, // R-331 — without it the hub cannot tell "empty" from "unmeasured" StatsKnown: t.StatsKnown, // R-331 — without it the hub cannot tell "empty" from "unmeasured"
LastIntegrityCheck: t.LastIntegrityCheck, // R-359 LastIntegrityCheck: t.LastIntegrityCheck, // R-359
LastIntegrityOK: t.LastIntegrityOK, LastIntegrityOK: t.LastIntegrityOK,
LastIntegrityDepth: t.LastIntegrityDepth, // R-399
AbandonPurgeRequested: t.AbandonPurgeRequested, // R-241: declared until the hub drops the package AbandonPurgeRequested: t.AbandonPurgeRequested, // R-241: declared until the hub drops the package
} }
} }
+123 -13
View File
@@ -41,14 +41,64 @@ import (
// any plausible structure check and far below "forever", so the failure mode of a wedged SFTP mount is // any plausible structure check and far below "forever", so the failure mode of a wedged SFTP mount is
// a released flag and a retry tomorrow, never a box whose backups stop because a check never returned. // a released flag and a retry tomorrow, never a box whose backups stop because a check never returned.
// //
// It is deliberately NOT sized for a `--read-data-subset` run, which downloads pack data and can take // R-399 CHANGED WHAT THIS NUMBER HAS TO COVER, and this paragraph replaces the one that said the
// hours. That option ships OFF (R-399); whoever turns it on must revisit this number, and this comment // opposite. Read-data now ships ON at 100% (see defaultIntegrityReadDataSubset below), so a check
// is the note that says so. // downloads and re-hashes the whole store every week. Measured on demo-hp 2026-08-30: 39.2 s at 100%
// against a 134 MB store, versus 35.0 s at structure depth. 30 minutes is still ~46x the only
// full-depth number that exists, so it is not resized here on the strength of one measurement — but
// it is now the number a LARGE store will meet first, and `integritySlowNoticeThreshold` exists to
// tell the operator long before that happens. R-401 owns the revisit.
const integrityCheckTimeout = 30 * time.Minute const integrityCheckTimeout = 30 * time.Minute
// integritySlowNoticeThreshold is the point at which a COMPLETED check has started costing real time
// and the depth setting needs revisiting (R-401).
//
// CHOSEN, and deliberately imprecise, from the single data point that exists: demo-hp's 134 MB store,
// 2026-08-30, 35.0 s at structure depth and 39.2 s at 100%. 5 minutes is ~7.6x the only full-depth
// number we have, so it cannot fire on anything resembling today's fleet; and it is well under
// integrityCheckTimeout, so the operator hears "this is getting slow" long before a check is killed
// for running too long. A notice changes no behaviour, so an imprecise number is cheap here — whereas
// a precise-looking threshold invented from one measurement on one small store would be the exact
// shape of the four production designs this project has already specced against nothing.
// A var, not a const, for ONE reason: the notice cannot otherwise be proven to fire through the real
// CheckOffboxIntegrity path — no test can make a check take five minutes. Tests lower it and restore
// it with defer. Nothing in production writes it.
var integritySlowNoticeThreshold = 5 * time.Minute
// defaultIntegrityMaxAgeDays is the max age of a SUCCESSFUL check before one is due again. // defaultIntegrityMaxAgeDays is the max age of a SUCCESSFUL check before one is due again.
const defaultIntegrityMaxAgeDays = 7 const defaultIntegrityMaxAgeDays = 7
// defaultIntegrityReadDataSubset is how deep an unconfigured box checks: ALL of it (R-399, Viktor's
// ruling of 2026-08-31).
//
// THE FACT THE DEFAULT RESTS ON, because it is the thing that stops someone turning it back down to
// save four seconds: **the structure check does not detect a size-preserving pack corruption.** On
// 2026-08-30 a pack in demo-hp's store was damaged WITHOUT changing its size; plain `restic check`
// reported `no errors were found` and exited clean, and every read-data form caught it. A store that
// is verified only structurally is a store whose rot is discovered at restore time, with a customer
// waiting.
//
// THE COST, measured the same day on the same store (140 829 678 B / 2 651 blobs / 67 snapshots):
// structure 35.0 s, 10% 35.9 s, 50% 37.3 s, 100% 39.2 s. Four seconds.
//
// WHAT IS NOT ESTABLISHED: how any of that behaves on a store one or two orders of magnitude larger.
// There is exactly ONE data point. That is why this is a constant and a notice (see
// integritySlowNoticeThreshold) rather than a rotation schedule, a size threshold or a bandwidth
// budget — every one of those would be a number invented from a single measurement. R-401.
//
// This default lives HERE and not in config.applyDefaults, deliberately, and defaultIntegrityMaxAgeDays
// beside it is the precedent: both integrity defaults are resolved in this package, in one accessor
// each, next to the reasoning that justifies them. Symmetry with the other Monitoring defaults is
// worth less than having the number and its argument in the same place.
const defaultIntegrityReadDataSubset = "100%"
// integrityOffToken switches the deep check back off without a code change.
//
// A setting with no off switch is not a setting. Without this token there would be no way to return a
// box to structure depth: an EMPTY value means "not configured" and therefore the default (§8), so
// emptiness cannot also mean "off". Matched case-insensitively.
const integrityOffToken = "off"
// readDataSubsetRe accepts the forms restic documents for --read-data-subset: "n/m", a percentage // readDataSubsetRe accepts the forms restic documents for --read-data-subset: "n/m", a percentage
// like "5%", or a size like "50M". Anything else is refused at read time rather than passed through — // like "5%", or a size like "50M". Anything else is refused at read time rather than passed through —
// a typo must not fail the whole check, which is what handing restic an unparsed value would do. // a typo must not fail the whole check, which is what handing restic an unparsed value would do.
@@ -80,21 +130,64 @@ type IntegrityResult struct {
// integrityReadDataSubset resolves the configured subset spec, refusing anything malformed. // integrityReadDataSubset resolves the configured subset spec, refusing anything malformed.
// //
// Fail-safe direction: an unrecognised value becomes "" (structure check only) with a WARN, never a // FOUR inputs, three outcomes (§8's table):
// passthrough. Handing restic `--read-data-subset=banana` fails the entire check, which would turn a // - absent or empty -> defaultIntegrityReadDataSubset. Empty is "not configured", never "off".
// typo in a config file into a store that silently stops being verified. // - "off" (any case) -> "" , the structure-and-index check only. The one way to switch it back.
// - a form restic accepts -> itself, unchanged. An explicit value always wins.
// - anything else -> defaultIntegrityReadDataSubset, with a WARN naming the bad value.
//
// THE MALFORMED CASE FALLS BACK TO THE DEFAULT, NOT TO STRUCTURE, and the direction is the point.
// Handing restic `--read-data-subset=banana` fails the whole check, so a typo must not be passed
// through — but downgrading to structure depth on a typo would ALSO silently remove the protection
// R-399 exists to add, which is R-357's shape exactly: a guard that opens quietly. Falling back to the
// default keeps the protection and still says loudly that the config is wrong.
func (m *Manager) integrityReadDataSubset() string { func (m *Manager) integrityReadDataSubset() string {
spec := strings.TrimSpace(m.cfg.Monitoring.Integrity.ReadDataSubset) spec := strings.TrimSpace(m.cfg.Monitoring.Integrity.ReadDataSubset)
if spec == "" { if spec == "" {
return defaultIntegrityReadDataSubset
}
if strings.EqualFold(spec, integrityOffToken) {
return "" return ""
} }
if !readDataSubsetRe.MatchString(spec) { if !readDataSubsetRe.MatchString(spec) {
m.logger.Printf("[WARN] [offbox] integrity: read_data_subset %q is not a form restic accepts (n/m, N%%, or a size like 50M) — running the STRUCTURE check only", spec) m.logger.Printf("[WARN] [offbox] integrity: read_data_subset %q is not a form restic accepts (n/m, N%%, a size like 50M, or %q) — falling back to the DEFAULT depth %q, not to a structure-only check, so a typo cannot quietly remove the protection",
return "" spec, integrityOffToken, defaultIntegrityReadDataSubset)
return defaultIntegrityReadDataSubset
} }
return spec return spec
} }
// IntegrityDepthCode is the depth as a short RECORDED value, for the persisted verdict and the wire.
//
// "" is reserved to mean NOT RECORDED — a box older than v0.228.0, whose stored verdict cannot say how
// deep it looked. That follows the StatsKnown precedent on the same object: absence means "cannot
// answer", never an answer. So structure depth is written as the word "structure", not as "".
func IntegrityDepthCode(subset string) string {
if subset == "" {
return "structure"
}
return subset
}
// noticeIfSlow logs an operator WARN when a COMPLETED check has started costing real time (R-401).
//
// COMPLETED ONLY. A skip has no duration to judge, and an unreachable store is "I could not look",
// which is not "I looked and it was slow" (§8). Pass and fail BOTH qualify: the notice and the failure
// alarm are independent facts and neither suppresses the other.
//
// It is a log line and NOTHING else — no hub event, no customer alarm. An event type costs the
// severity contract, the grain table and three registers, all to say "this took a while"; 08 §6.2's
// coarse-by-default rule points the other way. And it does NOT change the depth by itself: a notice
// that silently reconfigures the box would be a behaviour change wearing a notice's clothes.
func (m *Manager) noticeIfSlow(res IntegrityResult) {
if res.Skipped || res.Unreachable || res.Duration < integritySlowNoticeThreshold {
return
}
m.logger.Printf("[WARN] [offbox] integrity: the check took %s at depth %s (%s), over the %s notice threshold — R-401: the depth setting needs revisiting for a store this size. Nothing was changed automatically.",
res.Duration.Round(time.Second), IntegrityDepthCode(res.ReadDataSubset),
integrityDepthLabel(res.ReadDataSubset), integritySlowNoticeThreshold)
}
// integrityMaxAge returns the configured max age of a successful check, defaulting to 7 days. // integrityMaxAge returns the configured max age of a successful check, defaulting to 7 days.
func (m *Manager) integrityMaxAge() time.Duration { func (m *Manager) integrityMaxAge() time.Duration {
d := m.cfg.Monitoring.Integrity.MaxAgeDays d := m.cfg.Monitoring.Integrity.MaxAgeDays
@@ -133,16 +226,29 @@ func (m *Manager) IntegrityDue(now time.Time) (due bool, last time.Time) {
// the hourly operator cooldown already governs the mail, and the failure is already recorded where a // the hourly operator cooldown already governs the mail, and the failure is already recorded where a
// surface can read it. A skip or an unreachable repository does NOT reach here, so tomorrow tries // surface can read it. A skip or an unreachable repository does NOT reach here, so tomorrow tries
// again. // again.
// RecordIntegrityOutcome is the exported entry point; the caller in main.go owns the decision of WHEN // RecordIntegrityVerdict is the PRODUCTION entry point: it persists the verdict AND the depth it was
// a verdict counts, because only it knows whether the run was forced or scheduled. // reached at, in one write. The caller in main.go owns the decision of WHEN a verdict counts, because
func (m *Manager) RecordIntegrityOutcome(at time.Time, ok bool) { m.recordIntegrityOutcome(at, ok) } // only it knows whether the run was forced or scheduled.
//
// The depth travels with the verdict because a stored result that does not say how deep it looked
// cannot be judged later: "checked, OK" means two different things at structure depth and at 100%,
// and the whole of R-399 is that difference.
func (m *Manager) RecordIntegrityVerdict(res IntegrityResult) {
m.recordIntegrityOutcome(res.RanAt, res.OK, IntegrityDepthCode(res.ReadDataSubset))
}
func (m *Manager) recordIntegrityOutcome(at time.Time, ok bool) { // RecordIntegrityOutcome records a verdict whose depth is not stated. It writes "" to the depth field,
// which reads as NOT RECORDED rather than as structure depth — see integrityDepthCode. Kept as the
// due-ness surface the R-359 tests drive; production goes through RecordIntegrityVerdict above.
func (m *Manager) RecordIntegrityOutcome(at time.Time, ok bool) { m.recordIntegrityOutcome(at, ok, "") }
func (m *Manager) recordIntegrityOutcome(at time.Time, ok bool, depth string) {
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) { if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.LastIntegrityCheck = at.UTC().Format(time.RFC3339) o.LastIntegrityCheck = at.UTC().Format(time.RFC3339)
o.LastIntegrityOK = ok o.LastIntegrityOK = ok
o.LastIntegrityDepth = depth
}); err != nil { }); err != nil {
m.logger.Printf("[ERROR] [offbox] integrity: could not persist the check outcome: %v — the check RAN and its verdict was ok=%v, but due-ness did not advance, so it will run again tomorrow", err, ok) m.logger.Printf("[ERROR] [offbox] integrity: could not persist the check outcome: %v — the check RAN and its verdict was ok=%v at depth %q, but due-ness did not advance, so it will run again tomorrow", err, ok, depth)
} }
} }
@@ -198,6 +304,7 @@ func (m *Manager) CheckOffboxIntegrity(ctx context.Context) IntegrityResult {
if err == nil { if err == nil {
res.OK = true res.OK = true
m.logger.Printf("[INFO] [offbox] integrity: check PASSED in %s (%s)", res.Duration.Round(time.Second), integrityDepthLabel(res.ReadDataSubset)) m.logger.Printf("[INFO] [offbox] integrity: check PASSED in %s (%s)", res.Duration.Round(time.Second), integrityDepthLabel(res.ReadDataSubset))
m.noticeIfSlow(res)
return res return res
} }
@@ -221,6 +328,9 @@ func (m *Manager) CheckOffboxIntegrity(ctx context.Context) IntegrityResult {
} }
m.logger.Printf("[ERROR] [offbox] integrity: check FAILED after %s — restic reported: %s", res.Duration.Round(time.Second), res.Output) m.logger.Printf("[ERROR] [offbox] integrity: check FAILED after %s — restic reported: %s", res.Duration.Round(time.Second), res.Output)
// A slow FAILING check gets the notice too. The two facts are independent and suppressing one
// because the other fired is how the second fact stops existing.
m.noticeIfSlow(res)
return res return res
} }
@@ -157,21 +157,17 @@ func TestR359_TimeoutIsNotDamage(t *testing.T) {
} }
} }
func TestR359_StructureCheckPassesNoReadDataFlag(t *testing.T) { // TestR359_StructureCheckPassesNoReadDataFlag was DELETED on 2026-08-31, superseded by R-399.
m, cap := newIntegrityManager(t, okRepo(nil)) //
m.CheckOffboxIntegrity(context.Background()) // It asserted that an unconfigured box passes NO --read-data flag. That was the correct contract on
// 2026-08-30, when nothing had measured the cost of a deeper check. The next day a size-preserving
argv := cap.checkArgv() // pack corruption was shown to PASS that structure-only check on real hardware, and Viktor ruled the
if argv == nil { // default to full depth. The test is not weakened, it is inverted: its replacement is
t.Fatal("no check ran") // TestR399_AbsentConfigRunsFullDepth in r399_depth_test.go, and the off token it left room for is
} // pinned by TestR399_OffTokenRunsStructureOnly.
for _, a := range argv { //
if strings.HasPrefix(a, "--read-data") { // Recorded here rather than removed silently, so a later reader does not re-derive the old ruling
t.Fatalf("the DEFAULT check downloaded pack data (%q) — that is a bandwidth cost nobody "+ // from its absence.
"chose, and R-399 exists precisely so it is not chosen here", a)
}
}
}
func TestR359_ReadDataSubsetIsPassedWhenConfigured(t *testing.T) { func TestR359_ReadDataSubsetIsPassedWhenConfigured(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil)) m, cap := newIntegrityManager(t, okRepo(nil))
@@ -192,25 +188,12 @@ func TestR359_ReadDataSubsetIsPassedWhenConfigured(t *testing.T) {
} }
} }
func TestR359_MalformedReadDataSubsetIsTreatedAsOff(t *testing.T) { // TestR359_MalformedReadDataSubsetIsTreatedAsOff was DELETED on 2026-08-31, superseded by R-399.
m, cap := newIntegrityManager(t, okRepo(nil)) //
m.cfg.Monitoring.Integrity.ReadDataSubset = "banana" // Its NAME was the defect. Treating a typo as "off" downgrades the check silently, which is R-357's
res := m.CheckOffboxIntegrity(context.Background()) // shape — a guard that opens quietly. The half of it that still holds (a malformed value never
// reaches restic, and it WARNs) is asserted by TestR399_MalformedFallsBackToTheDefault, which also
for _, a := range cap.checkArgv() { // pins the new direction: the fallback is the DEFAULT depth, never structure.
if strings.HasPrefix(a, "--read-data") {
t.Fatalf("a malformed value was handed to restic (%q) — restic rejects it and the WHOLE "+
"check fails, so one typo silently stops the store being verified at all", a)
}
}
if res.ReadDataSubset != "" {
t.Errorf("a refused value was still recorded as the depth: %q", res.ReadDataSubset)
}
if !strings.Contains(cap.logBuf.String(), "WARN") {
t.Error("a refused config value must say so — silence makes a typo indistinguishable from a " +
"deliberate structure-only setting")
}
}
func TestR359_MessageNeverCarriesResticOutputOrCredentials(t *testing.T) { func TestR359_MessageNeverCarriesResticOutputOrCredentials(t *testing.T) {
// R-379: 615 bytes of raw database text reached a customer once. And offboxBaseArgs builds the repo // R-379: 615 bytes of raw database text reached a customer once. And offboxBaseArgs builds the repo
@@ -0,0 +1,193 @@
package backup
import (
"context"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// ── R-399 — the off-site check reads the DATA, not just the catalogue ────────────────────────────
//
// THE MEASUREMENT THESE TESTS DEFEND. On demo-hp, 2026-08-30, a pack in a real 134 MB store was
// damaged WITHOUT changing its size. Plain `restic check` — the structure-and-index check that every
// box in the fleet ran — reported `no errors were found` and exited clean. Every `--read-data*` form
// caught it. Cost of the deeper run on that store: 39.2 s versus 35.0 s.
//
// So the assertions below are on the ARGUMENT LIST, never on the absence of an error. "The check
// passed" is exactly what the broken configuration produced; only the argv can tell the two apart.
// readDataArg returns the --read-data* argument the check passed to restic, or "" when it passed none.
func readDataArg(argv []string) string {
for _, a := range argv {
if strings.HasPrefix(a, "--read-data") {
return a
}
}
return ""
}
// A1 — the fleet's actual configuration: no integrity block at all.
func TestR399_AbsentConfigRunsFullDepth(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
// Deliberately touch nothing: this is a box whose controller.yaml has no `integrity:` key.
res := m.CheckOffboxIntegrity(context.Background())
argv := cap.checkArgv()
if argv == nil {
t.Fatal("no check ran")
}
if got := readDataArg(argv); got != "--read-data-subset=100%" {
t.Fatalf("an unconfigured box ran at depth %q, want --read-data-subset=100%%; argv=%v\n"+
"A structure-only run is what every box did before R-399, and it PASSED a size-preserving "+
"pack corruption on real hardware — the store's rot would be found at restore time",
got, argv)
}
if res.ReadDataSubset != "100%" {
t.Errorf("the result did not record the depth it actually ran at: %+v", res)
}
}
// A2 — an empty string is NOT the off switch. Empty means "not configured", so it means the default.
func TestR399_EmptyStringIsNotOff(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = ""
m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "--read-data-subset=100%" {
t.Fatalf("an empty value was treated as OFF (%q) — emptiness must mean 'not configured', or "+
"there is no way to distinguish an unset key from a deliberate downgrade", got)
}
}
// A3 — the off switch. A setting with no off switch is not a setting.
func TestR399_OffTokenRunsStructureOnly(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = "off"
res := m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "" {
t.Fatalf("the off token still downloaded pack data (%q) — there would be no way to switch the "+
"deep check off without editing code", got)
}
if res.ReadDataSubset != "" {
t.Errorf("a structure-only run recorded a subset: %q", res.ReadDataSubset)
}
if code := IntegrityDepthCode(res.ReadDataSubset); code != "structure" {
t.Errorf("structure depth must be RECORDED as %q, not as an empty string a reader has to "+
"interpret; got %q", "structure", code)
}
}
// A4 — an operator writing "OFF" or "Off" in a YAML file means the same thing.
func TestR399_OffTokenIsCaseInsensitive(t *testing.T) {
for _, tok := range []string{"OFF", "Off", " oFf "} {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = tok
m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "" {
t.Errorf("%q was not recognised as the off token (argv carried %q)", tok, got)
}
}
}
// A5 — an explicit value wins over both the default and the off token.
func TestR399_ExplicitValueWins(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = "10%"
res := m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "--read-data-subset=10%" {
t.Fatalf("an explicit 10%% ran as %q — a chosen value must not be overridden by a default", got)
}
if res.ReadDataSubset != "10%" {
t.Errorf("result recorded %q, want 10%%", res.ReadDataSubset)
}
}
// A6 — a typo falls back to the DEFAULT, not to structure depth.
//
// The direction is the whole point. Passing "banana" through fails the entire check; downgrading to
// structure would silently remove the protection R-399 exists to add, which is R-357's shape exactly —
// a guard that opens quietly. Falling back to the default keeps the protection and still says loudly
// that the config is wrong.
func TestR399_MalformedFallsBackToTheDefault(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
m.cfg.Monitoring.Integrity.ReadDataSubset = "banana"
res := m.CheckOffboxIntegrity(context.Background())
argv := cap.checkArgv()
for _, a := range argv {
if strings.Contains(a, "banana") {
t.Fatalf("a malformed value was handed to restic (%q) — restic rejects it and the WHOLE "+
"check fails, so one typo would stop the store being verified at all", a)
}
}
if got := readDataArg(argv); got != "--read-data-subset=100%" {
t.Fatalf("a typo downgraded the check to %q instead of falling back to the default 100%% — "+
"a guard that opens quietly on a typo is R-357's failure", got)
}
if res.ReadDataSubset != "100%" {
t.Errorf("result recorded %q after a refused value, want the default", res.ReadDataSubset)
}
log := cap.logBuf.String()
if !strings.Contains(log, "WARN") || !strings.Contains(log, "banana") {
t.Errorf("a refused config value must WARN and NAME the bad value; log was:\n%s", log)
}
}
// A7 — the depth is stated in the outcome, or the result cannot be judged afterwards.
func TestR399_DepthIsStatedInTheOutcome(t *testing.T) {
m, cap := newIntegrityManager(t, okRepo(nil))
res := m.CheckOffboxIntegrity(context.Background())
if !strings.Contains(cap.logBuf.String(), "100%") {
t.Errorf("the PASSED log line does not say how deep the check looked:\n%s", cap.logBuf.String())
}
// And it is PERSISTED with the verdict, so a later reader of the stored state can judge it too.
m.RecordIntegrityVerdict(res)
tgt := m.settings.GetOffboxTarget()
if tgt == nil || tgt.LastIntegrityDepth != "100%" {
t.Fatalf("the stored verdict does not carry its depth: %+v — 'checked, OK' means two different "+
"things at structure depth and at 100%%", tgt)
}
}
// A8 — THE SEAM TEST. Everything above sets the config field directly; this one proves the default
// survives the PRODUCTION resolution path: real YAML bytes -> config.LoadFromBytes -> the accessor ->
// the argv. A default that only works when a test hand-builds the struct is the inert-seam failure
// this project has shipped four times.
func TestR399_DefaultResolvesThroughTheRealConfigPath(t *testing.T) {
// A minimal but VALID controller.yaml — LoadFromBytes validates, and a config that fails
// validation would never reach a running box, so a fixture that skipped the required keys would
// not be the production path.
const yaml = `
customer:
id: "demo-hp"
domain: "felhom.example.hu"
monitoring:
enabled: true
thresholds:
disk_warn_percent: 80
`
loaded, err := config.LoadFromBytes([]byte(yaml))
if err != nil {
t.Fatalf("the fixture config does not parse: %v", err)
}
if loaded.Monitoring.Integrity.ReadDataSubset != "" {
t.Fatalf("fixture wrong: the parsed config already carries a subset %q, so this test would "+
"prove nothing about the ABSENT case", loaded.Monitoring.Integrity.ReadDataSubset)
}
m, cap := newIntegrityManager(t, okRepo(nil))
// Only the parsed Monitoring block is transplanted; Paths must stay pointing at the test's temp
// dir. The seam under test is config-parse -> Monitoring.Integrity -> integrityReadDataSubset.
m.cfg.Monitoring = loaded.Monitoring
m.CheckOffboxIntegrity(context.Background())
if got := readDataArg(cap.checkArgv()); got != "--read-data-subset=100%" {
t.Fatalf("a real controller.yaml with no `integrity:` block resolved to depth %q, want "+
"--read-data-subset=100%% — that is every box in the fleet today", got)
}
}
@@ -0,0 +1,139 @@
package backup
import (
"context"
"errors"
"strings"
"testing"
"time"
)
// ── R-399 / R-401 — the check tells the operator when it starts costing real time ────────────────
//
// The 100% default rests on ONE measurement, on ONE 134 MB store: 39.2 s. It will not stay true. The
// notice is what makes that fact reach a person from the product rather than from a customer.
//
// It is a LOG LINE and nothing else — no hub event, no customer alarm, and it never changes the depth
// by itself. TestR399_SlownessRaisesNoHubEvent in cmd/controller pins the first of those; the other
// two are pinned here.
// slowNoticeFragment is the ASCII-only fingerprint of the notice, chosen so no other line in this
// file's logs can match it.
const slowNoticeFragment = "R-401: the depth setting needs revisiting"
// withSlowThreshold lowers the notice threshold for one test and restores it.
//
// No test can make a check take five minutes, so the threshold is the seam. Everything else runs
// through the real CheckOffboxIntegrity.
func withSlowThreshold(t *testing.T, d time.Duration) {
t.Helper()
prev := integritySlowNoticeThreshold
integritySlowNoticeThreshold = d
t.Cleanup(func() { integritySlowNoticeThreshold = prev })
}
// slowReply makes the `check` call take measurable time so a lowered threshold is genuinely exceeded
// rather than merely equalled.
func slowReply(out []byte, err error) func(args []string) ([]byte, error) {
return okRepo(func(args []string) ([]byte, error) {
time.Sleep(3 * time.Millisecond)
return out, err
})
}
// B1 — a slow check warns, and the warning names the duration and the depth.
func TestR399_SlowCheckWarns(t *testing.T) {
withSlowThreshold(t, time.Millisecond)
m, cap := newIntegrityManager(t, slowReply(nil, nil))
res := m.CheckOffboxIntegrity(context.Background())
if !res.OK {
t.Fatalf("fixture: this run should pass; got %+v", res)
}
log := cap.logBuf.String()
if !strings.Contains(log, slowNoticeFragment) {
t.Fatalf("a check over the threshold produced no notice — the operator would keep spending a "+
"customer's bandwidth every week and hear about it from the customer. Log:\n%s", log)
}
if !strings.Contains(log, "WARN") {
t.Errorf("the notice is not at WARN level:\n%s", log)
}
// It must name BOTH facts: how long, and how deep. Either alone is unactionable.
if !strings.Contains(log, "100%") {
t.Errorf("the notice does not name the depth that was slow:\n%s", log)
}
if !strings.Contains(log, "the check took") {
t.Errorf("the notice does not name the duration:\n%s", log)
}
// And it changes NOTHING by itself. A notice that silently reconfigured the box would be a
// behaviour change wearing a notice's clothes.
if m.integrityReadDataSubset() != "100%" {
t.Error("the notice altered the configured depth — it must only report")
}
}
// B2 — a fast check is silent. A notice that fires every week is not a notice.
func TestR399_FastCheckIsSilent(t *testing.T) {
withSlowThreshold(t, time.Hour)
m, cap := newIntegrityManager(t, okRepo(nil))
m.CheckOffboxIntegrity(context.Background())
if strings.Contains(cap.logBuf.String(), slowNoticeFragment) {
t.Fatalf("a check under the threshold warned anyway:\n%s", cap.logBuf.String())
}
}
// B3 — a SKIP has no duration to judge.
func TestR399_SkipNeverWarns(t *testing.T) {
withSlowThreshold(t, time.Nanosecond) // every non-zero duration would qualify
m, cap := newIntegrityManager(t, okRepo(nil))
if err := m.AcquireRunningForTest(); err != nil {
t.Fatalf("fixture: %v", err)
}
defer m.ReleaseRunningForTest()
res := m.CheckOffboxIntegrity(context.Background())
if !res.Skipped {
t.Fatalf("fixture: expected a skip, got %+v", res)
}
if strings.Contains(cap.logBuf.String(), slowNoticeFragment) {
t.Fatalf("a skipped check produced a slowness notice — it never ran, so there is no duration "+
"to judge:\n%s", cap.logBuf.String())
}
}
// B4 — "I could not look" is not "I looked and it was slow".
func TestR399_UnreachableNeverWarns(t *testing.T) {
withSlowThreshold(t, time.Nanosecond)
m, cap := newIntegrityManager(t, func(args []string) ([]byte, error) {
time.Sleep(3 * time.Millisecond)
return []byte("ssh: connect to host nas.local port 22: Connection refused"), errors.New("exit 1")
})
res := m.CheckOffboxIntegrity(context.Background())
if !res.Unreachable {
t.Fatalf("fixture: expected unreachable, got %+v", res)
}
if strings.Contains(cap.logBuf.String(), slowNoticeFragment) {
t.Fatalf("an unreachable store produced a slowness notice:\n%s", cap.logBuf.String())
}
}
// B5 — the notice and the failure verdict are INDEPENDENT facts. Neither suppresses the other.
func TestR399_SlowAndFailedProducesBoth(t *testing.T) {
withSlowThreshold(t, time.Millisecond)
const damaged = "pack 5b1f2c3d: not found in index\nrepository contains errors"
m, cap := newIntegrityManager(t, slowReply([]byte(damaged), errFake))
res := m.CheckOffboxIntegrity(context.Background())
if res.OK || res.Unreachable {
t.Fatalf("fixture: expected a readable-and-damaged verdict, got %+v", res)
}
log := cap.logBuf.String()
if !strings.Contains(log, "check FAILED") {
t.Fatalf("the failure was not reported:\n%s", log)
}
if !strings.Contains(log, slowNoticeFragment) {
t.Fatalf("a slow FAILING check lost its slowness notice — suppressing one fact because the "+
"other fired is how the second fact stops existing:\n%s", log)
}
}
+21 -5
View File
@@ -188,11 +188,27 @@ type PingUUIDsConfig struct {
// is on. R-341 is the failure that shape avoids: a dated check that was quietly missed for five days // is on. R-341 is the failure that shape avoids: a dated check that was quietly missed for five days
// because nothing asked again. // because nothing asked again.
// //
// IntegrityReadDataSubset is EMPTY by default and that is a decision, not an oversight (R-399). An // IntegrityReadDataSubset chooses HOW DEEP the check looks, and since 2026-08-31 it DEFAULTS TO
// empty value runs restic's structure-and-index check, which downloads no pack data. A value like // FULL DEPTH — an absent or empty value re-reads 100% of the stored data (R-399, Viktor's ruling).
// "5%" adds `--read-data-subset=5%`, which downloads and re-hashes that fraction of the store every //
// run — a bandwidth and money cost that nothing has yet measured against the real store, so the // This paragraph replaces one that argued the opposite. That argument was correct on the day it was
// default must not be chosen here. // written, when nothing had measured the cost; it was overtaken by a measurement the next day.
//
// **The structure check does not detect a size-preserving pack corruption.** On demo-hp, 2026-08-30, a
// pack was damaged WITHOUT changing its size: plain `restic check` reported `no errors were found` and
// exited clean, and every read-data form caught it. A structurally-verified store is one whose rot is
// found at restore time, with a customer waiting. That sentence is the reason for the default and it
// is what should stop anyone turning it back down to save four seconds.
//
// The cost, same store and day (140 829 678 B / 2 651 blobs / 67 snapshots): structure 35.0 s,
// 10% 35.9 s, 50% 37.3 s, 100% 39.2 s.
//
// Accepted values: absent or empty -> the default (100%); "off" (any case) -> structure and index
// only; any form restic accepts for --read-data-subset ("10%", "1/7", "50M") -> itself. Anything else
// WARNs and falls back to the default, never to structure — a typo must not quietly remove the
// protection. The default constant, the off token and that reasoning live in
// internal/backup/offbox_integrity.go, beside defaultIntegrityMaxAgeDays and NOT in applyDefaults:
// both integrity defaults are resolved in one accessor each, next to the argument that justifies them.
type IntegrityConfig struct { type IntegrityConfig struct {
MaxAgeDays int `yaml:"max_age_days"` MaxAgeDays int `yaml:"max_age_days"`
ReadDataSubset string `yaml:"read_data_subset"` ReadDataSubset string `yaml:"read_data_subset"`
+11 -3
View File
@@ -114,9 +114,17 @@ type BackupReport struct {
// protected?". // protected?".
// //
// **The live numbers are in `Offsite` (backup.OffboxReportStatus) — read that instead**, and // **The live numbers are in `Offsite` (backup.OffboxReportStatus) — read that instead**, and
// consult its `StatsKnown` before believing a zero. `IntegrityOK` has no source at all: the // consult its `StatsKnown` before believing a zero. That includes the integrity verdict: it is
// controller runs no integrity check, and `NotifyIntegrityOK`/`NotifyIntegrityFailed` exist and // published as `Offsite.LastIntegrityCheck` / `LastIntegrityOK` / `LastIntegrityDepth`.
// are called from nowhere — so it can only ever be a lie or a permanent "Unknown". //
// CORRECTED 2026-08-31 (R-399). The sentence here used to read "`IntegrityOK` has no source at
// all: the controller runs no integrity check, and `NotifyIntegrityOK`/`NotifyIntegrityFailed`
// exist and are called from nowhere". BOTH HALVES BECAME FALSE IN v0.227.0 — the check runs weekly
// and both notifiers are called. The fields below are still dead and still unrendered, but the
// reason changed: not "nothing produces it" but "the live verdict lives on OffboxReportStatus, and
// these are kept only so historical reports already in the hub's store keep parsing". Giving them
// a value now would resurrect the card R-331 removed. A warning block asserting a fact that has
// stopped being true is how the next reader concludes the opposite of what it means.
// //
// They are kept rather than deleted so historical reports already in the hub's store keep // They are kept rather than deleted so historical reports already in the hub's store keep
// parsing; pinned by TestBackupReport_DeadFieldsStayZero. // parsing; pinned by TestBackupReport_DeadFieldsStayZero.
+9
View File
@@ -304,6 +304,15 @@ type OffboxTarget struct {
// a dated check that is quietly missed and never catches up. // a dated check that is quietly missed and never catches up.
LastIntegrityCheck string `json:"last_integrity_check,omitempty"` // RFC3339 LastIntegrityCheck string `json:"last_integrity_check,omitempty"` // RFC3339
LastIntegrityOK bool `json:"last_integrity_ok,omitempty"` LastIntegrityOK bool `json:"last_integrity_ok,omitempty"`
// LastIntegrityDepth (R-399) records HOW DEEP that verdict looked: "structure" for the
// structure-and-index check, or the subset spec that was re-read ("100%", "10%", "1/7", ...).
//
// It is a third field rather than a flag for the same reason there are two above: "checked, OK"
// means two different things at structure depth and at 100%, and a verdict that cannot say which
// cannot be judged afterwards. EMPTY means NOT RECORDED — a verdict written by a controller older
// than v0.228.0 — and never "structure"; absence means the box cannot answer, exactly as StatsKnown
// below establishes for the counts.
LastIntegrityDepth string `json:"last_integrity_depth,omitempty"`
LastError string `json:"last_error,omitempty"` LastError string `json:"last_error,omitempty"`
LastDuration string `json:"last_duration,omitempty"` LastDuration string `json:"last_duration,omitempty"`
RepoSizeHuman string `json:"repo_size_human,omitempty"` RepoSizeHuman string `json:"repo_size_human,omitempty"`
+68
View File
@@ -65,6 +65,11 @@ func (s *Server) handleDebugAPI(w http.ResponseWriter, r *http.Request) {
// Section 3: Backup testing (app-data only; disk-tier moved to host agent) // Section 3: Backup testing (app-data only; disk-tier moved to host agent)
case subpath == "backup/dbdump" && r.Method == http.MethodPost: case subpath == "backup/dbdump" && r.Method == http.MethodPost:
s.debugTriggerDBDump(w, r) s.debugTriggerDBDump(w, r)
// R-400 — the „Csak cross-drive" button has posted here since it was added and nothing answered.
// The capability was never missing: Manager.RunTier2 is live and the app config page already calls
// it. Only the debug route was absent, so this is IMPLEMENTED rather than deleted.
case subpath == "backup/crossdrive" && r.Method == http.MethodPost:
s.debugRunCrossDrive(w, r)
// R-397 — the button at debug.html:83 has posted here since it was added and NOTHING answered. // R-397 — the button at debug.html:83 has posted here since it was added and NOTHING answered.
// Verified 2026-08-30: this dispatch had no such case, so pressing „Restic integritás" did // Verified 2026-08-30: this dispatch had no such case, so pressing „Restic integritás" did
// nothing at all. Seventh instance of built-but-never-wired in this project; filed as R-400 in its // nothing at all. Seventh instance of built-but-never-wired in this project; filed as R-400 in its
@@ -404,6 +409,66 @@ func (s *Server) debugTriggerDBDump(w http.ResponseWriter, r *http.Request) {
writeDebugJSON(w, http.StatusOK, true, "DB dump elindítva", nil) writeDebugJSON(w, http.StatusOK, true, "DB dump elindítva", nil)
} }
// debugRunCrossDrive runs the Tier-2 (cross-drive) copy for every deployed HDD app (R-400).
//
// ASYNCHRONOUS, following debugTriggerDBDump above rather than the integrity button below: a Tier-2
// sweep across every app copies real data and is bounded by disk speed, not by a timeout, so holding
// the operator's request open for it would time the browser out and teach nothing. The integrity
// button is synchronous because the operator pressed it to learn an ANSWER; this one is pressed to
// make something happen, and the log is where its outcome belongs.
//
// It reports WHICH apps it started for, not just „elindítva". A sweep that quietly matched zero apps
// and a sweep that matched eight are different facts, and a message that cannot tell them apart is
// how a button reads as working while doing nothing — the class this whole row exists to close.
func (s *Server) debugRunCrossDrive(w http.ResponseWriter, r *http.Request) {
if s.backupMgr == nil {
writeDebugJSON(w, http.StatusBadRequest, false, "Backup manager nincs konfigurálva", nil)
return
}
names := crossDriveTargets(s.stackMgr.GetStacks(), func(name string) bool {
return s.backupMgr.Tier2Info(name).IsHDDApp
})
if len(names) == 0 {
writeDebugJSON(w, http.StatusOK, true, "Nincs olyan telepített alkalmazás, amelynek 2. mentése futtatható lenne",
map[string]interface{}{"apps": names, "count": 0})
return
}
go func(apps []string) {
for _, name := range apps {
if err := s.backupMgr.RunTier2(name); err != nil {
s.logger.Printf("[WARN] [web] debug cross-drive run for %s failed: %v", name, err)
continue
}
s.logger.Printf("[INFO] [web] debug cross-drive run for %s completed", name)
}
}(names)
writeDebugJSON(w, http.StatusOK, true,
fmt.Sprintf("Cross-drive mentés elindítva %d alkalmazásra", len(names)),
map[string]interface{}{"apps": names, "count": len(names)})
}
// crossDriveTargets picks the apps a Tier-2 sweep should run for.
//
// A NAMED FUNCTION rather than an inline loop so both of its answers are assertable. The empty answer
// and the non-empty answer are the two outcomes a dead button and a working one are told apart by, and
// building a deployed HDD-backed stack inside a web test is not practical — so the selection is proven
// here and the dispatch is proven at the route.
func crossDriveTargets(all []stacks.Stack, isHDDApp func(name string) bool) []string {
names := make([]string, 0)
for _, st := range all {
if !st.Deployed {
continue
}
// Tier 2 is an off-DRIVE copy: an app with no HDD path has no second drive to copy to, and
// RunTier2 would return "no source drive" for every one of them.
if !isHDDApp(st.Name) {
continue
}
names = append(names, st.Name)
}
return names
}
// debugRunIntegrityCheck runs the off-site integrity check by hand (R-359/R-397). // debugRunIntegrityCheck runs the off-site integrity check by hand (R-359/R-397).
// //
// SYNCHRONOUS on purpose, unlike the DB-dump button beside it: the operator pressed this to learn an // SYNCHRONOUS on purpose, unlike the DB-dump button beside it: the operator pressed this to learn an
@@ -427,6 +492,9 @@ func (s *Server) debugRunIntegrityCheck(w http.ResponseWriter, r *http.Request)
"unreachable": res.Unreachable, "unreachable": res.Unreachable,
"duration_ms": res.Duration.Milliseconds(), "duration_ms": res.Duration.Milliseconds(),
"read_data_subset": res.ReadDataSubset, "read_data_subset": res.ReadDataSubset,
// R-399: the depth as it is RECORDED, so the JSON says "structure" rather than an empty string
// a reader has to interpret. read_data_subset above stays raw for anyone parsing the argv.
"depth": backup.IntegrityDepthCode(res.ReadDataSubset),
} }
switch { switch {
case res.Skipped: case res.Skipped:
@@ -0,0 +1,147 @@
package web
import (
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// ── R-400 — the debug page stops lying ───────────────────────────────────────────────────────────
//
// Verified 2026-08-31 against the shipped tree: debug.html referenced 24 `/api/debug/...` addresses
// and handleDebugAPI answered 17. Seven controls did nothing. Three of the seven were not buttons —
// `dr/infra-status` and both `storage/watchdog-status` calls fetch on page LOAD, so those panels had
// been permanently empty and nobody had to click anything to be misled. This is the page an operator
// opens when something is already wrong.
//
// The gate in scripts/debug_route_gate.py makes the CLASS impossible. These tests name the individual
// DECISIONS, so that re-adding one of them is deliberate rather than accidental.
// debugTemplateSource reads the shipped template from disk. The served page is rendered from this
// file, so a reference that is gone here is gone from the page — and reading the source is what lets
// a deletion be asserted without a browser (none is available on this host).
func debugTemplateSource(t *testing.T) string {
t.Helper()
b, err := os.ReadFile(filepath.Join("templates", "debug.html"))
if err != nil {
t.Fatalf("read debug.html: %v", err)
}
return string(b)
}
// D2 — every control DELETED in Part 2.1, by name, with its reason.
func TestR400_DeletedControlsAreGoneFromTheTemplate(t *testing.T) {
src := debugTemplateSource(t)
deleted := []struct{ ref, why string }{
{"/api/debug/backup/infra",
"the disk-tier infra backup moved to the host agent (slice 8C); no function in this repo backs it"},
{"/api/debug/hub/infra-push",
"Pusher.PushInfraBackup was removed 2026-06-16 — retired hub-side, and it had pushed plaintext secrets"},
{"/api/debug/dr/infra-status",
"it rendered local infra backups and the hub infra push, both of which are the two retired mechanisms above"},
{"/api/debug/storage/watchdog-status",
"the slice-8C storage watchdog is retired; the drive-gate reconcile replaced it and publishes no such status"},
{"/api/debug/storage/simulate-disconnect",
"no backing capability, and it WRITES storage state — a button that fakes a drive disconnect on a customer's machine is a foot-gun"},
{"/api/debug/storage/simulate-reconnect", "same as simulate-disconnect"},
}
for _, d := range deleted {
if strings.Contains(src, d.ref) {
t.Errorf("%s is still referenced by debug.html — it was deleted because %s", d.ref, d.why)
}
}
// POSITIVE CONTROL. Without it this test passes against a template it failed to read, or one that
// was emptied — the exact shape of an assertion that proves nothing.
for _, kept := range []string{"/api/debug/backup/integrity", "/api/debug/dr/trigger-setup"} {
if !strings.Contains(src, kept) {
t.Fatalf("positive control failed: %s is missing too, so the assertions above prove nothing "+
"about what was deliberately deleted", kept)
}
}
}
// D2b — the panels and the JavaScript went with the controls. A panel left behind renders nothing
// forever, which is how this whole class hides.
func TestR400_DeletedControlsLeftNoPanelOrScript(t *testing.T) {
src := debugTemplateSource(t)
// ASCII-only fragments (R-364): accented Hungarian through a template read is not the hazard here,
// but the rule is uniform and these identifiers are ASCII anyway.
for _, orphan := range []string{
"watchdog-status", // the panel div and its two loaders
"renderWatchdogStatus",
"loadWatchdogStatus",
"simulateDisconnect",
"simulateReconnect",
"loadDRStatus",
"dr-status",
"btn-infra-backup",
"btn-hub-infra",
"section-storage",
} {
if strings.Contains(src, orphan) {
t.Errorf("%q survived the deletion — an orphaned panel or handler renders nothing forever "+
"and reads as a working page", orphan)
}
}
// POSITIVE CONTROL: the neighbours that must stay.
for _, kept := range []string{"btn-dr-trigger", "triggerDR", "section-dr", "btn-crossdrive"} {
if !strings.Contains(src, kept) {
t.Fatalf("positive control failed: %q is gone too — the deletion took a neighbour with it", kept)
}
}
}
// D1 — the one control IMPLEMENTED in Part 2.1 dispatches and answers with its JSON shape.
func TestR400_CrossDriveRouteDispatches(t *testing.T) {
// backupMgr nil is the fixture on purpose: it reaches the handler's own guard, which proves the
// route was DISPATCHED. A 404 here is the defect — that is precisely what the button got before.
s := newDebugServer(t, nil)
req := httptest.NewRequest(http.MethodPost, "/api/debug/backup/crossdrive", nil)
w := httptest.NewRecorder()
s.handleDebugAPI(w, req)
if w.Code == http.StatusNotFound {
t.Fatal("POST /api/debug/backup/crossdrive returned 404 — the button still posts to nothing")
}
var env map[string]interface{}
if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
t.Fatalf("the route answered with something that is not the debug JSON envelope: %q", w.Body.String())
}
if _, has := env["ok"]; !has {
t.Errorf("no `ok` in the envelope: %v", env)
}
}
// D1b — and it answers with the app list when a manager IS present. The empty sweep and the non-empty
// sweep are different facts, and „elindítva" alone cannot tell them apart.
func TestR400_CrossDriveReportsWhichAppsItStarted(t *testing.T) {
// The selection itself, both answers, through the same function the handler calls.
all := []stacks.Stack{
{Name: "immich", Deployed: true},
{Name: "vaultwarden", Deployed: true},
{Name: "not-deployed", Deployed: false},
}
hdd := func(name string) bool { return name == "immich" }
got := crossDriveTargets(all, hdd)
if len(got) != 1 || got[0] != "immich" {
t.Fatalf("the sweep picked %v — it must take deployed HDD-backed apps only: an app with no "+
"second drive has nowhere to copy to, and an undeployed app has nothing to copy", got)
}
if none := crossDriveTargets(all, func(string) bool { return false }); len(none) != 0 {
t.Errorf("with no HDD app the sweep must be EMPTY, not a silent all-apps run: %v", none)
}
// NON-nil empty slice: it marshals as [] rather than null, so the JSON says "zero apps" instead of
// "no answer".
if none := crossDriveTargets(nil, hdd); none == nil {
t.Error("the empty answer is nil — it would marshal as `null`, which reads as 'unknown' rather " +
"than 'none', the same conflation R-331 cost a whole operator card")
}
}
@@ -82,24 +82,10 @@
<button class="btn btn-secondary btn-sm" id="btn-integrity" data-label="Restic integritás" onclick="triggerAction('btn-integrity','/api/debug/backup/integrity','POST')">Restic integritás</button> <button class="btn btn-secondary btn-sm" id="btn-integrity" data-label="Restic integritás" onclick="triggerAction('btn-integrity','/api/debug/backup/integrity','POST')">Restic integritás</button>
<span class="debug-result" id="btn-integrity-result"></span> <span class="debug-result" id="btn-integrity-result"></span>
<button class="btn btn-secondary btn-sm" id="btn-infra-backup" data-label="Infra mentés" onclick="triggerAction('btn-infra-backup','/api/debug/backup/infra','POST')">Infra mentés</button>
<span class="debug-result" id="btn-infra-backup-result"></span>
</div> </div>
</div> </div>
</div> </div>
<!-- Section 4: Storage Testing -->
<div class="card debug-section" id="section-storage">
<div class="card-header debug-section-header" onclick="toggleSection('storage')">
<h3>Tárhely teszt</h3>
<span class="section-toggle">▶</span>
</div>
<div class="card-body debug-section-body" style="display:none">
<div id="watchdog-status"><span class="text-muted">Betöltés...</span></div>
</div>
</div>
<!-- Section 5: Hub & Connectivity --> <!-- Section 5: Hub & Connectivity -->
<div class="card debug-section" id="section-hub"> <div class="card debug-section" id="section-hub">
<div class="card-header debug-section-header" onclick="toggleSection('hub')"> <div class="card-header debug-section-header" onclick="toggleSection('hub')">
@@ -112,9 +98,6 @@
<button class="btn btn-primary btn-sm" id="btn-hub-push" data-label="Hub jelentés küldése" onclick="triggerAction('btn-hub-push','/api/debug/hub/push','POST')">Hub jelentés küldése</button> <button class="btn btn-primary btn-sm" id="btn-hub-push" data-label="Hub jelentés küldése" onclick="triggerAction('btn-hub-push','/api/debug/hub/push','POST')">Hub jelentés küldése</button>
<span class="debug-result" id="btn-hub-push-result"></span> <span class="debug-result" id="btn-hub-push-result"></span>
<button class="btn btn-secondary btn-sm" id="btn-hub-infra" data-label="Infra backup küldése" onclick="triggerAction('btn-hub-infra','/api/debug/hub/infra-push','POST')">Infra backup küldése</button>
<span class="debug-result" id="btn-hub-infra-result"></span>
<button class="btn btn-secondary btn-sm" id="btn-hub-conn" data-label="Hub elérhetőség" onclick="triggerAction('btn-hub-conn','/api/debug/hub/test-connectivity','POST')">Hub elérhetőség</button> <button class="btn btn-secondary btn-sm" id="btn-hub-conn" data-label="Hub elérhetőség" onclick="triggerAction('btn-hub-conn','/api/debug/hub/test-connectivity','POST')">Hub elérhetőség</button>
<span class="debug-result" id="btn-hub-conn-result"></span> <span class="debug-result" id="btn-hub-conn-result"></span>
@@ -168,7 +151,6 @@
<span class="section-toggle">▶</span> <span class="section-toggle">▶</span>
</div> </div>
<div class="card-body debug-section-body" style="display:none"> <div class="card-body debug-section-body" style="display:none">
<div id="dr-status"><span class="text-muted">Betöltés...</span></div>
<div class="debug-actions" style="margin-top:1rem"> <div class="debug-actions" style="margin-top:1rem">
<div class="debug-dr-danger"> <div class="debug-dr-danger">
<p style="color:var(--crit);font-weight:600">Vészhelyzet szimuláció</p> <p style="color:var(--crit);font-weight:600">Vészhelyzet szimuláció</p>
@@ -305,11 +287,9 @@ function loadSectionData(id) {
case 'diagnostic': loadDiagnostic(); break; case 'diagnostic': loadDiagnostic(); break;
case 'events': loadEventHistory(); break; case 'events': loadEventHistory(); break;
case 'backup': loadBackupStatus(); break; case 'backup': loadBackupStatus(); break;
case 'storage': loadWatchdogStatus(); break;
case 'hub': loadHubStatus(); break; case 'hub': loadHubStatus(); break;
case 'telemetry': break; // no auto-load, user triggers manually case 'telemetry': break; // no auto-load, user triggers manually
case 'selfupdate': loadSelfUpdateStatus(); break; case 'selfupdate': loadSelfUpdateStatus(); break;
case 'dr': loadDRStatus(); break;
case 'appexport': loadAppExportStatus(); break; case 'appexport': loadAppExportStatus(); break;
case 'logs': initLogViewer(); break; case 'logs': initLogViewer(); break;
} }
@@ -432,65 +412,6 @@ function loadBackupStatus() {
}); });
} }
// ── Section 4: Storage ──
function loadWatchdogStatus() {
fetch('/api/debug/storage/watchdog-status', {headers: csrfHeaders()}).then(function(r){return r.json()}).then(function(data) {
if (!data.ok) { document.getElementById('watchdog-status').innerHTML = '<span class="text-muted">Nem elérhető</span>'; return; }
renderWatchdogStatus(data.data);
}).catch(function() {
document.getElementById('watchdog-status').innerHTML = '<span class="text-muted">Nem elérhető</span>';
});
startPolling('storage', 5000, function() {
fetch('/api/debug/storage/watchdog-status', {headers: csrfHeaders()}).then(function(r){return r.json()}).then(function(data) {
if (data.ok) renderWatchdogStatus(data.data);
}).catch(function(){});
});
}
function renderWatchdogStatus(paths) {
if (!paths || paths.length === 0) {
document.getElementById('watchdog-status').innerHTML = '<span class="text-muted">Nincs tárhely</span>';
return;
}
var html = '<table class="info-table debug-table"><tr><th>Útvonal</th><th>Cimke</th><th>Állapot</th><th>Probe</th><th>Debounce</th><th>Latency</th><th>Művelet</th></tr>';
paths.forEach(function(p) {
var dot = p.status === 'connected' ? '<span class="status-dot green"></span>' : '<span class="status-dot red"></span>';
var simBadge = p.simulated ? ' <span class="badge-warn">SIM</span>' : '';
var action = '';
if (p.status === 'connected' && !p.simulated) {
action = '<button class="btn btn-xs btn-secondary" onclick="simulateDisconnect(this,\'' + p.path + '\')">Leválasztás</button>';
} else if (p.simulated) {
action = '<button class="btn btn-xs btn-primary" onclick="simulateReconnect(\'' + p.path + '\')">Visszacsatl.</button>';
}
html += '<tr><td class="mono">' + p.path + '</td><td>' + (p.label||'-') + '</td><td>' + dot + ' ' + p.status + simBadge + '</td>';
html += '<td>' + (p.probe_ok_count||0) + '/' + (p.probe_count||0) + '</td><td>' + (p.debounce_count||0) + '/' + (p.debounce_max||3) + '</td>';
html += '<td>' + (p.avg_latency_ms ? p.avg_latency_ms.toFixed(1) + 'ms' : '-') + '</td><td>' + action + '</td></tr>';
});
html += '</table>';
document.getElementById('watchdog-status').innerHTML = html;
}
function simulateDisconnect(btn, path) {
felhomConfirm(btn, 'Biztosan szimulálja a leválasztást? Ez leállítja az érintett alkalmazásokat.', function () {
fetch('/api/debug/storage/simulate-disconnect', {
method: 'POST',
headers: Object.assign({'Content-Type':'application/json'}, csrfHeaders()),
body: JSON.stringify({path: path})
}).then(function(r){return r.json()}).then(function(data) {
if (!data.ok) showAlert('Hiba: ' + (data.error || 'ismeretlen'));
loadWatchdogStatus();
}).catch(function(e) { showAlert('Hiba: ' + e.message); });
});
}
function simulateReconnect(path) {
fetch('/api/debug/storage/simulate-reconnect', {
method: 'POST',
headers: Object.assign({'Content-Type':'application/json'}, csrfHeaders()),
body: JSON.stringify({path: path})
}).then(function(r){return r.json()}).then(function(data) {
if (!data.ok) showAlert('Hiba: ' + (data.error || 'ismeretlen'));
loadWatchdogStatus();
}).catch(function(e) { showAlert('Hiba: ' + e.message); });
}
// ── Section 5: Hub ── // ── Section 5: Hub ──
function loadHubStatus() { function loadHubStatus() {
fetch('/api/debug/dump', {headers: csrfHeaders()}).then(function(r){return r.json()}).then(function(data) { fetch('/api/debug/dump', {headers: csrfHeaders()}).then(function(r){return r.json()}).then(function(data) {
@@ -527,32 +448,6 @@ function loadSelfUpdateStatus() {
} }
// ── Section 7: DR ── // ── Section 7: DR ──
function loadDRStatus() {
fetch('/api/debug/dr/infra-status', {headers: csrfHeaders()}).then(function(r){return r.json()}).then(function(data) {
if (!data.ok) { document.getElementById('dr-status').innerHTML = '<span class="text-muted">Nem elérhető</span>'; return; }
var d = data.data || {};
var html = '<h4>Helyi infra backup</h4>';
if (d.drives && d.drives.length > 0) {
html += '<table class="info-table debug-table"><tr><th>Meghajtó</th><th>Cimke</th><th>Állapot</th><th>Utolsó módosítás</th><th>Fájlok</th></tr>';
d.drives.forEach(function(dr) {
var files = (dr.files || []).join(', ') || '-';
html += '<tr><td class="mono">' + dr.path + '</td><td>' + (dr.label||'-') + '</td><td>' + (dr.has_backup ? 'Van' : '<span class="state-text-warn">Nincs</span>') + '</td><td>' + (dr.last_modified ? fmtTime(dr.last_modified) : '-') + '</td><td class="text-muted" style="font-size:.75rem">' + files + '</td></tr>';
});
html += '</table>';
} else {
html += '<span class="text-muted">Nincs csatlakoztatott meghajtó</span>';
}
if (d.hub_infra_push) {
html += '<h4>Hub infra backup</h4><div class="debug-kv-grid">';
html += '<span>Utolsó push:</span><span>' + (d.hub_infra_push.last_success ? fmtTime(d.hub_infra_push.last_success) : '-') + '</span>';
if (d.hub_infra_push.last_error) html += '<span>Utolsó hiba:</span><span class="debug-result-error">' + d.hub_infra_push.last_error + '</span>';
html += '</div>';
}
document.getElementById('dr-status').innerHTML = html;
}).catch(function() {
document.getElementById('dr-status').innerHTML = '<span class="text-muted">Nem elérhető</span>';
});
}
function triggerDR() { function triggerDR() {
var input = document.getElementById('dr-confirm-input'); var input = document.getElementById('dr-confirm-input');
if (input.value !== 'RESET') { if (input.value !== 'RESET') {
+6 -1
View File
@@ -15,7 +15,8 @@ Gates, in order (all must pass; **non-zero exit on any failure**):
5. app-row-dedup no duplicated app-row markup in the dashboard templates 5. app-row-dedup no duplicated app-row markup in the dashboard templates
6. mojibake no double-encoded UTF-8 in Hungarian copy 6. mojibake no double-encoded UTF-8 in Hungarian copy
7. docker-v every `docker … -v` mount is a named volume or a proven host path 7. docker-v every `docker … -v` mount is a named volume or a proven host path
8. reuse-refs every path cited by this repo's REUSE.md still resolves 8. debug-routes every debug-page control resolves to a handler, and back (R-400)
9. reuse-refs every path cited by this repo's REUSE.md still resolves
WHY THIS FILE EXISTS (2026-08-02, closing R-29 leg (a) and half of leg (b)). WHY THIS FILE EXISTS (2026-08-02, closing R-29 leg (a) and half of leg (b)).
@@ -62,6 +63,10 @@ GATES = [
("docker-v", os.path.join(SCRIPTS, "docker_run_volume_path_gate.py"), [], True), ("docker-v", os.path.join(SCRIPTS, "docker_run_volume_path_gate.py"), [], True),
("secret-markup", os.path.join(SCRIPTS, "secret_in_markup_gate.py"), [], True), ("secret-markup", os.path.join(SCRIPTS, "secret_in_markup_gate.py"), [], True),
("retrieval-promise", os.path.join(SCRIPTS, "retrieval_promise_gate.py"), [], True), ("retrieval-promise", os.path.join(SCRIPTS, "retrieval_promise_gate.py"), [], True),
# R-400 — every debug-page control resolves to a handler, and every handler is reachable.
# Registered AFTER the seven dead controls were implemented or deleted: a registered-but-failing
# gate refuses every push, so the order matters here exactly as it did for instructions_gate.
("debug-routes", os.path.join(SCRIPTS, "debug_route_gate.py"), [], True),
("reuse-refs", SHARED_REUSE, [REPO], True), ("reuse-refs", SHARED_REUSE, [REPO], True),
("instructions", SHARED_INSTRUCTIONS, [REPO], True), ("instructions", SHARED_INSTRUCTIONS, [REPO], True),
# R-389 — a REPORT.md observation with no register row behind it. Fast: stdlib file reads. # R-389 — a REPORT.md observation with no register row behind it. Fast: stdlib file reads.
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Debug-route gate (R-400) — every control on the debug page must resolve to a handler, and every
handler must be reachable from the page.
WHY IT EXISTS. On 2026-08-31 the shipped debug page referenced 24 `/api/debug/...` addresses and the
dispatcher answered 17. Seven controls did nothing, and three of those seven were not buttons at all:
they fetch on page LOAD, so whole panels had been permanently empty and nobody had to click anything
to be misled. This is the page an operator opens when something is already wrong.
BOTH DIRECTIONS FAIL. A reference with no case is a dead control. A case with no reference is a
handler nothing reaches — the same defect mirrored, and the shape this project has now shipped eight
times. Neither is a warning here.
DELIBERATELY TEN LINES OF LOGIC. Two lists and a difference. Its value is that it cannot rot: a
cleverer gate that understood routing would need maintaining, and an unmaintained gate is how the
class hides in the first place. The dispatcher's EXACT-match switch with a NotFound default is what
made the original defect visible, and this gate assumes exactly that shape — do not make either clever.
Run from controller/: python3 scripts/debug_route_gate.py
"""
import io
import os
import re
import sys
TEMPLATE = os.path.join("internal", "web", "templates", "debug.html")
DISPATCH = os.path.join("internal", "web", "handler_debug.go")
REF_RE = re.compile(r"/api/debug/([A-Za-z0-9/_-]+)")
CASE_RE = re.compile(r'subpath\s*==\s*"([A-Za-z0-9/_-]+)"')
def read(path):
if not os.path.exists(path):
print("DEBUG ROUTE GATE INCONCLUSIVE: %s not found (run from controller/)" % path)
sys.exit(2)
return io.open(path, encoding="utf-8").read()
def main():
# Sets, not lists: the same address referenced by two controls is satisfied by one case (§8).
refs = set(REF_RE.findall(read(TEMPLATE)))
cases = set(CASE_RE.findall(read(DISPATCH)))
dead = sorted(refs - cases)
unreached = sorted(cases - refs)
for name in dead:
print("DEAD CONTROL %s references /api/debug/%s and %s has no case for it"
% (TEMPLATE, name, DISPATCH))
for name in unreached:
print("UNREACHED %s dispatches %r and %s never references it"
% (DISPATCH, name, TEMPLATE))
if dead or unreached:
print("DEBUG ROUTE GATE FAILED: %d dead control(s), %d unreached handler(s)"
% (len(dead), len(unreached)))
sys.exit(1)
print("debug route gate OK - %d referenced address(es), all dispatched, none orphaned"
% len(refs))
if __name__ == "__main__":
main()
@@ -30,6 +30,7 @@ FINGERPRINTS = [
("app-row-dedup", "app_row_dedup_gate: OK"), ("app-row-dedup", "app_row_dedup_gate: OK"),
("mojibake", "mojibake_gate: OK"), ("mojibake", "mojibake_gate: OK"),
("docker-v", "docker -v gate OK"), ("docker-v", "docker -v gate OK"),
("debug-routes", "debug route gate OK"),
("reuse-refs", "cited paths — exact"), ("reuse-refs", "cited paths — exact"),
] ]
+109
View File
@@ -0,0 +1,109 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Tests for scripts/debug_route_gate.py (R-400).
Run from controller/: python3 scripts/test_debug_route_gate.py
WHY THE RED-PROOFS ARE THE POINT. A gate that has never been seen failing is a gate that has not been
shown to gate anything. This is the second time that sentence has earned its place in this project, so
both directions are proven by MUTATING a real copy of the tree and watching the gate convict.
"""
import io
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
SCRIPTS = os.path.dirname(os.path.abspath(__file__))
CTRL = os.path.dirname(SCRIPTS)
GATE = os.path.join(SCRIPTS, "debug_route_gate.py")
TEMPLATE = os.path.join("internal", "web", "templates", "debug.html")
DISPATCH = os.path.join("internal", "web", "handler_debug.go")
RUNNER = os.path.join(SCRIPTS, "controller_gates.py")
def run_gate(cwd):
p = subprocess.run([sys.executable, GATE], cwd=cwd,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return p.returncode, p.stdout.decode("utf-8", "replace")
def sandbox():
"""A copy of just the two files the gate reads, in a throwaway tree.
Mutating the real tree and reverting is the version of this that leaves a broken repo behind when
an assertion fails mid-test.
"""
tmp = tempfile.mkdtemp(prefix="debugroutegate-")
for rel in (TEMPLATE, DISPATCH):
dst = os.path.join(tmp, rel)
os.makedirs(os.path.dirname(dst), exist_ok=True)
shutil.copyfile(os.path.join(CTRL, rel), dst)
return tmp
class DebugRouteGateTest(unittest.TestCase):
# C1 — the gate passes on the shipped tree.
def test_passes_on_the_shipped_tree(self):
rc, out = run_gate(CTRL)
self.assertEqual(rc, 0, out)
self.assertIn("debug route gate OK", out)
# C2 — Scenario G, the mandatory red-proof: a reference with no handler convicts, and is NAMED.
def test_fails_on_an_unwired_reference(self):
tmp = sandbox()
try:
path = os.path.join(tmp, TEMPLATE)
src = io.open(path, encoding="utf-8").read()
io.open(path, "w", encoding="utf-8").write(
src + '\n<!-- red-proof --><a href="/api/debug/storage/simulate-disconnect">x</a>\n')
rc, out = run_gate(tmp)
self.assertEqual(rc, 1, "a dead control did not convict the gate:\n" + out)
self.assertIn("storage/simulate-disconnect", out,
"the gate convicted without NAMING the reference:\n" + out)
finally:
shutil.rmtree(tmp)
# C3 — the mirror image: a handler nothing reaches is the same defect.
def test_fails_on_an_unreached_handler(self):
tmp = sandbox()
try:
path = os.path.join(tmp, DISPATCH)
src = io.open(path, encoding="utf-8").read()
marker = 'case subpath == "dump" && r.Method == http.MethodGet:'
self.assertIn(marker, src, "fixture drifted: the dispatch shape changed")
io.open(path, "w", encoding="utf-8").write(src.replace(
marker,
'case subpath == "ghost/handler" && r.Method == http.MethodGet:\n\t\ts.debugDump(w, r)\n\t' + marker,
1))
rc, out = run_gate(tmp)
self.assertEqual(rc, 1, "an unreached handler did not convict the gate:\n" + out)
self.assertIn("ghost/handler", out,
"the gate convicted without NAMING the handler:\n" + out)
finally:
shutil.rmtree(tmp)
# C4 — registration, read from the runner's own GATES table rather than by matching text.
# A commented-out row still contains the string; an entry in the parsed list does not.
def test_gate_is_registered_in_the_runner(self):
import ast
tree = ast.parse(io.open(RUNNER, encoding="utf-8").read())
labels = []
for node in ast.walk(tree):
if isinstance(node, ast.Assign):
for tgt in node.targets:
if isinstance(tgt, ast.Name) and tgt.id == "GATES":
for elt in node.value.elts:
first = elt.elts[0]
labels.append(first.value if hasattr(first, "value") else first.s)
self.assertTrue(labels, "the runner's GATES table could not be parsed")
self.assertIn("debug-routes", labels,
"debug_route_gate.py exists but the runner never lists it — a gate nothing runs "
"is the inert seam this repo has shipped four times. Listed: %r" % (labels,))
if __name__ == "__main__":
unittest.main(verbosity=2)