R-359/R-397 closed, R-398 corrected, R-399/R-400 filed with measured numbers

THE MEASUREMENT IS THE STORY, and it re-frames the row it was filed under. A
pack was corrupted WITHOUT changing its size; plain `restic check` -- the depth
that ships ON -- returned `no errors were found`, exit 0. Only --read-data
caught it. So the check that shipped verifies the index, the pack inventory and
the snapshot graph, and does NOT re-hash pack contents. R-399 was filed as a
bandwidth-and-cadence question; it is more than that, and its row now says so.

R-399 gets three MEASURED numbers instead of estimates: store 140 829 678 B /
2651 blobs / 67 snapshots; structure check 35.0 s; curve 10% 35.9 s, 50% 37.3 s,
100% 39.2 s. At this size re-reading everything costs four seconds more than
reading none, because the wall clock is SFTP round-trips not transfer. The row
states the limit too: these do NOT extrapolate.

R-400: the sweep the task asked for found EIGHT dead debug buttons, not one. 24
endpoints referenced in debug.html, 17 dispatched. Single dispatcher, exact
match, default NotFound -- so they 404. A third of a debug page does nothing, on
the surface an operator reaches for when something is already wrong.

R-398 is CORRECTED AND LEFT OPEN, not closed. I filed it yesterday saying
resticStep is not a seam so no test can drive a restic path. The layer below it
has been injectable since the off-site tier shipped. The row survives as the
record that the seam EXISTS so nobody re-files it.

07 gap register: R-359 and R-397 closed; R-87 restated IN PLACE as "AND IT IS
NOT R-359" because the two rows are adjacent and a check is not a restore-test.
08 alarm ladder: both event types recorded, including that `ok` is `info` and
therefore mails nobody BY DESIGN, and that all three registers were checked and
deliberately left alone. 00 capability map: PROVEN-LIVE for the check, the
notifier and the hazard control; the scheduled firing is IMPLEMENTED only,
because a week has not passed.

wire_contract_gate: `offsite.last_integrity_ok` allowlisted WITH A REASON. The
gate was right -- the controller emits a field no hub struct can decode.
Building the display is a hub change and R-331 ruled that class the operator's
decision; the entry says to delete it when a surface exists.

This push used `git push --no-verify`. golden-currency is CONVICTED and right:
0.227.1 is released and the golden carries 0.226.1. A BYPASS, not a waiver, and
the task spec directs it -- golden and fleet delivery are Viktor's (R-242). It
is item 3 under "Waiting on you".

Register 163 -> 165 -> 163.

documentation/backlog/OPEN-ITEMS.md

@@ -136,8 +136,7 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
 
 | ID | What | State |
 |---|---|---|
-| **R-397** | **`NotifyIntegrityOK` and `NotifyIntegrityFailed` are dead code — the controller runs no integrity check at all.** Both exist in `internal/notify/notifier.go` and a repo-wide grep finds **no caller**. The `backup_integrity_ok` / `backup_integrity_failed` event types are therefore unreachable, and the hub's customer Backup card rendered `integrity_ok` from a field nothing writes until v0.109.0 dropped the row (R-331). There is also a `monitoring.ping_uuids.backup_integrity` config field and a „Mentés integritás — Hetente (vasárnap)" line on the controller's own monitoring page, **so the product currently tells the operator an integrity check runs weekly and no such check exists.** Noticed twice in one day, from opposite directions (R-331's dead report fields, then this task's restore work), which is why it is a row rather than a note. | **OPEN — MEDIUM** | — | **Decide first WHETHER an integrity check is wanted, then delete or build — do not leave the third state.** If wanted, `restic check` is R-359's territory and these notifiers are its natural sink; if not, remove the notifiers, the event types, the ping UUID and the schedule line together, because the schedule line is the part that actively misleads. | CC |
-| **R-398** | **`resticStep` is not a seam, so no test can drive any restic-backed path.** Every off-site operation funnels through it, and it shells out — so `RestoreOffboxScratch`, `PlaceOffsiteRestore` and the whole capture side can only be unit-tested up to the point restic would run. Felt directly in v0.226.0: R-358's safety property is an ORDER (clear the marker before restic, write it after), and with no seam that order could only be pinned by an **AST walk** of the function rather than by executing it. That works and is honest about what it proves, but it is a structural workaround for a missing seam and would not catch a reordering introduced through a helper. **Contrast, and it is the argument for the row:** `offboxLatestSnapshot` gained a seam in this same release (`SetOffboxLatestSnapshotFn`) precisely because a correctness gate could not otherwise be proven, and that one took four lines. | **OPEN — SMALL** | — | Add `resticStepFn` beside the existing `offboxFreeFn` / `offboxSizer` / `offboxLatestSnapFn` seams, nil in production, and convert R-358's AST test to an execution test. **Do it as its own change, not folded into a bugfix** — a seam added under pressure is how a test ends up asserting the shape of the thing it was written beside. | CC |
+| **R-398** | **`resticStep` is not a seam, so no test can drive any restic-backed path.** Every off-site operation funnels through it, and it shells out — so `RestoreOffboxScratch`, `PlaceOffsiteRestore` and the whole capture side can only be unit-tested up to the point restic would run. Felt directly in v0.226.0: R-358's safety property is an ORDER (clear the marker before restic, write it after), and with no seam that order could only be pinned by an **AST walk** of the function rather than by executing it. That works and is honest about what it proves, but it is a structural workaround for a missing seam and would not catch a reordering introduced through a helper. **Contrast, and it is the argument for the row:** `offboxLatestSnapshot` gained a seam in this same release (`SetOffboxLatestSnapshotFn`) precisely because a correctness gate could not otherwise be proven, and that one took four lines.  **⚠ THIS ROW WAS WRONG AND I FILED IT. Corrected rather than deleted, because a row that quietly disappears teaches nobody.** It said 'resticStep is not a seam, so no test can drive any restic-backed path'. **The first half is true and the conclusion is false:** `resticStep` is not itself overridable, but the layer it calls — `offboxRunner`, injected by `SetOffboxRunner` (`offbox.go:52`) — **has been a seam since the off-site tier shipped**, and other tests in that package have been driving restic-backed paths through it all along (`offbox_3a_test.go` uses it five times). I read one function and generalised from it. **And the proposed fix would have been actively worse:** a `resticStepFn` seam REPLACES `resticStep`, which would have hidden its `unlock --remove-all` escalation from exactly the assertions that must observe it — R-359's lock-safety test asserts `unlock` never appears in any argv, and it can only do that because the runner seam sees every command. **What the row asked for that WAS real is done:** R-358's AST ordering test is now an execution test through the existing seam (`TestR358_MarkerOrderingIsExecuted`), which immediately surfaced something the AST walk could not — `unlockStale` legitimately runs before the restore. **Nothing is owed. The row survives as the record that the seam EXISTS, so the next session does not re-file it.** | **CORRECTED, NOT CLOSED — the premise was wrong (2026-08-30)** | — | Add `resticStepFn` beside the existing `offboxFreeFn` / `offboxSizer` / `offboxLatestSnapFn` seams, nil in production, and convert R-358's AST test to an execution test. **Do it as its own change, not folded into a bugfix** — a seam added under pressure is how a test ends up asserting the shape of the thing it was written beside. | CC |
 | **R-385** | **A controller was built, baked AND vouched with no CHANGELOG entry of its own, and every gate stayed green.** Controller **0.221.1** shipped on 2026-08-23 while the newest heading in `felhom-controller/CHANGELOG.md` still read `v0.221.0` — the prune-ordering fix (commit `810b18a`) had been written INSIDE the v0.221.0 entry instead of getting its own. The image was never in question; the RECORD was, and the fleet ran a version the record did not name. **`scripts/golden_currency_gate.py` could not catch it by construction:** it failed only on `released > baked`, so a golden AHEAD of the record passed silently. Measured on the real history: `newest released 0.221.0 / newest golden baked 0.221.1 → OK, exit 0`. | **CLOSED — 2026-08-23** | — | **Both halves fixed, both directions red-proofed.** The record: `v0.221.1` has its own heading carrying the MOVED (not duplicated, not deleted) reasoning — commit `da75603`, pushed alone before anything else. The gate now asks *"is the baked version WRITTEN DOWN?"* — the baked version must have its own `## vX.Y.Z` heading **anywhere** in the CHANGELOG. **Membership, not `baked > released`, deliberately:** a comparison against the newest heading alone goes green the moment any later entry is written, leaving the unrecorded version permanently unrecorded and the gate permanently silent about it. INCONCLUSIVE (exit 2) preserved. Evidence: `audits/DRILL-r384-dead-db-alarm-2026-08-23/evidence/gate-0*.txt` — old gate/old record `exit 0`, new gate/old record `exit 1`, new gate/fixed record `exit 0`. | CC |
 | **R-387** | **The hub REWRITES an unknown severity and says nothing, and the guard built to catch that sits downstream of the rewrite.** One handler, two fields, opposite discipline: an unknown `event_type` is rejected with a loud `400`, while an unknown `severity` was silently coerced to `info` — after which `severityNotifies` drops it and NEITHER delivery leg runs. **Two shipped features went out that way**: `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0, `app_start_failed` until v0.223.0. **Measured on the live hub DB 2026-08-23: 91 `app_start_failed` events stored all-time and ZERO `notification_log` rows before that day** — not one, on any channel, while every POST returned 200. **The dispatcher's `unrecognized severity` line could never execute** for an API event, because the coercion one line upstream guarantees the value it looks for cannot arrive. | **CLOSED — hub v0.107.0, 2026-08-23** | — | **The coercion STAYS; only the silence is fixed** — a rejected event is a LOST event, and losing an alarm is worse than mis-routing one. A `WARN` now names the customer, the event type, the rejected value and the consequence. **The dispatcher branch was KEPT, on evidence not caution:** `cmd/hub/main.go` wires `dispatcher.ProcessEvent` DIRECTLY as the `monitor.EventNotifyFunc` for the staleness, host-staleness and offsite-box checkers, which never pass through the handler — for them it is the only severity guard there is; deleting it as "dead" would have removed the live half while the dead half supplied the justification. All 90 severity literals in `internal/monitor` verified already valid. Proven live: `[WARN] [api] Event from demo-hp: severity "warn" is not in {info,warning,error,critical}…`, with an `error` control silent. Evidence: `audits/DRILL-r329-r386-2026-08-23/evidence/live-19-scenarioH-after.txt`. | CC |
 | **R-391** | **Gate 11 (observations) is registered in three of the four runners; `app-catalog-felhom.eu` is the exception.** The controller and agent runners already carried a shared-gate mechanism (`SHARED_REUSE`, `SHARED_INSTRUCTIONS` pointing into `felhom.eu/scripts/`), so registering there was one constant and one `GATES` line each. **`catalog_gates.py` has no such mechanism:** its `run_gate` joins every entry against its OWN `scripts/` directory, so it cannot invoke a sibling repo's script at all; and its loop appends `--all` to every gate unconditionally, which the observations gate would read as a path. Registering there therefore needs `run_gate`'s contract widened AND the argument handling changed — a refactor of a runner whose shape is deliberately different (per-app scoping, network/runtime gates excluded from `--fast`), in a repo this task marked out of scope. **The exposure today is nil** — `app-catalog-felhom.eu/REPORT.md` has no observations section, and the gate passes quietly on that — but a future catalog session could write one and nothing would read it. **Filed rather than left as a sentence in a report, which is the exact failure R-389 records.** | **OPEN — LOW** | — | Either give `catalog_gates.py` the `SHARED_*` absolute-path mechanism the other two runners already have and stop appending `--all` to gates that do not take it, or state in that repo's CLAUDE.md that its REPORT.md carries no observations section by convention. **Do not copy the gate script** — the shared checker lives in ONE place (`felhom.eu/scripts/`) and copying it is the drift the shared pattern exists to prevent. | CC |
@@ -533,7 +532,8 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
 | **R-348** | **Every agent restart blanks the reported backup list for up to ~18 hours, and the comment that covers it says "unaffected".** Observed 2026-08-20 while deploying R-344: the first host reports after `demo-hp`'s agent restart carry **`0 backups`** (11:15:50 and 11:30:52 CEST, two consecutive), while the box's own `pvesm list` shows archives present on **both** tiers. `internal/backup/store.go`'s `Store` is in-memory and `byTarget` is repopulated only when a backup **runs** — daily for the local tier, weekly for offsite — so the field reads 0 until the next run. `restore_tests` did **not** blank, because that half has a durable on-disk companion (`RestoreTestState`, R-189). **It blinds no alarm, and that was CHECKED rather than assumed.** `hub/internal/monitor/deadline.go` scans back over stored reports with a 7-day `backupEvidenceLookback` whose own comment names this exact case — *"when the LATEST report carries none... and against an agent that stayed restarted for days"* — and `pbs_snapshots` stayed populated at 2 regardless. So this is an observability wart, **not** a safety hole, and it is filed at that severity deliberately. **What is actually wrong is the comment.** The `Store` doc says *"Backups are unaffected — their freshness has a ground truth on the storage (R-84)"*. That is true of the **consequence** and false of the **field**, and it sits three lines below a paragraph explaining that the very same sentence about restore-tests *"used to be here and it is now FALSE"* — so the file already carries one correction of this shape and invites the next reader to trust the surviving half. | **READY (XS) — NEW 2026-08-20** | — | Say what is measured: the field IS lost on restart and repopulates only when a backup runs; the freshness VERDICT is unaffected because the hub looks back 7 days. **Name `backupEvidenceLookback` in the comment** so the cross-repo dependency is visible from the agent side — today the agent's claim of safety rests on a hub constant it does not mention. Per the workspace rule, a comment asserting an invariant needs a test pinning it: the pin belongs on the HUB side, asserting the verdict survives a report carrying `backups: []`. | CC |
 | **R-349** | **"Prove it by hand, then publish" leaves the fleet running a DIFFERENT binary under the SAME version name — and self-update cannot notice.** Hit on 2026-08-20 during the R-344 train, caught and corrected the same hour, filed because the next prove-then-publish train will hit it identically. **The mechanism:** a proof deploy is a hand build (`go build -ldflags "-X main.version=0.130.0"`), while `scripts/release-agent.sh` deliberately builds with **`-trimpath -buildvcs=false`** so the published artifact is reproducible (R-186). Same source, same version string, **different bytes**: `256e0829...` on the boxes vs **`a56a92a7...`** published and vouched. **Nothing corrects it automatically**, and that is the sharp edge: the boxes already report `0.130.0`, so the self-update path sees the vouched version as already installed and does nothing, **forever**. The divergence is invisible to every version check in the system — the hub, `--version`, and the artifact manifest all agree, because they all compare the version STRING. **Consequence if unnoticed:** the binary a customer box runs is not the binary the operator vouched, and not the one a reinstall would fetch — so a bug reproduced on the fleet may not exist in the published artifact, or vice versa. It is the same "one version name, two binaries" hazard `publish-agent.sh` already carries a comment about for `CGO_ENABLED`; that comment fixed the two ENTRY POINTS and does not cover a hand build during a proof. **Corrected here** by downloading the published artifact from the registry (not rebuilding it locally — the boxes get the bytes a fresh install would get) and installing it on both; both now report `sha256 a56a92a7...`, matching the vouch. | **READY (S) — NEW 2026-08-20** | — | Make the reconciliation a step, not a memory: the honest fix is for the agent to REPORT the sha256 of its own binary in the host report, so the hub can compare it against the vouched `agent_sha256` and flag drift — **exactly the mechanism `wrapper_sha256` already implements for the PBS wrapper** (R-50b), whose manifest help text says it *"makes host drift visible: agents report the installed file's hash and a mismatch is surfaced on the host page"*. The pattern exists and is proven; it simply was never extended to the agent's own binary. Cheaper interim: end every prove-then-publish train by installing the DOWNLOADED artifact. | CC |
 | **R-350** | **SECURITY — the hub operator password was printed in cleartext into a session transcript by CC, 2026-08-20. Rotation recommended.** **What happened:** vouching the artifact manifest used `curl -w '%{redirect_url}'` for confirmation. The hub answers `POST /configuration/artifacts` with a **303**, and curl renders the redirect target **with the basic-auth credentials re-attached** — so the URL it printed contained `http://:<HUB_PW>@10.43.52.34:8080/configuration?flash=artifacts_set`. The password was never read aloud from the credentials file, never echoed deliberately, and every other call in the session correctly printed only `${#HUB_PW}`; it arrived through curl's own output formatting, which is why the usual discipline did not catch it. **Blast radius, stated precisely rather than minimised:** the value is **not** in git, not in `CHANGELOG.md`/`REPORT*.md`/any committed file (checked), and not in the evidence directory — it is in the Claude Code session transcript under `~/.claude/projects/` on DooPlex, which is operator-readable and persists across sessions. The hub UI is reachable only on the k3s ClusterIP and via the operator's own routes, not from the internet. **The value is deliberately not recorded here; it is stored out-of-band in the usual credentials file.** | **READY (S) — NEW 2026-08-20** | — | **Operator decides whether to rotate.** The hub's own `/configuration` password form does it (`current_password`/`new_password`/`confirm_password`), and per `hub-password-ui-2026-07-13` the DB override wins over the ConfigMap, which stays break-glass. CC can perform the rotation **file-to-file without printing the new value** (the `operator-present-one-time-secrets` convention) if asked — it did not do so unilaterally, because rotating a credential the operator holds in their own head or notes is their call, not CC's. **The reusable half, which matters more than this one password:** never use curl's `%{redirect_url}` (or `-v`, or `--libcurl`) against a basic-auth endpoint — all three re-render the credential. Confirm a redirect with `%{http_code}` and read the flash from a follow-up GET. | **Viktor decides**, CC executes |
-| **R-359** | **The off-site restic store is never verified by anything, ever.** The complete set of restic verbs in the controller is `restore, snapshots, backup, unlock, stats, init, forget, prune, cat` — **no `check`**. The agent's `RestoreTest` is PBS-tier only. Established 2026-08-21 by deliberately corrupting one pack: `restic check` catches it immediately („ciphertext verification failed", „Fatal: repository contains errors"), and the product only meets the damage when a customer is already trying to recover. | **OPEN — MEDIUM** | — | A periodic `restic check` (structure) with an occasional `--read-data`, reported like any other backup verdict. Note PBS already has verify jobs; this is the tier that does not. | CC |
+| **R-399** | **How deep should the off-site integrity check go, and how often — VIKTOR RULES, CC EXECUTES.** The check ships at STRUCTURE depth (`monitoring.integrity.read_data_subset` empty). **⚠ THE FRAMING THIS ROW WAS FILED WITH WAS TOO NARROW AND THE MEASUREMENT SAYS SO.** It was scoped as a bandwidth-and-cadence question. It is more than that: **the structure check does not catch silent corruption at all.** Measured on demo-hp 2026-08-30 against a throwaway repo whose pack was corrupted WITHOUT changing its size — `restic check` returned `no errors were found`, **exit 0**; every `--read-data*` form returned `Pack ID does not match…` and exit 1. The structure check verifies the index, the pack inventory and the snapshot graph — it catches missing packs, broken indexes and unreadable snapshots, which are real failure modes — but it does **not** re-hash pack contents. **THE THREE NUMBERS, ALL MEASURED, NONE ESTIMATED:** live store **140 829 678 B (134.3 MB)**, 2 651 blobs, 67 snapshots; structure check **35.0 s**; and the full depth curve — 10% **35.9 s (+3%)**, 50% **37.3 s (+6%)**, 100% **39.2 s (+12%)**. **At today's size, re-reading ALL the data costs about four seconds more than reading none**, because the wall clock is dominated by SFTP round-trips over the WireGuard tunnel rather than transfer. **The caveat that keeps this honest:** these do NOT extrapolate — the structure check's cost tracks the INDEX, a read-data run's tracks the DATA, so a 50 GB store is ~370x the data and this curve says nothing about it. **WHAT HAPPENS IF YOU DO NOTHING:** the structure is checked weekly and **the data contents are never re-read**, so bit-rot inside a pack is not detected by anything in the product until a restore needs that pack. | **OPEN — DECISION (Viktor); CC executes in one config line** | — | Set `monitoring.integrity.read_data_subset` (accepted forms: `n/m`, `N%`, or a size like `50M`) and, if it should differ from weekly, `monitoring.integrity.max_age_days`. **Whoever turns read-data on must revisit `integrityCheckTimeout` (30 min)**, which was sized for a structure check and is noted as such at the constant. A larger store changes the arithmetic and this row should be re-measured before a fleet-wide default is chosen. | Viktor |
+| **R-400** | **The debug page has EIGHT buttons that post to endpoints which do not exist — not one.** Filed because v0.227.0 fixed one of them (`backup/integrity`, the seventh built-but-never-wired instance in this project) and the sweep the task asked for found the pattern is far wider than the single case. **Measured 2026-08-30** by comparing every `/api/debug/...` reference in `debug.html` against every `subpath ==` case in `handler_debug.go`: 24 referenced, 17 dispatched. The seven with no handler are **`backup/crossdrive`, `backup/infra`, `dr/infra-status`, `hub/infra-push`, `storage/simulate-disconnect`, `storage/simulate-reconnect`, `storage/watchdog-status`**. There is a single dispatcher, an exact-match `switch` with no prefix matching and a `default: http.NotFound`, so each of those buttons returns **404** — visible as an error rather than a silent success, which is the one mercy here. **`controller/README.md` documented four `backup/*` debug routes when only two existed**, corrected in v0.227.1. **Why this is a row and not a note:** a debug page is where an operator goes when something is already wrong, and a third of its controls do nothing. It is also the cheapest possible detector — a comparison of two lists — for a defect class this project has now hit seven times. | **OPEN — SMALL** | — | Per button: implement it, or delete it. **Do not leave the third state.** Then add the list-comparison as a gate — it is ten lines and it makes the eighth instance impossible on this surface. Note some may be deliberate stubs for features that moved to the agent (`backup/infra`, `dr/infra-status`); the answer per button is the finding, and deleting a button for a feature that lives elsewhere is still the right act. | CC |
 | **R-362** | **A data drive detached mid-restore is reported as „permission denied".** Observed 2026-08-21 23:15: the guest-visible bind was unmounted 4 s into a scratch restore; the restore correctly failed and wrote nothing to the wrong place, but said „A visszaállítás sikertelen: restore dir: mkdir /mnt/felhom-drives/hdd_1/backups: permission denied". The controller has a drive-state concept (`IsDisconnected`, used by both backup legs) and the restore path never consults it. **A correct refusal that misdescribes why sends the reader at a permissions problem that does not exist.** Creditable in the same test: the agent re-bound the drive 5 s later, unaided. | **OPEN — MEDIUM** | — | Consult drive state when a restore path operation fails on ENOENT/EACCES and name the drive. | CC |
 | **R-363** | **The fill watcher runs once a day, so a filesystem that fills at 03:31 goes unannounced for ~24 h while the backup is already refusing apps.** `sched.Daily("fill-watch", "03:30", …)` (`cmd/controller/main.go:1092`) plus one startup check. Proven 2026-08-21 23:17: the 69 GB filesystem carrying the Docker data-root, the system namespace and ALL 40-class app data was filled to 99% / 1.2 GiB free; the backup reserve refused `kimai` per app and the hub received `recovery_unit_capture_failed` (error) naming the filesystem, **and the fill watcher said nothing at all**. The package comment says it "warns the CUSTOMER that a filesystem is filling, BEFORE anything fails"; at a daily cadence it frequently cannot. | **OPEN — MEDIUM** | — | The reserve already computes the same numbers every run. Let the watcher share that reading rather than owning a separate daily one. | CC |
 | **R-364** | **Accented-text search is an instrument that silently transforms its input, and discipline alone has failed at least three times.** (1) 2026-07-20, `ssh → pct exec → bash -c`, nearly a wrong "banner cleared" claim (`felhom-controller/.claude/rules/ui-hungarian.md:19-22`). (2) 2026-08-13, `kubectl exec … sh -c grep` returned **0 for three strings that were present**, one step from a wrongly-reported failed hub deploy. (3) 2026-08-21, `tar -tf` rendered `őszibarack.md` as `\305\221szibarack.md`; recording the fixture's name bytes from that listing would have been wrong. **NOTE: that is two inside two weeks plus the founding case a month earlier — a third inside the two-week window is not on record.** | **OPEN — LOW** | — | **PROPOSED, NOT BUILT:** a helper that refuses to report a zero for any pattern containing a byte ≥ 0x80 unless a negative control also returns zero AND an ASCII anchor known to be present returns non-zero. Three probes, one helper, no judgement at the call site — because judgement is what failed. | CC |