From bdab80c93383a489d7692ad437fbb7740799d4a7 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 4 Aug 2026 13:56:01 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20R-200=20diagnostic=20=E2=80=94=20CONTEX?= =?UTF-8?q?T=20+=20REPORT=20(proven=20live=20on=20demo-felhom)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CONTEXT.md | 20 +++++++++ REPORT.md | 125 +++++++++++++++++++---------------------------------- 2 files changed, 64 insertions(+), 81 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 8ab3bf0..fa4d2b6 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -1672,6 +1672,26 @@ Last updated: 2026-06-13 (v0.60.0 backlog-Medium cleanup) --- +## R-200 (v0.195.0) — the offsite key recovery diagnostic + +`--recover-offsite-check` is a `docker exec` escape hatch (the `--print-reset-code` shape), NOT a page +or an API a browser can reach. R comes from **STDIN** — never argv, never `ps`, never shell history, +never a transcript. It asks the agent (>= v0.125.0) to fetch this host's sealed bundle and open it, +then reports whether the recovered repository password matches the on-disk one **by sha256**. + + docker exec -i felhom-controller /usr/local/bin/felhom-controller --recover-offsite-check < /path/to/code + +**IT COMPARES AND NEVER INSTALLS.** `CheckOffsiteKeyRecoverable` must stay free of any write — if a +future change makes it place the recovered password, it stops being a diagnostic and needs the drill's +supervision (that is link 9, R-200's remaining half). Pinned by +`TestCheckOffsiteKeyRecoverable_WritesNothing`, whose red-proof is adding the install call. + +Exit codes are load-bearing: **0** match, **2** a clean MISMATCH, **1** a step failed. A mismatch is a +finding about the system; a failure is a finding about the run, and they must never share a status. + +**Proven live on demo-felhom 2026-08-04** — recovered sha256 == on-disk sha256 == the hub's stored +hash. Nothing customer-facing ships with it: no card, no form, no preview. + ## About Viktor (project owner) - Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business diff --git a/REPORT.md b/REPORT.md index a6a950f..c3568a3 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,97 +1,60 @@ -# REPORT — R-182: one operator email per backup run, and nothing dropped without a trace +# REPORT — controller v0.195.0: prove the offsite key comes back (2026-08-04, R-200 plumbing half) -**Date:** 2026-08-03 · **Repo:** `felhom-controller` · **v0.193.1 → v0.194.0** · commit `88897a2` -**Baseline on arrival:** `db0d4b129d3b`, v0.193.1 — matched §1. +**Deployed:** `gitea.dooplex.hu/admin/felhom-controller:0.195.0` on guest 9201 (demo-felhom), `Up (healthy)`. +**Requires agent >= v0.125.0** (declared MinAgent) and hub >= v0.94.0. -## The defect +## What shipped -Nine per-app `recovery_unit_capture_failed` events reached the hub on 2026-08-03 and **two operator -emails went out**. The hub's operator cooldown key is `customerID:eventType(+tier)`, and that event -carries `app` but **no `tier`** — so the key held no app identifier. The first refused app took the -hour's slot and every other app's failure was discarded **before `LogNotification`**, leaving no row -on any channel. +`--recover-offsite-check` — a `docker exec` diagnostic in the shape of `--print-reset-code`. It reads +the customer's recovery code from **STDIN**, asks the agent to fetch this host's hub-held sealed bundle +and open it, and reports whether the recovered offsite repository password matches the one on disk +**by sha256**. Two hashes and a verdict; never a password, never the recovery code, never a blob. -**The obvious fix was ruled against:** putting `app` in the key produces one email per failing app, -which on a full disk is a dozen. +- `internal/backup/offbox_recovery_check.go` — `CheckOffsiteKeyRecoverable`, hashes only. +- `internal/backup/offbox_recovery_cli.go` — the entry point, R from stdin. +- `internal/agentapi/escrow.go` — `RecoverOffsiteRepoPassword` (the only place R crosses outbound). +- `cmd/controller/main.go` — the flag, alongside `--print-reset-code`. -## What ships +**R from stdin and not a flag** because a flag value is visible in `ps`, in shell history, in a +container's command line and in any session transcript. **It compares and never installs** (§8.5): +the recovered password is not written to `offbox/repo_password`, and link 9 — placing it so the +existing repository opens — is next session's work, with the drill around it. -`internal/backup/runsummary.go` — a per-run collector with exactly `admissionSet`'s lifetime, fed by -all three write legs, emitting `backup_run_failures` **once** at the end and **only when something -failed**. The per-app event stays and becomes the record. +## THE RESULT -- **A refusal is noted ONCE, inside `admitApp` where the verdict is taken** — not at the three legs - that consult it. R-181's contract is one verdict per app per run; noting per leg listed a single - refused app three times and produced *"2 of 1 apps failed"*. **Found by the digest's own test.** -- **Deliberate skips are excluded** — a disconnected or decommissioned drive has its own alert. -- **A manual run always reports:** a unique `run_id` the hub's cooldown cannot collapse. - -**A gap the spec did not anticipate, and its fix.** `recovery_unit_capture_failed` also fires from -`GetFullStatus`'s periodic sweep, outside any run. With it now record-only, those failures would have -been recorded and **never notified** — a new silence created while closing one. The sweep therefore -emits a digest too, deliberately with **no `run_id`**, so it stays under the ordinary hourly cooldown -exactly as before while the mail now lists every failing app instead of whichever was first. - -## §3's safety property — confirmed, not assumed - -`hub/internal/monitor/deadline.go:396` (`expected_backup_missed`) and `:417` -(`expected_dbdump_missed`) are raised by the **hub**, from `assessBackupFreshness(reportJSON,…)` and -`GetEventsByType` — i.e. from the box's report freshness and stored events, **independently of any -email the controller chooses to send**. A digest's silence therefore still means "the run finished -and found nothing wrong". This is what makes the whole design safe, and weakening that check -re-opens a silent-failure path. - -## Files - -`internal/backup/runsummary.go` (new) · `runsummary_test.go` (new) · `admission.go` · -`admission_test.go` · `backup.go` · `recovery_unit.go` · `internal/notify/notifier.go` · -`internal/web/handler_debug.go` · `internal/api/router.go` · `cmd/controller/main.go` · -`CHANGELOG.md` · `REUSE.md` - -## Tests — `go build && go vet && go test ./...` → **28 packages ok, rc=0** - -7 new tests. `controller_gates.py` → all 8 OK. - -### Red-proofs - -| # | Mutation | Result | -|---|---|---| -| A | **all four** `noteFailure` feeds removed (the pre-R-182 per-app-only path) | **RED** — Scenario A and the refresh-sweep test both fail; the digest is never emitted | -| I (1st attempt) | the `main.go` seam wiring commented out | **DID NOT FAIL — recorded as such.** The AST test walked the backup package and not `main.go`, so the seam could be disconnected with the suite green. **The test was fixed**, not the result recorded | -| I (re-run) | same mutation, against the fixed test | **RED** — and the string `SetRunSummaryNotify` is still present in the file, which is why this is an AST walk and not `strings.Contains` | - -## Live proof — demo-hp guest 9201 - -**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button -calls, running the production `RunDBDumps`. No browser on DooPlex. - -Filled for real with `fallocate` to **241 MB free / 100% used**; the thin pool held **30.78 → 30.78** -(instrument re-proven before use — demo-hp's pool is 53.93 GiB and a genuine fill would exhaust it). +On demo-felhom, 2026-08-04 13:49 CEST: ``` -[INFO] [backup] Run summary: 2 of 2 apps failed (manual run) — notifying the operator once -[INFO] Event pushed: backup_run_failures (error) — 2 of 2 apps failed to back up in this manual run: opengist, privatebin + on-disk sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb + recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb + [MATCH] the offsite repository password IS recoverable from the sealed escrow. ``` -Both apps refused, **one** digest naming both. Against the previous behaviour: two refusals, one -email naming one app, one vanishing. Hub-side evidence is in `felhom.eu/REPORT.md`. +Both equal the hash the hub independently stores as `restic_pw_sha256` — three sources agree. +Confirmed non-write: `offbox/repo_password` mtime is still `2026-08-03 07:18:02` afterwards. -**Scenario E:** a second run in the same hour produced a second digest (11:50:45 and 11:51:28). -**Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest. +Five minutes earlier the same path with a deliberately wrong code returned +`[FAIL] … the recovery code did not open the sealed bundle … nothing was written`, exit 1. -## Teardown +## Tests -Fill file removed, guest helper scripts and the credential file `shred`-ed, `pct fstrim 9201` -returned 63.3 GiB, thin pool at **30.76**. Disk back to 1.9G used / 64G free. +`go build ./... && go vet ./... && go test ./...` — rc=0; `controller_gates.py --fast` — all OK. +New: `offbox_recovery_check_test.go` — match/mismatch verdicts, the no-local-password shape, R passed +through verbatim, agent-failure-is-not-a-mismatch, and the four exit codes with a check that no +printed stream carries a secret. -## Observations — NOT acted on +**Red-proof observed:** adding `InjectOffboxPassword` to the check fails +`TestCheckOffsiteKeyRecoverable_WritesNothing` with *"the check INSTALLED the recovered password"*. -1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2. That is exactly the - measured case being closed (two refusals → one email), but a five-app demonstration would be - stronger and needs a box with more apps. -2. **`FormatOperatorEmail` puts an emoji in every operator subject**, including this digest. Part 3 - asked for no emoji; the icon comes from the shared formatter, and changing it globally would alter - every other operator mail and its tests. Left alone deliberately — the digest's own copy has none. -3. **The digest's leg name for a refusal is `whole app (refused before any write)`** rather than a - specific leg, because the reserve refuses all three at once. Accurate, but longer than the other - leg names and it widens the column. +## Not in this release + +No card, no form, no preview, no customer-facing text — building an interface on a chain nobody had +walked is how the preceding three weeks went wrong. The customer-facing shape the operator ruled on +(yell → recovery-code form → preview → proceed) is R-200's remaining half, now priced against a chain +that exists. **R-201 stays open:** no recovered password has been installed, no repository reopened, +no file restored. + +## Observation + +The controller binary is at `/usr/local/bin/felhom-controller`, not `/app/…` — a guessed path cost the +first diagnostic invocation. Worth a line in the runbook when one is written.