REPORT: hub v0.93.0 (R-198 retention fix, R-197 changed-key signal, R-192 honesty half, R-196 five comments, R-202 filed)
gates / gates (push) Successful in 7s

This commit is contained in:
2026-08-04 13:00:38 +02:00
parent 40687b0921
commit 9f31956201
+218 -98
View File
@@ -1,121 +1,241 @@
# REPORT — RECON: can a customer actually get their backups back? (2026-08-04) # REPORT — hub v0.93.0: the retention keeps the key it was built to keep (R-198) + the honesty pass (R-197, R-192, R-196)
**Class:** recon / spike. **No production code. No build, no deploy, no version bump.** **Date:** 2026-08-04 · **Class:** implementation. One additive schema change, no destructive operation.
**Baselines re-read on arrival:** `felhom-agent` `856a127cd60b` (v0.124.1) · `felhom-controller` **Deployed:** `gitea.dooplex.hu/admin/felhom-hub:0.93.0`, ArgoCD **Synced**, rollout complete, schema
`0887fd676d60` (v0.194.0) · `felhom.eu` `d26f49ad680e` (hub v0.92.0). All three trees clean. change **confirmed applied on the live database**.
> Supersedes the earlier 2026-08-04 report (R-193/R-192 spike + R-195), per the overwrite convention. > Supersedes the earlier 2026-08-04 report (the RECON session), per the overwrite convention. Nothing
> Nothing durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `OPEN-ITEMS.md` > durable from it lived only here — it is in `audits/RECON-offsite-dr-chain-2026-08-04.md`,
> R-192/R-193/R-195/R-196/R-197 and `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`. > `OPEN-ITEMS.md` R-198…R-202 and the capability map.
**Deliverable:** `documentation/audits/RECON-offsite-dr-chain-2026-08-04.md` — Q1Q8 with method,
measurement and ruling per question, the eleven-link chain table, the drill design, every query used,
and the operator's open decisions.
--- ---
## 1. What was established ## 1. Confirmed baselines, and the three landmarks
**The chain from a dead node to an open repository is not assembled, and the failure is deeper than | Repo | on arrival | end state |
"some links are manual".** |---|---|---|
| `felhom.eu` | `3f2b7bc02386`, clean, == `origin/main`, hub v0.92.0 | `40687b0`, hub **v0.93.0** deployed |
| `felhom-controller` | `0887fd676d60`, clean | **untouched** — Part 5 hit its gate (§7) |
1. **Links 14 are PROVEN-LIVE** — mint → stage to the agent → seal under R → store on the hub. **§3's three landmarks, re-confirmed in the real source before editing — none had drifted:**
2. **Link 5 is lossy, and this is the finding of the session.** `host_escrow_superseded` has **no
`identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`)
copies only the K-escrow `blob`. The **restic repository password lives in `identity_blob`** — so
the retention feature built for the reinstall-orphan incident does not retain the secret that
incident is about. The **next escrow ceremony destroys the last copy**, and that ceremony is exactly
what the system tells a rebuilt box's customer to run. Both demo boxes crossed that line on
2026-08-04 at 07:15:36 and 07:20:08. → **R-198**
3. **Link 6 has no client anywhere.** `handleReEnroll` / `handleGetRestoreDirective` serve the blobs;
zero callers in the agent, the hub UI, any script or any runbook. Retrieval is `sqlite3` by hand on
the hub pod. → **R-199**
4. **Links 78 are manual-only.** `UnwrapIdentityBundle`'s only production caller is
`--selftest=identity-consume`, reading R from `FELHOM_RECOVERY_CODE`; nothing extracts
`restic_repo_password` from the recovered bundle.
5. **Link 9's form does not exist.** `POST /backup/offbox/inject-password` is routed, handled and
unit-tested; **no template contains it.****R-200**
6. **Link 11 has never happened.** No blob served, no fork-4 bundle unsealed with a real R outside a
unit test, no recovered password injected, no repo reopened, no restore. The one live round-trip
proof (2026-06-10) **predates the `ResticRepoPassword` field by a month**. → **R-201**
**And the most useful thing settled: a fail-closed refusal IS implementable.** The hub already sends 1. `host_escrow_superseded` (`store.go:377`) had `blob` and **no** `identity_blob`.
every box `escrow{identity_blob_present, restic_pw_sha256, created_at}` on every report ACK 2. `identity_blob` was added to `host_escrow` by a **later** `ALTER TABLE` (`store.go:414`).
(`api/handler.go:504-510`), and the controller **discards it** whenever no offbox target exists 3. `demoteCurrentEscrowTx` (`store.go:2547`) selected six columns and not that one.
(`report/escrow_confirm.go:75-84`). Persisting it needs no new hub API and no new secret; the
`ClaimSync` set-only pattern is the precedent.
**The operator's ruled recovery-screen shape is buildable** and is priced row by row in §9 of the **Highest register ID on arrival: R-201.** Grepped `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
findings doc: fresh-install signal exists (the mint branch's own `os.Stat`); hub-has-a-blob exists on `CONTEXT.md` before minting. One new item: **R-202**.
the wire; the preview is cheap and genuinely read-only (`restic snapshots --json` +
`stats --mode raw-data --json` are already the box's own counters); the one real cost is crossing
agent→controller for the unseal, because the controller image ships no `age` — one new agent local-API
endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts/<id>/escrow`.
## 2. Corrections owed and made ## 2. §8.1's two facts — both LOOKED AT on the live database, not deduced
- **Yesterday's spike annotated, not rewritten** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): Read from a `hub.db` + `-wal` + `-shm` snapshot taken after the deploy (`PRAGMA integrity_check`
a dated block at the top plus an inline note on Q8(b). Its candidate (b) verdict *"not implementable `ok`; freshness proven positively: `MAX(host_reports.received_at)` = `2026-08-04 10:45:39` vs
against a zero-knowledge escrow"* is **too strong** — unattended recovery is impossible; a `datetime('now')` = `10:58:59`).
customer-present recovery is not. Everything else in that spike stands, and its Q3 remains unmeasured.
- **Capability map corrected** (`architecture/00-capability-map.md`, escrow-ceremony row): the hub
v0.60.0 retention claim *"a previous passphrase stays recoverable with its recovery code"* is struck
through and replaced with what the code does — K-escrow only, no read path, `ListSupersededEscrow`
has zero production callers.
- **The same false promise is made to the customer in Hungarian** on the orphan card
(`controller/internal/web/templates/backups_remote.html:66,69`). Recorded in R-198; **not edited**
this session ships no controller change.
- **The prompt's Q4 premise is a conflation, and it is corrected in the doc:** `escrow_confirm.go`
**does** fire on a mint (pending branch, hash mismatch, one WARN). The "wired to the wrong path"
finding was about the hub's `escrow_stale` event (R-196), a different mechanism. What is missing is
not the detection but the consequence — the warning reaches a log file and prescribes the act that
closes the recovery window.
## 3. Register **(a) Nothing is backfillable, and here is the looking:**
- **New: R-198, R-199, R-200, R-201** (highest previous ID was R-197; grepped across `OPEN-ITEMS.md`, | id | host_id | `identity_blob IS NOT NULL` | superseded_at |
`ROADMAP.md`, `STATUS.md`, `CONTEXT.md` before minting). |---|---|---|---|
- **Updated: R-193** (seven-part addendum: (b) overturned in part, the chain census, the retention | 3 | `demo-hp-bb76ea` | **0** | `2026-08-04 07:15:36` |
finding, fail-closed implementable, the operator ruling priced, the drill, Q3 still unmeasured) and | 4 | `demo-felhom-8363b5` | **0** | `2026-08-04 07:20:08` |
**R-192** (two inputs, no verdict change).
- **`STATUS.md`** updated in operator language: two new "what's broken" entries, three new
"waiting on you" items, a "changed since" line.
## 4. Live measurements taken (read-only) Both retained rows are empty of it, and their source rows were overwritten at those same timestamps —
`host_escrow` now holds the *new* generation. **No path preserved them.** (The prompt asked me to check
whether some path had, because that would have been a better answer than expected. It had not.)
Hub SQLite snapshot with the `-wal` (`kubectl exec … cat`), `PRAGMA integrity_check``ok`, freshness **(b) Who the fix protects, i.e. whose NEXT ceremony now retains a recoverable off-site key:**
proven positively (`MAX(host_reports.received_at)` = `2026-08-04 10:00:40` vs `datetime('now')` =
`10:03:48`). Escrow tables, `PRAGMA table_info` on both, latest-report `$.offsite.*` and `$.claimed`
per customer, the `claimed` 1→0→1 transition across both rebuilds, and the events table since
2026-08-03. Plus one `ls -la --time-style=full-iso` inside guest 9201 proving the data dir, the
`encryption.key` and `offbox/repo_password` are all newborn as of `2026-08-03 07:17:25``07:18:02`.
Every query with its field names and values is in §11 of the findings doc.
**Nothing was written, run, rotated, re-issued, deleted or reconfigured anywhere.** No ceremony, no | host_id | identity blob present | bytes |
`selftest=identity-consume`, no touch on the storage endpoint. |---|---|---|
| `demo-felhom-8363b5` | yes | 572 |
| `demo-hp-bb76ea` | yes | 572 |
## 5. What could not be established **2 of 2 hosts with a current escrow.** Both were one ceremony away from losing their repository
password again.
The 2026-08-05 ~02:15 UTC run's outcome (still the decisive measurement; neither box has run — **Schema change confirmed applied live**`PRAGMA table_info(host_escrow_superseded)` on the running
`last_run` absent on both); whether demo-hp's data dir was fresh (measured only on demo-felhom; pod's database returns `identity_blob BLOB` as column 8. It runs at startup, so this is the check that
demo-hp needs the break-glass credential this read-only session did not use); whether the orphaned the startup actually did it rather than the code merely containing it.
ciphertext still exists at the provider; whether `age` is present on either demo host. Named in §12
of the findings doc.
## 6. Observations recorded, not acted on ## 3. Files modified, and the commits
A fresh `encryption.key` on every rebuild makes every pre-rebuild `app.yaml` `ENC:` secret **Commit `91cabdd`** — code + docs:
undecryptable — circular dependency **C-1** of `_recovery-inventory-2026-07-28.md`, which fired live on
both demo boxes this week and is a strictly larger blast radius than the offsite repo. `Claimed`
returning to 1 with no event means "the customer claimed this box" has no audit trail.
`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting. `handleReEnroll` returns a
freshly minted API key in its response body on an endpoint nothing currently calls.
## 7. Conventions | File | Change |
|---|---|
| `hub/internal/store/store.go` | `identity_blob` on `host_escrow_superseded` (CREATE + additive `ALTER`); carried in `demoteCurrentEscrowTx`; `ListSupersededEscrow` reads it; `HostEscrow.IdentityBlob`; new `CountCurrentEscrowWithIdentity`; `SaveHostEscrow` returns the replaced hash; the table comment rewritten to describe what the code now does **and** to record that it once did not; three R-196 comment corrections |
| `hub/internal/api/handler.go` | `maybeEmitRepoKeyChanged` + `eventRepoKeyChanged`; wired into the superseded branch of the escrow PUT; one R-196 comment correction |
| `hub/internal/monitor/offsite_delivery.go` | `deliveryShape` (burned/regressed); two-shape message; `offsite_reports_since_consume` in the details; `recordHealRefusal`; refusal branches split so each names its reason |
| `hub/internal/notify/dispatcher.go` | `offsite_repo_key_changed`, `offsite_delivery_stuck`, `offsite_credential_restaged``operatorOnlyEvents` |
| `hub/internal/offsite/offsite.go` | one R-196 comment correction |
| new: `store/escrow_identity_retention_test.go`, `api/escrow_repo_key_changed_test.go`, `monitor/offsite_delivery_honesty_test.go` | Scenarios AG |
| 11 existing `_test.go` files | mechanical: `SaveHostEscrow` now returns three values |
| `hub/CHANGELOG.md`, `CONTEXT.md` (S-28), `REUSE.md`, `STATUS.md`, `00-capability-map.md`, `03-host-agent.md`, `OPEN-ITEMS.md`, `ROADMAP.md` | docs |
- **No `CHANGELOG.md` entry.** `felhom.eu` has no root changelog — they are per-area (`hub/`, **Commit `40687b0`**`manifests/hub.yaml` `0.92.0``0.93.0`.
`scripts/`, `website/`) and log code changes. This session changed none of those areas. Stated
rather than silently skipped. ## 4. Tests, and every red-proof outcome
- **`REUSE.md`** untouched — no shared helper or pattern moved.
- **Gates + CI run id** are recorded in the session summary. Green gate after each phase: `go build ./... && go vet ./... && go test ./...` in `hub/`**rc=0**,
full suite, run separately from every commit (never combined).
| Group | Scenario | Result | Red-proof — mutation → outcome |
|---|---|---|---|
| A | re-escrow retains both keys | PASS | drop `identity_blob` from the shared copy routine (**= production ≤ v0.92.0**) → **FAIL**: *"the retained row carries NO identity blob — the off-site repository password was destroyed by the ceremony that was supposed to preserve it"* |
| B | host delete demotes both | PASS | same mutation → **FAIL**. **Plus the §8.2 mutation**: fix only the re-escrow caller (patch the retained row in `SaveHostEscrow` instead of the shared routine) → **A passes, B FAILS** — the "fixed one caller" mistake, demonstrated |
| C | legacy host with no identity blob supersedes cleanly | PASS | — (a nullable-column check; no guard to remove) |
| D | changed key raises one signal, host named, no hash values | PASS | remove the comparison from the escrow PUT → **FAIL**: *"the repository key demonstrably changed and NO signal was raised"*, while E still passes |
| E | unchanged key is silent | PASS | — |
| F | message states the measured state | PASS | restore the single hardcoded sentence → **both message tests FAIL** |
| G | heal refusal leaves a durable record | PASS | replace the regressed branch with a bare `return` (**= production ≤ v0.92.0**) → **FAIL**, and its cadence test with it |
| — | `IsOperatorOnly` for all three event types | PASS | — |
| — | `CountCurrentEscrowWithIdentity` census | PASS | — |
**A mutation that did not remove every guard, recorded because rule §9.12 requires it.** My first
Scenario-F mutation kept the `switch`'s `default` branch, so the *regressed* test passed for the wrong
reason and only the *burned* one failed. That is not a red-proof. The mutation that does fail both is
deleting the shape switch entirely and emitting the one v0.92.0 sentence — done, observed, recorded.
**Both callers proven separately** (§8.2): `TestSaveHostEscrow_RetainsIdentityBlob` (re-escrow) and
`TestDeleteHost_DemotesIdentityBlob` (host delete). The pre-existing
`TestSaveHostEscrow_RetainsSuperseded` stayed **green throughout the two months the bug existed**,
because it asserts the mechanism (a retained row exists, carrying the old K-blob). The new tests assert
the consequence (the retained row can still yield a repository password) — `CLAUDE.md`'s rule, applied
to the case that earned it.
**Every `-run` filter was verified to have matched** (`-v`, `=== RUN` lines counted) before any result
was read as a proof.
## 5. The exact operator message text, quoted for review
**R-197 — `offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession):
> Offsite repository key CHANGED for host `<host_id>`: the new escrow seals a different repository
> password than the one it replaced. The previous off-site history is no longer opened by this box's
> current key. The superseding blob was retained (`<N>` held), so that history stays recoverable with
> the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an
> orphan card, and expect the next backup to start a fresh history.
Details JSON: `{"host_id": "...", "retained_count": N, "repo_key": "changed"}`**no hash values**.
**R-192 — `offsite_delivery_stuck`, burned shape** (unchanged in intent, corrected in wording):
> Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed `<age>` ago; of
> the first `<N>` report(s) after that consume, NONE carried an offbox target, and the latest report
> carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.
**R-192 — regressed shape** (the demo-hp case, which previously got the burned text):
> Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed `<age>` ago; of
> the first `<N>` report(s) after that consume, `<M>` DID carry an offbox target — and the latest report
> carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT
> the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic
> restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports
> after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).
**R-192 — the refusal record** (`notification_log`, channel `operator`, status `refused`), message
*"Automatic offsite credential restage was NOT performed."* with the reason in `error_message`, e.g.:
> regressed-apply shape: `<M>` of the first `<N>` report(s) after the consume DID carry an offbox
> target, so a burned credential is ruled out — a restage would treat a symptom whose cause is
> elsewhere. Operator's call (R-193).
## 6. The corrected comments — FIVE, not three, with their locations as found
The spec expected three and named two; a census of every `restic` mention in the hub found **five**.
| # | Location | What it claimed |
|---|---|---|
| 1 | `hub/internal/offsite/offsite.go:198` | *"the restic repo password just changed"* as the justification for `MarkEscrowStale` |
| 2 | `hub/internal/api/handler.go:1067` | *"The re-issuer resets the restic repo password, which makes the OLD escrow blob stale"* |
| 3 | `hub/internal/store/store.go:423` (the `stale_at` ALTER) | *"stale_at is set when the offsite repo password is re-issued"* |
| 4 | `hub/internal/store/store.go` `MarkEscrowStale` doc comment | *"called when the offsite repo password is re-issued"* |
| 5 | `hub/internal/store/store.go` `EscrowStatus.Stale` field comment | *"true when the offsite password was re-issued"* |
All five now state what the code does, name the correction with its date and item, and cite the recon.
The staleness mark is documented as **precautionary** — the box's re-apply may mint a fresh repository
password — with R-197's measured signal named as the evidential one. **The behaviour is unchanged and
R-196 stays OPEN**, because the false-staleness alarm is a behaviour change that must not ride a
comment-correction release. It also matters *more* now than when filed: under R-198 an unnecessary
ceremony is no longer harmless bookkeeping, it supersedes a blob.
## 7. Part 5 — the gate was hit; the card is untouched; R-202 filed
**The gate:** ship it iff the hub can tell a box what it needs with **one** additional boolean on the
escrow ACK it already sends.
**The hub can compute such a boolean cheaply** — *"≥1 retained blob for this host carries an identity
blob"*, one correlated predicate in `GetEscrowStatusForCustomer` — and the controller even has the right
seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer
the card's question.** The card renders on `RepoState == "orphaned"`, and its promise is about *the key
THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned
remote belongs to. A box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the
boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified,
which on a customer-facing card is worse than today's hedged one. Making it truthful needs the orphan's
generation, which is the same knowledge R-199/R-201's unassembled chain needs.
**So: stopped, filed R-202, left the card alone**, per §8.6. **Stated plainly rather than buried: the
false sentence is still live**, in Hungarian, on both demo boxes. The cheapest honest interim — drop the
recoverability clause and say only that the old history is set aside and not deleted, which is true
unconditionally — is recorded in R-202 and **not taken here**, because it is a customer-copy change and
the gate said leave it alone.
## 8. Register
- **R-198 → SHIPPED** (hub v0.93.0), with the §8.1 facts and both red-proof outcomes recorded.
- **R-197 → SHIPPED** (hub v0.93.0).
- **R-192 → HALF SHIPPED**: both honesty halves done; **the guard's 500-oldest-reports scoping stays
OPEN**, deliberately, because its correct shape depends on the recovery chain that is not yet
assembled. The window is named inside the alert text so the limitation travels with the number.
- **R-196 → comments corrected; the BEHAVIOUR stays OPEN.**
- **R-193 → still open**, updated with the three operator rulings.
- **R-202 → NEW** (Part 5's gate).
- **R-199, R-200 and R-201 remain OPEN and untouched** — the hub's blob-serving endpoints still have no
client, the injection seam still has no form, and nothing in the recovery path has ever been
exercised. v0.93.0 makes the key survive; it does not assemble the chain that hands it back.
- **The three operator rulings of 2026-08-04 are recorded** in a dedicated block at the top of
`OPEN-ITEMS.md` (a ruling that lives only in a conversation binds nobody — the R-96 rule):
run the drill after R-198 (**R-198 has shipped; the drill is next**); **delete the orphaned
ciphertext — STILL OWED**, deliberately not done here because a destructive act on a protected
endpoint does not ride a schema-change release; and **accept the risk on R-193(c)**, which is what
makes this retention load-bearing rather than tidy.
- Capability map, `03-host-agent.md` §8a (a new *Custody across generations* section — the contract
changed, so S-1 applies), `CONTEXT.md` S-28, `REUSE.md` and `ROADMAP.md` all updated in the same
commit.
## 9. Deploy & validation
- Clean-tree gate held per repo before the build; the image was built from a pushed tree.
- `./build.sh 0.93.0 --push``felhom-hub:0.93.0`, 25 MB. Manifest bumped in git, committed, pushed.
- ArgoCD hard-refresh + **deliberate sync** (never `kubectl set image`). **Synced**; `deploy/hub`
rolled out; live image `gitea.dooplex.hu/admin/felhom-hub:0.93.0`; startup log clean (all checkers
initialized, `Listening on :8080`, no errors).
- **Live validation was structural, not manufactured** (§9.3): the schema column read from the running
pod's database, and the two censuses in §2. **No ceremony was run on any box** — today, a ceremony is
the act that destroys the key, and it is also the one thing that would have "proved" the retention by
consuming a real customer's generation.
- Disk headroom checked before the build (`/mnt/5_hdd` 24%, `/` 86%).
- **Nothing provisioned; nothing to tear down.**
- Gates: `python3 scripts/repo_gates.py --fast` → all OK, after every docs change.
## 10. CI
- Code commit **`91cabdd`** and manifest commit **`40687b0`** — run numbers, task ids and conclusions
quoted in the session summary; **`--no-verify` was NOT used** (the pre-push hook ran `repo_gates.py
--fast` and passed on both pushes).
## 11. Observations — noticed, not acted on
1. **`ListSupersededEscrow` still has no production caller.** v0.93.0 makes it *return* the identity
blob, so a retained key is reachable from Go for the first time — but nothing reads it. That is
R-199's territory and was deliberately not widened here.
2. **`handleReEnroll` returns a freshly minted API key in its response body** on an endpoint nothing
currently calls. Worth a look *before* anything starts calling it (R-199).
3. **`CountReportsOffsiteSince`'s `LIMIT 500` is the only instrument in the delivery checker**, and its
window is now named in operator-facing text. If R-192's remaining half is ever fixed, that sentence
must change with it — the R-100 corollary (an alarm whose text stopped matching what its verdict
counts) applies directly.
4. **`offsite_delivery_stuck` had been reaching the customer channel by construction** for its whole
life; the only reason no customer row exists is that the demo customers have no configured
recipient. An absent notification row is not evidence a leg is blocked.
</content> </content>