222 lines
14 KiB
Markdown
222 lines
14 KiB
Markdown
# REPORT — the offsite backup key comes back (2026-08-04, R-199 closed, R-200 half)
|
|
|
|
**Class:** implementation across three repos + one decisive live proof. Nothing wiped, deleted,
|
|
rotated or reconfigured; no customer-facing change in any language.
|
|
|
|
---
|
|
|
|
## 1. SCENARIO A — THE RESULT, FIRST, BECAUSE IT IS THE SESSION
|
|
|
|
**On `demo-felhom`, 2026-08-04 13:49 CEST, through the real endpoints end to end:**
|
|
|
|
```
|
|
=== offsite key recovery check (R-200) — compares, never installs ===
|
|
on-disk sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
|
|
recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
|
|
[MATCH] the offsite repository password IS recoverable from the sealed escrow.
|
|
Nothing was written: this check compares and never installs.
|
|
EXIT=0
|
|
```
|
|
|
|
**The hashes matched.** And they match a **third, independent** source: `host_escrow.restic_pw_sha256`
|
|
on the hub reads `c60c8bc737a6…` for `demo-felhom-8363b5`, recorded at the ceremony and never involved
|
|
in this comparison. The box's file, the sealed bundle and the hub's record all agree.
|
|
|
|
**The offsite repository password is recoverable from the sealed escrow with the customer's recovery
|
|
code.** That had never been demonstrated — the one live round-trip on record (2026-06-10) predates the
|
|
field by a month.
|
|
|
|
## 2. Part 0 — the recovery code
|
|
|
|
**Answered by the operator before the session began:** the 2026-08-04 07:16/07:20 codes had not been
|
|
kept, so **the operator ran a fresh ceremony on both nodes and saved the codes** (supplied to this
|
|
session as `R_DEMO-FELHOM` in the DooPlex credentials file). No ceremony was run by CC.
|
|
|
|
Measured hub-side, and worth recording because it exercised two of yesterday's behaviours live:
|
|
the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z` (demo-felhom), and
|
|
`restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated them as
|
|
same-password re-ceremonies: **no superseded row was created** and **no `offsite_repo_key_changed`
|
|
fired**. That is v0.93.0's Scenario E, live. It also means yesterday's `identity_blob` retention is
|
|
still unit-proven and structurally verified, **not yet exercised live** — no supersession has happened.
|
|
|
|
Subject box: **demo-felhom** (Tier 0, reachable with a baked key, current escrow). Not demo-hp
|
|
(break-glass only), not DooPlex, not ep0.
|
|
|
|
## 3. The §8.2 decision as built, and where the gate is
|
|
|
|
**Built as §8.2:** `GET /api/v1/hosts/{host_id}/escrow`, authenticated with the box's own per-host key,
|
|
self-scoped (global may read any — the same asymmetry the PUT has).
|
|
|
|
**The single gate point is `api.escrowSelfServiceRetrieval`** (`hub/internal/api/handler.go`), a named
|
|
constant. `false` re-imposes the recovery-mode requirement on this endpoint and changes nothing else —
|
|
the same unseal, the same extraction, the same proof, minus the self-service property. The trade is
|
|
written above the handler in `dr.go`'s style: what it gives out, why it is safe, and **what it changes
|
|
about who is required** — obtaining the blob used to need an operator to arm recovery mode and now
|
|
needs only that box's credential. They still cannot open it. The mitigation is the audit row.
|
|
|
|
## 4. Files, commits, deployed versions
|
|
|
|
| Repo | Commits | Version | Deployed |
|
|
|---|---|---|---|
|
|
| `felhom.eu` | `435f4a5` (code) · `dd08926` (manifest) · docs commit below | hub **v0.94.0** | ArgoCD Synced/Healthy, `felhom-hub:0.94.0` |
|
|
| `felhom-agent` | `6d79047` | **v0.125.0** | published sha `f7d8339b53d9…`, **verified by independent download**, installed on demo-felhom (`felhom-agent --version` → 0.125.0, `active`). **NOT vouched** |
|
|
| `felhom-controller` | `9640e51` | **v0.195.0** | `felhom-controller:0.195.0` on guest 9201, `Up (healthy)` |
|
|
|
|
**Hub:** `internal/api/handler.go` (route + `handleHostEscrowGet` + `escrowSelfServiceRetrieval`),
|
|
`internal/notify/dispatcher.go` (`escrow_blob_served` → `operatorOnlyEvents`),
|
|
`internal/api/escrow_get_test.go` (new).
|
|
**Agent:** `internal/escrow/recover.go` (new — `OffsiteKeyRecoverer`, distinct errors),
|
|
`internal/hub/client.go` (`FetchIdentityEscrow`), `internal/localapi/escrow_recover.go` (new route),
|
|
`internal/localapi/server.go` (seam + registration), `cmd/felhom-agent/main.go` (production wiring +
|
|
the §8.6 selftest message fix), plus `recover_test.go` and `escrow_recover_wiring_test.go` (new).
|
|
**Controller:** `internal/backup/offbox_recovery_check.go` + `offbox_recovery_cli.go` (new),
|
|
`internal/agentapi/escrow.go` (`RecoverOffsiteRepoPassword`), `cmd/controller/main.go`
|
|
(`--recover-offsite-check`), plus `offbox_recovery_check_test.go` (new).
|
|
|
|
**Green gate in each repo after each phase** (`go build && go vet && go test ./...`, rc=0), plus
|
|
`repo_gates.py` / `agent_gates.py` / `controller_gates.py` — all OK. **No test run was combined with a
|
|
commit.**
|
|
|
|
## 5. Tests and every red-proof
|
|
|
|
| Scenario | Result | Red-proof — mutation → outcome |
|
|
|---|---|---|
|
|
| **A** recovered key == on-disk key | PASS (unit, real `age`) + **PROVEN LIVE** | return `bundle.PBSToken` instead of `ResticRepoPassword` → **FAIL**: *"the recovered key is not the sealed repository password (len 24 vs 64)"*. This is the mutation that mattered: every field of that bundle is a non-empty string that looks like a secret |
|
|
| **B** wrong R fails closed, nothing written | PASS + **PROVEN LIVE** | — (the property is age's KDF, asserted rather than guarded, so there is no guard to remove) |
|
|
| **C** a box fetches only its own blob | PASS | delete the `!isGlobal && authHostID != pathHostID` check → **FAIL**: host A was served host B's blob, body and all |
|
|
| **D** no blob → a clean "none" | PASS | — |
|
|
| **E** every retrieval leaves a record | PASS + **PROVEN LIVE** | `if false &&` on the audit block → **FAIL**: *"a sealed recovery blob was served and NOTHING recorded it"* |
|
|
| **F** R persists nowhere | PASS + **PROVEN LIVE** | write R to a temp file inside the unseal → **FAIL** (see the correction below) |
|
|
| **G** the operator DR path is untouched | PASS | — |
|
|
| **H** the seam is wired (AST) | PASS | `_ = escrowRecoverer` with the `Options` field removed → **FAIL**: *"localapi.Options … carries no EscrowRecovery field"* |
|
|
| §8.5 compare-never-install | PASS + **PROVEN LIVE** | add `InjectOffboxPassword` to the check → **FAIL**: *"the check INSTALLED the recovered password"* |
|
|
|
|
**TWO MUTATIONS DID NOT REMOVE EVERY GUARD ON THE FIRST ATTEMPT, AND BOTH ARE RECORDED RATHER THAN
|
|
QUIETLY REDONE** (§9.13 — two of the last four sessions found a red-proof that passed):
|
|
|
|
1. **Scenario F.** The planted leak wrote R into the walked directory and **the test still passed** —
|
|
because a later failing call OVERWROTE the leak file with the wrong code, and the test only scanned
|
|
for the first one. Instrumenting the mutation (printing where it wrote) is what found it. **The test
|
|
was wrong, not the mutation.** It now asserts **emptiness** of the TMPDIR tree as its primary check —
|
|
nothing under it is created by the test, so any survivor is a leak regardless of content — with the
|
|
content scan kept as defence in depth over all three secrets. Re-run: **FAIL**, *"the unseal left
|
|
1 file(s) behind under TMPDIR"*.
|
|
2. **Scenario H.** Commenting the wiring line out failed to **compile** (`declared and not used`), which
|
|
is not a red-proof. The mutation that compiles keeps the variable alive (`_ = escrowRecoverer`) and
|
|
drops the struct field. Re-run: **FAIL** as quoted above.
|
|
|
|
**Every `-run` filter was verified to have matched** (`-v`, `=== RUN` lines counted) before any result
|
|
was read as a proof.
|
|
|
|
## 6. The live validation trail, in order
|
|
|
|
1. **Wrong code first, deliberately** (13:44:44) — hub: `escrow blob SERVED to host demo-felhom-8363b5
|
|
(572 opaque bytes, self_scope=true) — recovery path in use`; agent: `offsite key recovery FAILED …
|
|
err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
|
|
blob): exit status 1"`; CLI: `[FAIL] … nothing was written.`, exit 1.
|
|
**This is Scenario B live AND it proves links 6 and 7 ran independently of the success** — the blob
|
|
was fetched and the unseal was attempted and refused.
|
|
2. **The real code** (13:49:47) — §1's output, exit 0.
|
|
3. **Nothing was written** — `offbox/repo_password` mtime on the live box is still
|
|
`2026-08-03 07:18:02`, unchanged by a check that ran at `11:49` UTC. The whole `offbox/` directory is
|
|
byte-for-byte the pre-check listing.
|
|
|
|
## 7. The retrieval record as it appears at the hub
|
|
|
|
```
|
|
customer_id severity created_at details_json
|
|
demo-felhom warning 2026-08-04 11:44:44 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
|
|
demo-felhom warning 2026-08-04 11:49:47 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
|
|
```
|
|
|
|
And the routing, which is the half that shows the register works:
|
|
|
|
```
|
|
customer_id channel status created_at
|
|
demo-felhom operator sent 2026-08-04 11:44:44
|
|
demo-felhom customer skipped 2026-08-04 11:44:44 ← operator_only
|
|
demo-felhom operator suppressed 2026-08-04 11:49:47 ← 1h operator cooldown, itself recorded (R-182)
|
|
demo-felhom customer skipped 2026-08-04 11:49:47
|
|
```
|
|
|
|
**Both retrievals raised an event; the operator was mailed for the first; the second's mail was
|
|
cooldown-suppressed and that suppression is written down.** The audit is per-retrieval and complete
|
|
even where the mail is not.
|
|
|
|
## 8. Evidence that R persisted nowhere — the search, not a claim
|
|
|
|
Searched on the subject host with the pattern read from a file (never in argv), across the agent
|
|
journal, the controller's container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`:
|
|
|
|
```
|
|
agent journal lines containing R: 0
|
|
controller log lines containing R: 0
|
|
host files containing R: 0
|
|
leftover felhom-idesc-* staging dirs: 0
|
|
staged escrow secret file: 0 entries
|
|
```
|
|
|
|
**With a positive control, because an absent match is not a measurement unless the instrument is shown
|
|
to work:** a copy of R was planted under `/tmp`, the same sweep found **1**, the copy was shredded, and
|
|
the sweep returned **0**. The instrument detects R when R is there.
|
|
|
|
R reached the box only as: the credentials-file value on DooPlex → a pipe → the container's stdin →
|
|
the request body → the agent's memory. It was never an argument, never a file on either machine, and
|
|
appears in no log line at any level.
|
|
|
|
## 9. Register
|
|
|
|
- **R-199 → SHIPPED + PROVEN-LIVE.** Links 6, 7 and 8 assembled and walked.
|
|
- **R-200 → plumbing shipped; the FORM is not built** and the row stays open for it and for link 9.
|
|
- **R-201 → OPEN, and the distinction is deliberate:** the KEY is proven recoverable; a recovered
|
|
password has never been **installed**, no repository has been **reopened** under one, and **no file
|
|
has ever been restored**. The drill's pass condition is unchanged — a byte-identical sentinel file,
|
|
not "the store opened".
|
|
- **R-202 → OPEN and untouched.** The orphan card still promises recoverability unconditionally.
|
|
- **The orphaned-ciphertext deletion (~1.2 GB) is STILL OWED** — operator ruling 2 of 2026-08-04,
|
|
deliberately not ridden along with a code release.
|
|
- Capability map, `03-host-agent.md` §8a, `CONTEXT.md` **S-29**, `REUSE.md` and `ROADMAP.md` updated;
|
|
`STATUS.md` rewritten for the operator.
|
|
|
|
## 10. The capability-map row, and what it deliberately does not claim
|
|
|
|
Added: *"The offsite repository password can be RECOVERED from the sealed escrow with the customer's
|
|
recovery code"* — **PROVEN-LIVE (2026-08-04)**, with the evidence above.
|
|
|
|
**What the row explicitly refuses to claim**, stated in the row itself because the previous over-claim
|
|
on this subject was struck out four hours earlier: it covers the **key**, not the **data**. No
|
|
recovered password has been installed, no repository reopened, no file restored. And the proof used a
|
|
box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is
|
|
exactly what the drill covers and it has not run.
|
|
|
|
## 11. CI
|
|
|
|
Run numbers, task ids and conclusions for all commits are quoted in the session summary.
|
|
**`--no-verify` was NOT used** — every push ran its repo's pre-push gate and passed.
|
|
|
|
## 12. Teardown
|
|
|
|
**Nothing was provisioned.** The temporary pattern file used by the R sweep and its positive control
|
|
were shredded on the host by the same command that created them (verified: 0 files remain). No
|
|
diagnostic artifact persists on either machine. The agent's previous binary is retained as
|
|
`/usr/local/bin/felhom-agent.bak-0.124.1`, per the documented deploy.
|
|
|
|
## 13. Observations — noticed, NOT acted on
|
|
|
|
1. **The `escrow_blob_served` mail is subject to the 1-hour operator cooldown** — visible above, where
|
|
the second retrieval's mail was suppressed. The EVENT rows are per-retrieval so the audit is
|
|
complete, but if retrieval ever becomes routine, a burst would produce one mail. Worth revisiting
|
|
**when** the customer-facing flow lands, not before.
|
|
2. **`ListSupersededEscrow` still has no production caller.** Recovery reads the CURRENT blob; a
|
|
customer needing a PREVIOUS generation's key (which is what R-198's retention exists for) has no
|
|
path at all. That is the next gap in this area after the drill.
|
|
3. **The controller binary is at `/usr/local/bin/felhom-controller`, not `/app/…`** — the first
|
|
diagnostic invocation failed on a guessed path. Worth a line in the runbook when one is written.
|
|
4. **`age` is at the hardcoded `/usr/bin/age` on both demo hosts** (1.2.1 on demo-felhom) — the recon
|
|
listed this as unestablished; it is now established for demo-felhom. `ageBinary` is a package
|
|
variable with no config override, so a host that installs `age` elsewhere would fail the unseal at
|
|
the last step.
|
|
5. **Yesterday's `identity_blob` retention has still not been exercised live** — the operator's fresh
|
|
ceremonies sealed the *same* password, so no supersession occurred. It remains unit-proven with a
|
|
verified schema; the first real supersession will be its live proof.
|