docs: the offsite key is proven recoverable (R-199 closed, R-200 half, R-201 scoped)
gates / gates (push) Successful in 8s

This commit is contained in:
2026-08-04 13:55:25 +02:00
parent dd089265e8
commit 846253baa8
8 changed files with 272 additions and 212 deletions
+35
View File
@@ -17,6 +17,41 @@
## Standing rulings
**S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR
path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller
v0.195.0).**
**The decision, so it is never re-litigated from scratch.** Self-service recovery needs the box to
obtain its own sealed package. The only retrieval that existed was operator-gated behind recovery mode
and the global key (`dr.go`), which cannot serve a customer-present flow. `GET /hosts/{id}/escrow` now
serves that ONE object to its authenticated owner.
- **Why it is safe:** the blob is opaque; the hub has never held R and has no decrypt path; a wrong
code fails closed at age's scrypt KDF (measured live — `exit status 1`, no plaintext, nothing
written); the caller already authenticates as this host for its report, desired state, WG
registration and PBS token, so this adds no new identity, only a new object.
- **What it changes, which is the whole of the trade:** obtaining the blob used to require the
OPERATOR to arm recovery mode. It now needs only that box's credential. **Fewer parties are
required.** The mitigation is that the capability is AUDITED, not silent: every retrieval raises
`escrow_blob_served` (warning, operator-only), written before the bytes leave.
- **The override is one constant.** `api.escrowSelfServiceRetrieval` — false re-imposes the
recovery-mode requirement on this endpoint and changes nothing else. Do not scatter the condition.
- **`handleReEnroll` / `handleGetRestoreDirective` are UNTOUCHED** and must stay so: they rotate the
host API key and serve the K-escrow and the directive as well. Different object, different gate.
Pinned by `TestEscrowGet_OperatorDRPathUnchanged`.
**Facts a future session needs before touching this area:**
- **The unseal lives in the AGENT and must stay there.** `age` is an agent runtime dependency
(`/usr/bin/age`, present on both demo hosts) and is deliberately absent from the controller image;
the blob is host-scoped and its only writer is the agent under the per-host key.
- **The agent returns ONE field.** Not the tunnel token, not the PBS token, not the WG key — the
controller is a trust tier down. Narrowing cost nothing and is not recoverable later.
- **R's rules:** in memory for one call, cleared on every path, never on disk, never in argv, never
logged, never echoed. The controller CLI reads it from STDIN for exactly that reason. Verified live:
0 log lines, 0 files, 0 leftover staging dirs, with a positive control proving the search worked.
- **The chain today: links 18 walked, 911 not.** The KEY comes back. Nothing installs it, reopens a
repository with it, or restores a file — R-200's remaining half and R-201.
**S-28 — the escrow retention now covers the OFF-SITE data key, and customer-present recovery is the
accepted design, which makes that retention load-bearing (2026-08-04, R-198/R-197; hub v0.93.0).**
+176 -196
View File
@@ -1,241 +1,221 @@
# REPORT — hub v0.93.0: the retention keeps the key it was built to keep (R-198) + the honesty pass (R-197, R-192, R-196)
# REPORT — the offsite backup key comes back (2026-08-04, R-199 closed, R-200 half)
**Date:** 2026-08-04 · **Class:** implementation. One additive schema change, no destructive operation.
**Deployed:** `gitea.dooplex.hu/admin/felhom-hub:0.93.0`, ArgoCD **Synced**, rollout complete, schema
change **confirmed applied on the live database**.
> Supersedes the earlier 2026-08-04 report (the RECON session), per the overwrite convention. Nothing
> durable from it lived only here — it is in `audits/RECON-offsite-dr-chain-2026-08-04.md`,
> `OPEN-ITEMS.md` R-198…R-202 and the capability map.
**Class:** implementation across three repos + one decisive live proof. Nothing wiped, deleted,
rotated or reconfigured; no customer-facing change in any language.
---
## 1. Confirmed baselines, and the three landmarks
## 1. SCENARIO A — THE RESULT, FIRST, BECAUSE IT IS THE SESSION
| Repo | on arrival | end state |
|---|---|---|
| `felhom.eu` | `3f2b7bc02386`, clean, == `origin/main`, hub v0.92.0 | `40687b0`, hub **v0.93.0** deployed |
| `felhom-controller` | `0887fd676d60`, clean | **untouched** — Part 5 hit its gate (§7) |
**On `demo-felhom`, 2026-08-04 13:49 CEST, through the real endpoints end to end:**
**§3's three landmarks, re-confirmed in the real source before editing — none had drifted:**
```
=== offsite key recovery check (R-200) — compares, never installs ===
on-disk sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
[MATCH] the offsite repository password IS recoverable from the sealed escrow.
Nothing was written: this check compares and never installs.
EXIT=0
```
1. `host_escrow_superseded` (`store.go:377`) had `blob` and **no** `identity_blob`.
2. `identity_blob` was added to `host_escrow` by a **later** `ALTER TABLE` (`store.go:414`).
3. `demoteCurrentEscrowTx` (`store.go:2547`) selected six columns and not that one.
**The hashes matched.** And they match a **third, independent** source: `host_escrow.restic_pw_sha256`
on the hub reads `c60c8bc737a6…` for `demo-felhom-8363b5`, recorded at the ceremony and never involved
in this comparison. The box's file, the sealed bundle and the hub's record all agree.
**Highest register ID on arrival: R-201.** Grepped `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
`CONTEXT.md` before minting. One new item: **R-202**.
**The offsite repository password is recoverable from the sealed escrow with the customer's recovery
code.** That had never been demonstrated — the one live round-trip on record (2026-06-10) predates the
field by a month.
## 2. §8.1's two facts — both LOOKED AT on the live database, not deduced
## 2. Part 0 — the recovery code
Read from a `hub.db` + `-wal` + `-shm` snapshot taken after the deploy (`PRAGMA integrity_check`
`ok`; freshness proven positively: `MAX(host_reports.received_at)` = `2026-08-04 10:45:39` vs
`datetime('now')` = `10:58:59`).
**Answered by the operator before the session began:** the 2026-08-04 07:16/07:20 codes had not been
kept, so **the operator ran a fresh ceremony on both nodes and saved the codes** (supplied to this
session as `R_DEMO-FELHOM` in the DooPlex credentials file). No ceremony was run by CC.
**(a) Nothing is backfillable, and here is the looking:**
Measured hub-side, and worth recording because it exercised two of yesterday's behaviours live:
the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z` (demo-felhom), and
`restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated them as
same-password re-ceremonies: **no superseded row was created** and **no `offsite_repo_key_changed`
fired**. That is v0.93.0's Scenario E, live. It also means yesterday's `identity_blob` retention is
still unit-proven and structurally verified, **not yet exercised live** — no supersession has happened.
| id | host_id | `identity_blob IS NOT NULL` | superseded_at |
Subject box: **demo-felhom** (Tier 0, reachable with a baked key, current escrow). Not demo-hp
(break-glass only), not DooPlex, not ep0.
## 3. The §8.2 decision as built, and where the gate is
**Built as §8.2:** `GET /api/v1/hosts/{host_id}/escrow`, authenticated with the box's own per-host key,
self-scoped (global may read any — the same asymmetry the PUT has).
**The single gate point is `api.escrowSelfServiceRetrieval`** (`hub/internal/api/handler.go`), a named
constant. `false` re-imposes the recovery-mode requirement on this endpoint and changes nothing else —
the same unseal, the same extraction, the same proof, minus the self-service property. The trade is
written above the handler in `dr.go`'s style: what it gives out, why it is safe, and **what it changes
about who is required** — obtaining the blob used to need an operator to arm recovery mode and now
needs only that box's credential. They still cannot open it. The mitigation is the audit row.
## 4. Files, commits, deployed versions
| Repo | Commits | Version | Deployed |
|---|---|---|---|
| 3 | `demo-hp-bb76ea` | **0** | `2026-08-04 07:15:36` |
| 4 | `demo-felhom-8363b5` | **0** | `2026-08-04 07:20:08` |
| `felhom.eu` | `435f4a5` (code) · `dd08926` (manifest) · docs commit below | hub **v0.94.0** | ArgoCD Synced/Healthy, `felhom-hub:0.94.0` |
| `felhom-agent` | `6d79047` | **v0.125.0** | published sha `f7d8339b53d9…`, **verified by independent download**, installed on demo-felhom (`felhom-agent --version` → 0.125.0, `active`). **NOT vouched** |
| `felhom-controller` | `9640e51` | **v0.195.0** | `felhom-controller:0.195.0` on guest 9201, `Up (healthy)` |
Both retained rows are empty of it, and their source rows were overwritten at those same timestamps —
`host_escrow` now holds the *new* generation. **No path preserved them.** (The prompt asked me to check
whether some path had, because that would have been a better answer than expected. It had not.)
**Hub:** `internal/api/handler.go` (route + `handleHostEscrowGet` + `escrowSelfServiceRetrieval`),
`internal/notify/dispatcher.go` (`escrow_blob_served``operatorOnlyEvents`),
`internal/api/escrow_get_test.go` (new).
**Agent:** `internal/escrow/recover.go` (new — `OffsiteKeyRecoverer`, distinct errors),
`internal/hub/client.go` (`FetchIdentityEscrow`), `internal/localapi/escrow_recover.go` (new route),
`internal/localapi/server.go` (seam + registration), `cmd/felhom-agent/main.go` (production wiring +
the §8.6 selftest message fix), plus `recover_test.go` and `escrow_recover_wiring_test.go` (new).
**Controller:** `internal/backup/offbox_recovery_check.go` + `offbox_recovery_cli.go` (new),
`internal/agentapi/escrow.go` (`RecoverOffsiteRepoPassword`), `cmd/controller/main.go`
(`--recover-offsite-check`), plus `offbox_recovery_check_test.go` (new).
**(b) Who the fix protects, i.e. whose NEXT ceremony now retains a recoverable off-site key:**
**Green gate in each repo after each phase** (`go build && go vet && go test ./...`, rc=0), plus
`repo_gates.py` / `agent_gates.py` / `controller_gates.py` — all OK. **No test run was combined with a
commit.**
| host_id | identity blob present | bytes |
## 5. Tests and every red-proof
| Scenario | Result | Red-proof — mutation → outcome |
|---|---|---|
| `demo-felhom-8363b5` | yes | 572 |
| `demo-hp-bb76ea` | yes | 572 |
| **A** recovered key == on-disk key | PASS (unit, real `age`) + **PROVEN LIVE** | return `bundle.PBSToken` instead of `ResticRepoPassword`**FAIL**: *"the recovered key is not the sealed repository password (len 24 vs 64)"*. This is the mutation that mattered: every field of that bundle is a non-empty string that looks like a secret |
| **B** wrong R fails closed, nothing written | PASS + **PROVEN LIVE** | — (the property is age's KDF, asserted rather than guarded, so there is no guard to remove) |
| **C** a box fetches only its own blob | PASS | delete the `!isGlobal && authHostID != pathHostID` check → **FAIL**: host A was served host B's blob, body and all |
| **D** no blob → a clean "none" | PASS | — |
| **E** every retrieval leaves a record | PASS + **PROVEN LIVE** | `if false &&` on the audit block → **FAIL**: *"a sealed recovery blob was served and NOTHING recorded it"* |
| **F** R persists nowhere | PASS + **PROVEN LIVE** | write R to a temp file inside the unseal → **FAIL** (see the correction below) |
| **G** the operator DR path is untouched | PASS | — |
| **H** the seam is wired (AST) | PASS | `_ = escrowRecoverer` with the `Options` field removed → **FAIL**: *"localapi.Options … carries no EscrowRecovery field"* |
| §8.5 compare-never-install | PASS + **PROVEN LIVE** | add `InjectOffboxPassword` to the check → **FAIL**: *"the check INSTALLED the recovered password"* |
**2 of 2 hosts with a current escrow.** Both were one ceremony away from losing their repository
password again.
**TWO MUTATIONS DID NOT REMOVE EVERY GUARD ON THE FIRST ATTEMPT, AND BOTH ARE RECORDED RATHER THAN
QUIETLY REDONE** (§9.13 — two of the last four sessions found a red-proof that passed):
**Schema change confirmed applied live** — `PRAGMA table_info(host_escrow_superseded)` on the running
pod's database returns `identity_blob BLOB` as column 8. It runs at startup, so this is the check that
the startup actually did it rather than the code merely containing it.
## 3. Files modified, and the commits
**Commit `91cabdd`** — code + docs:
| File | Change |
|---|---|
| `hub/internal/store/store.go` | `identity_blob` on `host_escrow_superseded` (CREATE + additive `ALTER`); carried in `demoteCurrentEscrowTx`; `ListSupersededEscrow` reads it; `HostEscrow.IdentityBlob`; new `CountCurrentEscrowWithIdentity`; `SaveHostEscrow` returns the replaced hash; the table comment rewritten to describe what the code now does **and** to record that it once did not; three R-196 comment corrections |
| `hub/internal/api/handler.go` | `maybeEmitRepoKeyChanged` + `eventRepoKeyChanged`; wired into the superseded branch of the escrow PUT; one R-196 comment correction |
| `hub/internal/monitor/offsite_delivery.go` | `deliveryShape` (burned/regressed); two-shape message; `offsite_reports_since_consume` in the details; `recordHealRefusal`; refusal branches split so each names its reason |
| `hub/internal/notify/dispatcher.go` | `offsite_repo_key_changed`, `offsite_delivery_stuck`, `offsite_credential_restaged``operatorOnlyEvents` |
| `hub/internal/offsite/offsite.go` | one R-196 comment correction |
| new: `store/escrow_identity_retention_test.go`, `api/escrow_repo_key_changed_test.go`, `monitor/offsite_delivery_honesty_test.go` | Scenarios AG |
| 11 existing `_test.go` files | mechanical: `SaveHostEscrow` now returns three values |
| `hub/CHANGELOG.md`, `CONTEXT.md` (S-28), `REUSE.md`, `STATUS.md`, `00-capability-map.md`, `03-host-agent.md`, `OPEN-ITEMS.md`, `ROADMAP.md` | docs |
**Commit `40687b0`** — `manifests/hub.yaml` `0.92.0``0.93.0`.
## 4. Tests, and every red-proof outcome
Green gate after each phase: `go build ./... && go vet ./... && go test ./...` in `hub/`**rc=0**,
full suite, run separately from every commit (never combined).
| Group | Scenario | Result | Red-proof — mutation → outcome |
|---|---|---|---|
| A | re-escrow retains both keys | PASS | drop `identity_blob` from the shared copy routine (**= production ≤ v0.92.0**) → **FAIL**: *"the retained row carries NO identity blob — the off-site repository password was destroyed by the ceremony that was supposed to preserve it"* |
| B | host delete demotes both | PASS | same mutation → **FAIL**. **Plus the §8.2 mutation**: fix only the re-escrow caller (patch the retained row in `SaveHostEscrow` instead of the shared routine) → **A passes, B FAILS** — the "fixed one caller" mistake, demonstrated |
| C | legacy host with no identity blob supersedes cleanly | PASS | — (a nullable-column check; no guard to remove) |
| D | changed key raises one signal, host named, no hash values | PASS | remove the comparison from the escrow PUT → **FAIL**: *"the repository key demonstrably changed and NO signal was raised"*, while E still passes |
| E | unchanged key is silent | PASS | — |
| F | message states the measured state | PASS | restore the single hardcoded sentence → **both message tests FAIL** |
| G | heal refusal leaves a durable record | PASS | replace the regressed branch with a bare `return` (**= production ≤ v0.92.0**) → **FAIL**, and its cadence test with it |
| — | `IsOperatorOnly` for all three event types | PASS | — |
| — | `CountCurrentEscrowWithIdentity` census | PASS | — |
**A mutation that did not remove every guard, recorded because rule §9.12 requires it.** My first
Scenario-F mutation kept the `switch`'s `default` branch, so the *regressed* test passed for the wrong
reason and only the *burned* one failed. That is not a red-proof. The mutation that does fail both is
deleting the shape switch entirely and emitting the one v0.92.0 sentence — done, observed, recorded.
**Both callers proven separately** (§8.2): `TestSaveHostEscrow_RetainsIdentityBlob` (re-escrow) and
`TestDeleteHost_DemotesIdentityBlob` (host delete). The pre-existing
`TestSaveHostEscrow_RetainsSuperseded` stayed **green throughout the two months the bug existed**,
because it asserts the mechanism (a retained row exists, carrying the old K-blob). The new tests assert
the consequence (the retained row can still yield a repository password) — `CLAUDE.md`'s rule, applied
to the case that earned it.
1. **Scenario F.** The planted leak wrote R into the walked directory and **the test still passed**
because a later failing call OVERWROTE the leak file with the wrong code, and the test only scanned
for the first one. Instrumenting the mutation (printing where it wrote) is what found it. **The test
was wrong, not the mutation.** It now asserts **emptiness** of the TMPDIR tree as its primary check —
nothing under it is created by the test, so any survivor is a leak regardless of content — with the
content scan kept as defence in depth over all three secrets. Re-run: **FAIL**, *"the unseal left
1 file(s) behind under TMPDIR"*.
2. **Scenario H.** Commenting the wiring line out failed to **compile** (`declared and not used`), which
is not a red-proof. The mutation that compiles keeps the variable alive (`_ = escrowRecoverer`) and
drops the struct field. Re-run: **FAIL** as quoted above.
**Every `-run` filter was verified to have matched** (`-v`, `=== RUN` lines counted) before any result
was read as a proof.
## 5. The exact operator message text, quoted for review
## 6. The live validation trail, in order
**R-197 — `offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession):
1. **Wrong code first, deliberately** (13:44:44) — hub: `escrow blob SERVED to host demo-felhom-8363b5
(572 opaque bytes, self_scope=true) — recovery path in use`; agent: `offsite key recovery FAILED …
err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt
blob): exit status 1"`; CLI: `[FAIL] … nothing was written.`, exit 1.
**This is Scenario B live AND it proves links 6 and 7 ran independently of the success** — the blob
was fetched and the unseal was attempted and refused.
2. **The real code** (13:49:47) — §1's output, exit 0.
3. **Nothing was written**`offbox/repo_password` mtime on the live box is still
`2026-08-03 07:18:02`, unchanged by a check that ran at `11:49` UTC. The whole `offbox/` directory is
byte-for-byte the pre-check listing.
> Offsite repository key CHANGED for host `<host_id>`: the new escrow seals a different repository
> password than the one it replaced. The previous off-site history is no longer opened by this box's
> current key. The superseding blob was retained (`<N>` held), so that history stays recoverable with
> the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an
> orphan card, and expect the next backup to start a fresh history.
## 7. The retrieval record as it appears at the hub
Details JSON: `{"host_id": "...", "retained_count": N, "repo_key": "changed"}`**no hash values**.
```
customer_id severity created_at details_json
demo-felhom warning 2026-08-04 11:44:44 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
demo-felhom warning 2026-08-04 11:49:47 {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
```
**R-192 — `offsite_delivery_stuck`, burned shape** (unchanged in intent, corrected in wording):
And the routing, which is the half that shows the register works:
> Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed `<age>` ago; of
> the first `<N>` report(s) after that consume, NONE carried an offbox target, and the latest report
> carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.
```
customer_id channel status created_at
demo-felhom operator sent 2026-08-04 11:44:44
demo-felhom customer skipped 2026-08-04 11:44:44 ← operator_only
demo-felhom operator suppressed 2026-08-04 11:49:47 ← 1h operator cooldown, itself recorded (R-182)
demo-felhom customer skipped 2026-08-04 11:49:47
```
**R-192 — regressed shape** (the demo-hp case, which previously got the burned text):
**Both retrievals raised an event; the operator was mailed for the first; the second's mail was
cooldown-suppressed and that suppression is written down.** The audit is per-retrieval and complete
even where the mail is not.
> Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed `<age>` ago; of
> the first `<N>` report(s) after that consume, `<M>` DID carry an offbox target — and the latest report
> carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT
> the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic
> restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports
> after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).
## 8. Evidence that R persisted nowhere — the search, not a claim
**R-192 — the refusal record** (`notification_log`, channel `operator`, status `refused`), message
*"Automatic offsite credential restage was NOT performed."* with the reason in `error_message`, e.g.:
Searched on the subject host with the pattern read from a file (never in argv), across the agent
journal, the controller's container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`:
> regressed-apply shape: `<M>` of the first `<N>` report(s) after the consume DID carry an offbox
> target, so a burned credential is ruled out — a restage would treat a symptom whose cause is
> elsewhere. Operator's call (R-193).
```
agent journal lines containing R: 0
controller log lines containing R: 0
host files containing R: 0
leftover felhom-idesc-* staging dirs: 0
staged escrow secret file: 0 entries
```
## 6. The corrected comments — FIVE, not three, with their locations as found
**With a positive control, because an absent match is not a measurement unless the instrument is shown
to work:** a copy of R was planted under `/tmp`, the same sweep found **1**, the copy was shredded, and
the sweep returned **0**. The instrument detects R when R is there.
The spec expected three and named two; a census of every `restic` mention in the hub found **five**.
R reached the box only as: the credentials-file value on DooPlex → a pipe → the container's stdin →
the request body → the agent's memory. It was never an argument, never a file on either machine, and
appears in no log line at any level.
| # | Location | What it claimed |
|---|---|---|
| 1 | `hub/internal/offsite/offsite.go:198` | *"the restic repo password just changed"* as the justification for `MarkEscrowStale` |
| 2 | `hub/internal/api/handler.go:1067` | *"The re-issuer resets the restic repo password, which makes the OLD escrow blob stale"* |
| 3 | `hub/internal/store/store.go:423` (the `stale_at` ALTER) | *"stale_at is set when the offsite repo password is re-issued"* |
| 4 | `hub/internal/store/store.go` `MarkEscrowStale` doc comment | *"called when the offsite repo password is re-issued"* |
| 5 | `hub/internal/store/store.go` `EscrowStatus.Stale` field comment | *"true when the offsite password was re-issued"* |
## 9. Register
All five now state what the code does, name the correction with its date and item, and cite the recon.
The staleness mark is documented as **precautionary** the box's re-apply may mint a fresh repository
password — with R-197's measured signal named as the evidential one. **The behaviour is unchanged and
R-196 stays OPEN**, because the false-staleness alarm is a behaviour change that must not ride a
comment-correction release. It also matters *more* now than when filed: under R-198 an unnecessary
ceremony is no longer harmless bookkeeping, it supersedes a blob.
- **R-199 → SHIPPED + PROVEN-LIVE.** Links 6, 7 and 8 assembled and walked.
- **R-200 → plumbing shipped; the FORM is not built** and the row stays open for it and for link 9.
- **R-201 → OPEN, and the distinction is deliberate:** the KEY is proven recoverable; a recovered
password has never been **installed**, no repository has been **reopened** under one, and **no file
has ever been restored**. The drill's pass condition is unchanged — a byte-identical sentinel file,
not "the store opened".
- **R-202 → OPEN and untouched.** The orphan card still promises recoverability unconditionally.
- **The orphaned-ciphertext deletion (~1.2 GB) is STILL OWED** — operator ruling 2 of 2026-08-04,
deliberately not ridden along with a code release.
- Capability map, `03-host-agent.md` §8a, `CONTEXT.md` **S-29**, `REUSE.md` and `ROADMAP.md` updated;
`STATUS.md` rewritten for the operator.
## 7. Part 5 — the gate was hit; the card is untouched; R-202 filed
## 10. The capability-map row, and what it deliberately does not claim
**The gate:** ship it iff the hub can tell a box what it needs with **one** additional boolean on the
escrow ACK it already sends.
Added: *"The offsite repository password can be RECOVERED from the sealed escrow with the customer's
recovery code"* — **PROVEN-LIVE (2026-08-04)**, with the evidence above.
**The hub can compute such a boolean cheaply** — *"≥1 retained blob for this host carries an identity
blob"*, one correlated predicate in `GetEscrowStatusForCustomer` — and the controller even has the right
seam already (`SetEscrowStale`/`StaleBlob` is exactly this shape). **But that boolean does not answer
the card's question.** The card renders on `RepoState == "orphaned"`, and its promise is about *the key
THIS orphaned repository was written under*. A box does not know which escrow generation the orphaned
remote belongs to. A box with a pre-v0.93.0 orphan and a post-v0.93.0 supersession would read the
boolean TRUE and the promise would still be false — a **conditional** falsehood that looks verified,
which on a customer-facing card is worse than today's hedged one. Making it truthful needs the orphan's
generation, which is the same knowledge R-199/R-201's unassembled chain needs.
**What the row explicitly refuses to claim**, stated in the row itself because the previous over-claim
on this subject was struck out four hours earlier: it covers the **key**, not the **data**. No
recovered password has been installed, no repository reopened, no file restored. And the proof used a
box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is
exactly what the drill covers and it has not run.
**So: stopped, filed R-202, left the card alone**, per §8.6. **Stated plainly rather than buried: the
false sentence is still live**, in Hungarian, on both demo boxes. The cheapest honest interim — drop the
recoverability clause and say only that the old history is set aside and not deleted, which is true
unconditionally — is recorded in R-202 and **not taken here**, because it is a customer-copy change and
the gate said leave it alone.
## 11. CI
## 8. Register
Run numbers, task ids and conclusions for all commits are quoted in the session summary.
**`--no-verify` was NOT used** — every push ran its repo's pre-push gate and passed.
- **R-198 → SHIPPED** (hub v0.93.0), with the §8.1 facts and both red-proof outcomes recorded.
- **R-197 → SHIPPED** (hub v0.93.0).
- **R-192 → HALF SHIPPED**: both honesty halves done; **the guard's 500-oldest-reports scoping stays
OPEN**, deliberately, because its correct shape depends on the recovery chain that is not yet
assembled. The window is named inside the alert text so the limitation travels with the number.
- **R-196 → comments corrected; the BEHAVIOUR stays OPEN.**
- **R-193 → still open**, updated with the three operator rulings.
- **R-202 → NEW** (Part 5's gate).
- **R-199, R-200 and R-201 remain OPEN and untouched** — the hub's blob-serving endpoints still have no
client, the injection seam still has no form, and nothing in the recovery path has ever been
exercised. v0.93.0 makes the key survive; it does not assemble the chain that hands it back.
- **The three operator rulings of 2026-08-04 are recorded** in a dedicated block at the top of
`OPEN-ITEMS.md` (a ruling that lives only in a conversation binds nobody — the R-96 rule):
run the drill after R-198 (**R-198 has shipped; the drill is next**); **delete the orphaned
ciphertext — STILL OWED**, deliberately not done here because a destructive act on a protected
endpoint does not ride a schema-change release; and **accept the risk on R-193(c)**, which is what
makes this retention load-bearing rather than tidy.
- Capability map, `03-host-agent.md` §8a (a new *Custody across generations* section — the contract
changed, so S-1 applies), `CONTEXT.md` S-28, `REUSE.md` and `ROADMAP.md` all updated in the same
commit.
## 12. Teardown
## 9. Deploy & validation
**Nothing was provisioned.** The temporary pattern file used by the R sweep and its positive control
were shredded on the host by the same command that created them (verified: 0 files remain). No
diagnostic artifact persists on either machine. The agent's previous binary is retained as
`/usr/local/bin/felhom-agent.bak-0.124.1`, per the documented deploy.
- Clean-tree gate held per repo before the build; the image was built from a pushed tree.
- `./build.sh 0.93.0 --push``felhom-hub:0.93.0`, 25 MB. Manifest bumped in git, committed, pushed.
- ArgoCD hard-refresh + **deliberate sync** (never `kubectl set image`). **Synced**; `deploy/hub`
rolled out; live image `gitea.dooplex.hu/admin/felhom-hub:0.93.0`; startup log clean (all checkers
initialized, `Listening on :8080`, no errors).
- **Live validation was structural, not manufactured** (§9.3): the schema column read from the running
pod's database, and the two censuses in §2. **No ceremony was run on any box** — today, a ceremony is
the act that destroys the key, and it is also the one thing that would have "proved" the retention by
consuming a real customer's generation.
- Disk headroom checked before the build (`/mnt/5_hdd` 24%, `/` 86%).
- **Nothing provisioned; nothing to tear down.**
- Gates: `python3 scripts/repo_gates.py --fast` → all OK, after every docs change.
## 13. Observations — noticed, NOT acted on
## 10. CI
- Code commit **`91cabdd`** and manifest commit **`40687b0`** — run numbers, task ids and conclusions
quoted in the session summary; **`--no-verify` was NOT used** (the pre-push hook ran `repo_gates.py
--fast` and passed on both pushes).
## 11. Observations — noticed, not acted on
1. **`ListSupersededEscrow` still has no production caller.** v0.93.0 makes it *return* the identity
blob, so a retained key is reachable from Go for the first time — but nothing reads it. That is
R-199's territory and was deliberately not widened here.
2. **`handleReEnroll` returns a freshly minted API key in its response body** on an endpoint nothing
currently calls. Worth a look *before* anything starts calling it (R-199).
3. **`CountReportsOffsiteSince`'s `LIMIT 500` is the only instrument in the delivery checker**, and its
window is now named in operator-facing text. If R-192's remaining half is ever fixed, that sentence
must change with it — the R-100 corollary (an alarm whose text stopped matching what its verdict
counts) applies directly.
4. **`offsite_delivery_stuck` had been reaching the customer channel by construction** for its whole
life; the only reason no customer row exists is that the demo customers have no configured
recipient. An absent notification row is not evidence a leg is blocked.
</content>
1. **The `escrow_blob_served` mail is subject to the 1-hour operator cooldown** — visible above, where
the second retrieval's mail was suppressed. The EVENT rows are per-retrieval so the audit is
complete, but if retrieval ever becomes routine, a burst would produce one mail. Worth revisiting
**when** the customer-facing flow lands, not before.
2. **`ListSupersededEscrow` still has no production caller.** Recovery reads the CURRENT blob; a
customer needing a PREVIOUS generation's key (which is what R-198's retention exists for) has no
path at all. That is the next gap in this area after the drill.
3. **The controller binary is at `/usr/local/bin/felhom-controller`, not `/app/…`** — the first
diagnostic invocation failed on a guessed path. Worth a line in the runbook when one is written.
4. **`age` is at the hardcoded `/usr/bin/age` on both demo hosts** (1.2.1 on demo-felhom) — the recon
listed this as unestablished; it is now established for demo-felhom. `ageBinary` is a package
variable with no config override, so a host that installs `age` elsewhere would fail the unseal at
the last step.
5. **Yesterday's `identity_blob` retention has still not been exercised live** — the operator's fresh
ceremonies sealed the *same* password, so no supersession occurred. It remains unit-proven with a
verified schema; the first real supersession will be its live proof.
+1
View File
@@ -78,6 +78,7 @@
| `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. |
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts**`blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
+18 -7
View File
@@ -36,11 +36,16 @@ Proven end to end on real hardware.
the old backups stay recoverable. Both keys are now kept. **What this cannot undo:** anything
superseded before today is gone for good, which includes both demo machines' pre-4-August keys — so
those 51 orphaned backups were beyond reach even if the recovery codes had been kept. *(R-198)*
- **Nothing in the recovery path has ever been performed, and that has not changed today.** Making a
recovery code is proven, live, with a real customer. **Using one is not.** No sealed package has ever
been handed back to a machine, no recovered key has ever been put back, no old backup store has ever
been reopened. Today's work makes the key **survive**; it does not build the path that hands it back.
That is what the proof exercise is for. *(R-199, R-200, R-201)*
- *(largely fixed 4 Aug)* ~~Nothing in the recovery path has ever been performed.~~ **The key now
demonstrably comes back — measured on a real machine this evening.** demo-felhom fetched its own
sealed package from the hub with its own credential, opened it with the recovery code you saved, and
the backup key that came out was **identical, character for character, to the one the machine is
using** — and to the fingerprint the hub had recorded separately. Three independent sources agreeing.
A deliberately wrong code, tried five minutes earlier, was refused outright and wrote nothing.
**What is still NOT done, and it is the half that matters to a customer:** nothing yet *puts the
recovered key back*, reopens the old backup store with it, or **restores a single file**. Today
proves the key survives and returns; it does not prove the backups do. *(R-199 closed; R-200 half;
R-201 open)*
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
says the old backups may later be restorable with the matching recovery code. From today that is true
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
@@ -90,7 +95,9 @@ Proven end to end on real hardware.
## What we're working on
- **Now:** the proof exercise — wipe a demo machine and recover it with a saved recovery code. It is
designed and waiting on your go-ahead. Both honesty fixes shipped today: a changed backup key now
designed, and it is now a much better bet than it was this morning: the first half of the path was
walked live today, so if the drill fails we will know *which* step failed instead of just "recovery
did not work". Waiting on your go-ahead. Both honesty fixes shipped today: a changed backup key now
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
what it actually saw.
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
@@ -142,7 +149,11 @@ Proven end to end on real hardware.
## Changed since last update
- **2026-08-04 (latest)****Fixed the worst of it.** The hub now keeps the off-site backup key when a
- **2026-08-04 (evening)****Proved the backup key comes back.** A demo machine fetched its own
sealed package, opened it with the saved recovery code, and produced a key identical to the one it
uses. A wrong code was refused and wrote nothing. The recovery code left no trace anywhere on the
machine — searched, with a planted copy first to prove the search worked. *(R-199, R-200)*
- **2026-08-04 (earlier)****Fixed the worst of it.** The hub now keeps the off-site backup key when a
machine re-seals, instead of only the whole-machine one — four lines of database change that stop the
recovery ceremony from destroying the thing it exists to protect. A changed backup key now raises an
alarm the same day. The daily email that described the opposite of what it measured now states what it
File diff suppressed because one or more lines are too long
@@ -409,6 +409,24 @@ retained row carries both.
predates the `ResticRepoPassword` field. Tracked as R-199/R-200/R-201; evidence
`documentation/audits/RECON-offsite-dr-chain-2026-08-04.md`.
#### Serving a blob back to its own box (hub v0.94.0 + agent v0.125.0, R-199)
Retention without retrieval is custody, not recovery. Since 2026-08-04 a host reads its own sealed
identity blob from `GET /api/v1/hosts/{host_id}/escrow` — self-scoped by the per-host key, the mirror
of the PUT that stored it — and opens it locally with the customer's recovery code via the agent's
`POST /escrow/recover-offsite-password`, which returns **only** the offsite repository password.
**This is a deliberate widening of who can obtain the ciphertext**, recorded in `CONTEXT.md` S-29 and
in the handler itself: it previously took an operator arming recovery mode, and now takes only that
box's own credential. The blob remains unopenable without R, and every retrieval raises an
operator-visible `escrow_blob_served` event. The operator-driven re-enroll/restore-directive path is
unchanged and keeps its recovery-mode gate.
**Proven end to end on demo-felhom, 2026-08-04:** the recovered repository password's sha256 was
byte-identical to the one on the box, and to the hash the hub had independently recorded. **The chain
stops there** — nothing installs a recovered password, reopens a repository with it, or restores a
file.
**The honesty property above is unchanged and now has a second edge:** losing `R` and the box makes
the offsite backups unrecoverable by anyone — *and* an older generation's data is recoverable only
with **that generation's** recovery code, which the customer must still hold. A current recovery code
File diff suppressed because one or more lines are too long
+2
View File
@@ -61,6 +61,8 @@
| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **comments corrected (hub v0.93.0); the BEHAVIOUR stays open** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping |
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 68 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L11811219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L21462156, reinstall preserves existing), verified at L23322337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor``06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0``06:57:19` `image file written, restarting bootstrap``06:57:21` container StartedAt + UpdateState `completed_at``06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |