RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Read-only recon of the escrow -> recovery chain, from a dead node to an open
repository. No production code, no build, no version bump.
Headline: the hub's superseded-escrow retention does NOT retain the offsite
repository password. host_escrow_superseded has no identity_blob column and
demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a
supersession is the PBS datastore key, not the restic repo password. The next
escrow ceremony -- which the system tells a rebuilt box's customer to run --
destroys the last copy. Both demo boxes crossed that line on 2026-08-04.
Also established:
- the hub's blob-serving endpoints (re-enroll / restore-directive) have zero
callers anywhere: agent, hub UI, scripts, runbooks (R-199)
- POST /backup/offbox/inject-password is routed and handled but no template
contains the form (R-200)
- nothing in the recovery path has ever been exercised; the one live
round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201)
- a fail-closed mint refusal IS implementable: the report ACK already carries
escrow{identity_blob_present, restic_pw_sha256} and the controller discards
it whenever no offbox target exists
Corrections: yesterday's spike annotated (candidate (b) overturned in part --
unattended recovery is impossible, customer-present is not); capability-map
retention claim struck through and replaced with what the code does.
Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.
This commit is contained in:
@@ -0,0 +1,531 @@
|
||||
# RECON — can a customer actually get their backups back?
|
||||
|
||||
**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
|
||||
**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.
|
||||
|
||||
> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
|
||||
> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
|
||||
> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
|
||||
> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
|
||||
> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
|
||||
> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
|
||||
> because the remedy closes it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed baselines (re-read on arrival)
|
||||
|
||||
| Repo | `main` @ commit | tree | Version |
|
||||
|------|-----------------|------|---------|
|
||||
| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
|
||||
| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
|
||||
| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |
|
||||
|
||||
All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.
|
||||
|
||||
**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
|
||||
R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
|
||||
`CONTEXT.md`. New items in this session therefore start at **R-198**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Instruments — what each can and cannot answer
|
||||
|
||||
| # | Instrument | Exact read | Known limit |
|
||||
|---|---|---|---|
|
||||
| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
|
||||
| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) |
|
||||
| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
|
||||
| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
|
||||
| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |
|
||||
|
||||
**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
|
||||
write of any kind, nothing touched on the storage endpoint.
|
||||
|
||||
---
|
||||
|
||||
## 3. Q1 — the chain, link by link
|
||||
|
||||
From a dead node to an open repository. **Status vocabulary as specified.**
|
||||
|
||||
| # | Link | Component / function | Who invokes it | Status |
|
||||
|---|---|---|---|---|
|
||||
| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
|
||||
| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
|
||||
| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
|
||||
| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
|
||||
| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
|
||||
| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
|
||||
| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
|
||||
| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
|
||||
| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
|
||||
| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
|
||||
| 11 | a backup or restore proves it | — | — | **never** |
|
||||
|
||||
### The chain in one sentence
|
||||
|
||||
Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually
|
||||
manual, and link 6 has no client at all. Link 11 has never happened.
|
||||
|
||||
---
|
||||
|
||||
## 4. Q2 — where the chain stops today
|
||||
|
||||
**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**
|
||||
|
||||
Two distinct stopping points, and the order matters:
|
||||
|
||||
**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
|
||||
still seals the *old* password. Recovery is possible in principle, and a person must:
|
||||
|
||||
1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
|
||||
or skip it entirely, because nothing consumes the endpoint it gates;
|
||||
2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
|
||||
3. copy the blob to a machine with the agent binary and `age`;
|
||||
4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
|
||||
5. open `bundle.json` by hand and read `restic_repo_password` out of it;
|
||||
6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
|
||||
session + CSRF) — there is no form;
|
||||
7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
|
||||
8. trigger a run and hope `restic cat config` succeeds.
|
||||
|
||||
Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
|
||||
to be present with R, and one (step 6) with no user interface at all.
|
||||
|
||||
**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
|
||||
rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
|
||||
`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
|
||||
has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
|
||||
**no procedure exists**, with or without R, at any level of effort. The password is gone.
|
||||
|
||||
Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
|
||||
`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
|
||||
snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.
|
||||
|
||||
---
|
||||
|
||||
## 5. Q3 — can a rebuilt controller know it should recover?
|
||||
|
||||
### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**
|
||||
|
||||
**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
|
||||
`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
|
||||
are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
|
||||
about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
|
||||
reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
|
||||
exactly the right line, but cannot distinguish "never configured" from "rebuilt".
|
||||
|
||||
**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
|
||||
(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
|
||||
(`internal/report/escrow_confirm.go:19-23`):
|
||||
|
||||
```go
|
||||
type EscrowStatus struct {
|
||||
IdentityBlobPresent bool `json:"identity_blob_present"`
|
||||
ResticPwSHA256 string `json:"restic_pw_sha256"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
```
|
||||
|
||||
That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
|
||||
every report cycle, on a channel that already exists, with no new API and no new secret.
|
||||
|
||||
**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
|
||||
immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
|
||||
non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
|
||||
there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
|
||||
exactly **one** reference to the ACK field in the whole controller
|
||||
(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).
|
||||
|
||||
**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:
|
||||
|
||||
- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
|
||||
caches the hub's claim-code state into `settings.json` idempotently by generation,
|
||||
`internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
|
||||
which is the correct shape here too);
|
||||
- at the mint branch, refuse when the password file is absent **and** the cached status says
|
||||
`identity_blob_present && restic_pw_sha256 != ""`.
|
||||
|
||||
**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
|
||||
ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
|
||||
`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
|
||||
design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
|
||||
status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
|
||||
nothing on a partial failure, `offsiteapply.go:1-6`).
|
||||
|
||||
**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
|
||||
9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):
|
||||
|
||||
| file | mtime |
|
||||
|---|---|
|
||||
| the data directory's parent | `2026-08-03 07:17:25` |
|
||||
| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
|
||||
| `offbox/` | `2026-08-03 07:18:02` |
|
||||
| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |
|
||||
|
||||
The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
|
||||
Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
|
||||
rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4
|
||||
reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
|
||||
submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.
|
||||
|
||||
> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
|
||||
> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
|
||||
> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).
|
||||
|
||||
---
|
||||
|
||||
## 6. Q4 — what the stale-escrow detector actually covers
|
||||
|
||||
### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**
|
||||
|
||||
**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
|
||||
that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
|
||||
event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
|
||||
and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
|
||||
detectors; only one of them was ever claimed to be misplaced.
|
||||
|
||||
**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
|
||||
with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
|
||||
true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
|
||||
is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:
|
||||
|
||||
```
|
||||
[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
|
||||
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
|
||||
```
|
||||
|
||||
**So: confirmed, it fires.** The detector is not absent from the mint path.
|
||||
|
||||
**Three limits, each of which matters more than the fact that it fires.**
|
||||
|
||||
1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
|
||||
flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
|
||||
`:130-165`); the pending branch sets no display flag. The operator learns nothing.
|
||||
2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
|
||||
`SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it
|
||||
→ **the old repo password ceases to exist**. The detector correctly identifies the situation and
|
||||
then tells the customer to close the door.
|
||||
3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
|
||||
deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
|
||||
identical warning.
|
||||
|
||||
So: **less needs building than the prompt feared for the *detection*, and more needs building for the
|
||||
*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
|
||||
signal or a recovery.
|
||||
|
||||
---
|
||||
|
||||
## 7. Q5 — is the superseded-escrow retention real, and what does it retain?
|
||||
|
||||
### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**
|
||||
|
||||
**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):
|
||||
|
||||
```sql
|
||||
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
|
||||
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
|
||||
FROM host_escrow WHERE host_id = ?
|
||||
```
|
||||
|
||||
`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):
|
||||
|
||||
| table | columns |
|
||||
|---|---|
|
||||
| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
|
||||
| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |
|
||||
|
||||
**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
|
||||
**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
|
||||
(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
|
||||
`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
|
||||
`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
|
||||
else.
|
||||
|
||||
**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**
|
||||
|
||||
- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
|
||||
- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
|
||||
- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.
|
||||
|
||||
**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
|
||||
production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
|
||||
for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
|
||||
comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
|
||||
this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.
|
||||
|
||||
**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:
|
||||
|
||||
> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
|
||||
> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
|
||||
> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*
|
||||
|
||||
The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
|
||||
own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):
|
||||
|
||||
> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
|
||||
> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
|
||||
> kóddal később visszaállítható lehet."*
|
||||
|
||||
**For the offsite restic repository — the incident it names — that is false.** The retained blob does
|
||||
not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
|
||||
asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
|
||||
→ **R-198**, and the capability-map correction in §9.
|
||||
|
||||
---
|
||||
|
||||
## 8. Q6 — has any part of this ever been exercised?
|
||||
|
||||
**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
|
||||
identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.
|
||||
|
||||
### Proven
|
||||
|
||||
| what | evidence | scope of the proof |
|
||||
|---|---|---|
|
||||
| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
|
||||
| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
|
||||
| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
|
||||
| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |
|
||||
|
||||
### Never exercised — named in those words
|
||||
|
||||
- **A blob has never been served to a box.** Link 6 has no client (§3).
|
||||
- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
|
||||
caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
|
||||
explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
|
||||
run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
|
||||
- **A recovered repo password has never been injected into a controller.**
|
||||
- **An existing offsite repository has never been reopened with a recovered password.**
|
||||
- **No restore of any kind has ever been performed from a recovered secret.**
|
||||
|
||||
`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
|
||||
*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
|
||||
operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
|
||||
exercised destructively**"*. **That document is right and this session confirms it.** What it did not
|
||||
catch — because it was written from `host_escrow`, not from the supersession path — is Q5.
|
||||
|
||||
**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
|
||||
**retention** claim quoted in §7. See §9.
|
||||
|
||||
---
|
||||
|
||||
## 9. Q7 — the operator's ruled shape, assessed row by row
|
||||
|
||||
> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
|
||||
> blob, then the controller should yell that recovery is available, and provide a form for the customer
|
||||
> to enter the recovery key. After unlocking the blob, the controller should show what will be
|
||||
> recovered before proceeding.*
|
||||
|
||||
| The ruling requires | Exists? | What it needs, and what it costs |
|
||||
|---|---|---|
|
||||
| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
|
||||
| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
|
||||
| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
|
||||
| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
|
||||
| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
|
||||
| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
|
||||
| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |
|
||||
|
||||
### Things the ruling implies that do not exist, priced
|
||||
|
||||
| missing piece | cost | note |
|
||||
|---|---|---|
|
||||
| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
|
||||
| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
|
||||
| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
|
||||
| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
|
||||
| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
|
||||
| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
|
||||
| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |
|
||||
|
||||
### The security question the ruling raises — stated, not answered
|
||||
|
||||
**What gates the surface today.** The controller dashboard is behind `RequireAuth`
|
||||
(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
|
||||
`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
|
||||
logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
|
||||
the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
|
||||
password**, on a controller reachable at whatever address that box's dashboard is reachable at.
|
||||
|
||||
**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
|
||||
snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
|
||||
the repository size.** That is a map of the household's digital life, and it is precisely the metadata
|
||||
the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.
|
||||
|
||||
**Three specifics the operator should weigh, without this session choosing:**
|
||||
|
||||
1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
|
||||
(~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
|
||||
online guessing is not a route. But **anyone with the dashboard password can already restore live
|
||||
offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is
|
||||
confined to the *orphaned/previous* repository, which today nobody can read at all.
|
||||
2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
|
||||
dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
|
||||
fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
|
||||
and should rate-limit like the login path does.
|
||||
3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
|
||||
root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
|
||||
set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
|
||||
the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
|
||||
smaller and time-bounded form**, and it should be decided alongside it rather than separately.
|
||||
|
||||
---
|
||||
|
||||
## 10. Q8 — the drill design
|
||||
|
||||
**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
|
||||
no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
|
||||
the full circle. **This section is the design; the drill is not run in this session.**
|
||||
|
||||
### Target and preconditions
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
|
||||
| **Guest** | the customer guest on that host (rebuild target) |
|
||||
| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
|
||||
| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
|
||||
| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
|
||||
| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |
|
||||
|
||||
### The cycle
|
||||
|
||||
| # | Step | Who | Observable that must be recorded |
|
||||
|---|---|---|---|
|
||||
| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
|
||||
| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
|
||||
| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
|
||||
| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
|
||||
| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
|
||||
| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
|
||||
| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
|
||||
| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
|
||||
| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
|
||||
| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
|
||||
| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |
|
||||
|
||||
### The single observation that makes this a FAILURE rather than a pass
|
||||
|
||||
> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**
|
||||
|
||||
A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
|
||||
not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
|
||||
pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
|
||||
customer's data came back. Only the sha256 comparison does.
|
||||
|
||||
### Secondary failures worth recording separately (each is a finding, not a pass/fail)
|
||||
|
||||
- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
|
||||
closes without a human act (would be new and serious).
|
||||
- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
|
||||
- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
|
||||
the transport, not the key.
|
||||
|
||||
### What could make it fail for uninteresting reasons
|
||||
|
||||
The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
|
||||
(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
|
||||
step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
|
||||
`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
|
||||
its GO line rather than retrying).
|
||||
|
||||
### Rollback, and whether the loss is acceptable
|
||||
|
||||
**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
|
||||
whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
|
||||
explicitly disposable, and it is the box the target-selection runbook names for drills. **The
|
||||
rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
|
||||
is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
|
||||
**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
|
||||
do not rely on the nightly.
|
||||
|
||||
**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
|
||||
box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.
|
||||
|
||||
---
|
||||
|
||||
## 11. Every query used
|
||||
|
||||
| # | Source | Query / read | Value read |
|
||||
|---|---|---|---|
|
||||
| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
|
||||
| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
|
||||
| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
|
||||
| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
|
||||
| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
|
||||
| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
|
||||
| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
|
||||
| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
|
||||
| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
|
||||
| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
|
||||
| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
|
||||
| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
|
||||
| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
|
||||
| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
|
||||
| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
|
||||
| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
|
||||
| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
|
||||
| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |
|
||||
|
||||
---
|
||||
|
||||
## 12. What could not be established
|
||||
|
||||
Named, not guessed.
|
||||
|
||||
1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
|
||||
runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
|
||||
Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
|
||||
2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
|
||||
(I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
|
||||
credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
|
||||
identical on both boxes, which is suggestive, not decisive.
|
||||
3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
|
||||
the endpoint needs the SFTP credential, which was not touched.
|
||||
4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
|
||||
holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
|
||||
have carried the identity blob anyway).
|
||||
5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
|
||||
agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
|
||||
demo host.
|
||||
|
||||
---
|
||||
|
||||
## 13. Observations — noticed, not acted on
|
||||
|
||||
1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
|
||||
pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
|
||||
dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
|
||||
week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
|
||||
**not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
|
||||
2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
|
||||
means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
|
||||
3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
|
||||
a trap for anyone reusing it as a read-only preview primitive (§9).
|
||||
4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
|
||||
the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
|
||||
nothing currently calls — worth a second look before anything starts calling it.
|
||||
|
||||
---
|
||||
|
||||
## 14. The operator's decisions — stated, and NOT answered here
|
||||
|
||||
1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
|
||||
*"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
|
||||
history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
|
||||
that walks the current chain will pass or fail on a chain that is missing a link everyone believed
|
||||
was there.
|
||||
2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
|
||||
**unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
|
||||
passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
|
||||
indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
|
||||
act on it and states it as the operator's.
|
||||
3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
|
||||
that every rebuild starts a new repository. **This session adds one input to that decision:** the
|
||||
"accept it, provided the system says so loudly" branch currently rests on a retention feature that
|
||||
does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
|
||||
stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
|
||||
to be re-described honestly to the customer.
|
||||
</content>
|
||||
</invoke>
|
||||
@@ -10,6 +10,39 @@ what it measured) · **Class:** spike — **no production code shipped for R-193
|
||||
|
||||
---
|
||||
|
||||
> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8
|
||||
>
|
||||
> A follow-up recon traced the escrow→recovery chain link by link
|
||||
> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull
|
||||
> in opposite directions.**
|
||||
>
|
||||
> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in
|
||||
> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is
|
||||
> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back —
|
||||
> customer-present, not unattended — is a real design, and the operator has since ruled on its shape.
|
||||
> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a
|
||||
> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row.
|
||||
>
|
||||
> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked
|
||||
> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx`
|
||||
> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained
|
||||
> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The
|
||||
> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony
|
||||
> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is
|
||||
> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and
|
||||
> 07:20:08. → **R-198**.
|
||||
>
|
||||
> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but
|
||||
> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as
|
||||
> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is
|
||||
> not complete without R-198.
|
||||
>
|
||||
> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d)
|
||||
> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is
|
||||
> still the decisive one.
|
||||
|
||||
---
|
||||
|
||||
## 0. Why this exists
|
||||
|
||||
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
|
||||
@@ -412,6 +445,12 @@ re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the exi
|
||||
|
||||
### (b) The credential becomes recoverable from escrow at re-bootstrap
|
||||
|
||||
> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.**
|
||||
> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a
|
||||
> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape.
|
||||
> The recon also found that the chain this option would use is clientless at three links and lossy at
|
||||
> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4).
|
||||
|
||||
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
|
||||
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
|
||||
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
|
||||
|
||||
Reference in New Issue
Block a user