hub v0.94.0: a box can fetch its own sealed recovery package (R-199 link 6)
gates / gates (push) Successful in 7s

Link 6 of the recovery chain had no client. The hub has served the identity blob since
slice 10D from handleReEnroll / handleGetRestoreDirective, gated on operator-armed recovery
mode and the global key -- and nothing in the agent, the hub UI, any script or any runbook
ever called either. The only documented retrieval was sqlite3 writefile() by hand on a
kubectl cp-ed database.

GET /api/v1/hosts/{host_id}/escrow is the box-authenticated mirror of the PUT that put the
blob there. Self-scoped (a per-host key reads only its own; global may read any). A host with
no bundle gets 200 {present:false} -- a 404 is indistinguishable from an unknown host and a
bare empty 200 from a zero-length blob.

THE TRADE IS RECORDED IN THE HANDLER, not inferred: obtaining the blob used to require the
operator to arm recovery mode; now whoever controls a rebuilt box can obtain it with that
box's own credential. They still cannot open it -- the hub has never held R and a wrong code
fails closed at age's scrypt KDF. The mitigation is that every retrieval raises
escrow_blob_served (warning, operator-only), recorded before the bytes leave.

escrowSelfServiceRetrieval is the single decision point: flip it to false and the endpoint
additionally requires recovery mode, changing nothing else.

The operator-driven DR path is untouched, pinned by a test. Red-proofs observed: removing the
ownership check serves host B's blob to host A; removing the record makes it silent.
This commit is contained in:
2026-08-04 13:39:27 +02:00
parent 9f31956201
commit 435f4a5229
4 changed files with 367 additions and 0 deletions
+125
View File
@@ -239,6 +239,12 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
h.handleHostEscrowPut(w, r, hostID)
// R-199 (v0.94.0): the box-authenticated MIRROR of the PUT above — a host reads back its own
// opaque identity blob so it can be unsealed with the customer's recovery code. Distinct from the
// operator-driven DR path in dr.go, which stays exactly as it is (see handleHostEscrowGet).
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
h.handleHostEscrowGet(w, r, hostID)
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
// operator retrieves it via the /admin/ path (global key only).
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
@@ -1198,6 +1204,125 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
// registered operator-only in notify.operatorOnlyEvents.
const eventRepoKeyChanged = "offsite_repo_key_changed"
// eventEscrowBlobServed (R-199) — a host retrieved its own sealed identity blob. Hub-internal,
// operator-only. See handleHostEscrowGet for why every retrieval is loud.
const eventEscrowBlobServed = "escrow_blob_served"
// escrowSelfServiceRetrieval is THE SINGLE DECISION POINT for the §8.2/§8.3 trade (R-199).
//
// true (§8.2, shipped v0.94.0) — a host may read its own blob whenever it authenticates as itself.
// false (§8.3, the fallback) — the same read additionally requires operator-armed recovery mode.
//
// It is one condition on purpose: the operator may overrule the trade below, and switching must cost a
// boolean rather than a redesign. Everything else in the recovery chain is identical either way.
const escrowSelfServiceRetrieval = true
// handleHostEscrowGet serves a host its OWN opaque identity-escrow blob (R-199, v0.94.0).
//
// WHAT THIS GIVES OUT, WHY IT IS SAFE, AND WHAT IT CHANGES ABOUT WHO IS REQUIRED — recorded here so the
// next reader finds the trade rather than inferring it (the dr.go header convention).
//
// WHAT: the age-wrapped `IdentityBundle` — opaque ciphertext. It carries the offsite restic repository
// password, the tunnel token, the PBS token and the WG key. The hub stores these bytes and has no
// decrypt path; the recovery code R that opens them exists only in the customer's hands.
//
// WHY IT IS SAFE TO GIVE OUT: the blob is useless without R (age scrypt + ChaCha20-Poly1305; a wrong R
// fails closed at the KDF, never to a plausible-but-wrong bundle), and a 10-word EFF code carries ~129
// bits. The caller already authenticates as this host for its report, its desired state, its WG
// registration and its PBS token — this adds no new identity, only a new object, and it is the exact
// MIRROR of the PUT above, which is how the blob got here in the first place.
//
// WHAT IT CHANGES, STATED PLAINLY BECAUSE IT IS THE WHOLE OF THE TRADE: before this, obtaining the blob
// required the OPERATOR to arm recovery mode with the global key (dr.go). Now whoever controls a
// rebuilt box can obtain it with that box's own credential. That is a real reduction in the number of
// parties required. They still cannot open it. The mitigation is that the capability is AUDITED rather
// than silent: every successful retrieval raises an operator event (below), because a silent capability
// is the shape this project has spent two weeks removing.
//
// THE OPERATOR-DRIVEN DR PATH IS UNTOUCHED. `handleReEnroll` / `handleGetRestoreDirective` keep their
// recovery-mode gate and their global-key arming, and they serve the K-escrow and the directive as
// well. This endpoint serves ONE object to ONE authenticated owner. Do not merge them.
func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pathHostID string) {
authHostID, _, isGlobal, ok := h.checkAuthHost(r)
if !ok {
http.Error(w, "Unauthorized", http.StatusUnauthorized)
return
}
if pathHostID == "" {
http.Error(w, "Missing host_id", http.StatusBadRequest)
return
}
// SELF-SCOPED: a per-host key reads only its OWN escrow. The global operator key may read any —
// the same asymmetry the PUT has. Without this line any host key is a fleet-wide blob reader.
if !isGlobal && authHostID != pathHostID {
h.logger.Printf("[WARN] escrow GET REFUSED: host %s asked for %s's blob (self-scope)", authHostID, pathHostID)
http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden)
return
}
host, err := h.store.GetHost(pathHostID)
if err != nil {
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
if host == nil {
http.Error(w, "Unknown host_id", http.StatusNotFound)
return
}
// §8.3 fallback lives here and nowhere else.
if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) {
h.logger.Printf("[WARN] escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID)
http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden)
return
}
bundle, berr := h.store.GetHostDRBundle(pathHostID)
if berr != nil {
h.logger.Printf("[ERROR] escrow GET for %s: %v", pathHostID, berr)
http.Error(w, "Internal error", http.StatusInternalServerError)
return
}
// A host with no sealed bundle gets a CLEAN ANSWER, not a fault: 200 with present=false. A 404
// here would be indistinguishable from an unknown host, and an empty 200 without the flag would be
// indistinguishable from a zero-length blob — both read as "something is broken" to a caller whose
// situation is simply "no ceremony has run yet".
if bundle == nil || len(bundle.IdentityBlob) == 0 {
h.logger.Printf("[INFO] escrow GET for %s: no identity blob stored (no ceremony has run)", pathHostID)
writeJSON(w, http.StatusOK, map[string]any{"host_id": pathHostID, "present": false, "identity_escrow_b64": ""})
return
}
// THE MITIGATION (§8.2). Recorded BEFORE the bytes leave, so a retrieval cannot be served without
// its audit row; a save failure is logged and does NOT block the response (the blob is opaque and
// refusing it would break a recovery over an audit hiccup — but the log line always exists).
//
// SEVERITY = warning, i.e. it reaches the operator by e-mail. Retrieval is not routine today: it
// happens during a recovery and nowhere else. IF a customer-facing self-service flow ever makes it
// routine, revisit this — but revisit it deliberately, do not let it decay to info because the
// mail became annoying.
if host.CustomerID != "" {
msg := fmt.Sprintf("Recovery blob served: host %s retrieved its own sealed identity escrow (%d opaque bytes). "+
"This is the recovery path in use — the blob cannot be opened without the customer's recovery code, which the hub never holds. "+
"If no recovery is in progress on that box, investigate.", pathHostID, len(bundle.IdentityBlob))
details, _ := json.Marshal(map[string]any{
"host_id": pathHostID,
"blob_bytes": len(bundle.IdentityBlob),
"self_scope": !isGlobal,
})
if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil {
h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr)
} else if h.dispatcher != nil {
go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub")
}
}
h.logger.Printf("[WARN] escrow blob SERVED to host %s (%d opaque bytes, self_scope=%v) — recovery path in use",
pathHostID, len(bundle.IdentityBlob), !isGlobal)
writeJSON(w, http.StatusOK, map[string]any{
"host_id": pathHostID,
"present": true,
"identity_escrow_b64": base64.StdEncoding.EncodeToString(bundle.IdentityBlob),
})
}
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password