hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
This commit is contained in:
@@ -0,0 +1,146 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-197 (v0.93.0) — a changed offsite repository key must be REPORTED. Both halves of the comparison
|
||||
// (host_escrow.restic_pw_sha256 and the row it replaces) have been in the database since SLICE 3 and
|
||||
// nothing read them: demo-felhom's repository password changed on 2026-08-03, orphaning 36 snapshots /
|
||||
// 1.14 GB, and no event, e-mail, card or log line said so for thirteen hours.
|
||||
//
|
||||
// Driven through the real endpoint (PUT /hosts/<id>/escrow), not by calling the emitter, so the wiring
|
||||
// is part of what is proven.
|
||||
|
||||
// escrowBodyWithHash is escrowBody plus the sealed-password hash and an identity blob — the fork-4
|
||||
// upload shape. Hash VALUES here are test fixtures, never real secrets.
|
||||
func escrowBodyWithHash(blob []byte, resticPwSHA, identity string) string {
|
||||
m := map[string]string{
|
||||
"blob_b64": base64.StdEncoding.EncodeToString(blob),
|
||||
"key_fingerprint": "ab:cd:ef",
|
||||
"posture": "zero_knowledge",
|
||||
"created_at": "2026-08-04T05:00:00Z",
|
||||
"restic_pw_sha256": resticPwSHA,
|
||||
}
|
||||
if identity != "" {
|
||||
m["identity_blob_b64"] = base64.StdEncoding.EncodeToString([]byte(identity))
|
||||
}
|
||||
b, _ := json.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// Scenario D — a new escrow sealing a DIFFERENT repo password raises exactly one operator signal,
|
||||
// naming the host, carrying NO hash value.
|
||||
// RED-PROOF: remove the maybeEmitRepoKeyChanged call from handleHostEscrowPut (or make its predicate
|
||||
// always fall through) → no event → this FAILS.
|
||||
func TestEscrowPut_ChangedRepoKey_RaisesSignal(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
|
||||
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-gen1")); rr.Code != http.StatusOK {
|
||||
t.Fatalf("first PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||
}
|
||||
// Generation 1 is onboarding, not a change — nothing may fire yet.
|
||||
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||
t.Fatalf("a FIRST escrow must not raise %s: %+v", eventRepoKeyChanged, ev)
|
||||
}
|
||||
|
||||
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_GEN2", "age-gen2")); rr.Code != http.StatusOK {
|
||||
t.Fatalf("second PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
ev, err := st.GetLatestEventByType("c1", eventRepoKeyChanged)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ev == nil {
|
||||
t.Fatal("R-197: the repository key demonstrably changed and NO signal was raised — this is the " +
|
||||
"thirteen-hour silence that let demo-felhom lose 1.14 GB unremarked")
|
||||
}
|
||||
if ev.Severity != "warning" {
|
||||
t.Errorf("severity = %q, want warning (info is an intentional non-notify — the operator would not be told)", ev.Severity)
|
||||
}
|
||||
if !strings.Contains(ev.Message, "h1") {
|
||||
t.Errorf("message must name the host, got %q", ev.Message)
|
||||
}
|
||||
// NO HASH VALUE may travel — not in the message, not in the details.
|
||||
for _, forbidden := range []string{"SHA_GEN1", "SHA_GEN2"} {
|
||||
if strings.Contains(ev.Message, forbidden) || strings.Contains(ev.DetailsJSON, forbidden) {
|
||||
t.Errorf("a repo-password hash value leaked into the event (%s): msg=%q details=%q", forbidden, ev.Message, ev.DetailsJSON)
|
||||
}
|
||||
}
|
||||
// Exactly one, not one per anything.
|
||||
evs, _ := st.GetRecentEvents("c1", 50)
|
||||
n := 0
|
||||
for _, e := range evs {
|
||||
if e.EventType == eventRepoKeyChanged {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("%s fired %d times for one supersession, want 1 (edge-triggered)", eventRepoKeyChanged, n)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario E — an UNCHANGED key says nothing. A customer re-running a ceremony for good reasons must
|
||||
// not be punished for it with an alarm.
|
||||
func TestEscrowPut_UnchangedRepoKey_Silent(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
|
||||
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_SAME", "age-1"))
|
||||
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_SAME", "age-2"))
|
||||
|
||||
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||
t.Fatalf("a same-hash re-ceremony must be SILENT, got: %+v", ev)
|
||||
}
|
||||
// And it must not have created a superseded row either (pre-existing contract, re-asserted here
|
||||
// because Scenario E's silence would also be produced by the supersede branch never running).
|
||||
if n, _ := st.CountSupersededEscrow("h1"); n != 0 {
|
||||
t.Fatalf("same-hash re-upload created %d superseded row(s), want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The in-between shapes must not be silent-and-indistinguishable either: a hash-less NEW blob cannot
|
||||
// show whether the key changed, and saying nothing at all would look identical to "unchanged".
|
||||
// Asserted at the level the code offers — no event (correct: nothing was measured) and no crash.
|
||||
func TestEscrowPut_HashlessSupersession_NoSignal(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
|
||||
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-1"))
|
||||
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "", "age-2")); rr.Code != http.StatusOK {
|
||||
t.Fatalf("hash-less supersession PUT = %d", rr.Code)
|
||||
}
|
||||
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||
t.Fatalf("a hash-less new blob cannot EVIDENCE a change; it must not claim one: %+v", ev)
|
||||
}
|
||||
// The retention still ran (the hashes differ, so this IS a supersession) — the key is kept even
|
||||
// though the change cannot be evidenced.
|
||||
if n, _ := st.CountSupersededEscrow("h1"); n != 1 {
|
||||
t.Fatalf("superseded rows = %d, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// R-97a's lesson, pinned in the SAME commit that mints the type: an operator-tier event that is not in
|
||||
// operatorOnlyEvents reaches customers as raw English, because a missing customerMessages entry is NOT
|
||||
// a block. Checked here rather than in notify so the type's two registers are asserted together.
|
||||
func TestRepoKeyChanged_IsOperatorOnly(t *testing.T) {
|
||||
if !notify.IsOperatorOnly(eventRepoKeyChanged) {
|
||||
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
|
||||
"about escrow custody", eventRepoKeyChanged)
|
||||
}
|
||||
// The two R-192 types have the same property and the same reason (see the register's comment).
|
||||
for _, et := range []string{"offsite_delivery_stuck", "offsite_credential_restaged"} {
|
||||
if !notify.IsOperatorOnly(et) {
|
||||
t.Errorf("%s is not registered operator-only", et)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -155,7 +155,7 @@ func TestReportACK_EscrowStatus(t *testing.T) {
|
||||
}
|
||||
|
||||
// escrow row with identity blob + hash → the ACK carries all three fields
|
||||
if _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {
|
||||
|
||||
@@ -1065,9 +1065,13 @@ func (h *Handler) reissueOnReenroll(cc *store.CustomerConfig) {
|
||||
}
|
||||
}
|
||||
// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||
// one-time password only ever reached the OLD controller). The re-issuer resets the restic repo
|
||||
// password, which makes the OLD escrow blob stale — the offsite provisioner invalidates the
|
||||
// escrow (2.3) and emits both events. Skips silently when offsite isn't provisioned/enabled.
|
||||
// one-time password only ever reached the OLD controller). ⚠ CORRECTED 2026-08-04 (R-196): this
|
||||
// used to claim "the re-issuer resets the restic repo password, which makes the OLD escrow blob
|
||||
// stale". It does not and cannot — the re-issuer resets the PROVIDER account password; the
|
||||
// repository password is generated on the box and never leaves it except sealed under R. The
|
||||
// provisioner does mark the escrow stale, but PRECAUTIONARILY (see the reasoning at
|
||||
// offsite.ReissueCredentials), not because this call rotated anything.
|
||||
// Skips silently when offsite isn't provisioned/enabled.
|
||||
if h.offsiteReissuer != nil {
|
||||
if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
|
||||
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
|
||||
@@ -1141,7 +1145,7 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
||||
// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
|
||||
// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
|
||||
// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
|
||||
superseded, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
||||
superseded, prevPwSHA, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
||||
if serr != nil {
|
||||
h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
@@ -1157,6 +1161,10 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
||||
"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
|
||||
h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
|
||||
}
|
||||
// R-197: the box's offsite DATA key demonstrably changed. Both halves of that comparison
|
||||
// have been stored since SLICE 3 and nothing read them — demo-felhom's key changed on
|
||||
// 2026-08-03 and nothing said so for thirteen hours.
|
||||
h.maybeEmitRepoKeyChanged(host.CustomerID, pathHostID, prevPwSHA, req.ResticPwSHA256, n)
|
||||
}
|
||||
}
|
||||
// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
|
||||
@@ -1185,6 +1193,69 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
||||
w.Write([]byte(`{"status":"ok"}`))
|
||||
}
|
||||
|
||||
// eventRepoKeyChanged (R-197) — the box's offsite restic REPOSITORY password changed, proven by the
|
||||
// hub's own stored hashes. Hub-internal (not in allowedEventTypes, like escrow_superseded) and
|
||||
// registered operator-only in notify.operatorOnlyEvents.
|
||||
const eventRepoKeyChanged = "offsite_repo_key_changed"
|
||||
|
||||
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
|
||||
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
|
||||
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
|
||||
// changed on 2026-08-03, orphaning 36 snapshots / 1.14 GB, and no event, e-mail, card or log line
|
||||
// said so for thirteen hours — the comparison that eventually found it is this one
|
||||
// (audits/RECON-offsite-dr-chain-2026-08-04.md, R-197).
|
||||
//
|
||||
// THE PREDICATE IS DELIBERATELY NARROW: both hashes known AND different. A first-ever hash (prev "")
|
||||
// is onboarding, not a change; a hash-less supersession (now "") cannot show a change happened; an
|
||||
// identical hash is a re-ceremony of the SAME password, which is a normal healthy act and must stay
|
||||
// silent or a customer is punished for re-running a ceremony. The in-between shapes are LOGGED rather
|
||||
// than dropped, so "we chose not to alarm" and "the check did not run" never look identical.
|
||||
//
|
||||
// SEVERITY = warning, chosen for the world v0.93.0 creates rather than the one it inherits. Before
|
||||
// R-198 a changed key meant the previous history was unopenable by anyone, ever — that would have
|
||||
// argued for error. From v0.93.0 the superseding ceremony RETAINS the old identity blob, so the
|
||||
// previous history stays recoverable with the recovery code that sealed it: the situation is "this
|
||||
// customer's off-site history now depends on an older recovery code", which is operator-actionable
|
||||
// (check the orphan card, expect a fresh repository) and is not a loss. warning also routes: the
|
||||
// dispatcher notifies on warning/error/critical and treats info as an intentional non-notify, and
|
||||
// the whole point of this row is that the operator learns on the day.
|
||||
//
|
||||
// EDGE-TRIGGERED: called only from the superseded branch of the escrow PUT, i.e. once per
|
||||
// supersession, never per report. No timer lives here — the dispatcher owns cooldown.
|
||||
//
|
||||
// NO HASH VALUE TRAVELS. The message and the details name the host, the customer and the retained
|
||||
// count only. The hashes are non-reversible, but a hash is still a fingerprint of a live secret and
|
||||
// this project's rule is that values do not leave the store.
|
||||
func (h *Handler) maybeEmitRepoKeyChanged(customerID, hostID, prevSHA, newSHA string, retained int) {
|
||||
switch {
|
||||
case prevSHA == "":
|
||||
h.logger.Printf("[INFO] escrow for host %s: no previous repo-password hash recorded (first hash, or a legacy hash-less blob) — repo-key-change check not applicable", hostID)
|
||||
return
|
||||
case newSHA == "":
|
||||
h.logger.Printf("[WARN] escrow for host %s: the NEW blob carries no repo-password hash (hash-less supersession) — whether the repository key changed CANNOT be determined from the hub's data", hostID)
|
||||
return
|
||||
case prevSHA == newSHA:
|
||||
return // same password re-sealed: a healthy re-ceremony (Scenario E — silence is correct)
|
||||
}
|
||||
msg := fmt.Sprintf("Offsite repository key CHANGED for host %s: the new escrow seals a different repository password than the one it replaced. "+
|
||||
"The previous off-site history is no longer opened by this box's current key. The superseding blob was retained (%d held), so that history stays recoverable with the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an orphan card, and expect the next backup to start a fresh history.",
|
||||
hostID, retained)
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"host_id": hostID,
|
||||
"retained_count": retained,
|
||||
"repo_key": "changed", // never the hash values
|
||||
})
|
||||
if _, err := h.store.SaveEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub"); err != nil {
|
||||
h.logger.Printf("[WARN] %s event save failed for %s: %v", eventRepoKeyChanged, hostID, err)
|
||||
return // audit row first: an e-mail without its event row lies (the OffsiteChecker convention)
|
||||
}
|
||||
h.logger.Printf("[WARN] offsite repository key CHANGED for host %s (customer %s) — previous history now depends on the superseded recovery code; %d retained blob(s)",
|
||||
hostID, customerID, retained)
|
||||
if h.dispatcher != nil {
|
||||
go h.dispatcher.ProcessEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
|
||||
// SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
|
||||
// host api_key. The secret is stored at rest and NEVER logged (only the username + a length are
|
||||
|
||||
Reference in New Issue
Block a user