a491abef6c
THE FACT WAS COMPUTED EVERY CYCLE AND KEPT NOWHERE. EscrowAutoConfirmer.Reconcile
has compared the hub's restic_pw_sha256 against the local key on every ACK since
SLICE 3. On the final-walk venue it logged, at 03:28:03Z and thirty-five minutes
before the customer looked, "the hub's escrow blob does not cover the CURRENT repo
password (hub hash 30ef574f != local 9b4a9a9d)" - and dropped it. The recovery
screen, evaluating in the same process, went on asking a question that could not
see it.
Now persisted: settings.HubEscrowKeySHA256 + HubEscrowKeyCheckedAt, recorded
UNCONDITIONALLY in Reconcile beside RecordPresence and RecordSuperseded - same
place, same reason: the box that needs it most is the rebuilt one with no target,
on which every gate below returns early.
OffsiteRecoveryOffer gains SHAPE (c): the hub holds a package for a key OTHER than
the one we are using. (a) and (b) are both proxies for that question and both have
now been wrong in opposite directions - (a) goes false the moment anything mints,
(b) is unreachable while the escrow is pending.
SEC 7.2, decided deliberately and stated in the code:
- a KNOWN DIFFERENCE offers, however old the reading. Age is not gated on. Both
sides are local; only the hub's half can be stale, and what the hub holds does
not change without a ceremony THIS box runs, which refreshes the hash on the
next ACK. Gating on age would make a box offline from the hub silently stop
offering - the exact failure this session removes. CheckedAt is persisted for
diagnosis, not as a gate.
- an ABSENT hash falls back to (a)/(b) and does NOT offer. "" is the hub
positively saying its package seals no repository password (legacy hash-less
escrow). Nothing to compare, and offering would put a permanent screen in
front of every legacy box.
The write damper: CheckedAt refreshes on every ack carrying a hash, but a save is
skipped when both the hash and the UTC day are unchanged, so an idle box does not
rewrite settings.json every fifteen minutes. It records WHEN WE LAST HEARD, not
when it last changed - the R-100 distinction.
Tests: Scenario C (a differing key offers, with both proxies asserted false first),
Scenario D (a matching key offers nothing), fact 1 still required, shape (a) still
works, and both SEC 7.2 halves.
RED-PROOFS, each with the mutation confirmed present in the file first:
D) hubHash != localHash conjunct dropped -> Scenario D FAILS (a healthy box
offered recovery forever); Scenario C still passes
WIRING) RecordEscrowKeyHash removed from the EscrowAutoConfirmer literal in
main.go -> TestMainWiresRecordEscrowKeyHash FAILS. This is the ships-inert
shape: unwired, everything compiles, every test in the package passes, the
auto-confirm still works, and shape (c) reads an empty hash forever.
Green: go build, go vet, go test ./... all pass.
251 lines
13 KiB
Go
251 lines
13 KiB
Go
package report
|
|
|
|
import (
|
|
"context"
|
|
"log"
|
|
"sync"
|
|
"time"
|
|
)
|
|
|
|
// SLICE 3 — hub-verified escrow auto-confirm. Replaces operator trust ("I ran the ceremony, click
|
|
// confirm") with a verified fact: the hub's report ACK carries the sha256 of the repo password the
|
|
// stored escrow blob COVERS (recorded at ceremony time); the controller flips pending→escrowed ONLY
|
|
// when that hash matches sha256 of its CURRENT local repo password. Blob-presence alone must never
|
|
// confirm — a blob can predate the current password (re-provision, inject, drive history) and a
|
|
// truthful-looking claim on a stale blob would re-open the exact un-recoverable-ciphertext gap fork-4
|
|
// closed. Hashes are non-reversible (256-bit random secrets) and safe to log; passwords never are.
|
|
|
|
// EscrowStatus mirrors the hub ACK's `escrow` object (nil when the hub has no escrow row).
|
|
type EscrowStatus struct {
|
|
IdentityBlobPresent bool `json:"identity_blob_present"`
|
|
ResticPwSHA256 string `json:"restic_pw_sha256"`
|
|
CreatedAt string `json:"created_at"`
|
|
// SupersededPresent / SupersededAt (v0.201.0, R-222) — the hub is ALSO keeping an earlier sealed
|
|
// package, and when it was set aside. Absent on a pre-0.97.0 hub, which reads as "no earlier
|
|
// package" and simply keeps today's message: an older hub cannot make the screen say anything new.
|
|
SupersededPresent bool `json:"superseded_present"`
|
|
SupersededAt string `json:"superseded_at"`
|
|
}
|
|
|
|
// EscrowAutoConfirmer runs the auto-confirm check on each report ACK. Long-lived (one per process) so
|
|
// the mismatch warning dedupes per distinct hash instead of firing every 15-minute cycle.
|
|
type EscrowAutoConfirmer struct {
|
|
// Pending reports whether the offbox target is configured AND EscrowState=="pending" — the
|
|
// confirm-flip state. "escrowed" is never flipped back (auto-UN-confirm does not exist), but
|
|
// since v0.127.0 it IS re-checked: see Escrowed + the stale-blob branch (Scenario F).
|
|
Pending func() bool
|
|
// Escrowed reports whether the offbox target is configured AND EscrowState=="escrowed" — the
|
|
// v0.127.0 stale-blob re-check state (Scenario F: a superseding ceremony that did NOT cover
|
|
// the current password — e.g. a CLI run without the staged secret — must be surfaced, not
|
|
// silently ignored; the spike left the drill box in exactly that state). nil → no re-check.
|
|
Escrowed func() bool
|
|
// LocalHash returns the canonical hash of the local repo password (ok=false → no password file).
|
|
LocalHash func() (hash string, ok bool)
|
|
// Flip transitions EscrowState pending→escrowed (settings.UpdateOffboxStatus).
|
|
Flip func() error
|
|
// Wipe removes the agent-staged secret (best-effort — the flip is the primary effect).
|
|
Wipe func(ctx context.Context) error
|
|
// RecordPresence persists the ACK's `identity_blob_present` — whether the HUB holds a sealed
|
|
// recovery package for this box (v0.199.0, R-204 item 4 / R-193).
|
|
//
|
|
// WHY IT LIVES HERE, in the auto-confirmer, rather than in its own ACK consumer: this is already
|
|
// the ONE place the ACK's escrow object arrives, and it is already wired. A second Reconcile call
|
|
// in main.go would be a second wiring point, and this project's count of features built but never
|
|
// wired is six. Pinned by TestEscrowConfirm_RecordsPresenceEvenWhenOffboxUnconfigured and by the
|
|
// wiring test.
|
|
//
|
|
// It is called FIRST, before every gate below, and that ordering is the whole fix: on a box with
|
|
// no off-site target `Pending()` and `Escrowed()` are both false and Reconcile returned
|
|
// immediately, so the one fact that distinguishes a REBUILT box from a box that never had
|
|
// off-site backups was thrown away on every cycle. nil → not recorded (older wiring, tests).
|
|
RecordPresence func(present bool) error
|
|
// RecordSuperseded persists the ACK's `superseded_present` / `superseded_at` (v0.201.0, R-222) —
|
|
// whether the hub is ALSO keeping an EARLIER sealed package for this box. Wired here for the same
|
|
// reason as RecordPresence: this is the one place the ACK's escrow object already arrives, and a
|
|
// second wiring point in main.go is how this project accumulated six features that were built and
|
|
// never wired. nil → not recorded (older wiring, tests).
|
|
RecordSuperseded func(present bool, at string) error
|
|
// RecordEscrowKeyHash persists the ACK's `restic_pw_sha256` — the hash of the repository password
|
|
// the hub's sealed package COVERS — with the time it was recorded (v0.206.0, R-241).
|
|
//
|
|
// WHY IT LIVES HERE, and it is the point of the whole change: the comparison between this hash and
|
|
// the local key is ALREADY MADE in Reconcile, on every ACK, and has been since SLICE 3 — and the
|
|
// result was used for one warning line and then discarded. On the final-walk venue that line
|
|
// (03:28:03Z) was the correct answer to the recovery screen's real question, thirty-five minutes
|
|
// before the customer looked at a screen that could not see it.
|
|
//
|
|
// Recorded UNCONDITIONALLY, before every gate below, for exactly the reason RecordPresence is: the
|
|
// box that needs this most is the rebuilt one with no configured target, on which `Pending()` and
|
|
// `Escrowed()` are both false and Reconcile used to return immediately. nil → not recorded.
|
|
RecordEscrowKeyHash func(sha, checkedAt string) error
|
|
// Now returns the current time; nil → time.Now. Injected so the persisted "when we last heard"
|
|
// stamp is testable without a sleep.
|
|
Now func() time.Time
|
|
Logger *log.Logger
|
|
|
|
mu sync.Mutex
|
|
warnedHash string // last mismatched hub hash we warned about (dedupe; shared by both branches)
|
|
stale bool // Scenario F: the hub blob does not cover the CURRENT password (display-only)
|
|
// sealedAt is the ACK's escrow created_at (v0.200.0, R-193) — the ONE non-secret fact the
|
|
// recovery screen may state before a code is entered. Recorded on every ACK that carries an
|
|
// escrow object, including on an unconfigured box, for the same reason RecordPresence is.
|
|
sealedAt string
|
|
}
|
|
|
|
// staleHashlessMarker is the warnedHash dedupe sentinel for the hash-less supersession case
|
|
// (the hub hash is EMPTY there, which must still warn exactly once, and must not collide with
|
|
// the zero value of warnedHash).
|
|
const staleHashlessMarker = "(hashless)"
|
|
|
|
// StaleBlob reports the Scenario-F display flag: EscrowState is escrowed but the hub's CURRENT
|
|
// blob does not cover the current repo password. In-memory only (recomputed from ACKs after a
|
|
// restart); NEVER blocks runs and NEVER flips state — the web card renders the warning + the
|
|
// re-ceremony CTA from it.
|
|
func (c *EscrowAutoConfirmer) StaleBlob() bool {
|
|
c.mu.Lock()
|
|
defer c.mu.Unlock()
|
|
return c.stale
|
|
}
|
|
|
|
// SealedAt returns the ACK-reported creation time of the hub's sealed recovery package ("" when no
|
|
// ACK has carried one). In-memory, recomputed from ACKs after a restart — the hub is the authority.
|
|
func (c *EscrowAutoConfirmer) SealedAt() string {
|
|
c.mu.Lock()
|
|
defer c.mu.Unlock()
|
|
return c.sealedAt
|
|
}
|
|
|
|
func (c *EscrowAutoConfirmer) logf(f string, a ...any) {
|
|
if c.Logger != nil {
|
|
c.Logger.Printf(f, a...)
|
|
}
|
|
}
|
|
|
|
// Reconcile applies one ACK's escrow status. Scenarios: match → flip+wipe (A); mismatch → stay pending
|
|
// + warn once per hash (B); no status / no hash / no local file → stay pending silently (C, normal
|
|
// onboarding); escrowed → the v0.127.0 stale-blob re-check (F — warn-only, never a state change);
|
|
// otherwise → no-op (E — offbox not configured).
|
|
func (c *EscrowAutoConfirmer) Reconcile(es *EscrowStatus) {
|
|
if es == nil {
|
|
return
|
|
}
|
|
// FIRST, unconditionally — see RecordPresence. Every gate below is allowed to skip the
|
|
// auto-confirm; none of them may skip this, because an unconfigured box is exactly the case that
|
|
// needs the fact. A record failure is logged and does NOT stop the auto-confirm: the two are
|
|
// independent, and swallowing it silently is the shape this project keeps removing.
|
|
if c.RecordPresence != nil {
|
|
if err := c.RecordPresence(es.IdentityBlobPresent); err != nil {
|
|
c.logf("[WARN] [escrow-confirm] could not record the hub's identity-blob presence (present=%v): %v", es.IdentityBlobPresent, err)
|
|
}
|
|
}
|
|
// Same discipline, same place, same reason (R-222): recorded unconditionally, because the box that
|
|
// needs it most is the unconfigured rebuilt one every gate below skips. Failure is logged, never
|
|
// swallowed, and never blocks the auto-confirm.
|
|
if c.RecordSuperseded != nil {
|
|
if err := c.RecordSuperseded(es.SupersededPresent, es.SupersededAt); err != nil {
|
|
c.logf("[WARN] [escrow-confirm] could not record the hub's superseded-package state (present=%v): %v", es.SupersededPresent, err)
|
|
}
|
|
}
|
|
// R-241: persist the hash the hub's package covers, with the moment we heard it. Same discipline,
|
|
// same place, same reason as the two above — and this one is the fact the recovery screen has been
|
|
// unable to see. A record failure is logged, never swallowed, and never blocks the auto-confirm.
|
|
if c.RecordEscrowKeyHash != nil {
|
|
if err := c.RecordEscrowKeyHash(es.ResticPwSHA256, c.now().UTC().Format(time.RFC3339)); err != nil {
|
|
c.logf("[WARN] [escrow-confirm] could not record the hub's escrowed-key hash (%.12s…): %v", es.ResticPwSHA256, err)
|
|
}
|
|
}
|
|
c.mu.Lock()
|
|
c.sealedAt = es.CreatedAt // in-memory only; a timestamp, never a secret
|
|
c.mu.Unlock()
|
|
if !c.Pending() {
|
|
// Scenario F (v0.127.0): an ESCROWED box re-checks the hash on every ACK — a superseding
|
|
// blob that does not cover the current password must be surfaced (warn + card flag), while
|
|
// runs continue and the state stays escrowed (no auto-UN-confirm, ever).
|
|
if c.Escrowed != nil && c.Escrowed() {
|
|
c.reconcileEscrowed(es)
|
|
}
|
|
return
|
|
}
|
|
// Fail-closed: the hash must exist AND ride a present identity blob (the hash-bearing container).
|
|
// A hash-less blob is a legacy/password-less escrow — the deprecated manual confirm covers those.
|
|
if es.ResticPwSHA256 == "" || !es.IdentityBlobPresent {
|
|
return
|
|
}
|
|
localHash, ok := c.LocalHash()
|
|
if !ok {
|
|
return // no local repo password file — nothing to verify against
|
|
}
|
|
if localHash != es.ResticPwSHA256 {
|
|
// The stored escrow does NOT cover the current key — flipping would be a false custody claim.
|
|
c.mu.Lock()
|
|
warned := c.warnedHash == es.ResticPwSHA256
|
|
c.warnedHash = es.ResticPwSHA256
|
|
c.mu.Unlock()
|
|
if !warned {
|
|
c.logf("[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password (hub hash %.12s… != local %.12s…) — run the escrow ceremony (wizard /backup/escrow, or felhom-agent --selftest=escrow-create --upload); staying pending", es.ResticPwSHA256, localHash)
|
|
}
|
|
return
|
|
}
|
|
if err := c.Flip(); err != nil {
|
|
c.logf("[ERROR] [escrow-confirm] hash matched but the escrowed flip failed (retries next cycle): %v", err)
|
|
return
|
|
}
|
|
c.logf("[INFO] [escrow-confirm] hub-verified: the escrow covers the current repo password (hash %.12s…) — EscrowState auto-confirmed escrowed; offsite runs enabled", es.ResticPwSHA256)
|
|
if c.Wipe != nil {
|
|
wctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
|
defer cancel()
|
|
if err := c.Wipe(wctx); err != nil {
|
|
c.logf("[ERROR] [escrow-confirm] escrowed but the agent-staged secret was NOT wiped: %v", err)
|
|
}
|
|
}
|
|
c.mu.Lock()
|
|
c.stale = false // a fresh hub-verified confirm clears any earlier stale flag
|
|
c.mu.Unlock()
|
|
}
|
|
|
|
// reconcileEscrowed is the Scenario-F branch (§8 truth table, escrowed rows): compare the ACK's
|
|
// hash exactly as the pending branch does; a mismatch OR a present blob with an EMPTY hash (the
|
|
// hash-less supersession — the spike's exact case) raises the stale flag + ONE warn per distinct
|
|
// hub hash (warnedHash reuse); a match clears the flag. State is never flipped; runs never block
|
|
// (offsite backups still protect against non-total loss).
|
|
func (c *EscrowAutoConfirmer) reconcileEscrowed(es *EscrowStatus) {
|
|
localHash, ok := c.LocalHash()
|
|
if !ok {
|
|
return // no local repo password file — nothing to compare against
|
|
}
|
|
hubHash := es.ResticPwSHA256
|
|
if hubHash != "" && hubHash == localHash {
|
|
c.mu.Lock()
|
|
c.stale = false
|
|
c.mu.Unlock()
|
|
return
|
|
}
|
|
// Stale: hash mismatch, or a blob whose hash is empty (hash-less supersession). Dedupe the
|
|
// warn per distinct hub hash; the empty hash dedupes under a sentinel so it still fires once.
|
|
dedupeKey := hubHash
|
|
if dedupeKey == "" {
|
|
dedupeKey = staleHashlessMarker
|
|
}
|
|
c.mu.Lock()
|
|
warned := c.warnedHash == dedupeKey
|
|
c.warnedHash = dedupeKey
|
|
c.stale = true
|
|
c.mu.Unlock()
|
|
if warned {
|
|
return
|
|
}
|
|
if hubHash == "" {
|
|
c.logf("[WARN] [escrow-confirm] STALE escrow: the hub's current blob carries NO password hash (hash-less supersession) — the stored recovery bundle does not cover the offsite password; create a new recovery code (wizard /backup/escrow). State stays escrowed; runs continue")
|
|
return
|
|
}
|
|
c.logf("[WARN] [escrow-confirm] STALE escrow: the hub's current blob does not cover the CURRENT repo password (hub hash %.12s… != local %.12s…) — create a new recovery code (wizard /backup/escrow). State stays escrowed; runs continue", hubHash, localHash)
|
|
}
|
|
|
|
// now returns the injected clock or time.Now.
|
|
func (c *EscrowAutoConfirmer) now() time.Time {
|
|
if c.Now != nil {
|
|
return c.Now()
|
|
}
|
|
return time.Now()
|
|
}
|