R-241 part 2: the comparison the box already makes becomes the thing that offers recovery

THE FACT WAS COMPUTED EVERY CYCLE AND KEPT NOWHERE. EscrowAutoConfirmer.Reconcile has compared the hub's restic_pw_sha256 against the local key on every ACK since SLICE 3. On the final-walk venue it logged, at 03:28:03Z and thirty-five minutes before the customer looked, "the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574f != local 9b4a9a9d)" - and dropped it. The recovery screen, evaluating in the same process, went on asking a question that could not see it. Now persisted: settings.HubEscrowKeySHA256 + HubEscrowKeyCheckedAt, recorded UNCONDITIONALLY in Reconcile beside RecordPresence and RecordSuperseded - same place, same reason: the box that needs it most is the rebuilt one with no target, on which every gate below returns early. OffsiteRecoveryOffer gains SHAPE (c): the hub holds a package for a key OTHER than the one we are using. (a) and (b) are both proxies for that question and both have now been wrong in opposite directions - (a) goes false the moment anything mints, (b) is unreachable while the escrow is pending. SEC 7.2, decided deliberately and stated in the code: - a KNOWN DIFFERENCE offers, however old the reading. Age is not gated on. Both sides are local; only the hub's half can be stale, and what the hub holds does not change without a ceremony THIS box runs, which refreshes the hash on the next ACK. Gating on age would make a box offline from the hub silently stop offering - the exact failure this session removes. CheckedAt is persisted for diagnosis, not as a gate. - an ABSENT hash falls back to (a)/(b) and does NOT offer. "" is the hub positively saying its package seals no repository password (legacy hash-less escrow). Nothing to compare, and offering would put a permanent screen in front of every legacy box. The write damper: CheckedAt refreshes on every ack carrying a hash, but a save is skipped when both the hash and the UTC day are unchanged, so an idle box does not rewrite settings.json every fifteen minutes. It records WHEN WE LAST HEARD, not when it last changed - the R-100 distinction. Tests: Scenario C (a differing key offers, with both proxies asserted false first), Scenario D (a matching key offers nothing), fact 1 still required, shape (a) still works, and both SEC 7.2 halves. RED-PROOFS, each with the mutation confirmed present in the file first: D) hubHash != localHash conjunct dropped -> Scenario D FAILS (a healthy box offered recovery forever); Scenario C still passes WIRING) RecordEscrowKeyHash removed from the EscrowAutoConfirmer literal in main.go -> TestMainWiresRecordEscrowKeyHash FAILS. This is the ships-inert shape: unwired, everything compiles, every test in the package passes, the auto-confirm still works, and shape (c) reads an empty hash forever. Green: go build, go vet, go test ./... all pass.
2026-08-07 11:33:42 +02:00
parent 763de3a025
commit a491abef6c
6 changed files with 346 additions and 3 deletions
@@ -1522,13 +1522,49 @@ func (m *Manager) needsOffsiteCredential(t *settings.OffboxTarget) bool {
 //
 // Scenario B still holds exactly: a healthy box has its own password and is not orphaned; a box that
 // never had off-site backups fails fact 1; an unclaimed box never reaches an authenticated page.
+// ── SHAPE (c), v0.206.0, R-241 — THE DISCRIMINATOR THAT ANSWERS THE REAL QUESTION ───────────────
+//
+// Shapes (a) and (b) are both PROXIES for one question — *does the hub hold a package for a key other
+// than the one I am using?* — and both have now been wrong, in opposite directions:
+//
+//   - (a) "no repository password" went false the moment anything minted one. Before v0.206.0's mint
+//     guard that happened by itself, ~30 minutes after a rebuild, and the customer who logged in the
+//     next morning never saw the screen. That is R-241.
+//   - (b) "a run proved the repo will not open" is unreachable on exactly that box: the only producer
+//     of RepoState=="orphaned" is ensureOffboxRepo, which is downstream of the escrow gate in
+//     runOffboxBackup, and the escrow can never confirm while the hub's package covers a different
+//     key. Self-locking.
+//
+// (c) asks the question directly, from two facts the box already holds: the hash the hub's package
+// covers (ACK-cached) and the hash of the key on disk. **This comparison was already computed on every
+// ACK and thrown away** — see settings.HubEscrowKeySHA256.
+//
+// ⚠ §7.2 — WHAT A STALE OR ABSENT READING RESOLVES TO, decided deliberately rather than by default:
+//
+//   - **A KNOWN DIFFERENCE OFFERS, however old the reading.** Age is not gated on. Both sides of the
+//     comparison are local; only the hub's half can be stale, and what the hub holds does not change
+//     without a ceremony THIS BOX runs — which refreshes the hash on the next ACK. Gating on age would
+//     add a second failure mode (a box offline from the hub silently stops offering) to fix a window
+//     that closes itself. `HubEscrowKeyCheckedAt` is persisted for diagnosis, not as a gate.
+//   - **AN ABSENT HASH FALLS BACK TO (a)/(b), it does not offer.** "" is what the hub sends for a
+//     legacy hash-less package — one that provably seals no repository password. There is nothing for
+//     (c) to compare, and offering on it would put a permanent screen in front of every legacy box.
+//     This is the one place where "not knowing" resolves to silence, and it does so because an empty
+//     hash is not an unknown: it is the hub positively saying the package covers no key.
+//
+// So: fail-closed (offer) on a known difference; fall back on a hash never learned. Pinned by
+// TestR241_ScenarioD_* and TestR241_StaleComparison_*.
 func (m *Manager) OffsiteRecoveryOffer() bool {
 	if m.settings == nil || !m.settings.GetHubEscrowIdentityPresent() {
 		return false // the hub holds nothing for us — nothing to recover
 	}
-	if _, ok := m.OffboxRepoPasswordHash(); !ok {
+	localHash, hasLocal := m.OffboxRepoPasswordHash()
+	if !hasLocal {
 		return true // (a) no repository password at all — the pristine rebuilt box
 	}
+	if hubHash, _ := m.settings.GetHubEscrowKeySHA256(); hubHash != "" && hubHash != localHash {
+		return true // (c) the hub's package covers a DIFFERENT key than the one we are using
+	}
 	return m.OffboxOrphaned() // (b) a password exists but the inherited history will not open under it
 }

@@ -0,0 +1,143 @@
+package backup
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
+)
+
+// R-241 shape (c) — the recovery offer is driven by the comparison the box already makes.
+//
+// Scenarios C and D from the task, plus §7.2's two staleness cases. The point of shape (c) is that
+// it asks the real question — *does the hub hold a package for a key other than the one I am
+// using?* — rather than the two proxies that have each now been wrong in opposite directions.
+
+// offerFixture builds a manager holding a repository password, with the hub's cached facts settable.
+// Returns the local key's hash so a test can make the hub's hash match or differ deliberately.
+func offerFixture(t *testing.T, hubHoldsPackage bool) (*Manager, *settings.Settings, string) {
+	t.Helper()
+	m, sett, pwPath := mintGuardManager(t, false) // mint freely first
+	if err := sett.SetOffboxTarget(&settings.OffboxTarget{
+		Enabled: true, Host: "nas.local", Port: 22, User: "felhom", RepoPath: "/srv/repo", Schedule: "daily",
+	}); err != nil {
+		t.Fatal(err)
+	}
+	if err := m.WriteOffboxSecrets("KEYMATERIAL", "nas.local ssh-ed25519 HOSTKEY"); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := os.Stat(pwPath); err != nil {
+		t.Fatalf("fixture should hold a repository password: %v", err)
+	}
+	local, ok := m.OffboxRepoPasswordHash()
+	if !ok {
+		t.Fatal("fixture should be able to hash its own key")
+	}
+	if err := sett.SetHubEscrowIdentityPresent(hubHoldsPackage); err != nil {
+		t.Fatal(err)
+	}
+	return m, sett, local
+}
+
+const otherKeyHash = "9b4a9a9dcec7898e7544f35b18470aac77c3d9064e5d3a302897617fa62edd65"
+
+// ── SCENARIO C — a differing key offers recovery, whatever the reason for the difference ────────
+//
+// This is the venue's exact state on 2026-08-07: a key present, no orphan recorded, escrow stuck
+// pending — and before shape (c), silence.
+func TestR241_ScenarioC_DifferingKeyOffersRecovery(t *testing.T) {
+	m, sett, local := offerFixture(t, true)
+	if local == otherKeyHash {
+		t.Fatal("fixture precondition: the local key must differ from the hub's")
+	}
+	if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2026-08-07T03:28:03Z"); err != nil {
+		t.Fatal(err)
+	}
+	// Neither proxy fires: a key EXISTS (so not shape (a)) and nothing is orphaned (so not shape (b)).
+	if _, ok := m.OffboxRepoPasswordHash(); !ok {
+		t.Fatal("precondition: shape (a) must be false")
+	}
+	if m.OffboxOrphaned() {
+		t.Fatal("precondition: shape (b) must be false")
+	}
+	if !m.OffsiteRecoveryOffer() {
+		t.Fatal("R-241: the hub holds a package for a DIFFERENT key and the screen was not offered — this is the defect")
+	}
+}
+
+// ── SCENARIO D — a healthy box is never offered recovery ────────────────────────────────────────
+//
+// RED-PROOF: drop the `hubHash != localHash` conjunct in shape (c) (make it `hubHash != ""`). A
+// healthy box is then offered recovery forever, and this test fails — which is how a screen stops
+// being read.
+func TestR241_ScenarioD_MatchingKeyOffersNothing(t *testing.T) {
+	m, sett, local := offerFixture(t, true)
+	if err := sett.SetHubEscrowKeySHA256(local, "2026-08-07T09:00:00Z"); err != nil {
+		t.Fatal(err)
+	}
+	if m.OffsiteRecoveryOffer() {
+		t.Fatal("a box whose key the hub's package covers must never be offered recovery")
+	}
+}
+
+// A box the hub holds nothing for is never offered, even if a stale hash lingers in settings. Fact 1
+// stays required — the spike's comment block calls dropping it "the plausible wrong fix".
+func TestR241_ShapeC_NeverHadOffsiteIsStillSilent(t *testing.T) {
+	m, sett, _ := offerFixture(t, false) // the hub holds NOTHING
+	if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2026-08-07T09:00:00Z"); err != nil {
+		t.Fatal(err)
+	}
+	if m.OffsiteRecoveryOffer() {
+		t.Fatal("a box that never had off-site backups must never be greeted by a recovery screen")
+	}
+}
+
+// ── §7.2 — the staleness decision, both halves ──────────────────────────────────────────────────
+
+// A KNOWN DIFFERENCE OFFERS, however old the reading. Age is deliberately not gated on: gating would
+// make a box offline from the hub silently stop offering, which is the failure this session exists
+// to remove.
+func TestR241_StaleComparison_KnownDifferenceStillOffers(t *testing.T) {
+	m, sett, _ := offerFixture(t, true)
+	if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2020-01-01T00:00:00Z"); err != nil { // ancient
+		t.Fatal(err)
+	}
+	if !m.OffsiteRecoveryOffer() {
+		t.Fatal("a known difference must offer regardless of how old the reading is (§7.2)")
+	}
+}
+
+// AN ABSENT HASH FALLS BACK TO (a)/(b) — it does not offer. "" is the hub positively saying its
+// package seals no repository password (legacy hash-less escrow); there is nothing to compare, and
+// offering would put a permanent screen in front of every legacy box.
+func TestR241_StaleComparison_AbsentHashFallsBackAndDoesNotOffer(t *testing.T) {
+	m, sett, _ := offerFixture(t, true)
+	if err := sett.SetHubEscrowKeySHA256("", ""); err != nil {
+		t.Fatal(err)
+	}
+	if m.OffsiteRecoveryOffer() {
+		t.Fatal("a hash never learned must fall back to (a)/(b), not offer (§7.2)")
+	}
+	// ...and the fallback still works: mark the repo orphaned and shape (b) fires as before.
+	if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) { o.RepoState = "orphaned" }); err != nil {
+		t.Fatal(err)
+	}
+	if !m.OffsiteRecoveryOffer() {
+		t.Fatal("shape (b) must still work when the hub's hash was never learned")
+	}
+}
+
+// Shape (a) is untouched: a box with no key at all is still offered, which is the pristine rebuild.
+func TestR241_ShapeAStillWorks(t *testing.T) {
+	m, sett, _ := offerFixture(t, true)
+	if err := os.Remove(filepath.Join(m.cfg.Paths.DataDir, "offbox", "repo_password")); err != nil {
+		t.Fatal(err)
+	}
+	if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2026-08-07T09:00:00Z"); err != nil {
+		t.Fatal(err)
+	}
+	if !m.OffsiteRecoveryOffer() {
+		t.Fatal("shape (a) — no repository password at all — must still offer")
+	}
+}