R-241 part 3: abandoning starts a 14-day countdown that ends the question
Until now "set aside" renamed the remote store and touched neither the escrow
nor the key, so the hub went on holding a sealed package for a key the box no
longer used. Shape (c) compares those two, finds them different, and offers
recovery - correctly, and for ever. A customer who had already said "I do not
want the old data" would be asked again at every login.
The operator's ruling is that the answer is NOT a "they decided" flag: fix the
state, do not remember that it is wrong. So the decision starts a countdown,
at the end of which the set-aside store and the sealed package that protects
it are removed TOGETHER. Afterwards shape (c) has nothing to compare and the
offer falls silent on its own - because the state is right, not because
something remembers it once was not.
THE GRACE IS REAL. The recovery offer stays reachable for the whole 14 days;
that is the change-of-mind path, and a grace in which recovery is impossible
would be decorative.
BOTH HALVES OR NEITHER. Removing only the store leaves a package that opens
nothing; removing only the package leaves ciphertext nobody can ever decrypt.
The two cannot be atomic across two machines, so it is a two-phase commit:
delete the store, record a durable marker, and keep DECLARING
offsite.abandon_purge_requested until the hub's ACK stops reporting a
superseded package. A crash between the halves re-declares on the next sweep;
it never leaves the pair half-removed and silent.
HUB HALF - SEC 8.2 ANSWERED: yes, the hub was needed, and only for this.
store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is
ever undone, and it never touches host_escrow (the package covering the key
the box uses now). The handler acts on the DECLARATION, never an inference,
and is placed immediately BEFORE the ACK is built - so
GetEscrowStatusForCustomer reads the effect and the SAME response closes the
box's two-phase commit. No second round-trip and no window where the box
thinks it is still owed. felhom-agent was NOT touched.
The countdown starts in ResetOrphanedRepo, NOT in the shared helper: the
helper is also the unclaimed auto-reset path, where nobody decided anything,
and an as-delivered box tidying a stranger's leftover store must not get a
customer's deletion clock. Pinned by a test.
Cancellation is wired into the recovery unlock, BEFORE the tier-up and the
listing - those can fail, and a countdown surviving a successful unlock
because a later step errored would delete the history the customer just
proved they can open.
The sweep is a Daily job at 05:10, not on the backup leg: it must run on a box
whose tier is not configured for runs. Quiet by construction on every box with
no countdown, and that silence is asserted.
Tests (all clock-injected; SEC 7.4 forbids shortening a live timer):
Scenario E (aside + package kept + countdown + offer still reachable, and
NOTHING deleted), Scenario F (both halves, the declaration repeating, the
close-out), Scenario G (cancel, path still nameable, no later deletion),
plus: not closed out while the package remains, a transport failure leaves the
countdown due and retrying, the no-op sweep issues zero remote commands, and
the unclaimed auto-reset starts no countdown.
RED-PROOFS, each with the mutation confirmed present in the file first:
F1) store deletion skipped -> Scenario F FAILS (no rm issued)
F2) declaration dropped from the report -> Scenario F FAILS (the hub is
never asked; the package would outlive the store for ever)
G) CancelAbandon made a no-op -> Scenario G FAILS (uncancellable countdown)
Green: controller and hub both build, vet and test clean; controller gates OK.
NOTHING WAS DELETED ANYWHERE - the terminal step has only ever run against
in-test fakes.
This commit is contained in:
@@ -0,0 +1,247 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-241 — abandoning starts a countdown that ENDS THE QUESTION (Scenarios E, F, G).
|
||||
//
|
||||
// The countdown is driven by an injected clock throughout. §7.4 forbids shortening a live timer to
|
||||
// watch the terminal step fire: it is the only thing in the product that deletes a customer's
|
||||
// off-site history, and a step tested once on real data is a step regretted once.
|
||||
|
||||
// abandonFixture: an orphaned, configured box holding a key, with the hub holding a package for a
|
||||
// DIFFERENT key — i.e. shape (c) is live and the customer is being offered recovery.
|
||||
// Returns the manager and a recorder of every remote shell command issued.
|
||||
type sshRecorder struct{ cmds []string }
|
||||
|
||||
func (r *sshRecorder) run(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error) {
|
||||
r.cmds = append(r.cmds, remoteCmd)
|
||||
return []byte(""), nil
|
||||
}
|
||||
|
||||
func abandonFixture(t *testing.T, now time.Time) (*Manager, *settings.Settings, *sshRecorder) {
|
||||
t.Helper()
|
||||
m, sett, _ := offerFixture(t, true)
|
||||
if err := sett.SetHubEscrowKeySHA256(otherKeyHash, now.Format(time.RFC3339)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sett.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
|
||||
o.EscrowState = "escrowed"
|
||||
o.RepoState = "orphaned"
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := &sshRecorder{}
|
||||
m.SetOffboxSSH(rec.run)
|
||||
m.SetOffboxRunner(func(ctx context.Context, env []string, args ...string) ([]byte, error) { return []byte(""), nil })
|
||||
m.SetOffboxClock(func() time.Time { return now })
|
||||
return m, sett, rec
|
||||
}
|
||||
|
||||
// ── SCENARIO E — abandoning sets aside, keeps the package, starts a countdown, stays reversible ──
|
||||
func TestR241_ScenarioE_AbandonStartsAReversibleCountdown(t *testing.T) {
|
||||
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
|
||||
m, _, rec := abandonFixture(t, start)
|
||||
|
||||
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
|
||||
t.Fatalf("abandon: %v", err)
|
||||
}
|
||||
// The store was MOVED, not deleted — no rm anywhere in this phase.
|
||||
joined := strings.Join(rec.cmds, " | ")
|
||||
if !strings.Contains(joined, "mv ") {
|
||||
t.Errorf("the old store must be moved aside; commands were: %s", joined)
|
||||
}
|
||||
if strings.Contains(joined, "rm -rf") {
|
||||
t.Fatalf("NOTHING may be deleted when the customer abandons — only at the end of the grace. Commands: %s", joined)
|
||||
}
|
||||
st := m.AbandonStatus()
|
||||
if !st.Active {
|
||||
t.Fatal("a countdown must be running after an abandonment")
|
||||
}
|
||||
if got := st.DueAt.Sub(start); got != abandonGraceDays*24*time.Hour {
|
||||
t.Errorf("countdown length = %v, want %d days", got, abandonGraceDays)
|
||||
}
|
||||
if st.DaysLeft != abandonGraceDays {
|
||||
t.Errorf("DaysLeft = %d, want %d", st.DaysLeft, abandonGraceDays)
|
||||
}
|
||||
if st.RepoPath == "" {
|
||||
t.Error("the set-aside path must be recorded, or the terminal step has nothing to delete")
|
||||
}
|
||||
// THE GRACE IS REAL: the recovery offer stays reachable for the whole window. A grace in which
|
||||
// recovery is impossible would be decorative.
|
||||
if !m.OffsiteRecoveryOffer() {
|
||||
t.Fatal("the recovery offer MUST stay reachable during the grace — that is the change-of-mind path")
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO G — changing your mind inside the window ───────────────────────────────────────────
|
||||
//
|
||||
// RED-PROOF: make the countdown uncancellable (delete the body of CancelAbandon). The countdown then
|
||||
// survives a successful recovery and this test fails — a customer who proved they hold their code
|
||||
// would still have the history deleted under them.
|
||||
func TestR241_ScenarioG_RecoveryInsideTheWindowCancelsTheCountdown(t *testing.T) {
|
||||
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
|
||||
m, _, _ := abandonFixture(t, start)
|
||||
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
day6 := start.AddDate(0, 0, 6)
|
||||
m.SetOffboxClock(func() time.Time { return day6 })
|
||||
if st := m.AbandonStatus(); !st.Active || st.DaysLeft != 8 {
|
||||
t.Fatalf("precondition: day 6 of 14 should leave 8 days, got %+v", st)
|
||||
}
|
||||
pathBefore := m.AbandonStatus().RepoPath
|
||||
|
||||
m.CancelAbandon("the customer recovered with their code")
|
||||
|
||||
st := m.AbandonStatus()
|
||||
if st.Active {
|
||||
t.Fatal("a countdown must be cancellable — the customer found their code")
|
||||
}
|
||||
if st.RepoPath != pathBefore {
|
||||
t.Errorf("the set-aside store must stay NAMEABLE after a cancel: got %q want %q", st.RepoPath, pathBefore)
|
||||
}
|
||||
// And a sweep now deletes nothing, on any later date.
|
||||
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 90) })
|
||||
deleted, err := m.AbandonSweep(context.Background())
|
||||
if err != nil || deleted {
|
||||
t.Fatalf("a cancelled countdown must never delete: deleted=%v err=%v", deleted, err)
|
||||
}
|
||||
}
|
||||
|
||||
// ── SCENARIO F — the countdown ends the question, and removes BOTH halves ────────────────────────
|
||||
//
|
||||
// RED-PROOF (store half): make AbandonSweep skip the rm. The first assertion fails.
|
||||
// RED-PROOF (package half): drop AbandonPurgeRequested from OffboxReportStatus. The declaration
|
||||
// assertion fails — the hub is never asked and the package outlives the store for ever.
|
||||
func TestR241_ScenarioF_TerminalStepRemovesBothHalvesTogether(t *testing.T) {
|
||||
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
|
||||
m, sett, rec := abandonFixture(t, start)
|
||||
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
setAside := m.AbandonStatus().RepoPath
|
||||
|
||||
// Not due yet — nothing happens, quietly.
|
||||
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 13) })
|
||||
if deleted, err := m.AbandonSweep(context.Background()); deleted || err != nil {
|
||||
t.Fatalf("day 13 must not delete: deleted=%v err=%v", deleted, err)
|
||||
}
|
||||
|
||||
// Due.
|
||||
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 14).Add(time.Minute) })
|
||||
rec.cmds = nil
|
||||
deleted, err := m.AbandonSweep(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("terminal step: %v", err)
|
||||
}
|
||||
if !deleted {
|
||||
t.Fatal("the terminal step must delete when due")
|
||||
}
|
||||
// HALF 1: the store is gone.
|
||||
joined := strings.Join(rec.cmds, " | ")
|
||||
if !strings.Contains(joined, "rm -rf") || !strings.Contains(joined, setAside) {
|
||||
t.Fatalf("the set-aside store at %s must be deleted; commands: %s", setAside, joined)
|
||||
}
|
||||
// HALF 2: the hub is ASKED for the package, and keeps being asked until it confirms.
|
||||
st := m.OffboxReportStatus()
|
||||
if st == nil || !st.AbandonPurgeRequested {
|
||||
t.Fatalf("the report must declare abandon_purge_requested until the hub drops the package, got %+v", st)
|
||||
}
|
||||
// It repeats — a lost request must retry rather than leave the pair half-removed.
|
||||
if d2, err2 := m.AbandonSweep(context.Background()); d2 || err2 != nil {
|
||||
t.Fatalf("a second sweep must be a quiet no-op while awaiting the hub: deleted=%v err=%v", d2, err2)
|
||||
}
|
||||
if st2 := m.OffboxReportStatus(); st2 == nil || !st2.AbandonPurgeRequested {
|
||||
t.Fatal("the declaration must persist across sweeps until confirmed")
|
||||
}
|
||||
|
||||
// The hub confirms by no longer reporting a superseded package → the question is over.
|
||||
m.ClearAbandonPurgeIfConfirmed(false)
|
||||
if got := sett.GetOffboxTarget(); got.AbandonPurgeRequested || got.AbandonRepoPath != "" || got.AbandonAt != "" {
|
||||
t.Errorf("the abandonment must be fully closed out, got %+v", got)
|
||||
}
|
||||
if st3 := m.OffboxReportStatus(); st3 != nil && st3.AbandonPurgeRequested {
|
||||
t.Error("the declaration must stop once the hub has confirmed")
|
||||
}
|
||||
}
|
||||
|
||||
// While the hub STILL reports a superseded package, the close-out must not fire — otherwise the box
|
||||
// stops asking and the package outlives the store silently, which is exactly half of Scenario F.
|
||||
func TestR241_PurgeIsNotClosedOutWhileThePackageRemains(t *testing.T) {
|
||||
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
|
||||
m, sett, _ := abandonFixture(t, start)
|
||||
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 15) })
|
||||
if _, err := m.AbandonSweep(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.ClearAbandonPurgeIfConfirmed(true) // the hub STILL holds a retained package
|
||||
if !sett.GetOffboxTarget().AbandonPurgeRequested {
|
||||
t.Fatal("the request must stand while the hub still reports a superseded package")
|
||||
}
|
||||
}
|
||||
|
||||
// A transport failure during the terminal step must NOT clear the countdown — it retries tomorrow.
|
||||
// Silently abandoning the abandonment would leave the store for ever with nothing counting down.
|
||||
func TestR241_TerminalStepFailureKeepsTheCountdownDue(t *testing.T) {
|
||||
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
|
||||
m, sett, _ := abandonFixture(t, start)
|
||||
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.SetOffboxSSH(func(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error) {
|
||||
return []byte("ssh: connect to host nas.local port 22: No route to host"), context.DeadlineExceeded
|
||||
})
|
||||
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 15) })
|
||||
deleted, err := m.AbandonSweep(context.Background())
|
||||
if deleted || err == nil {
|
||||
t.Fatalf("a failed deletion must be reported, not swallowed: deleted=%v err=%v", deleted, err)
|
||||
}
|
||||
got := sett.GetOffboxTarget()
|
||||
if got.AbandonAt == "" || got.AbandonPurgeRequested {
|
||||
t.Fatalf("a failed terminal step must leave the countdown DUE and unrequested, got %+v", got)
|
||||
}
|
||||
if !m.AbandonStatus().Active {
|
||||
t.Error("the countdown must still be active so tomorrow's sweep retries")
|
||||
}
|
||||
}
|
||||
|
||||
// Quiet by construction: a box with no countdown does no work and says nothing (§ the daily job's
|
||||
// own contract). Asserted, because "it probably does nothing" is how a sweep with a bug hides.
|
||||
func TestR241_Sweep_QuietWhenNothingDue(t *testing.T) {
|
||||
m, _, rec := abandonFixture(t, time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC))
|
||||
deleted, err := m.AbandonSweep(context.Background())
|
||||
if deleted || err != nil {
|
||||
t.Fatalf("a box with no countdown must be a pure no-op: deleted=%v err=%v", deleted, err)
|
||||
}
|
||||
if len(rec.cmds) != 0 {
|
||||
t.Fatalf("a no-op sweep must issue no remote commands, got %v", rec.cmds)
|
||||
}
|
||||
if m.AbandonStatus().Active {
|
||||
t.Error("no countdown should be reported")
|
||||
}
|
||||
}
|
||||
|
||||
// The UNCLAIMED auto-reset must NOT start a customer countdown — nobody decided anything there.
|
||||
// An as-delivered box tidying a stranger's leftover store must not put a 14-day deletion clock on it.
|
||||
func TestR241_UnclaimedAutoResetStartsNoCountdown(t *testing.T) {
|
||||
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
|
||||
m, _, _ := abandonFixture(t, start)
|
||||
t2 := m.settings.GetOffboxTarget()
|
||||
base, env := m.offboxBaseArgs(t2)
|
||||
if err := m.resetOrphanedRepo(context.Background(), base, env, "auto (unclaimed)"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m.AbandonStatus().Active {
|
||||
t.Fatal("the unclaimed auto-reset must not start a customer abandonment countdown")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user