R-241 part 3: abandoning starts a 14-day countdown that ends the question
Until now "set aside" renamed the remote store and touched neither the escrow
nor the key, so the hub went on holding a sealed package for a key the box no
longer used. Shape (c) compares those two, finds them different, and offers
recovery - correctly, and for ever. A customer who had already said "I do not
want the old data" would be asked again at every login.
The operator's ruling is that the answer is NOT a "they decided" flag: fix the
state, do not remember that it is wrong. So the decision starts a countdown,
at the end of which the set-aside store and the sealed package that protects
it are removed TOGETHER. Afterwards shape (c) has nothing to compare and the
offer falls silent on its own - because the state is right, not because
something remembers it once was not.
THE GRACE IS REAL. The recovery offer stays reachable for the whole 14 days;
that is the change-of-mind path, and a grace in which recovery is impossible
would be decorative.
BOTH HALVES OR NEITHER. Removing only the store leaves a package that opens
nothing; removing only the package leaves ciphertext nobody can ever decrypt.
The two cannot be atomic across two machines, so it is a two-phase commit:
delete the store, record a durable marker, and keep DECLARING
offsite.abandon_purge_requested until the hub's ACK stops reporting a
superseded package. A crash between the halves re-declares on the next sweep;
it never leaves the pair half-removed and silent.
HUB HALF - SEC 8.2 ANSWERED: yes, the hub was needed, and only for this.
store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is
ever undone, and it never touches host_escrow (the package covering the key
the box uses now). The handler acts on the DECLARATION, never an inference,
and is placed immediately BEFORE the ACK is built - so
GetEscrowStatusForCustomer reads the effect and the SAME response closes the
box's two-phase commit. No second round-trip and no window where the box
thinks it is still owed. felhom-agent was NOT touched.
The countdown starts in ResetOrphanedRepo, NOT in the shared helper: the
helper is also the unclaimed auto-reset path, where nobody decided anything,
and an as-delivered box tidying a stranger's leftover store must not get a
customer's deletion clock. Pinned by a test.
Cancellation is wired into the recovery unlock, BEFORE the tier-up and the
listing - those can fail, and a countdown surviving a successful unlock
because a later step errored would delete the history the customer just
proved they can open.
The sweep is a Daily job at 05:10, not on the backup leg: it must run on a box
whose tier is not configured for runs. Quiet by construction on every box with
no countdown, and that silence is asserted.
Tests (all clock-injected; SEC 7.4 forbids shortening a live timer):
Scenario E (aside + package kept + countdown + offer still reachable, and
NOTHING deleted), Scenario F (both halves, the declaration repeating, the
close-out), Scenario G (cancel, path still nameable, no later deletion),
plus: not closed out while the package remains, a transport failure leaves the
countdown due and retrying, the no-op sweep issues zero remote commands, and
the unclaimed auto-reset starts no countdown.
RED-PROOFS, each with the mutation confirmed present in the file first:
F1) store deletion skipped -> Scenario F FAILS (no rm issued)
F2) declaration dropped from the report -> Scenario F FAILS (the hub is
never asked; the package would outlive the store for ever)
G) CancelAbandon made a no-op -> Scenario G FAILS (uncancellable countdown)
Green: controller and hub both build, vet and test clean; controller gates OK.
NOTHING WAS DELETED ANYWHERE - the terminal step has only ever run against
in-test fakes.
This commit is contained in:
@@ -0,0 +1,215 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// ABANDONMENT — deciding to give up the old off-site history is a finishable thing (R-241, v0.206.0).
|
||||
//
|
||||
// THE PROBLEM THIS SOLVES. `resetOrphanedRepo` renamed the remote store aside and touched neither the
|
||||
// escrow nor the key, so the hub went on holding a sealed package for a key the box no longer used.
|
||||
// Shape (c) compares those two, finds them different, and offers recovery — correctly, and for ever.
|
||||
// A customer who has already said "I do not want the old data" would be asked again at every login.
|
||||
//
|
||||
// THE OPERATOR'S RULING (2026-08-07) is that the answer is NOT a "they decided" flag. A flag would
|
||||
// leave the box in a state that is genuinely wrong (the hub holding a package for a key nobody uses)
|
||||
// and paper over it. Instead the decision starts a **14-day countdown**, at the end of which the
|
||||
// set-aside store and the sealed package that protects it are removed TOGETHER — after which there is
|
||||
// nothing left to compare and nothing left to ask about. **Fix the state, do not remember that it is
|
||||
// wrong.**
|
||||
//
|
||||
// THE GRACE IS REAL, NOT DECORATIVE. The recovery offer stays reachable for the whole window; that is
|
||||
// the change-of-mind path (Scenario G). A grace period during which recovery is impossible would be
|
||||
// theatre.
|
||||
|
||||
// abandonGraceDays is the countdown the operator set. Reminders fire at 5, 3 and 1 days (see
|
||||
// AbandonRemindAtDays) — visible, reversible, and running out in public.
|
||||
const abandonGraceDays = 14
|
||||
|
||||
// AbandonRemindAtDays are the remaining-day marks at which the abandoning box reminds the customer.
|
||||
// Descending, so the surface can pick the first one that has been reached.
|
||||
var AbandonRemindAtDays = []int{5, 3, 1}
|
||||
|
||||
// abandonNow is the countdown's clock seam. Tests inject; nil → time.Now. It exists so the terminal
|
||||
// step can be driven deterministically — §7.4 forbids shortening a live timer to watch it fire,
|
||||
// because that is how an irreversible step gets tested once and regretted once.
|
||||
func (m *Manager) abandonNow() time.Time {
|
||||
if m.offboxNow != nil {
|
||||
return m.offboxNow()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
// SetOffboxClock injects the abandonment clock (tests only).
|
||||
func (m *Manager) SetOffboxClock(fn func() time.Time) { m.offboxNow = fn }
|
||||
|
||||
// startAbandonCountdown records the decision and the date the terminal step will run. Called by
|
||||
// resetOrphanedRepo AFTER the move-aside has succeeded — a countdown started before the store has
|
||||
// actually moved would count down to deleting a path that does not exist.
|
||||
func (m *Manager) startAbandonCountdown(setAsidePath string) {
|
||||
now := m.abandonNow().UTC()
|
||||
due := now.AddDate(0, 0, abandonGraceDays)
|
||||
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
|
||||
o.AbandonStartedAt = now.Format(time.RFC3339)
|
||||
o.AbandonAt = due.Format(time.RFC3339)
|
||||
o.AbandonRepoPath = setAsidePath
|
||||
o.AbandonPurgeRequested = false
|
||||
}); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] could not record the abandonment countdown: %v", err)
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] abandonment countdown started: the set-aside history at %s and the hub's sealed package "+
|
||||
"are removed together on %s (%d days). The recovery screen stays reachable until then.",
|
||||
setAsidePath, due.Format("2006-01-02"), abandonGraceDays)
|
||||
}
|
||||
|
||||
// AbandonState is the surface's read model. Zero value = nothing in progress.
|
||||
type AbandonState struct {
|
||||
Active bool // a countdown is running
|
||||
StartedAt time.Time //
|
||||
DueAt time.Time // when the terminal step runs
|
||||
DaysLeft int // ceiling, so "0 days left" only ever means "today"
|
||||
RepoPath string // the set-aside store awaiting deletion
|
||||
PurgeRequested bool // the store is gone; awaiting the hub to drop the sealed package
|
||||
}
|
||||
|
||||
// AbandonStatus reports the countdown for the UI and the report. It never mutates.
|
||||
func (m *Manager) AbandonStatus() AbandonState {
|
||||
t := m.settings.GetOffboxTarget()
|
||||
if t == nil {
|
||||
return AbandonState{}
|
||||
}
|
||||
st := AbandonState{RepoPath: t.AbandonRepoPath, PurgeRequested: t.AbandonPurgeRequested}
|
||||
if t.AbandonAt == "" {
|
||||
return st
|
||||
}
|
||||
due, err := time.Parse(time.RFC3339, t.AbandonAt)
|
||||
if err != nil {
|
||||
// A malformed stamp must not silently mean "never due" — that would strand the store for ever
|
||||
// with a countdown the customer can see and nothing behind it.
|
||||
m.logger.Printf("[WARN] [offbox] abandonment due-date is unparseable (%q) — treating the countdown as NOT running: %v", t.AbandonAt, err)
|
||||
return st
|
||||
}
|
||||
st.Active, st.DueAt = true, due
|
||||
if s, serr := time.Parse(time.RFC3339, t.AbandonStartedAt); serr == nil {
|
||||
st.StartedAt = s
|
||||
}
|
||||
// Ceiling: a countdown with 30 minutes left says "1 day", never "0". Zero is reserved for due.
|
||||
remaining := due.Sub(m.abandonNow())
|
||||
if remaining <= 0 {
|
||||
st.DaysLeft = 0
|
||||
} else {
|
||||
st.DaysLeft = int((remaining + 24*time.Hour - time.Nanosecond) / (24 * time.Hour))
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// CancelAbandon stops a running countdown — the change-of-mind path (Scenario G). Called when a
|
||||
// recovery succeeds: the customer has their code after all, and the history they were about to give
|
||||
// up is exactly what the code opens.
|
||||
//
|
||||
// It clears the schedule but KEEPS AbandonRepoPath, so the set-aside store remains nameable on the
|
||||
// backups page. Nothing has been deleted at this point by construction — the terminal step is the
|
||||
// only thing that deletes, and it has not run.
|
||||
func (m *Manager) CancelAbandon(reason string) {
|
||||
t := m.settings.GetOffboxTarget()
|
||||
if t == nil || (t.AbandonAt == "" && !t.AbandonPurgeRequested) {
|
||||
return // nothing running — silent, so a healthy recovery does not log about a countdown
|
||||
}
|
||||
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
|
||||
o.AbandonStartedAt, o.AbandonAt = "", ""
|
||||
o.AbandonPurgeRequested = false
|
||||
}); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] could not cancel the abandonment countdown: %v", err)
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] abandonment countdown CANCELLED (%s) — the set-aside history at %s is kept and nothing was deleted", reason, t.AbandonRepoPath)
|
||||
}
|
||||
|
||||
// AbandonSweep is the daily terminal step. It is the ONLY thing in the product that deletes a
|
||||
// customer's off-site history, and it does so on a date the customer was shown.
|
||||
//
|
||||
// ⚠ IT REMOVES BOTH HALVES OR NEITHER — Scenario F. The set-aside store and the sealed package that
|
||||
// protects it are the two halves of one thing; removing only the store leaves the hub holding a
|
||||
// package for a key that opens nothing, and removing only the package leaves ciphertext nobody can
|
||||
// ever decrypt. Either is a state that asks a question nobody can answer.
|
||||
//
|
||||
// The two halves cannot be made atomic across two machines, so this is a two-phase commit with the
|
||||
// STORE FIRST and a durable marker: delete the remote store, record AbandonPurgeRequested, and keep
|
||||
// declaring it in the report until the hub's ACK stops reporting a superseded package. A crash
|
||||
// between the two leaves the marker set and the next sweep re-declares — it never leaves the pair
|
||||
// half-removed and silent.
|
||||
//
|
||||
// Returns (deleted, err). deleted=false with err=nil is the normal "nothing due" case.
|
||||
func (m *Manager) AbandonSweep(ctx context.Context) (bool, error) {
|
||||
st := m.AbandonStatus()
|
||||
// Phase 2 outstanding: the store is gone, the hub has not confirmed. Re-declare and wait.
|
||||
if st.PurgeRequested {
|
||||
m.logger.Printf("[DEBUG] [offbox] abandonment: the set-aside store is deleted; awaiting the hub to drop the sealed package")
|
||||
return false, nil
|
||||
}
|
||||
if !st.Active || st.DueAt.After(m.abandonNow()) {
|
||||
return false, nil // not due — quiet by construction on every healthy box
|
||||
}
|
||||
t := m.settings.GetOffboxTarget()
|
||||
if t == nil || t.AbandonRepoPath == "" {
|
||||
m.logger.Printf("[WARN] [offbox] abandonment is due but no set-aside path is recorded — nothing deleted; clearing the countdown so it does not retry for ever")
|
||||
m.CancelAbandon("no set-aside path recorded")
|
||||
return false, fmt.Errorf("abandonment due with no recorded path")
|
||||
}
|
||||
port := t.Port
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
m.logger.Printf("[WARN] [offbox] abandonment DUE — deleting the set-aside off-site history at %s (chosen by the customer on %s; this is irreversible)",
|
||||
t.AbandonRepoPath, st.StartedAt.Format("2006-01-02"))
|
||||
out, err := m.sshRunner()(ctx, t.Host, t.User, port, m.offboxKeyPath(), m.offboxKnownHosts(),
|
||||
"rm -rf "+shellQuote(t.AbandonRepoPath))
|
||||
if err != nil {
|
||||
// NOT cleared: a transport failure must retry tomorrow, not silently abandon the abandonment.
|
||||
m.logger.Printf("[ERROR] [offbox] abandonment: deleting the set-aside history failed — the countdown stays due and retries: %v: %s", err, truncate(out))
|
||||
return false, fmt.Errorf("delete set-aside history: %w", err)
|
||||
}
|
||||
// Phase 1 done. Record it durably BEFORE anything else, so a crash here re-declares rather than
|
||||
// forgetting that the store is already gone.
|
||||
if uerr := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
|
||||
o.AbandonPurgeRequested = true
|
||||
o.AbandonAt = "" // the schedule has fired; the marker now drives the rest
|
||||
}); uerr != nil {
|
||||
m.logger.Printf("[ERROR] [offbox] abandonment: the store was deleted but the marker could not be saved — the hub's package may outlive it: %v", uerr)
|
||||
return true, uerr
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] abandonment: set-aside history deleted; requesting the hub to drop the sealed package that protected it")
|
||||
if m.offboxOrphanEvent != nil {
|
||||
m.offboxOrphanEvent("offbox_abandon_completed", t.AbandonRepoPath)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// ClearAbandonPurgeIfConfirmed closes the two-phase commit: once the hub's ACK stops reporting a
|
||||
// superseded package, both halves are gone and the abandonment is finished. Called from the ACK path.
|
||||
//
|
||||
// This is what makes §2.1 work without a "they decided" flag: afterwards the hub holds a package for
|
||||
// the key the box is actually using (or none at all), shape (c) has nothing to compare, and the
|
||||
// recovery offer falls silent on its own — because the state is right, not because something is
|
||||
// remembering that it once was not.
|
||||
func (m *Manager) ClearAbandonPurgeIfConfirmed(supersededPresent bool) {
|
||||
t := m.settings.GetOffboxTarget()
|
||||
if t == nil || !t.AbandonPurgeRequested || supersededPresent {
|
||||
return
|
||||
}
|
||||
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
|
||||
o.AbandonPurgeRequested = false
|
||||
o.AbandonRepoPath = ""
|
||||
o.AbandonStartedAt = ""
|
||||
o.OrphanedRenamedTo = ""
|
||||
}); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] could not close out the abandonment: %v", err)
|
||||
return
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] abandonment COMPLETE — the set-aside history and the sealed package that protected it are both gone; nothing further to ask about")
|
||||
}
|
||||
Reference in New Issue
Block a user