v0.289.0: off-site key cannot delete — append-only rclone transport, box sends only its public key (hub registrar), retention only inside a hub window behind the fake-snapshot guard (decisions 68-69, R-820, R-822)
gates / gates (push) Successful in 27s
gates / gates (push) Successful in 27s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -1,3 +1,33 @@
|
||||
## v0.289.0 — the off-site key cannot delete: append-only transport, no password on the box, retention only inside a hub window (decisions 68–69, R-820, R-822) (2026-10-03)
|
||||
|
||||
**MinAgent: 0.131.0** (unchanged). **Needs hub v0.127.0** (the key registrar and the window endpoints; an older hub
|
||||
answers 404 to `register-key` and the apply-bridge keeps retrying). No new household string.
|
||||
|
||||
- **The box never fetches the Storage Box password any more.** The apply-bridge (`offsiteapply`) sends its PUBLIC key
|
||||
to the hub's registrar (`HubRegistrar`: `register-key`, `confirm-key`), which pins it in the sub-account to
|
||||
`command="rclone serve restic --stdio --append-only <repo>",restrict`; the box then PROVES the key reaches the pinned
|
||||
server (`PinnedProber`: exit 0 and rclone's output — an unpinned key gets the restricted shell, exit 8, measured)
|
||||
before configuring anything. `HTTPConsumer`, `SSHCopyIDInstaller` and the `sshpass` path are gone. A box upgraded
|
||||
from v0.288.0 re-applies once (`descriptorHash` gains `|pinned-v1`) and re-registers its EXISTING key.
|
||||
- **Transport.** A hub-provisioned target (`Transport: "rclone-pinned"`, set by `ApplyOffsiteTarget`) uses restic's
|
||||
`rclone:` backend with `-o rclone.program="ssh -p 23 … -i <key> … rclone"` — restic 0.14.0 suffices, rclone is NOT in
|
||||
the image (it runs at the provider). An sftp-written repository reads, extends, restores and `check --read-data`s
|
||||
through it (measured on the provider). The household's own SFTP NAS target is unchanged.
|
||||
- **Retention leaves the box on the pinned tier (decision 68).** Both `forget --prune` sites (after a run; over quota)
|
||||
go through `offsiteWindowRetention`: no window → nothing deleted; inside a hub-granted window the **fake-snapshot
|
||||
guard (R-822)** refuses on any future-dated snapshot, any snapshot newer than the hub's bound, or a plan that would
|
||||
remove a snapshot younger than 8 days; otherwise the OLDEST `max_remove` planned snapshots are forgotten by id and the
|
||||
window is closed with counts. **Disagreement recorded:** the brief said abort when the plan exceeds a week's
|
||||
removal; the first window after the interim legitimately does, so the box caps and takes the oldest instead.
|
||||
- **Move-aside is the hub's** (`POST /offsite/move-aside`); **abandonment is deferred to the operator** on the pinned
|
||||
tier — nothing deleted, `offbox_abandon_deferred` (operator-only) — because the key cannot delete (R-823).
|
||||
- Transport failures of the `rclone:` backend (`error talking HTTP to rclone`) classify as transport.
|
||||
- Bug found by the existing suite and fixed before release: the NAS move-aside path assigned a shadowed `newPath`.
|
||||
- Tests: `offsiteapply` rewritten (fresh, upgraded, already-pinned, registered-but-not-pinned, wrong fingerprint,
|
||||
host-key mismatch, idempotent, confirm failure); `offbox_window_test.go` (the lab's 13 future fakes refused; recent
|
||||
removal refused; honest plan oldest-first capped; pinned run never forgets without a window; pinned move-aside asks the
|
||||
hub; pinned abandonment defers). Red-proofs: `felhom.eu/documentation/audits/offsite-lock-build-2026-10-03/partC/`.
|
||||
|
||||
## v0.288.0 — remove tells the truth about the household's files (decision 67, R-800) (2026-10-02)
|
||||
|
||||
**MinAgent: 0.131.0** (unchanged). Needs hub v0.123.0 (unchanged). New string: `layout.userdata_marad` (both languages).
|
||||
|
||||
@@ -245,6 +245,8 @@
|
||||
| `report.SetPendingLogTails` + `buildLogTailsSection` | controller/internal/report/logtail.go | ACK `log_tail_requests` → next report `log_tails` | THE pull-based ACK-flag pattern (hub asks, controller pushes next cycle) — copy for any new hub→box request | Consume-once drain at BuildReport; failed push re-arms from the hub's still-pending request; NEVER add a hub→controller push channel |
|
||||
| `metrics.FetchContainerLogTail` | controller/internal/metrics/logscanner.go | `(name, tailLines) (string, error)` | Raw per-container `docker logs --tail=N` | 15s timeout; caller caps/redacts (capTailLines) |
|
||||
| `ConfigRefresher.Reconcile` | controller/internal/report/config_refresh.go | `(ackVersion int)` | Pull-based config refresh | Re-pulls controller.yaml (re-merging local_api), then graceful self-restart; first-run = baseline, no restart |
|
||||
| `offsiteapply.HubRegistrar` / `HubWindowClient` / `PinnedProber` (v0.289.0, decisions 68–69) | controller/internal/offsiteapply/seams.go | `Register(ctx,pub)(fp,err)` · `Confirm` · `MoveAside` · `Open/Close` window · `Probe(ctx,host,user,port,kh,privPEM) bool` | EVERY off-site key install, the hub move-aside, the clean-up window | **The box never handles the sub-account password** — there is no consume path any more. `PinnedProber` is a POSITIVE observable (exit 0 + rclone output); "authenticates" is NOT enough — an unpinned key authenticates and can delete. |
|
||||
| `Manager.offsiteWindowRetention` + `offsiteGuard` (v0.289.0, R-822) | controller/internal/backup/offbox_window.go | `(ctx, base, env, why)` · pure `(all, plan, now, newestAllowed, max) (ids, refuse)` | THE retention step for both callers (after a run, over quota) | Pinned tier: no window → nothing deleted; the guard runs BEFORE any forget; forget is by explicit ids, oldest first. NAS tier: the old SP-2 policy, unchanged. |
|
||||
| `offsiteapply.SettleProvider` / `SettleFunc` / `Bridge.AwaitSettle` / `ReconcileWhenSettled` (R-71a, v0.162.0) | controller/internal/offsiteapply/offsiteapply.go + seams.go | `SettleState() (version, floor string, updateRunning, floorKnown bool)` | THE settle-gate: defers the offsite one-time-password consume past a managed day-0 floor-update (the F10 race). Wire the `SettleFunc` adapter over `updater.GetFloor()`/`IsUpdateRunning()` — **the updater's knowledge is the ONE floor source; never fetch the floor a second way**. Gate ONLY the bridge goroutine, and only when an updater exists (nil `Settle` = reconcile immediately). Bounds `settlePoll`/`settleFloorSubBound`/`settleOverallBound`; the floor is in-memory (report-ACK-derived, ~5–10 s), NOT persisted → unknown until the first ACK on any restart. Inject `Now`/`Sleep` in tests (no real sleeps). B′: at/above-floor GOes on the first poll, zero wait. Do NOT touch the consume/persist order or the 404 contract — ordering only |
|
||||
| `bootstrap.MaybeIngest` / `RefreshConfig` | controller/internal/bootstrap/bootstrap.go | bootstrap.json → controller.yaml | Day-0 + refresh | Overwrites controller.yaml, NEVER settings.json |
|
||||
| `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image |
|
||||
|
||||
@@ -155,6 +155,12 @@ backups, monitoring and notifications. All Proxmox/disk operations are delegated
|
||||
action block right; used by the dashboard installed-apps list, the Távoli mentés toggle list
|
||||
and the Visszaállítás restore-to-verify/.fab lists; the backups-apps expander header is
|
||||
ALIGNED to the same grammar (own markup — it carries the toggle). Protected infra stacks
|
||||
**The off-site key cannot delete (v0.289.0, decisions 68–69):** the hub-provisioned tier is reached through an
|
||||
APPEND-ONLY key the hub's registrar pins in the Storage Box sub-account (`rclone serve restic --stdio --append-only`);
|
||||
the box sends only its public key (`offsiteapply.HubRegistrar`) and never sees the sub-account password. Transport is
|
||||
restic `rclone:` over ssh port 23 (`settings.OffboxTarget.Transport = "rclone-pinned"`); the household's own SFTP NAS
|
||||
is unchanged. Retention runs only inside a hub-opened weekly window, behind the fake-snapshot guard
|
||||
(`backup/offbox_window.go`); the orphan move-aside is the hub's; a due abandonment is deferred to the operator.
|
||||
**Apps go off-site by themselves (v0.283.0, decision 50):** a fresh install on a box whose customer has off-site
|
||||
switches the app's off-site copy ON (`settings.DefaultOffboxOnForNewApp`, deploy-done hook); an earlier choice is
|
||||
kept. Older apps: one press on both backup pages (`/backup/offbox/enable-all`, „Nem most" dismisses). The size
|
||||
|
||||
@@ -772,14 +772,30 @@ func main() {
|
||||
// reconcile between placing a recovered key and reading the repository (R-219). nil when off-site
|
||||
// is not configured for this customer, which the web seam treats as "skip".
|
||||
var offsiteBridge *offsiteapply.Bridge
|
||||
offsiteRegistrar := offsiteapply.HubRegistrar{HubURL: cfg.Hub.URL, CustomerID: cfg.Customer.ID, APIKey: cfg.Hub.APIKey}
|
||||
if backupMgr != nil && cfg.Hub.URL != "" && cfg.Hub.APIKey != "" {
|
||||
// The orphan move-aside is the HUB's now (decision 69): the box's pinned key reaches only the
|
||||
// append-only rclone server and cannot rename a directory.
|
||||
backupMgr.SetOffsiteMoveAside(offsiteRegistrar.MoveAside)
|
||||
// Decision 68: retention on the append-only tier happens only inside a hub-opened window.
|
||||
backupMgr.SetOffsiteWindowClient(offsiteapply.HubWindowClient{Registrar: offsiteRegistrar})
|
||||
}
|
||||
if backupMgr != nil && cfg.Offsite.Enabled && cfg.Hub.URL != "" && cfg.Hub.APIKey != "" {
|
||||
bridge := &offsiteapply.Bridge{
|
||||
Cfg: cfg,
|
||||
Consumer: offsiteapply.HTTPConsumer{HubURL: cfg.Hub.URL, CustomerID: cfg.Customer.ID, APIKey: cfg.Hub.APIKey},
|
||||
// Decision 69 (v0.289.0): the box sends its PUBLIC key to the hub's registrar, which pins it
|
||||
// append-only; the box never receives the sub-account password.
|
||||
Registrar: offsiteRegistrar,
|
||||
Scanner: offsiteapply.KeyscanScanner{},
|
||||
KeyGen: offsiteapply.ED25519KeyGen{},
|
||||
Installer: offsiteapply.SSHCopyIDInstaller{},
|
||||
Prober: offsiteapply.SFTPKeyAuthProber{KeyPath: filepath.Join(cfg.Paths.DataDir, "offbox", "ssh_key")},
|
||||
Prober: offsiteapply.PinnedProber{},
|
||||
Existing: func() string {
|
||||
b, err := os.ReadFile(filepath.Join(cfg.Paths.DataDir, "offbox", "ssh_key"))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return string(b)
|
||||
},
|
||||
Enabler: offsiteapply.EnablerFunc(func(ctx context.Context, host, user string, port int, repoPath, priv, kh string, quotaGB int) error {
|
||||
tgt := &settings.OffboxTarget{Enabled: true, Host: host, User: user, Port: port, RepoPath: repoPath, Schedule: "daily", QuotaGB: quotaGB}
|
||||
stage := func(ctx context.Context, pw string) error {
|
||||
@@ -1306,6 +1322,11 @@ func main() {
|
||||
case "offbox_repo_reset":
|
||||
notifier.PushEvent("offbox_repo_reset", "info",
|
||||
"A távoli mentési tároló visszaállítva: a régi előzmény félretéve (nem törölve), és egy üres, új tároló jött létre a mostani kulccsal.", map[string]string{"renamed_to": renamedTo})
|
||||
case "offbox_abandon_deferred":
|
||||
// v0.289.0 (decision 69): the box's off-site key cannot delete, so the customer-chosen
|
||||
// deletion of the set-aside history is the operator's (R-823). Operator-only on the hub.
|
||||
notifier.PushEvent("offbox_abandon_deferred", "warning",
|
||||
"A félretett régi távoli mentések törlése esedékes, de a doboz távoli kulcsa csak hozzáadni tud (69. döntés): semmi nem törlődött. A félretett másolatot az üzemeltető távolítja el.", map[string]string{"set_aside_path": renamedTo})
|
||||
case "offbox_abandon_completed":
|
||||
// R-241: the ONLY event in the product that reports a customer's off-site history
|
||||
// being deleted. It is fired after the deletion, not before — the operator wants to
|
||||
|
||||
@@ -113,6 +113,10 @@ type Manager struct {
|
||||
// offboxSSH (v0.142.0) is the raw-ssh exec seam for the orphaned-repo move-aside (restic has no
|
||||
// rename); tests inject a fake. Nil → the real ssh invocation (defaultOffboxSSH).
|
||||
offboxSSH func(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error)
|
||||
// offsiteMoveAside (v0.289.0, decision 69) asks the hub to set the repository aside.
|
||||
offsiteMoveAside func(ctx context.Context) (string, error)
|
||||
// offsiteWindow (v0.289.0, decision 68) asks the hub for a clean-up window. nil → no box retention.
|
||||
offsiteWindow OffsiteWindowClient
|
||||
|
||||
// offboxSizer (3a) — the mandatory-set byte estimator for the pre-push enlargement gate, overridable
|
||||
// in tests so the gate is unit-testable without a real du. Nil → the real dirSizeBytes (du -sb).
|
||||
|
||||
@@ -69,6 +69,11 @@ func (m *Manager) SetOffboxOrphanEvent(fn func(eventType, renamedTo string)) {
|
||||
}
|
||||
|
||||
// SetOffboxSSH overrides the raw-ssh exec used for the orphaned-repo move-aside (tests).
|
||||
// SetOffsiteMoveAside wires the hub's move-aside (decision 69): the orphan reset asks the hub.
|
||||
func (m *Manager) SetOffsiteMoveAside(fn func(ctx context.Context) (string, error)) {
|
||||
m.offsiteMoveAside = fn
|
||||
}
|
||||
|
||||
func (m *Manager) SetOffboxSSH(fn func(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error)) {
|
||||
m.offboxSSH = fn
|
||||
}
|
||||
@@ -208,7 +213,8 @@ func ClassifyOffsiteFailure(err error) OffsiteFailureClass {
|
||||
strings.Contains(s, "no route to host"), strings.Contains(s, "i/o timeout"),
|
||||
strings.Contains(s, "timed out"), strings.Contains(s, "permission denied"),
|
||||
strings.Contains(s, "host key"), strings.Contains(s, "handshake"),
|
||||
strings.Contains(s, "could not resolve"), strings.Contains(s, "network is unreachable"):
|
||||
strings.Contains(s, "could not resolve"), strings.Contains(s, "network is unreachable"),
|
||||
strings.Contains(s, "error talking http to rclone"): // measured: the rclone: backend's dead-ssh shape
|
||||
return OffsiteFailTransport
|
||||
default:
|
||||
return OffsiteFailUnknown
|
||||
@@ -311,27 +317,44 @@ func (m *Manager) resetOrphanedRepo(ctx context.Context, base, env []string, rea
|
||||
if t == nil {
|
||||
return fmt.Errorf("no offsite target configured")
|
||||
}
|
||||
port := t.Port
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
// Choose a move-aside name that never overwrites an earlier orphaned copy (edge rule: -2, -3).
|
||||
date := time.Now().UTC().Format("20060102")
|
||||
base1 := t.RepoPath + ".orphaned-" + date
|
||||
newPath := base1
|
||||
for i := 2; i <= 20; i++ {
|
||||
// `test -e <p>` returns non-zero (exit 1) when absent — that is the name we want. A transport
|
||||
// error also lands here; we then just try the mv and let it fail loudly rather than loop.
|
||||
out, err := m.sshRunner()(ctx, t.Host, t.User, port, m.offboxKeyPath(), m.offboxKnownHosts(), "test -e "+shellQuote(newPath))
|
||||
if err != nil && !strings.Contains(strings.ToLower(string(out)), "denied") {
|
||||
break // absent (test -e exit 1) → free name
|
||||
var newPath string
|
||||
if t.Pinned() {
|
||||
// v0.289.0 (decision 69): the HUB sets the repository aside — the box's append-only key reaches only
|
||||
// the pinned rclone server and cannot rename a directory. The hub renames, never deletes, and picks a
|
||||
// name that never overwrites an earlier set-aside copy (`<repo>.orphaned-<date>[-n]`).
|
||||
if m.offsiteMoveAside == nil {
|
||||
return fmt.Errorf("offbox move-aside: the hub's key registrar is not configured on this box")
|
||||
}
|
||||
m.logger.Printf("[WARN] [offbox] resetting orphaned repo (%s): asking the hub to set %s aside, then re-init", reason, t.RepoPath)
|
||||
np, err := m.offsiteMoveAside(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("offbox move-aside failed: %w", err)
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] the hub set the orphaned repo aside: %s -> %s (nothing deleted)", t.RepoPath, newPath)
|
||||
newPath = np
|
||||
} else {
|
||||
port := t.Port
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
// Choose a move-aside name that never overwrites an earlier orphaned copy (edge rule: -2, -3).
|
||||
date := time.Now().UTC().Format("20060102")
|
||||
base1 := t.RepoPath + ".orphaned-" + date
|
||||
newPath = base1
|
||||
for i := 2; i <= 20; i++ {
|
||||
// `test -e <p>` returns non-zero (exit 1) when absent — that is the name we want. A transport
|
||||
// error also lands here; we then just try the mv and let it fail loudly rather than loop.
|
||||
out, err := m.sshRunner()(ctx, t.Host, t.User, port, m.offboxKeyPath(), m.offboxKnownHosts(), "test -e "+shellQuote(newPath))
|
||||
if err != nil && !strings.Contains(strings.ToLower(string(out)), "denied") {
|
||||
break // absent (test -e exit 1) → free name
|
||||
}
|
||||
newPath = fmt.Sprintf("%s-%d", base1, i)
|
||||
}
|
||||
m.logger.Printf("[WARN] [offbox] resetting orphaned repo (%s): move-aside %s -> %s, then re-init", reason, t.RepoPath, newPath)
|
||||
if out, err := m.sshRunner()(ctx, t.Host, t.User, port, m.offboxKeyPath(), m.offboxKnownHosts(),
|
||||
fmt.Sprintf("mv %s %s", shellQuote(t.RepoPath), shellQuote(newPath))); err != nil {
|
||||
return fmt.Errorf("offbox move-aside failed: %w: %s", err, truncate(out))
|
||||
}
|
||||
newPath = fmt.Sprintf("%s-%d", base1, i)
|
||||
}
|
||||
m.logger.Printf("[WARN] [offbox] resetting orphaned repo (%s): move-aside %s -> %s, then re-init", reason, t.RepoPath, newPath)
|
||||
if out, err := m.sshRunner()(ctx, t.Host, t.User, port, m.offboxKeyPath(), m.offboxKnownHosts(),
|
||||
fmt.Sprintf("mv %s %s", shellQuote(t.RepoPath), shellQuote(newPath))); err != nil {
|
||||
return fmt.Errorf("offbox move-aside failed: %w: %s", err, truncate(out))
|
||||
}
|
||||
// Fresh init under the current passphrase.
|
||||
ictx, icancel := context.WithTimeout(ctx, offboxProbeTimeout)
|
||||
@@ -628,6 +651,10 @@ func (m *Manager) ApplyOffsiteTarget(ctx context.Context, tgt *settings.OffboxTa
|
||||
if tgt.EscrowState != "escrowed" {
|
||||
tgt.EscrowState = "pending"
|
||||
}
|
||||
// Decision 69 (v0.289.0): a HUB-provisioned target is reached only through the append-only key the
|
||||
// registrar pinned — the apply-bridge proved that before calling here. The household's own NAS
|
||||
// (offboxConfigHandler) never comes through this function and stays Transport "".
|
||||
tgt.Transport = settings.TransportRclonePinned
|
||||
if err := m.settings.SetOffboxTarget(tgt); err != nil {
|
||||
return fmt.Errorf("apply offsite target: %w", err)
|
||||
}
|
||||
@@ -733,13 +760,32 @@ func (m *Manager) OffboxEscrowState() string {
|
||||
return t.EscrowState
|
||||
}
|
||||
|
||||
// offboxBaseArgs builds the restic global args (repo + sftp.args carrying the ConnectTimeout, key, pinned
|
||||
// known_hosts, port) and the env (RESTIC_PASSWORD_FILE). The ConnectTimeout is MANDATORY (fail-fast).
|
||||
// offboxBaseArgs builds the restic global args (the rclone: repo + rclone.program carrying the
|
||||
// ConnectTimeout, key, pinned known_hosts, port) and the env (RESTIC_PASSWORD_FILE).
|
||||
func (m *Manager) offboxBaseArgs(t *settings.OffboxTarget) ([]string, []string) {
|
||||
port := t.Port
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
// v0.289.0 (decision 69, R-820): the `rclone:` backend over the box's APPEND-ONLY key. The hub's key
|
||||
// registrar pins this key in the sub-account's authorized_keys to
|
||||
// `command="rclone serve restic --stdio --append-only <repo>",restrict`, so whatever we ask for, the
|
||||
// provider runs that server: backups, restores, `check` and lock removal work; every delete is
|
||||
// refused (403) — measured on the provider 2026-10-03 (audits/offsite-append-only-2026-10-03/,
|
||||
// audits/offsite-lock-build-2026-10-03/partA: an sftp-written repo reads, extends, restores and
|
||||
// `check --read-data`s through it). restic 0.14.0 is enough; rclone is NOT needed in the image (it
|
||||
// runs on the provider). The program is split on whitespace by restic; the trailing "rclone" is the
|
||||
// remote command an UNPINNED key would run (and the pin ignores). ConnectTimeout stays MANDATORY.
|
||||
if t.Pinned() {
|
||||
if t.Port == 0 {
|
||||
port = 23 // the provider accepts OpenSSH keys on 23 only (measured: 22 refuses them)
|
||||
}
|
||||
rcloneProg := fmt.Sprintf("ssh -p %d -oBatchMode=yes -oConnectTimeout=%d -oStrictHostKeyChecking=yes -oUserKnownHostsFile=%s -oIdentitiesOnly=yes -i %s %s@%s rclone",
|
||||
port, offboxConnectTimeoutSec, m.offboxKnownHosts(), m.offboxKeyPath(), t.User, t.Host)
|
||||
return []string{"-r", "rclone:" + t.RepoPath, "-o", "rclone.program=" + rcloneProg},
|
||||
[]string{"RESTIC_PASSWORD_FILE=" + m.offboxPwPath()}
|
||||
}
|
||||
// The household's own SFTP NAS target (Transport ""): unchanged since v0.142.0.
|
||||
// restic's sftp backend connects via the `-o sftp.command` SSH invocation (the portable form across
|
||||
// restic versions — `sftp.args` is not recognized by restic 0.14). The ConnectTimeout makes a dead NAS
|
||||
// fail in ~N s (the load-bearing spike Q8 knob); StrictHostKeyChecking + a pinned known_hosts avoid
|
||||
@@ -1412,17 +1458,12 @@ func (m *Manager) runOffboxInternal(ctx context.Context, apps, base, env []strin
|
||||
if firstErr != nil {
|
||||
return res, firstErr
|
||||
}
|
||||
// Retention: keep a sane window, prune the rest. SP-2: `--group-by host,tags` so an app's OLD
|
||||
// unit-only-shape snapshots share a group with its NEW enlarged shape (same <stack> tag) and age
|
||||
// out naturally — the default host,paths grouping would strand old-shape snapshots in their own
|
||||
// permanently-retained group. prune takes an EXCLUSIVE lock (the C2 stale-lock step) → resticStep.
|
||||
// Retention (decision 68, v0.289.0): the box's key is append-only, so it cannot prune on its own. It
|
||||
// asks the hub for a clean-up window; only inside one, and only past the fake-snapshot guard (R-822),
|
||||
// does it forget/prune. No window → nothing is deleted (the interim, option 3). SP-2's
|
||||
// `--group-by host,tags` policy is unchanged — it lives in offsiteWindowRetention now.
|
||||
m.offboxProgress.setPhase(OffboxPhaseRetention)
|
||||
fctx, cancel := context.WithTimeout(ctx, offboxBackupTimeout)
|
||||
defer cancel()
|
||||
if out, ferr := m.resticStep(fctx, env, base, "prune", "forget", "--group-by", "host,tags", "--keep-daily", "7", "--keep-weekly", "4", "--keep-monthly", "6", "--prune"); ferr != nil {
|
||||
// A prune failure is non-fatal to the backup itself (data is safe) — log, don't fail the run.
|
||||
m.logger.Printf("[WARN] [offbox] forget --prune failed (backups are safe): %v: %s", ferr, truncate(out))
|
||||
}
|
||||
m.offsiteWindowRetention(ctx, base, env, "after-run")
|
||||
return res, nil
|
||||
}
|
||||
|
||||
@@ -1778,24 +1819,16 @@ func OffboxQuotaPercent(t *settings.OffboxTarget) int {
|
||||
return pct
|
||||
}
|
||||
|
||||
// offboxPruneOnly runs ONLY the retention/prune step (the over-quota path: new backups are refused but
|
||||
// pruning must stay available — it is the only way back under the quota). Repo-ensure first so a fresh
|
||||
// target still fails loudly; errors are non-fatal (same as the regular run's prune).
|
||||
// offboxPruneOnly is the over-quota path: new backups are refused. Pruning is the only way back under the
|
||||
// quota — and since decision 68 it happens ONLY inside a hub-opened window, behind the R-822 guard. When
|
||||
// the hub grants none, nothing is deleted and the household's quota sentence stays (no delete attempt —
|
||||
// a refused delete costs ~48 s of restic retries per file and writes an index, measured).
|
||||
func (m *Manager) offboxPruneOnly(ctx context.Context, base, env []string) {
|
||||
if rerr := m.ensureOffboxRepo(ctx, base, env); rerr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] over-quota prune: repo unreachable: %v", rerr)
|
||||
return
|
||||
}
|
||||
fctx, cancel := context.WithTimeout(ctx, offboxBackupTimeout)
|
||||
defer cancel()
|
||||
// SP-2: `--group-by host,tags` (mirrors runOffboxInternal's forget) so old unit-only-shape snapshots
|
||||
// age out with the enlarged shape instead of stranding in a permanently-retained host,paths group.
|
||||
fargs := append(append([]string{}, base...), "forget", "--group-by", "host,tags", "--keep-daily", "7", "--keep-weekly", "4", "--keep-monthly", "6", "--prune")
|
||||
if out, ferr := m.runner()(fctx, env, fargs...); ferr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] over-quota prune failed: %v: %s", ferr, truncate(out))
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [offbox] over-quota: prune executed (new backups refused until under quota)")
|
||||
}
|
||||
m.offsiteWindowRetention(ctx, base, env, "over-quota")
|
||||
}
|
||||
|
||||
// offboxRecordStats reads the snapshot count (best-effort) for the UI; also fills repo size when stats works.
|
||||
|
||||
@@ -192,6 +192,19 @@ func (m *Manager) AbandonSweep(ctx context.Context) (bool, error) {
|
||||
m.CancelAbandon("no set-aside path recorded")
|
||||
return false, fmt.Errorf("abandonment due with no recorded path")
|
||||
}
|
||||
if t.Pinned() {
|
||||
// Decision 69 (v0.289.0): the box's off-site key is append-only and cannot delete — by design,
|
||||
// so that a broken-into box cannot erase history. The set-aside copy STAYS; the operator removes
|
||||
// it (R-823). The schedule is closed so the sweep stops; nothing was deleted.
|
||||
if uerr := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) { o.AbandonAt = "" }); uerr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] abandonment: could not close the schedule: %v", uerr)
|
||||
}
|
||||
m.logger.Printf("[WARN] [offbox] abandonment DUE for %s, but the off-site key is append-only (decision 69) — NOTHING deleted; the set-aside copy stays until the operator removes it", t.AbandonRepoPath)
|
||||
if m.offboxOrphanEvent != nil {
|
||||
m.offboxOrphanEvent("offbox_abandon_deferred", t.AbandonRepoPath)
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
port := t.Port
|
||||
if port == 0 {
|
||||
port = 22
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ── Decision 68 (v0.289.0): the box prunes its own repository ONLY inside a hub-opened window ──────────
|
||||
//
|
||||
// The hub-provisioned tier's key is append-only (decision 69): every delete is refused by the provider.
|
||||
// To keep retention working, the box ASKS the hub for a clean-up window after its run. The hub grants
|
||||
// one at most weekly (or on an operator's one-shot grant) by PREPENDING a deleting line for the box's own
|
||||
// key — OpenSSH uses the first matching line, measured on the provider — and closes it when the box
|
||||
// reports, or after 20 minutes on its own.
|
||||
//
|
||||
// THE FAKE-SNAPSHOT GUARD (R-822) runs BEFORE any forget, inside the window. Measured in the lab: 13
|
||||
// future-dated empty snapshots added through an add-only key make the box's own policy select EVERY real
|
||||
// snapshot for removal. So, refuse when:
|
||||
// - any snapshot is dated in the future (beyond offsiteGuardSkew), or after the hub's newest-allowed
|
||||
// bound (the moment the window opened, plus the same skew);
|
||||
// - the plan would remove a snapshot younger than offsiteGuardMinAge — the honest policy
|
||||
// (--keep-daily 7) never removes the newest snapshot of any of the last 7 days, while a poisoning
|
||||
// shape does exactly that.
|
||||
// And bound the damage of anything the guard cannot see: at most MaxRemove (the hub's number) snapshots
|
||||
// per window, OLDEST first. DISAGREEMENT RECORDED (R-96 rule 4): the brief asked to ABORT when the plan
|
||||
// exceeds a week's removal; the first window after the interim legitimately exceeds it (weeks of
|
||||
// unpruned history), so an abort would never prune at all. Capping and taking the oldest gives the
|
||||
// same bound on loss per window and still converges.
|
||||
//
|
||||
// The NAS tier (Transport "") is unchanged: the household's own disk, pruned by the box as before.
|
||||
//
|
||||
// Pinned by TestOffsiteGuard_* (offbox_window_test.go), including the lab's 13-fake shape.
|
||||
|
||||
const (
|
||||
offsiteGuardSkew = time.Hour
|
||||
offsiteGuardMinAge = 8 * 24 * time.Hour
|
||||
)
|
||||
|
||||
// OffsiteWindow is the hub's answer to "may I prune now?".
|
||||
type OffsiteWindow struct {
|
||||
Granted bool
|
||||
ID int64
|
||||
NewestAllowed time.Time
|
||||
MaxRemove int
|
||||
Reason string // why not granted (logged)
|
||||
}
|
||||
|
||||
// OffsiteWindowResult is what the box reports when it is done (the hub closes the window on it).
|
||||
type OffsiteWindowResult struct {
|
||||
ID int64 `json:"window_id"`
|
||||
CountBefore int `json:"count_before"`
|
||||
CountAfter int `json:"count_after"`
|
||||
Removed int `json:"removed"`
|
||||
Outcome string `json:"outcome"` // pruned | nothing | guard-refused | error
|
||||
Reason string `json:"reason,omitempty"`
|
||||
}
|
||||
|
||||
// OffsiteWindowClient is the hub side (offsiteapply.HubWindowClient in production).
|
||||
type OffsiteWindowClient interface {
|
||||
Open(ctx context.Context, countBefore int) (OffsiteWindow, error)
|
||||
Close(ctx context.Context, r OffsiteWindowResult) error
|
||||
}
|
||||
|
||||
// SetOffsiteWindowClient wires the hub's window (decision 68). nil → no box-side retention on the pinned tier.
|
||||
func (m *Manager) SetOffsiteWindowClient(c OffsiteWindowClient) { m.offsiteWindow = c }
|
||||
|
||||
// retentionPolicy is the ruled policy, unchanged since SP-2 (`--group-by host,tags`).
|
||||
var retentionPolicy = []string{"--group-by", "host,tags", "--keep-daily", "7", "--keep-weekly", "4", "--keep-monthly", "6"}
|
||||
|
||||
type guardSnap struct {
|
||||
ID string `json:"id"`
|
||||
ShortID string `json:"short_id"`
|
||||
Time time.Time `json:"time"`
|
||||
}
|
||||
|
||||
// offsiteGuard is the PURE decision: from all snapshots and the policy's remove-plan, either the ids to
|
||||
// remove (oldest first, at most maxRemove) or a refusal reason.
|
||||
func offsiteGuard(all, plan []guardSnap, now, newestAllowed time.Time, maxRemove int) ([]string, string) {
|
||||
for _, s := range all {
|
||||
if s.Time.After(now.Add(offsiteGuardSkew)) {
|
||||
return nil, fmt.Sprintf("snapshot %s is dated in the future (%s)", s.ShortID, s.Time.UTC().Format(time.RFC3339))
|
||||
}
|
||||
if !newestAllowed.IsZero() && s.Time.After(newestAllowed.Add(offsiteGuardSkew)) {
|
||||
return nil, fmt.Sprintf("snapshot %s (%s) is newer than the hub allows (%s)", s.ShortID, s.Time.UTC().Format(time.RFC3339), newestAllowed.UTC().Format(time.RFC3339))
|
||||
}
|
||||
}
|
||||
for _, s := range plan {
|
||||
if now.Sub(s.Time) < offsiteGuardMinAge {
|
||||
return nil, fmt.Sprintf("the policy would remove snapshot %s from %s — younger than %d days, which honest retention never does",
|
||||
s.ShortID, s.Time.UTC().Format(time.RFC3339), int(offsiteGuardMinAge.Hours()/24))
|
||||
}
|
||||
}
|
||||
sorted := append([]guardSnap{}, plan...)
|
||||
sort.Slice(sorted, func(i, j int) bool { return sorted[i].Time.Before(sorted[j].Time) })
|
||||
if maxRemove >= 0 && len(sorted) > maxRemove {
|
||||
sorted = sorted[:maxRemove]
|
||||
}
|
||||
ids := make([]string, 0, len(sorted))
|
||||
for _, s := range sorted {
|
||||
ids = append(ids, s.ID)
|
||||
}
|
||||
return ids, ""
|
||||
}
|
||||
|
||||
func (m *Manager) listGuardSnaps(ctx context.Context, base, env []string) ([]guardSnap, error) {
|
||||
sctx, cancel := context.WithTimeout(ctx, offboxProbeTimeout)
|
||||
defer cancel()
|
||||
out, err := m.runner()(sctx, env, append(append([]string{}, base...), "snapshots", "--json")...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list snapshots: %w: %s", err, truncate(out))
|
||||
}
|
||||
var snaps []guardSnap
|
||||
if err := json.Unmarshal(out, &snaps); err != nil {
|
||||
return nil, fmt.Errorf("parse snapshots: %w", err)
|
||||
}
|
||||
return snaps, nil
|
||||
}
|
||||
|
||||
func (m *Manager) planRemovals(ctx context.Context, base, env []string) ([]guardSnap, error) {
|
||||
pctx, cancel := context.WithTimeout(ctx, offboxProbeTimeout)
|
||||
defer cancel()
|
||||
args := append(append(append([]string{}, base...), "forget"), retentionPolicy...)
|
||||
args = append(args, "--dry-run", "--json")
|
||||
out, err := m.runner()(pctx, env, args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("forget --dry-run: %w: %s", err, truncate(out))
|
||||
}
|
||||
// restic 0.14.0 prints the JSON array on stdout; the runner may combine stderr — take the array.
|
||||
js := string(out)
|
||||
if i := strings.Index(js, "["); i > 0 {
|
||||
js = js[i:]
|
||||
}
|
||||
var groups []struct {
|
||||
Remove []guardSnap `json:"remove"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(strings.TrimSpace(js)), &groups); err != nil {
|
||||
return nil, fmt.Errorf("parse forget plan: %w", err)
|
||||
}
|
||||
var plan []guardSnap
|
||||
for _, g := range groups {
|
||||
plan = append(plan, g.Remove...)
|
||||
}
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
// offsiteWindowRetention is the ONE retention step for both callers (after a run, over quota).
|
||||
func (m *Manager) offsiteWindowRetention(ctx context.Context, base, env []string, why string) {
|
||||
t := m.settings.GetOffboxTarget()
|
||||
if !t.Pinned() {
|
||||
// The household's own SFTP NAS: the box prunes as it always did (SP-2 policy).
|
||||
fctx, cancel := context.WithTimeout(ctx, offboxBackupTimeout)
|
||||
defer cancel()
|
||||
args := append(append([]string{"forget"}, retentionPolicy...), "--prune")
|
||||
if out, ferr := m.resticStep(fctx, env, base, "prune", args...); ferr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] forget --prune failed (%s; backups are safe): %v: %s", why, ferr, truncate(out))
|
||||
}
|
||||
return
|
||||
}
|
||||
if m.offsiteWindow == nil {
|
||||
m.logger.Printf("[INFO] [offbox] retention skipped (%s): the off-site key is append-only and no clean-up window client is wired — nothing deleted (decision 68)", why)
|
||||
return
|
||||
}
|
||||
snaps, err := m.listGuardSnaps(ctx, base, env)
|
||||
if err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] retention skipped (%s): %v", why, err)
|
||||
return
|
||||
}
|
||||
w, err := m.offsiteWindow.Open(ctx, len(snaps))
|
||||
if err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] retention skipped (%s): asking the hub for a window failed: %v", why, err)
|
||||
return
|
||||
}
|
||||
if !w.Granted {
|
||||
m.logger.Printf("[INFO] [offbox] retention skipped (%s): no clean-up window now (%s) — nothing deleted (decision 68)", why, w.Reason)
|
||||
return
|
||||
}
|
||||
start := time.Now()
|
||||
res := OffsiteWindowResult{ID: w.ID, CountBefore: len(snaps), CountAfter: len(snaps)}
|
||||
defer func() {
|
||||
cctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
defer cancel()
|
||||
if cerr := m.offsiteWindow.Close(cctx, res); cerr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] closing clean-up window %d with the hub failed (the hub closes it by itself in 20 min): %v", w.ID, cerr)
|
||||
}
|
||||
}()
|
||||
plan, err := m.planRemovals(ctx, base, env)
|
||||
if err != nil {
|
||||
res.Outcome, res.Reason = "error", err.Error()
|
||||
m.logger.Printf("[WARN] [offbox] clean-up window %d: %v", w.ID, err)
|
||||
return
|
||||
}
|
||||
ids, refuse := offsiteGuard(snaps, plan, time.Now(), w.NewestAllowed, w.MaxRemove)
|
||||
if refuse != "" {
|
||||
res.Outcome, res.Reason = "guard-refused", refuse
|
||||
m.logger.Printf("[ERROR] [offbox] clean-up window %d: the fake-snapshot guard REFUSED — nothing deleted: %s (R-822)", w.ID, refuse)
|
||||
return
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
res.Outcome = "nothing"
|
||||
m.logger.Printf("[INFO] [offbox] clean-up window %d: the policy removes nothing", w.ID)
|
||||
return
|
||||
}
|
||||
fctx, cancel := context.WithTimeout(ctx, offboxBackupTimeout)
|
||||
defer cancel()
|
||||
args := append(append([]string{"forget"}, ids...), "--prune")
|
||||
if out, ferr := m.resticStep(fctx, env, base, "prune", args...); ferr != nil {
|
||||
res.Outcome, res.Reason = "error", truncate(out)
|
||||
m.logger.Printf("[WARN] [offbox] clean-up window %d: forget --prune failed (backups are safe): %v: %s", w.ID, ferr, truncate(out))
|
||||
} else {
|
||||
res.Outcome = "pruned"
|
||||
}
|
||||
if after, lerr := m.listGuardSnaps(ctx, base, env); lerr == nil {
|
||||
res.CountAfter = len(after)
|
||||
}
|
||||
res.Removed = res.CountBefore - res.CountAfter
|
||||
m.logger.Printf("[INFO] [offbox] clean-up window %d (%s): %d of %d planned snapshot(s) removed, %d -> %d, in %s",
|
||||
w.ID, why, res.Removed, len(plan), res.CountBefore, res.CountAfter, time.Since(start).Round(time.Second))
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// ── decision 68/69 (v0.289.0): the append-only tier ─────────────────────────────────────────────────
|
||||
|
||||
func pinTarget(t *testing.T, sett *settings.Settings) {
|
||||
t.Helper()
|
||||
if err := sett.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
|
||||
o.Transport = settings.TransportRclonePinned
|
||||
o.Port = 23
|
||||
o.Host, o.User, o.RepoPath = "u1-sub4.example", "u1-sub4", "/home/felhom-repo"
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func snapJSON(s []guardSnap) []byte { b, _ := json.Marshal(s); return b }
|
||||
|
||||
func planJSON(remove []guardSnap) []byte {
|
||||
b, _ := json.Marshal([]map[string]any{{"tags": []string{"app1"}, "remove": remove}})
|
||||
return b
|
||||
}
|
||||
|
||||
// windowRunner fakes restic for the retention step: snapshots, the dry-run plan, and records every
|
||||
// non-dry-run forget (the only call that deletes).
|
||||
type windowRunner struct {
|
||||
snaps []guardSnap
|
||||
plan []guardSnap
|
||||
forgets [][]string
|
||||
}
|
||||
|
||||
func (w *windowRunner) run(_ context.Context, _ []string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case contains(args, "forget") && contains(args, "--dry-run"):
|
||||
return planJSON(w.plan), nil
|
||||
case contains(args, "forget"):
|
||||
w.forgets = append(w.forgets, append([]string{}, args...))
|
||||
return nil, nil
|
||||
case contains(args, "snapshots"):
|
||||
return snapJSON(w.snaps), nil
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
type fakeWindow struct {
|
||||
grant OffsiteWindow
|
||||
opened int
|
||||
closed []OffsiteWindowResult
|
||||
}
|
||||
|
||||
func (f *fakeWindow) Open(context.Context, int) (OffsiteWindow, error) {
|
||||
f.opened++
|
||||
return f.grant, nil
|
||||
}
|
||||
func (f *fakeWindow) Close(_ context.Context, r OffsiteWindowResult) error {
|
||||
f.closed = append(f.closed, r)
|
||||
return nil
|
||||
}
|
||||
|
||||
func snap(id string, at time.Time) guardSnap {
|
||||
return guardSnap{ID: id + "-full", ShortID: id, Time: at}
|
||||
}
|
||||
|
||||
// The pinned transport: rclone over port 23, the pinned key; never sftp.
|
||||
func TestOffboxBaseArgs_PinnedUsesRcloneOnPort23(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
pinTarget(t, sett)
|
||||
sett.UpdateOffboxStatus(func(o *settings.OffboxTarget) { o.Port = 0 })
|
||||
args, _ := m.offboxBaseArgs(sett.GetOffboxTarget())
|
||||
j := strings.Join(args, " ")
|
||||
if !strings.Contains(j, "-r rclone:/home/felhom-repo") || !strings.Contains(j, "rclone.program=ssh -p 23 ") || strings.Contains(j, "sftp") {
|
||||
t.Fatalf("pinned args = %q", j)
|
||||
}
|
||||
if !argsContainTimeout(args) {
|
||||
t.Fatal("ConnectTimeout lost on the pinned transport")
|
||||
}
|
||||
// The household's NAS stays SFTP.
|
||||
m2, sett2 := newOffboxManager(t)
|
||||
if j2 := strings.Join(func() []string { a, _ := m2.offboxBaseArgs(sett2.GetOffboxTarget()); return a }(), " "); !strings.Contains(j2, "sftp:") {
|
||||
t.Fatalf("NAS args = %q", j2)
|
||||
}
|
||||
}
|
||||
|
||||
// THE CONSEQUENCE: on the pinned tier, a run with no window deletes NOTHING — no forget reaches restic.
|
||||
// RED-PROOF: the pre-v0.289.0 retention ran `forget … --prune` unconditionally after every run.
|
||||
func TestRetention_PinnedWithoutWindowDeletesNothing(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
pinTarget(t, sett)
|
||||
wr := &windowRunner{snaps: []guardSnap{snap("a", time.Now().Add(-40*24*time.Hour))}, plan: []guardSnap{snap("a", time.Now().Add(-40*24*time.Hour))}}
|
||||
m.SetOffboxRunner(wr.run)
|
||||
m.offsiteWindowRetention(context.Background(), nil, nil, "after-run") // no client wired
|
||||
fw := &fakeWindow{grant: OffsiteWindow{Granted: false, Reason: "not due"}}
|
||||
m.SetOffsiteWindowClient(fw)
|
||||
m.offsiteWindowRetention(context.Background(), nil, nil, "after-run")
|
||||
if len(wr.forgets) != 0 {
|
||||
t.Fatalf("a forget ran without a window: %v", wr.forgets)
|
||||
}
|
||||
if fw.opened != 1 || len(fw.closed) != 0 {
|
||||
t.Fatalf("opened=%d closed=%d", fw.opened, len(fw.closed))
|
||||
}
|
||||
}
|
||||
|
||||
// The full run path: RunOffboxBackup on a pinned target with no window never calls forget.
|
||||
func TestRunOffboxBackup_PinnedNeverForgetsWithoutWindow(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
pinTarget(t, sett)
|
||||
var forgets int
|
||||
m.SetOffboxRunner(func(ctx context.Context, env []string, args ...string) ([]byte, error) {
|
||||
if contains(args, "forget") {
|
||||
forgets++
|
||||
}
|
||||
rr := &recordingOffboxRunner{}
|
||||
return rr.run(ctx, env, args...)
|
||||
})
|
||||
_ = m.RunOffboxBackup(context.Background())
|
||||
if forgets != 0 {
|
||||
t.Fatalf("the pinned run reached forget %d time(s)", forgets)
|
||||
}
|
||||
}
|
||||
|
||||
// R-822, the lab's shape: 13 future-dated fakes. The guard REFUSES and nothing is deleted; the hub
|
||||
// is told why (window closed with outcome guard-refused).
|
||||
func TestOffsiteGuard_LabThirteenFutureFakes_Refused(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
pinTarget(t, sett)
|
||||
now := time.Now()
|
||||
real := []guardSnap{snap("r1", now.Add(-2*time.Hour)), snap("r2", now.Add(-26*time.Hour)), snap("r3", now.Add(-50*time.Hour))}
|
||||
all := append([]guardSnap{}, real...)
|
||||
for d := 1; d <= 7; d++ {
|
||||
all = append(all, snap(fmt.Sprintf("f%d", d), time.Date(2027, 1, d, 3, 0, 0, 0, time.UTC)))
|
||||
}
|
||||
for mth := 2; mth <= 7; mth++ {
|
||||
all = append(all, snap(fmt.Sprintf("m%d", mth), time.Date(2027, time.Month(mth), 15, 3, 0, 0, 0, time.UTC)))
|
||||
}
|
||||
wr := &windowRunner{snaps: all, plan: real} // the poisoned policy selects every REAL snapshot
|
||||
m.SetOffboxRunner(wr.run)
|
||||
fw := &fakeWindow{grant: OffsiteWindow{Granted: true, ID: 7, NewestAllowed: now, MaxRemove: 50}}
|
||||
m.SetOffsiteWindowClient(fw)
|
||||
m.offsiteWindowRetention(context.Background(), nil, nil, "after-run")
|
||||
if len(wr.forgets) != 0 {
|
||||
t.Fatalf("the guard let a poisoned plan delete: %v", wr.forgets)
|
||||
}
|
||||
if len(fw.closed) != 1 || fw.closed[0].Outcome != "guard-refused" || !strings.Contains(fw.closed[0].Reason, "future") {
|
||||
t.Fatalf("window close = %+v", fw.closed)
|
||||
}
|
||||
}
|
||||
|
||||
// Past-dated fakes that make the policy drop a RECENT real snapshot: refused too.
|
||||
func TestOffsiteGuard_RecentRemovalRefused(t *testing.T) {
|
||||
now := time.Now()
|
||||
all := []guardSnap{snap("old", now.Add(-60*24*time.Hour)), snap("recent", now.Add(-3*24*time.Hour))}
|
||||
_, why := offsiteGuard(all, []guardSnap{snap("recent", now.Add(-3*24*time.Hour))}, now, now, 50)
|
||||
if !strings.Contains(why, "younger than 8 days") {
|
||||
t.Fatalf("why = %q", why)
|
||||
}
|
||||
_, why = offsiteGuard(all, nil, now, now.Add(-10*24*time.Hour), 50)
|
||||
if !strings.Contains(why, "newer than the hub allows") {
|
||||
t.Fatalf("newest-allowed bound not enforced: %q", why)
|
||||
}
|
||||
}
|
||||
|
||||
// Honest retention inside a window: the oldest first, at most MaxRemove, and the forget names ids.
|
||||
func TestOffsiteGuard_HonestPlanPrunesOldestFirstCapped(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
pinTarget(t, sett)
|
||||
now := time.Now()
|
||||
plan := []guardSnap{snap("c", now.Add(-20*24*time.Hour)), snap("a", now.Add(-90*24*time.Hour)), snap("b", now.Add(-60*24*time.Hour))}
|
||||
all := append([]guardSnap{snap("keep", now.Add(-time.Hour))}, plan...)
|
||||
wr := &windowRunner{snaps: all, plan: plan}
|
||||
m.SetOffboxRunner(wr.run)
|
||||
fw := &fakeWindow{grant: OffsiteWindow{Granted: true, ID: 9, NewestAllowed: now, MaxRemove: 2}}
|
||||
m.SetOffsiteWindowClient(fw)
|
||||
m.offsiteWindowRetention(context.Background(), nil, nil, "after-run")
|
||||
if len(wr.forgets) != 1 {
|
||||
t.Fatalf("forgets = %v", wr.forgets)
|
||||
}
|
||||
got := strings.Join(wr.forgets[0], " ")
|
||||
if !strings.Contains(got, "forget a-full b-full --prune") || strings.Contains(got, "c-full") {
|
||||
t.Fatalf("forget = %q (want the two OLDEST, capped)", got)
|
||||
}
|
||||
if len(fw.closed) != 1 || fw.closed[0].Outcome != "pruned" || fw.closed[0].ID != 9 {
|
||||
t.Fatalf("close = %+v", fw.closed)
|
||||
}
|
||||
}
|
||||
|
||||
// The orphan reset on the pinned tier asks the HUB; no ssh `mv` from the box.
|
||||
func TestResetOrphaned_PinnedAsksTheHub(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
pinTarget(t, sett)
|
||||
m.SetOffboxSSH(func(context.Context, string, string, int, string, string, string) ([]byte, error) {
|
||||
t.Fatal("the box issued a raw ssh command on the pinned tier")
|
||||
return nil, nil
|
||||
})
|
||||
called := 0
|
||||
m.SetOffsiteMoveAside(func(context.Context) (string, error) { called++; return "/home/felhom-repo.orphaned-20261003", nil })
|
||||
m.SetOffboxRunner(func(context.Context, []string, ...string) ([]byte, error) { return nil, nil })
|
||||
if err := m.resetOrphanedRepo(context.Background(), nil, nil, "test"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if called != 1 || sett.GetOffboxTarget().OrphanedRenamedTo != "/home/felhom-repo.orphaned-20261003" {
|
||||
t.Fatalf("hub move-aside called=%d recorded=%q", called, sett.GetOffboxTarget().OrphanedRenamedTo)
|
||||
}
|
||||
}
|
||||
|
||||
// Abandonment on the pinned tier: due → nothing deleted, the operator is told, the sweep goes quiet.
|
||||
func TestAbandon_PinnedDefersToOperator(t *testing.T) {
|
||||
m, sett := newOffboxManager(t)
|
||||
pinTarget(t, sett)
|
||||
sett.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
|
||||
o.AbandonRepoPath = "/home/felhom-repo.orphaned-20260901"
|
||||
o.AbandonStartedAt = time.Now().Add(-20 * 24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
o.AbandonAt = time.Now().Add(-time.Hour).UTC().Format(time.RFC3339)
|
||||
})
|
||||
m.SetOffboxSSH(func(context.Context, string, string, int, string, string, string) ([]byte, error) {
|
||||
t.Fatal("the box tried to delete on the pinned tier")
|
||||
return nil, nil
|
||||
})
|
||||
var evs []string
|
||||
m.SetOffboxOrphanEvent(func(e, _ string) { evs = append(evs, e) })
|
||||
deleted, err := m.AbandonSweep(context.Background())
|
||||
if deleted || err != nil || len(evs) != 1 || evs[0] != "offbox_abandon_deferred" {
|
||||
t.Fatalf("deleted=%v err=%v events=%v", deleted, err, evs)
|
||||
}
|
||||
if again, _ := m.AbandonSweep(context.Background()); again {
|
||||
t.Fatal("second sweep deleted")
|
||||
}
|
||||
}
|
||||
@@ -1,9 +1,11 @@
|
||||
// Package offsiteapply is the controller-side apply-bridge (SLICE 2): it turns the hub-served offsite
|
||||
// descriptor + the one-time password into a working key-only offbox target. On config apply it consumes the
|
||||
// one-time password, VERIFIES the box host key against the hub-captured fingerprint (no blind TOFU), pins it,
|
||||
// installs the controller's own key, and configures the offbox target → EscrowState="pending" (the fork-4
|
||||
// enable path). Idempotent (a descriptor hash marker prevents re-consuming a spent password) and fail-safe
|
||||
// (any step fails → nothing persisted, retried next cycle; never a half-configured offbox).
|
||||
// Package offsiteapply is the controller-side apply-bridge: it turns the hub-served offsite descriptor into
|
||||
// a working off-site target reached through an APPEND-ONLY key. It VERIFIES the Storage Box host key against
|
||||
// the hub-captured fingerprint (no blind TOFU), sends the box's PUBLIC key to the hub's key registrar, which
|
||||
// writes it into the sub-account pinned to `rclone serve restic --stdio --append-only <repo>` (decision 69,
|
||||
// controller v0.289.0), proves the key reaches the pinned server, and configures the offbox target →
|
||||
// EscrowState="pending" (the fork-4 enable path). The box NEVER receives the sub-account password any more —
|
||||
// that password can rewrite authorized_keys and remove the pin (R-820). Idempotent and fail-safe (any step
|
||||
// fails → nothing persisted, retried next cycle).
|
||||
package offsiteapply
|
||||
|
||||
import (
|
||||
@@ -22,9 +24,11 @@ import (
|
||||
|
||||
// The apply-bridge seams (tests inject fakes — no live SSH / hub calls in unit tests).
|
||||
type (
|
||||
// PasswordConsumer fetches the one-time transient password from the hub (single-use).
|
||||
PasswordConsumer interface {
|
||||
Consume(ctx context.Context) (string, error)
|
||||
// KeyRegistrar is the hub's key registrar (decision 69): install a PUBLIC key pinned append-only,
|
||||
// then confirm the one the box uses (the hub drops every other line).
|
||||
KeyRegistrar interface {
|
||||
Register(ctx context.Context, pub string) (fingerprint string, err error)
|
||||
Confirm(ctx context.Context, fingerprint string) error
|
||||
}
|
||||
// HostKeyScanner returns the box's host-key fingerprint (SHA256:…) + the known_hosts line to pin.
|
||||
HostKeyScanner interface {
|
||||
@@ -34,25 +38,18 @@ type (
|
||||
KeyGenerator interface {
|
||||
Generate() (privPEM, pubAuthorized string, err error)
|
||||
}
|
||||
// KeyInstaller installs the pub line on the box using the one-time password, then verifies passwordless
|
||||
// key auth with the private key. It MUST pin the VERIFIED knownHosts line (from the scan) on the
|
||||
// connection — never blind-TOFU — so a MITM cannot swap the key between the scan and the install.
|
||||
KeyInstaller interface {
|
||||
Install(ctx context.Context, host, user string, port int, password, privPEM, pubAuthorized, knownHosts string) error
|
||||
}
|
||||
// OffboxEnabler configures the offbox target (key + known_hosts + target + soft quota) and goes
|
||||
// EscrowState="pending" (the fork-4 enable path). quotaGB=0 = no soft limit (dedicated boxes).
|
||||
OffboxEnabler interface {
|
||||
ConfigureOffbox(ctx context.Context, host, user string, port int, repoPath, privPEM, knownHosts string, quotaGB int) error
|
||||
}
|
||||
// KeyAuthProber checks whether an ALREADY-INSTALLED key authenticates to the target (pinned to the
|
||||
// freshly-scanned knownHosts). ok=true returns that key's PEM so the descriptor change is applied by
|
||||
// re-pinning + reconfiguring WITHOUT consuming a one-time password (key-auth-first — kills the
|
||||
// stale-descriptor consume-404 loop and shrinks the re-issue blast radius to genuinely-fresh guests).
|
||||
// ok=false (no key / auth refused) → the caller falls through to the full consume+install path.
|
||||
KeyAuthProber interface {
|
||||
Probe(ctx context.Context, host, user string, port int, knownHosts string) (privPEM string, ok bool)
|
||||
// PinnedKeyProber proves a key reaches the PINNED append-only server (not merely that it
|
||||
// authenticates — an unpinned key authenticates too, and can delete).
|
||||
PinnedKeyProber interface {
|
||||
Probe(ctx context.Context, host, user string, port int, knownHosts, privPEM string) bool
|
||||
}
|
||||
// ExistingKey returns the box's installed off-site private key, "" when none.
|
||||
ExistingKey func() string
|
||||
// SettleProvider reports the managed-update settle state so the bridge can DEFER consuming the
|
||||
// one-time password until any imminent managed floor-update has converged (R-71a — the structural
|
||||
// fix for the F10 day-0 race). The failure it prevents: a fresh box boots below the operator floor,
|
||||
@@ -106,13 +103,13 @@ func belowFloor(version, floor string) bool {
|
||||
// Bridge reconciles the offsite descriptor into a configured offbox target.
|
||||
type Bridge struct {
|
||||
Cfg *config.Config
|
||||
Consumer PasswordConsumer
|
||||
Registrar KeyRegistrar
|
||||
Scanner HostKeyScanner
|
||||
KeyGen KeyGenerator
|
||||
Installer KeyInstaller
|
||||
Enabler OffboxEnabler
|
||||
Prober KeyAuthProber // optional: key-auth-first (nil → always the full consume+install path)
|
||||
MarkerPath string // where the applied-descriptor-hash is persisted (e.g. <dataDir>/offbox/applied_marker)
|
||||
Prober PinnedKeyProber
|
||||
Existing ExistingKey // nil → no existing key (always a fresh keypair)
|
||||
MarkerPath string // where the applied-descriptor-hash is persisted (e.g. <dataDir>/offbox/applied_marker)
|
||||
Logger *log.Logger
|
||||
|
||||
// Settle gates the consume/install path behind managed-update convergence (R-71a). nil → no gate
|
||||
@@ -157,7 +154,9 @@ func (b *Bridge) logf(f string, a ...any) {
|
||||
// re-applies. QuotaGB is included (SLICE 4) so a hub-side quota raise reaches the target — on an
|
||||
// already-provisioned guest that re-apply is a cheap key-auth-first re-pin (no password consumed).
|
||||
func descriptorHash(o config.OffsiteConfig) string {
|
||||
s := fmt.Sprintf("%s|%s|%s|%d|%s|%s|%d", o.Type, o.Host, o.User, o.Port, o.RepoPath, o.HostFingerprint, o.QuotaGB)
|
||||
// "|pinned-v1" (v0.289.0): a box upgraded from the password era re-applies ONCE, which is what moves
|
||||
// its key to the pinned append-only line. Without it the old marker matches and nothing migrates.
|
||||
s := fmt.Sprintf("%s|%s|%s|%d|%s|%s|%d|pinned-v1", o.Type, o.Host, o.User, o.Port, o.RepoPath, o.HostFingerprint, o.QuotaGB)
|
||||
sum := sha256.Sum256([]byte(s))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -212,57 +211,64 @@ func (b *Bridge) Reconcile(ctx context.Context) error {
|
||||
return fmt.Errorf("offsite-apply: host-key MISMATCH for %s (got %s, want %s) — refusing to pin/install (possible MITM)", o.Host, scannedFP, o.HostFingerprint)
|
||||
}
|
||||
|
||||
// 1b) Key-auth-first: if an already-installed key still authenticates (pinned to the key we JUST
|
||||
// verified — the probe never weakens the identity check), the descriptor change is applied by
|
||||
// re-pinning + reconfiguring alone. NO one-time password is consumed — a stale/re-scanned descriptor
|
||||
// on an already-provisioned guest no longer loops on consume-404.
|
||||
if b.Prober != nil {
|
||||
if privPEM, ok := b.Prober.Probe(ctx, o.Host, o.User, port, knownHostsLine); ok {
|
||||
if err := b.Enabler.ConfigureOffbox(ctx, o.Host, o.User, port, o.RepoPath, privPEM, knownHostsLine, o.QuotaGB); err != nil {
|
||||
return fmt.Errorf("offsite-apply: reconfigure (key-auth-first): %w", err)
|
||||
// 2) The key: the one already installed (a running box, or one upgraded from the password era — the
|
||||
// SAME key is re-registered and comes back pinned), else a fresh pair.
|
||||
privPEM, pub, fresh := "", "", false
|
||||
if b.Existing != nil {
|
||||
if pem := b.Existing(); pem != "" {
|
||||
if p, perr := PublicKeyOf(pem); perr == nil {
|
||||
privPEM, pub = pem, p
|
||||
} else {
|
||||
b.logf("[WARN] [offsite-apply] the installed key does not parse (%v) — generating a fresh one", perr)
|
||||
}
|
||||
if err := b.writeMarker(h); err != nil {
|
||||
b.logf("[WARN] [offsite-apply] key-auth-first applied for %s but failed to persist the marker: %v", o.Host, err)
|
||||
return err
|
||||
}
|
||||
b.logf("[INFO] [offsite-apply] existing key still authenticates to %s@%s — re-pinned + reconfigured without consuming a password", o.User, o.Host)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// 2) Generate the controller keypair.
|
||||
privPEM, pubAuthorized, err := b.KeyGen.Generate()
|
||||
if privPEM == "" {
|
||||
if privPEM, pub, err = b.KeyGen.Generate(); err != nil {
|
||||
return fmt.Errorf("offsite-apply: keygen: %w", err)
|
||||
}
|
||||
fresh = true
|
||||
}
|
||||
fp, err := FingerprintOf(pub)
|
||||
if err != nil {
|
||||
return fmt.Errorf("offsite-apply: keygen: %w", err)
|
||||
return fmt.Errorf("offsite-apply: own key: %w", err)
|
||||
}
|
||||
|
||||
// 3) Consume the one-time password (single-use). After this the password is SPENT.
|
||||
password, err := b.Consumer.Consume(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("offsite-apply: consume one-time password: %w", err)
|
||||
// 3) Already pinned? (a descriptor change on a healthy box, or a restart) → no hub write needed.
|
||||
pinned := b.Prober.Probe(ctx, o.Host, o.User, port, knownHostsLine, privPEM)
|
||||
if !pinned {
|
||||
// 4) Ask the hub's registrar to install it pinned append-only. NO password crosses this call.
|
||||
hubFP, rerr := b.Registrar.Register(ctx, pub)
|
||||
if rerr != nil {
|
||||
return fmt.Errorf("offsite-apply: register key with the hub: %w", rerr)
|
||||
}
|
||||
if hubFP != fp {
|
||||
return fmt.Errorf("offsite-apply: hub installed key %s, ours is %s — refusing", hubFP, fp)
|
||||
}
|
||||
// 5) Prove the key now reaches the PINNED server (positive observable, not just auth).
|
||||
if !b.Prober.Probe(ctx, o.Host, o.User, port, knownHostsLine, privPEM) {
|
||||
return fmt.Errorf("offsite-apply: key %s registered but does not reach the pinned append-only server", fp)
|
||||
}
|
||||
b.logf("[INFO] [offsite-apply] the hub installed key %s append-only on %s@%s (fresh=%v)", fp, o.User, o.Host, fresh)
|
||||
}
|
||||
|
||||
// 4) Install the pubkey using the password (proven ssh-copy-id -s -f), verify key auth. Pin the
|
||||
// scanner-VERIFIED known_hosts line on the install/verify connections — never accept-new — so a MITM
|
||||
// cannot substitute a different key in the gap between the scan and the install.
|
||||
if err := b.Installer.Install(ctx, o.Host, o.User, port, password, privPEM, pubAuthorized, knownHostsLine); err != nil {
|
||||
// The password is now SPENT but install failed — a loud, distinct signal: the operator must reset
|
||||
// the box password on the hub and let the bridge retry. Do NOT mark applied.
|
||||
b.logf("[ERROR] [offsite-apply] key install FAILED after consuming the one-time password for %s@%s — the password is spent; reset it on the hub to retry: %v", o.User, o.Host, err)
|
||||
return fmt.Errorf("offsite-apply: install key (password spent — needs hub reset): %w", err)
|
||||
}
|
||||
|
||||
// 5) Configure the offbox target + go EscrowState="pending" (fork-4 enable path).
|
||||
// 6) Configure the offbox target (writes the key) + go EscrowState="pending" when new (fork-4).
|
||||
if err := b.Enabler.ConfigureOffbox(ctx, o.Host, o.User, port, o.RepoPath, privPEM, knownHostsLine, o.QuotaGB); err != nil {
|
||||
return fmt.Errorf("offsite-apply: configure offbox: %w", err)
|
||||
}
|
||||
|
||||
// 6) Persist the marker LAST — only a fully-applied descriptor is recorded (fail-safe).
|
||||
// 7) Confirm: the hub removes every other key line (rotation's last step). Best effort — a failed
|
||||
// confirm leaves an extra PINNED line, which cannot delete; the next apply retries it.
|
||||
if err := b.Registrar.Confirm(ctx, fp); err != nil {
|
||||
b.logf("[WARN] [offsite-apply] confirm key %s with the hub failed (an extra pinned line may remain; retried on the next apply): %v", fp, err)
|
||||
}
|
||||
|
||||
// 8) Persist the marker LAST — only a fully-applied descriptor is recorded (fail-safe).
|
||||
if err := b.writeMarker(h); err != nil {
|
||||
b.logf("[WARN] [offsite-apply] applied offsite for %s but failed to persist the marker (will re-apply next cycle — the password is spent, needs reset): %v", o.Host, err)
|
||||
b.logf("[WARN] [offsite-apply] applied offsite for %s but failed to persist the marker (re-applies next cycle; harmless — no password is involved): %v", o.Host, err)
|
||||
return err
|
||||
}
|
||||
b.logf("[INFO] [offsite-apply] offsite configured for %s@%s:%s (pending key escrow)", o.User, o.Host, o.RepoPath)
|
||||
b.logf("[INFO] [offsite-apply] offsite configured append-only for %s@%s:%s (key %s)", o.User, o.Host, o.RepoPath, fp)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -3,7 +3,10 @@ package offsiteapply
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -15,19 +18,36 @@ import (
|
||||
|
||||
// --- fakes ---
|
||||
|
||||
type fakeConsumer struct {
|
||||
pw string
|
||||
err error
|
||||
calls int
|
||||
panics bool
|
||||
// fakeRegistrar is the hub's key registrar. `calls` counts Register — the one call that writes to the
|
||||
// provider; it is the "did the bridge go to the hub" signal the settle-gate and retry tests count.
|
||||
type fakeRegistrar struct {
|
||||
calls int
|
||||
confirms []string
|
||||
gotPub string
|
||||
wrongFP bool
|
||||
err error
|
||||
confirmEr error
|
||||
panics bool
|
||||
}
|
||||
|
||||
func (f *fakeConsumer) Consume(_ context.Context) (string, error) {
|
||||
func (f *fakeRegistrar) Register(_ context.Context, pub string) (string, error) {
|
||||
if f.panics {
|
||||
panic("consume must NOT be called (idempotent no-op)")
|
||||
panic("register must NOT be called")
|
||||
}
|
||||
f.calls++
|
||||
return f.pw, f.err
|
||||
f.gotPub = pub
|
||||
if f.err != nil {
|
||||
return "", f.err
|
||||
}
|
||||
if f.wrongFP {
|
||||
return "SHA256:somebody-else", nil
|
||||
}
|
||||
return FingerprintOf(pub)
|
||||
}
|
||||
|
||||
func (f *fakeRegistrar) Confirm(_ context.Context, fp string) error {
|
||||
f.confirms = append(f.confirms, fp)
|
||||
return f.confirmEr
|
||||
}
|
||||
|
||||
type fakeScanner struct {
|
||||
@@ -39,40 +59,25 @@ func (f *fakeScanner) Scan(_ context.Context, _ string, _ int) (string, string,
|
||||
return f.fp, f.line, f.err
|
||||
}
|
||||
|
||||
type fakeKeyGen struct{ priv, pub string }
|
||||
// realKeyGen returns ONE real pair (the bridge parses keys now — fingerprints are real).
|
||||
type realKeyGen struct{ priv, pub string }
|
||||
|
||||
func (f *fakeKeyGen) Generate() (string, string, error) { return f.priv, f.pub, nil }
|
||||
|
||||
type fakeInstaller struct {
|
||||
err error
|
||||
calls int
|
||||
gotPub string
|
||||
gotPriv string
|
||||
gotPw string
|
||||
gotKnownHost string
|
||||
}
|
||||
|
||||
func (f *fakeInstaller) Install(_ context.Context, _, _ string, _ int, password, privPEM, pub, knownHosts string) error {
|
||||
f.calls++
|
||||
f.gotPub, f.gotPriv, f.gotPw, f.gotKnownHost = pub, privPEM, password, knownHosts
|
||||
return f.err
|
||||
}
|
||||
func (f *realKeyGen) Generate() (string, string, error) { return f.priv, f.pub, nil }
|
||||
|
||||
// fakeProber models the provider: a key reaches the pinned server once the registrar installed it
|
||||
// (pinnedAfterRegister) or from the start (pinnedInitially).
|
||||
type fakeProber struct {
|
||||
pem string
|
||||
ok bool
|
||||
panics bool
|
||||
calls int
|
||||
gotKH string
|
||||
reg *fakeRegistrar
|
||||
pinnedInitially bool
|
||||
pinnedAfterRegister bool
|
||||
calls int
|
||||
gotKH, gotPriv string
|
||||
}
|
||||
|
||||
func (f *fakeProber) Probe(_ context.Context, _, _ string, _ int, kh string) (string, bool) {
|
||||
if f.panics {
|
||||
panic("prober must NOT be called (verify must precede the probe)")
|
||||
}
|
||||
func (f *fakeProber) Probe(_ context.Context, _, _ string, _ int, kh, priv string) bool {
|
||||
f.calls++
|
||||
f.gotKH = kh
|
||||
return f.pem, f.ok
|
||||
f.gotKH, f.gotPriv = kh, priv
|
||||
return f.pinnedInitially || (f.pinnedAfterRegister && f.reg.calls > 0)
|
||||
}
|
||||
|
||||
type fakeEnabler struct {
|
||||
@@ -90,187 +95,200 @@ func (f *fakeEnabler) ConfigureOffbox(_ context.Context, host, _ string, _ int,
|
||||
return f.err
|
||||
}
|
||||
|
||||
func newBridge(t *testing.T, o config.OffsiteConfig) (*Bridge, *fakeConsumer, *fakeInstaller, *fakeEnabler, *bytes.Buffer) {
|
||||
var testPriv, testPub = func() (string, string) {
|
||||
p, q, err := ED25519KeyGen{}.Generate()
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return p, q
|
||||
}()
|
||||
|
||||
var otherPriv, otherPub = func() (string, string) {
|
||||
p, q, _ := ED25519KeyGen{}.Generate()
|
||||
return p, q
|
||||
}()
|
||||
|
||||
func newBridge(t *testing.T, o config.OffsiteConfig) (*Bridge, *fakeRegistrar, *fakeProber, *fakeEnabler, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
cfg := &config.Config{}
|
||||
cfg.Offsite = o
|
||||
cons := &fakeConsumer{pw: "the-transient-pw"}
|
||||
inst := &fakeInstaller{}
|
||||
reg := &fakeRegistrar{}
|
||||
pr := &fakeProber{reg: reg, pinnedAfterRegister: true}
|
||||
en := &fakeEnabler{}
|
||||
var logbuf bytes.Buffer
|
||||
b := &Bridge{
|
||||
Cfg: cfg,
|
||||
Consumer: cons,
|
||||
Registrar: reg,
|
||||
Scanner: &fakeScanner{fp: "SHA256:goodfp", line: "[h]:23 ssh-ed25519 AAAAKEY"},
|
||||
KeyGen: &fakeKeyGen{priv: "PRIVPEM", pub: "ssh-ed25519 AAAAPUB felhom"},
|
||||
Installer: inst,
|
||||
KeyGen: &realKeyGen{priv: testPriv, pub: testPub},
|
||||
Prober: pr,
|
||||
Enabler: en,
|
||||
MarkerPath: filepath.Join(t.TempDir(), "offbox", "applied_marker"),
|
||||
Logger: log.New(&logbuf, "", 0),
|
||||
}
|
||||
return b, cons, inst, en, &logbuf
|
||||
return b, reg, pr, en, &logbuf
|
||||
}
|
||||
|
||||
func goodOffsite() config.OffsiteConfig {
|
||||
return config.OffsiteConfig{Enabled: true, Type: "shared", Host: "h", User: "u", Port: 23, RepoPath: "/home/felhom-repo", QuotaGB: 50, HostFingerprint: "SHA256:goodfp"}
|
||||
}
|
||||
|
||||
// Scenario A — full apply: consume → verify-pin → install → configure offbox → marker persisted; pw not logged.
|
||||
func TestBridge_AppliesEndToEnd(t *testing.T) {
|
||||
b, cons, inst, en, logbuf := newBridge(t, goodOffsite())
|
||||
// A fresh box: verify host key → register the PUBLIC key → prove the pin → configure → confirm → marker.
|
||||
// What crosses to the hub is a public key and nothing else.
|
||||
func TestBridge_FreshBoxRegistersPublicKeyOnly(t *testing.T) {
|
||||
b, reg, pr, en, logbuf := newBridge(t, goodOffsite())
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("reconcile: %v", err)
|
||||
}
|
||||
if cons.calls != 1 {
|
||||
t.Fatalf("consume calls = %d, want 1", cons.calls)
|
||||
if reg.calls != 1 || reg.gotPub != testPub {
|
||||
t.Fatalf("register calls=%d pub=%q", reg.calls, reg.gotPub)
|
||||
}
|
||||
if inst.calls != 1 || inst.gotPw != "the-transient-pw" || inst.gotPub == "" {
|
||||
t.Fatalf("installer not called with pw+pub: %+v", inst)
|
||||
if strings.Contains(reg.gotPub, "PRIVATE") {
|
||||
t.Fatal("the private key was sent to the hub")
|
||||
}
|
||||
if inst.gotKnownHost != "[h]:23 ssh-ed25519 AAAAKEY" {
|
||||
t.Fatalf("installer must receive the scanner-verified known_hosts to pin (no TOFU), got %q", inst.gotKnownHost)
|
||||
if pr.calls != 2 || pr.gotKH != "[h]:23 ssh-ed25519 AAAAKEY" {
|
||||
t.Fatalf("probe calls=%d kh=%q (want before+after register, pinned to the scanned host key)", pr.calls, pr.gotKH)
|
||||
}
|
||||
if en.calls != 1 || en.gotHost != "h" || en.gotKnownHost != "[h]:23 ssh-ed25519 AAAAKEY" || en.gotPriv != "PRIVPEM" {
|
||||
t.Fatalf("enabler not called with the pinned known_hosts + key: %+v", en)
|
||||
fp, _ := FingerprintOf(testPub)
|
||||
if en.calls != 1 || en.gotPriv != testPriv || en.gotQuotaGB != 50 {
|
||||
t.Fatalf("enabler: %+v", en)
|
||||
}
|
||||
if en.gotQuotaGB != 50 {
|
||||
t.Fatalf("the bridge must map the descriptor's quota_gb into the target (SLICE 4), got %d", en.gotQuotaGB)
|
||||
if len(reg.confirms) != 1 || reg.confirms[0] != fp {
|
||||
t.Fatalf("confirm = %v, want [%s]", reg.confirms, fp)
|
||||
}
|
||||
if b.readMarker() != descriptorHash(b.Cfg.Offsite) {
|
||||
t.Fatal("marker not persisted after a successful apply")
|
||||
if _, err := os.Stat(b.MarkerPath); err != nil {
|
||||
t.Fatalf("marker not persisted: %v", err)
|
||||
}
|
||||
if strings.Contains(logbuf.String(), "the-transient-pw") {
|
||||
t.Fatal("the one-time password LEAKED into a log line")
|
||||
if !strings.Contains(logbuf.String(), "append-only") {
|
||||
t.Fatalf("log does not say append-only:\n%s", logbuf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// SLICE 4 — a quota-only descriptor change re-applies (the hash includes QuotaGB), and with a working
|
||||
// key it costs no password: key-auth-first re-pins + remaps the quota.
|
||||
func TestBridge_QuotaChangeReappliesWithoutConsume(t *testing.T) {
|
||||
b, cons, _, en, _ := newBridge(t, goodOffsite())
|
||||
cons.panics = true
|
||||
b.Prober = &fakeProber{pem: "EXISTINGPEM", ok: true}
|
||||
// marker for the OLD quota (25) already applied; the descriptor now says 50
|
||||
old := b.Cfg.Offsite
|
||||
old.QuotaGB = 25
|
||||
_ = os.MkdirAll(filepath.Dir(b.MarkerPath), 0o700)
|
||||
_ = os.WriteFile(b.MarkerPath, []byte(descriptorHash(old)), 0o600)
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("quota-change reconcile: %v", err)
|
||||
// THE MIGRATION: a box upgraded from the password era has an installed (unpinned) key and an applied
|
||||
// marker written under the OLD descriptor hash. The new hash ("|pinned-v1") re-applies once; the SAME
|
||||
// key is registered (no new keypair) and comes back pinned.
|
||||
func TestBridge_UpgradedBoxReRegistersSameKey(t *testing.T) {
|
||||
b, reg, _, en, _ := newBridge(t, goodOffsite())
|
||||
b.Existing = func() string { return otherPriv }
|
||||
b.KeyGen = nil // must not be needed
|
||||
o := goodOffsite()
|
||||
if err := os.MkdirAll(filepath.Dir(b.MarkerPath), 0o700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if en.calls != 1 || en.gotQuotaGB != 50 {
|
||||
t.Fatalf("a quota raise must re-apply and map the NEW quota (no consume): %+v", en)
|
||||
}
|
||||
}
|
||||
|
||||
// Key-auth-first (Scenario B) — the existing key still works: NO consume, NO install; re-verify + re-pin +
|
||||
// reconfigure with the EXISTING key, marker updated.
|
||||
func TestBridge_KeyAuthFirstSkipsConsume(t *testing.T) {
|
||||
b, cons, inst, en, _ := newBridge(t, goodOffsite())
|
||||
cons.panics = true // the whole point: a working key must NEVER consume the one-time password
|
||||
prober := &fakeProber{pem: "EXISTINGPEM", ok: true}
|
||||
b.Prober = prober
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("key-auth-first reconcile: %v", err)
|
||||
}
|
||||
if prober.calls != 1 || prober.gotKH != "[h]:23 ssh-ed25519 AAAAKEY" {
|
||||
t.Fatalf("probe must run once with the freshly-scanned pinned known_hosts: %+v", prober)
|
||||
}
|
||||
if inst.calls != 0 {
|
||||
t.Fatal("installer must NOT run when the existing key authenticates")
|
||||
}
|
||||
if en.calls != 1 || en.gotPriv != "EXISTINGPEM" || en.gotKnownHost != "[h]:23 ssh-ed25519 AAAAKEY" {
|
||||
t.Fatalf("enabler must reconfigure with the EXISTING key + fresh pin: %+v", en)
|
||||
}
|
||||
if b.readMarker() != descriptorHash(b.Cfg.Offsite) {
|
||||
t.Fatal("marker must be updated after a key-auth-first apply")
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario C — key-auth-first must NOT weaken the fresh path: probe fails → the full
|
||||
// verify→consume→install path runs unchanged (with the freshly GENERATED key).
|
||||
func TestBridge_FreshGuestFallsThroughToFullPath(t *testing.T) {
|
||||
b, cons, inst, en, _ := newBridge(t, goodOffsite())
|
||||
b.Prober = &fakeProber{ok: false} // fresh guest: no key / auth refused
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("fresh-guest reconcile: %v", err)
|
||||
}
|
||||
if cons.calls != 1 || inst.calls != 1 {
|
||||
t.Fatalf("fresh guest must consume+install exactly once: cons=%d inst=%d", cons.calls, inst.calls)
|
||||
}
|
||||
if en.calls != 1 || en.gotPriv != "PRIVPEM" {
|
||||
t.Fatalf("fresh guest must configure with the GENERATED key: %+v", en)
|
||||
}
|
||||
if b.readMarker() != descriptorHash(b.Cfg.Offsite) {
|
||||
t.Fatal("marker must be persisted after a full-path apply")
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario B — host-key mismatch → refuse: no consume, no install, no configure, no marker.
|
||||
func TestBridge_HostKeyMismatchRefuses(t *testing.T) {
|
||||
b, cons, inst, en, _ := newBridge(t, goodOffsite())
|
||||
b.Scanner = &fakeScanner{fp: "SHA256:ATTACKER", line: "[h]:23 ssh-ed25519 EVIL"}
|
||||
b.Prober = &fakeProber{panics: true} // the probe must NEVER run when the identity check failed
|
||||
err := b.Reconcile(context.Background())
|
||||
if err == nil || !strings.Contains(err.Error(), "MISMATCH") {
|
||||
t.Fatalf("mismatch must refuse, got %v", err)
|
||||
}
|
||||
if cons.calls != 0 || inst.calls != 0 || en.calls != 0 {
|
||||
t.Fatalf("nothing may proceed on a host-key mismatch: cons=%d inst=%d en=%d", cons.calls, inst.calls, en.calls)
|
||||
}
|
||||
if b.readMarker() != "" {
|
||||
t.Fatal("no marker may be written on a mismatch")
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario C — idempotent: marker already matches → no-op, consume is NOT called.
|
||||
func TestBridge_IdempotentNoReconsume(t *testing.T) {
|
||||
b, cons, inst, en, _ := newBridge(t, goodOffsite())
|
||||
cons.panics = true // Consume must not be called
|
||||
// pre-seed the marker with the current descriptor hash
|
||||
_ = os.MkdirAll(filepath.Dir(b.MarkerPath), 0o700)
|
||||
if err := os.WriteFile(b.MarkerPath, []byte(descriptorHash(b.Cfg.Offsite)), 0o600); err != nil {
|
||||
// The pre-v0.289.0 marker for this exact descriptor:
|
||||
if err := os.WriteFile(b.MarkerPath, []byte(legacyDescriptorHash(o)), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("idempotent reconcile must be a clean no-op, got %v", err)
|
||||
t.Fatalf("reconcile: %v", err)
|
||||
}
|
||||
if cons.calls != 0 || inst.calls != 0 || en.calls != 0 {
|
||||
t.Fatal("an already-applied descriptor must be a full no-op")
|
||||
if reg.calls != 1 || reg.gotPub != otherPub {
|
||||
t.Fatalf("the upgraded box must re-register its EXISTING key: calls=%d pub=%q", reg.calls, reg.gotPub)
|
||||
}
|
||||
if en.gotPriv != otherPriv {
|
||||
t.Fatal("the existing key was not kept")
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario D — install fails → fail-safe: marker NOT persisted, offbox NOT configured, loud log.
|
||||
func TestBridge_InstallFailIsFailSafe(t *testing.T) {
|
||||
b, cons, inst, en, logbuf := newBridge(t, goodOffsite())
|
||||
inst.err = errors.New("ssh-copy-id refused")
|
||||
err := b.Reconcile(context.Background())
|
||||
if err == nil {
|
||||
t.Fatal("install failure must error")
|
||||
// A restart / descriptor change on a box whose key is already pinned: no hub write at all.
|
||||
func TestBridge_AlreadyPinnedNeedsNoRegister(t *testing.T) {
|
||||
b, reg, pr, en, _ := newBridge(t, goodOffsite())
|
||||
b.Existing = func() string { return testPriv }
|
||||
pr.pinnedInitially = true
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("reconcile: %v", err)
|
||||
}
|
||||
if reg.calls != 0 || en.calls != 1 {
|
||||
t.Fatalf("register=%d enable=%d", reg.calls, en.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// The registrar answered but the key still does not reach the PINNED server (e.g. an unpinned line
|
||||
// won): refuse — never configure a key that can delete.
|
||||
func TestBridge_RegisteredButNotPinnedRefuses(t *testing.T) {
|
||||
b, reg, pr, en, _ := newBridge(t, goodOffsite())
|
||||
pr.pinnedAfterRegister = false
|
||||
if err := b.Reconcile(context.Background()); err == nil {
|
||||
t.Fatal("a key that does not reach the pinned server must refuse")
|
||||
}
|
||||
if reg.calls != 1 || en.calls != 0 {
|
||||
t.Fatalf("register=%d enable=%d", reg.calls, en.calls)
|
||||
}
|
||||
if _, err := os.Stat(b.MarkerPath); err == nil {
|
||||
t.Fatal("marker persisted after a refusal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBridge_HubInstalledADifferentKeyRefuses(t *testing.T) {
|
||||
b, reg, _, en, _ := newBridge(t, goodOffsite())
|
||||
reg.wrongFP = true
|
||||
if err := b.Reconcile(context.Background()); err == nil || en.calls != 0 {
|
||||
t.Fatalf("err=%v enable=%d", err, en.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// Host-key mismatch → refuse before anything touches the hub.
|
||||
func TestBridge_HostKeyMismatchRefuses(t *testing.T) {
|
||||
b, reg, pr, en, _ := newBridge(t, goodOffsite())
|
||||
b.Scanner = &fakeScanner{fp: "SHA256:EVIL", line: "x"}
|
||||
reg.panics = true
|
||||
if err := b.Reconcile(context.Background()); err == nil || !strings.Contains(err.Error(), "MISMATCH") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
if pr.calls != 0 || en.calls != 0 {
|
||||
t.Fatal("probe/configure ran after a host-key mismatch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBridge_IdempotentMarker(t *testing.T) {
|
||||
b, reg, _, _, _ := newBridge(t, goodOffsite())
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reg.panics = true
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("second reconcile: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBridge_RegisterFailIsFailSafe(t *testing.T) {
|
||||
b, reg, _, en, _ := newBridge(t, goodOffsite())
|
||||
reg.err = errors.New("hub down")
|
||||
if err := b.Reconcile(context.Background()); err == nil {
|
||||
t.Fatal("want error")
|
||||
}
|
||||
if en.calls != 0 {
|
||||
t.Fatal("offbox must NOT be configured when install fails")
|
||||
t.Fatal("configured after a failed register")
|
||||
}
|
||||
if b.readMarker() != "" {
|
||||
t.Fatal("marker must NOT be persisted on a failed apply (fail-safe)")
|
||||
}
|
||||
if cons.calls != 1 {
|
||||
t.Fatal("the password was consumed (spent) before install")
|
||||
}
|
||||
if !strings.Contains(logbuf.String(), "password is spent") {
|
||||
t.Fatal("a consumed-but-failed install must log the loud 'password is spent' signal")
|
||||
if _, err := os.Stat(b.MarkerPath); err == nil {
|
||||
t.Fatal("marker persisted after failure")
|
||||
}
|
||||
}
|
||||
|
||||
// A failed confirm leaves an extra PINNED line (cannot delete) — not a failure of the apply.
|
||||
func TestBridge_ConfirmFailureIsNotFatal(t *testing.T) {
|
||||
b, reg, _, en, logbuf := newBridge(t, goodOffsite())
|
||||
reg.confirmEr = errors.New("hub blip")
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatalf("reconcile: %v", err)
|
||||
}
|
||||
if en.calls != 1 || !strings.Contains(logbuf.String(), "confirm key") {
|
||||
t.Fatalf("enable=%d log=%s", en.calls, logbuf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// Disabled → no-op (no consume/install/configure).
|
||||
func TestBridge_DisabledNoOp(t *testing.T) {
|
||||
o := goodOffsite()
|
||||
o.Enabled = false
|
||||
b, cons, inst, en, _ := newBridge(t, o)
|
||||
if err := b.Reconcile(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cons.calls+inst.calls+en.calls != 0 {
|
||||
t.Fatal("disabled offsite must be a no-op")
|
||||
b, reg, _, en, _ := newBridge(t, o)
|
||||
reg.panics = true
|
||||
if err := b.Reconcile(context.Background()); err != nil || en.calls != 0 {
|
||||
t.Fatalf("err=%v enable=%d", err, en.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// legacyDescriptorHash is the pre-v0.289.0 marker key, byte for byte as v0.288.0 computed it.
|
||||
func legacyDescriptorHash(o config.OffsiteConfig) string {
|
||||
sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%s|%s|%d|%s|%s|%d", o.Type, o.Host, o.User, o.Port, o.RepoPath, o.HostFingerprint, o.QuotaGB)))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
@@ -17,16 +17,13 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"golang.org/x/crypto/ssh"
|
||||
"golang.org/x/crypto/ssh/knownhosts"
|
||||
)
|
||||
|
||||
// --- func adapters (convenient wiring in main.go) ---
|
||||
|
||||
type ConsumerFunc func(ctx context.Context) (string, error)
|
||||
|
||||
func (f ConsumerFunc) Consume(ctx context.Context) (string, error) { return f(ctx) }
|
||||
|
||||
type EnablerFunc func(ctx context.Context, host, user string, port int, repoPath, privPEM, knownHosts string, quotaGB int) error
|
||||
|
||||
func (f EnablerFunc) ConfigureOffbox(ctx context.Context, host, user string, port int, repoPath, privPEM, knownHosts string, quotaGB int) error {
|
||||
@@ -40,48 +37,125 @@ type SettleFunc func() (version, floor string, updateRunning, floorKnown bool)
|
||||
|
||||
func (f SettleFunc) SettleState() (string, string, bool, bool) { return f() }
|
||||
|
||||
// --- HTTPConsumer: POST the hub consume-password endpoint with the per-customer API key ---
|
||||
|
||||
type HTTPConsumer struct {
|
||||
// --- HubRegistrar: the box's half of the hub key registrar (decision 69, hub >= v0.127.0) ---
|
||||
//
|
||||
// The box sends ONLY its public key; the hub writes it into the Storage Box sub-account's
|
||||
// authorized_keys pinned to `rclone serve restic --stdio --append-only <repo>`. Until controller
|
||||
// v0.289.0 the box fetched the sub-account PASSWORD here (consume-password) — and that password can
|
||||
// rewrite authorized_keys, i.e. remove the pin (measured 2026-10-03, R-820). No response this client
|
||||
// reads can carry a password; TestHubRegistrar_NeverAsksForAPassword pins the paths it calls.
|
||||
type HubRegistrar struct {
|
||||
HubURL string
|
||||
CustomerID string
|
||||
APIKey string
|
||||
HC *http.Client
|
||||
}
|
||||
|
||||
func (c HTTPConsumer) Consume(ctx context.Context) (string, error) {
|
||||
func (c HubRegistrar) post(ctx context.Context, path string, body any) ([]byte, error) {
|
||||
if c.HubURL == "" || c.CustomerID == "" || c.APIKey == "" {
|
||||
return "", fmt.Errorf("offsite-apply: consume: hub url/customer/apikey not configured")
|
||||
return nil, fmt.Errorf("offsite-apply: hub url/customer/apikey not configured")
|
||||
}
|
||||
hc := c.HC
|
||||
if hc == nil {
|
||||
hc = &http.Client{Timeout: 20 * time.Second}
|
||||
hc = &http.Client{Timeout: 3 * time.Minute} // the hub does an SSH round trip to the provider
|
||||
}
|
||||
url := strings.TrimRight(c.HubURL, "/") + "/api/v1/offsite/consume-password/" + c.CustomerID
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, nil)
|
||||
var rd io.Reader
|
||||
if body != nil {
|
||||
b, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rd = strings.NewReader(string(b))
|
||||
}
|
||||
url := strings.TrimRight(c.HubURL, "/") + "/api/v1/offsite/" + path + "/" + c.CustomerID
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, rd)
|
||||
if err != nil {
|
||||
return "", err
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+c.APIKey)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := hc.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<16))
|
||||
if resp.StatusCode == http.StatusNotFound {
|
||||
return "", fmt.Errorf("no unconsumed offsite password (already consumed or none provisioned)")
|
||||
}
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return "", fmt.Errorf("consume: HTTP %d", resp.StatusCode)
|
||||
return nil, fmt.Errorf("hub %s: HTTP %d: %s", path, resp.StatusCode, truncate(raw))
|
||||
}
|
||||
var body struct {
|
||||
Password string `json:"password"`
|
||||
return raw, nil
|
||||
}
|
||||
|
||||
// Register asks the hub to install pub pinned append-only; returns the key's fingerprint as the hub saw it.
|
||||
func (c HubRegistrar) Register(ctx context.Context, pub string) (string, error) {
|
||||
raw, err := c.post(ctx, "register-key", map[string]string{"public_key": strings.TrimSpace(pub)})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := json.Unmarshal(raw, &body); err != nil || body.Password == "" {
|
||||
return "", fmt.Errorf("consume: malformed response")
|
||||
var r struct {
|
||||
Installed bool `json:"installed"`
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
}
|
||||
return body.Password, nil // NEVER logged
|
||||
if err := json.Unmarshal(raw, &r); err != nil || !r.Installed || r.Fingerprint == "" {
|
||||
return "", fmt.Errorf("hub register-key: malformed response")
|
||||
}
|
||||
return r.Fingerprint, nil
|
||||
}
|
||||
|
||||
// Confirm tells the hub the box now uses fp; the hub removes every other key line.
|
||||
func (c HubRegistrar) Confirm(ctx context.Context, fp string) error {
|
||||
_, err := c.post(ctx, "confirm-key", map[string]string{"fingerprint": fp})
|
||||
return err
|
||||
}
|
||||
|
||||
// MoveAside asks the hub to set the repository aside (never deletes) — the box's pinned key cannot.
|
||||
func (c HubRegistrar) MoveAside(ctx context.Context) (string, error) {
|
||||
raw, err := c.post(ctx, "move-aside", nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var r struct {
|
||||
MovedTo string `json:"moved_to"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &r); err != nil || r.MovedTo == "" {
|
||||
return "", fmt.Errorf("hub move-aside: malformed response")
|
||||
}
|
||||
return r.MovedTo, nil
|
||||
}
|
||||
|
||||
// --- HubWindowClient: the box's half of the clean-up window (decision 68) ---
|
||||
|
||||
type HubWindowClient struct{ Registrar HubRegistrar }
|
||||
|
||||
func (c HubWindowClient) Open(ctx context.Context, countBefore int) (backup.OffsiteWindow, error) {
|
||||
raw, err := c.Registrar.post(ctx, "window-open", map[string]int{"count_before": countBefore})
|
||||
if err != nil {
|
||||
return backup.OffsiteWindow{}, err
|
||||
}
|
||||
var r struct {
|
||||
Granted bool `json:"granted"`
|
||||
WindowID int64 `json:"window_id"`
|
||||
NewestAllowed string `json:"newest_allowed"`
|
||||
MaxRemove int `json:"max_remove"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &r); err != nil {
|
||||
return backup.OffsiteWindow{}, fmt.Errorf("hub window-open: malformed response")
|
||||
}
|
||||
w := backup.OffsiteWindow{Granted: r.Granted, ID: r.WindowID, MaxRemove: r.MaxRemove, Reason: r.Reason}
|
||||
if r.Granted {
|
||||
t, perr := time.Parse(time.RFC3339, r.NewestAllowed)
|
||||
if perr != nil {
|
||||
return backup.OffsiteWindow{}, fmt.Errorf("hub window-open: bad newest_allowed")
|
||||
}
|
||||
w.NewestAllowed = t
|
||||
}
|
||||
return w, nil
|
||||
}
|
||||
|
||||
func (c HubWindowClient) Close(ctx context.Context, res backup.OffsiteWindowResult) error {
|
||||
_, err := c.Registrar.post(ctx, "window-close", res)
|
||||
return err
|
||||
}
|
||||
|
||||
// --- KeyscanScanner: capture the box host key (x/crypto/ssh, no binary) → fingerprint + known_hosts line ---
|
||||
@@ -145,72 +219,21 @@ func (ED25519KeyGen) Generate() (string, string, error) {
|
||||
return privPEM, pubLine, nil
|
||||
}
|
||||
|
||||
// --- SSHCopyIDInstaller: install the pubkey via the proven `sshpass -e ssh-copy-id -p N -s -f`, verify ---
|
||||
|
||||
type SSHCopyIDInstaller struct{}
|
||||
|
||||
func (SSHCopyIDInstaller) Install(ctx context.Context, host, user string, port int, password, privPEM, pubAuthorized, knownHosts string) error {
|
||||
if strings.TrimSpace(knownHosts) == "" {
|
||||
return fmt.Errorf("ssh-copy-id: empty known_hosts — refusing to install without a pinned host key")
|
||||
}
|
||||
// ssh-copy-id -s (SFTP mode) mktemp's its batch file under ~/.ssh and dies LOCALLY if the directory
|
||||
// doesn't exist — the container image ships without /root/.ssh (live finding: the one-time password was
|
||||
// consumed, then the install failed before ever connecting).
|
||||
if home, err := os.UserHomeDir(); err == nil {
|
||||
if err := os.MkdirAll(filepath.Join(home, ".ssh"), 0o700); err != nil {
|
||||
return fmt.Errorf("ssh-copy-id: ensure ~/.ssh (needed by -s mode): %w", err)
|
||||
}
|
||||
}
|
||||
work, err := os.MkdirTemp("", "felhom-keyinstall-")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(work)
|
||||
pubPath := filepath.Join(work, "id.pub")
|
||||
privPath := filepath.Join(work, "id")
|
||||
khPath := filepath.Join(work, "known_hosts")
|
||||
if err := os.WriteFile(pubPath, []byte(pubAuthorized), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(privPath, []byte(privPEM), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
// Pin the scanner-VERIFIED host key: StrictHostKeyChecking=yes against this known_hosts refuses any
|
||||
// other key (no accept-new/TOFU) — the ssh-copy-id + verify sessions connect ONLY to the box whose
|
||||
// fingerprint the bridge already matched against the hub descriptor.
|
||||
if err := os.WriteFile(khPath, []byte(knownHosts+"\n"), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
// Install (SSHPASS env is read by `sshpass -e`; the password never appears on argv).
|
||||
install := exec.CommandContext(ctx, "sshpass", "-e", "ssh-copy-id", "-p", strconv.Itoa(port), "-s", "-f",
|
||||
"-i", pubPath, "-o", "StrictHostKeyChecking=yes", "-o", "UserKnownHostsFile="+khPath, user+"@"+host)
|
||||
install.Env = append(os.Environ(), "SSHPASS="+password)
|
||||
if out, err := install.CombinedOutput(); err != nil {
|
||||
return fmt.Errorf("ssh-copy-id: %w: %s", err, truncate(out))
|
||||
}
|
||||
// Verify passwordless key auth (an SFTP no-op; the box's restricted shell only offers SFTP).
|
||||
verify := exec.CommandContext(ctx, "sftp", "-b", "-", "-P", strconv.Itoa(port),
|
||||
"-i", privPath, "-oBatchMode=yes", "-oStrictHostKeyChecking=yes", "-oUserKnownHostsFile="+khPath, user+"@"+host)
|
||||
verify.Stdin = strings.NewReader("pwd\n")
|
||||
if out, err := verify.CombinedOutput(); err != nil {
|
||||
return fmt.Errorf("key-auth verify failed after install: %w: %s", err, truncate(out))
|
||||
}
|
||||
return nil
|
||||
// --- PinnedProber: does this key reach the PINNED append-only server? ---
|
||||
//
|
||||
// Measured on the provider 2026-10-03 (audits/offsite-lock-build-2026-10-03/partA/A5): `ssh -i <key>
|
||||
// host probe` with stdin closed exits 0 and prints rclone's NOTICE when the key is pinned (the forced
|
||||
// rclone starts, sees EOF, exits); an UNPINNED key gets the restricted shell → "Command not found",
|
||||
// exit 8. A refused key exits 255. So ok = exit 0 AND "rclone" in the output — a POSITIVE observable.
|
||||
type PinnedProber struct {
|
||||
Timeout time.Duration // 0 → 20 s
|
||||
// Run is the exec seam (tests); nil → real ssh.
|
||||
Run func(ctx context.Context, args []string) ([]byte, error)
|
||||
}
|
||||
|
||||
// --- SFTPKeyAuthProber: does the ALREADY-INSTALLED key still authenticate? (key-auth-first) ---
|
||||
|
||||
// SFTPKeyAuthProber probes passwordless auth with the existing installed key (KeyPath), pinned to the
|
||||
// freshly-verified knownHosts line. No key file → ok=false (fresh guest). The probe never logs secrets.
|
||||
type SFTPKeyAuthProber struct {
|
||||
KeyPath string // the installed key, e.g. <dataDir>/offbox/ssh_key
|
||||
Timeout time.Duration // per-probe budget; 0 → 20s
|
||||
}
|
||||
|
||||
func (p SFTPKeyAuthProber) Probe(ctx context.Context, host, user string, port int, knownHosts string) (string, bool) {
|
||||
pem, err := os.ReadFile(p.KeyPath)
|
||||
if err != nil {
|
||||
return "", false // no existing key — a fresh guest; take the full path
|
||||
func (p PinnedProber) Probe(ctx context.Context, host, user string, port int, knownHosts, privPEM string) bool {
|
||||
if strings.TrimSpace(privPEM) == "" {
|
||||
return false
|
||||
}
|
||||
timeout := p.Timeout
|
||||
if timeout == 0 {
|
||||
@@ -220,21 +243,44 @@ func (p SFTPKeyAuthProber) Probe(ctx context.Context, host, user string, port in
|
||||
defer cancel()
|
||||
work, err := os.MkdirTemp("", "felhom-keyprobe-")
|
||||
if err != nil {
|
||||
return "", false
|
||||
return false
|
||||
}
|
||||
defer os.RemoveAll(work)
|
||||
khPath := filepath.Join(work, "known_hosts")
|
||||
if err := os.WriteFile(khPath, []byte(knownHosts+"\n"), 0o600); err != nil {
|
||||
return "", false
|
||||
khPath, keyPath := filepath.Join(work, "known_hosts"), filepath.Join(work, "id")
|
||||
if os.WriteFile(khPath, []byte(knownHosts+"\n"), 0o600) != nil || os.WriteFile(keyPath, []byte(privPEM), 0o600) != nil {
|
||||
return false
|
||||
}
|
||||
probe := exec.CommandContext(pctx, "sftp", "-b", "-", "-P", strconv.Itoa(port),
|
||||
"-i", p.KeyPath, "-oBatchMode=yes", "-oConnectTimeout=10",
|
||||
"-oStrictHostKeyChecking=yes", "-oUserKnownHostsFile="+khPath, user+"@"+host)
|
||||
probe.Stdin = strings.NewReader("pwd\n")
|
||||
if err := probe.Run(); err != nil {
|
||||
return "", false // auth refused / unreachable — fall through to the full path
|
||||
args := []string{"-p", strconv.Itoa(port), "-i", keyPath, "-oBatchMode=yes", "-oConnectTimeout=10", "-oIdentitiesOnly=yes",
|
||||
"-oStrictHostKeyChecking=yes", "-oUserKnownHostsFile=" + khPath, user + "@" + host, "probe"}
|
||||
run := p.Run
|
||||
if run == nil {
|
||||
run = func(ctx context.Context, args []string) ([]byte, error) {
|
||||
cmd := exec.CommandContext(ctx, "ssh", args...)
|
||||
cmd.Stdin = strings.NewReader("")
|
||||
return cmd.CombinedOutput()
|
||||
}
|
||||
}
|
||||
return string(pem), true
|
||||
out, err := run(pctx, args)
|
||||
return err == nil && strings.Contains(string(out), "rclone")
|
||||
}
|
||||
|
||||
// PublicKeyOf derives the authorized_keys line of an OpenSSH private key (the migration path: a box
|
||||
// whose key predates the pin registers the SAME key, so the hub re-writes it pinned).
|
||||
func PublicKeyOf(privPEM string) (string, error) {
|
||||
signer, err := ssh.ParsePrivateKey([]byte(privPEM))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(ssh.MarshalAuthorizedKey(signer.PublicKey())), nil
|
||||
}
|
||||
|
||||
// FingerprintOf returns the SHA256 fingerprint of an authorized_keys line.
|
||||
func FingerprintOf(pub string) (string, error) {
|
||||
pk, _, _, _, err := ssh.ParseAuthorizedKey([]byte(pub))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return ssh.FingerprintSHA256(pk), nil
|
||||
}
|
||||
|
||||
func truncate(b []byte) string {
|
||||
|
||||
@@ -68,18 +68,18 @@ func (c *fakeClock) sleep(_ context.Context, d time.Duration) {
|
||||
|
||||
// settleBridge builds a bridge wired for a FULL Reconcile (so a released gate consumes exactly once)
|
||||
// plus the injectable settle-gate. Prober is nil → the fresh consume+install path runs on release.
|
||||
func settleBridge(t *testing.T, s *fakeSettle, clk *fakeClock) (*Bridge, *fakeConsumer, *bytes.Buffer) {
|
||||
func settleBridge(t *testing.T, s *fakeSettle, clk *fakeClock) (*Bridge, *fakeRegistrar, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
cfg := &config.Config{}
|
||||
cfg.Offsite = goodOffsite()
|
||||
cons := &fakeConsumer{pw: "the-transient-pw"}
|
||||
cons := &fakeRegistrar{}
|
||||
var logbuf bytes.Buffer
|
||||
b := &Bridge{
|
||||
Cfg: cfg,
|
||||
Consumer: cons,
|
||||
Registrar: cons,
|
||||
Scanner: &fakeScanner{fp: "SHA256:goodfp", line: "[h]:23 ssh-ed25519 AAAAKEY"},
|
||||
KeyGen: &fakeKeyGen{priv: "PRIVPEM", pub: "ssh-ed25519 AAAAPUB felhom"},
|
||||
Installer: &fakeInstaller{},
|
||||
KeyGen: &realKeyGen{priv: testPriv, pub: testPub},
|
||||
Prober: &fakeProber{reg: cons, pinnedAfterRegister: true},
|
||||
Enabler: &fakeEnabler{},
|
||||
MarkerPath: filepath.Join(t.TempDir(), "offbox", "applied_marker"),
|
||||
Logger: log.New(&logbuf, "", 0),
|
||||
|
||||
@@ -314,6 +314,10 @@ type OffboxTarget struct {
|
||||
// apply-bridge. 0 = no soft limit (dedicated boxes are Hetzner-enforced; manual targets unset).
|
||||
// Felhom-enforced: at ≥100% NEW backup runs are refused (prune/restore never are); ≥80% warns.
|
||||
QuotaGB int `json:"quota_gb,omitempty"`
|
||||
// Transport (v0.289.0, decision 69): "rclone-pinned" for the HUB-provisioned Storage Box tier — the
|
||||
// box's key reaches only an append-only rclone server and cannot delete; "" (the default) is the
|
||||
// household's own SFTP NAS target, unchanged. Set by ApplyOffsiteTarget, never by a form.
|
||||
Transport string `json:"transport,omitempty"`
|
||||
|
||||
// Runtime status (written by the off-box runner; never holds a secret).
|
||||
LastRun string `json:"last_run,omitempty"` // RFC3339
|
||||
@@ -1330,6 +1334,12 @@ func (s *Settings) GetOffboxTarget() *OffboxTarget {
|
||||
}
|
||||
|
||||
// SetOffboxTarget saves (or clears, on nil) the off-box target config.
|
||||
// TransportRclonePinned marks the hub-provisioned, append-only off-site tier (decision 69).
|
||||
const TransportRclonePinned = "rclone-pinned"
|
||||
|
||||
// Pinned reports whether the target is the append-only hub tier.
|
||||
func (t *OffboxTarget) Pinned() bool { return t != nil && t.Transport == TransportRclonePinned }
|
||||
|
||||
func (s *Settings) SetOffboxTarget(t *OffboxTarget) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
@@ -99,6 +99,11 @@ func (s *Server) offboxConfigHandler(w http.ResponseWriter, r *http.Request) {
|
||||
Host: host, Port: port, User: user, RepoPath: repoPath,
|
||||
Schedule: "daily",
|
||||
}
|
||||
if prev != nil && prev.Pinned() && prev.Host == host && prev.User == user {
|
||||
// A hand edit of the hub-provisioned tier keeps its append-only transport (decision 69); a
|
||||
// different host/user is the household's own NAS and starts as SFTP.
|
||||
tgt.Transport = prev.Transport
|
||||
}
|
||||
if prev != nil { // preserve runtime status fields across an edit
|
||||
tgt.LastRun, tgt.LastStatus, tgt.LastError = prev.LastRun, prev.LastStatus, prev.LastError
|
||||
// R-100: LastSuccess is runtime status like the rest — an edit to the host/path/schedule must
|
||||
|
||||
Reference in New Issue
Block a user