admin
332725a024
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
46 lines
3.5 KiB
Markdown
46 lines
3.5 KiB
Markdown
# REPORT — controller v0.110.0: offbox stale-lock self-heal (C2) + crash-truthful status (C1) — DEPLOYED
|
|
|
|
**Date:** 2026-07-10 · **Class:** implementation (controller only). Baseline v0.109.1 → **v0.110.0** (live
|
|
on 9201, healthy). Fixes the overnight campaign's HIGH finding (C2) + the cosmetic C1. No hub/agent change.
|
|
|
|
## What shipped
|
|
- **C2 self-heal (`internal/backup`):** `resticStep` wraps the backup/prune/restore restic calls — on a lock
|
|
error (`repository is already locked`) it escalates to `unlock --remove-all` and retries the step ONCE,
|
|
safe because the repo has a SINGLE legitimate writer (per-customer sub-account isolation + the in-process
|
|
single-flight mutex every caller holds). A second lock failure surfaces the error (never loops). Plus a
|
|
cheap pre-run `unlock` (stale-only) hygiene before every run + restore. **Boundary (documented in code +
|
|
README):** a DR-cloned second controller writing the same repo would defeat the single-writer premise —
|
|
operator-supervised, out of scope.
|
|
- **C1 (`NewManager.reconcileCrashedRun`):** a persisted `LastStatus="running"` at startup (a controller
|
|
that died mid-run) flips to `error` + "megszakadt futás (a vezérlő újraindult futás közben)"; the next
|
|
successful run clears it. Nil-settings guarded.
|
|
|
|
## Tests + companion red-proofs (ALL-GREEN, both red-proofs fired)
|
|
- A — lock-on-backup → `unlock --remove-all` → retry → ok. **Red-proof:** neuter the escalation → the exact
|
|
campaign failure `offbox backup rallly: exit status 1` → FAIL. Reverted.
|
|
- B — lock persists twice → exactly one `--remove-all` + one retry → error surfaced, no unlock loop.
|
|
- C — pre-run stale `unlock` issued on every run (asserted via the fake's call count).
|
|
- E — no lock → `--remove-all` NEVER fires (the escalation can't trigger spuriously).
|
|
- D (C1) — Manager built over `LastStatus="running"` flips to the Hungarian interrupted error. **Red-proof:**
|
|
drop the flip → status lies "running" forever → FAIL. Reverted.
|
|
|
|
## Live validation (bounded, demo-safe — mirrors the campaign)
|
|
- Deployed 0.110.0; demo untouched (escrowed, quota 50, ok).
|
|
- **C1 live:** crashed the controller mid-run (`docker restart`) → on startup the log showed "previous run
|
|
was interrupted by a controller restart…" and settings flipped to `last_status="error"`,
|
|
`last_error="megszakadt futás…"`. ✓
|
|
- **C2 live end-to-end:** crashed the controller DURING a backup (restic holding the exclusive lock) → the
|
|
NEXT run self-healed and completed "backup OK: 2 app(s) backed up, 43s" with **zero manual restic**. The
|
|
demo returned to `escrowed`/`ok`; the 04:15 schedule is safe. ✓
|
|
- **Refinement vs the task premise:** the live evidence (and, on re-reading, the campaign's own teardown log
|
|
where plain `restic unlock` already reported "successfully removed locks") shows the pre-run stale `unlock`
|
|
hygiene ALONE clears these crash locks — the container's hostname is stable (`demo-felhom`), so restic
|
|
detects the dead PID as stale. So in practice the lock is cleared before the backup step ever sees it; the
|
|
`--remove-all` escalation is the genuine can't-detect-staleness fallback (unit-tested + red-proofed, not
|
|
needed in this live run). Both layers ship — hygiene first, escalation as the safety net.
|
|
|
|
## NEXT
|
|
Peti onboarding runbook (the session-gated tracks: auto-confirm happy chain, fresh-enable full cycle, live
|
|
quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end); old-box archive retirement (Track E
|
|
hashes in hand — the new box restores byte-perfect).
|