felhom-controller/REPORT.md

REPORT — controller v0.110.0: offbox stale-lock self-heal (C2) + crash-truthful status (C1) — DEPLOYED

Date: 2026-07-10 · Class: implementation (controller only). Baseline v0.109.1 → v0.110.0 (live on 9201, healthy). Fixes the overnight campaign's HIGH finding (C2) + the cosmetic C1. No hub/agent change.

What shipped

• C2 self-heal (internal/backup): resticStep wraps the backup/prune/restore restic calls — on a lock error (repository is already locked) it escalates to unlock --remove-all and retries the step ONCE, safe because the repo has a SINGLE legitimate writer (per-customer sub-account isolation + the in-process single-flight mutex every caller holds). A second lock failure surfaces the error (never loops). Plus a cheap pre-run unlock (stale-only) hygiene before every run + restore. Boundary (documented in code + README): a DR-cloned second controller writing the same repo would defeat the single-writer premise — operator-supervised, out of scope.
• C1 (NewManager.reconcileCrashedRun): a persisted LastStatus="running" at startup (a controller that died mid-run) flips to error + "megszakadt futás (a vezérlő újraindult futás közben)"; the next successful run clears it. Nil-settings guarded.

Tests + companion red-proofs (ALL-GREEN, both red-proofs fired)

• A — lock-on-backup → unlock --remove-all → retry → ok. Red-proof: neuter the escalation → the exact campaign failure offbox backup rallly: exit status 1 → FAIL. Reverted.
• B — lock persists twice → exactly one --remove-all + one retry → error surfaced, no unlock loop.
• C — pre-run stale unlock issued on every run (asserted via the fake's call count).
• E — no lock → --remove-all NEVER fires (the escalation can't trigger spuriously).
• D (C1) — Manager built over LastStatus="running" flips to the Hungarian interrupted error. Red-proof: drop the flip → status lies "running" forever → FAIL. Reverted.

Live validation (bounded, demo-safe — mirrors the campaign)

• Deployed 0.110.0; demo untouched (escrowed, quota 50, ok).
• C1 live: crashed the controller mid-run (docker restart) → on startup the log showed "previous run was interrupted by a controller restart…" and settings flipped to last_status="error", last_error="megszakadt futás…". ✓
• C2 live end-to-end: crashed the controller DURING a backup (restic holding the exclusive lock) → the NEXT run self-healed and completed "backup OK: 2 app(s) backed up, 43s" with zero manual restic. The demo returned to escrowed/ok; the 04:15 schedule is safe. ✓
• Refinement vs the task premise: the live evidence (and, on re-reading, the campaign's own teardown log where plain restic unlock already reported "successfully removed locks") shows the pre-run stale unlock hygiene ALONE clears these crash locks — the container's hostname is stable (demo-felhom), so restic detects the dead PID as stale. So in practice the lock is cleared before the backup step ever sees it; the --remove-all escalation is the genuine can't-detect-staleness fallback (unit-tested + red-proofed, not needed in this live run). Both layers ship — hygiene first, escalation as the safety net.

NEXT

Peti onboarding runbook (the session-gated tracks: auto-confirm happy chain, fresh-enable full cycle, live quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end); old-box archive retirement (Track E hashes in hand — the new box restores byte-perfect).