From 332725a0242b7cea25d2117e2c9974671350426c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 10 Jul 2026 07:09:26 +0200 Subject: [PATCH] =?UTF-8?q?report:=20v0.110.0=20C2/C1=20fix=20deployed=20+?= =?UTF-8?q?=20live-validated=20(crash=20=E2=86=92=20next=20run=20self-heal?= =?UTF-8?q?s)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- REPORT.md | 82 ++++++++++++++++++++++++++----------------------------- 1 file changed, 39 insertions(+), 43 deletions(-) diff --git a/REPORT.md b/REPORT.md index fd6ddf8..6c26b5f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,49 +1,45 @@ -# REPORT — controller v0.109.1: SLICE 4 soft-quota + offsite health — DEPLOYED + live-verified +# REPORT — controller v0.110.0: offbox stale-lock self-heal (C2) + crash-truthful status (C1) — DEPLOYED -**Date:** 2026-07-10 · **Class:** implementation. Baseline v0.108.0 → v0.109.0 → **v0.109.1** (live on -9201, `Up (healthy)`). Pairs with **hub v0.41.0** (OffsiteChecker + freeze lever) — live. The offsite arc's -last build item. +**Date:** 2026-07-10 · **Class:** implementation (controller only). Baseline v0.109.1 → **v0.110.0** (live +on 9201, healthy). Fixes the overnight campaign's HIGH finding (C2) + the cosmetic C1. No hub/agent change. -## What shipped (controller) -- **Data:** `OffboxTarget.QuotaGB` (bridge maps it from the descriptor; the descriptor hash now includes it - so a hub-side quota change re-applies — via key-auth-first, no password consumed) + `RepoSizeBytes` - (persisted from `restic stats`; failed stats keeps last-known — stale-but-safe). -- **Pre-run soft-quota gate:** ≥100% refuses NEW backups (Hungarian: "A NAS-mentés túllépte a - tárhelykeretet (X/Y GB)…", `LastStatus=error`, operator alert via the existing notify) — but - **`offboxPruneOnly` still runs the retention step** (pruning is the only way back under quota) and - **restore is never gated**. ≥80% (<100%) sets the Hungarian usage `LastWarning`. Quota 0 = no gate. -- **UI:** `/backups` usage bar (used/quota + %, green/amber/red) when quota>0. Template gates green. -- **Report:** the hub report carries the non-secret - `offsite:{enabled, escrow_state, last_run, last_status, snapshot_count, repo_size_bytes, quota_gb}`. -- **v0.109.1 (live finding):** the deploy's own re-apply DEMOTED the escrowed demo to pending + wiped its - runtime status (`ApplyOffsiteTarget` replaced the whole target). Now a re-apply preserves the existing - `EscrowState` (custody tracks the preserved repo password, not the coords) + runtime status; a fresh - guest still lands pending. +## What shipped +- **C2 self-heal (`internal/backup`):** `resticStep` wraps the backup/prune/restore restic calls — on a lock + error (`repository is already locked`) it escalates to `unlock --remove-all` and retries the step ONCE, + safe because the repo has a SINGLE legitimate writer (per-customer sub-account isolation + the in-process + single-flight mutex every caller holds). A second lock failure surfaces the error (never loops). Plus a + cheap pre-run `unlock` (stale-only) hygiene before every run + restore. **Boundary (documented in code + + README):** a DR-cloned second controller writing the same repo would defeat the single-writer premise — + operator-supervised, out of scope. +- **C1 (`NewManager.reconcileCrashedRun`):** a persisted `LastStatus="running"` at startup (a controller + that died mid-run) flips to `error` + "megszakadt futás (a vezérlő újraindult futás közben)"; the next + successful run clears it. Nil-settings guarded. -## Tests + companion red-proofs (ALL-GREEN, all three fired) -- Scenario A: over-quota → backup 0 calls, **prune 1 call**, Hungarian status, restore reaches the runner. - **Red-proof:** gated the prune too → "prune MUST still run over quota, got 0" → FAIL. -- Scenario B: 84% → run ok + "A NAS-mentés a keret 84%-át használja (42/50 GB)." + bytes persisted. -- Scenario C: quota 0 → no gate/warn at any size. Bridge quota mapping + quota-change re-apply - (panicking consumer) tested. -- v0.109.1 preserve: **red-proof** dropped the EscrowState carry-over → "must NOT demote an escrowed - target, got pending" → FAIL. -- Report object present when enabled / nil when unconfigured (nil-safe hub-side). +## Tests + companion red-proofs (ALL-GREEN, both red-proofs fired) +- A — lock-on-backup → `unlock --remove-all` → retry → ok. **Red-proof:** neuter the escalation → the exact + campaign failure `offbox backup rallly: exit status 1` → FAIL. Reverted. +- B — lock persists twice → exactly one `--remove-all` + one retry → error surfaced, no unlock loop. +- C — pre-run stale `unlock` issued on every run (asserted via the fake's call count). +- E — no lock → `--remove-all` NEVER fires (the escalation can't trigger spuriously). +- D (C1) — Manager built over `LastStatus="running"` flips to the Hungarian interrupted error. **Red-proof:** + drop the flip → status lies "running" forever → FAIL. Reverted. -## Live verification (all through the real pipeline) -- **Key-auth-first proven LIVE:** the quota-hash change triggered a real re-apply — "existing key still - authenticates … re-pinned + reconfigured without consuming a password" against the real box. -- v0.109.1 deployed; demo repaired via the deprecated manual confirm (truthful — same escrowed password; - the confirm's wipe also removed the re-staged agent secret, verified empty) + a manual run: - **2 snapshots, 41s, `escrowed`, `quota_gb: 50`, `repo_size_bytes` 45 MB**. -- The next report carried the full offsite object hub-side; the **OffsiteChecker held it healthy** - (0 `offsite_*` events — 0% fill, not stale). The `/backups` usage bar renders live: - "Tárhelykeret: 43.2 MB / 50 GB (0%)" (curl through the in-guest UI). - -## NOT live-exercised (per the STOP) -The freeze lever (would write-lock the demo's working offsite) and the 100% quota-refusal (demo repo is -0.08% of quota) — unit red-proofs cover both; live exercise waits for a real need / the Peti window. +## Live validation (bounded, demo-safe — mirrors the campaign) +- Deployed 0.110.0; demo untouched (escrowed, quota 50, ok). +- **C1 live:** crashed the controller mid-run (`docker restart`) → on startup the log showed "previous run + was interrupted by a controller restart…" and settings flipped to `last_status="error"`, + `last_error="megszakadt futás…"`. ✓ +- **C2 live end-to-end:** crashed the controller DURING a backup (restic holding the exclusive lock) → the + NEXT run self-healed and completed "backup OK: 2 app(s) backed up, 43s" with **zero manual restic**. The + demo returned to `escrowed`/`ok`; the 04:15 schedule is safe. ✓ +- **Refinement vs the task premise:** the live evidence (and, on re-reading, the campaign's own teardown log + where plain `restic unlock` already reported "successfully removed locks") shows the pre-run stale `unlock` + hygiene ALONE clears these crash locks — the container's hostname is stable (`demo-felhom`), so restic + detects the dead PID as stale. So in practice the lock is cleared before the backup step ever sees it; the + `--remove-all` escalation is the genuine can't-detect-staleness fallback (unit-tested + red-proofed, not + needed in this live run). Both layers ship — hygiene first, escalation as the safety net. ## NEXT -Peti onboarding runbook (live-exercises auto-confirm + F4 + fresh-provision quota), NEW-box restore -round-trip → retire the old-box archive. +Peti onboarding runbook (the session-gated tracks: auto-confirm happy chain, fresh-enable full cycle, live +quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end); old-box archive retirement (Track E +hashes in hand — the new box restores byte-perfect).