report: v0.110.0 C2/C1 fix deployed + live-validated (crash → next run self-heals)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -1,49 +1,45 @@
|
|||||||
# REPORT — controller v0.109.1: SLICE 4 soft-quota + offsite health — DEPLOYED + live-verified
|
# REPORT — controller v0.110.0: offbox stale-lock self-heal (C2) + crash-truthful status (C1) — DEPLOYED
|
||||||
|
|
||||||
**Date:** 2026-07-10 · **Class:** implementation. Baseline v0.108.0 → v0.109.0 → **v0.109.1** (live on
|
**Date:** 2026-07-10 · **Class:** implementation (controller only). Baseline v0.109.1 → **v0.110.0** (live
|
||||||
9201, `Up (healthy)`). Pairs with **hub v0.41.0** (OffsiteChecker + freeze lever) — live. The offsite arc's
|
on 9201, healthy). Fixes the overnight campaign's HIGH finding (C2) + the cosmetic C1. No hub/agent change.
|
||||||
last build item.
|
|
||||||
|
|
||||||
## What shipped (controller)
|
## What shipped
|
||||||
- **Data:** `OffboxTarget.QuotaGB` (bridge maps it from the descriptor; the descriptor hash now includes it
|
- **C2 self-heal (`internal/backup`):** `resticStep` wraps the backup/prune/restore restic calls — on a lock
|
||||||
so a hub-side quota change re-applies — via key-auth-first, no password consumed) + `RepoSizeBytes`
|
error (`repository is already locked`) it escalates to `unlock --remove-all` and retries the step ONCE,
|
||||||
(persisted from `restic stats`; failed stats keeps last-known — stale-but-safe).
|
safe because the repo has a SINGLE legitimate writer (per-customer sub-account isolation + the in-process
|
||||||
- **Pre-run soft-quota gate:** ≥100% refuses NEW backups (Hungarian: "A NAS-mentés túllépte a
|
single-flight mutex every caller holds). A second lock failure surfaces the error (never loops). Plus a
|
||||||
tárhelykeretet (X/Y GB)…", `LastStatus=error`, operator alert via the existing notify) — but
|
cheap pre-run `unlock` (stale-only) hygiene before every run + restore. **Boundary (documented in code +
|
||||||
**`offboxPruneOnly` still runs the retention step** (pruning is the only way back under quota) and
|
README):** a DR-cloned second controller writing the same repo would defeat the single-writer premise —
|
||||||
**restore is never gated**. ≥80% (<100%) sets the Hungarian usage `LastWarning`. Quota 0 = no gate.
|
operator-supervised, out of scope.
|
||||||
- **UI:** `/backups` usage bar (used/quota + %, green/amber/red) when quota>0. Template gates green.
|
- **C1 (`NewManager.reconcileCrashedRun`):** a persisted `LastStatus="running"` at startup (a controller
|
||||||
- **Report:** the hub report carries the non-secret
|
that died mid-run) flips to `error` + "megszakadt futás (a vezérlő újraindult futás közben)"; the next
|
||||||
`offsite:{enabled, escrow_state, last_run, last_status, snapshot_count, repo_size_bytes, quota_gb}`.
|
successful run clears it. Nil-settings guarded.
|
||||||
- **v0.109.1 (live finding):** the deploy's own re-apply DEMOTED the escrowed demo to pending + wiped its
|
|
||||||
runtime status (`ApplyOffsiteTarget` replaced the whole target). Now a re-apply preserves the existing
|
|
||||||
`EscrowState` (custody tracks the preserved repo password, not the coords) + runtime status; a fresh
|
|
||||||
guest still lands pending.
|
|
||||||
|
|
||||||
## Tests + companion red-proofs (ALL-GREEN, all three fired)
|
## Tests + companion red-proofs (ALL-GREEN, both red-proofs fired)
|
||||||
- Scenario A: over-quota → backup 0 calls, **prune 1 call**, Hungarian status, restore reaches the runner.
|
- A — lock-on-backup → `unlock --remove-all` → retry → ok. **Red-proof:** neuter the escalation → the exact
|
||||||
**Red-proof:** gated the prune too → "prune MUST still run over quota, got 0" → FAIL.
|
campaign failure `offbox backup rallly: exit status 1` → FAIL. Reverted.
|
||||||
- Scenario B: 84% → run ok + "A NAS-mentés a keret 84%-át használja (42/50 GB)." + bytes persisted.
|
- B — lock persists twice → exactly one `--remove-all` + one retry → error surfaced, no unlock loop.
|
||||||
- Scenario C: quota 0 → no gate/warn at any size. Bridge quota mapping + quota-change re-apply
|
- C — pre-run stale `unlock` issued on every run (asserted via the fake's call count).
|
||||||
(panicking consumer) tested.
|
- E — no lock → `--remove-all` NEVER fires (the escalation can't trigger spuriously).
|
||||||
- v0.109.1 preserve: **red-proof** dropped the EscrowState carry-over → "must NOT demote an escrowed
|
- D (C1) — Manager built over `LastStatus="running"` flips to the Hungarian interrupted error. **Red-proof:**
|
||||||
target, got pending" → FAIL.
|
drop the flip → status lies "running" forever → FAIL. Reverted.
|
||||||
- Report object present when enabled / nil when unconfigured (nil-safe hub-side).
|
|
||||||
|
|
||||||
## Live verification (all through the real pipeline)
|
## Live validation (bounded, demo-safe — mirrors the campaign)
|
||||||
- **Key-auth-first proven LIVE:** the quota-hash change triggered a real re-apply — "existing key still
|
- Deployed 0.110.0; demo untouched (escrowed, quota 50, ok).
|
||||||
authenticates … re-pinned + reconfigured without consuming a password" against the real box.
|
- **C1 live:** crashed the controller mid-run (`docker restart`) → on startup the log showed "previous run
|
||||||
- v0.109.1 deployed; demo repaired via the deprecated manual confirm (truthful — same escrowed password;
|
was interrupted by a controller restart…" and settings flipped to `last_status="error"`,
|
||||||
the confirm's wipe also removed the re-staged agent secret, verified empty) + a manual run:
|
`last_error="megszakadt futás…"`. ✓
|
||||||
**2 snapshots, 41s, `escrowed`, `quota_gb: 50`, `repo_size_bytes` 45 MB**.
|
- **C2 live end-to-end:** crashed the controller DURING a backup (restic holding the exclusive lock) → the
|
||||||
- The next report carried the full offsite object hub-side; the **OffsiteChecker held it healthy**
|
NEXT run self-healed and completed "backup OK: 2 app(s) backed up, 43s" with **zero manual restic**. The
|
||||||
(0 `offsite_*` events — 0% fill, not stale). The `/backups` usage bar renders live:
|
demo returned to `escrowed`/`ok`; the 04:15 schedule is safe. ✓
|
||||||
"Tárhelykeret: 43.2 MB / 50 GB (0%)" (curl through the in-guest UI).
|
- **Refinement vs the task premise:** the live evidence (and, on re-reading, the campaign's own teardown log
|
||||||
|
where plain `restic unlock` already reported "successfully removed locks") shows the pre-run stale `unlock`
|
||||||
## NOT live-exercised (per the STOP)
|
hygiene ALONE clears these crash locks — the container's hostname is stable (`demo-felhom`), so restic
|
||||||
The freeze lever (would write-lock the demo's working offsite) and the 100% quota-refusal (demo repo is
|
detects the dead PID as stale. So in practice the lock is cleared before the backup step ever sees it; the
|
||||||
0.08% of quota) — unit red-proofs cover both; live exercise waits for a real need / the Peti window.
|
`--remove-all` escalation is the genuine can't-detect-staleness fallback (unit-tested + red-proofed, not
|
||||||
|
needed in this live run). Both layers ship — hygiene first, escalation as the safety net.
|
||||||
|
|
||||||
## NEXT
|
## NEXT
|
||||||
Peti onboarding runbook (live-exercises auto-confirm + F4 + fresh-provision quota), NEW-box restore
|
Peti onboarding runbook (the session-gated tracks: auto-confirm happy chain, fresh-enable full cycle, live
|
||||||
round-trip → retire the old-box archive.
|
quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end); old-box archive retirement (Track E
|
||||||
|
hashes in hand — the new box restores byte-perfect).
|
||||||
|
|||||||
Reference in New Issue
Block a user