admin
292a23bf39
Unattended live soak of the build-complete offsite arc. F4 re-issue + restore round-trip proved live (9/9 byte-identical, immich db-dump exact). Crash recovery surfaced a HIGH resilience gap: crash mid-prune leaves a stale restic exclusive lock the controller can't self-heal → subsequent runs fail until manual unlock. Demo restored to start-state. Track B (quota) + hub-orchestrated A blocked by session-gating → Peti runbook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
84 lines
6.3 KiB
Markdown
84 lines
6.3 KiB
Markdown
# CAMPAIGN — offsite arc overnight soak — 2026-07-10 (unattended)
|
||
|
||
**Class:** unattended live test campaign. **Baselines:** controller v0.109.1 (`0bd4cd0`), hub v0.41.0
|
||
(`09bd719`), agent v0.79.0 (`a6e8bcb`) — all confirmed live at start. **Operator action required:** none
|
||
overnight; two morning items below. **Demo end-state:** restored to start-state + newer clean snapshots.
|
||
|
||
## VERDICT: PASS (arc resilient) with 1 HIGH finding + 2 notes; 2 tracks blocked by session-gating
|
||
|
||
The build-complete offsite arc held up against reality. F4 credential re-issue and the restore round-trip
|
||
proved out live for the first time. One genuine resilience gap surfaced (crash-mid-prune stale lock). Two
|
||
tracks (the live quota-crossing and the hub-orchestrated provisioning) could not run: the hub operator
|
||
config surface is session/CSRF-gated and this run was unattended (no browser, no operator password) — they
|
||
join the auto-confirm happy chain in the "needs a supervised/Peti session" bucket.
|
||
|
||
## Per-track results
|
||
|
||
| Track | Scenario | Result |
|
||
|---|---|---|
|
||
| Pre-flight | baselines, demo snapshot, fence census | PASS (census: only demo 269008) |
|
||
| **A** | Hetzner API ops (create/reset/update-access/delete) | PASS — all API-valid against the real box |
|
||
| **A** | **F4 re-issue** (reset invalidates old pw, new works) | **PASS — first live proof** (throwaway 269090) |
|
||
| A | exactly-1 label guard data-condition | PASS (ListSubaccounts=2 → guard would refuse) |
|
||
| A | DNS-lag (F2) quantified | 3–48s clean; >30s / TIMEOUT under create/delete churn |
|
||
| A | Freeze readonly behavioral write-block | INCONCLUSIVE (unreliable write-probe; API op OK) |
|
||
| A | hub-orchestrated enable/consume/save | BLOCKED (session-gated) → supervised |
|
||
| **B** | live quota-crossing + descriptor churn | BLOCKED (quota change is session-gated) → supervised |
|
||
| **C** | single-flight double-trigger | PASS (run2 "skipped — another backup is running") |
|
||
| C | crash mid-run: escrow untouched | PASS (escrowed throughout) |
|
||
| C | crash restart: zero consume attempts | PASS (applied-marker idempotency held) |
|
||
| C | crash mid-run: settings.json integrity | PASS (valid after crash) |
|
||
| **C** | **crash mid-prune: repo recovery** | **FINDING C2 (HIGH)** — stale lock blocks all runs |
|
||
| **D** | alert pipeline | 1 intentional backup_failed; no flood; cooldown-dedupe not fully exercisable |
|
||
| **E** | **restore round-trip both apps vs USB** | **PASS — 9/9 byte-identical** incl. immich db-dump |
|
||
|
||
## Findings
|
||
|
||
- **C2 (HIGH — resilience gap).** A controller crash **mid-prune** leaves a stale restic *exclusive* lock
|
||
(confirmed: `repository is already locked exclusively by PID 33174 on demo-felhom`). The controller has
|
||
**no auto-unlock path**, so every subsequent offbox run FAILS (`exit status 1`) until the lock is cleared
|
||
manually or ages to ~30-minute staleness — the demo's scheduled 04:15 run would have failed. Repo itself
|
||
uncorrupted. Recovered in-campaign via `restic unlock --remove-all` (needed `--remove-all` because the
|
||
lock's hostname `demo-felhom` ≠ the container hostname, so restic couldn't auto-detect the dead PID as
|
||
stale), then a clean run succeeded. **Recommended follow-up (supervised, code change):** have
|
||
`ensureOffboxRepo` (or a pre-run guard) detect + `unlock` a stale lock it owns before failing, and/or the
|
||
run handler surface "repo locked — crash recovery" distinctly from a generic backup failure.
|
||
- **C1 (LOW — cosmetic).** A crash mid-run leaves `LastStatus="running"` persisted; the in-memory
|
||
single-flight mutex recovers (next run is attempted), and the status self-corrects on the next completed
|
||
run — but the UI shows a stale "running" until then.
|
||
- **A/F2 (confirms existing OPEN).** Fresh sub-account DNS lag is real and highly variable (3–48s clean,
|
||
>30s/TIMEOUT under churn) — the hub's scan-retry ladder is load-bearing; the ~60s default may occasionally
|
||
be tight under churn. (Already tracked from SLICE 4; this quantifies it.)
|
||
|
||
## Cannot-cover (needs a supervised/Peti session — honest list)
|
||
The live quota-crossing (Track B) and hub-orchestrated provisioning/consume-once (Track A) both require the
|
||
hub operator config surface, which is session/CSRF-gated (no unattended path). The Freeze behavioral
|
||
write-block needs a reliable SFTP write-probe. All join the previously-listed auto-confirm-happy-chain /
|
||
fresh-enable-full-cycle bucket for the Peti onboarding runbook.
|
||
|
||
## Fence audit (start vs end)
|
||
- Pool box 611714 sub-accounts: **start = {269008}**, **end = {269008}**, campaign-labelled = **0** both.
|
||
(Transient throwaways 269079–269090 created + deleted; Hetzner DELETE is async → census must poll — no
|
||
persistent leaks; every track ended campaign=0.)
|
||
- Demo offbox: escrow_state escrowed (unchanged), quota_gb 50 (unchanged — never touched), host/user/repo
|
||
unchanged, toggled apps {audiobookshelf, immich} preserved (the closing run backed up exactly those 2),
|
||
USB's 10 recovery units untouched, controller 0.109.1 healthy. Newer-only diffs: snapshot_count 2→4,
|
||
repo 45→86 MB, last_run advanced (within-quota accumulated snapshots — retention prunes over time).
|
||
- No throwaway or dummy byte left behind; Track-E restore scratch removed.
|
||
- Budgets: Hetzner API ~40 calls (≤600), upload ~0 (Track B blocked — no urandom staged; ≤12 GB),
|
||
throwaways ≤2 concurrent (≤8), alerts 1 email (≤100/day). No secret printed (throwaway passwords
|
||
generated/used/discarded; hashes only).
|
||
|
||
## MORNING — two items for Viktor
|
||
1. **One expected alert email** to tick off: **backup_failed** (~02:26 CEST) — the Track-C crash-recovery
|
||
test's locked-repo run. Already recovered; no action needed. (No quota/fill/staleness emails — those
|
||
tracks needed an operator session.)
|
||
2. **Old-box archive retirement decision** (Track E hashes in hand): the new box restores **byte-perfect**
|
||
(9/9 files, immich db-dump sha256 `d78466fcbf595b…4526b`). The old-project archive `u629193*` is safe to
|
||
retire whenever you're satisfied.
|
||
|
||
## Recommended follow-up (next supervised session)
|
||
- Fix C2: auto-unlock a self-owned stale restic lock on the offbox run path (small, high-value resilience fix).
|
||
- The Peti onboarding runbook remains the vehicle for the session-gated tracks (auto-confirm happy chain,
|
||
fresh-enable full cycle, live quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end).
|