# CAMPAIGN — offsite arc overnight soak — 2026-07-10 (unattended)

**Class:** unattended live test campaign. **Baselines:** controller v0.109.1 (`0bd4cd0`), hub v0.41.0
(`09bd719`), agent v0.79.0 (`a6e8bcb`) — all confirmed live at start. **Operator action required:** none
overnight; two morning items below. **Demo end-state:** restored to start-state + newer clean snapshots.

## VERDICT: PASS (arc resilient) with 1 HIGH finding + 2 notes; 2 tracks blocked by session-gating

The build-complete offsite arc held up against reality. F4 credential re-issue and the restore round-trip
proved out live for the first time. One genuine resilience gap surfaced (crash-mid-prune stale lock). Two
tracks (the live quota-crossing and the hub-orchestrated provisioning) could not run: the hub operator
config surface is session/CSRF-gated and this run was unattended (no browser, no operator password) — they
join the auto-confirm happy chain in the "needs a supervised/Peti session" bucket.

## Per-track results

| Track | Scenario | Result |
|---|---|---|
| Pre-flight | baselines, demo snapshot, fence census | PASS (census: only demo 269008) |
| **A** | Hetzner API ops (create/reset/update-access/delete) | PASS — all API-valid against the real box |
| **A** | **F4 re-issue** (reset invalidates old pw, new works) | **PASS — first live proof** (throwaway 269090) |
| A | exactly-1 label guard data-condition | PASS (ListSubaccounts=2 → guard would refuse) |
| A | DNS-lag (F2) quantified | 3–48s clean; >30s / TIMEOUT under create/delete churn |
| A | Freeze readonly behavioral write-block | INCONCLUSIVE (unreliable write-probe; API op OK) |
| A | hub-orchestrated enable/consume/save | BLOCKED (session-gated) → supervised |
| **B** | live quota-crossing + descriptor churn | BLOCKED (quota change is session-gated) → supervised |
| **C** | single-flight double-trigger | PASS (run2 "skipped — another backup is running") |
| C | crash mid-run: escrow untouched | PASS (escrowed throughout) |
| C | crash restart: zero consume attempts | PASS (applied-marker idempotency held) |
| C | crash mid-run: settings.json integrity | PASS (valid after crash) |
| **C** | **crash mid-prune: repo recovery** | **FINDING C2 (HIGH)** — stale lock blocks all runs |
| **D** | alert pipeline | 1 intentional backup_failed; no flood; cooldown-dedupe not fully exercisable |
| **E** | **restore round-trip both apps vs USB** | **PASS — 9/9 byte-identical** incl. immich db-dump |

## Findings

- **C2 (HIGH — resilience gap).** A controller crash **mid-prune** leaves a stale restic *exclusive* lock
  (confirmed: `repository is already locked exclusively by PID 33174 on demo-felhom`). The controller has
  **no auto-unlock path**, so every subsequent offbox run FAILS (`exit status 1`) until the lock is cleared
  manually or ages to ~30-minute staleness — the demo's scheduled 04:15 run would have failed. Repo itself
  uncorrupted. Recovered in-campaign via `restic unlock --remove-all` (needed `--remove-all` because the
  lock's hostname `demo-felhom` ≠ the container hostname, so restic couldn't auto-detect the dead PID as
  stale), then a clean run succeeded. **Recommended follow-up (supervised, code change):** have
  `ensureOffboxRepo` (or a pre-run guard) detect + `unlock` a stale lock it owns before failing, and/or the
  run handler surface "repo locked — crash recovery" distinctly from a generic backup failure.
- **C1 (LOW — cosmetic).** A crash mid-run leaves `LastStatus="running"` persisted; the in-memory
  single-flight mutex recovers (next run is attempted), and the status self-corrects on the next completed
  run — but the UI shows a stale "running" until then.
- **A/F2 (confirms existing OPEN).** Fresh sub-account DNS lag is real and highly variable (3–48s clean,
  >30s/TIMEOUT under churn) — the hub's scan-retry ladder is load-bearing; the ~60s default may occasionally
  be tight under churn. (Already tracked from SLICE 4; this quantifies it.)

## Cannot-cover (needs a supervised/Peti session — honest list)
The live quota-crossing (Track B) and hub-orchestrated provisioning/consume-once (Track A) both require the
hub operator config surface, which is session/CSRF-gated (no unattended path). The Freeze behavioral
write-block needs a reliable SFTP write-probe. All join the previously-listed auto-confirm-happy-chain /
fresh-enable-full-cycle bucket for the Peti onboarding runbook.

## Fence audit (start vs end)
- Pool box 611714 sub-accounts: **start = {269008}**, **end = {269008}**, campaign-labelled = **0** both.
  (Transient throwaways 269079–269090 created + deleted; Hetzner DELETE is async → census must poll — no
  persistent leaks; every track ended campaign=0.)
- Demo offbox: escrow_state escrowed (unchanged), quota_gb 50 (unchanged — never touched), host/user/repo
  unchanged, toggled apps {audiobookshelf, immich} preserved (the closing run backed up exactly those 2),
  USB's 10 recovery units untouched, controller 0.109.1 healthy. Newer-only diffs: snapshot_count 2→4,
  repo 45→86 MB, last_run advanced (within-quota accumulated snapshots — retention prunes over time).
- No throwaway or dummy byte left behind; Track-E restore scratch removed.
- Budgets: Hetzner API ~40 calls (≤600), upload ~0 (Track B blocked — no urandom staged; ≤12 GB),
  throwaways ≤2 concurrent (≤8), alerts 1 email (≤100/day). No secret printed (throwaway passwords
  generated/used/discarded; hashes only).

## MORNING — two items for Viktor
1. **One expected alert email** to tick off: **backup_failed** (~02:26 CEST) — the Track-C crash-recovery
   test's locked-repo run. Already recovered; no action needed. (No quota/fill/staleness emails — those
   tracks needed an operator session.)
2. **Old-box archive retirement decision** (Track E hashes in hand): the new box restores **byte-perfect**
   (9/9 files, immich db-dump sha256 `d78466fcbf595b…4526b`). The old-project archive `u629193*` is safe to
   retire whenever you're satisfied.

## Recommended follow-up (next supervised session)
- Fix C2: auto-unlock a self-owned stale restic lock on the offbox run path (small, high-value resilience fix).
- The Peti onboarding runbook remains the vehicle for the session-gated tracks (auto-confirm happy chain,
  fresh-enable full cycle, live quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end).