felhom.eu/documentation/audits/CAMPAIGN-offsite-overnight-2026-07-10.md

CAMPAIGN — offsite arc overnight soak — 2026-07-10 (unattended)

Class: unattended live test campaign. Baselines: controller v0.109.1 (0bd4cd0), hub v0.41.0 (09bd719), agent v0.79.0 (a6e8bcb) — all confirmed live at start. Operator action required: none overnight; two morning items below. Demo end-state: restored to start-state + newer clean snapshots.

VERDICT: PASS (arc resilient) with 1 HIGH finding + 2 notes; 2 tracks blocked by session-gating

The build-complete offsite arc held up against reality. F4 credential re-issue and the restore round-trip proved out live for the first time. One genuine resilience gap surfaced (crash-mid-prune stale lock). Two tracks (the live quota-crossing and the hub-orchestrated provisioning) could not run: the hub operator config surface is session/CSRF-gated and this run was unattended (no browser, no operator password) — they join the auto-confirm happy chain in the "needs a supervised/Peti session" bucket.

Per-track results

Track	Scenario	Result
Pre-flight	baselines, demo snapshot, fence census	PASS (census: only demo 269008)
A	Hetzner API ops (create/reset/update-access/delete)	PASS — all API-valid against the real box
A	F4 re-issue (reset invalidates old pw, new works)	PASS — first live proof (throwaway 269090)
A	exactly-1 label guard data-condition	PASS (ListSubaccounts=2 → guard would refuse)
A	DNS-lag (F2) quantified	3–48s clean; >30s / TIMEOUT under create/delete churn
A	Freeze readonly behavioral write-block	INCONCLUSIVE (unreliable write-probe; API op OK)
A	hub-orchestrated enable/consume/save	BLOCKED (session-gated) → supervised
B	live quota-crossing + descriptor churn	BLOCKED (quota change is session-gated) → supervised
C	single-flight double-trigger	PASS (run2 "skipped — another backup is running")
C	crash mid-run: escrow untouched	PASS (escrowed throughout)
C	crash restart: zero consume attempts	PASS (applied-marker idempotency held)
C	crash mid-run: settings.json integrity	PASS (valid after crash)
C	crash mid-prune: repo recovery	FINDING C2 (HIGH) — stale lock blocks all runs
D	alert pipeline	1 intentional backup_failed; no flood; cooldown-dedupe not fully exercisable
E	restore round-trip both apps vs USB	PASS — 9/9 byte-identical incl. immich db-dump

Findings

• C2 (HIGH — resilience gap). A controller crash mid-prune leaves a stale restic exclusive lock (confirmed: repository is already locked exclusively by PID 33174 on demo-felhom). The controller has no auto-unlock path, so every subsequent offbox run FAILS (exit status 1) until the lock is cleared manually or ages to ~30-minute staleness — the demo's scheduled 04:15 run would have failed. Repo itself uncorrupted. Recovered in-campaign via restic unlock --remove-all (needed --remove-all because the lock's hostname demo-felhom ≠ the container hostname, so restic couldn't auto-detect the dead PID as stale), then a clean run succeeded. Recommended follow-up (supervised, code change): have ensureOffboxRepo (or a pre-run guard) detect + unlock a stale lock it owns before failing, and/or the run handler surface "repo locked — crash recovery" distinctly from a generic backup failure.
• C1 (LOW — cosmetic). A crash mid-run leaves LastStatus="running" persisted; the in-memory single-flight mutex recovers (next run is attempted), and the status self-corrects on the next completed run — but the UI shows a stale "running" until then.
• A/F2 (confirms existing OPEN). Fresh sub-account DNS lag is real and highly variable (3–48s clean,

  30s/TIMEOUT under churn) — the hub's scan-retry ladder is load-bearing; the ~60s default may occasionally be tight under churn. (Already tracked from SLICE 4; this quantifies it.)

Cannot-cover (needs a supervised/Peti session — honest list)

The live quota-crossing (Track B) and hub-orchestrated provisioning/consume-once (Track A) both require the hub operator config surface, which is session/CSRF-gated (no unattended path). The Freeze behavioral write-block needs a reliable SFTP write-probe. All join the previously-listed auto-confirm-happy-chain / fresh-enable-full-cycle bucket for the Peti onboarding runbook.

Fence audit (start vs end)

• Pool box 611714 sub-accounts: start = {269008}, end = {269008}, campaign-labelled = 0 both. (Transient throwaways 269079–269090 created + deleted; Hetzner DELETE is async → census must poll — no persistent leaks; every track ended campaign=0.)
• Demo offbox: escrow_state escrowed (unchanged), quota_gb 50 (unchanged — never touched), host/user/repo unchanged, toggled apps {audiobookshelf, immich} preserved (the closing run backed up exactly those 2), USB's 10 recovery units untouched, controller 0.109.1 healthy. Newer-only diffs: snapshot_count 2→4, repo 45→86 MB, last_run advanced (within-quota accumulated snapshots — retention prunes over time).
• No throwaway or dummy byte left behind; Track-E restore scratch removed.
• Budgets: Hetzner API ~40 calls (≤600), upload ~0 (Track B blocked — no urandom staged; ≤12 GB), throwaways ≤2 concurrent (≤8), alerts 1 email (≤100/day). No secret printed (throwaway passwords generated/used/discarded; hashes only).

MORNING — two items for Viktor

1. One expected alert email to tick off: backup_failed (~02:26 CEST) — the Track-C crash-recovery test's locked-repo run. Already recovered; no action needed. (No quota/fill/staleness emails — those tracks needed an operator session.)
2. Old-box archive retirement decision (Track E hashes in hand): the new box restores byte-perfect (9/9 files, immich db-dump sha256 d78466fcbf595b…4526b). The old-project archive u629193* is safe to retire whenever you're satisfied.

Recommended follow-up (next supervised session)

• Fix C2: auto-unlock a self-owned stale restic lock on the offbox run path (small, high-value resilience fix).
• The Peti onboarding runbook remains the vehicle for the session-gated tracks (auto-confirm happy chain, fresh-enable full cycle, live quota-crossing, hub-orchestrated provisioning + F4/freeze end-to-end).