felhom.eu/REPORT-offsite-lock-build-2026-10-03.md

REPORT — off-site backups a box cannot delete: built and live (decisions 68–70) — 2026-10-03 (evening)

Architecture read: 07-backup-architecture.md (custody, threat rows 9–12, §D), 06-offsite-connectivity.md §3/§5, 09 §3. Baselines (re-verified): controller 09453325d1b2 (0.288.0), agent d766666ff8cf (0.138.0), felhom.eu 9268d9933b8f (hub 0.126.0 deployed), catalog 917a779cca67. Register 327 rows, highest R-822. Rulings recorded first as decisions 68–70 (5188dbd). Evidence: documentation/audits/offsite-lock-build-2026-10-03/.

The Part table

Part	Result	Notes
A — migration spike	done, passed	An sftp-written repo is listed, extended, restored from (bytes identical), checked and check --read-dataed through the pinned rclone: key with restic 0.14.0; a delete is refused (403). The same measurements also settled several facts: one key on two lines → the first line wins (so the window = prepend a deleting line); an absolute pinned path works; a probe signal (exit 0 + rclone output = pinned; exit 8 = unpinned); the restricted shell's dd/mv/cat (no test).
B — hub registrar + sealed password	done — hub v0.127.0 deployed	internal/offsitekeys; consume-password → 410; password AES-256-GCM at rest (4 live rows sealed, read back as enc:v1:); daily key check 07:10 + on demand. 3 red-proofs.
C — box on the locked key	done — controller v0.289.0, then v0.289.1	registrar client, pinned probe, rclone: transport (hub tier only — the household NAS stays sftp), all four deleting features off the box, window client + fake-snapshot guard. 4 red-proofs. v0.289.1 fixes a defect v0.289.0 put live (below).
D — live, both demo boxes	done	demo-felhom 11→13, demo-hp 91→100 (history kept); a delete from each box refused (403), count unchanged; one-file restore and check through the pinned key on each; the old endpoint answers 410 to demo-hp's own key. The hand-run key check is clean for both. Changed: the "unprefixed test line on a demo sub-account" decoy ran on tester-1's account instead, which already held 3 unpinned lines. The demo passwords are now sealed in the hub, and the hub DB is the only route to them.
— stop point	passed
E — window + guard	mechanics done; a real prune NOT done	Window 1 on demo-hp ran live: the hub opened it (deleting line first), the guard refused, the window closed in 3 s, the operator was mailed, and the key file read back clean. The refusal is a design defect (R-824): any manual run makes the plan remove a same-day snapshot younger than 8 days. Weekly windows stay OFF.
F — ep0 → DooPlex copy	done	ep0: one read-only token (the only change there). DooPlex: an SSH forward (felhom-ep0-pbs-tunnel.service — operator ruling in-session, because ep0's PBS listens on wg0 only), PBS remote, datastore ep0-copy, nightly pull 05:00 with remove-vanished false, Saturday verify, failures to admin@ via Resend (the test mail arrived). First pull: 201 s, 12 GB, 4 of 4 snapshots, matching ep0. Runbook: runbooks/ep0-datastore-copy.md.
Golden + floor	done	Golden 0.289.1 baked (subagent, runbook §4.1, token-leak grep 0 with a positive control), round-trip sha matches, vouched (agent 0.138.0, min_agent 0.131.0), floor 0.289.1 SERVED to both boxes.

Claims in the brief that turned out wrong

1. "An sftp-written repo reads through rclone" — confirmed: it was expected, and now it is measured.
2. "The box stores no password today" — true on disk (it was only in an env var during install), but the box could fetch the password at will; that route is closed now.
3. "One authorized_keys can hold two lines for the same key" — it can, but only the first line counts. That is what makes the window possible with a single key.
4. "The integrity check works through the forced key" — confirmed (check, check --read-data, the exclusive lock).
5. "DooPlex has room and a PBS that can pull from ep0" — it has the room (5.5 TB) and a PBS, but it cannot reach ep0's PBS (open on wg0 only). An SSH forward was added on an operator ruling.
6. "Every deleting feature leaves the box" — only on the hub tier. The same code serves the household's own SFTP NAS, which keeps box-side retention. A Transport flag separates the two.
7. "Abort when the plan exceeds a week's removal" — that would never prune after the interim. The box takes the oldest snapshots up to the cap instead (disagreement recorded in the code and the CHANGELOG).
8. The guard as written is too strict (R-824). It also cannot see past-dated poisoning (R-822, residual).

Found and fixed in-session

• R-825 (v0.289.1): the provider's rclone prints a NOTICE line on every connection, and restic forwards it into the output. Every --json parse failed, so demo-felhom recorded 0 snapshots as measured, and the hub mailed a false offsite_snapshots_dropped (11→0) at 17:17. The fix went live 15 min later. It strips the notice, and an unreadable count is never a measured zero. Red-proved.
• A shadowed newPath in the NAS move-aside path was caught by the existing suite before release.
• Stale unpinned keys sat in the sub-accounts: 4 on demo-felhom's and 5 on demo-hp's (every reinstall added one). The registrar removed them. tester-1's 3 remain, and the daily check alarms on them (R-826).

Deviations, stated

• Two controller releases, against the one-release rule: v0.289.1 fixes a false zero that v0.289.0 put live.
• The hub has a test-only commit after the release (window-sweep test + a test helper in the store). The deployed v0.127.0 image does not contain it; behaviour is unchanged.
• I read tester-1's sealed-era password from the hub DB again for Part A (operator ruling from the morning session; the copy was deleted, the value never printed).
• A Hetzner storage API token was printed into this session's transcript while I read manifests/storagebox.secret.yaml (a gitignored file; the redaction regex missed the quoted value). Rotate HETZNER_TOKEN — it is in Secret/storagebox.
• The window's red-proofs ran in unit tests and on one live window. A live real prune did not happen (R-824).

Records

• Closed: R-820, R-821, R-342 (+ R-825 opened and closed). Narrowed: R-95, R-822. Opened: R-823, R-824, R-826, R-827, R-828, R-830. Register 327 → 330.
• Decisions 68–70 in 09 §3, CONTEXT, 07, 06. 07 threat rows 9/10/12 and 06 §3.6 carry [FACT] lines.
• Runbooks: ep0-datastore-copy.md (new), secrets.md (offsite key, DooPlex PBS secrets), RUNBOOK-manual-build.md (pveam update).

Teardown, three layers

• Machines: helper scripts were removed from both demo guests, their containers and hosts (0 left). tester-1's sub-account is back to .ssh, felhom-repo, with authorized_keys byte-identical to the start (sha256 795e7153…). The scratch dirs spike-r436, spike-migrate and the rclone .config were removed. The drill VM is reverted to virgin, build guest 9100 destroyed, the bake token shredded.
• Host (DooPlex): kept on purpose: felhom-ep0-pbs-tunnel.service, the PBS remote/datastore/jobs/notification target (Part F). The scratchpad secret files (sub4 password, ep0 token, Resend key copy) were shredded.
• Hub: kept: v0.127.0, Secret/offsite-secret-key, floor 0.289.1, golden 0.289.1 vouched. Weekly windows OFF. The one-shot grant for demo-hp was consumed.