felhom.eu/documentation/audits/offsite-append-only-2026-10-03/DESIGN.md

Off-site append-only — who deletes old backups when the box cannot? (PROPOSAL, 2026-10-03)

Status: a PROPOSAL for the operator to rule on. Nothing here is built. Nothing here is [DESIGN]. Evidence: live/ (the provider, u629488-sub4) and lab/ (OpenSSH + rclone v1.75.1 on DooPlex). Baselines: felhom.eu f4c5466, controller 0945332 (v0.288.0, restic 0.14.0 Debian 0.14.0-1+b5).

1. What was measured (the facts this design stands on)

#	Fact	Where
F1	A key whose authorized_keys line is command="rclone serve restic --stdio --append-only <dir>",restrict can init, backup, snapshots, restore (bytes identical) and check.	live/E1-E3, live/E4-E6, live/C2-A5
F2	Through that key forget <id> --prune, forget --keep-last 1 and a real prune are REFUSED: blob not removed, server response: 403 Forbidden (403), rc=1, snapshot count unchanged. Each refusal costs ~45–48 s of restic retries. A refused prune has already written a new index first (check: duplicate index, non-critical).	live/E4-E6
F3	Control: the same forget through a key WITHOUT the forced command deletes (1/1 files deleted).	live/E4-E6 (C1)
F4	The client's requested path and flags are ignored: the pinned directory is served even when the client names another path, and the client never asked for --append-only (restic sends serve restic --stdio --b2-hard-delete).	live/C2-A5, live/B2
F5	The forced key cannot get a shell, sftp, scp, rsync or a port forward (administratively prohibited); a command such as rm -rf <dir> just starts the forced rclone.	live/B2, live/B1-B2
F6	Port 22 accepts no OpenSSH-format key at all (both test keys refused there); keys work on port 23 only — the port the product uses (hub/internal/offsite/offsite.go sftpPort = 23). The PASSWORD logs in on BOTH ports, and through it .ssh/authorized_keys was read and REWRITTEN (that is how the test keys were installed).	live/B1-B2, this session
F7	Crash locks: a killed backup leaves a lock. The next backup is not blocked; an exclusive operation (check — the weekly integrity job) is. Plain unlock prints successfully removed locks and removes nothing (the lock is not yet stale — by design, R-430 reproduced); unlock --remove-all does remove it — the append-only server allows lock deletion.	live/C2-A5, lab/A5
F8	A crashed upload leaves unreferenced packs (lab: 17) that only a deleting key can clean.	lab/A5
F9	The append-only server refuses delete/overwrite of existing files (403) and path escapes (../, %2e%2e, ..%2f → 400), but ACCEPTS new files of any name inside the repo, including a new keys/ entry (quota can be filled).	lab/B-rclone-server-surface (lab rclone; the provider's rclone version cannot be read — rclone version is not a shell command there)
F10	Retention poisoning. Through the add-only key, 13 future-dated empty snapshots with the same host+tag make the box's exact policy (--group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6) select all 3 real snapshots for removal.	lab/C3-retention-poisoning
F11	The provider's rclone creates ~/.config/rclone/ (empty) in the sub-account home on first use.	live/TEARDOWN

From the code (read, not run):

#	Fact	Where
K1	The box can obtain its sub-account password at will: it declares needs_credential in two reports, the hub's off-site self-heal re-arms the stored value (RestageOneTimeSecret; the value is never cleared after a consume) or escalates to a provider re-issue, and the box consumes it with its own API key.	hub/internal/offsiteheal/reconciler.go, hub/internal/store/store.go ConsumeOneTimeSecret/RestageOneTimeSecret, controller/internal/offsiteapply/seams.go
K2	The hub DB holds every sub-account password in the clear, indefinitely (one_time_secrets.value).	hub/internal/store/store.go
K3	Four box features need a normal (deleting) login, not just the two forget sites: forget --prune after a run (offbox.go ~1422), over quota (offbox.go ~1793), the orphan move-aside (mv, offbox.go resetOrphanedRepo), and the customer-chosen abandonment (rm -rf, offbox_abandon.go AbandonSweep).	controller 0945332
K4	restic has no prune-only credential: every key unlocks the same master key, so whoever prunes can read every backup in that repo.	restic design (documented, not measured)

2. The prerequisite every option shares — the lock is worthless until this is closed

F6 + K1: a box that is broken into can get the password, log in on port 23 with it, and rewrite authorized_keys — removing the forced command from its own key. So before any lock ships:

1. The box never receives the sub-account password. The hub becomes the key registrar: the box generates its key pair and sends the PUBLIC key to the hub; the hub (which already holds the password, K2) writes the forced line into authorized_keys over SFTP port 23. The off-site self-heal re-arms the HUB's install job, not a password for the box.
2. The hub checks the file. Each day the hub reads authorized_keys and alarms if any line lacks the forced prefix (outside an open clean-up window). This is also the answer to "an operator adds a key later": a key without the prefix re-opens deletion, so a check — not a rule — catches it.
3. Optional hardening, operator's call: rotate the sub-account password after each hub use and keep it only in the hub (K2 stays true either way: the hub can already delete every household's off-site history today — new row).

3. The options

Option 1 — a clean-up window opened by the hub (the box keeps pruning its own repo)

The box normally holds only its add-only key. Once a week the hub adds a second, deleting key line for the same box key pair (or a second key the box holds) for ~15 minutes; the box runs its retention; the hub removes the line and checks the file.

• Custody: unchanged. The repository password never leaves the box (07 §8a stays true).
• What a compromised box can do: delete during a window. Detection backstop: R-431 (a fall of more than half), plus a new per-window check — the hub compares the snapshot count before and after and alarms on a fall larger than the retention could cause.
• Poisoning (F10) must be guarded even here, because an attacker who was on the box and left can plant future snapshots that the next honest window then obeys: before forget, refuse when any snapshot is dated in the future or newer than the newest the hub has seen reported; run --dry-run first and abort if it would remove more snapshots than the policy can remove in a week.
• Build cost: hub: key registrar + window open/close + authorized_keys check (SFTP with the password it holds — no provider API, no main account). Controller: switch the transport from sftp: to rclone: with -o rclone.program="ssh -p 23 … -i <key> … rclone" (restic 0.14.0 is enough, F1; rclone is NOT needed in the image); move forget --prune (both sites, together — R-191) into the window; move the move-aside and the abandonment to the hub (K3).
• Money: none.

Option 2 — clean-up runs off the box

A Felhom-side worker prunes each repo with a deleting key.

• Custody: CHANGES. restic has no prune-only key (K4): the worker must hold every repository password in a form it can use unattended, and could read every household's backups. That changes a promise to the customer (07 §8a) — not CC's to change.
• Poisoning (F10): same guard needed.
• Build cost: a new always-on worker in the recovery path, its own credentials store, its own failure alarms — plus everything in §2.

Option 3 — never delete; let the quota grow

• Money: small. The pool box (BX11, 1 TB) was €4.06/month when recorded (SPIKE-ep0-storagebox- 2026-07-09; Hetzner raised prices in April 2026 — re-check) ≈ €0.004 per GB per month. A household adding 10 GB a year unpruned costs ≈ €0.50 a year. The demo boxes cannot calibrate this (0.5 GB and 0.0 GB used today), so 10 GB/year is an assumption, stated as one.
• The real cost is the quota (50–100 GB): at the quota offbox_fit.go refuses new pushes and the over-quota forget (K3) would itself be refused, so the household's off-site copy STOPS. Crash leftovers (F8) and refused-prune index files (F2) also accumulate.
• Good as an interim, not an end state.

4. Recommendation

Option 1, with Option 3 as the interim while it is built. Order:

1. §2 first (key registrar + the authorized_keys check) — without it nothing below protects anything.
2. Switch every box to the forced key with no box-side retention (Option 3 interim). Quotas today are 50–100 GB against ≤0.5 GB used, so this costs nothing for months.
3. Then the weekly window (Option 1) with the poisoning guard.

Why not Option 2: it trades a box-level risk for a custody change that lets one Felhom machine read every household's backups, and it adds an always-on service to the recovery path.

5. Migration, rotation, restore

• Existing repos (both demo boxes, any household): only the authorized_keys line and the box's transport change; the repository bytes are not touched. Not measured: reading a repo written over sftp: through rclone: — restic's on-disk layout is backend-independent, so it is expected to work; measure on the scratch account before any household. History is kept.
• Key rotation: the hub writes the new forced line, the box switches, the hub removes the old line — the same registrar path.
• Restore works through the add-only key (F1) — no second key is needed for the household.
• Locks: the self-heal (resticStep → unlock --remove-all) keeps working under this transport (F7). R-430's fear does not apply to it.
• Failure speed: with today's code a forced key makes each night's forget --prune fail after ~45 s per refused file and grow the index (F2) — the two forget sites must leave the box in the same change that switches the key.