felhom.eu/documentation/audits/SPIKE-r95-offsite-delete-2026-09-01.md

SPIKE — can the box be stopped from deleting its own off-site history? (R-95, 2026-09-01)

Read-only study. No production code, no version bump, no image, no golden. No delete verb was issued against any live store. ep0, DooPlex and Peti's box were not touched at all.

---

Q1 — Is the safety net real? NO — and worse than "no": it cannot be seen from the box at all, so the exposure must be treated as UNBOUNDED until Viktor looks at the control panel.

Measured on both live boxes, over the SFTP credential each box already holds, read-only.

probe	demo-hp (sub-account A, gid 1058)	demo-felhom (gid 1019)
positive control — the account's own home	lists .ssh, <repo>	lists .ssh, <repo>, <repo>.orphaned-20260810
negative control — ./zzz-no-such-r95	not found	not found
./.snapshots	not found	not found
repo path itself	config data index keys locks snapshots	data index keys locks snapshots
<repo>/.snapshots	not found	not found
/ (box root)	Permission denied — the account is jailed to /home	same

Two readings, and the box cannot distinguish them: either no snapshots exist, or they exist and are invisible to a sub-account. Both are bad, and the second is not a reprieve — a snapshot the box cannot see is a snapshot the box cannot restore from either. Recovery would be an operator act through the Hetzner panel, not something the product can do.

What the register actually claims, OPEN-ITEMS.md:233, verbatim:

| — | Storage Box **snapshots** on storage-box-pool-1 — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm size_snapshots > 0 tomorrow; ... until then the mitigation is armed, not proven |

Three facts about that row:

1. It has no R-number (| — |), so no gate and no grep-the-register rule can ever cite it.
2. Its "confirm tomorrow" was 2026-07-27 (git log -S, last touched in 72692e1). That is 36 days ago. Nobody confirmed.
3. The DUE-CHECKS block — the mechanism this project built (R-341) for exactly this — is EMPTY. The one dated commitment that mattered was never entered into it.

And R-95 leans on it: its own text reads "Mitigation now ARMED". That word is doing work it has not earned. → R-429.

Not answerable here: whether snapshots exist. The register's own confirming field, size_snapshots, is a Hetzner API field, and §11-D records the deliberate decision not to call that API with the production token. This spike stopped at that fence. → a question for Viktor.

Explicitly NOT done: no restore from a snapshot was attempted. Whether one can be restored is a separate question and is named as such.

What this does to the urgency: it raises it. The task supposed Q1 might bound the worst case to seven days. It does not. The bound is unverified, and unverifiable from the product side.

---

Q2 — What must the box write, and what must it delete? Ten verbs, not nine — and there are TWO forget --prune sites, not one.

Census by grep over internal/backup/*.go at 960d29b, each confirmed by reading its context.

verb	file:line	class
init	offbox.go:321, offbox.go:811	write
backup	offbox.go:927, offbox.go:1332, offbox_shares.go:107	write
forget … --prune	offbox.go:1388 (retention) and offbox.go:1759 (over-quota)	DELETE
unlock	offbox.go:746 (stale-only), offbox.go:768 (--remove-all)	DELETE
restore	offbox.go:1827, offbox_restore.go:393, offbox_proof.go:328 (--no-lock)	read
snapshots	offbox.go:1771, offbox_inventory.go:73, offbox_restore.go:126	read
stats	offbox.go:1786, offbox_restore.go:160	read (takes a lock)
cat	offbox.go:782	read
check	offbox_integrity.go:316	read (takes a lock)

Two corrections to the task's list, both mine to report:

• check is missing from it. It is a real off-site verb (offbox_integrity.go:316) and it is lock-taking, which makes it central to Q5.
• dump is NOT a verb. offbox_progress.go:185 is OffboxPhaseDump = "dump", a progress phase name. I withdrew it after reading the context rather than counting it.

Does the box need the delete verbs? forget/prune — no, that is retention and it is exactly what R-89 moved off the box for PBS. unlock — this is the whole of Q5.

---

Q3 — Can the sub-account express "write but never delete"? No. The API offers exactly one axis, and it is all-or-nothing.

DOCUMENTED, not measured — from this project's own mirror of the API, hub/internal/hetznerapi/hetznerapi.go:38-45, which is the authority we have without calling the provider:

type AccessSettings struct {
    SSHEnabled          bool `json:"ssh_enabled"`
    ReachableExternally bool `json:"reachable_externally"`
    SambaEnabled        bool `json:"samba_enabled"`
    WebDAVEnabled       bool `json:"webdav_enabled"`
    Readonly            bool `json:"readonly"`
}

Five booleans. readonly is the only permission axis, and a backup target cannot be read-only. There is no append-only, no per-directory grant, no write-without-unlink. This is the finding that forces Q6, and the register already suspected it.

Measured corroboration, no new action taken: demo-felhom's home contains a <repo>.orphaned-20260810 directory beside <repo>. That directory was renamed by the controller (the orphan guard). A rename is a delete-class operation. So the credential's reach is not a theory — the existing state on the live store is evidence of it.

Not established: whether the live API would report anything the struct omits. Settling it needs the provider token — fenced (§11-D).

---

Q4 — What did the PBS fix do, and what would copying it cost? The shape does not transfer, because the far end is a disk, not a server.

What moved (R-89, PROVEN-LIVE): retention became a hub-owned commercial attribute; boxes set keep_last: 0; ep0 runs the prune jobs; box tokens stay write-only. 07 §8 row 10 records the box being refused when deleting its own PBS snapshot.

Why it does not transfer as-is: PBS is a server that can refuse. The restic tier writes to a Storage Box over SFTP — a filesystem. A filesystem runs nothing. So "move retention to the far end" has no far end to move it to. Candidates, with their costs:

who runs retention instead	cost	new risk it creates
the hub	a scheduler + a per-customer credential	the hub gains a credential that can delete every customer's history — one compromise instead of N
DooPlex	a cron + the same credentials	same concentration, on a Tier-2 box that is itself the recovery chain
ep0	a service on a protected machine	an architecture change, not a config change; ep0 is protected
nobody — never prune	£0 today	the store grows without bound; the over-quota path at offbox.go:1759 exists precisely because quota is already a live concern

What the migration cost last time, and the trap is identical here. R-191: R-89 changed the contract, one client-side retention setting did not follow, and every successful weekly backup then reported as a failure — upload complete, 67.2 % reused, then TASK ERROR: job errors and a whole_guest_backup_failed alarm to the operator. "A doc that states the contract does not enforce it — the gate does."

The same trap, doubled: withdrawing delete without disarming retention would make every off-site run log [WARN] forget --prune failed — and there are two call sites, offbox.go:1388 and offbox.go:1759. The task's brief names only the first. Disarming one and not the other reproduces R-191 exactly.

---

Q5 — The lock problem. Measured, and it is NOT the blocker I expected — but it exposed something worse: unlock reports success on a deletion that did not happen.

Method. A throwaway local restic repo in the controller container's /tmp on demo-hp (60 MB, removed at the end, no live store touched). Append-only was modelled faithfully: a sticky locks directory (1777) owned by root, holding a root-owned lock, with restic run as nobody. That is exactly append-only semantics — create allowed, delete refused.

Both controls passed before anything was believed:

• nobody can create in the locks directory → the model is not simply "read-only".
• nobody cannot delete root's lock → the model really does refuse deletes.

A genuine restic lock was captured (copied out while a real check --read-data held it), not hand-forged.

verb, running as nobody against an undeletable stale lock	result
restic unlock --remove-all	prints successfully removed locks — and the lock is STILL THERE
restic backup	succeeds — snapshot a7928829 saved
restic check	succeeds — no errors were found
restic snapshots --no-lock	succeeds
restic restore --no-lock (the R-87 proof pattern)	succeeds

So the answer to "does the store wedge?" is: not in this case. restic 0.14.0 treats a lock whose owner is provably dead as stale and proceeds without needing to remove it. Withdrawing delete does not, by itself, wedge the store. That removes the constraint the task suspected would be deciding.

But the measurement found a different defect. unlock --remove-all reported success while deleting nothing. This project already has a named class for that — "Exit codes that lie" — and resticStep's crash-lock self-heal is built directly on top of this call. A self-heal that cannot fail is a self-heal that cannot be trusted. → R-430.

UNKNOWN, and I am not claiming otherwise: the crash-lock case — a lock left by a container that no longer exists, whose hostname restic cannot match, so it will not treat it as stale for ~30 minutes. That is documented in resticStep's own comment (offbox.go:~760) and is the reason --remove-all exists at all. My model could not reproduce it: the captured lock carried this container's own hostname. What would settle it: a lock captured from a container with a different hostname, replayed against an append-only endpoint. In that window, the only remedy is the very call that Q5 just showed reports success while doing nothing.

How far --no-lock reaches: restic's own help says it "allows some operations on read-only repositories". Measured: snapshots and restore work under it. backup and check still take a lock — but creating a lock is a write, which append-only permits. So the read paths are safe and the write paths are unaffected; only removal is denied, and only the crash-lock window depends on removal.

---

Q6 — Is an append-only transport reachable? Yes in principle — restic 0.14.0 does speak REST — but not without either moving the data or putting a machine in front of it.

Measured, in the controller container, with a control:

repo string	result
banana:http://127.0.0.1:1/x (control)	Fatal: parsing repository location failed: invalid backend
rest:http://127.0.0.1:1/	Fatal: unable to open config file: … dial tcp 127.0.0.1:1: connect: connection refused

A connection error, not a parse error — the REST backend IS recognised by restic 0.14.0.

Append-only is not a restic feature. Measured: restic help and restic backup --help contain zero occurrences of "append". It is a rest-server flag (--append-only). So the client is ready and the server does not exist yet.

In front, or move the data? rest-server serves a local directory. The Storage Box is a remote share. So either (a) a machine mounts the Storage Box and runs rest-server over that mount — keeping the bytes where they are, at the cost of a mount in the hot path — or (b) the data moves to storage attached to whatever runs rest-server. These are very different prices and the choice is not obvious; (a) keeps the current bill, (b) is a migration of every customer's history.

Where would it run? ep0 is protected. Adding a service to it is an architecture change, not a configuration change — and it also makes ep0 a single point of failure for both tiers, which is precisely the concern §11-D already has open.

Cost: one new always-on service in the recovery path, plus whatever machine hosts it. The moving part matters more than the money: if rest-server is down, backups stop — and this project's own history (R-191) is a warning about what happens when a change in the off-site contract is not carried everywhere at once.

---

Q7 — What does DETECTION cost? Almost nothing. The number already arrives at the hub, and the hub already keeps the history to compare it against.

Measured from source:

• The box already reports snapshot_count — controller/internal/report/types.go:131, and it is already rendered on the hub's Backup card (hub/internal/web/backup_card.go:118).
• The hub retains report history: store.go:968 is an INSERT INTO reports, and the read is SELECT … ORDER BY id DESC LIMIT 1 (store.go:1016). It appends; it does not replace. So every previous snapshot_count for every customer is already on disk.

So an unexplained drop is a comparison between two rows the hub already has. No new box code, no new credential, no new moving part, no new bill. The alarm vocabulary and the operator-only routing already exist.

The honest caveats: a legitimate forget also drops the count, so the rule needs to know retention (and today the box prunes itself, which is what makes the number noisy — the same change that disarms box-side retention is what makes this signal clean). And detection is not prevention: it tells you within a day that history was destroyed; it does not stop it. snapshot_count: 0 must mean UNKNOWN, not EMPTY — backup_card.go:29 records that exact defect being fixed once already.

---

Ranked options for Viktor

All four were considered. My recommendation is 4 + 2, in that order, and explicitly not 3 yet.

1. Do nothing — NOT acceptable as it stands, and Q1 is why

Before this spike, "do nothing" meant "exposed, but bounded to seven days by snapshots." That sentence is not supported by any evidence. The box cannot see a single snapshot, on either machine, and the register's confirmation step was never done. Doing nothing now means an unbounded exposure on the tier holding the customer's documents and photos. Cost: £0, no evenings. What happens if you choose it: the risk stays exactly where it is, and the register keeps saying "ARMED", which is the part I would not accept. One cheap act rescues most of this option: look at the Storage Box panel and answer Q1. If snapshots are real, "do nothing" becomes defensible again. That is ten minutes and it is yours to do — the fence in §11-D stopped me.

2. Copy the PBS shape — worth doing, but it is smaller than it sounds and it is not the safety it appears to be

Withdraw box-side retention; someone else prunes. Q5 says this no longer looks dangerous — a stale lock does not wedge the store. But Q3 says the credential still cannot be narrowed: the API has one axis, readonly, and a backup target cannot be read-only. So the box would keep the ability to delete, and simply stop using it. That is a discipline, not a guarantee — a compromised guest is unaffected by it. Cost: one evening, plus wherever retention moves (each candidate concentrates the credential). Trap: both forget sites (offbox.go:1388 and :1759) must be disarmed in the same change, or you get R-191 again — successful backups reported as failures.

3. Change the transport (rest-server --append-only) — the only real prevention, and I would not start it yet

Q6 says it is reachable: restic 0.14.0 speaks REST. This is the option that actually makes the box unable to delete. Cost: a new always-on service in the recovery path; a machine to host it (ep0 is protected — this is an architecture change); and either a mount in the hot path or moving every customer's history. Money is the small part. Why not yet: it should be chosen against a measured picture, and one measurement is still missing — Q1. Building the expensive prevention while nobody knows whether a seven-day net already exists is the wrong order.

4. Detect instead of prevent — cheapest by a wide margin, and I would do this first

Q7 measured that the material is already there: the count is on the wire, the hub keeps the history, the alarm path exists. An unexplained drop in snapshot_count becomes noticeable within a day. Cost: hub-side only. No box change, no credential change, no new service, no new bill. What it does not do: it does not stop the deletion. It converts "we would never know" into "we know tomorrow" — and against ransomware inside the guest, knowing tomorrow is the difference between losing a day and losing everything silently. This week's pattern held twice already (R-87, R-404): proving beats preventing when preventing is expensive. This is the third instance.

My pick: answer Q1 today (yours, ten minutes), then build 4, then 2. Revisit 3 once Q1 is answered — and if Q1 comes back "no snapshots", 3 moves up sharply.