felhom.eu/documentation/runbooks/provider-questions-2026-09-01.md

# Two questions for Hetzner — drafted, ready to send (2026-09-01)

**These are the only thing standing between us and finishing R-95 properly.** They cost nothing and
they are answerable by a support agent without escalation.

**Send them yourself.** CC drafted them and did not send them, and did not call the provider API —
`§11-D` is still the operator's fence. **No credential, password or token appears below, and none
should be added.** The account id and the product name are all either question needs.

**What to fill in:** the ticket needs the Storage Box account. Ours is **`u629488`** (the box the
register calls `storage-box-pool-1`, plan BX11). Nothing else.

**Why two separate tickets:** they go to different parts of the answer — one is about the snapshot
product, one is about the SSH endpoint's configuration — and a single ticket asking both tends to get
one answered and the other dropped.

---

## ⚠ READ THIS BEFORE YOU SEND, AND BEFORE YOU BELIEVE AN ANSWER: two Hetzner products, opposite answers

**Hetzner has two similarly-named storage products and their documentation says OPPOSITE things about
restoring a single file. We are a Storage BOX. The answer that says "no" belongs to the other one.**

| | **Storage Box** — *what we have* | **Storage Share** — *not us* |
|---|---|---|
| what it is | plain SFTP/SSH storage, sub-accounts, `u629488` | a managed **Nextcloud** instance |
| single file out of a snapshot | *"You can download individual files or entire directories as usual"* — `docs.hetzner.com/storage/storage-box/snapshots/` | *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* — `docs.hetzner.com/storage/storage-share/faq/backup-snapshot/` |

**How to tell the Storage Share page apart at a glance:** it talks about *Nextcloud's data cache*, a
*database dump*, and the *konsoleH* web interface, and it never mentions Storage Box.

**Why this is load-bearing rather than trivia.** A search for "Hetzner restore individual files from
snapshot" surfaces the Storage **Share** FAQ, and it reads like a definitive "no". **A support agent
answering Question 1 could reasonably reply from that page and give us a wrong answer** — one that
would send R-95 to the top of the register for no reason. Question 1 below therefore names the
product, quotes the Storage Box line, and asks about the **main account** specifically.
**If an answer comes back citing full-snapshot-only, check which product it is about before acting on
it.** Caught 2026-09-01 by the operator noticing the URL said `storage-share`.

---

## Question 1 — can the MAIN account retrieve individual files from a snapshot?

**Why it matters, in one line:** if it cannot, the only route back is a whole-box rollback that hits
every customer on the box and destroys every newer snapshot — which would mean the snapshots protect
almost nobody in practice. **This is the question that decides how urgent R-95 is.**

> **Subject:** Storage Box u629488 — retrieving individual files from a snapshot
>
> Hello,
>
> We use Storage Box `u629488` with sub-accounts, and daily automatic snapshots are enabled.
>
> We can reach `/.zfs/snapshot` from a sub-account, but it lists as empty, and no snapshot name we
> try can be entered. We understand sub-accounts may be restricted here.
>
> Your Storage Box documentation says, under Snapshots: *"You can download individual files or
> entire directories as usual."*
>
> Our question is about the **main account**: from the main account, over SSH or SFTP on port 23,
> can we **read or download individual files and directories out of a specific snapshot** — for
> example a single directory under one sub-account's home — **without** performing a snapshot
> restore of the whole Storage Box?
>
> To be clear, this question is about a **Storage Box**, not about Storage Share. We are aware the
> Storage Share FAQ says only full-snapshot restores are supported; we are asking whether that also
> applies to Storage Box, because the Storage Box snapshot documentation appears to say the
> opposite.
>
> If yes, please tell us the exact path we should use and how the snapshot directory is named.
>
> If no, please confirm that the only way to get data out of a snapshot is the full "restore
> snapshot" action on the whole Storage Box.
>
> Thank you.

**How to read the answer.**
* **"Yes, from the main account"** → per-file recovery exists, but it is an operator act in a
  browser or over the main account's own SSH, and it can never be something the product does for the
  customer. R-433 closes at that. R-95 stays where it is.
* **"No, only a full restore"** → the snapshots do **not** bound a single customer's exposure at all,
  because using them costs every other customer on the box their newer snapshots. **R-95 becomes
  urgent and the transport change stops being optional.**

---

## Question 2 — is `--append-only` enforced on the `rclone serve restic` endpoint?

**Why it matters, in one line:** if it is enforced server-side, a compromised box **cannot delete its
own backups**, with no new machine and no data migration. **This is the question that could make R-95
disappear.**

> **Subject:** Storage Box u629488 — rclone serve restic endpoint and --append-only
>
> Hello,
>
> Your Storage Box documentation states: *"Restic is natively supported with the SFTP backend. As
> another option, we support the restic backend, which is provided by Rclone over SSH."* The
> restricted SSH shell on Storage Box `u629488` (port 23) also lists `rclone serve restic --stdio`
> among the available server-side backends.
>
> Our question is about how that command is run on your side: **is `--append-only` enforced by you,
> or is the command line taken from what the client sends?**
>
> In other words, if a client connects and asks for `rclone serve restic --stdio` **without**
> `--append-only`, does it get a server that permits deletions?
>
> If the flag can be enforced, is there any way for us to request that for this account or for
> individual sub-accounts?
>
> Thank you.

**How to read the answer.**
* **"Enforced server-side" or "can be enabled per account"** → this is the cheap prevention the
  2026-09-01 spike priced at a new always-on service plus either a mount in the hot path or migrating
  every customer's history. **It needs none of that** — the server already runs at the provider, and
  restic 0.14.0 already speaks the `rclone:` backend (measured, with a control:
  `banana:` → `invalid backend`, `rclone:` → the helper was executed). The remaining work is putting
  `rclone` in the controller image and switching the repository URL. **R-95's root cause goes away.**
* **"The client supplies the command line"** → **the lead is worth nothing** and should be recorded
  as dead, not left looking promising. Prevention then still needs a machine in front of the store,
  and the decision reverts to the spike's option 3 at its original price.

---

## Where these came from

* **R-433** — no snapshot is reachable from a sub-account by any name. 777,600 exact names in the
  vendor's `YYYY-MM-DDTHH-MM-SS` format over nine days, zero hits, with a passing control; `/home`
  and `/.zfs` are different filesystems and `/home/.zfs` does not exist. Question 1 exists because
  that measurement can only speak for a sub-account.
* **R-436** — the `rclone serve restic --stdio` backend and restic's `rclone:` support, both
  measured, and since **confirmed as officially supported** by the vendor's own Storage Box access
  page (*"we support the restic backend, which is provided by Rclone over SSH"*). **That page says
  nothing at all about append-only**, in either its Rclone or its Restic section — so Question 2 is
  genuinely unanswered by the documentation and is not a question the docs could have saved us.
* **The two-product trap** above, caught 2026-09-01. The Storage Box command table on that same
  access page also matches, item for item, the `help` output measured live on our own sub-account —
  independent corroboration that we were reading the right product's surface.
* Evidence for both: `documentation/audits/evidence-drill-r95-recovery-2026-09-01/`.