felhom.eu/documentation/audits/SPIKE-ep0-storagebox-2026-07-09.md

# SPIKE — ep0 ↔ Hetzner Storage Box (SSHFS mount + PBS-datastore feasibility)

**Date:** 2026-07-09  **Executor:** Claude Code (live on ep0)  **Class:** operational + spike
**Scope boundary honoured:** synthetic test data only; no production datastore promoted, no real
backup job pointed at the mount, no customer data touched. All PBS scratch artifacts torn down.

---

## VERDICT

**Direct PBS datastore on the SSHFS mount: UNFIT (hard blocker).**
**PBS datastore via a loopback ext4 image on the SSHFS mount: FIT — with caveats.**
**Raw SSHFS mount as a durable file target (restic/borg/rsync/plain files): FIT.**

The Storage Box is now permanently mounted on ep0 at `/mnt/pbs-storagebox` (durable systemd unit,
key-auth, passwordless, reboot-ordered). The mount itself is fast and reliable. The nuance is
entirely about **how PBS wants to own its chunk store** — see below.

---

## 0. Verified inputs (confirmed live)

| Fact | Value |
|------|-------|
| Storage Box | BX11 `PBS-storage-1`, 1 TB, €4.06/mo |
| Host | `u629193.your-storagebox.de` (resolves **AAAA-only**: `2a01:4f8:2b01:d2d::2`) |
| Username / SSH port | `u629193` / **23** |
| ep0 | `167.233.158.164`, Debian 13 (trixie), kernel 6.12, **eu-central** (same zone as box) |
| ep0 PBS | `proxmox-backup-server 4.2.2-1` **already installed & in production** |
| Existing datastores on ep0 | `felhom-offsite` (`/srv/pbs-felhom`, prod offsite tier) + `scratch` — **untouched** |
| ep0 local free (`/`) | 29 GB of 38 GB (`/dev/sda1`) |
| Box free | 1.0 TB (1% used) |

**Reachability:** port 23 was **OPEN from ep0 over IPv6** with no console change — the runbook's
"External Reachability still OFF" concern did **not** block ep0→box. (The earlier DooPlex/Windows
cloud SSH keys were irrelevant; see §Auth.) The runbook's "this sandbox cannot reach ep0" caveat
was also stale — ep0 was reachable throughout.

## 1. Auth — how the key actually got onto the box

- Hetzner **Cloud-project SSH keys** (console → Security → SSH keys) do **NOT** authorize a Storage
  Box. The box keeps its own `authorized_keys`. Adding `ep0-storagebox` at the project level was
  proven inert: key auth to the box returned `Permission denied (publickey,password)`.
- A dedicated **ed25519** key was generated on ep0 (`/root/.ssh/storagebox_ed25519`, 0600,
  fingerprint `SHA256:HlIJl/WwDutWKVaxhPhdvwAGyMwhh4ztspqKZAgpIlc`).
- Installed to the box's `authorized_keys` via **`ssh-copy-id -p 23 -s -i …`**. The **`-s` (SFTP)
  flag is mandatory** on Hetzner Storage Boxes — the normal shell-append method is rejected, and
  `-i` is required because the key filename is non-default.
- Box password: set once by the operator in the console, used only for that one `ssh-copy-id`, and
  never entered this session. Key/cred material referenced here is **stored 0600 on ep0**, not in
  this doc. Post-install auth is fully passwordless.

## 2. The durable mount (the permanent deliverable)

systemd unit `/etc/systemd/system/mnt-pbs\x2dstoragebox.mount`, **enabled** (WantedBy
multi-user.target):

```
[Mount]
What=u629193@u629193.your-storagebox.de:/home
Where=/mnt/pbs-storagebox
Type=fuse.sshfs
Options=port=23,IdentityFile=/root/.ssh/storagebox_ed25519,idmap=user,uid=34,gid=34,
        allow_other,default_permissions,reconnect,ServerAliveInterval=15,
        ServerAliveCountMax=3,StrictHostKeyChecking=yes,_netdev
```

Non-obvious choices that were forced by testing, not guessed:
- **`:/home`, not `:/`** — the box root `/` is `dr-x--x--x root` (read-only to us); writes there
  fail `Operation not permitted`. Our writable area is `/home` (owned by `u629193`).
- **`uid=34,gid=34`** — PBS datastore I/O runs as the `backup` user (uid 34). The mount is mapped so
  everything appears owned by `backup`, otherwise PBS can't write.
- **`user_allow_other`** was added to `/etc/fuse.conf` so `allow_other` is honoured.
- `reconnect` + `ServerAliveInterval=15`/`CountMax=3` → survive transient drops; `_netdev` +
  `After=network-online.target` → correct boot ordering; `StrictHostKeyChecking=yes` → box host key
  pinned in `/root/.ssh/known_hosts` (MITM protection on the tunnel).

Raw-mount fitness (before any PBS layer):
- Write-integrity round-trip (10 MB urandom, sha256): **OK**.
- Sequential throughput: **~327 MB/s write**, **~69 MB/s read** (256 MiB, intra-Hetzner IPv6).

## 3. The hard blocker — why a *direct* PBS datastore fails

`proxmox-backup-manager datastore create … /mnt/pbs-storagebox/…` fails immediately:

```
TASK ERROR: unable to create chunk store 'spike-sb' … EACCES: Permission denied
```

Root cause isolated by reproducing PBS's steps by hand as the `backup` user: `mkdir`, nested
`mkdir`, `chmod`, file create, and directory `fsync` **all succeed** — but **`chown` is rejected by
the Storage Box for everyone, even same-owner, even as root** (`Permission denied`). Hetzner Storage
Box SSHFS is single-user SFTP with no `chown` support. PBS's `ChunkStore::create` chowns the new
chunk store to `backup:backup`, so datastore creation can never complete directly on the mount. This
is architectural, not tunable.

## 4. The workaround that works — loopback ext4 on the SSHFS mount

Put a real filesystem *inside a file* on the box; PBS then owns a normal ext4 where `chown` works.

```
truncate -s 20G /mnt/pbs-storagebox/spike-sb.img
mkfs.ext4 -F -E lazy_itable_init=1,lazy_journal_init=1 spike-sb.img   # instant (sparse+lazy)
mount -o loop spike-sb.img /mnt/pbs-sb-loop
# chown inside the ext4: OK  → datastore create: OK
```

### Measurements (throwaway `spike-sb` datastore, synthetic 750 MB known-SHA dataset, ×2 backups)

| Operation | Result |
|-----------|--------|
| Datastore create (65 536 chunk dirs) | **6 s** |
| Backup #1 (750 MB, cold) | **10.1 s** (~75 MB/s end-to-end through PBS→ext4→loop→sshfs→box) |
| Backup #2 (identical → dedup) | **1.8 s**, dedup factor **2.0** |
| **Garbage collection** (the flagged risk) | **7 s — completes, NO hang, `TASK OK`** |
| Verify (all snapshots) | **12 s, 0 errors** |
| Restore round-trip (750 MB → sha256 vs source) | **11 s, integrity OK** |
| GC after forget-all | completes in 10 s; `Removed chunks: 0` = **expected** (PBS ~24 h chunk grace period, not a mount fault) |

**The research's headline fear — GC latency/hangs on a high-latency mount — did not materialize**
here, because intra-Hetzner IPv6 latency is low and the loopback ext4 turns PBS's many small
metadata ops into local-fs ops that flush to the box in batches.

### Caveats on the loopback path (why "fit-*with-caveats*")
1. **Crash/disconnect integrity risk.** ext4-over-loopback-over-sshfs adds a journaled FS on top of a
   network file. A hard disconnect mid-write risks ext4 corruption of the whole image (vs. losing
   one chunk). `reconnect` mitigates transient drops but is not a durability guarantee. A real
   deployment needs: periodic `fsck` discipline, and treating the image as **replaceable** (PBS
   `verify` + a second copy elsewhere), never as the sole custodian of data.
2. **Fixed image size.** ext4 image is provisioned up-front (sparse, so it only consumes actual
   used bytes on the box, but the FS cap is fixed until grown). Growing = `truncate` + `resize2fs`.
3. **Single-mounter.** The loop image must be mounted by exactly one host; no concurrent access.
4. **Boot ordering.** Two layers to bring up in order at boot (sshfs mount → loop mount → PBS). The
   **sshfs layer is now reboot-proven** (§5); the **loop-mount + PBS auto-bring-up layering is still
   unproven at boot** — the loop image + a `.mount`/`.service` ordered after the sshfs unit would
   need adding and a reboot to confirm before any real loopback deployment.

## 5. Open / not-yet-done

- **Reboot-survival (Phase 3e) — DONE, PASSED (operator-approved reboot, 2026-07-09).** ep0 was
  rebooted; the SSHFS mount **auto-mounted at boot** (unit `active`+`enabled`), a post-boot write as
  the `backup` user succeeded, WG reconnected (handshake within 19 s), and `proxmox-backup-proxy`
  came back `active`. Boot id changed `40e87037… → 8dc83d82…`; downtime ~30–60 s.
  Still unproven at boot: the **loop-mount + PBS auto-bring-up** layering (the loop image was torn
  down before the reboot), which a real loopback deployment must add and re-test.
- **Cache-disk constraint (for the PBS-S3 alternative, if ever chosen):** PBS S3 datastores want a
  64–128 GB **local** cache. ep0's `/` has only **29 GB free** — the S3 path would need a bigger ep0
  disk first. Irrelevant to the SSHFS/loopback path, which needs no such local cache.

## 6. Decision this unblocks

Three offsite-tier options, now measured enough to choose between:

1. **SSHFS + loopback-ext4 PBS datastore on the 1 TB Storage Box** — cheapest (€4.06/mo, 1 TB),
   proven functional here (GC/verify/restore all green), but carries the loopback-integrity caveat.
   Best if the Storage Box is a *secondary* copy behind the existing `felhom-offsite` tier, not the
   only one.
2. **PBS-S3 on Hetzner Object Storage** — the separate future spike; needs the 64–128 GB local cache
   (→ grow ep0's disk first) and has the S3-refresh datastore-block window to measure. Cleaner
   integrity story (no loopback FS), different cost model.
3. **Promote ep0 cache-disk first**, then decide — only needed for option 2.

Recommendation for the immediate next step: if the Storage Box is wanted as a **cheap second offsite
copy**, adopt option 1 with the loopback image treated as replaceable and guarded by scheduled
`verify`. If it's meant to be a **primary** offsite datastore, do the PBS-S3 spike (option 2) before
committing, because the loopback-integrity caveat is not ideal for a sole custodian.

**Not a GO-LIVE-PACKAGE item.** This is the offsite arc, separate from the Peti pilot (D3). No
go-live flag flipped.

## 7. Teardown state (left clean)

- Throwaway `spike-sb` datastore removed; only `felhom-offsite` + `scratch` remain.
- Loop image unmounted and `spike-sb.img` deleted; box `/home` back to just `.ssh` (384 KB used).
- All scratch API tokens (`spiketok*`) deleted; scratch ACL removed.
- **Kept (durable deliverables):** the `/mnt/pbs-storagebox` SSHFS mount unit (enabled) and the
  `storagebox_ed25519` key on ep0.