felhom.eu/documentation/audits/SPIKE-ep0-storagebox-2026-07-09.md

SPIKE — ep0 ↔ Hetzner Storage Box (SSHFS mount + PBS-datastore feasibility)

Date: 2026-07-09 Executor: Claude Code (live on ep0) Class: operational + spike Scope boundary honoured: synthetic test data only; no production datastore promoted, no real backup job pointed at the mount, no customer data touched. All PBS scratch artifacts torn down.

---

VERDICT

Direct PBS datastore on the SSHFS mount: UNFIT (hard blocker). PBS datastore via a loopback ext4 image on the SSHFS mount: FIT — with caveats. Raw SSHFS mount as a durable file target (restic/borg/rsync/plain files): FIT.

The Storage Box is now permanently mounted on ep0 at /mnt/pbs-storagebox (durable systemd unit, key-auth, passwordless, reboot-ordered). The mount itself is fast and reliable. The nuance is entirely about how PBS wants to own its chunk store — see below.

---

0. Verified inputs (confirmed live)

Fact	Value
Storage Box	BX11 PBS-storage-1, 1 TB, €4.06/mo
Host	u629193.your-storagebox.de (resolves AAAA-only: 2a01:4f8:2b01:d2d::2)
Username / SSH port	u629193 / 23
ep0	167.233.158.164, Debian 13 (trixie), kernel 6.12, eu-central (same zone as box)
ep0 PBS	proxmox-backup-server 4.2.2-1 already installed & in production
Existing datastores on ep0	felhom-offsite (/srv/pbs-felhom, prod offsite tier) + scratch — untouched
ep0 local free (/)	29 GB of 38 GB (/dev/sda1)
Box free	1.0 TB (1% used)

Reachability: port 23 was OPEN from ep0 over IPv6 with no console change — the runbook's "External Reachability still OFF" concern did not block ep0→box. (The earlier DooPlex/Windows cloud SSH keys were irrelevant; see §Auth.) The runbook's "this sandbox cannot reach ep0" caveat was also stale — ep0 was reachable throughout.

1. Auth — how the key actually got onto the box

• Hetzner Cloud-project SSH keys (console → Security → SSH keys) do NOT authorize a Storage Box. The box keeps its own authorized_keys. Adding ep0-storagebox at the project level was proven inert: key auth to the box returned Permission denied (publickey,password).
• A dedicated ed25519 key was generated on ep0 (/root/.ssh/storagebox_ed25519, 0600, fingerprint SHA256:HlIJl/WwDutWKVaxhPhdvwAGyMwhh4ztspqKZAgpIlc).
• Installed to the box's authorized_keys via ssh-copy-id -p 23 -s -i …. The -s (SFTP) flag is mandatory on Hetzner Storage Boxes — the normal shell-append method is rejected, and -i is required because the key filename is non-default.
• Box password: set once by the operator in the console, used only for that one ssh-copy-id, and never entered this session. Key/cred material referenced here is stored 0600 on ep0, not in this doc. Post-install auth is fully passwordless.

2. The durable mount (the permanent deliverable)

systemd unit /etc/systemd/system/mnt-pbs\x2dstoragebox.mount, enabled (WantedBy multi-user.target):

[Mount]
What=u629193@u629193.your-storagebox.de:/home
Where=/mnt/pbs-storagebox
Type=fuse.sshfs
Options=port=23,IdentityFile=/root/.ssh/storagebox_ed25519,idmap=user,uid=34,gid=34,
        allow_other,default_permissions,reconnect,ServerAliveInterval=15,
        ServerAliveCountMax=3,StrictHostKeyChecking=yes,_netdev

Non-obvious choices that were forced by testing, not guessed:

• :/home, not :/ — the box root / is dr-x--x--x root (read-only to us); writes there fail Operation not permitted. Our writable area is /home (owned by u629193).
• uid=34,gid=34 — PBS datastore I/O runs as the backup user (uid 34). The mount is mapped so everything appears owned by backup, otherwise PBS can't write.
• user_allow_other was added to /etc/fuse.conf so allow_other is honoured.
• reconnect + ServerAliveInterval=15/CountMax=3 → survive transient drops; _netdev + After=network-online.target → correct boot ordering; StrictHostKeyChecking=yes → box host key pinned in /root/.ssh/known_hosts (MITM protection on the tunnel).

Raw-mount fitness (before any PBS layer):

• Write-integrity round-trip (10 MB urandom, sha256): OK.
• Sequential throughput: ~327 MB/s write, ~69 MB/s read (256 MiB, intra-Hetzner IPv6).

3. The hard blocker — why a direct PBS datastore fails

proxmox-backup-manager datastore create … /mnt/pbs-storagebox/… fails immediately:

TASK ERROR: unable to create chunk store 'spike-sb' … EACCES: Permission denied

Root cause isolated by reproducing PBS's steps by hand as the backup user: mkdir, nested mkdir, chmod, file create, and directory fsync all succeed — but chown is rejected by the Storage Box for everyone, even same-owner, even as root (Permission denied). Hetzner Storage Box SSHFS is single-user SFTP with no chown support. PBS's ChunkStore::create chowns the new chunk store to backup:backup, so datastore creation can never complete directly on the mount. This is architectural, not tunable.

4. The workaround that works — loopback ext4 on the SSHFS mount

Put a real filesystem inside a file on the box; PBS then owns a normal ext4 where chown works.

truncate -s 20G /mnt/pbs-storagebox/spike-sb.img
mkfs.ext4 -F -E lazy_itable_init=1,lazy_journal_init=1 spike-sb.img   # instant (sparse+lazy)
mount -o loop spike-sb.img /mnt/pbs-sb-loop
# chown inside the ext4: OK  → datastore create: OK

Measurements (throwaway spike-sb datastore, synthetic 750 MB known-SHA dataset, ×2 backups)

Operation	Result
Datastore create (65 536 chunk dirs)	6 s
Backup #1 (750 MB, cold)	10.1 s (~75 MB/s end-to-end through PBS→ext4→loop→sshfs→box)
Backup #2 (identical → dedup)	1.8 s, dedup factor 2.0
Garbage collection (the flagged risk)	7 s — completes, NO hang, TASK OK
Verify (all snapshots)	12 s, 0 errors
Restore round-trip (750 MB → sha256 vs source)	11 s, integrity OK
GC after forget-all	completes in 10 s; Removed chunks: 0 = expected (PBS ~24 h chunk grace period, not a mount fault)

The research's headline fear — GC latency/hangs on a high-latency mount — did not materialize here, because intra-Hetzner IPv6 latency is low and the loopback ext4 turns PBS's many small metadata ops into local-fs ops that flush to the box in batches.

Caveats on the loopback path (why "fit-with-caveats")

1. Crash/disconnect integrity risk. ext4-over-loopback-over-sshfs adds a journaled FS on top of a network file. A hard disconnect mid-write risks ext4 corruption of the whole image (vs. losing one chunk). reconnect mitigates transient drops but is not a durability guarantee. A real deployment needs: periodic fsck discipline, and treating the image as replaceable (PBS verify + a second copy elsewhere), never as the sole custodian of data.
2. Fixed image size. ext4 image is provisioned up-front (sparse, so it only consumes actual used bytes on the box, but the FS cap is fixed until grown). Growing = truncate + resize2fs.
3. Single-mounter. The loop image must be mounted by exactly one host; no concurrent access.
4. Boot ordering. Two layers to bring up in order at boot (sshfs mount → loop mount → PBS). The sshfs layer is now reboot-proven (§5); the loop-mount + PBS auto-bring-up layering is still unproven at boot — the loop image + a .mount/.service ordered after the sshfs unit would need adding and a reboot to confirm before any real loopback deployment.

5. Open / not-yet-done

• Reboot-survival (Phase 3e) — DONE, PASSED (operator-approved reboot, 2026-07-09). ep0 was rebooted; the SSHFS mount auto-mounted at boot (unit active+enabled), a post-boot write as the backup user succeeded, WG reconnected (handshake within 19 s), and proxmox-backup-proxy came back active. Boot id changed 40e87037… → 8dc83d82…; downtime ~30–60 s. Still unproven at boot: the loop-mount + PBS auto-bring-up layering (the loop image was torn down before the reboot), which a real loopback deployment must add and re-test.
• Cache-disk constraint (for the PBS-S3 alternative, if ever chosen): PBS S3 datastores want a 64–128 GB local cache. ep0's / has only 29 GB free — the S3 path would need a bigger ep0 disk first. Irrelevant to the SSHFS/loopback path, which needs no such local cache.

6. Decision this unblocks

Three offsite-tier options, now measured enough to choose between:

1. SSHFS + loopback-ext4 PBS datastore on the 1 TB Storage Box — cheapest (€4.06/mo, 1 TB), proven functional here (GC/verify/restore all green), but carries the loopback-integrity caveat. Best if the Storage Box is a secondary copy behind the existing felhom-offsite tier, not the only one.
2. PBS-S3 on Hetzner Object Storage — the separate future spike; needs the 64–128 GB local cache (→ grow ep0's disk first) and has the S3-refresh datastore-block window to measure. Cleaner integrity story (no loopback FS), different cost model.
3. Promote ep0 cache-disk first, then decide — only needed for option 2.

Recommendation for the immediate next step: if the Storage Box is wanted as a cheap second offsite copy, adopt option 1 with the loopback image treated as replaceable and guarded by scheduled verify. If it's meant to be a primary offsite datastore, do the PBS-S3 spike (option 2) before committing, because the loopback-integrity caveat is not ideal for a sole custodian.

Not a GO-LIVE-PACKAGE item. This is the offsite arc, separate from the Peti pilot (D3). No go-live flag flipped.

7. Teardown state (left clean)

• Throwaway spike-sb datastore removed; only felhom-offsite + scratch remain.
• Loop image unmounted and spike-sb.img deleted; box /home back to just .ssh (384 KB used).
• All scratch API tokens (spiketok*) deleted; scratch ACL removed.
• Kept (durable deliverables): the /mnt/pbs-storagebox SSHFS mount unit (enabled) and the storagebox_ed25519 key on ep0.