docs(audit): ep0 Hetzner Storage Box SSHFS + PBS-datastore feasibility spike
Wired the 1TB BX11 Storage Box to ep0 as a durable key-auth SSHFS mount (/mnt/pbs-storagebox, reboot-proven). Findings: direct PBS datastore UNFIT (box rejects chown -> ChunkStore::create fails); loopback-ext4-on-SSHFS FIT with caveats (GC 7s/verify 12s/restore 11s all green, no GC hang). Raw mount ~327MB/s write, ~69MB/s read. Spike only, synthetic data, torn down. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -0,0 +1,177 @@
|
||||
# SPIKE — ep0 ↔ Hetzner Storage Box (SSHFS mount + PBS-datastore feasibility)
|
||||
|
||||
**Date:** 2026-07-09 **Executor:** Claude Code (live on ep0) **Class:** operational + spike
|
||||
**Scope boundary honoured:** synthetic test data only; no production datastore promoted, no real
|
||||
backup job pointed at the mount, no customer data touched. All PBS scratch artifacts torn down.
|
||||
|
||||
---
|
||||
|
||||
## VERDICT
|
||||
|
||||
**Direct PBS datastore on the SSHFS mount: UNFIT (hard blocker).**
|
||||
**PBS datastore via a loopback ext4 image on the SSHFS mount: FIT — with caveats.**
|
||||
**Raw SSHFS mount as a durable file target (restic/borg/rsync/plain files): FIT.**
|
||||
|
||||
The Storage Box is now permanently mounted on ep0 at `/mnt/pbs-storagebox` (durable systemd unit,
|
||||
key-auth, passwordless, reboot-ordered). The mount itself is fast and reliable. The nuance is
|
||||
entirely about **how PBS wants to own its chunk store** — see below.
|
||||
|
||||
---
|
||||
|
||||
## 0. Verified inputs (confirmed live)
|
||||
|
||||
| Fact | Value |
|
||||
|------|-------|
|
||||
| Storage Box | BX11 `PBS-storage-1`, 1 TB, €4.06/mo |
|
||||
| Host | `u629193.your-storagebox.de` (resolves **AAAA-only**: `2a01:4f8:2b01:d2d::2`) |
|
||||
| Username / SSH port | `u629193` / **23** |
|
||||
| ep0 | `167.233.158.164`, Debian 13 (trixie), kernel 6.12, **eu-central** (same zone as box) |
|
||||
| ep0 PBS | `proxmox-backup-server 4.2.2-1` **already installed & in production** |
|
||||
| Existing datastores on ep0 | `felhom-offsite` (`/srv/pbs-felhom`, prod offsite tier) + `scratch` — **untouched** |
|
||||
| ep0 local free (`/`) | 29 GB of 38 GB (`/dev/sda1`) |
|
||||
| Box free | 1.0 TB (1% used) |
|
||||
|
||||
**Reachability:** port 23 was **OPEN from ep0 over IPv6** with no console change — the runbook's
|
||||
"External Reachability still OFF" concern did **not** block ep0→box. (The earlier DooPlex/Windows
|
||||
cloud SSH keys were irrelevant; see §Auth.) The runbook's "this sandbox cannot reach ep0" caveat
|
||||
was also stale — ep0 was reachable throughout.
|
||||
|
||||
## 1. Auth — how the key actually got onto the box
|
||||
|
||||
- Hetzner **Cloud-project SSH keys** (console → Security → SSH keys) do **NOT** authorize a Storage
|
||||
Box. The box keeps its own `authorized_keys`. Adding `ep0-storagebox` at the project level was
|
||||
proven inert: key auth to the box returned `Permission denied (publickey,password)`.
|
||||
- A dedicated **ed25519** key was generated on ep0 (`/root/.ssh/storagebox_ed25519`, 0600,
|
||||
fingerprint `SHA256:HlIJl/WwDutWKVaxhPhdvwAGyMwhh4ztspqKZAgpIlc`).
|
||||
- Installed to the box's `authorized_keys` via **`ssh-copy-id -p 23 -s -i …`**. The **`-s` (SFTP)
|
||||
flag is mandatory** on Hetzner Storage Boxes — the normal shell-append method is rejected, and
|
||||
`-i` is required because the key filename is non-default.
|
||||
- Box password: set once by the operator in the console, used only for that one `ssh-copy-id`, and
|
||||
never entered this session. Key/cred material referenced here is **stored 0600 on ep0**, not in
|
||||
this doc. Post-install auth is fully passwordless.
|
||||
|
||||
## 2. The durable mount (the permanent deliverable)
|
||||
|
||||
systemd unit `/etc/systemd/system/mnt-pbs\x2dstoragebox.mount`, **enabled** (WantedBy
|
||||
multi-user.target):
|
||||
|
||||
```
|
||||
[Mount]
|
||||
What=u629193@u629193.your-storagebox.de:/home
|
||||
Where=/mnt/pbs-storagebox
|
||||
Type=fuse.sshfs
|
||||
Options=port=23,IdentityFile=/root/.ssh/storagebox_ed25519,idmap=user,uid=34,gid=34,
|
||||
allow_other,default_permissions,reconnect,ServerAliveInterval=15,
|
||||
ServerAliveCountMax=3,StrictHostKeyChecking=yes,_netdev
|
||||
```
|
||||
|
||||
Non-obvious choices that were forced by testing, not guessed:
|
||||
- **`:/home`, not `:/`** — the box root `/` is `dr-x--x--x root` (read-only to us); writes there
|
||||
fail `Operation not permitted`. Our writable area is `/home` (owned by `u629193`).
|
||||
- **`uid=34,gid=34`** — PBS datastore I/O runs as the `backup` user (uid 34). The mount is mapped so
|
||||
everything appears owned by `backup`, otherwise PBS can't write.
|
||||
- **`user_allow_other`** was added to `/etc/fuse.conf` so `allow_other` is honoured.
|
||||
- `reconnect` + `ServerAliveInterval=15`/`CountMax=3` → survive transient drops; `_netdev` +
|
||||
`After=network-online.target` → correct boot ordering; `StrictHostKeyChecking=yes` → box host key
|
||||
pinned in `/root/.ssh/known_hosts` (MITM protection on the tunnel).
|
||||
|
||||
Raw-mount fitness (before any PBS layer):
|
||||
- Write-integrity round-trip (10 MB urandom, sha256): **OK**.
|
||||
- Sequential throughput: **~327 MB/s write**, **~69 MB/s read** (256 MiB, intra-Hetzner IPv6).
|
||||
|
||||
## 3. The hard blocker — why a *direct* PBS datastore fails
|
||||
|
||||
`proxmox-backup-manager datastore create … /mnt/pbs-storagebox/…` fails immediately:
|
||||
|
||||
```
|
||||
TASK ERROR: unable to create chunk store 'spike-sb' … EACCES: Permission denied
|
||||
```
|
||||
|
||||
Root cause isolated by reproducing PBS's steps by hand as the `backup` user: `mkdir`, nested
|
||||
`mkdir`, `chmod`, file create, and directory `fsync` **all succeed** — but **`chown` is rejected by
|
||||
the Storage Box for everyone, even same-owner, even as root** (`Permission denied`). Hetzner Storage
|
||||
Box SSHFS is single-user SFTP with no `chown` support. PBS's `ChunkStore::create` chowns the new
|
||||
chunk store to `backup:backup`, so datastore creation can never complete directly on the mount. This
|
||||
is architectural, not tunable.
|
||||
|
||||
## 4. The workaround that works — loopback ext4 on the SSHFS mount
|
||||
|
||||
Put a real filesystem *inside a file* on the box; PBS then owns a normal ext4 where `chown` works.
|
||||
|
||||
```
|
||||
truncate -s 20G /mnt/pbs-storagebox/spike-sb.img
|
||||
mkfs.ext4 -F -E lazy_itable_init=1,lazy_journal_init=1 spike-sb.img # instant (sparse+lazy)
|
||||
mount -o loop spike-sb.img /mnt/pbs-sb-loop
|
||||
# chown inside the ext4: OK → datastore create: OK
|
||||
```
|
||||
|
||||
### Measurements (throwaway `spike-sb` datastore, synthetic 750 MB known-SHA dataset, ×2 backups)
|
||||
|
||||
| Operation | Result |
|
||||
|-----------|--------|
|
||||
| Datastore create (65 536 chunk dirs) | **6 s** |
|
||||
| Backup #1 (750 MB, cold) | **10.1 s** (~75 MB/s end-to-end through PBS→ext4→loop→sshfs→box) |
|
||||
| Backup #2 (identical → dedup) | **1.8 s**, dedup factor **2.0** |
|
||||
| **Garbage collection** (the flagged risk) | **7 s — completes, NO hang, `TASK OK`** |
|
||||
| Verify (all snapshots) | **12 s, 0 errors** |
|
||||
| Restore round-trip (750 MB → sha256 vs source) | **11 s, integrity OK** |
|
||||
| GC after forget-all | completes in 10 s; `Removed chunks: 0` = **expected** (PBS ~24 h chunk grace period, not a mount fault) |
|
||||
|
||||
**The research's headline fear — GC latency/hangs on a high-latency mount — did not materialize**
|
||||
here, because intra-Hetzner IPv6 latency is low and the loopback ext4 turns PBS's many small
|
||||
metadata ops into local-fs ops that flush to the box in batches.
|
||||
|
||||
### Caveats on the loopback path (why "fit-*with-caveats*")
|
||||
1. **Crash/disconnect integrity risk.** ext4-over-loopback-over-sshfs adds a journaled FS on top of a
|
||||
network file. A hard disconnect mid-write risks ext4 corruption of the whole image (vs. losing
|
||||
one chunk). `reconnect` mitigates transient drops but is not a durability guarantee. A real
|
||||
deployment needs: periodic `fsck` discipline, and treating the image as **replaceable** (PBS
|
||||
`verify` + a second copy elsewhere), never as the sole custodian of data.
|
||||
2. **Fixed image size.** ext4 image is provisioned up-front (sparse, so it only consumes actual
|
||||
used bytes on the box, but the FS cap is fixed until grown). Growing = `truncate` + `resize2fs`.
|
||||
3. **Single-mounter.** The loop image must be mounted by exactly one host; no concurrent access.
|
||||
4. **Boot ordering.** Two layers to bring up in order at boot (sshfs mount → loop mount → PBS). The
|
||||
**sshfs layer is now reboot-proven** (§5); the **loop-mount + PBS auto-bring-up layering is still
|
||||
unproven at boot** — the loop image + a `.mount`/`.service` ordered after the sshfs unit would
|
||||
need adding and a reboot to confirm before any real loopback deployment.
|
||||
|
||||
## 5. Open / not-yet-done
|
||||
|
||||
- **Reboot-survival (Phase 3e) — DONE, PASSED (operator-approved reboot, 2026-07-09).** ep0 was
|
||||
rebooted; the SSHFS mount **auto-mounted at boot** (unit `active`+`enabled`), a post-boot write as
|
||||
the `backup` user succeeded, WG reconnected (handshake within 19 s), and `proxmox-backup-proxy`
|
||||
came back `active`. Boot id changed `40e87037… → 8dc83d82…`; downtime ~30–60 s.
|
||||
Still unproven at boot: the **loop-mount + PBS auto-bring-up** layering (the loop image was torn
|
||||
down before the reboot), which a real loopback deployment must add and re-test.
|
||||
- **Cache-disk constraint (for the PBS-S3 alternative, if ever chosen):** PBS S3 datastores want a
|
||||
64–128 GB **local** cache. ep0's `/` has only **29 GB free** — the S3 path would need a bigger ep0
|
||||
disk first. Irrelevant to the SSHFS/loopback path, which needs no such local cache.
|
||||
|
||||
## 6. Decision this unblocks
|
||||
|
||||
Three offsite-tier options, now measured enough to choose between:
|
||||
|
||||
1. **SSHFS + loopback-ext4 PBS datastore on the 1 TB Storage Box** — cheapest (€4.06/mo, 1 TB),
|
||||
proven functional here (GC/verify/restore all green), but carries the loopback-integrity caveat.
|
||||
Best if the Storage Box is a *secondary* copy behind the existing `felhom-offsite` tier, not the
|
||||
only one.
|
||||
2. **PBS-S3 on Hetzner Object Storage** — the separate future spike; needs the 64–128 GB local cache
|
||||
(→ grow ep0's disk first) and has the S3-refresh datastore-block window to measure. Cleaner
|
||||
integrity story (no loopback FS), different cost model.
|
||||
3. **Promote ep0 cache-disk first**, then decide — only needed for option 2.
|
||||
|
||||
Recommendation for the immediate next step: if the Storage Box is wanted as a **cheap second offsite
|
||||
copy**, adopt option 1 with the loopback image treated as replaceable and guarded by scheduled
|
||||
`verify`. If it's meant to be a **primary** offsite datastore, do the PBS-S3 spike (option 2) before
|
||||
committing, because the loopback-integrity caveat is not ideal for a sole custodian.
|
||||
|
||||
**Not a GO-LIVE-PACKAGE item.** This is the offsite arc, separate from the Peti pilot (D3). No
|
||||
go-live flag flipped.
|
||||
|
||||
## 7. Teardown state (left clean)
|
||||
|
||||
- Throwaway `spike-sb` datastore removed; only `felhom-offsite` + `scratch` remain.
|
||||
- Loop image unmounted and `spike-sb.img` deleted; box `/home` back to just `.ssh` (384 KB used).
|
||||
- All scratch API tokens (`spiketok*`) deleted; scratch ACL removed.
|
||||
- **Kept (durable deliverables):** the `/mnt/pbs-storagebox` SSHFS mount unit (enabled) and the
|
||||
`storagebox_ed25519` key on ep0.
|
||||
Reference in New Issue
Block a user