diff --git a/documentation/audits/SPIKE-ep0-storagebox-2026-07-09.md b/documentation/audits/SPIKE-ep0-storagebox-2026-07-09.md new file mode 100644 index 0000000..9ebc560 --- /dev/null +++ b/documentation/audits/SPIKE-ep0-storagebox-2026-07-09.md @@ -0,0 +1,177 @@ +# SPIKE — ep0 ↔ Hetzner Storage Box (SSHFS mount + PBS-datastore feasibility) + +**Date:** 2026-07-09 **Executor:** Claude Code (live on ep0) **Class:** operational + spike +**Scope boundary honoured:** synthetic test data only; no production datastore promoted, no real +backup job pointed at the mount, no customer data touched. All PBS scratch artifacts torn down. + +--- + +## VERDICT + +**Direct PBS datastore on the SSHFS mount: UNFIT (hard blocker).** +**PBS datastore via a loopback ext4 image on the SSHFS mount: FIT — with caveats.** +**Raw SSHFS mount as a durable file target (restic/borg/rsync/plain files): FIT.** + +The Storage Box is now permanently mounted on ep0 at `/mnt/pbs-storagebox` (durable systemd unit, +key-auth, passwordless, reboot-ordered). The mount itself is fast and reliable. The nuance is +entirely about **how PBS wants to own its chunk store** — see below. + +--- + +## 0. Verified inputs (confirmed live) + +| Fact | Value | +|------|-------| +| Storage Box | BX11 `PBS-storage-1`, 1 TB, €4.06/mo | +| Host | `u629193.your-storagebox.de` (resolves **AAAA-only**: `2a01:4f8:2b01:d2d::2`) | +| Username / SSH port | `u629193` / **23** | +| ep0 | `167.233.158.164`, Debian 13 (trixie), kernel 6.12, **eu-central** (same zone as box) | +| ep0 PBS | `proxmox-backup-server 4.2.2-1` **already installed & in production** | +| Existing datastores on ep0 | `felhom-offsite` (`/srv/pbs-felhom`, prod offsite tier) + `scratch` — **untouched** | +| ep0 local free (`/`) | 29 GB of 38 GB (`/dev/sda1`) | +| Box free | 1.0 TB (1% used) | + +**Reachability:** port 23 was **OPEN from ep0 over IPv6** with no console change — the runbook's +"External Reachability still OFF" concern did **not** block ep0→box. (The earlier DooPlex/Windows +cloud SSH keys were irrelevant; see §Auth.) The runbook's "this sandbox cannot reach ep0" caveat +was also stale — ep0 was reachable throughout. + +## 1. Auth — how the key actually got onto the box + +- Hetzner **Cloud-project SSH keys** (console → Security → SSH keys) do **NOT** authorize a Storage + Box. The box keeps its own `authorized_keys`. Adding `ep0-storagebox` at the project level was + proven inert: key auth to the box returned `Permission denied (publickey,password)`. +- A dedicated **ed25519** key was generated on ep0 (`/root/.ssh/storagebox_ed25519`, 0600, + fingerprint `SHA256:HlIJl/WwDutWKVaxhPhdvwAGyMwhh4ztspqKZAgpIlc`). +- Installed to the box's `authorized_keys` via **`ssh-copy-id -p 23 -s -i …`**. The **`-s` (SFTP) + flag is mandatory** on Hetzner Storage Boxes — the normal shell-append method is rejected, and + `-i` is required because the key filename is non-default. +- Box password: set once by the operator in the console, used only for that one `ssh-copy-id`, and + never entered this session. Key/cred material referenced here is **stored 0600 on ep0**, not in + this doc. Post-install auth is fully passwordless. + +## 2. The durable mount (the permanent deliverable) + +systemd unit `/etc/systemd/system/mnt-pbs\x2dstoragebox.mount`, **enabled** (WantedBy +multi-user.target): + +``` +[Mount] +What=u629193@u629193.your-storagebox.de:/home +Where=/mnt/pbs-storagebox +Type=fuse.sshfs +Options=port=23,IdentityFile=/root/.ssh/storagebox_ed25519,idmap=user,uid=34,gid=34, + allow_other,default_permissions,reconnect,ServerAliveInterval=15, + ServerAliveCountMax=3,StrictHostKeyChecking=yes,_netdev +``` + +Non-obvious choices that were forced by testing, not guessed: +- **`:/home`, not `:/`** — the box root `/` is `dr-x--x--x root` (read-only to us); writes there + fail `Operation not permitted`. Our writable area is `/home` (owned by `u629193`). +- **`uid=34,gid=34`** — PBS datastore I/O runs as the `backup` user (uid 34). The mount is mapped so + everything appears owned by `backup`, otherwise PBS can't write. +- **`user_allow_other`** was added to `/etc/fuse.conf` so `allow_other` is honoured. +- `reconnect` + `ServerAliveInterval=15`/`CountMax=3` → survive transient drops; `_netdev` + + `After=network-online.target` → correct boot ordering; `StrictHostKeyChecking=yes` → box host key + pinned in `/root/.ssh/known_hosts` (MITM protection on the tunnel). + +Raw-mount fitness (before any PBS layer): +- Write-integrity round-trip (10 MB urandom, sha256): **OK**. +- Sequential throughput: **~327 MB/s write**, **~69 MB/s read** (256 MiB, intra-Hetzner IPv6). + +## 3. The hard blocker — why a *direct* PBS datastore fails + +`proxmox-backup-manager datastore create … /mnt/pbs-storagebox/…` fails immediately: + +``` +TASK ERROR: unable to create chunk store 'spike-sb' … EACCES: Permission denied +``` + +Root cause isolated by reproducing PBS's steps by hand as the `backup` user: `mkdir`, nested +`mkdir`, `chmod`, file create, and directory `fsync` **all succeed** — but **`chown` is rejected by +the Storage Box for everyone, even same-owner, even as root** (`Permission denied`). Hetzner Storage +Box SSHFS is single-user SFTP with no `chown` support. PBS's `ChunkStore::create` chowns the new +chunk store to `backup:backup`, so datastore creation can never complete directly on the mount. This +is architectural, not tunable. + +## 4. The workaround that works — loopback ext4 on the SSHFS mount + +Put a real filesystem *inside a file* on the box; PBS then owns a normal ext4 where `chown` works. + +``` +truncate -s 20G /mnt/pbs-storagebox/spike-sb.img +mkfs.ext4 -F -E lazy_itable_init=1,lazy_journal_init=1 spike-sb.img # instant (sparse+lazy) +mount -o loop spike-sb.img /mnt/pbs-sb-loop +# chown inside the ext4: OK → datastore create: OK +``` + +### Measurements (throwaway `spike-sb` datastore, synthetic 750 MB known-SHA dataset, ×2 backups) + +| Operation | Result | +|-----------|--------| +| Datastore create (65 536 chunk dirs) | **6 s** | +| Backup #1 (750 MB, cold) | **10.1 s** (~75 MB/s end-to-end through PBS→ext4→loop→sshfs→box) | +| Backup #2 (identical → dedup) | **1.8 s**, dedup factor **2.0** | +| **Garbage collection** (the flagged risk) | **7 s — completes, NO hang, `TASK OK`** | +| Verify (all snapshots) | **12 s, 0 errors** | +| Restore round-trip (750 MB → sha256 vs source) | **11 s, integrity OK** | +| GC after forget-all | completes in 10 s; `Removed chunks: 0` = **expected** (PBS ~24 h chunk grace period, not a mount fault) | + +**The research's headline fear — GC latency/hangs on a high-latency mount — did not materialize** +here, because intra-Hetzner IPv6 latency is low and the loopback ext4 turns PBS's many small +metadata ops into local-fs ops that flush to the box in batches. + +### Caveats on the loopback path (why "fit-*with-caveats*") +1. **Crash/disconnect integrity risk.** ext4-over-loopback-over-sshfs adds a journaled FS on top of a + network file. A hard disconnect mid-write risks ext4 corruption of the whole image (vs. losing + one chunk). `reconnect` mitigates transient drops but is not a durability guarantee. A real + deployment needs: periodic `fsck` discipline, and treating the image as **replaceable** (PBS + `verify` + a second copy elsewhere), never as the sole custodian of data. +2. **Fixed image size.** ext4 image is provisioned up-front (sparse, so it only consumes actual + used bytes on the box, but the FS cap is fixed until grown). Growing = `truncate` + `resize2fs`. +3. **Single-mounter.** The loop image must be mounted by exactly one host; no concurrent access. +4. **Boot ordering.** Two layers to bring up in order at boot (sshfs mount → loop mount → PBS). The + **sshfs layer is now reboot-proven** (§5); the **loop-mount + PBS auto-bring-up layering is still + unproven at boot** — the loop image + a `.mount`/`.service` ordered after the sshfs unit would + need adding and a reboot to confirm before any real loopback deployment. + +## 5. Open / not-yet-done + +- **Reboot-survival (Phase 3e) — DONE, PASSED (operator-approved reboot, 2026-07-09).** ep0 was + rebooted; the SSHFS mount **auto-mounted at boot** (unit `active`+`enabled`), a post-boot write as + the `backup` user succeeded, WG reconnected (handshake within 19 s), and `proxmox-backup-proxy` + came back `active`. Boot id changed `40e87037… → 8dc83d82…`; downtime ~30–60 s. + Still unproven at boot: the **loop-mount + PBS auto-bring-up** layering (the loop image was torn + down before the reboot), which a real loopback deployment must add and re-test. +- **Cache-disk constraint (for the PBS-S3 alternative, if ever chosen):** PBS S3 datastores want a + 64–128 GB **local** cache. ep0's `/` has only **29 GB free** — the S3 path would need a bigger ep0 + disk first. Irrelevant to the SSHFS/loopback path, which needs no such local cache. + +## 6. Decision this unblocks + +Three offsite-tier options, now measured enough to choose between: + +1. **SSHFS + loopback-ext4 PBS datastore on the 1 TB Storage Box** — cheapest (€4.06/mo, 1 TB), + proven functional here (GC/verify/restore all green), but carries the loopback-integrity caveat. + Best if the Storage Box is a *secondary* copy behind the existing `felhom-offsite` tier, not the + only one. +2. **PBS-S3 on Hetzner Object Storage** — the separate future spike; needs the 64–128 GB local cache + (→ grow ep0's disk first) and has the S3-refresh datastore-block window to measure. Cleaner + integrity story (no loopback FS), different cost model. +3. **Promote ep0 cache-disk first**, then decide — only needed for option 2. + +Recommendation for the immediate next step: if the Storage Box is wanted as a **cheap second offsite +copy**, adopt option 1 with the loopback image treated as replaceable and guarded by scheduled +`verify`. If it's meant to be a **primary** offsite datastore, do the PBS-S3 spike (option 2) before +committing, because the loopback-integrity caveat is not ideal for a sole custodian. + +**Not a GO-LIVE-PACKAGE item.** This is the offsite arc, separate from the Peti pilot (D3). No +go-live flag flipped. + +## 7. Teardown state (left clean) + +- Throwaway `spike-sb` datastore removed; only `felhom-offsite` + `scratch` remain. +- Loop image unmounted and `spike-sb.img` deleted; box `/home` back to just `.ssh` (384 KB used). +- All scratch API tokens (`spiketok*`) deleted; scratch ACL removed. +- **Kept (durable deliverables):** the `/mnt/pbs-storagebox` SSHFS mount unit (enabled) and the + `storagebox_ed25519` key on ep0.