diff --git a/documentation/audits/SPIKE-storagebox-restic-direct-2026-07-09.md b/documentation/audits/SPIKE-storagebox-restic-direct-2026-07-09.md new file mode 100644 index 0000000..b0803e5 --- /dev/null +++ b/documentation/audits/SPIKE-storagebox-restic-direct-2026-07-09.md @@ -0,0 +1,210 @@ +# SPIKE — restic → Hetzner Storage Box, DIRECT from a residential (IPv4) customer box + +**Date:** 2026-07-09 **Executor:** Claude Code (live) **Class:** spike (empirical validation) +**Egress under test:** the real `felhom-controller` container in demo LXC guest 9201 on `felhom-pve` +(residential N100 — the true customer-egress profile). **No controller code changed; no version bump.** + +--- + +## 1. VERDICT + +**Direct restic-over-SFTP from a residential IPv4 customer box → FIT.** The reachability that matters +for the pilot is proven: **the box is reachable on :23 over IPv4** from the residential egress, and the +controller's existing offbox tier drives a full **config → run → restore** loop against the live box +through its **real endpoints**. Sub-account-per-customer **isolation holds**. A **concentrator (ep0/WG) +is NOT required** for reachability. + +**One blocking caveat for productionizing — a controller bug, not a box/restic problem (see F1):** the +offbox runner looks for each app's recovery unit under the app's **data-drive** namespace root, but on +any system whose primary backups live on a **separate backup drive** (this demo → the USB drive) it +finds nothing, **skips every app, and still reports `LastStatus=ok` with 0 snapshots**. Direct offbox is +mechanically sound; this path-resolution mismatch must be fixed (or the storage layout aligned) before it +backs up real customer data. + +--- + +## 2. Verified inputs (live) + +| Fact | Value | +|------|-------| +| Box | BX11 `PBS-storage-1` (#611421), `u629193`, **port 23**, FSN1, 1 TB, €4.06/mo, 0 B used at start | +| Main host | `u629193.your-storagebox.de` — **now resolves A `167.235.97.81`** (AAAA `2a01:4f8:2b01:d2d::2` exists but see §3) | +| External reachability | **ON** (was OFF in the prior ep0 spike) | +| restic in container | **0.14.0** (go1.19.8) — uses `sftp.command`, already handled by offbox.go | +| ssh in container | OpenSSH 9.2p1 | +| Controller | v0.103.0, healthy, LXC 9201, container `felhom-controller`, listens `:8080` (bridge, in-container), dashboard open (no password) | +| ep0 | still holds a box-authorized key + a live SSHFS mount of `/home` (reused for teardown only) | + +## 3. Reachability (Phase 1 — the gate) — **PASS over IPv4** + +From **inside the controller container** (the residential egress): +- **DNS:** `getent ahostsv4` → `167.235.97.81`. `getent ahostsv6` → **nothing**. The residential egress + is **IPv4-only** — no AAAA is returned to it, and an explicit `ssh -6` probe returns **"Network is + unreachable."** This is the **load-bearing result for the IPv4-only pilot (Peti).** +- **`ssh-keyscan -p 23`** returned host keys → reachable. +- **Explicit IPv4 probe** (`ssh -4 -p 23 … true`) → `Permission denied (publickey,password)` = + **reachable** (auth failure, not timeout/refused). Round-trip felt immediate. +- **Conclusion:** direct residential-IPv4 → box:23 works. No concentrator needed for reachability. + +## 4. Host-key pinning (Phase 2 — no blind TOFU) — **PASS** + +The three host keys seen from the residential container via `ssh-keyscan` are **byte-identical** to the +keys ep0 pinned over a **completely different network path** (intra-Hetzner) in the prior spike — two +independent paths agreeing rules out MITM. The verified line to pin in the controller config (ed25519, +fingerprint `SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM`): + +``` +[u629193.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs +``` + +Note: the **sub-account hostname presents the SAME host key** (same physical box) — one pinned line +covers both main and sub endpoints. + +## 5. Raw restic round-trip (Phase 3 — mechanism proof) — **PASS** + +Reproduced offbox.go's exact `sftp.command` from inside the container, RepoPath `/home/felhom-spike` +(box `/` is read-only; `/home` writable), scratch key authorized on the box (see §Auth): + +| Step | Result | +|------|--------| +| `restic init` | **OK — the chown-immunity headline.** restic writes plain files and never `chown`s, so it succeeds where **PBS-direct hit EACCES** (prior spike). This is *why* restic is the right tool for this box. | +| backup #1 (500 MB urandom) | **1:56 (116 s) → ~4.3 MB/s**; snapshot saved | +| restore latest + sha256 | **INTEGRITY-OK** (exact match) | +| backup #2 (identical) | **3 s, 704 B added** → dedup works over SFTP (factor 2.0) | +| `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune` | completes, **32 s** | + +### Auth path (Phase 3a) +Scratch ed25519 key generated in-container. Because ep0 already held a box-authorized key, the pubkey +was **appended to the box `authorized_keys` over SFTP using ep0's existing trust** — password-free, no +operator step (operator-approved alternative to `ssh-copy-id … -s`). Verified passwordless SFTP from the +residential container. (Observation: the box's main `authorized_keys` already held **11 keys** before +this — worth an operator cleanup.) + +## 6. Integration proof through the REAL endpoints (Phase 4 — the proof that counts) — **PASS (with F1)** + +Driven via curl to the in-container `:8080` (form POSTs via the public URL can silently no-op — a known +demo gotcha; localhost is authoritative). RepoPath `/home/felhom-offbox-spike`. + +1. **`POST /backup/offbox/config`** (Host, Port 23, User `u629193`, RepoPath, `ssh_key` via + `--data-urlencode @file`, `known_hosts`, `enabled=on`) → 302 success flash. The controller wrote its + own secret set: `ssh_key` (0600), `known_hosts` (0644), **auto-generated `repo_password` (0600)** — + fully controller-managed; the operator never supplies the repo password. +2. **`POST /backup/offbox/toggle`** (audiobookshelf) → included. +3. **`POST /backup/offbox/run`** → **F1 fired:** `LastStatus=ok`, **0 snapshots**, log: + `audiobookshelf: no backup data yet (/mnt/sys_drive/felhom-data/backups/primary/audiobookshelf) — + skipping`. The primary backups on this demo actually live on the **USB drive** + (`/mnt/felhom-drives/felhom-usb/backups/primary/…`); offbox looked under the app's **data-drive** + namespace root instead → found nothing. +4. After **staging the real (synthetic) recovery unit** at the path offbox expects, re-run → + **`LastStatus=ok`, 1 snapshot, 8 s.** Repo created correctly on the box (config/data/index/keys/ + snapshots — a proper restic repo). +5. **`POST /backup/offbox/restore`** (audiobookshelf) → restored non-destructively to + `data/offbox-restore/audiobookshelf/…`; the full recovery unit (`compose/*.yml` + `manifest.json`) + came back and the **manifest sha256 matched the source exactly**. + +**The config→run→restore loop is green through the customer/operator's real endpoints against the live +box over residential IPv4.** + +### F1 (the important finding) — offbox recovery-unit source path vs actual backup location +`runOffboxInternal` derives the source as `RecoveryUnitPath(AppNamespaceRoot(stack), stack)` = +`/backups/primary/` (`internal/backup/offbox.go:268-276`, +`backup.go:147`). But primary backups are written to the configured **backup-storage drive**, which can +differ from the app's data drive. When they differ (this demo: data on `sys_drive`, backups on the +`felhom-usb` drive) offbox finds no recovery unit and skips. **Impact:** on any multi-drive customer +(likely Peti, if his backups target a dedicated drive) direct offbox would silently back up nothing. +**Fix direction:** resolve the recovery-unit source from the *actual primary-backup storage location* +(the backup target), not the app's data-drive namespace root — or guarantee the two are aligned. +*Confirm intended source with the controller team; this is the one real blocker for productionizing.* + +### F2 — silent "ok / 0 snapshots" +A run that skips **every** app for missing recovery units still reports `LastStatus=ok`. There is no +warning surfaced to the operator/customer that nothing was backed up. Offbox should distinguish +"backed up N apps" from "0 apps had data" (warn/degraded status), else F1-class problems stay invisible. + +## 7. Sub-account isolation (Phase 5) — **PASS** + +Operator created sub-account **`u629193-sub1`** (own hostname `u629193-sub1.your-storagebox.de`, base +dir `/felhom-peti-spike`, SSH on). The **console SSH-key field did not take** (F3); the key was installed +with **`ssh-copy-id -p 23 -s -f -i `** using the sub-account password (both `-s` **and** `-f` are +required). Then, from the residential container: + +- **Jail:** `ls /` → *Permission denied*; the sub's `/home` shows **only its own `.ssh`** (its base dir + `/felhom-peti-spike` is presented as `/home`); **both main repos are "not found"** from the sub. +- **Round-trip:** init + backup + 1 snapshot with the sub's **distinct key + distinct repo password** — OK. +- **Isolation assert (b):** the **main** repo password on the **sub** repo → `Fatal: wrong password or no + key found` — repos are independently encrypted. +- **Isolation assert (a):** the sub endpoint pointed at the main repo path → *repository not found* — + the jail blocks cross-customer reads. + +**Isolation model for fork 2:** sub-account-per-customer is viable. RepoPath convention **`/home/`** +for both main and sub (the sub is chrooted, so `/home/` is physically isolated per customer). User +pattern `u629193` (main) / `u629193-subN` (sub, with its own `…-subN.your-storagebox.de` hostname). +Provisioning is **console/Robot-driven**; the console SSH-key field is currently unreliable (F3) — use +`ssh-copy-id … -s -f` until fixed. `ValidateOffboxTarget` already accepts sub targets (hyphen in +user/host and the `/home/` repo path all pass). + +## 8. Throughput (Phase 6) + +Residential upload is the sole bottleneck (intra-Hetzner box speed ~327 MB/s is irrelevant here). Clean +measurement: **500 MB in 116 s ≈ 4.3 MB/s (~34 Mbps upstream)**. Extrapolated first-seed: +**~20 GB ≈ 78 min, ~50 GB ≈ 195 min.** Restore is download-bound (faster on typical residential links). +**Implication:** the daily schedule is fine for steady-state deltas (dedup makes them tiny — 704 B/3 s +above), but the **first seed of a real app can take hours** — give the customer a heads-up on onboarding. +(A dedicated 5 GB timing was skipped as it only reconfirms the same upstream bottleneck.) + +## 9. Provisioning values (bake into a future task) + +| Field | Value | +|------|-------| +| Host | `u629193.your-storagebox.de` (main) · `u629193-subN.your-storagebox.de` (**sub has its own hostname**) | +| Port | **23** | +| User | `u629193` (main) · `u629193-subN` (sub) | +| RepoPath | **`/home/`** (sub is chrooted → physically isolated) | +| known_hosts line | `[HOST]:23 ssh-ed25519 AAAA…Oxs` (same key for main + sub hostnames) | +| Key install | **`ssh-copy-id -p 23 -s -f -i USER@HOST`** (both `-s` and `-f` required; console SSH-key field unreliable per F3) | +| Retention | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune` (offbox default) | + +## 10. Secrets inventory for custody design (fork 4 — NOT decided here) + +Per-customer, all controller-managed under `data/offbox/` (0600 key + repo_password, 0644 known_hosts): +- **restic repo password** — auto-generated by the controller on first config; **loss = unrecoverable + offsite data.** +- **SSH private key** — 0600. +Both ride DR via the PBS whole-CT snapshot. **Recovery-ordering coupling:** offsite-restic recovery +depends on first recovering these secrets from the PBS tier — the custody/escrow spec must sequence that. + +## 11. Decision this unblocks + +- **Topology (direct vs concentrator):** **DIRECT — recommended.** Residential IPv4 reaches the box + directly; no ep0/WG concentrator needed. The concentrator spike is **not** required. +- **Isolation (fork 2):** **sub-account-per-customer — recommended.** Jailed + independently encrypted. + Provisioning is console/Robot-driven (fix or work around the console SSH-key field, F3). +- **Relationship to the PBS tier:** restic offbox = the **app-data** offsite tier (backs up + `backups/primary/`), distinct from the PBS whole-CT tier. ep0 `felhom-offsite` capacity was + **not** re-observed in this spike (out of scope). + +## 12. NOT done / next + +- **F1 controller fix** — the recovery-unit source-path mismatch is the real blocker for direct offbox on + multi-drive customers; must be resolved (or storage layout aligned) before real data. **Highest priority.** +- **F2** — surface a warning when a run backs up 0 apps. +- **F3** — Hetzner console sub-account SSH-key field didn't apply the key; use `ssh-copy-id -s -f`. +- Custody/escrow design (fork 4) using the §10 inventory. +- Production provisioning task: small controller/UI polish + per-customer sub-account ceremony. +- Concentrator spike: **not needed** (direct works). + +## 13. Teardown (left clean) + +- **Box:** spike repos removed (`/home/felhom-spike`, `/home/felhom-offbox-spike`, sub `/home/peti-repo`); + main `authorized_keys` restored 12→**11** (spike key removed, ep0 key preserved); box `/home` back to + `.ssh` + the sub base dir. +- **Controller:** audiobookshelf un-toggled; staged recovery unit + `/mnt/sys_drive/felhom-data` removed; + `offbox-restore/` removed; `data/offbox/` secrets removed; **offbox target cleared from settings.json**; + controller restarted **healthy** (settings load clean, 0 offbox lines). + *(Mid-teardown a bad write truncated settings.json to 0 B; restored from the controller's own + `settings.json.bak` — no data lost. The daily `offbox-backup` job is still scheduled but no-ops with no config.)* +- **Container:** scratch keys/passwords (`/tmp/felhomspike`) removed. +- **OPERATOR TODO:** **delete (or rotate) sub-account `u629193-sub1`** — its password was shown on screen + during creation, so treat it as compromised scratch. Its base dir `/felhom-peti-spike/.ssh` (spike key) + goes away when the sub-account is deleted; keep it only if promoting to Peti's real sub-account (then + rotate the key). No secrets are recorded in this doc (all stored 0600 out-of-band).