felhom.eu/documentation/audits/SPIKE-storagebox-restic-direct-2026-07-09.md

# SPIKE — restic → Hetzner Storage Box, DIRECT from a residential (IPv4) customer box

**Date:** 2026-07-09  **Executor:** Claude Code (live)  **Class:** spike (empirical validation)
**Egress under test:** the real `felhom-controller` container in demo LXC guest 9201 on `felhom-pve`
(residential N100 — the true customer-egress profile).  **No controller code changed; no version bump.**

---

## 1. VERDICT

**Direct restic-over-SFTP from a residential IPv4 customer box → FIT.** The reachability that matters
for the pilot is proven: **the box is reachable on :23 over IPv4** from the residential egress, and the
controller's existing offbox tier drives a full **config → run → restore** loop against the live box
through its **real endpoints**. Sub-account-per-customer **isolation holds**. A **concentrator (ep0/WG)
is NOT required** for reachability.

**One blocking caveat for productionizing — a controller bug, not a box/restic problem (see F1):** the
offbox runner looks for each app's recovery unit under the app's **data-drive** namespace root, but on
any system whose primary backups live on a **separate backup drive** (this demo → the USB drive) it
finds nothing, **skips every app, and still reports `LastStatus=ok` with 0 snapshots**. Direct offbox is
mechanically sound; this path-resolution mismatch must be fixed (or the storage layout aligned) before it
backs up real customer data.

---

## 2. Verified inputs (live)

| Fact | Value |
|------|-------|
| Box | BX11 `PBS-storage-1` (#611421), `u629193`, **port 23**, FSN1, 1 TB, €4.06/mo, 0 B used at start |
| Main host | `u629193.your-storagebox.de` — **now resolves A `167.235.97.81`** (AAAA `2a01:4f8:2b01:d2d::2` exists but see §3) |
| External reachability | **ON** (was OFF in the prior ep0 spike) |
| restic in container | **0.14.0** (go1.19.8) — uses `sftp.command`, already handled by offbox.go |
| ssh in container | OpenSSH 9.2p1 |
| Controller | v0.103.0, healthy, LXC 9201, container `felhom-controller`, listens `:8080` (bridge, in-container), dashboard open (no password) |
| ep0 | still holds a box-authorized key + a live SSHFS mount of `/home` (reused for teardown only) |

## 3. Reachability (Phase 1 — the gate) — **PASS over IPv4**

From **inside the controller container** (the residential egress):
- **DNS:** `getent ahostsv4` → `167.235.97.81`. `getent ahostsv6` → **nothing**. The residential egress
  is **IPv4-only** — no AAAA is returned to it, and an explicit `ssh -6` probe returns **"Network is
  unreachable."** This is the **load-bearing result for the IPv4-only pilot (Peti).**
- **`ssh-keyscan -p 23`** returned host keys → reachable.
- **Explicit IPv4 probe** (`ssh -4 -p 23 … true`) → `Permission denied (publickey,password)` =
  **reachable** (auth failure, not timeout/refused). Round-trip felt immediate.
- **Conclusion:** direct residential-IPv4 → box:23 works. No concentrator needed for reachability.

## 4. Host-key pinning (Phase 2 — no blind TOFU) — **PASS**

The three host keys seen from the residential container via `ssh-keyscan` are **byte-identical** to the
keys ep0 pinned over a **completely different network path** (intra-Hetzner) in the prior spike — two
independent paths agreeing rules out MITM. The verified line to pin in the controller config (ed25519,
fingerprint `SHA256:XqONwb1S0zuj5A1CDxpOSuD2hnAArV1A3wKY7Z3sdgM`):

```
[u629193.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
```

Note: the **sub-account hostname presents the SAME host key** (same physical box) — one pinned line
covers both main and sub endpoints.

## 5. Raw restic round-trip (Phase 3 — mechanism proof) — **PASS**

Reproduced offbox.go's exact `sftp.command` from inside the container, RepoPath `/home/felhom-spike`
(box `/` is read-only; `/home` writable), scratch key authorized on the box (see §Auth):

| Step | Result |
|------|--------|
| `restic init` | **OK — the chown-immunity headline.** restic writes plain files and never `chown`s, so it succeeds where **PBS-direct hit EACCES** (prior spike). This is *why* restic is the right tool for this box. |
| backup #1 (500 MB urandom) | **1:56 (116 s) → ~4.3 MB/s**; snapshot saved |
| restore latest + sha256 | **INTEGRITY-OK** (exact match) |
| backup #2 (identical) | **3 s, 704 B added** → dedup works over SFTP (factor 2.0) |
| `forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune` | completes, **32 s** |

### Auth path (Phase 3a)
Scratch ed25519 key generated in-container. Because ep0 already held a box-authorized key, the pubkey
was **appended to the box `authorized_keys` over SFTP using ep0's existing trust** — password-free, no
operator step (operator-approved alternative to `ssh-copy-id … -s`). Verified passwordless SFTP from the
residential container. (Observation: the box's main `authorized_keys` already held **11 keys** before
this — worth an operator cleanup.)

## 6. Integration proof through the REAL endpoints (Phase 4 — the proof that counts) — **PASS (with F1)**

Driven via curl to the in-container `:8080` (form POSTs via the public URL can silently no-op — a known
demo gotcha; localhost is authoritative). RepoPath `/home/felhom-offbox-spike`.

1. **`POST /backup/offbox/config`** (Host, Port 23, User `u629193`, RepoPath, `ssh_key` via
   `--data-urlencode @file`, `known_hosts`, `enabled=on`) → 302 success flash. The controller wrote its
   own secret set: `ssh_key` (0600), `known_hosts` (0644), **auto-generated `repo_password` (0600)** —
   fully controller-managed; the operator never supplies the repo password.
2. **`POST /backup/offbox/toggle`** (audiobookshelf) → included.
3. **`POST /backup/offbox/run`** → **F1 fired:** `LastStatus=ok`, **0 snapshots**, log:
   `audiobookshelf: no backup data yet (/mnt/sys_drive/felhom-data/backups/primary/audiobookshelf) —
   skipping`. The primary backups on this demo actually live on the **USB drive**
   (`/mnt/felhom-drives/felhom-usb/backups/primary/…`); offbox looked under the app's **data-drive**
   namespace root instead → found nothing.
4. After **staging the real (synthetic) recovery unit** at the path offbox expects, re-run →
   **`LastStatus=ok`, 1 snapshot, 8 s.** Repo created correctly on the box (config/data/index/keys/
   snapshots — a proper restic repo).
5. **`POST /backup/offbox/restore`** (audiobookshelf) → restored non-destructively to
   `data/offbox-restore/audiobookshelf/…`; the full recovery unit (`compose/*.yml` + `manifest.json`)
   came back and the **manifest sha256 matched the source exactly**.

**The config→run→restore loop is green through the customer/operator's real endpoints against the live
box over residential IPv4.**

### F1 (the important finding) — offbox recovery-unit source path vs actual backup location
`runOffboxInternal` derives the source as `RecoveryUnitPath(AppNamespaceRoot(stack), stack)` =
`<app-data-drive namespace root>/backups/primary/<stack>` (`internal/backup/offbox.go:268-276`,
`backup.go:147`). But primary backups are written to the configured **backup-storage drive**, which can
differ from the app's data drive. When they differ (this demo: data on `sys_drive`, backups on the
`felhom-usb` drive) offbox finds no recovery unit and skips. **Impact:** on any multi-drive customer
(likely Peti, if his backups target a dedicated drive) direct offbox would silently back up nothing.
**Fix direction:** resolve the recovery-unit source from the *actual primary-backup storage location*
(the backup target), not the app's data-drive namespace root — or guarantee the two are aligned.
*Confirm intended source with the controller team; this is the one real blocker for productionizing.*

### F2 — silent "ok / 0 snapshots"
A run that skips **every** app for missing recovery units still reports `LastStatus=ok`. There is no
warning surfaced to the operator/customer that nothing was backed up. Offbox should distinguish
"backed up N apps" from "0 apps had data" (warn/degraded status), else F1-class problems stay invisible.

## 7. Sub-account isolation (Phase 5) — **PASS**

Operator created sub-account **`u629193-sub1`** (own hostname `u629193-sub1.your-storagebox.de`, base
dir `/felhom-peti-spike`, SSH on). The **console SSH-key field did not take** (F3); the key was installed
with **`ssh-copy-id -p 23 -s -f -i <pub>`** using the sub-account password (both `-s` **and** `-f` are
required). Then, from the residential container:

- **Jail:** `ls /` → *Permission denied*; the sub's `/home` shows **only its own `.ssh`** (its base dir
  `/felhom-peti-spike` is presented as `/home`); **both main repos are "not found"** from the sub.
- **Round-trip:** init + backup + 1 snapshot with the sub's **distinct key + distinct repo password** — OK.
- **Isolation assert (b):** the **main** repo password on the **sub** repo → `Fatal: wrong password or no
  key found` — repos are independently encrypted.
- **Isolation assert (a):** the sub endpoint pointed at the main repo path → *repository not found* —
  the jail blocks cross-customer reads.

**Isolation model for fork 2:** sub-account-per-customer is viable. RepoPath convention **`/home/<name>`**
for both main and sub (the sub is chrooted, so `/home/<name>` is physically isolated per customer). User
pattern `u629193` (main) / `u629193-subN` (sub, with its own `…-subN.your-storagebox.de` hostname).
Provisioning is **console/Robot-driven**; the console SSH-key field is currently unreliable (F3) — use
`ssh-copy-id … -s -f` until fixed. `ValidateOffboxTarget` already accepts sub targets (hyphen in
user/host and the `/home/<name>` repo path all pass).

## 8. Throughput (Phase 6)

Residential upload is the sole bottleneck (intra-Hetzner box speed ~327 MB/s is irrelevant here). Clean
measurement: **500 MB in 116 s ≈ 4.3 MB/s (~34 Mbps upstream)**. Extrapolated first-seed:
**~20 GB ≈ 78 min, ~50 GB ≈ 195 min.** Restore is download-bound (faster on typical residential links).
**Implication:** the daily schedule is fine for steady-state deltas (dedup makes them tiny — 704 B/3 s
above), but the **first seed of a real app can take hours** — give the customer a heads-up on onboarding.
(A dedicated 5 GB timing was skipped as it only reconfirms the same upstream bottleneck.)

## 9. Provisioning values (bake into a future task)

| Field | Value |
|------|-------|
| Host | `u629193.your-storagebox.de` (main) · `u629193-subN.your-storagebox.de` (**sub has its own hostname**) |
| Port | **23** |
| User | `u629193` (main) · `u629193-subN` (sub) |
| RepoPath | **`/home/<name>`** (sub is chrooted → physically isolated) |
| known_hosts line | `[HOST]:23 ssh-ed25519 AAAA…Oxs` (same key for main + sub hostnames) |
| Key install | **`ssh-copy-id -p 23 -s -f -i <pub> USER@HOST`** (both `-s` and `-f` required; console SSH-key field unreliable per F3) |
| Retention | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune` (offbox default) |

## 10. Secrets inventory for custody design (fork 4 — NOT decided here)

Per-customer, all controller-managed under `data/offbox/` (0600 key + repo_password, 0644 known_hosts):
- **restic repo password** — auto-generated by the controller on first config; **loss = unrecoverable
  offsite data.**
- **SSH private key** — 0600.
Both ride DR via the PBS whole-CT snapshot. **Recovery-ordering coupling:** offsite-restic recovery
depends on first recovering these secrets from the PBS tier — the custody/escrow spec must sequence that.

## 11. Decision this unblocks

- **Topology (direct vs concentrator):** **DIRECT — recommended.** Residential IPv4 reaches the box
  directly; no ep0/WG concentrator needed. The concentrator spike is **not** required.
- **Isolation (fork 2):** **sub-account-per-customer — recommended.** Jailed + independently encrypted.
  Provisioning is console/Robot-driven (fix or work around the console SSH-key field, F3).
- **Relationship to the PBS tier:** restic offbox = the **app-data** offsite tier (backs up
  `backups/primary/<stack>`), distinct from the PBS whole-CT tier. ep0 `felhom-offsite` capacity was
  **not** re-observed in this spike (out of scope).

## 12. NOT done / next

- **F1 controller fix** — the recovery-unit source-path mismatch is the real blocker for direct offbox on
  multi-drive customers; must be resolved (or storage layout aligned) before real data. **Highest priority.**
- **F2** — surface a warning when a run backs up 0 apps.
- **F3** — Hetzner console sub-account SSH-key field didn't apply the key; use `ssh-copy-id -s -f`.
- Custody/escrow design (fork 4) using the §10 inventory.
- Production provisioning task: small controller/UI polish + per-customer sub-account ceremony.
- Concentrator spike: **not needed** (direct works).

## 13. Teardown (left clean)

- **Box:** spike repos removed (`/home/felhom-spike`, `/home/felhom-offbox-spike`, sub `/home/peti-repo`);
  main `authorized_keys` restored 12→**11** (spike key removed, ep0 key preserved); box `/home` back to
  `.ssh` + the sub base dir.
- **Controller:** audiobookshelf un-toggled; staged recovery unit + `/mnt/sys_drive/felhom-data` removed;
  `offbox-restore/` removed; `data/offbox/` secrets removed; **offbox target cleared from settings.json**;
  controller restarted **healthy** (settings load clean, 0 offbox lines).
  *(Mid-teardown a bad write truncated settings.json to 0 B; restored from the controller's own
  `settings.json.bak` — no data lost. The daily `offbox-backup` job is still scheduled but no-ops with no config.)*
- **Container:** scratch keys/passwords (`/tmp/felhomspike`) removed.
- **OPERATOR TODO:** **delete (or rotate) sub-account `u629193-sub1`** — its password was shown on screen
  during creation, so treat it as compromised scratch. Its base dir `/felhom-peti-spike/.ssh` (spike key)
  goes away when the sub-account is deleted; keep it only if promoting to Peti's real sub-account (then
  rotate the key). No secrets are recorded in this doc (all stored 0600 out-of-band).