# VALIDATION — offbox v0.104.0 live against the Hetzner Storage Box (supervised)

**Date:** 2026-07-09 · **Class:** supervised operational validation (RUNBOOK) · **Executor:** Claude Code
live, with Viktor on the 🛑 STOP points. **No controller code changed; no version bump.** Closes the offsite
arc: proves v0.104.0's offbox (discovery + no-silent-success) end-to-end against the real box through the
real endpoints, in the exact scenario v0.103.0 got wrong.

---

## 1. VERDICT

**offbox v0.104.0 direct-to-Storage-Box is VALIDATED** through the real `/backup/offbox/*` endpoints. The
discovery fix works against the **real drive layout** (not just `t.TempDir()`): an **undeployed** app whose
recovery unit exists **only on the USB drive** — the precise case the old `systemDataPath` fallback
mis-resolved — was found and backed up. The no-silent-success behaviors both fired against the live box:
0/N → hard error + operator alert; partial → customer warning. A 45 MB db-dump round-tripped byte-perfect.

---

## 2. Confirmed inputs

| Thing | Value |
|-------|-------|
| controller | `:0.104.0` @ `908e4b9` on guest 9201, `Up (healthy)` |
| box | `u629193.your-storagebox.de`, port **23**, IPv4 `167.235.97.81`, ~869 GB free |
| account | fresh **sub-account `u629193-sub1`** (old compromised one deleted; console-created) @ own host `u629193-sub1.your-storagebox.de`, base dir `/felhom-demo` |
| RepoPath | **`/home/felhom-demo-repo`** (sub-account chroot presents its base dir as `/home` → RepoPath `/home/<name>`) |
| key install | console SSH-key field again didn't apply (**F3 persists**) → installed via `ssh-copy-id -p 23 **-s -f** -i <pub>` with the sub-account password (Viktor, 🛑) |
| known_hosts (public — safe) | `[u629193-sub1.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs` |
| host-key MITM cross-check | ed25519 key **byte-identical** to ep0's pinned `known_hosts` entry → PASS |
| method | config/toggle/run/restore all via **real endpoints** (curl to in-container `127.0.0.1:8080`; demo dashboard is open — localhost authoritative, avoids the public-URL POST no-op). Anti-F9 satisfied. |

**Actors:** DISCOVER = **immich** (undeployed; unit only on USB; real 45 MB `db-dumps/immich-postgres.sql`).
MISSING = **vaultwarden** (no unit anywhere). Partial 2nd = **audiobookshelf** (compose+manifest unit on USB).

## 3. Phase 3 — DISCOVERY PROOF ✅
immich toggled (NOT deployed; live `app.yaml` absent; unit only at `…/felhom-usb/backups/primary/immich`),
run via `POST /backup/offbox/run`:
```
[offbox] initialized restic repo
[offbox] backed up immich (/mnt/felhom-drives/felhom-usb/backups/primary/immich)
[offbox] backup OK: 1 app(s) backed up, 1 snapshot(s), 34s
```
`LastStatus=ok`, `SnapshotCount=1`, no error. The backed-up **src is the USB path** — proving discovery
scanned the storage registry (schedulable path ∪ systemDataPath) and found the USB unit. **Pre-fix
(v0.103.0)** this resolved to `systemDataPath` (`/mnt/sys_drive/felhom-data/backups/primary/immich`, which
does not exist) → 0 snapshots reported `ok`. The regression scenario is closed against the real box.

## 4. Phase 4 — NO-SILENT-SUCCESS ✅
- **0/N hard error:** only vaultwarden toggled (no unit). Run →
  `LastStatus=error`, `LastError="off-box backup produced no snapshots: 1 app(s) toggled but no recovery
  unit was found on any connected drive (missing: vaultwarden)"`, and the operator alert fired:
  `[ERROR] [offbox] backup failed after 5s: …` → `[INFO] Event pushed: backup_failed (error) — Off-box
  (NAS) mentés sikertelen`. (Controller-side alert pipeline confirmed via the pushed `backup_failed`
  event; downstream hub→Resend email delivery is the hub's leg — operator inbox confirmation is Viktor's.)
- **Partial warning:** immich + audiobookshelf + vaultwarden toggled. Run →
  `LastStatus=ok`, 2 backed up (`2 app(s) backed up, 1 skipped (no unit), 2 snapshot(s), 45s`),
  `LastWarning="Figyelmeztetés: 1 alkalmazásnak nincs elérhető mentése, ezek kimaradtak: vaultwarden"`,
  **rendered on `/backups`** (confirmed in the page HTML, `--warn` style). No false "all good".

## 5. Phase 5 — ROUND-TRIP INTEGRITY ✅
Restore immich via `POST /backup/offbox/restore` → non-destructive to `<DataDir>/offbox-restore/immich`.
Full recovery unit came back (`compose/{app.yaml,.felhom.yml,docker-compose.yml}`, `manifest.json`,
`db-dumps/immich-postgres.sql`). **sha256 of the restored 45 MB db-dump = the USB source, exactly**
(`d78466fcbf595b488ba8b962fcc521f68fbe6f36b8b8e57815a6124bccf4526b`). The box holds genuinely recoverable
data through the real restore endpoint.

## 6. Phase 6 — PERSISTENCE ✅
Controller restarted (bootstrap) → healthy. offbox config (enabled, host/user/repo_path), `LastStatus=ok`,
`SnapshotCount`, `LastWarning`, the per-app toggles, and the 0600 secret files (`ssh_key`, `repo_password`,
`known_hosts`) all survived — settings.json persistence intact.

## 7. Fork resolutions
1. **Topology (fork 1):** **direct — confirmed.** Residential IPv4 → box:23; no ep0 concentrator, no WG.
2. **Account model (fork 2):** **sub-account-per-customer — confirmed.** Fresh console sub-account with its
   own `…-subN.your-storagebox.de` hostname, chrooted (base dir presented as `/home`), **RepoPath
   convention `/home/<name>`**, host key shared with the main account (one pinned line covers both).
   Password-free *intent* holds, but **the console SSH-key field is unreliable (F3)** → the repeatable
   install path is `ssh-copy-id -p 23 -s -f` with the sub-account password.
3. **PBS-namespace / WG tier (fork 3):** deferred — restic-to-box is the pilot-era offsite. Not built.
4. **Custody (fork 4) — surface RECORDED, design OPEN.** Per-customer secrets, controller-managed, 0600 in
   the controller data dir: **restic repo password** (`data/offbox/repo_password`, 32-byte hex,
   auto-generated on first config) + **SSH private key** (`data/offbox/ssh_key`). **Neither is independently
   recoverable today** — both ride DR only via the PBS whole-CT snapshot of the guest rootfs. Recovery
   ordering: offsite-restic recovery depends on first recovering these from the PBS tier. An escrow/custody
   design is the load-bearing open follow-up before any real reliance on this tier.

## 8. Throughput (observed, residential upload)
immich cold backup (45 MB) + repo init = 34 s; a dedup run (immich unchanged) + audiobookshelf cold = 45 s;
a full-dedup run = 44 s. The 45 MB upload itself is fast (~consistent with the spike's ~4.3 MB/s residential
upstream); **`forget --prune` + `snapshots/stats` over SFTP dominate the wall-clock on a small repo** (dedup
backups finish in ~3 s but the run is ~44 s). Fine for a daily schedule; note the first seed of a real
tens-of-GB app is upstream-bound (hours) — heads-up on onboarding.

## 9. Teardown — KEPT (Viktor's decision, 🛑)
Left standing as a demo-offsite (€4.06/mo): the fresh sub-account `u629193-sub1`, the offbox config, and the
box repo `/home/felhom-demo-repo` (immich + audiobookshelf snapshots). Config tidied — the test-only
**vaultwarden** toggle removed and a final clean run left `LastStatus=ok`, 2 snapshots, **no warning**;
toggled = {audiobookshelf, immich}. Scratch removed (the `/tmp` keypair copy — the durable key lives 0600 in
`data/offbox/ssh_key`; the 45 MB verify-restore). Untouched: the main account's other keys, the USB real
units, ep0's `felhom-offsite`/mount.
**⚠️ Standing-reliance caveat:** the scratch restic repo password is now semi-permanent and **NOT
independently recoverable** → any real dependence on this demo-offsite needs the fork-4 custody decision first.

## 10. NEXT (to fully productionize)
- **Fork-4 custody/escrow spec** — the one blocker for real reliance (repo password + SSH key recovery
  independent of the PBS whole-CT).
- **Per-customer provisioning runbook** for Peti: sub-account create + key install (`ssh-copy-id -s -f`
  until the console key field is fixed / F3) + known_hosts pin + RepoPath `/home/<name>` + offbox config.
- **F3** (Hetzner console sub-account SSH-key field silently not applying) — recurred; work around with
  `ssh-copy-id -s -f`, or raise with Hetzner.
- **Tier-2 `TestBackupTier2Restore_DoubleClickRefused` flake** — separate backup-suite test-race debt
  (pre-existing; see controller v0.104.0 REPORT Observations).