felhom.eu/documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md

VALIDATION — offbox v0.104.0 live against the Hetzner Storage Box (supervised)

Date: 2026-07-09 · Class: supervised operational validation (RUNBOOK) · Executor: Claude Code live, with Viktor on the 🛑 STOP points. No controller code changed; no version bump. Closes the offsite arc: proves v0.104.0's offbox (discovery + no-silent-success) end-to-end against the real box through the real endpoints, in the exact scenario v0.103.0 got wrong.

---

1. VERDICT

offbox v0.104.0 direct-to-Storage-Box is VALIDATED through the real /backup/offbox/* endpoints. The discovery fix works against the real drive layout (not just t.TempDir()): an undeployed app whose recovery unit exists only on the USB drive — the precise case the old systemDataPath fallback mis-resolved — was found and backed up. The no-silent-success behaviors both fired against the live box: 0/N → hard error + operator alert; partial → customer warning. A 45 MB db-dump round-tripped byte-perfect.

---

2. Confirmed inputs

Thing	Value
controller	:0.104.0 @ 908e4b9 on guest 9201, Up (healthy)
box	u629193.your-storagebox.de, port 23, IPv4 167.235.97.81, ~869 GB free
account	fresh sub-account u629193-sub1 (old compromised one deleted; console-created) @ own host u629193-sub1.your-storagebox.de, base dir /felhom-demo
RepoPath	/home/felhom-demo-repo (sub-account chroot presents its base dir as /home → RepoPath /home/<name>)
key install	console SSH-key field again didn't apply (F3 persists) → installed via ssh-copy-id -p 23 **-s -f** -i <pub> with the sub-account password (Viktor, 🛑)
known_hosts (public — safe)	[u629193-sub1.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
host-key MITM cross-check	ed25519 key byte-identical to ep0's pinned known_hosts entry → PASS
method	config/toggle/run/restore all via real endpoints (curl to in-container 127.0.0.1:8080; demo dashboard is open — localhost authoritative, avoids the public-URL POST no-op). Anti-F9 satisfied.

Actors: DISCOVER = immich (undeployed; unit only on USB; real 45 MB db-dumps/immich-postgres.sql). MISSING = vaultwarden (no unit anywhere). Partial 2nd = audiobookshelf (compose+manifest unit on USB).

3. Phase 3 — DISCOVERY PROOF ✅

immich toggled (NOT deployed; live app.yaml absent; unit only at …/felhom-usb/backups/primary/immich), run via POST /backup/offbox/run:

[offbox] initialized restic repo
[offbox] backed up immich (/mnt/felhom-drives/felhom-usb/backups/primary/immich)
[offbox] backup OK: 1 app(s) backed up, 1 snapshot(s), 34s

LastStatus=ok, SnapshotCount=1, no error. The backed-up src is the USB path — proving discovery scanned the storage registry (schedulable path ∪ systemDataPath) and found the USB unit. Pre-fix (v0.103.0) this resolved to systemDataPath (/mnt/sys_drive/felhom-data/backups/primary/immich, which does not exist) → 0 snapshots reported ok. The regression scenario is closed against the real box.

4. Phase 4 — NO-SILENT-SUCCESS ✅

• 0/N hard error: only vaultwarden toggled (no unit). Run → LastStatus=error, LastError="off-box backup produced no snapshots: 1 app(s) toggled but no recovery unit was found on any connected drive (missing: vaultwarden)", and the operator alert fired: [ERROR] [offbox] backup failed after 5s: … → [INFO] Event pushed: backup_failed (error) — Off-box (NAS) mentés sikertelen. (Controller-side alert pipeline confirmed via the pushed backup_failed event; downstream hub→Resend email delivery is the hub's leg — operator inbox confirmation is Viktor's.)
• Partial warning: immich + audiobookshelf + vaultwarden toggled. Run → LastStatus=ok, 2 backed up (2 app(s) backed up, 1 skipped (no unit), 2 snapshot(s), 45s), LastWarning="Figyelmeztetés: 1 alkalmazásnak nincs elérhető mentése, ezek kimaradtak: vaultwarden", rendered on /backups (confirmed in the page HTML, --warn style). No false "all good".

5. Phase 5 — ROUND-TRIP INTEGRITY ✅

Restore immich via POST /backup/offbox/restore → non-destructive to <DataDir>/offbox-restore/immich. Full recovery unit came back (compose/{app.yaml,.felhom.yml,docker-compose.yml}, manifest.json, db-dumps/immich-postgres.sql). sha256 of the restored 45 MB db-dump = the USB source, exactly (d78466fcbf595b488ba8b962fcc521f68fbe6f36b8b8e57815a6124bccf4526b). The box holds genuinely recoverable data through the real restore endpoint.

6. Phase 6 — PERSISTENCE ✅

Controller restarted (bootstrap) → healthy. offbox config (enabled, host/user/repo_path), LastStatus=ok, SnapshotCount, LastWarning, the per-app toggles, and the 0600 secret files (ssh_key, repo_password, known_hosts) all survived — settings.json persistence intact.

7. Fork resolutions

1. Topology (fork 1): direct — confirmed. Residential IPv4 → box:23; no ep0 concentrator, no WG.
2. Account model (fork 2): sub-account-per-customer — confirmed. Fresh console sub-account with its own …-subN.your-storagebox.de hostname, chrooted (base dir presented as /home), RepoPath convention /home/<name>, host key shared with the main account (one pinned line covers both). Password-free intent holds, but the console SSH-key field is unreliable (F3) → the repeatable install path is ssh-copy-id -p 23 -s -f with the sub-account password.
3. PBS-namespace / WG tier (fork 3): deferred — restic-to-box is the pilot-era offsite. Not built.
4. Custody (fork 4) — surface RECORDED, design OPEN. Per-customer secrets, controller-managed, 0600 in the controller data dir: restic repo password (data/offbox/repo_password, 32-byte hex, auto-generated on first config) + SSH private key (data/offbox/ssh_key). Neither is independently recoverable today — both ride DR only via the PBS whole-CT snapshot of the guest rootfs. Recovery ordering: offsite-restic recovery depends on first recovering these from the PBS tier. An escrow/custody design is the load-bearing open follow-up before any real reliance on this tier.

8. Throughput (observed, residential upload)

immich cold backup (45 MB) + repo init = 34 s; a dedup run (immich unchanged) + audiobookshelf cold = 45 s; a full-dedup run = 44 s. The 45 MB upload itself is fast (~consistent with the spike's ~4.3 MB/s residential upstream); forget --prune + snapshots/stats over SFTP dominate the wall-clock on a small repo (dedup backups finish in ~3 s but the run is ~44 s). Fine for a daily schedule; note the first seed of a real tens-of-GB app is upstream-bound (hours) — heads-up on onboarding.

9. Teardown — KEPT (Viktor's decision, 🛑)

Left standing as a demo-offsite (€4.06/mo): the fresh sub-account u629193-sub1, the offbox config, and the box repo /home/felhom-demo-repo (immich + audiobookshelf snapshots). Config tidied — the test-only vaultwarden toggle removed and a final clean run left LastStatus=ok, 2 snapshots, no warning; toggled = {audiobookshelf, immich}. Scratch removed (the /tmp keypair copy — the durable key lives 0600 in data/offbox/ssh_key; the 45 MB verify-restore). Untouched: the main account's other keys, the USB real units, ep0's felhom-offsite/mount. ⚠️ Standing-reliance caveat: the scratch restic repo password is now semi-permanent and NOT independently recoverable → any real dependence on this demo-offsite needs the fork-4 custody decision first.

10. NEXT (to fully productionize)

• Fork-4 custody/escrow spec — the one blocker for real reliance (repo password + SSH key recovery independent of the PBS whole-CT).
• Per-customer provisioning runbook for Peti: sub-account create + key install (ssh-copy-id -s -f until the console key field is fixed / F3) + known_hosts pin + RepoPath /home/<name> + offbox config.
• F3 (Hetzner console sub-account SSH-key field silently not applying) — recurred; work around with ssh-copy-id -s -f, or raise with Hetzner.
• Tier-2 TestBackupTier2Restore_DoubleClickRefused flake — separate backup-suite test-race debt (pre-existing; see controller v0.104.0 REPORT Observations).