admin
463ea369c4
Real endpoints, fresh box sub-account. Discovery proof: immich (undeployed, USB-only unit — the v0.103.0 mis-resolution case) → ok/1 snapshot on the USB path; 45MB db-dump restore sha256-matched. No-silent-success on the live box: 0/N → error + backup_failed alert; partial → ok + LastWarning. Persists across restart. Forks 1+2 confirmed (direct; sub-account /home/<name>, ssh-copy-id -s -f, console key field still broken F3); fork-4 custody surface recorded, escrow design OPEN (repo pw + key not independently recoverable). Kept as standing demo-offsite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
114 lines
8.3 KiB
Markdown
114 lines
8.3 KiB
Markdown
# VALIDATION — offbox v0.104.0 live against the Hetzner Storage Box (supervised)
|
||
|
||
**Date:** 2026-07-09 · **Class:** supervised operational validation (RUNBOOK) · **Executor:** Claude Code
|
||
live, with Viktor on the 🛑 STOP points. **No controller code changed; no version bump.** Closes the offsite
|
||
arc: proves v0.104.0's offbox (discovery + no-silent-success) end-to-end against the real box through the
|
||
real endpoints, in the exact scenario v0.103.0 got wrong.
|
||
|
||
---
|
||
|
||
## 1. VERDICT
|
||
|
||
**offbox v0.104.0 direct-to-Storage-Box is VALIDATED** through the real `/backup/offbox/*` endpoints. The
|
||
discovery fix works against the **real drive layout** (not just `t.TempDir()`): an **undeployed** app whose
|
||
recovery unit exists **only on the USB drive** — the precise case the old `systemDataPath` fallback
|
||
mis-resolved — was found and backed up. The no-silent-success behaviors both fired against the live box:
|
||
0/N → hard error + operator alert; partial → customer warning. A 45 MB db-dump round-tripped byte-perfect.
|
||
|
||
---
|
||
|
||
## 2. Confirmed inputs
|
||
|
||
| Thing | Value |
|
||
|-------|-------|
|
||
| controller | `:0.104.0` @ `908e4b9` on guest 9201, `Up (healthy)` |
|
||
| box | `u629193.your-storagebox.de`, port **23**, IPv4 `167.235.97.81`, ~869 GB free |
|
||
| account | fresh **sub-account `u629193-sub1`** (old compromised one deleted; console-created) @ own host `u629193-sub1.your-storagebox.de`, base dir `/felhom-demo` |
|
||
| RepoPath | **`/home/felhom-demo-repo`** (sub-account chroot presents its base dir as `/home` → RepoPath `/home/<name>`) |
|
||
| key install | console SSH-key field again didn't apply (**F3 persists**) → installed via `ssh-copy-id -p 23 **-s -f** -i <pub>` with the sub-account password (Viktor, 🛑) |
|
||
| known_hosts (public — safe) | `[u629193-sub1.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs` |
|
||
| host-key MITM cross-check | ed25519 key **byte-identical** to ep0's pinned `known_hosts` entry → PASS |
|
||
| method | config/toggle/run/restore all via **real endpoints** (curl to in-container `127.0.0.1:8080`; demo dashboard is open — localhost authoritative, avoids the public-URL POST no-op). Anti-F9 satisfied. |
|
||
|
||
**Actors:** DISCOVER = **immich** (undeployed; unit only on USB; real 45 MB `db-dumps/immich-postgres.sql`).
|
||
MISSING = **vaultwarden** (no unit anywhere). Partial 2nd = **audiobookshelf** (compose+manifest unit on USB).
|
||
|
||
## 3. Phase 3 — DISCOVERY PROOF ✅
|
||
immich toggled (NOT deployed; live `app.yaml` absent; unit only at `…/felhom-usb/backups/primary/immich`),
|
||
run via `POST /backup/offbox/run`:
|
||
```
|
||
[offbox] initialized restic repo
|
||
[offbox] backed up immich (/mnt/felhom-drives/felhom-usb/backups/primary/immich)
|
||
[offbox] backup OK: 1 app(s) backed up, 1 snapshot(s), 34s
|
||
```
|
||
`LastStatus=ok`, `SnapshotCount=1`, no error. The backed-up **src is the USB path** — proving discovery
|
||
scanned the storage registry (schedulable path ∪ systemDataPath) and found the USB unit. **Pre-fix
|
||
(v0.103.0)** this resolved to `systemDataPath` (`/mnt/sys_drive/felhom-data/backups/primary/immich`, which
|
||
does not exist) → 0 snapshots reported `ok`. The regression scenario is closed against the real box.
|
||
|
||
## 4. Phase 4 — NO-SILENT-SUCCESS ✅
|
||
- **0/N hard error:** only vaultwarden toggled (no unit). Run →
|
||
`LastStatus=error`, `LastError="off-box backup produced no snapshots: 1 app(s) toggled but no recovery
|
||
unit was found on any connected drive (missing: vaultwarden)"`, and the operator alert fired:
|
||
`[ERROR] [offbox] backup failed after 5s: …` → `[INFO] Event pushed: backup_failed (error) — Off-box
|
||
(NAS) mentés sikertelen`. (Controller-side alert pipeline confirmed via the pushed `backup_failed`
|
||
event; downstream hub→Resend email delivery is the hub's leg — operator inbox confirmation is Viktor's.)
|
||
- **Partial warning:** immich + audiobookshelf + vaultwarden toggled. Run →
|
||
`LastStatus=ok`, 2 backed up (`2 app(s) backed up, 1 skipped (no unit), 2 snapshot(s), 45s`),
|
||
`LastWarning="Figyelmeztetés: 1 alkalmazásnak nincs elérhető mentése, ezek kimaradtak: vaultwarden"`,
|
||
**rendered on `/backups`** (confirmed in the page HTML, `--warn` style). No false "all good".
|
||
|
||
## 5. Phase 5 — ROUND-TRIP INTEGRITY ✅
|
||
Restore immich via `POST /backup/offbox/restore` → non-destructive to `<DataDir>/offbox-restore/immich`.
|
||
Full recovery unit came back (`compose/{app.yaml,.felhom.yml,docker-compose.yml}`, `manifest.json`,
|
||
`db-dumps/immich-postgres.sql`). **sha256 of the restored 45 MB db-dump = the USB source, exactly**
|
||
(`d78466fcbf595b488ba8b962fcc521f68fbe6f36b8b8e57815a6124bccf4526b`). The box holds genuinely recoverable
|
||
data through the real restore endpoint.
|
||
|
||
## 6. Phase 6 — PERSISTENCE ✅
|
||
Controller restarted (bootstrap) → healthy. offbox config (enabled, host/user/repo_path), `LastStatus=ok`,
|
||
`SnapshotCount`, `LastWarning`, the per-app toggles, and the 0600 secret files (`ssh_key`, `repo_password`,
|
||
`known_hosts`) all survived — settings.json persistence intact.
|
||
|
||
## 7. Fork resolutions
|
||
1. **Topology (fork 1):** **direct — confirmed.** Residential IPv4 → box:23; no ep0 concentrator, no WG.
|
||
2. **Account model (fork 2):** **sub-account-per-customer — confirmed.** Fresh console sub-account with its
|
||
own `…-subN.your-storagebox.de` hostname, chrooted (base dir presented as `/home`), **RepoPath
|
||
convention `/home/<name>`**, host key shared with the main account (one pinned line covers both).
|
||
Password-free *intent* holds, but **the console SSH-key field is unreliable (F3)** → the repeatable
|
||
install path is `ssh-copy-id -p 23 -s -f` with the sub-account password.
|
||
3. **PBS-namespace / WG tier (fork 3):** deferred — restic-to-box is the pilot-era offsite. Not built.
|
||
4. **Custody (fork 4) — surface RECORDED, design OPEN.** Per-customer secrets, controller-managed, 0600 in
|
||
the controller data dir: **restic repo password** (`data/offbox/repo_password`, 32-byte hex,
|
||
auto-generated on first config) + **SSH private key** (`data/offbox/ssh_key`). **Neither is independently
|
||
recoverable today** — both ride DR only via the PBS whole-CT snapshot of the guest rootfs. Recovery
|
||
ordering: offsite-restic recovery depends on first recovering these from the PBS tier. An escrow/custody
|
||
design is the load-bearing open follow-up before any real reliance on this tier.
|
||
|
||
## 8. Throughput (observed, residential upload)
|
||
immich cold backup (45 MB) + repo init = 34 s; a dedup run (immich unchanged) + audiobookshelf cold = 45 s;
|
||
a full-dedup run = 44 s. The 45 MB upload itself is fast (~consistent with the spike's ~4.3 MB/s residential
|
||
upstream); **`forget --prune` + `snapshots/stats` over SFTP dominate the wall-clock on a small repo** (dedup
|
||
backups finish in ~3 s but the run is ~44 s). Fine for a daily schedule; note the first seed of a real
|
||
tens-of-GB app is upstream-bound (hours) — heads-up on onboarding.
|
||
|
||
## 9. Teardown — KEPT (Viktor's decision, 🛑)
|
||
Left standing as a demo-offsite (€4.06/mo): the fresh sub-account `u629193-sub1`, the offbox config, and the
|
||
box repo `/home/felhom-demo-repo` (immich + audiobookshelf snapshots). Config tidied — the test-only
|
||
**vaultwarden** toggle removed and a final clean run left `LastStatus=ok`, 2 snapshots, **no warning**;
|
||
toggled = {audiobookshelf, immich}. Scratch removed (the `/tmp` keypair copy — the durable key lives 0600 in
|
||
`data/offbox/ssh_key`; the 45 MB verify-restore). Untouched: the main account's other keys, the USB real
|
||
units, ep0's `felhom-offsite`/mount.
|
||
**⚠️ Standing-reliance caveat:** the scratch restic repo password is now semi-permanent and **NOT
|
||
independently recoverable** → any real dependence on this demo-offsite needs the fork-4 custody decision first.
|
||
|
||
## 10. NEXT (to fully productionize)
|
||
- **Fork-4 custody/escrow spec** — the one blocker for real reliance (repo password + SSH key recovery
|
||
independent of the PBS whole-CT).
|
||
- **Per-customer provisioning runbook** for Peti: sub-account create + key install (`ssh-copy-id -s -f`
|
||
until the console key field is fixed / F3) + known_hosts pin + RepoPath `/home/<name>` + offbox config.
|
||
- **F3** (Hetzner console sub-account SSH-key field silently not applying) — recurred; work around with
|
||
`ssh-copy-id -s -f`, or raise with Hetzner.
|
||
- **Tier-2 `TestBackupTier2Restore_DoubleClickRefused` flake** — separate backup-suite test-race debt
|
||
(pre-existing; see controller v0.104.0 REPORT Observations).
|