From 463ea369c4a66db83ba58984d6d461600b3af6c0 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 9 Jul 2026 13:17:52 +0200 Subject: [PATCH] docs(audit): offbox v0.104.0 VALIDATED live against the Storage Box (supervised) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real endpoints, fresh box sub-account. Discovery proof: immich (undeployed, USB-only unit — the v0.103.0 mis-resolution case) → ok/1 snapshot on the USB path; 45MB db-dump restore sha256-matched. No-silent-success on the live box: 0/N → error + backup_failed alert; partial → ok + LastWarning. Persists across restart. Forks 1+2 confirmed (direct; sub-account /home/, ssh-copy-id -s -f, console key field still broken F3); fork-4 custody surface recorded, escrow design OPEN (repo pw + key not independently recoverable). Kept as standing demo-offsite. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 13 ++ ...VALIDATION-offbox-storagebox-2026-07-09.md | 113 ++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md diff --git a/CONTEXT.md b/CONTEXT.md index fcb98b2..98f850e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,19 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised).** Closed the offsite + arc: controller `:0.104.0`@`908e4b9` ran the real `/backup/offbox/*` endpoints against a fresh box + **sub-account** (`u629193-sub1`, RepoPath `/home/felhom-demo-repo`). The capstone proof — **immich** + (undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to `systemDataPath`) + → discovery found the USB unit → `ok`, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success + confirmed on the live box: 0/N → `error` + `backup_failed` operator alert; partial → `ok` + + Hungarian `LastWarning` on `/backups`. Persists across restart. **Forks:** (1) direct topology + (2) + sub-account-per-customer (RepoPath `/home/`, `ssh-copy-id -p 23 -s -f` — console SSH-key field still + broken, F3) confirmed; (3) PBS/WG tier deferred; **(4) custody surface RECORDED, escrow design OPEN** — + the per-customer restic repo password + SSH key are 0600 in the controller data dir, **NOT independently + recoverable** (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance. + **KEPT** as a standing demo-offsite (€4.06/mo). Record: `documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md`. + - **2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator (hub v0.36.0, `844fbfa7`; deploy `02c748eb`; LIVE, ArgoCD Synced/Healthy)** — go-live G7/G9/G10/G11. Two coupled, security-first changes to the operator customer page (`customer_unified.html` + diff --git a/documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md b/documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md new file mode 100644 index 0000000..5748624 --- /dev/null +++ b/documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md @@ -0,0 +1,113 @@ +# VALIDATION — offbox v0.104.0 live against the Hetzner Storage Box (supervised) + +**Date:** 2026-07-09 · **Class:** supervised operational validation (RUNBOOK) · **Executor:** Claude Code +live, with Viktor on the 🛑 STOP points. **No controller code changed; no version bump.** Closes the offsite +arc: proves v0.104.0's offbox (discovery + no-silent-success) end-to-end against the real box through the +real endpoints, in the exact scenario v0.103.0 got wrong. + +--- + +## 1. VERDICT + +**offbox v0.104.0 direct-to-Storage-Box is VALIDATED** through the real `/backup/offbox/*` endpoints. The +discovery fix works against the **real drive layout** (not just `t.TempDir()`): an **undeployed** app whose +recovery unit exists **only on the USB drive** — the precise case the old `systemDataPath` fallback +mis-resolved — was found and backed up. The no-silent-success behaviors both fired against the live box: +0/N → hard error + operator alert; partial → customer warning. A 45 MB db-dump round-tripped byte-perfect. + +--- + +## 2. Confirmed inputs + +| Thing | Value | +|-------|-------| +| controller | `:0.104.0` @ `908e4b9` on guest 9201, `Up (healthy)` | +| box | `u629193.your-storagebox.de`, port **23**, IPv4 `167.235.97.81`, ~869 GB free | +| account | fresh **sub-account `u629193-sub1`** (old compromised one deleted; console-created) @ own host `u629193-sub1.your-storagebox.de`, base dir `/felhom-demo` | +| RepoPath | **`/home/felhom-demo-repo`** (sub-account chroot presents its base dir as `/home` → RepoPath `/home/`) | +| key install | console SSH-key field again didn't apply (**F3 persists**) → installed via `ssh-copy-id -p 23 **-s -f** -i ` with the sub-account password (Viktor, 🛑) | +| known_hosts (public — safe) | `[u629193-sub1.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs` | +| host-key MITM cross-check | ed25519 key **byte-identical** to ep0's pinned `known_hosts` entry → PASS | +| method | config/toggle/run/restore all via **real endpoints** (curl to in-container `127.0.0.1:8080`; demo dashboard is open — localhost authoritative, avoids the public-URL POST no-op). Anti-F9 satisfied. | + +**Actors:** DISCOVER = **immich** (undeployed; unit only on USB; real 45 MB `db-dumps/immich-postgres.sql`). +MISSING = **vaultwarden** (no unit anywhere). Partial 2nd = **audiobookshelf** (compose+manifest unit on USB). + +## 3. Phase 3 — DISCOVERY PROOF ✅ +immich toggled (NOT deployed; live `app.yaml` absent; unit only at `…/felhom-usb/backups/primary/immich`), +run via `POST /backup/offbox/run`: +``` +[offbox] initialized restic repo +[offbox] backed up immich (/mnt/felhom-drives/felhom-usb/backups/primary/immich) +[offbox] backup OK: 1 app(s) backed up, 1 snapshot(s), 34s +``` +`LastStatus=ok`, `SnapshotCount=1`, no error. The backed-up **src is the USB path** — proving discovery +scanned the storage registry (schedulable path ∪ systemDataPath) and found the USB unit. **Pre-fix +(v0.103.0)** this resolved to `systemDataPath` (`/mnt/sys_drive/felhom-data/backups/primary/immich`, which +does not exist) → 0 snapshots reported `ok`. The regression scenario is closed against the real box. + +## 4. Phase 4 — NO-SILENT-SUCCESS ✅ +- **0/N hard error:** only vaultwarden toggled (no unit). Run → + `LastStatus=error`, `LastError="off-box backup produced no snapshots: 1 app(s) toggled but no recovery + unit was found on any connected drive (missing: vaultwarden)"`, and the operator alert fired: + `[ERROR] [offbox] backup failed after 5s: …` → `[INFO] Event pushed: backup_failed (error) — Off-box + (NAS) mentés sikertelen`. (Controller-side alert pipeline confirmed via the pushed `backup_failed` + event; downstream hub→Resend email delivery is the hub's leg — operator inbox confirmation is Viktor's.) +- **Partial warning:** immich + audiobookshelf + vaultwarden toggled. Run → + `LastStatus=ok`, 2 backed up (`2 app(s) backed up, 1 skipped (no unit), 2 snapshot(s), 45s`), + `LastWarning="Figyelmeztetés: 1 alkalmazásnak nincs elérhető mentése, ezek kimaradtak: vaultwarden"`, + **rendered on `/backups`** (confirmed in the page HTML, `--warn` style). No false "all good". + +## 5. Phase 5 — ROUND-TRIP INTEGRITY ✅ +Restore immich via `POST /backup/offbox/restore` → non-destructive to `/offbox-restore/immich`. +Full recovery unit came back (`compose/{app.yaml,.felhom.yml,docker-compose.yml}`, `manifest.json`, +`db-dumps/immich-postgres.sql`). **sha256 of the restored 45 MB db-dump = the USB source, exactly** +(`d78466fcbf595b488ba8b962fcc521f68fbe6f36b8b8e57815a6124bccf4526b`). The box holds genuinely recoverable +data through the real restore endpoint. + +## 6. Phase 6 — PERSISTENCE ✅ +Controller restarted (bootstrap) → healthy. offbox config (enabled, host/user/repo_path), `LastStatus=ok`, +`SnapshotCount`, `LastWarning`, the per-app toggles, and the 0600 secret files (`ssh_key`, `repo_password`, +`known_hosts`) all survived — settings.json persistence intact. + +## 7. Fork resolutions +1. **Topology (fork 1):** **direct — confirmed.** Residential IPv4 → box:23; no ep0 concentrator, no WG. +2. **Account model (fork 2):** **sub-account-per-customer — confirmed.** Fresh console sub-account with its + own `…-subN.your-storagebox.de` hostname, chrooted (base dir presented as `/home`), **RepoPath + convention `/home/`**, host key shared with the main account (one pinned line covers both). + Password-free *intent* holds, but **the console SSH-key field is unreliable (F3)** → the repeatable + install path is `ssh-copy-id -p 23 -s -f` with the sub-account password. +3. **PBS-namespace / WG tier (fork 3):** deferred — restic-to-box is the pilot-era offsite. Not built. +4. **Custody (fork 4) — surface RECORDED, design OPEN.** Per-customer secrets, controller-managed, 0600 in + the controller data dir: **restic repo password** (`data/offbox/repo_password`, 32-byte hex, + auto-generated on first config) + **SSH private key** (`data/offbox/ssh_key`). **Neither is independently + recoverable today** — both ride DR only via the PBS whole-CT snapshot of the guest rootfs. Recovery + ordering: offsite-restic recovery depends on first recovering these from the PBS tier. An escrow/custody + design is the load-bearing open follow-up before any real reliance on this tier. + +## 8. Throughput (observed, residential upload) +immich cold backup (45 MB) + repo init = 34 s; a dedup run (immich unchanged) + audiobookshelf cold = 45 s; +a full-dedup run = 44 s. The 45 MB upload itself is fast (~consistent with the spike's ~4.3 MB/s residential +upstream); **`forget --prune` + `snapshots/stats` over SFTP dominate the wall-clock on a small repo** (dedup +backups finish in ~3 s but the run is ~44 s). Fine for a daily schedule; note the first seed of a real +tens-of-GB app is upstream-bound (hours) — heads-up on onboarding. + +## 9. Teardown — KEPT (Viktor's decision, 🛑) +Left standing as a demo-offsite (€4.06/mo): the fresh sub-account `u629193-sub1`, the offbox config, and the +box repo `/home/felhom-demo-repo` (immich + audiobookshelf snapshots). Config tidied — the test-only +**vaultwarden** toggle removed and a final clean run left `LastStatus=ok`, 2 snapshots, **no warning**; +toggled = {audiobookshelf, immich}. Scratch removed (the `/tmp` keypair copy — the durable key lives 0600 in +`data/offbox/ssh_key`; the 45 MB verify-restore). Untouched: the main account's other keys, the USB real +units, ep0's `felhom-offsite`/mount. +**⚠️ Standing-reliance caveat:** the scratch restic repo password is now semi-permanent and **NOT +independently recoverable** → any real dependence on this demo-offsite needs the fork-4 custody decision first. + +## 10. NEXT (to fully productionize) +- **Fork-4 custody/escrow spec** — the one blocker for real reliance (repo password + SSH key recovery + independent of the PBS whole-CT). +- **Per-customer provisioning runbook** for Peti: sub-account create + key install (`ssh-copy-id -s -f` + until the console key field is fixed / F3) + known_hosts pin + RepoPath `/home/` + offbox config. +- **F3** (Hetzner console sub-account SSH-key field silently not applying) — recurred; work around with + `ssh-copy-id -s -f`, or raise with Hetzner. +- **Tier-2 `TestBackupTier2Restore_DoubleClickRefused` flake** — separate backup-suite test-race debt + (pre-existing; see controller v0.104.0 REPORT Observations).