docs(audit): offbox v0.104.0 VALIDATED live against the Storage Box (supervised)
Real endpoints, fresh box sub-account. Discovery proof: immich (undeployed, USB-only unit — the v0.103.0 mis-resolution case) → ok/1 snapshot on the USB path; 45MB db-dump restore sha256-matched. No-silent-success on the live box: 0/N → error + backup_failed alert; partial → ok + LastWarning. Persists across restart. Forks 1+2 confirmed (direct; sub-account /home/<name>, ssh-copy-id -s -f, console key field still broken F3); fork-4 custody surface recorded, escrow design OPEN (repo pw + key not independently recoverable). Kept as standing demo-offsite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
+13
@@ -3,6 +3,19 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised).** Closed the offsite
|
||||
arc: controller `:0.104.0`@`908e4b9` ran the real `/backup/offbox/*` endpoints against a fresh box
|
||||
**sub-account** (`u629193-sub1`, RepoPath `/home/felhom-demo-repo`). The capstone proof — **immich**
|
||||
(undeployed, recovery unit only on the USB drive, the exact case v0.103.0 mis-resolved to `systemDataPath`)
|
||||
→ discovery found the USB unit → `ok`, 1 snapshot; 45 MB db-dump restore sha256-matched. No-silent-success
|
||||
confirmed on the live box: 0/N → `error` + `backup_failed` operator alert; partial → `ok` +
|
||||
Hungarian `LastWarning` on `/backups`. Persists across restart. **Forks:** (1) direct topology + (2)
|
||||
sub-account-per-customer (RepoPath `/home/<name>`, `ssh-copy-id -p 23 -s -f` — console SSH-key field still
|
||||
broken, F3) confirmed; (3) PBS/WG tier deferred; **(4) custody surface RECORDED, escrow design OPEN** —
|
||||
the per-customer restic repo password + SSH key are 0600 in the controller data dir, **NOT independently
|
||||
recoverable** (ride DR only via the PBS whole-CT) → the load-bearing follow-up before real reliance.
|
||||
**KEPT** as a standing demo-offsite (€4.06/mo). Record: `documentation/audits/VALIDATION-offbox-storagebox-2026-07-09.md`.
|
||||
|
||||
- **2026-07-09 — TASK GL-7 SHIPPED: customer-page passphrase hardening + install-command generator
|
||||
(hub v0.36.0, `844fbfa7`; deploy `02c748eb`; LIVE, ArgoCD Synced/Healthy)** — go-live G7/G9/G10/G11.
|
||||
Two coupled, security-first changes to the operator customer page (`customer_unified.html` +
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# VALIDATION — offbox v0.104.0 live against the Hetzner Storage Box (supervised)
|
||||
|
||||
**Date:** 2026-07-09 · **Class:** supervised operational validation (RUNBOOK) · **Executor:** Claude Code
|
||||
live, with Viktor on the 🛑 STOP points. **No controller code changed; no version bump.** Closes the offsite
|
||||
arc: proves v0.104.0's offbox (discovery + no-silent-success) end-to-end against the real box through the
|
||||
real endpoints, in the exact scenario v0.103.0 got wrong.
|
||||
|
||||
---
|
||||
|
||||
## 1. VERDICT
|
||||
|
||||
**offbox v0.104.0 direct-to-Storage-Box is VALIDATED** through the real `/backup/offbox/*` endpoints. The
|
||||
discovery fix works against the **real drive layout** (not just `t.TempDir()`): an **undeployed** app whose
|
||||
recovery unit exists **only on the USB drive** — the precise case the old `systemDataPath` fallback
|
||||
mis-resolved — was found and backed up. The no-silent-success behaviors both fired against the live box:
|
||||
0/N → hard error + operator alert; partial → customer warning. A 45 MB db-dump round-tripped byte-perfect.
|
||||
|
||||
---
|
||||
|
||||
## 2. Confirmed inputs
|
||||
|
||||
| Thing | Value |
|
||||
|-------|-------|
|
||||
| controller | `:0.104.0` @ `908e4b9` on guest 9201, `Up (healthy)` |
|
||||
| box | `u629193.your-storagebox.de`, port **23**, IPv4 `167.235.97.81`, ~869 GB free |
|
||||
| account | fresh **sub-account `u629193-sub1`** (old compromised one deleted; console-created) @ own host `u629193-sub1.your-storagebox.de`, base dir `/felhom-demo` |
|
||||
| RepoPath | **`/home/felhom-demo-repo`** (sub-account chroot presents its base dir as `/home` → RepoPath `/home/<name>`) |
|
||||
| key install | console SSH-key field again didn't apply (**F3 persists**) → installed via `ssh-copy-id -p 23 **-s -f** -i <pub>` with the sub-account password (Viktor, 🛑) |
|
||||
| known_hosts (public — safe) | `[u629193-sub1.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs` |
|
||||
| host-key MITM cross-check | ed25519 key **byte-identical** to ep0's pinned `known_hosts` entry → PASS |
|
||||
| method | config/toggle/run/restore all via **real endpoints** (curl to in-container `127.0.0.1:8080`; demo dashboard is open — localhost authoritative, avoids the public-URL POST no-op). Anti-F9 satisfied. |
|
||||
|
||||
**Actors:** DISCOVER = **immich** (undeployed; unit only on USB; real 45 MB `db-dumps/immich-postgres.sql`).
|
||||
MISSING = **vaultwarden** (no unit anywhere). Partial 2nd = **audiobookshelf** (compose+manifest unit on USB).
|
||||
|
||||
## 3. Phase 3 — DISCOVERY PROOF ✅
|
||||
immich toggled (NOT deployed; live `app.yaml` absent; unit only at `…/felhom-usb/backups/primary/immich`),
|
||||
run via `POST /backup/offbox/run`:
|
||||
```
|
||||
[offbox] initialized restic repo
|
||||
[offbox] backed up immich (/mnt/felhom-drives/felhom-usb/backups/primary/immich)
|
||||
[offbox] backup OK: 1 app(s) backed up, 1 snapshot(s), 34s
|
||||
```
|
||||
`LastStatus=ok`, `SnapshotCount=1`, no error. The backed-up **src is the USB path** — proving discovery
|
||||
scanned the storage registry (schedulable path ∪ systemDataPath) and found the USB unit. **Pre-fix
|
||||
(v0.103.0)** this resolved to `systemDataPath` (`/mnt/sys_drive/felhom-data/backups/primary/immich`, which
|
||||
does not exist) → 0 snapshots reported `ok`. The regression scenario is closed against the real box.
|
||||
|
||||
## 4. Phase 4 — NO-SILENT-SUCCESS ✅
|
||||
- **0/N hard error:** only vaultwarden toggled (no unit). Run →
|
||||
`LastStatus=error`, `LastError="off-box backup produced no snapshots: 1 app(s) toggled but no recovery
|
||||
unit was found on any connected drive (missing: vaultwarden)"`, and the operator alert fired:
|
||||
`[ERROR] [offbox] backup failed after 5s: …` → `[INFO] Event pushed: backup_failed (error) — Off-box
|
||||
(NAS) mentés sikertelen`. (Controller-side alert pipeline confirmed via the pushed `backup_failed`
|
||||
event; downstream hub→Resend email delivery is the hub's leg — operator inbox confirmation is Viktor's.)
|
||||
- **Partial warning:** immich + audiobookshelf + vaultwarden toggled. Run →
|
||||
`LastStatus=ok`, 2 backed up (`2 app(s) backed up, 1 skipped (no unit), 2 snapshot(s), 45s`),
|
||||
`LastWarning="Figyelmeztetés: 1 alkalmazásnak nincs elérhető mentése, ezek kimaradtak: vaultwarden"`,
|
||||
**rendered on `/backups`** (confirmed in the page HTML, `--warn` style). No false "all good".
|
||||
|
||||
## 5. Phase 5 — ROUND-TRIP INTEGRITY ✅
|
||||
Restore immich via `POST /backup/offbox/restore` → non-destructive to `<DataDir>/offbox-restore/immich`.
|
||||
Full recovery unit came back (`compose/{app.yaml,.felhom.yml,docker-compose.yml}`, `manifest.json`,
|
||||
`db-dumps/immich-postgres.sql`). **sha256 of the restored 45 MB db-dump = the USB source, exactly**
|
||||
(`d78466fcbf595b488ba8b962fcc521f68fbe6f36b8b8e57815a6124bccf4526b`). The box holds genuinely recoverable
|
||||
data through the real restore endpoint.
|
||||
|
||||
## 6. Phase 6 — PERSISTENCE ✅
|
||||
Controller restarted (bootstrap) → healthy. offbox config (enabled, host/user/repo_path), `LastStatus=ok`,
|
||||
`SnapshotCount`, `LastWarning`, the per-app toggles, and the 0600 secret files (`ssh_key`, `repo_password`,
|
||||
`known_hosts`) all survived — settings.json persistence intact.
|
||||
|
||||
## 7. Fork resolutions
|
||||
1. **Topology (fork 1):** **direct — confirmed.** Residential IPv4 → box:23; no ep0 concentrator, no WG.
|
||||
2. **Account model (fork 2):** **sub-account-per-customer — confirmed.** Fresh console sub-account with its
|
||||
own `…-subN.your-storagebox.de` hostname, chrooted (base dir presented as `/home`), **RepoPath
|
||||
convention `/home/<name>`**, host key shared with the main account (one pinned line covers both).
|
||||
Password-free *intent* holds, but **the console SSH-key field is unreliable (F3)** → the repeatable
|
||||
install path is `ssh-copy-id -p 23 -s -f` with the sub-account password.
|
||||
3. **PBS-namespace / WG tier (fork 3):** deferred — restic-to-box is the pilot-era offsite. Not built.
|
||||
4. **Custody (fork 4) — surface RECORDED, design OPEN.** Per-customer secrets, controller-managed, 0600 in
|
||||
the controller data dir: **restic repo password** (`data/offbox/repo_password`, 32-byte hex,
|
||||
auto-generated on first config) + **SSH private key** (`data/offbox/ssh_key`). **Neither is independently
|
||||
recoverable today** — both ride DR only via the PBS whole-CT snapshot of the guest rootfs. Recovery
|
||||
ordering: offsite-restic recovery depends on first recovering these from the PBS tier. An escrow/custody
|
||||
design is the load-bearing open follow-up before any real reliance on this tier.
|
||||
|
||||
## 8. Throughput (observed, residential upload)
|
||||
immich cold backup (45 MB) + repo init = 34 s; a dedup run (immich unchanged) + audiobookshelf cold = 45 s;
|
||||
a full-dedup run = 44 s. The 45 MB upload itself is fast (~consistent with the spike's ~4.3 MB/s residential
|
||||
upstream); **`forget --prune` + `snapshots/stats` over SFTP dominate the wall-clock on a small repo** (dedup
|
||||
backups finish in ~3 s but the run is ~44 s). Fine for a daily schedule; note the first seed of a real
|
||||
tens-of-GB app is upstream-bound (hours) — heads-up on onboarding.
|
||||
|
||||
## 9. Teardown — KEPT (Viktor's decision, 🛑)
|
||||
Left standing as a demo-offsite (€4.06/mo): the fresh sub-account `u629193-sub1`, the offbox config, and the
|
||||
box repo `/home/felhom-demo-repo` (immich + audiobookshelf snapshots). Config tidied — the test-only
|
||||
**vaultwarden** toggle removed and a final clean run left `LastStatus=ok`, 2 snapshots, **no warning**;
|
||||
toggled = {audiobookshelf, immich}. Scratch removed (the `/tmp` keypair copy — the durable key lives 0600 in
|
||||
`data/offbox/ssh_key`; the 45 MB verify-restore). Untouched: the main account's other keys, the USB real
|
||||
units, ep0's `felhom-offsite`/mount.
|
||||
**⚠️ Standing-reliance caveat:** the scratch restic repo password is now semi-permanent and **NOT
|
||||
independently recoverable** → any real dependence on this demo-offsite needs the fork-4 custody decision first.
|
||||
|
||||
## 10. NEXT (to fully productionize)
|
||||
- **Fork-4 custody/escrow spec** — the one blocker for real reliance (repo password + SSH key recovery
|
||||
independent of the PBS whole-CT).
|
||||
- **Per-customer provisioning runbook** for Peti: sub-account create + key install (`ssh-copy-id -s -f`
|
||||
until the console key field is fixed / F3) + known_hosts pin + RepoPath `/home/<name>` + offbox config.
|
||||
- **F3** (Hetzner console sub-account SSH-key field silently not applying) — recurred; work around with
|
||||
`ssh-copy-id -s -f`, or raise with Hetzner.
|
||||
- **Tier-2 `TestBackupTier2Restore_DoubleClickRefused` flake** — separate backup-suite test-race debt
|
||||
(pre-existing; see controller v0.104.0 REPORT Observations).
|
||||
Reference in New Issue
Block a user