# VALIDATION — offsite provisioning LIVE end-to-end (hub → Hetzner → controller), 2026-07-09

**Class:** Risky / supervised operational validation (runbook execution). **Operator on 🛑 steps:** Viktor.
**Scope:** first live run of the SLICE 1+2 hub-driven offsite provisioning loop on the demo customer
(`demo-felhom`), against the new dedicated-project pool box (`storage-box-pool-1`, id **611714**, user
`u629488`, project 15282031, BX11 fsn1). Closed through the fork-4 escrow gate to first snapshots.

## VERDICT: **PASS** (with 2 live bugs found + fixed mid-run, 2 findings deferred)

The complete loop ran through the REAL pipeline — hub UI save → Hetzner sub-account provision →
descriptor + one-time password → controller config-pull → self-restart → apply-bridge (verify-pin →
consume-once → pinned key install → offbox configured → `EscrowState=pending`) → run-gate refusal →
escrow confirm → first run → **restic repo + 2 snapshots on the new box, key-only**. No hand-set state
anywhere (anti-F9 held: every transition via the shipped endpoints/jobs).

Shipped mid-run: **hub v0.38.1** (F1) + **controller v0.106.1** (F3). Final versions live:
hub 0.38.1, controller 0.106.1.

## Phase log

### Phase 0 — wiring (secret + manifest)
- Secret `felhom-system/storagebox` keys were already correct-shaped (`HETZNER_TOKEN`,
  `HETZNER_POOL_BOX_ID=611714` numeric); `HETZNER_LOCATION` optional (hub defaults `fsn1`).
- **Scope check FAILED first:** the stored token listed only the OLD project's box (611421/`u629193` — ep0's
  PBS box) → it was the shared-project token the runbook forbids. 🛑 Viktor swapped in the dedicated-project
  token (pasted via `read -rs` + `kubectl patch --stringData`, never on argv/logs). Re-check: token sees
  ONLY 611714/`u629488` → scoped correctly. **Scope-check method (reusable):** list `/v1/storage_boxes`
  with the token from a shell var; the visible set IS the blast radius.
- Manifest `manifests/hub.yaml`: `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` via `secretKeyRef` (optional:true)
  + literal `HETZNER_LOCATION=fsn1` (commit `0a65f2b`); synced only AFTER the token swap. Hub logged
  `Offsite provisioning enabled (pool_box=611714, location=fsn1)`.

### Phase 1 — pre-flight (all clean)
Controller 0.106.0 healthy, bridge dormant; customer id is **`demo-felhom`** (`demo-felhom-01` is the
host id); `config_version=3`, no `offsite` key; `one_time_secrets` empty; box 611714: 0 subaccounts.

### Phase 2 — provisioning saves (where the bugs lived)
- **Save 1 (3 presses, 20:50 CEST): F1.** The ~25s spinner-less save invited re-clicks. Press 1 created
  sub-account 268985 on Hetzner, but the abandoned request's `r.Context()` was canceled between
  `CreateSubaccount` and `SaveOneTimeSecret` → **password lost forever** (stranded resource). Press 2 hit
  Hetzner's box-level HTTP 423 action lock (fail-closed 502, correct). Press 3 took the idempotent path and
  saved the descriptor (v4) — with no stored password.
  **Fix: hub v0.38.1** — `applyOffsite` provisions on `context.WithoutCancel` + 3-min absolute timeout
  (create→wait→store is the atom). Regression test cancels the request mid-create; red-proof against the
  raw-ctx shape reproduced the exact live error. Recovery: deleted stranded 268985 (zero data).
- **Save 2 (single press, 21:08): F2 (open).** Fresh create 268988 + password STORED (v0.38.1 ordering
  proved itself), then the host-key scan failed: `lookup u629488-sub1.your-storagebox.de … no such host` —
  a fresh sub-account's DNS takes seconds-to-a-minute. Fail-closed 502, nothing saved, state recoverable.
- **Save 3 (21:10):** idempotent path (no 2nd sub-account — the runbook's idempotency probe, live),
  DNS now resolved, scan OK → descriptor v5 saved, password still unconsumed.

### Phase 3 — controller applies
- **v4 stale-descriptor noise (bonus proof):** the controller had picked up v4 earlier; the bridge ran
  scan+verify then consume → 404 → fail-safe abort with the designed retry note. Fail-safe proven live.
- **v5 attempt (21:25): F3.** Bridge verified the pin, CONSUMED the password, then `ssh-copy-id -s` died
  **locally**: SFTP mode mktemp's under `~/.ssh` and the container image has no `/root/.ssh`. The loud
  "password is spent — reset on the hub" fail-safe fired (no marker, no offbox config; the password never
  left the box). Container probes (no secrets: throwaway key + wrong password) isolated it AND proved the
  TOCTOU-hardened pin chain sound: with `~/.ssh` present, pinned known_hosts + `StrictHostKeyChecking=yes`
  verified cleanly and wrong-password failed as `Permission denied` (sshpass exit 5).
  **Fix: controller v0.106.1** — the installer ensures `~/.ssh` (0700) exists. Deployed to 9201.
  Recovery: deleted spent 268988; 🛑 save 4 (21:36, first press) → sub-account **269008** + fresh password
  → descriptor v6.
- **v6 apply (21:45:45–49 CEST) — CLEAN:** config-refresh → self-restart → bridge:
  `offsite configured for u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo (pending key escrow)`.

**Verification table (all ✅):** consumed exactly once (`consumed_at` set, single row, read-only DB check);
`data/offbox/`: `ssh_key` 0600 + pinned `known_hosts` + `applied_marker` (dir 0700); offbox target = the
descriptor with `escrow_state=pending`; agent escrow stage-push succeeded; **run-gate refusal** (manual
`POST /backup/offbox/run` → "A NAS-mentés a kulcs letétbe helyezésére vár."); **restart idempotency**
(bootstrap restart → zero bridge activity, no re-consume).

### Phase 4 — fingerprint-flip spot check: SKIPPED (time-boxed)
Covered by the unit red-proof (mismatch → refuse before consume); the adjacent fail-safe paths were proven
live twice above.

### Phase 5 — escrow + first run (Option A, 🛑 Viktor approved)
**Deviation from the runbook's premise:** `WriteOffboxSecrets` preserves an existing `repo_password` (the
DR-inject seam), so the new target reuses the password **already escrowed** in the production hub blob under
Viktor's R (validated same-day: the recovered password decrypted the real repo). The escrow claim was
therefore already truthful → **no new ceremony**: `POST /backup/offbox/confirm-escrow` → `escrowed` →
`POST /backup/offbox/run` → `initialized restic repo` → **2 apps (audiobookshelf, immich), 2 snapshots, 34s**.
Independent proof: SFTP with the bridge-installed key + pinned host key listed the repo structure and the
two snapshot objects on the new box. Old-box offsite (`u629193-sub1`) untouched — kept as frozen archive
until a NEW-box restore round-trip is verified (Viktor's standing decision).

## Findings
| # | What | Status |
|---|------|--------|
| F1 | Client disconnect mid-provision strands a sub-account with a lost password (`r.Context()` canceled between create and store) | **FIXED hub v0.38.1** (+ test + red-proof) |
| F2 | Fresh sub-account DNS lags creation → first-save host-key scan 502s (`no such host`); workaround: save again | **OPEN** — retry scan with backoff in `ProvisionOffsite` |
| F3 | `ssh-copy-id -s` requires local `~/.ssh`; container image ships without it → install failed AFTER consuming the password | **FIXED controller v0.106.1** |
| F4 | The designed recovery "reset the password on the hub" doesn't exist (no reset-password path); recovery today = delete sub-account + re-save | **OPEN** — hub needs a reset→re-store one-time-secret action (`ResetSubaccountPassword` already in CloudAPI) |
| F5 | The save button gives no in-flight feedback (~25s) — the direct cause of F1's re-clicks | **OPEN** (UX: disable button/spinner) |
| F6 | Concurrent saves hit Hetzner's box-level HTTP 423 action lock → fail-closed 502 (correct but confusing) | observed; optional retry-on-423 in hetznerapi |

## Operational facts (for the next operator)
- Env names the hub reads (main.go): `HETZNER_TOKEN`, `HETZNER_POOL_BOX_ID` (**numeric** box id, NOT the
  name), `HETZNER_LOCATION` (default fsn1). Secret: `felhom-system/storagebox`.
- The token MUST be the dedicated-project one — verify by listing boxes with it (must NOT see 611421).
- Sub-account home dirs: `felhom-<customer_id>` on the parent box; controller-facing repo path is always
  `/home/felhom-repo` (chroot). Deleted sub-account usernames are recycled (`-sub1` reused 3×).
- Hetzner box `stats` lag real usage — don't use them as a freshness probe; SFTP-list the repo instead.
- Controller report/refresh cadence: hub-report every 15 min → config-refresh on ACK → self-restart.
  Worst-case descriptor-to-applied latency ≈ 15 min.

## NEXT
SLICE 3 (escrow auto-confirm — replaces the manual confirm exercised here), SLICE 4 (soft-quota on
`quota_gb`), F2 scan-retry, F4 hub password-reset, F5 spinner, old-offsite retirement after a verified
NEW-box restore, Peti/pilot onboarding runbook.