admin
ecf9185605
Full hub->Hetzner->controller loop validated on demo-felhom against the new dedicated-project pool box; 2 snapshots on the new box, key-only. F1/F3 fixed mid-run (hub v0.38.1, controller v0.106.1); F2/F4/F5/F6 recorded open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
111 lines
8.7 KiB
Markdown
111 lines
8.7 KiB
Markdown
# VALIDATION — offsite provisioning LIVE end-to-end (hub → Hetzner → controller), 2026-07-09
|
||
|
||
**Class:** Risky / supervised operational validation (runbook execution). **Operator on 🛑 steps:** Viktor.
|
||
**Scope:** first live run of the SLICE 1+2 hub-driven offsite provisioning loop on the demo customer
|
||
(`demo-felhom`), against the new dedicated-project pool box (`storage-box-pool-1`, id **611714**, user
|
||
`u629488`, project 15282031, BX11 fsn1). Closed through the fork-4 escrow gate to first snapshots.
|
||
|
||
## VERDICT: **PASS** (with 2 live bugs found + fixed mid-run, 2 findings deferred)
|
||
|
||
The complete loop ran through the REAL pipeline — hub UI save → Hetzner sub-account provision →
|
||
descriptor + one-time password → controller config-pull → self-restart → apply-bridge (verify-pin →
|
||
consume-once → pinned key install → offbox configured → `EscrowState=pending`) → run-gate refusal →
|
||
escrow confirm → first run → **restic repo + 2 snapshots on the new box, key-only**. No hand-set state
|
||
anywhere (anti-F9 held: every transition via the shipped endpoints/jobs).
|
||
|
||
Shipped mid-run: **hub v0.38.1** (F1) + **controller v0.106.1** (F3). Final versions live:
|
||
hub 0.38.1, controller 0.106.1.
|
||
|
||
## Phase log
|
||
|
||
### Phase 0 — wiring (secret + manifest)
|
||
- Secret `felhom-system/storagebox` keys were already correct-shaped (`HETZNER_TOKEN`,
|
||
`HETZNER_POOL_BOX_ID=611714` numeric); `HETZNER_LOCATION` optional (hub defaults `fsn1`).
|
||
- **Scope check FAILED first:** the stored token listed only the OLD project's box (611421/`u629193` — ep0's
|
||
PBS box) → it was the shared-project token the runbook forbids. 🛑 Viktor swapped in the dedicated-project
|
||
token (pasted via `read -rs` + `kubectl patch --stringData`, never on argv/logs). Re-check: token sees
|
||
ONLY 611714/`u629488` → scoped correctly. **Scope-check method (reusable):** list `/v1/storage_boxes`
|
||
with the token from a shell var; the visible set IS the blast radius.
|
||
- Manifest `manifests/hub.yaml`: `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` via `secretKeyRef` (optional:true)
|
||
+ literal `HETZNER_LOCATION=fsn1` (commit `0a65f2b`); synced only AFTER the token swap. Hub logged
|
||
`Offsite provisioning enabled (pool_box=611714, location=fsn1)`.
|
||
|
||
### Phase 1 — pre-flight (all clean)
|
||
Controller 0.106.0 healthy, bridge dormant; customer id is **`demo-felhom`** (`demo-felhom-01` is the
|
||
host id); `config_version=3`, no `offsite` key; `one_time_secrets` empty; box 611714: 0 subaccounts.
|
||
|
||
### Phase 2 — provisioning saves (where the bugs lived)
|
||
- **Save 1 (3 presses, 20:50 CEST): F1.** The ~25s spinner-less save invited re-clicks. Press 1 created
|
||
sub-account 268985 on Hetzner, but the abandoned request's `r.Context()` was canceled between
|
||
`CreateSubaccount` and `SaveOneTimeSecret` → **password lost forever** (stranded resource). Press 2 hit
|
||
Hetzner's box-level HTTP 423 action lock (fail-closed 502, correct). Press 3 took the idempotent path and
|
||
saved the descriptor (v4) — with no stored password.
|
||
**Fix: hub v0.38.1** — `applyOffsite` provisions on `context.WithoutCancel` + 3-min absolute timeout
|
||
(create→wait→store is the atom). Regression test cancels the request mid-create; red-proof against the
|
||
raw-ctx shape reproduced the exact live error. Recovery: deleted stranded 268985 (zero data).
|
||
- **Save 2 (single press, 21:08): F2 (open).** Fresh create 268988 + password STORED (v0.38.1 ordering
|
||
proved itself), then the host-key scan failed: `lookup u629488-sub1.your-storagebox.de … no such host` —
|
||
a fresh sub-account's DNS takes seconds-to-a-minute. Fail-closed 502, nothing saved, state recoverable.
|
||
- **Save 3 (21:10):** idempotent path (no 2nd sub-account — the runbook's idempotency probe, live),
|
||
DNS now resolved, scan OK → descriptor v5 saved, password still unconsumed.
|
||
|
||
### Phase 3 — controller applies
|
||
- **v4 stale-descriptor noise (bonus proof):** the controller had picked up v4 earlier; the bridge ran
|
||
scan+verify then consume → 404 → fail-safe abort with the designed retry note. Fail-safe proven live.
|
||
- **v5 attempt (21:25): F3.** Bridge verified the pin, CONSUMED the password, then `ssh-copy-id -s` died
|
||
**locally**: SFTP mode mktemp's under `~/.ssh` and the container image has no `/root/.ssh`. The loud
|
||
"password is spent — reset on the hub" fail-safe fired (no marker, no offbox config; the password never
|
||
left the box). Container probes (no secrets: throwaway key + wrong password) isolated it AND proved the
|
||
TOCTOU-hardened pin chain sound: with `~/.ssh` present, pinned known_hosts + `StrictHostKeyChecking=yes`
|
||
verified cleanly and wrong-password failed as `Permission denied` (sshpass exit 5).
|
||
**Fix: controller v0.106.1** — the installer ensures `~/.ssh` (0700) exists. Deployed to 9201.
|
||
Recovery: deleted spent 268988; 🛑 save 4 (21:36, first press) → sub-account **269008** + fresh password
|
||
→ descriptor v6.
|
||
- **v6 apply (21:45:45–49 CEST) — CLEAN:** config-refresh → self-restart → bridge:
|
||
`offsite configured for u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo (pending key escrow)`.
|
||
|
||
**Verification table (all ✅):** consumed exactly once (`consumed_at` set, single row, read-only DB check);
|
||
`data/offbox/`: `ssh_key` 0600 + pinned `known_hosts` + `applied_marker` (dir 0700); offbox target = the
|
||
descriptor with `escrow_state=pending`; agent escrow stage-push succeeded; **run-gate refusal** (manual
|
||
`POST /backup/offbox/run` → "A NAS-mentés a kulcs letétbe helyezésére vár."); **restart idempotency**
|
||
(bootstrap restart → zero bridge activity, no re-consume).
|
||
|
||
### Phase 4 — fingerprint-flip spot check: SKIPPED (time-boxed)
|
||
Covered by the unit red-proof (mismatch → refuse before consume); the adjacent fail-safe paths were proven
|
||
live twice above.
|
||
|
||
### Phase 5 — escrow + first run (Option A, 🛑 Viktor approved)
|
||
**Deviation from the runbook's premise:** `WriteOffboxSecrets` preserves an existing `repo_password` (the
|
||
DR-inject seam), so the new target reuses the password **already escrowed** in the production hub blob under
|
||
Viktor's R (validated same-day: the recovered password decrypted the real repo). The escrow claim was
|
||
therefore already truthful → **no new ceremony**: `POST /backup/offbox/confirm-escrow` → `escrowed` →
|
||
`POST /backup/offbox/run` → `initialized restic repo` → **2 apps (audiobookshelf, immich), 2 snapshots, 34s**.
|
||
Independent proof: SFTP with the bridge-installed key + pinned host key listed the repo structure and the
|
||
two snapshot objects on the new box. Old-box offsite (`u629193-sub1`) untouched — kept as frozen archive
|
||
until a NEW-box restore round-trip is verified (Viktor's standing decision).
|
||
|
||
## Findings
|
||
| # | What | Status |
|
||
|---|------|--------|
|
||
| F1 | Client disconnect mid-provision strands a sub-account with a lost password (`r.Context()` canceled between create and store) | **FIXED hub v0.38.1** (+ test + red-proof) |
|
||
| F2 | Fresh sub-account DNS lags creation → first-save host-key scan 502s (`no such host`); workaround: save again | **OPEN** — retry scan with backoff in `ProvisionOffsite` |
|
||
| F3 | `ssh-copy-id -s` requires local `~/.ssh`; container image ships without it → install failed AFTER consuming the password | **FIXED controller v0.106.1** |
|
||
| F4 | The designed recovery "reset the password on the hub" doesn't exist (no reset-password path); recovery today = delete sub-account + re-save | **OPEN** — hub needs a reset→re-store one-time-secret action (`ResetSubaccountPassword` already in CloudAPI) |
|
||
| F5 | The save button gives no in-flight feedback (~25s) — the direct cause of F1's re-clicks | **OPEN** (UX: disable button/spinner) |
|
||
| F6 | Concurrent saves hit Hetzner's box-level HTTP 423 action lock → fail-closed 502 (correct but confusing) | observed; optional retry-on-423 in hetznerapi |
|
||
|
||
## Operational facts (for the next operator)
|
||
- Env names the hub reads (main.go): `HETZNER_TOKEN`, `HETZNER_POOL_BOX_ID` (**numeric** box id, NOT the
|
||
name), `HETZNER_LOCATION` (default fsn1). Secret: `felhom-system/storagebox`.
|
||
- The token MUST be the dedicated-project one — verify by listing boxes with it (must NOT see 611421).
|
||
- Sub-account home dirs: `felhom-<customer_id>` on the parent box; controller-facing repo path is always
|
||
`/home/felhom-repo` (chroot). Deleted sub-account usernames are recycled (`-sub1` reused 3×).
|
||
- Hetzner box `stats` lag real usage — don't use them as a freshness probe; SFTP-list the repo instead.
|
||
- Controller report/refresh cadence: hub-report every 15 min → config-refresh on ACK → self-restart.
|
||
Worst-case descriptor-to-applied latency ≈ 15 min.
|
||
|
||
## NEXT
|
||
SLICE 3 (escrow auto-confirm — replaces the manual confirm exercised here), SLICE 4 (soft-quota on
|
||
`quota_gb`), F2 scan-retry, F4 hub password-reset, F5 spinner, old-offsite retirement after a verified
|
||
NEW-box restore, Peti/pilot onboarding runbook.
|