felhom.eu/documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md

VALIDATION — offsite provisioning LIVE end-to-end (hub → Hetzner → controller), 2026-07-09

Class: Risky / supervised operational validation (runbook execution). Operator on 🛑 steps: Viktor. Scope: first live run of the SLICE 1+2 hub-driven offsite provisioning loop on the demo customer (demo-felhom), against the new dedicated-project pool box (storage-box-pool-1, id 611714, user u629488, project 15282031, BX11 fsn1). Closed through the fork-4 escrow gate to first snapshots.

VERDICT: PASS (with 2 live bugs found + fixed mid-run, 2 findings deferred)

The complete loop ran through the REAL pipeline — hub UI save → Hetzner sub-account provision → descriptor + one-time password → controller config-pull → self-restart → apply-bridge (verify-pin → consume-once → pinned key install → offbox configured → EscrowState=pending) → run-gate refusal → escrow confirm → first run → restic repo + 2 snapshots on the new box, key-only. No hand-set state anywhere (anti-F9 held: every transition via the shipped endpoints/jobs).

Shipped mid-run: hub v0.38.1 (F1) + controller v0.106.1 (F3). Final versions live: hub 0.38.1, controller 0.106.1.

Phase log

Phase 0 — wiring (secret + manifest)

• Secret felhom-system/storagebox keys were already correct-shaped (HETZNER_TOKEN, HETZNER_POOL_BOX_ID=611714 numeric); HETZNER_LOCATION optional (hub defaults fsn1).
• Scope check FAILED first: the stored token listed only the OLD project's box (611421/u629193 — ep0's PBS box) → it was the shared-project token the runbook forbids. 🛑 Viktor swapped in the dedicated-project token (pasted via read -rs + kubectl patch --stringData, never on argv/logs). Re-check: token sees ONLY 611714/u629488 → scoped correctly. Scope-check method (reusable): list /v1/storage_boxes with the token from a shell var; the visible set IS the blast radius.
• Manifest manifests/hub.yaml: HETZNER_TOKEN/HETZNER_POOL_BOX_ID via secretKeyRef (optional:true)
  • literal HETZNER_LOCATION=fsn1 (commit 0a65f2b); synced only AFTER the token swap. Hub logged Offsite provisioning enabled (pool_box=611714, location=fsn1).

Phase 1 — pre-flight (all clean)

Controller 0.106.0 healthy, bridge dormant; customer id is demo-felhom (demo-felhom-01 is the host id); config_version=3, no offsite key; one_time_secrets empty; box 611714: 0 subaccounts.

Phase 2 — provisioning saves (where the bugs lived)

• Save 1 (3 presses, 20:50 CEST): F1. The ~25s spinner-less save invited re-clicks. Press 1 created sub-account 268985 on Hetzner, but the abandoned request's r.Context() was canceled between CreateSubaccount and SaveOneTimeSecret → password lost forever (stranded resource). Press 2 hit Hetzner's box-level HTTP 423 action lock (fail-closed 502, correct). Press 3 took the idempotent path and saved the descriptor (v4) — with no stored password. Fix: hub v0.38.1 — applyOffsite provisions on context.WithoutCancel + 3-min absolute timeout (create→wait→store is the atom). Regression test cancels the request mid-create; red-proof against the raw-ctx shape reproduced the exact live error. Recovery: deleted stranded 268985 (zero data).
• Save 2 (single press, 21:08): F2 (open). Fresh create 268988 + password STORED (v0.38.1 ordering proved itself), then the host-key scan failed: lookup u629488-sub1.your-storagebox.de … no such host — a fresh sub-account's DNS takes seconds-to-a-minute. Fail-closed 502, nothing saved, state recoverable.
• Save 3 (21:10): idempotent path (no 2nd sub-account — the runbook's idempotency probe, live), DNS now resolved, scan OK → descriptor v5 saved, password still unconsumed.

Phase 3 — controller applies

• v4 stale-descriptor noise (bonus proof): the controller had picked up v4 earlier; the bridge ran scan+verify then consume → 404 → fail-safe abort with the designed retry note. Fail-safe proven live.
• v5 attempt (21:25): F3. Bridge verified the pin, CONSUMED the password, then ssh-copy-id -s died locally: SFTP mode mktemp's under ~/.ssh and the container image has no /root/.ssh. The loud "password is spent — reset on the hub" fail-safe fired (no marker, no offbox config; the password never left the box). Container probes (no secrets: throwaway key + wrong password) isolated it AND proved the TOCTOU-hardened pin chain sound: with ~/.ssh present, pinned known_hosts + StrictHostKeyChecking=yes verified cleanly and wrong-password failed as Permission denied (sshpass exit 5). Fix: controller v0.106.1 — the installer ensures ~/.ssh (0700) exists. Deployed to 9201. Recovery: deleted spent 268988; 🛑 save 4 (21:36, first press) → sub-account 269008 + fresh password → descriptor v6.
• v6 apply (21:45:45–49 CEST) — CLEAN: config-refresh → self-restart → bridge: offsite configured for u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo (pending key escrow).

Verification table (all ✅): consumed exactly once (consumed_at set, single row, read-only DB check); data/offbox/: ssh_key 0600 + pinned known_hosts + applied_marker (dir 0700); offbox target = the descriptor with escrow_state=pending; agent escrow stage-push succeeded; run-gate refusal (manual POST /backup/offbox/run → "A NAS-mentés a kulcs letétbe helyezésére vár."); restart idempotency (bootstrap restart → zero bridge activity, no re-consume).

Phase 4 — fingerprint-flip spot check: SKIPPED (time-boxed)

Covered by the unit red-proof (mismatch → refuse before consume); the adjacent fail-safe paths were proven live twice above.

Phase 5 — escrow + first run (Option A, 🛑 Viktor approved)

Deviation from the runbook's premise: WriteOffboxSecrets preserves an existing repo_password (the DR-inject seam), so the new target reuses the password already escrowed in the production hub blob under Viktor's R (validated same-day: the recovered password decrypted the real repo). The escrow claim was therefore already truthful → no new ceremony: POST /backup/offbox/confirm-escrow → escrowed → POST /backup/offbox/run → initialized restic repo → 2 apps (audiobookshelf, immich), 2 snapshots, 34s. Independent proof: SFTP with the bridge-installed key + pinned host key listed the repo structure and the two snapshot objects on the new box. Old-box offsite (u629193-sub1) untouched — kept as frozen archive until a NEW-box restore round-trip is verified (Viktor's standing decision).

Findings

#	What	Status
F1	Client disconnect mid-provision strands a sub-account with a lost password (r.Context() canceled between create and store)	FIXED hub v0.38.1 (+ test + red-proof)
F2	Fresh sub-account DNS lags creation → first-save host-key scan 502s (no such host); workaround: save again	OPEN — retry scan with backoff in ProvisionOffsite
F3	ssh-copy-id -s requires local ~/.ssh; container image ships without it → install failed AFTER consuming the password	FIXED controller v0.106.1
F4	The designed recovery "reset the password on the hub" doesn't exist (no reset-password path); recovery today = delete sub-account + re-save	OPEN — hub needs a reset→re-store one-time-secret action (ResetSubaccountPassword already in CloudAPI)
F5	The save button gives no in-flight feedback (~25s) — the direct cause of F1's re-clicks	OPEN (UX: disable button/spinner)
F6	Concurrent saves hit Hetzner's box-level HTTP 423 action lock → fail-closed 502 (correct but confusing)	observed; optional retry-on-423 in hetznerapi

Operational facts (for the next operator)

• Env names the hub reads (main.go): HETZNER_TOKEN, HETZNER_POOL_BOX_ID (numeric box id, NOT the name), HETZNER_LOCATION (default fsn1). Secret: felhom-system/storagebox.
• The token MUST be the dedicated-project one — verify by listing boxes with it (must NOT see 611421).
• Sub-account home dirs: felhom-<customer_id> on the parent box; controller-facing repo path is always /home/felhom-repo (chroot). Deleted sub-account usernames are recycled (-sub1 reused 3×).
• Hetzner box stats lag real usage — don't use them as a freshness probe; SFTP-list the repo instead.
• Controller report/refresh cadence: hub-report every 15 min → config-refresh on ACK → self-restart. Worst-case descriptor-to-applied latency ≈ 15 min.

NEXT

SLICE 3 (escrow auto-confirm — replaces the manual confirm exercised here), SLICE 4 (soft-quota on quota_gb), F2 scan-retry, F4 hub password-reset, F5 spinner, old-offsite retirement after a verified NEW-box restore, Peti/pilot onboarding runbook.