validation record: offsite provisioning live e2e PASS (2026-07-09) + CONTEXT/REPORT

Full hub->Hetzner->controller loop validated on demo-felhom against the new
dedicated-project pool box; 2 snapshots on the new box, key-only. F1/F3 fixed
mid-run (hub v0.38.1, controller v0.106.1); F2/F4/F5/F6 recorded open.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CONTEXT.md

@@ -3,6 +3,23 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 
+- **2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS.**
+  Full loop through the real pipeline on `demo-felhom` against the new dedicated-project pool box (611714 /
+  `u629488`, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password →
+  controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned `ssh-copy-id` →
+  `EscrowState=pending`) → run-gate refusal proven → escrow confirm (Option A: the repo password was already
+  escrowed — `WriteOffboxSecrets` preserves it) → first run: **restic repo + 2 snapshots on the new box,
+  key-only**. Two live bugs found+fixed mid-run: **F1** hub v0.38.1 (`applyOffsite` on `context.WithoutCancel` —
+  a re-click canceled the provision between create and password-store, stranding the sub-account) and **F3**
+  controller v0.106.1 (`ssh-copy-id -s` needs `~/.ssh` to exist in the container — died AFTER consuming the
+  password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no
+  password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait).
+  Phase-0 wiring: `manifests/hub.yaml` now injects `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` from
+  Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as
+  frozen archive until a NEW-box restore round-trip. Record:
+  `documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. NEXT: SLICE 3 (auto-confirm),
+  SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook.
+
 - **2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0).** The controller apply-bridge:
   on startup it reconciles the hub-served `offsite:` descriptor into a key-only offbox target
   (`controller/internal/offsiteapply.Bridge`) — **verify-pin the box host key against the hub-captured

REPORT.md

@@ -2,85 +2,32 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## TASK GL-7 — customer page: passphrase hardening + install-command generator (hub v0.36.0) — 2026-07-09
+## RUNBOOK — offsite provisioning LIVE end-to-end (supervised) — 2026-07-09 — **PASS**
 
-**Shipped:** felhom.eu `844fbfa7` (code) + `02c748eb` (manifest bump); **hub v0.36.0 LIVE** on k3s
-(ArgoCD app `felhom` Synced/Healthy, rollout confirmed, live image `felhom-hub:0.36.0`,
-`Listening on :8080`). felhom.eu only — agent + host-install untouched. Two coupled, security-first
-changes to the operator customer page, plus the Peti tester agreement.
+**Full record:** `documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`.
+**Shipped this run (felhom.eu):** manifest wiring `0a65f2b` (HETZNER_* env from Secret/storagebox),
+**hub v0.38.1** `7c70c54` + bump `6d5b3c1` (LIVE: Synced/Healthy, `Offsite provisioning enabled
+(pool_box=611714, location=fsn1)`). Pairs with controller v0.106.1 (that repo's REPORT).
 
-### Part 1 — passphrase hardening (the security win, ships first)
+The first live run of the SLICE 1+2 loop on `demo-felhom` against the new dedicated-project pool box:
+hub UI save → sub-account provisioned (269008, `u629488-sub1`) → non-secret descriptor + one-time password →
+controller config-refresh → apply-bridge (verify-pin, consume-once, pinned `ssh-copy-id`, offbox configured,
+`EscrowState=pending`) → fork-4 run-gate refusal proven → escrow confirm (Option A — the repo password was
+already escrowed under Viktor's R) → **first run: restic repo initialized, 2 apps, 2 snapshots on the new box,
+key-only** (independently verified by SFTP-listing the repo with the bridge-installed key + pinned host key).
 
-The per-customer retrieval passphrase was in cleartext **twice**: the visible `#retrieval-pw` node
-and baked into the Option-3 debug curl's `X-Retrieval-Password:` header (a copyable command carrying
-the secret). Now:
-- `#retrieval-pw` renders a **masked bullet run by default**, with a **Reveal** (`toggleSecret`) and
-  a **copy** (`copySecret`) control; the value lives in `data-secret` (the page's existing reveal
-  model — the value still ships in the DOM, unchanged from before).
-- The Option-3 command carries a **`<YOUR-RETRIEVAL-PASSWORD>` placeholder** — the secret is never in
-  a copyable command block.
-- A **zero-secret-in-DOM** reveal-on-demand fetch is a deliberate future follow-up, explicitly NOT
-  scoped here (kept the change tight and reviewable).
+**Live bugs found + fixed mid-run:**
+- **F1 (hub v0.38.1):** a re-click canceled `r.Context()` between `CreateSubaccount` and
+  `SaveOneTimeSecret` → sub-account stranded with its password lost. `applyOffsite` now provisions on
+  `context.WithoutCancel` + 3-min timeout. Test `TestApplyOffsite_ClientDisconnectMidProvision` + red-proof
+  (raw-ctx shape reproduces the exact live error).
+- **F3 (controller v0.106.1):** `ssh-copy-id -s` mktemp's under `~/.ssh`, absent from the container image —
+  install died locally AFTER consuming the password (fail-safe held; password never transmitted).
 
-### Part 2 — interactive install-command generator
+**Open findings:** F2 fresh-subaccount DNS lag → first-save scan 502 (retry-with-backoff wanted);
+F4 no hub password-reset path (recovery = delete sub-account + re-save); F5 no save spinner (the re-click
+bait); F6 Hetzner box-level 423 action lock on concurrent saves (fail-closed, correct).
 
-The three hard-coded install `<code>` blocks became a **client-side builder** (vanilla JS — no
-framework, CDN, or network call) that assembles a live-updating command from form controls:
-- Emits **only real host-install v1.12.0 flags** in a **download-then-run** shape (never `curl | bash`).
-- CustomerID prefilled from the server (`pageData.ScriptVersion` + `data-customer-id`); a **byo**
-  selection **requires** `--cores`/`--memory` (client-side enforcement via `.gen-req` + a `gen-msg`
-  prompt); caps/mode are placeholders, never silent defaults.
-- **Graceful JS-off static fallback**: the Option-1/2 code nodes retain a
-  `--customer-id … --mode <appliance|byo>` command.
-- **Curated control surface** (mode/cores/memory/vmid/node/acl-storages/operator-pubkey-file/
-  preserve-state-from + skip-provision/dry-run/preflight-only/allow-new-leaf). The **seven
-  dangerous/operator-only flags** (`--force`, `--rotate-recovery`, `--enable-oob`, `--remove-golden`,
-  `--uninstall`, `--adopt-pool`, `--rescope-acl`) are **never offered as controls**.
-
-`configs.go`: `const hostInstallVersion = "1.12.0"`; `pageData.ScriptVersion` added + populated.
-`style.css`: `.gen-controls`/`.gen-radios`/`.gen-radio`/`.gen-check(s)`/number-inputs/`.gen-msg`
-(dark palette, 2px radius).
-
-### Part 3 — tests (green gate passed on committed source)
-
-- **`TestTemplates_PassphraseHardened`** — the secret is NOT in the Option-3 command (placeholder is),
-  `#retrieval-pw` is masked-by-default (bullet-entity run), `data-secret` populated, reveal+copy
-  controls present. **Red-proof:** revert the Option-3 block to `{{.Config.RetrievalPassword}}` → the
-  "not-in-command" / masked assertions fail.
-- **`TestTemplates_InstallGenerator`** — every curated control id renders, script version +
-  `data-customer-id` present, the JS-off static fallback command present, and **none of the seven
-  excluded flags appear anywhere on the page**.
-- `go build ./... && go vet ./... && go test ./internal/web/` → **ok** on the committed source (`844fbfa7`).
-
-### Part 4 — deploy (GitOps)
-
-Built `felhom-hub:0.36.0` on 180 from the pushed source, bumped `manifests/hub.yaml` (0.35.0→0.36.0),
-ArgoCD hard-refresh + sync → Synced/Healthy, `deploy/hub` rolled out, live image + startup log
-confirmed. **Gotcha caught:** the first build produced a stale image because the Part-1/Part-2 commits
-were committed locally but not pushed before building (`build.sh` pulls origin) — fixed by pushing
-first, then rebuilding.
-
-### Part 5 — tester agreement + closeout
-
-`documentation/pilot/PETI-tester-agreement.md` written: the BYO trust boundary (break-glass/OOB/WG
-all OFF, non-root agent, pool-scoped token, TLS+per-guest-token local API), the honest limitations
-(no offsite backup yet; physically-removable-drive caveat F2; pool-reassert-bring-up-only F7;
-`:53`-must-be-free F6), exit rights (uninstall keeps drives/data/hub-record), Peti's box facts
-(80c/128 GB → caps **12/32768**, `sajatfelhom.hu` tunnel re-point), the byo install command, and the
-onboarding sequence (preflight → install → **G9 password-set** → tunnel → **backup-green**).
-GO-LIVE-PACKAGE updated: GL-7 ✔; **G9/G10 → done**, **G11 → done** (with the one open pilot question
-folded into the agreement's pre-install checklist); decision log + open-questions updated.
-
-### The one remaining pilot question (owned by the agreement)
-
-Peti's **local backup TARGET** — `felhom-pbs` is unreachable from his LAN, so the default
-`--acl-storages "local local-lvm felhom-pbs"` is wrong for him. Confirm at onboarding **whether his
-box has a second disk/pool**: if yes it becomes the `--acl-storages` backup target (real local
-resilience); if no, backups share the guest's pool (degraded — one drive failure loses both, must be
-stated to him). Changes only the `--acl-storages` value, nothing structural.
-
-### Verification note
-
-The hub UI is operator-password-gated — CC cannot log in, so per the repo policy UI changes are
-verified via the render tests (both green) plus the deploy checks (image/rollout/startup log), not a
-live browser session.
+**Standing state:** demo offsite now targets the NEW box, `escrowed`, un-paused; the old-box offsite
+(`u629193-sub1`) is a frozen archive until a NEW-box restore round-trip is verified.
+**NEXT:** SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota), F2/F4/F5, Peti/pilot onboarding runbook.

documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md

@@ -0,0 +1,110 @@
+# VALIDATION — offsite provisioning LIVE end-to-end (hub → Hetzner → controller), 2026-07-09
+
+**Class:** Risky / supervised operational validation (runbook execution). **Operator on 🛑 steps:** Viktor.
+**Scope:** first live run of the SLICE 1+2 hub-driven offsite provisioning loop on the demo customer
+(`demo-felhom`), against the new dedicated-project pool box (`storage-box-pool-1`, id **611714**, user
+`u629488`, project 15282031, BX11 fsn1). Closed through the fork-4 escrow gate to first snapshots.
+
+## VERDICT: **PASS** (with 2 live bugs found + fixed mid-run, 2 findings deferred)
+
+The complete loop ran through the REAL pipeline — hub UI save → Hetzner sub-account provision →
+descriptor + one-time password → controller config-pull → self-restart → apply-bridge (verify-pin →
+consume-once → pinned key install → offbox configured → `EscrowState=pending`) → run-gate refusal →
+escrow confirm → first run → **restic repo + 2 snapshots on the new box, key-only**. No hand-set state
+anywhere (anti-F9 held: every transition via the shipped endpoints/jobs).
+
+Shipped mid-run: **hub v0.38.1** (F1) + **controller v0.106.1** (F3). Final versions live:
+hub 0.38.1, controller 0.106.1.
+
+## Phase log
+
+### Phase 0 — wiring (secret + manifest)
+- Secret `felhom-system/storagebox` keys were already correct-shaped (`HETZNER_TOKEN`,
+  `HETZNER_POOL_BOX_ID=611714` numeric); `HETZNER_LOCATION` optional (hub defaults `fsn1`).
+- **Scope check FAILED first:** the stored token listed only the OLD project's box (611421/`u629193` — ep0's
+  PBS box) → it was the shared-project token the runbook forbids. 🛑 Viktor swapped in the dedicated-project
+  token (pasted via `read -rs` + `kubectl patch --stringData`, never on argv/logs). Re-check: token sees
+  ONLY 611714/`u629488` → scoped correctly. **Scope-check method (reusable):** list `/v1/storage_boxes`
+  with the token from a shell var; the visible set IS the blast radius.
+- Manifest `manifests/hub.yaml`: `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` via `secretKeyRef` (optional:true)
+  + literal `HETZNER_LOCATION=fsn1` (commit `0a65f2b`); synced only AFTER the token swap. Hub logged
+  `Offsite provisioning enabled (pool_box=611714, location=fsn1)`.
+
+### Phase 1 — pre-flight (all clean)
+Controller 0.106.0 healthy, bridge dormant; customer id is **`demo-felhom`** (`demo-felhom-01` is the
+host id); `config_version=3`, no `offsite` key; `one_time_secrets` empty; box 611714: 0 subaccounts.
+
+### Phase 2 — provisioning saves (where the bugs lived)
+- **Save 1 (3 presses, 20:50 CEST): F1.** The ~25s spinner-less save invited re-clicks. Press 1 created
+  sub-account 268985 on Hetzner, but the abandoned request's `r.Context()` was canceled between
+  `CreateSubaccount` and `SaveOneTimeSecret` → **password lost forever** (stranded resource). Press 2 hit
+  Hetzner's box-level HTTP 423 action lock (fail-closed 502, correct). Press 3 took the idempotent path and
+  saved the descriptor (v4) — with no stored password.
+  **Fix: hub v0.38.1** — `applyOffsite` provisions on `context.WithoutCancel` + 3-min absolute timeout
+  (create→wait→store is the atom). Regression test cancels the request mid-create; red-proof against the
+  raw-ctx shape reproduced the exact live error. Recovery: deleted stranded 268985 (zero data).
+- **Save 2 (single press, 21:08): F2 (open).** Fresh create 268988 + password STORED (v0.38.1 ordering
+  proved itself), then the host-key scan failed: `lookup u629488-sub1.your-storagebox.de … no such host` —
+  a fresh sub-account's DNS takes seconds-to-a-minute. Fail-closed 502, nothing saved, state recoverable.
+- **Save 3 (21:10):** idempotent path (no 2nd sub-account — the runbook's idempotency probe, live),
+  DNS now resolved, scan OK → descriptor v5 saved, password still unconsumed.
+
+### Phase 3 — controller applies
+- **v4 stale-descriptor noise (bonus proof):** the controller had picked up v4 earlier; the bridge ran
+  scan+verify then consume → 404 → fail-safe abort with the designed retry note. Fail-safe proven live.
+- **v5 attempt (21:25): F3.** Bridge verified the pin, CONSUMED the password, then `ssh-copy-id -s` died
+  **locally**: SFTP mode mktemp's under `~/.ssh` and the container image has no `/root/.ssh`. The loud
+  "password is spent — reset on the hub" fail-safe fired (no marker, no offbox config; the password never
+  left the box). Container probes (no secrets: throwaway key + wrong password) isolated it AND proved the
+  TOCTOU-hardened pin chain sound: with `~/.ssh` present, pinned known_hosts + `StrictHostKeyChecking=yes`
+  verified cleanly and wrong-password failed as `Permission denied` (sshpass exit 5).
+  **Fix: controller v0.106.1** — the installer ensures `~/.ssh` (0700) exists. Deployed to 9201.
+  Recovery: deleted spent 268988; 🛑 save 4 (21:36, first press) → sub-account **269008** + fresh password
+  → descriptor v6.
+- **v6 apply (21:45:45–49 CEST) — CLEAN:** config-refresh → self-restart → bridge:
+  `offsite configured for u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo (pending key escrow)`.
+
+**Verification table (all ✅):** consumed exactly once (`consumed_at` set, single row, read-only DB check);
+`data/offbox/`: `ssh_key` 0600 + pinned `known_hosts` + `applied_marker` (dir 0700); offbox target = the
+descriptor with `escrow_state=pending`; agent escrow stage-push succeeded; **run-gate refusal** (manual
+`POST /backup/offbox/run` → "A NAS-mentés a kulcs letétbe helyezésére vár."); **restart idempotency**
+(bootstrap restart → zero bridge activity, no re-consume).
+
+### Phase 4 — fingerprint-flip spot check: SKIPPED (time-boxed)
+Covered by the unit red-proof (mismatch → refuse before consume); the adjacent fail-safe paths were proven
+live twice above.
+
+### Phase 5 — escrow + first run (Option A, 🛑 Viktor approved)
+**Deviation from the runbook's premise:** `WriteOffboxSecrets` preserves an existing `repo_password` (the
+DR-inject seam), so the new target reuses the password **already escrowed** in the production hub blob under
+Viktor's R (validated same-day: the recovered password decrypted the real repo). The escrow claim was
+therefore already truthful → **no new ceremony**: `POST /backup/offbox/confirm-escrow` → `escrowed` →
+`POST /backup/offbox/run` → `initialized restic repo` → **2 apps (audiobookshelf, immich), 2 snapshots, 34s**.
+Independent proof: SFTP with the bridge-installed key + pinned host key listed the repo structure and the
+two snapshot objects on the new box. Old-box offsite (`u629193-sub1`) untouched — kept as frozen archive
+until a NEW-box restore round-trip is verified (Viktor's standing decision).
+
+## Findings
+| # | What | Status |
+|---|------|--------|
+| F1 | Client disconnect mid-provision strands a sub-account with a lost password (`r.Context()` canceled between create and store) | **FIXED hub v0.38.1** (+ test + red-proof) |
+| F2 | Fresh sub-account DNS lags creation → first-save host-key scan 502s (`no such host`); workaround: save again | **OPEN** — retry scan with backoff in `ProvisionOffsite` |
+| F3 | `ssh-copy-id -s` requires local `~/.ssh`; container image ships without it → install failed AFTER consuming the password | **FIXED controller v0.106.1** |
+| F4 | The designed recovery "reset the password on the hub" doesn't exist (no reset-password path); recovery today = delete sub-account + re-save | **OPEN** — hub needs a reset→re-store one-time-secret action (`ResetSubaccountPassword` already in CloudAPI) |
+| F5 | The save button gives no in-flight feedback (~25s) — the direct cause of F1's re-clicks | **OPEN** (UX: disable button/spinner) |
+| F6 | Concurrent saves hit Hetzner's box-level HTTP 423 action lock → fail-closed 502 (correct but confusing) | observed; optional retry-on-423 in hetznerapi |
+
+## Operational facts (for the next operator)
+- Env names the hub reads (main.go): `HETZNER_TOKEN`, `HETZNER_POOL_BOX_ID` (**numeric** box id, NOT the
+  name), `HETZNER_LOCATION` (default fsn1). Secret: `felhom-system/storagebox`.
+- The token MUST be the dedicated-project one — verify by listing boxes with it (must NOT see 611421).
+- Sub-account home dirs: `felhom-<customer_id>` on the parent box; controller-facing repo path is always
+  `/home/felhom-repo` (chroot). Deleted sub-account usernames are recycled (`-sub1` reused 3×).
+- Hetzner box `stats` lag real usage — don't use them as a freshness probe; SFTP-list the repo instead.
+- Controller report/refresh cadence: hub-report every 15 min → config-refresh on ACK → self-restart.
+  Worst-case descriptor-to-applied latency ≈ 15 min.
+
+## NEXT
+SLICE 3 (escrow auto-confirm — replaces the manual confirm exercised here), SLICE 4 (soft-quota on
+`quota_gb`), F2 scan-retry, F4 hub password-reset, F5 spinner, old-offsite retirement after a verified
+NEW-box restore, Peti/pilot onboarding runbook.