diff --git a/CONTEXT.md b/CONTEXT.md index d87220b..58dcbba 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,17 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-09 — offsite hardening bundle SHIPPED+LIVE (hub v0.39.0 + controller v0.107.0 + agent v0.78.0).** + The e2e's sharp edges closed before slices 3–4: **F4** "Re-issue offsite credentials" (explicit operator + recovery for consumed-password dead-ends — exactly-1 label scope, fresh one-time secret, ConfigVersion + bump; `hetznerapi.ResetBoxPassword` added for dedicated); **F2** host-key scan retry (~60s ladder — first + save survives fresh-subaccount DNS lag); **F5** submit-disable + in-flight notice; **key-auth-first bridge** + (working installed key → re-pin+reconfigure WITHOUT consuming; fingerprint verify never bypassed; kills the + stale-descriptor consume-404 loop); **staged-secret wipe** (agent `DELETE /escrow/stage-secret`; controller + calls it on every flip to `escrowed`; the e2e's lingering staged file wiped live through the new endpoint, + idempotency proven). All red-proofed. F4 reset NOT exercised live (demo credential untouchable — throwaway + customer under supervision when needed). NEXT: SLICE 3 auto-confirm, SLICE 4 soft-quota, Peti runbook. + - **2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS.** Full loop through the real pipeline on `demo-felhom` against the new dedicated-project pool box (611714 / `u629488`, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password → diff --git a/REPORT.md b/REPORT.md index b38fa85..89bf8a1 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,32 +2,32 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## RUNBOOK — offsite provisioning LIVE end-to-end (supervised) — 2026-07-09 — **PASS** +## TASK — offsite provisioning hardening (F2/F4/F5 + key-auth-first + staged-secret wipe) — 2026-07-09 — DEPLOYED -**Full record:** `documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. -**Shipped this run (felhom.eu):** manifest wiring `0a65f2b` (HETZNER_* env from Secret/storagebox), -**hub v0.38.1** `7c70c54` + bump `6d5b3c1` (LIVE: Synced/Healthy, `Offsite provisioning enabled -(pool_box=611714, location=fsn1)`). Pairs with controller v0.106.1 (that repo's REPORT). +**Shipped (felhom.eu):** hub **v0.39.0** `17cc67f` + bump `17d9af5` — LIVE (Synced/Healthy, provisioner +enabled). Bundle pairs: controller **v0.107.0** (key-auth-first bridge + wipe-on-escrowed; live on 9201) + +agent **v0.78.0** (`DELETE /escrow/stage-secret`; live on felhom-pve, 56/56). All three green-gated with +companion red-proofs. -The first live run of the SLICE 1+2 loop on `demo-felhom` against the new dedicated-project pool box: -hub UI save → sub-account provisioned (269008, `u629488-sub1`) → non-secret descriptor + one-time password → -controller config-refresh → apply-bridge (verify-pin, consume-once, pinned `ssh-copy-id`, offbox configured, -`EscrowState=pending`) → fork-4 run-gate refusal proven → escrow confirm (Option A — the repo password was -already escrowed under Viktor's R) → **first run: restic repo initialized, 2 apps, 2 snapshots on the new box, -key-only** (independently verified by SFTP-listing the repo with the bridge-installed key + pinned host key). +**Hub changes:** +- **F4 (pilot-gating) — "Re-issue offsite credentials":** `Provisioner.ReissueCredentials` + UI button + (confirm-gated, shown only when provisioned) + `POST /configs/{id}/offsite-reissue`. Resets the + sub-account password (or dedicated-box password via the new `hetznerapi.ResetBoxPassword`), stores a + FRESH one-time secret, re-saves the config unchanged → `ConfigVersion` bump → the stuck guest's next + refresh re-runs the bridge. **Hard-scoped:** refuses unless the `felhom-customer=` label lookup finds + exactly 1 resource (red-proofed). Explicit action only — never implicit rotation. Password never logged. +- **F2:** `scanWithRetry` — the host-key scan retries through fresh-subaccount DNS lag (2/4/8/16/30s ≈ 60s, + inside the F1 3-min detached ctx; fail-closed past budget; red-proofed). +- **F5:** the config form disables its submit buttons + shows an in-flight notice on submit (the re-click + bait that caused live F1). -**Live bugs found + fixed mid-run:** -- **F1 (hub v0.38.1):** a re-click canceled `r.Context()` between `CreateSubaccount` and - `SaveOneTimeSecret` → sub-account stranded with its password lost. `applyOffsite` now provisions on - `context.WithoutCancel` + 3-min timeout. Test `TestApplyOffsite_ClientDisconnectMidProvision` + red-proof - (raw-ctx shape reproduces the exact live error). -- **F3 (controller v0.106.1):** `ssh-copy-id -s` mktemp's under `~/.ssh`, absent from the container image — - install died locally AFTER consuming the password (fail-safe held; password never transmitted). +**One-off cleanup (executed):** the staged secret lingering on felhom-pve from the e2e's Option-A confirm +was wiped through the NEW agent endpoint (controller container → pinned local API → `{"removed":true}`), +verified gone; idempotent re-wipe returned `{"removed":false}`. No secret printed. -**Open findings:** F2 fresh-subaccount DNS lag → first-save scan 502 (retry-with-backoff wanted); -F4 no hub password-reset path (recovery = delete sub-account + re-save); F5 no save spinner (the re-click -bait); F6 Hetzner box-level 423 action lock on concurrent saves (fail-closed, correct). +**Deliberately NOT exercised live:** the F4 reset itself (it would rotate the demo's working credential — +touching it is forbidden; a live exercise needs a throwaway customer under supervision). Unit red-proofs +cover the guards. F5 is template JS — verified on the next operator save. -**Standing state:** demo offsite now targets the NEW box, `escrowed`, un-paused; the old-box offsite -(`u629193-sub1`) is a frozen archive until a NEW-box restore round-trip is verified. -**NEXT:** SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota), F2/F4/F5, Peti/pilot onboarding runbook. +**NEXT:** SLICE 3 (escrow auto-confirm — inherits wipe-on-escrowed), SLICE 4 (soft-quota on `quota_gb`), +NEW-box restore round-trip → retire the old-box archive, Peti onboarding runbook. diff --git a/hub/README.md b/hub/README.md index 272e6cc..e84df01 100644 --- a/hub/README.md +++ b/hub/README.md @@ -79,9 +79,17 @@ the controller re-pulls. The controller consumes the one-time password at `POST (customer-API-key auth, single-use) — then installs its key and the hub resets the box password (SLICE 2+). **Fail-closed:** a provisioning error returns 502 and saves nothing. **Secrets:** the Hetzner token (`HETZNER_TOKEN`, out-of-band) and every generated password are NEVER logged / committed / in `ConfigJSON`. -**PREREQUISITE for live use:** the token MUST be scoped to a **dedicated Hetzner project** (the current shared -project token can delete ep0 — spike §6). Absent token → the UI still renders; saving with offsite enabled -returns "not configured". SLICE 2 (controller apply-bridge), SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota) follow. +**PREREQUISITE for live use:** the token MUST be scoped to a **dedicated Hetzner project** (the shared +project token can delete ep0 — spike §6); check the scope by listing boxes with it. Absent token → the UI +still renders; saving with offsite enabled returns "not configured". **Live-validated end-to-end 2026-07-09** +(`documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`): SLICE 2 (controller apply-bridge) +is shipped and applied. +**Hardening (v0.38.1–v0.39.0):** provisioning runs on a client-disconnect-proof detached context (F1); the +host-key scan retries through fresh-subaccount DNS lag (~60s ladder, F2); the save button disables with an +in-flight notice (F5); and **"Re-issue offsite credentials"** (F4, `POST /configs/{id}/offsite-reissue`) is +the explicit recovery for a consumed-password dead-end — resets ONLY the resource labelled for that customer +(refuses unless exactly 1), stores a fresh one-time secret, bumps `ConfigVersion`. Never implicit rotation. +SLICE 3 (escrow auto-confirm) and SLICE 4 (soft-quota) follow. ### Infrastructure Backup — RETIRED (Phase-1, 2026-06-16, hub v0.12.0)