diff --git a/CONTEXT.md b/CONTEXT.md index ba1924e..d87220b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,23 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-09 — offsite provisioning VALIDATED LIVE end-to-end (hub v0.38.1 + controller v0.106.1) — PASS.** + Full loop through the real pipeline on `demo-felhom` against the new dedicated-project pool box (611714 / + `u629488`, project 15282031): UI save → sub-account 269008 provisioned → descriptor v6 + one-time password → + controller config-refresh (≤15 min) → apply-bridge (verify-pin → consume-once → pinned `ssh-copy-id` → + `EscrowState=pending`) → run-gate refusal proven → escrow confirm (Option A: the repo password was already + escrowed — `WriteOffboxSecrets` preserves it) → first run: **restic repo + 2 snapshots on the new box, + key-only**. Two live bugs found+fixed mid-run: **F1** hub v0.38.1 (`applyOffsite` on `context.WithoutCancel` — + a re-click canceled the provision between create and password-store, stranding the sub-account) and **F3** + controller v0.106.1 (`ssh-copy-id -s` needs `~/.ssh` to exist in the container — died AFTER consuming the + password). OPEN: F2 (fresh sub-account DNS lags → first-save scan 502s; retry-with-backoff), F4 (hub has no + password-reset path — recovery today is delete-subaccount + re-save), F5 (no save spinner → re-click bait). + Phase-0 wiring: `manifests/hub.yaml` now injects `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` from + Secret/storagebox; scoped-token check = list boxes with it (must see ONLY 611714). Old-box offsite kept as + frozen archive until a NEW-box restore round-trip. Record: + `documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. NEXT: SLICE 3 (auto-confirm), + SLICE 4 (soft-quota), F2/F4/F5, Peti onboarding runbook. + - **2026-07-09 — offsite provisioning SLICE 2 (hub v0.38.0 + controller v0.106.0).** The controller apply-bridge: on startup it reconciles the hub-served `offsite:` descriptor into a key-only offbox target (`controller/internal/offsiteapply.Bridge`) — **verify-pin the box host key against the hub-captured diff --git a/REPORT.md b/REPORT.md index 6b42052..b38fa85 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,85 +2,32 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## TASK GL-7 — customer page: passphrase hardening + install-command generator (hub v0.36.0) — 2026-07-09 +## RUNBOOK — offsite provisioning LIVE end-to-end (supervised) — 2026-07-09 — **PASS** -**Shipped:** felhom.eu `844fbfa7` (code) + `02c748eb` (manifest bump); **hub v0.36.0 LIVE** on k3s -(ArgoCD app `felhom` Synced/Healthy, rollout confirmed, live image `felhom-hub:0.36.0`, -`Listening on :8080`). felhom.eu only — agent + host-install untouched. Two coupled, security-first -changes to the operator customer page, plus the Peti tester agreement. +**Full record:** `documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`. +**Shipped this run (felhom.eu):** manifest wiring `0a65f2b` (HETZNER_* env from Secret/storagebox), +**hub v0.38.1** `7c70c54` + bump `6d5b3c1` (LIVE: Synced/Healthy, `Offsite provisioning enabled +(pool_box=611714, location=fsn1)`). Pairs with controller v0.106.1 (that repo's REPORT). -### Part 1 — passphrase hardening (the security win, ships first) +The first live run of the SLICE 1+2 loop on `demo-felhom` against the new dedicated-project pool box: +hub UI save → sub-account provisioned (269008, `u629488-sub1`) → non-secret descriptor + one-time password → +controller config-refresh → apply-bridge (verify-pin, consume-once, pinned `ssh-copy-id`, offbox configured, +`EscrowState=pending`) → fork-4 run-gate refusal proven → escrow confirm (Option A — the repo password was +already escrowed under Viktor's R) → **first run: restic repo initialized, 2 apps, 2 snapshots on the new box, +key-only** (independently verified by SFTP-listing the repo with the bridge-installed key + pinned host key). -The per-customer retrieval passphrase was in cleartext **twice**: the visible `#retrieval-pw` node -and baked into the Option-3 debug curl's `X-Retrieval-Password:` header (a copyable command carrying -the secret). Now: -- `#retrieval-pw` renders a **masked bullet run by default**, with a **Reveal** (`toggleSecret`) and - a **copy** (`copySecret`) control; the value lives in `data-secret` (the page's existing reveal - model — the value still ships in the DOM, unchanged from before). -- The Option-3 command carries a **`` placeholder** — the secret is never in - a copyable command block. -- A **zero-secret-in-DOM** reveal-on-demand fetch is a deliberate future follow-up, explicitly NOT - scoped here (kept the change tight and reviewable). +**Live bugs found + fixed mid-run:** +- **F1 (hub v0.38.1):** a re-click canceled `r.Context()` between `CreateSubaccount` and + `SaveOneTimeSecret` → sub-account stranded with its password lost. `applyOffsite` now provisions on + `context.WithoutCancel` + 3-min timeout. Test `TestApplyOffsite_ClientDisconnectMidProvision` + red-proof + (raw-ctx shape reproduces the exact live error). +- **F3 (controller v0.106.1):** `ssh-copy-id -s` mktemp's under `~/.ssh`, absent from the container image — + install died locally AFTER consuming the password (fail-safe held; password never transmitted). -### Part 2 — interactive install-command generator +**Open findings:** F2 fresh-subaccount DNS lag → first-save scan 502 (retry-with-backoff wanted); +F4 no hub password-reset path (recovery = delete sub-account + re-save); F5 no save spinner (the re-click +bait); F6 Hetzner box-level 423 action lock on concurrent saves (fail-closed, correct). -The three hard-coded install `` blocks became a **client-side builder** (vanilla JS — no -framework, CDN, or network call) that assembles a live-updating command from form controls: -- Emits **only real host-install v1.12.0 flags** in a **download-then-run** shape (never `curl | bash`). -- CustomerID prefilled from the server (`pageData.ScriptVersion` + `data-customer-id`); a **byo** - selection **requires** `--cores`/`--memory` (client-side enforcement via `.gen-req` + a `gen-msg` - prompt); caps/mode are placeholders, never silent defaults. -- **Graceful JS-off static fallback**: the Option-1/2 code nodes retain a - `--customer-id … --mode ` command. -- **Curated control surface** (mode/cores/memory/vmid/node/acl-storages/operator-pubkey-file/ - preserve-state-from + skip-provision/dry-run/preflight-only/allow-new-leaf). The **seven - dangerous/operator-only flags** (`--force`, `--rotate-recovery`, `--enable-oob`, `--remove-golden`, - `--uninstall`, `--adopt-pool`, `--rescope-acl`) are **never offered as controls**. - -`configs.go`: `const hostInstallVersion = "1.12.0"`; `pageData.ScriptVersion` added + populated. -`style.css`: `.gen-controls`/`.gen-radios`/`.gen-radio`/`.gen-check(s)`/number-inputs/`.gen-msg` -(dark palette, 2px radius). - -### Part 3 — tests (green gate passed on committed source) - -- **`TestTemplates_PassphraseHardened`** — the secret is NOT in the Option-3 command (placeholder is), - `#retrieval-pw` is masked-by-default (bullet-entity run), `data-secret` populated, reveal+copy - controls present. **Red-proof:** revert the Option-3 block to `{{.Config.RetrievalPassword}}` → the - "not-in-command" / masked assertions fail. -- **`TestTemplates_InstallGenerator`** — every curated control id renders, script version + - `data-customer-id` present, the JS-off static fallback command present, and **none of the seven - excluded flags appear anywhere on the page**. -- `go build ./... && go vet ./... && go test ./internal/web/` → **ok** on the committed source (`844fbfa7`). - -### Part 4 — deploy (GitOps) - -Built `felhom-hub:0.36.0` on 180 from the pushed source, bumped `manifests/hub.yaml` (0.35.0→0.36.0), -ArgoCD hard-refresh + sync → Synced/Healthy, `deploy/hub` rolled out, live image + startup log -confirmed. **Gotcha caught:** the first build produced a stale image because the Part-1/Part-2 commits -were committed locally but not pushed before building (`build.sh` pulls origin) — fixed by pushing -first, then rebuilding. - -### Part 5 — tester agreement + closeout - -`documentation/pilot/PETI-tester-agreement.md` written: the BYO trust boundary (break-glass/OOB/WG -all OFF, non-root agent, pool-scoped token, TLS+per-guest-token local API), the honest limitations -(no offsite backup yet; physically-removable-drive caveat F2; pool-reassert-bring-up-only F7; -`:53`-must-be-free F6), exit rights (uninstall keeps drives/data/hub-record), Peti's box facts -(80c/128 GB → caps **12/32768**, `sajatfelhom.hu` tunnel re-point), the byo install command, and the -onboarding sequence (preflight → install → **G9 password-set** → tunnel → **backup-green**). -GO-LIVE-PACKAGE updated: GL-7 ✔; **G9/G10 → done**, **G11 → done** (with the one open pilot question -folded into the agreement's pre-install checklist); decision log + open-questions updated. - -### The one remaining pilot question (owned by the agreement) - -Peti's **local backup TARGET** — `felhom-pbs` is unreachable from his LAN, so the default -`--acl-storages "local local-lvm felhom-pbs"` is wrong for him. Confirm at onboarding **whether his -box has a second disk/pool**: if yes it becomes the `--acl-storages` backup target (real local -resilience); if no, backups share the guest's pool (degraded — one drive failure loses both, must be -stated to him). Changes only the `--acl-storages` value, nothing structural. - -### Verification note - -The hub UI is operator-password-gated — CC cannot log in, so per the repo policy UI changes are -verified via the render tests (both green) plus the deploy checks (image/rollout/startup log), not a -live browser session. +**Standing state:** demo offsite now targets the NEW box, `escrowed`, un-paused; the old-box offsite +(`u629193-sub1`) is a frozen archive until a NEW-box restore round-trip is verified. +**NEXT:** SLICE 3 (escrow auto-confirm), SLICE 4 (soft-quota), F2/F4/F5, Peti/pilot onboarding runbook. diff --git a/documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md b/documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md new file mode 100644 index 0000000..44db0c7 --- /dev/null +++ b/documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md @@ -0,0 +1,110 @@ +# VALIDATION — offsite provisioning LIVE end-to-end (hub → Hetzner → controller), 2026-07-09 + +**Class:** Risky / supervised operational validation (runbook execution). **Operator on 🛑 steps:** Viktor. +**Scope:** first live run of the SLICE 1+2 hub-driven offsite provisioning loop on the demo customer +(`demo-felhom`), against the new dedicated-project pool box (`storage-box-pool-1`, id **611714**, user +`u629488`, project 15282031, BX11 fsn1). Closed through the fork-4 escrow gate to first snapshots. + +## VERDICT: **PASS** (with 2 live bugs found + fixed mid-run, 2 findings deferred) + +The complete loop ran through the REAL pipeline — hub UI save → Hetzner sub-account provision → +descriptor + one-time password → controller config-pull → self-restart → apply-bridge (verify-pin → +consume-once → pinned key install → offbox configured → `EscrowState=pending`) → run-gate refusal → +escrow confirm → first run → **restic repo + 2 snapshots on the new box, key-only**. No hand-set state +anywhere (anti-F9 held: every transition via the shipped endpoints/jobs). + +Shipped mid-run: **hub v0.38.1** (F1) + **controller v0.106.1** (F3). Final versions live: +hub 0.38.1, controller 0.106.1. + +## Phase log + +### Phase 0 — wiring (secret + manifest) +- Secret `felhom-system/storagebox` keys were already correct-shaped (`HETZNER_TOKEN`, + `HETZNER_POOL_BOX_ID=611714` numeric); `HETZNER_LOCATION` optional (hub defaults `fsn1`). +- **Scope check FAILED first:** the stored token listed only the OLD project's box (611421/`u629193` — ep0's + PBS box) → it was the shared-project token the runbook forbids. 🛑 Viktor swapped in the dedicated-project + token (pasted via `read -rs` + `kubectl patch --stringData`, never on argv/logs). Re-check: token sees + ONLY 611714/`u629488` → scoped correctly. **Scope-check method (reusable):** list `/v1/storage_boxes` + with the token from a shell var; the visible set IS the blast radius. +- Manifest `manifests/hub.yaml`: `HETZNER_TOKEN`/`HETZNER_POOL_BOX_ID` via `secretKeyRef` (optional:true) + + literal `HETZNER_LOCATION=fsn1` (commit `0a65f2b`); synced only AFTER the token swap. Hub logged + `Offsite provisioning enabled (pool_box=611714, location=fsn1)`. + +### Phase 1 — pre-flight (all clean) +Controller 0.106.0 healthy, bridge dormant; customer id is **`demo-felhom`** (`demo-felhom-01` is the +host id); `config_version=3`, no `offsite` key; `one_time_secrets` empty; box 611714: 0 subaccounts. + +### Phase 2 — provisioning saves (where the bugs lived) +- **Save 1 (3 presses, 20:50 CEST): F1.** The ~25s spinner-less save invited re-clicks. Press 1 created + sub-account 268985 on Hetzner, but the abandoned request's `r.Context()` was canceled between + `CreateSubaccount` and `SaveOneTimeSecret` → **password lost forever** (stranded resource). Press 2 hit + Hetzner's box-level HTTP 423 action lock (fail-closed 502, correct). Press 3 took the idempotent path and + saved the descriptor (v4) — with no stored password. + **Fix: hub v0.38.1** — `applyOffsite` provisions on `context.WithoutCancel` + 3-min absolute timeout + (create→wait→store is the atom). Regression test cancels the request mid-create; red-proof against the + raw-ctx shape reproduced the exact live error. Recovery: deleted stranded 268985 (zero data). +- **Save 2 (single press, 21:08): F2 (open).** Fresh create 268988 + password STORED (v0.38.1 ordering + proved itself), then the host-key scan failed: `lookup u629488-sub1.your-storagebox.de … no such host` — + a fresh sub-account's DNS takes seconds-to-a-minute. Fail-closed 502, nothing saved, state recoverable. +- **Save 3 (21:10):** idempotent path (no 2nd sub-account — the runbook's idempotency probe, live), + DNS now resolved, scan OK → descriptor v5 saved, password still unconsumed. + +### Phase 3 — controller applies +- **v4 stale-descriptor noise (bonus proof):** the controller had picked up v4 earlier; the bridge ran + scan+verify then consume → 404 → fail-safe abort with the designed retry note. Fail-safe proven live. +- **v5 attempt (21:25): F3.** Bridge verified the pin, CONSUMED the password, then `ssh-copy-id -s` died + **locally**: SFTP mode mktemp's under `~/.ssh` and the container image has no `/root/.ssh`. The loud + "password is spent — reset on the hub" fail-safe fired (no marker, no offbox config; the password never + left the box). Container probes (no secrets: throwaway key + wrong password) isolated it AND proved the + TOCTOU-hardened pin chain sound: with `~/.ssh` present, pinned known_hosts + `StrictHostKeyChecking=yes` + verified cleanly and wrong-password failed as `Permission denied` (sshpass exit 5). + **Fix: controller v0.106.1** — the installer ensures `~/.ssh` (0700) exists. Deployed to 9201. + Recovery: deleted spent 268988; 🛑 save 4 (21:36, first press) → sub-account **269008** + fresh password + → descriptor v6. +- **v6 apply (21:45:45–49 CEST) — CLEAN:** config-refresh → self-restart → bridge: + `offsite configured for u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo (pending key escrow)`. + +**Verification table (all ✅):** consumed exactly once (`consumed_at` set, single row, read-only DB check); +`data/offbox/`: `ssh_key` 0600 + pinned `known_hosts` + `applied_marker` (dir 0700); offbox target = the +descriptor with `escrow_state=pending`; agent escrow stage-push succeeded; **run-gate refusal** (manual +`POST /backup/offbox/run` → "A NAS-mentés a kulcs letétbe helyezésére vár."); **restart idempotency** +(bootstrap restart → zero bridge activity, no re-consume). + +### Phase 4 — fingerprint-flip spot check: SKIPPED (time-boxed) +Covered by the unit red-proof (mismatch → refuse before consume); the adjacent fail-safe paths were proven +live twice above. + +### Phase 5 — escrow + first run (Option A, 🛑 Viktor approved) +**Deviation from the runbook's premise:** `WriteOffboxSecrets` preserves an existing `repo_password` (the +DR-inject seam), so the new target reuses the password **already escrowed** in the production hub blob under +Viktor's R (validated same-day: the recovered password decrypted the real repo). The escrow claim was +therefore already truthful → **no new ceremony**: `POST /backup/offbox/confirm-escrow` → `escrowed` → +`POST /backup/offbox/run` → `initialized restic repo` → **2 apps (audiobookshelf, immich), 2 snapshots, 34s**. +Independent proof: SFTP with the bridge-installed key + pinned host key listed the repo structure and the +two snapshot objects on the new box. Old-box offsite (`u629193-sub1`) untouched — kept as frozen archive +until a NEW-box restore round-trip is verified (Viktor's standing decision). + +## Findings +| # | What | Status | +|---|------|--------| +| F1 | Client disconnect mid-provision strands a sub-account with a lost password (`r.Context()` canceled between create and store) | **FIXED hub v0.38.1** (+ test + red-proof) | +| F2 | Fresh sub-account DNS lags creation → first-save host-key scan 502s (`no such host`); workaround: save again | **OPEN** — retry scan with backoff in `ProvisionOffsite` | +| F3 | `ssh-copy-id -s` requires local `~/.ssh`; container image ships without it → install failed AFTER consuming the password | **FIXED controller v0.106.1** | +| F4 | The designed recovery "reset the password on the hub" doesn't exist (no reset-password path); recovery today = delete sub-account + re-save | **OPEN** — hub needs a reset→re-store one-time-secret action (`ResetSubaccountPassword` already in CloudAPI) | +| F5 | The save button gives no in-flight feedback (~25s) — the direct cause of F1's re-clicks | **OPEN** (UX: disable button/spinner) | +| F6 | Concurrent saves hit Hetzner's box-level HTTP 423 action lock → fail-closed 502 (correct but confusing) | observed; optional retry-on-423 in hetznerapi | + +## Operational facts (for the next operator) +- Env names the hub reads (main.go): `HETZNER_TOKEN`, `HETZNER_POOL_BOX_ID` (**numeric** box id, NOT the + name), `HETZNER_LOCATION` (default fsn1). Secret: `felhom-system/storagebox`. +- The token MUST be the dedicated-project one — verify by listing boxes with it (must NOT see 611421). +- Sub-account home dirs: `felhom-` on the parent box; controller-facing repo path is always + `/home/felhom-repo` (chroot). Deleted sub-account usernames are recycled (`-sub1` reused 3×). +- Hetzner box `stats` lag real usage — don't use them as a freshness probe; SFTP-list the repo instead. +- Controller report/refresh cadence: hub-report every 15 min → config-refresh on ACK → self-restart. + Worst-case descriptor-to-applied latency ≈ 15 min. + +## NEXT +SLICE 3 (escrow auto-confirm — replaces the manual confirm exercised here), SLICE 4 (soft-quota on +`quota_gb`), F2 scan-retry, F4 hub password-reset, F5 spinner, old-offsite retirement after a verified +NEW-box restore, Peti/pilot onboarding runbook.