report: offsite provisioning live e2e validation PASS (v0.106.1)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

REPORT.md

@@ -1,58 +1,44 @@
-# REPORT — controller v0.106.0: offsite provisioning SLICE 2 (apply-bridge)
+# REPORT — controller v0.106.1 + offsite provisioning LIVE end-to-end validation — **PASS**
 
-**Date:** 2026-07-09 · **Class:** implementation (controller) + risky. **Baseline:** `main` @ `fa9362f`
-(v0.105.0) → v0.106.0. Pairs with **hub v0.38.0** (adds `HostFingerprint` to the offsite descriptor).
+**Date:** 2026-07-09 · **Class:** operational validation (supervised runbook) + hotfix. Baseline v0.106.0 →
+**v0.106.1** (live on guest 9201). Pairs with hub v0.38.1. **Full record:**
+`felhom.eu/documentation/audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md`.
 
-## What shipped
-On startup the controller reconciles the hub-served `offsite:` descriptor into a working key-only offbox
-target — the controller half of hub-driven offsite provisioning.
-- `internal/config.OffsiteConfig` — the `offsite:` section (mirrors the hub descriptor incl. `host_fingerprint`).
-- `internal/offsiteapply.Bridge.Reconcile` — **verify-pin (no blind TOFU)** → generate keypair → **consume
-  the one-time password** (single-use, never logged) → install pubkey (`sshpass -e ssh-copy-id -p 23 -s -f`,
-  **pinning the scanner-verified `known_hosts` via `StrictHostKeyChecking=yes` — the install/verify sessions
-  refuse any key but the one the bridge already matched, closing the scan→install MITM gap**)
-  + verify → configure offbox → `EscrowState="pending"` → persist the descriptor-hash marker LAST.
-  **Idempotent** (marker prevents re-consuming a spent password) and **fail-safe** (any step fails → nothing
-  persisted, retried next cycle; consumed-but-failed install logs a loud "password is spent — reset on the hub").
-- `internal/backup.Manager.ApplyOffsiteTarget` — reuses the fork-4 enable primitives (best-effort escrow stage).
-- `cmd/controller` — wires the real seams + runs `Reconcile` async at startup. `Dockerfile` + `sshpass`.
+## What happened
+First live run of the SLICE 2 apply-bridge, driven end-to-end through the real pipeline (no hand-set state):
+hub provisioned sub-account 269008 (`u629488-sub1`) on the new dedicated-project pool box (611714) →
+descriptor v6 + one-time password → controller `config-refresh` (15-min report ACK) → self-restart →
+**bridge applied cleanly**: scan + fingerprint verify (BEFORE consume) → keypair → consume-once →
+`sshpass -e ssh-copy-id -s -f` with the **pinned known_hosts (`StrictHostKeyChecking=yes`)** → SFTP key-auth
+verify → offbox configured → `EscrowState="pending"` → marker persisted.
 
-## Files changed
-`internal/config/config.go`, `internal/offsiteapply/{offsiteapply.go,seams.go,offsiteapply_test.go}` (new),
-`internal/backup/offbox.go`, `cmd/controller/main.go`, `Dockerfile`, `CHANGELOG.md`.
+## v0.106.1 — the one live bug (F3)
+`ssh-copy-id -s` (SFTP mode) mktemp's its batch file under `~/.ssh`; the container image ships without
+`/root/.ssh` → the install died **locally** after the password was consumed. Fail-safe held exactly as
+designed (loud "password is spent" log, no marker, no offbox config; the password never left the box).
+Fix: `SSHCopyIDInstaller` ensures `~/.ssh` (0700) exists. Container probes (throwaway key + wrong password,
+no secrets) isolated the bug AND proved the pin chain sound: pinned single-line known_hosts verifies cleanly;
+wrong password fails as `Permission denied` (sshpass exit 5).
 
-## Tests + companion red-proofs
-Green gate `go build ./... && go vet ./... && go test ./...` — **ALL-GREEN** (both repos).
-- `TestBridge_AppliesEndToEnd` — consume→verify-pin→install→configure→marker; asserts **the installer AND
-  the enabler both receive the scanner-verified known_hosts** + the private key, and **the one-time password
-  never appears in a log line**. **Red-proof run:** passed an empty known_hosts to the installer (the pre-fix
-  TOFU shape) → test FAILED ("installer must receive the scanner-verified known_hosts to pin (no TOFU), got \"\""). Reverted.
-  Pinning-the-install-connection (not just the scan) is load-bearing — addresses the security-review TOFU-after-verify finding.
-- `TestBridge_HostKeyMismatchRefuses` — a scanned FP ≠ descriptor FP → refuse (no consume/install/configure/marker).
-  **Red-proof run:** dropped the verify (`if false`) → the mismatch proceeded to install a wrong key → test
-  FAILED ("mismatch must refuse, got <nil>"). Reverted. No-TOFU is load-bearing.
-- `TestBridge_IdempotentNoReconsume` — marker matches → `Consume` panics if called → clean no-op.
-- `TestBridge_InstallFailIsFailSafe` — install errors → marker NOT persisted, offbox NOT configured, loud
-  "password is spent" log. **Red-proof run:** persisted the marker before the install → a failed apply looked
-  done → test FAILED ("marker must NOT be persisted on a failed apply"). Reverted. Marker-after-success is load-bearing.
-- Hub `internal/offsite`: descriptor carries `HostFingerprint` from a faked scanner; a scan failure fails-closed.
+## Live proofs (all green)
+- **Consume-once:** hub `one_time_secrets.consumed_at` set, single row (read-only DB check).
+- **Fail-safe, twice:** the stale-v4 descriptor run hit consume→404 → clean abort + retry note; the F3 run
+  hit install-fail → "password is spent — reset on the hub", no marker, no half-configured offbox.
+- **fork-4 run gate:** manual `POST /backup/offbox/run` while pending → refused
+  ("A NAS-mentés a kulcs letétbe helyezésére vár."). No ciphertext before escrow.
+- **Restart idempotency:** bootstrap restart → zero bridge activity (marker no-op), no re-consume.
+- **Closing the loop (Option A, Viktor-approved):** the repo password was already escrowed (preserved by
+  `WriteOffboxSecrets` — the DR-inject seam), so confirm-escrow was truthful without a new ceremony →
+  `escrowed` → first run: **restic repo initialized, 2 apps (audiobookshelf, immich), 2 snapshots, 34s** —
+  independently verified by SFTP-listing the repo objects with the bridge-installed key + pinned host key.
 
-## Deploy verification
-Deployed to guest 9201 (golden/bootstrap mechanism): `:0.106.0 Up (healthy)`, `controller_started … (0.106.0)`,
-`errors=0 warnings=0`, startup hub report sent. Offsite is disabled in the demo config, so the bridge async
-goroutine (gated on `cfg.Offsite.Enabled`) correctly no-ops — no `offsite-apply` log lines. The image ships the
-new offbox-install tools: `sshpass` (`/usr/bin/sshpass`), `ssh-copy-id`, `sftp` all present in the container.
-This deploy proves the image + bridge wiring + `sshpass` dependency, not a live apply (that's the next runbook).
+## Standing state
+Demo offbox targets the NEW box (`u629488-sub1…:/home/felhom-repo`), `escrowed`, un-paused, daily schedule.
+Old-box offsite (`u629193-sub1`) untouched — frozen archive until a NEW-box restore round-trip is verified.
 
-## NOT yet live-applied
-The supervised end-to-end (hub provisions on the new pool box `u629488`/project `15282031` → controller
-consumes + installs its key + configures the offbox → `EscrowState="pending"`) is the **next runbook**, gated
-on the hub being wired with the new scoped `HETZNER_TOKEN` + `HETZNER_POOL_BOX_ID`. Unit tests (faked seams)
-are this slice's proof. Untested-live: the real `sshpass`/`ssh-copy-id -s -f` install + the x/crypto/ssh
-host-key scan against a live box (both proven in the API spike; re-confirmed in the supervised run).
-
-## Observations
-- The bridge runs once at startup; the config-refresh self-restart re-runs it after a descriptor change (no
-  separate post-refresh hook needed — the restart is the trigger).
-- The escrow stage-push is best-effort (agent-down leaves the offbox configured+pending, re-stage later) —
-  the offbox run-gate still holds until the operator confirms escrow (fork-4).
+## Open follow-ups (controller-relevant)
+- F2 (hub): fresh-subaccount DNS lag → first-save scan 502; retry-with-backoff in `ProvisionOffsite`.
+- F4 (hub): no password-reset path — the recovery the controller's "reset on the hub" message references
+  doesn't exist yet; today's recovery is delete-subaccount + re-save.
+- SLICE 3 (escrow auto-confirm — replaces the manual confirm exercised here), SLICE 4 (soft-quota).
+- NEW-box restore round-trip, then retire the old sub-account.