felhom-controller/REPORT.md

REPORT — controller v0.106.0: offsite provisioning SLICE 2 (apply-bridge)

Date: 2026-07-09 · Class: implementation (controller) + risky. Baseline: main @ fa9362f (v0.105.0) → v0.106.0. Pairs with hub v0.38.0 (adds HostFingerprint to the offsite descriptor).

What shipped

On startup the controller reconciles the hub-served offsite: descriptor into a working key-only offbox target — the controller half of hub-driven offsite provisioning.

• internal/config.OffsiteConfig — the offsite: section (mirrors the hub descriptor incl. host_fingerprint).
• internal/offsiteapply.Bridge.Reconcile — verify-pin (no blind TOFU) → generate keypair → consume the one-time password (single-use, never logged) → install pubkey (sshpass -e ssh-copy-id -p 23 -s -f, pinning the scanner-verified known_hosts via StrictHostKeyChecking=yes — the install/verify sessions refuse any key but the one the bridge already matched, closing the scan→install MITM gap)
  • verify → configure offbox → EscrowState="pending" → persist the descriptor-hash marker LAST. Idempotent (marker prevents re-consuming a spent password) and fail-safe (any step fails → nothing persisted, retried next cycle; consumed-but-failed install logs a loud "password is spent — reset on the hub").
• internal/backup.Manager.ApplyOffsiteTarget — reuses the fork-4 enable primitives (best-effort escrow stage).
• cmd/controller — wires the real seams + runs Reconcile async at startup. Dockerfile + sshpass.

Files changed

internal/config/config.go, internal/offsiteapply/{offsiteapply.go,seams.go,offsiteapply_test.go} (new), internal/backup/offbox.go, cmd/controller/main.go, Dockerfile, CHANGELOG.md.

Tests + companion red-proofs

Green gate go build ./... && go vet ./... && go test ./... — ALL-GREEN (both repos).

• TestBridge_AppliesEndToEnd — consume→verify-pin→install→configure→marker; asserts the installer AND the enabler both receive the scanner-verified known_hosts + the private key, and the one-time password never appears in a log line. Red-proof run: passed an empty known_hosts to the installer (the pre-fix TOFU shape) → test FAILED ("installer must receive the scanner-verified known_hosts to pin (no TOFU), got """). Reverted. Pinning-the-install-connection (not just the scan) is load-bearing — addresses the security-review TOFU-after-verify finding.
• TestBridge_HostKeyMismatchRefuses — a scanned FP ≠ descriptor FP → refuse (no consume/install/configure/marker). Red-proof run: dropped the verify (if false) → the mismatch proceeded to install a wrong key → test FAILED ("mismatch must refuse, got "). Reverted. No-TOFU is load-bearing.
• TestBridge_IdempotentNoReconsume — marker matches → Consume panics if called → clean no-op.
• TestBridge_InstallFailIsFailSafe — install errors → marker NOT persisted, offbox NOT configured, loud "password is spent" log. Red-proof run: persisted the marker before the install → a failed apply looked done → test FAILED ("marker must NOT be persisted on a failed apply"). Reverted. Marker-after-success is load-bearing.
• Hub internal/offsite: descriptor carries HostFingerprint from a faked scanner; a scan failure fails-closed.

Deploy verification

Deployed to guest 9201 (golden/bootstrap mechanism): :0.106.0 Up (healthy), controller_started … (0.106.0), errors=0 warnings=0, startup hub report sent. Offsite is disabled in the demo config, so the bridge async goroutine (gated on cfg.Offsite.Enabled) correctly no-ops — no offsite-apply log lines. The image ships the new offbox-install tools: sshpass (/usr/bin/sshpass), ssh-copy-id, sftp all present in the container. This deploy proves the image + bridge wiring + sshpass dependency, not a live apply (that's the next runbook).

NOT yet live-applied

The supervised end-to-end (hub provisions on the new pool box u629488/project 15282031 → controller consumes + installs its key + configures the offbox → EscrowState="pending") is the next runbook, gated on the hub being wired with the new scoped HETZNER_TOKEN + HETZNER_POOL_BOX_ID. Unit tests (faked seams) are this slice's proof. Untested-live: the real sshpass/ssh-copy-id -s -f install + the x/crypto/ssh host-key scan against a live box (both proven in the API spike; re-confirmed in the supervised run).

Observations

• The bridge runs once at startup; the config-refresh self-restart re-runs it after a descriptor change (no separate post-refresh hook needed — the restart is the trigger).
• The escrow stage-push is best-effort (agent-down leaves the offbox configured+pending, re-stage later) — the offbox run-gate still holds until the operator confirms escrow (fork-4).