felhom-controller/REPORT.md

REPORT — controller v0.40.0: bootstrap pull+merge onboarding (live-validated) (2026-06-11)

Lockstep two-repo change with felhom-agent v0.19.0. Fixes the onboarding 401 found last session: a freshly provisioned guest used to seed a "configured" controller.yaml from the agent's host hub key, which the hub's customer-scoped /api/v1/report rejects → the controller could never report ONLINE. Now, on first boot, the controller pulls its full controller.yaml from the hub (using the bootstrap's retrieval passphrase, which yields the customer-scoped key) and merges in the per-guest local_api block. Validated live end-to-end on the demo (guest 9201).

What changed (internal/bootstrap, cmd/controller/main.go)

• Contract v1 → v2 (felhom.bootstrap/v2): BootstrapCustomer keeps only id; BootstrapHub drops api_key/host_id, adds retrieval_password; local_api unchanged. Non-v2 → setup mode.
• MaybeIngest(configPath, cfg, logger, pull PullFunc) — pull injected (decision (b): keeps bootstrap free of the heavy internal/report package; main.go wires report.PullConfig). Flow: idempotent (configured → return, no pull) → parse+validate v2 → pull with bounded retry (1 + 3 backoff attempts, transient ErrPullTransient only; auth/not-found fail fast) → merge local_api at the YAML-map level (decision (c): preserves every hub-emitted field) → write 0600 atomic → reload. Fail-safe + never-crash (hub outage at first boot → setup mode).
• New sentinel ErrPullTransient; main.go's adapter maps report.ErrHubUnreachable → transient, passes auth/not-found through as permanent. Removed configFromBootstrap (the host-key path).

Cross-repo contract checksum-diff (rendered bootstrap.json field set)

The agent's v2 renderer output was ingested by the controller's json.Unmarshal — every field populated, exact match:

level	fields (agent emits == controller ingests)
top	schema, customer, hub, local_api
customer	id
hub	url, retrieval_password
local_api	endpoint, fingerprint, token

(Automated round-trip via a throwaway test in each package; removed after verifying.)

Tests — non-hollow (internal/bootstrap), all green

• Pull+merge: stub pull returns a hub yaml with hub.api_key: CUSTKEY_FROM_HUB, customer.domain, and an unmodeled assets.source_url. Asserts the written controller.yaml carries **the customer key
  • identity + the preserved unmodeled assets field** AND the bootstrap's local_api.{endpoint, fingerprint,token}, and contains no host key/id.
• Idempotency: preset cfg.Customer.ID → asserts pull never invoked, file untouched.
• Transient retry: stub returns ErrPullTransient always → asserts exactly 1+len(delays) calls, then setup mode, no file (backoff shrunk to ~1ms via the overridable pullRetryDelays).
• Permanent no-retry: stub returns a plain (auth-style) error → asserts a single call.
• Schema reject (non-v2), missing-required, malformed/absent → setup mode, no pull.

go build ./... && go test ./... green.

Live validation (demo Proxmox felhom-pve, guest 9201, golden baked :0.40.0)

Golden re-baked: local:backup/vzdump-lxc-9100-2026_06_11-13_26_45.tar.zst (baked image confirmed gitea.dooplex.hu/admin/felhom-controller:0.40.0). Provisioned fresh as demo-felhom via agent v0.19.0 --selftest=provision -customer-id demo-felhom -hub-password <passphrase> (passphrase read from the hub customer_configs and transported base64 to avoid UTF-8 mangling; stored out-of-band), then pct reboot + systemctl restart felhom-agent (the local-API token workaround, Finding #1).

• Bootstrap (v2) on the guest: hub keys = [url, retrieval_password] (no host key), customer keys = [id] only, 0600. ✓
• Pull+merge worked — the merged /opt/docker/felhom-controller/controller.yaml (secrets redacted) carries from the hub pull: hub.api_key: 4b11c0c3… (the customer-scoped key, matches the hub's customer_configs row), hub.enabled: true, customer.{id: demo-felhom, domain: demo-felhom.eu, name, email}, assets.source_url, git (catalog repo), infrastructure.cf_* (Cloudflare config); and merged from the bootstrap: local_api.{endpoint: 192.168.0.162:8443, fingerprint: 60b5974d…, token}. No host_id, no agent host key. ✓
• Hub ONLINE at v0.40.0 — [report] Hub report pushed successfully (3090 bytes) + Startup hub report sent, no 401. Hub reports row for demo-felhom: controller_version=0.40.0, received_at=2026-06-11 11:32:00 (fresh → online). 0 deployed apps (fresh guest — expected). ✓
• local_api survived the merge — GET /api/host-metrics → {ok:true}, cpu_temp_c=49 (real), 4 storage targets; GET /api/disks → {ok:true}, felhom-usb data_bearing:true. ✓
• 8C invariant intact — agent-direct POST /disks/format on data-bearing /dev/sdb1 → HTTP 403 {formatted:false, data_bearing:true, reason:"device is mounted", pending_op:{op:storage_wipe, durable_id:byid:wwn-…, …}} "operator signature required (pending_signature)". Disk untouched (/dev/sdb1 ext4 8G, still mounted). ✓

What broke / what's missing

• Bootstrap log line absent in docker logs (observability nit, reproduced from last session's seed-log). MaybeIngest's [INFO] bootstrap: pulled config … coming up configured does not surface in docker logs even though setupLogger writes to stdout and the pull demonstrably ran (customer key present, hub report OK, catalog repo configured). The first captured line is a later async local-api WARN — the early synchronous bootstrap log is being swallowed before docker attaches. Worth a follow-up (flush/sequence the logger before MaybeIngest, or log the pull result post-startup).
• Finding #1 still open (separate spec): the local-API channel 401s until systemctl restart felhom-agent after provisioning a live-daemon host (the running daemon didn't reload the freshly minted token). Reproduced (startup WARN at 11:31:55); workaround applied.
• Operational gotcha (mine, fixed): kubectl cp's "tar: removing leading '/'" warning polluted a captured base64 passphrase on the first attempt → a 2-char garbage passphrase → re-extracted with tail -1 and re-provisioned cleanly. The UTF-8 (Hungarian) passphrase must be transported byte-exact (base64), not through the Windows shell.
• Minor: guest 9201's hostname is felhom-golden (no -hostname passed); cosmetic, customer.id is correct.

Versions / artifacts

• Controller v0.40.0 (CHANGELOG updated). Pushed to main: commit 6a594f9 (code) — this REPORT in the follow-up commit.
• Lockstep agent v0.19.0 (commit e5a1819). New golden: local:backup/vzdump-lxc-9100-2026_06_11-13_26_45.tar.zst.
• No secrets committed (passphrase, customer key, CF tokens, local-api token — all out-of-band/redacted).