felhom-agent/REPORT.md

REPORT — Slice 7 Phase 1: unified bring-up reconcile job (v0.8.0) (2026-06-09)

Overwrite-latest report (most recent significant work only). Cumulative history lives in CHANGELOG.md. Implements TASK — Slice 7 Phase 1: unified bring-up reconcile job. Agent-only — no hub/wire change (the new guest auto-appears in the host-report via ListLXC).

Outcome

The shared front half of provision and guest-loss DR shipped as a journaled reconcile job (internal/reconcile/bringup.go), mirroring the slice-6 restore-test's crash-safety but KEEPING the guest on success and applying a scenario-specific identity policy. Built from the slice-7 spike findings (commit 3342993): F1 (MAC reset unconditional on provision), F3 (host keys via a baked golden first-boot unit, not an agent guest-internal op), F4 (transient config-lock retry).

What landed

• RunBringUp(BringUpSpec) BringUpResult — restore → identity reset → size → attach mounts → start LINK-UP, each mutation preceded by journaling the owning entry. Verdict is liveness (waitRunning), never the start exitstatus (reuses the v0.7.0 WARNINGS surface). Success keeps the guest (the key difference from the restore-test).
• Identity policy (doc 03 §9): provision resets MAC unconditionally (PUT net0, hwaddr omitted → PVE regenerates; F1) + hostname, host-side via the token; machine-id + SSH host keys regenerate guest-side on first boot (systemd + the baked unit) — the agent never touches guest internals. dr_guest_loss preserves continuity (keep hostname; keep MAC unless KeepMAC=false); never resets restic/tunnel/hub identity.
• Compensating rollback: any mid-flight failure destroys the just-created guest (ClassGuestDestroy benign via Provenance{SameTxnCreated:true}, gated). New journal flag Rollback + Recover.recoverBringUp reap a half-built guest from a mid-job crash (idempotent, via ListLXC) — distinct from the scratch path's audit label, same destroy machinery.
• F4: identity+sizing+mounts coalesced into ONE PUT config; rootfs grow kept separate; setConfigWithLockRetry retries ONLY the transient PVE config-lock 500 (pveConfigLock), never a real error.
• --selftest=bring-up (-mode provision|dr -archive -vmid -hostname [-keep]) — runs the real job after a Recover, then tears the guest down unless -keep.
• configs/build-golden.sh — the validated golden recipe incl. the F3 first-boot felhom-regen-hostkeys.service (Condition-gated: fires on provision, no-ops on DR). The spike's golden archive (no unit) is superseded.

Tests (assert the effect)

go test ./... green; -race green on the build server. Provision happy path (fresh MAC = net0 without hwaddr, hostname, coalesced sizing+mount, rootfs-grow separate, started, guest NOT destroyed); compensating rollback injected at each step (restore / config / start-task / waitRunning → asserts the guest was destroyed); DR continuity (MAC kept, hostname not reset) + DR KeepMAC=false resets MAC; liveness verdict (warnings+running pass / not-running fail); F4 (lock-500 → retry → proceed; non-lock 500 → fail without retry); owning entry journaled before restore; reserved/existing VMID refused; Recover rolls back / clean.

Live validation (demo-felhom) — all green; caught + fixed one real bug

Golden built via configs/build-golden.sh → local:backup/vzdump-lxc-9100-2026_06_09-21_32_58.tar.zst (Docker overlayfs, host-key unit baked + enabled). All runs via --selftest=bring-up.

• Bug caught live (then fixed + re-validated): the first golden's felhom-regen-hostkeys unit used ExecStart=/usr/sbin/ssh-keygen (as drafted in the task) → status=203/EXEC ("No such file"); on Debian 13 ssh-keygen is at /usr/bin/ssh-keygen. Fixed in build-golden.sh, rebuilt the golden, re-validated. (The unit tests mock the guest, so only the live run could surface this — exactly why the slice runs live.)
• provision (vmid 8100): restore → fresh MAC BC:24:11:4A:CF:3C (≠ golden's) → hostname felhom-prov-8100 → link-up, 8s. Verified: felhom-regen-hostkeys active and SSH host keys present (root@felhom-prov-8100) — regenerated by the baked unit, the agent issued no ssh-keygen; ssh.service active; machine-id unique (103a0325…); hostname propagated; clean DHCP lease 192.168.0.114; docker run hello-world OK → torn down.
• dr (vmid 8101, from a 9999 customer backup): continuity preserved — hostname stayed felhom-selftest-scratch (NOT reset) and SSH host keys preserved (not regenerated). The source had no net0, so MAC-keep was a no-op live (covered by the unit test instead).
• Recover (vmid 8102): killed a provision mid-restore (timeout -s KILL); the server-side restore left an orphan guest; the re-run's Recover logged rolled back half-built bring-up guest vmid=8102 (BringUpRolledBack:1), then re-provisioned cleanly → torn down. Idempotent.

Teardown: no spike guests linger; the validated golden is retained as the artifact; the service runs v0.8.0.

Deferred (stated, not built)

Provisioning BACK HALF (controller deploy, bootstrap, per-guest token mint) → slice 8; host-loss DR + escrow consumption → slice 10; the SOURCE of a BringUpSpec (hub desired-state) → slice 10 (GuestMount defined minimally, no hub coupling). No secrets committed.