felhom-agent/REPORT.md

# REPORT — Slice 7 Phase 1: unified bring-up reconcile job (v0.8.0) (2026-06-09)

> Overwrite-latest report (most recent significant work only). Cumulative history lives in
> [CHANGELOG.md](CHANGELOG.md). Implements `TASK — Slice 7 Phase 1: unified bring-up reconcile job`.
> **Agent-only — no hub/wire change** (the new guest auto-appears in the host-report via `ListLXC`).

## Outcome

The shared **front half** of provision and guest-loss DR shipped as a journaled reconcile job
(`internal/reconcile/bringup.go`), mirroring the slice-6 restore-test's crash-safety but KEEPING
the guest on success and applying a **scenario-specific identity policy**. Built from the slice-7
spike findings (commit `3342993`): F1 (MAC reset unconditional on provision), F3 (host keys via a
baked golden first-boot unit, not an agent guest-internal op), F4 (transient config-lock retry).

## What landed

- **`RunBringUp(BringUpSpec) BringUpResult`** — restore → identity reset → size → attach mounts →
  start LINK-UP, each mutation preceded by journaling the owning entry. **Verdict is liveness**
  (`waitRunning`), never the start exitstatus (reuses the v0.7.0 WARNINGS surface). **Success keeps
  the guest** (the key difference from the restore-test).
- **Identity policy (doc 03 §9):** *provision* resets MAC unconditionally (`PUT net0`, hwaddr
  omitted → PVE regenerates; F1) + hostname, host-side via the token; machine-id + SSH host keys
  regenerate guest-side on first boot (systemd + the baked unit) — the agent never touches guest
  internals. *dr_guest_loss* preserves continuity (keep hostname; keep MAC unless `KeepMAC=false`);
  never resets restic/tunnel/hub identity.
- **Compensating rollback:** any mid-flight failure destroys the just-created guest
  (`ClassGuestDestroy` benign via `Provenance{SameTxnCreated:true}`, gated). New journal flag
  `Rollback` + `Recover.recoverBringUp` reap a half-built guest from a mid-job crash (idempotent,
  via `ListLXC`) — distinct from the scratch path's audit label, same destroy machinery.
- **F4:** identity+sizing+mounts coalesced into ONE `PUT config`; rootfs grow kept separate;
  `setConfigWithLockRetry` retries ONLY the transient PVE config-lock 500 (`pveConfigLock`), never
  a real error.
- **`--selftest=bring-up`** (`-mode provision|dr -archive -vmid -hostname [-keep]`) — runs the real
  job after a `Recover`, then tears the guest down unless `-keep`.
- **`configs/build-golden.sh`** — the validated golden recipe incl. the F3 first-boot
  `felhom-regen-hostkeys.service` (Condition-gated: fires on provision, no-ops on DR). The spike's
  golden archive (no unit) is superseded.

## Tests (assert the effect)

`go test ./...` green; `-race` green on the build server. Provision happy path (fresh MAC = net0
without hwaddr, hostname, coalesced sizing+mount, rootfs-grow separate, started, **guest NOT
destroyed**); compensating rollback injected at each step (restore / config / start-task /
waitRunning → asserts the guest **was** destroyed); DR continuity (MAC kept, hostname not reset) +
DR `KeepMAC=false` resets MAC; liveness verdict (warnings+running pass / not-running fail); F4
(lock-500 → retry → proceed; non-lock 500 → fail without retry); owning entry journaled **before**
restore; reserved/existing VMID refused; `Recover` rolls back / clean.

## Live validation (demo-felhom) — all green; caught + fixed one real bug

Golden built via `configs/build-golden.sh` → `local:backup/vzdump-lxc-9100-2026_06_09-21_32_58.tar.zst`
(Docker `overlayfs`, host-key unit baked + enabled). All runs via `--selftest=bring-up`.

- **Bug caught live (then fixed + re-validated):** the first golden's `felhom-regen-hostkeys`
  unit used `ExecStart=/usr/sbin/ssh-keygen` (as drafted in the task) → `status=203/EXEC` ("No such
  file"); on Debian 13 `ssh-keygen` is at `/usr/bin/ssh-keygen`. Fixed in `build-golden.sh`,
  rebuilt the golden, re-validated. (The unit tests mock the guest, so only the live run could
  surface this — exactly why the slice runs live.)
- **provision** (vmid 8100): restore → fresh MAC `BC:24:11:4A:CF:3C` (≠ golden's) → hostname
  `felhom-prov-8100` → link-up, 8s. Verified: `felhom-regen-hostkeys` **active** and SSH host keys
  present (`root@felhom-prov-8100`) — **regenerated by the baked unit, the agent issued no
  ssh-keygen**; `ssh.service` active; machine-id unique (`103a0325…`); hostname propagated; clean
  DHCP lease `192.168.0.114`; `docker run hello-world` OK → torn down.
- **dr** (vmid 8101, from a 9999 customer backup): continuity preserved — hostname stayed
  `felhom-selftest-scratch` (NOT reset) and SSH host keys **preserved** (not regenerated). The
  source had no net0, so MAC-keep was a no-op live (covered by the unit test instead).
- **Recover** (vmid 8102): killed a provision mid-restore (`timeout -s KILL`); the server-side
  restore left an orphan guest; the re-run's `Recover` logged `rolled back half-built bring-up
  guest vmid=8102` (`BringUpRolledBack:1`), then re-provisioned cleanly → torn down. Idempotent.

Teardown: no spike guests linger; the validated golden is retained as the artifact; the service
runs v0.8.0.

## Deferred (stated, not built)

Provisioning BACK HALF (controller deploy, bootstrap, per-guest token mint) → slice 8; host-loss
DR + escrow consumption → slice 10; the SOURCE of a `BringUpSpec` (hub desired-state) → slice 10
(`GuestMount` defined minimally, no hub coupling). No secrets committed.