felhom-agent/REPORT.md

REPORT — v0.41.0: provisioned customer guests auto-start after a host reboot (onboot:1, F3)

Repo: felhom-agent · Version: v0.40.0 → v0.41.0 · Date: 2026-06-24 Baseline: main @ db95d51 (version var 0.40.0), trunk-based, no branches. Class: risky/supervised — touches the provisioning chain (back-half).

Finding (F3, from TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md)

The golden bakes --onboot 0 (configs/build-golden.sh:63, template safety) and the provision back-half never overrode it → every provisioned customer guest was onboot:0. After a host reboot/power-cut the customer's whole home-server (controller + all apps) stays down until a manual pct start. Confirmed live in the campaign (Phase 4.1: host rebooted → pct status 9201 = stopped).

The fix

internal/provision/backhalf.go, BackHalf.Provision: added a fatal step right after the config-mount attach (and before the guest-hook install):

if err := b.run(ctx, "pct", "set", strconv.Itoa(in.VMID), "-onboot", "1"); err != nil {
    return Result{}, fmt.Errorf("provision: set onboot: %w", err)
}

• Locus = the back-half, not the golden. build-golden.sh --onboot 0 is unchanged — a template must not auto-start; onboot is a per-guest property the back-half is the right place to set.
• Fatal, like the config-mount attach (a guest that won't auto-recover is a provisioning defect).
• No startup/boot-order/delay (operator decision): the v0.75 mountpoint-gate already covers the drive-bind race at boot (Phase 4.4). The agent never auto-pct starts anything — this is config only.

Test + red-proof

internal/provision/backhalf_test.go:

• Added TestProvision_SetsOnbootOne — asserts the exact pct set 8200 -onboot 1 invocation was recorded by the fake runner (via a new recRunner.hasExact helper, since several pct calls are now recorded and find only returns the first).
• Red-proof (run/confirm/revert): removed the b.run(... -onboot 1) call → TestProvision_SetsOnbootOne FAILED (expected 'pct set 8200 -onboot 1' to be issued) → restored the call → green.
• Existing back-half tests stay green (chown-failure-stops still holds: onboot is after the config-mount attach, so a chown failure never reaches it).
• Green gate: go build ./... ✓ · go vet ./... ✓ · go test ./... ✓ (all packages ok).

Deploy + remediate + verify

A. Built + deployed agent v0.41.0. Built on 192.168.0.180 (go1.26, -ldflags -X main.version=0.41.0, source @ 166a1c8) → scp 180→local→felhom-pve → backed up the prior binary to /usr/local/bin/felhom-agent.bak-0.40.0 → install -m0755 → systemctl restart felhom-agent. Verified: felhom-agent --version → 0.41.0, service active, logs clean (local-api leaf ready, both 9201 drives re-bound under the shared parent, hub desired-state updated).

B. Demo remediation (9201 was provisioned pre-fix → onboot:0). pct config 9201 before → onboot: 0; ran pct set 9201 -onboot 1 (non-destructive, config-only); after → onboot: 1 ✓.

C. Back-half fix proper — LIVE-VALIDATED (supervised capstone re-provision, 2026-06-24). Destroyed guest 9201 (pct stop + pct destroy — 3 owned volumes removed; bind dirs + physical-drive data survive) and re-provisioned it through the real provision chain with agent v0.41.0: felhom-agent --selftest=provision -archive local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst -vmid 9201 -customer-id demo-felhom -hub-password <out-of-band> -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42. Front-half bring-up OK (1m11s), back-half OK. Assertion: the freshly-provisioned pct config 9201 showed onboot: 1 with NO manual pct set — set by the back-half fix. Full 3-volume layout faithfully reproduced (rootfs 32G + mp0 docker 200G + mp1 /mnt/sys_drive 50G). Agent daemon restart then reasserted both enrolled drives (felhom-flash, felhom-usb) at stable paths under /mnt/felhom-drives (F9), skipped the ejected drive; controller + base infra came up healthy.

Optional — host-reboot scenario LIVE-VALIDATED (the exact Phase-4.1 failure, now passing). Rebooted felhom-pve (uptime went 16:37 → 1 min, confirming a real reboot). pct status 9201 = running with NO manual pct start (auto-started via onboot:1). The other guests (9001, 9999; onboot:0) correctly stayed stopped — proving the per-guest onboot:1 property drives the auto-start, not a blanket start-all. Post-reboot recovery confirmed: the agent (fresh PID) re-bound both drives at stable paths, /mnt/sys_drive + /var/lib/docker are separate mountpoints, and controller + traefik + cloudflared + filebrowser all came up healthy.

Demo state after the capstone: controller redeployed to v0.82.0 (the fresh provision came up on the golden-baked v0.77.0; pin bumped + bootstrap re-run). Standard post-re-provision residue (as every prior capstone): customer apps are not redeployed and the controller's storage paths need a one-time UI re-enroll of the two drives (their host-side data on felhom-usb is intact; the SSD mp0/mp1 app data was destroyed with the volumes). The F3 fix itself is fully validated.