admin
4725396c81
Destroyed + re-provisioned 9201 through the real provision chain with v0.41.0: fresh pct config showed onboot:1 with no manual set. Host reboot of felhom-pve -> 9201 auto-started (running, no manual pct start); onboot:0 scratch guests stayed stopped; drives re-bound at stable, sys_drive separate, controller + infra healthy. Demo controller redeployed to v0.82.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
78 lines
5.2 KiB
Markdown
78 lines
5.2 KiB
Markdown
# REPORT — v0.41.0: provisioned customer guests auto-start after a host reboot (`onboot:1`, F3)
|
|
|
|
**Repo:** `felhom-agent` · **Version:** `v0.40.0` → **`v0.41.0`** · **Date:** 2026-06-24
|
|
**Baseline:** `main` @ `db95d51` (version var `0.40.0`), trunk-based, no branches.
|
|
**Class:** risky/supervised — touches the provisioning chain (back-half).
|
|
|
|
## Finding (F3, from `TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md`)
|
|
|
|
The golden bakes `--onboot 0` (`configs/build-golden.sh:63`, template safety) and the provision
|
|
back-half never overrode it → **every provisioned customer guest was `onboot:0`**. After a host
|
|
reboot/power-cut the customer's whole home-server (controller + all apps) stays **down until a manual
|
|
`pct start`**. Confirmed live in the campaign (Phase 4.1: host rebooted → `pct status 9201` = stopped).
|
|
|
|
## The fix
|
|
|
|
`internal/provision/backhalf.go`, `BackHalf.Provision`: added a fatal step right after the config-mount
|
|
attach (and before the guest-hook install):
|
|
|
|
```go
|
|
if err := b.run(ctx, "pct", "set", strconv.Itoa(in.VMID), "-onboot", "1"); err != nil {
|
|
return Result{}, fmt.Errorf("provision: set onboot: %w", err)
|
|
}
|
|
```
|
|
|
|
- **Locus = the back-half, not the golden.** `build-golden.sh` `--onboot 0` is **unchanged** — a template
|
|
must not auto-start; `onboot` is a per-guest property the back-half is the right place to set.
|
|
- **Fatal**, like the config-mount attach (a guest that won't auto-recover is a provisioning defect).
|
|
- **No `startup`/boot-order/delay** (operator decision): the v0.75 mountpoint-gate already covers the
|
|
drive-bind race at boot (Phase 4.4). The agent never auto-`pct start`s anything — this is config only.
|
|
|
|
## Test + red-proof
|
|
|
|
`internal/provision/backhalf_test.go`:
|
|
- Added `TestProvision_SetsOnbootOne` — asserts the exact `pct set 8200 -onboot 1` invocation was
|
|
recorded by the fake runner (via a new `recRunner.hasExact` helper, since several `pct` calls are now
|
|
recorded and `find` only returns the first).
|
|
- **Red-proof (run/confirm/revert):** removed the `b.run(... -onboot 1)` call → `TestProvision_SetsOnbootOne`
|
|
**FAILED** (`expected 'pct set 8200 -onboot 1' to be issued`) → restored the call → **green**.
|
|
- Existing back-half tests stay green (chown-failure-stops still holds: onboot is after the config-mount
|
|
attach, so a chown failure never reaches it).
|
|
- Green gate: `go build ./...` ✓ · `go vet ./...` ✓ · `go test ./...` ✓ (all packages ok).
|
|
|
|
## Deploy + remediate + verify
|
|
|
|
**A. Built + deployed agent v0.41.0.** Built on 192.168.0.180 (go1.26, `-ldflags -X main.version=0.41.0`,
|
|
source @ `166a1c8`) → scp 180→local→felhom-pve → backed up the prior binary to
|
|
`/usr/local/bin/felhom-agent.bak-0.40.0` → `install -m0755` → `systemctl restart felhom-agent`. Verified:
|
|
`felhom-agent --version` → **0.41.0**, service **active**, logs clean (local-api leaf ready, both 9201
|
|
drives re-bound under the shared parent, hub desired-state updated).
|
|
|
|
**B. Demo remediation (9201 was provisioned pre-fix → `onboot:0`).** `pct config 9201` before → `onboot: 0`;
|
|
ran `pct set 9201 -onboot 1` (non-destructive, config-only); after → **`onboot: 1`** ✓.
|
|
|
|
**C. Back-half fix proper — LIVE-VALIDATED (supervised capstone re-provision, 2026-06-24).** Destroyed
|
|
guest 9201 (`pct stop` + `pct destroy` — 3 owned volumes removed; bind dirs + physical-drive data
|
|
survive) and re-provisioned it through the **real provision chain** with agent v0.41.0:
|
|
`felhom-agent --selftest=provision -archive local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst
|
|
-vmid 9201 -customer-id demo-felhom -hub-password <out-of-band> -rootfs-grow 0 -datavol-grow 184
|
|
-sysdata-grow 42`. Front-half bring-up OK (1m11s), back-half OK. **Assertion: the freshly-provisioned
|
|
`pct config 9201` showed `onboot: 1` with NO manual `pct set`** — set by the back-half fix. Full 3-volume
|
|
layout faithfully reproduced (rootfs 32G + mp0 docker 200G + mp1 /mnt/sys_drive 50G). Agent daemon restart
|
|
then reasserted both enrolled drives (felhom-flash, felhom-usb) at stable paths under
|
|
`/mnt/felhom-drives` (F9), skipped the ejected drive; controller + base infra came up healthy.
|
|
|
|
**Optional — host-reboot scenario LIVE-VALIDATED (the exact Phase-4.1 failure, now passing).** Rebooted
|
|
felhom-pve (uptime went 16:37 → 1 min, confirming a real reboot). **`pct status 9201` = running with NO
|
|
manual `pct start`** (auto-started via `onboot:1`). The other guests (9001, 9999; `onboot:0`) correctly
|
|
stayed **stopped** — proving the per-guest `onboot:1` property drives the auto-start, not a blanket
|
|
start-all. Post-reboot recovery confirmed: the agent (fresh PID) re-bound both drives at stable paths,
|
|
`/mnt/sys_drive` + `/var/lib/docker` are separate mountpoints, and controller + traefik + cloudflared +
|
|
filebrowser all came up healthy.
|
|
|
|
**Demo state after the capstone:** controller redeployed to **v0.82.0** (the fresh provision came up on
|
|
the golden-baked v0.77.0; pin bumped + `bootstrap` re-run). Standard post-re-provision residue (as every
|
|
prior capstone): customer apps are **not** redeployed and the controller's storage paths need a one-time
|
|
**UI re-enroll** of the two drives (their host-side data on felhom-usb is intact; the SSD mp0/mp1 app
|
|
data was destroyed with the volumes). The F3 fix itself is fully validated.
|