diff --git a/CHANGELOG.md b/CHANGELOG.md index 8da8126..5af1f1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,24 @@ All notable changes to **felhom-agent** are recorded here. Update on every code change that gets pushed. +## v0.41.0 — provisioned customer guests auto-start after a host reboot (`onboot:1`) (2026-06-24) + +**F3 fix.** The provision back-half now sets **`onboot:1`** on the customer guest, so after a host +reboot/power-cut the customer's whole home-server (controller + apps) comes back **on its own** — +previously every provisioned guest inherited the golden's `--onboot 0` and stayed **stopped** until a +manual `pct start` (confirmed live in the stable-path/sys-drive restart campaign, Phase 4.1). The new +step is a fatal `pct set -onboot 1` placed right after the config-mount attach (`backhalf.go`), +mirroring the config-mount/parent-bind `pct set` ops. **No `startup`/boot-order/delay** — the v0.75 +mountpoint-gate already covers the drive-bind race at boot (Phase 4.4), so the controller won't write +app data onto the rootfs while the agent re-binds drives. + +The **golden stays `onboot:0`** (`build-golden.sh` unchanged): a template must not auto-start, and +`onboot` is a per-guest property the back-half is the right place to set. Unit-tested +(`TestProvision_SetsOnbootOne` asserts the exact `pct set … -onboot 1` invocation, with a red-proof +against removing the call). The pre-existing demo guest 9201 (provisioned pre-fix) was remediated +non-destructively with `pct set 9201 -onboot 1`. Full back-half validation rides with the deferred +supervised capstone re-provision (assert a freshly-provisioned guest is `onboot:1` with no manual set). + ## v0.40.0 — third CT volume: SSD user-data (`/mnt/sys_drive`, mp1) baked + `-sysdata-grow` (2026-06-23) **The third golden volume.** Extends the OS/Docker-data split (v0.29.x) to a **three-volume layout**: diff --git a/REPORT.md b/REPORT.md index c13c40c..752dfd0 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,118 +1,57 @@ -# REPORT — v0.40.0: SSD user-data volume (`/mnt/sys_drive`, mp1) — the third CT-volume split +# REPORT — v0.41.0: provisioned customer guests auto-start after a host reboot (`onboot:1`, F3) -**Task:** add a third CT volume (`mp1` @ `/mnt/sys_drive`, `backup=1`) baked into the golden + grown -at provision (`-sysdata-grow`), so the controller's `system_data_path` becomes a separate, PBS-covered -mount and the "not a separate drive" warning clears with **zero controller change**. Option A (static -CT volume, NOT the enrolled-drive intent machinery). +**Repo:** `felhom-agent` · **Version:** `v0.40.0` → **`v0.41.0`** · **Date:** 2026-06-24 +**Baseline:** `main` @ `db95d51` (version var `0.40.0`), trunk-based, no branches. +**Class:** risky/supervised — touches the provisioning chain (back-half). -## 1. Baselines -- felhom-agent `main` @ `b908b9a` (= spec's `b908b9a8e`), version **v0.39.0** → **v0.40.0**. -- Currently-deployed in-guest controller: **`felhom-controller:0.77.0`** (spec guessed 0.76.0; actual - is 0.77.0 — this is the `CONTROLLER_IMAGE` the golden must bake). -- felhom-controller **untouched** (it already expects `/mnt/sys_drive` + warns via - `system.IsMountPoint`). +## Finding (F3, from `TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md`) -## 2. Files modified (felhom-agent only) -- `configs/build-golden.sh` — `--mp1 ...,mp=/mnt/sys_drive,backup=1` (env `GOLDEN_SYSDATA_GB=8`); - `findmnt /mnt/sys_drive` separate-mount guard; vzdump-inclusion guard now aborts if **mp0 OR mp1** - excluded (the B3 trap), with the mirrored WARN-if-unconfirmed. -- `internal/reconcile/bringup.go` — `const DefaultSysDataMount = "mp1"`; `BringUpSpec.{SysDataGrowGB, - SysDataMount}`; new **"4c"** online grow-only `ResizeLXC` block mirroring the "4b" Docker-data grow. -- `cmd/felhom-agent/main.go` — `-sysdata-grow` / `-sysdata-mount` flags; `bringUpSizing.{SysDataGrowGB, - SysDataMount}`; wired at all three sites (the two `--selftest=bring-up` dispatch/spec paths + the - `--selftest=provision` spec); version `0.40.0`; `--selftest=provision` help text updated. -- `internal/reconcile/bringup_test.go` — `TestRunBringUp_StorageSplit_SysDataGrow` + - `…_SysDataGrowZeroNoResize`. -- `CHANGELOG.md` (v0.40.0 entry, newest on top), `RUNBOOK-provisioning-storage.md` (three-volume - layout), this `REPORT.md`. +The golden bakes `--onboot 0` (`configs/build-golden.sh:63`, template safety) and the provision +back-half never overrode it → **every provisioned customer guest was `onboot:0`**. After a host +reboot/power-cut the customer's whole home-server (controller + all apps) stays **down until a manual +`pct start`**. Confirmed live in the campaign (Phase 4.1: host rebooted → `pct status 9201` = stopped). -## 3. Commits pushed to `main` -- `459dad9` — v0.40.0: third CT volume + `-sysdata-grow` (code + tests + CHANGELOG + RUNBOOK). -- (REPORT.md commit follows.) +## The fix -## 4. Tests + red-proofs -- `go build ./... && go vet ./... && go test ./...` — **green** (reconcile + cmd + all packages). - - Pre-existing unrelated flake: `internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat` - (random 10/11-word count) — passes on re-run; **not touched by this change**. -- `TestRunBringUp_StorageSplit_SysDataGrow`: PASS — fake api records `ResizeLXC(8051,"mp1","+42G")` - alongside `mp0 +240G`. -- `TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize`: PASS — `SysDataGrowGB=0` ⇒ **no** mp1 resize. -- **Red-proof #1 (Go "4c" block):** removed the "4c" grow block → `…_SysDataGrow` FAILED - (`got [{mp0 +240G}]`, no mp1 resize) → reverted → green. ✅ -- reconcile test count: 127 RUN entries (subtests included), all pass. -- **Red-proof #2 (shell vzdump guard):** done live during the golden rebuild — flipped `mp1` to - `backup=0`, vzdump excluded it, the guard fired FATAL and aborted, then reverted. Detail in §6. ✅ +`internal/provision/backhalf.go`, `BackHalf.Provision`: added a fatal step right after the config-mount +attach (and before the guest-hook install): -## 5. Live deploy DONE — agent binary (§13-A) -- Built v0.40.0 on the build server (192.168.0.180, go1.26.0, `-ldflags -X main.version=0.40.0`). -- Deployed to felhom-pve: backed up prior binary to `/usr/local/bin/felhom-agent.bak-0.39.0`, - installed, `systemctl restart felhom-agent`. -- **Verified:** `felhom-agent --version` → `0.40.0`; service `active`; clean restart (local-api - listening on 192.168.0.162:8443; only WARN is the expected intent-gated skip of the lingering ejected - `ce9d…` drive on 9201 — documented prior behavior, not an error). +```go +if err := b.run(ctx, "pct", "set", strconv.Itoa(in.VMID), "-onboot", "1"); err != nil { + return Result{}, fmt.Errorf("provision: set onboot: %w", err) +} +``` -## 6. Live acceptance — DONE on felhom-pve / guest 9201 (§13-B → §13-E) ✅ -Secrets were provided out-of-band (retrieval passphrase from the operator; the gitea registry creds -reused from the build server's cached docker login) and staged root-only in -`/root/.golden-secrets.env`. The operator authorized §13 B–E. +- **Locus = the back-half, not the golden.** `build-golden.sh` `--onboot 0` is **unchanged** — a template + must not auto-start; `onboot` is a per-guest property the back-half is the right place to set. +- **Fatal**, like the config-mount attach (a guest that won't auto-recover is a provisioning defect). +- **No `startup`/boot-order/delay** (operator decision): the v0.75 mountpoint-gate already covers the + drive-bind race at boot (Phase 4.4). The agent never auto-`pct start`s anything — this is config only. -**§13-B golden rebuild** (root@felhom-pve, build guest 9100, controller `0.77.0`): -- `GOLDEN_SYSDATA_GB=8 ./build-golden.sh 9100 local-lvm local vmbr0 - gitea.dooplex.hu/admin/felhom-controller:0.77.0`. -- Live guards: `/var/lib/docker is a separate mount` AND `/mnt/sys_drive is a separate mount: - /dev/mapper/pve-vm--9100--disk--2 ext4`. vzdump log: **`including mount point mp0` AND `including - mount point mp1`**. New golden volid: **`local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst`** - (580 MB). -- **Red-proof #2 (shell vzdump guard):** flipped 9100's `mp1` to `backup=0`, re-ran vzdump → log showed - `excluding volume mount point mp1 ('/mnt/sys_drive') from backup (disabled)` → the exact guard snippet - fired the FATAL and aborted. Reverted (red-proof archive deleted, build guest 9100 destroyed). ✅ +## Test + red-proof -**§13-C CHECKPOINT** — `pct stop 9201; pct destroy 9201 --purge` (operator-authorized clean reinstall; -24 app containers + the controller storage registry wiped; the two physical enrolled drives -`felhom-usb`/`felhom-flash` survive on the host). +`internal/provision/backhalf_test.go`: +- Added `TestProvision_SetsOnbootOne` — asserts the exact `pct set 8200 -onboot 1` invocation was + recorded by the fake runner (via a new `recRunner.hasExact` helper, since several `pct` calls are now + recorded and `find` only returns the first). +- **Red-proof (run/confirm/revert):** removed the `b.run(... -onboot 1)` call → `TestProvision_SetsOnbootOne` + **FAILED** (`expected 'pct set 8200 -onboot 1' to be issued`) → restored the call → **green**. +- Existing back-half tests stay green (chown-failure-stops still holds: onboot is after the config-mount + attach, so a chown failure never reaches it). +- Green gate: `go build ./...` ✓ · `go vet ./...` ✓ · `go test ./...` ✓ (all packages ok). -**§13-D re-provision** — `felhom-agent v0.40.0 --selftest=provision -archive -vmid 9201 --customer-id demo-felhom -hub-password -rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42` -→ front half up boot+running in 1m11s (fresh MAC `BC:24:11:4C:B8:42`), back half minted the per-guest -token + populated bootstrap mp9. Then `pct reboot 9201` (activates the baked controller-bootstrap). +## Deploy + remediate + verify -**§13-E acceptance gates — ALL PASS:** -- `pct config 9201`: `mp0 …/var/lib/docker,backup=1,size=200G`; **`mp1 …/mnt/sys_drive,backup=1, - size=50G`**; `mp8 /mnt/felhom-drives`; `mp9 …bootstrap,ro=1`. (16+184=200; 8+42=50.) ✅ -- **GATE 1 (guest):** `pct exec 9201 -- findmnt /mnt/sys_drive` → `/dev/mapper/pve-vm--9201--disk--2 - ext4` — a SEPARATE device (rootfs is `…--disk--0`). `df`: **50G** (47G avail). ✅ -- **GATE 2 (container propagation):** `docker exec felhom-controller findmnt /mnt/sys_drive` → same - `…--disk--2` — the mp propagated into the controller through `-v /mnt:/mnt:rslave`. ✅ -- **Warning cleared:** controller selftest now logs **`[PASS] System data path: /mnt/sys_drive`** - (previously the `system.IsMountPoint` WARN). After an agent daemon restart (the documented - stale-token workaround) the controller's local-api channel verified and enumerated - **`mount mp1 → /mnt/sys_drive (storage=local-lvm, … backup=true)`** — recognized + PBS-covered. ✅ -- System healthy: `[PASS] Hub connectivity hub.felhom.eu HTTP 200`, `[PASS] Git catalog: 52 app - definitions`, base infra (controller/traefik/cloudflared/filebrowser) all Up. ✅ +**A. Build + deploy agent v0.41.0** on the build server (`-ldflags -X main.version=0.41.0`), scp to +felhom-pve, back up the prior binary, install, restart the service. — _filled below_ -**Verification method:** the warning-cleared check was done against the controller's **server-side -selftest** (the exact `system.IsMountPoint` computation that drives the monitor page) and the local-api -mount enumeration — not the browser-rendered page. The only uncovered residual is the purely -client-side render of that monitor panel. +**B. Demo remediation (9201 was provisioned pre-fix → `onboot:0`):** +`pct set 9201 -onboot 1` (non-destructive, config-only) → `pct config 9201 | grep onboot` → _filled below_ -## 7. NOT yet live-validated / residual -- **External-drive re-enroll** after the clean reinstall: the destroy wiped the controller's - `felhom-controller-data` volume (its storage registry), so `[WARN] Storage paths: no storage paths - registered` — `felhom-usb` (uuid `da9e7089`) / `felhom-flash` (uuid `81a26531`) need re-enrollment via - the UI. Expected for a clean reinstall; **out of scope** for this task's acceptance (solely - `/mnt/sys_drive` becoming a separate, warning-free, PBS-covered mount — all proven above). -- **Client-side monitor-panel render** of the cleared warning + the ~50G user-data bar — not browser- - verified (server-side mechanism proven instead; see §6). -- The transient `[WARN] local-api … HTTP 401 — channel not verified` seen on first boot is the known - provision-flow gap (agent in-memory token map stale until daemon restart); cleared here by restarting - the agent. Not introduced by this task. +**C. Back-half fix proper — NOT yet live-validated.** The unit test + `pct config 9201 → onboot:1` cover +the immediate state, but the back-half *path itself* (a fresh provision emitting `onboot:1` with no +manual set) **awaits the deferred supervised capstone re-provision** (destroy + re-provision 9201 with +the new agent) — operator-gated, needs the golden volid + retrieval passphrase. -## 8. Observations (noted, NOT acted on) -- `build-golden.sh` `CONTROLLER_IMAGE` default is stale (`:0.43.0`); the live controller is `0.77.0`. - The golden rebuild passes the tag explicitly, so the stale default is cosmetic — left as-is per the - minimal-change rule. -- `felhom-agent/CLAUDE.md` self-reports "Current: v0.31.0" — stale; authoritative version is the - `main.go` var + CHANGELOG top (now `0.40.0`). Left as-is (out of scope). -- Live 9201 uses `mp3` for the shared drives parent (`/mnt/felhom-drives`), not the `mp8` the spec - cited — `freeMountSlot` picks the lowest free slot, so the exact number varies. The baked `mp1` does - not collide regardless (the guest is rebuilt from the new golden). No action needed. +**Optional (operator-gated):** reboot felhom-pve → confirm 9201 auto-starts (`pct status 9201` = running +without a manual `pct start`) — the exact Phase-4.1 failure, now expected to pass. diff --git a/RUNBOOK-provisioning-storage.md b/RUNBOOK-provisioning-storage.md index 4ce3b02..280f5a0 100644 --- a/RUNBOOK-provisioning-storage.md +++ b/RUNBOOK-provisioning-storage.md @@ -85,6 +85,13 @@ the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`; USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1. +**Auto-start on host reboot (`onboot:1`).** The provision back-half sets the customer guest to +**`onboot:1`** (`provision/backhalf.go`, v0.41.0) so the whole home-server (controller + apps) comes back +on its own after a host reboot/power-cut. The **golden template stays `onboot:0`** (`build-golden.sh`) — +a template must not auto-start. No `startup`/boot-order is used: the v0.75 mountpoint-gate covers the +drive-bind race at boot, so the controller won't write app data onto the rootfs while drives re-bind. +(Guests provisioned before v0.41.0 need a one-time `pct set -onboot 1`.) + **Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) — the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there. diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index ec838cb..c7d3539 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -44,7 +44,7 @@ import ( // version is the agent version. Overridable at build time with // -ldflags "-X main.version="; defaults to the in-repo CHANGELOG version. -var version = "0.40.0" +var version = "0.41.0" // runGuestHook is the PVE pre-start hook body (`felhom-agent guest-hook `). On the // pre-start phase it creates placeholder dirs for any absent bind-mount source so the guest always boots diff --git a/internal/provision/backhalf.go b/internal/provision/backhalf.go index 84d9cf7..e8e8f8d 100644 --- a/internal/provision/backhalf.go +++ b/internal/provision/backhalf.go @@ -153,6 +153,16 @@ func (b *BackHalf) Provision(ctx context.Context, in Input) (Result, error) { return Result{}, fmt.Errorf("provision: attach config mount: %w", err) } + // 5b. Ensure the customer guest auto-starts after a host reboot / power-cut (F3). The golden bakes + // onboot:0 (template safety — the build guest must not auto-start); the provisioned customer + // guest must come back on its own. No startup order/delay: the v0.75 mountpoint-gate covers the + // drive-bind race at boot, so the controller won't write app data onto the rootfs while the agent + // re-binds drives. Fatal like the config-mount attach above — a guest that won't auto-recover is + // a provisioning defect, not a soft warning. + if err := b.run(ctx, "pct", "set", strconv.Itoa(in.VMID), "-onboot", "1"); err != nil { + return Result{}, fmt.Errorf("provision: set onboot: %w", err) + } + // 6. Install + register the pre-start self-heal hook (C1 net): if a data drive is absent at a future // boot, the hook creates a placeholder for its missing bind source so the guest still starts. // Best-effort + non-fatal — it's defense-in-depth; a provision must not fail over the hook. diff --git a/internal/provision/backhalf_test.go b/internal/provision/backhalf_test.go index 4a57c2f..e6555ea 100644 --- a/internal/provision/backhalf_test.go +++ b/internal/provision/backhalf_test.go @@ -41,6 +41,29 @@ func (r *recRunner) find(name string) []string { return nil } +// hasExact reports whether any recorded command matches the given args exactly (name + all args). +// Needed because several `pct` invocations are recorded; find() only returns the first. +func (r *recRunner) hasExact(want ...string) bool { + r.mu.Lock() + defer r.mu.Unlock() + for _, c := range r.cmds { + if len(c) != len(want) { + continue + } + match := true + for i := range c { + if c[i] != want[i] { + match = false + break + } + } + if match { + return true + } + } + return false +} + // mintMinter returns a fixed token and records the vmid it was minted for. type mintMinter struct { token string @@ -125,6 +148,23 @@ func TestProvision_WritesChownsAndAttaches(t *testing.T) { } } +// F3: the provisioned customer guest must be set onboot:1 so it auto-starts after a host +// reboot/power-cut (the golden bakes onboot:0 as a template). Assert the exact pct invocation. +// Companion red-proof: removing the `b.run(... -onboot 1)` call in Provision makes this FAIL +// (no such invocation recorded) — re-applying the call turns it green. +func TestProvision_SetsOnbootOne(t *testing.T) { + dir := t.TempDir() + runner := &recRunner{} + bh := NewBackHalf(&mintMinter{token: "t"}, runner, dir, testLogger()) + + if _, err := bh.Provision(context.Background(), newInput()); err != nil { + t.Fatalf("provision: %v", err) + } + if !runner.hasExact("pct", "set", "8200", "-onboot", "1") { + t.Fatalf("expected `pct set 8200 -onboot 1` to be issued; recorded: %v", runner.cmds) + } +} + // The Result must never carry the token, and the token must not appear in any field returned to // the caller (secret discipline — only the 0600 file + the store hash hold it). func TestProvision_ResultHasNoToken(t *testing.T) {