From aaa276a7b934bfc0c38cf550fabdf2a948943ea3 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 27 Jun 2026 11:56:40 +0200 Subject: [PATCH] =?UTF-8?q?build-golden.sh:=20default=20controller=20image?= =?UTF-8?q?=20=E2=86=92=20current=20(0.85.1);=20golden=20rebuilt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CONTROLLER_IMAGE default (arg 6) was a stale :0.43.0, so an argument-less golden build baked an ancient controller (fresh Day-0 boxes started at 0.77). Bumped the default to the current :0.85.1; always pass it explicitly per rebuild. Golden rebuilt at 0.85.1 on felhom-pve (volid vzdump-lxc-9100-2026_06_27-11_42_51); baked-image verify confirmed :0.85.1 in the build guest. No agent binary change. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK --- CHANGELOG.md | 20 ++++++++ REPORT.md | 103 +++++++++++++++++----------------------- configs/build-golden.sh | 5 +- 3 files changed, 67 insertions(+), 61 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3615722..61ab7cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,26 @@ All notable changes to **felhom-agent** are recorded here. Update on every code change that gets pushed. +## build-golden.sh — default controller image bumped to current; golden rebuilt at 0.85.1 (2026-06-27) + +**Operational + a default fix (no agent binary change — version stays v0.42.0).** + +- `configs/build-golden.sh`: the `CONTROLLER_IMAGE` default (positional arg 6) was a stale + `…/felhom-controller:0.43.0` — an argument-less golden build baked a wildly old controller, so fresh + Day-0 boxes started old (the demo started at 0.77). Bumped the default to the **current** + `…/felhom-controller:0.85.1` so the worst case (no explicit arg) is merely "current", not ancient. + - **Always pass the controller version explicitly at each rebuild** — this default only bounds the + worst case. A future `make golden` that resolves the latest pullable tag would remove the need for a + hand-bumped default (Observation, not this task). +- **Golden rebuilt at 0.85.1** on `felhom-pve` with the image passed **explicitly** + (`build-golden.sh 9100 … gitea.dooplex.hu/admin/felhom-controller:0.85.1`). New archive volid: + `local:backup/vzdump-lxc-9100-2026_06_27-11_42_51.tar.zst` (rootfs 32G + Docker-data 16G + user-data + 8G, all in the archive; mp0+mp1 inclusion confirmed in the vzdump log). + - **Baked-image verify (cheap, mandatory):** in the build guest `/etc/felhom-controller-image` = + `…:0.85.1` and `docker images` showed it baked (379 MB). New Day-0 provisions now ship current. + - The host-bootstrap script auto-discovers the newest golden, so it picks up this rebuild + automatically. The real demo 9201 was **not** re-provisioned (it is the Phase-2 floor test box). + ## v0.42.0 — agentic controller update: in-guest image swap + rollback (Phase 1) (2026-06-26) The host agent now owns the in-guest controller image **swap** — the new-architecture replacement for diff --git a/REPORT.md b/REPORT.md index 8d01aaf..0baa675 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,69 +1,52 @@ -# REPORT — v0.42.0: agentic controller update — in-guest image swap + rollback (Phase 1) +# REPORT — build-golden.sh default fix + golden rebuild at current (2026-06-27) -**Repo:** `felhom-agent` · **Version:** `v0.41.0` → **`v0.42.0`** · **Date:** 2026-06-26 +**Task class:** Risky/supervised, Part A only. **No agent binary change** — version stays **v0.42.0**. -## What & why -The controller's old self-update (`docker compose up -d` on an in-container compose path) is dead in the -LXC architecture. Phase 1 moves the **container swap** to the host agent: the in-guest controller pulls -the target image, then the agent — external to the controller container, so it survives the controller -being killed mid-swap — rewrites `/etc/felhom-controller-image`, restarts the golden's -`felhom-controller-bootstrap.service`, verifies health, and **rolls back** if the new controller doesn't -come up. The guest is never left without a controller. +## What changed -## Baseline -felhom-agent `4725396` v0.41.0 → v0.42.0. Cross-repo: felhom-controller v0.85.0 (the pull + delegate half). +`configs/build-golden.sh` — the `CONTROLLER_IMAGE` default (positional arg 6) was a stale +`gitea.dooplex.hu/admin/felhom-controller:0.43.0`. An argument-less build therefore baked a wildly old +controller, so fresh Day-0 boxes booted old (the demo started at 0.77). Bumped the default to the +**current** `…/felhom-controller:0.85.1`, with a comment that the version should still be passed +explicitly at each rebuild (a `make golden` that resolves the latest tag is a future Observation). -## Changes -- **`internal/localapi/controllerswap.go` (new)** — `ControllerSwapper` (testable, over a `GuestExecutor` - seam) + handlers: - - `POST /controller/swap {image}` (`withGuest`) → **202** `{status, previous_image, target_image}`, then - async: record previous to `/var/lib/felhom-agent/controller-swap-.json` (crash-safety) → confirm - the target image is present in the guest (else abort, **no swap**) → write the image file → restart the - bootstrap unit → poll the new controller to healthy (`docker inspect`, ≤90s) → **roll back** + restart - if not. Strict image-ref gate (`gitea.dooplex.hu/admin/felhom-controller:`); single-flight per - guest (409). - - `GET /controller/swap/status` → `{state, current, previous, target, error}`. -- **`internal/localapi/guestbind.go`** — `GuestBinder.GuestExec` (the single `pct exec` seam). -- **`internal/localapi/server.go`** — `ControllerSwap` option + route wiring + single-flight map. -- **`cmd/felhom-agent/main.go`** — `ControllerSwap: guestBinder`; `--selftest=controller-swap -vmid -image` - (+ the allowlist fix — see below); version → 0.42.0. +## Green gate -## Tests (`internal/localapi/controllerswap_test.go`) — all green -happy swap; **rollback-on-unhealthy** (image reverted to previous + restart called ≥2×); image-absent → -no swap (no file write); running-with-no-healthcheck → healthy; bad image → 400; single-flight → 409. -- **Companion red-proof:** disabling the verify→rollback left the guest on the bad image and **failed** - `TestControllerSwap_RollbackOnUnhealthy` (`image file = …:0.84.0 after rollback, want previous …:0.77.0`) - → reverted → green. The rollback assertion is load-bearing. -- localapi package: 70 → **71** tests. `go build ./... && go vet ./... && go test ./...` — all `ok`. +- `bash -n configs/build-golden.sh` → OK. -## Live test on demo 9201 (operator-authorized; agent v0.42.0 deployed to felhom-pve) -- **Bug found live + fixed:** `--selftest=controller-swap` printed flag-usage — `controller-swap` was - missing from `selftestFlag.Set`'s allowlist. Fixed (commit `b17d07b`), rebuilt, redeployed. -- **Swap 0.77→0.84** (`--selftest=controller-swap`, after pre-pulling 0.84): state=**done**, new controller - `Up (healthy)`, `/etc/felhom-controller-image` = 0.84.0. (~9s to healthy.) -- **Rollback** (swap to a present-but-broken `:99.99.99` built from a controller base with `ENTRYPOINT - /bin/false`): 90s verify timeout → **rolled back to 0.84.0 healthy**, state=failed, current=0.84.0 — the - guest was never left without a controller. -- **E2E (real UI button)** via claude-in-chrome on `felhom.demo-felhom.eu/settings`: with the demo on 0.85.0 - (new code) and 0.85.1 pushed as latest, clicked **"Frissítés telepítése"** → the controller pulled 0.85.1 - in-guest and called the agent → agent log: swap → "new controller healthy" `0.85.1` → UI reloaded showing - **"Sikeres (0.85.0 → 0.85.1)"**. No "docker-compose.yml nem elérhető". -- Hub reflects the version via the controller's report path: `0.84.0 → 0.85.0 → 0.85.1`. +## Golden rebuild (live on felhom-pve) -## Observations (out of scope — not acted on) -1. **Agent host-report `controller_version` is hardcoded `""`** (`internal/hub/collect.go:180`, "slice 8 - fills" — never done). So the hub's `guests.controller_version` stays empty; the controller's own report - path carries the real version (verified `0.85.1`). Pre-existing, untouched here. -2. **Agent-restart-mid-swap auto-resume:** the swap records previous to a state file (so a manual/operator - rollback is possible), but the agent does not yet auto-reconcile a `swapping` state file on its own - restart. The common failure (controller killed mid-swap) is fully handled; agent-crash-mid-swap leaves a - recoverable record. A startup reconcile is a candidate follow-up. -3. **Golden freshness:** the golden bakes controller 0.77.0; a golden rebuild at current is wanted so new - provisions start current (independent of this feature). +Ran the rebuild with the controller image passed **explicitly**: -## Commits -- `6f14b66` — v0.42.0 swap + rollback + tests. -- `b17d07b` — fix: register `controller-swap` in the `--selftest` allowlist (found live). +``` +build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst \ + local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.85.1 +``` +(`REGISTRY_USER`/`REGISTRY_TOKEN` sourced out-of-band from `/root/.golden-secrets.env`.) -No secrets in any committed file; the registry token used for the live pre-pull stayed on the host -(stdin login, logged out). +- **New golden archive volid:** `local:backup/vzdump-lxc-9100-2026_06_27-11_42_51.tar.zst` + (rootfs 32G + Docker-data 16G + user-data 8G; archive size ~574 MB; the vzdump log confirmed + *including mount point mp0 (/var/lib/docker)* and *mp1 (/mnt/sys_drive)* — neither excluded). + +### Baked-image verify (cheap, mandatory) — PASS + +In the build guest (started briefly, then stopped again): + +``` +/etc/felhom-controller-image → gitea.dooplex.hu/admin/felhom-controller:0.85.1 +docker images (felhom-controller) → gitea.dooplex.hu/admin/felhom-controller:0.85.1 379MB +``` + +New Day-0 provisions now ship **current** (the host-bootstrap script auto-discovers the newest golden, +so it picks up this rebuild with no further change). + +### Full verify (throwaway provision) — not performed + +Operator-gated/recommended only. The cheap verify already proves the archive carries 0.85.1; the real +demo 9201 was deliberately **not** re-provisioned (it is the Phase-2 floor test box). The build guest +9100 is left stopped (destroy with `pct destroy 9100 --purge` when no longer needed). + +## Observations + +- A `make golden` (or a `--latest` flag) that resolves the newest pullable controller tag at build time + would make the hand-bumped default unnecessary and prevent this class of drift entirely. diff --git a/configs/build-golden.sh b/configs/build-golden.sh index 2a87cd1..2198d8f 100644 --- a/configs/build-golden.sh +++ b/configs/build-golden.sh @@ -40,7 +40,10 @@ TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}" ROOTFS_STORAGE="${3:-local-lvm}" ARCHIVE_STORAGE="${4:-local}" BRIDGE="${5:-vmbr0}" -CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.43.0}" +# Default to the CURRENT controller so an argument-less build is never wildly stale. ALWAYS pass the +# controller version explicitly at each rebuild (this default only bounds the worst case); a future +# `make golden` that resolves the latest pullable tag would remove the need for a hand-bumped default. +CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.85.1}" REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}" # OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just # large enough for the baked images + headroom; provision grows it to the per-customer target.