build-golden.sh: default controller image → current (0.85.1); golden rebuilt

The CONTROLLER_IMAGE default (arg 6) was a stale :0.43.0, so an argument-less
golden build baked an ancient controller (fresh Day-0 boxes started at 0.77).
Bumped the default to the current :0.85.1; always pass it explicitly per rebuild.
Golden rebuilt at 0.85.1 on felhom-pve (volid vzdump-lxc-9100-2026_06_27-11_42_51);
baked-image verify confirmed :0.85.1 in the build guest. No agent binary change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
This commit is contained in:
2026-06-27 11:56:40 +02:00
parent 2dcee2d7e8
commit aaa276a7b9
3 changed files with 67 additions and 61 deletions
+20
View File
@@ -3,6 +3,26 @@
All notable changes to **felhom-agent** are recorded here. Update on every code
change that gets pushed.
## build-golden.sh — default controller image bumped to current; golden rebuilt at 0.85.1 (2026-06-27)
**Operational + a default fix (no agent binary change — version stays v0.42.0).**
- `configs/build-golden.sh`: the `CONTROLLER_IMAGE` default (positional arg 6) was a stale
`…/felhom-controller:0.43.0` — an argument-less golden build baked a wildly old controller, so fresh
Day-0 boxes started old (the demo started at 0.77). Bumped the default to the **current**
`…/felhom-controller:0.85.1` so the worst case (no explicit arg) is merely "current", not ancient.
- **Always pass the controller version explicitly at each rebuild** — this default only bounds the
worst case. A future `make golden` that resolves the latest pullable tag would remove the need for a
hand-bumped default (Observation, not this task).
- **Golden rebuilt at 0.85.1** on `felhom-pve` with the image passed **explicitly**
(`build-golden.sh 9100 … gitea.dooplex.hu/admin/felhom-controller:0.85.1`). New archive volid:
`local:backup/vzdump-lxc-9100-2026_06_27-11_42_51.tar.zst` (rootfs 32G + Docker-data 16G + user-data
8G, all in the archive; mp0+mp1 inclusion confirmed in the vzdump log).
- **Baked-image verify (cheap, mandatory):** in the build guest `/etc/felhom-controller-image` =
`…:0.85.1` and `docker images` showed it baked (379 MB). New Day-0 provisions now ship current.
- The host-bootstrap script auto-discovers the newest golden, so it picks up this rebuild
automatically. The real demo 9201 was **not** re-provisioned (it is the Phase-2 floor test box).
## v0.42.0 — agentic controller update: in-guest image swap + rollback (Phase 1) (2026-06-26)
The host agent now owns the in-guest controller image **swap** — the new-architecture replacement for
+43 -60
View File
@@ -1,69 +1,52 @@
# REPORT — v0.42.0: agentic controller update — in-guest image swap + rollback (Phase 1)
# REPORT — build-golden.sh default fix + golden rebuild at current (2026-06-27)
**Repo:** `felhom-agent` · **Version:** `v0.41.0`**`v0.42.0`** · **Date:** 2026-06-26
**Task class:** Risky/supervised, Part A only. **No agent binary change** — version stays **v0.42.0**.
## What & why
The controller's old self-update (`docker compose up -d` on an in-container compose path) is dead in the
LXC architecture. Phase 1 moves the **container swap** to the host agent: the in-guest controller pulls
the target image, then the agent — external to the controller container, so it survives the controller
being killed mid-swap — rewrites `/etc/felhom-controller-image`, restarts the golden's
`felhom-controller-bootstrap.service`, verifies health, and **rolls back** if the new controller doesn't
come up. The guest is never left without a controller.
## What changed
## Baseline
felhom-agent `4725396` v0.41.0 → v0.42.0. Cross-repo: felhom-controller v0.85.0 (the pull + delegate half).
`configs/build-golden.sh` — the `CONTROLLER_IMAGE` default (positional arg 6) was a stale
`gitea.dooplex.hu/admin/felhom-controller:0.43.0`. An argument-less build therefore baked a wildly old
controller, so fresh Day-0 boxes booted old (the demo started at 0.77). Bumped the default to the
**current** `…/felhom-controller:0.85.1`, with a comment that the version should still be passed
explicitly at each rebuild (a `make golden` that resolves the latest tag is a future Observation).
## Changes
- **`internal/localapi/controllerswap.go` (new)** — `ControllerSwapper` (testable, over a `GuestExecutor`
seam) + handlers:
- `POST /controller/swap {image}` (`withGuest`) → **202** `{status, previous_image, target_image}`, then
async: record previous to `/var/lib/felhom-agent/controller-swap-<vmid>.json` (crash-safety) → confirm
the target image is present in the guest (else abort, **no swap**) → write the image file → restart the
bootstrap unit → poll the new controller to healthy (`docker inspect`, ≤90s) → **roll back** + restart
if not. Strict image-ref gate (`gitea.dooplex.hu/admin/felhom-controller:<semver>`); single-flight per
guest (409).
- `GET /controller/swap/status``{state, current, previous, target, error}`.
- **`internal/localapi/guestbind.go`** — `GuestBinder.GuestExec` (the single `pct exec` seam).
- **`internal/localapi/server.go`** — `ControllerSwap` option + route wiring + single-flight map.
- **`cmd/felhom-agent/main.go`** — `ControllerSwap: guestBinder`; `--selftest=controller-swap -vmid -image`
(+ the allowlist fix — see below); version → 0.42.0.
## Green gate
## Tests (`internal/localapi/controllerswap_test.go`) — all green
happy swap; **rollback-on-unhealthy** (image reverted to previous + restart called ≥2×); image-absent →
no swap (no file write); running-with-no-healthcheck → healthy; bad image → 400; single-flight → 409.
- **Companion red-proof:** disabling the verify→rollback left the guest on the bad image and **failed**
`TestControllerSwap_RollbackOnUnhealthy` (`image file = …:0.84.0 after rollback, want previous …:0.77.0`)
→ reverted → green. The rollback assertion is load-bearing.
- localapi package: 70 → **71** tests. `go build ./... && go vet ./... && go test ./...` — all `ok`.
- `bash -n configs/build-golden.sh` → OK.
## Live test on demo 9201 (operator-authorized; agent v0.42.0 deployed to felhom-pve)
- **Bug found live + fixed:** `--selftest=controller-swap` printed flag-usage — `controller-swap` was
missing from `selftestFlag.Set`'s allowlist. Fixed (commit `b17d07b`), rebuilt, redeployed.
- **Swap 0.77→0.84** (`--selftest=controller-swap`, after pre-pulling 0.84): state=**done**, new controller
`Up (healthy)`, `/etc/felhom-controller-image` = 0.84.0. (~9s to healthy.)
- **Rollback** (swap to a present-but-broken `:99.99.99` built from a controller base with `ENTRYPOINT
/bin/false`): 90s verify timeout → **rolled back to 0.84.0 healthy**, state=failed, current=0.84.0 — the
guest was never left without a controller.
- **E2E (real UI button)** via claude-in-chrome on `felhom.demo-felhom.eu/settings`: with the demo on 0.85.0
(new code) and 0.85.1 pushed as latest, clicked **"Frissítés telepítése"** → the controller pulled 0.85.1
in-guest and called the agent → agent log: swap → "new controller healthy" `0.85.1` → UI reloaded showing
**"Sikeres (0.85.0 → 0.85.1)"**. No "docker-compose.yml nem elérhető".
- Hub reflects the version via the controller's report path: `0.84.0 → 0.85.0 → 0.85.1`.
## Golden rebuild (live on felhom-pve)
## Observations (out of scope — not acted on)
1. **Agent host-report `controller_version` is hardcoded `""`** (`internal/hub/collect.go:180`, "slice 8
fills" — never done). So the hub's `guests.controller_version` stays empty; the controller's own report
path carries the real version (verified `0.85.1`). Pre-existing, untouched here.
2. **Agent-restart-mid-swap auto-resume:** the swap records previous to a state file (so a manual/operator
rollback is possible), but the agent does not yet auto-reconcile a `swapping` state file on its own
restart. The common failure (controller killed mid-swap) is fully handled; agent-crash-mid-swap leaves a
recoverable record. A startup reconcile is a candidate follow-up.
3. **Golden freshness:** the golden bakes controller 0.77.0; a golden rebuild at current is wanted so new
provisions start current (independent of this feature).
Ran the rebuild with the controller image passed **explicitly**:
## Commits
- `6f14b66` — v0.42.0 swap + rollback + tests.
- `b17d07b` — fix: register `controller-swap` in the `--selftest` allowlist (found live).
```
build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst \
local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.85.1
```
(`REGISTRY_USER`/`REGISTRY_TOKEN` sourced out-of-band from `/root/.golden-secrets.env`.)
No secrets in any committed file; the registry token used for the live pre-pull stayed on the host
(stdin login, logged out).
- **New golden archive volid:** `local:backup/vzdump-lxc-9100-2026_06_27-11_42_51.tar.zst`
(rootfs 32G + Docker-data 16G + user-data 8G; archive size ~574 MB; the vzdump log confirmed
*including mount point mp0 (/var/lib/docker)* and *mp1 (/mnt/sys_drive)* — neither excluded).
### Baked-image verify (cheap, mandatory) — PASS
In the build guest (started briefly, then stopped again):
```
/etc/felhom-controller-image → gitea.dooplex.hu/admin/felhom-controller:0.85.1
docker images (felhom-controller) → gitea.dooplex.hu/admin/felhom-controller:0.85.1 379MB
```
New Day-0 provisions now ship **current** (the host-bootstrap script auto-discovers the newest golden,
so it picks up this rebuild with no further change).
### Full verify (throwaway provision) — not performed
Operator-gated/recommended only. The cheap verify already proves the archive carries 0.85.1; the real
demo 9201 was deliberately **not** re-provisioned (it is the Phase-2 floor test box). The build guest
9100 is left stopped (destroy with `pct destroy 9100 --purge` when no longer needed).
## Observations
- A `make golden` (or a `--latest` flag) that resolves the newest pullable controller tag at build time
would make the hand-bumped default unnecessary and prevent this class of drift entirely.
+4 -1
View File
@@ -40,7 +40,10 @@ TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
ROOTFS_STORAGE="${3:-local-lvm}"
ARCHIVE_STORAGE="${4:-local}"
BRIDGE="${5:-vmbr0}"
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.43.0}"
# Default to the CURRENT controller so an argument-less build is never wildly stale. ALWAYS pass the
# controller version explicitly at each rebuild (this default only bounds the worst case); a future
# `make golden` that resolves the latest pullable tag would remove the need for a hand-bumped default.
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.85.1}"
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just
# large enough for the baked images + headroom; provision grows it to the per-customer target.