build-golden.sh: default controller image → current (0.85.1); golden rebuilt
The CONTROLLER_IMAGE default (arg 6) was a stale :0.43.0, so an argument-less golden build baked an ancient controller (fresh Day-0 boxes started at 0.77). Bumped the default to the current :0.85.1; always pass it explicitly per rebuild. Golden rebuilt at 0.85.1 on felhom-pve (volid vzdump-lxc-9100-2026_06_27-11_42_51); baked-image verify confirmed :0.85.1 in the build guest. No agent binary change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
This commit is contained in:
@@ -3,6 +3,26 @@
|
||||
All notable changes to **felhom-agent** are recorded here. Update on every code
|
||||
change that gets pushed.
|
||||
|
||||
## build-golden.sh — default controller image bumped to current; golden rebuilt at 0.85.1 (2026-06-27)
|
||||
|
||||
**Operational + a default fix (no agent binary change — version stays v0.42.0).**
|
||||
|
||||
- `configs/build-golden.sh`: the `CONTROLLER_IMAGE` default (positional arg 6) was a stale
|
||||
`…/felhom-controller:0.43.0` — an argument-less golden build baked a wildly old controller, so fresh
|
||||
Day-0 boxes started old (the demo started at 0.77). Bumped the default to the **current**
|
||||
`…/felhom-controller:0.85.1` so the worst case (no explicit arg) is merely "current", not ancient.
|
||||
- **Always pass the controller version explicitly at each rebuild** — this default only bounds the
|
||||
worst case. A future `make golden` that resolves the latest pullable tag would remove the need for a
|
||||
hand-bumped default (Observation, not this task).
|
||||
- **Golden rebuilt at 0.85.1** on `felhom-pve` with the image passed **explicitly**
|
||||
(`build-golden.sh 9100 … gitea.dooplex.hu/admin/felhom-controller:0.85.1`). New archive volid:
|
||||
`local:backup/vzdump-lxc-9100-2026_06_27-11_42_51.tar.zst` (rootfs 32G + Docker-data 16G + user-data
|
||||
8G, all in the archive; mp0+mp1 inclusion confirmed in the vzdump log).
|
||||
- **Baked-image verify (cheap, mandatory):** in the build guest `/etc/felhom-controller-image` =
|
||||
`…:0.85.1` and `docker images` showed it baked (379 MB). New Day-0 provisions now ship current.
|
||||
- The host-bootstrap script auto-discovers the newest golden, so it picks up this rebuild
|
||||
automatically. The real demo 9201 was **not** re-provisioned (it is the Phase-2 floor test box).
|
||||
|
||||
## v0.42.0 — agentic controller update: in-guest image swap + rollback (Phase 1) (2026-06-26)
|
||||
|
||||
The host agent now owns the in-guest controller image **swap** — the new-architecture replacement for
|
||||
|
||||
@@ -1,69 +1,52 @@
|
||||
# REPORT — v0.42.0: agentic controller update — in-guest image swap + rollback (Phase 1)
|
||||
# REPORT — build-golden.sh default fix + golden rebuild at current (2026-06-27)
|
||||
|
||||
**Repo:** `felhom-agent` · **Version:** `v0.41.0` → **`v0.42.0`** · **Date:** 2026-06-26
|
||||
**Task class:** Risky/supervised, Part A only. **No agent binary change** — version stays **v0.42.0**.
|
||||
|
||||
## What & why
|
||||
The controller's old self-update (`docker compose up -d` on an in-container compose path) is dead in the
|
||||
LXC architecture. Phase 1 moves the **container swap** to the host agent: the in-guest controller pulls
|
||||
the target image, then the agent — external to the controller container, so it survives the controller
|
||||
being killed mid-swap — rewrites `/etc/felhom-controller-image`, restarts the golden's
|
||||
`felhom-controller-bootstrap.service`, verifies health, and **rolls back** if the new controller doesn't
|
||||
come up. The guest is never left without a controller.
|
||||
## What changed
|
||||
|
||||
## Baseline
|
||||
felhom-agent `4725396` v0.41.0 → v0.42.0. Cross-repo: felhom-controller v0.85.0 (the pull + delegate half).
|
||||
`configs/build-golden.sh` — the `CONTROLLER_IMAGE` default (positional arg 6) was a stale
|
||||
`gitea.dooplex.hu/admin/felhom-controller:0.43.0`. An argument-less build therefore baked a wildly old
|
||||
controller, so fresh Day-0 boxes booted old (the demo started at 0.77). Bumped the default to the
|
||||
**current** `…/felhom-controller:0.85.1`, with a comment that the version should still be passed
|
||||
explicitly at each rebuild (a `make golden` that resolves the latest tag is a future Observation).
|
||||
|
||||
## Changes
|
||||
- **`internal/localapi/controllerswap.go` (new)** — `ControllerSwapper` (testable, over a `GuestExecutor`
|
||||
seam) + handlers:
|
||||
- `POST /controller/swap {image}` (`withGuest`) → **202** `{status, previous_image, target_image}`, then
|
||||
async: record previous to `/var/lib/felhom-agent/controller-swap-<vmid>.json` (crash-safety) → confirm
|
||||
the target image is present in the guest (else abort, **no swap**) → write the image file → restart the
|
||||
bootstrap unit → poll the new controller to healthy (`docker inspect`, ≤90s) → **roll back** + restart
|
||||
if not. Strict image-ref gate (`gitea.dooplex.hu/admin/felhom-controller:<semver>`); single-flight per
|
||||
guest (409).
|
||||
- `GET /controller/swap/status` → `{state, current, previous, target, error}`.
|
||||
- **`internal/localapi/guestbind.go`** — `GuestBinder.GuestExec` (the single `pct exec` seam).
|
||||
- **`internal/localapi/server.go`** — `ControllerSwap` option + route wiring + single-flight map.
|
||||
- **`cmd/felhom-agent/main.go`** — `ControllerSwap: guestBinder`; `--selftest=controller-swap -vmid -image`
|
||||
(+ the allowlist fix — see below); version → 0.42.0.
|
||||
## Green gate
|
||||
|
||||
## Tests (`internal/localapi/controllerswap_test.go`) — all green
|
||||
happy swap; **rollback-on-unhealthy** (image reverted to previous + restart called ≥2×); image-absent →
|
||||
no swap (no file write); running-with-no-healthcheck → healthy; bad image → 400; single-flight → 409.
|
||||
- **Companion red-proof:** disabling the verify→rollback left the guest on the bad image and **failed**
|
||||
`TestControllerSwap_RollbackOnUnhealthy` (`image file = …:0.84.0 after rollback, want previous …:0.77.0`)
|
||||
→ reverted → green. The rollback assertion is load-bearing.
|
||||
- localapi package: 70 → **71** tests. `go build ./... && go vet ./... && go test ./...` — all `ok`.
|
||||
- `bash -n configs/build-golden.sh` → OK.
|
||||
|
||||
## Live test on demo 9201 (operator-authorized; agent v0.42.0 deployed to felhom-pve)
|
||||
- **Bug found live + fixed:** `--selftest=controller-swap` printed flag-usage — `controller-swap` was
|
||||
missing from `selftestFlag.Set`'s allowlist. Fixed (commit `b17d07b`), rebuilt, redeployed.
|
||||
- **Swap 0.77→0.84** (`--selftest=controller-swap`, after pre-pulling 0.84): state=**done**, new controller
|
||||
`Up (healthy)`, `/etc/felhom-controller-image` = 0.84.0. (~9s to healthy.)
|
||||
- **Rollback** (swap to a present-but-broken `:99.99.99` built from a controller base with `ENTRYPOINT
|
||||
/bin/false`): 90s verify timeout → **rolled back to 0.84.0 healthy**, state=failed, current=0.84.0 — the
|
||||
guest was never left without a controller.
|
||||
- **E2E (real UI button)** via claude-in-chrome on `felhom.demo-felhom.eu/settings`: with the demo on 0.85.0
|
||||
(new code) and 0.85.1 pushed as latest, clicked **"Frissítés telepítése"** → the controller pulled 0.85.1
|
||||
in-guest and called the agent → agent log: swap → "new controller healthy" `0.85.1` → UI reloaded showing
|
||||
**"Sikeres (0.85.0 → 0.85.1)"**. No "docker-compose.yml nem elérhető".
|
||||
- Hub reflects the version via the controller's report path: `0.84.0 → 0.85.0 → 0.85.1`.
|
||||
## Golden rebuild (live on felhom-pve)
|
||||
|
||||
## Observations (out of scope — not acted on)
|
||||
1. **Agent host-report `controller_version` is hardcoded `""`** (`internal/hub/collect.go:180`, "slice 8
|
||||
fills" — never done). So the hub's `guests.controller_version` stays empty; the controller's own report
|
||||
path carries the real version (verified `0.85.1`). Pre-existing, untouched here.
|
||||
2. **Agent-restart-mid-swap auto-resume:** the swap records previous to a state file (so a manual/operator
|
||||
rollback is possible), but the agent does not yet auto-reconcile a `swapping` state file on its own
|
||||
restart. The common failure (controller killed mid-swap) is fully handled; agent-crash-mid-swap leaves a
|
||||
recoverable record. A startup reconcile is a candidate follow-up.
|
||||
3. **Golden freshness:** the golden bakes controller 0.77.0; a golden rebuild at current is wanted so new
|
||||
provisions start current (independent of this feature).
|
||||
Ran the rebuild with the controller image passed **explicitly**:
|
||||
|
||||
## Commits
|
||||
- `6f14b66` — v0.42.0 swap + rollback + tests.
|
||||
- `b17d07b` — fix: register `controller-swap` in the `--selftest` allowlist (found live).
|
||||
```
|
||||
build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst \
|
||||
local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.85.1
|
||||
```
|
||||
(`REGISTRY_USER`/`REGISTRY_TOKEN` sourced out-of-band from `/root/.golden-secrets.env`.)
|
||||
|
||||
No secrets in any committed file; the registry token used for the live pre-pull stayed on the host
|
||||
(stdin login, logged out).
|
||||
- **New golden archive volid:** `local:backup/vzdump-lxc-9100-2026_06_27-11_42_51.tar.zst`
|
||||
(rootfs 32G + Docker-data 16G + user-data 8G; archive size ~574 MB; the vzdump log confirmed
|
||||
*including mount point mp0 (/var/lib/docker)* and *mp1 (/mnt/sys_drive)* — neither excluded).
|
||||
|
||||
### Baked-image verify (cheap, mandatory) — PASS
|
||||
|
||||
In the build guest (started briefly, then stopped again):
|
||||
|
||||
```
|
||||
/etc/felhom-controller-image → gitea.dooplex.hu/admin/felhom-controller:0.85.1
|
||||
docker images (felhom-controller) → gitea.dooplex.hu/admin/felhom-controller:0.85.1 379MB
|
||||
```
|
||||
|
||||
New Day-0 provisions now ship **current** (the host-bootstrap script auto-discovers the newest golden,
|
||||
so it picks up this rebuild with no further change).
|
||||
|
||||
### Full verify (throwaway provision) — not performed
|
||||
|
||||
Operator-gated/recommended only. The cheap verify already proves the archive carries 0.85.1; the real
|
||||
demo 9201 was deliberately **not** re-provisioned (it is the Phase-2 floor test box). The build guest
|
||||
9100 is left stopped (destroy with `pct destroy 9100 --purge` when no longer needed).
|
||||
|
||||
## Observations
|
||||
|
||||
- A `make golden` (or a `--latest` flag) that resolves the newest pullable controller tag at build time
|
||||
would make the hand-bumped default unnecessary and prevent this class of drift entirely.
|
||||
|
||||
@@ -40,7 +40,10 @@ TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
|
||||
ROOTFS_STORAGE="${3:-local-lvm}"
|
||||
ARCHIVE_STORAGE="${4:-local}"
|
||||
BRIDGE="${5:-vmbr0}"
|
||||
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.43.0}"
|
||||
# Default to the CURRENT controller so an argument-less build is never wildly stale. ALWAYS pass the
|
||||
# controller version explicitly at each rebuild (this default only bounds the worst case); a future
|
||||
# `make golden` that resolves the latest pullable tag would remove the need for a hand-bumped default.
|
||||
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.85.1}"
|
||||
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
|
||||
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just
|
||||
# large enough for the baked images + headroom; provision grows it to the per-customer target.
|
||||
|
||||
Reference in New Issue
Block a user