diff --git a/README.md b/README.md index 9891718..9770c75 100644 --- a/README.md +++ b/README.md @@ -141,6 +141,23 @@ See the table in [`internal/proxmox/doc.go`](internal/proxmox/doc.go). Summary: lifecycle **including restore** is API-token-covered; OS-root is confined to golden-image `keyctl` create, host mounts, and SMART/sensors (phase3 §B3). +### Controller swap (agentic controller update, Phase 1 — v0.42.0) + +The local API (`internal/localapi/`) owns the in-guest controller image **swap** — the new-architecture +replacement for the controller's dead in-container `docker compose` self-update. The in-guest controller +pre-pulls the target image then calls the agent: + +- `POST /controller/swap {image}` (`withGuest`-scoped) → **202**, then async: record previous + (`/var/lib/felhom-agent/controller-swap-.json`) → confirm the target is present in the guest → + write `/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service` → poll + the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the + guest is never left without a controller). Strict image-ref gate; single-flight per guest (409). +- `GET /controller/swap/status` → `{state, current, previous, target, error}`. + +The agent is **external** to the controller container, so it survives the controller being killed +mid-swap (which the controller cannot do to itself). `GuestBinder.GuestExec` is the single `pct exec` +seam. Exercise directly with `--selftest=controller-swap -vmid -image `. + ### TLS trust The host serves a self-signed cert. Verification is **not** blanket-disabled. Pick one in diff --git a/REPORT.md b/REPORT.md index cd7fb28..8d01aaf 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,77 +1,69 @@ -# REPORT — v0.41.0: provisioned customer guests auto-start after a host reboot (`onboot:1`, F3) +# REPORT — v0.42.0: agentic controller update — in-guest image swap + rollback (Phase 1) -**Repo:** `felhom-agent` · **Version:** `v0.40.0` → **`v0.41.0`** · **Date:** 2026-06-24 -**Baseline:** `main` @ `db95d51` (version var `0.40.0`), trunk-based, no branches. -**Class:** risky/supervised — touches the provisioning chain (back-half). +**Repo:** `felhom-agent` · **Version:** `v0.41.0` → **`v0.42.0`** · **Date:** 2026-06-26 -## Finding (F3, from `TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md`) +## What & why +The controller's old self-update (`docker compose up -d` on an in-container compose path) is dead in the +LXC architecture. Phase 1 moves the **container swap** to the host agent: the in-guest controller pulls +the target image, then the agent — external to the controller container, so it survives the controller +being killed mid-swap — rewrites `/etc/felhom-controller-image`, restarts the golden's +`felhom-controller-bootstrap.service`, verifies health, and **rolls back** if the new controller doesn't +come up. The guest is never left without a controller. -The golden bakes `--onboot 0` (`configs/build-golden.sh:63`, template safety) and the provision -back-half never overrode it → **every provisioned customer guest was `onboot:0`**. After a host -reboot/power-cut the customer's whole home-server (controller + all apps) stays **down until a manual -`pct start`**. Confirmed live in the campaign (Phase 4.1: host rebooted → `pct status 9201` = stopped). +## Baseline +felhom-agent `4725396` v0.41.0 → v0.42.0. Cross-repo: felhom-controller v0.85.0 (the pull + delegate half). -## The fix +## Changes +- **`internal/localapi/controllerswap.go` (new)** — `ControllerSwapper` (testable, over a `GuestExecutor` + seam) + handlers: + - `POST /controller/swap {image}` (`withGuest`) → **202** `{status, previous_image, target_image}`, then + async: record previous to `/var/lib/felhom-agent/controller-swap-.json` (crash-safety) → confirm + the target image is present in the guest (else abort, **no swap**) → write the image file → restart the + bootstrap unit → poll the new controller to healthy (`docker inspect`, ≤90s) → **roll back** + restart + if not. Strict image-ref gate (`gitea.dooplex.hu/admin/felhom-controller:`); single-flight per + guest (409). + - `GET /controller/swap/status` → `{state, current, previous, target, error}`. +- **`internal/localapi/guestbind.go`** — `GuestBinder.GuestExec` (the single `pct exec` seam). +- **`internal/localapi/server.go`** — `ControllerSwap` option + route wiring + single-flight map. +- **`cmd/felhom-agent/main.go`** — `ControllerSwap: guestBinder`; `--selftest=controller-swap -vmid -image` + (+ the allowlist fix — see below); version → 0.42.0. -`internal/provision/backhalf.go`, `BackHalf.Provision`: added a fatal step right after the config-mount -attach (and before the guest-hook install): +## Tests (`internal/localapi/controllerswap_test.go`) — all green +happy swap; **rollback-on-unhealthy** (image reverted to previous + restart called ≥2×); image-absent → +no swap (no file write); running-with-no-healthcheck → healthy; bad image → 400; single-flight → 409. +- **Companion red-proof:** disabling the verify→rollback left the guest on the bad image and **failed** + `TestControllerSwap_RollbackOnUnhealthy` (`image file = …:0.84.0 after rollback, want previous …:0.77.0`) + → reverted → green. The rollback assertion is load-bearing. +- localapi package: 70 → **71** tests. `go build ./... && go vet ./... && go test ./...` — all `ok`. -```go -if err := b.run(ctx, "pct", "set", strconv.Itoa(in.VMID), "-onboot", "1"); err != nil { - return Result{}, fmt.Errorf("provision: set onboot: %w", err) -} -``` +## Live test on demo 9201 (operator-authorized; agent v0.42.0 deployed to felhom-pve) +- **Bug found live + fixed:** `--selftest=controller-swap` printed flag-usage — `controller-swap` was + missing from `selftestFlag.Set`'s allowlist. Fixed (commit `b17d07b`), rebuilt, redeployed. +- **Swap 0.77→0.84** (`--selftest=controller-swap`, after pre-pulling 0.84): state=**done**, new controller + `Up (healthy)`, `/etc/felhom-controller-image` = 0.84.0. (~9s to healthy.) +- **Rollback** (swap to a present-but-broken `:99.99.99` built from a controller base with `ENTRYPOINT + /bin/false`): 90s verify timeout → **rolled back to 0.84.0 healthy**, state=failed, current=0.84.0 — the + guest was never left without a controller. +- **E2E (real UI button)** via claude-in-chrome on `felhom.demo-felhom.eu/settings`: with the demo on 0.85.0 + (new code) and 0.85.1 pushed as latest, clicked **"Frissítés telepítése"** → the controller pulled 0.85.1 + in-guest and called the agent → agent log: swap → "new controller healthy" `0.85.1` → UI reloaded showing + **"Sikeres (0.85.0 → 0.85.1)"**. No "docker-compose.yml nem elérhető". +- Hub reflects the version via the controller's report path: `0.84.0 → 0.85.0 → 0.85.1`. -- **Locus = the back-half, not the golden.** `build-golden.sh` `--onboot 0` is **unchanged** — a template - must not auto-start; `onboot` is a per-guest property the back-half is the right place to set. -- **Fatal**, like the config-mount attach (a guest that won't auto-recover is a provisioning defect). -- **No `startup`/boot-order/delay** (operator decision): the v0.75 mountpoint-gate already covers the - drive-bind race at boot (Phase 4.4). The agent never auto-`pct start`s anything — this is config only. +## Observations (out of scope — not acted on) +1. **Agent host-report `controller_version` is hardcoded `""`** (`internal/hub/collect.go:180`, "slice 8 + fills" — never done). So the hub's `guests.controller_version` stays empty; the controller's own report + path carries the real version (verified `0.85.1`). Pre-existing, untouched here. +2. **Agent-restart-mid-swap auto-resume:** the swap records previous to a state file (so a manual/operator + rollback is possible), but the agent does not yet auto-reconcile a `swapping` state file on its own + restart. The common failure (controller killed mid-swap) is fully handled; agent-crash-mid-swap leaves a + recoverable record. A startup reconcile is a candidate follow-up. +3. **Golden freshness:** the golden bakes controller 0.77.0; a golden rebuild at current is wanted so new + provisions start current (independent of this feature). -## Test + red-proof +## Commits +- `6f14b66` — v0.42.0 swap + rollback + tests. +- `b17d07b` — fix: register `controller-swap` in the `--selftest` allowlist (found live). -`internal/provision/backhalf_test.go`: -- Added `TestProvision_SetsOnbootOne` — asserts the exact `pct set 8200 -onboot 1` invocation was - recorded by the fake runner (via a new `recRunner.hasExact` helper, since several `pct` calls are now - recorded and `find` only returns the first). -- **Red-proof (run/confirm/revert):** removed the `b.run(... -onboot 1)` call → `TestProvision_SetsOnbootOne` - **FAILED** (`expected 'pct set 8200 -onboot 1' to be issued`) → restored the call → **green**. -- Existing back-half tests stay green (chown-failure-stops still holds: onboot is after the config-mount - attach, so a chown failure never reaches it). -- Green gate: `go build ./...` ✓ · `go vet ./...` ✓ · `go test ./...` ✓ (all packages ok). - -## Deploy + remediate + verify - -**A. Built + deployed agent v0.41.0.** Built on 192.168.0.180 (go1.26, `-ldflags -X main.version=0.41.0`, -source @ `166a1c8`) → scp 180→local→felhom-pve → backed up the prior binary to -`/usr/local/bin/felhom-agent.bak-0.40.0` → `install -m0755` → `systemctl restart felhom-agent`. Verified: -`felhom-agent --version` → **0.41.0**, service **active**, logs clean (local-api leaf ready, both 9201 -drives re-bound under the shared parent, hub desired-state updated). - -**B. Demo remediation (9201 was provisioned pre-fix → `onboot:0`).** `pct config 9201` before → `onboot: 0`; -ran `pct set 9201 -onboot 1` (non-destructive, config-only); after → **`onboot: 1`** ✓. - -**C. Back-half fix proper — LIVE-VALIDATED (supervised capstone re-provision, 2026-06-24).** Destroyed -guest 9201 (`pct stop` + `pct destroy` — 3 owned volumes removed; bind dirs + physical-drive data -survive) and re-provisioned it through the **real provision chain** with agent v0.41.0: -`felhom-agent --selftest=provision -archive local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst --vmid 9201 -customer-id demo-felhom -hub-password -rootfs-grow 0 -datavol-grow 184 --sysdata-grow 42`. Front-half bring-up OK (1m11s), back-half OK. **Assertion: the freshly-provisioned -`pct config 9201` showed `onboot: 1` with NO manual `pct set`** — set by the back-half fix. Full 3-volume -layout faithfully reproduced (rootfs 32G + mp0 docker 200G + mp1 /mnt/sys_drive 50G). Agent daemon restart -then reasserted both enrolled drives (felhom-flash, felhom-usb) at stable paths under -`/mnt/felhom-drives` (F9), skipped the ejected drive; controller + base infra came up healthy. - -**Optional — host-reboot scenario LIVE-VALIDATED (the exact Phase-4.1 failure, now passing).** Rebooted -felhom-pve (uptime went 16:37 → 1 min, confirming a real reboot). **`pct status 9201` = running with NO -manual `pct start`** (auto-started via `onboot:1`). The other guests (9001, 9999; `onboot:0`) correctly -stayed **stopped** — proving the per-guest `onboot:1` property drives the auto-start, not a blanket -start-all. Post-reboot recovery confirmed: the agent (fresh PID) re-bound both drives at stable paths, -`/mnt/sys_drive` + `/var/lib/docker` are separate mountpoints, and controller + traefik + cloudflared + -filebrowser all came up healthy. - -**Demo state after the capstone:** controller redeployed to **v0.82.0** (the fresh provision came up on -the golden-baked v0.77.0; pin bumped + `bootstrap` re-run). Standard post-re-provision residue (as every -prior capstone): customer apps are **not** redeployed and the controller's storage paths need a one-time -**UI re-enroll** of the two drives (their host-side data on felhom-usb is intact; the SSD mp0/mp1 app -data was destroyed with the volumes). The F3 fix itself is fully validated. +No secrets in any committed file; the registry token used for the live pre-pull stayed on the host +(stdin login, logged out).