docs: v0.42.0 controller-swap — README endpoint + REPORT (live-validated)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7

README.md

@@ -141,6 +141,23 @@ See the table in [`internal/proxmox/doc.go`](internal/proxmox/doc.go). Summary:
 lifecycle **including restore** is API-token-covered; OS-root is confined to golden-image
 `keyctl` create, host mounts, and SMART/sensors (phase3 §B3).
 
+### Controller swap (agentic controller update, Phase 1 — v0.42.0)
+
+The local API (`internal/localapi/`) owns the in-guest controller image **swap** — the new-architecture
+replacement for the controller's dead in-container `docker compose` self-update. The in-guest controller
+pre-pulls the target image then calls the agent:
+
+- `POST /controller/swap {image}` (`withGuest`-scoped) → **202**, then async: record previous
+  (`/var/lib/felhom-agent/controller-swap-<vmid>.json`) → confirm the target is present in the guest →
+  write `/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service` → poll
+  the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the
+  guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
+- `GET /controller/swap/status` → `{state, current, previous, target, error}`.
+
+The agent is **external** to the controller container, so it survives the controller being killed
+mid-swap (which the controller cannot do to itself). `GuestBinder.GuestExec` is the single `pct exec`
+seam. Exercise directly with `--selftest=controller-swap -vmid <id> -image <ref>`.
+
 ### TLS trust
 
 The host serves a self-signed cert. Verification is **not** blanket-disabled. Pick one in

REPORT.md

@@ -1,77 +1,69 @@
-# REPORT — v0.41.0: provisioned customer guests auto-start after a host reboot (`onboot:1`, F3)
+# REPORT — v0.42.0: agentic controller update — in-guest image swap + rollback (Phase 1)
 
-**Repo:** `felhom-agent` · **Version:** `v0.40.0` → **`v0.41.0`** · **Date:** 2026-06-24
-**Baseline:** `main` @ `db95d51` (version var `0.40.0`), trunk-based, no branches.
-**Class:** risky/supervised — touches the provisioning chain (back-half).
+**Repo:** `felhom-agent` · **Version:** `v0.41.0` → **`v0.42.0`** · **Date:** 2026-06-26
 
-## Finding (F3, from `TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md`)
+## What & why
+The controller's old self-update (`docker compose up -d` on an in-container compose path) is dead in the
+LXC architecture. Phase 1 moves the **container swap** to the host agent: the in-guest controller pulls
+the target image, then the agent — external to the controller container, so it survives the controller
+being killed mid-swap — rewrites `/etc/felhom-controller-image`, restarts the golden's
+`felhom-controller-bootstrap.service`, verifies health, and **rolls back** if the new controller doesn't
+come up. The guest is never left without a controller.
 
-The golden bakes `--onboot 0` (`configs/build-golden.sh:63`, template safety) and the provision
-back-half never overrode it → **every provisioned customer guest was `onboot:0`**. After a host
-reboot/power-cut the customer's whole home-server (controller + all apps) stays **down until a manual
-`pct start`**. Confirmed live in the campaign (Phase 4.1: host rebooted → `pct status 9201` = stopped).
+## Baseline
+felhom-agent `4725396` v0.41.0 → v0.42.0. Cross-repo: felhom-controller v0.85.0 (the pull + delegate half).
 
-## The fix
+## Changes
+- **`internal/localapi/controllerswap.go` (new)** — `ControllerSwapper` (testable, over a `GuestExecutor`
+  seam) + handlers:
+  - `POST /controller/swap {image}` (`withGuest`) → **202** `{status, previous_image, target_image}`, then
+    async: record previous to `/var/lib/felhom-agent/controller-swap-<vmid>.json` (crash-safety) → confirm
+    the target image is present in the guest (else abort, **no swap**) → write the image file → restart the
+    bootstrap unit → poll the new controller to healthy (`docker inspect`, ≤90s) → **roll back** + restart
+    if not. Strict image-ref gate (`gitea.dooplex.hu/admin/felhom-controller:<semver>`); single-flight per
+    guest (409).
+  - `GET /controller/swap/status` → `{state, current, previous, target, error}`.
+- **`internal/localapi/guestbind.go`** — `GuestBinder.GuestExec` (the single `pct exec` seam).
+- **`internal/localapi/server.go`** — `ControllerSwap` option + route wiring + single-flight map.
+- **`cmd/felhom-agent/main.go`** — `ControllerSwap: guestBinder`; `--selftest=controller-swap -vmid -image`
+  (+ the allowlist fix — see below); version → 0.42.0.
 
-`internal/provision/backhalf.go`, `BackHalf.Provision`: added a fatal step right after the config-mount
-attach (and before the guest-hook install):
+## Tests (`internal/localapi/controllerswap_test.go`) — all green
+happy swap; **rollback-on-unhealthy** (image reverted to previous + restart called ≥2×); image-absent →
+no swap (no file write); running-with-no-healthcheck → healthy; bad image → 400; single-flight → 409.
+- **Companion red-proof:** disabling the verify→rollback left the guest on the bad image and **failed**
+  `TestControllerSwap_RollbackOnUnhealthy` (`image file = …:0.84.0 after rollback, want previous …:0.77.0`)
+  → reverted → green. The rollback assertion is load-bearing.
+- localapi package: 70 → **71** tests. `go build ./... && go vet ./... && go test ./...` — all `ok`.
 
-```go
-if err := b.run(ctx, "pct", "set", strconv.Itoa(in.VMID), "-onboot", "1"); err != nil {
-    return Result{}, fmt.Errorf("provision: set onboot: %w", err)
-}
-```
+## Live test on demo 9201 (operator-authorized; agent v0.42.0 deployed to felhom-pve)
+- **Bug found live + fixed:** `--selftest=controller-swap` printed flag-usage — `controller-swap` was
+  missing from `selftestFlag.Set`'s allowlist. Fixed (commit `b17d07b`), rebuilt, redeployed.
+- **Swap 0.77→0.84** (`--selftest=controller-swap`, after pre-pulling 0.84): state=**done**, new controller
+  `Up (healthy)`, `/etc/felhom-controller-image` = 0.84.0. (~9s to healthy.)
+- **Rollback** (swap to a present-but-broken `:99.99.99` built from a controller base with `ENTRYPOINT
+  /bin/false`): 90s verify timeout → **rolled back to 0.84.0 healthy**, state=failed, current=0.84.0 — the
+  guest was never left without a controller.
+- **E2E (real UI button)** via claude-in-chrome on `felhom.demo-felhom.eu/settings`: with the demo on 0.85.0
+  (new code) and 0.85.1 pushed as latest, clicked **"Frissítés telepítése"** → the controller pulled 0.85.1
+  in-guest and called the agent → agent log: swap → "new controller healthy" `0.85.1` → UI reloaded showing
+  **"Sikeres (0.85.0 → 0.85.1)"**. No "docker-compose.yml nem elérhető".
+- Hub reflects the version via the controller's report path: `0.84.0 → 0.85.0 → 0.85.1`.
 
-- **Locus = the back-half, not the golden.** `build-golden.sh` `--onboot 0` is **unchanged** — a template
-  must not auto-start; `onboot` is a per-guest property the back-half is the right place to set.
-- **Fatal**, like the config-mount attach (a guest that won't auto-recover is a provisioning defect).
-- **No `startup`/boot-order/delay** (operator decision): the v0.75 mountpoint-gate already covers the
-  drive-bind race at boot (Phase 4.4). The agent never auto-`pct start`s anything — this is config only.
+## Observations (out of scope — not acted on)
+1. **Agent host-report `controller_version` is hardcoded `""`** (`internal/hub/collect.go:180`, "slice 8
+   fills" — never done). So the hub's `guests.controller_version` stays empty; the controller's own report
+   path carries the real version (verified `0.85.1`). Pre-existing, untouched here.
+2. **Agent-restart-mid-swap auto-resume:** the swap records previous to a state file (so a manual/operator
+   rollback is possible), but the agent does not yet auto-reconcile a `swapping` state file on its own
+   restart. The common failure (controller killed mid-swap) is fully handled; agent-crash-mid-swap leaves a
+   recoverable record. A startup reconcile is a candidate follow-up.
+3. **Golden freshness:** the golden bakes controller 0.77.0; a golden rebuild at current is wanted so new
+   provisions start current (independent of this feature).
 
-## Test + red-proof
+## Commits
+- `6f14b66` — v0.42.0 swap + rollback + tests.
+- `b17d07b` — fix: register `controller-swap` in the `--selftest` allowlist (found live).
 
-`internal/provision/backhalf_test.go`:
-- Added `TestProvision_SetsOnbootOne` — asserts the exact `pct set 8200 -onboot 1` invocation was
-  recorded by the fake runner (via a new `recRunner.hasExact` helper, since several `pct` calls are now
-  recorded and `find` only returns the first).
-- **Red-proof (run/confirm/revert):** removed the `b.run(... -onboot 1)` call → `TestProvision_SetsOnbootOne`
-  **FAILED** (`expected 'pct set 8200 -onboot 1' to be issued`) → restored the call → **green**.
-- Existing back-half tests stay green (chown-failure-stops still holds: onboot is after the config-mount
-  attach, so a chown failure never reaches it).
-- Green gate: `go build ./...` ✓ · `go vet ./...` ✓ · `go test ./...` ✓ (all packages ok).
-
-## Deploy + remediate + verify
-
-**A. Built + deployed agent v0.41.0.** Built on 192.168.0.180 (go1.26, `-ldflags -X main.version=0.41.0`,
-source @ `166a1c8`) → scp 180→local→felhom-pve → backed up the prior binary to
-`/usr/local/bin/felhom-agent.bak-0.40.0` → `install -m0755` → `systemctl restart felhom-agent`. Verified:
-`felhom-agent --version` → **0.41.0**, service **active**, logs clean (local-api leaf ready, both 9201
-drives re-bound under the shared parent, hub desired-state updated).
-
-**B. Demo remediation (9201 was provisioned pre-fix → `onboot:0`).** `pct config 9201` before → `onboot: 0`;
-ran `pct set 9201 -onboot 1` (non-destructive, config-only); after → **`onboot: 1`** ✓.
-
-**C. Back-half fix proper — LIVE-VALIDATED (supervised capstone re-provision, 2026-06-24).** Destroyed
-guest 9201 (`pct stop` + `pct destroy` — 3 owned volumes removed; bind dirs + physical-drive data
-survive) and re-provisioned it through the **real provision chain** with agent v0.41.0:
-`felhom-agent --selftest=provision -archive local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst
--vmid 9201 -customer-id demo-felhom -hub-password <out-of-band> -rootfs-grow 0 -datavol-grow 184
--sysdata-grow 42`. Front-half bring-up OK (1m11s), back-half OK. **Assertion: the freshly-provisioned
-`pct config 9201` showed `onboot: 1` with NO manual `pct set`** — set by the back-half fix. Full 3-volume
-layout faithfully reproduced (rootfs 32G + mp0 docker 200G + mp1 /mnt/sys_drive 50G). Agent daemon restart
-then reasserted both enrolled drives (felhom-flash, felhom-usb) at stable paths under
-`/mnt/felhom-drives` (F9), skipped the ejected drive; controller + base infra came up healthy.
-
-**Optional — host-reboot scenario LIVE-VALIDATED (the exact Phase-4.1 failure, now passing).** Rebooted
-felhom-pve (uptime went 16:37 → 1 min, confirming a real reboot). **`pct status 9201` = running with NO
-manual `pct start`** (auto-started via `onboot:1`). The other guests (9001, 9999; `onboot:0`) correctly
-stayed **stopped** — proving the per-guest `onboot:1` property drives the auto-start, not a blanket
-start-all. Post-reboot recovery confirmed: the agent (fresh PID) re-bound both drives at stable paths,
-`/mnt/sys_drive` + `/var/lib/docker` are separate mountpoints, and controller + traefik + cloudflared +
-filebrowser all came up healthy.
-
-**Demo state after the capstone:** controller redeployed to **v0.82.0** (the fresh provision came up on
-the golden-baked v0.77.0; pin bumped + `bootstrap` re-run). Standard post-re-provision residue (as every
-prior capstone): customer apps are **not** redeployed and the controller's storage paths need a one-time
-**UI re-enroll** of the two drives (their host-side data on felhom-usb is intact; the SSD mp0/mp1 app
-data was destroyed with the volumes). The F3 fix itself is fully validated.
+No secrets in any committed file; the registry token used for the live pre-pull stayed on the host
+(stdin login, logged out).