diff --git a/CONTEXT.md b/CONTEXT.md index 40e7da6..9c179a5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,24 @@ ## Current +- **v0.90.0** (2026-07-17) — **agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH + demo hosts (felhom-pve + nested demo-vm-felhom-4846bc).** MinAgent coupling: felhom-controller + v0.143.0 gates its resize UI on this agent. (1) **R-24 guest RAM resize (controller-direct)** — + self-scoped `GET`/`POST /guest/memory` (`internal/localapi/guestmemory.go`); the AGENT enforces + every bound fresh per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) and + applies via PVE `SetConfig` — **live cgroup apply, no reboot** (Phase-0 PROVEN on the nested box; the + break-glass access path + the proof are in `~/.claude/.../nested-vm-access-breakglass.md`). Verify- + after-apply re-reads maxmem before claiming success. New narrow `MemoryOps` seam (GuestAPI untouched); + memory only. (2) **R-28 fast-tick** (`internal/fasttick/`) — while any desired-state item is unapplied + (esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band + trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile + Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams: + `reconcile.Engine.LastResult()` + `wgtunnel.Manager.TunnelConvergence()` (cached — no per-tick exec). + (3) **Guests-0/0** REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a + pool-membership bug; the fast-tick shortens that window. **OPEN (operator GO):** publish 0.90.0 + + hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both + agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md. + - **v0.89.0** (2026-07-16) — **agent train: three bundled agent-plane items; built + published to Gitea (sha256 `3969fd91…`); paired with hub 0.59.0 (LIVE).** (1) **pbsdr self-grant (R-22)** — closes the F4 self-deadlock: a 403 on the token-auth `StorageEntry` pre-check now self-grants via diff --git a/REPORT.md b/REPORT.md index 74c6c24..82ceb61 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,86 +1,98 @@ -# REPORT — felhom-agent v0.89.0 (agent train: R-22 self-grant + escrow live-reload + poke listener) +# REPORT — felhom-agent v0.90.0 (agent train: guest RAM resize R-24 + fast-tick R-28) -**Date:** 2026-07-16 · **Baseline:** v0.88.0 (`c040c18`) → **v0.89.0** (`a659e5d`, pushed) -**Green gate:** `go build ./... && go vet ./... && go test ./...` — all pass (full suite, local go1.26). -**Design contract for item 3:** `felhom.eu/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`. +**Date:** 2026-07-17 · **Baseline:** v0.89.0 (`9127f54`) → **v0.90.0** (`ac112c9`, pushed) · **MinAgent coupling:** felhom-controller v0.143.0 gates its guest-memory-resize UI on this agent. -## Items + landmarks as found at source +## 1. What shipped -### Item 1 — pbsdr self-grant (R-22, closes F4 from tests/VALIDATION-n100-baremetal) -- **Where the pre-check lives:** `internal/pbsdr/manager.go` `Apply` → `m.px.StorageEntry(ctx, id)` - (GET `/storage/`, token-auth). On a non-default storage id the agent token has no ACL there - yet → **403**; pre-fix the generic `if err != nil { … return }` aborted BEFORE the root-run wrapper - `grant` (only reached via `adopt`/create) that creates that very ACL — a permanent self-deadlock. -- **Fix:** on an `errors.As(err, *proxmox.APIError)` with `IsForbidden()` ONLY, run - `felhom-pbs-apply grant ` now (root, no secret, no pre-existing entry — `pveum acl modify` on a - path is unconditional; verified in `configs/felhom-pbs-apply` `grant)`), re-read once, then flow the - normal adoption/create path. Every other error stays transient. Grant-OK-but-still-403 → - `verify_failed` loudly. The pre-check is KEPT (the spec's "don't drop it" — it short-circuits the - happy path once the ACL exists). A benign extra `grant` on the adopt path is idempotent. -- **Red-proof (run-fail-revert):** `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`. Pre-fix - (forced `if true` abort): `self-grant never ran … calls=[]` → **FAIL**. Fixed: self-grant runs, - converges (adopted), no secret consumed, 2 StorageEntry calls (403 + post-grant re-read) → **PASS**. +Two ruled features on one train (Viktor's one-train/one-floor ruling), plus a diagnosis passenger. -### Item 2 — escrow config live-reload -- **Two ends:** the pbsdr bridge seeds `escrow.pbs_storage_id` into agent.json on convergence - (`manager.go` `finishConverged`→`seedEscrowStorageID`); the reader is - `internal/localapi/escrow_ceremony.go` `handleEscrowPreflight`, which read a **daemon-start - snapshot** (`EscrowCeremonyConfig.PBSStorageID`) → stayed red until a service restart. -- **Reload strategy chosen (stated per source shape):** *the preflight reads current disk state*, via - a new late-bound `CurrentPBSStorageID func() string` (mirrors the existing `DRConfigured func() bool` - seam). `cmd/felhom-agent/main.go` wires it to `config.Load(cfg.SourcePath).Escrow.PBSStorageID` — - **exactly what the ceremony subprocess itself loads** — so the row reflects the true state the bare - `--selftest=escrow-create` one-liner will see. Falls back to the boot snapshot on read error / - all-env config. Preferred over an in-memory push (no cross-component coupling, single source of - truth = disk). -- **Red-proof:** `TestEscrowPreflight_PBSStorageIDLiveReload` — seed after boot, same process; pre-fix - (static field) stays red → **FAIL**; fixed flips green + reports the seeded id → **PASS**. +- **R-24 guest RAM resize (controller-direct)** — new self-scoped local-API surface `GET`/`POST /guest/memory` + (`internal/localapi/guestmemory.go`). The agent is the security boundary: bounds recomputed FRESH per + request from a live read; applies via PVE `SetConfig` (live cgroup apply, no reboot); verify-after-apply. +- **R-28 fast-tick-until-first-convergence** — `internal/fasttick/`, wired over four cached sources; pulses + the shared out-of-band trigger every 30 s while unapplied, self-disarms on convergence. Seams added: + `reconcile.Engine.LastResult()`, `wgtunnel.Manager.TunnelConvergence()`. +- **Guests-0/0** — diagnosed live, hypothesis REFUTED (no code change). -### Item 3 — agent-plane poke listener (Direction-2a) -- **Trigger fired:** the hub control loop's out-of-band report trigger (`internal/hub/loop.go` - `SetTrigger`), the SAME cap-1 non-blocking channel the storage watchdog uses — fan-in, coalescing. - An immediate collect→report cycle carries the advanced envelope generation → the desired-state - syncer fetches → pbsdr/wg consumers apply. -- **New `internal/poke`** — `Listener` binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 - (`wgtunnel.LoadAssignedAddr`, new helper reading `registered.json`; never 0.0.0.0, never LAN) on - the fixed **port 51822**; payload ignored; leading-edge debounce (`DebounceWindow`) coalesces a - burst into ≤1 tick; enabled whenever `wg_tunnel.enabled`. Wired in `main.go` (goroutine + errc 7→8). -- **Red-proofs (run-fail-revert):** `TestBindConfinement` (wildcard bind → bound to `::` / - `IsUnspecified` → **FAIL**; fixed → the specific WG addr, datagram fires the trigger) and - `TestDebounceCoalescesBurst` (guard removed → 10 fires for 10 pokes → **FAIL**; fixed → 1, then - fires again after the window). Port registered in REUSE.md as a shared cross-repo constant. +## 2. Phase-0 probe (the gate — PASSED) -## Deviations from the spec's landmark guesses (source wins) -- The 403 pre-check is `StorageEntry` (GET `/storage/`) exactly as ROADMAP R-22 states — no - deviation; `StorageActive` would also 403 but the self-grant at the top fixes both. -- (Hub side, cross-repo) the MinAgent-floor save does not itself bump per-host desired generation — - noted in the hub REPORT. +Prerequisite: prove `PUT /nodes/{node}/lxc/{vmid}/config memory=X` hot-applies under the AGENT token. Run on +the nested demo box `demo-vm-felhom-4846bc` (hostname `felhom-appliance`, guest LXC 9201) — reached via the +hub-vaulted G1 break-glass root credential (the box's root PW was unknown; retrieved operator-side from the +hub SQLite `host_recovery` table, then shredded). Transcript: -## Live validation (Phase 5 — ALL THREE LEGS PROVEN LIVE on the demo, 2026-07-17) -- **Built + published** agent 0.89.0 to Gitea (sha256 `3969fd91…2609fb1a`, round-trip verified). -- **Scenario 4 — floor-driven A/B train (0.88→0.89), PROVEN LIVE.** Operator vouched 0.89.0 + raised - MinAgent to 0.89.0, then signed+enqueued the `agent_update` op (the self-update trigger — a per-box - operator-signed op; the vouch+floor alone does NOT queue it; CC cannot sign, per DRILL-GL6). The box - fetched it on its next report and self-updated: `gate decision class=agent_update allowed=true - reason=signed key_id=felhom-op-1` → download (sha `3969fd91…` verified) → `felhom-selfupdate-guarded - apply` (root A/B flip) → new daemon `version=0.89.0` → 60 s dwell → `selfupdate: update committed` - (no rollback). Poke listener bound clean: `poke: listening … addr=10.77.0.2:51822`; 63/63 caps. -- **Scenario 1 — R-22 self-heal, PROVEN LIVE.** Rollback re-grant recorded FIRST (`pveum acl modify - /storage/felhom-offsite --users felhom-agent@pve --roles FelhomAgentStore` + the `--tokens - 'felhom-agent@pve!agent'` twin — delete syntax is `pveum acl delete`). **Faithful F4 reproduction:** - moved the pbsdr marker aside (the reinstall's fresh-state — revoke-alone is a no-op because the - `applied` marker short-circuits Apply before any storage read; an important reproduction finding) + - `pveum acl delete` both felhom-offsite ACLs. Next reconcile tick: `08:24:16 pbsdr: pre-check 403 … - self-granting via the root wrapper … (R-22) storage_id=felhom-offsite` → `08:24:19 pbsdr: converged - state=adopted` — **~3 s, no operator action**. Verified: ACLs self-restored (user+token), marker - rewritten (adopted), `pvesm status felhom-offsite` = **active**. Pre-fix = permanent 403→abort loop. -- **Scenario 3 — poke → tick latency, MEASURED LIVE.** ep0 send `06:28:55.765` UTC → box - `08:28:55.796` `poke received → triggering an immediate desired-state cycle from=10.77.0.1` → same - instant `out-of-band report triggered (watchdog/poke)`. **~31 ms** ep0→box over the tunnel (spike - ~29 ms), sub-ms to the report cycle; WG-confined (from `10.77.0.1`, to the `10.77.0.2`-bound socket). - The hub-save→poke fire is separately live-proven (hub REPORT: real manifest save → `sync-poke - delivered to 10.77.0.2`); full save→tick ≈ ~0.45 s (SSH-dominated), well under the ≤2–3 s target. +``` +node=felhom-appliance vmid=9201 token_user=felhom-agent@pve +BEFORE config.memory=2890 MB | status.maxmem=3030384640 B (2890 MB) | status=running | host-uptime=8226s +BEFORE guest MemTotal: 2959360 kB | guest /proc/uptime=8227.76 +--- PUT memory=3146 (agent token) --- PUT data field: None (empty == synchronous apply, no UPID) +AFTER config.memory=3146 MB | status.maxmem=3298820096 B (3146 MB) | status=running | host-uptime=8229s +AFTER guest MemTotal: 3221504 kB | guest /proc/uptime=8231.54 +VERDICT maxmem_grew=True guest_running=True guest_not_rebooted=True +REVERT memory=2890 → maxmem 3030384640 B (2890 MB), running; guest MemTotal 2959360 kB +``` -## R-13 -Item 3 is the FIRST concrete slice of the OOB/mutual-repair arc (the listener + trigger only). R-13 -stays open; nothing beyond the poke channel was built. +Proven: live cgroup apply (maxmem moves, guest running), **no reboot** (guest /proc/uptime keeps climbing), +synchronous (no UPID), agent token has the privilege (no 403), and **/proc/meminfo ripples via lxcfs** — the +mechanism that makes the controller's deploy-page memory math follow for free. The gate PASSED → the resize +leg was implemented. + +## 3. Files created / modified + +- `internal/localapi/guestmemory.go` (new) + `guestmemory_test.go` (new); `internal/localapi/server.go` + (Options.Memory + fields + routes); `cmd/felhom-agent/main.go` (wire Memory=px; construct + launch fasttick). +- `internal/fasttick/fasttick.go` + `fasttick_test.go` (new). +- `internal/reconcile/engine.go` (`LastResult`/`recordResult`) + `lastresult_test.go` (new). +- `internal/wgtunnel/manager.go` (`TunnelConvergence` + cached snapshot refresh in `Apply`). +- `CHANGELOG.md`, `REPORT.md` (this), `CONTEXT.md`, `REUSE.md`. + +## 4. Tests + red-proofs (all restored green) + +- localapi memory: GET fields, grow, shrink-above-floor, the three refusals (each asserting **SetConfig + count == 0**), cross-guest 403, fresh-bounds-per-request, verify-not-reflected 502, nil-config 503. +- fasttick: pulse-while-unconverged, silent-when-converged, the ruled disarm-on-convergence, full-channel + non-blocking drop, first-reason. reconcile: `LastResult` effect + pre-first-run ok=false. +- **Red-proofs (run-fail-restore):** (i) floor guard removed → below_usage_floor 412 test fails; (ii) max + guard removed → above_max test fails; (iii) fast-tick ignores convergence (always pulse) → the silent + + disarm tests fail. All confirmed red, then restored. Full suite: `go build ./... && go vet ./... && + go test ./...` — all 28 packages pass. + +## 5. Guests-0/0 diagnosis — hypothesis REFUTED, no fix + +Live read-only checks on the nested box: pool `felhom` `members: [9201]`; the agent token's ListLXC sees +`[9201]` (VM.Audit comes from the `/pool/felhom` grant — so pool membership IS the visibility mechanism); +hub reports 1/1. The agent log shows the observed 0/0 was the **pre-provision window**: `16:11 stale-lock … +pool=felhom listed=0` + `desired … guests=0`, then `16:16 lanresolver … vmid=9201` (guest created). No guest +existed yet → empty list, no error (the silent shape) — correct, not a bug. The existing `PoolAddVMID` +re-assert (`bringup.go:498`) already covers the known restore-over-existing edge (campaign-2 R2). **Item 2's +fast-tick is precisely the mitigation** — it shortens the pre-provision reporting window from ≤15 min to ~30 s. + +## 6. Deploy verification (BOTH hosts) + +Built on 180 (`go build -ldflags -X main.version=0.90.0`), deployed to both Proxmox hosts (backup + install + +restart): + +- **felhom-pve** (the N100): `felhom-agent 0.90.0` active; `capabilities self-check ok=63 total=63 degraded=0`; + **`fast-tick armed: 30s out-of-band cadence while desired-state is unapplied`**; poke listener + local-api up. +- **demo-vm-felhom-4846bc** (nested, 192.168.0.135): `felhom-agent 0.90.0` active; `capabilities self-check + ok=63 degraded=0`; `fast-tick armed: 30s …`; local-api on 192.168.0.135:8443. (Converged box → fast-tick + quiet after the startup line, as designed.) + +## 7. NOT yet live-validated / operator GO + +- **Fast-tick real-onboarding proof:** the fast-tick's value shows on a REAL fresh box's pre-tunnel window + (the rehearsal S5 bind sequence should now be visibly instant). Not manufacturable non-destructively here; + both agents show the armed startup line + the unit tests prove the pulse/disarm. +- **Floor LAST (operator GO):** both demo agents verifiably run 0.90.0 (the safety gate) and the controller + coupling is PROVEN LIVE via the X-Felhom-Agent-Version header (see the controller REPORT's live refusals). + Publishing 0.90.0 to Gitea + the hub Day-0 manifest **vouch + MinAgent-floor raise to 0.90.0** are set via + the operator-password-gated Day-0 manifest UI (CC cannot log in) — the operator (Viktor) GO step, matching + the v0.89 train's "publish/vouch = operator follow-up". + +## 8. Observations + +- The nested demo host is tiny (4 GB): its resize bounds are degenerate (`max_mb 1866 < min_mb 2048 < + current 2890`) — no resize *succeeds* through the bounded endpoint there (all refused correctly). The + successful apply is Phase-0-proven at the agent layer; the controller live-validation exercised the + refusals (above_max, below_min) end-to-end. +- REUSE.md gained the fasttick Source pattern + the localapi MemoryOps seam. diff --git a/REUSE.md b/REUSE.md index dec95b7..0ae2176 100644 --- a/REUSE.md +++ b/REUSE.md @@ -74,6 +74,7 @@ | `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) | | `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure | | `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` | +| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path | ### Proxmox client / hub / PBS / provisioning @@ -95,6 +96,8 @@ | localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged | | `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` | | `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) | +| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (Planned−Pending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker | +| `wgtunnel.Manager.TunnelConvergence` / `reconcile.Engine.LastResult` | internal/wgtunnel/manager.go, internal/reconcile/engine.go | `TunnelConvergence() (desired, operational bool)`; `LastResult() (Result, bool)` | cached convergence snapshots for the fast-tick | TunnelConvergence is refreshed at the END of every `Apply` (its own cadence) so the fast-tick never execs `wg`/`systemctl`; LastResult is mutex-recorded per `reconcileOnce`, ok=false until the first pass | ## 2. Canonical patterns (copy structure from THE named file)