docs: v0.90.0 REPORT + CONTEXT + REUSE (guest RAM resize R-24 + fast-tick R-28; Phase-0 proof; deploy verified both hosts; floor = operator GO)

This commit is contained in:
2026-07-17 19:25:54 +02:00
parent ac112c956e
commit 8208a80cc7
3 changed files with 111 additions and 78 deletions
+18
View File
@@ -5,6 +5,24 @@
## Current
- **v0.90.0** (2026-07-17) — **agent train: guest RAM resize (R-24) + fast-tick (R-28); LIVE on BOTH
demo hosts (felhom-pve + nested demo-vm-felhom-4846bc).** MinAgent coupling: felhom-controller
v0.143.0 gates its resize UI on this agent. (1) **R-24 guest RAM resize (controller-direct)**
self-scoped `GET`/`POST /guest/memory` (`internal/localapi/guestmemory.go`); the AGENT enforces
every bound fresh per request (min 2048 / max host_total2048 / shrink floor max(2048, usage+512)) and
applies via PVE `SetConfig`**live cgroup apply, no reboot** (Phase-0 PROVEN on the nested box; the
break-glass access path + the proof are in `~/.claude/.../nested-vm-access-breakglass.md`). Verify-
after-apply re-reads maxmem before claiming success. New narrow `MemoryOps` seam (GuestAPI untouched);
memory only. (2) **R-28 fast-tick** (`internal/fasttick/`) — while any desired-state item is unapplied
(esp. the pre-tunnel WG-registration window a hub poke can't reach) pulse the shared out-of-band
trigger every 30 s, self-disarm on convergence; four cached sources (desired-gen==0, reconcile
PlannedPending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational). Seams:
`reconcile.Engine.LastResult()` + `wgtunnel.Manager.TunnelConvergence()` (cached — no per-tick exec).
(3) **Guests-0/0** REFUTED live: the 0/0 was the pre-provision window (guest not yet created), not a
pool-membership bug; the fast-tick shortens that window. **OPEN (operator GO):** publish 0.90.0 +
hub Day-0 manifest vouch + MinAgent-floor raise to 0.90.0 (password-gated UI; the safety gate — both
agents on 0.90.0 — is satisfied and the coupling is proven live via the version header). See REPORT.md.
- **v0.89.0** (2026-07-16) — **agent train: three bundled agent-plane items; built + published to
Gitea (sha256 `3969fd91…`); paired with hub 0.59.0 (LIVE).** (1) **pbsdr self-grant (R-22)**
closes the F4 self-deadlock: a 403 on the token-auth `StorageEntry` pre-check now self-grants via
+90 -78
View File
@@ -1,86 +1,98 @@
# REPORT — felhom-agent v0.89.0 (agent train: R-22 self-grant + escrow live-reload + poke listener)
# REPORT — felhom-agent v0.90.0 (agent train: guest RAM resize R-24 + fast-tick R-28)
**Date:** 2026-07-16 · **Baseline:** v0.88.0 (`c040c18`) → **v0.89.0** (`a659e5d`, pushed)
**Green gate:** `go build ./... && go vet ./... && go test ./...` — all pass (full suite, local go1.26).
**Design contract for item 3:** `felhom.eu/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`.
**Date:** 2026-07-17 · **Baseline:** v0.89.0 (`9127f54`) → **v0.90.0** (`ac112c9`, pushed) · **MinAgent coupling:** felhom-controller v0.143.0 gates its guest-memory-resize UI on this agent.
## Items + landmarks as found at source
## 1. What shipped
### Item 1 — pbsdr self-grant (R-22, closes F4 from tests/VALIDATION-n100-baremetal)
- **Where the pre-check lives:** `internal/pbsdr/manager.go` `Apply``m.px.StorageEntry(ctx, id)`
(GET `/storage/<id>`, token-auth). On a non-default storage id the agent token has no ACL there
yet → **403**; pre-fix the generic `if err != nil { … return }` aborted BEFORE the root-run wrapper
`grant` (only reached via `adopt`/create) that creates that very ACL — a permanent self-deadlock.
- **Fix:** on an `errors.As(err, *proxmox.APIError)` with `IsForbidden()` ONLY, run
`felhom-pbs-apply grant <id>` now (root, no secret, no pre-existing entry — `pveum acl modify` on a
path is unconditional; verified in `configs/felhom-pbs-apply` `grant)`), re-read once, then flow the
normal adoption/create path. Every other error stays transient. Grant-OK-but-still-403 →
`verify_failed` loudly. The pre-check is KEPT (the spec's "don't drop it" — it short-circuits the
happy path once the ACL exists). A benign extra `grant` on the adopt path is idempotent.
- **Red-proof (run-fail-revert):** `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`. Pre-fix
(forced `if true` abort): `self-grant never ran … calls=[]`**FAIL**. Fixed: self-grant runs,
converges (adopted), no secret consumed, 2 StorageEntry calls (403 + post-grant re-read) → **PASS**.
Two ruled features on one train (Viktor's one-train/one-floor ruling), plus a diagnosis passenger.
### Item 2 — escrow config live-reload
- **Two ends:** the pbsdr bridge seeds `escrow.pbs_storage_id` into agent.json on convergence
(`manager.go` `finishConverged``seedEscrowStorageID`); the reader is
`internal/localapi/escrow_ceremony.go` `handleEscrowPreflight`, which read a **daemon-start
snapshot** (`EscrowCeremonyConfig.PBSStorageID`) → stayed red until a service restart.
- **Reload strategy chosen (stated per source shape):** *the preflight reads current disk state*, via
a new late-bound `CurrentPBSStorageID func() string` (mirrors the existing `DRConfigured func() bool`
seam). `cmd/felhom-agent/main.go` wires it to `config.Load(cfg.SourcePath).Escrow.PBSStorageID`
**exactly what the ceremony subprocess itself loads** — so the row reflects the true state the bare
`--selftest=escrow-create` one-liner will see. Falls back to the boot snapshot on read error /
all-env config. Preferred over an in-memory push (no cross-component coupling, single source of
truth = disk).
- **Red-proof:** `TestEscrowPreflight_PBSStorageIDLiveReload` — seed after boot, same process; pre-fix
(static field) stays red → **FAIL**; fixed flips green + reports the seeded id → **PASS**.
- **R-24 guest RAM resize (controller-direct)** — new self-scoped local-API surface `GET`/`POST /guest/memory`
(`internal/localapi/guestmemory.go`). The agent is the security boundary: bounds recomputed FRESH per
request from a live read; applies via PVE `SetConfig` (live cgroup apply, no reboot); verify-after-apply.
- **R-28 fast-tick-until-first-convergence**`internal/fasttick/`, wired over four cached sources; pulses
the shared out-of-band trigger every 30 s while unapplied, self-disarms on convergence. Seams added:
`reconcile.Engine.LastResult()`, `wgtunnel.Manager.TunnelConvergence()`.
- **Guests-0/0** — diagnosed live, hypothesis REFUTED (no code change).
### Item 3 — agent-plane poke listener (Direction-2a)
- **Trigger fired:** the hub control loop's out-of-band report trigger (`internal/hub/loop.go`
`SetTrigger`), the SAME cap-1 non-blocking channel the storage watchdog uses — fan-in, coalescing.
An immediate collect→report cycle carries the advanced envelope generation → the desired-state
syncer fetches → pbsdr/wg consumers apply.
- **New `internal/poke`**`Listener` binds a contentless UDP socket EXCLUSIVELY to the box's WG /32
(`wgtunnel.LoadAssignedAddr`, new helper reading `registered.json`; never 0.0.0.0, never LAN) on
the fixed **port 51822**; payload ignored; leading-edge debounce (`DebounceWindow`) coalesces a
burst into ≤1 tick; enabled whenever `wg_tunnel.enabled`. Wired in `main.go` (goroutine + errc 7→8).
- **Red-proofs (run-fail-revert):** `TestBindConfinement` (wildcard bind → bound to `::` /
`IsUnspecified`**FAIL**; fixed → the specific WG addr, datagram fires the trigger) and
`TestDebounceCoalescesBurst` (guard removed → 10 fires for 10 pokes → **FAIL**; fixed → 1, then
fires again after the window). Port registered in REUSE.md as a shared cross-repo constant.
## 2. Phase-0 probe (the gate — PASSED)
## Deviations from the spec's landmark guesses (source wins)
- The 403 pre-check is `StorageEntry` (GET `/storage/<id>`) exactly as ROADMAP R-22 states — no
deviation; `StorageActive` would also 403 but the self-grant at the top fixes both.
- (Hub side, cross-repo) the MinAgent-floor save does not itself bump per-host desired generation —
noted in the hub REPORT.
Prerequisite: prove `PUT /nodes/{node}/lxc/{vmid}/config memory=X` hot-applies under the AGENT token. Run on
the nested demo box `demo-vm-felhom-4846bc` (hostname `felhom-appliance`, guest LXC 9201) — reached via the
hub-vaulted G1 break-glass root credential (the box's root PW was unknown; retrieved operator-side from the
hub SQLite `host_recovery` table, then shredded). Transcript:
## Live validation (Phase 5 — ALL THREE LEGS PROVEN LIVE on the demo, 2026-07-17)
- **Built + published** agent 0.89.0 to Gitea (sha256 `3969fd91…2609fb1a`, round-trip verified).
- **Scenario 4 — floor-driven A/B train (0.88→0.89), PROVEN LIVE.** Operator vouched 0.89.0 + raised
MinAgent to 0.89.0, then signed+enqueued the `agent_update` op (the self-update trigger — a per-box
operator-signed op; the vouch+floor alone does NOT queue it; CC cannot sign, per DRILL-GL6). The box
fetched it on its next report and self-updated: `gate decision class=agent_update allowed=true
reason=signed key_id=felhom-op-1` → download (sha `3969fd91…` verified) → `felhom-selfupdate-guarded
apply` (root A/B flip) → new daemon `version=0.89.0` → 60 s dwell → `selfupdate: update committed`
(no rollback). Poke listener bound clean: `poke: listening … addr=10.77.0.2:51822`; 63/63 caps.
- **Scenario 1 — R-22 self-heal, PROVEN LIVE.** Rollback re-grant recorded FIRST (`pveum acl modify
/storage/felhom-offsite --users felhom-agent@pve --roles FelhomAgentStore` + the `--tokens
'felhom-agent@pve!agent'` twin — delete syntax is `pveum acl delete`). **Faithful F4 reproduction:**
moved the pbsdr marker aside (the reinstall's fresh-state — revoke-alone is a no-op because the
`applied` marker short-circuits Apply before any storage read; an important reproduction finding) +
`pveum acl delete` both felhom-offsite ACLs. Next reconcile tick: `08:24:16 pbsdr: pre-check 403 …
self-granting via the root wrapper … (R-22) storage_id=felhom-offsite` → `08:24:19 pbsdr: converged
state=adopted` — **~3 s, no operator action**. Verified: ACLs self-restored (user+token), marker
rewritten (adopted), `pvesm status felhom-offsite` = **active**. Pre-fix = permanent 403→abort loop.
- **Scenario 3 — poke → tick latency, MEASURED LIVE.** ep0 send `06:28:55.765` UTC → box
`08:28:55.796` `poke received → triggering an immediate desired-state cycle from=10.77.0.1` → same
instant `out-of-band report triggered (watchdog/poke)`. **~31 ms** ep0→box over the tunnel (spike
~29 ms), sub-ms to the report cycle; WG-confined (from `10.77.0.1`, to the `10.77.0.2`-bound socket).
The hub-save→poke fire is separately live-proven (hub REPORT: real manifest save → `sync-poke
delivered to 10.77.0.2`); full save→tick ≈ ~0.45 s (SSH-dominated), well under the ≤23 s target.
```
node=felhom-appliance vmid=9201 token_user=felhom-agent@pve
BEFORE config.memory=2890 MB | status.maxmem=3030384640 B (2890 MB) | status=running | host-uptime=8226s
BEFORE guest MemTotal: 2959360 kB | guest /proc/uptime=8227.76
--- PUT memory=3146 (agent token) --- PUT data field: None (empty == synchronous apply, no UPID)
AFTER config.memory=3146 MB | status.maxmem=3298820096 B (3146 MB) | status=running | host-uptime=8229s
AFTER guest MemTotal: 3221504 kB | guest /proc/uptime=8231.54
VERDICT maxmem_grew=True guest_running=True guest_not_rebooted=True
REVERT memory=2890 → maxmem 3030384640 B (2890 MB), running; guest MemTotal 2959360 kB
```
## R-13
Item 3 is the FIRST concrete slice of the OOB/mutual-repair arc (the listener + trigger only). R-13
stays open; nothing beyond the poke channel was built.
Proven: live cgroup apply (maxmem moves, guest running), **no reboot** (guest /proc/uptime keeps climbing),
synchronous (no UPID), agent token has the privilege (no 403), and **/proc/meminfo ripples via lxcfs** — the
mechanism that makes the controller's deploy-page memory math follow for free. The gate PASSED → the resize
leg was implemented.
## 3. Files created / modified
- `internal/localapi/guestmemory.go` (new) + `guestmemory_test.go` (new); `internal/localapi/server.go`
(Options.Memory + fields + routes); `cmd/felhom-agent/main.go` (wire Memory=px; construct + launch fasttick).
- `internal/fasttick/fasttick.go` + `fasttick_test.go` (new).
- `internal/reconcile/engine.go` (`LastResult`/`recordResult`) + `lastresult_test.go` (new).
- `internal/wgtunnel/manager.go` (`TunnelConvergence` + cached snapshot refresh in `Apply`).
- `CHANGELOG.md`, `REPORT.md` (this), `CONTEXT.md`, `REUSE.md`.
## 4. Tests + red-proofs (all restored green)
- localapi memory: GET fields, grow, shrink-above-floor, the three refusals (each asserting **SetConfig
count == 0**), cross-guest 403, fresh-bounds-per-request, verify-not-reflected 502, nil-config 503.
- fasttick: pulse-while-unconverged, silent-when-converged, the ruled disarm-on-convergence, full-channel
non-blocking drop, first-reason. reconcile: `LastResult` effect + pre-first-run ok=false.
- **Red-proofs (run-fail-restore):** (i) floor guard removed → below_usage_floor 412 test fails; (ii) max
guard removed → above_max test fails; (iii) fast-tick ignores convergence (always pulse) → the silent +
disarm tests fail. All confirmed red, then restored. Full suite: `go build ./... && go vet ./... &&
go test ./...` — all 28 packages pass.
## 5. Guests-0/0 diagnosis — hypothesis REFUTED, no fix
Live read-only checks on the nested box: pool `felhom` `members: [9201]`; the agent token's ListLXC sees
`[9201]` (VM.Audit comes from the `/pool/felhom` grant — so pool membership IS the visibility mechanism);
hub reports 1/1. The agent log shows the observed 0/0 was the **pre-provision window**: `16:11 stale-lock …
pool=felhom listed=0` + `desired … guests=0`, then `16:16 lanresolver … vmid=9201` (guest created). No guest
existed yet → empty list, no error (the silent shape) — correct, not a bug. The existing `PoolAddVMID`
re-assert (`bringup.go:498`) already covers the known restore-over-existing edge (campaign-2 R2). **Item 2's
fast-tick is precisely the mitigation** — it shortens the pre-provision reporting window from ≤15 min to ~30 s.
## 6. Deploy verification (BOTH hosts)
Built on 180 (`go build -ldflags -X main.version=0.90.0`), deployed to both Proxmox hosts (backup + install +
restart):
- **felhom-pve** (the N100): `felhom-agent 0.90.0` active; `capabilities self-check ok=63 total=63 degraded=0`;
**`fast-tick armed: 30s out-of-band cadence while desired-state is unapplied`**; poke listener + local-api up.
- **demo-vm-felhom-4846bc** (nested, 192.168.0.135): `felhom-agent 0.90.0` active; `capabilities self-check
ok=63 degraded=0`; `fast-tick armed: 30s …`; local-api on 192.168.0.135:8443. (Converged box → fast-tick
quiet after the startup line, as designed.)
## 7. NOT yet live-validated / operator GO
- **Fast-tick real-onboarding proof:** the fast-tick's value shows on a REAL fresh box's pre-tunnel window
(the rehearsal S5 bind sequence should now be visibly instant). Not manufacturable non-destructively here;
both agents show the armed startup line + the unit tests prove the pulse/disarm.
- **Floor LAST (operator GO):** both demo agents verifiably run 0.90.0 (the safety gate) and the controller
coupling is PROVEN LIVE via the X-Felhom-Agent-Version header (see the controller REPORT's live refusals).
Publishing 0.90.0 to Gitea + the hub Day-0 manifest **vouch + MinAgent-floor raise to 0.90.0** are set via
the operator-password-gated Day-0 manifest UI (CC cannot log in) — the operator (Viktor) GO step, matching
the v0.89 train's "publish/vouch = operator follow-up".
## 8. Observations
- The nested demo host is tiny (4 GB): its resize bounds are degenerate (`max_mb 1866 < min_mb 2048 <
current 2890`) — no resize *succeeds* through the bounded endpoint there (all refused correctly). The
successful apply is Phase-0-proven at the agent layer; the controller live-validation exercised the
refusals (above_max, below_min) end-to-end.
- REUSE.md gained the fasttick Source pattern + the localapi MemoryOps seam.
+3
View File
@@ -74,6 +74,7 @@
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC``Client.Pool`), fail-safe skip on pool-read failure |
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
| `MemoryOps` + `Server.readMemoryBounds` | internal/localapi/guestmemory.go | `readMemoryBounds(ctx, vmid) (memoryBounds, err)` | guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory | NEW narrow seam (never extend `GuestAPI` — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config `memory`=MB, status/node=bytes); verify maxmem==target after `SetConfig` before claiming success; SetConfig NEVER called on a refusal path |
### Proxmox client / hub / PBS / provisioning
@@ -95,6 +96,8 @@
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
| `wgtunnel.LoadAssignedAddr` | internal/wgtunnel/manager.go | `LoadAssignedAddr(stateDir) (netip.Addr, bool)` | the box's own WG /32 without a Manager | Reads `registered.json`; ok=false until registered; strips the /32 → bare addr (the poke bind target) |
| `fasttick.Loop` + `fasttick.SourceFunc` | internal/fasttick/fasttick.go | `New(out chan<- struct{}, interval, logger, sources...)`; `Source.Unconverged() (bool, reason)` | agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence | STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — `desiredProvider.Generation()`, `reconcile.Engine.LastResult()` (PlannedPending>0), `pbsdrLoop.PBSDRStatus().State=="waiting_secret"` ONLY, `wgtunnel.Manager.TunnelConvergence()`. The LOUD pbsdr states + destructive `pending_signature` are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch `MinPollSeconds`/`clampInterval`/the ticker |
| `wgtunnel.Manager.TunnelConvergence` / `reconcile.Engine.LastResult` | internal/wgtunnel/manager.go, internal/reconcile/engine.go | `TunnelConvergence() (desired, operational bool)`; `LastResult() (Result, bool)` | cached convergence snapshots for the fast-tick | TunnelConvergence is refreshed at the END of every `Apply` (its own cadence) so the fast-tick never execs `wg`/`systemctl`; LastResult is mutex-recorded per `reconcileOnce`, ok=false until the first pass |
## 2. Canonical patterns (copy structure from THE named file)