docs: v0.90.0 REPORT + CONTEXT + REUSE (guest RAM resize R-24 + fast-tick R-28; Phase-0 proof; deploy verified both hosts; floor = operator GO)

This commit is contained in:
2026-07-17 19:25:54 +02:00
parent ac112c956e
commit 8208a80cc7
3 changed files with 111 additions and 78 deletions
+90 -78
View File
@@ -1,86 +1,98 @@
# REPORT — felhom-agent v0.89.0 (agent train: R-22 self-grant + escrow live-reload + poke listener)
# REPORT — felhom-agent v0.90.0 (agent train: guest RAM resize R-24 + fast-tick R-28)
**Date:** 2026-07-16 · **Baseline:** v0.88.0 (`c040c18`) → **v0.89.0** (`a659e5d`, pushed)
**Green gate:** `go build ./... && go vet ./... && go test ./...` — all pass (full suite, local go1.26).
**Design contract for item 3:** `felhom.eu/documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`.
**Date:** 2026-07-17 · **Baseline:** v0.89.0 (`9127f54`) → **v0.90.0** (`ac112c9`, pushed) · **MinAgent coupling:** felhom-controller v0.143.0 gates its guest-memory-resize UI on this agent.
## Items + landmarks as found at source
## 1. What shipped
### Item 1 — pbsdr self-grant (R-22, closes F4 from tests/VALIDATION-n100-baremetal)
- **Where the pre-check lives:** `internal/pbsdr/manager.go` `Apply``m.px.StorageEntry(ctx, id)`
(GET `/storage/<id>`, token-auth). On a non-default storage id the agent token has no ACL there
yet → **403**; pre-fix the generic `if err != nil { … return }` aborted BEFORE the root-run wrapper
`grant` (only reached via `adopt`/create) that creates that very ACL — a permanent self-deadlock.
- **Fix:** on an `errors.As(err, *proxmox.APIError)` with `IsForbidden()` ONLY, run
`felhom-pbs-apply grant <id>` now (root, no secret, no pre-existing entry — `pveum acl modify` on a
path is unconditional; verified in `configs/felhom-pbs-apply` `grant)`), re-read once, then flow the
normal adoption/create path. Every other error stays transient. Grant-OK-but-still-403 →
`verify_failed` loudly. The pre-check is KEPT (the spec's "don't drop it" — it short-circuits the
happy path once the ACL exists). A benign extra `grant` on the adopt path is idempotent.
- **Red-proof (run-fail-revert):** `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`. Pre-fix
(forced `if true` abort): `self-grant never ran … calls=[]`**FAIL**. Fixed: self-grant runs,
converges (adopted), no secret consumed, 2 StorageEntry calls (403 + post-grant re-read) → **PASS**.
Two ruled features on one train (Viktor's one-train/one-floor ruling), plus a diagnosis passenger.
### Item 2 — escrow config live-reload
- **Two ends:** the pbsdr bridge seeds `escrow.pbs_storage_id` into agent.json on convergence
(`manager.go` `finishConverged``seedEscrowStorageID`); the reader is
`internal/localapi/escrow_ceremony.go` `handleEscrowPreflight`, which read a **daemon-start
snapshot** (`EscrowCeremonyConfig.PBSStorageID`) → stayed red until a service restart.
- **Reload strategy chosen (stated per source shape):** *the preflight reads current disk state*, via
a new late-bound `CurrentPBSStorageID func() string` (mirrors the existing `DRConfigured func() bool`
seam). `cmd/felhom-agent/main.go` wires it to `config.Load(cfg.SourcePath).Escrow.PBSStorageID`
**exactly what the ceremony subprocess itself loads** — so the row reflects the true state the bare
`--selftest=escrow-create` one-liner will see. Falls back to the boot snapshot on read error /
all-env config. Preferred over an in-memory push (no cross-component coupling, single source of
truth = disk).
- **Red-proof:** `TestEscrowPreflight_PBSStorageIDLiveReload` — seed after boot, same process; pre-fix
(static field) stays red → **FAIL**; fixed flips green + reports the seeded id → **PASS**.
- **R-24 guest RAM resize (controller-direct)** — new self-scoped local-API surface `GET`/`POST /guest/memory`
(`internal/localapi/guestmemory.go`). The agent is the security boundary: bounds recomputed FRESH per
request from a live read; applies via PVE `SetConfig` (live cgroup apply, no reboot); verify-after-apply.
- **R-28 fast-tick-until-first-convergence** — `internal/fasttick/`, wired over four cached sources; pulses
the shared out-of-band trigger every 30 s while unapplied, self-disarms on convergence. Seams added:
`reconcile.Engine.LastResult()`, `wgtunnel.Manager.TunnelConvergence()`.
- **Guests-0/0** — diagnosed live, hypothesis REFUTED (no code change).
### Item 3 — agent-plane poke listener (Direction-2a)
- **Trigger fired:** the hub control loop's out-of-band report trigger (`internal/hub/loop.go`
`SetTrigger`), the SAME cap-1 non-blocking channel the storage watchdog uses — fan-in, coalescing.
An immediate collect→report cycle carries the advanced envelope generation → the desired-state
syncer fetches → pbsdr/wg consumers apply.
- **New `internal/poke`** — `Listener` binds a contentless UDP socket EXCLUSIVELY to the box's WG /32
(`wgtunnel.LoadAssignedAddr`, new helper reading `registered.json`; never 0.0.0.0, never LAN) on
the fixed **port 51822**; payload ignored; leading-edge debounce (`DebounceWindow`) coalesces a
burst into ≤1 tick; enabled whenever `wg_tunnel.enabled`. Wired in `main.go` (goroutine + errc 7→8).
- **Red-proofs (run-fail-revert):** `TestBindConfinement` (wildcard bind → bound to `::` /
`IsUnspecified`**FAIL**; fixed → the specific WG addr, datagram fires the trigger) and
`TestDebounceCoalescesBurst` (guard removed → 10 fires for 10 pokes → **FAIL**; fixed → 1, then
fires again after the window). Port registered in REUSE.md as a shared cross-repo constant.
## 2. Phase-0 probe (the gate — PASSED)
## Deviations from the spec's landmark guesses (source wins)
- The 403 pre-check is `StorageEntry` (GET `/storage/<id>`) exactly as ROADMAP R-22 states — no
deviation; `StorageActive` would also 403 but the self-grant at the top fixes both.
- (Hub side, cross-repo) the MinAgent-floor save does not itself bump per-host desired generation —
noted in the hub REPORT.
Prerequisite: prove `PUT /nodes/{node}/lxc/{vmid}/config memory=X` hot-applies under the AGENT token. Run on
the nested demo box `demo-vm-felhom-4846bc` (hostname `felhom-appliance`, guest LXC 9201) — reached via the
hub-vaulted G1 break-glass root credential (the box's root PW was unknown; retrieved operator-side from the
hub SQLite `host_recovery` table, then shredded). Transcript:
## Live validation (Phase 5 — ALL THREE LEGS PROVEN LIVE on the demo, 2026-07-17)
- **Built + published** agent 0.89.0 to Gitea (sha256 `3969fd91…2609fb1a`, round-trip verified).
- **Scenario 4 — floor-driven A/B train (0.88→0.89), PROVEN LIVE.** Operator vouched 0.89.0 + raised
MinAgent to 0.89.0, then signed+enqueued the `agent_update` op (the self-update trigger — a per-box
operator-signed op; the vouch+floor alone does NOT queue it; CC cannot sign, per DRILL-GL6). The box
fetched it on its next report and self-updated: `gate decision class=agent_update allowed=true
reason=signed key_id=felhom-op-1` → download (sha `3969fd91…` verified) → `felhom-selfupdate-guarded
apply` (root A/B flip) → new daemon `version=0.89.0` → 60 s dwell → `selfupdate: update committed`
(no rollback). Poke listener bound clean: `poke: listening … addr=10.77.0.2:51822`; 63/63 caps.
- **Scenario 1 — R-22 self-heal, PROVEN LIVE.** Rollback re-grant recorded FIRST (`pveum acl modify
/storage/felhom-offsite --users felhom-agent@pve --roles FelhomAgentStore` + the `--tokens
'felhom-agent@pve!agent'` twin — delete syntax is `pveum acl delete`). **Faithful F4 reproduction:**
moved the pbsdr marker aside (the reinstall's fresh-state — revoke-alone is a no-op because the
`applied` marker short-circuits Apply before any storage read; an important reproduction finding) +
`pveum acl delete` both felhom-offsite ACLs. Next reconcile tick: `08:24:16 pbsdr: pre-check 403 …
self-granting via the root wrapper … (R-22) storage_id=felhom-offsite` → `08:24:19 pbsdr: converged
state=adopted` — **~3 s, no operator action**. Verified: ACLs self-restored (user+token), marker
rewritten (adopted), `pvesm status felhom-offsite` = **active**. Pre-fix = permanent 403→abort loop.
- **Scenario 3 — poke → tick latency, MEASURED LIVE.** ep0 send `06:28:55.765` UTC → box
`08:28:55.796` `poke received → triggering an immediate desired-state cycle from=10.77.0.1` → same
instant `out-of-band report triggered (watchdog/poke)`. **~31 ms** ep0→box over the tunnel (spike
~29 ms), sub-ms to the report cycle; WG-confined (from `10.77.0.1`, to the `10.77.0.2`-bound socket).
The hub-save→poke fire is separately live-proven (hub REPORT: real manifest save → `sync-poke
delivered to 10.77.0.2`); full save→tick ≈ ~0.45 s (SSH-dominated), well under the ≤23 s target.
```
node=felhom-appliance vmid=9201 token_user=felhom-agent@pve
BEFORE config.memory=2890 MB | status.maxmem=3030384640 B (2890 MB) | status=running | host-uptime=8226s
BEFORE guest MemTotal: 2959360 kB | guest /proc/uptime=8227.76
--- PUT memory=3146 (agent token) --- PUT data field: None (empty == synchronous apply, no UPID)
AFTER config.memory=3146 MB | status.maxmem=3298820096 B (3146 MB) | status=running | host-uptime=8229s
AFTER guest MemTotal: 3221504 kB | guest /proc/uptime=8231.54
VERDICT maxmem_grew=True guest_running=True guest_not_rebooted=True
REVERT memory=2890 → maxmem 3030384640 B (2890 MB), running; guest MemTotal 2959360 kB
```
## R-13
Item 3 is the FIRST concrete slice of the OOB/mutual-repair arc (the listener + trigger only). R-13
stays open; nothing beyond the poke channel was built.
Proven: live cgroup apply (maxmem moves, guest running), **no reboot** (guest /proc/uptime keeps climbing),
synchronous (no UPID), agent token has the privilege (no 403), and **/proc/meminfo ripples via lxcfs** — the
mechanism that makes the controller's deploy-page memory math follow for free. The gate PASSED → the resize
leg was implemented.
## 3. Files created / modified
- `internal/localapi/guestmemory.go` (new) + `guestmemory_test.go` (new); `internal/localapi/server.go`
(Options.Memory + fields + routes); `cmd/felhom-agent/main.go` (wire Memory=px; construct + launch fasttick).
- `internal/fasttick/fasttick.go` + `fasttick_test.go` (new).
- `internal/reconcile/engine.go` (`LastResult`/`recordResult`) + `lastresult_test.go` (new).
- `internal/wgtunnel/manager.go` (`TunnelConvergence` + cached snapshot refresh in `Apply`).
- `CHANGELOG.md`, `REPORT.md` (this), `CONTEXT.md`, `REUSE.md`.
## 4. Tests + red-proofs (all restored green)
- localapi memory: GET fields, grow, shrink-above-floor, the three refusals (each asserting **SetConfig
count == 0**), cross-guest 403, fresh-bounds-per-request, verify-not-reflected 502, nil-config 503.
- fasttick: pulse-while-unconverged, silent-when-converged, the ruled disarm-on-convergence, full-channel
non-blocking drop, first-reason. reconcile: `LastResult` effect + pre-first-run ok=false.
- **Red-proofs (run-fail-restore):** (i) floor guard removed → below_usage_floor 412 test fails; (ii) max
guard removed → above_max test fails; (iii) fast-tick ignores convergence (always pulse) → the silent +
disarm tests fail. All confirmed red, then restored. Full suite: `go build ./... && go vet ./... &&
go test ./...` — all 28 packages pass.
## 5. Guests-0/0 diagnosis — hypothesis REFUTED, no fix
Live read-only checks on the nested box: pool `felhom` `members: [9201]`; the agent token's ListLXC sees
`[9201]` (VM.Audit comes from the `/pool/felhom` grant — so pool membership IS the visibility mechanism);
hub reports 1/1. The agent log shows the observed 0/0 was the **pre-provision window**: `16:11 stale-lock …
pool=felhom listed=0` + `desired … guests=0`, then `16:16 lanresolver … vmid=9201` (guest created). No guest
existed yet → empty list, no error (the silent shape) — correct, not a bug. The existing `PoolAddVMID`
re-assert (`bringup.go:498`) already covers the known restore-over-existing edge (campaign-2 R2). **Item 2's
fast-tick is precisely the mitigation** — it shortens the pre-provision reporting window from ≤15 min to ~30 s.
## 6. Deploy verification (BOTH hosts)
Built on 180 (`go build -ldflags -X main.version=0.90.0`), deployed to both Proxmox hosts (backup + install +
restart):
- **felhom-pve** (the N100): `felhom-agent 0.90.0` active; `capabilities self-check ok=63 total=63 degraded=0`;
**`fast-tick armed: 30s out-of-band cadence while desired-state is unapplied`**; poke listener + local-api up.
- **demo-vm-felhom-4846bc** (nested, 192.168.0.135): `felhom-agent 0.90.0` active; `capabilities self-check
ok=63 degraded=0`; `fast-tick armed: 30s …`; local-api on 192.168.0.135:8443. (Converged box → fast-tick
quiet after the startup line, as designed.)
## 7. NOT yet live-validated / operator GO
- **Fast-tick real-onboarding proof:** the fast-tick's value shows on a REAL fresh box's pre-tunnel window
(the rehearsal S5 bind sequence should now be visibly instant). Not manufacturable non-destructively here;
both agents show the armed startup line + the unit tests prove the pulse/disarm.
- **Floor LAST (operator GO):** both demo agents verifiably run 0.90.0 (the safety gate) and the controller
coupling is PROVEN LIVE via the X-Felhom-Agent-Version header (see the controller REPORT's live refusals).
Publishing 0.90.0 to Gitea + the hub Day-0 manifest **vouch + MinAgent-floor raise to 0.90.0** are set via
the operator-password-gated Day-0 manifest UI (CC cannot log in) — the operator (Viktor) GO step, matching
the v0.89 train's "publish/vouch = operator follow-up".
## 8. Observations
- The nested demo host is tiny (4 GB): its resize bounds are degenerate (`max_mb 1866 < min_mb 2048 <
current 2890`) — no resize *succeeds* through the bounded endpoint there (all refused correctly). The
successful apply is Phase-0-proven at the agent layer; the controller live-validation exercised the
refusals (above_max, below_min) end-to-end.
- REUSE.md gained the fasttick Source pattern + the localapi MemoryOps seam.