felhom-agent/REPORT.md

REPORT — v0.74.0: re-assert pool membership after restore-over-existing (campaign-2 R2)

Date: 2026-07-07 · Class: implementation (agent). Baseline: main @ e04b75e (v0.73.0) → ca0b169. Part B of the campaign-2 R1/R2 fix bundle (C = controller F-C2-1 v0.103.0; A = operator pool one-liner; D = DR-source spike; E = report correction).

Root cause (the campaign report's R1 was a symptom)

Pool membership is what lets the pool-scoped FelhomAgentGuest token reach a guest — the grant at /pool/<pool> applies only to pool MEMBERS. pct restore --pool sets membership at CREATE, but a restore OVER AN EXISTING VMID (the host-loss/finale path) never re-applies it, and no code re-added a guest to the pool. So every destroy-restore silently dropped membership → the guest lost VM.Audit → the NEXT --selftest=restore-test/DR couldn't read the source config → its existing, correct bindMountOverrides never ran → raw bind mounts hit pct restore → "mp8 … only possible for root" (the campaign's mislabelled "R1"). The role + ACL were correct all along; membership was the fault.

The fix

• Client.PoolAddVMID(ctx, pool, vmid) (internal/proxmox/mutate.go): PUT /pools/{pool} vms={vmid} — PVE-additive (merge, not replace; delete=1 removes), idempotent (already-member swallowed), needs Pool.Allocate (the token has it). Sync (no UPID).
• bring-up re-asserts (internal/reconcile/bringup.go): after liveness is proven, if spec.Pool != "", call PoolAddVMID. A pool-add hiccup is surfaced LOUD + into res.StartWarnings but must NOT flip a healthy running guest's verdict (membership matters for the NEXT op).
• B3 (scratch teardown 403) — diagnosed, no code: restoretest.go already passes Pool: DefaultPool for scratch restores → the campaign's VM.Allocate teardown 403 was a CASCADE of the failed bind-mount restore (a half-built guest outside any pool), not an independent gap.
• Role/ACL untouched (correct); bindMountOverrides untouched (correct — it just needed to run).

Tests + red-proofs

pool_test.go: PoolAddVMID PUT shape + idempotent-on-already-member + real-error-surfaces + validation. bringup_test.go: re-asserts when Pool!="" (red-proof: pre-fix no-call → poolAdds=[] FAIL, demonstrated + reverted), no-pool→no-call, pool-add-failure warns-but-passes (liveness wins, guest kept). Full gate go build/vet/test ./... = PASS.

Deploy + live acceptance (the headline)

Built -X main.version=0.74.0, deployed to felhom-pve (.bak-0.73.0 kept); --version 0.74.0, active, capabilities 56/56 degraded=0. Part A run (pveum pool modify felhom --vms 9201, per operator go): 9201 now a pool member; VM.Audit+VM.Allocate+VM.Backup present on /vms/9201. Live --selftest=restore-test then PASSED for the first time: neutralizing source bind-mount mountpoints … bind_mounts=2 → scratch 990000 restored + verified boot+running + torn down clean (no VM.Allocate 403) in 4m35s. B3 confirmed as a cascade; restore-test needs no bind code.

Observations

• DR bring-up -mode dr has a SEPARATE real gap (no overrides, source guest gone) — scoped in felhom.eu/documentation/audits/SPIKE-dr-bindmount-source-2026-07-07.md; recommended fix is a small known-constant override reusing bindMountOverrides (mp8/mp9 are structural constants).
• The empty pool predated the campaign (07:41 403 before the 14:09 finale) — an earlier drill's restore-over-existing; v0.74.0 stops the recurrence going forward.