MinAgent coupling: felhom-controller v0.143.0 gates its guest-memory-resize UI on
this agent (FeatureGuestMemoryResize, MinAgent 0.90.0).
R-24 guest RAM resize (internal/localapi/guestmemory.go): self-scoped GET/POST
/guest/memory. Agent enforces every bound FRESH per request (min 2048, max
host_total-2048, shrink floor max(2048, usage+512)); applies via PVE SetConfig —
live cgroup apply, no reboot (Phase-0 proven on the nested demo box). Verify-after-apply
re-reads maxmem before claiming success. New narrow MemoryOps seam (GuestAPI untouched);
Options.Memory nil -> 503. Memory only.
R-28 fast-tick (internal/fasttick): while any desired-state item is unapplied -
including the pre-tunnel window a hub poke can't reach - pulse the shared out-of-band
trigger every 30s, self-disarm on convergence. Four cached sources (desired-gen==0,
reconcile Planned-Pending>0, pbsdr waiting_secret only, wgtunnel desired-not-operational);
LOUD pbsdr states + pending_signature excluded. Seams: reconcile.Engine.LastResult() +
wgtunnel.Manager.TunnelConvergence() (cached, no per-tick exec).
Guests-0/0: hypothesis REFUTED live (9201 IS a pool member; 0/0 was the pre-provision
window; PoolAddVMID re-assert already covers restore-over-existing). No code change; the
fast-tick mitigates the window.
Tests + red-proofs (i floor guard, ii max guard, iii always-pulse) all restored green.
The restore-test had GL-5 finding #2's mirror image: its live-source-config
bind-override path tripped PVE's drop-unlisted-mountpoints rule, so scratch
guests boot-verified WITHOUT their storage mpN - weaker verification than
claimed. Params now derive from the ARCHIVE's own embedded config via
ExtractArchiveConfig + drRestoreOverrides (the object under test; full
layout, content genuinely extracted - the added runtime IS the
verification); unreadable/unknown-topology archives refuse up front. NEW
mount-parity assert (2b, pre-start): restored mpN set vs the archive's -
missing/mispathed/undersized/extra mpN fail the test naming the delta, so
constraint (b) can never regress into a green light. MountParity +
MountInventory ride the result + hub wire record (additive). Dead
bindMountOverrides/archiveVMID path deleted with its tests (no reachable
lookalike). DR bring-up untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The live scratch DR exposed the second half of PVE's all-or-nothing
explicit-params restore: mountpoints NOT named in the params are silently
DROPPED - the DR guest came up without its mp0/mp1 data volumes (boot
passed; the customer's world did not ride along). drRestoreOverrides now
derives the COMPLETE param set from the archive's extracted config:
explicit rootfs, every storage-backed mpN passed through (size + in-guest
path + backup flag preserved so vzrestore extracts its content), the two
structural binds replaced by 4d-swapped throwaways; unknown bind mpN or
unparseable size refuses loudly. Snapshot sections never shadow the
current config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The live validation hit PVE's all-or-nothing restore rule: mpN params
without an explicit rootfs -> HTTP 500 "mount points configured, but
'rootfs' not set" (the same constraint restoretest.go:211 documents for the
live-config path; the spike never ran an override restore). The lost guest
has no live config, so the rootfs SIZE now comes from the archive's own
embedded config via NEW Client.ExtractArchiveConfig (GET vzdump/
extractconfig - verified live: answers 200 under the scoped agent token;
PBS keys stay server-side, the spike's candidate-1 rejection holds; used
for the SIZE ONLY - the bind layout stays the platform constants).
Unparseable/unreadable archive config -> clean refusal before any restore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
ModeDRGuestLoss now passes restore-time MountOverrides for the two
platform-constant structural binds (mp8 parent, mp9 bootstrap) via the
shared throwaway-volume format helper - without them a customer-archive
restore under the privsep token fails outright ("restoring 'mp8' to bind
mount is only possible for root"). New post-restore step 4d swaps the real
binds in via the host runner (root pct set, one slot per call), deletes the
displaced unusedN volumes (API config PUT; a scoped-token refusal logs the
residue loudly instead of widening privileges), and respects the
committed/launched rollback envelope. Provision passes nil overrides -
byte-identical behavior (regression contract test).
Engine grows an optional HostRunner + StateDir seam (DR refuses up front
without a runner); selftest bring-up wires the ExecRunner + cleans the
scratch mp9 host dir on teardown; proxmox.GuestConfig.Unused() added.
6 new tests incl. C2 mid-swap rollback + C3 older-archive + 403-warn paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pool membership is what lets the pool-scoped token reach a guest; pct restore
--pool sets it only at CREATE, so a restore over an existing VMID drops the guest
from the felhom pool and 403s the next restore-test/DR on VM.Audit. This empty-pool
state is the true root cause of the campaign's "R1" (bind-mount restore failing was
a symptom — restore-test's existing bind neutralization never ran without config-read).
Add Client.PoolAddVMID (PUT /pools, additive+idempotent, Pool.Allocate) and call it
in bring-up after liveness when spec.Pool!="" — warn-not-fail on a hiccup (liveness
wins). B3 scratch-teardown 403 diagnosed as a cascade (restoretest already passes
Pool). Role/ACL untouched. Tests + red-proof.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
- reconcile: ClassAgentUpdate op class; always Destructive (no provenance
blesses replacing the root-adjacent binary). classify test + companion
(TestClassify_AgentUpdateAlwaysDestructive).
- opsign: `-op agent_update` with -agent-version + -sha256 (isHex64-validated);
params {version,sha256}. isHex64 test (Group D).
- config: SelfUpdateConfig{URLTemplate,Username,Token,StateDir,DwellSeconds}
+ WithDefaults + Token redaction.
- internal/selfupdate: Executor (download → verify vs the SIGNED sha → sudo -n
wrapper `apply`; sha is the only integrity root — mismatch refuses + removes,
agent untouched); Manager (startup dwell → `commit`; version-mismatch → no
commit + loud WARN + marker left for report visibility; shutdown-before-dwell
leaves pending). WrapperRunner seam → tests never shell out.
- hub report: additive selfupdate_pending(+version) via SetSelfUpdateReporter
seam; both omitempty (Wireguard precedent) so the cross-repo golden contract
stays byte-stable — no hub change.
- capability manifest: 3 non-critical FELHOM_SELFUPDATE probes.
- main.go: updateExec appended to the executor chain; commit-manager wired to
the report seam + MaybeCommit goroutine after core init.
Tests: Group A (executor happy/sha-mismatch+companion/bad-params/wrapper-fail),
B (agent_update rides the real gate: pinned-key executes, non-pinned +
retarget rejected), C (commit/version-mismatch/no-pending/shutdown), D (opsign).
C2 companion red-proof verified (neutered Go verify → bad binary reaches apply
→ test fails), reverted. Full go test ./... green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
A WAN (pbs-tier) restore of a large guest exceeds the restore-task wait's 10m
default → the wait expired mid-restore, teardown fired against a still-restoring
(not-yet-pool-associated) scratch guest → leak + a phantom VM.Allocate 403.
- RestoreTestSpec.RestoreTaskTimeout (0→10m default); the restore WaitTask passes
it. Local tier unchanged (10m).
- config RestoreTestPBSRestoreTimeoutSeconds + accessor (default 120m).
- main restoreTaskTimeout(cfg,tier): configured PBS timeout only when tier==pbs,
else 0. Both scheduler + selftest spec builds.
- Tests + WaitOptions red-proof + accessor contract.
The "grant scratch-band VM.Allocate" follow-up is diagnosed not blind-applied:
the scratch is restored INTO /pool/felhom (ACL already grants VM.Allocate), so
the earlier 403 was a consequence of the timeout. No ACL/host-install change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Campaign pool-effects F1 (HIGH): the bring-up compensating rollback and the
restore-test teardown destroyed the target vmid even when RestoreLXC failed
synchronously without creating anything — destroying a guest the transaction
never made (only the pool ACL 403 contained it). A RestoreLXC UPID is now the
sole destroy authorization in all three destroy paths (in-process bring-up
defer, in-process restore-test teardown, Recover). F2: the restore-test
advances past an 'already exists' band vmid (invisible squatter) instead of
failing + false-alerting; a fully-occupied band Skips.
Red-proof verified: with the gates reverted, the four new tests fail with the
innocent-guest destroy. go build/vet/test clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RestoreLXCOptions.Pool → pct restore --pool (omit-when-empty). New
reconcile.DefaultPool="felhom"; BringUpSpec.Pool threaded to the bring-up
restore; BOTH restore sites pool the guest (provision/DR via spec.Pool set to
DefaultPool by the CLI; restore-test scratch via DefaultPool = SPIKE residual
#2). No agent ACL change (ships in host-install v1.6.0); the pool param is inert
until the token has Pool.Allocate + the pool exists, so publishing is safe ahead
of the coordinated swap. Tests + red-proofs; go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New -cores N / -memory M (MiB) flags for --selftest=bring-up|provision (0 = keep
golden default), flowed through bringUpSizing into reconcile.BringUpSpec so the
existing buildBringUpConfig emits cores/memory into the pre-start config PUT (the
cap lands before first boot). No engine change. New pure-function test
TestBuildBringUpConfig_ResourceCaps asserts the set + omit-when-zero cases
(red-proof run + reverted). go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).
- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).
Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume
(mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go
grows the golden's data volume to the per-customer target (DataVolGrowGB) and
emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent
DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New
RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PVE refuses a restore carrying mountpoint params unless rootfs is also set ("mount
points configured, but 'rootfs' not set"). Size the rootfs override from the source
rootfs (rootfsSizeGB/sizeToGB, round up). Validated manually on the host: restore of
bind-mounted 9201 with --rootfs + --mp0/--mp9 overrides boots + tears down cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A slice-10 enrolled guest's data drive is a host bind-mount mp0 that the privsep
token can't vzrestore ("bind mount is only possible for root") — so the restore-test
failed for every enrolled guest regardless of backup tier. The restore-test now reads
the source guest config (vmid from the archive volid) and passes RestoreLXC mp
overrides converting each bind-mount mpN to a throwaway 1G volume on the restore
storage (no root needed; boot-verify doesn't need the data). proxmox.RestoreLXC gains
MountOverrides. + unit tests (archiveVMID, bindMountOverrides).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The control envelope becomes live: the agent caches the hub's desired-state +
generation and re-fetches GET /hosts/{id}/desired-state only when the
generation advances. A new internal/desired Syncer maps the wire shape into a
reconcile.CachingProvider feeding the engine; benign deltas reconcile, an
explicit guest decommission is gated pending_signature (exec is 10B). Adds the
DesiredStateResponse/WireDesiredState wire types + Client.FetchDesiredState +
the loop EnvelopeObserver seam. Cross-repo golden (envelope + desired-state)
byte-identical with the hub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook:
PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting
advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict
was decided by an advisory exit code before the real boot check ran. Every
modern-distro restore-test reported pass:false.
- proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict)
- restore-test start step accepts warnings, surfaces them, verdict stays waitRunning
- RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable
nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail
- hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5)
- scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings
- tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free
regression guard (systemd 256-300)
Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The security core of slice 4: hub-supplied intent is no longer trusted for
destructive change. The gate fronts the per-guest queue's executor, so every
mutation passes it. Reuses internal/authz for all crypto (surface untouched).
- Classifier (doc 03 §4): benign vs destructive by provenance + data-bearing-
ness, NOT by verb. Destroy/overwrite of customer data is destructive unless
agent-internal provenance (same-journaled-txn create, or agent-tagged scratch)
makes it benign — and that provenance is journal-recorded, NEVER hub-sourced.
Unknown op class fails safe to destructive.
- Reversibility gate: benign -> allowed unsigned; destructive -> requires a
verified, role-scoped, action-bound operator signature, else pending_signature
and never executed. Every decision audited (signal, never the guard).
- Signed-op consuming layer over authz.Verifier.Verify (locked pipeline
untouched): role-scoping (doc 04 §4 — recovery=rotation only, operational=
ordinary destructive + planned rotation) + op-to-action binding (op+host+
guest+params must match the gated action).
- Signed-job orchestration: idempotency dedupe by nonce + journal-wrapped
execution via an injected DestructiveExecutor (nil this slice — inert).
- Crash recovery (Note 1): Engine.Recover consumes the journal InFlight() set at
startup (resume-or-rollback) — covers an op that crashed after the POST and
before its terminal record, which idempotency dedupe alone cannot. Added
TaskStatusOnce to the GuestAPI seam. Wired into daemon startup.
- Note 2: memory comparison canonicalized to MiB (desiredMemoryMiB) so a
non-MiB-aligned MemoryBytes converges in one pass, not perpetual drift.
- Daemon: builds the verifier from config signers (none = nil verifier, the
common slice-4 state), the gate (+SlogAudit), runs Recover before mutating.
Adversarial matrix proven against the REAL authz.Verifier with in-test-minted
SSHSIGs (framing replicated in reconcile's test binary; authz untouched, no
signing added to the verify-only package): unsigned job + unsigned desired-state
delta -> pending_signature; unknown signer/expired/replay-across-restart/wrong
host -> typed authz rejections; wrong guest/op/params -> binding_mismatch;
recovery key on ordinary destructive -> role_denied; hub-supplied scratch tag
ignored -> refused; valid+role+target+fresh nonce -> accepted then replay
rejected. Full module race-clean + vet-clean on the Linux build server.
Inert this slice: no destructive deltas served until slice 10; the destructive
path is classified, gated, and tested but not wired to live execution.
CHECKPOINT: Phase B complete (slice 4 done). Awaiting validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New internal/reconcile package: the agent-side control core's structural half.
- Per-guest serializer Queue (doc 03 §10): the single choke point all mutation
sources funnel through; same-vmid serial in submit order, different vmids
parallel (cond-var FIFO lanes).
- Desired-state model + DesiredProvider seam; EmptyProvider is the only live
source at slice 4 (no hub serving until slice 10) so the live engine computes
an empty action set and performs zero mutations.
- Normalization layer (FieldNormalizers): normalized desired-vs-actual so
Proxmox round-trip quirks don't read as drift. normDesc promoted out of
main.go to reconcile.NormDescription; selftest uses the shared helper.
- Plan (pure diff): minimal benign action set (Start/Stop/SetConfig) for guests
in both desired and actual; provision/destroy out of scope here.
- Engine: dispatches onto the shared queue; honors the dual-mode SetConfig
contract (UPID -> WaitTask; empty UPID -> synchronous success).
- Durable op journal + idempotency store (mirrors authz.FileNonceStore):
in-flight task ids for crash detection + AlreadyApplied dedupe across restart.
- Wired into runDaemon alongside the hub loop, sharing the queue; runs cleanly
with no desired state and no signers.
Full module race-clean and vet-clean on the Linux build server.
CHECKPOINT: Phase A only. Awaiting validation before Phase B (the reversibility
gate + signed-op consuming layer, landing v0.4.0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin