Compare commits

..

4 Commits

17 changed files with 451 additions and 25 deletions
+44 -3
View File
@@ -1,7 +1,48 @@
## Unreleased (2026-10-19) — three fixes for the next release (R-921, R-1070, R-1010); two texts made true after 0.160.0
## Unreleased (2026-10-21) — every guest may swap as much as its RAM (R-1192, the 8 GB freeze) — COMMITTED, NOT RELEASED (the release is the operator's word)
Not released (the night brief of 2026-10-19: agent code + tests + commit only; the release is the operator's word). No
sudoers, wrapper or config-bundle change.
The freeze of the rehearsal's first afternoon, reproduced and measured on demo-hp (felhom.eu
`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets `swap`, and the agent
never set it. On the 8 GB machine (guest 4096 MB) the first uploads made Immich load its analysis models; the guest hit
its limit with the swap full and could evict only program code — 150 000–600 000 refaults and ~1 GB of disk reads per
10 s, memory „full" pressure 92–95 %, load 155, the dashboard dead for 6½ minutes, no app killed. Raising the swap cap
to 4096 MB live ended it within a minute while the uploads ran on.
- `reconcile.buildBringUpConfig`: a new guest gets `swap` = its memory.
- `POST /guest/memory` (RAM resize): `swap` moves with `memory` (`localapi.GuestSwapMB`).
- At agent start (one minute in): `localapi.EnsureGuestSwap` raises every LXC in the `felhom` pool whose swap is below
its RAM to it — a live cgroup change, no restart; never lowers a higher swap; one guest's failure is logged and the
others are still done.
- Tests `TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM`, `TestR1192_BringUpSetsSwapToTheRAM`, the resize test's
new swap assertion — each red-proved (the write removed → `raised 0, want 2`; `swap=""` on both other paths).
- Privilege: `swap` is under `VM.Config.Memory`, which the RAM resize already uses; no role, sudoers or bundle change.
Added the night of 2026-10-21 (still unreleased, the same release):
- **The host's swap is reported and checked (R-1192 follow-up).** The guest fix moves a memory burst onto the HOST's
swap; a host with none gets nothing from it. The host report carries `host.host_swap_mb` (Proxmox node status — the
kernel's SwapTotal, the same number as `/proc/meminfo`) and `host.guest_swap_mb` (the sum of the guests' `swap`; each
guest spec also carries `swap_bytes`). When the first is below the second the agent logs one WARN (and one INFO when
it is covered again); the hub raises `host_swap_short`. `GET /guest/memory` carries `host_swap_mb` and
`guest_swap_mb` for the controller's install page. The agent never creates host swap (a decision — felhom.eu
register). Measured the same night: our installer gives an 8 GB / 256 GB machine 7.75 GB of host swap.
- **R-1194 — no false „operator access degraded" on a fresh box.** The first host report was sent before the
felhom-sshd loop's first pass, read a config that was never rendered („config invalid, port 0") and the hub mailed
`oob_degraded` — reproduced on a fresh install 2026-10-21 (the report at 20:34:00, the apply 0.4 s later, the daemon
healthy; the check runs `sshd -t` through sudo, so the non-root theory was false). The loop now reports no OOB stanza
until its first pass has run (a failed pass counts, so a real lasting failure still reaches the hub).
- Tests `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass` — each red-proved.
## v0.161.0 — three fixes (R-921, R-1070, R-1010); two texts made true after 0.160.0 (2026-10-20) — RELEASED on the operator's yes, NOT VOUCHED
Released 2026-10-20 (operator yes, the morning answers) with `scripts/release-agent.sh` from a clean clone at `5152123`:
binary sha256 `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle sha256
`485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47`, tag `v0.161.0`. Delivered by one signed
`agent_update` job per box to demo-hp (06:38 UTC), Tester 1 (06:54) and demo-felhom (06:55): each `signed op COMPLETED`,
`capabilities self-check ok=76 total=76`, `update committed`. **The vouch is the operator's.** Live on demo-hp: `host_busy`
answers `{"busy": true, "what": "backup:local"}` during a „Mentés most" whole-guest backup and `{"busy": false}` before
and after (read from inside 9201 with its own token; felhom.eu `audits/night2-2026-10-20/P1/`). R-1083 (a key for an
existing guest) is NOT in this release: see the R-1083 row. No sudoers, wrapper or config-bundle change.
- **R-921 — `GET /backup/status` says when the host is busy.** New field `host_busy` (`{busy, what}`): whether the
host-wide heavy-operation gate is held right now (another guest's backup, the restore test, a night OS step, the
+5
View File
@@ -1,5 +1,10 @@
# CONTEXT — felhom-agent working state
> **2026-10-20 — v0.161.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom** (signed jobs, all
> committed, 76/76). R-921 `host_busy` (proven live on demo-hp during a press), R-1070 the tunnel `absent` state (9202's
> real Docker 29 answer is the fixture row), R-1010 the janitor's one `force=1`. Vouch: version 0.161.0, sha256
> `ff4e6dc5…99d3b5`, bundle `485f36c9…12c47`. R-1083 left out on purpose: an agent key for 9202 would make its
> controller book whole-guest backups through the agent, one tier writing to ep0 — the R-1083 row says what is needed.
> **2026-10-19 — v0.160.0 RELEASED (operator yes), NOT VOUCHED; on demo-hp, Tester 1, demo-felhom.** R-1093, R-978,
> R-973 (below) plus R-1149 (the local tier prunes `keep-daily=N`, `BackupTier.PruneSpec`) and R-1151 (a claim with
> `{"defer_upload": true}` keeps the package until `POST /escrow/ceremony/confirm` brings the two words at the drawn
+16 -18
View File
@@ -1,20 +1,18 @@
# REPORT — agent, evening/night brief 2026-10-19 (code + tests + commit only, NOT released)
# REPORT — agent, the night of 2026-10-21: the host's swap is reported; no false emergency-door alarm on a new box — COMMITTED, NOT RELEASED
Commit `56aabc7` (CHANGELOG „Unreleased"), scanner wiring `84ef581`. No sudoers, wrapper or config-bundle change.
The release is the operator's word (STATUS morning list: „Release agent 0.161.0?"), then the vouch.
**Brief:** the unattended night brief of 2026-10-21 (Parts 1, 2, 5). Architecture: felhom.eu `03-host-agent.md` §9 (the
new swap paragraph). Both join the swap fix (`286a7e3`) in the coming 0.162.0 — the operator's release and vouch (D26 = A).
- **R-921 — `host_busy` on `GET /backup/status`:** whether the host-wide heavy-operation gate is held (`{busy, what}`),
the state in which `POST /backup` answers 409. Controller 0.335.0 (the floor) reads it before any stop. Test
`TestBackupStatus_R921_HostBusy`, red-proved (assignment removed → `free gate: host_busy = <nil>`).
- **R-1070 — tunnel `absent`:** Docker 29 prints „error: no such object" in lower case; the case-sensitive match read
a missing cloudflared container as `unknown`, which failed the host OS-update rule (Tester4, 2026-10-15). Now a
missing container is `absent`, and the host rule passes it only when the reading from before the step also had no
cloudflared container. Tests `TestHostHealthVerdict_R1070_NoTunnelSetUp`, `TestClassifyTunnel` (Docker 29's wording,
measured on demo-hp), red-proved both ways.
- **R-1010 — the janitor's forced snapshot delete:** on „Failed to find logical volume …_vzdump" only, one retry with
the API's `force=1` (`DeleteSnapshotForce`; `pvesh usage … -command delete` lists `--force`). Tests
`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone` (6 rows), `TestDeleteSnapshotForce_R1010`, red-proved.
- **R-986 — the security-scan gate** (tooling): the shared `felhom.eu/scripts/security_scan_gate.py` in the fast set;
`.security-accepted` holds 15 accepted test constants; decoys in `test_gate_decoys.py`.
Gates: `go build/vet/test ./...` green; `agent_gates.py --fast` OK; decoys 36 cases OK. CI 2103, 2110 success.
- **Host swap (R-1192 follow-up), `0da792e`:** the host report carries `host.host_swap_mb` (Proxmox node status = the
kernel's SwapTotal) and `host.guest_swap_mb` (the guests' `swap` sum; each guest spec `swap_bytes`); one WARN when the
first is below the second, one INFO when covered again; `GET /guest/memory` carries both for the controller's install
page. The agent never creates host swap (register R-1207). Proven live on the soak box: the hub stored
`host_swap_mb 7935, guest_swap_mb 4096`.
- **R-1194, `0da792e`:** no OOB stanza until the felhom-sshd loop's first pass has run — the first report on a fresh box
read a config not yet rendered and the hub mailed `oob_degraded` (reproduced 2026-10-21 20:34 CEST on a fresh install).
- **The swap fix itself** ran as a hand-placed fixture on the 8 GB soak box: guest swap 512 → 4096 MB one minute after
start; the 2-hour household load answered every probe (felhom.eu `audits/night-2026-10-21/P1-soak/`).
- Release candidate built from a clean clone at `0da792e` with the release flags, version `0.162.0`:
sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (not published).
- Tests (red-proved): `TestR1192_HostReportCarriesSwapAndWarnsOnce`, `TestR1192_GuestMemoryCarriesHostAndGuestSwap`,
`TestR1194_NoOOBStanzaBeforeTheFirstPass`. CI job 2176 green.
+16
View File
@@ -936,6 +936,22 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// R-1192: every guest may swap as much as its RAM — raised once at start (a live change, no restart). A minute
// after start so the guest has settled; a failure is logged and the next start retries.
go func() {
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
sctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
if n, err := localapi.EnsureGuestSwap(sctx, px, reconcile.DefaultPool, logger); err != nil {
logger.Warn("guest-swap: check skipped", "err", err)
} else {
logger.Info("guest-swap: checked", "raised", n)
}
}()
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
+22 -1
View File
@@ -22,6 +22,11 @@ type Loop struct {
mu sync.Mutex
fetched bool // a desired-state document has been delivered (false = never a teardown signal)
block *hub.WireWireguard
// passed (R-1194) is set once the first reconcile pass has run. Until then the instance has never been rendered,
// so its health says nothing — and the first host report, sent before that pass, used to read „config invalid,
// port 0" and raise a false oob_degraded on EVERY fresh box (rehearsal 2026-10-20; reproduced 2026-10-21: the
// report at 20:34:00 read an absent config, the apply landed 0.4 s later).
passed bool
nudge chan struct{}
}
@@ -85,6 +90,7 @@ func (l *Loop) reconcile(ctx context.Context) {
if !fetched {
applyBlock = nil
}
defer l.markPassed()
port, err := l.mgr.Apply(ctx, applyBlock)
if err != nil {
return // Apply logged; a claim/exhaustion or install error — retry next tick
@@ -106,8 +112,23 @@ func oobPeerIP(block *hub.WireWireguard) string {
return block.OOBPeerIP
}
// OOBStatus implements the hub collector's reporter seam (Part 4).
// markPassed records that a reconcile pass has run (R-1194) — after a failed Apply too, so a real, lasting failure
// is still reported from the next report on.
func (l *Loop) markPassed() {
l.mu.Lock()
l.passed = true
l.mu.Unlock()
}
// OOBStatus implements the hub collector's reporter seam (Part 4). nil — no stanza, which the hub does not evaluate —
// until the first reconcile pass has run (R-1194).
func (l *Loop) OOBStatus(ctx context.Context) *hub.OOBStatus {
l.mu.Lock()
passed := l.passed
l.mu.Unlock()
if !passed {
return nil
}
_, block := l.snapshot()
return l.mgr.Status(ctx, block)
}
@@ -0,0 +1,29 @@
package felhomsshd
import (
"context"
"testing"
"time"
)
// R-1194: on every fresh box the first host report — sent before the OOB loop's first pass — read „config invalid,
// port 0" (the config was not rendered yet) and the hub mailed oob_degraded. Reproduced 2026-10-21 on a fresh box
// (felhom.eu audits/night-2026-10-21/P5-checks/): the report at 20:34:00, the apply 0.4 s later, the daemon healthy.
// The loop now reports NO stanza (the hub does not evaluate a missing one) until a pass has run — a failed pass
// counts, so a real lasting failure is still reported from the next report on.
// RED-PROOF: with OOBStatus not gated on `passed`, the first assertion fails (a stanza with config_invalid).
func TestR1194_NoOOBStanzaBeforeTheFirstPass(t *testing.T) {
r := newScriptRunner()
m := NewManager(r, t.TempDir(), nil)
m.isActive = func(context.Context) bool { return false }
m.isFailed = func(context.Context) bool { return false }
m.now = func() time.Time { return time.Unix(1783270000, 0) }
l := NewLoop(m, nil, time.Minute, nil)
if st := l.OOBStatus(context.Background()); st != nil {
t.Fatalf("a stanza before the first pass reads a config that was never rendered: %+v", st)
}
l.reconcile(context.Background()) // fails or succeeds — either way a pass has run
if st := l.OOBStatus(context.Background()); st == nil {
t.Fatal("after the first pass the health must be reported (a real failure must reach the hub)")
}
}
+7 -1
View File
@@ -11,6 +11,7 @@ import (
"os"
"strings"
"sync"
"sync/atomic"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -136,6 +137,8 @@ type Collector struct {
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger
now func() time.Time
// swapShortWarned (R-1192): the host-swap-short WARN is logged once per state change, not every report.
swapShortWarned atomic.Bool
}
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
@@ -375,6 +378,8 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
}
host := hostMetrics(c.px.Node(), ns)
guests := c.collectGuests(ctx)
c.applySwap(&host, ns, guests) // R-1192: host swap vs the guests' swap allowance
host.CPUTempC = c.cpuTempC(ctx) // slice 9: operator freebie — temp now rides the hub report too
host.WrapperSHA256 = pbsWrapperSHA256() // R-50b(a): make privileged-artifact drift answerable
report := &HostReport{
@@ -383,7 +388,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(),
Host: host,
Guests: c.collectGuests(ctx),
Guests: guests,
// storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx),
@@ -567,6 +572,7 @@ func (c *Collector) collectGuests(ctx context.Context) []Guest {
Cores: cfg.Cores,
MemoryBytes: g.MaxMem,
DiskBytes: g.MaxDisk,
SwapBytes: cfg.Swap * (1 << 20),
}
}
guests = append(guests, entry)
+39
View File
@@ -0,0 +1,39 @@
package hub
import (
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the guest swap fix (every guest may swap as much as its RAM,
// localapi/guestswap.go) moves a memory burst onto the HOST's swap. Measured on demo-hp the same day: our installer
// gives an 8 GB / 256 GB machine 7.75 GB of host swap, so a 4096 MB guest's 4096 MB swap fits. A host with no swap
// (a BYO install, or swap turned off) gets nothing from the fix. So the report carries both numbers, the agent says it
// once in its log, and the hub raises host_swap_short. The agent never creates host swap itself: changing a host its
// owner set up is a decision (felhom.eu register).
// HostSwapShort reports whether the host's swap is below the guests' swap allowance. A host whose guests have no
// swap allowance (0) is never short.
func HostSwapShort(hostMB, guestMB int64) bool { return guestMB > 0 && hostMB < guestMB }
// applySwap fills the host block's swap numbers from the node status and the guests' specs, and logs the short
// state once per change.
func (c *Collector) applySwap(h *HostMetrics, ns proxmox.NodeStatus, guests []Guest) {
hostMB := ns.Swap.Total / (1 << 20)
var guestMB int64
for _, g := range guests {
if g.Spec != nil {
guestMB += g.Spec.SwapBytes / (1 << 20)
}
}
h.HostSwapMB, h.GuestSwapMB = &hostMB, &guestMB
if HostSwapShort(hostMB, guestMB) {
if c.swapShortWarned.CompareAndSwap(false, true) {
c.logger.Warn("host-swap: the host's swap is below its guests' swap allowance — a memory burst can freeze the box (R-1192); the hub is told (host_swap_short)",
"host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
return
}
if c.swapShortWarned.CompareAndSwap(true, false) {
c.logger.Info("host-swap: the host's swap covers its guests' swap allowance again", "host_swap_mb", hostMB, "guest_swap_mb", guestMB)
}
}
+42
View File
@@ -0,0 +1,42 @@
package hub
import (
"bytes"
"log/slog"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192 follow-up (night 2026-10-21): the host report carries the host's swap and the guests' swap allowance, and the
// agent says ONCE in its log when the host's swap is below it (and once when it is covered again).
// RED-PROOF: with applySwap not called the numbers are nil; with the CompareAndSwap removed the WARN repeats.
func TestR1192_HostReportCarriesSwapAndWarnsOnce(t *testing.T) {
var buf bytes.Buffer
c := &Collector{logger: slog.New(slog.NewTextHandler(&buf, nil))}
guests := []Guest{{VMID: 9201, Spec: &GuestSpec{SwapBytes: 4096 << 20}}, {VMID: 9202, Spec: nil}}
var ns proxmox.NodeStatus // a host with no swap
for i := 0; i < 3; i++ {
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if h.HostSwapMB == nil || h.GuestSwapMB == nil || *h.HostSwapMB != 0 || *h.GuestSwapMB != 4096 {
t.Fatalf("report numbers: host %v guest %v", h.HostSwapMB, h.GuestSwapMB)
}
}
if n := strings.Count(buf.String(), "host-swap: the host's swap is below"); n != 1 {
t.Fatalf("the short WARN must be logged once, got %d:\n%s", n, buf.String())
}
ns.Swap.Total = 7935 << 20 // the installer's 8 GB host
h := HostMetrics{}
c.applySwap(&h, ns, guests)
if *h.HostSwapMB != 7935 || HostSwapShort(*h.HostSwapMB, *h.GuestSwapMB) {
t.Fatalf("a covered host must not be short: %v / %v", *h.HostSwapMB, *h.GuestSwapMB)
}
if !strings.Contains(buf.String(), "covers its guests' swap allowance again") {
t.Fatalf("the recovery must be said once:\n%s", buf.String())
}
if HostSwapShort(0, 0) {
t.Fatal("a host whose guests have no swap allowance is never short")
}
}
+6
View File
@@ -338,6 +338,11 @@ type HostMetrics struct {
//
// Empty = unreadable/absent, which the hub treats as UNKNOWN, never as drift.
WrapperSHA256 string `json:"wrapper_sha256,omitempty"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up, night 2026-10-21): the host's swap (Proxmox node status — the
// kernel's SwapTotal) and the sum of its guests' swap allowance. The guest swap fix (swap = RAM) needs swap ON THE
// HOST behind it; the hub raises host_swap_short when the first is below the second. nil = not read (older agent).
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
}
// Guest is one LXC. The agent reports vmid; the hub derives the guest PK
@@ -355,6 +360,7 @@ type GuestSpec struct {
Cores int `json:"cores"`
MemoryBytes int64 `json:"memory_bytes"`
DiskBytes int64 `json:"disk_bytes"`
SwapBytes int64 `json:"swap_bytes,omitempty"` // R-1192: the guest's swap allowance (PVE config `swap`, MB → bytes)
}
// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
+12 -1
View File
@@ -57,6 +57,10 @@ type MemoryInfo struct {
MaxMB int64 `json:"max_mb"`
FloorMB int64 `json:"floor_mb"`
Running bool `json:"running"`
// HostSwapMB / GuestSwapMB (R-1192 follow-up): the host's swap and this guest's swap allowance, so the controller
// can tell the household when the host has too little swap behind the guest (its install page). nil = not read.
HostSwapMB *int64 `json:"host_swap_mb,omitempty"`
GuestSwapMB *int64 `json:"guest_swap_mb,omitempty"`
}
// memoryResizeRequest is the POST /guest/memory body.
@@ -94,6 +98,8 @@ type memoryBounds struct {
maxMB int64
floorMB int64
running bool
hostSwapMB int64
guestSwapMB int64
}
func (b memoryBounds) refusal(code string) memoryRefusal {
@@ -104,7 +110,9 @@ func (b memoryBounds) refusal(code string) memoryRefusal {
}
func (b memoryBounds) info(vmid int) MemoryInfo {
hostSwap, guestSwap := b.hostSwapMB, b.guestSwapMB
return MemoryInfo{
HostSwapMB: &hostSwap, GuestSwapMB: &guestSwap,
VMID: vmid, AllocatedMB: b.allocatedMB, UsageMB: b.usageMB, HostTotalMB: b.hostTotalMB,
MinMB: b.minMB, MaxMB: b.maxMB, FloorMB: b.floorMB, Running: b.running,
}
@@ -139,6 +147,8 @@ func (s *Server) readMemoryBounds(ctx context.Context, vmid int) (memoryBounds,
hostTotalMB: node.Memory.Total / mib, // bytes → MB, floor (conservative for max)
minMB: minGuestMemoryMB,
running: st.Status == "running",
hostSwapMB: node.Swap.Total / mib, // R-1192 follow-up: the kernel's SwapTotal, as Proxmox reports it
guestSwapMB: cfg.Swap, // PVE config swap is MB
}
b.maxMB = b.hostTotalMB - hostReserveMB
b.floorMB = b.usageMB + shrinkUsageMarginMB
@@ -225,7 +235,8 @@ func (s *Server) handleGuestMemoryResize(w http.ResponseWriter, r *http.Request,
}
// Apply. SetConfig may be synchronous (empty UPID) or return a task to wait on.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target)})
// R-1192: the swap moves with the memory (GuestSwapMB) — a grown guest with the old swap cap thrashes the same way.
upid, err := s.mem.SetConfig(r.Context(), vmid, map[string]string{"memory": fmt.Sprintf("%d", target), "swap": fmt.Sprintf("%d", GuestSwapMB(target))})
if err != nil {
s.logger.Error("local-api: guest-memory SetConfig", "vmid", vmid, "target_mb", target, "err", err)
writeErr(w, http.StatusBadGateway, "memory resize failed: "+err.Error())
+8 -1
View File
@@ -25,12 +25,14 @@ type fakeMemory struct {
setUPID string
setErr error
applyReflect bool
hostSwapB int64 // R-1192 follow-up: node swap total
guestSwapMB int64
}
func (f *fakeMemory) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
f.mu.Lock()
defer f.mu.Unlock()
return proxmox.GuestConfig{Memory: f.allocMB}, nil
return proxmox.GuestConfig{Memory: f.allocMB, Swap: f.guestSwapMB}, nil
}
func (f *fakeMemory) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
f.mu.Lock()
@@ -42,6 +44,7 @@ func (f *fakeMemory) NodeStatus(_ context.Context) (proxmox.NodeStatus, error) {
defer f.mu.Unlock()
var ns proxmox.NodeStatus
ns.Memory.Total = f.hostTotalB
ns.Swap.Total = f.hostSwapB
return ns, nil
}
func (f *fakeMemory) SetConfig(_ context.Context, _ int, params map[string]string) (string, error) {
@@ -151,6 +154,10 @@ func TestGuestMemory_GrowAndShrink(t *testing.T) {
if f.lastParams["memory"] != "12288" {
t.Errorf("SetConfig param = %q, want 12288", f.lastParams["memory"])
}
// R-1192: the swap moves with the memory.
if f.lastParams["swap"] != "12288" {
t.Errorf("SetConfig swap = %q, want 12288 (swap = RAM)", f.lastParams["swap"])
}
})
t.Run("shrink above floor", func(t *testing.T) {
f := stdFixture() // floor = max(2048, 3000+512) = 3512
+73
View File
@@ -0,0 +1,73 @@
package localapi
import (
"context"
"fmt"
"log/slog"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// ── R-1192 (2026-10-21): the guest may swap as much as its RAM ────────────────────────────────────────────────────
//
// MEASURED (felhom.eu audits/fixes-2026-10-21/P4-freeze/): on the minimum machine (8 GB host, guest 4096 MB) the
// household's first uploads made Immich load its analysis models (+1.15 GB) on top of 8 apps' 1.75 GB; the guest hit
// its 4096 MB limit with its swap capped at Proxmox's LXC default of 512 MB — and full. The kernel could then evict
// only file pages: ~360 MB was left for the program code of ~25 containers, re-read from disk without end (150 000–
// 600 000 refaults and ~1 GB of reads per 10 s; memory „full" pressure 92–95 %, load 155, the dashboard dead, 0 OOM
// kills). Raising the cap to 4096 MB live ended it in under a minute (memory „full" 96 % → 1.7 %, dashboard back
// 42 s later) while the uploads ran on. The host had 7.4 GB of swap free the whole time.
//
// The agent never set `swap` at all, so every guest had the 512 MB default. Now:
// - a new guest is created with swap = its memory (reconcile.bringup);
// - a RAM resize sets swap with the memory (handleGuestMemoryResize);
// - at agent start, every LXC in the felhom pool whose swap is below its memory is raised to it (this file).
// Raising swap is a live cgroup change (memory.swap.max), no restart. It never lowers a swap someone set higher.
// GuestSwapMB is the swap a guest of memoryMB gets: the same as its RAM.
func GuestSwapMB(memoryMB int64) int64 { return memoryMB }
// SwapOps is the Proxmox surface EnsureGuestSwap needs (satisfied by *proxmox.Client).
type SwapOps interface {
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
}
// EnsureGuestSwap raises the swap of every LXC guest in pool to GuestSwapMB(memory) when it is below. Returns how
// many guests were raised. One guest's failure is logged and the others are still handled.
func EnsureGuestSwap(ctx context.Context, ops SwapOps, pool string, logger *slog.Logger) (int, error) {
p, err := ops.Pool(ctx, pool)
if err != nil {
return 0, fmt.Errorf("guest swap: read pool %s: %w", pool, err)
}
raised := 0
for _, m := range p.Members {
if m.Type != "lxc" || m.VMID <= 0 {
continue
}
cfg, err := ops.GuestConfig(ctx, m.VMID)
if err != nil {
logger.Warn("guest-swap: config read failed (left as it is)", "vmid", m.VMID, "err", err)
continue
}
want := GuestSwapMB(cfg.Memory)
if cfg.Memory <= 0 || cfg.Swap >= want {
logger.Debug("guest-swap: nothing to do", "vmid", m.VMID, "memory_mb", cfg.Memory, "swap_mb", cfg.Swap)
continue
}
upid, err := ops.SetConfig(ctx, m.VMID, map[string]string{"swap": fmt.Sprintf("%d", want)})
if err == nil && upid != "" {
_, err = ops.WaitTask(ctx, upid, proxmox.WaitOptions{})
}
if err != nil {
logger.Warn("guest-swap: raise failed (left as it is)", "vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want, "err", err)
continue
}
raised++
logger.Info("guest-swap: raised (R-1192 — a guest swap cap below its RAM turned a memory burst into a freeze)",
"vmid", m.VMID, "from_mb", cfg.Swap, "to_mb", want)
}
return raised, nil
}
+71
View File
@@ -0,0 +1,71 @@
package localapi
import (
"context"
"errors"
"io"
"log/slog"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: the freeze measured on the minimum machine (felhom.eu audits/fixes-2026-10-21/P4-freeze/) — a guest whose
// swap is capped below its RAM evicts program code instead of idle memory. EnsureGuestSwap raises every pool guest's
// swap to its RAM, never lowers one, skips non-LXC members, and survives one guest's failure.
// COMPANION RED-PROOF: return 0 before the SetConfig call → „raised 0, want 2".
type fakeSwapOps struct {
cfg map[int]proxmox.GuestConfig
failSet map[int]bool
set map[int]map[string]string
}
func (f *fakeSwapOps) Pool(context.Context, string) (proxmox.PoolInfo, error) {
return proxmox.PoolInfo{Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}, {VMID: 9202, Type: "lxc"}, {VMID: 300, Type: "qemu"},
{Type: "storage"}, {VMID: 9203, Type: "lxc"}, {VMID: 9204, Type: "lxc"}}}, nil
}
func (f *fakeSwapOps) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
c, ok := f.cfg[vmid]
if !ok {
return c, errors.New("no such guest")
}
return c, nil
}
func (f *fakeSwapOps) SetConfig(_ context.Context, vmid int, p map[string]string) (string, error) {
if f.failSet[vmid] {
return "", errors.New("pve said no")
}
if f.set == nil {
f.set = map[int]map[string]string{}
}
f.set[vmid] = p
return "", nil
}
func (f *fakeSwapOps) WaitTask(context.Context, string, proxmox.WaitOptions) (proxmox.TaskStatus, error) {
return proxmox.TaskStatus{}, nil
}
func TestR1192_EnsureGuestSwapRaisesEveryLowSwapToItsRAM(t *testing.T) {
f := &fakeSwapOps{cfg: map[int]proxmox.GuestConfig{
9201: {Memory: 4096, Swap: 512}, // the measured box: raised
9202: {Memory: 2048, Swap: 4096}, // already higher: never lowered
9203: {Memory: 6144, Swap: 0}, // none: raised
9204: {Memory: 4096, Swap: 512}, // the write fails: logged, the others still done
}, failSet: map[int]bool{9204: true}}
n, err := EnsureGuestSwap(context.Background(), f, "felhom", slog.New(slog.NewTextHandler(io.Discard, nil)))
if err != nil {
t.Fatal(err)
}
if n != 2 {
t.Fatalf("raised %d, want 2 (9201, 9203)", n)
}
if f.set[9201]["swap"] != "4096" || f.set[9203]["swap"] != "6144" {
t.Fatalf("writes = %v", f.set)
}
if _, touched := f.set[9202]; touched {
t.Fatal("a swap above the RAM was lowered")
}
if len(f.set[9201]) != 1 {
t.Fatalf("the swap raise wrote more than the swap: %v", f.set[9201])
}
}
+39
View File
@@ -0,0 +1,39 @@
package localapi
import (
"encoding/json"
"testing"
)
// R-1192 follow-up (night 2026-10-21): GET /guest/memory carries the host's swap and this guest's swap allowance, so
// the controller's install page can say when the host has no swap behind the guest. 0 is a real answer (no swap);
// absent means an older agent.
// RED-PROOF: without the two MemoryInfo fields the keys are missing → fails on „host_swap_mb absent".
func TestR1192_GuestMemoryCarriesHostAndGuestSwap(t *testing.T) {
for _, c := range []struct {
name string
hostB, guestMB int64
wantHost, wantG int64
}{
{"installer host (7.75 GB swap), guest swap = RAM", 7935 * mib, 8192, 7935, 8192},
{"a host with NO swap", 0, 8192, 0, 8192},
} {
f := stdFixture()
f.hostSwapB, f.guestSwapMB = c.hostB, c.guestMB
w := do(t, memServer(t, f).Handler(), "GET", "/guest/memory", "A", "")
var m struct {
Data map[string]any `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &m); err != nil {
t.Fatal(err)
}
hs, ok := m.Data["host_swap_mb"].(float64)
if !ok {
t.Fatalf("%s: host_swap_mb absent: %s", c.name, w.Body.String())
}
gs, _ := m.Data["guest_swap_mb"].(float64)
if int64(hs) != c.wantHost || int64(gs) != c.wantG {
t.Fatalf("%s: host %v guest %v, want %d / %d", c.name, hs, gs, c.wantHost, c.wantG)
}
}
}
+3
View File
@@ -645,6 +645,9 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
}
if spec.MemoryMB > 0 {
params["memory"] = strconv.Itoa(spec.MemoryMB)
// R-1192: swap = RAM. Left unset, Proxmox gives an LXC 512 MB, and a guest that fills its RAM can then evict
// only program code — the measured freeze (localapi.GuestSwapMB carries the reasoning and the numbers).
params["swap"] = strconv.Itoa(spec.MemoryMB)
}
for i, m := range spec.Mounts {
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
+19
View File
@@ -0,0 +1,19 @@
package reconcile
import (
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-1192: a new guest is created with swap = its RAM (left unset, Proxmox gives an LXC 512 MB — the measured freeze,
// felhom.eu audits/fixes-2026-10-21/P4-freeze/). COMPANION RED-PROOF: drop params["swap"] in buildBringUpConfig.
func TestR1192_BringUpSetsSwapToTheRAM(t *testing.T) {
p := buildBringUpConfig(BringUpSpec{MemoryMB: 4096}, proxmox.GuestConfig{})
if p["memory"] != "4096" || p["swap"] != "4096" {
t.Fatalf("bring-up params memory=%q swap=%q, want 4096/4096", p["memory"], p["swap"])
}
if q := buildBringUpConfig(BringUpSpec{}, proxmox.GuestConfig{}); q["swap"] != "" {
t.Fatalf("no memory in the spec → no swap written either, got %q", q["swap"])
}
}