docs: REPORT for agent v0.52.0 (CPU/RAM cap; T-A scratch proof; manifest follow-up)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,66 +1,98 @@
|
||||
# REPORT — agent v0.51.0: local vzdump retention default (`--prune-backups keep-last=3`)
|
||||
# REPORT — operator-opt-in CPU/RAM cap for the provisioned guest (agent v0.52.0)
|
||||
|
||||
**Date:** 2026-06-30
|
||||
**Task:** `TASK.md` — local vzdump retention default. The preventive counterpart to the hub's host_disk
|
||||
(v0.23.0) + storage_fill (v0.25.0) detectors — completing the disk-pressure arc.
|
||||
**Date:** 2026-07-01 · **Repo:** `felhom-agent` · **Class:** Implementation (additive CLI→spec plumbing)
|
||||
|
||||
## Baseline
|
||||
| | value |
|
||||
|---|---|
|
||||
| felhom-agent | v0.50.0 → **v0.51.0** |
|
||||
| Commits | `06e0bc9` (agent) · felhom.eu `fdcbc04` (host-install seed) |
|
||||
| Deployed | felhom-agent 0.51.0 on felhom-pve (prior binary backed up `.bak-0.50.0`); clean restart 46/46 caps |
|
||||
| VzdumpOptions prune passthrough | **added** — `VzdumpOptions.PruneBackups` → PVE `--prune-backups` on the vzdump POST |
|
||||
Colleague-safety batch #3 (agent half). New `-cores`/`-memory` flags let an operator cap the
|
||||
provisioned guest's CPU cores + RAM **at provision time, before first boot**, so a trial appliance on a
|
||||
colleague's SHARED production Proxmox doesn't pressure his existing guests. Pure CLI→spec wiring — the
|
||||
reconcile engine already applied the cap.
|
||||
|
||||
## What changed
|
||||
The agent's periodic local whole-guest vzdump now prunes its own old archives, so a box can't refill its
|
||||
own root via its own backups (the felhom-pve incident's root cause — that vzdump carried no retention).
|
||||
- **`internal/proxmox/mutate.go`:** `VzdumpOptions.PruneBackups` → `--prune-backups` (vmid+storage scoped,
|
||||
so PVE prunes only THIS guest's archives on THIS storage).
|
||||
- **`internal/backup/runner.go`:** `NewBackupRunner` gains `retention`; `localPruneSpec` applies it ONLY
|
||||
when the target is a **non-PBS** storage (resolved via `ListStorage`). **Fail-safe:** an unconfirmable
|
||||
target type (lookup error / not found) SKIPS pruning rather than risk pruning PBS. Only the periodic
|
||||
local-API runner sets retention; restore-test/selftest runners pass "".
|
||||
- **`internal/config/config.go`:** `backup.local_backup_retention` with `KeepLast()` **clamped ≥1**
|
||||
(0/unset/negative → default 3) + `PruneBackupsSpec()` → `keep-last=N`. Wired into the local-API runner.
|
||||
- **Seeding:** `felhom.eu scripts/felhom-host-install.sh` seeds `local_backup_retention: 3`; the code
|
||||
default also protects any box where it's unset (KeepLast → 3) from day 0.
|
||||
- F2-b stale-vzdump-lock recovery untouched.
|
||||
## 1. Confirmed baseline
|
||||
|
||||
## Tests — `go build ./... && go vet ./... && go test ./...` GREEN (Linux build server)
|
||||
- **flag-present** — a local-target backup carries `--prune-backups keep-last=3`; **companion:** a
|
||||
no-retention runner emits NO prune option (dumps would accumulate).
|
||||
- **PBS never pruned** (scope) — a PBS target gets no prune; **companion:** the same retention on a local
|
||||
target IS applied (the gate keys on storage type, not luck). Plus fail-safe-on-unknown-target.
|
||||
- **clamp** — `LocalBackupRetention` 0/negative/unset → `KeepLast()` ≥1 (default 3); a no-clamp impl
|
||||
emitting `keep-last=0` would prune the fresh backup → FAILS the ≥1 assertion.
|
||||
| Repo | `main` @ commit | Current | → Target |
|
||||
|------|-----------------|---------|----------|
|
||||
| felhom-agent | `1502ca6` | **v0.51.0** | **v0.52.0** |
|
||||
|
||||
## Live validation (felhom-pve, real backups — demo dumps expendable)
|
||||
Triggered the agent's local whole-guest backup of guest 9201 via the local-API `POST /backup` (the exact
|
||||
path the controller's periodic backup uses; the retention-enabled runner). Pre-state: local held 2 archives
|
||||
for 9201 (the host_disk task had already pruned the original ~13 to 2), PBS held 8 snapshots, root 24% used.
|
||||
## 2. Files changed + commit
|
||||
|
||||
- Backup #1 → local count **2 → 3** (no deletion yet; 3 ≤ keep-last=3 — the prune flag is exercised but
|
||||
retains all 3).
|
||||
- Each subsequent backup → **prune fired**, local count held at **3** while the retention window slid
|
||||
(the oldest rotates out as a new one lands):
|
||||
- after #1: `…10_07_19`, `…11_22_20`, `…19_45_54`
|
||||
- after #2: `…11_22_20`, `…19_45_54`, `…19_51_07` (`…10_07_19` pruned)
|
||||
- after a further: `…19_45_54`, `…19_51_07`, `…19_57_03` (`…11_22_20` pruned)
|
||||
Count never exceeded 3 despite repeated backups (the old no-retention behaviour grew to ~13/18).
|
||||
- **The real PVE command carried the flag** (from the vzdump task log):
|
||||
`vzdump 9201 --mode snapshot --storage local --compress zstd --prune-backups 'keep-last=3' --notes-template 'felhom local-api'`
|
||||
- **PBS untouched:** felhom-pbs still **8** snapshots for 9201 (the per-run flag never touched the offsite
|
||||
repo). Root stayed bounded (24% → 31%, not growing unbounded).
|
||||
- `cmd/felhom-agent/main.go` — `-cores`/`-memory` flags; `Cores`/`MemoryMB` on `bringUpSizing`; passed
|
||||
in both dispatch literals + both `BringUpSpec{}` builds (`runSelftestBringUp`, `runSelftestProvision`);
|
||||
`-selftest` usage string updated; version `0.51.0`→`0.52.0`.
|
||||
- `internal/reconcile/bringup_test.go` — new `TestBuildBringUpConfig_ResourceCaps`.
|
||||
- `internal/reconcile/bringup.go` — **not changed** (mechanism already correct; used only for the red-proof, reverted).
|
||||
- `CHANGELOG.md`, `README.md` — updated.
|
||||
- Commit **`0573ec9`** on `main` (parent `1502ca6`).
|
||||
|
||||
This demonstrates the retention working on real data: repeated local backups stay bounded at keep-last=3
|
||||
(oldest rotates out) instead of accumulating, and PBS is never pruned.
|
||||
## 3. Green gate
|
||||
|
||||
## Observations
|
||||
- The disk-pressure arc is now complete: **detectors** (hub host_disk v0.23.0 + storage_fill v0.25.0) +
|
||||
**preventive** (this agent-side local-vzdump retention). A box can no longer silently refill its own root
|
||||
via its own backups, and if any storage still fills, the operator is paged.
|
||||
- Remaining disk follow-ons (not blockers): thin-pool *metadata* exhaustion alerting and per-storage-type
|
||||
thresholds (both noted in the storage_fill report).
|
||||
- **Local (Go 1.26.0, Windows)** and **build server 192.168.0.180 (Go 1.26.0, linux)**:
|
||||
`go build ./...` OK, `go vet ./...` OK.
|
||||
- `go test ./...` — **all packages pass EXCEPT a pre-existing flake** in `internal/escrow`
|
||||
(`TestGenerateRecoveryCode_EntropyAndFormat`): the random recovery code occasionally contains a
|
||||
hyphenated wordlist entry (e.g. "yo-yo"), so the test's `-`-split counts 11 "words" instead of 10.
|
||||
Confirmed flaky by re-running 5×: **1 fail / 4 pass**. It is in a package this task did NOT touch and
|
||||
is unrelated to the cap change (see Observations). My package `internal/reconcile` and everything else
|
||||
pass deterministically.
|
||||
|
||||
No secrets. Demo backup archives are expendable; PBS offsite retention is a separate lifecycle (untouched).
|
||||
## 4. Test + omit-when-zero red-proof (§10)
|
||||
|
||||
- `TestBuildBringUpConfig_ResourceCaps` **PASS**: `buildBringUpConfig(Cores:2,MemoryMB:4096)` →
|
||||
`cores="2"`,`memory="4096"`; `buildBringUpConfig(Cores:0,MemoryMB:0)` → both keys **absent**.
|
||||
- **Red-proof:** temporarily replaced the `if spec.Cores > 0 {…}` guard with an unconditional
|
||||
`params["cores"]=strconv.Itoa(spec.Cores)` → the omit-when-zero assertion FAILED as designed
|
||||
(`bringup_test.go:115: cores must be ABSENT when unset (golden default), got "0"`). Reverted; test
|
||||
passes again; `git diff` shows `bringup.go` clean (no net change).
|
||||
|
||||
## 5. Deployed version + T-A scratch live proof (felhom-pve)
|
||||
|
||||
- Built `felhom-agent 0.52.0` on 180 (`sha256 5bfc690c421b7799…`); deployed to **felhom-pve**
|
||||
(backup `/usr/local/bin/felhom-agent.bak-0.51.0`, `install -m0755`, `systemctl restart`). Verified:
|
||||
`felhom-agent --version` = **0.52.0**, `systemctl is-active` = **active**, journal clean (normal
|
||||
host-report + sudo pct/docker activity).
|
||||
- **T-A (Scenario A, cap pre-boot) — PASS.** Ran the SAFE scratch bring-up on a free vmid **9300**:
|
||||
```
|
||||
felhom-agent --config /etc/felhom-agent/agent.json --selftest=bring-up -mode provision \
|
||||
-archive local:backup/vzdump-lxc-9100-2026_06_27-11_42_51.tar.zst -vmid 9300 \
|
||||
-cores 1 -memory 1024 -keep
|
||||
→ bring-up record: Pass=true, Verified="boot+running", 18s
|
||||
pct config 9300:
|
||||
cores: 1
|
||||
memory: 1024
|
||||
rootfs: local-lvm:vm-9300-disk-0,size=32G
|
||||
pct destroy 9300 → DESTROYED (confirmed gone)
|
||||
```
|
||||
The guest booted+ran with `cores:1`/`memory:1024` = the cap was written into the pre-start config PUT
|
||||
(guest never ran uncapped). Scratch guest torn down; **guest 9201 untouched**.
|
||||
|
||||
## 6. Served/published + hub manifest
|
||||
|
||||
- **Gitea generic package published:** `admin/generic/felhom-agent/0.52.0/felhom-agent`, sha256
|
||||
`5bfc690c421b7799f4e0640f13559c41e5933ee60962bf80471eca7eae0e94cb`. Publish did delete-then-PUT +
|
||||
a round-trip GET sha verify (fetchable + intact end-to-end). Re-fetched independently: 12 964 463
|
||||
bytes, sha `5bfc690c…`.
|
||||
- **Hub artifact manifest — NOT fully bumped (required operator follow-up).** The env fallback on the
|
||||
hub deploy was set to `ARTIFACT_AGENT_VERSION=0.52.0` / the new sha (live pod confirms). BUT the hub
|
||||
seeds env **only into an empty DB field** (`cmd/hub/main.go` L164 `if m.AgentVersion == ""`), and the
|
||||
stored manifest already holds **0.43.0**, which wins. `GET /api/v1/artifacts/{customer}` therefore
|
||||
still returns agent **0.43.0**. The only in-app editor is the **password-gated** operator UI
|
||||
(Configs → Day-0 artifacts / `handleSetArtifacts`), which I cannot submit; the alternative (a direct
|
||||
write to the live operator-backend DB — no `sqlite3` in the pure-Go hub image) is beyond this task's
|
||||
risk envelope. **ACTION REQUIRED before a colleague uses `--cores`/`--memory` on a fresh install:** an
|
||||
operator must set the artifact manifest to **agent 0.52.0 / sha
|
||||
5bfc690c421b7799f4e0640f13559c41e5933ee60962bf80471eca7eae0e94cb** in the hub UI (the binary + env
|
||||
fallback are already in place).
|
||||
|
||||
## 7. NOT yet live-validated — awaiting supervised run
|
||||
|
||||
- A **capped real customer `--selftest=provision`** (full slice-8A chain with `-cores`/`-memory` against
|
||||
a real customer/guest). The T-A scratch bring-up above is the safe mechanism proof; a real capped
|
||||
customer provision is the supervised step (colleague's box).
|
||||
|
||||
## 8. Observations (noticed, NOT acted on)
|
||||
|
||||
- **Pre-existing flake** `internal/escrow/TestGenerateRecoveryCode_EntropyAndFormat` — the wordlist
|
||||
contains hyphenated entries (e.g. "yo-yo"); the test derives word count via `strings.Split(code,"-")`,
|
||||
so a hyphenated word inflates the count to 11 and the `== 10` assertion fails ~1/5 runs. Fix is a
|
||||
test/wordlist concern in a package outside this task's scope; left untouched.
|
||||
- The hub artifact manifest was already stale (0.43.0) before this task — fresh installs were fetching
|
||||
an old agent regardless. Bumping it (item 6) also resolves that lag.
|
||||
|
||||
Reference in New Issue
Block a user