Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
d4be12ca95 | ||
|
admin
|
7403c2a838 | ||
|
admin
|
309e368731 | ||
|
admin
|
0722b2cdb0 | ||
|
admin
|
4fe2f81a32 | ||
|
admin
|
9bdb4dae8f | ||
|
admin
|
d9864a94bf | ||
|
admin
|
77cd70f7c0 | ||
|
admin
|
1030abd7d6 |
+53
-1
@@ -1,4 +1,56 @@
|
||||
## Unreleased (to become v0.132.0) — a controller that dies slowly is reported, not just restarted (2026-09-17, R-539)
|
||||
## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
|
||||
|
||||
> **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).
|
||||
|
||||
**MinAgent impact:** none required by any controller.
|
||||
|
||||
- **R-685 — the backup space preflight** (`backup/runner.go` `spaceFits`). Before a vzdump to a LOCAL (non-PBS)
|
||||
target, the newest archive of that guest on that target × 1.25 + 1 GiB must be free — PVE prunes old archives
|
||||
only AFTER a successful backup, so the kept ones still stand during the run. A shortfall is a skip: nothing
|
||||
starts, the record's Error reads `skipped: not enough space: <target> has X GiB free; the last archive of guest N
|
||||
was Y GiB, so a new one needs about Z GiB …` (stable prefix `BackupSkipNoSpacePrefix`), and it reaches the
|
||||
controller's tier view and, through the quiesce loop's tier notifier, the operator's `whole_guest_backup_failed`.
|
||||
It FAILS OPEN on what is not known (PBS target, first backup, unreadable usage).
|
||||
- **Free space is read from `GET /nodes/<node>/storage`** (`NodeStorage`), never `GET /storage` — the latter is the
|
||||
cluster DEFINITIONS and carries no usage. **Found live, before release:** the first build read `/storage`,
|
||||
failed open, and a real vzdump of demo-hp 9201 started from the live test; it was aborted after 5 min 16 s, no
|
||||
archive left (`felhom.eu/documentation/audits/night-2026-09-25/F/`). The test fake's `ListStorage` now strips
|
||||
usage like production. Red-proofs: two (`…/F/redproof-r685-*.txt`).
|
||||
- Live proof (demo-hp, safe builds with a hard stop before vzdump): ×10 margin → refused, "local has 14.9 GiB
|
||||
free … needs about 77.2 GiB"; release margin → "space preflight passed" need 11.3 GB, avail 16.0 GB.
|
||||
|
||||
## v0.133.0 — a restore-test can never fill a box's disk; leftovers retried on a timer (2026-09-24, R-672, R-673)
|
||||
|
||||
> **RELEASED 2026-09-24** by `scripts/release-agent.sh` — tag `v0.133.0` (`9bdb4da`), sha256 `3aa303452b8c6be58573d00af01a0ab4a0d97e4f885ffecd0144a18ac24e69b6`, verified by an independent anonymous download. **NOT delivered** (needs an operator-signed `agent_update` job per box, R-530) and **NOT vouched**.
|
||||
|
||||
**MinAgent impact:** none required by any controller. Hub **v0.124.0** makes a thin pool CRITICAL at 90 %
|
||||
(data or metadata) and keys the storage-fill alarm per pool per 6 h; an older hub still raises its generic
|
||||
90/95 % storage-fill events from the same report.
|
||||
|
||||
- **R-672 — the space preflight** (`reconcile/restoretest_space.go`, provider `internal/restorespace`). Before
|
||||
anything is journaled or created, a restore-test needs free data ≥ restored × 1.2 + 5 GiB on its target
|
||||
(`backup.restore_test_space_factor` / `backup.restore_test_space_reserve_gib`) and room in a thin pool's
|
||||
metadata. `restored` is the UNCOMPRESSED size — the vzdump log's "Total bytes written" (a file-backed
|
||||
archive) or the PBS snapshot size; the archive FILE is never used (9201: 6.9 GB file, 22.6 GB restore). The
|
||||
target moves OFF the tested guest's own pool when another storage is eligible (active, `rootdir`, and the
|
||||
agent holds Datastore.AllocateSpace there) and fits. Anything unknown refuses. A refusal is reported to the
|
||||
hub as the test's result (`pass=false`, `skipped=true`, "skipped: not enough space on …") — never a pass,
|
||||
never dropped. The archive config is read once (it was read twice). Measured case: demo-hp 2026-09-24 —
|
||||
the restore-test that filled `local-lvm` would now be refused (needs 32.1 GB, 23.2 GB free).
|
||||
- **R-672 — a failed scratch teardown is retried every 10 minutes** (`Engine.RetryScratchTeardown`, the
|
||||
daemon's janitor), not only by Recover at agent start; never a vmid a running test owns; after 3 failed
|
||||
tries the operator is told through a failed restore-test record naming the scratch guest.
|
||||
- **R-672 — a thin pool crossing 90 % requests an immediate host report** (`Observer.SetThinHighTrigger`, the
|
||||
storage watchdog's read path, every few seconds; re-armed below 85 %), so the hub's alarm sees it in
|
||||
seconds, not at the next 15-minute report.
|
||||
- **R-673 — the stale-lock sweep runs on the same 10-minute timer** (it ran only at start: a stale
|
||||
`snapshot-delete` lock blocked 9201's whole-box backups for five hours), holding the one-heavy-operation
|
||||
gate so no agent backup can start between its "no vzdump running" check and its unlock.
|
||||
- Red-proofs: six, each seen failing (REPORT).
|
||||
|
||||
## v0.132.0 — a controller that dies slowly is reported, not just restarted (2026-09-17, R-539)
|
||||
|
||||
> **RELEASED 2026-09-17** by `scripts/release-agent.sh` — tag `v0.132.0`, sha256 `4afe815749a41b327ebe4a98a4557ad2acffb1fa71740a3a8b8003473b835321`. **Vouched 2026-09-17** for Day-0 installs on the operator's word, together with golden **0.246.0** (the hub's R-120 gate required the newer golden first). Delivered to demo-hp and the N100 by an operator-signed `agent_update` job each (operator ruling 3 of 2026-09-16), not by a floor.
|
||||
|
||||
**MinAgent impact:** none required by any controller. Hub **v0.117.0** turns the new fields into
|
||||
`controller_slow_crashloop`; an older hub ignores them.
|
||||
|
||||
+17
@@ -1,5 +1,22 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
|
||||
> **2026-09-25 night — v0.133.0 AND v0.134.0 DELIVERED to both demo boxes (CC-signed `agent_update`, ruling 1);
|
||||
> restore test back ON (the `-1` config kept as `agent.json.night-0925-off`). v0.134.0 = R-685:** `backup/runner.go`
|
||||
> `spaceFits` — a vzdump to a LOCAL target needs free ≥ newest archive of that guest × 1.25 + 1 GiB (PVE prunes
|
||||
> only after success); a shortfall is a named skip (`BackupSkipNoSpacePrefix`), fail-open on PBS / first backup /
|
||||
> unknown usage. **Free space comes from `NodeStorage` (`GET /nodes/<n>/storage`) — `ListStorage` (`GET /storage`)
|
||||
> has NO usage**; the first build read it and a real vzdump started in its own live test (aborted, no archive).
|
||||
> demo-hp: `local_backup_retention` 1 (operator option A, saved `agent.json.pre-a4-retention`). Peti's box: nothing.
|
||||
|
||||
> **2026-09-24 — v0.133.0 RELEASED, NOT DELIVERED (R-672, R-673).** Restore-test space preflight
|
||||
> (`reconcile/restoretest_space.go`, `internal/restorespace`): uncompressed size from the vzdump log / PBS size,
|
||||
> × 1.2 + 5 GiB, thin metadata, off the tested guest's pool, unknown refuses, reported as `skipped` non-pass.
|
||||
> Janitor (`cmd/felhom-agent/janitor.go`) every 10 min: `Engine.RetryScratchTeardown` + stale-lock sweep under
|
||||
> the heavy-op gate. Thin pool ≥ 90 % → immediate report (hub v0.124.0 alarms). **Operator rulings 2026-09-24
|
||||
> (evening):** the scheduled restore-test is OFF on both demo hosts (`backup.restore_test_eval_interval_seconds:
|
||||
> -1` — note: 0 means the 6 h DEFAULT, only a negative disables) until v0.133.0 is delivered there; saved configs
|
||||
> `/etc/felhom-agent/agent.json.pre-r672`. demo-hp 9201 was repaired (stop, fsck, start; two Redis AOF tails cut).
|
||||
> Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
|
||||
> entry = current); the end-of-task detail lives in `REPORT.md`.
|
||||
|
||||
|
||||
@@ -1,47 +1,17 @@
|
||||
# REPORT — agent v0.131.0: the controller comes back by itself; backup status per tier (2026-09-15)
|
||||
# REPORT — agents v0.133.0 + v0.134.0 delivered; v0.134.0 released (night 2026-09-25)
|
||||
|
||||
Task: *before the volunteer — the big night's P1 fixes*, Parts A and C (agent half). Architecture: `03-host-agent.md`
|
||||
§4 (the "healing a crashed controller" sentence), `07-backup-architecture.md` §6.
|
||||
**Delivered** to demo-felhom and demo-hp by CC-signed `agent_update` jobs (ruling 1, 2026-09-16), each verified
|
||||
against the published package's sha256 first: v0.133.0 (committed 21:35 / 21:50 CEST), then v0.134.0 (22:52 /
|
||||
22:58). Hub reads 0.134.0 on both. Peti's box: nothing. Restore test back ON on both (A3: demo-felhom PASS 85 s;
|
||||
demo-hp refused for space, correctly). demo-hp `local_backup_retention: 1` (operator option A).
|
||||
|
||||
## Measured first (A.1)
|
||||
Docker 29.8.0, throwaway containers on scratch 9202: after `docker kill`, **both** `--restart unless-stopped` and
|
||||
`--restart always` stayed `exited (137)` 60 s later. The task's claim was right; a policy change is not a fix.
|
||||
**v0.134.0 = R-685 (agent half):** a vzdump to a LOCAL target needs free ≥ newest archive × 1.25 + 1 GiB; a
|
||||
shortfall is a named skip before anything starts; fail-open on PBS / first backup / unknown usage. Tag `v0.134.0`
|
||||
(`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download; CI jobs
|
||||
975–977 success. **Not vouched** for Day-0.
|
||||
|
||||
## What shipped
|
||||
- **Controller supervisor** (`internal/localapi/controllersupervisor.go`): every 30 s, for provisioned felhom-pool guests
|
||||
that are running, restart `felhom-controller-bootstrap.service` on the second not-running observation. Guards: swap
|
||||
in flight, host-side park marker, locked / vzdump-busy / stopped guest, unknown docker answer, unprovisioned guest,
|
||||
3 restarts in 15 min → 30 min pause. Record rides the host report as `controller_supervisor`; hub v0.114.0 mints the
|
||||
events. Same GuestExecutor and sudoers grants as the swap — no new privilege.
|
||||
- **Per-tier backup status**: `GET /backup/status` (untargeted) gains `tiers[]` — newest success (record, or storage
|
||||
after a restart), last attempt kept apart, storage presence; `GET /backup/tiers` gains `storage`.
|
||||
- **Golden script**: `--restart always`. No golden baked (R-468); existing boxes keep `unless-stopped`.
|
||||
|
||||
## Red-proofs (each seen failing, then restored)
|
||||
- remove the restart call → `the killed controller was NOT restarted — this is R-523 (restarts=0)`
|
||||
- remove the backoff block → `crash-looping controller restarted 10 times in 10 minutes — want exactly 3`
|
||||
- `last_success` from the newest attempt → `pbs tier reports a failed attempt as its last success`
|
||||
|
||||
## Release and delivery
|
||||
`scripts/release-agent.sh 0.131.0`: tag `v0.131.0`, sha256 `1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c`,
|
||||
verified by download. **The task's "the floor delivers the agent" was wrong** (R-530): the hub holds a floor above the
|
||||
box's agent; agents update only by an operator-signed job. On the operator's keys: `felhom-opsign -op agent_update` for
|
||||
`demo-hp-bb76ea` only → authorized 08:44:16Z, committed 08:45:21Z, `controller-supervisor: started`. demo-felhom and
|
||||
Peti's box stay on 0.130.0.
|
||||
|
||||
## Live validation on demo-hp guest 9201 (agent 0.131.0, controller 0.243.0)
|
||||
| moment | result |
|
||||
|---|---|
|
||||
| idle kill 08:53:27Z | restarted 08:54:22Z; dashboard 200 **59 s** after the kill |
|
||||
| parked + kill | stayed dead 100 s, `the guest is PARKED — leaving it` every sweep; unpark → 200 in **25 s** |
|
||||
| kill 10 s into a swap | `during a controller SWAP — the swap owns it` ×3; the swap rolled back itself, healthy 09:02:06Z |
|
||||
| kill 5 s into a deploy | **not measured**: the three test restarts had filled the budget, so the guard paused (as designed) and the hub mailed `controller_crashloop` |
|
||||
| resume after the pause | pause held to 09:35:52Z; restarted 09:36:24Z; dashboard 200 at 09:36:30Z |
|
||||
Two invalid attempts, both marked in the evidence: a swap POST sent over plain HTTP (400, no swap), and a "health 200"
|
||||
line during the pause that the container state contradicts.
|
||||
|
||||
## Teardown
|
||||
Machine: 9201 back on its controller (see resume); the throwaway homebox deploy left no container. Host: park marker
|
||||
removed; nothing else changed. Hub: demo-hp floor override 0.243.0 kept (it delivers this release).
|
||||
|
||||
Evidence: `felhom.eu/documentation/audits/evidence-p1fixes-2026-09-15/A*`.
|
||||
**Found live before the release:** the first build read free space from `GET /storage` (no usage) and failed open —
|
||||
its own "must refuse" test started a real vzdump of demo-hp 9201, aborted by CC after 5 min 16 s, no archive left.
|
||||
Fixed (`NodeStorage`), test fake made honest (`ListStorage` strips usage), two red-proofs, re-proven live with safe
|
||||
builds (a hard stop before vzdump): ×10 refused with real numbers, ×1.25 passed. Evidence:
|
||||
`felhom.eu/documentation/audits/night-2026-09-25/F/`.
|
||||
|
||||
@@ -85,6 +85,8 @@
|
||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||
| `Client.PoolAddVMID` | internal/proxmox/mutate.go | `PoolAddVMID(ctx, pool, vmid) error` | re-assert pool membership after a restore-over-existing (campaign-2 R2) | SYNC (no UPID, don't WaitTask); PVE `PUT /pools` is additive (merge, not replace) — `delete=1` removes; idempotent (already-member swallowed); needs `Pool.Allocate` at `/pool/<pool>`. `pct restore --pool` sets membership only at CREATE — a restore over an existing vmid drops it, so bring-up re-asserts post-restore |
|
||||
| `reconcile.PreflightRestoreSpace` + `restorespace.Provider` (v0.133.0) | internal/reconcile/restoretest_space.go, internal/restorespace/restorespace.go | `PreflightRestoreSpace(ctx, space, policy, archive, rawCfg, configured) SpaceVerdict` | ANY step that restores or copies a guest onto a storage — size it first | The restored size is UNCOMPRESSED (vzdump log "Total bytes written" / PBS snapshot size) — never the archive FILE (6.9 GB file → 22.6 GB restore, R-672); an unknown refuses; eligibility needs Datastore.AllocateSpace on `/storage/<id>` specifically (the `/` grant answers every path) |
|
||||
| `Engine.RetryScratchTeardown` + the daemon janitor (v0.133.0) | internal/reconcile/restoretest_retry.go, cmd/felhom-agent/janitor.go | `RetryScratchTeardown(ctx) ScratchRetryResult` | retrying a leftover on a TIMER instead of only at start | Never `Recover` on a timer — it also resolves generic in-flight ops; a periodic sweep that unlocks guests holds the one-heavy-op gate (`InFlight.TryAcquire`) |
|
||||
| `TLSConfig.build` / `normalizeFingerprint` | internal/proxmox/tls.go | `build() (*tls.Config, error)` | PVE leaf-cert SHA-256 pinning | No insecure default |
|
||||
| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
|
||||
| `httpx.NewTransport` | internal/httpx/transport.go | `NewTransport(tlsCfg, idleConnTimeout) *http.Transport` | **EVERY** hand-rolled `http.Transport` in this repo — pbs, hub and proxmox all pin TLS, so none can use `http.DefaultTransport` | **R-344: never inline `&http.Transport{TLSClientConfig: ...}` again.** A composite literal takes `IdleConnTimeout` **zero, which means retain idle connections FOREVER** — `http.DefaultTransport` sets 90s and a literal does not inherit it. Combined with a client rebuilt per cycle and dropped (`pbsTargetsFromPVE`), that stranded **388 sockets on ep0 in 46 h**, held open on BOTH sides. `idleConnTimeout <= 0` means **use the default**, never "no timeout". Returns a **FRESH** transport every call — a shared one would pool connections across differently pinned endpoints. Pinned by `internal/pbs/client_leak_test.go` (server-side connection counting) + `internal/httpx/transport_test.go` |
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// janitorInterval is how often the leftovers of an interrupted restore-test or backup are retried
|
||||
// (R-672 rule 3, R-673). Both used to be resolved ONLY at agent start: on 2026-09-24 a failed scratch
|
||||
// teardown kept a full thin pool full for 2.5 h, and a stale `snapshot-delete` lock blocked 9201's
|
||||
// whole-box backups for five hours — each cleared within a minute of an agent restart.
|
||||
const janitorInterval = 10 * time.Minute
|
||||
|
||||
// janitorDeps are the janitor's seams (tests drive one pass with fakes).
|
||||
type janitorDeps struct {
|
||||
retryScratch func(ctx context.Context) reconcile.ScratchRetryResult
|
||||
staleLocks func(ctx context.Context) // localapi Server.RecoverStaleLockedGuests; nil when the local API is off
|
||||
heavy *backup.InFlight
|
||||
record func(hub.RestoreTest)
|
||||
now func() time.Time
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
// janitorPass is one pass. The stale-lock sweep runs only while holding the one-heavy-operation gate, so
|
||||
// no agent backup can START between its "no vzdump is running" check and its unlock (at start-up the
|
||||
// sweep ran before the backup loop existed; on a timer that ordering must be made, not assumed). A busy
|
||||
// gate skips the sweep this pass — the next pass retries.
|
||||
func janitorPass(ctx context.Context, d janitorDeps) {
|
||||
if d.retryScratch != nil {
|
||||
r := d.retryScratch(ctx)
|
||||
if r.Examined > 0 {
|
||||
d.logger.Info("janitor: restore-test scratch retry pass", "examined", r.Examined,
|
||||
"destroyed", r.Destroyed, "already_gone", r.Clean, "failed", r.Failed)
|
||||
}
|
||||
for _, vmid := range r.GaveUp {
|
||||
// The operator is told through the existing restore-test failure path: the hub raises
|
||||
// restore_test_failed (operator) once per distinct archive — this record's archive names
|
||||
// the stuck scratch guest.
|
||||
if d.record != nil {
|
||||
d.record(hub.RestoreTest{
|
||||
SourceArchive: fmt.Sprintf("scratch-teardown:%d", vmid),
|
||||
ScratchVMID: vmid,
|
||||
Pass: false,
|
||||
Error: fmt.Sprintf("restore-test scratch guest %d could not be torn down after %d retries — it holds its disks; remove it by hand (pct destroy %d) after checking what keeps it busy",
|
||||
vmid, reconcile.MaxTeardownTries, vmid),
|
||||
TestedAt: d.now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
if d.staleLocks != nil {
|
||||
release, busy, ok := d.heavy.TryAcquire("stale-lock-sweep")
|
||||
if !ok {
|
||||
d.logger.Info("janitor: stale-lock sweep deferred — a heavy operation is in flight", "busy", busy)
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
d.staleLocks(ctx)
|
||||
}
|
||||
}
|
||||
|
||||
// runJanitor runs janitorPass every janitorInterval until ctx ends.
|
||||
func runJanitor(ctx context.Context, d janitorDeps) {
|
||||
d.logger.Info("janitor: starting (restore-test scratch retry + stale-lock sweep)", "interval", janitorInterval)
|
||||
t := time.NewTicker(janitorInterval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
janitorPass(ctx, d)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// R-672 / R-673 (v0.133.0): one janitor pass, driven with fakes.
|
||||
|
||||
func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
|
||||
|
||||
// A scratch the engine gave up on reaches the hub as a failed restore-test record naming it — the
|
||||
// existing operator path (restore_test_failed). Never a pass.
|
||||
func TestJanitor_GaveUpIsReportedAsAFailure(t *testing.T) {
|
||||
var got []hub.RestoreTest
|
||||
janitorPass(context.Background(), janitorDeps{
|
||||
retryScratch: func(context.Context) reconcile.ScratchRetryResult {
|
||||
return reconcile.ScratchRetryResult{Examined: 1, Failed: 1, GaveUp: []int{990000}}
|
||||
},
|
||||
heavy: &backup.InFlight{}, record: func(r hub.RestoreTest) { got = append(got, r) },
|
||||
now: time.Now, logger: quiet(),
|
||||
})
|
||||
if len(got) != 1 || got[0].Pass || got[0].ScratchVMID != 990000 || !strings.Contains(got[0].Error, "990000") {
|
||||
t.Fatalf("records = %+v — want one FAILED record naming scratch 990000", got)
|
||||
}
|
||||
}
|
||||
|
||||
// R-673: the stale-lock sweep runs only while holding the one-heavy-operation gate, so no agent backup can
|
||||
// start between its "no vzdump running" check and its unlock.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT): drop the TryAcquire → "the sweep ran while a backup held the gate".
|
||||
func TestJanitor_StaleLockSweepWaitsForTheHeavyGate(t *testing.T) {
|
||||
heavy := &backup.InFlight{}
|
||||
swept := 0
|
||||
d := janitorDeps{staleLocks: func(context.Context) { swept++ }, heavy: heavy, now: time.Now, logger: quiet()}
|
||||
release, _, ok := heavy.TryAcquire("backup")
|
||||
if !ok {
|
||||
t.Fatal("setup")
|
||||
}
|
||||
janitorPass(context.Background(), d)
|
||||
if swept != 0 {
|
||||
t.Fatal("the sweep ran while a backup held the gate")
|
||||
}
|
||||
release()
|
||||
janitorPass(context.Background(), d)
|
||||
if swept != 1 {
|
||||
t.Fatalf("swept %d times with the gate free — want 1", swept)
|
||||
}
|
||||
if _, _, ok := heavy.TryAcquire("after"); !ok {
|
||||
t.Fatal("the sweep did not release the gate")
|
||||
}
|
||||
}
|
||||
@@ -50,6 +50,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfheal"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/selfupdate"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
@@ -897,6 +898,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
// it finds nothing to flag. The re-mount dispatch is off the poll path (a goroutine).
|
||||
storageTrigger := make(chan struct{}, 1)
|
||||
loop.SetTrigger(storageTrigger)
|
||||
// R-672: a thin pool crossing 90 % requests a report at once (the hub's storage-fill alarm).
|
||||
observer.SetThinHighTrigger(func() {
|
||||
select {
|
||||
case storageTrigger <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
})
|
||||
remounter := &gateRemounter{gate: gate, ops: hostOps, hostID: cfg.Hub.HostID, logger: logger}
|
||||
// Drive intent store (slice 10 P3 self-heal): persisted, durable-id-keyed enroll/eject/decommission
|
||||
// state. Gates the watchdog's self-heal re-mount to ENROLLED drives, and the local API records
|
||||
@@ -963,6 +971,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
Logger: logger,
|
||||
})
|
||||
|
||||
rtSpace, rtPolicy := restoreSpaceFor(cfg, px, hostOps)
|
||||
engine := reconcile.NewEngine(reconcile.EngineOptions{
|
||||
API: px,
|
||||
Queue: queue,
|
||||
@@ -971,6 +980,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
Gate: gate,
|
||||
HostID: cfg.Hub.HostID,
|
||||
Logger: logger,
|
||||
// R-672: the restore-test's space preflight (nil would refuse every test — fail-closed).
|
||||
RestoreSpace: rtSpace,
|
||||
SpacePolicy: rtPolicy,
|
||||
})
|
||||
|
||||
// Crash recovery (doc 03 §10): resolve any op that was in flight when the agent
|
||||
@@ -1407,6 +1419,16 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
go localSrv.WatchControllers(ctx)
|
||||
go func() { errc <- localSrv.Run(ctx) }()
|
||||
}
|
||||
// R-672 / R-673: retry a failed restore-test teardown and sweep stale backup locks on a timer, not
|
||||
// only at start-up (janitor.go).
|
||||
{
|
||||
jd := janitorDeps{retryScratch: engine.RetryScratchTeardown, heavy: heavyOps,
|
||||
record: backupStore.RecordRestoreTest, now: time.Now, logger: logger}
|
||||
if localSrv != nil {
|
||||
jd.staleLocks = localSrv.RecoverStaleLockedGuests
|
||||
}
|
||||
go runJanitor(ctx, jd)
|
||||
}
|
||||
if lanLoop != nil {
|
||||
lanServers = 1
|
||||
go func() { errc <- lanLoop.Run(ctx) }()
|
||||
@@ -2209,6 +2231,17 @@ func formatOrDash(t time.Time) string {
|
||||
return t.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
// restoreSpaceFor builds the restore-test's space preflight (R-672): the production provider over the
|
||||
// Proxmox API + the privileged `lvs` metadata read, and the configured margin.
|
||||
func restoreSpaceFor(cfg config.Config, px *proxmox.Client, ops *storage.SudoHostOps) (reconcile.RestoreSpace, reconcile.SpacePolicy) {
|
||||
factor, reserve := cfg.Backup.RestoreTestSpace()
|
||||
p := &restorespace.Provider{API: px}
|
||||
if ops != nil {
|
||||
p.ThinMeta = ops.ThinPoolMetadata
|
||||
}
|
||||
return p, reconcile.SpacePolicy{Factor: factor, ReserveBytes: reserve}
|
||||
}
|
||||
|
||||
func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog.Logger, archive string) int {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
||||
@@ -2239,8 +2272,10 @@ func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog
|
||||
}
|
||||
}
|
||||
gate := reconcile.NewGate(nil, cfg.Hub.HostID, reconcile.SlogAudit{Logger: logger}, logger)
|
||||
rtSpace, rtPolicy := restoreSpaceFor(cfg, px, newHostOps(cfg, logger))
|
||||
engine := reconcile.NewEngine(reconcile.EngineOptions{
|
||||
API: px, Queue: queue, Journal: journal, Gate: gate, HostID: cfg.Hub.HostID, Logger: logger,
|
||||
RestoreSpace: rtSpace, SpacePolicy: rtPolicy,
|
||||
})
|
||||
|
||||
fmt.Printf("=== felhom-agent %s selftest=restore-test ===\n", version)
|
||||
@@ -2270,10 +2305,16 @@ func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog
|
||||
RestoreTaskTimeout: restoreTaskTimeout(cfg, rtTier),
|
||||
})
|
||||
printJSON("restore-test record", backup.ToHubRestoreTest(res, time.Now().UTC()))
|
||||
if res.Skipped && res.SkipReason != "" {
|
||||
fmt.Printf(" space preflight: storage=%s required=%d avail=%d\n", res.TargetStorage, res.RequiredBytes, res.AvailBytes)
|
||||
fmt.Printf("=== selftest=restore-test SKIPPED — %s ===\n", res.SkipReason)
|
||||
return 4
|
||||
}
|
||||
if res.Skipped {
|
||||
fmt.Println("=== selftest=restore-test SKIPPED (no free scratch VMID in band) ===")
|
||||
return 0
|
||||
}
|
||||
fmt.Printf(" space preflight passed: storage=%s required=%d avail=%d\n", res.TargetStorage, res.RequiredBytes, res.AvailBytes)
|
||||
if res.Err != nil || !res.Pass {
|
||||
fmt.Fprintf(os.Stderr, " [FAIL] restore-test (scratch %d): %v\n", res.ScratchVMID, res.Err)
|
||||
return 1
|
||||
|
||||
@@ -24,11 +24,13 @@ type fakeBackupAPI struct {
|
||||
cfgErr error
|
||||
content []proxmox.StorageContent
|
||||
contentErr error
|
||||
storages []proxmox.Storage // returned by ListStorage (the local-prune scope gate)
|
||||
storageErr error
|
||||
vzdumps []proxmox.VzdumpOptions
|
||||
logLines []string // returned by TaskLogTail (e.g. "INFO: backup mode: stop")
|
||||
waitGate chan struct{} // if non-nil, WaitTask blocks until closed (8B.2 watcher timing)
|
||||
storages []proxmox.Storage // returned by ListStorage (the local-prune scope gate) — DEFINITIONS: like
|
||||
// production's GET /storage, it never carries usage; ListStorage strips Avail/Used (R-685's live lesson)
|
||||
nodeStorages []proxmox.Storage // returned by NodeStorage (GET /nodes/{node}/storage — WITH usage)
|
||||
storageErr error
|
||||
vzdumps []proxmox.VzdumpOptions
|
||||
logLines []string // returned by TaskLogTail (e.g. "INFO: backup mode: stop")
|
||||
waitGate chan struct{} // if non-nil, WaitTask blocks until closed (8B.2 watcher timing)
|
||||
}
|
||||
|
||||
func (f *fakeBackupAPI) Vzdump(_ context.Context, o proxmox.VzdumpOptions) (string, error) {
|
||||
@@ -48,6 +50,17 @@ func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.S
|
||||
return f.content, f.contentErr
|
||||
}
|
||||
func (f *fakeBackupAPI) ListStorage(_ context.Context) ([]proxmox.Storage, error) {
|
||||
out := make([]proxmox.Storage, len(f.storages))
|
||||
for i, s := range f.storages {
|
||||
s.Avail, s.Used, s.Total = 0, 0, 0 // GET /storage has no usage — a fake that had it hid R-685's defect
|
||||
out[i] = s
|
||||
}
|
||||
return out, f.storageErr
|
||||
}
|
||||
func (f *fakeBackupAPI) NodeStorage(_ context.Context) ([]proxmox.Storage, error) {
|
||||
if f.nodeStorages != nil {
|
||||
return f.nodeStorages, f.storageErr
|
||||
}
|
||||
return f.storages, f.storageErr
|
||||
}
|
||||
func (f *fakeBackupAPI) TaskLogTail(_ context.Context, _ string, _ int) ([]string, error) {
|
||||
@@ -137,13 +150,14 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
|
||||
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
|
||||
const big = 4 << 30 // a plausible whole-guest archive
|
||||
api := &fakeBackupAPI{content: []proxmox.StorageContent{
|
||||
{VolID: "a", Content: "backup", CTime: 10, Size: big},
|
||||
{VolID: "b", Content: "backup", CTime: 99, Size: big},
|
||||
// R-689: real vzdump names with their vmid — only a backup OF A GUEST is a candidate.
|
||||
{VolID: "local:backup/vzdump-lxc-9001-a.tar.zst", VMID: 9001, Content: "backup", CTime: 10, Size: big},
|
||||
{VolID: "local:backup/vzdump-lxc-9001-b.tar.zst", VMID: 9001, Content: "backup", CTime: 99, Size: big},
|
||||
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
|
||||
}}
|
||||
r := NewBackupRunner(api, "local", "", "", "", quiet())
|
||||
vol, err := r.PickRestoreCandidate(context.Background())
|
||||
if err != nil || vol != "b" {
|
||||
if err != nil || vol != "local:backup/vzdump-lxc-9001-b.tar.zst" {
|
||||
t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
|
||||
}
|
||||
// no backups → "".
|
||||
@@ -163,12 +177,12 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
|
||||
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
|
||||
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
|
||||
api := &fakeBackupAPI{content: []proxmox.StorageContent{
|
||||
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
|
||||
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
|
||||
{VolID: "felhom-pbs:backup/ct/9001/real", VMID: 9001, Content: "backup", CTime: 10, Size: 4 << 30},
|
||||
{VolID: "felhom-pbs:backup/ct/9001/phantom", VMID: 9001, Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
|
||||
}}
|
||||
r := NewBackupRunner(api, "local", "", "", "", quiet())
|
||||
vol, err := r.PickRestoreCandidate(context.Background())
|
||||
if err != nil || vol != "real" {
|
||||
if err != nil || vol != "felhom-pbs:backup/ct/9001/real" {
|
||||
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// R-672 (v0.133.0): a restore-test the SPACE preflight refused is the test's RESULT — recorded for the
|
||||
// hub as pass=false with the reason, never dropped (a band skip still is) and never a pass.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT): the scheduler's pre-v0.133.0 `if res.Skipped { return }` → "a space
|
||||
// refusal never reached the host report".
|
||||
func TestR672_SpaceSkipIsReportedNotDropped(t *testing.T) {
|
||||
store := NewStore()
|
||||
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Skipped: true,
|
||||
SkipReason: "skipped: not enough space on local-lvm: restoring 21.1 GiB (vzdump log) needs 30.3 GiB free, has 21.6 GiB"}}
|
||||
s := NewScheduler(SchedulerOptions{
|
||||
Runner: rt, Pick: func(context.Context) (string, error) { return "vol", nil }, Store: store,
|
||||
Spec: func(context.Context, string) reconcile.RestoreTestSpec {
|
||||
return reconcile.RestoreTestSpec{RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009}
|
||||
},
|
||||
Cadence: time.Hour, Logger: quiet(),
|
||||
})
|
||||
s.tick(context.Background())
|
||||
got := store.RestoreTests(context.Background())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("a space refusal never reached the host report: %+v", got)
|
||||
}
|
||||
if got[0].Pass || !got[0].Skipped || !strings.HasPrefix(got[0].Error, "skipped: not enough space") {
|
||||
t.Fatalf("record = %+v — want pass=false, skipped, the reason as the error", got[0])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-685 (v0.134.0) — a whole-box backup that cannot fit its LOCAL target is a named skip BEFORE anything
|
||||
// starts, with the numbers in the record's Error, never a vzdump that fills the disk and fails.
|
||||
|
||||
const gib = int64(1) << 30
|
||||
|
||||
func spaceAPI(avail int64, lastArchive int64, typ string) *fakeBackupAPI {
|
||||
api := &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1",
|
||||
storages: []proxmox.Storage{{Storage: "local", Type: typ, Content: "backup", Avail: avail}}}
|
||||
if lastArchive > 0 {
|
||||
api.content = []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst",
|
||||
Content: "backup", VMID: 9201, Size: lastArchive, CTime: 1790280000}}
|
||||
}
|
||||
return api
|
||||
}
|
||||
|
||||
// TestR685_BackupThatCannotFitIsSkipped — demo-hp's shape: an 8.2 GB archive, 4 GiB free. No vzdump is
|
||||
// started; the record says why, with the numbers, under a stable prefix.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT.md): drop the spaceFits call from backup() — this test fails at "a vzdump
|
||||
// was started on a target that cannot hold it".
|
||||
func TestR685_BackupThatCannotFitIsSkipped(t *testing.T) {
|
||||
api := spaceAPI(4*gib, 8182759056, "dir")
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
rec, err := r.Backup(context.Background(), 9201)
|
||||
if len(api.vzdumps) != 0 {
|
||||
t.Fatalf("a vzdump was started on a target that cannot hold it: %+v", api.vzdumps)
|
||||
}
|
||||
if err == nil || rec.Success || !strings.HasPrefix(rec.Error, BackupSkipNoSpacePrefix) {
|
||||
t.Fatalf("want a named skip, got err=%v rec=%+v", err, rec)
|
||||
}
|
||||
for _, want := range []string{"4.0 GiB free", "7.6 GiB", "10.5 GiB"} {
|
||||
if !strings.Contains(rec.Error, want) {
|
||||
t.Errorf("the reason must carry the numbers (%q missing): %s", want, rec.Error)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestR685_BackupThatFitsRuns — the same archive with 16 GiB free (demo-hp after tonight's prune) runs.
|
||||
func TestR685_BackupThatFitsRuns(t *testing.T) {
|
||||
api := spaceAPI(16*gib, 8182759056, "dir")
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
_, _ = r.Backup(context.Background(), 9201)
|
||||
if len(api.vzdumps) != 1 {
|
||||
t.Fatalf("a backup that fits must run, vzdumps=%d", len(api.vzdumps))
|
||||
}
|
||||
}
|
||||
|
||||
// TestR685_FailsOpen — never refuse on what is not KNOWN: a PBS target, a first backup (no archive to
|
||||
// size from), an unknown free figure, or a storage list that cannot be read.
|
||||
func TestR685_FailsOpen(t *testing.T) {
|
||||
cases := map[string]*fakeBackupAPI{
|
||||
"pbs target": spaceAPI(1*gib, 8*gib, "pbs"),
|
||||
"first backup": spaceAPI(1*gib, 0, "dir"),
|
||||
"avail unknown": spaceAPI(0, 8*gib, "dir"),
|
||||
"storage list error": func() *fakeBackupAPI {
|
||||
a := spaceAPI(1*gib, 8*gib, "dir")
|
||||
a.storageErr = context.DeadlineExceeded
|
||||
return a
|
||||
}(),
|
||||
}
|
||||
for name, api := range cases {
|
||||
target := "local"
|
||||
if name == "pbs target" {
|
||||
api.storages[0].Storage = "felhom-pbs"
|
||||
target = "felhom-pbs"
|
||||
}
|
||||
r := NewBackupRunner(api, target, proxmox.ModeSnapshot, "", "", quiet())
|
||||
_, _ = r.Backup(context.Background(), 9201)
|
||||
if len(api.vzdumps) != 1 {
|
||||
t.Errorf("%s: the preflight must fail OPEN, but no vzdump ran", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-689 (v0.135.0) — demo-hp keeps its golden template in `local:backup/`. It is content "backup",
|
||||
// 654 MB and so "plausibly complete", and it was the newest SETTLED entry: the restore test picked it
|
||||
// every 6 h and failed extractconfig with a 403 (measured 2026-09-24 10:36, 09-25 04:57 and 10:57),
|
||||
// while the guest's own archive — younger than the 24 h settle — went untested and nothing was proven.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT.md): drop the guestBackupArchive call from PickSettledRestoreCandidateOn —
|
||||
// this test then picks `local:backup/felhom-golden-0.236.0.tar.zst`.
|
||||
func TestR689_TheRestoreTestNeverPicksTheGolden(t *testing.T) {
|
||||
const day = int64(86400)
|
||||
now := int64(1790370000) // 2026-09-25 ~19:00Z
|
||||
api := &fakeBackupAPI{content: []proxmox.StorageContent{
|
||||
// the guest's real archive, settled (older than the cutoff below)
|
||||
{VolID: "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 3*day},
|
||||
// the golden: newer, settled, big, and NOT a backup of a guest
|
||||
{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 2*day},
|
||||
// a hand-copied tarball that PVE happens to attribute to a vmid — the name is not a vzdump's
|
||||
{VolID: "local:backup/copy-of-9201.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 2*day},
|
||||
}}
|
||||
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
|
||||
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst" {
|
||||
t.Fatalf("picked %q — the restore test must prove a backup OF A GUEST", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestR689_GuestBackupArchiveShapes(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
e proxmox.StorageContent
|
||||
ok bool
|
||||
}{
|
||||
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst", VMID: 9201}, true},
|
||||
{proxmox.StorageContent{VolID: "local:backup/vzdump-qemu-300-2026_09_24-21_59_25.vma.zst", VMID: 300}, true},
|
||||
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z", VMID: 9201}, true},
|
||||
{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
|
||||
{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
|
||||
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
|
||||
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
|
||||
} {
|
||||
if ok, why := guestBackupArchive(c.e); ok != c.ok {
|
||||
t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -22,6 +23,10 @@ type BackupAPI interface {
|
||||
StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error)
|
||||
// ListStorage enumerates storages (name+type) — used to scope local-only retention (never prune PBS).
|
||||
ListStorage(ctx context.Context) ([]proxmox.Storage, error)
|
||||
// NodeStorage is GET /nodes/{node}/storage — the storages WITH live usage (avail/used). R-685's space
|
||||
// preflight reads free space HERE: ListStorage (GET /storage) is the cluster DEFINITIONS and carries no
|
||||
// usage at all — measured live 2026-09-24 on demo-hp, where reading it let a backup through.
|
||||
NodeStorage(ctx context.Context) ([]proxmox.Storage, error)
|
||||
// TaskLogTail reads trailing task-log lines — used to read the ACTUAL vzdump mode
|
||||
// (PVE may downgrade a requested snapshot to stop for a stopped guest — spike B1).
|
||||
TaskLogTail(ctx context.Context, upid string, limit int) ([]string, error)
|
||||
@@ -170,6 +175,16 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
rec.UncoveredVolumes = []string{}
|
||||
}
|
||||
|
||||
// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
|
||||
// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
|
||||
// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
|
||||
if ok, why := r.spaceFits(ctx, vmid); !ok {
|
||||
rec.Error = BackupSkipNoSpacePrefix + why
|
||||
rec.DurationSeconds = time.Since(start).Seconds()
|
||||
r.logger.Warn("backup SKIPPED by the space preflight (R-685) — nothing was started", "vmid", vmid, "target", r.target, "reason", why)
|
||||
return rec, fmt.Errorf("backup: %s", rec.Error)
|
||||
}
|
||||
|
||||
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
|
||||
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
|
||||
PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
|
||||
@@ -217,6 +232,56 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
|
||||
return rec, nil
|
||||
}
|
||||
|
||||
// BackupSkipNoSpacePrefix starts a backup record's Error when the space preflight refused (R-685) — a
|
||||
// stable prefix the controller's page and the hub can key on.
|
||||
const BackupSkipNoSpacePrefix = "skipped: not enough space: "
|
||||
|
||||
// Space preflight margins (R-685): the new archive is predicted as the newest archive of this guest on the
|
||||
// target × backupSpaceGrowth, plus backupSpaceFloorBytes of headroom for the host. MEASURED 2026-09-24:
|
||||
// demo-hp 9201's archives grew 5.8 → 6.2 → 6.9 → 7.6 GB in four nights (+10 % a night at worst), so 1.25
|
||||
// covers two nights' growth. PVE prunes old archives only AFTER a successful backup, so the free space
|
||||
// must hold the new archive while every kept one still exists.
|
||||
const (
|
||||
backupSpaceGrowth = 1.25
|
||||
backupSpaceFloorBytes = int64(1) << 30
|
||||
)
|
||||
|
||||
// spaceFits answers whether a new archive of vmid fits on a LOCAL (non-PBS) target. It FAILS OPEN — a
|
||||
// backup is the thing being protected, so an unreadable storage, an unknown type or a first backup (no
|
||||
// previous archive to size from) proceeds and says so; only a POSITIVE "it does not fit" refuses.
|
||||
func (r *BackupRunner) spaceFits(ctx context.Context, vmid int) (bool, string) {
|
||||
// NodeStorage, never ListStorage: only the node view carries avail (see BackupAPI.NodeStorage).
|
||||
stores, err := r.api.NodeStorage(ctx)
|
||||
if err != nil {
|
||||
r.logger.Warn("backup: space preflight could not read storage usage — proceeding (fail-open)", "target", r.target, "err", err)
|
||||
return true, ""
|
||||
}
|
||||
var st *proxmox.Storage
|
||||
for i := range stores {
|
||||
if stores[i].Storage == r.target {
|
||||
st = &stores[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if st == nil || st.Type == "pbs" || st.Avail <= 0 {
|
||||
return true, "" // PBS dedups and has its own lifecycle; an unknown avail never refuses
|
||||
}
|
||||
_, last, err := r.latestArchive(ctx, vmid)
|
||||
if err != nil || last <= 0 {
|
||||
r.logger.Info("backup: space preflight has no previous archive to size from — proceeding", "vmid", vmid, "target", r.target)
|
||||
return true, ""
|
||||
}
|
||||
need := int64(float64(last)*backupSpaceGrowth) + backupSpaceFloorBytes
|
||||
if st.Avail >= need {
|
||||
r.logger.Info("backup: space preflight passed", "vmid", vmid, "target", r.target, "last_archive_bytes", last, "need_bytes", need, "avail_bytes", st.Avail)
|
||||
return true, ""
|
||||
}
|
||||
return false, fmt.Sprintf("%s has %s free; the last archive of guest %d was %s, so a new one needs about %s (old archives are removed only after a successful backup)",
|
||||
r.target, humanGiB(st.Avail), vmid, humanGiB(last), humanGiB(need))
|
||||
}
|
||||
|
||||
func humanGiB(b int64) string { return fmt.Sprintf("%.1f GiB", float64(b)/(1<<30)) }
|
||||
|
||||
// watchForSnapshot polls the running backup's task log until it sees the storage-snapshot marker
|
||||
// (→ onSnapshot once) or the requested mode is reported as `stop` (→ downgraded; the marker will
|
||||
// never come, so stop watching) or ctx is cancelled (backup finished). Best-effort: a log-read
|
||||
@@ -298,6 +363,14 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
|
||||
if e.Content != "backup" {
|
||||
continue
|
||||
}
|
||||
// R-689 (v0.135.0): only a backup OF A GUEST is a restore-test candidate. demo-hp keeps its golden
|
||||
// template in `local:backup/` — content "backup", 654 MB, plausibly complete — and it was picked as
|
||||
// the newest settled archive every 6 h and failed extractconfig (403) each time, while the guest's
|
||||
// real archive went untested.
|
||||
if ok, why := guestBackupArchive(e); !ok {
|
||||
r.noteNotAGuestBackupOnce(e, why)
|
||||
continue
|
||||
}
|
||||
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
|
||||
continue // not settled yet — a newer archive is not a reason to re-prove an older one
|
||||
}
|
||||
@@ -520,5 +593,53 @@ func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.R
|
||||
if res.Err != nil {
|
||||
rt.Error = res.Err.Error()
|
||||
}
|
||||
if res.SkipReason != "" { // R-672: the space preflight refused — reported, never a pass
|
||||
rt.Pass = false
|
||||
rt.Skipped = true
|
||||
rt.Error = res.SkipReason
|
||||
}
|
||||
return rt
|
||||
}
|
||||
|
||||
// guestBackupArchive reports whether a storage entry is a whole-guest backup of a known guest — a
|
||||
// `vzdump-<type>-<vmid>-…` file on a dir storage, or a `backup/{ct,vm}/<vmid>/<time>` snapshot on a PBS
|
||||
// datastore — whose vmid the storage itself reports. Anything else in a backup content type (a golden
|
||||
// template, a hand-copied tarball) is not a backup of a guest and is never restore-tested (R-689).
|
||||
// Pure, so the rule is unit-tested without a storage.
|
||||
func guestBackupArchive(e proxmox.StorageContent) (bool, string) {
|
||||
if e.VMID <= 0 {
|
||||
return false, "not a backup of a guest (the storage reports no vmid)"
|
||||
}
|
||||
vol := e.VolID
|
||||
if i := strings.Index(vol, ":"); i >= 0 {
|
||||
vol = vol[i+1:]
|
||||
}
|
||||
vol = strings.TrimPrefix(vol, "backup/")
|
||||
vmid := strconv.Itoa(e.VMID)
|
||||
switch {
|
||||
case strings.HasPrefix(vol, "vzdump-lxc-"+vmid+"-"), strings.HasPrefix(vol, "vzdump-qemu-"+vmid+"-"):
|
||||
return true, ""
|
||||
case strings.HasPrefix(vol, "ct/"+vmid+"/"), strings.HasPrefix(vol, "vm/"+vmid+"/"):
|
||||
return true, ""
|
||||
}
|
||||
return false, "not a vzdump archive or a PBS snapshot of guest " + vmid
|
||||
}
|
||||
|
||||
// noteNotAGuestBackupOnce logs, once per volid, that a backup-content entry is not a restore-test
|
||||
// candidate because it is not a backup of a guest (R-689). INFO, not WARN: a golden template kept in
|
||||
// the backup directory is the operator's, and not a fault.
|
||||
func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why string) {
|
||||
r.rejectedMu.Lock()
|
||||
if r.rejected == nil {
|
||||
r.rejected = map[string]struct{}{}
|
||||
}
|
||||
_, seen := r.rejected[e.VolID]
|
||||
if !seen {
|
||||
r.rejected[e.VolID] = struct{}{}
|
||||
}
|
||||
r.rejectedMu.Unlock()
|
||||
if !seen {
|
||||
r.logger.Info("backup: restore-test skips an entry that is not a backup of a guest",
|
||||
"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -208,9 +208,11 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
spec := s.spec(ctx, archive)
|
||||
spec.Archive = archive
|
||||
res := s.runner.RunRestoreTest(ctx, spec)
|
||||
if res.Skipped {
|
||||
if res.Skipped && res.SkipReason == "" {
|
||||
return // already logged by the engine (no free scratch VMID)
|
||||
}
|
||||
// R-672: a SPACE refusal is the test's result — reported (pass=false, the reason as the error),
|
||||
// never dropped and never a pass. It earns no rotation credit, so the tier stays due.
|
||||
rt := ToHubRestoreTest(res, s.now())
|
||||
s.store.RecordRestoreTest(rt)
|
||||
// Rotation credit is given ONLY on success. A failing tier must keep sorting first, or a tier
|
||||
|
||||
@@ -345,6 +345,12 @@ type BackupConfig struct {
|
||||
// between an archive settling and its proof, and the retry rate of a tier whose restore-test
|
||||
// keeps failing. See defaultRestoreTestEvalInterval for the measurement it was chosen from.
|
||||
RestoreTestEvalIntervalSeconds int `json:"restore_test_eval_interval_seconds"`
|
||||
// RestoreTestSpaceFactor / RestoreTestSpaceReserveGiB are the restore-test's space margin (R-672,
|
||||
// v0.133.0): a test starts only when the target storage has free ≥ restored × factor + reserve,
|
||||
// `restored` being the UNCOMPRESSED size. 0/unset → 1.2 and 5 GiB. A test config may raise them to
|
||||
// watch the refusal (the brief's live case a).
|
||||
RestoreTestSpaceFactor float64 `json:"restore_test_space_factor,omitempty"`
|
||||
RestoreTestSpaceReserveGiB float64 `json:"restore_test_space_reserve_gib,omitempty"`
|
||||
// RestoreTestSettleSeconds is how long an archive must have sat on its tier before it is a
|
||||
// restore-test candidate (R-86); 0 → default (24h), negative → 0 (no settle requirement).
|
||||
// Restore-testing an archive a backup is still writing proves nothing about the backup that
|
||||
@@ -615,6 +621,20 @@ func (b BackupConfig) RestoreTestEvalInterval() time.Duration {
|
||||
}
|
||||
}
|
||||
|
||||
// RestoreTestSpace returns the restore-test's space margin (R-672): factor (≥ 1) and reserve bytes.
|
||||
// Unset or out-of-range → 1.2 and 5 GiB.
|
||||
func (b BackupConfig) RestoreTestSpace() (factor float64, reserveBytes int64) {
|
||||
factor = b.RestoreTestSpaceFactor
|
||||
if factor < 1 {
|
||||
factor = 1.2
|
||||
}
|
||||
reserveBytes = int64(b.RestoreTestSpaceReserveGiB * float64(1<<30))
|
||||
if reserveBytes <= 0 {
|
||||
reserveBytes = 5 << 30
|
||||
}
|
||||
return factor, reserveBytes
|
||||
}
|
||||
|
||||
// RestoreTestSettle returns how long an archive must have sat before it is a restore-test
|
||||
// candidate (R-86): a positive value as-is, negative → 0 (no settle requirement), 0 → the default.
|
||||
//
|
||||
|
||||
@@ -470,6 +470,10 @@ type RestoreTest struct {
|
||||
// mount layout, not just booted. Additive — a hub that predates them ignores the unknown keys.
|
||||
MountParity string `json:"mount_parity,omitempty"`
|
||||
MountInventory []string `json:"mount_inventory,omitempty"`
|
||||
// Skipped (R-672, v0.133.0): the test did NOT run — the space preflight refused, and Error says
|
||||
// why ("skipped: not enough space on …"). Pass is false. A hub that predates the key reads a
|
||||
// failed test with that error, which is the honest reading.
|
||||
Skipped bool `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
// PBSSnapshot is one PBS (offsite) snapshot's inventory + integrity state (doc 03 §8, slice
|
||||
|
||||
@@ -52,7 +52,7 @@ func newDREngine(t *testing.T, api GuestAPI) (*Engine, *fakeRunner, string, *Que
|
||||
t.Cleanup(q.Close)
|
||||
fr := &fakeRunner{}
|
||||
sd := t.TempDir()
|
||||
e := NewEngine(EngineOptions{API: api, Queue: q, Journal: j, Provider: EmptyProvider{}, HostRunner: fr, StateDir: sd})
|
||||
e := NewEngine(EngineOptions{API: api, Queue: q, Journal: j, Provider: EmptyProvider{}, HostRunner: fr, StateDir: sd, RestoreSpace: roomySpace{}})
|
||||
return e, fr, sd, q
|
||||
}
|
||||
|
||||
|
||||
@@ -44,6 +44,17 @@ type Engine struct {
|
||||
|
||||
opSeq uint64 // atomic; makes each op id unique per attempt
|
||||
|
||||
// restoreSpace + spacePolicy are the restore-test's space preflight (R-672). nil space REFUSES
|
||||
// every restore-test (fail-closed) — see restoretest_space.go.
|
||||
restoreSpace RestoreSpace
|
||||
spacePolicy SpacePolicy
|
||||
|
||||
// scratchMu guards activeScratch (the vmids a running restore-test owns — the retry timer never
|
||||
// touches those) and teardownTries (failed timer retries per journal op, R-672 rule 3).
|
||||
scratchMu sync.Mutex
|
||||
activeScratch map[int]bool
|
||||
teardownTries map[string]int
|
||||
|
||||
// lastRes records the most recent successful Reconcile Result (v0.90.0, R-28 fast-tick source).
|
||||
// The fast-tick reads it to decide convergence: actionable drift is Planned − Pending > 0 (a
|
||||
// destructive pending_signature refusal is EXPECTED state, not drift to hammer on). lastOK is
|
||||
@@ -86,6 +97,10 @@ type EngineOptions struct {
|
||||
HostRunner proxmox.Runner
|
||||
// StateDir is the agent state dir ("" → /var/lib/felhom-agent); only the 4d swap reads it.
|
||||
StateDir string
|
||||
// RestoreSpace is the restore-test's space preflight (R-672). nil → every restore-test is REFUSED
|
||||
// with its reason (fail-closed). SpacePolicy zero → DefaultSpacePolicy.
|
||||
RestoreSpace RestoreSpace
|
||||
SpacePolicy SpacePolicy
|
||||
}
|
||||
|
||||
// NewEngine builds an Engine. The Queue is shared (the single §10 choke point); the
|
||||
@@ -124,9 +139,24 @@ func NewEngine(opts EngineOptions) *Engine {
|
||||
logger: logger,
|
||||
hostRun: opts.HostRunner,
|
||||
stateDir: stateDir,
|
||||
|
||||
restoreSpace: opts.RestoreSpace,
|
||||
spacePolicy: policyOrDefault(opts.SpacePolicy),
|
||||
activeScratch: map[int]bool{},
|
||||
teardownTries: map[string]int{},
|
||||
}
|
||||
}
|
||||
|
||||
func policyOrDefault(p SpacePolicy) SpacePolicy {
|
||||
if p.Factor < 1 {
|
||||
p.Factor = DefaultSpacePolicy.Factor
|
||||
}
|
||||
if p.ReserveBytes <= 0 {
|
||||
p.ReserveBytes = DefaultSpacePolicy.ReserveBytes
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// Result summarizes one Reconcile pass.
|
||||
type Result struct {
|
||||
Planned int
|
||||
|
||||
@@ -213,7 +213,7 @@ func newEngine(t *testing.T, api GuestAPI, provider DesiredProvider) (*Engine, *
|
||||
t.Cleanup(func() { j.Close() })
|
||||
q := NewQueue()
|
||||
t.Cleanup(q.Close)
|
||||
e := NewEngine(EngineOptions{API: api, Queue: q, Journal: j, Provider: provider})
|
||||
e := NewEngine(EngineOptions{API: api, Queue: q, Journal: j, Provider: provider, RestoreSpace: roomySpace{}})
|
||||
return e, j, q
|
||||
}
|
||||
|
||||
|
||||
@@ -50,10 +50,18 @@ type RestoreTestResult struct {
|
||||
ScratchVMID int
|
||||
Pass bool
|
||||
Verified string // "boot+running" this slice
|
||||
Skipped bool // no free scratch VMID in band → test not run
|
||||
Err error
|
||||
StartedAt time.Time
|
||||
Duration time.Duration
|
||||
Skipped bool // test not run: no free scratch VMID in band, or the space preflight refused (R-672)
|
||||
// SkipReason is set when the SPACE PREFLIGHT refused (R-672): the test did not run, and this is
|
||||
// reported to the hub as the test's result (pass=false), never as a pass. Empty for a band skip.
|
||||
SkipReason string
|
||||
// TargetStorage is where the restore went (rule 2 may move it off the tested guest's pool);
|
||||
// RequiredBytes/AvailBytes are rule 1's figures.
|
||||
TargetStorage string
|
||||
RequiredBytes int64
|
||||
AvailBytes int64
|
||||
Err error
|
||||
StartedAt time.Time
|
||||
Duration time.Duration
|
||||
// StartWarnings holds the warning line(s) the guest-start task emitted (e.g. the
|
||||
// systemd-nesting advisory). Populated only when the start exited "WARNINGS: N";
|
||||
// always surfaced, NEVER used to decide pass/fail (the verdict is liveness — waitRunning).
|
||||
@@ -136,6 +144,29 @@ func (e *Engine) RunRestoreTest(ctx context.Context, spec RestoreTestSpec) Resto
|
||||
return res
|
||||
}
|
||||
|
||||
// R-672: the space preflight, BEFORE anything is journaled or created.
|
||||
rawCfg, err := e.api.ExtractArchiveConfig(ctx, spec.Archive)
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: restore-test extract archive config: %w", err)
|
||||
return res
|
||||
}
|
||||
v := PreflightRestoreSpace(ctx, e.restoreSpace, e.spacePolicy, spec.Archive, rawCfg, spec.RestoreStorage)
|
||||
res.TargetStorage, res.RequiredBytes, res.AvailBytes = v.Storage, v.Required, v.Avail
|
||||
if !v.OK {
|
||||
res.Skipped = true
|
||||
res.SkipReason = "skipped: " + v.Reason
|
||||
e.logger.Warn("restore-test SKIPPED by the space preflight (R-672) — nothing was created",
|
||||
"archive", spec.Archive, "storage", v.Storage, "required_bytes", v.Required, "avail_bytes", v.Avail, "reason", v.Reason)
|
||||
res.Duration = time.Since(now)
|
||||
return res
|
||||
}
|
||||
if v.Storage != spec.RestoreStorage {
|
||||
e.logger.Info("restore-test: restoring OFF the tested guest's own pool (R-672 rule 2)",
|
||||
"configured", spec.RestoreStorage, "avoided", v.Avoided, "target", v.Storage)
|
||||
}
|
||||
e.logger.Info("restore-test: space preflight passed", "storage", v.Storage, "required_bytes", v.Required, "avail_bytes", v.Avail)
|
||||
spec.RestoreStorage = v.Storage
|
||||
|
||||
lxc, err := e.api.ListLXC(ctx)
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: restore-test list guests: %w", err)
|
||||
@@ -164,7 +195,9 @@ func (e *Engine) RunRestoreTest(ctx context.Context, spec RestoreTestSpec) Resto
|
||||
// Serialize on the scratch VMID's lane (inherits §10), and capture the result.
|
||||
var vmidOccupied bool
|
||||
ch := e.queue.Submit(vmid, func() error {
|
||||
vmidOccupied = e.runScratchTest(ctx, vmid, spec, &res)
|
||||
e.markScratch(vmid, true)
|
||||
defer e.markScratch(vmid, false)
|
||||
vmidOccupied = e.runScratchTest(ctx, vmid, spec, rawCfg, &res)
|
||||
return res.Err
|
||||
})
|
||||
<-ch
|
||||
@@ -182,7 +215,7 @@ func (e *Engine) RunRestoreTest(ctx context.Context, spec RestoreTestSpec) Resto
|
||||
// runScratchTest is the journaled body (runs on vmid's queue lane). The occupied return is true
|
||||
// ONLY when PVE synchronously refused the restore because the vmid already holds a guest (one
|
||||
// the pool-blind band scan couldn't see) — the caller then advances to the next band vmid (F2).
|
||||
func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestSpec, res *RestoreTestResult) (occupied bool) {
|
||||
func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestSpec, rawCfg string, res *RestoreTestResult) (occupied bool) {
|
||||
base := JournalEntry{OpID: e.scratchOpID(vmid), VMID: vmid, Kind: scratchKind, Scratch: true}
|
||||
|
||||
// OWN the scratch guest's cleanup BEFORE any mutation. From here, a crash is recoverable.
|
||||
@@ -215,11 +248,7 @@ func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestS
|
||||
// genuinely EXTRACTED — full fidelity; the added runtime IS the verification), the two
|
||||
// structural binds → throwaway stand-ins. An unreadable archive config or an unknown
|
||||
// topology REFUSES up front — never restore a partial guest to "verify" it.
|
||||
rawCfg, err := e.api.ExtractArchiveConfig(ctx, spec.Archive)
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: restore-test extract archive config: %w", err)
|
||||
return false
|
||||
}
|
||||
// The archive's config was read ONCE, by the space preflight (R-672), and is passed in.
|
||||
mountOverrides, err := drRestoreOverrides(rawCfg, spec.RestoreStorage)
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: restore-test: %w", err)
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
package reconcile
|
||||
|
||||
import "context"
|
||||
|
||||
// ── A failed scratch teardown is retried on a TIMER, not only at agent start (R-672 rule 3) ────────
|
||||
//
|
||||
// MEASURED 2026-09-24 on demo-hp: the scheduled restore-test's teardown failed (`lvremove … contains a
|
||||
// filesystem in use`, a transient hold) and logged "left for Recover" — and Recover runs ONLY at agent
|
||||
// start, so the 22 GiB scratch guest sat in the full pool for 2.5 hours until an agent restart. The
|
||||
// timer calls RetryScratchTeardown every 10 minutes: the SAME resolution as Recover (recoverScratch —
|
||||
// the gate's benign scratch destroy, idempotent when the guest is already gone), restricted to Scratch
|
||||
// entries that carry a launch-proof UPID and that no running restore-test owns. After
|
||||
// MaxTeardownTries failed attempts for one entry the operator is told (the caller reports it); the
|
||||
// timer keeps trying.
|
||||
|
||||
// MaxTeardownTries is how many failed timer retries of one scratch entry happen before the operator
|
||||
// is told.
|
||||
const MaxTeardownTries = 3
|
||||
|
||||
// ScratchRetryResult summarizes one timer pass.
|
||||
type ScratchRetryResult struct {
|
||||
Examined int
|
||||
Destroyed int
|
||||
Clean int // already gone
|
||||
Failed int
|
||||
// GaveUp lists the scratch vmids whose failed tries reached MaxTeardownTries IN THIS PASS — each
|
||||
// is reported exactly once (the caller tells the operator).
|
||||
GaveUp []int
|
||||
}
|
||||
|
||||
func (e *Engine) markScratch(vmid int, active bool) {
|
||||
e.scratchMu.Lock()
|
||||
defer e.scratchMu.Unlock()
|
||||
if active {
|
||||
e.activeScratch[vmid] = true
|
||||
} else {
|
||||
delete(e.activeScratch, vmid)
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Engine) scratchActive(vmid int) bool {
|
||||
e.scratchMu.Lock()
|
||||
defer e.scratchMu.Unlock()
|
||||
return e.activeScratch[vmid]
|
||||
}
|
||||
|
||||
// RetryScratchTeardown is the timer's pass. It never touches a non-Scratch entry (unlike Recover,
|
||||
// which also resolves generic in-flight operations and must therefore run only at start), never an
|
||||
// entry without a launch-proof UPID (nothing was created), and never a vmid a running test owns.
|
||||
func (e *Engine) RetryScratchTeardown(ctx context.Context) ScratchRetryResult {
|
||||
var out ScratchRetryResult
|
||||
if e.journal == nil {
|
||||
return out
|
||||
}
|
||||
for _, entry := range e.journal.InFlight() {
|
||||
if !entry.Scratch || entry.UPID == "" || e.scratchActive(entry.VMID) {
|
||||
continue
|
||||
}
|
||||
out.Examined++
|
||||
var r RecoverResult
|
||||
e.recoverScratch(ctx, entry, &r)
|
||||
switch {
|
||||
case r.ScratchDestroyed > 0:
|
||||
out.Destroyed++
|
||||
e.forgetTries(entry.OpID)
|
||||
case r.ScratchClean > 0:
|
||||
out.Clean++
|
||||
e.forgetTries(entry.OpID)
|
||||
default:
|
||||
out.Failed++
|
||||
n := e.addTry(entry.OpID)
|
||||
e.logger.Warn("restore-test: scratch teardown retry failed (timer)", "vmid", entry.VMID, "op_id", entry.OpID, "try", n)
|
||||
if n == MaxTeardownTries {
|
||||
out.GaveUp = append(out.GaveUp, entry.VMID)
|
||||
e.logger.Error("restore-test: scratch guest still NOT torn down after repeated retries — telling the operator",
|
||||
"vmid", entry.VMID, "tries", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (e *Engine) addTry(op string) int {
|
||||
e.scratchMu.Lock()
|
||||
defer e.scratchMu.Unlock()
|
||||
e.teardownTries[op]++
|
||||
return e.teardownTries[op]
|
||||
}
|
||||
|
||||
func (e *Engine) forgetTries(op string) {
|
||||
e.scratchMu.Lock()
|
||||
defer e.scratchMu.Unlock()
|
||||
delete(e.teardownTries, op)
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package reconcile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-672 rule 3 (v0.133.0): a failed scratch teardown is retried on a TIMER. The consequence asserted:
|
||||
// the leaked scratch guest is destroyed by a timer pass (not only by a restart's Recover), the operator
|
||||
// is told exactly once after MaxTeardownTries failures, and a vmid a running test owns is never touched.
|
||||
|
||||
// leakScratch runs a restore-test whose teardown fails, leaving scratch 990000 in-flight — the
|
||||
// 2026-09-24 shape ("lvremove … contains a filesystem in use").
|
||||
func leakScratch(t *testing.T) (*Engine, *fakeAPI, *Journal) {
|
||||
t.Helper()
|
||||
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}, restoreUPID: "UPID:r", destroyErr: errors.New("lvremove: contains a filesystem in use")}
|
||||
e, j := spaceEngine(t, api, roomySpace{})
|
||||
e.RunRestoreTest(context.Background(), RestoreTestSpec{Archive: "local:backup/x.tar.zst", RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009})
|
||||
if len(j.InFlight()) != 1 {
|
||||
t.Fatalf("setup: want the scratch left in-flight after a failed teardown, got %+v", j.InFlight())
|
||||
}
|
||||
api.lxc = []proxmox.Guest{{VMID: 990000}}
|
||||
return e, api, j
|
||||
}
|
||||
|
||||
// COMPANION RED-PROOF (REPORT): make RetryScratchTeardown return without touching the journal (the
|
||||
// v0.132.0 shape — only Recover at start resolved a leak) → "the leaked scratch was not destroyed by
|
||||
// the timer".
|
||||
func TestRetry_TheTimerDestroysALeakedScratch(t *testing.T) {
|
||||
e, api, j := leakScratch(t)
|
||||
api.destroyErr = nil // the transient hold is gone
|
||||
before := len(api.destroys)
|
||||
r := e.RetryScratchTeardown(context.Background())
|
||||
if r.Destroyed != 1 || len(api.destroys) != before+1 || api.destroys[len(api.destroys)-1] != 990000 {
|
||||
t.Fatalf("the leaked scratch was not destroyed by the timer: result=%+v destroys=%v", r, api.destroys)
|
||||
}
|
||||
if len(j.InFlight()) != 0 {
|
||||
t.Fatalf("the entry is still in flight after a successful retry: %+v", j.InFlight())
|
||||
}
|
||||
if r2 := e.RetryScratchTeardown(context.Background()); r2.Examined != 0 {
|
||||
t.Fatalf("a resolved entry was examined again: %+v", r2)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetry_OperatorToldOnceAfterThreeFailures(t *testing.T) {
|
||||
e, _, _ := leakScratch(t)
|
||||
var gave [][]int
|
||||
for i := 0; i < MaxTeardownTries+2; i++ {
|
||||
gave = append(gave, e.RetryScratchTeardown(context.Background()).GaveUp)
|
||||
}
|
||||
for i, g := range gave {
|
||||
want := 0
|
||||
if i == MaxTeardownTries-1 {
|
||||
want = 1
|
||||
}
|
||||
if len(g) != want {
|
||||
t.Fatalf("pass %d gave up on %v — want the operator told exactly once, on pass %d", i+1, g, MaxTeardownTries)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetry_NeverTouchesARunningTest(t *testing.T) {
|
||||
e, api, _ := leakScratch(t)
|
||||
api.destroyErr = nil
|
||||
e.markScratch(990000, true) // a restore-test is (again) working on this vmid
|
||||
before := len(api.destroys)
|
||||
if r := e.RetryScratchTeardown(context.Background()); r.Examined != 0 || len(api.destroys) != before {
|
||||
t.Fatalf("the timer touched a scratch a running test owns: %+v destroys=%v", r, api.destroys)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
package reconcile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ── The restore-test's space preflight (R-672, agent v0.133.0) ─────────────────────────────────────
|
||||
//
|
||||
// MEASURED 2026-09-24 on demo-hp: the scheduled restore-test restored 9201's archive into `local-lvm`
|
||||
// — the SAME thin pool that holds 9201 — with no free-space check. The pool reached 100 %
|
||||
// (`out_of_data_space`, `error_if_no_space`), and 9201's rootfs and data volume remounted READ-ONLY.
|
||||
// Evidence: felhom.eu `documentation/audits/night-2026-09-24/C-02…C-07`, `audits/r672-2026-09-24/`.
|
||||
//
|
||||
// THE THREE RULES, all decided here before ANY mutation (before the scratch entry is even journaled):
|
||||
// 1. SPACE FIRST. The target storage must have free data ≥ restored × factor + reserve (defaults 1.2 and
|
||||
// 5 GiB, `backup.restore_test_space_factor` / `backup.restore_test_space_reserve_gib`), and a thin
|
||||
// pool's metadata must have room for the same share. `restored` is the UNCOMPRESSED size — the
|
||||
// archive FILE is the wrong number: 9201's archive was 6.9 GB and its restore wrote 22.6 GB, so
|
||||
// "file × 1.2 + 5 GiB" (14.3 GB) would have let the 2026-09-24 test run into a pool with 23 GB free.
|
||||
// 2. KEEP OFF THE TESTED GUEST'S POOL when another eligible storage (active, takes `rootdir`, and the
|
||||
// agent holds Datastore.AllocateSpace on it) passes rule 1. With only one, rule 1 decides.
|
||||
// 3. UNKNOWN REFUSES. An unreadable size, an unreadable storage or an unknown thin-pool metadata fill is
|
||||
// a skip with its reason, never a guess — the fail-safe direction of every guard in this project.
|
||||
// A refusal is reported to the hub as the test's RESULT ("skipped: …", pass=false), never as a pass.
|
||||
// Pinned by restoretest_space_test.go.
|
||||
|
||||
// RestoreSpace is the preflight's seam onto the host. Production: internal/restorespace.
|
||||
type RestoreSpace interface {
|
||||
// RestoredBytes is how many bytes restoring `archive` will write (uncompressed), and where that
|
||||
// figure came from (for the log and the refusal).
|
||||
RestoredBytes(ctx context.Context, archive string) (bytes int64, source string, err error)
|
||||
// Free reports the storage's free data bytes and, for a thin pool, its metadata-used fraction.
|
||||
Free(ctx context.Context, storage string) (StorageFree, error)
|
||||
// Eligible lists the storages a restore-test may target: active, content `rootdir`, and the agent
|
||||
// holds Datastore.AllocateSpace there.
|
||||
Eligible(ctx context.Context) ([]string, error)
|
||||
}
|
||||
|
||||
// StorageFree is one storage's free space as the preflight judges it.
|
||||
type StorageFree struct {
|
||||
AvailBytes int64
|
||||
UsedBytes int64
|
||||
Thin bool
|
||||
// MetaUsedFraction is the thin pool's metadata use (0..1); MetaKnown false = could not be read.
|
||||
MetaUsedFraction float64
|
||||
MetaKnown bool
|
||||
}
|
||||
|
||||
// SpacePolicy is rule 1's margin.
|
||||
type SpacePolicy struct {
|
||||
Factor float64 // ≥ 1
|
||||
ReserveBytes int64
|
||||
}
|
||||
|
||||
// DefaultSpacePolicy is 1.2 × restored + 5 GiB.
|
||||
var DefaultSpacePolicy = SpacePolicy{Factor: 1.2, ReserveBytes: 5 << 30}
|
||||
|
||||
// SpaceVerdict is the preflight's answer.
|
||||
type SpaceVerdict struct {
|
||||
OK bool
|
||||
Storage string // the storage the restore goes to (when OK) or was judged (when not)
|
||||
Required int64
|
||||
Avail int64
|
||||
Reason string // empty when OK
|
||||
// Avoided is the tested guest's own storage, when rule 2 moved the restore off it.
|
||||
Avoided string
|
||||
}
|
||||
|
||||
// requiredBytes is rule 1's figure.
|
||||
func (p SpacePolicy) requiredBytes(restored int64) int64 {
|
||||
f := p.Factor
|
||||
if f < 1 {
|
||||
f = DefaultSpacePolicy.Factor
|
||||
}
|
||||
return int64(float64(restored)*f) + p.ReserveBytes
|
||||
}
|
||||
|
||||
// fits judges one storage against rule 1 (data AND thin metadata). An unknown metadata fill on a thin
|
||||
// pool refuses (rule 3).
|
||||
func fits(fr StorageFree, required int64) (bool, string) {
|
||||
if fr.AvailBytes < required {
|
||||
return false, fmt.Sprintf("needs %s free, has %s", gib(required), gib(fr.AvailBytes))
|
||||
}
|
||||
if fr.Thin {
|
||||
if !fr.MetaKnown {
|
||||
return false, "thin-pool metadata fill unknown"
|
||||
}
|
||||
// The metadata a restore of `required` bytes needs, in the pool's own proportion of metadata to
|
||||
// data. A pool with no data yet has no proportion to read → only the absolute ceiling applies.
|
||||
need := 0.0
|
||||
if fr.UsedBytes > 0 {
|
||||
need = fr.MetaUsedFraction * float64(required) / float64(fr.UsedBytes)
|
||||
}
|
||||
if fr.MetaUsedFraction+need > 0.9 {
|
||||
return false, fmt.Sprintf("thin-pool metadata would reach %.0f%% (now %.0f%%)", 100*(fr.MetaUsedFraction+need), 100*fr.MetaUsedFraction)
|
||||
}
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
func gib(b int64) string { return fmt.Sprintf("%.1f GiB", float64(b)/(1<<30)) }
|
||||
|
||||
// sourceStorages returns the storage ids that hold the ARCHIVED guest's volumes (rootfs and every mpN
|
||||
// that names a `storage:volume`), read from the archive's own embedded config — the guest under test.
|
||||
// Bind mounts (a leading "/") carry no storage.
|
||||
func sourceStorages(rawCfg string) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
for k, v := range archiveCurrentConfig(rawCfg) {
|
||||
if k != "rootfs" && !(strings.HasPrefix(k, "mp") && len(k) > 2 && k[2] >= '0' && k[2] <= '9') {
|
||||
continue
|
||||
}
|
||||
vol := strings.TrimSpace(strings.SplitN(strings.TrimSpace(v), ",", 2)[0])
|
||||
if vol == "" || strings.HasPrefix(vol, "/") {
|
||||
continue
|
||||
}
|
||||
if st, _, ok := strings.Cut(vol, ":"); ok && st != "" {
|
||||
out[st] = true
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// PreflightRestoreSpace applies the three rules. `configured` is `backup.restore_storage`.
|
||||
func PreflightRestoreSpace(ctx context.Context, space RestoreSpace, policy SpacePolicy, archive, rawCfg, configured string) SpaceVerdict {
|
||||
if space == nil {
|
||||
return SpaceVerdict{Storage: configured, Reason: "no space check is wired — refusing (fail-closed)"}
|
||||
}
|
||||
restored, src, err := space.RestoredBytes(ctx, archive)
|
||||
if err != nil || restored <= 0 {
|
||||
return SpaceVerdict{Storage: configured, Reason: fmt.Sprintf("cannot tell how much the restore writes (%v)", err)}
|
||||
}
|
||||
required := policy.requiredBytes(restored)
|
||||
own := sourceStorages(rawCfg)
|
||||
|
||||
// Rule 2: the configured storage holds the guest under test → try the others first.
|
||||
var order []string
|
||||
avoided := ""
|
||||
if own[configured] {
|
||||
eligible, eerr := space.Eligible(ctx)
|
||||
if eerr == nil {
|
||||
sort.Strings(eligible)
|
||||
for _, s := range eligible {
|
||||
if s != configured && !own[s] {
|
||||
order = append(order, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(order) > 0 {
|
||||
avoided = configured
|
||||
}
|
||||
}
|
||||
order = append(order, configured)
|
||||
|
||||
var last SpaceVerdict
|
||||
for _, s := range order {
|
||||
fr, ferr := space.Free(ctx, s)
|
||||
if ferr != nil {
|
||||
last = SpaceVerdict{Storage: s, Required: required, Reason: fmt.Sprintf("cannot read free space on %s (%v)", s, ferr)}
|
||||
continue
|
||||
}
|
||||
ok, why := fits(fr, required)
|
||||
v := SpaceVerdict{OK: ok, Storage: s, Required: required, Avail: fr.AvailBytes}
|
||||
if ok {
|
||||
if s != configured {
|
||||
v.Avoided = avoided
|
||||
}
|
||||
return v
|
||||
}
|
||||
v.Reason = fmt.Sprintf("not enough space on %s: restoring %s (%s) %s", s, gib(restored), src, why)
|
||||
last = v
|
||||
}
|
||||
return last
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
package reconcile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-672 (v0.133.0) — the restore-test's space preflight. Every test asserts the CONSEQUENCE: whether the
|
||||
// Proxmox API was asked to restore anything, where to, and what the result says — never only the verdict.
|
||||
|
||||
const gb = int64(1000 * 1000 * 1000)
|
||||
|
||||
// fakeSpace is a configurable RestoreSpace.
|
||||
type fakeSpace struct {
|
||||
restored int64
|
||||
restoredErr error
|
||||
free map[string]StorageFree
|
||||
freeErr map[string]error
|
||||
eligible []string
|
||||
}
|
||||
|
||||
func (f fakeSpace) RestoredBytes(context.Context, string) (int64, string, error) {
|
||||
return f.restored, "fake", f.restoredErr
|
||||
}
|
||||
func (f fakeSpace) Free(_ context.Context, s string) (StorageFree, error) {
|
||||
if err := f.freeErr[s]; err != nil {
|
||||
return StorageFree{}, err
|
||||
}
|
||||
fr, ok := f.free[s]
|
||||
if !ok {
|
||||
return StorageFree{}, errors.New("unknown storage")
|
||||
}
|
||||
return fr, nil
|
||||
}
|
||||
func (f fakeSpace) Eligible(context.Context) ([]string, error) { return f.eligible, nil }
|
||||
|
||||
// thin9201 is demo-hp's local-lvm at 10:29 on 2026-09-24, just before the restore-test that filled it:
|
||||
// 23.2 GB free, 33.3 GB used, metadata 2.65 %.
|
||||
var thin9201 = StorageFree{AvailBytes: 23210892 * 1024, UsedBytes: 33277043 * 1024, Thin: true, MetaUsedFraction: 0.0265, MetaKnown: true}
|
||||
|
||||
// archive9201 is 9201's archive config: both volumes on local-lvm.
|
||||
const archive9201 = "hostname: demo-hp\nrootfs: local-lvm:vm-9201-disk-0,size=32G\nmp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G\nmp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\n"
|
||||
|
||||
func spaceEngine(t *testing.T, api *fakeAPI, sp RestoreSpace) (*Engine, *Journal) {
|
||||
t.Helper()
|
||||
j, err := OpenJournal(filepath.Join(t.TempDir(), "journal.log"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { j.Close() })
|
||||
q := NewQueue()
|
||||
t.Cleanup(q.Close)
|
||||
return NewEngine(EngineOptions{API: api, Queue: q, Journal: j, RestoreSpace: sp}), j
|
||||
}
|
||||
|
||||
func run9201(e *Engine) RestoreTestResult {
|
||||
return e.RunRestoreTest(context.Background(), RestoreTestSpec{
|
||||
Archive: "local:backup/vzdump-lxc-9201-2026_09_23-06_55_25.tar.zst", RestoreStorage: "local-lvm",
|
||||
ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local",
|
||||
})
|
||||
}
|
||||
|
||||
// TestSpace_The2026_09_24TestIsRefused replays R-672: 9201's archive restores 22.6 GB (its vzdump log),
|
||||
// the pool has 23.2 GB free. The test must NOT start — no restore call, no journaled scratch — and the
|
||||
// result must say why, as a non-pass.
|
||||
//
|
||||
// COMPANION RED-PROOFS (REPORT): (1) the preflight removed (v0.132.0's shape) → a restore into local-lvm
|
||||
// is issued; (2) `restored` taken from the archive FILE (6.9 GB, the brief's "archive × 1.2 + 5 GiB") →
|
||||
// 6.9×1.2+5.4 = 13.7 GB < 23.2 GB free, so the test starts — the defect the uncompressed size exists for.
|
||||
func TestSpace_The2026_09_24TestIsRefused(t *testing.T) {
|
||||
api := &fakeAPI{extractCfg: archive9201, cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}}
|
||||
e, j := spaceEngine(t, api, fakeSpace{restored: 22607360000, free: map[string]StorageFree{"local-lvm": thin9201}})
|
||||
res := run9201(e)
|
||||
if len(api.restores) != 0 {
|
||||
t.Fatalf("a restore was issued into a pool that cannot take it: %+v", api.restores)
|
||||
}
|
||||
if len(j.InFlight()) != 0 {
|
||||
t.Fatalf("a scratch entry was journaled for a test that must not start: %+v", j.InFlight())
|
||||
}
|
||||
if res.Pass || !res.Skipped || !strings.Contains(res.SkipReason, "not enough space on local-lvm") {
|
||||
t.Fatalf("result = pass=%v skipped=%v reason=%q — want a non-pass skip naming the storage", res.Pass, res.Skipped, res.SkipReason)
|
||||
}
|
||||
if res.RequiredBytes < 32*gb || res.AvailBytes != thin9201.AvailBytes {
|
||||
t.Fatalf("required=%d avail=%d — want ≥ 32 GB required (22.6 × 1.2 + 5 GiB) against 23.2 GB", res.RequiredBytes, res.AvailBytes)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSpace_KeepsOffTheTestedGuestsPool — rule 2: another eligible storage that fits takes the restore.
|
||||
func TestSpace_KeepsOffTheTestedGuestsPool(t *testing.T) {
|
||||
api := &fakeAPI{extractCfg: archive9201, cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}}
|
||||
e, _ := spaceEngine(t, api, fakeSpace{restored: 22607360000, eligible: []string{"local-lvm", "big-dir"},
|
||||
free: map[string]StorageFree{"local-lvm": {AvailBytes: 900 * gb, UsedBytes: 10 * gb, Thin: true, MetaKnown: true}, "big-dir": {AvailBytes: 500 * gb}}})
|
||||
res := run9201(e)
|
||||
if len(api.restores) != 1 || api.restores[0].Storage != "big-dir" {
|
||||
t.Fatalf("restores = %+v — want ONE restore onto big-dir, off 9201's own pool", api.restores)
|
||||
}
|
||||
if res.TargetStorage != "big-dir" {
|
||||
t.Fatalf("target=%q", res.TargetStorage)
|
||||
}
|
||||
for k, v := range api.restores[0].MountOverrides {
|
||||
if strings.HasPrefix(v, "local-lvm:") {
|
||||
t.Fatalf("%s still lands on the tested guest's pool: %s", k, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSpace_OnlyOnePool_RuleOneDecides — no other eligible storage: the tested guest's pool is used when it
|
||||
// fits (demo-hp's real shape: nvme-scratch takes rootdir but the agent holds no AllocateSpace there).
|
||||
func TestSpace_OnlyOnePool_RuleOneDecides(t *testing.T) {
|
||||
api := &fakeAPI{extractCfg: archive9201, cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}}
|
||||
e, _ := spaceEngine(t, api, fakeSpace{restored: 2 * gb, eligible: []string{"local-lvm"}, free: map[string]StorageFree{"local-lvm": thin9201}})
|
||||
res := run9201(e)
|
||||
if len(api.restores) != 1 || api.restores[0].Storage != "local-lvm" || res.Skipped || res.TargetStorage != "local-lvm" {
|
||||
t.Fatalf("restores=%+v skipped=%v — a 2 GB restore fits 23 GB free on the only pool", api.restores, res.Skipped)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSpace_UnknownRefuses — rule 3, one case per unknown. Nothing is restored in any of them.
|
||||
func TestSpace_UnknownRefuses(t *testing.T) {
|
||||
cases := map[string]RestoreSpace{
|
||||
"no space check wired": nil,
|
||||
"restore size unknown": fakeSpace{restoredErr: errors.New("no vzdump log"), free: map[string]StorageFree{"local-lvm": thin9201}},
|
||||
"free space unreadable": fakeSpace{restored: gb, freeErr: map[string]error{"local-lvm": errors.New("api down")}},
|
||||
"thin metadata unknown": fakeSpace{restored: gb, free: map[string]StorageFree{"local-lvm": {AvailBytes: 900 * gb, UsedBytes: gb, Thin: true}}},
|
||||
"metadata would overrun": fakeSpace{restored: 10 * gb, free: map[string]StorageFree{"local-lvm": {AvailBytes: 900 * gb, UsedBytes: 10 * gb, Thin: true, MetaUsedFraction: 0.5, MetaKnown: true}}},
|
||||
}
|
||||
for name, sp := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
api := &fakeAPI{extractCfg: archive9201, cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}}
|
||||
var e *Engine
|
||||
if sp == nil {
|
||||
e, _ = spaceEngine(t, api, nil)
|
||||
} else {
|
||||
e, _ = spaceEngine(t, api, sp)
|
||||
}
|
||||
res := run9201(e)
|
||||
if len(api.restores) != 0 || res.Pass || !res.Skipped || res.SkipReason == "" {
|
||||
t.Fatalf("restores=%d pass=%v skipped=%v reason=%q — an unknown must refuse before anything moves",
|
||||
len(api.restores), res.Pass, res.Skipped, res.SkipReason)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestSpace_SourceStorages reads the tested guest's pools from the ARCHIVE's config, binds excluded.
|
||||
func TestSpace_SourceStorages(t *testing.T) {
|
||||
got := sourceStorages(archive9201 + "mp1: other:vm-9201-disk-2,mp=/x,size=1G\n[snap]\nrootfs: snapstore:x\n")
|
||||
if !got["local-lvm"] || !got["other"] || got["snapstore"] || len(got) != 2 {
|
||||
t.Fatalf("sourceStorages = %v — want local-lvm + other, binds and snapshot sections excluded", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package reconcile
|
||||
|
||||
import "context"
|
||||
|
||||
// roomySpace is the permissive RestoreSpace the pre-R-672 restore-test tests run with: 1 GiB restored,
|
||||
// 1 TiB free, thin metadata known and low. The space rules themselves are pinned in
|
||||
// restoretest_space_test.go.
|
||||
type roomySpace struct{}
|
||||
|
||||
func (roomySpace) RestoredBytes(context.Context, string) (int64, string, error) {
|
||||
return 1 << 30, "test", nil
|
||||
}
|
||||
func (roomySpace) Free(context.Context, string) (StorageFree, error) {
|
||||
return StorageFree{AvailBytes: 1 << 40, UsedBytes: 1 << 30, Thin: true, MetaUsedFraction: 0.01, MetaKnown: true}, nil
|
||||
}
|
||||
func (roomySpace) Eligible(context.Context) ([]string, error) { return nil, nil }
|
||||
@@ -0,0 +1,176 @@
|
||||
// Package restorespace is the production seam behind reconcile.RestoreSpace (R-672, agent v0.133.0):
|
||||
// how much a restore of an archive writes, how much a storage has free, and which storages a
|
||||
// restore-test may target. Every read that cannot answer returns an error — the preflight then
|
||||
// REFUSES (reconcile/restoretest_space.go rule 3); nothing here guesses.
|
||||
package restorespace
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
)
|
||||
|
||||
// API is the Proxmox subset the provider reads.
|
||||
type API interface {
|
||||
ListStorage(ctx context.Context) ([]proxmox.Storage, error)
|
||||
NodeStorage(ctx context.Context) ([]proxmox.Storage, error)
|
||||
StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error)
|
||||
Permissions(ctx context.Context, aclPath string) (map[string]int, error)
|
||||
}
|
||||
|
||||
// Provider implements reconcile.RestoreSpace.
|
||||
type Provider struct {
|
||||
API API
|
||||
// ThinMeta reads a thin pool's metadata-used fraction (storage.HostOps.ThinPoolMetadata).
|
||||
ThinMeta func(ctx context.Context, vg, pool string) (float64, bool)
|
||||
// ReadFile reads a vzdump log; nil → os.ReadFile.
|
||||
ReadFile func(name string) ([]byte, error)
|
||||
}
|
||||
|
||||
var _ reconcile.RestoreSpace = (*Provider)(nil)
|
||||
|
||||
// totalWrittenRe is vzdump's own count of the bytes tar wrote into the archive — the UNCOMPRESSED size,
|
||||
// i.e. what a restore writes back ("INFO: Total bytes written: 22607360000 (22GiB, 49MiB/s)").
|
||||
var totalWrittenRe = regexp.MustCompile(`Total bytes written:\s*(\d+)`)
|
||||
|
||||
// archiveExts are the vzdump archive suffixes; the log is the archive name without it + ".log".
|
||||
var archiveExts = []string{".tar.zst", ".tar.gz", ".tar.lzo", ".tgz", ".tar"}
|
||||
|
||||
func (p *Provider) readFile(name string) ([]byte, error) {
|
||||
if p.ReadFile != nil {
|
||||
return p.ReadFile(name)
|
||||
}
|
||||
return os.ReadFile(name)
|
||||
}
|
||||
|
||||
// RestoredBytes: a file-backed archive → its vzdump log's "Total bytes written"; a PBS archive → the
|
||||
// size Proxmox reports for the snapshot (its logical, uncompressed size). The archive FILE size is never
|
||||
// used: it is compressed (6.9 GB for a 22.6 GB restore, measured 2026-09-24).
|
||||
func (p *Provider) RestoredBytes(ctx context.Context, archive string) (int64, string, error) {
|
||||
id, vol, ok := strings.Cut(archive, ":")
|
||||
if !ok || id == "" || vol == "" {
|
||||
return 0, "", fmt.Errorf("not a storage volid: %q", archive)
|
||||
}
|
||||
st, err := p.storageConfig(ctx, id)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
switch st.Type {
|
||||
case "pbs":
|
||||
items, err := p.API.StorageContent(ctx, id)
|
||||
if err != nil {
|
||||
return 0, "", fmt.Errorf("list %s: %w", id, err)
|
||||
}
|
||||
for _, it := range items {
|
||||
if it.VolID == archive && it.Size > 0 {
|
||||
return it.Size, "pbs snapshot size", nil
|
||||
}
|
||||
}
|
||||
return 0, "", fmt.Errorf("archive %s not listed on %s with a size", archive, id)
|
||||
default:
|
||||
if st.Path == "" {
|
||||
return 0, "", fmt.Errorf("storage %s (%s) has no path to read a vzdump log from", id, st.Type)
|
||||
}
|
||||
base := path.Base(vol) // "backup/vzdump-lxc-…tar.zst" → "vzdump-lxc-…tar.zst"
|
||||
stem := ""
|
||||
for _, ext := range archiveExts {
|
||||
if strings.HasSuffix(base, ext) {
|
||||
stem = strings.TrimSuffix(base, ext)
|
||||
break
|
||||
}
|
||||
}
|
||||
if stem == "" {
|
||||
return 0, "", fmt.Errorf("unknown archive suffix: %s", base)
|
||||
}
|
||||
logPath := path.Join(st.Path, "dump", stem+".log")
|
||||
b, err := p.readFile(logPath)
|
||||
if err != nil {
|
||||
return 0, "", fmt.Errorf("read vzdump log %s: %w", logPath, err)
|
||||
}
|
||||
m := totalWrittenRe.FindSubmatch(b)
|
||||
if m == nil {
|
||||
return 0, "", fmt.Errorf("vzdump log %s carries no \"Total bytes written\"", logPath)
|
||||
}
|
||||
n, err := strconv.ParseInt(string(m[1]), 10, 64)
|
||||
if err != nil || n <= 0 {
|
||||
return 0, "", fmt.Errorf("vzdump log %s: bad byte count %q", logPath, m[1])
|
||||
}
|
||||
return n, "vzdump log: total bytes written", nil
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Provider) storageConfig(ctx context.Context, id string) (proxmox.Storage, error) {
|
||||
all, err := p.API.ListStorage(ctx)
|
||||
if err != nil {
|
||||
return proxmox.Storage{}, fmt.Errorf("list storage config: %w", err)
|
||||
}
|
||||
for _, s := range all {
|
||||
if s.Storage == id {
|
||||
return s, nil
|
||||
}
|
||||
}
|
||||
return proxmox.Storage{}, fmt.Errorf("storage %s not configured", id)
|
||||
}
|
||||
|
||||
// Free reads the node's live usage for `storage`; a thin pool adds its metadata fill.
|
||||
func (p *Provider) Free(ctx context.Context, storage string) (reconcile.StorageFree, error) {
|
||||
live, err := p.API.NodeStorage(ctx)
|
||||
if err != nil {
|
||||
return reconcile.StorageFree{}, fmt.Errorf("node storage: %w", err)
|
||||
}
|
||||
for _, s := range live {
|
||||
if s.Storage != storage {
|
||||
continue
|
||||
}
|
||||
if s.Active != 1 {
|
||||
return reconcile.StorageFree{}, fmt.Errorf("storage %s is not active", storage)
|
||||
}
|
||||
fr := reconcile.StorageFree{AvailBytes: s.Avail, UsedBytes: s.Used, Thin: s.Type == "lvmthin"}
|
||||
if fr.Thin {
|
||||
cfg, cerr := p.storageConfig(ctx, storage)
|
||||
if cerr == nil && p.ThinMeta != nil && cfg.VGName != "" && cfg.ThinPool != "" {
|
||||
fr.MetaUsedFraction, fr.MetaKnown = p.ThinMeta(ctx, cfg.VGName, cfg.ThinPool)
|
||||
}
|
||||
}
|
||||
return fr, nil
|
||||
}
|
||||
return reconcile.StorageFree{}, fmt.Errorf("storage %s not reported by the node", storage)
|
||||
}
|
||||
|
||||
// Eligible: active, content includes `rootdir`, and the agent holds Datastore.AllocateSpace on
|
||||
// /storage/<id>. The permission is read for the SPECIFIC privilege — the box-wide grant answers every
|
||||
// path with inherited privileges (proxmox.Client.Permissions).
|
||||
func (p *Provider) Eligible(ctx context.Context) ([]string, error) {
|
||||
live, err := p.API.NodeStorage(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("node storage: %w", err)
|
||||
}
|
||||
var out []string
|
||||
for _, s := range live {
|
||||
if s.Active != 1 || !hasContent(s.Content, "rootdir") {
|
||||
continue
|
||||
}
|
||||
privs, perr := p.API.Permissions(ctx, "/storage/"+s.Storage)
|
||||
if perr != nil || privs["Datastore.AllocateSpace"] != 1 {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Storage)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func hasContent(list, want string) bool {
|
||||
for _, c := range strings.Split(list, ",") {
|
||||
if strings.TrimSpace(c) == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
package restorespace
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-672 (v0.133.0). The provider behind the restore-test's space preflight. No test reaches a real
|
||||
// Proxmox or a real file: the API and ReadFile are fakes.
|
||||
|
||||
type fakeAPI struct {
|
||||
cfg []proxmox.Storage
|
||||
live []proxmox.Storage
|
||||
content map[string][]proxmox.StorageContent
|
||||
perms map[string]map[string]int
|
||||
}
|
||||
|
||||
func (f fakeAPI) ListStorage(context.Context) ([]proxmox.Storage, error) { return f.cfg, nil }
|
||||
func (f fakeAPI) NodeStorage(context.Context) ([]proxmox.Storage, error) { return f.live, nil }
|
||||
func (f fakeAPI) StorageContent(_ context.Context, s string) ([]proxmox.StorageContent, error) {
|
||||
return f.content[s], nil
|
||||
}
|
||||
func (f fakeAPI) Permissions(_ context.Context, p string) (map[string]int, error) {
|
||||
if m, ok := f.perms[p]; ok {
|
||||
return m, nil
|
||||
}
|
||||
return map[string]int{}, nil
|
||||
}
|
||||
|
||||
const archive = "local:backup/vzdump-lxc-9201-2026_09_23-06_55_25.tar.zst"
|
||||
|
||||
// The real log's tail (demo-hp, 2026-09-23): 22.6 GB written, a 6.91 GB archive file.
|
||||
const vzdumpLog = "2026-09-23 07:02:47 INFO: Total bytes written: 22607360000 (22GiB, 49MiB/s)\n2026-09-23 07:02:47 INFO: archive file size: 6.91GB\n"
|
||||
|
||||
// demoHP is demo-hp's storage layout: `local` (dir, backups), `local-lvm` (thin), `nvme-scratch` (dir,
|
||||
// rootdir — but the agent holds NO grant there), a pbs.
|
||||
func demoHP() fakeAPI {
|
||||
return fakeAPI{
|
||||
cfg: []proxmox.Storage{
|
||||
{Storage: "local", Type: "dir", Path: "/var/lib/vz"},
|
||||
{Storage: "local-lvm", Type: "lvmthin", VGName: "pve", ThinPool: "data"},
|
||||
{Storage: "nvme-scratch", Type: "dir", Path: "/mnt/hdd_1"},
|
||||
{Storage: "felhom-pbs", Type: "pbs"},
|
||||
},
|
||||
live: []proxmox.Storage{
|
||||
{Storage: "local", Type: "dir", Content: "vztmpl,backup,iso,import", Active: 1, Avail: 4 << 30, Used: 34 << 30},
|
||||
{Storage: "local-lvm", Type: "lvmthin", Content: "images,rootdir", Active: 1, Avail: 23210892 * 1024, Used: 33277043 * 1024},
|
||||
{Storage: "nvme-scratch", Type: "dir", Content: "images,rootdir", Active: 1, Avail: 800 << 30},
|
||||
{Storage: "felhom-pbs", Type: "pbs", Content: "backup", Active: 0},
|
||||
},
|
||||
content: map[string][]proxmox.StorageContent{
|
||||
"local": {{VolID: archive, Size: 7417540996}},
|
||||
"felhom-pbs": {{VolID: "felhom-pbs:backup/ct/9201/2026-09-23T02:00:00Z", Size: 21 << 30}},
|
||||
},
|
||||
perms: map[string]map[string]int{
|
||||
"/storage/local": {"Datastore.Audit": 1, "Datastore.AllocateSpace": 1},
|
||||
"/storage/local-lvm": {"Datastore.Audit": 1, "Datastore.AllocateSpace": 1},
|
||||
// nvme-scratch: only the inherited box-wide Datastore.Audit — the trap proxmox.Permissions names.
|
||||
"/storage/nvme-scratch": {"Datastore.Audit": 1},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// TestRestoredBytes_ReadsTheUncompressedSize — the vzdump log's "Total bytes written", never the archive
|
||||
// FILE size (6.9 GB for a 22.6 GB restore).
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT): return the storage content's Size for a dir storage → 7417540996, and
|
||||
// this test fails at "the compressed file size was used".
|
||||
func TestRestoredBytes_ReadsTheUncompressedSize(t *testing.T) {
|
||||
var asked string
|
||||
p := &Provider{API: demoHP(), ReadFile: func(n string) ([]byte, error) { asked = n; return []byte(vzdumpLog), nil }}
|
||||
n, src, err := p.RestoredBytes(context.Background(), archive)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n == 7417540996 {
|
||||
t.Fatal("the compressed file size was used — the restore writes 3× that")
|
||||
}
|
||||
if n != 22607360000 || asked != "/var/lib/vz/dump/vzdump-lxc-9201-2026_09_23-06_55_25.log" {
|
||||
t.Fatalf("n=%d from %q (log %q)", n, src, asked)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoredBytes_UnknownIsAnError(t *testing.T) {
|
||||
cases := map[string]*Provider{
|
||||
"no log": {API: demoHP(), ReadFile: func(string) ([]byte, error) { return nil, errors.New("ENOENT") }},
|
||||
"log without size": {API: demoHP(), ReadFile: func(string) ([]byte, error) { return []byte("ERROR: failed\n"), nil }},
|
||||
}
|
||||
for name, p := range cases {
|
||||
if n, _, err := p.RestoredBytes(context.Background(), archive); err == nil {
|
||||
t.Fatalf("%s: got %d, want an error (the preflight then refuses)", name, n)
|
||||
}
|
||||
}
|
||||
if _, _, err := (&Provider{API: demoHP()}).RestoredBytes(context.Background(), "local:backup/weird.vma"); err == nil {
|
||||
t.Fatal("an unknown archive suffix must be an error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoredBytes_PBS(t *testing.T) {
|
||||
p := &Provider{API: demoHP()}
|
||||
n, _, err := p.RestoredBytes(context.Background(), "felhom-pbs:backup/ct/9201/2026-09-23T02:00:00Z")
|
||||
if err != nil || n != 21<<30 {
|
||||
t.Fatalf("n=%d err=%v", n, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEligible_NeedsTheSpecificGrant — nvme-scratch takes rootdir but the agent holds only the inherited
|
||||
// Datastore.Audit there, so it is NOT eligible; `local` holds no rootdir.
|
||||
func TestEligible_NeedsTheSpecificGrant(t *testing.T) {
|
||||
got, err := (&Provider{API: demoHP()}).Eligible(context.Background())
|
||||
if err != nil || len(got) != 1 || got[0] != "local-lvm" {
|
||||
t.Fatalf("eligible = %v (%v) — want only local-lvm on demo-hp", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFree_ThinCarriesMetadata(t *testing.T) {
|
||||
p := &Provider{API: demoHP(), ThinMeta: func(_ context.Context, vg, pool string) (float64, bool) {
|
||||
if vg != "pve" || pool != "data" {
|
||||
t.Fatalf("metadata read for %s/%s", vg, pool)
|
||||
}
|
||||
return 0.0265, true
|
||||
}}
|
||||
fr, err := p.Free(context.Background(), "local-lvm")
|
||||
if err != nil || !fr.Thin || !fr.MetaKnown || fr.MetaUsedFraction != 0.0265 || fr.AvailBytes != 23210892*1024 {
|
||||
t.Fatalf("free = %+v err=%v", fr, err)
|
||||
}
|
||||
if _, err := p.Free(context.Background(), "felhom-pbs"); err == nil {
|
||||
t.Fatal("an inactive storage must be an error")
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
@@ -35,6 +36,45 @@ type Observer struct {
|
||||
host HostReader
|
||||
ops HostOps
|
||||
logger *slog.Logger
|
||||
|
||||
// R-672 (v0.133.0): a thin pool crossing thinPoolAlarmFraction (data OR metadata) requests an
|
||||
// out-of-band host report at once, so the hub's storage-fill alarm sees it in seconds instead of at
|
||||
// the next 15-minute report. Rising edge per pool; re-armed below thinPoolRearmFraction.
|
||||
highMu sync.Mutex
|
||||
high map[string]bool
|
||||
onThinHigh func()
|
||||
}
|
||||
|
||||
const (
|
||||
thinPoolAlarmFraction = 0.90
|
||||
thinPoolRearmFraction = 0.85
|
||||
)
|
||||
|
||||
// SetThinHighTrigger wires the out-of-band report request (main: the storage trigger channel).
|
||||
func (o *Observer) SetThinHighTrigger(f func()) { o.onThinHigh = f }
|
||||
|
||||
// noteThinFill is the edge detector. key separates data from metadata so each has its own edge.
|
||||
func (o *Observer) noteThinFill(key string, frac float64) {
|
||||
o.highMu.Lock()
|
||||
if o.high == nil {
|
||||
o.high = map[string]bool{}
|
||||
}
|
||||
fire := false
|
||||
switch {
|
||||
case frac >= thinPoolAlarmFraction && !o.high[key]:
|
||||
o.high[key] = true
|
||||
fire = true
|
||||
case frac < thinPoolRearmFraction && o.high[key]:
|
||||
o.high[key] = false
|
||||
}
|
||||
o.highMu.Unlock()
|
||||
if fire {
|
||||
o.logger.Error("storage: thin pool crossed 90% — requesting an immediate host report (the hub alarms)",
|
||||
"pool", key, "fraction", frac)
|
||||
if o.onThinHigh != nil {
|
||||
o.onThinHigh()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// NewObserver builds an Observer. host defaults to a ProcHostReader; logger to the
|
||||
@@ -260,6 +300,7 @@ func (o *Observer) build(s proxmox.Storage, mounts []Mount) observed {
|
||||
o.logger.Warn("storage: lvmthin pool data fill is high (a full pool corrupts every guest on it)",
|
||||
"storage", s.Storage, "data_used_fraction", frac)
|
||||
}
|
||||
o.noteThinFill(s.Storage+"/data", frac)
|
||||
}
|
||||
|
||||
// SMART-only device hint (v0.95.0): a dir-storage that lives INSIDE a shared filesystem (the
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// R-672 (v0.133.0): a thin pool crossing 90 % requests an out-of-band host report ONCE, through the
|
||||
// watchdog's own read path (Known — every few seconds), so the hub's storage-fill alarm sees the pool in
|
||||
// seconds, not at the next 15-minute report. Re-armed below 85 %.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT): remove the noteThinFill call from the data path → "no report was
|
||||
// requested when the pool crossed 90 %".
|
||||
func TestThinHigh_RequestsOneReportPerCrossing(t *testing.T) {
|
||||
pool := proxmox.Storage{Storage: "local-lvm", Type: "lvmthin", Content: "rootdir,images", Total: 1000, Active: 1}
|
||||
api := &fakeStorageAPI{node: "n", cluster: []proxmox.Storage{pool}}
|
||||
o := NewObserver(api, &fakeHostReader{}, nil, quietLogger())
|
||||
asked := 0
|
||||
o.SetThinHighTrigger(func() { asked++ })
|
||||
for i, used := range []int64{800, 910, 950, 1000, 840, 920} {
|
||||
p := pool
|
||||
p.Used, p.Avail, p.UsedFraction = used, 1000-used, float64(used)/1000
|
||||
api.nodeSt = []proxmox.Storage{p}
|
||||
if _, err := o.Known(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[int]int{0: 0, 1: 1, 2: 1, 3: 1, 4: 1, 5: 2}[i]
|
||||
if asked != want {
|
||||
t.Fatalf("after %d/1000 used: %d report requests, want %d", used, asked, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user