REPORT: v0.136.0 and v0.137.0

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

CHANGELOG.md

@@ -1,3 +1,82 @@
+## v0.136.0 — … of a guest that still EXISTS (2026-09-27, R-689 second half)
+
+> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.136.0` (`16dbc83`), sha256 `2eb0b5ebe253defd68b322312bbac12418c051b0a7a0d2d1831310d97fa6d755`, verified by download. **NOT vouched** (the operator's act).
+
+**MinAgent impact:** none required by any controller.
+
+- **R-689, second half.** Read on demo-hp right after v0.135.0 with the read-only `-selftest=restore-test-due`: the
+  golden was skipped, and the pick fell to `vzdump-lxc-9100-2026_08_21…` — a leftover of a guest deleted in August. A
+  candidate's guest must now exist on this node (`GuestConfig`): "does not exist" skips the archive (INFO once per
+  volid); any other lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove". Tests
+  `TestR689_AnArchiveOfADeletedGuestIsNeverPicked` (red-proofed: without the check it picks the 9100 leftover),
+  `TestR689_AGuestLookupFailureIsUnknownNotEmpty`. A second agent release in one session — the first half was found
+  incomplete on the box.
+
+## v0.135.0 — the restore test proves only backups OF A GUEST (2026-09-27, R-689)
+
+> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.135.0` (`d4be12c`), sha256 `ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (the operator's act).
+
+**MinAgent impact:** none required by any controller.
+
+- **R-689** (`backup/runner.go` `guestBackupArchive`). demo-hp keeps its golden template in `local:backup/` — content
+  "backup", 654 MB, plausibly complete, the newest settled entry — and the scheduled restore test picked it every 6 h and
+  failed `extractconfig` with a 403, while the guest's real archive went untested. A restore-test candidate is now a
+  `vzdump-{lxc,qemu}-<vmid>-…` file or a PBS `backup/{ct,vm}/<vmid>/…` snapshot whose vmid the storage reports; anything
+  else is skipped with one INFO line per volid (not the "INCOMPLETE archive" WARN). Tests
+  `TestR689_TheRestoreTestNeverPicksTheGolden` (red-proof: without the check it picks the golden) and
+  `TestR689_GuestBackupArchiveShapes`; two older picker tests' fixtures moved to real archive names.
+  Evidence: `felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`.
+
+## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
+
+> **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).
+
+**MinAgent impact:** none required by any controller.
+
+- **R-685 — the backup space preflight** (`backup/runner.go` `spaceFits`). Before a vzdump to a LOCAL (non-PBS)
+  target, the newest archive of that guest on that target × 1.25 + 1 GiB must be free — PVE prunes old archives
+  only AFTER a successful backup, so the kept ones still stand during the run. A shortfall is a skip: nothing
+  starts, the record's Error reads `skipped: not enough space: <target> has X GiB free; the last archive of guest N
+  was Y GiB, so a new one needs about Z GiB …` (stable prefix `BackupSkipNoSpacePrefix`), and it reaches the
+  controller's tier view and, through the quiesce loop's tier notifier, the operator's `whole_guest_backup_failed`.
+  It FAILS OPEN on what is not known (PBS target, first backup, unreadable usage).
+- **Free space is read from `GET /nodes/<node>/storage`** (`NodeStorage`), never `GET /storage` — the latter is the
+  cluster DEFINITIONS and carries no usage. **Found live, before release:** the first build read `/storage`,
+  failed open, and a real vzdump of demo-hp 9201 started from the live test; it was aborted after 5 min 16 s, no
+  archive left (`felhom.eu/documentation/audits/night-2026-09-25/F/`). The test fake's `ListStorage` now strips
+  usage like production. Red-proofs: two (`…/F/redproof-r685-*.txt`).
+- Live proof (demo-hp, safe builds with a hard stop before vzdump): ×10 margin → refused, "local has 14.9 GiB
+  free … needs about 77.2 GiB"; release margin → "space preflight passed" need 11.3 GB, avail 16.0 GB.
+
+## v0.133.0 — a restore-test can never fill a box's disk; leftovers retried on a timer (2026-09-24, R-672, R-673)
+
+> **RELEASED 2026-09-24** by `scripts/release-agent.sh` — tag `v0.133.0` (`9bdb4da`), sha256 `3aa303452b8c6be58573d00af01a0ab4a0d97e4f885ffecd0144a18ac24e69b6`, verified by an independent anonymous download. **NOT delivered** (needs an operator-signed `agent_update` job per box, R-530) and **NOT vouched**.
+
+**MinAgent impact:** none required by any controller. Hub **v0.124.0** makes a thin pool CRITICAL at 90 %
+(data or metadata) and keys the storage-fill alarm per pool per 6 h; an older hub still raises its generic
+90/95 % storage-fill events from the same report.
+
+- **R-672 — the space preflight** (`reconcile/restoretest_space.go`, provider `internal/restorespace`). Before
+  anything is journaled or created, a restore-test needs free data ≥ restored × 1.2 + 5 GiB on its target
+  (`backup.restore_test_space_factor` / `backup.restore_test_space_reserve_gib`) and room in a thin pool's
+  metadata. `restored` is the UNCOMPRESSED size — the vzdump log's "Total bytes written" (a file-backed
+  archive) or the PBS snapshot size; the archive FILE is never used (9201: 6.9 GB file, 22.6 GB restore). The
+  target moves OFF the tested guest's own pool when another storage is eligible (active, `rootdir`, and the
+  agent holds Datastore.AllocateSpace there) and fits. Anything unknown refuses. A refusal is reported to the
+  hub as the test's result (`pass=false`, `skipped=true`, "skipped: not enough space on …") — never a pass,
+  never dropped. The archive config is read once (it was read twice). Measured case: demo-hp 2026-09-24 —
+  the restore-test that filled `local-lvm` would now be refused (needs 32.1 GB, 23.2 GB free).
+- **R-672 — a failed scratch teardown is retried every 10 minutes** (`Engine.RetryScratchTeardown`, the
+  daemon's janitor), not only by Recover at agent start; never a vmid a running test owns; after 3 failed
+  tries the operator is told through a failed restore-test record naming the scratch guest.
+- **R-672 — a thin pool crossing 90 % requests an immediate host report** (`Observer.SetThinHighTrigger`, the
+  storage watchdog's read path, every few seconds; re-armed below 85 %), so the hub's alarm sees it in
+  seconds, not at the next 15-minute report.
+- **R-673 — the stale-lock sweep runs on the same 10-minute timer** (it ran only at start: a stale
+  `snapshot-delete` lock blocked 9201's whole-box backups for five hours), holding the one-heavy-operation
+  gate so no agent backup can start between its "no vzdump running" check and its unlock.
+- Red-proofs: six, each seen failing (REPORT).
+
 ## v0.132.0 — a controller that dies slowly is reported, not just restarted (2026-09-17, R-539)
 
 > **RELEASED 2026-09-17** by `scripts/release-agent.sh` — tag `v0.132.0`, sha256 `4afe815749a41b327ebe4a98a4557ad2acffb1fa71740a3a8b8003473b835321`. **Vouched 2026-09-17** for Day-0 installs on the operator's word, together with golden **0.246.0** (the hub's R-120 gate required the newer golden first). Delivered to demo-hp and the N100 by an operator-signed `agent_update` job each (operator ruling 3 of 2026-09-16), not by a floor.

CONTEXT.md

@@ -1,5 +1,22 @@
 # CONTEXT — felhom-agent working state
 
+
+> **2026-09-25 night — v0.133.0 AND v0.134.0 DELIVERED to both demo boxes (CC-signed `agent_update`, ruling 1);
+> restore test back ON (the `-1` config kept as `agent.json.night-0925-off`). v0.134.0 = R-685:** `backup/runner.go`
+> `spaceFits` — a vzdump to a LOCAL target needs free ≥ newest archive of that guest × 1.25 + 1 GiB (PVE prunes
+> only after success); a shortfall is a named skip (`BackupSkipNoSpacePrefix`), fail-open on PBS / first backup /
+> unknown usage. **Free space comes from `NodeStorage` (`GET /nodes/<n>/storage`) — `ListStorage` (`GET /storage`)
+> has NO usage**; the first build read it and a real vzdump started in its own live test (aborted, no archive).
+> demo-hp: `local_backup_retention` 1 (operator option A, saved `agent.json.pre-a4-retention`). Peti's box: nothing.
+
+> **2026-09-24 — v0.133.0 RELEASED, NOT DELIVERED (R-672, R-673).** Restore-test space preflight
+> (`reconcile/restoretest_space.go`, `internal/restorespace`): uncompressed size from the vzdump log / PBS size,
+> × 1.2 + 5 GiB, thin metadata, off the tested guest's pool, unknown refuses, reported as `skipped` non-pass.
+> Janitor (`cmd/felhom-agent/janitor.go`) every 10 min: `Engine.RetryScratchTeardown` + stale-lock sweep under
+> the heavy-op gate. Thin pool ≥ 90 % → immediate report (hub v0.124.0 alarms). **Operator rulings 2026-09-24
+> (evening):** the scheduled restore-test is OFF on both demo hosts (`backup.restore_test_eval_interval_seconds:
+> -1` — note: 0 means the 6 h DEFAULT, only a negative disables) until v0.133.0 is delivered there; saved configs
+> `/etc/felhom-agent/agent.json.pre-r672`. demo-hp 9201 was repaired (stop, fsck, start; two Redis AOF tails cut).
 > Snapshot of the current state + open threads. Authoritative history lives in `CHANGELOG.md` (top
 > entry = current); the end-of-task detail lives in `REPORT.md`.
 

REPORT.md

@@ -1,65 +1,24 @@
-# REPORT — agent v0.132.0: a controller that dies slowly is reported, not just restarted
+# REPORT — agent v0.135.0: the restore test proves only backups of a guest (R-689, 2026-09-27)
 
-**2026-09-17.** R-539, operator ruling 3 of 2026-09-16. Architecture: `felhom.eu/documentation/architecture/03-host-agent.md` (the supervisor and its budget).
+**Baseline:** `main` `7403c2a838db`, v0.134.0 on both demo hosts. **Commits:** `d4be12c` (fix + tests), `9ff937d`
+(CHANGELOG, after the release). **Released** by `scripts/release-agent.sh`: tag `v0.135.0`, sha256
+`ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (operator).
 
-## Claims in the brief that turned out wrong — named first
+**Tests:** full suite rc=0; agent gates OK after the release. `TestR689_TheRestoreTestNeverPicksTheGolden` red-proofed
+(without the check it picks `local:backup/felhom-golden-0.236.0.tar.zst`); `TestR689_GuestBackupArchiveShapes`; two older
+picker fixtures moved to real archive names.
 
-1. **„Emit `controller_slow_crashloop` from the agent."** The agent has no event channel. It carries
-   timestamps in its heartbeat stanza and the **hub** mints the event when one moves. So the build was
-   three wire fields here **plus** a hub checker change (hub v0.117.0), not an event registration alone.
-2. **„Five kills 20 min apart is too long — make the interval configurable for the test."** Not needed,
-   and not done: the proof restart from B.4(a) was already persisted, so four more real kills ~8 minutes
-   apart reached five in 24 hours inside the session, with the **production** window. 8 minutes keeps any
-   15-minute window at two restarts, so the fast brake never interferes.
+**Delivered** by signed `agent_update` (felhom-opsign, key `felhom-op-1`): demo-felhom committed 12:22:53 CEST, demo-hp
+12:29:02 CEST (`felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`).
 
-## What shipped
+**Then v0.136.0 (`16dbc83`, sha256 `2eb0b5eb…fa6d755`, NOT vouched):** the scheduler's read-only verdict on demo-hp
+(`-selftest=restore-test-due`, `D1/D1-selftest-due-demo-hp.txt`) skipped the golden but picked a leftover archive of
+guest 9100 (deleted in August). The guest must now exist; red-proof `D1/RP-r689-deleted-guest.txt`. Signed-delivered to
+both hosts. The verdict after 0.136.0: `D1/D1-selftest-due-after-0136.txt`.
 
-`internal/localapi/controllersupervisor.go`: beside the unchanged 3-in-15 brake, restarts the supervisor
-performed in the last 24 hours. At the fifth, `slow_crashloop_since` moves (at most once per 24 h);
-`slow_crashloop` and `restarts_24h` ride the stanza (`internal/hub/report.go`). Persisted per guest in
-`/var/lib/felhom-agent/guests/<vmid>/controller-restarts-24h.json` (tmp + rename, 0600); unreadable or
-corrupt → WARN and a clean start. Never stops restarting. Deliberate kills count. The startup line prints
-the new limits.
-
-## Red-proofs — each seen failing, then passing
-
-| test | break | failure seen |
-|---|---|---|
-| `TestControllerSupervisor_SlowCrashloop` | no counter | `five restarts 20 minutes apart did not raise slow_crashloop — this is R-539` |
-| same | once-per-24h guard removed | `the raise moved again on the 6th restart … mailed per restart` |
-| `…SlowCounterSurvivesAgentRestart` | save removed | `the agent restart reset the slow counter … Restarts24h:1` |
-
-Negative control: restarts 7 h apart never raise it. Wire shape extended (`restarts_24h`, `slow_crashloop`).
-
-## Gates and release
-
-`go build ./... && go vet ./... && go test ./...` — green, 30 packages. `agent_gates.py --fast` — all OK.
-Released by `scripts/release-agent.sh`: tag `v0.132.0`, sha256
-`4afe815749a41b327ebe4a98a4557ad2acffb1fa71740a3a8b8003473b835321`, verified by independent download.
-**Not vouched** for Day-0 installs (the operator's act). Code `18d03bd`, release record `CHANGELOG` commit after it.
-
-## Delivery — operator-signed, per box (ruling 1 of 2026-09-16)
-
-| box | signed | authorised → completed | committed | startup line |
-|---|---|---|---|---|
-| demo-hp (`demo-hp-bb76ea`) | 08:27:36Z | 08:29:25Z | 08:30:29Z | `slow_crashloop_max=5 slow_crashloop_window=24h0m0s` |
-| N100 (`demo-felhom-8363b5`) | 08:27:36Z | 08:34:48Z | 08:35:52Z | same |
-
-Peti's box: not touched. Evidence: `felhom.eu/documentation/audits/evidence-chaos-fixes-2026-09-17/partC-agent-update.txt`.
-
-## Live validation — PASS, production window
-
-On demo-hp guest 9201: five controller kills (08:51:53Z from B.4(a), then 08:57, 09:05, 09:13, 09:21),
-each restarted by the agent in 42–56 s, never by hand; the fast brake never armed. At #5:
-`SLOW CRASH-LOOP — … restarts_24h=5 window=24h0m0s threshold=5`; persisted file holds the five times and the
-raise. The hub minted `controller_slow_crashloop` at 09:29:40Z and **exactly one** operator mail arrived.
-Evidence: `…/partC-live-slow-crashloop.txt`.
-
-## Teardown
-
-Nothing provisioned. Stated effect on the demo box: 9201's counter holds five restarts and stays raised
-until 09:22Z on 2026-09-18; it cannot mail again inside that window.
-
-## Observations
-
-1. The persisted file writes the zero raise time as `"0001-01-01T00:00:00Z"` (Go `omitempty` does not omit a zero `time.Time`). **NOT-A-FINDING: the loader reads it back as zero (`IsZero`), which the live file and the restart test both exercised; cosmetic only.**
+**Then v0.137.0 (`3ef095f`):** 0.136.0's lookup met PVE's 403 "permission denied" (the token sees only its pool), not
+"does not exist", so the local tier read UNKNOWN every evaluation (`D1/D1-selftest-due-after-0136.txt`) — a regression of
+0.136.0, fixed: a guest the agent cannot read is not one it manages. Red-proof `D1/RP-r689-acl.txt`; verified read-only on
+demo-hp with the pre-release binary before the release (`D1/D1-selftest-due-0137-pre.txt`): both leftovers skipped, the
+off-site tier due on 9201, the local tier waiting for today's 9201 archive to settle. Three agent releases in one session
+— each the smallest fix of what the box showed.

internal/backup/backup_test.go

@@ -2,6 +2,7 @@ package backup
 
 import (
 	"context"
+	"fmt"
 	"encoding/json"
 	"errors"
 	"io"
@@ -21,14 +22,18 @@ type fakeBackupAPI struct {
 	vzdumpErr  error
 	waitErr    error
 	cfg        proxmox.GuestConfig
+	goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
+	aclGuests  map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
 	cfgErr     error
 	content    []proxmox.StorageContent
 	contentErr error
-	storages   []proxmox.Storage // returned by ListStorage (the local-prune scope gate)
-	storageErr error
-	vzdumps    []proxmox.VzdumpOptions
-	logLines   []string      // returned by TaskLogTail (e.g. "INFO: backup mode: stop")
-	waitGate   chan struct{} // if non-nil, WaitTask blocks until closed (8B.2 watcher timing)
+	storages   []proxmox.Storage // returned by ListStorage (the local-prune scope gate) — DEFINITIONS: like
+	// production's GET /storage, it never carries usage; ListStorage strips Avail/Used (R-685's live lesson)
+	nodeStorages []proxmox.Storage // returned by NodeStorage (GET /nodes/{node}/storage — WITH usage)
+	storageErr   error
+	vzdumps      []proxmox.VzdumpOptions
+	logLines     []string      // returned by TaskLogTail (e.g. "INFO: backup mode: stop")
+	waitGate     chan struct{} // if non-nil, WaitTask blocks until closed (8B.2 watcher timing)
 }
 
 func (f *fakeBackupAPI) Vzdump(_ context.Context, o proxmox.VzdumpOptions) (string, error) {
@@ -41,13 +46,30 @@ func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOpti
 	}
 	return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
 }
-func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
+func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
+	if f.aclGuests[vmid] {
+		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
+	}
+	if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
+		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
+	}
 	return f.cfg, f.cfgErr
 }
 func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
 	return f.content, f.contentErr
 }
 func (f *fakeBackupAPI) ListStorage(_ context.Context) ([]proxmox.Storage, error) {
+	out := make([]proxmox.Storage, len(f.storages))
+	for i, s := range f.storages {
+		s.Avail, s.Used, s.Total = 0, 0, 0 // GET /storage has no usage — a fake that had it hid R-685's defect
+		out[i] = s
+	}
+	return out, f.storageErr
+}
+func (f *fakeBackupAPI) NodeStorage(_ context.Context) ([]proxmox.Storage, error) {
+	if f.nodeStorages != nil {
+		return f.nodeStorages, f.storageErr
+	}
 	return f.storages, f.storageErr
 }
 func (f *fakeBackupAPI) TaskLogTail(_ context.Context, _ string, _ int) ([]string, error) {
@@ -137,13 +159,14 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
 func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
 	const big = 4 << 30 // a plausible whole-guest archive
 	api := &fakeBackupAPI{content: []proxmox.StorageContent{
-		{VolID: "a", Content: "backup", CTime: 10, Size: big},
-		{VolID: "b", Content: "backup", CTime: 99, Size: big},
+		// R-689: real vzdump names with their vmid — only a backup OF A GUEST is a candidate.
+		{VolID: "local:backup/vzdump-lxc-9001-a.tar.zst", VMID: 9001, Content: "backup", CTime: 10, Size: big},
+		{VolID: "local:backup/vzdump-lxc-9001-b.tar.zst", VMID: 9001, Content: "backup", CTime: 99, Size: big},
 		{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
 	}}
 	r := NewBackupRunner(api, "local", "", "", "", quiet())
 	vol, err := r.PickRestoreCandidate(context.Background())
-	if err != nil || vol != "b" {
+	if err != nil || vol != "local:backup/vzdump-lxc-9001-b.tar.zst" {
 		t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
 	}
 	// no backups → "".
@@ -163,12 +186,12 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
 // `pick = "phantom" want the newest COMPLETE archive 'real'`.
 func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
 	api := &fakeBackupAPI{content: []proxmox.StorageContent{
-		{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
-		{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
+		{VolID: "felhom-pbs:backup/ct/9001/real", VMID: 9001, Content: "backup", CTime: 10, Size: 4 << 30},
+		{VolID: "felhom-pbs:backup/ct/9001/phantom", VMID: 9001, Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
 	}}
 	r := NewBackupRunner(api, "local", "", "", "", quiet())
 	vol, err := r.PickRestoreCandidate(context.Background())
-	if err != nil || vol != "real" {
+	if err != nil || vol != "felhom-pbs:backup/ct/9001/real" {
 		t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
 	}
 }

internal/backup/r685_space_test.go

@@ -0,0 +1,83 @@
+package backup
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// R-685 (v0.134.0) — a whole-box backup that cannot fit its LOCAL target is a named skip BEFORE anything
+// starts, with the numbers in the record's Error, never a vzdump that fills the disk and fails.
+
+const gib = int64(1) << 30
+
+func spaceAPI(avail int64, lastArchive int64, typ string) *fakeBackupAPI {
+	api := &fakeBackupAPI{vzdumpUPID: "UPID:vzdump:1",
+		storages: []proxmox.Storage{{Storage: "local", Type: typ, Content: "backup", Avail: avail}}}
+	if lastArchive > 0 {
+		api.content = []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst",
+			Content: "backup", VMID: 9201, Size: lastArchive, CTime: 1790280000}}
+	}
+	return api
+}
+
+// TestR685_BackupThatCannotFitIsSkipped — demo-hp's shape: an 8.2 GB archive, 4 GiB free. No vzdump is
+// started; the record says why, with the numbers, under a stable prefix.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the spaceFits call from backup() — this test fails at "a vzdump
+// was started on a target that cannot hold it".
+func TestR685_BackupThatCannotFitIsSkipped(t *testing.T) {
+	api := spaceAPI(4*gib, 8182759056, "dir")
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	rec, err := r.Backup(context.Background(), 9201)
+	if len(api.vzdumps) != 0 {
+		t.Fatalf("a vzdump was started on a target that cannot hold it: %+v", api.vzdumps)
+	}
+	if err == nil || rec.Success || !strings.HasPrefix(rec.Error, BackupSkipNoSpacePrefix) {
+		t.Fatalf("want a named skip, got err=%v rec=%+v", err, rec)
+	}
+	for _, want := range []string{"4.0 GiB free", "7.6 GiB", "10.5 GiB"} {
+		if !strings.Contains(rec.Error, want) {
+			t.Errorf("the reason must carry the numbers (%q missing): %s", want, rec.Error)
+		}
+	}
+}
+
+// TestR685_BackupThatFitsRuns — the same archive with 16 GiB free (demo-hp after tonight's prune) runs.
+func TestR685_BackupThatFitsRuns(t *testing.T) {
+	api := spaceAPI(16*gib, 8182759056, "dir")
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	_, _ = r.Backup(context.Background(), 9201)
+	if len(api.vzdumps) != 1 {
+		t.Fatalf("a backup that fits must run, vzdumps=%d", len(api.vzdumps))
+	}
+}
+
+// TestR685_FailsOpen — never refuse on what is not KNOWN: a PBS target, a first backup (no archive to
+// size from), an unknown free figure, or a storage list that cannot be read.
+func TestR685_FailsOpen(t *testing.T) {
+	cases := map[string]*fakeBackupAPI{
+		"pbs target":    spaceAPI(1*gib, 8*gib, "pbs"),
+		"first backup":  spaceAPI(1*gib, 0, "dir"),
+		"avail unknown": spaceAPI(0, 8*gib, "dir"),
+		"storage list error": func() *fakeBackupAPI {
+			a := spaceAPI(1*gib, 8*gib, "dir")
+			a.storageErr = context.DeadlineExceeded
+			return a
+		}(),
+	}
+	for name, api := range cases {
+		target := "local"
+		if name == "pbs target" {
+			api.storages[0].Storage = "felhom-pbs"
+			target = "felhom-pbs"
+		}
+		r := NewBackupRunner(api, target, proxmox.ModeSnapshot, "", "", quiet())
+		_, _ = r.Backup(context.Background(), 9201)
+		if len(api.vzdumps) != 1 {
+			t.Errorf("%s: the preflight must fail OPEN, but no vzdump ran", name)
+		}
+	}
+}

internal/backup/r689_golden_test.go

@@ -0,0 +1,112 @@
+package backup
+
+import (
+	"context"
+	"fmt"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// R-689 (v0.135.0) — demo-hp keeps its golden template in `local:backup/`. It is content "backup",
+// 654 MB and so "plausibly complete", and it was the newest SETTLED entry: the restore test picked it
+// every 6 h and failed extractconfig with a 403 (measured 2026-09-24 10:36, 09-25 04:57 and 10:57),
+// while the guest's own archive — younger than the 24 h settle — went untested and nothing was proven.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the guestBackupArchive call from PickSettledRestoreCandidateOn —
+// this test then picks `local:backup/felhom-golden-0.236.0.tar.zst`.
+func TestR689_TheRestoreTestNeverPicksTheGolden(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790370000) // 2026-09-25 ~19:00Z
+	api := &fakeBackupAPI{content: []proxmox.StorageContent{
+		// the guest's real archive, settled (older than the cutoff below)
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 3*day},
+		// the golden: newer, settled, big, and NOT a backup of a guest
+		{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 2*day},
+		// a hand-copied tarball that PVE happens to attribute to a vmid — the name is not a vzdump's
+		{VolID: "local:backup/copy-of-9201.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 2*day},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got != "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst" {
+		t.Fatalf("picked %q — the restore test must prove a backup OF A GUEST", got)
+	}
+}
+
+func TestR689_GuestBackupArchiveShapes(t *testing.T) {
+	for _, c := range []struct {
+		e  proxmox.StorageContent
+		ok bool
+	}{
+		{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst", VMID: 9201}, true},
+		{proxmox.StorageContent{VolID: "local:backup/vzdump-qemu-300-2026_09_24-21_59_25.vma.zst", VMID: 300}, true},
+		{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z", VMID: 9201}, true},
+		{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
+		{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
+		{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
+		{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false},       // no vmid reported
+	} {
+		if ok, why := guestBackupArchive(c.e); ok != c.ok {
+			t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
+		}
+	}
+}
+
+// R-689 (v0.136.0) — the measured demo-hp shape right after v0.135.0: the golden (skipped), a leftover archive
+// of guest 9100 deleted in August (settled), and today's archive of 9201 (not settled yet). The pick must be
+// NOTHING — never the deleted guest's archive. With 9201's archive settled, that one.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the known-guest check — the pick is the 9100 leftover.
+func TestR689_AnArchiveOfADeletedGuestIsNeverPicked(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790476000)
+	api := &fakeBackupAPI{goneGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
+		{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 14*day},
+		{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil || got != "" {
+		t.Fatalf("picked %q err=%v — a deleted guest's archive proves nothing about this box", got, err)
+	}
+	got, _, _ = r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now, 0).UTC())
+	if got != "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst" {
+		t.Fatalf("with 9201's archive settled the pick is %q", got)
+	}
+}
+
+// Any OTHER lookup failure is not "the guest is gone": the tier must read UNKNOWN (an error), never
+// "nothing to prove".
+func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
+	api := &fakeBackupAPI{cfgErr: fmt.Errorf("proxmox: connection refused"), content: []proxmox.StorageContent{
+		{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: 10},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err == nil {
+		t.Fatal("a failed guest lookup read as a clean answer")
+	}
+}
+
+// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
+// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
+// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
+func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790476000)
+	api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
+		{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil || got != "" {
+		t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
+	}
+}

internal/backup/runner.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"log/slog"
 	"sort"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -22,6 +23,10 @@ type BackupAPI interface {
 	StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error)
 	// ListStorage enumerates storages (name+type) — used to scope local-only retention (never prune PBS).
 	ListStorage(ctx context.Context) ([]proxmox.Storage, error)
+	// NodeStorage is GET /nodes/{node}/storage — the storages WITH live usage (avail/used). R-685's space
+	// preflight reads free space HERE: ListStorage (GET /storage) is the cluster DEFINITIONS and carries no
+	// usage at all — measured live 2026-09-24 on demo-hp, where reading it let a backup through.
+	NodeStorage(ctx context.Context) ([]proxmox.Storage, error)
 	// TaskLogTail reads trailing task-log lines — used to read the ACTUAL vzdump mode
 	// (PVE may downgrade a requested snapshot to stop for a stopped guest — spike B1).
 	TaskLogTail(ctx context.Context, upid string, limit int) ([]string, error)
@@ -170,6 +175,16 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
 		rec.UncoveredVolumes = []string{}
 	}
 
+	// R-685 (v0.134.0): will the new archive FIT on a local target? Asked before anything runs, so a
+	// target that cannot hold it is a named SKIP with the numbers, not a nightly "No space left on
+	// device" that only the vzdump log explains (demo-hp, every night from 2026-09-23 — R-684).
+	if ok, why := r.spaceFits(ctx, vmid); !ok {
+		rec.Error = BackupSkipNoSpacePrefix + why
+		rec.DurationSeconds = time.Since(start).Seconds()
+		r.logger.Warn("backup SKIPPED by the space preflight (R-685) — nothing was started", "vmid", vmid, "target", r.target, "reason", why)
+		return rec, fmt.Errorf("backup: %s", rec.Error)
+	}
+
 	upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
 		VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
 		PruneBackups: r.localPruneSpec(ctx), // local target → keep-last=N; PBS/unknown → "" (no prune)
@@ -217,6 +232,56 @@ func (r *BackupRunner) backup(ctx context.Context, vmid int, onSnapshot func())
 	return rec, nil
 }
 
+// BackupSkipNoSpacePrefix starts a backup record's Error when the space preflight refused (R-685) — a
+// stable prefix the controller's page and the hub can key on.
+const BackupSkipNoSpacePrefix = "skipped: not enough space: "
+
+// Space preflight margins (R-685): the new archive is predicted as the newest archive of this guest on the
+// target × backupSpaceGrowth, plus backupSpaceFloorBytes of headroom for the host. MEASURED 2026-09-24:
+// demo-hp 9201's archives grew 5.8 → 6.2 → 6.9 → 7.6 GB in four nights (+10 % a night at worst), so 1.25
+// covers two nights' growth. PVE prunes old archives only AFTER a successful backup, so the free space
+// must hold the new archive while every kept one still exists.
+const (
+	backupSpaceGrowth     = 1.25
+	backupSpaceFloorBytes = int64(1) << 30
+)
+
+// spaceFits answers whether a new archive of vmid fits on a LOCAL (non-PBS) target. It FAILS OPEN — a
+// backup is the thing being protected, so an unreadable storage, an unknown type or a first backup (no
+// previous archive to size from) proceeds and says so; only a POSITIVE "it does not fit" refuses.
+func (r *BackupRunner) spaceFits(ctx context.Context, vmid int) (bool, string) {
+	// NodeStorage, never ListStorage: only the node view carries avail (see BackupAPI.NodeStorage).
+	stores, err := r.api.NodeStorage(ctx)
+	if err != nil {
+		r.logger.Warn("backup: space preflight could not read storage usage — proceeding (fail-open)", "target", r.target, "err", err)
+		return true, ""
+	}
+	var st *proxmox.Storage
+	for i := range stores {
+		if stores[i].Storage == r.target {
+			st = &stores[i]
+			break
+		}
+	}
+	if st == nil || st.Type == "pbs" || st.Avail <= 0 {
+		return true, "" // PBS dedups and has its own lifecycle; an unknown avail never refuses
+	}
+	_, last, err := r.latestArchive(ctx, vmid)
+	if err != nil || last <= 0 {
+		r.logger.Info("backup: space preflight has no previous archive to size from — proceeding", "vmid", vmid, "target", r.target)
+		return true, ""
+	}
+	need := int64(float64(last)*backupSpaceGrowth) + backupSpaceFloorBytes
+	if st.Avail >= need {
+		r.logger.Info("backup: space preflight passed", "vmid", vmid, "target", r.target, "last_archive_bytes", last, "need_bytes", need, "avail_bytes", st.Avail)
+		return true, ""
+	}
+	return false, fmt.Sprintf("%s has %s free; the last archive of guest %d was %s, so a new one needs about %s (old archives are removed only after a successful backup)",
+		r.target, humanGiB(st.Avail), vmid, humanGiB(last), humanGiB(need))
+}
+
+func humanGiB(b int64) string { return fmt.Sprintf("%.1f GiB", float64(b)/(1<<30)) }
+
 // watchForSnapshot polls the running backup's task log until it sees the storage-snapshot marker
 // (→ onSnapshot once) or the requested mode is reported as `stop` (→ downgraded; the marker will
 // never come, so stop watching) or ctx is cancelled (backup finished). Best-effort: a log-read
@@ -294,10 +359,42 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
 	}
 	var best string
 	var bestCTime int64 = -1
+	known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
 	for _, e := range contents {
 		if e.Content != "backup" {
 			continue
 		}
+		// R-689 (v0.135.0): only a backup OF A GUEST is a restore-test candidate. demo-hp keeps its golden
+		// template in `local:backup/` — content "backup", 654 MB, plausibly complete — and it was picked as
+		// the newest settled archive every 6 h and failed extractconfig (403) each time, while the guest's
+		// real archive went untested.
+		if ok, why := guestBackupArchive(e); !ok {
+			r.noteNotAGuestBackupOnce(e, why)
+			continue
+		}
+		// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
+		// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
+		// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
+		// archive; any OTHER lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove".
+		if _, seen := known[e.VMID]; !seen {
+			_, err := r.api.GuestConfig(ctx, e.VMID)
+			switch {
+			case err == nil:
+				known[e.VMID] = true
+			case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
+				// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
+				// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
+				// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
+				// read is not one it manages; its archive is not a candidate.
+				known[e.VMID] = false
+			default:
+				return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
+			}
+		}
+		if !known[e.VMID] {
+			r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
+			continue
+		}
 		if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
 			continue // not settled yet — a newer archive is not a reason to re-prove an older one
 		}
@@ -527,3 +624,46 @@ func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.R
 	}
 	return rt
 }
+
+// guestBackupArchive reports whether a storage entry is a whole-guest backup of a known guest — a
+// `vzdump-<type>-<vmid>-…` file on a dir storage, or a `backup/{ct,vm}/<vmid>/<time>` snapshot on a PBS
+// datastore — whose vmid the storage itself reports. Anything else in a backup content type (a golden
+// template, a hand-copied tarball) is not a backup of a guest and is never restore-tested (R-689).
+// Pure, so the rule is unit-tested without a storage.
+func guestBackupArchive(e proxmox.StorageContent) (bool, string) {
+	if e.VMID <= 0 {
+		return false, "not a backup of a guest (the storage reports no vmid)"
+	}
+	vol := e.VolID
+	if i := strings.Index(vol, ":"); i >= 0 {
+		vol = vol[i+1:]
+	}
+	vol = strings.TrimPrefix(vol, "backup/")
+	vmid := strconv.Itoa(e.VMID)
+	switch {
+	case strings.HasPrefix(vol, "vzdump-lxc-"+vmid+"-"), strings.HasPrefix(vol, "vzdump-qemu-"+vmid+"-"):
+		return true, ""
+	case strings.HasPrefix(vol, "ct/"+vmid+"/"), strings.HasPrefix(vol, "vm/"+vmid+"/"):
+		return true, ""
+	}
+	return false, "not a vzdump archive or a PBS snapshot of guest " + vmid
+}
+
+// noteNotAGuestBackupOnce logs, once per volid, that a backup-content entry is not a restore-test
+// candidate because it is not a backup of a guest (R-689). INFO, not WARN: a golden template kept in
+// the backup directory is the operator's, and not a fault.
+func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why string) {
+	r.rejectedMu.Lock()
+	if r.rejected == nil {
+		r.rejected = map[string]struct{}{}
+	}
+	_, seen := r.rejected[e.VolID]
+	if !seen {
+		r.rejected[e.VolID] = struct{}{}
+	}
+	r.rejectedMu.Unlock()
+	if !seen {
+		r.logger.Info("backup: restore-test skips an entry that is not a backup of a guest",
+			"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
+	}
+}