REPORT: v0.136.0 and v0.137.0

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

CHANGELOG.md

@@ -1,3 +1,32 @@
+## v0.136.0 — … of a guest that still EXISTS (2026-09-27, R-689 second half)
+
+> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.136.0` (`16dbc83`), sha256 `2eb0b5ebe253defd68b322312bbac12418c051b0a7a0d2d1831310d97fa6d755`, verified by download. **NOT vouched** (the operator's act).
+
+**MinAgent impact:** none required by any controller.
+
+- **R-689, second half.** Read on demo-hp right after v0.135.0 with the read-only `-selftest=restore-test-due`: the
+  golden was skipped, and the pick fell to `vzdump-lxc-9100-2026_08_21…` — a leftover of a guest deleted in August. A
+  candidate's guest must now exist on this node (`GuestConfig`): "does not exist" skips the archive (INFO once per
+  volid); any other lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove". Tests
+  `TestR689_AnArchiveOfADeletedGuestIsNeverPicked` (red-proofed: without the check it picks the 9100 leftover),
+  `TestR689_AGuestLookupFailureIsUnknownNotEmpty`. A second agent release in one session — the first half was found
+  incomplete on the box.
+
+## v0.135.0 — the restore test proves only backups OF A GUEST (2026-09-27, R-689)
+
+> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.135.0` (`d4be12c`), sha256 `ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (the operator's act).
+
+**MinAgent impact:** none required by any controller.
+
+- **R-689** (`backup/runner.go` `guestBackupArchive`). demo-hp keeps its golden template in `local:backup/` — content
+  "backup", 654 MB, plausibly complete, the newest settled entry — and the scheduled restore test picked it every 6 h and
+  failed `extractconfig` with a 403, while the guest's real archive went untested. A restore-test candidate is now a
+  `vzdump-{lxc,qemu}-<vmid>-…` file or a PBS `backup/{ct,vm}/<vmid>/…` snapshot whose vmid the storage reports; anything
+  else is skipped with one INFO line per volid (not the "INCOMPLETE archive" WARN). Tests
+  `TestR689_TheRestoreTestNeverPicksTheGolden` (red-proof: without the check it picks the golden) and
+  `TestR689_GuestBackupArchiveShapes`; two older picker tests' fixtures moved to real archive names.
+  Evidence: `felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`.
+
 ## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
 
 > **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).

REPORT.md

@@ -1,17 +1,24 @@
-# REPORT — agents v0.133.0 + v0.134.0 delivered; v0.134.0 released (night 2026-09-25)
+# REPORT — agent v0.135.0: the restore test proves only backups of a guest (R-689, 2026-09-27)
 
-**Delivered** to demo-felhom and demo-hp by CC-signed `agent_update` jobs (ruling 1, 2026-09-16), each verified
+**Baseline:** `main` `7403c2a838db`, v0.134.0 on both demo hosts. **Commits:** `d4be12c` (fix + tests), `9ff937d`
-against the published package's sha256 first: v0.133.0 (committed 21:35 / 21:50 CEST), then v0.134.0 (22:52 /
+(CHANGELOG, after the release). **Released** by `scripts/release-agent.sh`: tag `v0.135.0`, sha256
-22:58). Hub reads 0.134.0 on both. Peti's box: nothing. Restore test back ON on both (A3: demo-felhom PASS 85 s;
+`ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (operator).
-demo-hp refused for space, correctly). demo-hp `local_backup_retention: 1` (operator option A).
 
-**v0.134.0 = R-685 (agent half):** a vzdump to a LOCAL target needs free ≥ newest archive × 1.25 + 1 GiB; a
+**Tests:** full suite rc=0; agent gates OK after the release. `TestR689_TheRestoreTestNeverPicksTheGolden` red-proofed
-shortfall is a named skip before anything starts; fail-open on PBS / first backup / unknown usage. Tag `v0.134.0`
+(without the check it picks `local:backup/felhom-golden-0.236.0.tar.zst`); `TestR689_GuestBackupArchiveShapes`; two older
-(`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download; CI jobs
+picker fixtures moved to real archive names.
-975–977 success. **Not vouched** for Day-0.
 
-**Found live before the release:** the first build read free space from `GET /storage` (no usage) and failed open —
+**Delivered** by signed `agent_update` (felhom-opsign, key `felhom-op-1`): demo-felhom committed 12:22:53 CEST, demo-hp
-its own "must refuse" test started a real vzdump of demo-hp 9201, aborted by CC after 5 min 16 s, no archive left.
+12:29:02 CEST (`felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`).
-Fixed (`NodeStorage`), test fake made honest (`ListStorage` strips usage), two red-proofs, re-proven live with safe
+
-builds (a hard stop before vzdump): ×10 refused with real numbers, ×1.25 passed. Evidence:
+**Then v0.136.0 (`16dbc83`, sha256 `2eb0b5eb…fa6d755`, NOT vouched):** the scheduler's read-only verdict on demo-hp
-`felhom.eu/documentation/audits/night-2026-09-25/F/`.
+(`-selftest=restore-test-due`, `D1/D1-selftest-due-demo-hp.txt`) skipped the golden but picked a leftover archive of
+guest 9100 (deleted in August). The guest must now exist; red-proof `D1/RP-r689-deleted-guest.txt`. Signed-delivered to
+both hosts. The verdict after 0.136.0: `D1/D1-selftest-due-after-0136.txt`.
+
+**Then v0.137.0 (`3ef095f`):** 0.136.0's lookup met PVE's 403 "permission denied" (the token sees only its pool), not
+"does not exist", so the local tier read UNKNOWN every evaluation (`D1/D1-selftest-due-after-0136.txt`) — a regression of
+0.136.0, fixed: a guest the agent cannot read is not one it manages. Red-proof `D1/RP-r689-acl.txt`; verified read-only on
+demo-hp with the pre-release binary before the release (`D1/D1-selftest-due-0137-pre.txt`): both leftovers skipped, the
+off-site tier due on 9201, the local tier waiting for today's 9201 archive to settle. Three agent releases in one session
+— each the smallest fix of what the box showed.

internal/backup/backup_test.go

@@ -2,6 +2,7 @@ package backup
 
 import (
 	"context"
+	"fmt"
 	"encoding/json"
 	"errors"
 	"io"
@@ -21,6 +22,8 @@ type fakeBackupAPI struct {
 	vzdumpErr  error
 	waitErr    error
 	cfg        proxmox.GuestConfig
+	goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
+	aclGuests  map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
 	cfgErr     error
 	content    []proxmox.StorageContent
 	contentErr error
@@ -43,7 +46,13 @@ func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOpti
 	}
 	return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
 }
-func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
+func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
+	if f.aclGuests[vmid] {
+		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
+	}
+	if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
+		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
+	}
 	return f.cfg, f.cfgErr
 }
 func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {

internal/backup/r689_golden_test.go

@@ -2,6 +2,7 @@ package backup
 
 import (
 	"context"
+	"fmt"
 	"testing"
 	"time"
 
@@ -54,3 +55,58 @@ func TestR689_GuestBackupArchiveShapes(t *testing.T) {
 		}
 	}
 }
+
+// R-689 (v0.136.0) — the measured demo-hp shape right after v0.135.0: the golden (skipped), a leftover archive
+// of guest 9100 deleted in August (settled), and today's archive of 9201 (not settled yet). The pick must be
+// NOTHING — never the deleted guest's archive. With 9201's archive settled, that one.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the known-guest check — the pick is the 9100 leftover.
+func TestR689_AnArchiveOfADeletedGuestIsNeverPicked(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790476000)
+	api := &fakeBackupAPI{goneGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
+		{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 14*day},
+		{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil || got != "" {
+		t.Fatalf("picked %q err=%v — a deleted guest's archive proves nothing about this box", got, err)
+	}
+	got, _, _ = r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now, 0).UTC())
+	if got != "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst" {
+		t.Fatalf("with 9201's archive settled the pick is %q", got)
+	}
+}
+
+// Any OTHER lookup failure is not "the guest is gone": the tier must read UNKNOWN (an error), never
+// "nothing to prove".
+func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
+	api := &fakeBackupAPI{cfgErr: fmt.Errorf("proxmox: connection refused"), content: []proxmox.StorageContent{
+		{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: 10},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err == nil {
+		t.Fatal("a failed guest lookup read as a clean answer")
+	}
+}
+
+// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
+// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
+// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
+func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790476000)
+	api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
+		{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil || got != "" {
+		t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
+	}
+}

internal/backup/runner.go

@@ -359,6 +359,7 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
 	}
 	var best string
 	var bestCTime int64 = -1
+	known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
 	for _, e := range contents {
 		if e.Content != "backup" {
 			continue
@@ -371,6 +372,29 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
 			r.noteNotAGuestBackupOnce(e, why)
 			continue
 		}
+		// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
+		// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
+		// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
+		// archive; any OTHER lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove".
+		if _, seen := known[e.VMID]; !seen {
+			_, err := r.api.GuestConfig(ctx, e.VMID)
+			switch {
+			case err == nil:
+				known[e.VMID] = true
+			case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
+				// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
+				// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
+				// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
+				// read is not one it manages; its archive is not a candidate.
+				known[e.VMID] = false
+			default:
+				return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
+			}
+		}
+		if !known[e.VMID] {
+			r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
+			continue
+		}
 		if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
 			continue // not settled yet — a newer archive is not a reason to re-prove an older one
 		}