restore test takes only this box's archives (R-727): an archive encrypted with another key is another box's

Red-proof RP39. Released as v0.138.0 by release-agent.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

CHANGELOG.md

@@ -1,3 +1,33 @@
+## v0.137.0 — a guest outside the agent's ACL is not a known guest (2026-09-27, R-689, v0.136.0 regression)
+
+> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.137.0` (`3ef095f`), sha256 `766c9166916a1bd3674b0dc69081f8a7619e770f1402d8ad7705b395937e7627`, verified by download. **NOT vouched** (the operator's act).
+>
+> **VOUCHED 2026-09-28** with golden 0.276.0 (`min_agent` 0.131.0), on the operator's word of 2026-09-27: the hub logged
+> `Artifact manifest set: agent=0.137.0 golden=0.276.0 min_agent="0.131.0"`, and a Day-0 test install fetched this binary
+> through the manifest and sha-verified it. Evidence: `felhom.eu/documentation/audits/evidence-golden-0276-2026-09-28/`.
+
+**MinAgent impact:** none required by any controller.
+
+- v0.136.0 asked `GuestConfig` whether an archive's guest exists and treated anything but "does not exist" as a lookup
+  failure. PVE answers **403 "permission denied at /vms/<id>"** for a vmid outside the token's pool — so on demo-hp the
+  deleted guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot read is not one it
+  manages; its archive is skipped. `TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest`, red-proofed. Verified read-only
+  on demo-hp with the pre-release binary before the release.
+
+## v0.136.0 — … of a guest that still EXISTS (2026-09-27, R-689 second half)
+
+> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.136.0` (`16dbc83`), sha256 `2eb0b5ebe253defd68b322312bbac12418c051b0a7a0d2d1831310d97fa6d755`, verified by download. **NOT vouched** (the operator's act).
+
+**MinAgent impact:** none required by any controller.
+
+- **R-689, second half.** Read on demo-hp right after v0.135.0 with the read-only `-selftest=restore-test-due`: the
+  golden was skipped, and the pick fell to `vzdump-lxc-9100-2026_08_21…` — a leftover of a guest deleted in August. A
+  candidate's guest must now exist on this node (`GuestConfig`): "does not exist" skips the archive (INFO once per
+  volid); any other lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove". Tests
+  `TestR689_AnArchiveOfADeletedGuestIsNeverPicked` (red-proofed: without the check it picks the 9100 leftover),
+  `TestR689_AGuestLookupFailureIsUnknownNotEmpty`. A second agent release in one session — the first half was found
+  incomplete on the box.
+
 ## v0.135.0 — the restore test proves only backups OF A GUEST (2026-09-27, R-689)
 
 > **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.135.0` (`d4be12c`), sha256 `ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (the operator's act).

REPORT.md

@@ -1,14 +1,15 @@
-# REPORT — agent v0.135.0: the restore test proves only backups of a guest (R-689, 2026-09-27)
+# REPORT — 2026-09-28: agent v0.137.0 vouched with golden 0.276.0
 
-**Baseline:** `main` `7403c2a838db`, v0.134.0 on both demo hosts. **Commits:** `d4be12c` (fix + tests), `9ff937d`
-(CHANGELOG, after the release). **Released** by `scripts/release-agent.sh`: tag `v0.135.0`, sha256
-`ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (operator).
+No agent code changed. `configs/build-golden.sh` (v3.0.0) was USED, not changed, to bake golden 0.276.0 in the
+drill VM on DooPlex (RUNBOOK-manual-build §4.0–§4.1).
 
-**Tests:** full suite rc=0; agent gates OK after the release. `TestR689_TheRestoreTestNeverPicksTheGolden` red-proofed
-(without the check it picks `local:backup/felhom-golden-0.236.0.tar.zst`); `TestR689_GuestBackupArchiveShapes`; two older
-picker fixtures moved to real archive names.
+- **Vouched** in the hub's own form: `golden_version` 0.276.0, `agent_version` 0.137.0, `min_agent` 0.131.0 (the
+  MinAgent in controller v0.276.0's CHANGELOG header). The R-120 gate refused golden 0.258.0 first (negative
+  control). Rollback = the values before: agent 0.132.0, golden 0.258.0, min_agent 0.131.0.
+- **One sha, three places:** `766c9166…7627` in Gitea, on felhom-pve and on demo-hp.
+- **A new box comes up on it:** a Day-0 install (installer 1.28.0, `--force-gitea-golden`) in the drill VM fetched
+  agent 0.137.0 and golden 0.276.0 through the manifest, both sha-verified, and the controller answered healthy at
+  0.276.0 behind an armed claim gate. Not claimed.
+- CHANGELOG: the v0.137.0 entry carries the vouch.
 
-**Delivered** by signed `agent_update` (felhom-opsign, key `felhom-op-1`): demo-felhom committed 12:22:53 CEST, demo-hp
-12:29:02 CEST (`felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`).
-
-**NOT yet validated:** one scheduled restore-test cycle on demo-hp (due ~18:28 CEST, read into `D1/D1-cycle-demo-hp.txt`).
+Evidence: `felhom.eu/documentation/audits/evidence-golden-0276-2026-09-28/`.

internal/backup/backup_test.go

@@ -23,6 +23,7 @@ type fakeBackupAPI struct {
 	waitErr    error
 	cfg        proxmox.GuestConfig
 	goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
+	aclGuests  map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
 	cfgErr     error
 	content    []proxmox.StorageContent
 	contentErr error
@@ -46,6 +47,9 @@ func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOpti
 	return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
 }
 func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
+	if f.aclGuests[vmid] {
+		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
+	}
 	if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
 		return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
 	}

internal/backup/r689_golden_test.go

@@ -91,3 +91,22 @@ func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
 		t.Fatal("a failed guest lookup read as a clean answer")
 	}
 }
+
+// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
+// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
+// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
+//
+// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
+func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
+	const day = int64(86400)
+	now := int64(1790476000)
+	api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
+		{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
+		{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
+	}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
+	if err != nil || got != "" {
+		t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
+	}
+}

internal/backup/r727_own_archives_test.go

@@ -0,0 +1,74 @@
+package backup
+
+import (
+	"context"
+	"errors"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+const (
+	thisBoxKey  = "de:51:7a:18:cb:39:22:30:2c:84:f5:8b:d1:91:4b:7e:81:bb:69:b8:89:0f:57:ac:d3:59:e1:1a:62:25:11:2c"
+	earlierBox1 = "6b:ca:5f:3f:ca:0f:e2:3f:fb:24:62:89:bf:e7:64:59:9a:41:c5:e6:e3:9f:3f:5f:e1:71:7b:a1:9d:24:67:82"
+	earlierBox2 = "fe:3d:db:95:d4:df:ab:e1:7d:4a:89:fa:2b:07:53:6a:e4:d2:85:95:d1:90:27:4b:d9:c6:92:20:95:04:e5:d4"
+)
+
+// R-727 (v0.138.0) — the 2026-09-30 shape, measured on a fresh box for a returning customer: the PBS
+// namespace held two archives of earlier boxes (same guest 9201, same token) and this box's own, which was not
+// settled yet. The old picker chose the earlier box's newest settled archive and failed `wrong key`.
+// The CONSEQUENCE asserted: no archive of another box is ever picked; with this box's archive settled it is picked.
+// COMPANION RED-PROOF: remove the `ownKey != "" && !EqualFold(...)` skip → the first case picks 2026-09-16T21:59:54Z.
+func TestR727_TheRestoreTestTakesOnlyThisBoxsArchives(t *testing.T) {
+	day := int64(86400)
+	now := int64(1790740000) // 2026-09-30 ~04:00Z
+	own := proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}
+	api := &fakeBackupAPI{
+		storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
+		content: []proxmox.StorageContent{
+			{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
+			{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
+			own,
+		},
+	}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+
+	// 1. The night of 2026-09-30: this box's own archive is ~6 h old, not settled (cutoff 24 h) — nothing to prove.
+	got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now-day, 0).UTC())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got != "" {
+		t.Fatalf("picked %q — an archive of ANOTHER box is never this box's proof (R-727)", got)
+	}
+	// 2. A day later this box's own archive is settled — it is the one picked.
+	got, _, err = r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now+day, 0).UTC())
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got != own.VolID {
+		t.Fatalf("picked %q, want this box's own %q", got, own.VolID)
+	}
+}
+
+// An unencrypted storage (a local dir) holds only this box's vzdumps — no key filter applies.
+func TestR727_UnencryptedStorageIsNotFiltered(t *testing.T) {
+	api := &fakeBackupAPI{
+		storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
+		content:  []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_29-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1790710025}},
+	}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	if got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err != nil || got == "" {
+		t.Fatalf("got %q err %v", got, err)
+	}
+}
+
+// A storage-list failure makes the tier UNKNOWN (an error), never "nothing to prove".
+func TestR727_KeyLookupFailureIsUnknown(t *testing.T) {
+	api := &fakeBackupAPI{storageErr: errors.New("proxmox: GET /storage -> HTTP 500"), content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/x", Content: "backup", VMID: 9201}}}
+	r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
+	if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err == nil {
+		t.Fatal("a failed key lookup must surface as an error (tier UNKNOWN)")
+	}
+}

internal/backup/runner.go

@@ -357,6 +357,16 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
 	if err != nil {
 		return "", time.Time{}, err
 	}
+	// R-727 (v0.138.0): on an ENCRYPTED storage, only archives written with THIS storage's key are this box's.
+	// Measured 2026-09-30 on a fresh box for a returning customer: the PBS namespace still held two archives
+	// of earlier boxes (same guest id 9201, same token), the newest settled one was an earlier box's, and the
+	// test failed `wrong key` every evaluation. The archive carries no host id; its key fingerprint is the
+	// discriminator (PVE's content `encrypted`, the storage's `encryption-key`). A lookup failure returns
+	// the error — the tier reads UNKNOWN, never "nothing to prove".
+	ownKey, err := r.storageKeyFingerprint(ctx, target)
+	if err != nil {
+		return "", time.Time{}, fmt.Errorf("reading the key fingerprint of storage %s: %w", target, err)
+	}
 	var best string
 	var bestCTime int64 = -1
 	known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
@@ -372,6 +382,10 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
 			r.noteNotAGuestBackupOnce(e, why)
 			continue
 		}
+		if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
+			r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
+			continue
+		}
 		// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
 		// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
 		// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
@@ -381,14 +395,18 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
 			switch {
 			case err == nil:
 				known[e.VMID] = true
-			case strings.Contains(err.Error(), "does not exist"):
+			case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
+				// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
+				// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
+				// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
+				// read is not one it manages; its archive is not a candidate.
 				known[e.VMID] = false
 			default:
 				return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
 			}
 		}
 		if !known[e.VMID] {
-			r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d no longer exists on this node", e.VMID))
+			r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
 			continue
 		}
 		if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
@@ -663,3 +681,29 @@ func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why str
 			"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
 	}
 }
+
+// storageKeyFingerprint returns the named storage's client-side encryption key fingerprint ("" when the
+// storage is not encrypted — a local dir holds only this box's own vzdumps).
+func (r *BackupRunner) storageKeyFingerprint(ctx context.Context, target string) (string, error) {
+	sts, err := r.api.ListStorage(ctx)
+	if err != nil {
+		return "", err
+	}
+	for _, st := range sts {
+		if st.Storage == target {
+			return strings.TrimSpace(st.EncryptionKey), nil
+		}
+	}
+	return "", nil
+}
+
+// shortFP is the first 8 bytes of a key fingerprint, for a log line.
+func shortFP(fp string) string {
+	if fp == "" {
+		return "none"
+	}
+	if len(fp) > 23 {
+		return fp[:23] + "…"
+	}
+	return fp
+}

internal/proxmox/types.go

@@ -219,15 +219,18 @@ type Storage struct {
 	UsedFraction float64 `json:"used_fraction,omitempty"`
 
 	// Type-specific config (durable_id sources).
-	Server      string `json:"server,omitempty"`      // nfs/cifs/pbs server host
-	Export      string `json:"export,omitempty"`      // nfs export path
-	Share       string `json:"share,omitempty"`       // cifs share name
-	Datastore   string `json:"datastore,omitempty"`   // pbs datastore name
-	Fingerprint string `json:"fingerprint,omitempty"` // pbs server cert fingerprint
-	Username    string `json:"username,omitempty"`    // pbs auth id, e.g. "felhom@pbs!n100"
-	Namespace   string `json:"namespace,omitempty"`   // pbs namespace ("" = root; per-customer tenancy = S4)
-	VGName      string `json:"vgname,omitempty"`      // lvm/lvmthin volume group
-	ThinPool    string `json:"thinpool,omitempty"`    // lvmthin pool LV name
+	Server string `json:"server,omitempty"` // nfs/cifs/pbs server host
+	// EncryptionKey is the storage's own client-side key FINGERPRINT (pbs; the key itself stays in
+	// /etc/pve/priv). R-727: an archive encrypted with any other key was written by another box.
+	EncryptionKey string `json:"encryption-key,omitempty"`
+	Export        string `json:"export,omitempty"`      // nfs export path
+	Share         string `json:"share,omitempty"`       // cifs share name
+	Datastore     string `json:"datastore,omitempty"`   // pbs datastore name
+	Fingerprint   string `json:"fingerprint,omitempty"` // pbs server cert fingerprint
+	Username      string `json:"username,omitempty"`    // pbs auth id, e.g. "felhom@pbs!n100"
+	Namespace     string `json:"namespace,omitempty"`   // pbs namespace ("" = root; per-customer tenancy = S4)
+	VGName        string `json:"vgname,omitempty"`      // lvm/lvmthin volume group
+	ThinPool      string `json:"thinpool,omitempty"`    // lvmthin pool LV name
 }
 
 // StorageContent is one entry of GET /nodes/{node}/storage/{store}/content
@@ -239,4 +242,7 @@ type StorageContent struct {
 	Size    int64  `json:"size"`
 	CTime   int64  `json:"ctime"`
 	VMID    int    `json:"vmid,omitempty"`
+	// Encrypted is the fingerprint of the key a PBS archive was encrypted with ("" = not encrypted). R-727:
+	// the restore test reads it to tell THIS box's archives from an earlier box's in the same namespace.
+	Encrypted string `json:"encrypted,omitempty"`
 }