Compare commits

..

16 Commits

Author SHA1 Message Date
admin 475bdce7e4 DR bring-up refuses beside a live original (R-834): source guest present, drives bind, or unreadable config
gates / gates (push) Successful in 18s
The DR route keeps onboot 1, binds the real drives and starts the guest: right on a replaced
host, a second box on the same drives beside a live original. The restore-test's no-host-bind
half is now pinned too (measured safe live on demo-hp 2026-10-04).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-04 08:52:35 +02:00
admin d766666ff8 CHANGELOG: v0.138.0 vouched with golden 0.283.1
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 11:40:20 +02:00
admin a4c09a7c11 REPORT: v0.138.0 (R-727)
gates / gates (push) Successful in 16s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 11:00:45 +02:00
admin 904dc20466 CHANGELOG: v0.138.0 released (R-727)
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 10:34:25 +02:00
admin e1b8269be0 restore test takes only this box's archives (R-727): an archive encrypted with another key is another box's
gates / gates (push) Successful in 16s
Red-proof RP39. Released as v0.138.0 by release-agent.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-30 10:33:52 +02:00
admin 5c68c869b6 docs: v0.137.0 vouched with golden 0.276.0 (2026-09-28)
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-28 09:45:08 +02:00
admin 728d12b1a0 CHANGELOG: v0.137.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:01:26 +02:00
admin dd81866b16 REPORT: v0.136.0 and v0.137.0
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:01:02 +02:00
admin 3ef095fb71 agent: a guest outside the agent's ACL is not a known guest (R-689, v0.136.0 regression)
gates / gates (push) Successful in 14s
PVE answers 403 permission denied, not "does not exist", for a vmid outside the felhom pool;
v0.136.0 turned that into a lookup failure and the local tier read UNKNOWN every evaluation
(measured on demo-hp). Such an archive is skipped. Red-proofed; verified read-only on demo-hp
with the pre-release binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 14:00:42 +02:00
admin 7c986915ca CHANGELOG: v0.136.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:46:22 +02:00
admin 16dbc83221 agent: the restore test takes only archives of a guest that still exists (R-689, second half)
gates / gates (push) Successful in 14s
Measured on demo-hp right after v0.135.0: with the golden skipped the pick fell to a leftover
archive of guest 9100, deleted in August. "does not exist" skips it; any other lookup error
makes the tier unknown. Red-proofed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:45:56 +02:00
admin 9555a7f93b REPORT: v0.135.0 released and signed-delivered to both demo hosts
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 13:02:10 +02:00
admin 9ff937d8fb CHANGELOG: v0.135.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 15s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 12:16:17 +02:00
admin d4be12ca95 agent: the restore test proves only backups of a guest (R-689)
gates / gates (push) Successful in 13s
The golden template in local:backup/ was picked as the newest settled archive on demo-hp and
failed every 6 h. Candidates are now vzdump-<type>-<vmid> files or PBS ct|vm/<vmid> snapshots
with a reported vmid. Red-proofed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-27 12:15:42 +02:00
admin 7403c2a838 REPORT + CONTEXT: v0.133.0 and v0.134.0 delivered to both demo boxes; restore test back on
gates / gates (push) Successful in 13s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-25 04:55:26 +02:00
admin 309e368731 CHANGELOG: v0.134.0 released (tag + package verified by download), not vouched
gates / gates (push) Successful in 14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-24 22:49:19 +02:00
10 changed files with 636 additions and 43 deletions
+89
View File
@@ -1,3 +1,92 @@
## v0.138.0 — the restore test takes only THIS box's archives (2026-09-30, R-727, `09` §3 decision 51)
> **RELEASED 2026-09-30** by `scripts/release-agent.sh` — tag `v0.138.0` (`e1b8269`), sha256
> `55916026001790a79ebf97d32c032610cfde8e09d02979b9b9d8c2cbc5d88195`, verified by download. **Not vouched** (the golden
> keeps 0.137.0 until the next bake); delivered to the demo boxes by signed `agent_update` jobs.
>
> **VOUCHED 2026-09-30** with golden 0.283.1 (`min_agent` 0.131.0), on the operator's word: the hub logged
> `Artifact manifest set: agent=0.138.0 golden=0.283.1 min_agent="0.131.0"`. Evidence:
> `felhom.eu/documentation/audits/evidence-golden-0283-2026-09-30/`.
**MinAgent impact:** none required by any controller.
- A returning customer's PBS namespace can hold archives of EARLIER boxes: same guest id (9201), same token, written
with a different key. Measured 2026-09-30: the newest SETTLED archive was an earlier box's, and the test failed
`wrong key … manifest's key 6b:ca:5f:3f… does not match provided key de:51:7a:18…` every evaluation. The archive
carries no host id; it carries its key fingerprint (PVE content `encrypted`), and the storage carries its own
(`GET /storage` → `encryption-key`). `PickSettledRestoreCandidateOn` now skips — and logs by name, once — an archive
whose fingerprint is not the storage's own; an unencrypted storage is not filtered; a failed storage read is an
error (tier UNKNOWN), never "nothing to prove".
- Tests: `TestR727_TheRestoreTestTakesOnlyThisBoxsArchives` (the 2026-09-30 shape: nothing picked while this box's
archive settles, then exactly it), `TestR727_UnencryptedStorageIsNotFiltered`, `TestR727_KeyLookupFailureIsUnknown`.
Red-proof RP39: the skip removed → the earlier box's `2026-09-16T21:59:54Z` is picked.
## v0.137.0 — a guest outside the agent's ACL is not a known guest (2026-09-27, R-689, v0.136.0 regression)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.137.0` (`3ef095f`), sha256 `766c9166916a1bd3674b0dc69081f8a7619e770f1402d8ad7705b395937e7627`, verified by download. **NOT vouched** (the operator's act).
>
> **VOUCHED 2026-09-28** with golden 0.276.0 (`min_agent` 0.131.0), on the operator's word of 2026-09-27: the hub logged
> `Artifact manifest set: agent=0.137.0 golden=0.276.0 min_agent="0.131.0"`, and a Day-0 test install fetched this binary
> through the manifest and sha-verified it. Evidence: `felhom.eu/documentation/audits/evidence-golden-0276-2026-09-28/`.
**MinAgent impact:** none required by any controller.
- v0.136.0 asked `GuestConfig` whether an archive's guest exists and treated anything but "does not exist" as a lookup
failure. PVE answers **403 "permission denied at /vms/<id>"** for a vmid outside the token's pool — so on demo-hp the
deleted guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot read is not one it
manages; its archive is skipped. `TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest`, red-proofed. Verified read-only
on demo-hp with the pre-release binary before the release.
## v0.136.0 — … of a guest that still EXISTS (2026-09-27, R-689 second half)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.136.0` (`16dbc83`), sha256 `2eb0b5ebe253defd68b322312bbac12418c051b0a7a0d2d1831310d97fa6d755`, verified by download. **NOT vouched** (the operator's act).
**MinAgent impact:** none required by any controller.
- **R-689, second half.** Read on demo-hp right after v0.135.0 with the read-only `-selftest=restore-test-due`: the
golden was skipped, and the pick fell to `vzdump-lxc-9100-2026_08_21…` — a leftover of a guest deleted in August. A
candidate's guest must now exist on this node (`GuestConfig`): "does not exist" skips the archive (INFO once per
volid); any other lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove". Tests
`TestR689_AnArchiveOfADeletedGuestIsNeverPicked` (red-proofed: without the check it picks the 9100 leftover),
`TestR689_AGuestLookupFailureIsUnknownNotEmpty`. A second agent release in one session — the first half was found
incomplete on the box.
## v0.135.0 — the restore test proves only backups OF A GUEST (2026-09-27, R-689)
> **RELEASED 2026-09-27** by `scripts/release-agent.sh` — tag `v0.135.0` (`d4be12c`), sha256 `ad4e75f16d338552f4588d3fe64c51cbf9651220b9d223b5386b85b6c37fd4c3`, verified by download. **NOT vouched** (the operator's act).
**MinAgent impact:** none required by any controller.
- **R-689** (`backup/runner.go` `guestBackupArchive`). demo-hp keeps its golden template in `local:backup/` — content
"backup", 654 MB, plausibly complete, the newest settled entry — and the scheduled restore test picked it every 6 h and
failed `extractconfig` with a 403, while the guest's real archive went untested. A restore-test candidate is now a
`vzdump-{lxc,qemu}-<vmid>-…` file or a PBS `backup/{ct,vm}/<vmid>/…` snapshot whose vmid the storage reports; anything
else is skipped with one INFO line per volid (not the "INCOMPLETE archive" WARN). Tests
`TestR689_TheRestoreTestNeverPicksTheGolden` (red-proof: without the check it picks the golden) and
`TestR689_GuestBackupArchiveShapes`; two older picker tests' fixtures moved to real archive names.
Evidence: `felhom.eu/documentation/audits/version-travel-2026-09-26/D1/`.
## v0.134.0 — a whole-box backup that cannot fit is skipped with a reason, before anything starts (2026-09-25 night, R-685)
> **RELEASED 2026-09-24 night** by `scripts/release-agent.sh` — tag `v0.134.0` (`0722b2c`), sha256 `7593bebe03234c7d22f3ade384e7ed7787dc659aa8c8594b3ee19af2ce81c72d`, verified by download. **NOT vouched** (Day-0 stays on the previous version).
**MinAgent impact:** none required by any controller.
- **R-685 — the backup space preflight** (`backup/runner.go` `spaceFits`). Before a vzdump to a LOCAL (non-PBS)
target, the newest archive of that guest on that target × 1.25 + 1 GiB must be free — PVE prunes old archives
only AFTER a successful backup, so the kept ones still stand during the run. A shortfall is a skip: nothing
starts, the record's Error reads `skipped: not enough space: <target> has X GiB free; the last archive of guest N
was Y GiB, so a new one needs about Z GiB …` (stable prefix `BackupSkipNoSpacePrefix`), and it reaches the
controller's tier view and, through the quiesce loop's tier notifier, the operator's `whole_guest_backup_failed`.
It FAILS OPEN on what is not known (PBS target, first backup, unreadable usage).
- **Free space is read from `GET /nodes/<node>/storage`** (`NodeStorage`), never `GET /storage` — the latter is the
cluster DEFINITIONS and carries no usage. **Found live, before release:** the first build read `/storage`,
failed open, and a real vzdump of demo-hp 9201 started from the live test; it was aborted after 5 min 16 s, no
archive left (`felhom.eu/documentation/audits/night-2026-09-25/F/`). The test fake's `ListStorage` now strips
usage like production. Red-proofs: two (`…/F/redproof-r685-*.txt`).
- Live proof (demo-hp, safe builds with a hard stop before vzdump): ×10 margin → refused, "local has 14.9 GiB
free … needs about 77.2 GiB"; release margin → "space preflight passed" need 11.3 GB, avail 16.0 GB.
## v0.133.0 — a restore-test can never fill a box's disk; leftovers retried on a timer (2026-09-24, R-672, R-673)
> **RELEASED 2026-09-24** by `scripts/release-agent.sh` — tag `v0.133.0` (`9bdb4da`), sha256 `3aa303452b8c6be58573d00af01a0ab4a0d97e4f885ffecd0144a18ac24e69b6`, verified by an independent anonymous download. **NOT delivered** (needs an operator-signed `agent_update` job per box, R-530) and **NOT vouched**.
+8
View File
@@ -1,6 +1,14 @@
# CONTEXT — felhom-agent working state
> **2026-09-25 night — v0.133.0 AND v0.134.0 DELIVERED to both demo boxes (CC-signed `agent_update`, ruling 1);
> restore test back ON (the `-1` config kept as `agent.json.night-0925-off`). v0.134.0 = R-685:** `backup/runner.go`
> `spaceFits` — a vzdump to a LOCAL target needs free ≥ newest archive of that guest × 1.25 + 1 GiB (PVE prunes
> only after success); a shortfall is a named skip (`BackupSkipNoSpacePrefix`), fail-open on PBS / first backup /
> unknown usage. **Free space comes from `NodeStorage` (`GET /nodes/<n>/storage`) — `ListStorage` (`GET /storage`)
> has NO usage**; the first build read it and a real vzdump started in its own live test (aborted, no archive).
> demo-hp: `local_backup_retention` 1 (operator option A, saved `agent.json.pre-a4-retention`). Peti's box: nothing.
> **2026-09-24 — v0.133.0 RELEASED, NOT DELIVERED (R-672, R-673).** Restore-test space preflight
> (`reconcile/restoretest_space.go`, `internal/restorespace`): uncompressed size from the vzdump log / PBS size,
> × 1.2 + 5 GiB, thin metadata, off the tested guest's pool, unknown refuses, reported as `skipped` non-pass.
+10 -27
View File
@@ -1,29 +1,12 @@
# REPORT — agent v0.133.0: a restore-test can never fill a box's disk (2026-09-24, R-672, R-673)
# REPORT — 2026-09-30: v0.138.0 (R-727)
Full record: `felhom.eu/documentation/audits/r672-2026-09-24/README.md`.
Full session report: `felhom.eu/REPORT-fixes-first-tester-2026-09-30.md`.
## Shipped (released, not delivered)
Tag `v0.133.0` (`9bdb4da`), package sha256 `3aa30345…e69b6`, verified by anonymous download. Delivery needs the
operator's signed `agent_update` job per box.
- **Space preflight** before anything is created: free ≥ restored × 1.2 + 5 GiB, `restored` UNCOMPRESSED (vzdump
log "Total bytes written" / PBS snapshot size), thin metadata with room, off the tested guest's pool when
another eligible storage fits, unknown refuses, reported as a non-pass (`skipped`).
- **Scratch teardown retried every 10 min**; operator told once after 3 failed tries.
- **Thin pool ≥ 90 %** → immediate host report (hub v0.124.0 alarms `storage_fill_critical`, per pool per 6 h).
- **R-673:** the stale-lock sweep on the same timer, under the one-heavy-op gate.
## Red-proofs (each seen failing; files in `felhom.eu/documentation/audits/r672-2026-09-24/redproofs/`)
1. preflight removed → the 2026-09-24 restore issued again; 2. archive FILE size used → "the compressed file size
was used"; 3. timer pass a no-op → "the leaked scratch was not destroyed by the timer"; 4. sweep without the gate →
"the sweep ran while a backup held the gate"; 5. no 90 % edge → "0 report requests, want 1"; 6. every skip dropped →
"a space refusal never reached the host report". Full suite `go test ./...` green; `agent_gates.py` OK.
## Live (demo-hp, the on-demand `--selftest=restore-test`, cadence OFF)
(a) factor 10 → refused: needs 215.5 GiB, has 22.1 GiB. (b) normal margin → refused: restoring 21.1 GiB needs 30.3 GiB,
has 22.1 GiB — correct: no full restore-test fits demo-hp under 80 % pool use. (c) a forced teardown failure could not
run live (it needs a scratch guest, which (b) shows cannot be created within the 80 % rule) — unit tests only. Pool
58.99 % before and after every run; nothing created.
## Observations
1. demo-hp has no second eligible storage for a restore-test: `nvme-scratch` takes `rootdir`, but the agent holds no grant there. FILED: R-672
- **Measured:** a PBS archive carries its key FINGERPRINT (PVE content `encrypted`), not a host id; the storage
carries its own (`GET /storage` → `encryption-key`). The restore test now skips an archive written with another
key and logs it by name; an unencrypted storage is not filtered; a failed storage read is UNKNOWN.
- Tests `TestR727_*`; red-proof RP39 (the skip removed → the 2026-09-16 archive of an earlier box is picked).
- Released by `release-agent.sh` (tag `v0.138.0`, sha256 `55916026…8195`, verified by download); **not vouched**.
Delivered by signed `agent_update` jobs to `demo-hp-bb76ea` and `demo-felhom-8363b5` (both committed within 340 s);
`-selftest=restore-test-due` on both reads each tier normally on 0.138.0.
- ep0 (decision 51): the three drill archives in `tester-1`'s namespace removed; other namespaces byte-identical.
+17 -7
View File
@@ -2,6 +2,7 @@ package backup
import (
"context"
"fmt"
"encoding/json"
"errors"
"io"
@@ -21,6 +22,8 @@ type fakeBackupAPI struct {
vzdumpErr error
waitErr error
cfg proxmox.GuestConfig
goneGuests map[int]bool // R-689: vmids whose config lookup answers "does not exist"
aclGuests map[int]bool // R-689: vmids outside the token's ACL — PVE answers 403 "permission denied"
cfgErr error
content []proxmox.StorageContent
contentErr error
@@ -43,7 +46,13 @@ func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOpti
}
return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
}
func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
func (f *fakeBackupAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
if f.aclGuests[vmid] {
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 403: permission denied at /vms/%d (missing privilege VM.Audit)", vmid, vmid)
}
if f.goneGuests[vmid] { // R-689: PVE's answer for a deleted guest
return proxmox.GuestConfig{}, fmt.Errorf("proxmox: GET /nodes/n/lxc/%d/config -> HTTP 500: Configuration file 'nodes/n/lxc/%d.conf' does not exist", vmid, vmid)
}
return f.cfg, f.cfgErr
}
func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
@@ -150,13 +159,14 @@ func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
const big = 4 << 30 // a plausible whole-guest archive
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "a", Content: "backup", CTime: 10, Size: big},
{VolID: "b", Content: "backup", CTime: 99, Size: big},
// R-689: real vzdump names with their vmid — only a backup OF A GUEST is a candidate.
{VolID: "local:backup/vzdump-lxc-9001-a.tar.zst", VMID: 9001, Content: "backup", CTime: 10, Size: big},
{VolID: "local:backup/vzdump-lxc-9001-b.tar.zst", VMID: 9001, Content: "backup", CTime: 99, Size: big},
{VolID: "iso", Content: "iso", CTime: 999, Size: big}, // not a backup → ignored
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "b" {
if err != nil || vol != "local:backup/vzdump-lxc-9001-b.tar.zst" {
t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
}
// no backups → "".
@@ -176,12 +186,12 @@ func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
// `pick = "phantom" want the newest COMPLETE archive 'real'`.
func TestPickRestoreCandidate_SkipsImplausibleArchives(t *testing.T) {
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "real", Content: "backup", CTime: 10, Size: 4 << 30},
{VolID: "phantom", Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
{VolID: "felhom-pbs:backup/ct/9001/real", VMID: 9001, Content: "backup", CTime: 10, Size: 4 << 30},
{VolID: "felhom-pbs:backup/ct/9001/phantom", VMID: 9001, Content: "backup", CTime: 99, Size: 1}, // newest, and impossible
}}
r := NewBackupRunner(api, "local", "", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "real" {
if err != nil || vol != "felhom-pbs:backup/ct/9001/real" {
t.Fatalf("pick = %q,%v want the newest COMPLETE archive 'real'", vol, err)
}
}
+112
View File
@@ -0,0 +1,112 @@
package backup
import (
"context"
"fmt"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-689 (v0.135.0) — demo-hp keeps its golden template in `local:backup/`. It is content "backup",
// 654 MB and so "plausibly complete", and it was the newest SETTLED entry: the restore test picked it
// every 6 h and failed extractconfig with a 403 (measured 2026-09-24 10:36, 09-25 04:57 and 10:57),
// while the guest's own archive — younger than the 24 h settle — went untested and nothing was proven.
//
// COMPANION RED-PROOF (REPORT.md): drop the guestBackupArchive call from PickSettledRestoreCandidateOn —
// this test then picks `local:backup/felhom-golden-0.236.0.tar.zst`.
func TestR689_TheRestoreTestNeverPicksTheGolden(t *testing.T) {
const day = int64(86400)
now := int64(1790370000) // 2026-09-25 ~19:00Z
api := &fakeBackupAPI{content: []proxmox.StorageContent{
// the guest's real archive, settled (older than the cutoff below)
{VolID: "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 3*day},
// the golden: newer, settled, big, and NOT a backup of a guest
{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 2*day},
// a hand-copied tarball that PVE happens to attribute to a vmid — the name is not a vzdump's
{VolID: "local:backup/copy-of-9201.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 2*day},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != "local:backup/vzdump-lxc-9201-2026_09_22-21_59_25.tar.zst" {
t.Fatalf("picked %q — the restore test must prove a backup OF A GUEST", got)
}
}
func TestR689_GuestBackupArchiveShapes(t *testing.T) {
for _, c := range []struct {
e proxmox.StorageContent
ok bool
}{
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-2026_09_24-21_59_25.tar.zst", VMID: 9201}, true},
{proxmox.StorageContent{VolID: "local:backup/vzdump-qemu-300-2026_09_24-21_59_25.vma.zst", VMID: 300}, true},
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z", VMID: 9201}, true},
{proxmox.StorageContent{VolID: "felhom-pbs:backup/vm/300/2026-07-28T05:31:14Z", VMID: 300}, true},
{proxmox.StorageContent{VolID: "local:backup/felhom-golden-0.236.0.tar.zst"}, false},
{proxmox.StorageContent{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", VMID: 9202}, false}, // vmid disagrees with the name
{proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z"}, false}, // no vmid reported
} {
if ok, why := guestBackupArchive(c.e); ok != c.ok {
t.Errorf("%s vmid=%d: ok=%v (%s), want %v", c.e.VolID, c.e.VMID, ok, why, c.ok)
}
}
}
// R-689 (v0.136.0) — the measured demo-hp shape right after v0.135.0: the golden (skipped), a leftover archive
// of guest 9100 deleted in August (settled), and today's archive of 9201 (not settled yet). The pick must be
// NOTHING — never the deleted guest's archive. With 9201's archive settled, that one.
//
// COMPANION RED-PROOF (REPORT.md): drop the known-guest check — the pick is the 9100 leftover.
func TestR689_AnArchiveOfADeletedGuestIsNeverPicked(t *testing.T) {
const day = int64(86400)
now := int64(1790476000)
api := &fakeBackupAPI{goneGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
{VolID: "local:backup/felhom-golden-0.236.0.tar.zst", Content: "backup", Size: 654115664, CTime: now - 14*day},
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil || got != "" {
t.Fatalf("picked %q err=%v — a deleted guest's archive proves nothing about this box", got, err)
}
got, _, _ = r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now, 0).UTC())
if got != "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst" {
t.Fatalf("with 9201's archive settled the pick is %q", got)
}
}
// Any OTHER lookup failure is not "the guest is gone": the tier must read UNKNOWN (an error), never
// "nothing to prove".
func TestR689_AGuestLookupFailureIsUnknownNotEmpty(t *testing.T) {
api := &fakeBackupAPI{cfgErr: fmt.Errorf("proxmox: connection refused"), content: []proxmox.StorageContent{
{VolID: "local:backup/vzdump-lxc-9201-x.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: 10},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err == nil {
t.Fatal("a failed guest lookup read as a clean answer")
}
}
// v0.137.0 — THE MEASURED ANSWER: the agent's token sees only its pool, so for the deleted guest PVE says 403
// "permission denied at /vms/9100", not "does not exist" (demo-hp, right after v0.136.0 — the local tier read
// UNKNOWN). Such a guest is not one this agent manages: its archive is skipped, the tier is not an error.
//
// COMPANION RED-PROOF (REPORT.md): drop the "permission denied" case — the pick errors.
func TestR689_AGuestOutsideTheAgentsACLIsNotAKnownGuest(t *testing.T) {
const day = int64(86400)
now := int64(1790476000)
api := &fakeBackupAPI{aclGuests: map[int]bool{9100: true}, content: []proxmox.StorageContent{
{VolID: "local:backup/vzdump-lxc-9100-2026_08_21-17_59_15.tar.zst", Content: "backup", VMID: 9100, Size: 656970239, CTime: now - 37*day},
{VolID: "local:backup/vzdump-lxc-9201-2026_09_27-04_35_47.tar.zst", Content: "backup", VMID: 9201, Size: 8 << 30, CTime: now - 7*3600},
}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Unix(now-day, 0).UTC())
if err != nil || got != "" {
t.Fatalf("picked %q err=%v — want nothing and no error (the only settled archive is not ours)", got, err)
}
}
+74
View File
@@ -0,0 +1,74 @@
package backup
import (
"context"
"errors"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
const (
thisBoxKey = "de:51:7a:18:cb:39:22:30:2c:84:f5:8b:d1:91:4b:7e:81:bb:69:b8:89:0f:57:ac:d3:59:e1:1a:62:25:11:2c"
earlierBox1 = "6b:ca:5f:3f:ca:0f:e2:3f:fb:24:62:89:bf:e7:64:59:9a:41:c5:e6:e3:9f:3f:5f:e1:71:7b:a1:9d:24:67:82"
earlierBox2 = "fe:3d:db:95:d4:df:ab:e1:7d:4a:89:fa:2b:07:53:6a:e4:d2:85:95:d1:90:27:4b:d9:c6:92:20:95:04:e5:d4"
)
// R-727 (v0.138.0) — the 2026-09-30 shape, measured on a fresh box for a returning customer: the PBS
// namespace held two archives of earlier boxes (same guest 9201, same token) and this box's own, which was not
// settled yet. The old picker chose the earlier box's newest settled archive and failed `wrong key`.
// The CONSEQUENCE asserted: no archive of another box is ever picked; with this box's archive settled it is picked.
// COMPANION RED-PROOF: remove the `ownKey != "" && !EqualFold(...)` skip → the first case picks 2026-09-16T21:59:54Z.
func TestR727_TheRestoreTestTakesOnlyThisBoxsArchives(t *testing.T) {
day := int64(86400)
now := int64(1790740000) // 2026-09-30 ~04:00Z
own := proxmox.StorageContent{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
own,
},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
// 1. The night of 2026-09-30: this box's own archive is ~6 h old, not settled (cutoff 24 h) — nothing to prove.
got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now-day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != "" {
t.Fatalf("picked %q — an archive of ANOTHER box is never this box's proof (R-727)", got)
}
// 2. A day later this box's own archive is settled — it is the one picked.
got, _, err = r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Unix(now+day, 0).UTC())
if err != nil {
t.Fatal(err)
}
if got != own.VolID {
t.Fatalf("picked %q, want this box's own %q", got, own.VolID)
}
}
// An unencrypted storage (a local dir) holds only this box's vzdumps — no key filter applies.
func TestR727_UnencryptedStorageIsNotFiltered(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "local", Type: "dir"}},
content: []proxmox.StorageContent{{VolID: "local:backup/vzdump-lxc-9201-2026_09_29-21_27_05.tar.zst", Content: "backup", VMID: 9201, Size: 955425507, CTime: 1790710025}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if got, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "local", time.Time{}); err != nil || got == "" {
t.Fatalf("got %q err %v", got, err)
}
}
// A storage-list failure makes the tier UNKNOWN (an error), never "nothing to prove".
func TestR727_KeyLookupFailureIsUnknown(t *testing.T) {
api := &fakeBackupAPI{storageErr: errors.New("proxmox: GET /storage -> HTTP 500"), content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/x", Content: "backup", VMID: 9201}}}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err == nil {
t.Fatal("a failed key lookup must surface as an error (tier UNKNOWN)")
}
}
+116
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"log/slog"
"sort"
"strconv"
"strings"
"sync"
"time"
@@ -356,12 +357,58 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
if err != nil {
return "", time.Time{}, err
}
// R-727 (v0.138.0): on an ENCRYPTED storage, only archives written with THIS storage's key are this box's.
// Measured 2026-09-30 on a fresh box for a returning customer: the PBS namespace still held two archives
// of earlier boxes (same guest id 9201, same token), the newest settled one was an earlier box's, and the
// test failed `wrong key` every evaluation. The archive carries no host id; its key fingerprint is the
// discriminator (PVE's content `encrypted`, the storage's `encryption-key`). A lookup failure returns
// the error — the tier reads UNKNOWN, never "nothing to prove".
ownKey, err := r.storageKeyFingerprint(ctx, target)
if err != nil {
return "", time.Time{}, fmt.Errorf("reading the key fingerprint of storage %s: %w", target, err)
}
var best string
var bestCTime int64 = -1
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
for _, e := range contents {
if e.Content != "backup" {
continue
}
// R-689 (v0.135.0): only a backup OF A GUEST is a restore-test candidate. demo-hp keeps its golden
// template in `local:backup/` — content "backup", 654 MB, plausibly complete — and it was picked as
// the newest settled archive every 6 h and failed extractconfig (403) each time, while the guest's
// real archive went untested.
if ok, why := guestBackupArchive(e); !ok {
r.noteNotAGuestBackupOnce(e, why)
continue
}
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
continue
}
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
// v0.135.0: with the golden skipped, the pick fell to `vzdump-lxc-9100-2026_08_21…`, a leftover of a
// guest deleted in August — proving nothing about any guest this box runs. "Does not exist" skips the
// archive; any OTHER lookup failure is returned, so the tier reads UNKNOWN, never "nothing to prove".
if _, seen := known[e.VMID]; !seen {
_, err := r.api.GuestConfig(ctx, e.VMID)
switch {
case err == nil:
known[e.VMID] = true
case strings.Contains(err.Error(), "does not exist"), strings.Contains(err.Error(), "permission denied"):
// v0.137.0: PVE answers 403 "permission denied at /vms/<id>" — not "does not exist" — for a guest
// outside the agent's ACL (the `felhom` pool). Measured on demo-hp after v0.136.0: the deleted
// guest 9100's archive made the local tier UNKNOWN every evaluation. A guest the agent cannot
// read is not one it manages; its archive is not a candidate.
known[e.VMID] = false
default:
return "", time.Time{}, fmt.Errorf("checking whether guest %d still exists: %w", e.VMID, err)
}
}
if !known[e.VMID] {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("guest %d does not exist on this node or is not one this agent manages", e.VMID))
continue
}
if !notAfter.IsZero() && e.CTime > notAfter.Unix() {
continue // not settled yet — a newer archive is not a reason to re-prove an older one
}
@@ -591,3 +638,72 @@ func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.R
}
return rt
}
// guestBackupArchive reports whether a storage entry is a whole-guest backup of a known guest — a
// `vzdump-<type>-<vmid>-…` file on a dir storage, or a `backup/{ct,vm}/<vmid>/<time>` snapshot on a PBS
// datastore — whose vmid the storage itself reports. Anything else in a backup content type (a golden
// template, a hand-copied tarball) is not a backup of a guest and is never restore-tested (R-689).
// Pure, so the rule is unit-tested without a storage.
func guestBackupArchive(e proxmox.StorageContent) (bool, string) {
if e.VMID <= 0 {
return false, "not a backup of a guest (the storage reports no vmid)"
}
vol := e.VolID
if i := strings.Index(vol, ":"); i >= 0 {
vol = vol[i+1:]
}
vol = strings.TrimPrefix(vol, "backup/")
vmid := strconv.Itoa(e.VMID)
switch {
case strings.HasPrefix(vol, "vzdump-lxc-"+vmid+"-"), strings.HasPrefix(vol, "vzdump-qemu-"+vmid+"-"):
return true, ""
case strings.HasPrefix(vol, "ct/"+vmid+"/"), strings.HasPrefix(vol, "vm/"+vmid+"/"):
return true, ""
}
return false, "not a vzdump archive or a PBS snapshot of guest " + vmid
}
// noteNotAGuestBackupOnce logs, once per volid, that a backup-content entry is not a restore-test
// candidate because it is not a backup of a guest (R-689). INFO, not WARN: a golden template kept in
// the backup directory is the operator's, and not a fault.
func (r *BackupRunner) noteNotAGuestBackupOnce(e proxmox.StorageContent, why string) {
r.rejectedMu.Lock()
if r.rejected == nil {
r.rejected = map[string]struct{}{}
}
_, seen := r.rejected[e.VolID]
if !seen {
r.rejected[e.VolID] = struct{}{}
}
r.rejectedMu.Unlock()
if !seen {
r.logger.Info("backup: restore-test skips an entry that is not a backup of a guest",
"target", r.target, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
}
}
// storageKeyFingerprint returns the named storage's client-side encryption key fingerprint ("" when the
// storage is not encrypted — a local dir holds only this box's own vzdumps).
func (r *BackupRunner) storageKeyFingerprint(ctx context.Context, target string) (string, error) {
sts, err := r.api.ListStorage(ctx)
if err != nil {
return "", err
}
for _, st := range sts {
if st.Storage == target {
return strings.TrimSpace(st.EncryptionKey), nil
}
}
return "", nil
}
// shortFP is the first 8 bytes of a key fingerprint, for a log line.
func shortFP(fp string) string {
if fp == "" {
return "none"
}
if len(fp) > 23 {
return fp[:23] + "…"
}
return fp
}
+15 -9
View File
@@ -219,15 +219,18 @@ type Storage struct {
UsedFraction float64 `json:"used_fraction,omitempty"`
// Type-specific config (durable_id sources).
Server string `json:"server,omitempty"` // nfs/cifs/pbs server host
Export string `json:"export,omitempty"` // nfs export path
Share string `json:"share,omitempty"` // cifs share name
Datastore string `json:"datastore,omitempty"` // pbs datastore name
Fingerprint string `json:"fingerprint,omitempty"` // pbs server cert fingerprint
Username string `json:"username,omitempty"` // pbs auth id, e.g. "felhom@pbs!n100"
Namespace string `json:"namespace,omitempty"` // pbs namespace ("" = root; per-customer tenancy = S4)
VGName string `json:"vgname,omitempty"` // lvm/lvmthin volume group
ThinPool string `json:"thinpool,omitempty"` // lvmthin pool LV name
Server string `json:"server,omitempty"` // nfs/cifs/pbs server host
// EncryptionKey is the storage's own client-side key FINGERPRINT (pbs; the key itself stays in
// /etc/pve/priv). R-727: an archive encrypted with any other key was written by another box.
EncryptionKey string `json:"encryption-key,omitempty"`
Export string `json:"export,omitempty"` // nfs export path
Share string `json:"share,omitempty"` // cifs share name
Datastore string `json:"datastore,omitempty"` // pbs datastore name
Fingerprint string `json:"fingerprint,omitempty"` // pbs server cert fingerprint
Username string `json:"username,omitempty"` // pbs auth id, e.g. "felhom@pbs!n100"
Namespace string `json:"namespace,omitempty"` // pbs namespace ("" = root; per-customer tenancy = S4)
VGName string `json:"vgname,omitempty"` // lvm/lvmthin volume group
ThinPool string `json:"thinpool,omitempty"` // lvmthin pool LV name
}
// StorageContent is one entry of GET /nodes/{node}/storage/{store}/content
@@ -239,4 +242,7 @@ type StorageContent struct {
Size int64 `json:"size"`
CTime int64 `json:"ctime"`
VMID int `json:"vmid,omitempty"`
// Encrypted is the fingerprint of the key a PBS archive was encrypted with ("" = not encrypted). R-727:
// the restore test reads it to tell THIS box's archives from an earlier box's in the same namespace.
Encrypted string `json:"encrypted,omitempty"`
}
+57
View File
@@ -302,6 +302,19 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
}
}
// R-834: DR keeps the archive's `onboot: 1`, binds the host's REAL drives (4d) and STARTS the guest
// — right on a replaced host, where the original is gone. Beside a LIVE original it would be a
// second controller for the same household on the same drives. So DR refuses when this host
// still carries the original (the archive's source VMID) or any guest that binds the drives.
// A copy beside the original is the restore-test's job (onboot=0, throwaway stand-ins, torn
// down) or the runbook's beside-restore. Pinned by TestRunBringUp_DRRefusesBesideALiveOriginal.
if spec.Mode == ModeDRGuestLoss {
if why := e.liveOriginalBeside(ctx, lxc, spec.Archive); why != "" {
res.Err = fmt.Errorf("reconcile: dr bring-up refused: %s — a DR restore beside a live original would run two boxes on the same drives (R-834)", why)
return
}
}
base := JournalEntry{OpID: e.bringUpOpID(spec.VMID), VMID: spec.VMID, Kind: bringUpKind, Rollback: true}
// OWN the rollback BEFORE any mutation. From here a crash leaves an in-flight Rollback
@@ -734,3 +747,47 @@ func net0MAC(cfg proxmox.GuestConfig) string {
}
return ""
}
// archiveSourceVMID reads the source guest's VMID from a backup volid: a vzdump file
// (`…/vzdump-lxc-<vmid>-<date>.tar.zst`) or a PBS snapshot (`…:backup/ct/<vmid>/<time>`). 0 = unknown.
func archiveSourceVMID(archive string) int {
if i := strings.Index(archive, "vzdump-lxc-"); i >= 0 {
rest := archive[i+len("vzdump-lxc-"):]
if j := strings.Index(rest, "-"); j > 0 {
if n, err := strconv.Atoi(rest[:j]); err == nil {
return n
}
}
}
if i := strings.Index(archive, "ct/"); i >= 0 {
rest := archive[i+len("ct/"):]
if j := strings.Index(rest, "/"); j > 0 {
if n, err := strconv.Atoi(rest[:j]); err == nil {
return n
}
}
}
return 0
}
// liveOriginalBeside says why a DR bring-up would land beside a live original ("" = it would not):
// the archive's source guest still exists here, or a guest binds the drives parent. Fails CLOSED: a
// guest whose config cannot be read cannot be ruled out.
func (e *Engine) liveOriginalBeside(ctx context.Context, lxc []proxmox.Guest, archive string) string {
src := archiveSourceVMID(archive)
for _, g := range lxc {
if src > 0 && g.VMID == src {
return fmt.Sprintf("the archive's source guest %d still exists on this host (status %s)", g.VMID, g.Status)
}
cfg, err := e.api.GuestConfig(ctx, g.VMID)
if err != nil {
return fmt.Sprintf("guest %d's config could not be read to rule out a live original: %v", g.VMID, err)
}
for slot, v := range cfg.MountPoints() {
if source, _, _ := strings.Cut(v, ","); source == structuralParentDir {
return fmt.Sprintf("guest %d binds the household drives (%s %s)", g.VMID, slot, structuralParentDir)
}
}
}
return ""
}
+138
View File
@@ -0,0 +1,138 @@
package reconcile
import (
"context"
"encoding/json"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-834: a whole-guest restore BESIDE a live original must never come up as a second box on the same
// drives. The DR route keeps `onboot: 1`, binds the real drives and starts the guest, so it refuses
// when the original (or any guest binding the drives) is still on this host; on a replaced host it
// proceeds and keeps its binds. Red-proof: make liveOriginalBeside return "" and the refusals pass
// the restore through (the "refused" sub-tests fail).
func TestRunBringUp_DRRefusesBesideALiveOriginal(t *testing.T) {
const target = 9299
drivesBind := proxmox.GuestConfig{Extra: map[string]json.RawMessage{
"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`),
}}
cases := []struct {
name string
archive string
lxc []proxmox.Guest
cfg map[int]proxmox.GuestConfig
refuse string // substring of the refusal; "" = must proceed
}{
{"the source guest still exists", "local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst",
[]proxmox.Guest{{VMID: 9201, Status: "running"}}, map[int]proxmox.GuestConfig{9201: scratchCfg()}, "source guest 9201"},
{"another guest binds the drives (PBS archive)", "felhom-pbs:backup/ct/9201/2026-10-04T02:34:55Z",
[]proxmox.Guest{{VMID: 9300, Status: "stopped"}}, map[int]proxmox.GuestConfig{9300: drivesBind}, "binds the household drives"},
{"a guest whose config cannot be read", "local:backup/vzdump-lxc-9201-x.tar.zst",
[]proxmox.Guest{{VMID: 9400, Status: "running"}}, map[int]proxmox.GuestConfig{}, "could not be read"},
{"replaced host: only an unrelated scratch guest", "local:backup/vzdump-lxc-9201-x.tar.zst",
[]proxmox.Guest{{VMID: 9202, Status: "running"}}, map[int]proxmox.GuestConfig{9202: scratchCfg()}, ""},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
cfg := c.cfg
cfg[target] = scratchCfg()
api := &fakeAPI{lxc: c.lxc, cfg: cfg}
e, fr, _, q := newDREngine(t, api)
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{
Mode: ModeDRGuestLoss, Archive: c.archive, VMID: target, RestoreStorage: "local-lvm", KeepMAC: true,
})
if c.refuse != "" {
if res.Err == nil || !strings.Contains(res.Err.Error(), c.refuse) || !strings.Contains(res.Err.Error(), "R-834") {
t.Fatalf("want a refusal naming %q, got %+v", c.refuse, res)
}
if len(api.restores) != 0 || len(api.starts) != 0 || len(fr.cmds) != 0 {
t.Fatalf("a refused DR touched the host: restores=%d starts=%v cmds=%v", len(api.restores), api.starts, fr.cmds)
}
return
}
if res.Err != nil || !res.Pass {
t.Fatalf("a DR on a replaced host must proceed, got %+v", res)
}
// … and there it keeps the REAL drives bind (the right binds on a replaced host).
joined := strings.Join(fr.cmds, "\n")
if !strings.Contains(joined, "-mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives") {
t.Fatalf("the DR guest lost its drives bind: %v", fr.cmds)
}
})
}
}
// Provisioning restores the GOLDEN (no drives, onboot set by the back-half on purpose): a drives-
// binding guest on the host does not block it — the rule is DR's alone.
func TestRunBringUp_ProvisionNotBlockedByADrivesBind(t *testing.T) {
api := &fakeAPI{
lxc: []proxmox.Guest{{VMID: 9201, Status: "running"}},
cfg: map[int]proxmox.GuestConfig{
9201: {Extra: map[string]json.RawMessage{"mp8": json.RawMessage(`"/mnt/felhom-drives,mp=/mnt/felhom-drives"`)}},
9203: scratchCfg(),
},
}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{Mode: ModeProvision, Archive: "local:vztmpl/felhom-golden.tar.zst", VMID: 9203, RestoreStorage: "local-lvm"})
if res.Err != nil || !res.Pass {
t.Fatalf("provision must proceed, got %+v", res)
}
}
func TestArchiveSourceVMID(t *testing.T) {
for in, want := range map[string]int{
"local:backup/vzdump-lxc-9201-2026_10_04-04_34_55.tar.zst": 9201,
"felhom-pbs:backup/ct/9201/2026-10-04T02:34:55Z": 9201,
"tmp-dooplex-copy:backup/ct/9201/2026-10-03T19:00:00Z": 9201,
"local:vztmpl/felhom-golden.tar.zst": 0,
"vol": 0,
} {
if got := archiveSourceVMID(in); got != want {
t.Errorf("archiveSourceVMID(%q) = %d, want %d", in, got, want)
}
}
}
// R-834, the restore-test route: its scratch guest sits BESIDE the live original by design, so it must
// carry no host-path bind — the archive's mp8 (the household's drives) and mp9 (the original's
// bootstrap) are replaced by throwaway volumes AT restore time. Measured live 2026-10-04 on demo-hp
// (`audits/backup-close-2026-10-04/partA/`): onboot 0 and no host bind on every poll. Red-proof:
// make drRestoreOverrides return the archive's own mp8 value and this fails.
func TestRestoreTest_NoHostPathBindBesideTheOriginal(t *testing.T) {
api := &fakeAPI{
cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()},
extractCfg: "hostname: demo-hp\nonboot: 1\nrootfs: local-lvm:vm-9201-disk-0,size=16G\n" +
"mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G\n" +
"mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\n" +
"mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1\n",
}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
_ = e.RunRestoreTest(context.Background(), RestoreTestSpec{
Archive: "local:backup/vzdump-lxc-9201-x.tar.zst", RestoreStorage: "local-lvm",
ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local",
})
if len(api.restores) != 1 {
t.Fatalf("want one restore, got %+v", api.restores)
}
r := api.restores[0]
for _, slot := range []string{"mp8", "mp9"} {
v, ok := r.MountOverrides[slot]
if !ok || strings.HasPrefix(v, "/") {
t.Fatalf("%s = %q (present=%v): the scratch beside the original must get a throwaway volume, never the host path", slot, v, ok)
}
}
for slot, v := range r.MountOverrides {
if strings.HasPrefix(v, "/") {
t.Fatalf("%s carries a host path %q into the scratch guest", slot, v)
}
}
if r.ConfigOverrides["onboot"] != "0" {
t.Fatalf("onboot = %q, want 0", r.ConfigOverrides["onboot"])
}
}