felhom.eu/documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md

RUNBOOK RECORD — the local whole-guest backup moved off the guest's own device (2026-07-28)

Class: supervised operational change, one box at a time. No code, no version bump — the change is a PVE storage definition, a per-storage ACL grant, and a one-line target switch in agent.json. Nothing was deleted.

Boxes: demo-hp (felhom-host, HP t740) first, then demo-felhom (N100). Change: primary backup tier local → felhom-backup, a dir storage on each box's secondary drive. Window: 2026-07-28 21:11 → 21:35 CEST.

Deliverable note. The spec is RUNBOOK-vzdump-target-move-2026-07-29.md; the work ran on the evening of 07-28. The filename is kept as specified.

---

1. Phase 0 — ground truth, established read-only before anything was touched

1.1 The premise holds: local and local-lvm are the same physical device

	demo-felhom	demo-hp
local (dir, /var/lib/vz)	pve-root → sda3	pve-root → sda3
local-lvm (lvmthin, pve/data)	sda3	sda3
System drive	AirDisk 512 GB SATA SSD (sda)	SanDisk X600 M.2 128 GB (sda)

A drive failure took the guest and its only local backup together. That is the fault being closed.

1.2 Physical inventory and the durable id

	demo-felhom	demo-hp
Secondary drive	/dev/sdb TOSHIBA MQ04ABF100, USB, rotational	/dev/nvme0n1 KXG50PNV1T02, NVMe
Size / free before	916 GB / 866 GB	938 GB / 891 GB
Mountpoint	/mnt/hdd_1	/mnt/nvme-1tb
Durable id (fs UUID)	47a3361a-91e0-4831-a69d-27f540ed3f48	91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
/sys/block/*/removable	0	0
SMART at close	—	PASSED, 50 °C, 7 % wear, 0 media errors

Both drives are already enrolled Felhom drives, mounted by filesystem UUID through agent-generated systemd units (mnt-hdd_1.mount, mnt-nvme\x2d1tb.mount), each marked "Managed by felhom-agent — do not edit by hand", WantedBy=multi-user.target. Neither is in /etc/fstab. The UUID in the unit is the same durable id the DR recipe carries, so the target drive was already durably identified before this change.

This is §2's principle meeting the reference hardware head-on: demo-felhom's target is an external USB HDD and both drives report removable=0. A rule keyed on transport would disqualify demo-felhom; a rule keyed on the removable flag would see no removable drive at all on either box. Neither attribute can decide a role. Both are fine as hints.

1.3 Space

The runbook's ~14.5 GB figure is the PBS-side measurement. The local zstd archive is far smaller:

	newest local archive before	× keep-last=3	share of the target drive's free space
demo-felhom (9201)	5.93 GB (trend 5.82 → 5.84 → 5.93)	~17.8 GB	2.1 % of 866 GB
demo-hp (9201)	1.68 GB (1.61 → 1.63 → 1.68)	~5.0 GB	0.6 % of 891 GB

Retention is local_backup_retention: 3 on both boxes and is unchanged by this runbook.

The move also relieves the system drive — 18 GB on demo-felhom, 5.2 GB on demo-hp once the old archives are eventually removed. On demo-hp that matters: its root is 40 GB with 24 GB free.

1.4 How the target is set — configuration, not converged. The runbook holds.

The primary tier is built from backup.local_backup_target (felhom-agent/internal/config/config.go:541, BackupTarget()), which was "local" on both boxes. The extra tier is felhom-pbs, weekly.

Every writer of agent.json was enumerated. There is exactly one: pbsdr.seedEscrowStorageID (internal/pbsdr/manager.go:454). It sets only escrow.pbs_storage_id, refuses to clobber a differing existing value (warn-and-keep), and performs a map[string]json.RawMessage read-modify-write that preserves unknown keys verbatim. Nothing on the hub pushes agent configuration. Seven prior hand-edit backups sit beside the live file.

A hand edit sticks. No code change was required, so the STOP in §3 did not fire.

1.5 What else reads the target — no hardcoded "local" on the backup path

Consumer	Verdict
Backup runners	One runner per tier, each constructed with its own TargetID (cmd/felhom-agent/main.go:1355). Parameterised.
NewestArchiveTime	Reads r.target (internal/backup/runner.go:390). Parameterised.
Restore-test rotation	Iterates BackupTiers(); restoreTierForArchive classifies from the archive, never from config (the R-82/R-85 fix). Parameterised.
Per-run prune	localPruneSpec suppresses only for PBS-type targets; a dir target still prunes at keep-last=3. Correct.
Customer-facing label	backupIsPBS (felhom-controller/.../backup_handlers.go:174) matches the substring pbs. felhom-backup correctly renders „Helyi tároló (felhom-backup)". Naming trap recorded — a target named with pbs in it would be mislabelled as offsite.

Two Phase 0 findings that shaped the change are in §4 and §6.

---

2. The two findings Phase 0 turned up before any command ran

F-1 — the storage path must BE the mountpoint, or the target reports disconnected forever

reportType (internal/storage/observe.go:365) calls a dir storage local only when it is literally named local; anything else becomes usb or local-dir. For those two types reachable() (:321) requires exactMount — the storage path must exactly equal a mountpoint (exactMountDevice, :422).

Point the storage at a subdirectory of the drive and exactMount is false, so reachable=false, state=disconnected, permanently — a false alarm designed in on day one, and the durable id degrades to a path fallback instead of the filesystem UUID.

Decision: path = the drive's mountpoint. PVE creates dump/ at the drive root, beside the existing felhom-data/. This also gives a real safety property: the guest is bind-mounted only the drive's felhom-data subpath (/mnt/felhom-drives/<drive>), so the archives sit outside anything the customer's guest can reach.

F-2 — is_mountpoint is what stops a silent retarget onto the system drive

man pvesm: "Assume the given path is an externally managed mountpoint and consider the storage offline if it is not mounted." Without it, an unplugged or late-mounting drive leaves /mnt/<drive> a bare directory on the root filesystem and vzdump writes the backup onto the system drive — the exact device this change exists to escape, silently. Proven live in §6.

Decision: --is_mountpoint 1 on both boxes.

---

3. The exact commands used, per box

Identical on both, differing only in the mountpoint. Run as root on the PVE host.

# 1. the storage, on the drive's OWN mountpoint, guarded (F-1 + F-2)
pvesm add dir felhom-backup --path /mnt/nvme-1tb --content backup --is_mountpoint 1   # demo-hp
pvesm add dir felhom-backup --path /mnt/hdd_1    --content backup --is_mountpoint 1   # demo-felhom

# 2. the per-storage grant — REQUIRED, and the reason the first backup failed (§4)
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --users  felhom-agent@pve
pveum acl modify /storage/felhom-backup --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'

# 3. back up the config, then repoint the primary tier
cd /etc/felhom-agent
cp -p agent.json agent.json.pre-e-target-move
#   in-place truncate-write (NOT sed -i): the directory is root-owned while the file is
#   agent-owned 0600, so a rename would flip ownership and the non-root agent could not read it.
python3 - <<'EOF'
import json
p='/etc/felhom-agent/agent.json'
raw=open(p).read()
old='"local_backup_target": "local"'
new='"local_backup_target": "felhom-backup"'
assert raw.count(old)==1, 'occurrences: %d' % raw.count(old)
out=raw.replace(old,new); json.loads(out)          # validate BEFORE writing
with open(p,'r+') as f: f.seek(0); f.write(out); f.truncate()
EOF

# 4. restart only with nothing in flight — check the PVE task list, NOT pgrep (see §8)
pvesh get /nodes/<node>/tasks --limit 20 --output-format json | python3 -c \
  'import sys,json;print([t for t in json.load(sys.stdin) if "endtime" not in t] or "none running")'
systemctl restart felhom-agent

Resulting stanza, both boxes:

dir: felhom-backup
	path /mnt/<drive>
	content backup
	is_mountpoint 1

Verification that Part 1 took, both boxes — the diff was exactly one line, ownership and mode preserved (-rw------- felhom-agent felhom-agent), and both tiers armed with no rejection:

backup tier armed target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s primary=true
backup tier armed target=felhom-pbs    cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s primary=false

GET /backup/due echoes the new tier, and the old target no longer exists — the cleanest proof the switch took:

GET /backup/due                        → {"due":true,"reason":"no successful backup recorded yet","age_state":"absent"}
GET /backup/due?target=felhom-backup   → {..., "target":"felhom-backup", "age_state":"absent"}
GET /backup/due?target=felhom-pbs      → {"due":false,"reason":"within cadence window","age_seconds":44362}
GET /backup/due?target=local           → {"ok":false,"error":"unknown backup target: local"}

---

4. Finding F-3 — the agent had no PVE privilege on the new storage (found by the first real backup)

The first triggered backup on demo-hp failed:

local-api: backup job failed vmid=9201 target=felhom-backup
  err="proxmox: POST /nodes/felhom-host/vzdump -> HTTP 403: permission denied
       at /storage/felhom-backup (missing privilege Datastore.Allocate)"

FelhomAgentStore (Datastore.Allocate,Datastore.AllocateSpace) is granted per storage path, to both felhom-agent@pve and the felhom-agent@pve!agent token — local, local-lvm and felhom-pbs each had their own pair. A new backup target needs the same pair or every backup 403s.

The grant in §3 step 2 mirrors the existing ones exactly; no privilege was widened, and the storage-scoped shape (rather than a grant at /) is preserved.

This is an E-2 requirement, not a one-off: felhom-host-install.sh must issue the grant for whatever target it provisions on a new install, or every new box ships with a backup tier that 403s on its first run.

---

5. Part 2 — the proof

5.1 A backup lands on the secondary drive — via the real path

Triggered through the endpoint the UI's „Mentés most" button invokes, not --selftest.

Route correction worth recording: the backups page has two triggers. POST /api/backup/run is the Tier-1 app-data backup (DB dumps + volume dumps); the whole-guest vzdump is POST /api/guest-backup/trigger, which goes through the controller's quiesce loop (ServeBackupAPI, backup_handlers.go). The first attempt used the former and correctly produced an app-data run and no vzdump.

	demo-hp	demo-felhom
Archive	felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst	felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst
Size	2,256,044,492 B (2.26 GB)	5,957,878,962 B (5.96 GB)
Duration	86.3 s	~197 s
Mode	snapshot — snapshotted at 21:18:16, app resumed before the copy finished	snapshot — snapshotted at 21:26:29
df on the target drive	used 92,344,320 → 2,348,572,672 (+2,256,228,352 B)	used 3,600,314,368 → 9,558,208,512 (+5,957,894,144 B)

On both boxes the df delta matches the archive size, so the bytes demonstrably landed on the secondary drive and not somewhere else.

On demo-felhom the backup was taken by the controller's first-backup safety valve rather than by the manual trigger: the agent restart made the new tier report age_state: absent, the controller correctly refused to withhold a first backup, and the manual trigger that followed was answered {"error":"mentés már folyamatban van"}. Same code path, same quiesce loop — the trigger simply lost the race to the mechanism that exists for exactly this state.

uncovered_volumes: ["/etc/felhom-bootstrap", "/mnt/felhom-drives"] — both are host binds and are correctly outside the guest archive. Pre-existing behaviour, unchanged by this move, but see §7.

5.2 A restore works from that archive

Run with no -archive, so the candidate selection is part of the proof. The rotation needed no nudge — it picked the new storage on its own.

restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst
          into scratch band [990000,990009] on local-lvm …
{
  "source_archive": "felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_18_15.tar.zst",
  "source_tier": "local",
  "scratch_vmid": 990000,
  "pass": true,
  "verified": "boot+running",
  "mount_parity": "ok",
  "duration_seconds": 114.39,
  "mount_inventory": [
    "mp0=/var/lib/docker (50G)", "mp1=/mnt/sys_drive (20G)",
    "mp8=/mnt/felhom-drives (throwaway for the archived bind)",
    "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
  ]
}
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m54s) ===

mount_parity: ok, pass: true, scratch torn down cleanly.

demo-felhom, same method, same result — the bigger guest (200 G + 50 G volumes):

restoring felhom-backup:backup/vzdump-lxc-9201-2026_07_28-21_26_28.tar.zst …
  "pass": true, "verified": "boot+running", "mount_parity": "ok",
  "duration_seconds": 84.10, "scratch_vmid": 990000,
  "mount_inventory": ["mp0=/var/lib/docker (200G)", "mp1=/mnt/sys_drive (50G)", …]
=== selftest=restore-test OK (restored+booted+verified+torn-down in 1m24s) ===

	demo-hp	demo-felhom
Candidate picked automatically from the new storage	yes	yes
pass / verified	true / boot+running	true / boot+running
mount_parity	ok	ok
Duration	1 m 54 s	1 m 24 s
Scratch torn down	yes (990000)	yes (990000)

Run with no -archive on both boxes, so the candidate selection is part of the proof. The rotation needed no nudge on either box — it picked the new storage on its own.

source_tier: "local" is correct, not a stale reference to the old storage name. restoreTierForArchive classifies by the archive's storage type; felhom-backup is a dir storage, so the restore correctly earns the local (10-minute) wait bound rather than the PBS one.

Method caveat: --selftest=restore-test is a separate one-shot process, so it does not write the daemon's rtState. GET /restore-test/status still reads null on both boxes and the customer-visible „utoljára ellenőrizve" is unchanged. The proof is real; the daemon's own 84 h rotation is undisturbed and will record its next run normally.

5.3 Freshness follows the target

Immediately after the backup: {"due":false,"age_seconds":159,"target":"felhom-backup","age_state":"known"}.

That alone does not separate the storage read from this process's in-memory record, so the agent was restarted — which empties the in-memory store (R-84) — and re-asked. Both boxes:

demo-hp       GET /backup/status → {"phase":"idle"}            ← no in-memory record at all
              GET /backup/due    → {"due":false,"age_seconds":188,
                                    "target":"felhom-backup","age_state":"known"}
demo-felhom   GET /backup/status → {"phase":"idle"}
              GET /backup/due    → {"due":false,"age_seconds":267,
                                    "target":"felhom-backup","age_state":"known"}

A known age with an empty in-memory store can only have come from reading the new storage. The tier neither looks absent (which would re-backup every cycle) nor stale. This is the positive observable, not the absence of a complaint.

This restart is also where the one operational error of the run happened — see §8.

5.4 The DR recipe does not record the new target — and one field improved

The finding: BuildDRRecipeHostHalf (internal/hub/dr_recipe.go:86) records guests, drives, pve_storage and the PBS coordinate. Nothing in the recipe says which storage holds the local archives. While the target was local that was guessable; it is not any more, and it is now actively misleading — after this change local still holds the stale archives while the live ones are on felhom-backup. A restorer reading the recipe sees two backup-capable storages and cannot tell which is which. Filed as R-109, alongside R-105/R-106.

The improvement — this change traces and partly closes R-105. R-105 recorded dr_recipe.host_half.drives = [] on every customer including two with enrolled data drives, with the cause explicitly untraced. It is now traced: the enrolled drives were never PVE storages at all, so they never entered report.StorageTargets, so isUserDataDrive (which needs type usb/local-dir and a durable id and a mount path) never saw them. Making the drive a storage populated it. Live on demo-hp after the change:

"drives": [{ "durable_id": "uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae",
             "mount_path": "/mnt/nvme-1tb", "intent": "enrolled",
             "total_bytes": 1006980812800 }]

The storage observation also gained a real device, a real UUID and SMART for the backup drive — health=PASSED temp=50 poh=28451 media_err=0 pct_used=7 — none of which existed when the drive was not a PVE storage:

- felhom-backup type=local-dir state=attached reach=true class=fast
    durable=uuid:91d2dc2d-… mount="/mnt/nvme-1tb" dev="/dev/nvme0n1"

demo-felhom shows the identical shape — drives populated, SMART now readable on the USB HDD:

- felhom-backup type=local-dir state=attached reach=true class=slow
    durable=uuid:47a3361a-… mount="/mnt/hdd_1" dev="/dev/sdb"
    smart: health=PASSED temp=35 poh=3581 realloc=0 pending=0 offline_unc=0

"drives": [{ "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48",
             "mount_path": "/mnt/hdd_1", "intent": "enrolled",
             "total_bytes": 983351140352 }]

dr_recipe.host_half.drives is now non-empty on both demo boxes, where R-105 recorded [] fleet-wide. Note the classification detail: demo-felhom's drive is USB-attached but reports removable=0, so it is typed local-dir rather than usb. That does not matter here — both types take the same reachability branch and both satisfy isUserDataDrive — but it is one more reason §2's "do not classify by transport" rule is right.

R-106 is untouched and still live on both boxes: the recipe still records "namespace": "root" where the real namespaces are demo-hp / demo-felhom.

---

6. Part 3 — what happens when the target drive is absent

Established with throwaway storages, so neither live drive was unmounted (unmounting would break the guest's felhom-data bind and the customer's app data on a remote box).

Live-proven: the PVE half

	storage on a non-mount path, no is_mountpoint	same path, is_mountpoint 1
Creation	succeeded silently	refused: unable to activate storage 'absent-test-b' - directory is expected to be a mount point but is not mounted: '/mnt/absent-b'
pvesm status	active, 40516856 KiB total	never created
Backing device	/dev/mapper/pve-root — the system drive	—
What it created	/mnt/absent-a/dump/ on the root filesystem	nothing

The unguarded storage reports active with plausible free space that is really the root filesystem's, and it had already created a dump/ directory there. A vzdump aimed at it would write the whole-guest backup onto the system drive with no error and no alarm — silent retargeting onto the exact device this change exists to escape, which §6 names as the worst outcome. is_mountpoint 1 converts that into a loud refusal. Both test storages were removed and /mnt/absent-a deleted.

Source-traced, not live-proven: the agent half

With the guard in place and the drive absent, the storage stays in storage.cfg but goes inactive. targetStoragePresent (internal/localapi/server.go:1212) checks the target's name presence only, never Reachable, so the tier is not deferred; newestArchiveOn cannot read the storage and degrades to archiveUnknown, which stays DUE by design. The controller therefore quiesces the apps and fires a vzdump, PVE refuses because the storage is not active, and the run fails loudly — arming the R-88 breaker and the backup_failed operator mail.

So today's behaviour is: fail and alarm; no silent retarget. That is the safe half.

It does not match §6's intended design (fall back to the system drive and alarm): there is no fallback at all, so an absent drive means no local backup whatsoever until a human intervenes, and each attempt costs an app quiesce until the breaker backs off. Whether a corruption-only copy on the system drive beats no copy is a product decision; the honest label matters either way. Filed as E-2.

This half is traced through source, not reproduced on hardware — the boxes are remote and unmounting a live enrolled drive would break the guest bind and the customer's data. Recorded as outstanding proof in §9.

Single-drive boxes

A single-drive box has nowhere to move the target. There the backup stays on the system drive and protects against corruption only — it is not drive-loss protection and must not be presented as a backup tier that survives hardware failure. Two drives is effectively a hardware requirement for drive-loss protection, and that belongs in the sales/onboarding material as well as in the UI label. Filed under E-2.

---

7. Scope recorded for E-2 — filed, not built

#	Item
1	A backup-target role on StoragePath, alongside Schedulable / IsDefault / Kind.
2	Assignment in the storage wizard — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or removable (§1.2 shows both fail on the reference hardware).
3	Unassigned drives do nothing automatically — §2's rule, enforced in code. A drive must never acquire a role by appearing.
4	Stickiness — an assigned target must not move because a new drive appeared, and must never silently retarget when absent.
5	New installs: felhom-host-install.sh must create the target storage with --is_mountpoint 1 and issue the FelhomAgentStore grant (§4), or a new box's first backup 403s. ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at. The installer's CREATE arm did issue the grant, exactly as this item asked. Its reuse arm — "the target already exists, leave it as it is" — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing local_backup_target at a storage its own token could not read. CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong. demo-felhom's local-api backup jobs 403'd six times between 09:24 and 17:34 CEST: POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate). The hub raised whole_guest_backup_failed at the first one ("retrying with backoff") and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's read returning {"data":[]} while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer 1.24.0 (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent v0.123.0 (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn).
6	Absent-target policy per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier.
7	Retention and space accounting on a drive the customer also uses — today keep-last=3 competes with customer data with no reservation and no ceiling.
8	The honest single-drive label.
9	Migrating the remaining fleet — peti-felhom and any box not covered here.
10	Naming guard: backupIsPBS matches the substring pbs, so a target so named would be mislabelled to the customer as offsite (§1.5).

---

8. Observations

8.1 An operational error in this run: the agent was restarted with a backup in flight

§4.3 and §8 of the spec both say not to. It happened anyway, on demo-hp at 21:21:12.

The manual „Mentés most" correctly fires every tier, not just the primary, so the trigger at 21:18 started the local vzdump and a felhom-pbs one. The local tier finished at 21:19:41. The PBS tier was still running. The in-flight check had been done before the first restart (21:12, clean) and was not repeated before the second one — the restart taken for the §5.3 freshness proof — so it landed on top of a live PBS backup.

What it produced:

level=ERROR msg="local-api: backup job failed" vmid=9201 target=felhom-pbs
  err="backup: vzdump task vmid 9201: proxmox: waiting for task UPID:…:vzdump:9201:… : context canceled"

The backup had not failed. context canceled is the agent being shut down while waiting, not the task failing. Checked rather than assumed, against two independent authorities:

• the PVE task 2026-07-28T21:19:45 … vzdump 9201 → OK;
• the PBS snapshot felhom-pbs:backup/ct/9201/2026-07-28T19:19:45Z → 6,264,034,053 B, a real, plausibly-complete archive (nine orders of magnitude above the F-CRIT-2 1 MiB floor).

So the run recorded a spurious tier failure — a self-inflicted instance of exactly the F-A1 class ("a healthy backup reported as FAILED") that the project fixed in controller v0.179.0.

It reached no channel, and the reason is worth keeping. The controller's event trail for the whole window is:

19:15:43Z  whole_guest_backup_failed     (error)  felhom-backup tier — the REAL 403, backoff 15m
19:19:45Z  whole_guest_backup_recovered  (info)   felhom-backup tier — cleared after the ACL grant

— and nothing for the 21:21 PBS failure. The agent process died with its own in-memory failure record, and by the time the controller next polled, NewestArchiveTime had read ground truth from PBS and found the successful 21:19:45 archive. The R-84 ground-truth design absorbed it: no breaker armed on the offsite tier, no operator mail, no spurious staleness. That is the design working, not a reason the mistake was harmless — on a slower tier, or with the restart a minute earlier, the same slip could have aborted a multi-hour WAN upload.

The rule that would have caught it: re-check in flight immediately before every restart, not once at the start of the procedure — and remember that a manual trigger fires all tiers, so the offsite tier can still be running long after the local one has finished.

8.2 Method and tooling

• pgrep -f vzdump is not a safe in-flight check. A polling script whose own command line contains the string matches itself, and it reported a backup in flight when the PVE task list showed none. It caused one false "ABORT: vzdump in flight" and one wrong reading in this run. Use the PVE task list (pvesh get /nodes/<node>/tasks, filter for a missing endtime).
• sed -i must not be used on agent.json. The directory is root-owned and the file is agent-owned 0600; sed -i renames, which would flip the file to root:root and leave the non-root agent unable to read its own config. The in-place truncate-write in §3 preserves both.
• A felhom-pbs vzdump failed at 17:58, hours before this change, with Can't connect to 10.77.0.1:8007 (Connection refused). Pre-existing and unrelated — checked rather than assumed: ep0 is up (uptime 1 d 2 h, proxmox-backup-proxy active), the WG handshake is fresh, and 8007 is open from demo-felhom now.
• The controller's first-backup safety valve behaved exactly as designed on demo-felhom: the agent restart made the new tier report age_state: absent, and the controller took a backup on its own before the manual trigger could ({"error":"mentés már folyamatban van"}).
• The archives are outside the customer's reach by construction — the guest is bound only the drive's felhom-data subpath, while dump/ sits at the drive root.

---

9. State at close

9.1 Old archives — left in place, as instructed

Nothing was deleted. They remain the rollback and the only evidence of what the previous configuration produced.

Box	Path	Contents	Size
demo-felhom	/var/lib/vz/dump/	3 × 9201 (5.82 / 5.84 / 5.93 GB) + 2 × 9100 (612 / 649 MB)	18 GB
demo-hp	/var/lib/vz/dump/	3 × 9201 (1.61 / 1.63 / 1.68 GB) + 1 × 9100 (649 MB)	5.2 GB

These are now stale and will never be refreshed — the tier that wrote them no longer points here, and keep-last=3 prunes only the new target. They are also the reason R-109 matters: a restorer reading the DR recipe sees both local and felhom-backup carrying content=backup and cannot tell that one holds live archives and the other holds a frozen snapshot of 2026-07-28. Deleting them is a separate, deliberate decision.

9.2 Fleet state — both boxes healthy, no thrash, no spurious staleness

	demo-felhom	demo-hp
Agent	active, v0.110.0	active, v0.110.0
Primary tier	felhom-backup, due=false, age_state=known	felhom-backup, due=false, age_state=known
Offsite tier	felhom-pbs, due=false, age_state=known	felhom-pbs, due=false, age_state=known
New target	5.96 GB used of 916 GB (2 %)	2.26 GB used of 938 GB (1 %)
Target drive SMART	PASSED, 35 °C	PASSED, 50 °C, 7 % wear
System drive	24 G / 94 G (27 %)	14 G / 39 G (38 %)
Agent errors, last 15 min	0	2, both accounted for (§4 403, §8.1 spurious)
Breaker	clear (whole_guest_backup_recovered 19:19:45Z)	clear

No tier is stale, no tier is thrashing, and both offsite tiers report a known, recent age.

9.3 Outstanding proof

1. Full drive-loss recovery — pull the system drive, restore the guest from the vzdump on the secondary drive, confirm it returns with named volumes, secrets and config. Needs physical access; the boxes are remote until ~08-02. This is the proof that flips matrix row 4 from PARTIAL to a real local route. Everything up to it is now proven: the archive exists on separate hardware, and a restore from it boots and passes mount_parity.
2. The agent half of §6 — with the drive absent, that the tier fails loudly rather than silently retargeting. The PVE half is live-proven (§6); the agent half is traced through source only, because unmounting a live enrolled drive would break the guest's felhom-data bind and the customer's app data on a remote box.

9.4 Backlog rows filed

ID	What
E-2	The machinery around this change — backup-target role on StoragePath, wizard assignment, no automatic roles, stickiness, felhom-host-install.sh creating the target and its ACL, absent-target policy, retention/space accounting on a shared drive, the honest single-drive label, remaining fleet migration. Full scope in §7.
R-109	The DR recipe records no backup target. BuildDRRecipeHostHalf lists every storage's name/type/content but never says which one holds the local archives — and after this change the box carries two content=backup dir storages, one live and one stale. §5.4.
R-105	Partially closed and fully traced for its third field: dr_recipe.host_half.drives is now populated on both demo boxes, and the cause of the fleet-wide [] is identified — the enrolled drives were never PVE storages, so isUserDataDrive never saw them. The other two fields (hosts.dr_record_json, host_escrow.directive_json) are untouched by this run.