felhom.eu/REPORT.md

REPORT — the local whole-guest backup moved off the guest's own device (2026-07-28)

Overwritten per the standing rule (the prior C9-F1/C9-F2 text stays in git history). Supervised operational run — no code, no version bump. Fleet unchanged: hub v0.80.0, agent v0.110.0, controller v0.183.0. peti-felhom untouched. Nothing was deleted.

Record: documentation/runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md.

---

What changed

On both demo boxes the primary backup tier moved from local — a dir storage on /var/lib/vz, i.e. the same physical device as the guest itself — to felhom-backup, a dir storage on the secondary drive.

	demo-felhom	demo-hp
Target drive	/dev/sdb USB HDD → /mnt/hdd_1	/dev/nvme0n1 → /mnt/nvme-1tb
Durable id	uuid:47a3361a-91e0-4831-a69d-27f540ed3f48	uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
Archive proven	5,957,878,962 B	2,256,044,492 B
Restore-test	pass, mount_parity: ok, 1 m 24 s	pass, mount_parity: ok, 1 m 54 s

Three steps per box: pvesm add dir … --is_mountpoint 1, two pveum acl modify, and a one-line local_backup_target edit in agent.json.

Drive loss is now locally recoverable in principle on both boxes — the whole-guest archive lives on different hardware from the guest, and a restore from it boots and passes mount parity. The proof that closes matrix row 4 (pull the system drive and restore for real) needs physical access and is outstanding.

Phase 0 verdict

The target is configuration, not converged — the runbook held and the §3 STOP did not fire. Exactly one writer of agent.json exists (pbsdr.seedEscrowStorageID); it touches only escrow.pbs_storage_id and preserves unknown keys verbatim via a map[string]json.RawMessage read-modify-write. Nothing on the hub pushes agent config. No consumer hardcodes "local" on the backup path — runners are one-per-tier, NewestArchiveTime reads its own target, and restoreTierForArchive classifies from the archive, not from config.

What the run found

• F-1 (before any command). The storage path must be the drive's mountpoint. A subdirectory fails exactMount (internal/storage/observe.go:321), so the target would report disconnected permanently and its durable id would degrade off the filesystem UUID.
• F-2 (before any command). --is_mountpoint 1 is load-bearing. Proven live: an unguarded dir storage on a non-mounted path reports active, advertises the root filesystem's free space, and had already created dump/ on /dev/mapper/pve-root — a silent retarget onto the exact device this change exists to escape. The guarded one refuses outright.
• F-3 (found by the first real backup). FelhomAgentStore is granted per storage path; a new target without its own grant 403s every backup. felhom-host-install.sh must issue it for new installs, or a new box ships with a tier that fails on its first run.
• R-109 (new). The DR recipe records no backup target — and each box now carries two content=backup dir storages, one live and one holding frozen 2026-07-28 archives.
• R-105 narrowed and traced. dr_recipe.host_half.drives was [] fleet-wide with the cause untraced. Cause: the enrolled drives were never PVE storages, so isUserDataDrive never saw them. Both demo boxes now populate drives, and the backup drives gained SMART reporting. R-105's other two fields are untouched.

Absent-drive behaviour (Part 3)

Today: fail loudly, no silent retarget. The PVE half is live-proven with throwaway storages (no live drive was unmounted). The agent half is source-traced: targetStoragePresent checks name presence only, never Reachable, so the tier stays DUE, the controller quiesces, vzdump is refused, and the run fails and alarms.

This is not §6's intended design (fall back to the system drive and alarm). There is no fallback at all, so an absent drive means no local backup until a human intervenes. Filed as E-2, together with the honest single-drive label — a one-drive box protects against corruption only, and two drives is effectively a hardware requirement for drive-loss protection.

One operational error, reported in full

The agent was restarted on demo-hp with a felhom-pbs backup in flight, against §4.3. The in-flight check was done before the first restart and not repeated before the second. It produced a spurious tier failure (context canceled while waiting) — the F-A1 class the project already fixed once.

The backup had not failed: the PVE task returned OK and the PBS snapshot 2026-07-28T19:19:45Z is 6,264,034,053 B. The spurious failure reached no channel — the agent died with its in-memory record and R-84 ground truth superseded it; the controller's event trail for the window shows only the real 403 failure and its recovery. The design absorbed it; the mistake was still a mistake, and on a slower tier the same slip could have aborted a multi-hour WAN upload.

Also recorded: pgrep -f vzdump self-matches a polling script's own command line and is not a safe in-flight check — use the PVE task list.

State at close

Both boxes healthy. Primary and offsite tiers due=false with age_state=known on both; breakers clear; agents active; no thrash and no spurious staleness. Target drives at 1–2 % used with SMART PASSED. The old archives (18 GB demo-felhom, 5.2 GB demo-hp) are left in place as the rollback and as the only evidence of what the previous configuration produced.

Outstanding: full drive-loss recovery (physical access), and the agent half of the absent-drive behaviour (needs a drive unmount that would break the guest bind on a remote box).