R-106 + R-109: the DR recipe records the resolved namespace and names the backup target (v0.118.0)

Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.

R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.

R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.

Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.

Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.

9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
This commit is contained in:
2026-07-30 13:11:08 +02:00
parent 1913e12031
commit 1c8a67eece
13 changed files with 670 additions and 103 deletions
+73
View File
@@ -1,3 +1,76 @@
## v0.118.0 — R-106 + R-109: the DR recipe stops guessing (2026-07-30)
**The recipe is read at the worst possible moment — by an operator rebuilding a machine that is gone —
and it was wrong about the PBS namespace and silent about the backup target.** Both were confirmed live
on both demo boxes before the fix, in the recipe the hub actually serves:
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
(no backup_target field at all)
while `/etc/pve/storage.cfg` on those same boxes reads `namespace demo-felhom` / `namespace demo-hp`.
**R-106 — the namespace.** `DRPBSCoord.Namespace` was taken from the LISTED SNAPSHOT. PBS does not echo
`ns` per item once the request is already namespace-scoped via `?ns=` (`internal/pbs/client.go:118-120`),
so `Snapshot.Namespace` was always empty, `ToHub` normalised empty → `"root"` (`internal/pbs/report.go:22-25`),
and `latestPBSCoord` wrote that into the recipe. Every per-customer box therefore reported the root
namespace while its backups lived in a tenant one. It now resolves from the pbs STORAGE — `PBSNamespace`,
carried on `StorageTarget` from storage.cfg's `namespace` — which is the same field `vzdump --storage <pbs>`
makes PVE read, so the recipe cannot disagree with the backup that produced the snapshot. One state, one
owner. An unconfigured namespace still reads `"root"`: that is an ANSWER (the box is genuinely in the root
namespace), and `namespace_state: resolved` distinguishes it from not knowing.
**R-109 — the backup target.** The recipe listed every storage's name/type/content and never said which
one holds the local whole-guest archives. Harmless while that was the well-known `local`; not harmless
since the 2026-07-28 vzdump-target move, after which every box carries TWO `content=backup` dir storages —
`felhom-backup` (live) and `local` (archives frozen at the move) — indistinguishable by name, type and
content. Picking the frozen one restores a guest that restores cleanly and is silently months stale. The
host-half now carries `backup_target`, resolved from the PRIMARY tier of `cfg.Backup.BackupTiers()` — the
same function the scheduler consults, not a re-derivation — plus the mountpoint, which is what actually
separates `/mnt/hdd_1` from `/var/lib/vz`.
**Absence is recorded as absence.** Neither field emits a default, an empty string, or a placeholder when
it cannot be resolved. `backup_target` has three outcomes and the two unknowns are deliberately distinct:
`agent_backup_config_unavailable` (the collector has no config seam) and `not_a_known_storage` (the config
names a storage this host does not have — the id IS still recorded, because that sends an operator
somewhere useful while silence does not). `namespace_state` likewise refuses to default to root when there
is no storage row to read.
**The resolver reports the tier IN EFFECT, not the file.** `SetBackupTargetResolver` closes over the
daemon-start config on purpose. A backup-target move rewrites `agent.json` and deliberately does NOT restart
the agent (the E-1 lesson: restarting mid-backup records a spurious failure for a run that succeeded), so
between the write and the restart the file names a target no backup is writing to yet. The live-reload shape
used for `escrow.pbs_storage_id` would have made the recipe point at the new storage while every archive
still landed on the old one.
**A cross-repo half of this was load-bearing and is easy to miss:** the hub's `AssembleDRRecipe` allow-lists
top-level keys, so `backup_target` would have been *stored intact and silently dropped* before reaching any
operator. Shipped together with hub v0.83.0; neither half is useful alone.
Not touched: the backup machinery. This corrects the RECORD, not the doing — and the record was never
acted on programmatically (the host_loss plan is surface-only, `internal/dr/plan.go:125`), so the blast
radius was exactly one operator reading a wrong value. `plan.PBS` inherits the fix for free.
- `internal/hub/report.go``StorageTarget.PBSNamespace` (`json:"-"`, the ConfigPath precedent: the struct
is a cross-repo contract and nothing off-box needs this value).
- `internal/storage/observe.go` — populate it from `proxmox.Storage.Namespace`.
- `internal/hub/dr_recipe.go``DRBackupTarget`, `ConfiguredBackupTarget`, `DRState*`/`DRReason*`,
`namespace_state`/`namespace_reason`, `resolveBackupTarget`, namespace from the storage.
- `internal/hub/collect.go``SetBackupTargetResolver` seam; an unwired seam reports unknown, never a guess.
- `cmd/felhom-agent/main.go``primaryBackupTargetOf` wired at BOTH collector sites (the daemon and
`selftest=hub`, so "the report it would send" really matches what the daemon sends).
- `testdata/host-report.golden.json` — new keys; kept byte-identical with the hub's copy (`f4bc3554…`).
Tests: 9 new. The consequence ones assert a box with two `content=backup` storages names the LIVE one and
does not name the frozen one — `assertBackupCandidateAmbiguity` fails the test if the fixture ever stops
posing that problem, so it cannot pass hollow. Fixtures are the storage set demo-felhom really had
(provenance recorded in the file: the box's own pre-fix recipe + its `storage.cfg`), not composed structs.
Two run the REAL path, `Collector.Collect()`, one of them specifically pinning that an unwired seam yields
unknown — a seam built and never wired is the failure mode this repo has hit four times. Red-proofs: 4,
each mutation asserted to have landed before running (revert R-106 → the live `"root"` symptom reproduces
in both the unit and production-path tests; make the unknown case guess → 5 assertions fire; drop the
field → the cross-repo contract guard fires too; hollow the fixture → the ambiguity guard fires).
Suite rc=0, 29 packages, 0 FAIL.
## v0.117.0 — R-117: the liveness signal now tests liveness (2026-07-30)
**`BoundUnderParent` reported a namespace that returned `EIO` on every read and write as healthy, and the