admin
1c8a67eece
Both defects were live on both demo boxes: the recipe said namespace "root" while storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup dir storages holds the local archives. R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once the list is namespace-scoped, so it was always empty and normalised to "root". It now resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes PVE read, so the recipe cannot disagree with the backup. R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1 from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not agent.json on disk — a target move rewrites the file and deliberately does not restart. Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason, never a default, an empty string, or a placeholder. Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target would otherwise be stored intact and dropped before any operator saw it. 9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
77 lines
4.1 KiB
Markdown
77 lines
4.1 KiB
Markdown
# REPORT — R-106 + R-109: the DR recipe stops guessing (agent v0.118.0, 2026-07-30)
|
|
|
|
**Overwritten** per the standing rule. Paired with **hub v0.83.0** — neither half is useful alone, see §5.
|
|
|
|
## 1. What was wrong, confirmed live BEFORE the fix
|
|
|
|
Downloaded from the hub, `GET /customers/<id>/dr-recipe.json`, both demo boxes:
|
|
|
|
```json
|
|
"pbs": { "repo_id": "felhom-pbs", "namespace": "root", "latest_snapshot_id": "9201" }
|
|
```
|
|
|
|
and **no `backup_target` key at all**, while `pve_storage` offered two plausible candidates:
|
|
|
|
```json
|
|
{ "name": "felhom-backup", "type": "local-dir", "content": "backup" }
|
|
{ "name": "local", "type": "local", "content": "backup,import,vztmpl,iso" }
|
|
```
|
|
|
|
`/etc/pve/storage.cfg` on those same boxes says `namespace demo-felhom` and `namespace demo-hp`.
|
|
|
|
## 2. Root causes, at source
|
|
|
|
- **R-106** — `Snapshot.Namespace` decodes `ns` (`internal/pbs/client.go:97`), which PBS does not echo per
|
|
item once the list is namespace-scoped via `?ns=` (`:118-120`). Always empty → `ToHub` normalises empty
|
|
to `"root"` (`internal/pbs/report.go:22-25`) → `latestPBSCoord` writes it into the recipe.
|
|
- **R-109** — `BuildDRRecipeHostHalf` emitted `guests`/`drives`/`pve_storage`/`pbs` and nothing naming the
|
|
vzdump target. Benign until the 2026-07-28 target move gave every box a second `content=backup` dir
|
|
storage holding archives frozen at that date.
|
|
|
|
## 3. The fix
|
|
|
|
The namespace now resolves from the pbs **storage** (`StorageTarget.PBSNamespace`, from storage.cfg's
|
|
`namespace`) — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree with
|
|
the backup that produced the snapshot. `backup_target` resolves from the **primary tier of
|
|
`cfg.Backup.BackupTiers()`** — the function the scheduler itself consults — and carries the mountpoint,
|
|
which is what actually separates `/mnt/hdd_1` from `/var/lib/vz`.
|
|
|
|
**Unresolvable is recorded as unresolvable**, never defaulted: `namespace_state` and `backup_target.state`
|
|
are `resolved | unknown`, with distinct reasons (`agent_backup_config_unavailable`, `not_a_known_storage`,
|
|
`no_pbs_storage_observed`). An unconfigured namespace still reads `"root"` — that is an answer, not a gap.
|
|
|
|
**The resolver reports the tier IN EFFECT, not the file on disk.** A backup-target move rewrites
|
|
`agent.json` and deliberately does not restart the agent (the E-1 lesson), so a disk re-read would name a
|
|
storage no archive had reached yet.
|
|
|
|
## 4. Files
|
|
|
|
`internal/hub/report.go`, `internal/storage/observe.go`, `internal/hub/dr_recipe.go`,
|
|
`internal/hub/collect.go`, `cmd/felhom-agent/main.go`, `internal/hub/testdata/host-report.golden.json`,
|
|
`REUSE.md`; tests in `internal/hub/{dr_recipe,contract,report}_test.go`, `internal/pbs/live_reporter_test.go`.
|
|
|
|
## 5. The cross-repo half that is easy to miss
|
|
|
|
The hub's `AssembleDRRecipe` **allow-lists** top-level keys. `backup_target` would have been stored intact
|
|
and dropped before any operator saw it. The same mechanism already cost `offsite_restic` its whole
|
|
existence (**R-122**, found this session). `REUSE.md` now records this as a two-repo change.
|
|
|
|
## 6. Tests
|
|
|
|
9 new. Consequence-level: a box with two `content=backup` storages names the **live** one and not the
|
|
frozen one; `assertBackupCandidateAmbiguity` fails if the fixture ever stops posing that problem. Fixtures
|
|
are the storage set demo-felhom really had, provenance recorded in-file. Two tests run the real
|
|
`Collector.Collect()` path, one pinning that an **unwired seam yields unknown**.
|
|
|
|
**4 red-proofs**, each mutation asserted to have landed before running: revert R-106 → the live `"root"`
|
|
symptom reproduces; make unknown guess a default → 5 assertions fire; drop the field → the cross-repo
|
|
contract guard fires; hollow the fixture → the ambiguity guard fires.
|
|
|
|
Suite: **rc=0, 29 packages ok, 0 FAIL** (run separately from the commit).
|
|
|
|
## 7. Not touched
|
|
|
|
The backup machinery. This corrects the record, not the doing — and the record was never acted on
|
|
programmatically: the `host_loss` plan is surface-only (`internal/dr/plan.go:125`), so the blast radius was
|
|
exactly one operator reading a wrong value. `plan.PBS` inherits the fix for free.
|