admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

R-106 + R-109: the DR recipe records the resolved namespace and names the backup target (v0.118.0)

Browse Source 
Both defects were live on both demo boxes: the recipe said namespace "root" while
storage.cfg said demo-felhom/demo-hp, and it never named which of two content=backup
dir storages holds the local archives.

R-106: the namespace came from the listed snapshot, but PBS omits `ns` per item once
the list is namespace-scoped, so it was always empty and normalised to "root". It now
resolves from the pbs STORAGE (storage.cfg's `namespace`) — the same field vzdump makes
PVE read, so the recipe cannot disagree with the backup.

R-109: backup_target resolves from the primary tier of cfg.Backup.BackupTiers(), the
function the scheduler consults, and carries the mountpoint that separates /mnt/hdd_1
from /var/lib/vz. The resolver reports the tier IN EFFECT (daemon-start config), not
agent.json on disk — a target move rewrites the file and deliberately does not restart.

Unresolvable is recorded as unresolvable: resolved|unknown plus a distinct reason,
never a default, an empty string, or a placeholder.

Needs hub v0.83.0 — AssembleDRRecipe allow-lists top-level keys, so backup_target
would otherwise be stored intact and dropped before any operator saw it.

9 tests, 4 red-proofs (each mutation asserted to have landed). Suite rc=0, 29 ok.
This commit is contained in:
authentik Default Admin
2026-07-30 13:11:08 +02:00
parent 1913e12031
commit 1c8a67eece
13 changed files with 670 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/hub/dr_recipe.go
+138 -10
View File
@@ -30,6 +30,36 @@ import "sort"
// ignore-unknown (encoding/json default) for forward-compat, mirroring storage_manifest.
const DRRecipeVersion = 1
// Recipe field states (R-106/R-109). A recipe is read at the worst possible moment — by an operator
// rebuilding a machine that is gone — so a field the agent cannot resolve must SAY SO rather than emit
// a default, an empty string, or a plausible-looking placeholder. A guess read as fact costs more than
// an admitted gap: it sends the restore at the wrong archive and nothing contradicts it. This is the
// same cannot-tell-must-not-lie rule R-117 needed a third state for.
const (
DRStateResolved = "resolved"
DRStateUnknown = "unknown"
)
// Reasons a resolved-value field is unknown. Enum-shaped, never free text, so the wire stays pinnable
// and TestDRRecipeHostHalf_NoSecrets has a fixed vocabulary to walk.
const (
// DRReasonNoBackupConfig: the collector was built without a backup-config seam, so the agent could
// not consult the very config its own scheduler reads. Nothing is guessed.
DRReasonNoBackupConfig = "agent_backup_config_unavailable"
// DRReasonNoSuchStorage: the configured target id matches no storage this host observes. The id is
// still recorded (it IS what the config says) and the state says it could not be corroborated.
DRReasonNoSuchStorage = "not_a_known_storage"
// DRReasonNoPBSStorage: snapshots exist but no pbs storage was observed, so there is no storage.cfg
// row to read the namespace from.
DRReasonNoPBSStorage = "no_pbs_storage_observed"
)
// PBSRootNamespace is how the recipe spells PBS's root namespace. The PBS API spells it as the EMPTY
// string (and `pct restore --ns root` would name a namespace that does not exist) — "root" is a display
// convention this wire has always used, kept here so the field's meaning did not change under R-106.
// Only a box with no `namespace` line in its pbs storage.cfg stanza ever emits it.
const PBSRootNamespace = "root"
// DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely
// from facts the report already collects — no new privileged reads.
type DRRecipeHostHalf struct {
@@ -38,6 +68,45 @@ type DRRecipeHostHalf struct {
PBS *DRPBSCoord `json:"pbs,omitempty"`
Drives []DRDrive `json:"drives"`
PVEStorage []DRPVEStorage `json:"pve_storage"`
// BackupTarget names WHICH storage holds the local whole-guest archives (R-109). Always present —
// its own State field carries "I could not tell", so the section is never simply absent.
BackupTarget *DRBackupTarget `json:"backup_target"`
}
// DRBackupTarget answers the one question pve_storage cannot: of every storage listed there, WHICH one
// does this box's primary backup tier actually write its whole-guest archives to?
//
// Before R-109 the recipe listed each storage's name/type/content and said nothing about the target.
// That was harmless while the target was the well-known `local`; the 2026-07-28 vzdump-target move
// ended that. Every box now carries TWO content=backup dir storages — `felhom-backup` (live) and
// `local` (archives frozen at the move, never refreshed since) — and they are indistinguishable by
// name, type and content alone. A restorer picking the frozen one gets a guest that restores cleanly
// and is silently months out of date, which is the worst shape a backup defect can take.
type DRBackupTarget struct {
// State is DRStateResolved | DRStateUnknown. A reader MUST consult it before trusting StorageID:
// the id is also recorded in one unknown case (see DRReasonNoSuchStorage).
State string `json:"state"`
// StorageID is the PVE storage id of the PRIMARY backup tier. Empty only when the config could not
// be consulted at all.
StorageID string `json:"storage_id,omitempty"`
// MountPath is where that storage's archives land on the host — the disambiguation a restorer
// actually needs, since it is what separates felhom-backup's /mnt/hdd_1 from local's /var/lib/vz.
// "" for a pbs target (no host mount) and when unresolved.
MountPath string `json:"mount_path,omitempty"`
// Reason is why State is unknown (one of the DRReason* constants); "" when resolved.
Reason string `json:"reason,omitempty"`
}
// ConfiguredBackupTarget is what the agent's own backup config says the PRIMARY tier writes to.
//
// Known=false is a REAL state, not a nil-guard: it means the collector was constructed without the
// backup-config seam (the --selftest one-shots did exactly this before v0.118.0), and the recipe then
// records unknown instead of inventing a target. Deliberately a struct rather than a `(string, bool)`
// return — the (value, ok) shape is what made "errors degrade to unknown, never to no-backup"
// unimplementable in newestArchiveOn, and this field has the same three-way reading.
type ConfiguredBackupTarget struct {
StorageID string
Known bool
}
// DRGuest is the sizing needed to recreate the LXC at the right size (GuestSpec, already on the wire).
@@ -51,8 +120,21 @@ type DRGuest struct {
// DRPBSCoord is WHERE the whole-CT snapshot lives — COORDINATES ONLY. The encryption key is escrow-only;
// the access token is identity-escrow-only. Neither is here.
type DRPBSCoord struct {
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
Namespace string `json:"namespace"` // PBS namespace the restore targets
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
// Namespace is the PBS namespace the restore targets, resolved from the pbs storage's storage.cfg
// stanza — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree
// with the backup that produced the snapshot. PBSRootNamespace when the box has no namespace
// configured; "" when NamespaceState is unknown.
//
// R-106: this used to come from the listed snapshot's own `ns`, which PBS does not echo per item once
// the request is already namespace-scoped via `?ns=` (internal/pbs/client.go). The field was
// therefore always empty, ToHub normalised empty → "root", and every per-customer box reported the
// root namespace while its backups were really in `demo-hp` / `demo-felhom`.
Namespace string `json:"namespace"`
// NamespaceState is DRStateResolved | DRStateUnknown — consult it before trusting Namespace.
NamespaceState string `json:"namespace_state"`
// NamespaceReason is why NamespaceState is unknown; "" when resolved.
NamespaceReason string `json:"namespace_reason,omitempty"`
LatestSnapshotID string `json:"latest_snapshot_id"` // most-recent snapshot's backup_id (a coordinate)
}
@@ -83,7 +165,7 @@ const driveIntentEnrolled = "enrolled"
// it is unit-tested directly (no live reads). drives[] = the user-data external drives (usb/local-dir
// with a durable-id); pve_storage[] = every storage target (the storage.cfg scaffolding); pbs = the
// latest PBS snapshot's coordinates; guests[] = each guest's sizing (skip guests with no spec).
func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSnapshot) *DRRecipeHostHalf {
func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSnapshot, backupTarget ConfiguredBackupTarget) *DRRecipeHostHalf {
h := &DRRecipeHostHalf{
RecipeVersion: DRRecipeVersion,
Guests: []DRGuest{},
@@ -103,11 +185,14 @@ func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSna
})
}
var pbsRepoID string
var pbsRepoID, pbsNamespace string
var pbsStorageFound bool
for _, t := range targets {
h.PVEStorage = append(h.PVEStorage, DRPVEStorage{Name: t.Name, Type: t.Type, Content: t.Content})
if t.Type == StorageTypePBS && pbsRepoID == "" {
pbsRepoID = t.Name // the pbs storage id is a coordinate, not the key
if t.Type == StorageTypePBS && !pbsStorageFound {
pbsStorageFound = true
pbsRepoID = t.Name // the pbs storage id is a coordinate, not the key
pbsNamespace = t.PBSNamespace // storage.cfg's namespace — "" here means the ROOT namespace
}
if isUserDataDrive(t) {
h.Drives = append(h.Drives, DRDrive{
@@ -119,12 +204,41 @@ func BuildDRRecipeHostHalf(guests []Guest, targets []StorageTarget, pbs []PBSSna
}
}
if c := latestPBSCoord(pbs, pbsRepoID); c != nil {
h.BackupTarget = resolveBackupTarget(targets, backupTarget)
if c := latestPBSCoord(pbs, pbsRepoID, pbsNamespace, pbsStorageFound); c != nil {
h.PBS = c
}
return h
}
// resolveBackupTarget records WHICH storage the primary backup tier writes to (R-109), or records
// explicitly that it could not tell. Three outcomes, and the two unknowns are deliberately distinct —
// "I could not read my own config" and "my config names a storage that is not here" send an operator
// to different places.
//
// MountPath prefers the live mount and falls back to the CONFIGURED path: during a rebuild the drive is
// frequently absent, and when it is, MountPath empties out while ConfigPath is the only thing left that
// still says which drive the row was about (the R-116 lesson). The storage's absence from the host is a
// separate signal (E-2's backup_target_absent); it does not make the recipe's answer unknown, because
// the question here is which storage.cfg row to restore FROM, and that is still known.
func resolveBackupTarget(targets []StorageTarget, cfg ConfiguredBackupTarget) *DRBackupTarget {
if !cfg.Known || cfg.StorageID == "" {
return &DRBackupTarget{State: DRStateUnknown, Reason: DRReasonNoBackupConfig}
}
for _, t := range targets {
if t.Name != cfg.StorageID {
continue
}
mount := t.MountPath
if mount == "" {
mount = t.ConfigPath
}
return &DRBackupTarget{State: DRStateResolved, StorageID: cfg.StorageID, MountPath: mount}
}
return &DRBackupTarget{State: DRStateUnknown, StorageID: cfg.StorageID, Reason: DRReasonNoSuchStorage}
}
// isUserDataDrive selects the external user-data drives the recipe enumerates (felhom-usb / felhom-flash
// class): a usb or local-dir storage with a filesystem-UUID durable id and a host mount path. local /
// lvmthin / pbs / nfs / cifs are scaffolding (they land in pve_storage) but are not user-data drives.
@@ -137,16 +251,30 @@ func isUserDataDrive(t StorageTarget) bool {
// latestPBSCoord picks the most-recent snapshot (lexical max of the RFC3339 backup_time) and returns
// its coordinates. Returns nil when there is no snapshot to target.
func latestPBSCoord(snaps []PBSSnapshot, repoID string) *DRPBSCoord {
//
// The namespace comes from the pbs STORAGE (storage.cfg), never from the snapshot — see DRPBSCoord's
// Namespace comment for why the snapshot's own field cannot answer it (R-106). storageFound=false with
// snapshots present is a genuine unknown: something listed snapshots, but there is no storage row to
// read a namespace from, so the recipe says so rather than defaulting to root.
func latestPBSCoord(snaps []PBSSnapshot, repoID, namespace string, storageFound bool) *DRPBSCoord {
if len(snaps) == 0 {
return nil
}
sorted := append([]PBSSnapshot(nil), snaps...)
sort.Slice(sorted, func(i, j int) bool { return sorted[i].BackupTime > sorted[j].BackupTime })
latest := sorted[0]
return &DRPBSCoord{
c := &DRPBSCoord{
RepoID: repoID,
Namespace: latest.Namespace,
LatestSnapshotID: latest.BackupID,
NamespaceState: DRStateUnknown,
NamespaceReason: DRReasonNoPBSStorage,
}
if storageFound {
c.NamespaceState, c.NamespaceReason = DRStateResolved, ""
// An empty configured namespace is not a missing answer — it IS the root namespace.
if c.Namespace = namespace; c.Namespace == "" {
c.Namespace = PBSRootNamespace
}
}
return c
}