Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.
Three compounding defects let a box report `applied` while every PBS request 401'd:
1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
content hash never moved — the converged agent short-circuited and never consumed the
fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
hash now, because descriptorHash marshals this struct.
2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
read verb — so the target resolver got "permission denied" every cycle, warned, and
skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.
3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
sentinel) runs on the 15-minute collect path and its verdict becomes a loud
`auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
unauthorized, since re-keying a too-narrow token would mint forever without fixing
anything. A transport error is UNKNOWN, never a rejection — otherwise every network
blip burns a credential. Recovery self-clears.
R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.
Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
Phase-1 live probe (felhom-hetzner) proved backup/restore/list/isolation over
the tunnel with a per-customer DatastoreBackup token, but the agent's PBS client
was namespace-unaware: Snapshots hit the datastore root (403 for a scoped token)
and Verify was whole-datastore (needs Datastore.Verify ~ admin). Operator-
approved fix.
- pbs.Config.Namespace + Client.namespace; Snapshots appends ?ns=; Verify sends
ns= (ns-scoped verify works with DatastoreBackup on the own ns — no admin
widening, Phase-1 confirmed). Root-ns clients unchanged (whole-datastore).
- proxmox.Storage.Namespace (parsed from /storage `namespace`).
- pbsTargetsFromPVE threads s.Namespace into the client.
Confirmed tenant ACL: DatastoreBackup on /datastore/felhom-offsite/<ns> (NOT
/ns/<ns>) to BOTH felhom@pbs (user) AND felhom@pbs!<ns> (token) — PBS privsep =
intersection; isolation holds (cross-ns 403 proven). TestClient_NamespaceScoping
red-proofed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live PBS runbook surfaced two gaps: (1) PBS verify defaults to ignore-verified=true and
SKIPS already-verified snapshots, so corruption after the first verify is never caught —
the agent's integrity check now POSTs ignore-verified=false to actually re-read+re-check.
(2) restore-test source_tier was hardcoded 'local'; now derived from the source storage
type ('pbs' for a PBS datastore). Adds a form-POST path to the PBS client.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin