Every client here pins TLS, so none can use http.DefaultTransport and each
hand-rolls its own. A composite literal takes IdleConnTimeout ZERO, which
means retain idle connections forever -- not "use a sane default".
pbsTargetsFromPVE builds a fresh pbs.Client every cycle and drops the
previous one, and an abandoned http.Transport does not close its
connections. One stranded socket per cycle, on both sides, forever.
Measured: 388 established connections on ep0 over 46 h, 194 per box, zero
closed in a 31-minute window. pvestatd and proxmox-backup-client made
162,404 requests in the same window and leaked none.
New leaf package internal/httpx owns the default (90s, http.DefaultTransport's
own value) and NewTransport, which returns a FRESH transport per call and
treats <=0 as "use the default", never "no timeout". pbs.Config gains
IdleConnTimeout for tests only.
hub and proxmox carried the same missing default and are corrected here for
consistency. Neither contributed to the ep0 leak -- both are built once per
process and neither talks to ep0:8007.
Tests count connections SERVER-side and model the abandonment, so they pin
the consequence rather than the field. Two red-proofs, both seen failing:
removing the timeout -> "still holds 5 open connection(s), want 0";
DisableKeepAlives -> "3 sequential requests over 3 connection(s), want 1"
(the leak test PASSES under that one -- it is the worse-fix guard that
catches it).
Not released: hand-installed on demo-hp only so demo-felhom stays the
control. CHANGELOG heading stays UNRELEASED until the publish is authorised.
Closes the agent half of R-39's fleet fix. Requires hub >=0.68.0 for the re-arm signal;
that hub is safe for 0.90.0 agents (unknown key dropped), so it deploys first.
Three compounding defects let a box report `applied` while every PBS request 401'd:
1. The re-key was INVISIBLE. An ep0 re-issue rotates the secret of an existing token, so
token_id/fingerprint/datastore/namespace come back byte-identical and the descriptor
content hash never moved — the converged agent short-circuited and never consumed the
fresh secret. WirePBSDR.SecretGeneration (field-exact with the hub) is what moves the
hash now, because descriptorHash marshals this struct.
2. The agent could not READ its own credential. It writes /etc/pve/priv/storage/<id>.pw
through the root wrapper, but that dir is 0700 root:www-data and the wrapper had no
read verb — so the target resolver got "permission denied" every cycle, warned, and
skipped. The one loop that could have caught the 401 was blind BY CONSTRUCTION. Adds a
narrow `read` verb (+ exactly one sudoers line, + a pbsdr-read capability row): one
secret to stdout, no network, no mutation, never in argv (sudo logs argv), traversal
refused by the id grammar, the dir allowlist AND a resolved-path prefix assertion.
3. Nothing probed AUTHENTICATION. pbs.ProbeAuth (GET /version + an ErrUnauthorized
sentinel) runs on the 15-minute collect path and its verdict becomes a loud
`auth_failed` the hub escalates to a fresh mint. /version needs no datastore, namespace
or privilege, so a 401 means the CREDENTIAL is bad; 403 is deliberately NOT treated as
unauthorized, since re-keying a too-narrow token would mint forever without fixing
anything. A transport error is UNKNOWN, never a rejection — otherwise every network
blip burns a credential. Recovery self-clears.
R-50b(a): the report now carries the installed wrapper's sha256 so drift against the
vouched manifest value is answerable. Empty = unknown, never drift.
Three red-proofs, all at the assertion level. Removing SecretGeneration fails the re-arm
test with "consume calls=1, want 2". Swallowing the probe result leaves State:applied
AuthFailed:false — the July-18 shape exactly. Notably, deleting the wrapper's id charset
guard alone does NOT open a traversal hole (readlink + the prefix assertion still catch
it), so the isolating red-proof removes BOTH and shows the out-of-tree secret printed —
the layering is real, and a single-guard red-proof would have passed vacuously.
Phase-1 live probe (felhom-hetzner) proved backup/restore/list/isolation over
the tunnel with a per-customer DatastoreBackup token, but the agent's PBS client
was namespace-unaware: Snapshots hit the datastore root (403 for a scoped token)
and Verify was whole-datastore (needs Datastore.Verify ~ admin). Operator-
approved fix.
- pbs.Config.Namespace + Client.namespace; Snapshots appends ?ns=; Verify sends
ns= (ns-scoped verify works with DatastoreBackup on the own ns — no admin
widening, Phase-1 confirmed). Root-ns clients unchanged (whole-datastore).
- proxmox.Storage.Namespace (parsed from /storage `namespace`).
- pbsTargetsFromPVE threads s.Namespace into the client.
Confirmed tenant ACL: DatastoreBackup on /datastore/felhom-offsite/<ns> (NOT
/ns/<ns>) to BOTH felhom@pbs (user) AND felhom@pbs!<ns> (token) — PBS privsep =
intersection; isolation holds (cross-ns 403 proven). TestClient_NamespaceScoping
red-proofed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live PBS runbook surfaced two gaps: (1) PBS verify defaults to ignore-verified=true and
SKIPS already-verified snapshots, so corruption after the first verify is never caught —
the agent's integrity check now POSTs ignore-verified=false to actually re-read+re-check.
(2) restore-test source_tier was hardcoded 'local'; now derived from the source storage
type ('pbs' for a PBS datastore). Adds a form-POST path to the PBS client.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin